Change default scheduler for qthreads and build with --enable-condwait-queue

After a lot of investigation into qthreads performance regressions we found two
main causes. The first is that qthreads default scheduler (sherwood) has
significant overhead. Sherwood is a work stealing scheduler with no real way to
turn off work stealing (we tried, but it relies on work stealing to divvy up
tasks so with work stealing off you only use part of a machine.) Work stealing
and some other factors add too much overhead. As a result we're switching the
default to be nemesis for all configurations except for numa. For numa we rely
on the semantics that the sherwood scheduler provides (essentially 1 shepherd =
1 numa domain = 1 sublocale.)

The second source of overhead was for higher core count machines only and is
because qthreads workers that are idle busy wait for work. This can cause
interference for other workers, and in particular really hurt our single
threaded and low task count applications. By building with
--enable-condwait-queue we avoid that penalty and idle workers call into the os
to be swapped rather than busy waiting.

With these two changes the only remaining performance regressions are for tests
that perform better with hyperthreading since we don't use hyperthreads by
default for qthreads but we do for fifo.

This also pulls --disable-spawn-cache out of the main config line into it's own
line so I could add a comment as to why we set it since I always forget. See
commit 2df9f3006a0bea3fca477c71aaacb1b626855eed (svn r19695) for original commit
of --disable-spawn-cache.

Note that this configuration has already been tested for the machines doing
qthreads performance runs and this is just the global change.
Change qthreads topology/affinity setup to work with nemesis

qthreads seems to setup the number of workers and workers per shepherd assuming
the shwerwood scheduler is being used when configured with hwloc. This changes
some of the logic to work with the nemesis scheduler. I have the patch out to
Dylan as well and will make changes that he suggests. I may go with this for a
little while if I don't hear back from him just to get some testing in.
Remove special qthreads setup for nemesis scheduler

Code was temporarily put into our qthreads setup to allow us to test out the
performance of the nemesis scheduler. Now that we're going to nemesis by
default this logic has been moved into the qthreads source (as a bug fix) and
is no longer needed in our qthreds setup. This reverts the tasks-qthreads.c
changes in commit 095f933f71792f7655a6b223aa2d6dc26e8d777c
Get rid of special setup for chap03/04 qthreads testing

Simply removes env var setup that was required to test either the nemesis
scheduler or modifications to the sherwood scheduler. The scheduler,
enable-condwait-queue, and shepherds/workers per shepherd all just get set
correctly without these env vars now.
Merge pull request #144 from ronawho/qthreads-default-scheduler

Use nemesis scheduler for qthreads and build with --enable-condwait-queue

For performance reasons we want to use the nemesis scheduler over the default
sherwood scheduler for qthreads. For single threaded and low task count
applications --enable-condwait-queue is needed for good performance. This
changes the default scheduler, builds with --enable-condwait-queue and removes
special code in our qthreads interface and in our qthreads performance testing
scripts that was used to test out this configuration without making it the
default for all other builds.

The other change was in the qthreads source itself to correctly set the number
of shepherds and workers per shepherd for single threaded schedulers (like
nemesis.) This is currently out for review with the qthreads team, but we want
to start testing now.

Note that there may be some deadlocks in testing temporarily. For a few tests
like domains/sungeun/assoc/stress.chpl we create tasks that increment a sync
var while the main task waits on the sync var to be a certain value. Those
tasks that are incrementing don't get to run because the main task never
yields. A fix for that is being worked on, but we want to start getting
testing going so we can switch to qthreads as our default tasking layer soon.
Remove ability to turn off qthreads work stealing (for good this time)

The ability to turn off work stealing has been added and removed twice now.
Both times to try out the performance of the sherwood scheduler with work
stealing off. The first time it was turned off unconditionally and the second
only if an env var was set.

We are going to use the nemesis scheduler by default and only use sherwood for
numa now. I'm removing this not because we're using nemesis by default as it
might have been useful for numa but because it doesn't do what we expect it to.
It seems sherwood is highly intertwined with work stealing. I believe that when
you create tasks it places them all on a local shepherd and relies on work
stealing to distribute work (where as nemesis or something deals out work to
shepherds in a round robin fashion.) So turning off work stealing the way we
were didn't have the desired behavior.
Merge pull request #145 from ronawho/qthreads-remove-no-work-stealing

Remove ability to turn off qthreads work stealing (for good this time)

[trivial, not reviewed] 

The ability to turn off work stealing has been added and removed twice now.
Both times to try out the performance of the sherwood scheduler with work
stealing off. The first time it was turned off unconditionally and the second
only if an env var was set.

We are going to use the nemesis scheduler by default and only use sherwood for
numa now. I'm removing this not because we're using nemesis by default as it
might have been useful for numa but because it doesn't do what we expect it to.
It seems sherwood is highly intertwined with work stealing. I believe that when
you create tasks it places them all on a local shepherd and relies on work
stealing to distribute work (where as nemesis or something deals out work to
shepherds in a round robin fashion.) So turning off work stealing the way we
were didn't have the desired behavior.
Add missing generated variables to runtime/include/chplcgfns.h

runtime/include/chplcgfns.h is home to extern variables and function
declarations that are defined in the generated code. It hasn't been updated in
a while so I added all of the current ones and added a note in codegen that new
values should be placed in this file.

I ran into this when I wanted to check CHPL_STACK_CHECKS. I didn't originally
realize this file existed and while adding CHPL_STACK_CHECKS I figured I'd
update all of them.
Merge pull request #147 from ronawho/update-externs-for-generated-vars

Add missing generated variables to runtime/include/chplcgfns.h

runtime/include/chplcgfns.h is home to extern variables and function
declarations that are defined in the generated code. It hasn't been updated in
a while so I added all of the current ones and added a note in codegen that new
values should be placed in this file.

I ran into this when I wanted to check CHPL_STACK_CHECKS. I didn't originally
realize this file existed and while adding CHPL_STACK_CHECKS I figured I'd
update all of them.
Change the default value of fNoStackChecks

Stack checks are not really used currently, the variable and compiler flag were
added to support turning off qthread guard pages. While working on turning off
qthreads guard pages I noticed the default value was wrong. We want stack
checks on by default, and only want to turn them off if the user requested
no-stack-checks (or implicitly through --no-checks and --fast)
Disable qthread guard pages if user compiled with stack checks off

Turn off guard pages if the user turned off stack checks. That can happen
either explicitly through (--no-stack-checks) or implicitly through --no-checks
and thus --fast. Either way, CHPL_STACK_CHECKS (variable that gets generated to
chpl_compilation_config.h) gets set to 0. This adds an extern reference to that
variable and a check for it to determine if we should set qthread guard pages
off. Note that we do NOT override if the user explicitly set either of the
qthread guard pages env vars.
Remove extern declaration of CHPL_STACK_CHECKS from tasks-qthreads.c

This extern (actual definition in the generated code) is now in
runtime/include/chplcgfns.h. This is the proper place for it so it was added
there, and can now be removed from tasks-qthreads.c
Merge pull request #146 from ronawho/qthreads-stack-check

Bind qthreads guard pages to CHPL_STACK_CHECKS

We want to disable qthread guard pages if the user requested stack checks be
off at compile time. The infrastructure for adding a stack checks option to the
compiler and generated code was added previously and this just uses that to
actually turn QTHREAD_GUARD_PAGES off (unless the user explicitly set
QTHREAD_GUARD_PAGES themselves.)

This also changes the default value of fNoStackChecks checks since we want 
them on by default.
Add volatile to *(int *)(0) = 0; in qthreads source

"*(int *)(0) = 0;" is used in sections that should never be executed and if
somehow you did get there this causes a deliberate segfault so you can get a
good core dump. Clang complains if this are non-volatile, which breaks the
build if CHPL_DEVELOPER (warnings => errors) is set.
Merge pull request #148 from ronawho/qthreads-darwin-build-issue

Add volatile to *(int *)(0) = 0; in qthreads source

[ok'd by qthreads team]

"*(int *)(0) = 0;" is used in sections that should never be executed and if
somehow you did get there this causes a deliberate segfault so you can get a
good core dump. Clang complains if this are non-volatile, which breaks the
build if CHPL_DEVELOPER (warnings => errors) is set.
Merged chapel-lang/master
Checkpoint 8/13