Change default scheduler for qthreads and build with --enable-condwait-queue

After a lot of investigation into qthreads performance regressions we found two
main causes. The first is that qthreads default scheduler (sherwood) has
significant overhead. Sherwood is a work stealing scheduler with no real way to
turn off work stealing (we tried, but it relies on work stealing to divvy up
tasks so with work stealing off you only use part of a machine.) Work stealing
and some other factors add too much overhead. As a result we're switching the
default to be nemesis for all configurations except for numa. For numa we rely
on the semantics that the sherwood scheduler provides (essentially 1 shepherd =
1 numa domain = 1 sublocale.)

The second source of overhead was for higher core count machines only and is
because qthreads workers that are idle busy wait for work. This can cause
interference for other workers, and in particular really hurt our single
threaded and low task count applications. By building with
--enable-condwait-queue we avoid that penalty and idle workers call into the os
to be swapped rather than busy waiting.

With these two changes the only remaining performance regressions are for tests
that perform better with hyperthreading since we don't use hyperthreads by
default for qthreads but we do for fifo.

This also pulls --disable-spawn-cache out of the main config line into it's own
line so I could add a comment as to why we set it since I always forget. See
commit 2df9f3006a0bea3fca477c71aaacb1b626855eed (svn r19695) for original commit
of --disable-spawn-cache.

Note that this configuration has already been tested for the machines doing
qthreads performance runs and this is just the global change.
Change qthreads topology/affinity setup to work with nemesis

qthreads seems to setup the number of workers and workers per shepherd assuming
the shwerwood scheduler is being used when configured with hwloc. This changes
some of the logic to work with the nemesis scheduler. I have the patch out to
Dylan as well and will make changes that he suggests. I may go with this for a
little while if I don't hear back from him just to get some testing in.
Remove special qthreads setup for nemesis scheduler

Code was temporarily put into our qthreads setup to allow us to test out the
performance of the nemesis scheduler. Now that we're going to nemesis by
default this logic has been moved into the qthreads source (as a bug fix) and
is no longer needed in our qthreds setup. This reverts the tasks-qthreads.c
changes in commit 095f933f71792f7655a6b223aa2d6dc26e8d777c
Get rid of special setup for chap03/04 qthreads testing

Simply removes env var setup that was required to test either the nemesis
scheduler or modifications to the sherwood scheduler. The scheduler,
enable-condwait-queue, and shepherds/workers per shepherd all just get set
correctly without these env vars now.
Merge pull request #144 from ronawho/qthreads-default-scheduler

Use nemesis scheduler for qthreads and build with --enable-condwait-queue

For performance reasons we want to use the nemesis scheduler over the default
sherwood scheduler for qthreads. For single threaded and low task count
applications --enable-condwait-queue is needed for good performance. This
changes the default scheduler, builds with --enable-condwait-queue and removes
special code in our qthreads interface and in our qthreads performance testing
scripts that was used to test out this configuration without making it the
default for all other builds.

The other change was in the qthreads source itself to correctly set the number
of shepherds and workers per shepherd for single threaded schedulers (like
nemesis.) This is currently out for review with the qthreads team, but we want
to start testing now.

Note that there may be some deadlocks in testing temporarily. For a few tests
like domains/sungeun/assoc/stress.chpl we create tasks that increment a sync
var while the main task waits on the sync var to be a certain value. Those
tasks that are incrementing don't get to run because the main task never
yields. A fix for that is being worked on, but we want to start getting
testing going so we can switch to qthreads as our default tasking layer soon.