Always tell qthreads how many threads we want to use

Previously we only told qthreads how many threads to use if a user set
CHPL_RT_NUM_THREADS_PER_LOCALE or if they set the qthreads variables
themselves.

However this became a problem (led to performance regressions) when switching
to the nemesis scheduler. The affinity layer seems to assume a multi-threaded
shepherd is being used.  Previously we patched qthreads to correct the number
of workers and workers per shepherd after the affinity layer setup, but the
problem was that the affinity layer was still using the bad information to pin
threads to cores.

With this we now always specify the number of threads to use. We default to num
cores on a node (which is what we used to try and get qthreads to do
automatically.)

Previously we always set worker_unit=core (unless the user set it explicitly.)
Now we only set that if num threads <= num cores, otherwise we set pu. Note
that pu is the qthreads default, but we explicitly set it, since that default
is documented, but might change in the future.

The only policy this changes for us is that we no longer always set
worker_unit=core, other than that this is just cleanup/fixing things so we get
our previous policy correct again.

Some more details:
 - Added an include for qt_shepherd_innards.h to get the threadqueue policy (so
   we either set num_shepherds/num_wps, or set hwpar an let qthreads figure it
   out.)

 - Added macro for the size we use for looking at qthread environment
   variables. We were pretty good at using 100, but this seems easier. 100
   matches the size that qthreads uses, although they just use the literal
   everywhere rather than having a marco.

 - Adds a function to check if the user set the qthreads level variables for
   the number of threads.

 - Clean up some comments.

 - Always unset qthread variables for controlling number of threads. We either
   override them in the case that CHPL_RT_NUM_THREADS_PER_LOCALE was set, reset
   them (and maybe limit to commMax) if the user set them themselves, or for
   the default they were never set, so it doesn't hurt to unset them.
Whoops, put the macro for env var size in an ifdef for profiling
readd comment that we limit CHPL_RT_NUM_THREADS_PER_LOCALE to numPUsPerNode
Some updates

Move setting up the available parallelism and call stacks into their own
functions

Replace some uses of QT_ENV_S with sizeof(actual_var)

Remove the comments on the include files. Some of them aren't accurate and we

change verbosity == 2 to verbosity >= 2

No longer set number of threads, or WORKER_UNIT if the user set QT_NUM_WORKERS
or QT_NUM_WORKERS_PER_SHEPHERD (assume "expert" user.)
Clean up comments
Fix spelling error
Merge pull request #195 from ronawho/qthreads-set-hwpar-hints

Ensure qthreads is always told how many threads to use

[reviewed by @gbtitus]

Previously we only told qthreads how many threads to use if a user set
CHPL_RT_NUM_THREADS_PER_LOCALE or if they set the qthreads variables
themselves. Otherwise we just let qthreads figure it out on it's own.

However this became a problem (led to performance regressions) when switching
to the nemesis scheduler. The affinity layer seems to assume a multi-threaded
shepherd is being used. Previously we patched qthreads to correct the number
of workers and workers per shepherd after the affinity layer setup, but the
problem was that the affinity layer was still using the bad information to pin
threads to cores.

With this we now always specify the number of threads to use. We default to num
cores on a node (which is what we used to try and get qthreads to do
automatically.) If a user specified CHPL_RT_NUM_THREADS_PER_LOCALE we limit
this to numPUs and commMax, and then set the appropriate qthreads variables
(which ones depends on the scheduler type.) If a user sets hwpar we do
something similar to what we do for CHPL_RT_NUM_THREADS_PER_LOCALE, we just
don't limit to numPU (this is how a user can oversubscribe the hardware if they
really want to.) And lastly if the user set QT_NUM_SHEPHERDS or
QT_NUM_WORKERS_PER_SHEPHERD we assume they are an expert user and so we don't
do anything else to setup the number of threads or the worker unit.

Previously we always set worker_unit=core (unless the user set it explicitly.)
Now we only set that if the user didn't set QT_NUM_SHEP and/or QT_WPS and num
threads <= num cores, otherwise we set pu. Note that pu is the qthreads
default, but we explicitly set it since it might change in the future.

The only policy this changes for us is that we don't always set
worker_unit=core, other than that this is just cleanup/fixing things so we get
our previous policy correct again.

Some more details:

 - Added an include for qt_shepherd_innards.h to get the threadqueue policy (so
   we either set num_shepherds/num_wps, or set hwpar to let qthreads figure it
   out.)

 - Added macro for the size we use for looking at qthread environment
   variables. We were pretty good at using 100, but this seems easier. 100
   matches the size that qthreads uses, although they just use the literal
   everywhere rather than having a marco.

 - Adds a function to check if the user set the qthreads level variables for
   the number of threads.

 - Clean up some comments.

 - Split parallelism and call stack setup into two separate functions.