chapel-lang/chapel — pull request #195 by ronawho (2014-08-22 18:56:11+00:00)
Title: Always tell qthreads how many threads we want to use

Previously we only told qthreads how many threads to use if a user set
CHPL_RT_NUM_THREADS_PER_LOCALE or if they set the qthreads variables
themselves.

However this became a problem (led to performance regressions) when switching
to the nemesis scheduler. The affinity layer seems to assume a multi-threaded
shepherd is being used. Previously we patched qthreads to correct the number
of workers and workers per shepherd after the affinity layer setup, but the
problem was that the affinity layer was still using the bad information to pin
threads to cores.

With this we now always specify the number of threads to use. We default to num
cores on a node (which is what we used to try and get qthreads to do
automatically.)

Previously we always set worker_unit=core (unless the user set it explicitly.)
Now we only set that if num threads <= num cores, otherwise we set pu. Note
that pu is the qthreads default, but we explicitly set it, since that default
is documented, but might change in the future.

The only policy this changes for us is that we no longer always set
worker_unit=core, other than that this is just cleanup/fixing things so we get
our previous policy correct again.

Some more details:
- Added an include for qt_shepherd_innards.h to get the threadqueue policy (so
we either set num_shepherds/num_wps, or set hwpar an let qthreads figure it
out.)

- Added macro for the size we use for looking at qthread environment
variables. We were pretty good at using 100, but this seems easier. 100
matches the size that qthreads uses, although they just use the literal
everywhere rather than having a marco.

- Adds a function to check if the user set the qthreads level variables for
the number of threads.

- Clean up some comments.

- Always unset qthread variables for controlling number of threads. We either
override them in the case that CHPL_RT_NUM_THREADS_PER_LOCALE was set, reset
them (and maybe limit to commMax) if the user set them themselves, or for
the default they were never set, so it doesn't hurt to unset them.