chapel-lang/chapel — pull request #51 by ronawho (2014-07-24 03:07:02+00:00)
Title: Enable "--enable-condwait-queue" for chap03/04 qthreads performance testing

[more chap03/04 qthreads performance tinkering, not reviewed]

Looking at some of the current performance regressions on chap04 for qthreads
it seemed that there was a pretty high performance penalty for purely serial
code. There doesn't seem to be a penalty on chap03 (or if there is, it's very
small.) However, I'm making the change there just to see what happens.

Using the "Loop Time Compare" graph as a case study:
- This is a purely serial code, however with qthreads all 8 cores were getting
pegged at 100%.
- Manually setting QTHREAD_NUM_SHEPHERD=1 resolved the performance hit,
indicating that there is interference from other shepherds.
- enabling qthreads over-subscription also resolved the performance hit. Then
by going through each setting that over-subscription set, I saw the
--enable-condwait-queue was responsible.

From qthreads:
--enable-condwait-queue: force the use of a pthread condwait queue, instead
of a spin-based queue for inter-thread communication (important if spinning
shepherds interfere with each other). Default enabled on sparc/solaris, but
default disabled elsewhere.

So it would seem that for serial code we "want" this set, but for highly
parallel code we wouldn't. I think the real solution should be that qthreads
should be able to see that it only has 1 task, and back the other workers off.

This is to see if the serial performance regressions are resolved, and to see
if and how much this hurts us for highly parallel cases.