Add real comments to yields in qthreads sync var lock and unlock

These were added to resolve deadlock that cropped up when we switched to the
nemesis scheduler. I never commented them properly because we weren't sure it
was the final solution or if the values were good. Things seem to be in good
shape some I'm adding extensive comments.
Switch to hwloc v1.9.1 as default.
Stop performance testing if OPTIMIZED=1 build failed

* If the compiler OPTIMIZED=1 build fails, turn off compiler perf testing.
* If the runtime OPTIMIZED=1 build fails and it is a performance test run, it
  is a fatal error, so send the failure message and exit.
* Added the ability to exit on a fatal error back to mysystem after it was
  accidentally removed in a previous commit.

Tested by doing: nightly -debug -performance -notest
with and without things to warn about added to the compiler and runtime.
Merge pull request #265 from gbtitus/hwloc-1.9.1-is-default

Switch to hwloc v1.9.1 as default.

Reviewed by @bradcray, @sungeunchoi, @ronawho over email.
Wordsmithing based on gregs review
Merge pull request #260 from ronawho/qthreads-syncvar-deadlock

Add real comments to yields in qthreads sync var lock and unlock

[reviewed by @gbtitus]

These were added to resolve deadlock that cropped up when we switched to the
nemesis scheduler. I never commented them properly because we weren't sure it
was the final solution or if the values were good. Things seem to be in good
shape some I'm adding extensive comments.
Turn off guard pages for stress/deitz/test_10k_coforall

I see this test timeout sometimes when I do a paratest when other people are
also testing. It creates a lot of small tasks so it makes sense to turn off
guard pages for it.
Unify stress and stress.numthr execopts and turn off guard pages

There are two versions of this test because we don't have a way to use a
different execenv for a single test. I believe they are intended to be
identical so I made the execopts the same.

This also turns off guard pages for both as it is a stress test that creates a
large number of tasks and takes a long amount of time to complete with guard
pages. stress.numthr already had guard pages off but as a compopt instead of
as part of the execenv.

The compopts is nice because it's general and the way we want users to do it,
but the execenv is nice because you don't have to copy it for each compopt. For
now I decided to stick with using QT_GUARD_PAGES=false over the compopt, and
once CHPL_STACK_CHECKS also backs whether guard pages are used at execution time
I will replace all uses of QT_GUARD_PAGES
Copied the execenv, but didn't clear the unwanted env vars
Merge pull request #266 from ronawho/qthreads-timeouts

Turn off guard pages for two stress tests

[reviewed by @sungeunchoi]

Turn off guard pages for stress/deitz/test_10k_coforall:

I see this test timeout sometimes when I do a paratest when other people are
also testing. It creates a lot of small tasks so it makes sense to turn off
guard pages for it.

Unify stress and stress.numthr execopts and turn off guard pages:

There are two versions of this test because we don't have a way to use a
different execenv for a single test. I believe they are intended to be
identical so I made the execopts the same.

This also turns off guard pages for both as it is a stress test that creates a
large number of tasks and takes a long amount of time to complete with guard
pages. stress.numthr already had guard pages off but as a compopt instead of
as part of the execenv.

The compopts is nice because it's general and the way we want users to do it,
but the execenv is nice because you don't have to copy it for each compopt. For
now I decided to stick with using QT_GUARD_PAGES=false over the compopt, and
once CHPL_STACK_CHECKS also backs whether guard pages are used at execution time
I will replace all uses of QT_GUARD_PAGES
Turn off guard pages for distributions/robust/associative/stress/many_domain

Guard pages are already off for
distributions/robust/associative/stress/many_array and this is a similar test,
that takes way too much time with guard pages on.
Merge pull request #268 from ronawho/qthreads-timeouts

Turn off guard pages for distributions/robust/associative/stress/many_domain

[trivial, not reviewed]

Guard pages are already off for
distributions/robust/associative/stress/many_array and this is a similar test,
that takes way too much time with guard pages on.
Add an option to sub_test to warn if a test took over X seconds to execute

This functionality is motivated by trying to find stress tests for which guard
pages should be turned off, and applications that may need a higher timeout as
a result of increased execution time from guard pages.

I think it is useful beyond that too, and can easily be extended to report long
compilations times as well.
Merge pull request #269 from ronawho/qthreads-timeouts

Add option to sub_test to warn if a test took over X seconds to execute

[reviewed by @noakesmichael]

This functionality is motivated by trying to find stress tests for which guard
pages should be turned off, and applications that may need a higher timeout as
a result of increased execution time from guard pages.

I think it is useful beyond that too, and can easily be extended to report long
compilations times as well.

Note that this currently must be manually set by a user, and nothing in start_test 
sets it. It's also unaffected by the timeout value, it's just a simple "did this test
take over X seconds to execute?"
Turn on warnings for nightly for tests that took > 60 seconds to execute

I'm using this to try to find stress tests that need guard pages turned off and
codes that may need timeouts added because they now take longer with guard
pages. This is going to make testing noisy for the next two days.

It will not report results for valgrind
Merge pull request #271 from ronawho/qthreads-timeouts

Turn on warnings for nightly for tests that took > 60 seconds to execute

[ok'd by sungeunchoi and thomasvandoren]

I'm using this to try to find stress tests that need guard pages turned off and
codes that may need timeouts added because they now take longer with guard
pages. This is going to make testing noisy for the next two days.

It will not report results for valgrind

Note this is temporary and will only run for tonight.
Forgot to lower stack size for stress like we so for stress.numthr
Merge pull request #272 from ronawho/qthreads-timeouts

Forgot to lower stack size for stress like we so for stress.numthr

[trivial, not reviewed]
Merge pull request #264 from daviditen/nightlyPerf

Stop performance testing if OPTIMIZED=1 build failed

* If the compiler OPTIMIZED=1 build fails, turn off compiler perf testing.
* If the runtime OPTIMIZED=1 build fails and it is a performance test run, it is a fatal error, so send the failure      
  message and exit.
* Added the ability to exit on a fatal error back to mysystem after it was accidentally removed in a
  previous commit.

Tested by doing: nightly -debug -performance -notest
with and without things to warn about added to the compiler and runtime.
[reviewed by gbtitus]
Fixed some regressions from last nightly testing