Stop performance testing if OPTIMIZED=1 build failed

* If the compiler OPTIMIZED=1 build fails, turn off compiler perf testing.
* If the runtime OPTIMIZED=1 build fails and it is a performance test run, it
  is a fatal error, so send the failure message and exit.
* Added the ability to exit on a fatal error back to mysystem after it was
  accidentally removed in a previous commit.

Tested by doing: nightly -debug -performance -notest
with and without things to warn about added to the compiler and runtime.
Quiet TAGS complaints when building Qthreads

We've had some noise in the runtime's build of the TAGS file due
to the fact that the generic glob patterns that were designed to
cover the header files for the task/ and thread/ layers didn't
match any files.

The fix I've taken here is to move the specification of the .h
files out of the tag command itself, in Makefile.help, and into
the Makefile.include files in question.  So far I've only done
it for the default (Qthreads) configuration, which seems to be
working.

Next step is to do the same for other task/thread options, and
arguably for other aspects of the runtime that utilize header
files.
Updated fifo and massivethreads Makefile.includes to name .h files
Made the threads Makefile.include files include their header files
Add an option to sub_test to warn if a test took over X seconds to execute

This functionality is motivated by trying to find stress tests for which guard
pages should be turned off, and applications that may need a higher timeout as
a result of increased execution time from guard pages.

I think it is useful beyond that too, and can easily be extended to report long
compilations times as well.
Merge pull request #269 from ronawho/qthreads-timeouts

Add option to sub_test to warn if a test took over X seconds to execute

[reviewed by @noakesmichael]

This functionality is motivated by trying to find stress tests for which guard
pages should be turned off, and applications that may need a higher timeout as
a result of increased execution time from guard pages.

I think it is useful beyond that too, and can easily be extended to report long
compilations times as well.

Note that this currently must be manually set by a user, and nothing in start_test 
sets it. It's also unaffected by the timeout value, it's just a simple "did this test
take over X seconds to execute?"
Turn on warnings for nightly for tests that took > 60 seconds to execute

I'm using this to try to find stress tests that need guard pages turned off and
codes that may need timeouts added because they now take longer with guard
pages. This is going to make testing noisy for the next two days.

It will not report results for valgrind
Merge pull request #271 from ronawho/qthreads-timeouts

Turn on warnings for nightly for tests that took > 60 seconds to execute

[ok'd by sungeunchoi and thomasvandoren]

I'm using this to try to find stress tests that need guard pages turned off and
codes that may need timeouts added because they now take longer with guard
pages. This is going to make testing noisy for the next two days.

It will not report results for valgrind

Note this is temporary and will only run for tonight.
Forgot to lower stack size for stress like we so for stress.numthr
Merge pull request #272 from ronawho/qthreads-timeouts

Forgot to lower stack size for stress like we so for stress.numthr

[trivial, not reviewed]
Merge pull request #264 from daviditen/nightlyPerf

Stop performance testing if OPTIMIZED=1 build failed

* If the compiler OPTIMIZED=1 build fails, turn off compiler perf testing.
* If the runtime OPTIMIZED=1 build fails and it is a performance test run, it is a fatal error, so send the failure      
  message and exit.
* Added the ability to exit on a fatal error back to mysystem after it was accidentally removed in a
  previous commit.

Tested by doing: nightly -debug -performance -notest
with and without things to warn about added to the compiler and runtime.
[reviewed by gbtitus]
Fixed some regressions from last nightly testing
Merge pull request #273 from vasslitvinov/fixup-tests-for-coercion-wrapper-elimination

Fixed some regressions from last nightly testing.

.good changes in the robustness suite are due to improved fork counts, possibly due to coercion wrapper elimination. I did not investigate.

I added a missing .good file.

For the rest:


functions/vass/setter-passed-to-another-fun.bad

=> I verified that the new contents of .bad are legitimate


types/enum/sungeun/cast_enum_weird_4.bad

=> This might not be related to my coercion-wrapper commit.
At any rate, the updated line number makes more sense
than the one that was there before.


types/records/const-checking/scenario-1-array-of-record-with-const-fld.bad
types/records/const-checking/scenario-2-init-array-of-rec-with-const-fld.bad
types/records/const-checking/scenario-3-assoc-dom-of-record-with-const-fld.bad
types/records/const-checking/todo-linenumber-in-array-assignment-error.future
types/records/const-checking/todo-linenumber-in-array-assignment-error.bad

=> The new line number reported by the compiler is not better than the old one.
Neither is perfect; passing --print-callstack-on-error gives the desired info.
The following future requests fixing this:
  todo-linenumber-in-array-assignment-error.bad
Merge pull request #267 from bradcray/fixTags

Fix runtime tasks/threads TAGS

This change replaces some catch-all glob patterns that we were using to
get source files from the tasking and threading libraries into the
emacs TAGS (or vim ctags) files.  For tasks=qthreads, threads=none,
the globs didn't match anything resulting in warnings at TAGS creation
time.

The fix taken here is to

(a) push the glob patterns for header files from the top-level
    Makefile.help file down into the task- and thread-specific
    Makefile.include files so that they can be more precise
    about pointing to the files in question (if any).  The only
    real downside of doing this is that TAGS files should generally
    process source (.c) files before header (.h) files in order
    to take a TAGS surfer to the definition of a symbol rather than
    its prototype.  The current approach has the downside of
    mixing .c files and .h files in the ALL_SRCS variable used to
    create the TAGS.  However, since each tasking/threading layer
    gives its .c files first and then its .h files, I expect things
    will still work out OK.  If they didn't we could create an
    ALL_HEADERS variable that each layer contributes to and then
    have the tags command name ALL_SRCS before ALL_HEADERS.

(b) Had Qthreads point to the sources in third-party

(c) Had threads=none not list any files (left the pattern in the
    file, commented out)


The only testing I did on this was to make sure that the various
tasking configurations built and created TAGS without warnings.


Further TODOs:

* looking at this, it's apparent that we don't include any .h
  files in ALL_SRCS beyond those in tasks/ and threads/ (such
  as comm/, mem/, atomics/, etc.  These should be added into
  ALL_SRCS as well.  I didn't do so here primarily because it's
  5pm on a Friday holiday weekend.

* Sung had a request to rename ALL_SRCS something like ALL_RUNTIME_SRCS
  or possibly ALL_RUNTIME_TAG_SRCS to make its role clearer.  I
  forgot about that until I was typing this up (see previous note
  about 5pm on a holiday weekend)
Made some cleanup edits to the PERFORMANCE file

* created section headers to give the file more structure and make
  the --fast flag more visible.

* did some wordsmithing
Merge pull request #276 from bradcray/perfNoteRevs

Made some cleanup edits to the PERFORMANCE file

* created section headers to give the file more structure and make
  the --fast flag more visible.

* did some wordsmithing
Revert "Turn on warnings for nightly for tests that took > 60 seconds to execute"

This reverts commit 097345ca7ca48592b97b73d9cec4b5643a1ab871.

I got the results I wanted, turning this off now.
Merge pull request #277 from ronawho/qthreads-timeouts

Revert "Turn on warnings for nightly for tests that took > 60 seconds to execute"

This reverts commit 097345ca7ca48592b97b73d9cec4b5643a1ab871.

I got the results I wanted, turning this off now.
Get access to the chpl_getNumCoresOnThisNode() declaration.

When I updated tasks-massivethreads recently I missed the fact that it
didn't already #include "chplsys.h", so it didn't have a declaration for
the chpl_getNumCoresOnThisNode() call I added.  Add the #include.
Merge pull request #278 from gbtitus/fix-massivethreads-getNumCores

Get access to the chpl_getNumCoresOnThisNode() declaration.

When I updated tasks-massivethreads recently I missed the fact that it
didn't already #include "chplsys.h", so it didn't have a declaration for
the chpl_getNumCoresOnThisNode() call I added. Add the #include.

Not reviewed.