Remove -j make flags in gmp and llvm makefiles.
Support parallel execution in makefiles.

Update makefiles such that calling `make -j` in $CHPL_HOME successfully
builds. The one exception is llvm, which has so many targets it is advisable to
set a finite number of jobs (e.g. `make -j16` instead of just `make -j` which
uses infinite jobs).

Adds .NOTPARALLEL to several makefiles that have order dependencies in order to
support this.

For the compiler, disables parallel recipe execution in the top-level
compiler/Makefile. This is needed so the object files exist before the $(CHPL)
prequisites are evaluated. The objects are built in a recursive make call, so
the files do not exist until that recipe completes. Fortunately, the expensive
recipes (the .o compiles) are not done from that file, so they are still done
in parallel. [.NOTPARALLEL][0] only affects the current file and recursive make
calls still use parallel execution.

For the runtime, both runtime/Makefile and runtime/Makefile.help disable
parallel execution. Makefile.help has the same issue as the compiler, but like
the compiler is still able to parallelize the compiles. The runtime/Makefile
disables parallel execution so the third-party recipes are built before the
runtime recipes. This is required for things like qthreads and tcmalloc that
the runtime expects to be available.

The top level third-party makefiles, like third-party/gmp/Makefile, disable
parallel execution because they typically have order dependencies. This still
allows the actual third party builds to use parallel execution. For example,
the unpacking and setup of the gmp directory is done sequentially, but the
actual make call that builds gmp uses parallel execution.

For details on how parallel execution works, especially with recursive make
calls, see the [Parallel Execution][1] section of the GNU make documentation.

[0]: https://www.gnu.org/software/make/manual/html_node/Special-Targets.html#Special-Targets
[1]: https://www.gnu.org/software/make/manual/html_node/Parallel.html#Parallel
Build all third party targets in parallel before runtime.

If hwloc is one of the packages, built it first, though. Otherwise, qthreads
will fail as it expects it to be built first when CHPL_HWLOC=hwloc (the default
when CHPL_TASKS=qthreads).
Update README to mention parallel build support.
Merge branch 'master' of https://github.com/chapel-lang/chapel into para-make
Fix execopts and compopts processing for paratest.server

Mike noticed that if you have something like paratest.server -compopts --llvm
-compopts --verify, that only --verify was picked up. The reason is pretty
simple and was that every time a compopts/execopts was found, it overrode the
previous value instead of concatenating. This just updates execopts and
compopts processing to concatenate instead of overriding. For all the other
arguments, overriding makes sense.
Make compopts/execopts have leading instead of trailing space for paratest

Makes it easier to read the code, and matches what we do in start_test, has no
affect on functionality
whoops, committed .swp files
Hack around an issue with chapel.tab.h/cpp recipe.

The `$(OBJ_SUBDIR)/chapel.tab.o` recipe in `compiler/parser/Makefile` has a
prerequisite `chapel.tab.cpp`. There are several recipes in that Makefile that
declare `chapel.tab.h` (note .h as opposed to .cpp). "chapel.tab.h
chapel.tab.cpp" is a single recipe since both files are output when bison is
called. And, both files should be rebuilt when chapel.ypp, their prerequisite,
is changed.

With parallel execution, the fact that both chapel.tab.h and chapel.tab.cpp are
separate files causes a race condition. Make see them each as dependencies for
other recipes, so it invokes the recipe to create chapel.tab.h and, in a
separate job, the recipe to build chapel.tab.cpp. This creates two instances of
the bison call. The final state is ok because the second bison call will
eventually finish and write out the file.

The issue arises when another job (that was depending on either chapel.tab.h or
chapel.tab.cpp) begins after the first bison call completes, but before the
second one completes. It then gets a file that is incomplete (the second bison
call is in the process of writing it out).

To workaround this issue, change the `$(OBJ_SUBDIR)/chapel.tab.o` recipe to
specify chapel.tab.h as a prerequisite. And, instead of using `$<` to pass the
files to the CXX call, list each file individually and use chapel.tab.cpp
(which is what it wants). IOW, use chapel.tab.h as the prequisite so bison only
is invoked once. And then use chapel.tab.cpp once the chapel.tab.h prerequisite
is met.

(I am open to better solutions, but this is the best one I could come up with.)
Merge branch 'master' of github.com:chapel-lang/chapel into para-make
Last update prior to new naming scheme
Merge pull request #648 from ronawho/fix-paratest-arg-processing

Fix execopts and compopts processing for paratest.server

[reviewed by @noakesmichael]

Mike noticed that if you have something like paratest.server -compopts --llvm
-compopts --verify, that only --verify was picked up. The reason is pretty
simple and was that every time a compopts/execopts was found, it overrode the
previous value instead of concatenating. This just updates execopts and
compopts processing to concatenate instead of overriding. For all the other
arguments, overriding makes sense.
Differentiate XC from XE whitebox testing.
Add comment based on review feedback.
Merge pull request #649 from thomasvandoren/fix-xe-wb-test-label

Differentiate XC from XE whitebox testing.
Significant reformatting/refactoring of REGRESSIONS

To go along with our nice, clean, new cron job identifiers, I've
refactored the REGRESSIONS file, in part inspired by the cleanup, in
part enabled by it.  This doesn't do anything like get us to a
completely happy place with REGRESSIONS, but hopefully people will
understand it better and it'll be useful for regressions week.
(And hopefully after REGRESSIONS week, it'll be much emptier).

There's still more to do here (esp. since not all tests have come
in yet under the new system), but this is a good enough start
that it's worth snapshotting and pushing out for people to see.

The major changes were:

* Updated comments at top of file to reflect current thinking/format

* Reformatted category entries to match nightly runs

* Made the implied/mental inheritance between categories explicit in
  the category header itself rather than vague entries like "along
  with the above"

* Introduced new "wildcard/abstract categories" (other than just
  "general" before) to serve as roots for common configurations
  like "gasnet*" rather than relying on "gasnet-everything" to
  be the base case for all GASNet testing (it wasn't always
  accurate in this respect)

* Tightened up the wording for the category names, reviewed by
  dates, etc. to be pithier.

* Reordered tests to reflect what seems like a more logical ordering:
  Primary environments, interesting configurations, gasnet, xe/xc,
  lower-priority environments, then distribution testing.

* Also tried to unify more on the name "sporadic" when describing
  nondeterministic cases and move these to the end of sections,
  but more work remains here.
Merge pull request #650 from bradcray/triage-sep-blc

Significant REGRESSIONS reformatting/cleanup

To go along with our nice, clean, new cron job identifiers, I've
refactored the REGRESSIONS file, in part inspired by the cleanup, in
part enabled by it.  This doesn't do anything like get us to a
completely happy place with REGRESSIONS, but hopefully people will
understand it better and it'll be useful for regressions week.
(And hopefully after REGRESSIONS week, it'll be much emptier).

There's still more to do here (esp. since not all tests have come
in yet under the new system), but this is a good enough start
that it's worth snapshotting and pushing out for people to see.

The major changes were:

* Updated comments at top of file to reflect current thinking/format

* Reformatted category entries to match nightly runs

* Made the implied/mental inheritance between categories explicit in
  the category header itself rather than vague entries like "along
  with the above"

* Introduced new "wildcard/abstract categories" (other than just
  "general" before) to serve as roots for common configurations
  like "gasnet*" rather than relying on "gasnet-everything" to
  be the base case for all GASNet testing (it wasn't always
  accurate in this respect)

* Tightened up the wording for the category names, reviewed by
  dates, etc. to be pithier.

* Reordered tests to reflect what seems like a more logical ordering:
  Primary environments, interesting configurations, gasnet, xe/xc,
  lower-priority environments, then distribution testing.

* Also tried to unify more on the name "sporadic" when describing
  nondeterministic cases and move these to the end of sections,
  but more work remains here.

Here are some excerpts for reference:

    ===================
    general
    Reviewed 2014-10-16
    ===================
    
    sporadic failure (2-3x a month -- vass)
    ---------------------------------------
    [Error matching program output for parallel/taskPar/taskIntents/21-capture-in-cobegin]


    ...

    ===================
    no-local
    Inherits 'general'
    Reviewed ????-??-??
    ===================


    ===================
    gasnet* regressions
    Inherits 'no-local'
    Reviewed 2014-10-16
    ===================

    annoying occasional failures even after Sung calmed down (09/18/14 -- sungeun)
    ------------------------------------------------------------------------------
    [Error matching program output for types/string/StringImpl/memLeaks/coforall]

    error: cannot create C string from remote string (due to r22900 -- sungeun)
    ---------------------------------------------------------------------------
    [Error matching program output for classes/sungeun/remoteDynamicDispatch (compopts: 1)]
    [Error matching program output for multilocale/diten/needMultiLocales/remoteStringTuple] (due to r23558)
    [Error matching program output for optimizations/sungeun/RADOpt/access1d (compopts: 1)] (2014/06/14)
    [Error matching program output for optimizations/sungeun/RADOpt/access2d (compopts: 1)] (2014/06/14)
    [Error matching program output for optimizations/sungeun/RADOpt/access3d (compopts: 1)] (2014/06/14)
    
    
    ===================
    gasnet-fast
    Inherits 'gasnet*'
    Reviewed ????-??-??
    ===================
    
    noisy failure due to 'slave got an unknown command on coord socket' (09/27/14)
    ------------------------------------------------------------------------------
    [Error matching program output for multilocale/diten/localBlock/needMultiLocales/localBlock5]
    
    noisy unresolved access of range by (2) (10/07/14)
    --------------------------------------------------
    [Error matching program output for users/jglewis/SSCA2_sync_array_initialization_bug]
    
    consistent timeouts
    -------------------
    [Error: Timed out executing program associative/bharshbarg/parSafeAdd (compopts: 1)]
    [Error: Timed out executing program domains/sungeun/assoc/parSafeMember]
    [Error: Timed out executing program memory/shannon/outofmemory/mallocOutOfMemory]
    [Error: Timed out executing program studies/lulesh/sungeun/lulesh (compopts: 1)]
    
    
    
    ================================
    gasnet.linux32
    inherits 'linux32' and 'gasnet*'
    Reviewed 2014-10-16
    ================================

    fairly consistent timeouts due to qthreads oversubscription (09/15/14)
    ----------------------------------------------------------------------
    [Error: Timed out executing program release/examples/benchmarks/hpcc/hpl]
    [Error: Timed out executing program release/examples/benchmarks/hpcc/ptrans]
    [Error: Timed out executing program release/examples/benchmarks/ssca2/SSCA2_main (compopts: 3, execopts: 1)]
    [Error: Timed out executing program release/examples/benchmarks/ssca2/SSCA2_main (compopts: 4, execopts: 1)]
    [Error: Timed out executing program release/examples/benchmarks/ssca2/SSCA2_main (compopts: 5, execopts: 1)]

    sporadic timeout (infrequent)
    -----------------------------
    [Error: Timed out executing program release/examples/benchmarks/ssca2/SSCA2_main (compopts: 2, execopts: 1)]
Merge pull request #632 from thomasvandoren/para-make

Support parallel execution in makefiles.

Update makefiles such that calling `make -j` in $CHPL_HOME successfully
builds. The one exception is llvm, which has so many targets it is advisable to
set a finite number of jobs (e.g. `make -j16` instead of just `make -j` which
uses infinite jobs).

Adds `.NOTPARALLEL` to several makefiles that have order dependencies in order
to support this.

For the compiler, disables parallel recipe execution in the top-level
compiler/Makefile. This is needed so the object files exist before the $(CHPL)
prequisites are evaluated. The objects are built in a recursive make call, so
the files do not exist until that recipe completes. Fortunately, the expensive
recipes (the .o compiles) are not done from that file, so they are still done
in parallel. [.NOTPARALLEL][0] only affects the current file and recursive make
calls still use parallel execution.

For the runtime, both runtime/Makefile and runtime/Makefile.help disable
parallel execution. Makefile.help has the same issue as the compiler, but like
the compiler is still able to parallelize the compiles. The runtime/Makefile
disables parallel execution so the third-party recipes are built before the
runtime recipes. This is required for things like qthreads and tcmalloc that
the runtime expects to be available.

Also add small improvement to third-party-pkgs recipe runtime/Makefile. Instead
of iterating through each pkg calling make individually, call make once and
pass all the pkgs targets. This allows make to execute all of them in parallel,
when possible. hwloc has to be called first, when present, since it is required
to be built before qthreads. (I believe that is the only inter-third-party
build dependency.)

The top level third-party makefiles, like third-party/gmp/Makefile, disable
parallel execution because they typically have order dependencies. This still
allows the actual third party builds to use parallel execution. For example,
the unpacking and setup of the gmp directory is done sequentially, but the
actual make call that builds gmp uses parallel execution.

Also remove the `-j$(NPROCS)` args in gmp and llvm makefiles. These are no
longer needed since -j can be set at the top level make, and will use available
jobs for parallel execution when building llvm and gmp.

For details on how parallel execution works, especially with recursive make
calls, see the [Parallel Execution][1] section of the GNU make documentation.

### Hack around an issue with chapel.tab.h/cpp recipe.

The `$(OBJ_SUBDIR)/chapel.tab.o` recipe in `compiler/parser/Makefile` had a
prerequisite `chapel.tab.cpp`. There are several recipes in that Makefile that also
declare `chapel.tab.h` (note .h as opposed to .cpp). "chapel.tab.h
chapel.tab.cpp" is a single recipe since both files are output when bison is
called. And, both files should be rebuilt when chapel.ypp, their prerequisite,
is changed.

With parallel execution, the fact that both chapel.tab.h and chapel.tab.cpp are
separate files for one recipe causes a race condition. Make sees them each as
dependencies for other recipes, so it invokes the recipe to create chapel.tab.h and, 
in a separate job, the recipe to build chapel.tab.cpp. This creates two instances of
the bison call. The final state is ok because the second bison call will
eventually finish and write out both files (both bison calls output both files).

The issue arises when another job (that was depending on either chapel.tab.h or
chapel.tab.cpp) begins after the first bison call completes, but before the
second one completes. It then gets a file that is incomplete (the second bison
call is in the process of writing it out).

To workaround this issue, change the `$(OBJ_SUBDIR)/chapel.tab.o` recipe to
specify chapel.tab.h as a prerequisite. And, instead of using `$<` to pass the
files to the CXX call, list each file individually and use chapel.tab.cpp
(which is what it wants). IOW, use chapel.tab.h as the prerequisite so bison only
is invoked once. And then use chapel.tab.cpp once the chapel.tab.h prerequisite
is met.

Timings with Parallel and Sequential Execution
----------------------------------------------

Times in the table below are in seconds.

Each run was done once on my mac laptop with 4 cores and hyperthreading (8
logical cores). `time -p` was used to collect time values. `git clean -dxf` was
run before each timing to make sure the repo was completely clean.

The -j value used for the parallel execution is in parenthesis below. With
llvm, a finite number of jobs must be specified on my laptop, otherwise it runs
out of resources during the llvm build and fails.

|Environment      | Sequential | Parallel |
|---------------- | ----------:| --------:|
| default (-j)          | 232.57 | 136.26 |
| quickstart env (-j)   |  33.28 |  11.28 |
| tcmalloc, gasnet (-j) | 312.13 | 150.06 |
| llvm (-j8)            | 727.37 | 666.00 |
| llvm (-j16)           | 727.37 | 615.31 |

Clearly parallel execution helps a lot (unsuprisingly :-)).

The llvm configuration only improves a little with parallel execution (compared
to the other configs), because the llvm makefile previously set
`-j<number_logical_cores>`. I didn't bother running without that, mostly
because I did not want to wait for it to finish.

### Configuration details

**default**: No extra environment settings. Use defaults for everything.
**quickstart**: `source util/quickstart/setchplenv.bash` before calling make.
**tcmalloc, gasnet**: Set `CHPL_MEM=tcmalloc CHPL_COMM=gasnet` in environment before calling make.
**llvm**: Set `CHPL_LLVM=llvm` in environment before calling make.

[0]: https://www.gnu.org/software/make/manual/html_node/Special-Targets.html#Special-Targets
[1]: https://www.gnu.org/software/make/manual/html_node/Parallel.html#Parallel

TODO:
-----
* [x] full test pass (building with `make -j DEBUG=0 OPTIMIZE=1 WARNINGS=1`)
* [x] verify parallel build on several platforms (run both `make` and `make -j` with default configuration):
  * [x] linux64 (242s vs 398s)
  * [x] linux32 (211s vs 411s)
  * [x] mac os x (161s vs 282s)
  * [x] cygwin (629s vs 1351s)
  * [x] cray XC (with cce) (223s vs 463s)
* [x] Should the `.NOTPARALLEL` targets go towards the top of the makefiles to make them more obvious?
* ~~[ ] update travis testing to call make with `-j<number_logical_cores>` to improve build time there~~
* ~~[ ] update nightly to call make with `-j<number_logical_cores>` to improve build time~~

See #634 for test system changes.