chapel-lang/chapel — pull request #632 by thomasvandoren (2014-10-11 08:06:56+00:00)
Title: Support parallel execution in makefiles.

Update makefiles such that calling `make -j` in $CHPL_HOME successfully
builds. The one exception is llvm, which has so many targets it is advisable to
set a finite number of jobs (e.g. `make -j16` instead of just `make -j` which
uses infinite jobs).

Adds `.NOTPARALLEL` to several makefiles that have order dependencies in order
to support this.

For the compiler, disables parallel recipe execution in the top-level
compiler/Makefile. This is needed so the object files exist before the $(CHPL)
prequisites are evaluated. The objects are built in a recursive make call, so
the files do not exist until that recipe completes. Fortunately, the expensive
recipes (the .o compiles) are not done from that file, so they are still done
in parallel. [.NOTPARALLEL][0] only affects the current file and recursive make
calls still use parallel execution.

For the runtime, both runtime/Makefile and runtime/Makefile.help disable
parallel execution. Makefile.help has the same issue as the compiler, but like
the compiler is still able to parallelize the compiles. The runtime/Makefile
disables parallel execution so the third-party recipes are built before the
runtime recipes. This is required for things like qthreads and tcmalloc that
the runtime expects to be available.

Also add small improvement to third-party-pkgs recipe runtime/Makefile. Instead
of iterating through each pkg calling make individually, call make once and
pass all the pkgs targets. This allows make to execute all of them in parallel,
when possible. hwloc has to be called first, when present, since it is required
to be built before qthreads. (I believe that is the only inter-third-party
build dependency.)

The top level third-party makefiles, like third-party/gmp/Makefile, disable
parallel execution because they typically have order dependencies. This still
allows the actual third party builds to use parallel execution. For example,
the unpacking and setup of the gmp directory is done sequentially, but the
actual make call that builds gmp uses parallel execution.

Also remove the `-j$(NPROCS)` args in gmp and llvm makefiles. These are no
longer needed since -j can be set at the top level make, and will use available
jobs for parallel execution when building llvm and gmp.

For details on how parallel execution works, especially with recursive make
calls, see the [Parallel Execution][1] section of the GNU make documentation.

Timings with Parallel and Sequential Execution
----------------------------------------------

Times in the table below are in seconds.

Each run was done once on my mac laptop with 4 cores and hyperthreading (8
logical cores). `time -p` was used to collect time values. `git clean -dxf` was
run before each timing to make sure the repo was completely clean.

The -j value used for the parallel execution is in parenthesis below. With
llvm, a finite number of jobs must be specified on my laptop, otherwise it runs
out of resources during the llvm build and fails.

|Environment | Sequential | Parallel |
|---------------- | ----------:| --------:|
| default (-j) | 232.57 | 136.26 |
| quickstart env (-j) | 33.28 | 11.28 |
| tcmalloc, gasnet (-j) | 312.13 | 150.06 |
| llvm (-j8) | 727.37 | 666.00 |
| llvm (-j16) | 727.37 | 615.31 |

Clearly parallel execution helps a lot (unsuprisingly :-)).

The llvm configuration only improves a little with parallel execution (compared
to the other configs), because the llvm makefile previously set
`-j`. I didn't bother running without that, mostly
because I did not want to wait for it to finish.

### Configuration details

**default**: No extra environment settings. Use defaults for everything.
**quickstart**: `source util/quickstart/setchplenv.bash` before calling make.
**tcmalloc, gasnet**: Set `CHPL_MEM=tcmalloc CHPL_COMM=gasnet` in environment before calling make.
**llvm**: Set `CHPL_LLVM=llvm` in environment before calling make.

[0]: https://www.gnu.org/software/make/manual/html_node/Special-Targets.html#Special-Targets
[1]: https://www.gnu.org/software/make/manual/html_node/Parallel.html#Parallel