cfengine/core — pull request #1920 by kacfengine (2014-08-27 11:29:42+00:00)
Title: Implement parallel acceptance testing.

The implementation is based on generating a custom Makefile from
within testall, and then invoking a sub make command to execute the
tests in parallel. Since not all tests can run in parallel for various
reasons, tests that reside in a /serial/ or /unsafe/ directory will be
carried out serially.

Care has also been taken to ensure that timed tests will not hog the
make job slots, but unfortunately it has to occupy at least one slot,
which means that if your make -j flag contains exactly the number of
cores you have, one core will be mostly idle as long as any timed
tests are running. The recommended workaround is to raise the number
of jobs a bit above your core count (which is recommended practice
regardless).

The way the makefile is constructed is so that each parallel test has
its own rule, all the timed and serial tests each have one group rule,
and the serial rule is made to depend on all the other rules, so that
it runs last, and in isolation.

Note that for the sub make invokation to communicate with the job
server correctly, the testall script has to be run with a '+' prefix
from within the Makefile, hence it was removed from the TESTS variable
and executed manually instead.

The parallel test harness of automake was considered (see
http://www.gnu.org/software/automake/manual/automake.html#Parallel-Test-Harness)
as an alternative, but was disqualified for a number of reasons. The
two main obstacles were:

- The list of tests has to be generated in configure, and cannot be
easily regenerated on every run, which means that if you add a test
it will not be picked up unless you reconfigure the project.

- Timed tests would be problematic because they have to run together
in order to be efficient. If they don't, each test will wait its
entire wait period before allowing the next test to run. Automake
however, requires each test to be executed by a separate rule.