python-imaging/Pillow — pull request #632 by hugovk (2014-04-18 12:00:04+00:00)
Title: Run test sets in different Travis jobs

Experiment
===

This didn't work out, but I'll document it here for future reference.

Hypothesis
---

Running tests in separate jobs will speed up the Travis CI builds.

Results
---

Initially, build slowed down. Probably due to the overhead of installing dependencies and building Pillow at the start. If that bit can be sped up (possibly via caching) then this approach may help.

Changing Python order sped up build a bit (total waiting time), but uses more CPU time.

Details
---

Using an environment variable, we can run different test sets in different build jobs.

Reference:
http://docs.travis-ci.com/user/speeding-up-the-build/#Parallelizing-your-builds-across-virtual-machines

Currently we have two main tests sets: self-test, and the main tests.

Using an environment variable, Travis will run a different job for each value.
```
env:
- TEST_CMD="coverage run --append --include=PIL/* selftest.py"
- TEST_CMD="python Tests/run.py --coverage"
```

Then instead of running both in `script:`
```
- coverage run --append --include=PIL/* selftest.py
- python Tests/run.py --coverage
```

just run one:
```
- $TEST_CMD
```

This changes build times from:

26:32 https://travis-ci.org/hugovk/Pillow/builds/22906842
24:25 https://travis-ci.org/hugovk/Pillow/builds/23269205

to:

36:52 https://travis-ci.org/hugovk/Pillow/builds/23268580
33:26 https://travis-ci.org/hugovk/Pillow/builds/23269260

Hmm. That may be partly because the pypy build is 2-3x slower and run last. We're just sitting around waiting for it to finish on the last job-runner. If we run pypy first, we can make better use of the parallel job-runners:

39:48 https://travis-ci.org/hugovk/Pillow/builds/23269381
35:48 https://travis-ci.org/hugovk/Pillow/builds/23269381

Still bad. Most of this is probably from the dependency install and Pillow build overhead at the start of each job.

But hang on! These times are the total CPU time used across all jobs. The actual time from start to finish is much less.

Here's the actual elapsed start-to-end times:

Before: ~12m (about the time of the slowest pypy job)
With env: ~13m
pypy at end: ~10m (about the time of the slowest pypp job)

It uses more CPU time, but us humans get the build results in a bit less time.

Conclusion
---

Should we include this change? Perhaps it'd be better to until #540 has been finalised and test again, as the dependency/build overhead will remain.

At the very least, pypy should come first as that's the slowest and needs a head start. I'll make another PR for that.