TechEmpower/FrameworkBenchmarks — pull request #939 by hamiltont (2014-07-24 08:27:02+00:00)
Title: Setup continuous verification using travis-ci

This uses [Travis](https://travis-ci.org/) to automatically run a "verify" on each modified framework when new commits are pushed or pull requests are opened. Each framework gets a clean VM for this verify. It's very useful, so I have made this hotfix that can be quickly applied to master. However, this will become much more useful when we get something like #920 merged into master as well so that all subprocess error/output logs are displayed on the console. For a full example including #920, see [here](https://travis-ci.org/hamiltont/FrameworkBenchmarks/builds/30719645).

The required `toolset/` modifications are very minor - are a number of return statements indicating success or failure. TechEmpower will need to link their github account with Travis-CI and update the .travis.yml to include a github token.

This makes a best-effort to avoid wasting travis' resources by examining the commits and only running verify tests on a framework when the commits have resulted in changes to that framework. Examples:
* Modify `toolset`: Triggers verification of every framework. Takes 2-3 hours
* Modify one framework e.g. `beego`: Triggers verification of just that one. Takes about 5 minutes
* Modify file in root: Triggers nothing. Takes about 5 minutes for the build to be fully canceled

No promises that this works perfectly for pull requests at the moment, it's my first time doing non-trivial travis-ci stuff

### All the tests are failing!

[This](https://travis-ci.org/hamiltont/FrameworkBenchmarks/builds/30719645) shows a huge number of failing tests. I've spot checked this enough to be fairly confident this is not the result of this PR, but obviously this is up for comment. A number appear to be very minor issues that have gone unnoticed

### Travis Limits

Travis allows a hard limit of 50 minutes per job (each push to github triggers a new "build", and each "build" contains multiple "jobs", one per test folder for us) and 4 megabytes of console output (only relevant because #920 is quite verbose...I'll need to fix that). We are well within the 50 minutes per job - verifying a single framework takes about 5-10 minutes. Neither of these limits impose any problem.

Travis allows a small number of concurrent "build"s based on their workload - this is typically 1 or 2, and is the main limiting factor for this project, because one push that triggers a slow build will delay all future builds. One change to `toolset/` will trigger ~100 "jobs" to be completed before other builds can be run, which delays other builds for 2-3 hours :-( You can use the flag "[ci skip]" anywhere inside of your commit message to not run travis, which avoids this if you want to modify `toolset/` without triggering a full rebuild.

I would really prefer a better approach than just checking for modifications inside `toolset/`, because right now just adding a comment anywhere inside `toolset/` will trigger a full re-verification and delay all later builds. Any thoughts on how to improve this are appreciated. I've considered just spot-checking a few frameworks (which ones?), and searching the commit message for something like "[tfb fullbuild]" to trigger a full build.

**Note:** Ok, I lied a bit. This doesn't verify every singe framework test. It only verifies one test per directory. Specifically it verifies the first one listed in the benchmark_config. I know that's not perfect, but it's good enough ATM