puppetlabs-operations/puppet-puppet — pull request #123 by danieldreier (2014-11-15 22:57:13+00:00)
Title: Refactor acceptance tests

The first iteration of acceptance tests lumped all tests into one big run on a single VM, which caused problems because the cleanup tasks were imperfect. For example, a failed cleanup could leave a process listening on port 8140, which caused the subsequent test to fail incorrectly.

This moves the tests into separate files and tags them, so that rspec tags can be used to run an more limited set of tests (e.g. just unicorn, just passenger, just the agent tests, etc). I've also removed the cleanup tasks because they are no longer necessary and I don't think we should support both approaches to testing.

The advantage here is no chance of bleed-over between test cases, whereas the previous way required imperfect automated cleanup between test cases. There are two major downsides: first, comprehensive testing of all cases requires considerably longer because a new VM must be provisioned for each case. Running all tests will take ~1.5 hours instead of ~20 minutes. Second, tests can no longer be run via a convenient rake task. The new syntax has been documented in `CONTRIBUTING.md but` it's less obvious.

I believe the tradeoff is acceptance because this is a step toward running reliable, trustable integration tests via jenkins (or similar). In that case, the tests can be parallelized by test case and OS. Given sufficient parallelization, this approach will run faster than the more linear approach, because each individual test is faster.

I'm planning on improving the quality of test coverage as well, but this is enough change for one PR.