Add runner.cleanup on runner creation

Runners that are directly created need to run their cleanup steps.
Cleanup on failure in test

By default, there is no cleanup on failure.
Merge pull request #936 from Yelp/test_runner_cleanup

Test runner cleanup