deis/deis — pull request #1913 by gabrtv (2014-09-25 16:43:15+00:00)
Title: Chaos test framework with improved error handling

This PR greatly improves error handling during the controller's interactions with the scheduler. In order to prove out the desired behavior and ensure we handle errors reliably, we now simulate random failures in a mock `chaos` scheduler. Specifically, this PR includes:

* New mock `chaos` scheduler with random error rates configurable for each scheduler API method
* Controller HTTP endpoints that interact with the scheduler now return inline HTTP 503 response on errors
* `deis logs` output now includes more detailed feedback on container errors and warnings
* Deploys of new releases are now truly zero-downtime and automatically rollback on any scheduling errors
* Errors during the creation of new releases (build or config changes) automatically rollback all changes
* Updated FSM module while adding support for swallowing exceptions `on_error` (kmmbvnr/django-fsm#59)
* New FSM states including `ERROR` and `CRASHED` to more closely model Heroku

Most of these changes are the result of TDD-based refactoring. Changes are mainly under-the-hood of the controller. As a result, there are no _specific_ testing instructions.