deis/deis — pull request #1834 by gabrtv (2014-09-14 20:14:01+00:00)
Title: Remove Celery from the Controller

This PR removes Celery asynchronous tasks from the Controller component. The great Celery purge:

* Dramatically simplifies the call stack throughout the controller
* Significantly reduces the memory footprint of the controller
* Fixes database leaks discovered during container FSM transitions
* Stops using `deis/cache` for Celery task synchronization

Cache remains available the controller at `CACHE_URL`, though it is no longer used.

Long-running methods (such as those interacting with the scheduler abstraction) are now performed inline in the model logic, often inside threads. This assumes that 8 workers are sufficient to prevent worker exhaustion with gunicorn `sync` workers. While this is suitable for most cases, we can also:

1. Make number of workers configurable via `etcd`
2. Invest resources in getting `psycopg2` to work with `gevent` (currently just hangs on db operations)

Note, this PR was tested heavily on #1833.