flynn/flynn-controller — pull request #22 by lmars (2014-06-02 15:34:23+00:00)
Title: Flap detection

This is an initial attempt at a restart back-off policy when jobs crash:

* If the job is new (i.e. was created by a new / updated formation) restart it immediately and increment its restart count
* If the job was started more than 10 minutes ago, reset its restart count to zero and restart it immediately
* If the job was started less than 10 minutes ago, wait 10 * 2 ^ (restart count - 1) minutes before restarting it

I have not added a test for this, but I have tried it out manually by switching the `backoffPeriod` to 30 seconds, manually killing jobs and seeing they are rescheduled as expected, the results of which can be seen in the following gist:

https://gist.github.com/lmars/a54a791e1434a0a311a8

Things that need addressing:

* Job restarts are not visible in the controller, so whilst the scheduler knows that jobs are queued to be restarted in the future, the controller will report that not enough jobs are running for a given formation
* When scaling down a formation, queued jobs should be removed first by just stopping their timer
* There is no upper limit on restart attempts

This depends on https://github.com/flynn/flynn-host/pull/17