lytics/metafora — pull request #76 by schmichael (2014-12-08 23:09:25+00:00)
Title: Retry etcd errors

Sorry for sneaking multiple changes into one PR:

* The main fix: Make etcd coordinator handle errors better (since AWS's network is the bane of our existence)
* If the task refresher (`taskmgr`) gets an error trying to update a claim key, it treats the task as "Lost" and stops it.
* If `watch` gets an error from etcd, it can just keep retrying until it succeeds or gets a stop signalled via `ec.stop`.
* If the node refresher gets an error, retry until the TTL deadline is up. If it can't succeed by the deadline, close everything down cleanly.
* **tl;dr** - node refresher is now the canary: everything else retries unless it fails and shuts everything down.
* If no tasks were running `consumer.Shutdown` could exit before `consumer.Run` which lead to `coordinator.Close` not completing and leaving an orphaned etcd node key around.
* Print *local* stack traces on task handler panics.