lytics/metafora — pull request #62 by schmichael (2014-11-04 23:15:42+00:00)
Title: Refresh node dir while running and cleanup on exit

**Please read #61**

Unfortunately this is more complexity, more etcd traffic, *and an interface change.* UGH.

That being said I think it accomplishes two critical goals:

1. Nodes properly cleanup after themselves.
2. (side-effect) Coordinators can fail in such a way that Metafora shuts down cleanly.

2 seems really important as otherwise we could have a metafora daemon running that's completely unable to communicate with the coordinator -- or worse: just the Task Watcher dies but the Command Watcher and Node Dir Refresher happily keep working away. This could allow a cluster to get into a really strange and hard to detect state.

Now when pieces die they should *gracefully* take the whole shebang down with them.