lytics/metafora — pull request #69 by schmichael (2014-11-18 19:06:17+00:00)
Title: etcd coordinator became too resilient

Sadly due to my last changes, metafora would happily continue trying to work even after fatal errors like not being able to update its node key in etcd.

The **desired behavior** when an error talking to etcd occurs is to **gracefully shutdown because another node may be claiming the tasks that used to be owned by the node experiencing connectivity issues.**

Right now Metafora's Consumer will just retry task and command watches on errors.

This change makes the node key refresher (node key ~= `/namespace/nodes/node-name`) the canary. If it fails, it Closes the coordinator which makes ``coordinator.Watch()`` and ``coordinator.Command()`` return their "exited" sentinel values `"", nil` and `nil, nil` respectively.

This in turn lets Metafora's Run() method exit gracefully, shutting down tasks on the way out.

**Consumer changes**

All of this made me realize if `Run()` exits in koalemosd and our own internal consumer, nothing happens! The consumer isn't running, tasks aren't running, but the main entry point will continue waiting for a signal! This fixes that at the expense of making our "Getting Started" boiler plate a bit more verbose.