Resilient watches

Got some help in #coreos to find out what to do about 401 errors. Turns out the answer is to restart the watch with a fresh Get. That ensures metafora will never miss events.

The diff looks like an absolute mess. Basically all I did was put *another* for loop around the initial Get when we start watching, and have 401s continue that outer loop.


bleh
No clue sadly. Pure guesswork at this point. Despite the misleading name this is the tick time: every TTL - 5 seconds we refresh the TTL. etcd's client does some retrying on network errors internally, hence the 5 seconds of leeway.

This is another one of the magic numbers I don't think any of us know the exact right answer for and whether or not it should be configurable.