progrium/ambassadord — pull request #13 by mikljohansson (2014-12-17 10:25:17+00:00)
Title: Fixed endless loop and added exponential backoff in case of errors

* Fixed endless loop when etcd returns "401: The event in requested index is outdated and cleared". Fix resets the waitIndex on each List()/Get() request.
* Added exponential backoff in configBackends.WatchToUpdate() goroutine to avoid DoS on etcd/consul in event of transient problems. Maximum backoff time is 30s.

Please note that the same problem could be present in the consul backend where a waitIndex is also used in ConsulStore.Watch(). I've got close to zero knowledge about consul and no easy way to test it, but if consul returns errors for too old waitIndexes (like etcd) does it could get stuck in a similar loop.