GoogleCloudPlatform/kubernetes — pull request #1090 by bcwaldon (2014-08-28 20:47:32+00:00)
Title: Start kubelet watch index at zero

1. Start up a cluster
2. Wait a long enough time for 1000+ etcd writes to happen in your cluster outside of the kubernetes keyspace
3. Restart a kubelet

The kubelet will repeatedly log something like this:

```
I0828 16:29:06.188275 00008 logs.go:39] etcd DEBUG: recv.response.fromhttp://127.0.0.1:4001/v2/keys/registry/hosts/10.141.164.134/kubelet?consistent=true&wait=true&waitIndex=253363
I0828 16:29:06.188435 00008 logs.go:39] etcd DEBUG: recv.success.http://127.0.0.1:4001/v2/keys/registry/hosts/10.141.164.134/kubelet?consistent=true&wait=true&waitIndex=253363
E0828 16:29:06.188933 00008 etcd.go:69] Unable to extract from the response (/registry/hosts/10.141.164.134/kubelet): %v%!(EXTRA *etcd.EtcdError=401: The event in requested index is outdated and cleared (the requested history has been cleared [254788/253363]) [255787])
```

This etcd error occurs when a user attempts to establish an watch with an index that is 1000 or more behind the current cluster-wide index. Here's what the kubelet is currently doing:

1. on startup, GET the kubelet config without `wait=true`
2. establish a watcher on the kubelet config key with `wait=true` and `waitIndex` equal to the ModifiedIndex of the etcd Node returned in step 1

The ModifiedIndex ends up being more than 1000 behind the current cluster index and the watch immediately fails. To work around this, we should just start the watcher using `waitIndex=0`.