Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): etcd watcher Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): HEAD / commit Environment: Cloud provider or hardware configuration: local VM OS (e.g. from /etc/os-release): Fedora 24 Kernel (e.g. ): Linux localhost.localdomain 4.7.7-200.fc24.x8664 SMP Sat Oct 8 00:21:59 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: N/A Others: N/A What happened: If you do a watch from resource version "0" using (the etcd v2 client code), it initially performs a get of the key in question and returns Added or Modified events, based on a comparison of each node's createdIndex and modifiedIndex. This works all the time. If you do a watch from resource version "0" using (the etcd v3 client code), it attempts to get for each kv returned, in an attempt to provide both the previous and current versions of each kv/object. This works as long as has not been compacted. If compaction has occurred, the initial will fail, and it's not possible to watch from "0"' for this particular key. What you expected to happen: Either: v3 watcher should not change the behavior from the v2 watcher; namely, it should not attempt to get the previous revision of each object and instead just return the current version as a event agree that it's ok to change the behavior, and in the event that the database has been compacted and the previous revision of an object cannot be retrieved, that object is sent as a event instead of . How to reproduce it (as minimally and precisely as possible): I'm working on a unit test and fix. Anything else do we need to know: cc
I will take a look and get back to you tomorrow.
If you have a pr ready, just assign to me or hongchao.
I'll have one tonight hopefully
Although it depends on if we pick option 1 or 2
The behavior of etcd2 is not technically correct. Watch from 0 and will give different results if watch from 0 always return "Added" even if 0 and should be the same. But we do not really have defined what does "0" mean in the interface. Also for etcd3, we cannot guarantee that watch can always return the previous value. Option 1 seems to be easier to achieve. If we do option2, then we will have a "Modified" event without having previous value. I am not sure if that is acceptable or not.
Just give an update here: I talked with on this a bit. We agreed that the behavior should be the same as etcd2 watcher. Of course the API seems non-standard right now. We need to rewrite it in better API once this gets more stable.
Actually, it looks like the event type is a red herring. The etcd2 client has always set Added/Modified based on the equality/inequality of the node's createdindex/modifiedindex: the watch cache always sets the event type to Added when watching from resourceVersion=0 etcd3 (when watching from 0) is still doing a Get() of the previous revision of every KeyValue: The extra get of the previous version is unnecessary if ModRevision == ModCreation, and can fail if the Get returns an ErrCompacted (it also seems like a potential performance problem to fan out N Get() calls during a single watch establishment
potential fix in
I pretty dislike the fact that: on 0 will first do a sync and then start to watch are forced to return prev value for all modification events watch on 0 should just start watch from the latest revision. recovery the initial state (thus the sync or the initial get) should be a separate operation. But, yea, I guess we need to preserve this at least for now.
- yes, I think there is general agreement for it, though we can't change that in v1 API. There is more-than-1-year old issue for that:
For this case, it won't fail. It returns empty result.

Discussion:
isCreated bool } <del> func parseKV(kv *mvccpb.KeyValue, prevVal []byte) *event { </del> <ins> // parseKV converts a KeyValue retrieved from an initial sync() listing to a synthetic isCreated event. func parseKV(kv *mvccpb.KeyValue) *event { </ins> return &event{ key:       string(kv.Key), value:     kv.Value, <del> prevValue: prevVal, </del> <ins> prevValue: nil, </ins> rev:       kv.ModRevision, isDeleted: false, <del> isCreated: kv.ModRevision == kv.CreateRevision, </del> <ins> isCreated: true, </ins> } }

// sync tries to retrieve existing data and send them to process. // The revision to watch will be set to the revision in response. <ins> // All events sent will have isCreated=true </ins> func (wc *watchChan) sync() error { opts := []clientv3.OpOption{} if wc.recursive {

return err } wc.initialRev = getResp.Header.Revision for _, kv := range getResp.Kvs { <del> prevResp, err := wc.watcher.client.Get(wc.ctx, string(kv.Key), clientv3.WithRev(kv.ModRevision-1), clientv3.WithSerializable()) if err != nil { return err } var prevVal []byte if len(prevResp.Kvs) > 0 { prevVal = prevResp.Kvs[0].Value } wc.sendEvent(parseKV(kv, prevVal)) </del> <ins> wc.sendEvent(parseKV(kv)) </ins> } return nil }

"errors" "fmt" "reflect" <ins> "strconv" </ins> "sync" "testing" "time"

// TestWatchFromZero tests that // - watch from 0 should sync up and grab the object added before <ins> // - watch from 0 is able to return events for objects whose previous version has been compacted </ins> // - watch from non-0 should just watch changes after given version func TestWatchFromZero(t *testing.T) { ctx, store, cluster := testSetup(t) defer cluster.Terminate(t) <del> key, storedObj := testPropogateStore(ctx, t, store, &api.Pod{ObjectMeta: api.ObjectMeta{Name: "foo"}}) </del> <ins> key, storedObj := testPropogateStore(ctx, t, store, &api.Pod{ObjectMeta: api.ObjectMeta{Name: "foo", Namespace: "ns"}}) </ins> w, err := store.Watch(ctx, key, "0", storage.Everything) if err != nil { t.Fatalf("Watch failed: %v", err) } testCheckResult(t, 0, watch.Added, w, storedObj) <ins> w.Stop() // Update out := &api.Pod{} err = store.GuaranteedUpdate(ctx, key, out, true, nil, storage.SimpleUpdate( func(runtime.Object) (runtime.Object, error) { return &api.Pod{ObjectMeta: api.ObjectMeta{Name: "foo", Namespace: "ns", Annotations: map[string]string{"a": "1"}}}, nil })) if err != nil { t.Fatalf("GuaranteedUpdate failed: %v", err) } // Make sure when we watch from 0 we receive an ADDED event w, err = store.Watch(ctx, key, "0", storage.Everything) if err != nil { t.Fatalf("Watch failed: %v", err) } testCheckResult(t, 1, watch.Added, w, out) w.Stop() // Update again out = &api.Pod{} err = store.GuaranteedUpdate(ctx, key, out, true, nil, storage.SimpleUpdate( func(runtime.Object) (runtime.Object, error) { return &api.Pod{ObjectMeta: api.ObjectMeta{Name: "foo", Namespace: "ns"}}, nil })) if err != nil { t.Fatalf("GuaranteedUpdate failed: %v", err) } // Compact previous versions revToCompact, err := strconv.Atoi(out.ResourceVersion) if err != nil { t.Fatalf("Error converting %q to an int: %v", storedObj.ResourceVersion, err) } _, err = cluster.RandClient().Compact(ctx, int64(revToCompact), clientv3.WithCompactPhysical()) if err != nil { t.Fatalf("Error compacting: %v", err) } // Make sure we can still watch from 0 and receive an ADDED event w, err = store.Watch(ctx, key, "0", storage.Everything) if err != nil { t.Fatalf("Watch failed: %v", err) } testCheckResult(t, 2, watch.Added, w, out) </ins> } // TestWatchFromNoneZero tests that