Change:
Opening as an follow up analysis from Merging unexpectedly resulted in a huge flakiness of at least one of the apimachinery tests: That is definitely unexpected, because that PR only changes the size of watchcache, which no e2e tests should rely on. So I started digging deeper into it and indeed that PR also uncovered the bug that was introduced earlier. Here is what is happening: made the watch cache size dynamic however, there was a place in watch cache that relied on the following invariant: "when watchache becomes full, it is full until the next relist" that place is this one: with the change that made watchcache size dynamic, it may no longer be true, i.e.: (a) watch cache was full (b) we are increasing its size (c) it is no longer full, but we no longer can't serve (at least not always) everything since listResourceVersion This is exactly what was happening in the failed tests: the service was updated well back in the past the watchcache initially rotated couple times (no need to increase it) [which pulled our change out of it] the its size was increased that resulted in cache becoming not full so the incoming watch should have returned "too old", but was accepting it (due to the listResourceVersion 0 condition being true), but in fact wasn't able to serve it Bonus point: in "kind" tests we were observing tests taking 30+ minutes - this seem to be a bug in kube-apiserver configuration, because the watch in the test doesn't set TimeoutSeconds, and: (a) in GCE it is default in kube-apiserver to sth random from [5m, 30m]: (b) in kind apparently it isn't... /kind bug /sig apimachinery /assign - FYI
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Opening as an follow up analysis from That is definitely unexpected, because that PR only changes the size of watchcache, which no e2e tests should rely on. made the watch cache size dynamic however, there was a place in watch cache that relied on the following invariant: the service was updated well back in the past the watchcache initially rotated couple times (no need to increase it) [which pulled our change out of it] the its size was increased that resulted in cache becoming not full so the incoming watch should have returned "too old", but was accepting it (due to the listResourceVersion 0 condition being true), but in fact wasn't able to serve it Bonus point: in "kind" tests we were observing tests taking 30+ minutes - this seem to be a bug in kube-apiserver configuration, because the watch in the test doesn't set TimeoutSeconds, and: (a) in GCE it is default in kube-apiserver to sth random from [5m, 30m]: (b) in kind apparently it isn't... /kind bug /sig apimachinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Working on the fix right now
for now contains a test reproducing the issue Fix coming soon.
is ready for review
it seems TimeoutSeconds is configured explicitely on GCE tests to 300 sec KIND is using Kubeadm under the hood and I could not find any place where those flags are configured, but then I think it should be taken the default values for TimeoutSeconds, no? EDIT Actually, it seems that it does take the default, and min-request-timeout takes precedence, that means it takes 1800s it seems we should set this apiserver option to a lower value, should we do it in KIND itself?
I would want to speak to the kubeadm team first, we avoid deviating from their defaults as much as possible so we can test them. cc
i see the tests under "client-go should negotiate" are not conformance yet, which makes them less likely to be run outside of k8s testing. i personally don't think our tools used for e2e (GCE/kube-up included) should be configuring these kube-apiserver flags. the tests should behave well with any deployer-X that relies on the apiserver sane defaults.

Discussion:
size := w.endIndex - w.startIndex var oldest uint64 switch { <del> case size >= w.capacity: // Once the watch event buffer is full, the oldest watch event we can deliver // is the first one in the buffer. oldest = w.cache[w.startIndex%w.capacity].ResourceVersion case w.listResourceVersion > 0: // If the watch event buffer isn't full, the oldest watch event we can deliver // is one greater than the resource version of the last full list. </del> <ins> case w.listResourceVersion > 0 && w.startIndex == 0: // If no event was removed from the buffer since last relist, the oldest watch // event we can deliver is one greater than the resource version of the list. </ins> oldest = w.listResourceVersion + 1 case size > 0: <del> // If we've never completed a list, use the resourceVersion of the oldest event // in the buffer. // This should only happen in unit tests that populate the buffer without // performing list/replace operations. </del> <ins> // If the previous condition is not satisfied: either some event was already // removed from the buffer or we've never completed a list (the latter can // only happen in unit tests that populate the buffer without performing // list/replace operations), the oldest watch event we can deliver is the first // one in the buffer. </ins> oldest = w.cache[w.startIndex%w.capacity].ResourceVersion default: return nil, fmt.Errorf("watch cache isn't correctly initialized")

return true } <ins> func TestCacheIncreaseDoesNotBreakWatch(t *testing.T) { store := newTestWatchCache(2, &cache.Indexers{}) now := store.clock.Now() addEvent := func(key string, rv uint64, t time.Time) { event := &watchCacheEvent{ Key:             key, ResourceVersion: rv, RecordTime:      t, } store.updateCache(event) } // Initial LIST comes from the moment of RV=10. store.Replace(nil, "10") addEvent("key1", 20, now) // Force "key1" to rotate our of cache. later := now.Add(2 * eventFreshDuration) addEvent("key2", 30, later) addEvent("key3", 40, later) // Force cache resize. addEvent("key4", 50, later.Add(time.Second)) _, err := store.GetAllEventsSince(15) if err == nil || !strings.Contains(err.Error(), "too old resource version") { t.Errorf("unexpected error: %v", err) } } </ins> func BenchmarkWatchCache_updateCache(b *testing.B) { store := newTestWatchCache(defaultUpperBoundCapacity, &cache.Indexers{}) store.cache = store.cache[:0]