Change:
return false, err } <del> noxuWatch, err := resourceClient.Watch(metav1.ListOptions{ResourceVersion: createdInstance.GetResourceVersion()}) if err != nil { return false, err } defer noxuWatch.Stop() select { case watchEvent := <-noxuWatch.ResultChan(): if watch.Error == watchEvent.Type { return false, nil </del> <ins> // Wait for all versions of watch cache to be primed and also make sure we consumed the DELETE event for all // versions so that any new watch with ResourceVersion=0 does not get those events. This is source of some flaky tests. // When a client creates a watch with resourceVersion=0, it will get an ADD event for any existing objects // but because they specified resourceVersion=0, there is no starting point in the cache buffer to return existing events // from, thus the server will return anything from current head of the cache to the end. By accessing the delete // events for all versions here, we make sure that the head of the cache is passed those events and they will not being // delivered to any future watch with resourceVersion=0. for _, v := range versions { noxuWatch, err := resourceClientForVersion(crd, dynamicClientSet, ns, v).Watch( metav1.ListOptions{ResourceVersion: createdInstance.GetResourceVersion()}) if err != nil { return false, err </ins> } <del> if watch.Deleted != watchEvent.Type { return false, fmt.Errorf("expected DELETE, but got %#v", watchEvent) </del> <ins> defer noxuWatch.Stop() select { case watchEvent := <-noxuWatch.ResultChan(): if watch.Error == watchEvent.Type { return false, nil } if watch.Deleted != watchEvent.Type { return false, fmt.Errorf("expected DELETE, but got %#v", watchEvent) } case <-time.After(5 * time.Second): return false, fmt.Errorf("gave up waiting for watch event") </ins> } <del> return true, nil case <-time.After(5 * time.Second): return false, fmt.Errorf("gave up waiting for watch event") </del> } <ins> return true, nil </ins> } func DeleteCustomResourceDefinition(crd *apiextensionsv1beta1.CustomResourceDefinition, apiExtensionsClient clientset.Interface) error {

Discussion:
both in PR and CI runs see /assign
/assign
Interesting fact: is not flaky, only . The test codepath for these two are almost identical. I wonder if this is a real bug not a flaky test. still investigating.
still seeing flakes like https://k8s- the loop that ensures the watch cache is primed goes version-by-version, and creates and observes the delete event for that version. it does not ensure the delete event is drained from the watch caches for the other versions.
a better approach would be probably be to:
why the poll? Because the watch is not reliable? Otherwise, we have that structure now.
Moves the version loop inside the poll Today we loop over versions and poll per version, which can leave unconsumed delete events in prior versions
Cannot follow. Now we have the following without a poll:
At minimum, I would move the version loop inside the poll so that when we exit we know the deletion events from isWatchCachePrimed are consumed by the watch caches for all versions. If we wanted, we could move the create/delete up out of the isWatchCachePrimed function or pass versions into it to avoid creating/deleting an object per version
Ok, so we were talking about different code. good. To understand: we do a call per version, each creating and deleting an object, and waiting for the delete event with a watch call. a following watch will not see the events of the calls of itself (because they use the same versioned watch cache) and of any version in front of it in the version slice (is this true?), but it might see events of calls of versions later in the version slice. So with your outer poll and inner version loop we would make sure that we drain every version's cache for following rv=0 watch requests. Sounds plausible.
Where do we have the official watch axioms?
I think I know what is the problem here. We make sure the watch is primed for . then we move on to and make sure the watch is primed by creating another object in . The watch cache is drained for but the watch cache for is not and that is the problem here. will send a fix.
yep, same thing I described above. The solution looks good.
I didn't see your comment when I commented. Your description is accurate.