Change:
// For this test, we'll actually cycle, "list/watch/create/delete" until we get an RV from list that observes the create and not an error. // This way all the tests that are checking for watches don't have to worry about RV too old problems because crazy things *could* happen // before like the created RV could be too old to watch. <del> for _, version := range servedVersions(crd) { err := wait.PollImmediate(500*time.Millisecond, 30*time.Second, func() (bool, error) { return isWatchCachePrimed(crd, dynamicClientSet, version) }) if err != nil { return nil, err } </del> <ins> err = wait.PollImmediate(500*time.Millisecond, 30*time.Second, func() (bool, error) { return isWatchCachePrimed(crd, dynamicClientSet) }) if err != nil { return nil, err </ins> } return crd, nil } <ins> func resourceClientForVersion(crd *apiextensionsv1beta1.CustomResourceDefinition, dynamicClientSet dynamic.Interface, namespace, version string) dynamic.ResourceInterface { gvr := schema.GroupVersionResource{Group: crd.Spec.Group, Version: version, Resource: crd.Spec.Names.Plural} if crd.Spec.Scope != apiextensionsv1beta1.ClusterScoped { return dynamicClientSet.Resource(gvr).Namespace(namespace) } else { return dynamicClientSet.Resource(gvr) } } </ins> // isWatchCachePrimed returns true if the watch is primed for an specified version of CRD watch <del> func isWatchCachePrimed(crd *apiextensionsv1beta1.CustomResourceDefinition, dynamicClientSet dynamic.Interface, version string) (bool, error) { </del> <ins> func isWatchCachePrimed(crd *apiextensionsv1beta1.CustomResourceDefinition, dynamicClientSet dynamic.Interface) (bool, error) { </ins> ns := "" if crd.Spec.Scope != apiextensionsv1beta1.ClusterScoped { ns = "aval" } <del> gvr := schema.GroupVersionResource{Group: crd.Spec.Group, Version: version, Resource: crd.Spec.Names.Plural} var resourceClient dynamic.ResourceInterface if crd.Spec.Scope != apiextensionsv1beta1.ClusterScoped { resourceClient = dynamicClientSet.Resource(gvr).Namespace(ns) } else { resourceClient = dynamicClientSet.Resource(gvr) </del> <ins> versions := servedVersions(crd) if len(versions) == 0 { return true, nil </ins> } <ins> resourceClient := resourceClientForVersion(crd, dynamicClientSet, ns, versions[0]) </ins> instanceName := "setup-instance" instance := &unstructured.Unstructured{ Object: map[string]interface{}{ <del> "apiVersion": crd.Spec.Group + "/" + version, </del> <ins> "apiVersion": crd.Spec.Group + "/" + versions[0], </ins> "kind":       crd.Spec.Names.Kind, "metadata": map[string]interface{}{ "namespace": ns,

Discussion:
both in PR and CI runs see /assign
/assign
Interesting fact: is not flaky, only . The test codepath for these two are almost identical. I wonder if this is a real bug not a flaky test. still investigating.
still seeing flakes like https://k8s- the loop that ensures the watch cache is primed goes version-by-version, and creates and observes the delete event for that version. it does not ensure the delete event is drained from the watch caches for the other versions.
a better approach would be probably be to:
why the poll? Because the watch is not reliable? Otherwise, we have that structure now.
Moves the version loop inside the poll Today we loop over versions and poll per version, which can leave unconsumed delete events in prior versions
Cannot follow. Now we have the following without a poll:
At minimum, I would move the version loop inside the poll so that when we exit we know the deletion events from isWatchCachePrimed are consumed by the watch caches for all versions. If we wanted, we could move the create/delete up out of the isWatchCachePrimed function or pass versions into it to avoid creating/deleting an object per version
Ok, so we were talking about different code. good. To understand: we do a call per version, each creating and deleting an object, and waiting for the delete event with a watch call. a following watch will not see the events of the calls of itself (because they use the same versioned watch cache) and of any version in front of it in the version slice (is this true?), but it might see events of calls of versions later in the version slice. So with your outer poll and inner version loop we would make sure that we drain every version's cache for following rv=0 watch requests. Sounds plausible.
Where do we have the official watch axioms?
I think I know what is the problem here. We make sure the watch is primed for . then we move on to and make sure the watch is primed by creating another object in . The watch cache is drained for but the watch cache for is not and that is the problem here. will send a fix.
yep, same thing I described above. The solution looks good.
I didn't see your comment when I commented. Your description is accurate.