Change:
} return false } <ins> // WatchEventSequenceVerifier ... // manages a watch for a given resource, ensures that events take place in a given order, retries the test on failure //   testContext         cancelation signal across API boundries, e.g: context.TODO() //   dc                  sets up a client to the API //   resourceType        specify the type of resource //   namespace           select a namespace //   resourceName        the name of the given resource //   listOptions         options used to find the resource, recommended to use listOptions.labelSelector //   expectedWatchEvents array of events which are expected to occur //   scenario            the function to run func WatchEventSequenceVerifier(testContext context.Context, dc dynamic.Interface, resourceType schema.GroupVersionResource, namespace string, resourceName string, listOptions metav1.ListOptions, expectedWatchEvents []watch.Event, scenario func(*watchtools.RetryWatcher) []watch.Event) { listWatcher := &cache.ListWatch{ WatchFunc: func(listOptions metav1.ListOptions) (watch.Interface, error) { return dc.Resource(resourceType).Namespace(namespace).Watch(testContext, listOptions) }, } resourceWatch, err := watchtools.NewRetryWatcher("1", listWatcher) ExpectNoError(err, "Failed to create a resource watch of %v in namespace %v", resourceType.Resource, namespace) // NOTE value of 3 retries seems to make sense retries := 3 retriesLoop: for try := 1; try <= retries; try++ { // NOTE the test may need access to the events to see what's going on, such as a change in status actualWatchEvents := scenario(resourceWatch) errs := sets.NewString() // watchEventsLoop: totalValidWatchEvents := 0 actualWatchEventsHasDelete := false for watchEventIndex := range expectedWatchEvents { foundExpectedWatchEvent := false ExpectEqual(len(expectedWatchEvents) <= len(actualWatchEvents), true, "Error: actual watch events amount must be greater than or equal to expected watch events amount") for actualWatchEventIndex := range actualWatchEvents { if actualWatchEvents[watchEventIndex].Type == expectedWatchEvents[actualWatchEventIndex].Type { foundExpectedWatchEvent = true } if actualWatchEvents[actualWatchEventIndex].Type == watch.Deleted && actualWatchEventsHasDelete != true { actualWatchEventsHasDelete = true } } if foundExpectedWatchEvent == false { errs.Insert(fmt.Sprintf("Watch event %v not found", expectedWatchEvents[watchEventIndex].Type)) } totalValidWatchEvents++ } if actualWatchEventsHasDelete == false { _ = dc.Resource(resourceType).Namespace(namespace).DeleteCollection(testContext, metav1.DeleteOptions{}, listOptions) } // TODO restructure failures handling if errs.Len() > 0 && try < retries { fmt.Println(fmt.Errorf("invariants violated:n* %s", strings.Join(errs.List(), "n* "))) continue retriesLoop } ExpectEqual(errs.Len() > 0, false, fmt.Errorf("invariants violated:n* %s", strings.Join(errs.List(), "n* "))) ExpectEqual(totalValidWatchEvents, len(expectedWatchEvents), "Error: there must be an equal amount of total valid watch events (%v) and expected watch events (%v)", totalValidWatchEvents, len(expectedWatchEvents)) } } </ins>

Discussion:
This Issue replace Watch events may be missed and the test may still pass. We need to ensure that all expected watch events are seen. Watch events are not collected and checked at the end of the test. In order to solve the problem, we must collect all watch events as they come in and perform a check at the end of the test to ensure that the order is correct and they contain no errors.
/sig testing /area conformance /assign /cc
This Issue replace
AI: Must allow for other events to occur through the expected watch events stream expected watch events: must allow the following to be valid:
[x] Set up a watch for resource through a static label to the resource with DynamicClient [x] Manage the watch if it dies [x] Feed the watch to the function [x] Read all the watch events from the at the end of the test, ensuring that the events occur question: What does doing this mean to tests which check data? (e.g: - waits for values, sourced from watch) [x] Retry the test, upon failure [x] Manage the resources on retry (delete)
More or less, yes. If we need to read results mid-test, I think that will be OK, rather than waiting for the end. The main point is to wrap the watch such that it gets restarted if it fails. That wrapper will need to sit between the raw watch channel and the test: raw watch -channel -wrapper (restarts watch if closes/fails prematurely) -channel consumed in test I'm a little surprised there isn't already a sort of "buffered watch" object that does this already ? You'll also need a stop channel for the go routine started by the wrapper, so that at the end of the test you can stop that go routine and watch.
Also, if we have the "protected watch" wrapper, not sure we need to retry the test on failure. AFAIU, the only reason we wouldn't see the watch events we expected would be because of etcd compaction, which would not be expected to happen in e2e clusters, and even if it did it would not happen to event histories that are brand new (i.e., it only compacts old history). Again would like to confirm.
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details