Change:
} return false } <ins> // WatchEventSequenceVerifier ... // manages a watch for a given resource, ensures that events take place in a given order, retries the test on failure //   testContext         cancelation signal across API boundries, e.g: context.TODO() //   dc                  sets up a client to the API //   resourceType        specify the type of resource //   namespace           select a namespace //   resourceName        the name of the given resource //   listOptions         options used to find the resource, recommended to use listOptions.labelSelector //   expectedWatchEvents array of events which are expected to occur //   scenario            the function to run func WatchEventSequenceVerifier(testContext context.Context, dc dynamic.Interface, resourceType schema.GroupVersionResource, namespace string, resourceName string, listOptions metav1.ListOptions, expectedWatchEvents []watch.Event, scenario func(*watchtools.RetryWatcher) []watch.Event) { listWatcher := &cache.ListWatch{ WatchFunc: func(listOptions metav1.ListOptions) (watch.Interface, error) { return dc.Resource(resourceType).Namespace(namespace).Watch(testContext, listOptions) }, } resourceWatch, err := watchtools.NewRetryWatcher("1", listWatcher) ExpectNoError(err, "Failed to create a resource watch of %v in namespace %v", resourceType.Resource, namespace) // NOTE value of 3 retries seems to make sense retries := 3 retriesLoop: for try := 1; try <= retries; try++ { // NOTE the test may need access to the events to see what's going on, such as a change in status actualWatchEvents := scenario(resourceWatch) errs := sets.NewString() // watchEventsLoop: totalValidWatchEvents := 0 actualWatchEventsHasDelete := false for watchEventIndex := range expectedWatchEvents { foundExpectedWatchEvent := false ExpectEqual(len(expectedWatchEvents) <= len(actualWatchEvents), true, "Error: actual watch events amount must be greater than or equal to expected watch events amount") for actualWatchEventIndex := range actualWatchEvents { if actualWatchEvents[watchEventIndex].Type == expectedWatchEvents[actualWatchEventIndex].Type { foundExpectedWatchEvent = true } if actualWatchEvents[actualWatchEventIndex].Type == watch.Deleted && actualWatchEventsHasDelete != true { actualWatchEventsHasDelete = true } } if foundExpectedWatchEvent == false { errs.Insert(fmt.Sprintf("Watch event %v not found", expectedWatchEvents[watchEventIndex].Type)) } totalValidWatchEvents++ } if actualWatchEventsHasDelete == false { _ = dc.Resource(resourceType).Namespace(namespace).DeleteCollection(testContext, metav1.DeleteOptions{}, listOptions) } // TODO restructure failures handling if errs.Len() > 0 && try < retries { fmt.Println(fmt.Errorf("invariants violated:n* %s", strings.Join(errs.List(), "n* "))) continue retriesLoop } ExpectEqual(errs.Len() > 0, false, fmt.Errorf("invariants violated:n* %s", strings.Join(errs.List(), "n* "))) ExpectEqual(totalValidWatchEvents, len(expectedWatchEvents), "Error: there must be an equal amount of total valid watch events (%v) and expected watch events (%v)", totalValidWatchEvents, len(expectedWatchEvents)) } } </ins>

Discussion:
This Issue replace Watch events may be missed and the test may still pass. We need to ensure that all expected watch events are seen. Watch events are not collected and checked at the end of the test. In order to solve the problem, we must collect all watch events as they come in and perform a check at the end of the test to ensure that the order is correct and they contain no errors.
/sig testing /area conformance /assign /cc
This Issue replace
AI: Must allow for other events to occur through the expected watch events stream expected watch events: must allow the following to be valid:
[x] Set up a watch for resource through a static label to the resource with DynamicClient [x] Manage the watch if it dies [x] Feed the watch to the function [x] Read all the watch events from the at the end of the test, ensuring that the events occur question: What does doing this mean to tests which check data? (e.g: - waits for values, sourced from watch) [x] Retry the test, upon failure [x] Manage the resources on retry (delete)
More or less, yes. If we need to read results mid-test, I think that will be OK, rather than waiting for the end. The main point is to wrap the watch such that it gets restarted if it fails. That wrapper will need to sit between the raw watch channel and the test: raw watch -channel -wrapper (restarts watch if closes/fails prematurely) -channel consumed in test I'm a little surprised there isn't already a sort of "buffered watch" object that does this already ? You'll also need a stop channel for the go routine started by the wrapper, so that at the end of the test you can stop that go routine and watch.
Also, if we have the "protected watch" wrapper, not sure we need to retry the test on failure. AFAIU, the only reason we wouldn't see the watch events we expected would be because of etcd compaction, which would not be expected to happen in e2e clusters, and even if it did it would not happen to event histories that are brand new (i.e., it only compacts old history). Again would like to confirm.
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

[x] Original approval issue: [x] Two weeks soak end date: 8 May 2020 [x] Two weeks soak end date: 23 June 2020 This issue is created to allow edit by /wip /hold This issue proposes the writing of a new test for the API endpoints , with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this APIsnoop query, there are still some remaining ConfigMap endpoints which are untested. a ConfigMap with a static label the ConfigMap with a new Label and updated data the ConfigMap to ensure it's patched all ConfigMaps in all Namespaces find the ConfigMap(1) ensure that the ConfigMap is found and is patched Namespaced ConfigMap(1) via a Collection with a LabelSelector This mock test uses client-go's standard functions for talking to the Event area of the API. Discover useragents: Display endpoints hit by the test: Show the change in coverage after the mock test: If a test with these calls gets merged, Conformance coverage will go up by 2 points
/sig testing /sig architecture /area conformance
/assign Issue created to replace
GitHub didn't allow me to assign the following users: raymonddeng99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen On hold & soaking
Reopened this issue. <detailsIn response to : /reopen On hold & soaking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details