Change:
What happened: If you establish a WATCH request on a list of CRs, then edit the CRD's object, you will stop getting MODIFIED events on the WATCH request. The WATCH request doesn't fail or report an error, but will not return more MODIFIED events until the request is restarted. This causes problems for custom controllers, which become unresponsive until their requests expire. What you expected to happen: Either the request should end (so it can be restarted safely) or it should continue to return MODIFIED events. How to reproduce it (as minimally and precisely as possible): I've attached . It does the following: Create CRD and an initial CR. Start a proxy, wait for it to come up, then start a watch request. Change the CR. (This change to the CR results in a MODIFIED event) Change the section of the CRD. Make another change to the CR. (This change to the CR doesn't result in an event). I've tested this against GKE masters on 1.9, 1.10 and 1.11, as well as a kubeadm 1.12.2 cluster. The output looks like: Note that you only see MODIFIED once despite the CR being changed twice. Environment: Kubernetes version (use ): 1.11.2-gke.18, 1.12.2 Cloud provider or hardware configuration: GKE, kubeadm on amd64 OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.15.0-36-generic ~16.04.1-Ubuntu SMP <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc
i suppose it's related w/ yes, b/c we switched to another CR storage when you updated CRD spec.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
fixed in

Discussion:
"k8s.io/apiextensions-apiserver/test/integration/fixtures" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" </ins> "k8s.io/client-go/dynamic" )

} }() <del> // Set up 100 loops creating and reading custom resources </del> <ins> // Set up 100 loops creating and reading and watching custom resources </ins> for i := 0; i < 100; i++ { wg.Add(1) go func(i int) {

time.Sleep(10 * time.Millisecond) } }(i) <ins> wg.Add(1) go func(i int) { defer wg.Done() for { select { case <-stopChan: return default: w, err := noxuNamespacedResourceClient.Watch(metav1.ListOptions{}) if err != nil { t.Fatalf("unexpected error establishing watch: %v", err) } for event := range w.ResultChan() { switch event.Type { case watch.Added, watch.Modified, watch.Deleted: // all expected default: t.Errorf("unexpected watch event: %#v", event) } } } time.Sleep(10 * time.Millisecond) } }(i) </ins> } // Let all the established get request loops soak

close(stopChan) // Let loops drain <del> wg.Wait() </del> <ins> drained := make(chan struct{}) go func() { defer close(drained) wg.Wait() }() select { case <-drained: case <-time.After(wait.ForeverTestTimeout): t.Error("timed out waiting for clients to complete") } </ins> }

cacher.stopWg.Add(1) go func() { defer cacher.stopWg.Done() <ins> defer cacher.terminateAllWatchers() </ins> wait.Until( func() { if !cacher.isStopped() {

} } <ins> func TestCacheWatcherStoppedOnDestroy(t *testing.T) { backingStorage := &dummyStorage{} cacher, _ := newTestCacher(backingStorage, 1000) defer cacher.Stop() // Wait until cacher is initialized. cacher.ready.wait() w, err := cacher.Watch(context.Background(), "pods/ns", "0", storage.Everything) if err != nil { t.Fatalf("Failed to create watch: %v", err) } watchClosed := make(chan struct{}) go func() { defer close(watchClosed) for event := range w.ResultChan() { switch event.Type { case watch.Added, watch.Modified, watch.Deleted: // ok default: t.Errorf("unexpected event %#v", event) } } }() cacher.Stop() select { case <-watchClosed: case <-time.After(wait.ForeverTestTimeout): t.Errorf("timed out waiting for watch to close") } } </ins> func TestTimeBucketWatchersBasic(t *testing.T) { filter := func(_ string, _ labels.Set, _ fields.Set) bool { return true