Change:
} } <ins> func TestCacheWatcherStoppedOnDestroy(t *testing.T) { backingStorage := &dummyStorage{} cacher, _ := newTestCacher(backingStorage, 1000) defer cacher.Stop() // Wait until cacher is initialized. cacher.ready.wait() w, err := cacher.Watch(context.Background(), "pods/ns", "0", storage.Everything) if err != nil { t.Fatalf("Failed to create watch: %v", err) } watchClosed := make(chan struct{}) go func() { defer close(watchClosed) for event := range w.ResultChan() { switch event.Type { case watch.Added, watch.Modified, watch.Deleted: // ok default: t.Errorf("unexpected event %#v", event) } } }() cacher.Stop() select { case <-watchClosed: case <-time.After(wait.ForeverTestTimeout): t.Errorf("timed out waiting for watch to close") } } </ins> func TestTimeBucketWatchersBasic(t *testing.T) { filter := func(_ string, _ labels.Set, _ fields.Set) bool { return true

Discussion:
Reproduced on 1.11.7 as well as master-ish () Deleting a CRD doesn't seem to cause the API server to terminate watches referencing the corresponding CRs, but the watches also don't properly resume if the CRD is created again, meaning that you have to wait for watch timeouts to actually get new watch events. We should probably kill open watches on CRD deletion, so that clients are notified and are forced to re-watch. Either that, or seamlessly resume watches if a CRD is recreated. <details</summary</details<details</summary</detailsthe above CRD: the above CR: (elsewhere) and grab a resource version (elsewhere) the lack of watch events After watches time out, things can re-establish, but that can be a while.
/assign
/cc sttts
/area custom-resources
it's sort a related thread w/ . for now CRD's update will terminate CR's watch connection, the other way round we might fix'em together. one possible fix suggested by months ago is that we make the storage to be a runnable object, e.g. by which we can clearily define the lifecycle of CR's storage
fixed in

What happened: If you establish a WATCH request on a list of CRs, then edit the CRD's object, you will stop getting MODIFIED events on the WATCH request. The WATCH request doesn't fail or report an error, but will not return more MODIFIED events until the request is restarted. This causes problems for custom controllers, which become unresponsive until their requests expire. What you expected to happen: Either the request should end (so it can be restarted safely) or it should continue to return MODIFIED events. How to reproduce it (as minimally and precisely as possible): I've attached . It does the following: Create CRD and an initial CR. Start a proxy, wait for it to come up, then start a watch request. Change the CR. (This change to the CR results in a MODIFIED event) Change the section of the CRD. Make another change to the CR. (This change to the CR doesn't result in an event). I've tested this against GKE masters on 1.9, 1.10 and 1.11, as well as a kubeadm 1.12.2 cluster. The output looks like: Note that you only see MODIFIED once despite the CR being changed twice. Environment: Kubernetes version (use ): 1.11.2-gke.18, 1.12.2 Cloud provider or hardware configuration: GKE, kubeadm on amd64 OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.15.0-36-generic ~16.04.1-Ubuntu SMP <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc
i suppose it's related w/ yes, b/c we switched to another CR storage when you updated CRD spec.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
fixed in