Code change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): watch leak Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 0 Environment: Cloud provider or hardware configuration: vagrant on Mac OS X OS (e.g. from /etc/os-release): CoreOS 1164.1.0 Kernel (e.g. ): Install tools: Others: What happened: We ran controller which handles our own ThirdPartyResource. When we hit "watch of ... ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared" error, "forcing resync" log is duplicated. This log message comes from . It looks like this goroutine does not exit when we hit the above mentioned error, and continued to work, and running "resync" more frequently. What you expected to happen: The resync goroutine must be terminated when watch is closed (i.e., ListAndWatch function returned), instead of leaking it. How to reproduce it (as minimally and precisely as possible): I just run our controller for ~15 minutes. I'm not sure why "401: The event in requested index is outdated and cleared (the requested history has been cleared" happens, but from the issues I read, it should be harmless. Anything else do we need to know:
I think I see what is going on here. If exits for any reason beyond an error from the resync loop no one will tell the resync loop to exit. Furthermore, there is no "local" way to do this as it would be unexpected for this function to signal the Done channel as other things outside of it may be relying on it. There are some useful functions around Context () that help to create a "local" copy of context that can be used in this situation.
Thank you for reply. Isn't is possible to create local channel in ListAndWatch, and defer close it, and then make resync goroutine "select" it too?
This issue was labelled as component/kubectl, but actually it is not limited to kubectl. All controllers suffer from this. Only people notice this if they enable V(4) logging.
The really annoying thing caused by this issue is that eventually resync is done in every few seconds due to the leaked goroutine. Even if you set 30 second resync period in controller, leaked goroutines call resync too, and you end up having resync call very very frequently. If you have many resources in cache, it really performs unnecessary sync loop. This is the last thing you want.
this looks like a real issue. Could be related to Could you help triage? looks like a proper fix.
I agree - this looks like a real issue. And indeed LGTM as a fix for it.

Related discussion:
text: r.setLastSyncResourceVersion(resourceVersion) resyncerrc := make(chan error, 1) <ins> cancelCh := make(chan struct{}) defer close(cancelCh) </ins> go func() { for { select { case <-resyncCh: case <-stopCh: return <ins> case <-cancelCh: return </ins> } glog.V(4).Infof("%s: forcing resync", r.name) if err := r.store.Resync(); err != nil {