Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I am running etcd-operator in kubernetes, and noticed that when I deleted the etcdcluster (Custom resource definition) the pods & services did not get cleaned up. I looked on another kube environment I had running, and on that env everything worked as expected. I checked logs, and could see on the env that was working I saw the following messages in Controller manager logs: I didn't see these in the env that wasn't working, so I started looking at the garbage collector on there. I noticed that there hadn't been any log message from garbage collecor since 31st Jan, when this was logged: Around this time we experienced some environmental network issues, which caused lots of errors around that time, such as: It looks like these issues may have caused the garbage collector to get into some sort of hung state? When I restarted controller manager everything started working as expected. What you expected to happen: Garbage collector should have kept running, and working as expected. How to reproduce it (as minimally and precisely as possible): Not sure on the exact sequence that caused it, but suspect it was related to a time period when errors were occurring. Anything else we need to know?: We have set, and had previously hit this garbage collector issue - However, we are running with Kube 1.8.6 with the fix for that issue cherry-picked (commit ) Environment: Kubernetes version (use ):Cloud provider or hardware configuration: IBM Cloud OS (e.g. from /etc/os-release): Ubuntu "16.04.3 LTS (Xenial Xerus)" Kernel (e.g. ): 0-78-generic -Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: I will attach logs from Controller manager.
/sig api-machinery
/assign /assign
cc
I saw garbage collector stopped working in 1.9.2 too. Most controllers (such as node controllers or deployment controllers) stopped reporting logs in controller- when this happened. Manually restarting controller-manager fixed the problem. I think somehow some controllers got stuck (could it be because of some bugs in watch cache?) and it's not specific to garbage collector. When the issue happened, still reported healthy for controller-manager.
tracks this
This is happening in production for us too.
did this happen to you to? Or is it that only the garbage collector disappeared from the logs?
For me the other controllers still have logs; it was just the GC that crashed.
does your cluster also use custom resources backed by Aggregated API servers?
Yes, although this happened in one cluster where we don't have any resources yet (the type exists). I'm not sure what that means
I still see messages from and and after the garbage collector stops working - so looks like some controllers are still working. Is there anything specific you were looking for?
I think the cause of this garbage collector hanging here is this: function periodically calls to the comment above GetDeletableResources, , an error in discovery causes the discovery client to no longer discover resources in the cluster, but instead of failing and returning an error, it simply logs the error as and returns an empty list of resources Sync function, upon recieving an empty resource list from discovery, detects that the resources have changed, and calls , which calls with as the argument as shown in the log as , which sets the list of monitors to an empty list because it thinks there are no resources to monitor. the Sync function calls controller.WaitForCacheSync, which calls , which will continually retry the function until it returns true, but it will always return false because is 0. So I think this was caused by which enabled number 2.

Discussion:
"//vendor/k8s.io/client-go/discovery:go_default_library", "//vendor/k8s.io/client-go/dynamic:go_default_library", "//vendor/k8s.io/client-go/informers:go_default_library", <ins> "//vendor/k8s.io/client-go/kubernetes:go_default_library", </ins> "//vendor/k8s.io/client-go/kubernetes/fake:go_default_library", "//vendor/k8s.io/client-go/rest:go_default_library", "//vendor/k8s.io/client-go/util/workqueue:go_default_library",

// Note that discoveryClient should NOT be shared with gc.restMapper, otherwise // the mapper's underlying discovery client will be unnecessarily reset during // the course of detecting new resources. <del> func (gc *GarbageCollector) Sync(discoveryClient discovery.DiscoveryInterface, period time.Duration, stopCh <-chan struct{}) { </del> <ins> func (gc *GarbageCollector) Sync(discoveryClient discovery.ServerResourcesInterface, period time.Duration, stopCh <-chan struct{}) { </ins> oldResources := make(map[schema.GroupVersionResource]struct{}) wait.Until(func() { // Get the current resource list from discovery. newResources := GetDeletableResources(discoveryClient) <ins> // This can occur if there is an internal error in GetDeletableResources. // If the gc attempts to sync with 0 resources it will block forever. // TODO: Implement a more complete solution for the garbage collector hanging. if len(newResources) == 0 { glog.V(5).Infof("no resources reported by discovery, skipping garbage collector sync") return } </ins> // Decide whether discovery has reported a change. if reflect.DeepEqual(oldResources, newResources) { glog.V(5).Infof("no resource updates from discovery, skipping garbage collector sync")

"strings" "sync" "testing" <ins> "time" </ins> "github.com/stretchr/testify/assert"

"k8s.io/client-go/discovery" "k8s.io/client-go/dynamic" "k8s.io/client-go/informers" <ins> "k8s.io/client-go/kubernetes" </ins> "k8s.io/client-go/kubernetes/fake" restclient "k8s.io/client-go/rest" "k8s.io/client-go/util/workqueue"

// ServeHTTP logs the action that occurred and always returns the associated status code func (f *fakeActionHandler) ServeHTTP(response http.ResponseWriter, request *http.Request) { <del> f.lock.Lock() defer f.lock.Unlock() </del> <ins> func() { f.lock.Lock() defer f.lock.Unlock() f.actions = append(f.actions, fakeAction{method: request.Method, path: request.URL.Path, query: request.URL.RawQuery}) fakeResponse, ok := f.response[request.Method+request.URL.Path] if !ok { fakeResponse.statusCode = 200 fakeResponse.content = []byte("{"kind": "List"}") } response.Header().Set("Content-Type", "application/json") response.WriteHeader(fakeResponse.statusCode) response.Write(fakeResponse.content) }() </ins> <del> f.actions = append(f.actions, fakeAction{method: request.Method, path: request.URL.Path, query: request.URL.RawQuery}) fakeResponse, ok := f.response[request.Method+request.URL.Path] if !ok { fakeResponse.statusCode = 200 fakeResponse.content = []byte("{"kind": "List"}") </del> <ins> // This is to allow the fakeActionHandler to simulate a watch being opened if strings.Contains(request.URL.RawQuery, "watch=true") { hijacker, ok := response.(http.Hijacker) if !ok { return } connection, _, err := hijacker.Hijack() if err != nil { return } defer connection.Close() time.Sleep(30 * time.Second) </ins> } <del> response.Header().Set("Content-Type", "application/json") response.WriteHeader(fakeResponse.statusCode) response.Write(fakeResponse.content) </del> } // testServerAndClientConfig returns a server that listens and a config that can reference it

} } <ins> // TestGarbageCollectorSync ensures that a discovery client error // will not cause the garbage collector to block infinitely. func TestGarbageCollectorSync(t *testing.T) { serverResources := []*metav1.APIResourceList{ { GroupVersion: "v1", APIResources: []metav1.APIResource{ {Name: "pods", Namespaced: true, Kind: "Pod", Verbs: metav1.Verbs{"delete", "list", "watch"}}, }, }, } fakeDiscoveryClient := &fakeServerResources{ PreferredResources: serverResources, Error:              nil, Lock:               sync.Mutex{}, InterfaceUsedCount: 0, } testHandler := &fakeActionHandler{ response: map[string]FakeResponse{ "GET" + "/api/v1/pods": { 200, []byte("{}"), }, }, } srv, clientConfig := testServerAndClientConfig(testHandler.ServeHTTP) defer srv.Close() clientConfig.ContentConfig.NegotiatedSerializer = nil client, err := kubernetes.NewForConfig(clientConfig) if err != nil { t.Fatal(err) } rm := &testRESTMapper{legacyscheme.Registry.RESTMapper()} metaOnlyClientPool := dynamic.NewClientPool(clientConfig, rm, dynamic.LegacyAPIPathResolverFunc) clientPool := dynamic.NewClientPool(clientConfig, rm, dynamic.LegacyAPIPathResolverFunc) podResource := map[schema.GroupVersionResource]struct{}{ {Group: "", Version: "v1", Resource: "pods"}: {}, } sharedInformers := informers.NewSharedInformerFactory(client, 0) alwaysStarted := make(chan struct{}) close(alwaysStarted) gc, err := NewGarbageCollector(metaOnlyClientPool, clientPool, rm, podResource, map[schema.GroupResource]struct{}{}, sharedInformers, alwaysStarted) if err != nil { t.Fatal(err) } stopCh := make(chan struct{}) defer close(stopCh) go gc.Run(1, stopCh) go gc.Sync(fakeDiscoveryClient, 10*time.Millisecond, stopCh) // Wait until the sync discovers the initial resources fmt.Printf("Test output") time.Sleep(1 * time.Second) err = expectSyncNotBlocked(fakeDiscoveryClient) if err != nil { t.Fatalf("Expected garbagecollector.Sync to be running but it is blocked: %v", err) } // Simulate the discovery client returning an error fakeDiscoveryClient.setPreferredResources(nil) fakeDiscoveryClient.setError(fmt.Errorf("Error calling discoveryClient.ServerPreferredResources()")) // Wait until sync discovers the change time.Sleep(1 * time.Second) // Remove the error from being returned and see if the garbage collector sync is still working fakeDiscoveryClient.setPreferredResources(serverResources) fakeDiscoveryClient.setError(nil) err = expectSyncNotBlocked(fakeDiscoveryClient) if err != nil { t.Fatalf("Expected garbagecollector.Sync to still be running but it is blocked: %v", err) } } func expectSyncNotBlocked(fakeDiscoveryClient *fakeServerResources) error { before := fakeDiscoveryClient.getInterfaceUsedCount() t := 1 * time.Second time.Sleep(t) after := fakeDiscoveryClient.getInterfaceUsedCount() if before == after { return fmt.Errorf("discoveryClient.ServerPreferredResources() called %d times over %v", after-before, t) } return nil } </ins> type fakeServerResources struct { PreferredResources []*metav1.APIResourceList Error              error <ins> Lock               sync.Mutex InterfaceUsedCount int </ins> } func (_ *fakeServerResources) ServerResourcesForGroupVersion(groupVersion string) (*metav1.APIResourceList, error) {

} func (f *fakeServerResources) ServerPreferredResources() ([]*metav1.APIResourceList, error) { <ins> f.Lock.Lock() defer f.Lock.Unlock() f.InterfaceUsedCount++ </ins> return f.PreferredResources, f.Error } <ins> func (f *fakeServerResources) setPreferredResources(resources []*metav1.APIResourceList) { f.Lock.Lock() defer f.Lock.Unlock() f.PreferredResources = resources } func (f *fakeServerResources) setError(err error) { f.Lock.Lock() defer f.Lock.Unlock() f.Error = err } func (f *fakeServerResources) getInterfaceUsedCount() int { f.Lock.Lock() defer f.Lock.Unlock() return f.InterfaceUsedCount } </ins> func (_ *fakeServerResources) ServerPreferredNamespacedResources() ([]*metav1.APIResourceList, error) { return nil, nil }