Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCP OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: What happened: Memory usage of federation-control-manager keeps on increasing to very large size overtime. What you expected to happen: Memory usage is stable How to reproduce it (as minimally and precisely as possible): Register one or more cluster to federation control plane and watch the top output for the federation-control-manager Also can check the pprof, here is the one such output after federation-control-manager is run for an hour. We can see there are around goroutines created in an hour which are leaked and would consume memory. Anything else do we need to know: I do have a fix for the issue and would be happy to submit it, if this issue is accepted.
plz check the issue
feel free to just submit a pull request, we don't really have a concept of an issue being "accepted" :)
This looks like a legitimate bug. could you please clarify what version of the federated service controller you are running? The image path and version of the container, or a git hash of the source should be sufficient.
i am taking the latest code from master branch.

Discussion:
// worker runs a worker thread that just dequeues items, processes them, and marks them done. // It enforces that the syncHandler is never invoked concurrently with the same key. func (sc *ServiceController) clusterEndpointWorker() { <del> fedClient := sc.federationClient </del> <ins> // process all pending events in endpointWorkerDoneChan ForLoop: for { select { case clusterName := <-sc.endpointWorkerDoneChan: sc.endpointWorkerMap[clusterName] = false default: // non-blocking, comes here if all existing events are processed break ForLoop } } </ins> for clusterName, cache := range sc.clusterCache.clientMap { <ins> workerExist, found := sc.endpointWorkerMap[clusterName] if found && workerExist { continue } // create a worker only if the previous worker has finished and gone out of scope </ins> go func(cache *clusterCache, clusterName string) { <ins> fedClient := sc.federationClient </ins> for { func() { key, quit := cache.endpointQueue.Get() // update endpoint cache if quit { <ins> // send signal that current worker has finished tasks and is going out of scope sc.endpointWorkerDoneChan <- clusterName </ins> return } defer cache.endpointQueue.Done(key)

}() } }(cache, clusterName) <ins> sc.endpointWorkerMap[clusterName] = true </ins> } }

// worker runs a worker thread that just dequeues items, processes them, and marks them done. // It enforces that the syncHandler is never invoked concurrently with the same key. func (sc *ServiceController) clusterServiceWorker() { <del> fedClient := sc.federationClient </del> <ins> // process all pending events in serviceWorkerDoneChan ForLoop: for { select { case clusterName := <-sc.serviceWorkerDoneChan: sc.serviceWorkerMap[clusterName] = false default: // non-blocking, comes here if all existing events are processed break ForLoop } } </ins> for clusterName, cache := range sc.clusterCache.clientMap { <ins> workerExist, found := sc.serviceWorkerMap[clusterName] if found && workerExist { continue } // create a worker only if the previous worker has finished and gone out of scope </ins> go func(cache *clusterCache, clusterName string) { <ins> fedClient := sc.federationClient </ins> for { func() { key, quit := cache.serviceQueue.Get() <del> defer cache.serviceQueue.Done(key) </del> if quit { <ins> // send signal that current worker has finished tasks and is going out of scope sc.serviceWorkerDoneChan <- clusterName </ins> return } <ins> defer cache.serviceQueue.Done(key) </ins> err := sc.clusterCache.syncService(key.(string), clusterName, cache, sc.serviceCache, fedClient, sc) if err != nil { glog.Errorf("Failed to sync service: %+v", err)

}() } }(cache, clusterName) <ins> sc.serviceWorkerMap[clusterName] = true </ins> } }

UserAgentName = "federation-service-controller" KubeAPIQPS    = 20.0 KubeAPIBurst  = 30 <ins> maxNoOfClusters = 100 </ins> ) type cachedService struct {

// services that need to be synced queue           *workqueue.Type knownClusterSet sets.String <ins> // endpoint worker map contains all the clusters registered with an indication that worker exist // key clusterName endpointWorkerMap map[string]bool // channel for worker to signal that it is going out of existence endpointWorkerDoneChan chan string // service worker map contains all the clusters registered with an indication that worker exist // key clusterName serviceWorkerMap map[string]bool // channel for worker to signal that it is going out of existence serviceWorkerDoneChan chan string </ins> } // New returns a new service controller to keep DNS provider service resources

}, }, ) <ins> s.endpointWorkerMap = make(map[string]bool) s.serviceWorkerMap = make(map[string]bool) s.endpointWorkerDoneChan = make(chan string, maxNoOfClusters) s.serviceWorkerDoneChan = make(chan string, maxNoOfClusters) </ins> return s }