Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I try to backport bookmark feature to our internal branch. When I enable bookmark feature, apiserver leaks memory. I spend some time to dig the root cause and find the bookmarkWatchers of service cacher is keep increasing. I am not sure if it exists at upstream code. Please let me know if it is ok at latest code. What you expected to happen: Avoid memory leaking How to reproduce it (as minimally and precisely as possible): a cluster. not update service. will find kube-apiserver's memory keep increasing. Anything else we need to know?: Kubelet will watch service with bookmark, and add a watcher to bookmarkWatchers. and service init resource version is 0 Without updating service, we cannot update "lastProcessedResourceVersion", it will keep 0, bookmarkTimer branch will always be continued. As a result, it cannot trigger popExpiredWatchers, so the service watcher will keep increasing every watching. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scalability PLAT :)
How about making this change ?
- can you also please take a look?
Agree with u. That's same way I fix in our branch. But I wanna hear the suggestion of and .
! With the same bugfix wrote above, we can see the memory not increasing any more.
Given that watch-bookmarks doesn't create any additional go-routines (well, creates 1, but cannot leak them), it has to leak the data. The only data that can be leaked is exactly bookmarkWatchers. So I looked into the code and got into exactly the same conclusion - that we must be leaking watchers for resources that aren't changing. The fix as above makes sense (though we also need a unit test for it).
Thanks a lot.
has the fix by
Thanks for reporting the bug, I will look at this day later. In addition, also need to think carefully about whether need to send a bookmark when there are no events at all.
cc

Discussion:
// if you set fire time at X, you can get the bookmark within (X-1,X+1) period. // This is NOT thread-safe. type watcherBookmarkTimeBuckets struct { <ins> lock            sync.Mutex </ins> watchersBuckets map[int64][]*cacheWatcher startBucketID   int64 clock           clock.Clock

return false } bucketID := nextTime.Unix() <ins> t.lock.Lock() defer t.lock.Unlock() </ins> if bucketID < t.startBucketID { bucketID = t.startBucketID }

currentBucketID := t.clock.Now().Unix() // There should be one or two elements in almost all cases expiredWatchers := make([][]*cacheWatcher, 0, 2) <ins> t.lock.Lock() defer t.lock.Unlock() </ins> for ; t.startBucketID <= currentBucketID; t.startBucketID++ { if watchers, ok := t.watchersBuckets[t.startBucketID]; ok { delete(t.watchersBuckets, t.startBucketID)

// Never send a bookmark event if we did not see an event here, this is fine // because we don't provide any guarantees on sending bookmarks. if lastProcessedResourceVersion == 0 { <ins> // pop expired watchers in case there has been no update c.bookmarkWatchers.popExpiredWatchers() </ins> continue } bookmarkEvent := &watchCacheEvent{

} } <ins> func TestCacherNoLeakWithMultipleWatchers(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.WatchBookmark, true)() backingStorage := &dummyStorage{} cacher, _, err := newTestCacher(backingStorage, 1000) if err != nil { t.Fatalf("Couldn't create cacher: %v", err) } defer cacher.Stop() // Wait until cacher is initialized. cacher.ready.wait() pred := storage.Everything pred.AllowWatchBookmarks = true // run the collision test for 3 seconds to let ~2 buckets expire stopCh := make(chan struct{}) time.AfterFunc(3*time.Second, func() { close(stopCh) }) wg := &sync.WaitGroup{} wg.Add(1) go func() { defer wg.Done() for { select { case <-stopCh: return default: ctx, _ := context.WithTimeout(context.Background(), 3*time.Second) w, err := cacher.Watch(ctx, "pods/ns", "0", pred) if err != nil { t.Fatalf("Failed to create watch: %v", err) } w.Stop() } } }() wg.Add(1) go func() { defer wg.Done() for { select { case <-stopCh: return default: cacher.bookmarkWatchers.popExpiredWatchers() } } }() // wait for adding/removing watchers to end wg.Wait() // wait out the expiration period and pop expired watchers time.Sleep(2 * time.Second) cacher.bookmarkWatchers.popExpiredWatchers() cacher.bookmarkWatchers.lock.Lock() defer cacher.bookmarkWatchers.lock.Unlock() if len(cacher.bookmarkWatchers.watchersBuckets) != 0 { t.Errorf("unexpected bookmark watchers %v", len(cacher.bookmarkWatchers.watchersBuckets)) } } </ins> func testCacherSendBookmarkEvents(t *testing.T, watchCacheEnabled, allowWatchBookmarks, expectedBookmarks bool) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.WatchBookmark, watchCacheEnabled)() backingStorage := &dummyStorage{}