Change:
What happened: Consider this use case, a k8s cluster with many short lived name spaces with a high cardinality. (Think UUIDs). Over time, some of the indices contain string sets for each name space with no objects in the backing string sets; which are never reused. Leads to process memory increasing over time until the process exceeds available memory. Potentially causing issues as an api-server or controller is terminated by the operating system. What you expected to happen: Process memory does not increase unbounded as the number of unique items then deleted from a thread safe map, causing indices to hold on to zero length string sets for the key. This could be mitigated when an index's string set for a key is empty, that string set is removed from the index. How to reproduce it (as minimally and precisely as possible): Thanks to for this reproduction code: Anything else we need to know?: Can be mitigated with the following patch against master at : has a code path for creating sets that don't exist, so if a set is temporarily reduced to zero; it will be re-created as if it was the initial case where the set was never created. Here's an apiserver pprof heap base png from a clean start for a test case of ≈800k name space create/delete in kube v1.14.7: Note the increase in heap utilisation from base of and nodes in the hot path. Similar increases were observed in v1.13.11 and v1.15.5. Same as above, but with the suggested fix and ≈100k name space create/delete: Note lack of and nodes in the hot path. (Was not able to directly attach the images to this issue due to Something went really wrong, and we can't process that file.) Environment: Kubernetes version (use ): Issue reproduced in v1.13.11, v1.14.7, 1.15.5 directly by adding/deleting namespaces. Above mitigation directly tested against a v1.14.7 cluster.
/sig api-machinery
Edit: I found threadSafeMap.updateIndices() - was searching with misspelled func name.
I'm not sure I follow, but is still present -
Nah, my bad; didn't spell the method right originally in the text.
One option is to mark the time when set becomes empty: We can clean Index entry when certain period has passed since the set becomes empty.
I really don't think that solution is a viable and adds unneeded complexity. You would then need to have something running in the background to prune empty sets. The solution proposed in the original post is the most straight forward, when the set is empty - delete it. If something is then created that needs the set, it will be recreated when needed.
I think that is fine.

Discussion:
"reflector_test.go", "shared_informer_test.go", "store_test.go", <ins> "thread_safe_store_test.go", </ins> "undelta_store_test.go", ], embed = [":go_default_library"],

set := index[indexValue] if set != nil { set.Delete(key) <ins> // If we don't delete the set when zero, indices with high cardinality // short lived resources can cause memory to increase over time from // unused empty sets. See `kubernetes/kubernetes/issues/84959`. if len(set) == 0 { delete(index, indexValue) } </ins> } } }

<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cache import ( "testing" ) func TestThreadSafeStoreDeleteRemovesEmptySetsFromIndex(t *testing.T) { testIndexer := "testIndexer" indexers := Indexers{ testIndexer: func(obj interface{}) (strings []string, e error) { indexes := []string{obj.(string)} return indexes, nil }, } indices := Indices{} store := NewThreadSafeStore(indexers, indices).(*threadSafeMap) testKey := "testKey" store.Add(testKey, testKey) // Assumption check, there should be a set for the `testKey` with one element in the added index set := store.indices[testIndexer][testKey] if len(set) != 1 { t.Errorf("Initial assumption of index backing string set having 1 element failed. Actual elements: %d", len(set)) return } store.Delete(testKey) set, present := store.indices[testIndexer][testKey] if present { t.Errorf("Index backing string set not deleted from index. Set length: %d", len(set)) } } func TestThreadSafeStoreAddKeepsNonEmptySetPostDeleteFromIndex(t *testing.T) { testIndexer := "testIndexer" testIndex := "testIndex" indexers := Indexers{ testIndexer: func(obj interface{}) (strings []string, e error) { indexes := []string{testIndex} return indexes, nil }, } indices := Indices{} store := NewThreadSafeStore(indexers, indices).(*threadSafeMap) store.Add("retain", "retain") store.Add("delete", "delete") // Assumption check, there should be a set for the `testIndex` with two elements set := store.indices[testIndexer][testIndex] if len(set) != 2 { t.Errorf("Initial assumption of index backing string set having 2 elements failed. Actual elements: %d", len(set)) return } store.Delete("delete") set, present := store.indices[testIndexer][testIndex] if !present { t.Errorf("Index backing string set erroneously deleted from index.") return } if len(set) != 1 { t.Errorf("Index backing string set has incorrect length, expect 1. Set length: %d", len(set)) } } </ins>