Change:
qps of configmap get request is too high in kube-apiserver. when there are many configmaps in cluster mounted by running pod, the qps of configmap get request will be high and the request will also need to get configmap resource from etcd instead of returned from kube-apiserver cache. DO NOT get configmap from etcd. Get it from cache of kube-apiserver. kubelet configMapAndSecretChangeDetectionStrategy to Cache; kube-apiserver --v=3 lots of configmaps mount to pods kube-apiserver metrics kube-apiserver log I have tested it in v1.22.13 and v1.28.0. The logic to set resourceversion=0 seems never be triggered <details</details<details</details<details</details<details</details<details</details<details</details
/sig node
Thanks can you also fill-in at least the kubernetes version, but the more info the better in your initial comment?
k8s version.
/remove-sig node /sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /remove-sig node /sig apimachinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apiserver
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig apiserver Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
I think this issue is related to kubelet not kube-apiserver and i have found the root cause i think.
/sig node
Could you add concrete data for some metrics? the request rate you describe timing for the requests. Is there a slowdown?
request rate depends on the total number of configmaps mounted to pods. what i care is not reduce the qps but to use kube-apiserver cache instead of direct request to etcd. for now, the request will redirect to etcd. there is no resourceVersion in the request query.
/cc /triage accepted
/cc
can you add some steps to reproduce the issue? and a link to the code you think is causing the problem?
steps: code: The invocation chain is quite lengthy and the reason like blow: kubelet has a ticker with 1s interval and will try to get pods from queue to sync if there is data in syncCh; is default set to 1m. pod will be enqueued with a random time between 1m ~ 1.5m by default; is background reconcicler in file. it will run every 100ms to try to mountAttachVolumes which could trigger configmap get operation when the volume is not mounted or require remount; .object will be set to nil each time pod to sync via . pay attention to the comment. to mount or remount volume, a configmap get request will be triggered and because the is nil, the resourceversion query will not be which result to the request goes to etcd finally. volume mounted, the flag will be set to false in ; when next mountAttachVolumes called but volume has been mounted and not require remount, no need to get configmap from kubelet's cache. therefore there will no http request to apiserver to get configmap; may be some configmap get operation can be satisfied by cachebasedmanager by directly returned from local cache or get it from apiserver's cache when cache expired because same configmap can be used by many pods and each pod has a goroutine to execute syncPod but only one goroutine to execute mountAttachVolumes. I think we can always set resourceversion=0 to the query and ignore whether it's the first time to fetch the configmap or just the local cache expired. The implementation of informer is also just to list all specified resources with resourceversion=0 when to start.
That was intentional, so that a newly scheduled pod would be sure to get latest content from the API. However, RegisterPod is called on every resync, and the cacheBasedManager#RegisterPod implementation is calling AddReference on all references every time, not just changed references. I agree we don't want to do that. No, we want to actually fetch a live object the first time we get data for a new pod. I think the fix needs to happen in cacheBasedManager#RegisterPod to only call AddReference for items which are not in c.getReferencedObjects(prev)
make sense to me. let me try to fix it. /assign
/assign

Discussion:
names := c.getReferencedObjects(pod) c.lock.Lock() defer c.lock.Unlock() <del> for name := range names { c.objectStore.AddReference(pod.Namespace, name, pod.UID) } </del> var prev *v1.Pod key := objectKey{namespace: pod.Namespace, name: pod.Name, uid: pod.UID} prev = c.registeredPods[key] c.registeredPods[key] = pod <del> if prev != nil { for name := range c.getReferencedObjects(prev) { // On an update, the .Add() call above will have re-incremented the // ref count of any existing object, so any objects that are in both // names and prev need to have their ref counts decremented. Any that // are only in prev need to be completely removed. This unconditional // call takes care of both cases. c.objectStore.DeleteReference(prev.Namespace, name, prev.UID) </del> <ins> // To minimize unnecessary API requests to the API server for the configmap/secret get API // only invoke AddReference the first time RegisterPod is called for a pod. if prev == nil { for name := range names { c.objectStore.AddReference(pod.Namespace, name, pod.UID) } } else { prevNames := c.getReferencedObjects(prev) // Add new references for name := range names { if !prevNames.Has(name) { c.objectStore.AddReference(pod.Namespace, name, pod.UID) } } // Remove dropped references for prevName := range prevNames { if !names.Has(prevName) { c.objectStore.DeleteReference(pod.Namespace, prevName, pod.UID) } </ins> } } }

} func podWithSecrets(ns, podName string, toAttach secretsToAttach) *v1.Pod { <del> return podWithSecretsAndUID(ns, podName, "", toAttach) </del> <ins> return podWithSecretsAndUID(ns, podName, fmt.Sprintf("%s/%s", ns, podName), toAttach) </ins> } func podWithSecretsAndUID(ns, podName, podUID string, toAttach secretsToAttach) *v1.Pod {

}, } manager.RegisterPod(podWithSecrets("ns1", "name1", s2)) <del> // All secrets should be invalidated - this should trigger get operations. </del> <ins> // Fetch only s3 and s20 secrets - this should trigger get operations. </ins> store.Get("ns1", "s1") store.Get("ns1", "s2") store.Get("ns1", "s20") store.Get("ns1", "s3") actions = fakeClient.Actions() <del> assert.Equal(t, 4, len(actions), "unexpected actions: %#v", actions) </del> <ins> assert.Equal(t, 2, len(actions), "unexpected actions: %#v", actions) </ins> fakeClient.ClearActions() // Create a new pod that is refencing the first three secrets - those should