Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --For single node with hundreds and thousands of pods and containers, the cpu usage of kubelet becomes high, one bottleneck is the cadvisor stats provider as below: the method iterate all containerInfos for every podStats cause high cpu the method should not iterate allcontainerInfos for every podStats, use map instead Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
I’ve done an analysis on the code and this is what I have found, I believe we have two issues to address. There are multiple range loops being performed over v2.ContainerInfo / PodStat maps within the ListPodStatus method which can be contributing to the high cpu utilisation. The allInfos argument which is supplied to the getCadvisorContainerInfo function is a map which uses a cgroup path as the key, in order to find matching container info to podUID, processing of the keys in the map needs to be performed until a match is found. Examples of keys are as follows: /kubepods/burstable/podf9e3a173-1470-451a-9ca2-/d8426b82582fbcfae16085bc794c49769d3a9648bd5463976aca4071f9342754 /docker/2dfa17fc81e781212bcc26e189ebb724721fe097394b4de3cbeb7e069bf23290 My recommendation would be to apply the following updates: Refactor the getCadvisorContainerInfo function to allow for a filter and key type parameter to be supplied by the caller, this enables reducing the number of range loops being performed over the container info map by removing elements that are not required and the ability to set the key being used in the returned container infos map to enable fast lookup of container info. My initial thoughts are to use an enumeration which has two values KEYCGROUPPATH to provide backwards compatibility and KEYPODUID to enable fast lookups via podUID Refactor ListPodStats method to be more efficient in constructing the PodStats slice and taking into account the fast lookup via podUID I think this is the best approach instead of pushing the data structure implementation for the container info upstream to the cadviser clients, this will at least initially provide a better optimisation scoped within the kubelet/stats package and reduce other possible impacts to changing the cadvisor clients Please advise.
Just realized this looks like a duplicate issue and looking as fixed by PR Both this issue and were raised by Zac
/unassign
/close
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Didn't think I could close this, leaving this with
this will be automatically closed once the PR merges

Discussion:
if err != nil { return nil, fmt.Errorf("failed to get container info from cadvisor: %v", err) } <del> // removeTerminatedContainerInfo will also remove pod level cgroups, so save the infos into allInfos first allInfos := infos infos = removeTerminatedContainerInfo(infos) </del> <ins> filteredInfos, allInfos := filterTerminatedContainerInfoAndAssembleByPodCgroupKey(infos) </ins> // Map each container to a pod and update the PodStats with container data. podToStats := map[statsapi.PodReference]*statsapi.PodStats{} <del> for key, cinfo := range infos { </del> <ins> for key, cinfo := range filteredInfos { </ins> // On systemd using devicemapper each mount into the container has an // associated cgroup. We ignore them to ensure we do not get duplicate // entries in our summary. For details on .mount units:

// getCadvisorPodInfoFromPodUID returns a pod cgroup information by matching the podUID with its CgroupName identifier base name func getCadvisorPodInfoFromPodUID(podUID types.UID, infos map[string]cadvisorapiv2.ContainerInfo) *cadvisorapiv2.ContainerInfo { <del> for key, info := range infos { if cm.IsSystemdStyleName(key) { // Convert to internal cgroup name and take the last component only. internalCgroupName := cm.ParseSystemdToCgroupName(key) key = internalCgroupName[len(internalCgroupName)-1] } else { // Take last component only. key = path.Base(key) } if cm.GetPodCgroupNameSuffix(podUID) == key { return &info } </del> <ins> if info, found := infos[cm.GetPodCgroupNameSuffix(podUID)]; found { return &info </ins> } return nil } <del> // removeTerminatedContainerInfo returns the specified containerInfo but with // the stats of the terminated containers removed. // </del> <ins> // filterTerminatedContainerInfoAndAssembleByPodCgroupKey returns the specified containerInfo but with // the stats of the terminated containers removed and all containerInfos assembled by pod cgroup key. // the first return map is container cgroup name <-> ContainerInfo and // the second return map is pod cgroup key <-> ContainerInfo. </ins> // A ContainerInfo is considered to be of a terminated container if it has an // older CreationTime and zero CPU instantaneous and memory RSS usage. <del> func removeTerminatedContainerInfo(containerInfo map[string]cadvisorapiv2.ContainerInfo) map[string]cadvisorapiv2.ContainerInfo { </del> <ins> func filterTerminatedContainerInfoAndAssembleByPodCgroupKey(containerInfo map[string]cadvisorapiv2.ContainerInfo) (map[string]cadvisorapiv2.ContainerInfo, map[string]cadvisorapiv2.ContainerInfo) { </ins> cinfoMap := make(map[containerID][]containerInfoWithCgroup) <ins> cinfosByPodCgroupKey := make(map[string]cadvisorapiv2.ContainerInfo) </ins> for key, cinfo := range containerInfo { <ins> var podCgroupKey string if cm.IsSystemdStyleName(key) { // Convert to internal cgroup name and take the last component only. internalCgroupName := cm.ParseSystemdToCgroupName(key) podCgroupKey = internalCgroupName[len(internalCgroupName)-1] } else { // Take last component only. podCgroupKey = path.Base(key) } cinfosByPodCgroupKey[podCgroupKey] = cinfo </ins> if !isPodManagedContainer(&cinfo) { continue }

} } } <del> return result </del> <ins> return result, cinfosByPodCgroupKey </ins> } // ByCreationTime implements sort.Interface for []containerInfoWithCgroup based

statustest "k8s.io/kubernetes/pkg/kubelet/status/testing" ) <del> func TestRemoveTerminatedContainerInfo(t *testing.T) { </del> <ins> func TestFilterTerminatedContainerInfoAndAssembleByPodCgroupKey(t *testing.T) { </ins> const ( seedPastPod0Infra      = 1000 seedPastPod0Container0 = 2000

"/pod0-i":  getTestContainerInfo(seedPod0Infra, pName0, namespace, leaky.PodInfraContainerName), "/pod0-c0": getTestContainerInfo(seedPod0Container0, pName0, namespace, cName00), } <del> output := removeTerminatedContainerInfo(infos) assert.Len(t, output, 2) </del> <ins> filteredInfos, allInfos := filterTerminatedContainerInfoAndAssembleByPodCgroupKey(infos) assert.Len(t, filteredInfos, 2) assert.Len(t, allInfos, 6) </ins> for _, c := range []string{"/pod0-i", "/pod0-c0"} { <del> if _, found := output[c]; !found { </del> <ins> if _, found := filteredInfos[c]; !found { t.Errorf("%q is expected to be in the outputn", c) } } for _, c := range []string{"pod0-i-terminated-1", "pod0-c0-terminated-1", "pod0-i-terminated-2", "pod0-c0-terminated-2", "pod0-i", "pod0-c0"} { if _, found := allInfos[c]; !found { </ins> t.Errorf("%q is expected to be in the outputn", c) } }

if err != nil { return nil, fmt.Errorf("failed to fetch cadvisor stats: %v", err) } <del> caInfos := getCRICadvisorStats(allInfos) </del> <ins> caInfos, allInfos := getCRICadvisorStats(allInfos) </ins> // get network stats for containers. // This is only used on Windows. For other platforms, (nil, nil) should be returned.

if err != nil { return nil, fmt.Errorf("failed to fetch cadvisor stats: %v", err) } <del> caInfos := getCRICadvisorStats(allInfos) </del> <ins> caInfos, allInfos := getCRICadvisorStats(allInfos) </ins> for _, stats := range resp { containerID := stats.Attributes.Id

} } <del> func getCRICadvisorStats(infos map[string]cadvisorapiv2.ContainerInfo) map[string]cadvisorapiv2.ContainerInfo { </del> <ins> func getCRICadvisorStats(infos map[string]cadvisorapiv2.ContainerInfo) (map[string]cadvisorapiv2.ContainerInfo, map[string]cadvisorapiv2.ContainerInfo) { </ins> stats := make(map[string]cadvisorapiv2.ContainerInfo) <del> infos = removeTerminatedContainerInfo(infos) for key, info := range infos { </del> <ins> filteredInfos, cinfosByPodCgroupKey := filterTerminatedContainerInfoAndAssembleByPodCgroupKey(infos) for key, info := range filteredInfos { </ins> // On systemd using devicemapper each mount into the container has an // associated cgroup. We ignore them to ensure we do not get duplicate // entries in our summary. For details on .mount units:

} stats[path.Base(key)] = info } <del> return stats </del> <ins> return stats, cinfosByPodCgroupKey </ins> }