Code change:
// getCadvisorPodInfoFromPodUID returns a pod cgroup information by matching the podUID with its CgroupName identifier base name func getCadvisorPodInfoFromPodUID(podUID types.UID, infos map[string]cadvisorapiv2.ContainerInfo) *cadvisorapiv2.ContainerInfo { <del> for key, info := range infos { if cm.IsSystemdStyleName(key) { // Convert to internal cgroup name and take the last component only. internalCgroupName := cm.ParseSystemdToCgroupName(key) key = internalCgroupName[len(internalCgroupName)-1] } else { // Take last component only. key = path.Base(key) } if cm.GetPodCgroupNameSuffix(podUID) == key { return &info } </del> <ins> if info, found := infos[cm.GetPodCgroupNameSuffix(podUID)]; found { return &info </ins> } return nil } <del> // removeTerminatedContainerInfo returns the specified containerInfo but with // the stats of the terminated containers removed. // </del> <ins> // filterTerminatedContainerInfoAndAssembleByPodCgroupKey returns the specified containerInfo but with // the stats of the terminated containers removed and all containerInfos assembled by pod cgroup key. // the first return map is container cgroup name <-> ContainerInfo and // the second return map is pod cgroup key <-> ContainerInfo. </ins> // A ContainerInfo is considered to be of a terminated container if it has an // older CreationTime and zero CPU instantaneous and memory RSS usage. <del> func removeTerminatedContainerInfo(containerInfo map[string]cadvisorapiv2.ContainerInfo) map[string]cadvisorapiv2.ContainerInfo { </del> <ins> func filterTerminatedContainerInfoAndAssembleByPodCgroupKey(containerInfo map[string]cadvisorapiv2.ContainerInfo) (map[string]cadvisorapiv2.ContainerInfo, map[string]cadvisorapiv2.ContainerInfo) { </ins> cinfoMap := make(map[containerID][]containerInfoWithCgroup) <ins> cinfosByPodCgroupKey := make(map[string]cadvisorapiv2.ContainerInfo) </ins> for key, cinfo := range containerInfo { <ins> var podCgroupKey string if cm.IsSystemdStyleName(key) { // Convert to internal cgroup name and take the last component only. internalCgroupName := cm.ParseSystemdToCgroupName(key) podCgroupKey = internalCgroupName[len(internalCgroupName)-1] } else { // Take last component only. podCgroupKey = path.Base(key) } cinfosByPodCgroupKey[podCgroupKey] = cinfo </ins> if !isPodManagedContainer(&cinfo) { continue }

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --For single node with hundreds and thousands of pods and containers, the cpu usage of kubelet becomes high, one bottleneck is the cadvisor stats provider as below: the method iterate all containerInfos for every podStats cause high cpu the method should not iterate allcontainerInfos for every podStats, use map instead Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
I’ve done an analysis on the code and this is what I have found, I believe we have two issues to address. There are multiple range loops being performed over v2.ContainerInfo / PodStat maps within the ListPodStatus method which can be contributing to the high cpu utilisation. The allInfos argument which is supplied to the getCadvisorContainerInfo function is a map which uses a cgroup path as the key, in order to find matching container info to podUID, processing of the keys in the map needs to be performed until a match is found. Examples of keys are as follows: /kubepods/burstable/podf9e3a173-1470-451a-9ca2-/d8426b82582fbcfae16085bc794c49769d3a9648bd5463976aca4071f9342754 /docker/2dfa17fc81e781212bcc26e189ebb724721fe097394b4de3cbeb7e069bf23290 My recommendation would be to apply the following updates: Refactor the getCadvisorContainerInfo function to allow for a filter and key type parameter to be supplied by the caller, this enables reducing the number of range loops being performed over the container info map by removing elements that are not required and the ability to set the key being used in the returned container infos map to enable fast lookup of container info.