Change:
Perf counters are currently updated every second which doesn't match the default cadvisor collection period as 10 seconds. Increase the update interval of perf counters to 10 seconds to match the house keeping interval of cadvisor. The fresh install of our internal AKS clusters. The perf counter query (pdh!PdhCollectQueryData) is responsible for 20% of CPU time in kubelet process on internal clusters, which can be reduced by lowering the perf counter update interval. <details</details<detailsAzure Kubernetes Service (AKS) </details<detailsOperating System: Windows Server 2019 Datacenter OS Build Number: .3770.amd64fre.rs5release.-1434 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /sig windows
/area cadvisor Since it is a performance overhead and not a bug per se, I will change the kind of this issue: /remove-kind bug /kind feature I bet there is an issue for this already. Can you comment/dedup?
I believe the ask here is to increase the which is only used on Windows to fetch stats from windows specific per counters. Windows does not use cAdvisor and instead uses which reads from the perf counters directly. /cc
(removing cadvisor since it is ask for windows stats) /remove-area cadvisor
agreed that is the ask. I don't have any major issue with it but would be worried about change in behavior for existing users. What are our options for making this configurable?
That would be great if making it configurable.
It might not be worth the overhead to pass this as configuration through kubelet parameters. I am unsure what affect it would have on folks who rely on it for current reporting or if it would be ok to bump the querying time. if you increase it to 10s do you see a large drop in cpu usage?
might have thoughts as well
Based on internal cloud profiling data from thousands of Windows clusters, pdh!PdhCollectQueryData takes 33% CPU usage as of today, 01/20/2023. If the updating interval increases from 1s to 10s, pdh!PdhCollectQueryData usage can be reduced 90% and CPU usage of kubelet can potentially drop 30%.
that is a compelling improvement. I am still concerned about changing the default on folks who might rely on it, I am not sure if it is a valid concern though
To understand the impact of changing the interval, what is the purpose for querying those Windows counters?
- Can you share a little bit more information about your setup? What node size are you using in AKS? Are you overprovisioning your nodes? Do you have any addons running in your AKS cluster? Does the drop eventually go down? Have you seen this issue on Windows Server 2022 too? We don't see this spike in CPU usage in our clusters provisioned with CAPZ (cluster api provider for azure) so I suspect something else is going on too. https://perf- I'm not opposed to increasing the timeout but currently I think this issue may be environmental.
1 The node size is D32_Flex which has 32 vCPUs and 128 GiB memory. 2 The perf counter query usage is the same across all clusters regardless of the cluster is running hot or cold. 3 azurepolicy is the only addon enabled. 4 The trending of perf counter query usage is quite stable. 5 Not applicable at this moment, all clusters are running with Windows Server 2019. Do you have CPU traces from CAPZ to check the time spent on PdhCollectQueryData? For our internal profiling data, perf counter usage is pretty stable in kubelet, and it's not related to the CPU spike.
I am not sure I understand this statement. The cpu usage from the the winstats pkg would be attributed to the kubelet since it is the own process. What is the owning process for usage you are seeing?
usage is from kubelet process itself. One thing I misunderstood is that, the usage in kubelet is not stable. Actually, it keeps growing. The usage is almost zero originally, but it grows up to 30+% cpu usage in kubelet. Looking into it...
The tool wasn't getting the correct data at the first time. It turns out usage in kubelet is growing from 20% to 30+% across all clusters, and the usage is never close to zero. I will investigate the growing cost of . It'd be great if we can increase the query interval to reduce the cost.

Discussion:
cpuQuery                  = "Processor(_Total)% Processor Time" memoryPrivWorkingSetQuery = "Process(_Total)Working Set - Private" memoryCommittedBytesQuery = "MemoryCommitted Bytes" <del> // Perf counters are updated every second. This is the same as the default cadvisor collection period // see https://github.com/google/cadvisor/blob/master/docs/runtime_options.md#housekeeping perfCounterUpdatePeriod = 1 * time.Second </del> <ins> // Perf counters are updated 10 seconds. This is the same as the default cadvisor housekeeping interval // set at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go perfCounterUpdatePeriod = 10 * time.Second </ins> // defaultCachePeriod is the default cache period for each cpuUsage. // This matches with the cadvisor setting and the time interval we use for containers. // see https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go#L63