Change:
} // podPreviouslyProcessed returns true if the volumes for this pod have already <del> // been processed by the populator </del> <ins> // been processed/reprocessed by the populator. Otherwise, the volumes for this pod need to // be reprocessed. </ins> func (dswp *desiredStateOfWorldPopulator) podPreviouslyProcessed( podName volumetypes.UniquePodName) bool { dswp.pods.RLock() defer dswp.pods.RUnlock() <del> _, exists := dswp.pods.processedPods[podName] return exists </del> <ins> return dswp.pods.processedPods[podName] } // markPodProcessingFailed marks the specified pod from processedPods as false to indicate that it failed processing func (dswp *desiredStateOfWorldPopulator) markPodProcessingFailed( podName volumetypes.UniquePodName) { dswp.pods.Lock() dswp.pods.processedPods[podName] = false dswp.pods.Unlock() } // podHasBeenSeenOnce returns true if the pod has been seen by the popoulator // at least once. func (dswp *desiredStateOfWorldPopulator) podHasBeenSeenOnce( podName volumetypes.UniquePodName) bool { dswp.pods.RLock() _, exist := dswp.pods.processedPods[podName] dswp.pods.RUnlock() return exist </ins> } // markPodProcessed records that the volumes for the specified pod have been

Discussion:
It happens during the "delete" phase in the load test. ! In perf-dash we can see that in total there are ~20M requests like that during the load test. ! It started in Sep 2019 (sic!), so probably when we enabled PVs in our load tests. We haven't detected it so far as, strangely, it doesn't seem to be impacting our scalability SLIs. We should investigate this. /sig scalability
I checked and we create 35 PersistentVolumeClaims per 100 nodes. So in case of 5K node test there will be 1750 PVCs in the cluster. This is close to the 2K of GET PVC requests we see.
Let's analyze the last run of 5K node test in more depths - Grafana: ! As I wrote already somewhere here, it happens during the delete phase. We reach the peak of ~2K QPS around 2:30 AM UTC in that run. Let's check the logs (api-server, kubelets) around that time.
Looking at the api-server logs one can easily tell that something is wrong. It's swamped by GET persistentvolumeclaim requests:
Restricting the logs to a single stateful set on a single node (single client ip) - it also doesn't look right. Total such requests from this IP address: So it looks like a single kubelet issued almost 20K GET requests for this PVC within ~75min, which gives us ~4 QPS from a single kubelet...
From the scheduler logs it looks like the statefulset pod for this PVC was bound on the node Let's check the kubelet logs there
Kubelet logs: Bingo, around the time when the issue happens, it's full of the following logs:
Looks like the errors disappeared exactly when the pod was deleted So, what happens here is partially caused by our test as it may happen that we'll first delete PVC and won't delete the corresponding pod for the next O(1h). Nevertheless, the code in the should be fixed so it doesn't cause kubelet to send ~4QPS of GET PVC requests in such case.
/sig node /sig storage
There is a pending discussion about ways to fix it in
Thanks for fixing that! I know how much work was it and I really appreciate!