Change:
I ran a deployment with replicas 1 and use to set container's subpath dir, which uses NFS as the storage backend, and create about dirs in container's subPath dir,when delete the pod, it spent much more time to end status, and when the replicas increase to 10 and do the rolling update, at the same time dir nums in containers increase to , it spent about 30 minutes to totally remove pod, which is intolerant. The pod is deleted fast. a pod using subPathExpr as the volume mount. dirs in the backend storage corresponds to the container volume mount the pod kubelet log We can find that it spent about 17s to clean the mount dir in kuelet. No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After some debugging, I inspected the pprof with flame graph when delete a pod, it shows that and cost much more time, here is the graph: ! Apparently the Walk func spent much time when doCleanPath, what we care is the top level dir,and this pr has some fast path to avoid much recursive time, but before gol1.16, the filepath.Walk is still less efficient, because when used in kubelet, it reads all the dir in the mount point before call the callback func in Walk even we don't use it after that, so it spends much time. And after go 1.16, filepath.WalkDir is introduced, which is more efficient than Walk, as the comment said in golang WalkDirFunc is called before reading a directory, so it can save much time when there are lots of dirs in the first level of mount points. So it's time wasting and meaningless when we use Walk so far because we traverse the dir but we didn't use it and return SkipDir err, the code is After I replaced it with , the took only about 41ms, which is a huge boost than 17s before.
/sig storage
I tried to fix this, and pr is here

Discussion:
// scan /var/lib/kubelet/pods/<uid>/volume-subpaths/<volume>/<container name>/* fullContainerDirPath := filepath.Join(subPathDir, containerDir.Name()) <del> err = filepath.Walk(fullContainerDirPath, func(path string, info os.FileInfo, _ error) error { </del> <ins> // The original traversal method here was ReadDir, which was not so robust to handle some error such as "stale NFS file handle", // so it was replaced with filepath.Walk in a later patch, which can pass through error and handled by the callback WalkFunc. // After go 1.16, WalkDir was introduced, it's more effective than Walk because the callback WalkDirFunc is called before // reading a directory, making it save some time when a container's subPath contains lots of dirs. // See https://github.com/kubernetes/kubernetes/pull/71804 and https://github.com/kubernetes/kubernetes/issues/107667 for more details. err = filepath.WalkDir(fullContainerDirPath, func(path string, info os.DirEntry, _ error) error { </ins> if path == fullContainerDirPath { // Skip top level directory return nil