Change:
when kubelet cleanupOrphanedPodDirsit may fail when the path is a file the log is ` when can see ,this path is a file ,not a directory kubelet can delete the file and path when deleting orphaned pod kubernetes 1.20.2+docker+nvidia-dockerand the pod has GPU allocated No response <details20.5 </details<detailsnone </details<details</details<details</details<details</details<details</details
kubelet only delete dirwhen the path is file ,it will fail my pod volume is like this
/sig node storage
/assign
Mounting a file using a subpath is fine, however I think mounting the same using twice isn't. Why do you need both? Can you try deleting and use in both places?
thanks, is this the reason for the problem?
It might be... can you do the test and confirm?
/triage needs-information let's wait for information on how things behave when not mounted twice
oki have a try
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
i delete cudagpu,but it still has same problem
Thanks let me come back to this soon...
Are you still using 1.20? Would it be possible to retry with a supported version of k8s, best would be 1.23?
using 1.20.
can you reproduce with 1.23? otherwise it'd be difficult to propose fixes to a k8s out of support.
ok. i will test in 1.23
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, we got the same issue, could someone confirm whether this issue has been fixed? we're using 1.20.7.
/reopen I am able to reproduce it with v1.29, but only with dirty hacks. Something bad must happen during node reboot. a Pod that uses a CSI volume + file on the volume as a subpath. kubelet service on the node where the pod runs. -delete the Pod. remove from the pod local volume directory, i.e. . I don't know how to reproduce the issue without this step. How else can it happen that the file gets missing or its reconstruction fails?! reboot the machine. After reboot, newly started kubelet will not see the Pod in the API server, and it will try to reconstruct its CSI volume mount from That file is missing, and therefore it will declare the volume as unmounted and cleaned, without cleaning its subpaths. The orphan cleanup routine will then try to remove subpaths of the volume, but it can handle only directories as subpath, not files -will fail.
Reopened this issue. <detailsIn response to : /reopen Stop kubelet service on the node. Force-delete the Pod. Manually remove from the pod local volume directory, i.e. . I don't know how to reproduce the issue without this step. How it can happen that the file gets missing or its reconstruction fails?! Hard reboot the machine. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
/triage accepted
/assign

Discussion:
// removeOrphanedPodVolumeDirs attempts to remove the pod volumes directory and // its subdirectories. There should be no files left under normal conditions // when this is called, so it effectively does a recursive rmdir instead of <del> // RemoveAll to ensure it only removes directories and not regular files. </del> <ins> // RemoveAll to ensure it only removes empty directories and files that were // used as mount points, but not content of the mount points. </ins> func (kl *Kubelet) removeOrphanedPodVolumeDirs(uid types.UID) []error { orphanVolumeErrors := []error{}

} } <del> // If there are any volume-subpaths, attempt to rmdir them </del> <ins> // If there are any volume-subpaths, attempt to remove them </ins> subpathVolumePaths, err := kl.getPodVolumeSubpathListFromDisk(uid) if err != nil { orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but error occurred during reading of volume-subpaths dir from disk: %v", uid, err))

} if len(subpathVolumePaths) > 0 { for _, subpathVolumePath := range subpathVolumePaths { <del> if err := syscall.Rmdir(subpathVolumePath); err != nil { </del> <ins> // Remove both files and empty directories here, as the subpath may have been a bind-mount of a file or a directory. if err := os.Remove(subpathVolumePath); err != nil { </ins> orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but failed to rmdir() subpath at path %v: %v", uid, subpathVolumePath, err)) } else { klog.InfoS("Cleaned up orphaned volume subpath from pod", "podUID", uid, "path", subpathVolumePath)