Code change:
In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs. Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

Related discussion:
text: }) } } <ins> func TestPodVolumesExistWithMount(t *testing.T) { poduid := types.UID("poduid") testCases := map[string]struct { prepareFunc func(kubelet *Kubelet) error expected    bool }{ "noncsivolume-dir-not-exist": { prepareFunc: func(kubelet *Kubelet) error { return nil }, expected: false, }, "noncsivolume-dir-exist-noplugins": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) return os.MkdirAll(filepath.Join(podDir, "volumes/"), 0750) }, expected: false, }, "noncsivolume-dir-exist-nomount": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) return os.MkdirAll(filepath.Join(podDir, "volumes/plugin/name"), 0750) }, expected: false, }, "noncsivolume-dir-exist-with-mount": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) volumePath := filepath.Join(podDir, "volumes/plugin/name") if err := os.MkdirAll(volumePath, 0750); err != nil { return err } fm := mount.NewFakeMounter( []mount.MountPoint{ {Device: "/dev/sdb", Path: volumePath}, }) kubelet.mounter = fm return nil }, expected: true, }, "noncsivolume-dir-exist-nomount-withcsimountpath": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) volumePath := filepath.Join(podDir, "volumes/plugin/name/mount") if err := os.MkdirAll(volumePath, 0750); err != nil { return err } fm := mount.NewFakeMounter( []mount.MountPoint{ {Device: "/dev/sdb", Path: volumePath}, }) kubelet.mounter = fm return nil }, expected: false, }, "csivolume-dir-exist-nomount": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) volumePath := filepath.Join(podDir, "volumes/kubernetes.io~csi/name") return os.MkdirAll(volumePath, 0750) }, expected: false, }, "csivolume-dir-exist-mount-nocsimountpath": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) volumePath := filepath.Join(podDir, "volumes/kubernetes.io~csi/name/mount") return os.MkdirAll(volumePath, 0750) }, expected: false, }, "csivolume-dir-exist-withcsimountpath": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) volumePath := filepath.Join(podDir, "volumes/kubernetes.io~csi/name/mount") if err := os.MkdirAll(volumePath, 0750); err != nil { return err } fm := mount.NewFakeMounter( []mount.MountPoint{ {Device: "/dev/sdb", Path: volumePath}, }) kubelet.mounter = fm return nil }, expected: true, }, } for name, tc := range testCases { t.Run(name, func(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kubelet := testKubelet.kubelet if tc.prepareFunc != nil { if err := tc.prepareFunc(kubelet); err != nil { t.Fatalf("%s failed preparation: %v", name, err) } } exist := kubelet.podVolumesExist(poduid) if tc.expected != exist { t.Errorf("%s failed: expected %t, got %t", name, tc.expected, exist) } }) } } </ins>