Change:
When a pod is marked as deleted while kubelet is down / being restarted, newly started kubelet does not clean up CSI filesystem volumes of the pod. Newly started kubelet tries to reconstruct the volume using CSI's function. This part looks working, CSI volume plugin loads its json file. But then VolumeManager checks if the volume is still mounted in directory. There are two issues: does not require volumes to be presented as mounts. They can be just directories with files on them. This will be case of the most of in-line volumes. Kubelet checks the right directory given by
/sig storage related to cc
Maybe a stupid idea... Reconciler sync has found a directory in . Reconciler has no way how to check that CSI volume (or any other volume) is really mounted / published there. Depending on DSW: The volume is in DSW -from comment here it seems that the SetUp (=NodePublish) will be called on the volume: The volume is not in DSW and it will be to ASW, so it's unmounted / unpublished in the next sync: In both cases, we can expect that the volume plugin / CSI driver is idempotent and SetUp() won't do anything if the volume is already set up / TearDown() will not do anything if the volume has been torn down already. Are our volume plugins really idempotent? IMO they are. We perhaps don't need to check for presence of mount point anywhere, just presence of directory should be enough to reconstruct the volume and set it up / tear it down. Adding to the loop.
Filed , PTAL
I'm curious how the reconstruction e2e tests passed
https://k8s-*force
They start kubelet with a pod already deleted. If the pod still exists, volume reconstruction can see it in DSW and does nothing when fails: [Reconstruction fails because it checks for a mount in a wrong directory.] In this case, kubelet waits for the volume to appear in VolumesInUse before calling . The pod is deleted before that and kubelet then just deletes DSW entry and the volume is never cleaned up.
They pass because their container does not handle SIGTERM and it takes 30 seconds to kill them. Kubelet has enough time to the volume during reconciliation.
It turns out that I tested with broken version of and reconstruction is broken only for ephemeral volumes that don't mount.
So the conclusion is that we only need to fix for ephemeral volumes? Thanks!
Yes, sorry again about the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle rotten
/lifecycle frozen
From this issue surfaced through e2e tests using the hostPath CSI driver. The cause is (which is used in for reconstruction) - it doesn't play well with volumes that bind mount from a directory instead of a device.
To work around this bug here, volume life cycle checking was disabled for all tests, not just the subpath test: This check is useful and should be enabled again.
Hey everyone! I arrived at this conversation by investigating on issue (seems to me like a duplicate of ). I'm wondering if there's something I can help with? I'm still a bit new to the codebase, though. I was able to reproduce the problem by using the following ginkgo focus command in the e2e framework: And do see that if we manually add the --check-volume-lifecycle=false to the statefulset/csi-hostpathplugin tests will pass correctly. I also see that during the process, something in the cleanup function doesn't correctly delete the pv's after the tests are done, so they remain alive if we execute a kubectl get pv command.
I think I found a bit of more info on the issue probably could be of some help. I think that the volume is not being correclty marked as unmounted inside in GenerateUnmountVolumeFunc. I'm not sure why but I'll continue to investigate on it. I think this error is coming from DeletePodFromVolume in
I a couple of extra debug lines to that lines in the file, and can see that when DeletePodFromVolume tries to see if the volume exists, it doesn't find it in the asw.attachedVolumes struct, like so: Probably we should search for the name of the attached volume differently for ephemeral volumes?
/assign
There are two problems: 1) The IsLikelyNotMountPoint() call that , which is known not to work for bind mounts. The mountpoint never gets found, so volume reconstruction fails with "... is not mounted", the PVC never never gets to the list of attached volumes, and that's why we end up hitting this error: 2) CSI inline volumes hit a different error (in addition to the first problem). The call path looks like this (starting in ): Inline volumes do not have a PV spec, so it fails . We should be calling instead.
There is an ongoing that changes to utilize syscall to detect mount point by using . By using , bind mount will be properly detected fast, but it requires kernel version is v5.6 or later. So, we would be able to also utilize in . Then, the issue of bind mount can be resolved at least for kernel v5.6+, and we will be able to focus on how we solve this issue for old kernels.

Discussion:
if err != nil { return nil, err } <del> attachablePlugin, err := rc.volumePluginMgr.FindAttachablePluginByName(volume.pluginName) if err != nil { return nil, err } deviceMountablePlugin, err := rc.volumePluginMgr.FindDeviceMountablePluginByName(volume.pluginName) if err != nil { return nil, err } </del> // Create pod object pod := &v1.Pod{

return nil, err } <ins> // We have to find the plugins by volume spec (NOT by plugin name) here // in order to correctly reconstruct ephemeral volume types. // Searching by spec checks whether the volume is actually attachable // (i.e. has a PV) whereas searching by plugin name can only tell whether // the plugin supports attachable volumes. attachablePlugin, err := rc.volumePluginMgr.FindAttachablePluginBySpec(volumeSpec) if err != nil { return nil, err } deviceMountablePlugin, err := rc.volumePluginMgr.FindDeviceMountablePluginBySpec(volumeSpec) if err != nil { return nil, err } </ins> var uniqueVolumeName v1.UniqueVolumeName if attachablePlugin != nil || deviceMountablePlugin != nil { uniqueVolumeName, err = util.GetUniqueVolumeNameFromSpec(plugin, volumeSpec)

// Filesystem Volume case if volumeMode == v1.PersistentVolumeFilesystem { // Create volumeSpec from mount path <del> klog.V(5).Infof("Starting operationExecutor.ReconstructVolumepodName") </del> <ins> klog.V(5).Infof("Starting operationExecutor.ReconstructVolume for file volume on pod %q", podName) </ins> volumeSpec, err := plugin.ConstructVolumeSpec(volumeSpecName, volumePath) if err != nil { return nil, err

// Block Volume case // Create volumeSpec from mount path <del> klog.V(5).Infof("Starting operationExecutor.ReconstructVolume") </del> <ins> klog.V(5).Infof("Starting operationExecutor.ReconstructVolume for block volume on pod %q", podName) </ins> // volumePath contains volumeName on the path. In the case of block volume, {volumeName} is symbolic link // corresponding to raw block device.

if mounter == nil { return false, fmt.Errorf("mounter was not set for a filesystem volume") } <del> if isNotMount, mountCheckErr = mounter.IsLikelyNotMountPoint(mountPath); mountCheckErr != nil { </del> <ins> if isNotMount, mountCheckErr = mount.IsNotMountPoint(mounter, mountPath); mountCheckErr != nil { </ins> return false, fmt.Errorf("could not check whether the volume %q (spec.Name: %q) pod %q (UID: %q) is mounted with: %v", uniqueVolumeName, volumeName,

"encoding/json" "errors" "fmt" <del> "regexp" </del> "strconv" "strings" "sync"

NodeName:                 node.Name, } <del> // Disable volume lifecycle checks due to issue #103651 for the one // test that it breaks. // TODO: enable this check once issue is resolved for csi-host-path driver // (https://github.com/kubernetes/kubernetes/pull/104858). if regexp.MustCompile("should unmount if pod is.*deleted while kubelet is down").MatchString(ginkgo.CurrentGinkgoTestDescription().FullTestText) { o.DriverContainerArguments = append(o.DriverContainerArguments, "--check-volume-lifecycle=false") } </del> cleanup, err := utils.CreateFromManifests(config.Framework, driverNamespace, func(item interface{}) error { if err := utils.PatchCSIDeployment(config.Framework, o, item); err != nil { return err