Change:
What happened: A custom CSI driver implementing returns a failure response, indicating that it did not complete its intended steps to prepare the volume for the mounting pod. Let's say that these steps involve a filesystem at the target directory, data from a service, and that data to a set of files in the mounted filesystem. If the first filesystem mounting step succeeds, but either of the subsequent two steps fail, the volume is still not ready for use by the pod. However, if this CSI driver returns a failure response, and the CSI mounter comes back and, before invoking the RPC again, it finds a filesystem already mounted at the target directory, , and ceases further calls to to allow the CSI driver to try again. What you expected to happen: If a CSI driver returns a failure response to a RPC, Kubernetes should honor that declaration of failure and invoke the RPC again, disallowing the mounting pod from starting until this RPC succeeds. Even though it's possible to partially work around the current behavior by having the driver mount the filesystem as late as possible, and then attempting to unmount it if fails to populate the volume with its intended data, doing so makes writing drivers more complicated and forces them to do extra work. How to reproduce it (as minimally and precisely as possible): In a CSI driver (even one that serves just ephemeral, inline volumes), implement to mount a filesystem at the target path, then return a failure response. Confirm that Kubernetes does not invoke again, because there's now a filesystem mounted at the target path. Anything else we need to know?: This topic came up for discussion in the "Kubernetes" Slack team. There, helped investigate this complaint, and to discuss changing this behavior. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Others: csi-node-driver-registrar container image tag: v1.0.2
/sig storage
While the plugin should try to clean up as much as possible, I agree that kubelet should not be assuming that if there's a mount point, the publish is successful. NodePublish is supposed to be idempotent so it should theoretically be safe to call it when it's already mounted, although there is the possibility that plugins may have written against the K8s behavior.
For the mount point checking here before calling NodePublishVolume, I am wondering it is just for performance optimization, or it has other purposes. Have concern about some drivers might depend on this check if they are not implemented with idenpotency.
Those drivers would be operating in violation of the specification. I'd prefer that we don't penalize already conforming and yet-to-be written drivers in order to accommodate these existing drivers—if they even exist.
More fallout in last night's testing: I introduced a bug into my driver that caused it to panic during ("Writing that data to a set of files in the mounted filesystem"). My handler function is set up to unmount a successfully mounted volume via if it doesn't complete steps 2 and 3 above, in order to deal with this issue's principal complaint. My driver container would crash due to this panic. The kubelet would call on my driver twice, it would crash twice, but the pod still started running. I'm not yet sure whether or not the filesystem got unmounted successfully via the on the first call, but somehow the pod started despite my handler function never completing successfully. I'm going to try to get some more debugging time on this later today to see if I can figure out what the kubelet was doing there.
Studying my code more, I see that my function only unmounted the mounted directory when the containing handler function was returning an error. When unwinding due to a panic, the named return value of type I was inspecting was still nil. Revising the criteria in that function covers both failure reasons (returning an and unwinding due to a panic). Regardless, this further underscores the problem: If a driver makes it as far as mounting a filesystem at the target directory, but fails afterward—including the process crashing and never responding positively to the RPC—the mounting pod will start. That's likely to then cause the pod to fail, for lack of the files it expected to find mounted.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

Discussion:
return errors.New(log("attacher.MountDevice failed, deviceMountPath is empty")) } <del> corruptedDir := false mounted, err := isDirMounted(c.plugin, deviceMountPath) if err != nil { klog.Error(log("attacher.MountDevice failed while checking mount status for dir [%s]", deviceMountPath)) if isCorruptedDir(deviceMountPath) { corruptedDir = true // leave to CSI driver to handle corrupted mount klog.Warning(log("attacher.MountDevice detected corrupted mount for dir [%s]", deviceMountPath)) } else { return err } } if mounted && !corruptedDir { klog.V(4).Info(log("attacher.MountDevice skipping mount, dir already mounted [%s]", deviceMountPath)) return nil } </del> // Setup if spec == nil { return errors.New(log("attacher.MountDevice failed, spec is nil"))

// Store volume metadata for UnmountDevice. Keep it around even if the // driver does not support NodeStage, UnmountDevice still needs it. <del> if err = os.MkdirAll(deviceMountPath, 0750); err != nil && !corruptedDir { </del> <ins> if err = os.MkdirAll(deviceMountPath, 0750); err != nil { </ins> return errors.New(log("attacher.MountDevice failed to create dir %#v:  %v", deviceMountPath, err)) } klog.V(4).Info(log("created target path successfully [%s]", deviceMountPath))

t.Errorf("expected mount options: %v, got: %v", tc.spec.PersistentVolume.Spec.MountOptions, vol.MountFlags) } } <ins> // Verify the deviceMountPath was created by the plugin if tc.stageUnstageSet { s, err := os.Stat(tc.deviceMountPath) if err != nil { t.Errorf("expected staging directory %s to be created and be a directory, got error: %s", tc.deviceMountPath, err) } else { if !s.IsDir() { t.Errorf("expected staging directory %s to be directory, got something else", tc.deviceMountPath) } } } </ins> }) } }

func (c *csiMountMgr) SetUpAt(dir string, mounterArgs volume.MounterArgs) error { klog.V(4).Infof(log("Mounter.SetUpAt(%s)", dir)) <del> corruptedDir := false mounted, err := isDirMounted(c.plugin, dir) if err != nil { if isCorruptedDir(dir) { corruptedDir = true // leave to CSI driver to handle corrupted mount klog.Warning(log("mounter.SetUpAt detected corrupted mount for dir [%s]", dir)) } else { return errors.New(log("mounter.SetUpAt failed while checking mount status for dir [%s]: %v", dir, err)) } } if mounted && !corruptedDir { klog.V(4).Info(log("mounter.SetUpAt skipping mount, dir already mounted [%s]", dir)) return nil } </del> csi, err := c.csiClientGetter.Get() if err != nil { return volumetypes.NewTransientOperationFailure(log("mounter.SetUpAt failed to get CSI client: %v", err))

} // create target_dir before call to NodePublish <del> if err := os.MkdirAll(dir, 0750); err != nil && !corruptedDir { return errors.New(log("mounter.SetUpAt failed to create dir %#v:  %v", dir, err)) </del> <ins> parentDir := filepath.Dir(dir) if err := os.MkdirAll(parentDir, 0750); err != nil { return errors.New(log("mounter.SetUpAt failed to create dir %#v:  %v", parentDir, err)) </ins> } <del> klog.V(4).Info(log("created target path successfully [%s]", dir)) </del> <ins> klog.V(4).Info(log("created target path successfully [%s]", parentDir)) </ins> nodePublishSecrets = map[string]string{} if secretRef != nil {

DevicePath: devicePath, } <del> if volumeDeviceMounter != nil { </del> <ins> if volumeDeviceMounter != nil && actualStateOfWorld.GetDeviceMountState(volumeToMount.VolumeName) != DeviceGloballyMounted { </ins> deviceMountPath, err := volumeDeviceMounter.GetDeviceMountPath(volumeToMount.VolumeSpec) if err != nil {

if resizeError != nil { klog.Errorf("MountVolume.NodeExpandVolume failed with %v", resizeError) <ins> // Resize failed. To make sure NodeExpand is re-tried again on the next attempt // *before* SetUp(), mark the mounted device as uncertain. markDeviceUncertainErr := actualStateOfWorld.MarkDeviceAsUncertain( volumeToMount.VolumeName, devicePath, deviceMountPath) if markDeviceUncertainErr != nil { // just log, return the resizeError error instead klog.Infof(volumeToMount.GenerateMsgDetailed( "MountVolume.MountDevice failed to mark volume as uncertain", markDeviceUncertainErr.Error())) } </ins> return volumeToMount.GenerateError("MountVolume.MountDevice failed while expanding volume", resizeError) } }