Change:
/kind bug What happened: Kubelet failed to unmount volumes if the backend ISCSI volume is not available. This also applies to other network attached volumes. What you expected to happen: Kubelet should be able to unmount the volumes. A manual unmount works even when the backend network volume is not available. How to reproduce it (as minimally and precisely as possible): Mount an ISCSI/network volume and disconnect/de-provision ISCSI target. Anything else we need to know?: XFS shuts down the filesystem due to I/O errors. Kubelet uses "stat" calls to figure out if it is a valid mount point or it is already unmounted. Stat calls fail with EIO and Kubelet never proceeds to unmount the volume. Environment: Kubernetes version (use ): Kubernetes 1.10.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): Centos 7.4 Kernel (e.g. ): Linux appserv37 3.10.0-693.el7.x8664 SMP Tue Aug 22 21:09:27 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Manual bring up Others:
/sig storage
IsLikelyNotMountPoint is using stat call to figure the filesystem is mounted or not. Stat calls are failing with EIO when there are I/O errors on the backend and Kubelet never proceeds to unmount the filesystem. Also, isLikelyNotMountPoint is not looking for "isCorruptedMnt" errors. All other calls have it.
looks like the nfs plugin is calling . Can iscsi use that?
didn't include error "Input/output error" () We should add it. I make a fix, PTAL, Thank you!
Thanks. Trying out the change. Will keep you posted if we need any other changes.
Stock ISCSI driver is working fine with EIO change. But more changes are required for making it work with Flexvolume drivers. I have the change list. I can club your change in or let me know if you want to go ahead and push this fix. I will push another PR to fix flexvolume issues.
Please club my change, I will close my PR. Thanks!

Discussion:
// UnmountDevice is part of the volume.Detacher interface. func (d *flexVolumeDetacher) UnmountDevice(deviceMountPath string) error { <del> if pathExists, pathErr := util.PathExists(deviceMountPath); pathErr != nil { return fmt.Errorf("Error checking if path exists: %v", pathErr) } else if !pathExists { </del> <ins> pathExists, pathErr := util.PathExists(deviceMountPath) if !pathExists { </ins> glog.Warningf("Warning: Unmount skipped because path does not exist: %v", deviceMountPath) return nil } <ins> if pathErr != nil && !util.IsCorruptedMnt(pathErr) { return fmt.Errorf("Error checking path: %v", pathErr) } </ins> notmnt, err := isNotMounted(d.plugin.host.GetMounter(d.plugin.GetPluginName()), deviceMountPath) if err != nil { <del> return err </del> <ins> if util.IsCorruptedMnt(err) { notmnt = false // Corrupted error is assumed to be mounted. } else { return err } </ins> } if notmnt { glog.Warningf("Warning: Path: %v already unmounted", deviceMountPath) } else {

func (f *flexVolumeUnmounter) TearDownAt(dir string) error { <del> if pathExists, pathErr := util.PathExists(dir); pathErr != nil { return fmt.Errorf("Error checking if path exists: %v", pathErr) } else if !pathExists { </del> <ins> pathExists, pathErr := util.PathExists(dir) if !pathExists { </ins> glog.Warningf("Warning: Unmount skipped because path does not exist: %v", dir) return nil } <ins> if pathErr != nil && !util.IsCorruptedMnt(pathErr) { return fmt.Errorf("Error checking path: %v", pathErr) } </ins> call := f.plugin.NewDriverCall(unmountCmd) call.Append(dir) _, err := call.Run()

glog.Warningf("Warning: Unmount skipped because path does not exist: %v", mountPath) return nil } <del> corruptedMnt := isCorruptedMnt(pathErr) </del> <ins> corruptedMnt := IsCorruptedMnt(pathErr) </ins> if pathErr != nil && !corruptedMnt { return fmt.Errorf("Error checking path: %v", pathErr) }

return true, nil } else if os.IsNotExist(err) { return false, nil <del> } else if isCorruptedMnt(err) { </del> <ins> } else if IsCorruptedMnt(err) { </ins> return true, err } else { return false, err } } <del> // isCorruptedMnt return true if err is about corrupted mount point func isCorruptedMnt(err error) bool { </del> <ins> // IsCorruptedMnt return true if err is about corrupted mount point func IsCorruptedMnt(err error) bool { </ins> if err == nil { return false }

case *os.SyscallError: underlyingError = pe.Err } <del> return underlyingError == syscall.ENOTCONN || underlyingError == syscall.ESTALE </del> <ins> return underlyingError == syscall.ENOTCONN || underlyingError == syscall.ESTALE || underlyingError == syscall.EIO </ins> } // GetSecretForPod locates secret by name in the pod's namespace and returns secret map