Change:
return true, nil } else if os.IsNotExist(err) { return false, nil <ins> } else if isCorruptedMnt(err) { return true, err </ins> } else { return false, err } } <ins> // isCorruptedMnt return true if err is about corrupted mount point func isCorruptedMnt(err error) bool { if err == nil { return false } var underlyingError error switch pe := err.(type) { case nil: return false case *os.PathError: underlyingError = pe.Err case *os.LinkError: underlyingError = pe.Err case *os.SyscallError: underlyingError = pe.Err } return underlyingError == syscall.ENOTCONN || underlyingError == syscall.ESTALE } </ins> // GetSecretForPod locates secret by name in the pod's namespace and returns secret map func GetSecretForPod(pod *v1.Pod, secretName string, kubeClient clientset.Interface) (map[string]string, error) { secret := make(map[string]string)

Discussion:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): glusterfs, umount Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): Linux hostname 4.4.0-59-generic -Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: terraform/ansible Others: What happened: Kubernetes allows users to delete a glusterfs pvc before it is has been umounted from all kubelets. This leaves orphaned mount points on the kubelet boxes and the kubelets looping trying and remove them.

What you expected to happen: The kubelet should successfully remove the mountpoints. How to reproduce it (as minimally and precisely as possible): Example deployment with with a heketi/glusterfs storageclass. gluster- To reproduce : kubectl create -f gluster- <wait for all containers to come upkubectl delete -f gluster- This will cause heketi to remove the glusterfs volume before some of the pods have umounted it. All mount points still online will error with: 'transport endpoint is not connected' These can still be umounted, however the way the code is written it will try to "stat" the mount point before trying the umount. This will error with something like the following: This will prevent the umount and pod cleanup. Anything else we need to know: The code looks something like this: I realise the importance of checking to see if a mountpoint is still up before removing it. My first thoughts were to restrict the stat checks to only error if the mount point didn't actually exist. Also checking /proc/mounts seems like a more reliable method for checking if something is mounted on linux.
Would prevent this scenario?
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details