Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): glusterfs, umount Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): Linux hostname 4.4.0-59-generic -Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: terraform/ansible Others: What happened: Kubernetes allows users to delete a glusterfs pvc before it is has been umounted from all kubelets. This leaves orphaned mount points on the kubelet boxes and the kubelets looping trying and remove them. What you expected to happen: The kubelet should successfully remove the mountpoints. How to reproduce it (as minimally and precisely as possible): Example deployment with with a heketi/glusterfs storageclass. gluster- To reproduce : kubectl create -f gluster- <wait for all containers to come upkubectl delete -f gluster- This will cause heketi to remove the glusterfs volume before some of the pods have umounted it. All mount points still online will error with: 'transport endpoint is not connected' These can still be umounted, however the way the code is written it will try to "stat" the mount point before trying the umount. This will error with something like the following: This will prevent the umount and pod cleanup. Anything else we need to know: The code looks something like this: I realise the importance of checking to see if a mountpoint is still up before removing it. My first thoughts were to restrict the stat checks to only error if the mount point didn't actually exist. Also checking /proc/mounts seems like a more reliable method for checking if something is mounted on linux.
Would prevent this scenario?
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

Discussion:
deps = [ "//pkg/apis/core/install:go_default_library", "//pkg/apis/core/v1/helper:go_default_library", <ins> "//pkg/util/mount:go_default_library", </ins> "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library", <del> ] + select({ "@io_bazel_rules_go//go/platform:linux": [ "//vendor/k8s.io/client-go/util/testing:go_default_library", ], "//conditions:default": [], }), </del> <ins> "//vendor/k8s.io/client-go/util/testing:go_default_library", ], </ins> ) filegroup(

"path" "path/filepath" "strings" <ins> "syscall" </ins> "github.com/golang/glog" "k8s.io/api/core/v1"

// IsNotMountPoint will be called instead of IsLikelyNotMountPoint. // IsNotMountPoint is more expensive but properly handles bind mounts. func UnmountMountPoint(mountPath string, mounter mount.Interface, extensiveMountPointCheck bool) error { <del> if pathExists, pathErr := PathExists(mountPath); pathErr != nil { return fmt.Errorf("Error checking if path exists: %v", pathErr) } else if !pathExists { </del> <ins> pathExists, pathErr := PathExists(mountPath) if !pathExists { </ins> glog.Warningf("Warning: Unmount skipped because path does not exist: %v", mountPath) return nil } <del> var notMnt bool var err error if extensiveMountPointCheck { notMnt, err = mount.IsNotMountPoint(mounter, mountPath) } else { notMnt, err = mounter.IsLikelyNotMountPoint(mountPath) </del> <ins> corruptedMnt := isCorruptedMnt(pathErr) if pathErr != nil && !corruptedMnt { return fmt.Errorf("Error checking path: %v", pathErr) </ins> } <ins> return doUnmountMountPoint(mountPath, mounter, extensiveMountPointCheck, corruptedMnt) } </ins> <del> if err != nil { return err } </del> <ins> // doUnmountMountPoint is a common unmount routine that unmounts the given path and // deletes the remaining directory if successful. // if extensiveMountPointCheck is true // IsNotMountPoint will be called instead of IsLikelyNotMountPoint. // IsNotMountPoint is more expensive but properly handles bind mounts. // if corruptedMnt is true, it means that the mountPath is a corrupted mountpoint, Take it as an argument for convenience of testing func doUnmountMountPoint(mountPath string, mounter mount.Interface, extensiveMountPointCheck bool, corruptedMnt bool) error { if !corruptedMnt { var notMnt bool var err error if extensiveMountPointCheck { notMnt, err = mount.IsNotMountPoint(mounter, mountPath) } else { notMnt, err = mounter.IsLikelyNotMountPoint(mountPath) } </ins> <del> if notMnt { glog.Warningf("Warning: %q is not a mountpoint, deleting", mountPath) return os.Remove(mountPath) </del> <ins> if err != nil { return err } if notMnt { glog.Warningf("Warning: %q is not a mountpoint, deleting", mountPath) return os.Remove(mountPath) } </ins> } // Unmount the mount path

} notMnt, mntErr := mounter.IsLikelyNotMountPoint(mountPath) if mntErr != nil { <del> return err </del> <ins> return mntErr </ins> } if notMnt { glog.V(4).Infof("%q is unmounted, deleting the directory", mountPath)

return true, nil } else if os.IsNotExist(err) { return false, nil <ins> } else if isCorruptedMnt(err) { return true, err </ins> } else { return false, err } } <ins> // isCorruptedMnt return true if err is about corrupted mount point func isCorruptedMnt(err error) bool { if err == nil { return false } var underlyingError error switch pe := err.(type) { case nil: return false case *os.PathError: underlyingError = pe.Err case *os.LinkError: underlyingError = pe.Err case *os.SyscallError: underlyingError = pe.Err } return underlyingError == syscall.ENOTCONN || underlyingError == syscall.ESTALE } </ins> // GetSecretForPod locates secret by name in the pod's namespace and returns secret map func GetSecretForPod(pod *v1.Pod, secretName string, kubeClient clientset.Interface) (map[string]string, error) { secret := make(map[string]string)

"k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/sets" <ins> utiltesting "k8s.io/client-go/util/testing" </ins> // util.go uses api.Codecs.LegacyCodec so import this package to do some // resource initialization. _ "k8s.io/kubernetes/pkg/apis/core/install" "k8s.io/kubernetes/pkg/apis/core/v1/helper" <ins> "k8s.io/kubernetes/pkg/util/mount" </ins> ) var nodeLabels map[string]string = map[string]string{

} } } <ins> func TestDoUnmountMountPoint(t *testing.T) { tmpDir1, err1 := utiltesting.MkTmpdir("umount_test1") if err1 != nil { t.Fatalf("error creating temp dir: %v", err1) } defer os.RemoveAll(tmpDir1) tmpDir2, err2 := utiltesting.MkTmpdir("umount_test2") if err2 != nil { t.Fatalf("error creating temp dir: %v", err2) } defer os.RemoveAll(tmpDir2) // Second part: want no error tests := []struct { mountPath    string corruptedMnt bool }{ { mountPath:    tmpDir1, corruptedMnt: true, }, { mountPath:    tmpDir2, corruptedMnt: false, }, } fake := &mount.FakeMounter{} for _, tt := range tests { err := doUnmountMountPoint(tt.mountPath, fake, false, tt.corruptedMnt) if err != nil { t.Errorf("err Expected nil, but got: %v", err) } } } </ins>