Change:
glog.Warningf("Warning: Unmount skipped because path does not exist: %v", deviceMountPath) return nil } <del> devicePath, cnt, err := mount.GetDeviceNameFromMount(detacher.mounter, deviceMountPath) </del> <ins> devicePath, _, err := mount.GetDeviceNameFromMount(detacher.mounter, deviceMountPath) </ins> if err != nil { return err } <del> if cnt > 1 { return fmt.Errorf("rbd: more than 1 reference counts at %s", deviceMountPath) } if cnt == 1 { // Unmount the device from the device mount point. glog.V(4).Infof("rbd: unmouting device mountpoint %s", deviceMountPath) if err = detacher.mounter.Unmount(deviceMountPath); err != nil { return err } glog.V(3).Infof("rbd: successfully umount device mountpath %s", deviceMountPath) </del> <ins> // Unmount the device from the device mount point. glog.V(4).Infof("rbd: unmouting device mountpoint %s", deviceMountPath) if err = detacher.mounter.Unmount(deviceMountPath); err != nil { return err </ins> } <ins> glog.V(3).Infof("rbd: successfully umount device mountpath %s", deviceMountPath) </ins> glog.V(4).Infof("rbd: detaching device %s", devicePath) err = detacher.manager.DetachDisk(detacher.plugin, deviceMountPath, devicePath) if err != nil {

Discussion:
Some mount propagation magic in Kubernetes on Ubuntu 16.04 on GCE duplicates every mount. Steps to reproduce: On recent master. install Kubernetes on GCE on Ubuntu: Run a pod with a GCE PD volume: The volume looks like it's mounted twice in output on the node: Deleting this pod will unmount everything correctly in GCE PD case, however it won't unmount Ceph RBD: There seems to be some bind-mount && rshared propagation magic happening in Google's helper mounter container that makes a mount appear twice. Note that MountPropagation and MountContainers are disabled in this cluster, so it can't be the culprit. I can probably fix reference counting in RBD, however this doubling of mounts looks very ugly and may have other consequences than just failed RBD teardown. /kind bug /sig storage
cc
I found the culprit, it's simple bind-mount , it's not just below it. Any mount in will appear twice in . It's enough to unmount it once, both of the mounts will then disappear. It's the same on GCI and on Ubuntu. We can either add an option to disable on Ubuntu or fix all the volume plugins not to depend on a volume mounted only once (e.g. ). Fixing the plugins will fix it on all deployments that may be bind-mounting stuff to weird places.
Hi Jan, I was trying to better understand --bind and --rbind in mount. Reading [1] I see the following: "Note that the filesystem mount options will remain the same as those on the original mount point, and cannot be changed by passing the -o option along with --bind/--rbind. The mount options can be changed by a separate remount command" Yet L1463 [2] and L14634 do exactly this, eg: mount --bind -o ro /proc "${CONTAINERIZEDMOUNTERROOTFS}/proc" Would you mind explaining? Thanks, Jeff [1] [2]
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity.

Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
sorry that I missed this issue earlier. Have you fixed it or is it still an issue?
I worked around it in