Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pod was deleted successfully but there is still a lock on the ceph image. I find two logs,one for that lock not removed and another for that lock removed. I find that there are little difference between them. the log that lock not removed: and the log that lock removed: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.6.7 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): centos7 Kernel (e.g. ): 3.10.0-327.el7.x86_64 Install tools: Others:
/sig storage
Error 16 is EBUSY. similar to
were you running kubelet with cadvisor on?
/assign
yes, every kubelet run with cadvisor,and i know what code 16 represent for,but i still don't know the reason, is this a bug? How can i avoid this kind of situation from happening,thank you
It is probably related to
We see some other case may cause , kubelet run with nodeexporter and nodeexporter is deployed by k8s daemonset with -v /:/rootfs:ro (mount kubelet host rootfs into nodeexporter container). rbd device is mounted before nodeexporter started, because k8s 1.6.7 not support mount propagation, when the rbd plugin unmount /dev/rbd0 on the host namespace, the mount info will still leave over in node_exporter container namespace, may cause . hope this would help
thanks, does the problem persist if node_exporter stops collecting filesystem metrics?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
I'm experiencing this issue. From what I can read above, if the rbd mount happens before the node_exporter starts; we'll always experience status 16?
We're getting this issue. This is because the mounted volume is mounted on Cadvisor container too. So the /dev/rbdX file is still in use. But still didn't find any solution to fix.
rbd volume will stuck when failed as will always returen
My solution: container use a volumeMount. make a script file to be executed every a minute:

Discussion:
return err } // Unmount the device from the device mount point. <del> klog.V(4).Infof("rbd: unmouting device mountpoint %s", deviceMountPath) if err = detacher.mounter.Unmount(deviceMountPath); err != nil { </del> <ins> notMnt, err := detacher.mounter.IsLikelyNotMountPoint(deviceMountPath) if err != nil { </ins> return err } <del> klog.V(3).Infof("rbd: successfully umount device mountpath %s", deviceMountPath) </del> <ins> if !notMnt { klog.V(4).Infof("rbd: unmouting device mountpoint %s", deviceMountPath) if err = detacher.mounter.Unmount(deviceMountPath); err != nil { return err } klog.V(3).Infof("rbd: successfully umount device mountpath %s", deviceMountPath) } </ins> <del> klog.V(4).Infof("rbd: detaching device %s", devicePath) err = detacher.manager.DetachDisk(detacher.plugin, deviceMountPath, devicePath) if err != nil { return err </del> <ins> // Get devicePath from deviceMountPath if devicePath is empty if devicePath == "" { rbdImageInfo, err := getRbdImageInfo(deviceMountPath) if err != nil { return err } found := false devicePath, found = getRbdDevFromImageAndPool(rbdImageInfo.pool, rbdImageInfo.name) if !found { klog.Warningf("rbd: can't found devicePath for %v. Device is already unmounted, Image %v, Pool %v", deviceMountPath, rbdImageInfo.pool, rbdImageInfo.name) } } if devicePath != "" { klog.V(4).Infof("rbd: detaching device %s", devicePath) err = detacher.manager.DetachDisk(detacher.plugin, deviceMountPath, devicePath) if err != nil { return err } klog.V(3).Infof("rbd: successfully detach device %s", devicePath) </ins> } <del> klog.V(3).Infof("rbd: successfully detach device %s", devicePath) </del> err = os.Remove(deviceMountPath) if err != nil { return err

} } } <ins> func TestGetRbdImageInfo(t *testing.T) { tmpDir, err := utiltesting.MkTmpdir("rbd_test") if err != nil { t.Fatalf("error creating temp dir: %v", err) } defer os.RemoveAll(tmpDir) for i, c := range []struct { DeviceMountPath    string TargetRbdImageInfo *rbdImageInfo }{ { DeviceMountPath:    fmt.Sprintf("%s/plugins/kubernetes.io/rbd/rbd/pool1-image-image1", tmpDir), TargetRbdImageInfo: &rbdImageInfo{pool: "pool1", name: "image1"}, }, { DeviceMountPath:    fmt.Sprintf("%s/plugins/kubernetes.io/rbd/mounts/pool2-image-image2", tmpDir), TargetRbdImageInfo: &rbdImageInfo{pool: "pool2", name: "image2"}, }, } { rbdImageInfo, err := getRbdImageInfo(c.DeviceMountPath) if err != nil { t.Errorf("Case %d: getRbdImageInfo failed: %v", i, err) continue } if !reflect.DeepEqual(rbdImageInfo, c.TargetRbdImageInfo) { t.Errorf("Case %d: unexpected RbdImageInfo, wanted %v, got %v", i, c.TargetRbdImageInfo, rbdImageInfo) } } } </ins>

rbdImageSizeUnitMiB      = 1024 * 1024 ) <ins> // A struct contains rbd image info. type rbdImageInfo struct { pool string name string } </ins> func getDevFromImageAndPool(pool, image string) (string, bool) { device, found := getRbdDevFromImageAndPool(pool, image) if found {

return false, output, nil } } <ins> // getRbdImageInfo try to get rbdImageInfo from deviceMountPath. func getRbdImageInfo(deviceMountPath string) (*rbdImageInfo, error) { deviceMountedPathSeps := strings.Split(filepath.Base(deviceMountPath), "-image-") if len(deviceMountedPathSeps) != 2 { return nil, fmt.Errorf("Can't found devicePath for %s ", deviceMountPath) } return &rbdImageInfo{ pool: deviceMountedPathSeps[0], name: deviceMountedPathSeps[1], }, nil } </ins>