Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In a customer , customer is unable to use azure disk in StatefulSet since /dev/sd changed after detach/attach disk. Currently we are using to reference the data disk which could be changed in some condition, see code: What you expected to happen: How to reproduce it (as minimally and precisely as possible): 5 StatefulSet with 1 replica and 1 azure disk mount each 1 StatefulSet and recreate the same StatefulSet would be azure disks could not be usable since /dev/sd changed Update: It seems that on azure china, this issue could repro very easily. Anything else we need to know?: I have checked the official doc: Environment: Kubernetes version (use ): from v1.6.0 to v1.10.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign I am working on a fix for this issue, would use instead of
We changed how we find disks on nodes quite some time back. What server version is this?
yes, I have already checked the code, the point is we are still using (code is ) to "mount -bind" k8s path, while could be changed when VM is attach/detaching data disks, see And I have also checked related AWS, GCE code, they are using other than ,And I have also checked related AWS, GCE code, they are using other than , see This issue could be stably repro on Azure China, while I could not repro that on other region up to now. I have already got a fix for this, still in testing now.
also add for awareness, I am working on a fix to use other than .
I would not use "/dev/disk/by-id" instead utilize the special UDEV rules we have created for our VMs -- They have the advantage that they create symlinks for all of the disks attached to the VM and the LUN assigned to a disk remains the same. The sysmlinks you find under /dev/disk/azure. For a master-node you will see a layout like the following. In scsi1 you find the additonal (data) disks A newly assigned disk is goign to appear as lun1,lun2,... Since we are not able to guaranty the order in which disks get assigned to a VM I would go and base the code on this already created layout
thanks for the check. So do you mean we use , instead? actually in Azure VM with UDEV rules created, they have no difference, they link to the same /dev/sd* (see below layout). While I also got some customer cases(e.g. coreos, see ), they don't have such UDEV rules created, which means does not exist, I would say would be more common, every linux distro have that path. From debugging perspective, the path is more intuitive.
BTW, I have a azure disk debugging doc: which uses , it's really intuitive for debugging perpurse.
it is right that at the end we point to /dev/sdi like in the sample above the advantage we have with the lun0 is that it acts as an abstraction layer where we can be sure that an application is always utilizing the disk with lun0, independent what actuall device name gets assigned to it. So at the moment it is ../../../sdi and after a reboot it could become ../../../sdk. But thanks to these UDEV rules /dev/disk/azure/scsi1/lun0 will always point to the right device behind it. I also would favour the usage of the lun over the disk-id if from a maintanance/support point of view as we can directly correlate what lun belongs to what disk. Otherwise I have at first fo to the VM directly to find out what disk might have an issue Regarding the centos issue, if we have the waagent installed on it then those new UDEV rules should be available as well. So it wonders me that we do not have them working well. At least I had not seen any customer SR who had reported an issue with those rules specifically on CentOS
It's not centos, it's coreos issue, see If we use , then we would heavily depends on UDEV rules.
I agree that one depends on these special UDEV rules in this case , but on each Azure VM the agent is to be installed anyway, so that is background why I suggested to use the LUNs
I think proposal is the cleanest and safest. However we will have to tell all Azure users that they have to have azure UDEV rules configured in nodes, even if WAAGENT is not installed.
I did hit the issue, using an ACS based Kubernetes cluster running 1.7.7. We are already in the process of moving over to AKS based clusters, so was simply hoping that there would be some escalated effort to bring 1.8.7 into AKS given that the azure-disk provisioner is the default StorageClass.Not sure if you are looking for any more debug info at this time, but I still have the affected pod active and should be able to get you some more information, so just let me know. that's great, I would like to take a look to make sure it's that issue, could you follow this to collect the data.
Sadly the node became NotReady, and the cluster redeployed the pod to a separate node, which has rectified the issue. We have been seeing the issue with far frequency, so I will make sure to gather the data next time it pops up.
That said, the reason I was confident it was this issue previously is that when I exec'd into the shell, I got the "Input / Output error" upon trying to "ls" the mount point. Furthermore, attempting to re-mount that point gave an error to the extent of "Special device /dev/sde does not exist"
Could you recall the VM had restarted before you hit that disk issue? Or just after attach/detach disk, that issue come up?

Discussion:
package azure_dd import ( <ins> "fmt" </ins> "path" "strconv" libstrings "strings"

return azureDiskList } <ins> // getDiskLinkByDevName get disk link by device name from devLinkPath, e.g. /dev/disk/azure/, /dev/disk/by-id/ func getDiskLinkByDevName(io ioHandler, devLinkPath, devName string) (string, error) { dirs, err := io.ReadDir(devLinkPath) glog.V(12).Infof("azureDisk - begin to find %s from %s", devName, devLinkPath) if err == nil { for _, f := range dirs { diskPath := devLinkPath + f.Name() glog.V(12).Infof("azureDisk - begin to Readlink: %s", diskPath) link, linkErr := io.Readlink(diskPath) if linkErr != nil { glog.Warningf("azureDisk - read link (%s) error: %v", diskPath, linkErr) continue } if libstrings.HasSuffix(link, devName) { return diskPath, nil } } return "", fmt.Errorf("device name(%s) is not found under %s", devName, devLinkPath) } return "", fmt.Errorf("read %s error: %v", devLinkPath, err) } </ins> func scsiHostRescan(io ioHandler, exec mount.Exec) { scsi_path := "/sys/class/scsi_host/" if dirs, err := io.ReadDir(scsi_path); err == nil {

dir := path.Join(sys_path, name, "block") if dev, err := io.ReadDir(dir); err == nil { found := false <ins> devName := dev[0].Name() </ins> for _, diskName := range azureDisks { <del> glog.V(12).Infof("azure disk - validating disk %q with sys disk %q", dev[0].Name(), diskName) if string(dev[0].Name()) == diskName { </del> <ins> glog.V(12).Infof("azureDisk - validating disk %q with sys disk %q", devName, diskName) if devName == diskName { </ins> found = true break } } if !found { <del> return "/dev/" + dev[0].Name(), nil </del> <ins> devLinkPaths := []string{"/dev/disk/azure/scsi1/", "/dev/disk/by-id/"} for _, devLinkPath := range devLinkPaths { diskPath, err := getDiskLinkByDevName(io, devLinkPath, devName) if err == nil { glog.V(4).Infof("azureDisk - found %s by %s under %s", diskPath, devName, devLinkPath) return diskPath, nil } glog.Warningf("azureDisk - getDiskLinkByDevName by %s under %s failed, error: %v", devName, devLinkPath, err) } return "/dev/" + devName, nil </ins> } } }