Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There is a drawback after PR: following code will fetch vmCache may lead to below error and the vmCache refresh time period would be 1 min, and may cost 1 min which is too long time How to fix this At least in Linux, we may still use parameter to get LUN, which is almost zero time. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
k8s 1.10.4, Azure (deployed via ACS-engine) do you know about any directly applicable workaround? This is hitting us quite hard. logs: logs: The last Attach operation reports success, yet we enter the same error loop over and over again.
thanks for reporting, I am fixing this issue, code should be ready in this week.
We have seen what looks like this issue - possible regression? It took 11m for the volume to attach.
it's more likely that disk could not be detected under dir inside vmss instance, is it a transient issue? Can this issue repro easily?
Seems to happen consistently on the 2nd pod of a new deploy. We have seen 3-4 times in a row deleting and re-deploying this project. Doesn't seem to be an issue for the kafka-0 and kafka-2 pods, just kafka-1. And only on a fresh deploy of the helm chart, deleting the pod does not reproduce.
what's your region and k8s version? it only happened on vmss, right?
The version we’re running is k8s 1.13.12 in useast2. And right this is on vmss. EDIT: I should also note that we're also using multi fault zones for each of the vmss pools we're deploying.
I'm having a similar issue, but I'm on 1.14.8. I don't know if is related, but these disk attachments seem to not be working properly on aks with vmms.
confirm volume attach with vmss is failing bad..
pls open a separate issue, provide detailed error info, e.g. k8s version, pod error description, etc. thanks.
This took too long AKS - kubernetes version -1.14.8 location -west europe
thanks for the reporting, I could confirm that the is due to PR: , which is going to mitigate vmss API throttling issue, in that PR, it enlarge the getVMSS TTL from 1min to 10min. Fortunately, from k8s 1.15.0, won't have this issue, since we have refactored the disk detect logic, it won't depend on getVMSS operation to get the disk LUN. In brief, pls upgrade to latest version 1.15.5, 1.15.x won't have this issue, thanks.
cherry-pick PR: to fix this issue on 1.14
You mention a 1.14 cherry pick but I'm also seeing Is this correct to assume 1.13 will have this fix as well ?
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! --/kind bug Uncomment only one, leave it on its own line: /kind bug /kind feature What happened: There is a drawback after PR: following code will fetch vmCache may lead to below error and the vmCache refresh time period would be 1 min, and may cost 1 min which is too long time At least in Linux, we may still use parameter to get LUN, which is almost zero time. Anything else we need to know?: Kubernetes version (use ): v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
1.13 don't accept any cherry-pick, so only 1.14 could be cherry-picked now, target fixed version is 1.14.10
the delay still can be very big this is because azure api call for disk attach returns after at 35s-2min and kubelet goes to its exp. backoff retries so it can reach 2min timeout adding attached recording: called 3 disk simultaneous attach with interval ~2s and saw and So actual attachment takes ~10-30s (device appears in /dev/disk/azure/scsi1) , but due to azure api we can loop on it I saw also that disk attach timing is unstable and somtemes it can take 2.5min , what do you think about logic like below? is not waiting for attachdisk to return, instead it calls GET until it returns lun id (we saw it takes few seconds after attachdisk). after GET returns lun id , it updates node's disks with lunId and kubelet can continue with mount after waiting for /dev/disk/azure/scsi1/lun*
same here
It takes about 10 minutes to recover from the FailedMount messages.
what's your k8s version?
there is a new vhd disk feature based on azure file which could attach/detach disk < 1s, consider that as an option if you have concern about disk attach/detach time cost:
AKS v1.14.8
for this specific issue, aks 1.15.x won't have this issue.
hi ,I am not dare to upgrade.because my stateful rabbitmq with pvc can not started because of failmount of PVC!
you could scale your statefulset to 0 replica, and pvc disk would be detached. After upgrade, scale back to original count.
okay,let me have a try.
I use helm to can I restore to two replicas after upgrading k8s version?

Discussion:
} func (a *azureDiskAttacher) WaitForAttach(spec *volume.Spec, devicePath string, _ *v1.Pod, timeout time.Duration) (string, error) { <del> var err error </del> volumeSource, _, err := getVolumeSource(spec) if err != nil { return "", err

nodeName := types.NodeName(a.plugin.host.GetHostName()) diskName := volumeSource.DiskName <del> glog.V(2).Infof("azureDisk - WaitForAttach: begin to GetDiskLun by diskName(%s), DataDiskURI(%s), nodeName(%s), devicePath(%s)", diskName, volumeSource.DataDiskURI, nodeName, devicePath) lun, err := diskController.GetDiskLun(diskName, volumeSource.DataDiskURI, nodeName) if err != nil { return "", err </del> <ins> var lun int32 if runtime.GOOS == "windows" { glog.V(2).Infof("azureDisk - WaitForAttach: begin to GetDiskLun by diskName(%s), DataDiskURI(%s), nodeName(%s), devicePath(%s)", diskName, volumeSource.DataDiskURI, nodeName, devicePath) lun, err = diskController.GetDiskLun(diskName, volumeSource.DataDiskURI, nodeName) if err != nil { return "", err } glog.V(2).Infof("azureDisk - WaitForAttach: GetDiskLun succeeded, got lun(%v)", lun) } else { lun, err = getDiskLUN(devicePath) if err != nil { return "", err } </ins> } <del> glog.V(2).Infof("azureDisk - WaitForAttach: GetDiskLun succeeded, got lun(%v)", lun) </del> exec := a.plugin.host.GetExec(a.plugin.GetPluginName()) io := &osIOHandler{}

"io/ioutil" "os" "path/filepath" <ins> "regexp" "strconv" </ins> libstrings "strings" "github.com/Azure/azure-sdk-for-go/services/storage/mgmt/2018-07-01/storage"

string(api.AzureSharedBlobDisk), string(api.AzureDedicatedBlobDisk), string(api.AzureManagedDisk)) <ins> lunPathRE = regexp.MustCompile(`/dev/disk/azure/scsi(?:.*)/lun(.+)`) </ins> ) func getPath(uid types.UID, volName string, host volume.VolumeHost) string {

} return libstrings.ToUpper(string(str[0])) + str[1:] } <ins> // getDiskLUN : deviceInfo could be a LUN number or a device path, e.g. /dev/disk/azure/scsi1/lun2 func getDiskLUN(deviceInfo string) (int32, error) { var diskLUN string if len(deviceInfo) <= 2 { diskLUN = deviceInfo } else { // extract the LUN num from a device path matches := lunPathRE.FindStringSubmatch(deviceInfo) if len(matches) == 2 { diskLUN = matches[1] } else { return -1, fmt.Errorf("cannot parse deviceInfo: %s", deviceInfo) } } lun, err := strconv.Atoi(diskLUN) if err != nil { return -1, err } return int32(lun), nil } </ins>

assert.Equal(t, err != nil, test.expectError, fmt.Sprintf("error msg: %v", err)) } } <ins> func TestGetDiskLUN(t *testing.T) { tests := []struct { deviceInfo  string expectedLUN int32 expectError bool }{ { deviceInfo:  "0", expectedLUN: 0, expectError: false, }, { deviceInfo:  "10", expectedLUN: 10, expectError: false, }, { deviceInfo:  "11d", expectedLUN: -1, expectError: true, }, { deviceInfo:  "999", expectedLUN: -1, expectError: true, }, { deviceInfo:  "", expectedLUN: -1, expectError: true, }, { deviceInfo:  "/dev/disk/azure/scsi1/lun2", expectedLUN: 2, expectError: false, }, { deviceInfo:  "/dev/disk/azure/scsi0/lun12", expectedLUN: 12, expectError: false, }, { deviceInfo:  "/dev/disk/by-id/scsi1/lun2", expectedLUN: -1, expectError: true, }, } for _, test := range tests { result, err := getDiskLUN(test.deviceInfo) assert.Equal(t, result, test.expectedLUN) assert.Equal(t, err != nil, test.expectError, fmt.Sprintf("error msg: %v", err)) } } </ins>