Change:
What happened: After running a rolling-update on a StatefulSet that uses Azure Disks, part of the replicas remain in a state forever, exhibiting the following events: Restarting the occasionally unblocks the situation: What you expected to happen: Attach/detach operations succeed consistently. How to reproduce it (as minimally and precisely as possible): Here is a copy of the manifest of the affected StatefulSet: <details<summarysolr-</summary</details</brAnd the default StorageClass: <details</summary</details</brExtra information: Two other kind of Pod events that are regularly associated with the error above: kubelet logs of a node where attaching the volume is failing kube-controller-manager logs in an environment where attaching the volume is failing Environment: Kubernetes version (use ): v1.10.10 Cloud provider or hardware configuration: Azure (westeurope) OS (e.g. from /etc/os-release): 16.04.4 LTS Kernel (e.g. ): 4.13.0-1018-azure Install tools: ACS-Engine Others: /kind bug /sig storage /sig azure this might be of interest for you, since you were involved in , which was supposed to fix attach/detach issues in Azure.
/assign
fix the attach disk issue in kubelet, while in your case, you restarted the controller manager, and then it works. It looks like disk volume is never attached to vm successfully before you restart controller manager.
Indeed. For completeness I updated the description and 2 more type of events that are regularly associated to this issue.
could you check activity logs in that resource group, the time should be before you restart controller manager, you could check whether there is any disk related issue. Since you have already restarted controller manager, we cannot get all logs before your restart.
I see a bunch of the following failures in the activity log, but the errors stopped 4h before I gathered the logs and restarted the controller-manager: <details<summaryactivity log</summary</details</brI managed to grab the kube-controller-manager logs from an environment where the situation is currently happening. See the updated description.
Hi would you add my slack: andyzhang workspace?
I think your issue is here, you are using strategy According to , if statefulset replica is using azure disk pvc, is not the right way, pls change the strategy to 'Recreate' Update: may be not related to that issue, still in investigation.
In our deployment every Pod uses a distinct Azure Disk, so there is not multi-attachment. I'm still waiting for feedback from customers, hoping this happens again today.
I have tried in my testing env with v1.10.10 on vmss nodes by , there could be error like following after disk attached to that VM, the disk detect action in vmss node seems slow sometimes, while in the end it could be recovered:
thanks for looking into it. In our case the disk never gets attached (1d in ContainerCreating state).
Hi I found an regression issue of , will send a PR soon. Thanks for reporting.

Discussion:
klog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk(%s)", nodeResourceGroup, vmName, diskName) ctx, cancel := getContextWithCancel() defer cancel() <del> if _, err := as.VirtualMachinesClient.CreateOrUpdate(ctx, nodeResourceGroup, vmName, newVM); err != nil { </del> <ins> _, err = as.VirtualMachinesClient.CreateOrUpdate(ctx, nodeResourceGroup, vmName, newVM) if err != nil { </ins> klog.Errorf("azureDisk - attach disk(%s) failed, err: %v", diskName, err) detail := err.Error() if strings.Contains(detail, errLeaseFailed) || strings.Contains(detail, errDiskBlobNotFound) {

klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk(%s)", nodeResourceGroup, vmName, diskName) ctx, cancel := getContextWithCancel() defer cancel() <del> if _, err := as.VirtualMachinesClient.CreateOrUpdate(ctx, nodeResourceGroup, vmName, newVM); err != nil { </del> <ins> _, err = as.VirtualMachinesClient.CreateOrUpdate(ctx, nodeResourceGroup, vmName, newVM) if err != nil { </ins> klog.Errorf("azureDisk - detach disk(%s) failed, err: %v", diskName, err) } else { klog.V(2).Infof("azureDisk - detach disk(%s) succeeded", diskName)

ctx, cancel := getContextWithCancel() defer cancel() klog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk(%s)", nodeResourceGroup, nodeName, diskName) <del> if _, err := ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, vm); err != nil { </del> <ins> _, err = ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, vm) if err != nil { </ins> detail := err.Error() if strings.Contains(detail, errLeaseFailed) || strings.Contains(detail, errDiskBlobNotFound) { // if lease cannot be acquired or disk not found, immediately detach the disk and return the original error

ctx, cancel := getContextWithCancel() defer cancel() klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk(%s)", nodeResourceGroup, nodeName, diskName) <del> if _, err := ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, vm); err != nil { </del> <ins> _, err = ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, vm) if err != nil { </ins> klog.Errorf("azureDisk - detach disk(%s) from %s failed, err: %v", diskName, nodeName, err) } else { klog.V(2).Infof("azureDisk - detach disk(%s) succeeded", diskName)