Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Same symtoms as The detach is not successful until controller-manager is restarted controller-manager logs: What you expected to happen: The detach operation should know that a node is not active or its been deleted and should assume the disk is detached. How to reproduce it (as minimally and precisely as possible): This seems quite hard to reproduce, but we do run into it quite a few times where the pod is stuck in , because the controller-manager is unable to detach disks from deleted nodes. Anything else we need to know?: Environment: Kubernetes version (use ): v1.15.11 Cloud provider or hardware configuration: azure (self-managed) OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): 5.0.0-1027-azure Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /area provider/azure paging
/sig azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
hi have you done sth manually on the node? if node is deleted finally, the disk would be in unattached state finally, then disk attach to another node would succeed finally.
as discussed offline, the issue is probably due to vmss instance compute name change(incorrect format: , correct format: ), user should not change vmss instance compute name, otherwise it would break disk attach/detach, since compute name is used in vmss node name passing, code is here:
Hi We don't change the instance compute name - however what i do see in the azure metadata service is in the form - What you're referring to seems to be the computer name, and we do use the defaults as provided by azure. See The attach/detach operations works fairly well most of the time, and I think we're running into some edge case here. One thing i might have missed out is the fact that there seems to have been an issue with the VM at the time. So the VM was present, but azure platform would return 400 bad request for a detach operation which probably led to this .. But when the VM was eventually deleted by the autoscaler, the re-attach was never attempted by the reconciler it would seem. and the last message about that disk was
should be a valid compute name according to , do you know why we define the format like this?
The function is used to get the name from IMDS, and its format is .
From the above errors, seems the request has been sent to VMSS but failed because it's not active. Do you know the VM state during that time?
I am not sure of the state at the time, but i believe the VM was in some kind of failed state. The VM has since been deleted, but we had errors on the scaleset at the time. The events on the VMSS look like this. The VM was eventually deleted by the autoscaler, but the disk was never re-attached to a different node, and the pod remained in for about 2 days until controller-manager was restarted.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Apparently we hit this issue today as a result of replacing the nodepool. We 2 new nodepools and deleted the old one, and now we have 718 of 1672 pods stuck with this error in one of our clusters. The volumes are failing to be attached to new pods, the VMSS for old nodepool is removed. In AKS portal disks are still reported to be attached to VM instances in the VMSS that was deleted. hundreds of pods are stuck in Init state won't likely go forward. /remove-lifecycle rotten
/remove-lifecycle rotten
I have seen similar issue when handling one customer's case, it's probably caused by broken vmss cluster, pls file a ticket to VMSS team to fix the issue.
I happened to find this issue in one customer case, this issue is caused by obsolete cache in kube-controller-manager, should skip attach/detach retry when it's 400, below are the details, following two instances were deleted 3min ago before kube-controller-manager trying to attach/detach: workaround restart kube-controller-manager should work
/assign

Related discussion:
text: c.diskAttachDetachMap.Delete(strings.ToLower(diskURI)) c.vmLockMap.UnlockEntry(strings.ToLower(string(nodeName))) <del> if err != nil && retry.IsErrorRetriable(err) && c.cloud.CloudProviderBackoff { klog.V(2).Infof("azureDisk - update backing off: detach disk(%s, %s), err: %v", diskName, diskURI, err) retryErr := kwait.ExponentialBackoff(c.cloud.RequestBackoff(), func() (bool, error) { c.vmLockMap.LockEntry(strings.ToLower(string(nodeName))) c.diskAttachDetachMap.Store(strings.ToLower(diskURI), "detaching") err := vmset.DetachDisk(diskName, diskURI, nodeName) c.diskAttachDetachMap.Delete(strings.ToLower(diskURI)) c.vmLockMap.UnlockEntry(strings.ToLower(string(nodeName))) retriable := false if err != nil && retry.IsErrorRetriable(err) { retriable = true </del> <ins> if err != nil { if isInstanceNotFoundError(err) { // if host doesn't exist, no need to detach klog.Warningf("azureDisk - got InstanceNotFoundError(%v), DetachDisk(%s) will assume disk is already detached", err, diskURI) return nil } if retry.IsErrorRetriable(err) && c.cloud.CloudProviderBackoff { klog.Warningf("azureDisk - update backing off: detach disk(%s, %s), err: %v", diskName, diskURI, err) retryErr := kwait.ExponentialBackoff(c.cloud.RequestBackoff(), func() (bool, error) { c.vmLockMap.LockEntry(strings.ToLower(string(nodeName))) c.diskAttachDetachMap.Store(strings.ToLower(diskURI), "detaching") err := vmset.DetachDisk(diskName, diskURI, nodeName) c.diskAttachDetachMap.Delete(strings.ToLower(diskURI)) c.vmLockMap.UnlockEntry(strings.ToLower(string(nodeName))) retriable := false if err != nil && retry.IsErrorRetriable(err) { retriable = true } return !retriable, err }) if retryErr != nil { err = retryErr klog.V(2).Infof("azureDisk - update abort backoff: detach disk(%s, %s), err: %v", diskName, diskURI, err) </ins> } <del> return !retriable, err }) if retryErr != nil { err = retryErr klog.V(2).Infof("azureDisk - update abort backoff: detach disk(%s, %s), err: %v", diskName, diskURI, err) </del> } } if err != nil {