Change:
/kind bug What happened: I'm running a POD with three VolumeMount (AzureDisk PVC). If I delete the pod, it gets rescheduled on a different node. The 3 disks are unmounted from the original node (thanks to kubelet) and the disks are starting to get unattached from the VM. Out of the three disks, only one was able to get unattached, it was properly reattached to the new node waiting to run the freshly restarted pod. The other two Disks couldn't be detached. The controller manager keeps trying to attach from this point on while never trying to re-detach. What you expected to happen: Kube-controller-manager should retry detaching disks after failures. How to reproduce it (as minimally and precisely as possible): create a pod with 3 volume mounts (AzureDisk). Kill the running pod. watch the status If it works, retry Anything else we need to know?: kube-controller-manager logs (--v=4) Environment: Kubernetes version (use ): 1.7.12 Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): centOS 7.4 Kernel (e.g. ): 3.10.0-693.11.6.el7.x86_64 Install tools: in-house (terraform + salt) cc:
/sig azure /sig storage
/assign let me know if you need more logs.
Three disk ID. one was detached / reattached properly the other twos had errors since they were detached at the same time (3 concurrent detach, 1 pass, 2 fails) After that the remaining two start looping with
This is one problem. the disk is still attached to the VM.
Azure Portal shows that a bunch of errors occurred, followed by three success. Each of those should have been the removal of a single disk. In the end, only 1 disk disappeared. <img width="508" alt="screen shot 2018-02-21 at 01 21 23" src="https://user-"<img width="591" alt="screen shot 2018-02-21 at 01 21 32" src="https://user-"My gut feeling is that the backoff logic doesn't refresh the list of disks to apply on the VM. So we have three functions running all trying to delete a single disk. The deletion is performed using a VMUpdate call with a list of Disks (minus the one we want to remove) The new VM is construction based on the list of disks (-1) All three deletion logics start at the same time, causing race conditions with the azure api. or Eventually, the backoff logic will get through and the last VM update will re-add the two disks that were not part of the original deletion request. One solution could be to lock on disk deletion if they touch the same NodeName? Or queue the deletion request when an existing request is ongoing for the same node.
you are right, there is no locking for detach disk, I will try to repro it and then use the locking fix to verify it, thanks for reporting.
/assign
let me know if you can’t reproduce. It’s easy for me. I could cook you up a quick manifest. Thanks for looking into it.
Your repro shows obvious evidence of race condition when detaching azure disk, I have commited a PR, pls take a look. Thanks for your reporting, it's really helpful.

Discussion:
return fmt.Errorf("invalid disk to detach: %q", diskURI) } <del> _, err := d.cloud.InstanceID(context.TODO(), nodeName) </del> <ins> instanceid, err := d.cloud.InstanceID(context.TODO(), nodeName) </ins> if err != nil { glog.Warningf("no instance id for node %q, skip detaching (%v)", nodeName, err) return nil

if err != nil { return err } <ins> getLunMutex.LockKey(instanceid) defer getLunMutex.UnlockKey(instanceid) </ins> err = diskController.DetachDiskByName("", diskURI, nodeName) if err != nil { glog.Errorf("failed to detach azure disk %q, err %v", diskURI, err)