Change:
During testing a AWS cluster, found AWS volume once was pending for detach about 30 minutes. According to the log, the following happened, controller issues a detach operation after detach response returns, it periodically checks the status until it reaches "detached". However, in this test, the status is "busy" all the time. after reaching the timeout 30 mins, detach function returns error because the status is not "detached" The following is the log related to the failed test I0915 18:29:46. 5 ] Verified volume is safe to detach for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-". E0915 18:59:54. 5 ] Error detaching volumeID "vol-": Timeout waiting for volume state: actual=busy, desired=detached E0915 18:59:54. 5 ] Operation for """" failed. No retries permitted until 2016-09-15 18:59:55. +0000 UTC (durationBeforeRetry 500ms). Error: DetachVolume.Detach failed for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-" with: Timeout waiting for volume state: actual=busy, desired=detached
Thoughts on what the ideal timeout should be. Too short would be bad because it would give a false-negative.
I know this is from a test but I'm seeing the same problem on a production cluster (1.4.8). I'm not sure exactly what triggers it yet. Kubelet only unmounts the pod directory, not the mount in "aws-ebs/mounts". As a result, the EBS gets stuck in "busy" state when detaching since it is still mounted to the VM. Manually unmounting the EBS allows it to get unstuck and detach safely. In controller manager I see: It then proceeds to get stuck detaching (volume goes to busy state), after which it starts trying to force detach which of course hangs indefinitely. Inspecting the node directly, I see it's still mounted which is why detach hangs: So something in kubelet is messed up and not unmounting both mount points correctly.
I think this might be the same issue as
On kubelet, it looks like the volume failed to mark as mounted, which is why it subsequently got stuck. This may have to do with the fact this pod () got rolled back while waiting for the previous deployment to succeed.
^ Looks like it might be a race condition between the controller manager and reconciler: manager attaches volume starts attachment process (pod rollback/delete) controller manager thinks its safe to detach, so removes volume from node's volume list, then tries to detach mounts the volume as part of the ongoing attach/mount process from (2) tries to update the mount list, this fails because something (reconciler?) has removed the volume from attached list gets stuck in (busy) state detaching, as its mounted to the kubelet node. it never gets unmounted again in kubelet because state doesn't reflect it is mounted.
Looks like there exists race conditions between external updates of and 's asynchronous kick off of operations like MountVolume, which is the case I'm seeing. Seems like a lot of these edge cases could crop up here - maybe reconciler should lock changes to the desiredStateOfWorld while it's running?
thank you for providing detailed information and analysis. I think there is a race condition between asynchronous operation MountVolume and updating actualStateOfWorld directly by reconciler I will work on a fix. Please let me know if you have any issue/comments.

Discussion:
// Ensure devices that should be detached/unmounted are detached/unmounted. for _, attachedVolume := range rc.actualStateOfWorld.GetUnmountedVolumes() { <del> if !rc.desiredStateOfWorld.VolumeExists(attachedVolume.VolumeName) { </del> <ins> // Check IsOperationPending to avoid marking a volume as detached if it's in the process of mounting. if !rc.desiredStateOfWorld.VolumeExists(attachedVolume.VolumeName) && !rc.operationExecutor.IsOperationPending(attachedVolume.VolumeName, nestedpendingoperations.EmptyUniquePodName) { </ins> if attachedVolume.GloballyMounted { // Volume is globally mounted to device, unmount it glog.V(12).Infof("Attempting to start UnmountDevice for volume %q (spec.Name: %q)",

} } else { // Volume is attached to node, detach it <ins> // Kubelet not responsible for detaching or this volume has a non-attachable volume plugin. </ins> if rc.controllerAttachDetachEnabled || !attachedVolume.PluginIsAttachable { <del> // Kubelet not responsible for detaching or this volume has a non-attachable volume plugin, // so just remove it to actualStateOfWorld without attach. rc.actualStateOfWorld.MarkVolumeAsDetached( attachedVolume.VolumeName, rc.nodeName) </del> <ins> rc.actualStateOfWorld.MarkVolumeAsDetached(attachedVolume.VolumeName, attachedVolume.NodeName) glog.Infof("Detached volume %q (spec.Name: %q) devicePath: %q", attachedVolume.VolumeName, attachedVolume.VolumeSpec.Name(), attachedVolume.DevicePath) </ins> } else { // Only detach if kubelet detach is enabled glog.V(12).Infof("Attempting to start DetachVolume for volume %q (spec.Name: %q)",

) const ( <del> // emptyUniquePodName is a UniquePodName for empty string. emptyUniquePodName types.UniquePodName = types.UniquePodName("") </del> <ins> // EmptyUniquePodName is a UniquePodName for empty string. EmptyUniquePodName types.UniquePodName = types.UniquePodName("") </ins> <del> // emptyUniqueVolumeName is a UniqueVolumeName for empty string emptyUniqueVolumeName v1.UniqueVolumeName = v1.UniqueVolumeName("") </del> <ins> // EmptyUniqueVolumeName is a UniqueVolumeName for empty string EmptyUniqueVolumeName v1.UniqueVolumeName = v1.UniqueVolumeName("") </ins> ) // NestedPendingOperations defines the supported set of operations.

podName types.UniquePodName) (bool, int) { // If volumeName is empty, operation can be executed concurrently <del> if volumeName == emptyUniqueVolumeName { </del> <ins> if volumeName == EmptyUniqueVolumeName { </ins> return false, -1 }

continue } <del> if previousOp.podName != emptyUniquePodName && podName != emptyUniquePodName && </del> <ins> if previousOp.podName != EmptyUniquePodName && podName != EmptyUniquePodName && </ins> previousOp.podName != podName { // No match, keep searching continue

func getOperationName( volumeName v1.UniqueVolumeName, podName types.UniquePodName) string { podNameStr := "" <del> if podName != emptyUniquePodName { </del> <ins> if podName != EmptyUniquePodName { </ins> podNameStr = fmt.Sprintf(" (%q)", podName) }

return err } <del> podName := volumetypes.UniquePodName("") </del> <ins> podName := nestedpendingoperations.EmptyUniquePodName </ins> // TODO: remove this -- not necessary if !volumeToMount.PluginIsAttachable { // Non-attachable volume plugins can execute mount for multiple pods