Code change:
// The first detach will be triggered after at least 50ms (maxWaitForUnmountDuration in test). // Right before detach operation is performed, the volume will be first removed from being reported // as attached on node status (RemoveVolumeFromReportAsAttached). After detach operation which is expected to fail, <del> // controller then added the volume back as attached. </del> <ins> // controller then treats the attachment as Uncertain. </ins> // Here it sleeps 100ms so that detach should be triggered already at this point. <del> // verifyVolumeReportedAsAttachedToNode will check volume is in the list of volume attached that needs to be updated // in node status. By calling this function (GetVolumesToReportAttached), node status should be updated, and the volume // will not need to be updated until new changes are applied (detach is triggered again) </del> time.Sleep(100 * time.Millisecond) <del> verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </del> <ins> verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateUncertain, asw) verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, false, asw, volumeAttachedCheckTimeout) </ins> // Add a second pod which tries to attach the volume to the same node. <del> // After adding pod to the same node, detach will not be triggered any more. </del> <ins> // After adding pod to the same node, detach will not be triggered any more, // the volume gets attached and reported as attached to the node. </ins> generatedVolumeName, podAddErr = dsw.AddPod(types.UniquePodName(podName2), controllervolumetesting.NewPod(podName2, podName2), volumeSpec, nodeName1) if podAddErr != nil { t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podAddErr)

Related discussion:
text: After KCM restart, any Pod that reuses a volume that failed to detach gets stuck ContainerCreating with message: User "system:node:ip-10-0-243-" cannot get resource "volumeattachments" in API group "" at the cluster scope: no relationship found between node 'ip-10-0-243-' and this object See steps to reproduce below, it's quite a complex condition. It happens when a pod is restarted, a CSI driver "busy" (i.e. times out) + KCM restarted at a wrong time. Pods start just fine. a Pod that uses an attachable CSI volume. the Pod. Make sure it unmounts successfully, but any fails with a transient error, such as timeout. The external attacher continues to detach the volume at this point. kube-controller-manager (KCM) and start a new one. Wait for any leader election etc - it's important for the new KCM to populate A/D controller with no pod using the detaching volume. Let the A/D controller to observe detach error. a new Pod on the same node that uses the same volume. the CSI driver to detach the volume (i.e. stop injecting any transient errors). At this point: The external attacher succeeds detaching the volume, VolumeAttachment is deleted. The volume is marked as attached in node status (!!!) despite no VolumeAttachment exists. A/D controller does not even try to create a new VolumeAttachment, the volume is marked as attached in its Actual State of World (ASW). Kubelet tries to mount the volume, but WaitForAttach fails with: , basically forever. I created . The script expects that the cluster is created by , because it restarts KCM in the middle. It uses my hacked hostpath CSI driver to inject detach errors. Sources are at / the last commit(s) there. There are two issues that together lead to this situation: fails, , i.e. puts it back to . With the volume in , kubelet may try to mount it, while the external attacher tries to detach it. It is not as bad as it sounds, because kubelet can't read VolumeAttachment with the deletion timestamp and it will not call / of un-attached volume.