Change:
After KCM restart, any Pod that reuses a volume that failed to detach gets stuck ContainerCreating with message: User "system:node:ip-10-0-243-" cannot get resource "volumeattachments" in API group "" at the cluster scope: no relationship found between node 'ip-10-0-243-' and this object See steps to reproduce below, it's quite a complex condition. It happens when a pod is restarted, a CSI driver "busy" (i.e. times out) + KCM restarted at a wrong time. Pods start just fine. a Pod that uses an attachable CSI volume. the Pod. Make sure it unmounts successfully, but any fails with a transient error, such as timeout. The external attacher continues to detach the volume at this point. kube-controller-manager (KCM) and start a new one. Wait for any leader election etc - it's important for the new KCM to populate A/D controller with no pod using the detaching volume. Let the A/D controller to observe detach error. a new Pod on the same node that uses the same volume. the CSI driver to detach the volume (i.e. stop injecting any transient errors). At this point: The external attacher succeeds detaching the volume, VolumeAttachment is deleted. The volume is marked as attached in node status (!!!) despite no VolumeAttachment exists. A/D controller does not even try to create a new VolumeAttachment, the volume is marked as attached in its Actual State of World (ASW). Kubelet tries to mount the volume, but WaitForAttach fails with: , basically forever. I created . The script expects that the cluster is created by , because it restarts KCM in the middle. It uses my hacked hostpath CSI driver to inject detach errors. Sources are at / the last commit(s) there. There are two issues that together lead to this situation: fails, , i.e. puts it back to . With the volume in , kubelet may try to mount it, while the external attacher tries to detach it. It is not as bad as it sounds, because kubelet can't read VolumeAttachment with the deletion timestamp and it will not call / of un-attached volume. KCM starts, all VolumeAttachments that don't have a Pod (that's good), but also (that's probably bad). When the newly started KCM re-tries to detach the volume and gets an error again, it marks the volume as fully attached (same as the previous point) and adds it to . Now the volume is both mounted + attached in ASW and A/D controller actually stops here and waits forever for kubelet to unmount the volume. <detailsToday's k/k master, </details<detailsNone </details<detailsN/A, it's KCM issue. </details<details, but seen in a downstream Kubernetes distro too. </details<details</details<details</details
/sig storage
/assign
/triage accepted
I have the same problem: Seems restart of Kubelet solves this problem. Kubernetes v1.25.13

Discussion:
// The first detach will be triggered after at least 50ms (maxWaitForUnmountDuration in test). // Right before detach operation is performed, the volume will be first removed from being reported // as attached on node status (RemoveVolumeFromReportAsAttached). After detach operation which is expected to fail, <del> // controller then added the volume back as attached. </del> <ins> // controller then treats the attachment as Uncertain. </ins> // Here it sleeps 100ms so that detach should be triggered already at this point. <del> // verifyVolumeReportedAsAttachedToNode will check volume is in the list of volume attached that needs to be updated // in node status. By calling this function (GetVolumesToReportAttached), node status should be updated, and the volume // will not need to be updated until new changes are applied (detach is triggered again) </del> time.Sleep(100 * time.Millisecond) <del> verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </del> <ins> verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateUncertain, asw) verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, false, asw, volumeAttachedCheckTimeout) </ins> // Add a second pod which tries to attach the volume to the same node. <del> // After adding pod to the same node, detach will not be triggered any more. </del> <ins> // After adding pod to the same node, detach will not be triggered any more, // the volume gets attached and reported as attached to the node. </ins> generatedVolumeName, podAddErr = dsw.AddPod(types.UniquePodName(podName2), controllervolumetesting.NewPod(podName2, podName2), volumeSpec, nodeName1) if podAddErr != nil { t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podAddErr)

// Sleep 1s to verify no detach are triggered after second pod is added in the future. time.Sleep(1000 * time.Millisecond) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // verifyVolumeNoStatusUpdateNeeded(t, logger, generatedVolumeName, nodeName1, asw) // Add a third pod which tries to attach the volume to a different node.

migrated := getMigratedStatusBySpec(volumeToDetach.VolumeSpec) if err != nil { <del> // On failure, add volume back to ReportAsAttached list actualStateOfWorld.AddVolumeToReportAsAttached( logger, volumeToDetach.VolumeName, volumeToDetach.NodeName) </del> <ins> // On failure, mark the volume as uncertain. Attach() must succeed before adding the volume back // to node status as attached. uncertainError := actualStateOfWorld.MarkVolumeAsUncertain( logger, volumeToDetach.VolumeName, volumeToDetach.VolumeSpec, volumeToDetach.NodeName) if uncertainError != nil { klog.Errorf("DetachVolume.MarkVolumeAsUncertain failed to add the volume %q to actual state after detach error: %s", volumeToDetach.VolumeName, uncertainError) } </ins> eventErr, detailedErr := volumeToDetach.GenerateError("DetachVolume.Detach failed", err) return volumetypes.NewOperationContext(eventErr, detailedErr, migrated) }