Change:
Recently, the cpu load of my kube-apiserver gets higher after I expanded more pods and nodes into the cluster. And I found that some volumes declared by pods could not be mounted after the restart of pods, even though the pods were successful scheduled. here's the kubectl describe information: After checking the logs of attachdetachcontroller, I found that in , return an error, thus the was skipped. That should be related to the high load on my kube-apiserver as it occasionally returns 500 http code (but this may not be the root cause). Here is the error message: Here comes the problem: In my opinion, the next loop of would retry and , since the state of actualStateOfWorld is different from the state of desiredStateOfWorld. but it didn't happened. By recording some additional logs, I notice that the pod was scheduled to the destination node by the scheduler right after above error message appeared, which cause executed. That would write the volume info back to . After that, the next loop of would skip the detach operation for that volume, since desiredStateOfWorld was 'equal to' actualStateOfWorld. The log snippet is as follows: Finally, attachdetachcontroller would not attach the volume neither, because the states of desiredStateOfWorld and actualStateOfWorld are the same. attachdetachcontroller mistakenly thought the volume was attached, but it' not the case. when fail, attachdetachcontroller should retry to detach the volume until the the detach operation completed. However, it was interrupted by the update of desiredStateOfWorld. a log and before a pod mounting pv Then you'll found log message "force skip detaching" would disappear after a few loops of No response <details</details<detailsN/A </detailsNo response <details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
If fails, that means the volume has not been detached and the volumeattachment is still on the nodeStatus, then kubelet should see that the volume still attaches to the node, so I don't understand why your pod failed to mount, it may not be caused by the controller. what's the error for kubelet?
! There are . The one before was failed, but another which is at the end of was successfully executed. Thus it cause the kubelet coundn't get the volume in .
kubelet log repeat the message below infinitely:
Okay, I know what you said, actually this problem is similar with the one which was fixed by , but it is not completely. Give another pr below to fix this.
Can we have the fix () back-ported to 1.21? Given that this completes the , the initial fix for which was , it would make sense to have this follow-up fix also be to 1.21. Looks like there is a that we could get this into. We have a 100+ clusters deployed in production that are running 1.21. We see this bug at least once a week and it's a headache to deal with. Upgrading to 1.22 or higher is a big undertaking for us.

Discussion:
if err != nil { // Skip detaching this volume if unable to update node status klog.ErrorS(err, "UpdateNodeStatusForNode failed while attempting to report volume as attached", "volume", attachedVolume) <ins> // Add volume back to ReportAsAttached if UpdateNodeStatusForNode call failed so that node status updater will add it back to VolumeAttached list. // It is needed here too because DetachVolume is not call actually and we keep the data consistency for every reconcile. rc.actualStateOfWorld.AddVolumeToReportAsAttached(attachedVolume.VolumeName, attachedVolume.NodeName) </ins> continue }

} <ins> func Test_Run_UpdateNodeStatusFailBeforeOneVolumeDetachNodeWithReadWriteOnce(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) dsw := cache.NewDesiredStateOfWorld(volumePluginMgr) asw := cache.NewActualStateOfWorld(volumePluginMgr) fakeKubeClient := controllervolumetesting.CreateTestClient() fakeRecorder := &record.FakeRecorder{} fakeHandler := volumetesting.NewBlockVolumePathHandler() ad := operationexecutor.NewOperationExecutor(operationexecutor.NewOperationGenerator( fakeKubeClient, volumePluginMgr, fakeRecorder, fakeHandler)) informerFactory := informers.NewSharedInformerFactory(fakeKubeClient, controller.NoResyncPeriodFunc()) nodeLister := informerFactory.Core().V1().Nodes().Lister() nsu := statusupdater.NewFakeNodeStatusUpdater(false /* returnError */) rc := NewReconciler( reconcilerLoopPeriod, maxWaitForUnmountDuration, syncLoopPeriod, false, dsw, asw, ad, nsu, nodeLister, fakeRecorder) reconciliationLoopFunc := rc.(*reconciler).reconciliationLoopFunc() podName1 := "pod-uid1" volumeName := v1.UniqueVolumeName("volume-name") volumeSpec := controllervolumetesting.GetTestVolumeSpec(string(volumeName), volumeName) volumeSpec.PersistentVolume.Spec.AccessModes = []v1.PersistentVolumeAccessMode{v1.ReadWriteOnce} nodeName1 := k8stypes.NodeName("node-name1") dsw.AddNode(nodeName1, false /*keepTerminatedPodVolumes*/) // Add the pod in which the volume is attached to the FailDetachNode generatedVolumeName, podAddErr := dsw.AddPod(types.UniquePodName(podName1), controllervolumetesting.NewPod(podName1, podName1), volumeSpec, nodeName1) if podAddErr != nil { t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podAddErr) } // Act reconciliationLoopFunc() // Volume is added to asw, volume should be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeReportedAsAttachedToNode(t, generatedVolumeName, nodeName1, true, asw) // Delete the pod dsw.DeletePod(types.UniquePodName(podName1), generatedVolumeName, nodeName1) // Mock NodeStatusUpdate fail rc.(*reconciler).nodeStatusUpdater = statusupdater.NewFakeNodeStatusUpdater(true /* returnError */) reconciliationLoopFunc() // The first detach will be triggered after at leaset 50ms (maxWaitForUnmountDuration in test). time.Sleep(100 * time.Millisecond) reconciliationLoopFunc() // Right before detach operation is performed, the volume will be first removed from being reported // as attached on node status (RemoveVolumeFromReportAsAttached). After UpdateNodeStatus operation which is expected to fail, // controller then added the volume back as attached. // verifyVolumeReportedAsAttachedToNode will check volume is in the list of volume attached that needs to be updated // in node status. By calling this function (GetVolumesToReportAttached), node status should be updated, and the volume // will not need to be updated until new changes are applied (detach is triggered again) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeReportedAsAttachedToNode(t, generatedVolumeName, nodeName1, true, asw) } </ins> func Test_Run_OneVolumeDetachFailNodeWithReadWriteOnce(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t)