Code change:
A/D controller tries to detach volumes that are still mounted after 6 minutes. The conditions to trigger the detach are: I.e. all containers are finished. But that does not guarantee that volumes are unmounted. When a CSI driver can't unmount a volume for 6 minutes, the volume is force detached by A/D controller, corrupting the volume and leaving volume mounts on the node in a bad state. Note that the node is perfectly healthy, it just can't unmount a volume from whatever reason. A/D controller detaches volumes after a Pod is deleted from the API server, i.e. the volumes are unmounted or someone force-deleted the pods. Very artificial reproducer: a Pod with an attachable CSI volume. the node, make the volume mount busy, e.g.: Every unmount of the volume fails and kubelet/the CSI driver periodically retry. the pod 6 minutes, see the volume is detached without unmounting first. KCM logs: Note that the Pod is still in the API server: And the node is healthy: Note that some CSI drivers (e.g. the cloud ones) do not allow detaching a volume while it's still mounted. Not all CSI driver have such possibility and they rely on Kubernetes not to call ControllerUnpublish before NodeUnpublish / NodeUnstage succeeds. <detailsToday's master </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-longterm
Right now, a volume is detached from a node when: (all pods that use the volume are deleted from the API server OR all their container are stopped) AND (the volume is unmounted OR 6 minutes passed). Potential solution 1: Issue detach of a mounted volume only when all pods are force-deleted from the API server. More formally: Detach when [(All pods that use the volume are deleted from the API server OR all their container are stopped) AND the volume is unmounted] OR [all pods that use the volume are deleted from the API server AND 6 minutes passed]. Is this change subtle enough that it does not break existing behavior (e.g. tooling on top of Kubernetes)?
Potential solution 2: Add a new flag to CSIDriver, say . We would need to extend internal Detach() call to include a flag if the volume to detach is mounted or not (A/D controller knows that) and the CSI volume plugin would check CSIDriver and potentially reject detach of a mounted volume. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 3: As a variant, we could add a cmdline option to the external-attacher, like . When that cmdline option is specified, the external attacher would check , if the volume to detach is still mounted, and refuse such detach (with a reasonable error message). CSI driver vendors could opt-in into this behavior. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 4: As a variant, we could add to CSIDriver, where each driver could put their own timeout before force-detaching a volume, with an option to use infinity.
Note to self: check , it might help on the CSI side.
Solution 3 will not work well with ungraceful node shutdown - attach/detach controller does not watch pods and force-deleting pods is the main point of ungraceful node shutdown KEP. /me will try if solution 1 would work when NodeStage times out (is Pod deleted from the API server before or after NodeUnstage is called?)
was right, a pod is deleted (without force) from the API server after NodeUnpublish succeeds, it does not wait for NodeUnstage to finish. Therefore we cannot distinguish if a pod was force-deleted (and we should start 6 minute timer) or pod was deleted without any force and only NodeUnstage is taking more time than expected. Solution 1 is then not possible.
Now we are down to solution 2 and 4. For solution 2, if driver does not have a way to check whether volume is mounted or not, the behavior will not change and we still force detach after 6 mins. And it might happen in the middle of unstage ..
Summing up yesterday's call: A/D controller should check if a node is unhealthy before force detaching volumes from it. It should never force detach a volume from a healthy node, not even after 6 minutes. Question what healthy means is open and we should talk to sig-node. is a working assumption.
I agree with both of these. I'd just emphasize that that particular definition of unhealthy definitely includes perfectly healthy nodes in a network partition. This change is no worse than the status quo ante for those nodes, and a definite improvement on the healthy nodes, so I wholeheartedly support it.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
/assign

Related discussion:
text: waitForDetachCallCount(t, 0 /* expectedDetachCallCount */, fakePlugin) } <ins> // Populates desiredStateOfWorld cache with one node/volume/pod tuple. // The node starts as healthy. // // Calls Run() // Verifies there is one attach call and no detach calls. // Deletes the pod from desiredStateOfWorld cache without first marking the node/volume as unmounted. // Verifies that the volume is NOT detached after maxWaitForUnmountDuration. // Marks the node as unhealthy. // Verifies that the volume is detached after maxWaitForUnmountDuration. func Test_Run_OneVolumeDetachOnUnhealthyNode(t *testing.T) { // Arrange volumePluginMgr, fakePlugin := volumetesting.GetTestVolumePluginMgr(t) dsw := cache.NewDesiredStateOfWorld(volumePluginMgr) asw := cache.NewActualStateOfWorld(volumePluginMgr) fakeKubeClient := controllervolumetesting.CreateTestClient() fakeRecorder := &record.FakeRecorder{} fakeHandler := volumetesting.NewBlockVolumePathHandler() ad := operationexecutor.NewOperationExecutor(operationexecutor.NewOperationGenerator( fakeKubeClient, volumePluginMgr, fakeRecorder, fakeHandler)) informerFactory := informers.NewSharedInformerFactory(fakeKubeClient, controller.NoResyncPeriodFunc()) nsu := statusupdater.NewFakeNodeStatusUpdater(false /* returnError */) nodeLister := informerFactory.Core().V1().Nodes().Lister() reconciler := NewReconciler( reconcilerLoopPeriod, maxWaitForUnmountDuration, syncLoopPeriod, false, dsw, asw, ad, nsu, nodeLister, fakeRecorder) podName1 := "pod-uid1" volumeName1 := v1.UniqueVolumeName("volume-name1") volumeSpec1 := controllervolumetesting.GetTestVolumeSpec(string(volumeName1), volumeName1) nodeName1 := k8stypes.NodeName("worker-0") node1 := &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: string(nodeName1)}, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:   v1.NodeReady, Status: v1.ConditionTrue, }, }, }, } informerFactory.Core().V1().Nodes().Informer().GetStore().Add(node1) dsw.AddNode(nodeName1, false /*keepTerminatedPodVolumes*/) volumeExists := dsw.VolumeExists(volumeName1, nodeName1) if volumeExists { t.Fatalf( "Volume %q/node %q should not exist, but it does.", volumeName1, nodeName1) } generatedVolumeName, podErr := dsw.AddPod(types.UniquePodName(podName1), controllervolumetesting.NewPod(podName1, podName1), volumeSpec1, nodeName1) if podErr != nil { t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podErr) } // Act ch := make(chan struct{}) go reconciler.Run(ch) defer close(ch) // Assert waitForNewAttacherCallCount(t, 1 /* expectedCallCount */, fakePlugin) verifyNewAttacherCallCount(t, false /* expectZeroNewAttacherCallCount */, fakePlugin) waitForAttachCallCount(t, 1 /* expectedAttachCallCount */, fakePlugin) verifyNewDetacherCallCount(t, true /* expectZeroNewDetacherCallCount */, fakePlugin) waitForDetachCallCount(t, 0 /* expectedDetachCallCount */, fakePlugin) // Act // Delete the pod and the volume will be detached even after the maxWaitForUnmountDuration expires as volume is // not unmounted and the node is healthy. dsw.DeletePod(types.UniquePodName(podName1), generatedVolumeName, nodeName1) time.Sleep(maxWaitForUnmountDuration * 5) // Assert waitForNewDetacherCallCount(t, 0 /* expectedCallCount */, fakePlugin) verifyNewAttacherCallCount(t, false /* expectZeroNewAttacherCallCount */, fakePlugin) waitForAttachCallCount(t, 1 /* expectedAttachCallCount */, fakePlugin) verifyNewDetacherCallCount(t, true /* expectZeroNewDetacherCallCount */, fakePlugin) waitForDetachCallCount(t, 0 /* expectedDetachCallCount */, fakePlugin) // Act // Mark the node unhealthy node2 := node1.DeepCopy() node2.Status.Conditions[0].Status = v1.ConditionFalse informerFactory.Core().V1().Nodes().Informer().GetStore().Update(node2) // Assert -- Detach was triggered after maxWaitForUnmountDuration waitForNewDetacherCallCount(t, 1 /* expectedCallCount */, fakePlugin) verifyNewAttacherCallCount(t, false /* expectZeroNewAttacherCallCount */, fakePlugin) waitForAttachCallCount(t, 1 /* expectedAttachCallCount */, fakePlugin) verifyNewDetacherCallCount(t, false /* expectZeroNewDetacherCallCount */, fakePlugin) waitForDetachCallCount(t, 1 /* expectedDetachCallCount */, fakePlugin) } </ins> func Test_ReportMultiAttachError(t *testing.T) { type nodeWithPods struct { name     k8stypes.NodeName