Code change:
A/D controller tries to detach volumes that are still mounted after 6 minutes. The conditions to trigger the detach are: I.e. all containers are finished. But that does not guarantee that volumes are unmounted. When a CSI driver can't unmount a volume for 6 minutes, the volume is force detached by A/D controller, corrupting the volume and leaving volume mounts on the node in a bad state. Note that the node is perfectly healthy, it just can't unmount a volume from whatever reason. A/D controller detaches volumes after a Pod is deleted from the API server, i.e. the volumes are unmounted or someone force-deleted the pods. Very artificial reproducer: a Pod with an attachable CSI volume. the node, make the volume mount busy, e.g.: Every unmount of the volume fails and kubelet/the CSI driver periodically retry. the pod 6 minutes, see the volume is detached without unmounting first. KCM logs: Note that the Pod is still in the API server: And the node is healthy: Note that some CSI drivers (e.g. the cloud ones) do not allow detaching a volume while it's still mounted. Not all CSI driver have such possibility and they rely on Kubernetes not to call ControllerUnpublish before NodeUnpublish / NodeUnstage succeeds. <detailsToday's master </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-longterm
Right now, a volume is detached from a node when: (all pods that use the volume are deleted from the API server OR all their container are stopped) AND (the volume is unmounted OR 6 minutes passed). Potential solution 1: Issue detach of a mounted volume only when all pods are force-deleted from the API server. More formally: Detach when [(All pods that use the volume are deleted from the API server OR all their container are stopped) AND the volume is unmounted] OR [all pods that use the volume are deleted from the API server AND 6 minutes passed]. Is this change subtle enough that it does not break existing behavior (e.g. tooling on top of Kubernetes)?
Potential solution 2: Add a new flag to CSIDriver, say . We would need to extend internal Detach() call to include a flag if the volume to detach is mounted or not (A/D controller knows that) and the CSI volume plugin would check CSIDriver and potentially reject detach of a mounted volume. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 3: As a variant, we could add a cmdline option to the external-attacher, like . When that cmdline option is specified, the external attacher would check , if the volume to detach is still mounted, and refuse such detach (with a reasonable error message). CSI driver vendors could opt-in into this behavior. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 4: As a variant, we could add to CSIDriver, where each driver could put their own timeout before force-detaching a volume, with an option to use infinity.
Note to self: check , it might help on the CSI side.
Solution 3 will not work well with ungraceful node shutdown - attach/detach controller does not watch pods and force-deleting pods is the main point of ungraceful node shutdown KEP. /me will try if solution 1 would work when NodeStage times out (is Pod deleted from the API server before or after NodeUnstage is called?)
was right, a pod is deleted (without force) from the API server after NodeUnpublish succeeds, it does not wait for NodeUnstage to finish. Therefore we cannot distinguish if a pod was force-deleted (and we should start 6 minute timer) or pod was deleted without any force and only NodeUnstage is taking more time than expected. Solution 1 is then not possible.
Now we are down to solution 2 and 4. For solution 2, if driver does not have a way to check whether volume is mounted or not, the behavior will not change and we still force detach after 6 mins. And it might happen in the middle of unstage ..
Summing up yesterday's call: A/D controller should check if a node is unhealthy before force detaching volumes from it. It should never force detach a volume from a healthy node, not even after 6 minutes. Question what healthy means is open and we should talk to sig-node. is a working assumption.
I agree with both of these. I'd just emphasize that that particular definition of unhealthy definitely includes perfectly healthy nodes in a network partition. This change is no worse than the status quo ante for those nodes, and a definite improvement on the healthy nodes, so I wholeheartedly support it.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
/assign

Related discussion:
text: // Check whether timeout has reached the maximum waiting time timeout := elapsedTime > rc.maxWaitForUnmountDuration <ins> isHealthy, err := rc.nodeIsHealthy(attachedVolume.NodeName) if err != nil { klog.Errorf("failed to get health of node %s: %s", attachedVolume.NodeName, err.Error()) } // Force detach volumes from unhealthy nodes after maxWaitForUnmountDuration. forceDetach := !isHealthy && timeout </ins> hasOutOfServiceTaint, err := rc.hasOutOfServiceTaint(attachedVolume.NodeName) if err != nil { klog.Errorf("failed to get taint specs for node %s: %s", attachedVolume.NodeName, err.Error()) } <del> // Check whether volume is still mounted. Skip detach if it is still mounted unless timeout </del> <ins> // Check whether volume is still mounted. Skip detach if it is still mounted unless force detach timeout </ins> // or the node has `node.kubernetes.io/out-of-service` taint. <del> if attachedVolume.MountedByNode && !timeout && !hasOutOfServiceTaint { </del> <ins> if attachedVolume.MountedByNode && !forceDetach && !hasOutOfServiceTaint { </ins> klog.V(5).InfoS("Cannot detach volume because it is still mounted", "volume", attachedVolume) continue }