Code change:
A/D controller tries to detach volumes that are still mounted after 6 minutes. The conditions to trigger the detach are: I.e. all containers are finished. But that does not guarantee that volumes are unmounted. When a CSI driver can't unmount a volume for 6 minutes, the volume is force detached by A/D controller, corrupting the volume and leaving volume mounts on the node in a bad state. Note that the node is perfectly healthy, it just can't unmount a volume from whatever reason. A/D controller detaches volumes after a Pod is deleted from the API server, i.e. the volumes are unmounted or someone force-deleted the pods. Very artificial reproducer: a Pod with an attachable CSI volume. the node, make the volume mount busy, e.g.: Every unmount of the volume fails and kubelet/the CSI driver periodically retry. the pod 6 minutes, see the volume is detached without unmounting first. KCM logs: Note that the Pod is still in the API server: And the node is healthy: Note that some CSI drivers (e.g. the cloud ones) do not allow detaching a volume while it's still mounted. Not all CSI driver have such possibility and they rely on Kubernetes not to call ControllerUnpublish before NodeUnpublish / NodeUnstage succeeds. <detailsToday's master </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-longterm
Right now, a volume is detached from a node when: (all pods that use the volume are deleted from the API server OR all their container are stopped) AND (the volume is unmounted OR 6 minutes passed). Potential solution 1: Issue detach of a mounted volume only when all pods are force-deleted from the API server. More formally: Detach when [(All pods that use the volume are deleted from the API server OR all their container are stopped) AND the volume is unmounted] OR [all pods that use the volume are deleted from the API server AND 6 minutes passed]. Is this change subtle enough that it does not break existing behavior (e.g. tooling on top of Kubernetes)?
Potential solution 2: Add a new flag to CSIDriver, say . We would need to extend internal Detach() call to include a flag if the volume to detach is mounted or not (A/D controller knows that) and the CSI volume plugin would check CSIDriver and potentially reject detach of a mounted volume. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 3: As a variant, we could add a cmdline option to the external-attacher, like . When that cmdline option is specified, the external attacher would check , if the volume to detach is still mounted, and refuse such detach (with a reasonable error message). CSI driver vendors could opt-in into this behavior. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 4: As a variant, we could add to CSIDriver, where each driver could put their own timeout before force-detaching a volume, with an option to use infinity.
Note to self: check , it might help on the CSI side.
Solution 3 will not work well with ungraceful node shutdown - attach/detach controller does not watch pods and force-deleting pods is the main point of ungraceful node shutdown KEP. /me will try if solution 1 would work when NodeStage times out (is Pod deleted from the API server before or after NodeUnstage is called?)
was right, a pod is deleted (without force) from the API server after NodeUnpublish succeeds, it does not wait for NodeUnstage to finish. Therefore we cannot distinguish if a pod was force-deleted (and we should start 6 minute timer) or pod was deleted without any force and only NodeUnstage is taking more time than expected. Solution 1 is then not possible.
Now we are down to solution 2 and 4. For solution 2, if driver does not have a way to check whether volume is mounted or not, the behavior will not change and we still force detach after 6 mins. And it might happen in the middle of unstage ..
Summing up yesterday's call: A/D controller should check if a node is unhealthy before force detaching volumes from it. It should never force detach a volume from a healthy node, not even after 6 minutes. Question what healthy means is open and we should talk to sig-node. is a working assumption.
I agree with both of these. I'd just emphasize that that particular definition of unhealthy definitely includes perfectly healthy nodes in a network partition. This change is no worse than the status quo ante for those nodes, and a definite improvement on the healthy nodes, so I wholeheartedly support it.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
/assign

Related discussion:
text: return false, nil } <ins> // nodeIsHealthy returns true if the node looks healthy. func (rc *reconciler) nodeIsHealthy(nodeName types.NodeName) (bool, error) { node, err := rc.nodeLister.Get(string(nodeName)) if err != nil { return false, err } return nodeutil.IsNodeReady(node), nil } </ins> func (rc *reconciler) reconcile() { // Detaches are triggered before attaches so that volumes referenced by // pods that are rescheduled to a different node are detached first.