Change:
} for _, pod := range dswp.podManager.GetPods() { <del> if dswp.podStateProvider.ShouldPodContainersBeTerminating(pod.UID) { </del> <ins> // Keep consistency of adding pod during reconstruction if dswp.hasAddedPods && dswp.podStateProvider.ShouldPodContainersBeTerminating(pod.UID) { </ins> // Do not (re)add volumes for pods that can't also be starting containers continue } <ins> if !dswp.hasAddedPods && dswp.podStateProvider.ShouldPodRuntimeBeRemoved(pod.UID) { // When kubelet restarts, we need to add pods to dsw if there is a possibility // that the container may still be running continue } </ins> dswp.processPodVolumes(pod, mountedVolumesForPod) } }

Discussion:
We observed that while a container is in the process of being torn down, but not completed, the volume gets unmounted and then subsequently detached, causing the fs to enter read-only mode. This happened after kubelet restarted. We observed that on the restart, the volume was not to DSW because the Pod was marked for deletion. In the normal case, we only remove the pod from DSW when the containers have finished teardown: However, in the reconstruction case, it looks like if a pod has started termination, but not necessarily completed, then we don't add the pod to DSW: We probably need to be consistent and change the add pod criteria to also add even if it's started terminating. The side effect is that in the normal case we may do unnecessary mounts if a pod is quickly created and deleted. We can also consider a flag to only use the stricter check during reconstruction. What happened in our env was that kubelet was crash looping, and then all the pods on the node were deleted. It looks like there was a good 30 seconds between volume unmount and container teardown. I think this may be easily reproducible if the pod has a long graceful termination period and the container ignores the stop signal. No response <details22 </details<details</details<details</details<details</details<details</details<details</details
/triage accepted
Yeah this looks problematic. What was the exact kube version btw? So it was possible that container is still writing to the volume? Do we know why fs enters read-only mode when this happens? I know the behaviour of kubelet is problematic regardless, but still. I think adding volumes belonging to terminating pod doesn't sound so bad. I wonder if we could change this to .
Does this mean instead of checking ShouldPodContainersBeTerminating to decide if we add the pod or not, we should check IsPodTerminationRequested or not to decide? Or we should use the same check ShouldPodRuntimeBeRemoved with reconstruction(this one is more consistent)? pointed out that there were already scaling challenges with volume manager processing pods it doesn't need to.
For reconstruction, we need to add the volume in DSW if there is a chance that containers are running. So is the best call to determine that.
We observed a similar issue but could only reproduce it with emptydir volumes. Do you think is the same issue?

/assign sunnylovestiramisu
thanks for the report. I think this issue could impact emptydir volumes too.
Is there any recommended way to make a pod having a long graceful termination period? Tried with some simple pods but it just terminated right away.
You can use the sample pod and instructions from here
cc
Here is my repro pod and steps: the pod shell, another window (window 2), ssh to the node another window (window 3), window 2, in window 1, exec back into the pod . The file will be gone. try . It will return When looking at kubelet logs, you'll see the log messages that indicate the volume was reconstructed without being to dsw. and UnmountVolume happen soon after: Pod lifecycle logs:
Tested locally via kind cluster. /k master is showing error of touch: cannot touch '/scratch/test': No such file or directory. cluster with the fix in this pr shows no error. And running shows no result in the log.