Change:
We observed that while a container is in the process of being torn down, but not completed, the volume gets unmounted and then subsequently detached, causing the fs to enter read-only mode. This happened after kubelet restarted. We observed that on the restart, the volume was not to DSW because the Pod was marked for deletion. In the normal case, we only remove the pod from DSW when the containers have finished teardown: However, in the reconstruction case, it looks like if a pod has started termination, but not necessarily completed, then we don't add the pod to DSW: We probably need to be consistent and change the add pod criteria to also add even if it's started terminating. The side effect is that in the normal case we may do unnecessary mounts if a pod is quickly created and deleted. We can also consider a flag to only use the stricter check during reconstruction. What happened in our env was that kubelet was crash looping, and then all the pods on the node were deleted. It looks like there was a good 30 seconds between volume unmount and container teardown. I think this may be easily reproducible if the pod has a long graceful termination period and the container ignores the stop signal. No response <details22 </details<details</details<details</details<details</details<details</details<details</details
/triage accepted
Yeah this looks problematic. What was the exact kube version btw? So it was possible that container is still writing to the volume? Do we know why fs enters read-only mode when this happens? I know the behaviour of kubelet is problematic regardless, but still. I think adding volumes belonging to terminating pod doesn't sound so bad. I wonder if we could change this to .
Does this mean instead of checking ShouldPodContainersBeTerminating to decide if we add the pod or not, we should check IsPodTerminationRequested or not to decide? Or we should use the same check ShouldPodRuntimeBeRemoved with reconstruction(this one is more consistent)? pointed out that there were already scaling challenges with volume manager processing pods it doesn't need to.
For reconstruction, we need to add the volume in DSW if there is a chance that containers are running. So is the best call to determine that.
We observed a similar issue but could only reproduce it with emptydir volumes. Do you think is the same issue?
/assign sunnylovestiramisu
thanks for the report. I think this issue could impact emptydir volumes too.
Is there any recommended way to make a pod having a long graceful termination period? Tried with some simple pods but it just terminated right away.
You can use the sample pod and instructions from here
cc
Here is my repro pod and steps: the pod shell, another window (window 2), ssh to the node another window (window 3), window 2, in window 1, exec back into the pod . The file will be gone. try . It will return When looking at kubelet logs, you'll see the log messages that indicate the volume was reconstructed without being to dsw. and UnmountVolume happen soon after: Pod lifecycle logs:
Tested locally via kind cluster. /k master is showing error of touch: cannot touch '/scratch/test': No such file or directory. cluster with the fix in this pr shows no error. And running shows no result in the log.

Discussion:
} for _, pod := range dswp.podManager.GetPods() { <del> if dswp.podStateProvider.ShouldPodContainersBeTerminating(pod.UID) { </del> <ins> // Keep consistency of adding pod during reconstruction if dswp.hasAddedPods && dswp.podStateProvider.ShouldPodContainersBeTerminating(pod.UID) { </ins> // Do not (re)add volumes for pods that can't also be starting containers continue } <ins> if !dswp.hasAddedPods && dswp.podStateProvider.ShouldPodRuntimeBeRemoved(pod.UID) { // When kubelet restarts, we need to add pods to dsw if there is a possibility // that the container may still be running continue } </ins> dswp.processPodVolumes(pod, mountedVolumesForPod) } }

"k8s.io/kubernetes/pkg/volume/util/types" ) <ins> const ( Removed     string = "removed" Terminating string = "terminating" Other       string = "other" ) </ins> func pluginPVOmittingClient(dswp *desiredStateOfWorldPopulator) { fakeClient := &fake.Clientset{} fakeClient.AddReactor("get", "persistentvolumeclaims", func(action core.Action) (bool, runtime.Object, error) {

dswp.kubeClient = fakeClient } <del> func prepareDswpWithVolume(t *testing.T) (*desiredStateOfWorldPopulator, kubepod.Manager) { </del> <ins> func prepareDswpWithVolume(t *testing.T) (*desiredStateOfWorldPopulator, kubepod.Manager, *fakePodStateProvider) { </ins> // create dswp mode := v1.PersistentVolumeFilesystem pv := &v1.PersistentVolume{

Phase: v1.ClaimBound, }, } <del> dswp, fakePodManager, _, _, _ := createDswpWithVolume(t, pv, pvc) return dswp, fakePodManager </del> <ins> dswp, fakePodManager, _, _, fakePodStateProvider := createDswpWithVolume(t, pv, pvc) return dswp, fakePodManager, fakePodStateProvider </ins> } func TestFindAndAddNewPods_WithRescontructedVolume(t *testing.T) {

// (i.e. with SELinuxMountReadWriteOncePod disabled) defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SELinuxMountReadWriteOncePod, false)() // create dswp <del> dswp, fakePodManager := prepareDswpWithVolume(t) </del> <ins> dswp, fakePodManager, _ := prepareDswpWithVolume(t) </ins> // create pod fakeOuterVolumeName := "dswp-test-volume-name"

break } } <ins> if dswp.hasAddedPods { t.Fatalf("HasAddedPod should be false but it is true") } </ins> if !found { t.Fatalf( "Could not found pod volume %v in the list of actual state of world volumes to mount.",

} <ins> func TestFindAndAddNewPods_WithDifferentConditions(t *testing.T) { tests := []struct { desc          string hasAddedPods  bool podState      string expectedFound bool // Found pod is added to DSW }{ { desc:          "HasAddedPods is false, ShouldPodRuntimeBeRemoved and ShouldPodContainerBeTerminating are both true", hasAddedPods:  false, podState:      Removed, expectedFound: false, // Pod should not be added to DSW }, { desc:          "HasAddedPods is false, ShouldPodRuntimeBeRemoved is false, ShouldPodContainerBeTerminating is true", hasAddedPods:  false, podState:      Terminating, expectedFound: true, // Pod should be added to DSW }, { desc:          "HasAddedPods is false, other condition", hasAddedPods:  false, podState:      Other, expectedFound: true, // Pod should be added to DSW }, { desc:          "HasAddedPods is true, ShouldPodRuntimeBeRemoved is false, ShouldPodContainerBeTerminating is true", hasAddedPods:  true, podState:      Terminating, expectedFound: false, // Pod should not be added to DSW }, { desc:          "HasAddedPods is true, ShouldPodRuntimeBeRemoved and ShouldPodContainerBeTerminating are both true", hasAddedPods:  true, podState:      Removed, expectedFound: false, // Pod should not be added to DSW }, { desc:          "HasAddedPods is true, other condition", hasAddedPods:  true, podState:      Other, expectedFound: true, // Pod should be added to DSW }, } for _, tc := range tests { t.Run(tc.desc, func(t *testing.T) { // create dswp dswp, fakePodManager, fakePodState := prepareDswpWithVolume(t) // create pod containers := []v1.Container{ { VolumeMounts: []v1.VolumeMount{ { Name:      "dswp-test-volume-name", MountPath: "/mnt", }, }, }, } pod := createPodWithVolume("dswp-test-pod", "dswp-test-volume-name", "file-bound", containers) fakePodManager.AddPod(pod) switch tc.podState { case Removed: fakePodState.removed = map[kubetypes.UID]struct{}{pod.UID: {}} case Terminating: fakePodState.terminating = map[kubetypes.UID]struct{}{pod.UID: {}} case Other: break } dswp.hasAddedPods = tc.hasAddedPods // Action dswp.findAndAddNewPods() // Verify podsInDSW := dswp.desiredStateOfWorld.GetPods() found := false if podsInDSW[types.UniquePodName(pod.UID)] { found = true } if found != tc.expectedFound { t.Fatalf( "Pod with uid %v has expectedFound value %v in pods in DSW %v", pod.UID, tc.expectedFound, podsInDSW) } }) } } </ins> func TestFindAndAddNewPods_WithReprocessPodAndVolumeRetrievalError(t *testing.T) { // create dswp <del> dswp, fakePodManager := prepareDswpWithVolume(t) </del> <ins> dswp, fakePodManager, _ := prepareDswpWithVolume(t) </ins> // create pod containers := []v1.Container{

func TestFindAndAddNewPods_WithVolumeRetrievalError(t *testing.T) { // create dswp <del> dswp, fakePodManager := prepareDswpWithVolume(t) </del> <ins> dswp, fakePodManager, _ := prepareDswpWithVolume(t) </ins> pluginPVOmittingClient(dswp)

func (p *fakePodStateProvider) ShouldPodContainersBeTerminating(uid kubetypes.UID) bool { _, ok := p.terminating[uid] <ins> // if ShouldPodRuntimeBeRemoved returns true, ShouldPodContainerBeTerminating should also return true if !ok { _, ok = p.removed[uid] } </ins> return ok }