Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When a pod is evicted under resource pressure, it goes into an error state. The pod terminates normally. Note that the 130GB file is based on 90% of total disk space - current utilization to hit the default eviction threshold. Full pod spec after going into error state: <details</detailsSee also Eviction tests () are currently broken (), there's a PR up to fix: Kubernetes version (use ): HEAD/1.23.x (kubeletVersion="v1.23.0-alpha.3.47+") Cloud provider or hardware configuration: local-up-cluster OS (e.g: ), Kernel (e.g. ):Install tools: local-up- Network plugin and version (if this is a network-related bug): N/A Others: Also reproducible on OpenShift 4.9 (1.23-based) /sig node /cc
mentions this looks very similar to
Confirmed that this is a regression from 1.21. When running the reproducer against the release-1.21 branch, the pod above correctly stays in Evicted status (since pods are mortal). On 1.22, it tries to restart, entering either an Error or ContainerCreating state. /kind regression
/assign
/triage accepted
From : is the first bad commit This was
Seems likely that wasn't actually fixed. The broken eviction tests made it much harder to verify.
Attaching logs from HEAD from reproducer: v=4 Let me also try to get a repro where it goes into Error
Logs from HEAD for reproducer when pod goes into Error: v=4
Roughly, kubelet syncTerminatingPod sends "Failed" to status manager (which looks like it generates the correct apiserver patch for phase running -failed), then somehow by the time Kubelet syncTerminatedPod is called it sends "Running but with errors" to status manager, and status manager allows that transition (internally) and then sends it to the apiserver. So the pod goes Running -Failed -Running which should never happen (we "lost" that the pod was stopped). Bug 1: pod patch status probably shouldn't allow terminal -nonterminal phase transitions (didn't we fix that??) Bug 2: pod worker is supposed to be getting most up to date status from status manager before calling syncTerminatedPod, which should include "Failed", but that isn't happening? thus the status manager then sends a "non-terminal state" (shouldn't ever happen) Roughly, figure out why status passed to syncTerminatedPod is older than what was sent by syncTerminatingPod, then figure out why status manager allowed that transition back to running, then figure out why apiserver didn't reject.
Staring at logs, I've figured out where the illegal phase transition is happening: If the container failed to properly terminate and generate a status saying as much, generateAPIPodStatus won't have any hints available to it that the pod died because it was killed. (This is happening in the reproducer case because the container won't exit before the termination grace period, and needs to be force killed.) Thus, it will assume that since the container used to be running, its current status is running. This results in an illegal phase transition.
From what I can tell, the issue is that is being passed a pod that still has running containers. From kubelet logs: which gets set as part of . It looks like that will always run so long as succeeds. Thus, I feel like syncTerminatingPod must have the bug: it returns successfully, but the containers of the pod aren't all actually terminated.
/priority critical-urgent based on priority of and the impact of this regression (Kubelet-initiated evictions frequently result in error state pods)
The containers ARE terminated, we just haven't retrieved their status. This is because of a decision to not refresh the container status cache in syncTerminatingPods. So syncTerminatingPod is not a bug.
So then perhaps the right thing to do here would be to refresh the container status cache before syncing another status to the API server in ? Then we can avoid syncing the outdated cached status
Updating from an internal discussion last night - bug 2 isn't actually a bug. Lantau convinced me we wouldn't refresh the status cache, so what happens is we call generatedAPIStatus with: API status (mostly irrelevant) status manager status to date container cache (but 2s) after The key thing is that phase from runtime status is irrelevant when 1 or 2 are terminal - because runtime has no idea about whether a pod is terminal. So roughly we should be ignoring phase from 3 except when determining when to go from pending to running. The bug here is that we're thinking that this is an "illegal transition" but still copying the latest reason and message (runtime status doesn't get to weigh in on overal terminal phases). We shouldn't need to refresh status at all.
Okay, so there is definitely a bug in what's being synced to the API server for evicted pod statuses right now, possibly for other terminated pods. In 1.21 and earlier, when a pod is evicted, it looks like this: <details</detailsNote that the status field is very minimal, and there are no container statuses. In 1.22+, we attempt to sync container statuses on termination, and sometimes they are wrong (e.g. changing phase from Failed -Running). Here's an example of what an evicted pod could look like on 1.22: <details</details
This appears to be broken even for runOnce pods, tested against latest HEAD (). With spec: After scheduling the pod... And the final pod status: <details</details
Elana identified the issue on status here - previously we NEVER reported container status after an eviction, so the refactor (which correctly ensured container exit codes showed up) was returning containers with state=waiting,lastState=terminated, which is incorrect. generateAPIStatus needs to know the difference between "current container status, waiting for more to show up" and "any status we have is the best we have". So once killPods succeeds in syncTerminatingPods and in syncTerminatedPods, we're no longer "waiting". So generating pod status should correctly summarize the container status as and not inject .

Discussion:
allStatus := append(append([]v1.ContainerStatus{}, s.ContainerStatuses...), s.InitContainerStatuses...) s.Phase = getPhase(&pod.Spec, allStatus) klog.V(4).InfoS("Got phase for pod", "pod", klog.KObj(pod), "oldPhase", oldPodStatus.Phase, "phase", s.Phase) <ins> // Perform a three-way merge between the statuses from the status manager, // runtime, and generated status to ensure terminal status is correctly set. if s.Phase != v1.PodFailed && s.Phase != v1.PodSucceeded { switch { case oldPodStatus.Phase == v1.PodFailed || oldPodStatus.Phase == v1.PodSucceeded: klog.V(4).InfoS("Status manager phase was terminal, updating phase to match", "pod", klog.KObj(pod), "phase", oldPodStatus.Phase) s.Phase = oldPodStatus.Phase case pod.Status.Phase == v1.PodFailed || pod.Status.Phase == v1.PodSucceeded: klog.V(4).InfoS("API phase was terminal, updating phase to match", "pod", klog.KObj(pod), "phase", pod.Status.Phase) s.Phase = pod.Status.Phase } } </ins> if s.Phase == oldPodStatus.Phase { // preserve the reason and message which is associated with the phase s.Reason = oldPodStatus.Reason

}, }, { <ins> name: "terminal phase from previous status must remain terminal, restartAlways", pod: &v1.Pod{ Spec: desiredState, Status: v1.PodStatus{ Phase: v1.PodRunning, ContainerStatuses: []v1.ContainerStatus{ runningState("containerA"), runningState("containerB"), }, }, }, currentStatus: &kubecontainer.PodStatus{}, previousStatus: v1.PodStatus{ Phase: v1.PodSucceeded, ContainerStatuses: []v1.ContainerStatus{ runningState("containerA"), runningState("containerB"), }, // Reason and message should be preserved Reason:  "Test", Message: "test", }, expected: v1.PodStatus{ Phase:    v1.PodSucceeded, HostIP:   "127.0.0.1", QOSClass: v1.PodQOSBestEffort, Conditions: []v1.PodCondition{ {Type: v1.PodInitialized, Status: v1.ConditionTrue, Reason: "PodCompleted"}, {Type: v1.PodReady, Status: v1.ConditionFalse, Reason: "PodCompleted"}, {Type: v1.ContainersReady, Status: v1.ConditionFalse, Reason: "PodCompleted"}, {Type: v1.PodScheduled, Status: v1.ConditionTrue}, }, ContainerStatuses: []v1.ContainerStatus{ ready(waitingWithLastTerminationUnknown("containerA", 1)), ready(waitingWithLastTerminationUnknown("containerB", 1)), }, Reason:  "Test", Message: "test", }, }, { name: "terminal phase from previous status must remain terminal, restartNever", pod: &v1.Pod{ Spec: v1.PodSpec{ NodeName: "machine", Containers: []v1.Container{ {Name: "containerA"}, {Name: "containerB"}, }, RestartPolicy: v1.RestartPolicyNever, }, Status: v1.PodStatus{ Phase: v1.PodRunning, ContainerStatuses: []v1.ContainerStatus{ runningState("containerA"), runningState("containerB"), }, }, }, currentStatus: &kubecontainer.PodStatus{}, previousStatus: v1.PodStatus{ Phase: v1.PodSucceeded, ContainerStatuses: []v1.ContainerStatus{ succeededState("containerA"), succeededState("containerB"), }, // Reason and message should be preserved Reason:  "Test", Message: "test", }, expected: v1.PodStatus{ Phase:    v1.PodSucceeded, HostIP:   "127.0.0.1", QOSClass: v1.PodQOSBestEffort, Conditions: []v1.PodCondition{ {Type: v1.PodInitialized, Status: v1.ConditionTrue, Reason: "PodCompleted"}, {Type: v1.PodReady, Status: v1.ConditionFalse, Reason: "PodCompleted"}, {Type: v1.ContainersReady, Status: v1.ConditionFalse, Reason: "PodCompleted"}, {Type: v1.PodScheduled, Status: v1.ConditionTrue}, }, ContainerStatuses: []v1.ContainerStatus{ ready(succeededState("containerA")), ready(succeededState("containerB")), }, Reason:  "Test", Message: "test", }, }, { </ins> name: "running can revert to pending", pod: &v1.Pod{ Spec: desiredState,

} status := *oldStatus.DeepCopy() for i := range status.ContainerStatuses { <del> if status.ContainerStatuses[i].State.Terminated != nil || status.ContainerStatuses[i].State.Waiting != nil { </del> <ins> if status.ContainerStatuses[i].State.Terminated != nil { </ins> continue } status.ContainerStatuses[i].State = v1.ContainerState{

} } for i := range status.InitContainerStatuses { <del> if status.InitContainerStatuses[i].State.Terminated != nil || status.InitContainerStatuses[i].State.Waiting != nil { </del> <ins> if status.InitContainerStatuses[i].State.Terminated != nil { </ins> continue } status.InitContainerStatuses[i].State = v1.ContainerState{

assert.Equal(t, newStatus.Message, firstStatus.Message) } <ins> func TestTerminatePodWaiting(t *testing.T) { syncer := newTestManager(&fake.Clientset{}) testPod := getTestPod() t.Logf("update the pod's status to Failed.  TerminatePod should preserve this status update.") firstStatus := getRandomPodStatus() firstStatus.Phase = v1.PodFailed firstStatus.InitContainerStatuses = []v1.ContainerStatus{ {Name: "init-test-1"}, {Name: "init-test-2", State: v1.ContainerState{Terminated: &v1.ContainerStateTerminated{Reason: "InitTest", ExitCode: 0}}}, {Name: "init-test-3", State: v1.ContainerState{Waiting: &v1.ContainerStateWaiting{Reason: "InitTest"}}}, } firstStatus.ContainerStatuses = []v1.ContainerStatus{ {Name: "test-1"}, {Name: "test-2", State: v1.ContainerState{Terminated: &v1.ContainerStateTerminated{Reason: "Test", ExitCode: 2}}}, {Name: "test-3", State: v1.ContainerState{Waiting: &v1.ContainerStateWaiting{Reason: "Test"}}}, } syncer.SetPodStatus(testPod, firstStatus) t.Logf("set the testPod to a pod with Phase running, to simulate a stale pod") testPod.Status = getRandomPodStatus() testPod.Status.Phase = v1.PodRunning syncer.TerminatePod(testPod) t.Logf("we expect the container statuses to have changed to terminated") newStatus := expectPodStatus(t, syncer, testPod) for i := range newStatus.ContainerStatuses { assert.False(t, newStatus.ContainerStatuses[i].State.Terminated == nil, "expected containers to be terminated") } for i := range newStatus.InitContainerStatuses { assert.False(t, newStatus.InitContainerStatuses[i].State.Terminated == nil, "expected init containers to be terminated") } expectUnknownState := v1.ContainerState{Terminated: &v1.ContainerStateTerminated{Reason: "ContainerStatusUnknown", Message: "The container could not be located when the pod was terminated", ExitCode: 137}} if !reflect.DeepEqual(newStatus.InitContainerStatuses[0].State, expectUnknownState) { t.Errorf("terminated container state not defaulted: %s", diff.ObjectReflectDiff(newStatus.InitContainerStatuses[0].State, expectUnknownState)) } if !reflect.DeepEqual(newStatus.InitContainerStatuses[1].State, firstStatus.InitContainerStatuses[1].State) { t.Errorf("existing terminated container state not preserved: %#v", newStatus.ContainerStatuses) } if !reflect.DeepEqual(newStatus.InitContainerStatuses[2].State, expectUnknownState) { t.Errorf("waiting container state not defaulted: %s", diff.ObjectReflectDiff(newStatus.InitContainerStatuses[2].State, expectUnknownState)) } if !reflect.DeepEqual(newStatus.ContainerStatuses[0].State, expectUnknownState) { t.Errorf("terminated container state not defaulted: %s", diff.ObjectReflectDiff(newStatus.ContainerStatuses[0].State, expectUnknownState)) } if !reflect.DeepEqual(newStatus.ContainerStatuses[1].State, firstStatus.ContainerStatuses[1].State) { t.Errorf("existing terminated container state not preserved: %#v", newStatus.ContainerStatuses) } if !reflect.DeepEqual(newStatus.ContainerStatuses[2].State, expectUnknownState) { t.Errorf("waiting container state not defaulted: %s", diff.ObjectReflectDiff(newStatus.ContainerStatuses[2].State, expectUnknownState)) } t.Logf("we expect the previous status update to be preserved.") assert.Equal(t, newStatus.Phase, firstStatus.Phase) assert.Equal(t, newStatus.Message, firstStatus.Message) } </ins> func TestSetContainerReadiness(t *testing.T) { cID1 := kubecontainer.ContainerID{Type: "test", ID: "1"} cID2 := kubecontainer.ContainerID{Type: "test", ID: "2"}

switch { case t.ExitCode == 1: // expected <ins> case t.ExitCode == 137 && (t.Reason == "ContainerStatusUnknown" || t.Reason == "Error"): // expected, pod was force-killed after grace period </ins> case t.ExitCode == 128 && (t.Reason == "StartError" || t.Reason == "ContainerCannotRun") && reBug88766.MatchString(t.Message): // pod volume teardown races with container start in CRI, which reports a failure framework.Logf("pod %s on node %s failed with the symptoms of https://github.com/kubernetes/kubernetes/issues/88766", pod.Name, pod.Spec.NodeName) default: data, _ := json.MarshalIndent(pod.Status, "", "  ") <del> framework.Logf("pod %s on node %s had incorrect final status:n%s", string(data)) </del> <ins> framework.Logf("pod %s on node %s had incorrect final status:n%s", pod.Name, pod.Spec.NodeName, string(data)) </ins> return fmt.Errorf("pod %s on node %s container unexpected exit code %d: start=%s end=%s reason=%s message=%s", pod.Name, pod.Spec.NodeName, t.ExitCode, t.StartedAt, t.FinishedAt, t.Reason, t.Message) } switch {

if !hasTerminalPhase { var names []string for _, status := range pod.Status.ContainerStatuses { <del> if status.State.Terminated != nil || status.State.Running != nil { </del> <ins> if status.State.Running != nil { </ins> names = append(names, status.Name) } }