Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When testing against OpenShift CI, I discovered a panic in the kubelet: Further investigation revealed that it is possible for to panic under certain circumstances: We hit this case when (podStatus remains nil) and (nil will be passed along to ). I'm not sure why exactly that would happen but it seems possible. Don't panic! Unclear, I've been trying to trace the code/find evidence of this happening in k8s CI but haven't found a match yet. ( would be nice :smile:) /sig node /cc /priority important-soon /milestone v1.22 Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/triage accepted
/assign
The bug is that running pods should terminate and then stop, not move to terminated. So something is getting into terminated but should never be in any other work type than terminating. We'll need more debug to determine why but I suspect that an orphaned pod is still in podSyncStatus at the time it UpdatePod is called, which means it's not seen as being in Terminating. isRuntimePod forces podSyncStatus to be a certain state, but perhaps it's should resett a terminated bool. Asked elana to add a v4 min level in the test jobs that reproduced this so we can see.
has bumped, here's a v=4 log with a panic: https://gcsweb-
Elana got the version with more logs and it is indeed the case that the pod is terminating (in this case because a CSI volume hasn't been torn down). So the bug is: worker for the pod is still running (terminating has completed, but is executing syncTerminatedPod) runs, calls , gets a stale result for a pod (thinks the pod is still running, but that's just a stale cache result) asks pod worker to terminate the pod with a runningPod worker sees that the pod is actually not terminating, but terminated, and so RunningPod is inconsistently handled (UpdatePod is not defensive enough) worker passes the kill pod request through to syncTerminatedPod which has a null pod status, causing the panic In earlier versions of we explicitly refreshed the runtime cache in , but when and I went through review we removed it and I forgot this race was possible. The race becomes more possible here because the particular workload Elana is testing involves CSI drivers that have a bad cleanup loop, so syncTerminatedPod is reaching timeout on volume unmount (3m) before finally succeeding. This means the race window is huge (invoke UpdatePod(runningPod) on an terminated but not yet cleaned up pod). I think the real bug is that UpdatePod is not defensive enough - the difference between this race condition (seeing an older runtime container) and a human somehow starting the container up again before it is terminated is pretty similar. The important difference is that the UpdatePod{... running pod } is intended to clean up orphaned pods - those the pod worker is not tracking. But we only go into terminated when we are certain the workload is stopped, so if we simply ignore until the pod goes away it might be a while before we kill that container. We absolutely need a background worker tracking that, which is best to be a pod worker, but it's a bit more work. So easy fix (remove the possibility of the very common race) is to simply have syncTerminatingPod refresh the pod cache for that one pod. A slightly less easy fix is to have UpdatePod simply ignore the runningPod request until the next housekeeping loop (which in some cases with bad csi drivers could be a few minutes or so and in the worst case might block). The most correct fix is to ensure syncTerminated pod has an opportunity to maintain its invariants (no running containers) or have that be maintained alongside. Pondering variations here of fixes. The easy fixes will defiintely be "safe", but "slight regression in latency in a few rare cases". Serializing the pod worker caused this, but it's in theory rare.
My understanding is that we make pod worker the source of truth for pod state. In this kind of race condition, as long as the request is sent to the pod worker, and the pod UID matches, it should be able to reconcile the state. Even if we do this, it is not atomic, the race condition still exists in theory. We probably should make the pod worker be able to reconcile the state. I don't quite understand what you mean by "until the next housekeeping loop"? Since the pod is being terminated in , what problem would it cause if we just ignore the runningPod request? If the termination is still in progress, we need to wait for it to finish anyway.
Here's the most concerning scenario: is terminated and then deleted from apiserver kills the container is taking a very long time after 2, housekeeping fires and sees the stale cache, triggers a kill (proposed change) ignore the kill request (since terminated pod is running) restarts container in the pod that references a volume mount can't make progress because volume detach can't make progress because the restarted container is holding a device lock / has a file open / w/e pod worker for that pod retries forever is the minimal fix that ignores (5) but I'm thinking through how to handle 8. I don't think we should do the easy fix (refreshing the cache isn't enough because you are vulnerable to 6-8). I'm potentially considering having the pod worker invoke syncTerminatingPod in the event syncTerminatedPod exits with an error, or simply have syncTerminatedPod do it if it would return an error to address 8.
But this basically means that after , there could still be running containers. In that case, this wouldn't be the only problem, the whole assumption the refactor builds on becomes invalid? BTW, restarting container may happen to docker, but containerd doesn't support it (at least it can't be done easily), not sure about cri-o.
cri-o doesn't support restarting a container, either.
The assumption in the kubelet is there are no more containers started by the kubelet, which is indeed safe. That doesn’t prevent bugs in CRI implementations, user error, cosmic rays, basically computers. It’s unlikely and not something we really designed for before, but the prerefactor code would survive it. Appears to have gotten past the issue as expected. I’ll look at what it would take to mitigate the possibility of zombie containers such that the pod worker converges even if volume mount can’t be completed.
In general a little defensiveness here to unreliable external components has only minor cost, and some mitigation seems reasonable.
Opened the first part of the fix in
SG. Yeah, that should be considered a rare case. However, adding some defensive cleanup also SGTM as long as it doesn't introduce too much overhead or complexity. :)

Discussion:
var wasGracePeriodShortened bool switch { case status.IsTerminated(): <ins> // A terminated pod may still be waiting for cleanup - if we receive a runtime pod kill request // due to housekeeping seeing an older cached version of the runtime pod simply ignore it until // after the pod worker completes. if isRuntimePod { klog.V(3).InfoS("Pod is waiting for termination, ignoring runtime-only kill until after pod worker is fully terminated", "pod", klog.KObj(pod), "podUID", pod.UID) return } </ins> workType = TerminatedPodWork if options.KillPodOptions != nil {

} } <ins> func TestUpdatePodForTerminatedRuntimePod(t *testing.T) { podWorkers, processed := createPodWorkers() now := time.Now() podWorkers.podSyncStatuses[types.UID("1")] = &podSyncStatus{ startedTerminating: true, terminatedAt:       now.Add(-time.Second), terminatingAt:      now.Add(-2 * time.Second), gracePeriod:        1, } // creates synthetic pod podWorkers.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodKill, RunningPod: &kubecontainer.Pod{ID: "1", Name: "1", Namespace: "test"}, }) drainAllWorkers(podWorkers) if len(processed) != 0 { t.Fatalf("Not all pods processed: %v", processed) } updates := processed["1"] if len(updates) != 0 { t.Fatalf("unexpected updates: %v", updates) } if len(podWorkers.lastUndeliveredWorkUpdate) != 0 { t.Fatalf("Unexpected undelivered work") } } </ins> func TestUpdatePodDoesNotForgetSyncPodKill(t *testing.T) { podWorkers, processed := createPodWorkers() numPods := 20