Code change:
} } <ins> func TestUpdatePodForTerminatedRuntimePod(t *testing.T) { podWorkers, processed := createPodWorkers() now := time.Now() podWorkers.podSyncStatuses[types.UID("1")] = &podSyncStatus{ startedTerminating: true, terminatedAt:       now.Add(-time.Second), terminatingAt:      now.Add(-2 * time.Second), gracePeriod:        1, } // creates synthetic pod podWorkers.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodKill, RunningPod: &kubecontainer.Pod{ID: "1", Name: "1", Namespace: "test"}, }) drainAllWorkers(podWorkers) if len(processed) != 0 { t.Fatalf("Not all pods processed: %v", processed) } updates := processed["1"] if len(updates) != 0 { t.Fatalf("unexpected updates: %v", updates) } if len(podWorkers.lastUndeliveredWorkUpdate) != 0 { t.Fatalf("Unexpected undelivered work") } } </ins> func TestUpdatePodDoesNotForgetSyncPodKill(t *testing.T) { podWorkers, processed := createPodWorkers() numPods := 20

Related discussion:
text: So the bug is: worker for the pod is still running (terminating has completed, but is executing syncTerminatedPod) runs, calls , gets a stale result for a pod (thinks the pod is still running, but that's just a stale cache result) asks pod worker to terminate the pod with a runningPod worker sees that the pod is actually not terminating, but terminated, and so RunningPod is inconsistently handled (UpdatePod is not defensive enough) worker passes the kill pod request through to syncTerminatedPod which has a null pod status, causing the panic In earlier versions of we explicitly refreshed the runtime cache in , but when and I went through review we removed it and I forgot this race was possible. The race becomes more possible here because the particular workload Elana is testing involves CSI drivers that have a bad cleanup loop, so syncTerminatedPod is reaching timeout on volume unmount (3m) before finally succeeding. This means the race window is huge (invoke UpdatePod(runningPod) on an terminated but not yet cleaned up pod). I think the real bug is that UpdatePod is not defensive enough - the difference between this race condition (seeing an older runtime container) and a human somehow starting the container up again before it is terminated is pretty similar. The important difference is that the UpdatePod{... running pod } is intended to clean up orphaned pods - those the pod worker is not tracking. But we only go into terminated when we are certain the workload is stopped, so if we simply ignore until the pod goes away it might be a while before we kill that container. We absolutely need a background worker tracking that, which is best to be a pod worker, but it's a bit more work. So easy fix (remove the possibility of the very common race) is to simply have syncTerminatingPod refresh the pod cache for that one pod. A slightly less easy fix is to have UpdatePod simply ignore the runningPod request until the next housekeeping loop (which in some cases with bad csi drivers could be a few minutes or so and in the worst case might block). The most correct fix is to ensure syncTerminated pod has an opportunity to maintain its invariants (no running containers) or have that be maintained alongside. Pondering variations here of fixes.