Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Restart: Always pods went into OutOfCpu error state after kubelet restart. Pods should remain running as node had sufficient capacity for them. Completed, RestartNever pod should not revert from Completed to OutOfCpu. Launch kubelet with: Run the following script against the local cluster: <details</detailsObserve pod status: Restart the kubelet: Observe pods: xref /sig node Kubernetes version (use ): v1.23.0-alpha.3.183+ - Cloud provider or hardware configuration: Local machine OS (e.g: ) & Kernel (e.g. ):Install tools: local-up- Network plugin and version (if this is a network-related bug): N/A Others:
/triage accepted /priority critical-urgent Reproducible on 1.22
From the reproducer: yields However, in the kubelet logs, it sees 7100m cpu allocated?
Looks like the topology admit handler thinks that the completed pods are using CPU which is why when it gets to the 8th it fails it with OutOfCpu. Must be a bug in there somewhere.
/assign
Unclear if could be related but linking for posterity
Confirmed this only affects 1.22, repro on 1.21 has expected behaviour: /sig scheduling I'll start a bisect.
is
Added some logging With this we see I think that the terminated pods are getting filtered, but for whatever reason they are still getting their resource requests to NodeInfo?
points out that my logging language was a bit backwards and that the completed pods should not have been returned, because they already successfully completed and exited. This code is suspect: The logs show that the completed pods are never filtered, so they are failing one of these checks. I am going to try to write a test case that hits this issue so we can test fixes against it.
The completed pods definitely show up as complete, but for some reason they do not get filtered (note I have changed my patch above for the log lines to make sense now): So I guess isPodTerminationRequested=true on already exited pods, but I would only expect that to be true if the pod was actively in the process of termination and not already exited. cc Log patch: <details</details
I tried changing to but apparently it thinks that's true :( Edit: doesn't work either.
The bug is that the only component in the kubelet that is allowed to authoritatively state a container is terminated (pod worker) takes non-zero time to do so during startup (because we have to call kill container to guarantee it's stopped). This races with the pod config loop and admission, which sends all the containers at once. In order for the race to be eliminated, some equivalent of a sync has to be completed during startup on terminal but potentially non-terminated pod workers BEFORE other adds are processed. That could be running HandlePodCleanups once all sources are synced but before HandlePod* methods are called, but HandlePodAdditions() cannot be invoked until after the pod worker has had a chance to verify all terminal pods are actually terminal. EDIT: Another potential approach is for the pod worker to hold starting pods until SyncKnownPods observes all terminal pods are accounted for at least once, then allowing starts.
Also, we appear to be missing a kubelet restart test that verifies admission with completed pods (which I thought was on our list of "missing e2e tests that would have caught these regressions" already). Such a test would have to: pods that consume resources for admission until they are running some of them with failed and` successful states the kubelet some of the running pods the kubelet none of the terminal pods are restarted, all running pods are restarted, and that only non-terminal pods are accounted to admission

Discussion:
"k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/tools/record" <ins> runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1alpha2" </ins> "k8s.io/klog/v2" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/events"

return ok } <ins> func isPodStatusCacheTerminal(status *kubecontainer.PodStatus) bool { runningContainers := 0 runningSandboxes := 0 for _, container := range status.ContainerStatuses { if container.State == kubecontainer.ContainerStateRunning { runningContainers++ } } for _, sb := range status.SandboxStatuses { if sb.State == runtimeapi.PodSandboxState_SANDBOX_READY { runningSandboxes++ } } return runningContainers == 0 && runningSandboxes == 0 } </ins> // UpdatePod carries a configuration change or termination state to a pod. A pod is either runnable, // terminating, or terminated, and will transition to terminating if deleted on the apiserver, it is // discovered to have a terminal phase (Succeeded or Failed), or if it is evicted by the kubelet.

status = &podSyncStatus{ syncedAt: now, } <ins> // if this pod is being synced for the first time, we need to make sure it is an active pod if !isRuntimePod && (pod.Status.Phase == v1.PodFailed || pod.Status.Phase == v1.PodSucceeded) { // check to see if the pod is not running and the pod is terminal. // If this succeeds then record in the podWorker that it is terminated. if statusCache, err := p.podCache.Get(pod.UID); err == nil { if isPodStatusCacheTerminal(statusCache) { status = &podSyncStatus{ terminatedAt:       now, terminatingAt:      now, syncedAt:           now, startedTerminating: true, finished:           true, } } } } </ins> p.podSyncStatuses[uid] = status }

} } <ins> func newPodWithPhase(uid, name string, phase v1.PodPhase) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID:  types.UID(uid), Name: name, }, Status: v1.PodStatus{ Phase: phase, }, } } </ins> // syncPodRecord is a record of a sync pod call type syncPodRecord struct { name       string

} } <ins> func TestUpdatePodWithTerminatedPod(t *testing.T) { podWorkers, _ := createPodWorkers() terminatedPod := newPodWithPhase("0000-0000-0000", "done-pod", v1.PodSucceeded) runningPod := &kubecontainer.Pod{ID: "0000-0000-0001", Name: "done-pod"} pod := newPod("0000-0000-0002", "running-pod") podWorkers.UpdatePod(UpdatePodOptions{ Pod:        terminatedPod, UpdateType: kubetypes.SyncPodCreate, }) podWorkers.UpdatePod(UpdatePodOptions{ Pod:        pod, UpdateType: kubetypes.SyncPodCreate, }) podWorkers.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodKill, RunningPod: runningPod, }) if podWorkers.IsPodKnownTerminated(pod.UID) == true { t.Errorf("podWorker state should not be terminated") } if podWorkers.IsPodKnownTerminated(terminatedPod.UID) == false { t.Errorf("podWorker state should be terminated") } if podWorkers.IsPodKnownTerminated(runningPod.ID) == true { t.Errorf("podWorker state should not be marked terminated for a running pod") } } </ins> func TestUpdatePodForRuntimePod(t *testing.T) { podWorkers, processed := createPodWorkers()

"time" "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/api/resource" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/test/e2e/framework" <ins> e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" </ins> testutils "k8s.io/kubernetes/test/utils" imageutils "k8s.io/kubernetes/test/utils/image"

runningPods = []*v1.Pod{} for _, pod := range podList.Items { <del> if r, err := testutils.PodRunningReady(&pod); err != nil || !r { </del> <ins> if r, err := testutils.PodRunningReadyOrSucceeded(&pod); err != nil || !r { </ins> continue } runningPods = append(runningPods, &pod)

return runningPods } <del> var _ = SIGDescribe("Restart [Serial] [Slow] [Disruptive] [NodeFeature:ContainerRuntimeRestart]", func() { </del> <ins> var _ = SIGDescribe("Restart [Serial] [Slow] [Disruptive]", func() { </ins> const ( // Saturate the node. It's not necessary that all these pods enter // Running/Ready, because we don't know the number of cores in the

}) }) }) <ins> ginkgo.Context("Kubelet", func() { ginkgo.It("should correctly account for terminated pods after restart", func() { node := getLocalNode(f) cpus := node.Status.Allocatable[v1.ResourceCPU] numCpus := int((&cpus).Value()) if numCpus < 1 { e2eskipper.Skipf("insufficient CPU available for kubelet restart test") } // create as many restartNever pods as there are allocatable CPU // nodes; if they are not correctly accounted for as terminated // later, this will fill up all node capacity podCountRestartNever := numCpus ginkgo.By(fmt.Sprintf("creating %d RestartNever pods on node", podCountRestartNever)) restartNeverPods := newTestPods(podCountRestartNever, false, imageutils.GetE2EImage(imageutils.BusyBox), "restart-kubelet-test") for _, pod := range restartNeverPods { pod.Spec.RestartPolicy = "Never" pod.Spec.Containers[0].Command = []string{"echo", "hi"} pod.Spec.Containers[0].Resources.Limits = v1.ResourceList{ v1.ResourceCPU: resource.MustParse("1"), } } createBatchPodWithRateControl(f, restartNeverPods, podCreationInterval) defer deletePodsSync(f, restartNeverPods) completedPods := waitForPods(f, podCountRestartNever, time.Minute) if len(completedPods) < podCountRestartNever { framework.Failf("Failed to run sufficient restartNever pods, got %d but expected %d", len(completedPods), podCountRestartNever) } podCountRestartAlways := (numCpus / 2) + 1 ginkgo.By(fmt.Sprintf("creating %d RestartAlways pods on node", podCountRestartAlways)) restartAlwaysPods := newTestPods(podCountRestartAlways, false, imageutils.GetPauseImageName(), "restart-kubelet-test") for _, pod := range restartAlwaysPods { pod.Spec.Containers[0].Resources.Limits = v1.ResourceList{ v1.ResourceCPU: resource.MustParse("1"), } } createBatchPodWithRateControl(f, restartAlwaysPods, podCreationInterval) defer deletePodsSync(f, restartAlwaysPods) numAllPods := podCountRestartNever + podCountRestartAlways allPods := waitForPods(f, numAllPods, startTimeout) if len(allPods) < numAllPods { framework.Failf("Failed to run sufficient restartAlways pods, got %d but expected %d", len(allPods), numAllPods) } ginkgo.By("killing and restarting kubelet") // We want to kill the kubelet rather than a graceful restart startKubelet := stopKubelet() startKubelet() // If this test works correctly, each of these pods will exit // with no issue. But if accounting breaks, pods scheduled after // restart may think these old pods are consuming CPU and we // will get an OutOfCpu error. ginkgo.By("verifying restartNever pods succeed and restartAlways pods stay running") for start := time.Now(); time.Since(start) < startTimeout; time.Sleep(10 * time.Second) { postRestartRunningPods := waitForPods(f, numAllPods, time.Minute) if len(postRestartRunningPods) < numAllPods { framework.Failf("less pods are running after node restart, got %d but expected %d", len(postRestartRunningPods), numAllPods) } } }) }) </ins> })