Change:
We observed the following series of events: 1) Pod is created with some attributes that later result in the pod being rejected at kubelet admission time 2) The pod is scheduled to a node 3) The pod is gracefully deleted by a controller, i.e. a deletion timestamp is set on the pod 4) The pod is rejected at kubelet admission time 5) The pod never gets deleted and is stuck: despite having the deletion timestamp set. It seems that kubelet does not issue the final force delete for the pod. This is problematic in many scenarios, but one specific case it was hit was when the pod is backed by a daemonset controller. This issue manifested in way where since the pod never got deleted in step 5 above, the daemonset controller did not create a new pod to run the on the node and thus node was was not running a replica of the daemonset controller pod. This seems to a be regression in k8s 1.27, I was not able to repro this behavior in 1.26. It is expected at during the series of events above, after the pod is failed at kubelet admission time, since the pod has a deletion timestamp set, the kubelet should forcefully delete the pod. The pod should not hang in deletion forever. Create a 1.27 kind cluster: Stop kubelet Create a pod bound to the node which will trigger a admission error. The below node selector label does not exist, so we expect that the pod should be rejected in admission. While kubelet is down issue a graceful deletion to set a deletion timestamp on the pod Verify deletion timestamp is set (pod is pending) Start kubelet Observe pod status, the pod is now in failed phase (as expected). However, the hangs, the pod is never force deleted by kubelet. No response 27 <detailsn/a </details<details</details<details</details<details</details<details</details
/sig node
cc
The kubelet logs have the following in the above repro: and there is no log about the pod being deleted by the status manager. Full kubelet logs - The log seems unexpected :) - the pod is not being deleted because the phase is "non-terminal", but the phase is failed which is a contradiction. It points that in the code something is wrong because
/assign I've checked that this is regression caused by this (the scenario works when this PR is reverted on 1.27.2). One relevant difference is that the function now returns false, because the . However, even if we make it return , still the deletion will not happen because . Wondering about the preferred solution. I guess ideally if we could trigger the . OTOH it wasn't triggered before and a pod rejected due to admission does not really need termination of containers, so maybe a shortcut will work. FYI
Opened PR:
Sigh... then it sounds like the daemonset controller is not handling correctly pods with a terminal phase. It looks like its waiting for them to disappear. Can you open another issue for that?
The pod that revealed the kubelet regression was indeed created by DaemonSet. However, this issue is reproducible without DS. Moreover, I cannot reproduce it with Kubelet fixed.
But this is the same problem as The DS controller should react to a Pod in a terminal phase and create a replacement Pod. I agree, if we fix kubelet, it's a non-issue, but the DS controller is not behaving correctly.
In this scenario the pod already has a , so DS ignores it waiting for it to complete (which would normally happen if Kubelet works ok): Not sure what is the expected behavior of DS here. I guess DS could look at what is the expected , and if exceeded substantially (like a minute or two) then force-delete the pod itself.
DS should consider the Pod terminal and, as such, create a replacement. Note that I'm referring to a state of the Pod where it has a terminal phase, regardless of whether it has a deletionTimestamp.
I see, going to open a dedicated DS Issue to move the discussion on priority and the shape of the fix there. Let's focus here on the Kubelet.
/cc
/triage accepted /priority important-soon
/cc
A pod that has been rejected due to admission should be continually readded to the pod worker by HandlePodCleanups until it succeeds at being terminated. That's because the pod is not in the desired set. Is HandlePodCleanups not adding it it in "Kill" state to the pod worker?
I took a closer look at this and is not handling this case. This is because the pod has been rejected at kubelet admission time, so a pod worker has not started for it. Additionally, the pod is not visible in the runtime, so it's not an orphan pod either. The main reason is because the pod has never started, so there is no pod worker driving it. The pod is returned from the desired state, pod manager (as part of . However, the pod is filtered out as part of because there was never a pod worker driving it, so there no for it. And since the pod is not part of and it's not an orphan pod (since it doesn't exist in runtime because we haven't started it), we end up not do anything with it in .
Should it be? I think right now our answer is no - but if we moved admission closer to the pod worker, we'd say yes? Pros and cons? We start pod workers for terminating a pod - is rejecting a pod a type of rejection? For the short term, why wouldn't HandlePodCleanups manage this case? It's the "resync" action for the kubelet as controller, and so it's nominally responsible for forcing all other components (pod worker, status manager, runtime) to return to state. Should it dispatch a TerminatePod call for these rejected pods that are not yet deleted, not known to the pod worker, and not known to the status manager?
Long term this might be the best option - having every path go through pod worker and the regular pod termination (i.e. instead of special casing special situations like admission rejection, may be simpler to reason about. Adding this to is definitely one option and I think we could expand it to cover this case. The only issue I see handling it there, is there could potentially be a large number of pods that are in terminal phase, not have a deletion timestamp set, and unknown pod worker and pod runtime. For each of these pods, we would have to call in every iteration of until someone eventually set the deletion timestamp and the pod would be deleted. Looking at the code, it should short-circuit , but we would need to confirm. Me and were also considering two other options: during kubelet rejection. I think it should be fine overall, but has a risk that after a kubelet restart, if a previous pod was still in runtime and it's running, could lead to the pod being deleted by status manager before the orphaned pod is terminated. by pod worker during kubelet rejection. It should go through pod worker and check that all containers are terminated, so it should avoid issue of the first approach. It sounds like option 2 or adding this HandlePodCleanups is the optimal paths forward. Curious to get your thoughts on the options here.
/retitle Pod graceful deletion hangs if the kubelet already rejected that pod at admission time There are 2 admission phases for Pods (API server admission, kubelet admission). This issue is about the second phase only.
Hi, are there any workarounds in 1.27?
The latest version of GKE contains a mitigation for this issue when it's related to Spot VM preemptions. It's still rolling out in some regions/zones. Please ensure you are in the latest patch version available in GKE.
The fix was cherry-picked for the 1.27 branch in , so it should be included in 1.27.4.
; we're on the latest patch () and we see this.
Can you clarify what you observe? We fixed a few different things that were related. Some of the fixes should be available in 1.27.3 and the last one in 1.27.4.
Hey no problem, we upgraded our cluster from to today, a new node came up and 37 pods were scheduled to it. One of those failed with: We've never seen this before in the 3+ years we've used GKE and upgraded in this manner, so appears to be related to the 1.27 release. I assume you work for Google? If it would help you I can raise a support case and give you the case number so you can inspect our logs.
That sounds unrelated to this issue. Pod failures (even if uncommon) are expected in a distributed system such as kubernetes. What's important is that the system should recover. This usually means that a replacement pod is created. In your case, was a replacement pod not created? (for Pods backed by Daemonset, StatefulSet, Job, Deployment). Nevertheless, just the error message you describe sounds more like , not this issue.
I am getting at 1.27.4-gke.900 running at GCP.

Discussion:
metrics.RestartedPodTotal.WithLabelValues("true").Add(float64(restartCountStatic)) metrics.RestartedPodTotal.WithLabelValues("").Add(float64(restartCount)) <ins> // Complete termination of deleted pods that are not runtime pods (don't have // running containers), are terminal, and are not known to pod workers. // An example is pods rejected during kubelet admission that have never // started before (i.e. does not have an orphaned pod). // Adding the pods with SyncPodKill to pod workers allows to proceed with // force-deletion of such pods, yet preventing re-entry of the routine in the // next invocation of HandlePodCleanups. for _, pod := range kl.filterTerminalPodsToDelete(allPods, runningRuntimePods, workingPods) { klog.V(3).InfoS("Handling termination and deletion of the pod to pod workers", "pod", klog.KObj(pod), "podUID", pod.UID) kl.podWorkers.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodKill, Pod:        pod, }) } </ins> // Finally, terminate any pods that are observed in the runtime but not present in the list of // known running pods from config. If we do terminate running runtime pods that will happen // asynchronously in the background and those will be processed in the next invocation of

return nil } <ins> // filterTerminalPodsToDelete returns terminal pods which are ready to be // deleted by the status manager, but are not in pod workers. // First, the check for deletionTimestamp is a performance optimization as we // don't need to do anything with terminal pods without deletionTimestamp. // Second, the check for terminal pods is to avoid race conditions of triggering // deletion on Pending pods which are not yet added to pod workers. // Third, the check to skip pods known to pod workers is that the lifecycle of // such pods is already handled by pod workers. // Finally, we skip runtime pods as their termination is handled separately in // the HandlePodCleanups routine. func (kl *Kubelet) filterTerminalPodsToDelete(allPods []*v1.Pod, runningRuntimePods []*kubecontainer.Pod, workingPods map[types.UID]PodWorkerSync) map[types.UID]*v1.Pod { terminalPodsToDelete := make(map[types.UID]*v1.Pod) for _, pod := range allPods { if pod.DeletionTimestamp == nil { // skip pods which don't have a deletion timestamp continue } if !podutil.IsPodPhaseTerminal(pod.Status.Phase) { // skip the non-terminal pods continue } if _, knownPod := workingPods[pod.UID]; knownPod { // skip pods known to pod workers continue } terminalPodsToDelete[pod.UID] = pod } for _, runningRuntimePod := range runningRuntimePods { // skip running runtime pods - they are handled by a dedicated routine // which terminates the containers delete(terminalPodsToDelete, runningRuntimePod.ID) } return terminalPodsToDelete } </ins> // splitPodsByStatic separates a list of desired pods from the pod manager into // regular or static pods. Mirror pods are not valid config sources (a mirror pod // being created cannot cause the Kubelet to start running a static pod) and are

if pod.DeletionTimestamp == nil || kubetypes.IsMirrorPod(pod) { return false } <del> // Delay deletion of pods until the phase is terminal. </del> <ins> // Delay deletion of pods until the phase is terminal, based on pod.Status // which comes from pod manager. </ins> if !podutil.IsPodPhaseTerminal(pod.Status.Phase) { <del> klog.V(3).InfoS("Delaying pod deletion as the phase is non-terminal", "phase", status.Phase, "pod", klog.KObj(pod), "podUID", pod.UID) </del> <ins> // For debugging purposes we also log the kubelet's local phase, when the deletion is delayed. klog.V(3).InfoS("Delaying pod deletion as the phase is non-terminal", "phase", pod.Status.Phase, "localPhase", status.Phase, "pod", klog.KObj(pod), "podUID", pod.UID) </ins> return false } // If this is an update completing pod termination then we know the pod termination is finished. if podIsFinished { <del> klog.V(3).InfoS("The pod termination is finished as SyncTerminatedPod completes its execution", "phase", status.Phase, "pod", klog.KObj(pod), "podUID", pod.UID) </del> <ins> klog.V(3).InfoS("The pod termination is finished as SyncTerminatedPod completes its execution", "phase", pod.Status.Phase, "localPhase", status.Phase, "pod", klog.KObj(pod), "podUID", pod.UID) </ins> return true } return false

return checkMirrorPodDisappear(ctx, f.ClientSet, pod.Name, pod.Namespace) }, f.Timeouts.PodDelete, f.Timeouts.Poll).Should(gomega.BeNil()) }) <ins> // Regression test for https://issues.k8s.io/118472 ginkgo.It("should force-delete non-admissible pods created and deleted during kubelet restart", func(ctx context.Context) { podName := "rejected-deleted-pod" + string(uuid.NewUUID()) gracePeriod := int64(30) nodeName := getNodeName(ctx, f) podSpec := e2epod.MustMixinRestrictedPodSecurity(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      podName, Namespace: f.Namespace.Name, }, Spec: v1.PodSpec{ NodeName: nodeName, NodeSelector: map[string]string{ "this-label": "does-not-exist-on-any-nodes", }, TerminationGracePeriodSeconds: &gracePeriod, RestartPolicy:                 v1.RestartPolicyNever, Containers: []v1.Container{ { Name:  podName, Image: imageutils.GetPauseImageName(), }, }, }, }) ginkgo.By("Stopping the kubelet") startKubelet := stopKubelet() // wait until the kubelet health check will fail gomega.Eventually(ctx, func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, f.Timeouts.PodStart, f.Timeouts.Poll).Should(gomega.BeFalse()) // Create the pod bound to the node. It will remain in the Pending // phase as Kubelet is down. ginkgo.By(fmt.Sprintf("Creating a pod (%v/%v)", f.Namespace.Name, podName)) pod := e2epod.NewPodClient(f).Create(ctx, podSpec) ginkgo.By(fmt.Sprintf("Deleting the pod (%v/%v) to set a deletion timestamp", pod.Namespace, pod.Name)) err := e2epod.NewPodClient(f).Delete(ctx, pod.Name, metav1.DeleteOptions{GracePeriodSeconds: &gracePeriod}) framework.ExpectNoError(err, "Failed to delete the pod: %q", pod.Name) // Restart Kubelet so that it proceeds with deletion ginkgo.By("Starting the kubelet") startKubelet() // wait until the kubelet health check will succeed gomega.Eventually(ctx, func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, f.Timeouts.PodStart, f.Timeouts.Poll).Should(gomega.BeTrue()) // Wait for the Kubelet to be ready. gomega.Eventually(ctx, func(ctx context.Context) bool { nodes, err := e2enode.TotalReady(ctx, f.ClientSet) framework.ExpectNoError(err) return nodes == 1 }, time.Minute, f.Timeouts.Poll).Should(gomega.BeTrue()) ginkgo.By(fmt.Sprintf("After the kubelet is restarted, verify the pod (%v/%v) is deleted by kubelet", pod.Namespace, pod.Name)) gomega.Eventually(ctx, func(ctx context.Context) error { return checkMirrorPodDisappear(ctx, f.ClientSet, pod.Name, pod.Namespace) }, f.Timeouts.PodDelete, f.Timeouts.Poll).Should(gomega.BeNil()) }) // Regression test for an extended scenario for https://issues.k8s.io/118472 ginkgo.It("should force-delete non-admissible pods that was admitted and running before kubelet restart", func(ctx context.Context) { nodeLabelKey := "custom-label-key-required" nodeLabelValueRequired := "custom-label-value-required-for-admission" podName := "rejected-deleted-run" + string(uuid.NewUUID()) gracePeriod := int64(30) nodeName := getNodeName(ctx, f) pod := e2epod.MustMixinRestrictedPodSecurity(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      podName, Namespace: f.Namespace.Name, }, Spec: v1.PodSpec{ NodeSelector: map[string]string{ nodeLabelKey: nodeLabelValueRequired, }, NodeName:                      nodeName, TerminationGracePeriodSeconds: &gracePeriod, RestartPolicy:                 v1.RestartPolicyNever, Containers: []v1.Container{ { Name:  podName, Image: imageutils.GetPauseImageName(), }, }, }, }) ginkgo.By(fmt.Sprintf("Adding node label for node (%v) to allow admission of pod (%v/%v)", nodeName, f.Namespace.Name, podName)) e2enode.AddOrUpdateLabelOnNode(f.ClientSet, nodeName, nodeLabelKey, nodeLabelValueRequired) ginkgo.DeferCleanup(func() { e2enode.RemoveLabelOffNode(f.ClientSet, nodeName, nodeLabelKey) }) // Create the pod bound to the node. It will start, but will be rejected after kubelet restart. ginkgo.By(fmt.Sprintf("Creating a pod (%v/%v)", f.Namespace.Name, podName)) pod = e2epod.NewPodClient(f).Create(ctx, pod) ginkgo.By(fmt.Sprintf("Waiting for the pod (%v/%v) to be running", f.Namespace.Name, pod.Name)) err := e2epod.WaitForPodNameRunningInNamespace(ctx, f.ClientSet, pod.Name, f.Namespace.Name) framework.ExpectNoError(err, "Failed to await for the pod to be running: (%v/%v)", f.Namespace.Name, pod.Name) ginkgo.By("Stopping the kubelet") startKubelet := stopKubelet() // wait until the kubelet health check will fail gomega.Eventually(ctx, func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, f.Timeouts.PodStart, f.Timeouts.Poll).Should(gomega.BeFalse()) ginkgo.By(fmt.Sprintf("Deleting the pod (%v/%v) to set a deletion timestamp", pod.Namespace, pod.Name)) err = e2epod.NewPodClient(f).Delete(ctx, pod.Name, metav1.DeleteOptions{GracePeriodSeconds: &gracePeriod}) framework.ExpectNoError(err, "Failed to delete the pod: %q", pod.Name) ginkgo.By(fmt.Sprintf("Removing node label for node (%v) to ensure the pod (%v/%v) is rejected after kubelet restart", nodeName, f.Namespace.Name, podName)) e2enode.RemoveLabelOffNode(f.ClientSet, nodeName, nodeLabelKey) // Restart Kubelet so that it proceeds with deletion ginkgo.By("Starting the kubelet") startKubelet() // wait until the kubelet health check will succeed gomega.Eventually(ctx, func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, f.Timeouts.PodStart, f.Timeouts.Poll).Should(gomega.BeTrue()) // Wait for the Kubelet to be ready. gomega.Eventually(ctx, func(ctx context.Context) bool { nodes, err := e2enode.TotalReady(ctx, f.ClientSet) framework.ExpectNoError(err) return nodes == 1 }, time.Minute, f.Timeouts.Poll).Should(gomega.BeTrue()) ginkgo.By(fmt.Sprintf("Once Kubelet is restarted, verify the pod (%v/%v) is deleted by kubelet", pod.Namespace, pod.Name)) gomega.Eventually(ctx, func(ctx context.Context) error { return checkMirrorPodDisappear(ctx, f.ClientSet, pod.Name, pod.Namespace) }, f.Timeouts.PodDelete, f.Timeouts.Poll).Should(gomega.BeNil()) }) </ins> }) })