Change:
I can reproduce this reliably with Calico v3.25.0 CNI, installed straight from the upstream . It seems to be a regression in Kubernetes 1.27.0-rc.0 compared to 1.27.0-beta.0, at least I can't reproduce it with beta.0. Cluster is deployed, and everything is ready & healthy. After a series of node reboots some pods are stuck in state and never recover: This of course disrupts the CNI and affects the scheduled workloads. The daemonset is "happy" with 2 pods being out: If I delete the pod manually, the pod recovers without any issues. The pods are restarted. Bring up a cluster with graceful node shutdown, reboot nodes a few times (including multiple node at anoce). Additional detailed output: <details</details<details</details<detailsnone </details<detailsTalos Linux 1.4.0-alpha.3 </details<details</details<details</details<detailsCalico v3.25.0, also reproducible with 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I wonder if that is related to the change
cc this might a regression in 1.27.0-rc.0 with this PR
Even if it is, it sounds like a bug in daemonset instead. Why wouldn't it replace pods that are terminated? It can probably be reproduced in integration.
I think there might be something with the fact that Calico terminates with exit code '0', so it feels like a "successful" run. Still such pod should be restarted, so I'm not sure why it doesn't get restarted after a reboot.
let me ask to get more details about reproducing this locally. Are you running this on Kind? How do you trigger the the restarts? Can you share mode details on the node configuration, such as the graceful shutdown period?
I initially hit this problem under AWS, but then I narrowed it down to Calico CNI. I'm using Talos Linux, this is the full kubelet config, using 30s/10s graceful shutdown period: <details</detailsLet me try to build a reproduce instruction in the next comment.
I also have complete output of under the details in the issue itself.
The daemonset controller should still recreate the pod in that scenario. /sig apps
The reproducer is to follow the , but it should be same under any Talos platforms except Docker (which doesn't have a graceful node shutdown, and any kind of shutdown). I verified it with the assets from the latest . The smallest I can reproduce it with is 1+1 cluster, but it looks same way e.g. in 3+2 cluster: Once the cluster is up, all the pods are running & healthy (). If I issue a reboot of the worker node with , with shows that pod on the worker node quickly enters state, and never recovers.
At the same time I don't quite understand what is exactly wrong, as if I just do a daemonset with command, the pod also enters state, but it gets quickly restarted by the via transition.
I can also send kubelet logs, and anything else which might be helpful to debug the issue, I'm also open to do a debug session with screen sharing.
This is because of . The pod doesn't reach the phase in this case. The kubelet just recreates the containers. However, when a pod is actually marked as "Succeeded" then kubelet would stop looking at it. However, it should be the responsibility of the daemonset controller to recreate a pod.
Following the sig-apps advise opened the PR () with an integration test for DaemonSet. It passes for Failed pods, but fails for Succeeded. In other words, the succeeded pods are not restarted. I'm yet to investigate (probably tomorrow) why this is so, but it seems like a bug. In the meanwhile some expertise about DaemonSets would be helpful.
Thanks for reporting this. I suppose this might be related to the fact that pods (created by daemonset) can now be marked as in v1.27. When graceful node shutdown happens, we terminate the pod with : which will trigger kubelet to terminate the pod in pod worker () and eventually I assume set a terminal phase on the pod. We should verify if that is what is happening here. In either case, +1 on comment that I would expect that daemonset controller should recreate a pod if it is marked completed.
I will note that the intent for graceful shutdown was always that pods could be marked Succeeded (vs just Failed), since the pod is indeed terminal. It's likely that this was previously racy (no guarantee succeeded made it to the apiserver) and fixed that. However, outside of graceful shutdown this should never occur - do we have any evidence a RestartAlways pod went succeeded in that case (wasn't clear from the thread)? That would be a serious bug.
I think Succeeded would also be possible in other scenarios, like deletion (deletionTimestamp!=nil). This I've tested for Job pods, but I think DS pods would behave the same. Except for deletion I think the behavior is limited to graceful node shutdown, which invokes the kill function, but respects the Succeeded phase: Other uses of the kill function unconditionally set Failed, such as the eviction manager: EDIT: oh, actually for Jobs we don't use restartPolicy=Always, but still deleted pods with restartPolicy=Always may still terminate with Succeeded phase as shown in this test:
Deletion with RestartAlways is ok, because the pod goes away reasonably fast (but we are adding latency in 1.27 to that path) as you note.

Discussion:
FailedPlacementReason = "FailedPlacement" // FailedDaemonPodReason is added to an event when the status of a Pod of a DaemonSet is 'Failed'. FailedDaemonPodReason = "FailedDaemonPod" <ins> // SucceededDaemonPodReason is added to an event when the status of a Pod of a DaemonSet is 'Succeeded'. SucceededDaemonPodReason = "SucceededDaemonPod" </ins> ) // controllerKind contains the schema.GroupVersionKind for this controller type.

// Emit an event so that it's discoverable to users. dsc.eventRecorder.Eventf(ds, v1.EventTypeWarning, FailedDaemonPodReason, msg) podsToDelete = append(podsToDelete, pod.Name) <ins> } else if pod.Status.Phase == v1.PodSucceeded { msg := fmt.Sprintf("Found succeeded daemon pod %s/%s on node %s, will try to delete it", pod.Namespace, pod.Name, node.Name) logger.V(2).Info("Found succeeded daemon pod on node, will try to delete it", "pod", klog.KObj(pod), "node", klog.KObj(node)) // Emit an event so that it's discoverable to users. dsc.eventRecorder.Eventf(ds, v1.EventTypeNormal, SucceededDaemonPodReason, msg) podsToDelete = append(podsToDelete, pod.Name) </ins> } else { daemonPodsRunning = append(daemonPodsRunning, pod) }

} } <ins> func validateDaemonSetPodsActive( podClient corev1client.PodInterface, podInformer cache.SharedIndexInformer, numberPods int, t *testing.T, ) { if err := wait.Poll(time.Second, 60*time.Second, func() (bool, error) { objects := podInformer.GetIndexer().List() if len(objects) < numberPods { return false, nil } podsActiveCount := 0 for _, object := range objects { pod := object.(*v1.Pod) ownerReferences := pod.ObjectMeta.OwnerReferences if len(ownerReferences) != 1 { return false, fmt.Errorf("Pod %s has %d OwnerReferences, expected only 1", pod.Name, len(ownerReferences)) } controllerRef := ownerReferences[0] if got, want := controllerRef.Kind, "DaemonSet"; got != want { t.Errorf("controllerRef.Kind = %q, want %q", got, want) } if controllerRef.Controller == nil || *controllerRef.Controller != true { t.Errorf("controllerRef.Controller is not set to true") } if pod.Status.Phase == v1.PodRunning || pod.Status.Phase == v1.PodPending { podsActiveCount += 1 } } return podsActiveCount == numberPods, nil }); err != nil { t.Fatal(err) } } </ins> // podUnschedulable returns a condition function that returns true if the given pod // gets unschedulable status. func podUnschedulable(c clientset.Interface, podNamespace, podName string) wait.ConditionFunc {

}) } <ins> func TestSimpleDaemonSetRestartsPodsOnTerminalPhase(t *testing.T) { for _, podPhase := range []v1.PodPhase{v1.PodSucceeded, v1.PodFailed} { t.Run(string(podPhase), func(tt *testing.T) { forEachStrategy(tt, func(t *testing.T, strategy *apps.DaemonSetUpdateStrategy) { ctx, closeFn, dc, informers, clientset := setup(t) defer closeFn() ns := framework.CreateNamespaceOrDie(clientset, "daemonset-restart-terminal-pod-test", t) defer framework.DeleteNamespaceOrDie(clientset, ns, t) dsClient := clientset.AppsV1().DaemonSets(ns.Name) podClient := clientset.CoreV1().Pods(ns.Name) nodeClient := clientset.CoreV1().Nodes() podInformer := informers.Core().V1().Pods().Informer() informers.Start(ctx.Done()) go dc.Run(ctx, 2) ds := newDaemonSet("restart-terminal-pod", ns.Name) ds.Spec.UpdateStrategy = *strategy if _, err := dsClient.Create(ctx, ds, metav1.CreateOptions{}); err != nil { t.Fatalf("Failed to create DaemonSet: %v", err) } defer cleanupDaemonSets(t, clientset, ds) numNodes := 3 addNodes(nodeClient, 0, numNodes, nil, t) validateDaemonSetPodsAndMarkReady(podClient, podInformer, numNodes, t) validateDaemonSetStatus(dsClient, ds.Name, int32(numNodes), t) podToMarkAsTerminal := podInformer.GetIndexer().List()[0].(*v1.Pod) podCopy := podToMarkAsTerminal.DeepCopy() podCopy.Status.Phase = podPhase if _, err := podClient.UpdateStatus(ctx, podCopy, metav1.UpdateOptions{}); err != nil { t.Fatalf("Failed to mark the pod as terminal with phase: %v. Error: %v", podPhase, err) } // verify all pods are active. They either continue Running or are Pending after restart validateDaemonSetPodsActive(podClient, podInformer, numNodes, t) validateDaemonSetPodsAndMarkReady(podClient, podInformer, numNodes, t) validateDaemonSetStatus(dsClient, ds.Name, int32(numNodes), t) }) }) } } </ins> func TestDaemonSetWithNodeSelectorLaunchesPods(t *testing.T) { forEachStrategy(t, func(t *testing.T, strategy *apps.DaemonSetUpdateStrategy) { ctx, closeFn, dc, informers, clientset := setup(t)