Change:
It was consistently observed in a testing environment that a node-critical pod failed to start. event was present but was missing( is an init container). Also the following error was present but strangely only once: After some digging it seemed throwing an error matched the observations above. Eventually the init container and the pod to start The following tests in recreate the problem by and show how depending on the behavior is different. When the feature gate is enabled the last init container gets stuck in and never gets started. Therefore the pod get's stuck and never progresses The problem seems to stem from: There case get's skipped and the created container is expected to get started by the previous one by the following lines or But the provided tests show that this does not happen. Also if there is only one init container with the following logic should execute but won't because in this case this get's executed <details</details<details</details<details</details<details</details<details</details<details</details
/sig node can you help checking this issue report as according to it is a regression introduced with Thanks in advance!
I'll take a look. /assign
I guess this is a valid bug. /triage accepted /priority important-soon Can you provide a more specific scenario for this? Why do container runtime fail to start the container? (How can I reproduce it in e2e test?)
^ I guess this is the wrong assumption. If the container is in the state, the kubelet should resync it.
During node bootstrapping we had a bug where got restarted. So it is possible the following to happen: container gets created -restarts -container fails to start
We have a similar case. I have not yet looked at how containerd behaves at the moment, because it happens rarely and only at node startup. Pod stuck in the status. In the suppress status we have the following message: Alse we have the next events: Node characteristics: And it's like the error is reproduced on 1.29. The neighbouring cluster with k8s 1.28 is not affected by this problem
we have observed a similar behaviour but intermittently while upgrading our k8s cluster from k8s 1.29.0 to 1.29.6. During upgrade process, when the node gets rebooted the "calico-node" pod of that node got stuck with below error forever until i restart the pod manually. Note: We are not observing this every time. Please let me know for any additional info required.
/cc

Discussion:
switch status.State { case kubecontainer.ContainerStateCreated: <del> // nothing to do but wait for it to start </del> <ins> // The main sync loop should have created and started the container // in one step. If the init container is in the 'created' state, // it is likely that the container runtime failed to start it. To // prevent the container from getting stuck in the 'created' state, // restart it. changes.InitContainersToStart = append(changes.InitContainersToStart, i) </ins> case kubecontainer.ContainerStateRunning: if !types.IsRestartableInitContainer(container) {

ContainersToStart: []int{1}, }, }, <ins> "Restart the container if the container is in created state": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyNever }, mutateStatusFn: func(status *kubecontainer.PodStatus) { status.ContainerStatuses[1].State = kubecontainer.ContainerStateCreated }, actions: podActions{ SandboxID:         baseStatus.SandboxStatuses[0].Id, ContainersToKill:  map[kubecontainer.ContainerID]containerToKillInfo{}, ContainersToStart: []int{1}, }, }, </ins> } { pod, status := makeBasePodAndStatus() if test.mutatePodFn != nil {

ContainersToKill:      getKillMapWithInitContainers(basePod, baseStatus, []int{}), }, }, <del> "initialization in progress; do nothing": { </del> <ins> "an init container is stuck in the created state; restart it": { </ins> mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyAlways }, mutateStatusFn: func(pod *v1.Pod, status *kubecontainer.PodStatus) { status.ContainerStatuses[2].State = kubecontainer.ContainerStateCreated }, <del> actions: noAction, </del> <ins> actions: podActions{ SandboxID:             baseStatus.SandboxStatuses[0].Id, InitContainersToStart: []int{2}, ContainersToStart:     []int{}, ContainersToKill:      getKillMapWithInitContainers(basePod, baseStatus, []int{}), }, </ins> }, "restartable init container has started; start the next": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyAlways },