Change:
ContainersToKill:  map[kubecontainer.ContainerID]containerToKillInfo{}, }, }, <ins> "Verify we create a pod sandbox if no ready sandbox for pod with RestartPolicy=Never and no containers have ever been created": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyNever }, mutateStatusFn: func(status *kubecontainer.PodStatus) { // no ready sandbox status.SandboxStatuses[0].State = runtimeapi.PodSandboxState_SANDBOX_NOTREADY status.SandboxStatuses[0].Metadata.Attempt = uint32(2) // no visible containers status.ContainerStatuses = []*kubecontainer.ContainerStatus{} }, actions: podActions{ SandboxID:         baseStatus.SandboxStatuses[0].Id, Attempt:           uint32(3), CreateSandbox:     true, KillPod:           true, ContainersToStart: []int{0, 1, 2}, ContainersToKill:  map[kubecontainer.ContainerID]containerToKillInfo{}, }, }, </ins> "Kill and recreate the container if the container is in unknown state": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyNever }, mutateStatusFn: func(status *kubecontainer.PodStatus) {

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If a pod's restart policy is "Never", and kubelet failed to create the PodSandbox (for whatever reason) once, the pod will be stuck at "ContainerCreating" forever. What you expected to happen: Since the user/application containers were never created, kubelet should retry the PodSandbox creation. The fact that the pod remains in a ContainerCreating state while kubelet explicitly does nothing to resolve the situation also confuses the user. How to reproduce it (as minimally and precisely as possible): I don't have an easy way to trigger a podsandbox creation failure off the top of my head, but this should be doable by messing with the CNI binaries/configurations. Anything else we need to know?: Relevant code snippet: Environment: Kubernetes version (use ): Only checked the master branch (1.16), but all supported release branches should have the issue Cloud provider or hardware configuration: not relevant OS (e.g: ): not relevant Kernel (e.g. ): not relevant Install tools: Network plugin and version (if this is a network-related bug): not relevant; though the initial PodSandbox creation failure could be caused by networking issues. Others:
thanks for posting! I'm looking at the code, and it seems like one fix could be checking if this pod has an associated sandbox - if it does, we want to respect the current behavior. If it doesn't, we want to attempt to create the sandbox. I was thinking something similar to: Thoughts? Is there a simplex fix?
Hi! Thank you for filing this - it matches the behaviour we've been seeing. Am I correct that this is a regression from (), and so only affects Kubernetes =1.13 ?
Your solution may not work for the one specific failure I checked, where the PodSandbox container was created, but call to the CNI plugin failed (for some reason; could be caused by a race condition). This left us a not-ready PodSandbox with no IP. Probably have to actually determine whether the user containers were created before. Need to check the code closer though.
Thanks. That's probably correct. I saw this in happening in 1.13
Gotcha - thanks! Taking a look at your pr now :)