Change:
If you create a pod with a restart policy of never, and you have a container that fails to start (e.g. you set its command to a nonexistent file), the Kubelet currently tries to restart it infinitely because the container never started successfully, and therefore never received an exit code. You end up with something like this from Docker: Is it worth modifying the code to consider a container that failed to start as actually failed, so the pod's restart policy can govern it? Originally reported at
+1. Also, I'm not sure why we would even apply a restart policy, since the container never started. , Andy Goldstein wrote:
One issue that comes up with marking it failed is if its governed by a Replication Controller, a consequence of marking it Failed is a proliferation of Pods being rapidly created and then failed. I know pod garbage collection would help that, but it would be good if we can update the dependent controller to recognize in common cases to back-off if pods keep failing because of failure to start scenarios. , Vish Kannan wrote:
I thought Kubelet already surfaces both creation and start failure from docker. Let me find the old issue I filed.
Here is the issue and the initial fix is PR: This is a bug we should fix anyway.
what is the output of kubectl describe pods? Are container at waiting state with proper error message from docker?
With respect to replication controller, see also . It should back off.
kubelet does surface the error (see the status below). The problem is that it doesn't categorize the container as dead/terminated, which affects the restart policy. A container would be considered "terminated" in kubelet if its state is not Running and the . In the case where a container failed to start, the FinishedAt time is zero (see example below), and the container state would be . This may cause kubelet to not even back off restarting at all (didn't check thoroughly), let alone complying to the restart policy. We should make sure kubelet categorize these containers properly.
Ok, I reproduced the issue easily. Here are the current situation: When container cannot start by docker due to various reasons, Kubelet marked the container as Waiting state with proper reason (RunContainerError) and detail message for docker, so Pod will stuck at Pending state. I copied & pasted a PodStatus example below: When above situation occurs, Kubelet will retry in next syncLoop, which is the original issue reported here. Please note that I call this retry, instead of restart, which arguably align with our current , and do nothing. So that upstream layer can pick up such issue, for example, job controller in this case, and delete such bad pods. Or Kubelet can introspect ContainerStatus's reason: With restartPolicy never, Kubelet marks such Container as terminated with above proper reason. With restartPolicy onFailure, Kubelet marks such Container as terminated with above proper reason, and pod stays Running. Retry / Restart will be backoff by the same backoff policy on the node. With restartPolicy Always, Kubelet marks such Container as terminated with above proper reason, and pod stays Running. etry / Restart will be backoff by the same backoff policy on the node. For replication controlled pods, since we didn't mark Pod as Failed, there is no issue mentioned at
I think kubelet should not rely on the reason (which is best-effort), and simply stops relying solely on to categorize "terminated" containers.
what would you look at in addition to ?
in my example above, the container which failed to start had . To me, it's clear that there was a failed attempt trying to start the container. Even though it's finish time is zero, this container should be considered "terminated".
ah, good point! I hadn't noticed that ExitCode was nonzero.

Discussion:
} } } <del> } else if !inspectResult.State.FinishedAt.IsZero() { </del> <ins> } else if !inspectResult.State.FinishedAt.IsZero() || inspectResult.State.ExitCode != 0 { // When a container fails to start State.ExitCode is non-zero, FinishedAt and StartedAt are both zero </ins> reason := "" <del> message := "" </del> <ins> message := inspectResult.State.Error finishedAt := unversioned.NewTime(inspectResult.State.FinishedAt) startedAt := unversioned.NewTime(inspectResult.State.StartedAt) </ins> // Note: An application might handle OOMKilled gracefully. // In that case, the container is oom killed, but the exit // code could be 0. if inspectResult.State.OOMKilled { reason = "OOMKilled" <del> } else { </del> <ins> } else if inspectResult.State.ExitCode == 0 { reason = "Completed" } else if !inspectResult.State.FinishedAt.IsZero() { </ins> reason = "Error" <del> message = inspectResult.State.Error </del> <ins> } else { // finishedAt is zero and ExitCode is nonZero occurs when docker fails to start the container reason = ErrContainerCannotRun.Error() // Adjust time to the time docker attempted to run the container, otherwise startedAt and finishedAt will be set to epoch, which is misleading finishedAt = unversioned.NewTime(inspectResult.Created) startedAt = unversioned.NewTime(inspectResult.Created) </ins> } result.status.State.Terminated = &api.ContainerStateTerminated{ <del> ExitCode: inspectResult.State.ExitCode, Message:  message, Reason:   reason, StartedAt:   unversioned.NewTime(inspectResult.State.StartedAt), FinishedAt:  unversioned.NewTime(inspectResult.State.FinishedAt), </del> <ins> ExitCode:    inspectResult.State.ExitCode, Message:     message, Reason:      reason, StartedAt:   startedAt, FinishedAt:  finishedAt, </ins> ContainerID: DockerPrefix + dockerID, } if tPath != "" {

} } } <del> } else { // TODO(dchen1107): Separate issue docker/docker#8294 was filed result.status.State.Waiting = &api.ContainerStateWaiting{ Reason: ErrContainerCannotRun.Error(), } </del> } return &result }

continue } // first failure <del> if containerStatus.State.Terminated != nil { </del> <ins> if containerStatus.State.Terminated != nil && !containerStatus.State.Terminated.FinishedAt.IsZero() { </ins> ts = containerStatus.State.Terminated.FinishedAt break }