Change:
Kubelet issues a docker inspect call after creating a container, but for very short lived containers there is a window in which they could exit before being inspected in the docker manager. If the container PID is 0, we should check the ExitCode and other relevant fields and handle them appropriately. [1] This is the suspected cause of several flakes: /cc
looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: I'd expect the second one to not have run in this case? Node logs:
Init containers have an implicit assumption that container statuses include results of previous SyncPod invocations. If that's not the case we may need to fix that (or at least mitigate it). On May 18, 2016, at 8:28 PM, Prashanth B wrote: looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: May 18 15:58:33.380: INFO: pod-init--1d4b-11e6-80a4- started at 2016-05-18T15:58:04-07:00 (2+1 container statuses recorded) May 18 15:58:33.380: INFO: Init container init1 ready: false, restart count 0 May 18 15:58:33.380: INFO: Init container init2 ready: true, restart count 0 I'd expect the second one to not have run in this case? Node logs: — You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub
This should be true. Is there any evidence that suggests the opposite? only shows conflicting view of whether init1 has run.
If the container exits already and it's not an infra container, kubelet should not error out. Although even if it does, I failed to see how this could affect the readiness of the init container without checking the code further.
Each syncPod invocation, we inspect the list of init container in order looking for lastTerminationState. There could be a bug in that logic, or if we didn't get the full status, we might make a bad decision. When determining which container to run next, we always look for the last init container in the chain that has success, then run the next container (unless the pod is RestartNever). , Yu-Ju Hong wrote:
Saw this flake again. How do we verify that container status is not lost between syn pods?
... between calls to syncPods
The second sync clearly saw the init1 container, and decided to launch init2. See "InitContainersToKeep" in the message below.
containers to keep can include failed containers. , Yu-Ju Hong wrote:
Discussed on slack. In summary, the problem has to do with how we convert the kubecontainer.PodStatus to the api.PodStatus. If a container is terminated, the function makes the decisions: the pod restart policy is "Always", and there is a "failure reason" recorded for the given container, this means the pod worker tried to (re)start the container but failed. Set the state to "Waiting" and populate the reason. , the container state is simply "terminated". Init containers don't obey the restart policy like regular containers, and the logic will need to be amended to work with them. is working on a PR. In addition to that, kubelet should not error out if a user container exited before it could apply the oom score (which is best effort in this case). fixes that.
Ahh, thanks for digging into the issue. This is the exact the concern I had when I reviewed the initial implementation of init container, which blocks pr for a while. I even asked to help me verify this case, unfortunately we didn't reproduce the issue back then. :-(
I wish I'd seen the flakes in the runs prior to merge.
We didn't notice this until we started running the tests against docker v1.11.1, which is much faster and hit the race condition differently.
I suppose that's a good thing...
It is hard to reproduce this. The extra check of last terminated container in your initial code makes me nervous, but we couldn't reproduce it due to that small race windows until we have a faster docker running on the node. :-)
I got "Pods should invoke init containers on a RestartAlways pod" flake (referenced from ): Logs:
All the fixes went in. Please reopen this issue if you see any more flakes.

Discussion:
// Handle the containers failed to be started, which should be in Waiting state. for _, container := range containers { <ins> if isInitContainer { // If the init container is terminated with exit code 0, it won't be restarted. // TODO(random-liu): Handle this in a cleaner way. s := podStatus.FindContainerStatusByName(container.Name) if s != nil && s.State == kubecontainer.ContainerStateExited && s.ExitCode == 0 { continue } } </ins> // If a container should be restarted in next syncpod, it is *Waiting*. if !kubecontainer.ShouldContainerBeRestarted(&container, pod, podStatus) { continue

cadvisorapi "github.com/google/cadvisor/info/v1" cadvisorapiv2 "github.com/google/cadvisor/info/v2" <ins> "github.com/stretchr/testify/assert" </ins> "k8s.io/kubernetes/pkg/api" apierrors "k8s.io/kubernetes/pkg/api/errors" "k8s.io/kubernetes/pkg/api/resource"

} } <ins> func verifyContainerStatuses(statuses []api.ContainerStatus, state, lastTerminationState map[string]api.ContainerState) error { for _, s := range statuses { if !reflect.DeepEqual(s.State, state[s.Name]) { return fmt.Errorf("unexpected state: %s", diff.ObjectDiff(state[s.Name], s.State)) } if !reflect.DeepEqual(s.LastTerminationState, lastTerminationState[s.Name]) { return fmt.Errorf("unexpected last termination state %s", diff.ObjectDiff( lastTerminationState[s.Name], s.LastTerminationState)) } } return nil } // Test generateAPIPodStatus with different reason cache and old api pod status. </ins> func TestGenerateAPIPodStatusWithReasonCache(t *testing.T) { // The following waiting reason and message  are generated in convertStatusToAPIStatus() startWaitingReason := "ContainerCreating" <ins> initWaitingReason := "PodInitializing" </ins> testTimestamp := time.Unix(123456789, 987654321) testErrorReason := fmt.Errorf("test-error") emptyContainerID := (&kubecontainer.ContainerID{}).String() <del> verifyStatus := func(t *testing.T, c int, podStatus api.PodStatus, state, lastTerminationState map[string]api.ContainerState) { statuses := podStatus.ContainerStatuses for _, s := range statuses { if !reflect.DeepEqual(s.State, state[s.Name]) { t.Errorf("case #%d, unexpected state: %s", c, diff.ObjectDiff(state[s.Name], s.State)) } if !reflect.DeepEqual(s.LastTerminationState, lastTerminationState[s.Name]) { t.Errorf("case #%d, unexpected last termination state %s", c, diff.ObjectDiff( lastTerminationState[s.Name], s.LastTerminationState)) } } } </del> testKubelet := newTestKubelet(t) kubelet := testKubelet.kubelet pod := podWithUidNameNs("12345678", "foo", "new")

Namespace: pod.Namespace, } tests := []struct { <del> containers                   []api.Container statuses                     []*kubecontainer.ContainerStatus reasons                      map[string]error oldStatuses                  []api.ContainerStatus expectedState                map[string]api.ContainerState </del> <ins> containers    []api.Container statuses      []*kubecontainer.ContainerStatus reasons       map[string]error oldStatuses   []api.ContainerStatus expectedState map[string]api.ContainerState // Only set expectedInitState when it is different from expectedState expectedInitState            map[string]api.ContainerState </ins> expectedLastTerminationState map[string]api.ContainerState }{ // For container with no historical record, State should be Waiting, LastTerminationState should be retrieved from // old status from apiserver. { <del> []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, []*kubecontainer.ContainerStatus{}, map[string]error{}, []api.ContainerStatus{{ </del> <ins> containers: []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, statuses:   []*kubecontainer.ContainerStatus{}, reasons:    map[string]error{}, oldStatuses: []api.ContainerStatus{{ </ins> Name:                 "with-old-record", LastTerminationState: api.ContainerState{Terminated: &api.ContainerStateTerminated{}}, }}, <del> map[string]api.ContainerState{ </del> <ins> expectedState: map[string]api.ContainerState{ </ins> "without-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: startWaitingReason, }},

Reason: startWaitingReason, }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedInitState: map[string]api.ContainerState{ "without-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: initWaitingReason, }}, "with-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: initWaitingReason, }}, }, expectedLastTerminationState: map[string]api.ContainerState{ </ins> "with-old-record": {Terminated: &api.ContainerStateTerminated{}}, }, }, // For running container, State should be Running, LastTerminationState should be retrieved from latest terminated status. { <del> []api.Container{{Name: "running"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "running"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:      "running", State:     kubecontainer.ContainerStateRunning,

ExitCode: 1, }, }, <del> map[string]error{}, []api.ContainerStatus{}, map[string]api.ContainerState{ </del> <ins> reasons:     map[string]error{}, oldStatuses: []api.ContainerStatus{}, expectedState: map[string]api.ContainerState{ </ins> "running": {Running: &api.ContainerStateRunning{ StartedAt: unversioned.NewTime(testTimestamp), }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedLastTerminationState: map[string]api.ContainerState{ </ins> "running": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID,

// recent start error or not, State should be Terminated, LastTerminationState should be retrieved from second latest // terminated status. { <del> []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:     "without-reason", State:    kubecontainer.ContainerStateExited,

ExitCode: 5, }, }, <del> map[string]error{"with-reason": testErrorReason, "succeed": testErrorReason}, []api.ContainerStatus{}, map[string]api.ContainerState{ </del> <ins> reasons:     map[string]error{"with-reason": testErrorReason, "succeed": testErrorReason}, oldStatuses: []api.ContainerStatus{}, expectedState: map[string]api.ContainerState{ </ins> "without-reason": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID,

ContainerID: emptyContainerID, }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedLastTerminationState: map[string]api.ContainerState{ </ins> "without-reason": {Terminated: &api.ContainerStateTerminated{ ExitCode:    3, ContainerID: emptyContainerID,

pod.Status.ContainerStatuses = test.oldStatuses podStatus.ContainerStatuses = test.statuses apiStatus := kubelet.generateAPIPodStatus(pod, podStatus) <del> verifyStatus(t, i, apiStatus, test.expectedState, test.expectedLastTerminationState) </del> <ins> assert.NoError(t, verifyContainerStatuses(apiStatus.ContainerStatuses, test.expectedState, test.expectedLastTerminationState), "case %d", i) } // Everything should be the same for init containers for i, test := range tests { kubelet.reasonCache = NewReasonCache() for n, e := range test.reasons { kubelet.reasonCache.add(pod.UID, n, e, "") } pod.Spec.InitContainers = test.containers pod.Status.InitContainerStatuses = test.oldStatuses podStatus.ContainerStatuses = test.statuses apiStatus := kubelet.generateAPIPodStatus(pod, podStatus) expectedState := test.expectedState if test.expectedInitState != nil { expectedState = test.expectedInitState } assert.NoError(t, verifyContainerStatuses(apiStatus.InitContainerStatuses, expectedState, test.expectedLastTerminationState), "case %d", i) } } // Test generateAPIPodStatus with different restart policies. func TestGenerateAPIPodStatusWithDifferentRestartPolicies(t *testing.T) { testErrorReason := fmt.Errorf("test-error") emptyContainerID := (&kubecontainer.ContainerID{}).String() testKubelet := newTestKubelet(t) kubelet := testKubelet.kubelet pod := podWithUidNameNs("12345678", "foo", "new") containers := []api.Container{{Name: "succeed"}, {Name: "failed"}} podStatus := &kubecontainer.PodStatus{ ID:        pod.UID, Name:      pod.Name, Namespace: pod.Namespace, ContainerStatuses: []*kubecontainer.ContainerStatus{ { Name:     "succeed", State:    kubecontainer.ContainerStateExited, ExitCode: 0, }, { Name:     "failed", State:    kubecontainer.ContainerStateExited, ExitCode: 1, }, { Name:     "succeed", State:    kubecontainer.ContainerStateExited, ExitCode: 2, }, { Name:     "failed", State:    kubecontainer.ContainerStateExited, ExitCode: 3, }, }, } kubelet.reasonCache.add(pod.UID, "succeed", testErrorReason, "") kubelet.reasonCache.add(pod.UID, "failed", testErrorReason, "") for c, test := range []struct { restartPolicy                api.RestartPolicy expectedState                map[string]api.ContainerState expectedLastTerminationState map[string]api.ContainerState // Only set expectedInitState when it is different from expectedState expectedInitState map[string]api.ContainerState // Only set expectedInitLastTerminationState when it is different from expectedLastTerminationState expectedInitLastTerminationState map[string]api.ContainerState }{ { restartPolicy: api.RestartPolicyNever, expectedState: map[string]api.ContainerState{ "succeed": {Terminated: &api.ContainerStateTerminated{ ExitCode:    0, ContainerID: emptyContainerID, }}, "failed": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID, }}, }, expectedLastTerminationState: map[string]api.ContainerState{ "succeed": {Terminated: &api.ContainerStateTerminated{ ExitCode:    2, ContainerID: emptyContainerID, }}, "failed": {Terminated: &api.ContainerStateTerminated{ ExitCode:    3, ContainerID: emptyContainerID, }}, }, }, { restartPolicy: api.RestartPolicyOnFailure, expectedState: map[string]api.ContainerState{ "succeed": {Terminated: &api.ContainerStateTerminated{ ExitCode:    0, ContainerID: emptyContainerID, }}, "failed": {Waiting: &api.ContainerStateWaiting{Reason: testErrorReason.Error()}}, }, expectedLastTerminationState: map[string]api.ContainerState{ "succeed": {Terminated: &api.ContainerStateTerminated{ ExitCode:    2, ContainerID: emptyContainerID, }}, "failed": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID, }}, }, }, { restartPolicy: api.RestartPolicyAlways, expectedState: map[string]api.ContainerState{ "succeed": {Waiting: &api.ContainerStateWaiting{Reason: testErrorReason.Error()}}, "failed":  {Waiting: &api.ContainerStateWaiting{Reason: testErrorReason.Error()}}, }, expectedLastTerminationState: map[string]api.ContainerState{ "succeed": {Terminated: &api.ContainerStateTerminated{ ExitCode:    0, ContainerID: emptyContainerID, }}, "failed": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID, }}, }, // If the init container is terminated with exit code 0, it won't be restarted even when the // restart policy is RestartAlways. expectedInitState: map[string]api.ContainerState{ "succeed": {Terminated: &api.ContainerStateTerminated{ ExitCode:    0, ContainerID: emptyContainerID, }}, "failed": {Waiting: &api.ContainerStateWaiting{Reason: testErrorReason.Error()}}, }, expectedInitLastTerminationState: map[string]api.ContainerState{ "succeed": {Terminated: &api.ContainerStateTerminated{ ExitCode:    2, ContainerID: emptyContainerID, }}, "failed": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID, }}, }, }, } { pod.Spec.RestartPolicy = test.restartPolicy // Test normal containers pod.Spec.Containers = containers apiStatus := kubelet.generateAPIPodStatus(pod, podStatus) expectedState, expectedLastTerminationState := test.expectedState, test.expectedLastTerminationState assert.NoError(t, verifyContainerStatuses(apiStatus.ContainerStatuses, expectedState, expectedLastTerminationState), "case %d", c) pod.Spec.Containers = nil // Test init containers pod.Spec.InitContainers = containers apiStatus = kubelet.generateAPIPodStatus(pod, podStatus) if test.expectedInitState != nil { expectedState = test.expectedInitState } if test.expectedInitLastTerminationState != nil { expectedLastTerminationState = test.expectedInitLastTerminationState } assert.NoError(t, verifyContainerStatuses(apiStatus.InitContainerStatuses, expectedState, expectedLastTerminationState), "case %d", c) pod.Spec.InitContainers = nil </ins> } }