Change:
After , we observed pods moving from phase to . Presents on master and release-1.8 as it was picked in test script has an command for basically doing a CLI watch or you can look in the kubelet log to see the phase transition from to Thanks to for helping isolate this :+1: /sig node
/assign
taking a look. although at first glance it is strange this didnt happen with evicted pods prior to this.
Thanks. FYI, I did confirm that reverting the mentioned PR on master corrected the issue.
This showed up breaking a job controller that depended on the pod not transitioning back to pending. It's possible this can cause applications to fail / run multiple times in potentially dangerous ways. I'm not sure we should allow a status update that moves back to a non terminal state via validation. It's not a legal transition as we have defined phase. We should separately put a guard in place that will report a clear error to clients.
after thinking on it, I think that the situation with eviction is a little different in that the containers are terminated but they are not immediately cleaned up. It is the proximity between pod termination time and container removal time that is uncovering the issue, I think. AFAICT, some path is attempting to get the container status after the container has been removed. When this fails, something in the status manager gets messed up and it under the impression that the pod is new and the status needs to be reinitialized.
Here are the relevant logs showing the bug: Still trying to figure out what caused the status update.
Ok, I figured out why this is happening. When we compute a container's status, we assume that . So deleting a pod's containers before the pod object is removed causes the state to return to pending.
Maybe we should expand this that I to prevent init container status reset to include all containers with an oldStatus
Would that prevent updating a container that crashed and restarted? Edit: nvm, I think that might work
I a comment at
i also commented here that it would be good if syncPod in status manager protected against an invalid transfer of phase... see: can we do that in this same PR?
My hope is to cherrypick this to 1.8. Is that check something we want to cherrypick as well?
i dont see why not... absent that its hard to know this fix actually fixes it... or that we dont regress.
do we want to do it in or ? is on the other side of a channel where a stack trace on the caller of could be more informative.
both? i just know that we get the latest state as told to api server on the other end of the channel...

Discussion:
// set status for Pods created on versions of kube older than 1.6 apiPodStatus.QOSClass = v1qos.GetPodQOS(pod) <ins> oldPodStatus, found := kl.statusManager.GetPodStatus(pod.UID) if !found { oldPodStatus = pod.Status } </ins> apiPodStatus.ContainerStatuses = kl.convertToAPIContainerStatuses( pod, podStatus, <del> pod.Status.ContainerStatuses, </del> <ins> oldPodStatus.ContainerStatuses, </ins> pod.Spec.Containers, len(pod.Spec.InitContainers) > 0, false, ) apiPodStatus.InitContainerStatuses = kl.convertToAPIContainerStatuses( pod, podStatus, <del> pod.Status.InitContainerStatuses, </del> <ins> oldPodStatus.InitContainerStatuses, </ins> pod.Spec.InitContainers, len(pod.Spec.InitContainers) > 0, true,

} oldStatus, found := oldStatuses[container.Name] if found { <del> if isInitContainer && oldStatus.State.Terminated != nil { </del> <ins> if oldStatus.State.Terminated != nil { </ins> // Do not update status on terminated init containers as // they be removed at any time. status = &oldStatus