Change:
https://k8s- Only one green run out of the past 11 runs. Potential offenders: /cc (saw you guys discussed this on sig-node slack)
Related test failures: , ,
assigning this to you. and I can help out if necessary.
Still not sure what's going on. In 1 downward api modification failure, I saw a pod created, container started, container killed 1 second later by the kubelet, when it's supposed to run for 120s. Very strange.
See e.g.
So far I've only found the volume tests failing in the ubuntu trusty docker 1.10 vms. Not sure if that's 100% of them or just what I've looked at so far.
One idea is that cgroups are cleaned up before the pod is started, and that somehow causes the pod to fail. See . I'll try to verify my theory later...
That's consistent with my observation. Ubuntu+docker 1.10 seems to be the only one failing. If anyone wants to try reproducing this locally, this would be the image to use.
I noticed that too. The "pod ran to completion" is "pod started and immediately exited". I've tried to reproduce locally and can't find a commit around last Wednesday that obviously would have caused this. I'll post any information I find here.
I tried to reproduce locally (not on GCE) with trusty & docker 1.10 but the tests are all passing so far. Either I didn't wait long enough or it's something specific to the GCE trusty image being used.
-- i am not sure how that is possible. the odd bit to me is we ran w/ pod cgroups on in node e2e since 1.5 if i am not mistaken.
-- ignore my previous comment as previously we only deleted the cgroup when the pod was removed from the api server, and not just terminated. so yeah, its possible something is happening here...
maybe we need to pass in both activePods and runningPods to get a more accurate listing?
see from . Search for pod , which has cgroups cleaned up within 2 seconds of starting. May not be the culplit, but its worth looking into.
the runningPods cache appears like it could have a 2 second latency before updating its cache so it could be related and could be helped by passing in activePods as well...
see:
That is possible. I am also not sure when a pod is considered running, and in what order pods are created. Do we create cgroups before starting the container? Would a pod be returned by GetPods (from above comment) if it had cgroups set up, but the container wasn't running yet? It could also simply be that we are cleaning up a pod's cgroups before it is ever running (if that is possible based on above questions).
-- we create the cgroup before the container is started, so I think we need to pass activePods. see:

Discussion:
// restarted. status = pod.Status } <del> if status.Phase == v1.PodFailed || status.Phase == v1.PodSucceeded { return true } return false </del> <ins> return status.Phase == v1.PodFailed || status.Phase == v1.PodSucceeded || (pod.DeletionTimestamp != nil && notRunning(status.ContainerStatuses)) </ins> } // OkToDeletePod returns true if all required node-level resources that a pod was consuming have

// Remove any cgroups in the hierarchy for pods that are no longer running. if kl.cgroupsPerQOS { <del> kl.cleanupOrphanedPodCgroups(cgroupPods, runningPods) </del> <ins> kl.cleanupOrphanedPodCgroups(cgroupPods, activePods) </ins> } kl.backOff.GC()

// cleanupOrphanedPodCgroups removes cgroups that should no longer exist. // it reconciles the cached state of cgroupPods with the specified list of runningPods <del> func (kl *Kubelet) cleanupOrphanedPodCgroups(cgroupPods map[types.UID]cm.CgroupName, runningPods []*kubecontainer.Pod) { </del> <ins> func (kl *Kubelet) cleanupOrphanedPodCgroups(cgroupPods map[types.UID]cm.CgroupName, activePods []*v1.Pod) { </ins> // Add all running pods to the set that we want to preserve podSet := sets.NewString() <del> for _, pod := range runningPods { podSet.Insert(string(pod.ID)) </del> <ins> for _, pod := range activePods { podSet.Insert(string(pod.UID)) </ins> } pcm := kl.containerManager.NewPodContainerManager() // Iterate over all the found pods to verify if they should be running