Change:
cAdvisor API is not accessible from the node. It is accessible from inside the cadvisor container though. This is the reason for the cadvisor e2e test to fail and the recent heapster errors reported by users. The cadvisor version itself hasn't changed recently though.
cc
There were some changes recently around port forwarding ( that may be the cause.
From the node: From the cadvisor container:
reported the same issue today. Restarting the container seems to fix it. So, its something to do with the initial setup.
Could this be due to Docker 1.5? On Feb 23, 2015 6:12 PM, "Rohit Jnagal" wrote:
test machine is running Docker 1.4.1 It looks like cadvisor container is sharing its networking with the pause container as usual, but the pause container got restarted, so the networking setup isn't working anymore. Restarting cadvisor connects it to the newer instance of the pause container and it starts working again. However docker inspect shows that cadvisor is connected to network of a different pause container: And that container died soon after it started Kubelet killed the pause container. I think this is kubelet's doing. It tried to restart both pause and cadvisor container. It kills and restarts cadvisor container which connects to existing pause container. Then, it goes ahead and kills/restarts the pause container, leaving the new cadvisor container hanging. Maybe this is related to infra pods issue reported by earlier, or maybe just an ordering issue in kubelet.
Thanks for the detailed debugging :) with your report I think I found the issue. In we check if the container has changed and print the message above. After that we the POD container. We should be doing this before the container restart as you pointed out. I'm assigning this to myself and I will fix tomorrow. Will also rename the bug and re-purpose it.
cool, I was just staring at that code. All yours :+1:
Thanks for debugging the issue. This is a regression we introduced lately.
And when the test, we all assumed it was the test failing. Sorry, :)
I fell for that until I realized that the basic networking can also fail :)

Discussion:
glog.Errorf("Unable to get pod with name %q and uid %q info with error(%v)", podFullName, uid, err) } <del> podChanged := false </del> for _, container := range pod.Spec.Containers { expectedHash := dockertools.HashContainer(&container) dockerContainerName := dockertools.BuildDockerName(uid, podFullName, &container)

glog.V(3).Infof("pod %q container %q exists as %v", podFullName, container.Name, containerID) // look for changes in the container. <del> containerChanged := hash != 0 && hash != expectedHash if !containerChanged { </del> <ins> podChanged := hash != 0 && hash != expectedHash if !podChanged { </ins> // TODO: This should probably be separated out into a separate goroutine. // If the container's liveness probe is unsuccessful, set readiness to false. If liveness is succesful, do a readiness check and set // readiness accordingly. If the initalDelay since container creation on liveness probe has not passed the probe will return Success.

} glog.Infof("pod %q container %q is unhealthy. Container will be killed and re-created.", podFullName, container.Name, live) } else { <del> podChanged = true </del> glog.Infof("pod %q container %q hash changed (%d vs %d). Container will be killed and re-created.", podFullName, container.Name, hash, expectedHash) } if err := kl.killContainer(dockerContainer); err != nil {

continue } killedContainers[containerID] = empty{} <ins> if podChanged { // Also kill associated pod infra container if the pod changed. if podInfraContainer, found, _ := dockerContainers.FindPodContainer(podFullName, uid, dockertools.PodInfraContainerName); found { if err := kl.killContainer(podInfraContainer); err != nil { glog.V(1).Infof("Failed to kill pod infra container %q: %v", podInfraContainer.ID, err) continue } } } </ins> } // Check RestartPolicy for container

containersToKeep[containerID] = empty{} } <del> if podChanged { // Also kill associated pod infra container if the pod changed. if podInfraContainer, found, _ := dockerContainers.FindPodContainer(podFullName, uid, dockertools.PodInfraContainerName); found { if err := kl.killContainer(podInfraContainer); err != nil { glog.V(1).Infof("Failed to kill pod infra container %q: %v", podInfraContainer.ID, err) } } podInfraContainerID, err = kl.createPodInfraContainer(pod) if err != nil { glog.Errorf("Failed to recreate pod infra container: %v for pod %q", err, podFullName) } containersToKeep[podInfraContainerID] = empty{} } </del> // Kill any containers in this pod which were not identified above (guards against duplicates). for id, container := range dockerContainers { curPodFullName, curUUID, _, _ := dockertools.ParseDockerName(container.Names[0])

t.Errorf("unexpected error: %v", err) } <del> verifyCalls(t, fakeDocker, []string{"list", "stop", "list", "create", "start", "stop", "create", "start", "inspect_container"}) </del> <ins> //verifyCalls(t, fakeDocker, []string{"list", "stop", "list", "create", "start", "stop", "create", "start", "inspect_container"}) verifyCalls(t, fakeDocker, []string{"list", "stop", "stop", "list", "create", "start"}) </ins> // A map interation is used to delete containers, so must not depend on // order here.