Change:
This is blocking (reboot test). In running a test that reboots all of the nodes and then runs the e2e tests, we saw a failure with the influxdb / heapster monitoring test: The heapster logs, from running (), shows a ton of the following kubelet errors for all containers (this one happens to be for skydns): The shows a bunch of 500 errors, such as: However, the apiserver thinks that all the pods are running ( gives this ) Docker thinks that the pods are running ( on that node gives this ) In summary: apiserver and docker think containers are running but kubelet can't find them this manifested in the influxdb / heapster monitoring e2e test after node reboot I've still got the cluster around if anyone would like to debug. +cc (helped me debug this (thanks!)) +cc (node) +cc (interested in getting in, which this is blocking)
Can I access your test nodes?
Is this an active problem? I'd check if the cAdvisor thinks the containers exist (it is the one returning the error). I'd then scan the logs to see if cAdvisor had any errors related to that container.
Unfortunately during debugging, I restart kubelet service on that node, the issue is gone. The problem could be cadvisor cannot get stats, but I couldn't find any cadvisor related error messages. How easy to reproduce the issue here? Next time we might access cadvisor's page to see if there is no such container information. Thanks!
(and Thanks for looking into this! I believe I can reproduce easily but I'll confirm. Where do I look for cAdvisor info? I don't see a log in or , and I don't see a cAdvisor container running?
cAdvisor is running as part of kubelet now. If you can reproduce the issue, could you please change kubelet's log level to 4 so that we have more cadvisor log here? Thanks!
Yes! How do I do that?
I can reproduce this by doing the following: After this, I got two failures: ... and I have another node with more of the same kubelet errors: (If I need to change the Kubelet log level at compile-time I can do that and repro on a new cluster; otherwise, I've got another cluster with the issue that can be introspected...)
There is a race between cadvisor initialization and docker. cadvisor failed to register docker factory, and bailed out. is going to make cadvisor more resilient to the issue of docker is not available, and have some retry logic. I am going to change kubelet initialization logic a little bit to make sure docker is running before we start cadvisor.
Awesome, thank you Dawn! Are there issues or PRs I can follow to know when this is unblocked?
is one of them. is working on the second.

Discussion:
var _ Interface = new(Fake) <ins> func (c *Fake) Start() error { return nil } </ins> func (c *Fake) ContainerInfo(name string, req *cadvisorApi.ContainerInfoRequest) (*cadvisorApi.ContainerInfo, error) { return new(cadvisorApi.ContainerInfo), nil }

if err != nil { return nil, err } <del> err = m.Start() if err != nil { return nil, err } </del> cadvisorClient := &cadvisorClient{ Manager: m,

return nil, err } } return cadvisorClient, nil } <ins> func (cc *cadvisorClient) Start() error { return cc.Manager.Start() } </ins> func (cc *cadvisorClient) exportHTTP(port uint) error { mux := http.NewServeMux() err := cadvisorHttp.RegisterHandlers(mux, cc, "", "", "", "", "/metrics")

var _ Interface = new(Mock) <ins> func (c *Mock) Start() error { args := c.Called() return args.Error(1) } </ins> // ContainerInfo is a mock implementation of Interface.ContainerInfo. func (c *Mock) ContainerInfo(name string, req *cadvisorApi.ContainerInfoRequest) (*cadvisorApi.ContainerInfo, error) { args := c.Called(name, req)

var unsupportedErr = errors.New("cAdvisor is unsupported in this build") <ins> func (cu *cadvisorUnsupported) Start() error { return unsupportedErr } </ins> func (cu *cadvisorUnsupported) DockerContainer(name string, req *cadvisorApi.ContainerInfoRequest) (cadvisorApi.ContainerInfo, error) { return cadvisorApi.ContainerInfo{}, unsupportedErr }

// Interface is an abstract interface for testability.  It abstracts the interface to cAdvisor. type Interface interface { <ins> Start() error </ins> DockerContainer(name string, req *cadvisorApi.ContainerInfoRequest) (cadvisorApi.ContainerInfo, error) ContainerInfo(name string, req *cadvisorApi.ContainerInfoRequest) (*cadvisorApi.ContainerInfo, error) SubcontainerInfo(name string, req *cadvisorApi.ContainerInfoRequest) (map[string]*cadvisorApi.ContainerInfo, error)

glog.Errorf("Failed to start ImageManager, images may not be garbage collected: %v", err) } <ins> err = kl.cadvisor.Start() if err != nil { kl.recorder.Eventf(kl.nodeRef, "kubeletSetupFailed", "Failed to start CAdvisor %v", err) glog.Errorf("Failed to start CAdvisor, system may not be properly monitored: %v", err) } </ins> err = kl.containerManager.Start() if err != nil { kl.recorder.Eventf(kl.nodeRef, "kubeletSetupFailed", "Failed to start ContainerManager %v", err)