Change:
I launched a cluster that was unhealthy and was trying to diagnose why when I ran across an interesting kubelet error. On the master, The validate endpoint is showing that the single node in the cluster cannot be reached: From on the node I see: and the kubelet appears to keep running but never recovers.
/cc
If I manually kill the kubelet () and then run on the master (to restart it) then it runs without re-printing the above error and the master's validate endpoint shows that the node is healthy. It would be much nicer if the kubelet could recover on its own without needing me to manually intervene.
I encountered this a second time this evening but with a slightly different error in the file The symptoms and remediation were the same.
This was on the 0.12.0 release.
Ok, the initial error reported above is due to docker is not running. The issue here is not kubelet is not running, instead of docker is shutdown. Why docker is shutdown here? because we upgrade to 1.5 docker?
I lowered the priority to P1. We recently monit to babysitting both docker and kubelet. Monit will check both docker and kubelet if running every 5 mins (). In your failure case, docker should be restart by monit. We can discuss the interval here. Maybe reduce it to 1 min.
Reducing the monitoring period SGTM.
How long you have wait when you query /validate? If it takes too long for a cluster being initialized, that is a separate issue and we should fix.
Ok, introduced DockerCache in kubelet, which makes thing worse. Basically it constructs a DockerCache at kubelet initialization time. If docker is not running, kubelet bailed out too. I am assigning this to you. I think we shouldn't fail kubelet initialization here just because of docker is shutdown. Kubelet should report NodeStatus including docker's status.
I will address the monit' s period issue in
Trying to address your questions in order: We have intermitent issues with docker not starting. is working on tracking it down (see ). On these nodes docker was running as was the kubelet, but the kubelet wasn't reachable by the master because it wasn't listening on the expected port. What does monit check? That the process is alive? Can it be configured to make sure it is "healthy" (for some definition of healthy)? If it just checks that the process is up, maybe kubelet should exit when it can't initialize properly and let monit restart it. We wait up to 10 minutes for /validate to report that all cluster components are healthy. The cluster is "initialized" in the sense that all components are running, but the cluster isn't functional because all components are not healthy. In this case manually bouncing the kubelet process pretty much immediately moved the node to healthy making the entire cluster functional. I did the manual checks 20-30 minutes after the initial 10 minute timeout, meaning that even at a 5 minute interval monit had plenty of chances to rectify the broken kubelet (and appeared to not do so).
Another data point: I just looked at a cluster that was launched at 8:23 this morning (5 hours ago) using version 0.12.0 and it has this same problem. I killed the kubelet and waited a couple of minutes, after which it was restarted and running happily (as reported by the master's /validate endpoint).
appears to be out of office the rest of this week and this is causing a significant number of cluster creation failures. Is there someone else who can own this and drive it to completion sooner?
cc/ Can you take a quick look at this?
what do you mean cluster creation failure? How big your cluster is? If you run 0.11 release, we should upgrade docker. also 's pr (() is not in 0.11 release. But 0.11 release doesn't have monit yet. We all observed sometimes docker daemon couldn't restart. With 0.11 that node is gone when docker failed to restart. For 0.12 release, monit is installed and configured to take care both docker and kubelet processes. pr is just making kubelet initialization failed when docker is not running. But both docker and kubelet are monitored by monit, eventually they are coming up. Things is definitely better than 0.11.
After 10 minutes the master's /validate endpoint is not reporting that all components are healthy 1 master, 1 node. I have only observed this on the 0.12.0 release. My question about monit is whether is does anything more than keep the process running. I have observed that it will restart the kubelet if the process exists (restoring it to working order) but it seems content to leave a broken kubelet process running for hours, making the cluster unusable.
I will take a look and send a fix, assigning to myself. I believe that it only checks whether the PID is alive. I will file a separate issue for using /healthz or an alternative ().
It looks like monit is capable of health checking:
Monit just detects if process is running or not today, but we could config it to do so.
Please just handle issue introduced by DockerCache here. We are going to handle healthcheck and interval in separate issue.
+1 :)
Thanks a lot for taking this issue and sorry for introducing the problem.

Discussion:
} func NewDockerCache(client DockerInterface) (DockerCache, error) { <del> containers, err := GetKubeletDockerContainers(client, false) if err != nil { return nil, err } </del> return &dockerCache{ client:        client, <del> cacheTime:     time.Now(), containers:    containers, </del> updatingCache: false, }, nil }