Change:
I've seen a bunch of different failure, with number of tests failing due to: The example failure is here: [but there were 5+ more today in different places & suites]
I guess we need a more specific diagnosis. to diagnose/delegate.
Yes, I checked the instance posted above yesterday, and nothing is really outstanding. Meanwhile, I couldn't find other instances you mentioned since there is no link.
- what do you mean by "nothing is really outstanding". This shouldn't happen... Regarding other failures - I can probably paste more tomorrow.
Ahh, I should make my comment more clear here. I didn't mean there is no issue. From the log posted above, I saw after the test finished, test checked all nodes ready or not, and one node is not ready thus the test is marked not ready, and the node is not self-healing within a reasonable time, which is not acceptable. But from all available information, nothing is outstanding to me to figure out the root cause.
I managed finding one node not ready after test, and checked Looks like kubelet is failed the healthcheck, and killed by supervisord: By checking kubelet log, I noticed that after last SYNCLOOP, kubelet doesn't nothing except responding /healthz, and after 5m, it is killed by supervisord: 5m here makes me think kubelet might stuck at SyncPods because docker is not responding within a reasonable time, Kubelet failed healthz after 5ms. Unfortunately, supervisord omit all health failure reasons, I cannot prove my theory here. (1) I will file a separate issue to fix supervisord scripts to collect the failure reason, and collect those logs for our jenkins. Another found I have is that right before this, supervisord also killed docker daemon: recently changed docker version to docker ps so that we could reliably detect docker health, especially for But caveat is that docker ps might take long time occasionally, and supervisord might kill docker daemon unnecessarily, which might augment this issue here. (2) has a pending pr () to revert that change back, so that we can rule out docker healthcheck issue here. Also we need to have docker log available for debugging. Above two issues make me think we might hit docker 1.9.X performance issue mentioned at: , but our performance stress test couldn't reproduce it: We need to verify this. (3) Also reminded me that recently we saw kubelet stuck at syncloop in but with docker 1.8.3 then. In this case, we might run into the hairpin mode bug. Anyway, my pending pr () should help on this case. cc/
fwiw, I am trying to reproduce and I keep running into this. So this one seems easy to reproduce manually if you think that helps.
we have two NotReady nodes (as of 10 am CET), but there were 210 NotReady ones during cluster validation phase. I disabled enormous cluster test, so you can debug.
- you can take a look into machines from:
The problematic run we are talking about is this one: where only 789/1000 of nodes get ready within timeout.
and I filed a separate issue for the enormous cluster failure: Basically it is caused by crashlooping etcd-container due to oom killing.
- thanks for investigation!
Do you have more incidents of NotReady nodes after the initial cluster validation? I am looking for those failures :-)
I checked nikhiljindal's cluster mentioned: It is a dup issue of , the kernel deadlock issue triggered by hairpin mode: I keep this one open to make sure the situation of the node is becoming NotReady is not caused by docker 1.9.1.
I think http://pr- might be another example of this?
I believe the failure you mentioned at is also hairpin mode related issue. Unfortunately there is no logs from worker nodes for that jenkins build.
Recent e2es have failed with this symptom.
can't tell without system logs which we don't seem to collect when build times out. there were quite a few cases of node not ready in , docker was crashlooping. And at least the first few weren't the hung task kernel bug from I'm collecting in () because I want to know if it was some corrupt docker-local-db state type thing.
- another instance:
And one more:
IMO this is blocked on the lack of logs when a build times out, just like (and they're probably the same root cause)
A little bit summary here: docker used to kill pre-running docker containers at startup stage, but recent releases, they move this logic to shutdown stage, which is good. But docker 1.9.1, since the rearchitect of docker daemon including separate docker network, persistent network & storage state requires much long time. Meanwhile, our supervisor babysitter script does respect such change in docker without waiting for docker gracefully terminated. In this case, we might run into the issue of supervisor keep shutdown the docker daemon, and immediately start it, but start failed due to previous process is not terminated. Then supervisor repeat above procedure. I am going to send a fix shortly.

Discussion:
# which in turn restarts docker. /etc/init.d/docker stop <ins> # Make sure docker gracefully terminated before start again while pidof docker > /dev/null; do echo "waiting clean shutdown" sleep 10 done </ins> /etc/init.d/docker start <del> echo "waiting a minute for startup" sleep 60 </del> <ins> echo "waiting 30s for startup" sleep 30 </ins> while true; do if ! sudo timeout 10 docker version > /dev/null; then