Code change:
I'll take a look at the failed builds and triage.
I went through the recent reboot failures in kubernetes-e2e-gke-ci-reboot and here is a summary. [6481, 6486, 6488] Kubelet never came back from the reboot, and stopped posting node status. Throughout the test suite, the failed node stayed silent: It's hard to dig further due to the lack of clarity. Gathering log would at least help us rule out some possible causes . [6485] Heapster seems to crash loop quite frequently, and sometimes it failed the test (due to being not ready).. cc [6487] kube-proxy pod is running but not ready before the reboot test even started: This is not really a reboot test failure since the cluster was already in a bad state. We should make sure all the pods are ready after each test (e.g. "should work after restarting kube-proxy" ran before the failed reboot test). Another thing that would help is to increase the log level of the gke cluster. The log level of a gce cluster is 4, but I think gke is at the regular 2. Due to the low log level, it's unclear when kube-proxy started failing the readiness probe.
Filed for (3).
The reboot tests basically haven't worked since 9pm last night: Interestingly, this lines up with the GKE equivalent for , but in that same window, has had no trouble. cc
Digging into logs from the runs linked.
All of the gke-ci failures in are heapster crash-looping after a reboot.
Changes to the heapster config in the last day: , . Nothing sticks out to me in those PRs.
Heapster could be in crashloop if it cannot reach influxdb. But that is a bad design, and should fix it in today's heapster with image 18. I double checked them, it is image version 18.
From , could be the num_nodes is unset, we give heapster no resource to run? If that is true, it could be point to the same version screw issue of our test infra.
Heapster v0.18.4 will crashloop if InfluxDB isn't available. However we can update HEAD to v0.19.0, which will not crash if the backend goes down. I will send a PR to update InfluxDB version. , Dawn Chen wrote:
GKE doesn't use InfluxDB, right?
Heapster is the only kube-system pod that's not ready in the tests. If it relies on influxdb, shouldn't influxdb be not ready too?
I like theory. My salt suggestion might've been horrible for unset ? I thought I did that correctly, but maybe my Python/Jinja was bad.
Yes, GKE doesn't use influxDB. My point is that if the backend storage is not ready, heapster is in crashloop. Vishnu made some changes to make heapster can run standalone, but looks like that fix is not included in heapster 18 based on his comment.
Okay. I'm still trying to catch up. Why is heapster on GKE waiting on backend storage? And if is unset (e.g. we haven't finished running configure-), it looks like we just give the
Catching others up: is seeing nodes that are actually hung with CPU pegged on reboot, so the heapster conversation may be a red herring.
There are plenty of tests that failed because kubelet didn't come back (and post status) after the reboot. There are a few that failed only because heapster was crash looping. They may be related or not...
I spun up a GKE test cluster: and the heapster manifest looks fine, so I think I ruled out anything regarding
I think is the winner. The reboot tests are a red herring.
I'm actually just seeing heapster flap on a fresh cluster, no reboots.
Heapster crashlooping is one problem. There are other cases, however, where kubelet just stops posting status all together. If heapster crashlooping can render kubelet unresponsive, that's a bug.
is the serial console output from the node that got pegged (in ). I rebooted the node to see if it would repro, and it came back up fine. EDIT: the link to the jenkins run.
I guess the serial console you posted is from build , the very recent one. I saw a similar one and the node never successfully come back. At this moment, I ignore those failures. The reboot failure caused by heapster crashloop is similar to what I observed 2 days ago for influxdb in gce cluster. The comment is at Can we retitle this one for (1)? Reboot test flakiness is caused by many reasons: version screw which is handled by test-infra, heapster and influxdb crashloop is handled by , kube-proxy not ready is handled by cluster team.
Ok. So we think we have the heapster flakes, versioning stuff, and kube-proxy stuff all taken care of elsewhere? I'm fine with targeting this issue at the "node doesn't come back after reboot" case. Is that what you were recommending?
Yes. 1) heapster in crashloop if backend is not ready is a known issue for a while, and fix is merged to heapster repositry for a while. We just need to upgrade our heapster after more testing (PR: ) 2) influxdb one requires more information, but we can handle them through 3) version screw is handled by test-infra through . already test/flake label for it 4) kube-proxy readiness issue is handled through
where else should I be looking for hints on why that VM didn't come back to life until I kicked it? Or did I destroy any evidence we might have had?
Fun fact: I thought this might be related the timing of when I made the ContainerVM change on the GKE side, but I made that change for all releases 1.1.4. That picks up this test suite, too, because it's at , and it's green: So this is a divergence from master to 1.1, or between projects.
That suite does not run any reboot tests...
Dammit, I thought I scanned for that. Good call. We need to fix that.
Here's the set of suites that run the reboot tests (thanks kubernetes-e2e-gce kubernetes-e2e-gce-reboot kubernetes-e2e-gke-ci kubernetes-e2e-gke-ci-reboot
Shouldn't we decided only X-reboot test suites run those reboot tests since they are disruptive?
Hm. builds 6478-6480 are all green and all after the (GKE-side) ContainerVM change, so we're looking for something else.
The container-vm change was merged at , which lines up with kubernetes-e2e-gke-ci-reboot 6481 (the beginning of the failures).
6481 is the one with the new containervm change. It includes kubernetes-build/7822, which has the change.
/ That's the OSS side change, which is irrelevant. GKE flipped at 19:45:55.
Ahhh. Fair enough.
Actually, and I just determined that the GKE-side change didn't make it out until 01:45:44, it looks like, so indeed nothing to do with ContainerVM.
I took a more careful look at the serial output pasted, and this is interesting: We're actually in the middle of a clean shutdown, trying to unmount and failing. This same stacktrace repeats every 120s.
I spun up an e2e cluster and with minimal testing have a node in the state of It's the same kernel hang on shutdown. ... and then I accidentally deleted it when I went to bisect, but ohwell. I'm sure I'll be able to get one back if I need.
Okay, so I've confirmed that at the build level at , I can spin up a GKE cluster and reboot tests fail. I'm going to peel back to and see if they pass (because that matches ). ETA: For reference, I'm able to tickle this with just:
7819 had a kernel hang on the second run as well, though slightly different signature (still aufs in the end):
For my next test, I'm peeling back to v1.1.4. We have no test coverage on reboots on the release branch of GKE right now, and I just want to guarantee this isn't some weird GCE hypervisor issue. And v1.1.4 is clean, at least for two runs worth.
This failed quickly on v1.1.5. Hrm. I'm wondering if my v1.1.4 test was a fluke, peeling back to try it there again. v1.1.5 only differs by ContainerVM (and go 1.4.3 vs 1.4.2), but all of our other timelines show that not being directly related. (Alternately, it's a lot easier on the newer kernel.) (Still no idea why the tests aren't triggered, but is.)
was very solid. I'm now trying on GKE test. This is a GKE unique thing, because after Friday morning at 1am, anything = will get deployed using the new ContainerVM. So I'm ret-conning a way to effectively try with the new ContainerVM, just to see if it's a contributing factor. It doesn't look like it should be. One of the other commits in the stream is a heapster version bump (from / ). If there's additional flapping or something caused by that (unique to GKE), it may be causing the hang. But that timeline doesn't match the line (relevant to , etc.)
Bad news bears: failed reboot tests. I'm having trouble rectifying this information with what we know about the timeline. But it does suggest there's a problem with ContainerVM at the very least, so there might be two problems here, so I'm rolling one back on GKE and we'll see what happens.
I've manually confirmed that by reverting deployment of on GKE, I can run reboot tests at and a latest build. Using a special build of latest that will still trigger deployment of , reboot tests fail. Well, damn.
GKE has been repushed and should, in theory, go back green soon. There's a lot I don't understand about this bug, and it's effectively going to stop us shipping and until we can figure it out.
Okay, I think that reverting was not all that's necessary. We unfortunately haven't managed to collect a green on yet. That gives me some comfort, because it means that the timeline-discomfort I had previously was founded.
Heh, I lied and right right writing that we collected our first one. Woo. I suspect will follow soon.
Haven't we updated go to 1.5 on Friday as well? I don't know if we use the same version of go for GKE as we do in GKE, but in general we know it causes trouble. cc for Heapster thing.
The timeline there is off, too. The golang1.5.3 PR went in at , which is well after the trouble started.
Morning status: Looks like we made a little greener, and didn't touch , so the patient had two viruses.
Okay, this is my suspicion, but I can't quite piece together what's going on: For some reason, something in (? but that busted the Heapster config, so it's confusing me as to why), caused this. I'm basing this on: the timeline of the timeline of If you suppose that we injected a bug in build 7820 that caused about a 60-70% cointoss as to whether we regress (which is similar to the rate we're seeing now that has been rolled back on GKE), it's not unreasonable that was green for , and that's the only one you have to explain away on this theory. After that, it goes kind of red, and then at about 1am, comes in and wrecks completely as well. Pulling it back much further, to, say, build 7819, and you have to explain the entire range from 6477-6480 as green, which is a much lower probability. So my money is on build 7820. (7821 is revision-equivalent, so boring to talk about.)
Oh. It is Heapster. I spun up a 7819 cluster and a 7820 cluster, and the difference is that even though still had slightly busted Salt syntax, it was functional enough to create the replication controller, whereas on 7819, there's no Heapster replication controller at all. So I think the Heapster flapping is entirely the cause of the remaining issues, so we really need to get in ASAP.
So yesterday afternoon I was wondering why build 7820 triggered such a large regression when, say, the bump to heapster in didn't regress us at all. Build 7820 finally let the the fix in , which was actually important because GKE at HEAD wasn't deploying (until last night), it was deploying only , so the pillar wasn't configured. I'm still not exactly sure what's going on here, and I had intentionally left blank for a while during this investigation because it didn't seem like it should matter, but it's obvious that if 7820 is the culprit that it's interesting. I pushed a fix last night to GKE that adds on 1.2 clusters as well, and we've collected our first green dot. I don't know why is still flakey, though.
Er, correction to my last comment. My change didn't make it out last night and won't be relevant until . The green is a fluke. Sad panda.
PR introduced syntax error, PR fixed the error although it introduced another syntax issue which was finally fixed in . if you are testing anything make sure you are using version with merged. I agree with that the problem is not with Heapster accessing InfluxDB. Also the same problem takes places in autoscaling suite (described here ). We started observing such problems around Jan 19th. AFAIK once the container crashes kubelet invoke docker inspect on the failing container. How about invoking also docker logs then? It would be very useful for debugging purposes.
Yes, I understand that fixed the issue fully, but at build 7820 the cluster that's created actually spins up a cluster with a replication controller .yaml with resources. At build 7819, it doesn't. The syntax in is busted, but it's apparently not so busted. Previous to , in build 7819 (and builds previous), there was no controller .yaml at all, so there was a window where GKE wasn't spinning up Heapster at all. Hence my conclusion.
The syntax in is correct from Salt point of view, although the generated file has empty line in the begging and is not accepted by Api Server, so the RC is not created.
That's inconsistent with the fact that Heapster is starting on a 7820 (GKE) cluster.
I have a new theory, which is that Heapster hasn't been running on GKE in 1.2 since , until .
It's not clear to me that upgrading heapster will solve this issue, since heapster doesn't rely on any backend on gke. Instead of asking kubelet to do that, it makes more sense for the test itself to get the logs through the apiserver. I think we should add that to the e2e test.
I spun up a cluster from just prior to and this is the set of RCs I see: So basically, when went in, Heapster was enabled for the first time in GKE (master, not the release branches) in months and we started flipping out. This is a serious test coverage issue, obviously.
I edited the above comment, but we would obviously catch this in the release branches because we have monitoring tied to Heapster related things. We need to be better here for CI.
I made some investigation. The problem with crashlooping Heapster is due to: The issue was fixed in I'll create a patch release and bump the version.
It looks like is green. I'm hoping that , which is the first build past , will go green as well. (The cycle time on that build is waaaaay too long.)
It looks like we're good! is green now!
for node problem detector.

Related discussion:
text: apiVersion: v1 kind: ReplicationController metadata: <del> name: heapster-v11 </del> <ins> name: heapster-v12 </ins> namespace: kube-system labels: k8s-app: heapster <del> version: v11 </del> <ins> version: v12 </ins> kubernetes.io/cluster-service: "true" spec: replicas: 1 selector: k8s-app: heapster <del> version: v11 </del> <ins> version: v12 </ins> template: metadata: labels: k8s-app: heapster <del> version: v11 </del> <ins> version: v12 </ins> kubernetes.io/cluster-service: "true" spec: containers: <del> - image: gcr.io/google_containers/heapster:v0.18.4 </del> <ins> - image: gcr.io/google_containers/heapster:v0.18.5 </ins> name: heapster resources: # keep request = limit to keep this container in guaranteed class