Code change:
Appears to deadlock or otherwise hang after a few seconds and then hit the 5-minute timeout. From start of log: followed by stack traces of zillions of goroutines that mostly report being blocked for 4+ minutes. Recent instances: https://travis- https://travis- cc
I tried to reproduce the issue by run kubelet unittests 200 times (100 for v1beta1, and 100 for v1beta3), but failed to reproduce the issue. I poked around, and found the only difference between my local machine and test machine is that I am running go 1.4.2 and test machine is running 1.3 (both failure cases posted by are with 1.3.0). I remembered from online I saw a deadlock issue reported against old golang version, but haven't found that issue yet. I will continue trying to reproduce the issue.
Saw this in travis runs too: https://travis-
Yes, there have been multiple failures today. Check shippable, also.
FWIW, these are not all kubelet tests failing, but all of them have many kubelet go routines in the trace - mostly stuck in managePodLoop().
I reran a couple of thousands time last night before I leave, and all passed. Yu-Ju just pointed out that the failure posted above by is go1.4. Shippable is in a really bad shape today, tons of different tests are failed, which should be handled separately.
Ok, with some hacky debugging message into kubelet package, even I couldn't reproduce the very failure reported here, I made some progress on understanding the issue here. Basically there are two issues: 1) The flaky tests reported here are TestServeExecInContainer. All reports posted here have the same stack trace below: 2) There are tons of goroutines, especially one created for each pod. Kubelet leak those goroutines. One can see when test initial run, there are 106 goroutines. Test itself create another 41 goroutines, but none of them are created for podWorkder.
FWIW, the tests in were creating go routines it didn't need at all. I just removed them in
Thanks for cleanup. Now the number down to 91:
cc/
Did something change related to this recently? I haven't heard any flakiness reports for a long time, and now they're starting up again. Any ideas? Sent from my iPhone
I didn't see any changes related to this feature. Actually I even couldn't reproduce it on my desktop. Please note that the comment I made above all those leftover goroutines are separate issue thought.
This has exposed a deadlock in spdystream. I'll work with to come up with a fix.
Upstream issue
Cool ncdc@ ! dchen1107 by the way, I've run across a few similar bugs recently where their reproducibility hinges mainly on how the test cluster was brought up. If the test cluster comes up right, then the tests pass 100% forever. If the cluster comes up bad, the tests fail, sometimes forever. Were you rerunning your tests against the same test cluster? Or rebuilding your test cluster between every test invocation?
This is unittests, not e2e. We saw test flakiness through travis and shippable, but never on my desktop.
This will hopefully fix the deadlock: I will do a bump commit once it's merged.
Thanks for your quick fix. I will clean up goroutines after that.
it's entirely possible there's some bugs in my exec & port forwarding code around leaking goroutines. Are you going to take an initial look to see what's sticking around?
Yes, your test leak some goroutine, but other tests too.

Related discussion:
text: if timer != nil { timer.Stop() } <ins> // Start a goroutine to drain resetChan. This is needed because we've seen // some unit tests with large numbers of goroutines get into a situation // where resetChan fills up, at least 1 call to Write() is still trying to // send to resetChan, the connection gets closed, and this case statement // attempts to grab the write lock that Write() already has, causing a // deadlock. // // See https://github.com/docker/spdystream/issues/49 for more details. go func() { for _ = range resetChan { } }() </ins> i.writeLock.Lock() close(resetChan) i.resetChan = nil i.writeLock.Unlock() break Loop } }