Code change:
Both the GKE and Parallel tests recently had a run where they timed out. (after 90 minutes!) This is like 10x longer than the parallel run should be taking.
And the build project, too:
any ideas?
p0
yup, investigating this is next on my queue, right after I figure out why the PR jenkins is failing so many runs.
This happened again: In fact I've seen at least few cases like that: "at some point tests just stop and completely nothing is happening to the end" The same happened here - completely nothing happened during last hour of the run,
While the failure rates of the individual test jobs is not very high (e.g. kubernetes-e2e-gce-parallel : MTTF (Last 7 Days) 7 hr 27 min ... the fact that we gate auto-merging on about 7 of these jobs means that a merge fails almost every hour due to this.
I looked at 4 of the builds in kubernetes-e2e-gce-parallel, comparing the test cases marked "N/A" (i.e. no test result given) instead of their usual "PASSED" or "SKIPPED" report. So far, there's been only one test case that's been consistently N/A in all four of the builds I looked at: I'm not sure what might be happening, but I'm tempted to disable this test in parallel runs now while we investigate what's actually wrong.
OK, that test has been disabled. I'm hoping that the PR builds recover (along with some git fixes as referenced in ). Now on to figuring out why this is causing us problems...
Had a successful failure in build 8902. Grepping out the Ginkgo node number shows this test being the problem: So it looks like it got stuck in ? Is there anything in here that should hang without deadline?
I had another run get stuck in exactly the same place: What's worrying is that another parallel build (http://go/k8s-test/job/kubernetes-e2e-gce-parallel/7960/) timed out again early this morning, but this networking test was not enabled. This suggests that there's still something else triggering the problem.
I may have a partial fix for this in .
Have we seen this since merged? Can we close this bug?
- I think we had a lot of other problems during that time. I would suggest leaving it open during the weekend and close it on Monday if we don't see any problems.
I don't think we've seen this in a while. Please reopen if we're still having mysterious timeouts in Jenkins.
We're seeing this again in . It looks like someone put "Networking should function for intra-pod communication" test back in the parallel run recently. I think that we should revert whatever that PR was.
That test had been fine. More worrying is that my fix from seems to have vanished.
OK, looks like that was removed in , referencing issue . I'm not sure if the same issue is happening again (where operations mysteriously hang forever) or if it's something else. any thoughts?
I think your answer is here: The timeout field, and all references to it in clients were removed, based on the fact that clients should use request.Timeout() instead, but the clients were not updated to do that. I will send a PR shortly.
It seems like very few of the e2e tests are using this method, but we aren't seeing issues on most of the tests. I wonder if there's something in particular we're hitting with the networking test?
Could this be this magic number in ? I guess this means the checks only happen for about a 75 seconds. If one or two of the nodes are spinning up very slowly, this will trip over . Just a thought. Either way I think this needs to be configurable, I'll work on that angle.
I agree this "15" should probably be bigger. However, I'm pretty sure, that this shouldn't cause the test hang forever - it may cause test failures, but hanging forever test shouldn't happen...
Anyway - I've sent out out for review - we will see if that helps...
OK - it won't help, because it doesn't work :( - I asked if there is any reasonably easy way to do it...
Another instance: Again - the "Networking should function for intra-pod communication" is among tests that didn't run...
assigning this issue to you since you're working on a PR
- sure, although I'm not sure if this will solve the issue...
you can assign back to me if it doesn't :)
Unfortunately my PR didn't help. This just happened again: - reassigning back to you
Another instance:
And again:
Are folks still seeing this? I'm having a really hard time getting it to occur on-demand.
OK, I did find another timeout - The list of suspicious tests does not include the networking test this time: Kubectl client Kubectl patch should add annotations for pods in rc [Conformance] Kubectl client Simple pod should support exec through an HTTP proxy Pods should be submitted and removed [Conformance] Secrets should be consumable from pods [Conformance] Will have to keep searching for builds timing out to see if any new pattern emerges.
Looking more carefully, is another timeout. (I really wish it'd fail on my debug PR...) Suspicious tests: Job should scale a job down Kubectl client Simple pod should support exec through an HTTP proxy I'm gonna start suspecting that Kubectl HTTP proxy test now... are we doing anything in this test that might be blocking?
has a very curious failure: only 3 tests ran at all. It seems we are running the cluster-sanity checks in for every Ginkgo node, so there are probably cases like this run where it'll race and fail in mysterious ways (or possibly even deadlock)? We should probably use and (as documented ) to run the sanity checks, cleanup, and log dumping only once, rather than once per Ginkgo node. This may solve the mysterious timeouts.
Bumping down to P1 because this seems pretty rare nowadays (but I'll still address the issues I noted).
It seems like this issue is way too general: jobs timing out without a clear reason should have an action of making that reason clear. Is there any action we can take there?
I think the fixes I mentioned in will address the mysterious timeouts.
Happened here:
Hoping this one is gone for good, but I'll track and see.
Reopening; saw this again with
Final lines of that before being killed: Another run which timed out is kubernetes-pull-build-test-e2e-gce/. Final lines from that: Plus my earlier suspicions in make me think the kubectl proxy test is lacking a necessary timeout somewhere.
Moving discussion of that test case to .
With open, can we close this in favor of ?
Closing for now. If fixing doesn't fix, I'll keep looking...

Related discussion:
text: // test pods from running, and tests that ensure all pods are running and // ready will fail). if err := waitForPodsRunningReady(api.NamespaceSystem, testContext.MinStartupPods, podStartupTimeout); err != nil { <del> t.Errorf("Error waiting for all pods to be running and ready: %v", err) return </del> <ins> Failf("Error waiting for all pods to be running and ready: %v", err) </ins> } <ins> return nil }, func(data []byte) { // Run on all Ginkgo nodes }) // Similar to SynchornizedBeforeSuite, we want to run some operations only once (such as collecting cluster logs). // Here, the order of functions is reversed; first, the function which runs everywhere, // and then the function that only runs on the first Ginkgo node. var _ = ginkgo.SynchronizedAfterSuite(func() { // Run on all Ginkgo nodes }, func() { // Run only Ginkgo on node 1 if testContext.ReportDir != "" { CoreDump(testContext.ReportDir) } }) // TestE2E checks configuration parameters (specified through flags) and then runs // E2E tests using the Ginkgo runner. // If a "report directory" is specified, one or more JUnit test reports will be // generated in this directory, and cluster logs will also be saved. // This function is called on each Ginkgo node in parallel mode. func TestE2E(t *testing.T) { util.ReallyCrash = true util.InitLogs() defer util.FlushLogs() // We must call setupProviderConfig first since SynchronizedBeforeSuite needs // cloudConfig to be set up already. if err := setupProviderConfig(); err != nil { glog.Fatalf(err.Error()) } gomega.RegisterFailHandler(ginkgo.Fail) // Disable skipped tests unless they are explicitly requested. if config.GinkgoConfig.FocusString == "" && config.GinkgoConfig.SkipString == "" { // TODO(ihmccreery) Remove [Skipped] once all [Skipped] labels have been reclassified. config.GinkgoConfig.SkipString = `[Flaky]|[Skipped]|[Feature:.+]` } </ins> // Run tests through the Ginkgo runner with output to console + JUnit for Jenkins var r []ginkgo.Reporter if testContext.ReportDir != "" { <del> r = append(r, reporters.NewJUnitReporter(path.Join(testContext.ReportDir, fmt.Sprintf("junit_%02d.xml", config.GinkgoConfig.ParallelNode)))) </del> <ins> // TODO: we should probably only be trying to create this directory once // rather than once-per-Ginkgo-node. if err := os.MkdirAll(testContext.ReportDir, 0755); err != nil { glog.Errorf("Failed creating report directory: %v", err) } else { r = append(r, reporters.NewJUnitReporter(path.Join(testContext.ReportDir, fmt.Sprintf("junit_%02d.xml", config.GinkgoConfig.ParallelNode)))) } </ins> } <del> glog.Infof("Starting e2e run; %q", runId) </del> <ins> glog.Infof("Starting e2e run %q on Ginkgo node %d", runId, config.GinkgoConfig.ParallelNode) </ins> ginkgo.RunSpecsWithDefaultAndCustomReporters(t, "Kubernetes e2e suite", r) }