Code change:
Both the GKE and Parallel tests recently had a run where they timed out. (after 90 minutes!) This is like 10x longer than the parallel run should be taking.
And the build project, too:
any ideas?
p0
yup, investigating this is next on my queue, right after I figure out why the PR jenkins is failing so many runs.
This happened again: In fact I've seen at least few cases like that: "at some point tests just stop and completely nothing is happening to the end" The same happened here - completely nothing happened during last hour of the run,
While the failure rates of the individual test jobs is not very high (e.g. kubernetes-e2e-gce-parallel : MTTF (Last 7 Days) 7 hr 27 min ... the fact that we gate auto-merging on about 7 of these jobs means that a merge fails almost every hour due to this.
I looked at 4 of the builds in kubernetes-e2e-gce-parallel, comparing the test cases marked "N/A" (i.e. no test result given) instead of their usual "PASSED" or "SKIPPED" report. So far, there's been only one test case that's been consistently N/A in all four of the builds I looked at: I'm not sure what might be happening, but I'm tempted to disable this test in parallel runs now while we investigate what's actually wrong.
OK, that test has been disabled. I'm hoping that the PR builds recover (along with some git fixes as referenced in ). Now on to figuring out why this is causing us problems...
Had a successful failure in build 8902. Grepping out the Ginkgo node number shows this test being the problem: So it looks like it got stuck in ? Is there anything in here that should hang without deadline?
I had another run get stuck in exactly the same place: What's worrying is that another parallel build (http://go/k8s-test/job/kubernetes-e2e-gce-parallel/7960/) timed out again early this morning, but this networking test was not enabled. This suggests that there's still something else triggering the problem.
I may have a partial fix for this in .
Have we seen this since merged? Can we close this bug?
- I think we had a lot of other problems during that time. I would suggest leaving it open during the weekend and close it on Monday if we don't see any problems.
I don't think we've seen this in a while. Please reopen if we're still having mysterious timeouts in Jenkins.
We're seeing this again in . It looks like someone put "Networking should function for intra-pod communication" test back in the parallel run recently. I think that we should revert whatever that PR was.
That test had been fine. More worrying is that my fix from seems to have vanished.
OK, looks like that was removed in , referencing issue . I'm not sure if the same issue is happening again (where operations mysteriously hang forever) or if it's something else. any thoughts?
I think your answer is here: The timeout field, and all references to it in clients were removed, based on the fact that clients should use request.Timeout() instead, but the clients were not updated to do that. I will send a PR shortly.
It seems like very few of the e2e tests are using this method, but we aren't seeing issues on most of the tests. I wonder if there's something in particular we're hitting with the networking test?
Could this be this magic number in ? I guess this means the checks only happen for about a 75 seconds. If one or two of the nodes are spinning up very slowly, this will trip over . Just a thought. Either way I think this needs to be configurable, I'll work on that angle.
I agree this "15" should probably be bigger. However, I'm pretty sure, that this shouldn't cause the test hang forever - it may cause test failures, but hanging forever test shouldn't happen...
Anyway - I've sent out out for review - we will see if that helps...
OK - it won't help, because it doesn't work :( - I asked if there is any reasonably easy way to do it...
Another instance: Again - the "Networking should function for intra-pod communication" is among tests that didn't run...
assigning this issue to you since you're working on a PR
- sure, although I'm not sure if this will solve the issue...
you can assign back to me if it doesn't :)
Unfortunately my PR didn't help. This just happened again: - reassigning back to you
Another instance:
And again:
Are folks still seeing this? I'm having a really hard time getting it to occur on-demand.
OK, I did find another timeout - The list of suspicious tests does not include the networking test this time: Kubectl client Kubectl patch should add annotations for pods in rc [Conformance] Kubectl client Simple pod should support exec through an HTTP proxy Pods should be submitted and removed [Conformance] Secrets should be consumable from pods [Conformance] Will have to keep searching for builds timing out to see if any new pattern emerges.
Looking more carefully, is another timeout. (I really wish it'd fail on my debug PR...) Suspicious tests: Job should scale a job down Kubectl client Simple pod should support exec through an HTTP proxy I'm gonna start suspecting that Kubectl HTTP proxy test now... are we doing anything in this test that might be blocking?
has a very curious failure: only 3 tests ran at all. It seems we are running the cluster-sanity checks in for every Ginkgo node, so there are probably cases like this run where it'll race and fail in mysterious ways (or possibly even deadlock)? We should probably use and (as documented ) to run the sanity checks, cleanup, and log dumping only once, rather than once per Ginkgo node. This may solve the mysterious timeouts.
Bumping down to P1 because this seems pretty rare nowadays (but I'll still address the issues I noted).
It seems like this issue is way too general: jobs timing out without a clear reason should have an action of making that reason clear. Is there any action we can take there?
I think the fixes I mentioned in will address the mysterious timeouts.
Happened here:
Hoping this one is gone for good, but I'll track and see.
Reopening; saw this again with
Final lines of that before being killed: Another run which timed out is kubernetes-pull-build-test-e2e-gce/. Final lines from that: Plus my earlier suspicions in make me think the kubectl proxy test is lacking a necessary timeout somewhere.
Moving discussion of that test case to .
With open, can we close this in favor of ?
Closing for now. If fixing doesn't fix, I'll keep looking...

Related discussion:
text: zone := testContext.CloudConfig.Zone region, err := gcecloud.GetGCERegion(zone) if err != nil { <del> glog.Fatalf("error parsing GCE region from zone %q: %v", zone, err) </del> <ins> return fmt.Errorf("error parsing GCE/GKE region from zone %q: %v", zone, err) </ins> } managedZones := []string{zone} // Only single-zone for now cloudConfig.Provider, err = gcecloud.CreateGCECloud(testContext.CloudConfig.ProjectID, region, zone, managedZones, "" /* networkUrl */, tokenSource, false /* useMetadataServer */) if err != nil { <del> glog.Fatal("Error building GCE provider: ", err) </del> <ins> return fmt.Errorf("Error building GCE/GKE provider: ", err) </ins> } <del> } if testContext.Provider == "aws" { </del> <ins> case "aws": </ins> awsConfig := "[Global]n" if cloudConfig.Zone == "" { <del> glog.Fatal("gce-zone must be specified for AWS") </del> <ins> return fmt.Errorf("gce-zone must be specified for AWS") </ins> } awsConfig += fmt.Sprintf("Zone=%sn", cloudConfig.Zone) if cloudConfig.ClusterTag == "" { <del> glog.Fatal("--cluster-tag must be specified for AWS") </del> <ins> return fmt.Errorf("--cluster-tag must be specified for AWS") </ins> } awsConfig += fmt.Sprintf("KubernetesClusterTag=%sn", cloudConfig.ClusterTag) var err error cloudConfig.Provider, err = cloudprovider.GetCloudProvider(testContext.Provider, strings.NewReader(awsConfig)) if err != nil { <del> glog.Fatal("Error building AWS provider: ", err) </del> <ins> return fmt.Errorf("Error building AWS provider: ", err) </ins> } <del> } // Disable skipped tests unless they are explicitly requested. if config.GinkgoConfig.FocusString == "" && config.GinkgoConfig.SkipString == "" { // TODO(ihmccreery) Remove [Skipped] once all [Skipped] labels have been reclassified. config.GinkgoConfig.SkipString = `[Flaky]|[Skipped]|[Feature:.+]` </del> } <del> gomega.RegisterFailHandler(ginkgo.Fail) c, err := loadClient() if err != nil { glog.Fatal("Error loading client: ", err) } </del> <ins> return nil } // There are certain operations we only want to run once per overall test invocation // (such as deleting old namespaces, or verifying that all system pods are running. // Because of the way Ginkgo runs tests in parallel, we must use SynchronizedBeforeSuite // to ensure that these operations only run on the first parallel Ginkgo node. // // This function takes two parameters: one function which runs on only the first Ginkgo node, // returning an opaque byte array, and then a second function which runs on all Ginkgo nodes, // accepting the byte array. var _ = ginkgo.SynchronizedBeforeSuite(func() []byte { // Run only on Ginkgo node 1 </ins> // Delete any namespaces except default and kube-system. This ensures no // lingering resources are left over from a previous test run. if testContext.CleanStart { <ins> c, err := loadClient() if err != nil { glog.Fatal("Error loading client: ", err) } </ins> deleted, err := deleteNamespaces(c, nil /* deleteFilter */, []string{api.NamespaceSystem, api.NamespaceDefault}) if err != nil { <del> t.Errorf("Error deleting orphaned namespaces: %v", err) </del> <ins> Failf("Error deleting orphaned namespaces: %v", err) </ins> } glog.Infof("Waiting for deletion of the following namespaces: %v", deleted) if err := waitForNamespacesDeleted(c, deleted, namespaceCleanupTimeout); err != nil { <del> glog.Fatalf("Failed to delete orphaned namespaces %v: %v", deleted, err) </del> <ins> Failf("Failed to delete orphaned namespaces %v: %v", deleted, err) </ins> } }