Change:
Which jobs are failing: Which test(s) are failing: The tests are failing in the . Since when has it been failing: 10/23/2020 Testgrid link: Reason for failure: It started failing at commit , but it's unknown if it's related to that. Anything else we need to know: We discussed in the Kubernetes SIG-Node CI subgroup and will start taking a look at it.
/assign
/sig node
Investigation notes: between successful and unsuccessful runs and found no notable differences. , and had a couple of failed health checks on initialization, which doesn't happen in the successful test runs, though there's only 2 seconds between starting readiness checks and an eventual successful check. config the same between runs looking through the kubelet logs for and , but I haven't found anything interesting yet. kubelet- failed health checks The below is a list of errors that occur in runs that succeeded and failed, which may be due to startup or specific testing scenarios. Bunch of these for numerous containers:
Both new and old test runs have a bunch of logs like this, implying that it's been having issues for a while: The failed one has this additional line: However, the doesn't have the same issues. I don't see the connection timeouts at all. I also don't see anything else in the kubelet logs that appears different between successful and failed runs. Unfortunately, I don't know enough about the infrastructure setup, but here are my current theories: about the infrastructure has gotten progressively more overloaded and now it's falling over. The tests run on 8 nodes versus 1, so they have more capacity there. Not totally buying this because why did it go from mostly successful to 100% failures rather than a mix for a bit. changed in the GCE configuration. Not sure how that is managed.
Diff there suggests three merges from k/k Only merge in test-infra is an image bump <img width="215" alt="image" src="https://user-"Interesting that a similar ci version and pull version of the job passes and the one ci version does not. pull: passing ci: failing ci: The failing ones do not have a node_e2e , but I don't see any of the expected log messages for failure of the services to start.
It is interesting that one passes and one does not. Primary differences are the number of nodes, resource settings, and the fact that the conformance tests use Docker.
In addition to the services log missing, the overall system run logs are also missing. May be related to any decision we make for
Would we be able to kick off a Prow job with custom configuration? In other words, to rule in or rule out something in the image change, can we kick off a job with the latest everything else but use the older image? In reference to the decision on , are you suggesting we pause investigation and see if we decide to just move or change the conformance tests?
/triage accepted /priority critical-urgent
Catching up on this. qq: has anyone vetted ? Ise that pr went in around the time this job began to fail The beforesuite (AI: we should list out what it does), fails to communicate with nodes the mention of 6443 ports looks interesting here.
Good find. Definitely looks suspicious. I'll dig into that.
From : This comes from , which has some node-conformance specific test setup, so maybe something interesting there.
quick interruption for instructions on how to reproduce this one: I think this may work Focus and skip The FOCUS and SKIP arguments im getting from Image config file The job also uses the following config file Make sure to copy it and name it (the argument to IMAGECONFIG_FILE above). Additional arguments The test args to this job are I think these are the defaults already but just in case may be actually useful to add the following to the above command Miscellaneous If you run into trouble with the above commands try going through The info here came from
You'll also need to use the flag to make sure it runs in Docker, I believe.
ah yeah, good catch! Thank you. Don't know about your availability, but if you are all busy ima allocate sometime later today to dive deeper into the info we have here
I'm going to try to spend time of this in the afternoon (PDT). Specifically, I was going to try to verify that the logs from starting up the test suite look as expected and possibly try to run the case to reproduce.
My first suspicion was to have the server running in another port than 6443 (which Kubelet has configured), this is not the case: Only o.SecureServing.BindAddress is being set explicitly, but the port comes from o.NewServerRunOptions().
I think it's related to the movement of this line. I couldn't figure out how to link to the PR so I made a comment.
as you notice before there's no , and we should be seeing Before:
I can repro the issue now, and I think the line was moved upward to generate the token before starting the kubelet. Moving it up out of the if-statement may help partially, but in the conformance test the kubelet is manually started ahead of this code, so it may not matter.
Make sense. Do you mean moving this out of the conformance check?
Haha, yea, that message is deceptive... I think I'll make a PR to at least let it be more correct and give us more info.
for alerts to subscribers, PR created. Please comment.
looking green!
The output looks very strange, I have no idea why. Output has double-underscore-number for the different runs, instead of the image used. [__08] vs [cos-lts]
Ya, I noticed that. It's weird. Not sure how it's set, but you can see that the junit config names have a similar issue: -junitcos- -
Ok, found the root of the problem. Here's the command for the master suite: Note , which is not present in the conformance tests. but not sure how it eventually gets here.

Discussion:
return configCmd } <del> func generateSecureToken(tokenLen int) (string, error) { </del> <ins> // GenerateSecureToken returns a string of length tokenLen, consisting // of random bytes encoded as base64 for use as a Bearer Token during // communication with an APIServer func GenerateSecureToken(tokenLen int) (string, error) { </ins> // Number of bytes to be tokenLen when base64 encoded. tokenSize := math.Ceil(float64(tokenLen) * 6 / 8) rawToken := make([]byte, int(tokenSize))

} if len(t.BearerToken) == 0 { var err error <del> t.BearerToken, err = generateSecureToken(16) </del> <ins> t.BearerToken, err = GenerateSecureToken(16) </ins> if err != nil { klog.Fatalf("Failed to generate bearer token: %v", err) } } // Allow 1% of nodes to be unready (statistically) - relevant for large clusters. if t.AllowedNotReadyNodes == 0 { t.AllowedNotReadyNodes = t.CloudConfig.NumNodes / 100

// If the services are expected to keep running after test, they should not monitor the test process. e2es = services.NewE2EServices(*stopServices) gomega.Expect(e2es.Start()).To(gomega.Succeed(), "should be able to start node services.") <del> klog.Infof("Node services started.  Running tests...") </del> } else { klog.Infof("Running tests without starting services.") }

importpath = "k8s.io/kubernetes/test/e2e_node/remote", deps = [ "//staging/src/k8s.io/apimachinery/pkg/util/errors:go_default_library", <ins> "//test/e2e/framework:go_default_library", </ins> "//test/e2e_node/builder:go_default_library", "//test/e2e_node/system:go_default_library", "//test/utils:go_default_library",

"k8s.io/klog/v2" <ins> "k8s.io/kubernetes/test/e2e/framework" </ins> "k8s.io/kubernetes/test/e2e_node/builder" "k8s.io/kubernetes/test/utils" )

// loadConformanceImage loads node conformance image from tar file. func loadConformanceImage(host, workspace string) error { <ins> klog.Info("Loading conformance image from tarfile") </ins> tarfile := filepath.Join(workspace, conformanceTarfile) if output, err := SSH(host, "timeout", conformanceImageLoadTimeout.String(), "docker", "load", "-i", tarfile); err != nil {

// with cluster e2e and launch kubelet outside of the test for both regular node e2e and // node conformance test. // TODO(random-liu): Switch to use standard node bootstrap script. <del> func launchKubelet(host, workspace, results, testArgs string) error { </del> <ins> func launchKubelet(host, workspace, results, testArgs, bearerToken string) error { </ins> podManifestPath := getPodPath(workspace) if output, err := SSH(host, "mkdir", podManifestPath); err != nil { return fmt.Errorf("failed to create kubelet pod manifest path %q: error - %v output - %q", podManifestPath, err, output) } startKubeletCmd := fmt.Sprintf("./%s --run-kubelet-mode --logtostderr --node-name=%s"+ <ins> " --bearer-token=%s"+ </ins> " --report-dir=%s %s --kubelet-flags=--pod-manifest-path=%s > %s 2>&1", <del> conformanceTestBinary, host, results, testArgs, podManifestPath, filepath.Join(results, kubeletLauncherLog)) </del> <ins> conformanceTestBinary, host, bearerToken, results, testArgs, podManifestPath, filepath.Join(results, kubeletLauncherLog)) </ins> var cmd []string systemd, err := isSystemd(host) if err != nil {

return "", err } <ins> bearerToken, err := framework.GenerateSecureToken(16) if err != nil { return "", err } </ins> // Launch kubelet. <del> if err := launchKubelet(host, workspace, results, testArgs); err != nil { </del> <ins> if err := launchKubelet(host, workspace, results, testArgs, bearerToken); err != nil { </ins> return "", err } // Stop kubelet.

// Run the tests klog.V(2).Infof("Starting tests on %q", host) podManifestPath := getPodPath(workspace) <del> cmd := fmt.Sprintf("'timeout -k 30s %fs docker run --rm --privileged=true --net=host -v /:/rootfs -v %s:%s -v %s:/var/result -e TEST_ARGS=--report-prefix=%s -e EXTRA_ENVS=%s %s'", timeout.Seconds(), podManifestPath, podManifestPath, results, junitFilePrefix, extraEnvs, getConformanceTestImageName(systemSpecName)) testOutput, err := SSH(host, "sh", "-c", cmd) if err != nil { return testOutput, err } return testOutput, nil </del> <ins> cmd := fmt.Sprintf("'timeout -k 30s %fs docker run --rm --privileged=true --net=host -v /:/rootfs -v %s:%s -v %s:/var/result -e TEST_ARGS=--report-prefix=%s -e EXTRA_ENVS=%s -e TEST_ARGS=--bearer-token=%s %s'", timeout.Seconds(), podManifestPath, podManifestPath, results, junitFilePrefix, extraEnvs, bearerToken, getConformanceTestImageName(systemSpecName)) return SSH(host, "sh", "-c", cmd) </ins> }

// standard kubelet launcher) func (e *E2EServices) Start() error { var err error <del> if !framework.TestContext.NodeConformance { if e.services, err = e.startInternalServices(); err != nil { return fmt.Errorf("failed to start internal services: %v", err) } </del> <ins> if e.services, err = e.startInternalServices(); err != nil { return fmt.Errorf("failed to start internal services: %v", err) } klog.Infof("Node services started.") // running the kubelet depends on whether we are running conformance test-suite if framework.TestContext.NodeConformance { klog.Info("nothing to do in node-e2e-services, running conformance suite") } else { </ins> // Start kubelet e.kubelet, err = e.startKubelet() if err != nil { return fmt.Errorf("failed to start kubelet: %v", err) } <ins> klog.Infof("Kubelet started.") </ins> } return nil }