Change:
All the crio serial jobs (presubmits and periodics) are not displaying the test grid results and timing out. periodics: node-kubelet-serial-crio presubmits: pull-kubernetes-node-kubelet-serial-crio-cgroupv1 pull-kubernetes-node-kubelet-serial-crio-cgroupv2 At least 6 months If you look into the artifacts the jobs are running but there seems to be hitting a timeout in the e2e_node tests. This causes the grid to not post the results so it marks the job as failed. We tried increasing the timeout to 6 hours from 4 hours but the timeout still occurs. Something is not right and we could use some help on figuring out why this can be happening. is one example that works without issue. /sig node testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
Looks like our recent PR didn't fix this.
Looking into the serial logs, it looks like these tests are hitting OOM. [ 3276.] Out of memory: Killed process (dd) total-vm:10490032kB, anon-rss:3218048kB, file-rss:80kB, shmem-rss:0kB, UID:0 pgtables:6352kB oomscoreadj:1000 [ 3276.] Out of memory: Killed process (dd) total-vm:10490032kB, anon-rss:3218048kB, file-rss:80kB, shmem-rss:0kB, UID:0 pgtables:6352kB oomscoreadj:1000
/reopen Memory limits did not help..
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Bumping machine size didn't fix this.
/cc :eyes:
dd is run in a couple of tests as far as I can see: running dd to simulate disk pressure, is similar. So, I'd start with The only Serial test I can find is this one: I've removed label from it to see it helps the test, here is a test PR: Let's see how it goes.
This looks quite dangerous and explains why bigger instances and limits didn't help: It tries to allocate 100Gb of memory.
Successful run has 'Out of memory: Killed process' as well, e.g.
It looks like OOMKiller message is expected and doesn't impact the issue. It's produced by the test case mentioned above. Without the test case the job is still getting timeouted and doesn't produce artifacts. Here is an example of failed and timeouted job run without the test case:
I don't think there is something wrong with long running tests. There are a lot of slow tests in the job and if infra is loaded with other tasks all of the tests would run slower. Here is a list of test durations for successful and failed job runs. I don't see anything that can't be explained by the busyness of the infra. Please correct me if I'm wrong here. successful run ()failed run (): One strange thing though is that the total time for the failed test(.577 sec / 60 = 365.2596m) is less than the job timeout (420m). Any ideas why the job was timed out?
Without slow tests finished in around 1h 25m and produced valid artifacts.
Without slow tests and with OOMKiller test the job still works ok:
With all tests the issue persisted despite the 10h timeout: I'll start bisecting. Hopefully will spot the test that causes everything to slow down.
One thing I noticed is that these tests are running two times. You can search for the start tests banner and see that we restart if the test failed.
One of the failures in the failed link was because kubelet restart took more time than ImageMaximumGCAge of 1 minute, is this expected ? Attempt to fix this
Looking into what tests take the longest, I notice that the *Manager jobs are all pretty beefy. WDYT of moving these to a separate serial job?
An interesting datapoint I found between containerd serial and crio serial: in containerd: On crio, it takes 974 seconds.
Running this test on gcp with crio I don't see why it is taking up that long. I think that there is something else going on in this test. In the tests that were passing this test only take 14 seconds also.
With all but last 3 slow tests excluded the job still run more than 8h: Remaining slow tests: Resource-usage, Restart and SystemNodeCriticalPod In contrast the job with all slow tests excluded took only 1h25m: I'm not sure if the reason of slowness is in one of those 3 tests or it's something else. I'll remove them and run the job again.
I don't think they're running 2 times. The build log is duplicated if testrun fails, I guess. Look at the duration time for the same test case. They're exactly the same with a high precision, which wouldn't be possible for two different test runs.
Job run with only and slow tests enabled took more than 5 hours and didn't produce enough artifacts:
Successful run is executed on , failed is executed on . Could it be something related with this datapoint?
job is a current suspect. Test run with enabled and takes 1h 46 min and looks good: I'll try to enable more slow jobs with excluded job.
Job run with enabled resourceusage, systemnodecritical, log rotation, density and eviction tests takes 1h51m and looks good: So far so good. Side note: I'm not sure I understand why logrotation, density and eviction tests marked slow as they altogether take around 5 minutes.
Job run with all tests except of takes 2h and still looks good: I'll try to investigate test.
Job run with all tests except of and takes 2h 4m. and still looks good:
Can you point out to those runs, please? In my testing both pull jobs use n1-standard-4 and results don't differ.
I was referring to the job shared in the description
Job run with all tests except of takes 2h 9m and looks good: So, test case seems to slow down everything.
Nice! So we can just filter out that job then? Not sure if we have a slow lane for crio/containerd?
checking the code it seems it is performing an Ubuntu specific task. Is this applicable/needed also for Fedora?
is a test case in the test: We can either remove/comment out it or investigate why it slows down everything.
It looks like it restarts dbus and systemd, which theoretically should work on any linux with systemd and dbus. However, that comment could mean that it was never tested on anything except Ubuntu.
The Dbus test case was by Can you help us to investigate why test case slows everything down on Fedora/CoreOS ?
The test case was by and tested only with as crio pull jobs didn't exist at that time. However, it should be failing/slowing down ci jobs if they exist.
I opened up I talked with a few people at Red hat and it was suggested that this test is bad. Going to try just skipping it on fedora. Restarting dbus is not recommended
I asked as he has been working on gracefulshutdown and asked about a few of these problems on RHEL like OS. I think skipping should be fine for this.
Fixed remaining failed tests. Both jobs are green now:
/assign
Nice job

Discussion:
"fmt" "os" "os/exec" <del> "path/filepath" </del> "regexp" "strconv" "time"

return isNodeReady(&nodeList.Items[0]) } <del> func systemctlDaemonReload() error { cmd := "systemctl daemon-reload" _, err := runCommand("sh", "-c", cmd) return err } var ( dbusConfPath = "/etc/systemd/system/dbus.service.d/k8s-graceful-node-shutdown-e2e.conf" dbusConf     = ` [Unit] RefuseManualStart=no RefuseManualStop=no [Service] KillMode=control-group ExecStop= ` ) func overlayDbusConfig() error { err := os.MkdirAll(filepath.Dir(dbusConfPath), 0755) if err != nil { return err } err = os.WriteFile(dbusConfPath, []byte(dbusConf), 0644) if err != nil { return err } return systemctlDaemonReload() } func restoreDbusConfig() error { err := os.Remove(dbusConfPath) if err != nil { return err } return systemctlDaemonReload() } </del> const ( // https://github.com/kubernetes/kubernetes/blob/1dd781ddcad454cc381806fbc6bd5eba8fa368d7/pkg/kubelet/nodeshutdown/nodeshutdown_manager_linux.go#L43-L44 podShutdownReason  = "Terminated"

}) }) }) <del> ginkgo.Context("Dbus", func() { ginkgo.It("should continue to run pods after a restart", func(ctx context.Context) { // Allow dbus to be restarted on ubuntu err := overlayDbusConfig() framework.ExpectNoError(err) defer func() { err := restoreDbusConfig() framework.ExpectNoError(err) }() preRestartPodCount := 2 ginkgo.By(fmt.Sprintf("creating %d RestartAlways pods on node", preRestartPodCount)) restartAlwaysPods := newTestPods(preRestartPodCount, false, imageutils.GetPauseImageName(), "restart-dbus-test") createBatchPodWithRateControl(ctx, f, restartAlwaysPods, podCreationInterval) ginkgo.DeferCleanup(deletePodsSync, f, restartAlwaysPods) allPods := waitForPodsCondition(ctx, f, preRestartPodCount, startTimeout, testutils.PodRunningReadyOrSucceeded) if len(allPods) < preRestartPodCount { framework.Failf("Failed to run sufficient restartAlways pods, got %d but expected %d", len(allPods), preRestartPodCount) } ginkgo.By("restarting dbus and systemd", func() { stdout, err := exec.Command("sudo", "systemctl", "reset-failed", "dbus").CombinedOutput() framework.ExpectNoError(err, "Failed to reset dbus start-limit with systemctl: %v, %s", err, string(stdout)) stdout, err = exec.Command("sudo", "systemctl", "restart", "dbus").CombinedOutput() framework.ExpectNoError(err, "Failed to restart dbus with systemctl: %v, %s", err, string(stdout)) stdout, err = exec.Command("sudo", "systemctl", "daemon-reexec").CombinedOutput() framework.ExpectNoError(err, "Failed to restart systemd with systemctl: %v, %s", err, string(stdout)) }) ginkgo.By("verifying restartAlways pods stay running", func() { for start := time.Now(); time.Since(start) < startTimeout && ctx.Err() == nil; time.Sleep(10 * time.Second) { postRestartRunningPods := waitForPodsCondition(ctx, f, preRestartPodCount, recoverTimeout, testutils.PodRunningReadyOrSucceeded) if len(postRestartRunningPods) < preRestartPodCount { framework.Failf("fewer pods are running after systemd restart, got %d but expected %d", len(postRestartRunningPods), preRestartPodCount) } } }) ginkgo.By("verifying new pods can be started after a dbus restart") postRestartPodCount := 2 postRestartPods := newTestPods(postRestartPodCount, false, imageutils.GetPauseImageName(), "restart-dbus-test") createBatchPodWithRateControl(ctx, f, postRestartPods, podCreationInterval) ginkgo.DeferCleanup(deletePodsSync, f, postRestartPods) allPods = waitForPodsCondition(ctx, f, preRestartPodCount+postRestartPodCount, startTimeout, testutils.PodRunningReadyOrSucceeded) if len(allPods) < preRestartPodCount+postRestartPodCount { framework.Failf("Failed to run pods after restarting dbus, got %d but expected %d", len(allPods), preRestartPodCount+postRestartPodCount) } }) }) </del> ginkgo.Context("Kubelet", func() { ginkgo.It("should correctly account for terminated pods after restart", func(ctx context.Context) { node := getLocalNode(ctx, f)

} } <del> // runCommand runs the cmd and returns the combined stdout and stderr, or an // error if the command failed. func runCommand(cmd ...string) (string, error) { output, err := exec.Command(cmd[0], cmd[1:]...).CombinedOutput() if err != nil { return "", fmt.Errorf("failed to run %q: %s (%s)", strings.Join(cmd, " "), err, output) } return string(output), nil } </del> // getCRIClient connects CRI and returns CRI runtime service clients and image service client. func getCRIClient() (internalapi.RuntimeService, internalapi.ImageManagerService, error) { // connection timeout for CRI service connection