Change:
node-kubelet-serial Kubelet should correctly account for terminated pods after restart Since test was created, flakes 10-20% of the time. Not entirely clear why, I haven't looked. No response /assign /sig node
Introduced by , maybe we should use to replace See the log below, a pod becomes Succeed may exhaust tens of seconds
/triage accepted
Yeah I only tested this locally and not in CI, it seems that in CI the pods take much longer. Timings just need to be tweaked.
/reopen still a number of flakes since the attached PR merged, I don't think that fixed it.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from this link, I see all suites passed, which flakes do you mean?
I see this failure in
so this problem should have been fixed by
From gubernator: In all cases, it's timing out. I'll try upping the startTimeout.
/reopen still flaking
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Luckily the kubelet log is clear as to what's going on. From (flake ) The test should work fine if we drop the CPU request a bit, let me tweak it to make up for the possible extra 100m scheduled. I'm not sure what's causing that (aren't these tests serial? what the heck is getting scheduled) but we should be resilient to it. Also, needs to be fixed. This test will no longer detect the regression it was designed to show after that change. I will revert it and also fix the resource requirements and that should fix this test.
/milestone v1.23 need to fix this before test freeze

Discussion:
podCount            = 100 podCreationInterval = 100 * time.Millisecond recoverTimeout      = 5 * time.Minute <del> startTimeout        = 5 * time.Minute </del> <ins> startTimeout        = 3 * time.Minute </ins> // restartCount is chosen so even with minPods we exhaust the default // allocation of a /24. minPods      = 50

if numCpus < 1 { e2eskipper.Skipf("insufficient CPU available for kubelet restart test") } <ins> if numCpus > 18 { // 950m * 19 = 1805 CPUs -> not enough to block the scheduling of another 950m pod e2eskipper.Skipf("test will return false positives on a machine with >18 cores") } </ins> // create as many restartNever pods as there are allocatable CPU // nodes; if they are not correctly accounted for as terminated

pod.Spec.RestartPolicy = "Never" pod.Spec.Containers[0].Command = []string{"echo", "hi"} pod.Spec.Containers[0].Resources.Limits = v1.ResourceList{ <del> v1.ResourceCPU: resource.MustParse("1"), </del> <ins> v1.ResourceCPU: resource.MustParse("950m"), // leave a little room for other workloads </ins> } } createBatchPodWithRateControl(f, restartNeverPods, podCreationInterval)

// restart may think these old pods are consuming CPU and we // will get an OutOfCpu error. ginkgo.By("verifying restartNever pods succeed and restartAlways pods stay running") <del> postRestartRunningPods := waitForPods(f, numAllPods, recoverTimeout) if len(postRestartRunningPods) < numAllPods { framework.Failf("less pods are running after node restart, got %d but expected %d", len(postRestartRunningPods), numAllPods) </del> <ins> for start := time.Now(); time.Since(start) < startTimeout; time.Sleep(10 * time.Second) { postRestartRunningPods := waitForPods(f, numAllPods, recoverTimeout) if len(postRestartRunningPods) < numAllPods { framework.Failf("less pods are running after node restart, got %d but expected %d", len(postRestartRunningPods), numAllPods) } </ins> } }) })