Change:
kubelet-gce-e2e-lock-contention ! 8/12/2021 It appears that kubelet is not coming up after restart W0225 06:45:50.261] [0m[sig-node] Lock contention [Slow] [Disruptive] [NodeSpecialFeature:LockContention][0m W0225 06:45:50.261] [1mKubelet should stop when the test acquires the lock on lock file and restart once the lock is released[0m W0225 06:45:50.262] [[0m W0225 06:45:50.262] [It] Kubelet should stop when the test acquires the lock on lock file and restart once the lock is released W0225 06:45:50.262] W0225 06:45:50.262] [1mSTEP[0m: perform kubelet health check to check if kubelet is healthy and running. W0225 06:45:50.262] [1mSTEP[0m: acquiring the lock on lock file i.e W0225 06:45:50.262] [1mSTEP[0m: verifying the kubelet is not healthy as there was a lock contention. W0225 06:45:50.263] W0225 06:45:40. 2231 ] Health check on "http://127.0.0.1:/healthz" failed, error=Head "http://127.0.0.1:/healthz": dial tcp 127.0.0.1:: connect: connection refused W0225 06:45:50.263] [1mSTEP[0m: releasing the lock on lock file i.e , triggering kubelet restart. W0225 06:45:50.263] [1mSTEP[0m: verifying the kubelet is healthy after restart. W0225 06:45:50.263] W0225 06:45:40. 2231 ] Health check on "http://127.0.0.1:/healthz" failed, error=Head "http://127.0.0.1:/healthz": dial tcp 127.0.0.1:: connect: connection refused W0225 06:45:50.263] W0225 06:45:41. 2231 ] Health check on "http://127.0.0.1:/healthz" failed, error=Head "http://127.0.0.1:/healthz": dial tcp 127.0.0.1:: connect: connection refused W0225 06:45:50.263] W0225 06:45:42. 2231 ] Health check on "http://127.0.0.1:/healthz" failed, error=Head "http://127.0.0.1:/healthz": dial tcp 127.0.0.1:: connect: connection refused W0225 06:45:50.264] W0225 06:45:43. 2231 ] Health check on "http://127.0.0.1:/healthz" failed, error=Head "http://127.0.0.1:/healthz": dial tcp 127.0.0.1:: connect: connection refused W0225 06:45:50.264] W0225 06:45:44. 2231 ] Health check on "http://127.0.0.1:/healthz" failed, error=Head "http://127.0.0.1:/healthz": dial tcp 127.0.0.1:: connect: connection refused the test is recently migrated to containerd /sig node
/assign any progress on this?
/triage accepted
I think the problem is this The lock contention arguments should have been part of but instead they were mistakenly to (my bad) I think this would have been caught if the job was merged first and then the e2e test, but I assume the reverse order happened hence it didn't get caught. Anyhow, to test the above with the correct params, how can I do it ? so that we know thats the reason and nothing else. /cc
its a kubelet flag, they should be part of only, see above job Although should be single quoted as done everywhere else, possibly it could be a problem? Create a PR with correct in k/test-infra we need to merge and keep an eye on test grid, testing the prow job configuration locally is bit tedious
Thanks for the insight. Yep it seems the is the correct place. I've created a PR to add correct args. PTAL.
Thanks for the PR. After the PR submit, the test is still failing somehow with
Thanks, I know about that, hence
The test passes https://k8s-

Discussion:
ginkgo.By("verifying the kubelet is not healthy as there was a lock contention.") // Once the lock is acquired, check if the kubelet is in healthy state or not. <del> // It should not be. </del> <ins> // It should not be as the lock contention forces the kubelet to stop. </ins> gomega.Eventually(func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, 10*time.Second, time.Second).Should(gomega.BeFalse()) <del> ginkgo.By("releasing the lock on lock file i.e /var/run/kubelet.lock, triggering kubelet restart.") // Release the lock. err = unix.Flock(fd, unix.LOCK_UN) framework.ExpectNoError(err) // Releasing the lock triggers kubelet to re-acquire the lock and restart. ginkgo.By("verifying the kubelet is healthy after restart.") // Kubelet should report healthy state. gomega.Eventually(func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, 10*time.Second, time.Second).Should(gomega.BeTrue()) </del> }) })