Change:
ci-kubernetes-node-kubelet-serial-hugepages when start the pod with the resources requests that contain multiple hugepages resources should set correct hugetlb mount and limit under the container cgroup Unexpected error: "pod ran to completion", Multiple sizes huge pages is going to be graduated to GA, so having this test job fixed is quite important.
/sig node
/assign
/triage accepted
/cc
I did a small analysis and I can see that the pod failed to start, I can not find any failure messages under the kubelet log, so it really hard to say why it failed.
were you able to reproduce it locally? btw, there is a "right way" to fix this - to fix dynamic update of a hugepage resources. This would eliminate need in a kubelet restart. There is even a PR for that: , we just need to make it working again.
after conversation with on slack I'd say we should stick to the current procedure - restarting kubelet when hugepages allocation changes. So, we should investigate why kubelet doesn't restart properly in this test case and fix it.
Nope, I failed to reproduce it locally, also under the CI, it does not appear often. The best thing will be to describe the pod in case if it failed, so we will see the reason.
under the code we have so from the log , I can see that the kubelet received the pod before the kubelet stopped, so the problem that pod dropped to Failed state because of the kubelet stop.
So some additional details, the one who responsible to restart the kubelet again, is the server service: restart the kubelet service are waiting for hugepages to be available are starting the pod server service restart the kubelet again because it was not healthy for too long - failed to start because of the kubelet restart
great findings! Should we just stop the kubelet and let the server service to start it again? We can also check if the kubelet is restarted successfully before starting the pod.
I tried it under the PR , also I unsure how can we check if the kubelet restarted successfully. All tests that I wrote before I checked if the node has NotReady state and waited until it will be Ready, but in case of the restart probably the node will always have Ready state because the time period is very short. I will continue to investigate and will check locally how can I fix it.
So the only reliable solution that I found, it just trying to start the pod number of times. I tested it 5 times and the test passed. Can you please review
Can we access kubelet service from hugepage test cases? If we can we can use its APIs like or to check if kubelet is ready after the restart. Overall I have a feeling of duplication test/e2enode/services functionality in This is a main reason for this and similar bugs I believe. It would be nice to get rid of this duplication if possible. That would nicely isolate kubelet service controls in one place and eliminate conflicts like this one. wdyt?
not really, the used mostly to start all needed components to emulate the cluster(not only the kubelet) and the test does not have access to it. I think you can separate the readiness check under the importable method, but it worth the separate PR.
I updated the PR to check the healthiness of the kubelet and looks like it works just fine.
thank you. it looks good to me so far. I'll add lgtm label when is merged and tests are all green.
the job is still flaky. can you reopen this issue?
I opened a different one - , the issue can affect any job, not only the hugepages one.
I can open a fix to W/A an issue(start the pod until it will succeed), it should fix all problems, WDYT?
i think it's ok as a workaround, until is fixed
would it be better to wait until api server reports correct resource value before running the pod?
We are waiting for it, the problem that the scheduler and the kubelet see a different picture regarding resources requested by pods.
ok. In this case restarting the pod until it succeeds seems reasonable to me to fix the flakiness until is fixed.
any progress on the workaround fix?

Discussion:
setHugepages() ginkgo.By("restarting kubelet to pick up pre-allocated hugepages") <del> restartKubelet() </del> <ins> // stop the kubelet and wait until the server will restart it automatically stopKubelet() // wait until the kubelet health check will fail gomega.Eventually(func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, time.Minute, time.Second).Should(gomega.BeFalse()) // wait until the kubelet health check will pass gomega.Eventually(func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, 2*time.Minute, 10*time.Second).Should(gomega.BeTrue()) </ins> waitForHugepages()

releaseHugepages() ginkgo.By("restarting kubelet to pick up pre-allocated hugepages") <del> restartKubelet() </del> <ins> // stop the kubelet and wait until the server will restart it automatically stopKubelet() </ins> waitForHugepages() })

import ( "context" <ins> "crypto/tls" </ins> "encoding/json" "flag" "fmt"

defaultPodResourcesPath    = "/var/lib/kubelet/pod-resources" defaultPodResourcesTimeout = 10 * time.Second defaultPodResourcesMaxSize = 1024 * 1024 * 16 // 16 Mb <ins> kubeletReadOnlyPort        = "10255" kubeletHealthCheckURL      = "http://127.0.0.1:" + kubeletReadOnlyPort + "/healthz" </ins> ) func getNodeSummary() (*stats.Summary, error) {

// stopKubelet will kill the running kubelet, and returns a func that will restart the process again func stopKubelet() func() { kubeletServiceName := findRunningKubletServiceName() <del> stdout, err := exec.Command("sudo", "systemctl", "kill", kubeletServiceName).CombinedOutput() </del> <ins> // reset the kubelet service start-limit-hit stdout, err := exec.Command("sudo", "systemctl", "reset-failed", kubeletServiceName).CombinedOutput() framework.ExpectNoError(err, "Failed to reset kubelet start-limit-hit with systemctl: %v, %v", err, stdout) stdout, err = exec.Command("sudo", "systemctl", "kill", kubeletServiceName).CombinedOutput() </ins> framework.ExpectNoError(err, "Failed to stop kubelet with systemctl: %v, %v", err, stdout) return func() { stdout, err := exec.Command("sudo", "systemctl", "start", kubeletServiceName).CombinedOutput() framework.ExpectNoError(err, "Failed to restart kubelet with systemctl: %v, %v", err, stdout) } } <ins> func kubeletHealthCheck(url string) bool { insecureTransport := http.DefaultTransport.(*http.Transport).Clone() insecureTransport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true} insecureHTTPClient := &http.Client{ Transport: insecureTransport, } req, err := http.NewRequest("HEAD", url, nil) if err != nil { return false } req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", framework.TestContext.BearerToken)) resp, err := insecureHTTPClient.Do(req) if err != nil { klog.Warningf("Health check on %q failed, error=%v", url, err) } else if resp.StatusCode != http.StatusOK { klog.Warningf("Health check on %q failed, status=%d", url, resp.StatusCode) } return err == nil && resp.StatusCode == http.StatusOK } </ins> func toCgroupFsName(cgroupName cm.CgroupName) string { if framework.TestContext.KubeletConfig.CgroupDriver == "systemd" { return cgroupName.ToSystemd()