Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The pod passed the scheduler(moved from the Pending phase) and failed to start because of the kubelet with the error: It means that the scheduler and kubelet get a different view of requested resources. I expect that if the scheduler passed the pod and the pod was started it should not fail under the kubelet because of insufficient resources. It failed under the CI - Steps: configure hugepages under the node(5 - 2Mi-hugepages) start the pod, that requests 6Mi of 2Mi-hugepages delete the namespace where the pod started restart the kubelet start an additional pod, that requests 6Mi of 2Mi-hugepages It happens very rarely, something around 10%. The problem that the scheduler and kubelet are using different mechanisms and structs to store active pods requests. Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am very interested in this question, I will try to analyze it.
I think some delay between the time when kubelet discovers the resource and the time when resource value is updated on the API server is expected. We can't guarantee that kubelet sends updated resource to the API server, API server receives the update and updates resource value in no time, can we?
the scheduler should see the same picture as the kubelet, so why does it pass the scheduler and failed under the kubelet? Do you know why do we need the same checks under two different places(kubelet and scheduler), why it not enough to check available resources only under the scheduler?
/sig node scheduling
are there any static pods being created in that test? static pods may cause this.
Nope, regular pods with the
/cc
I can not see the issue happens under the 1.21, I will re-open the issue if will encounter it again.
I can still see it in the recent logs. Here is an example from
Feel free to re-open it. I just checked that the hugepages job is much more stable than it was under 1.20.
I can't reopen it it seems. Can you?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue became a showstopper for promoting . Can you help to fix it?
I've noticed a lot of similar error messages in the kubelet log of successfull tests. Here is an example: taken from the latest available test run: The fact that node capacity for the requested hugepages is 0 for 3 pods in a row makes me suspect that the test creates pods requesting hugepages that are not allocated or something similar. Can you help to investigate this?

Discussion:
pod := getHugepagesTestPod(f, limits, mounts, volumes) <del> ginkgo.By("by running a guarantee pod that requests hugepages") </del> <ins> ginkgo.By("by running a test pod that requests hugepages") </ins> testpod = f.PodClient().CreateSync(pod) }) // we should use JustAfterEach because framework will teardown the client under the AfterEach method ginkgo.JustAfterEach(func() { <ins> ginkgo.By(fmt.Sprintf("deleting test pod %s", testpod.Name)) f.PodClient().DeleteSync(testpod.Name, metav1.DeleteOptions{}, 2*time.Minute) </ins> releaseHugepages() ginkgo.By("restarting kubelet to pick up pre-allocated hugepages")