Change:
/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node started to fail since 09-13.
/cc
/assign
/cc
/triage accepted /priority important-soon it's indeed failing too often
/reopen The test is still failing: Now timing out at 120s i.e. 2 minutes as we increased the default timeout . We need to understand the root cause and get to the bottom of this.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Snippet from the build log () corresponding from this failure We see from the above logs that: node has two numa nodes, 8 cpus per NUMA so the root cause of the failure is not because the tests are running on a node that doesn't have multi-numa support. is started and the health check fails a couple of times but eventually succeeds. pod is created with 9 cpus (cpusperNUMA +1 i.e. 8+1) which is expected to fail with Topology Affinity Error as we cannot allocate resources for such a pod from a single NUMAnode. We fail to get matching metrics. It is not clear if the check fails because we don't get kubelet metrics at all or the expected metrics but based on point 6 below it seems like it is because of latter. We can add additional logs for more clarity here. each is executed and Kubelet is restarted and health check is done. are collected from the namespace corresponding to the test pod which shows that the pod started even though it was expected to fail with Topology Affinity Error. We need to understand why the pod got successfully created even though it requested CPUs that couldn't be allocated from a single NUMAnode.

Discussion:
}) ginkgo.By("Giving the Kubelet time to start up and produce metrics") <del> gomega.Eventually(ctx, getKubeletMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) </del> <ins> gomega.Eventually(ctx, getKubeletMetrics, 2*time.Minute, 10*time.Second).Should(matchResourceMetrics) </ins> ginkgo.By("Ensuring the metrics match the expectations a few more times") <del> gomega.Consistently(ctx, getKubeletMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) </del> <ins> gomega.Consistently(ctx, getKubeletMetrics, 2*time.Minute, 10*time.Second).Should(matchResourceMetrics) </ins> }) ginkgo.It("should report admission failures when the topology manager alignment is known to fail", func(ctx context.Context) {

}) ginkgo.By("Giving the Kubelet time to start up and produce metrics") <del> gomega.Eventually(ctx, getKubeletMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) </del> <ins> gomega.Eventually(ctx, getKubeletMetrics, 2*time.Minute, 10*time.Second).Should(matchResourceMetrics) </ins> ginkgo.By("Ensuring the metrics match the expectations a few more times") <del> gomega.Consistently(ctx, getKubeletMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) </del> <ins> gomega.Consistently(ctx, getKubeletMetrics, 2*time.Minute, 10*time.Second).Should(matchResourceMetrics) </ins> }) ginkgo.It("should not report any admission failures when the topology manager alignment is expected to succeed", func(ctx context.Context) {

}) ginkgo.By("Giving the Kubelet time to start up and produce metrics") <del> gomega.Eventually(ctx, getKubeletMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) </del> <ins> gomega.Eventually(ctx, getKubeletMetrics, 2*time.Minute, 10*time.Second).Should(matchResourceMetrics) </ins> ginkgo.By("Ensuring the metrics match the expectations a few more times") <del> gomega.Consistently(ctx, getKubeletMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) </del> <ins> gomega.Consistently(ctx, getKubeletMetrics, 2*time.Minute, 10*time.Second).Should(matchResourceMetrics) </ins> }) }) })