Change:
/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node started to fail since 09-13.
/cc
/assign
/cc
/triage accepted /priority important-soon it's indeed failing too often
/reopen The test is still failing: Now timing out at 120s i.e. 2 minutes as we increased the default timeout . We need to understand the root cause and get to the bottom of this.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Snippet from the build log () corresponding from this failure We see from the above logs that: node has two numa nodes, 8 cpus per NUMA so the root cause of the failure is not because the tests are running on a node that doesn't have multi-numa support. is started and the health check fails a couple of times but eventually succeeds. pod is created with 9 cpus (cpusperNUMA +1 i.e. 8+1) which is expected to fail with Topology Affinity Error as we cannot allocate resources for such a pod from a single NUMAnode. We fail to get matching metrics. It is not clear if the check fails because we don't get kubelet metrics at all or the expected metrics but based on point 6 below it seems like it is because of latter. We can add additional logs for more clarity here. each is executed and Kubelet is restarted and health check is done. are collected from the namespace corresponding to the test pod which shows that the pod started even though it was expected to fail with Topology Affinity Error. We need to understand why the pod got successfully created even though it requested CPUs that couldn't be allocated from a single NUMAnode.

Discussion:
ginkgo.Context("when querying /metrics", func() { var oldCfg *kubeletconfig.KubeletConfiguration var testPod *v1.Pod <del> var cpusNumPerNUMA, numaNodes int </del> <ins> var cpusNumPerNUMA, coresNumPerNUMA, numaNodes, threadsPerCore int </ins> ginkgo.BeforeEach(func(ctx context.Context) { var err error

framework.ExpectNoError(err) } <del> numaNodes, cpusNumPerNUMA = hostCheck() </del> <ins> numaNodes, coresNumPerNUMA, threadsPerCore = hostCheck() cpusNumPerNUMA = coresNumPerNUMA * threadsPerCore </ins> // It is safe to assume that the CPUs are distributed equally across // NUMA nodes and therefore number of CPUs on all NUMA nodes are same // so we just check the CPUs on the first NUMA node framework.Logf("numaNodes on the system %d", numaNodes) <ins> framework.Logf("Cores per NUMA on the system %d", coresNumPerNUMA) framework.Logf("Threads per Core on the system %d", threadsPerCore) </ins> framework.Logf("CPUs per NUMA on the system %d", cpusNumPerNUMA) policy := topologymanager.PolicySingleNumaNode

}) }) <del> func hostCheck() (int, int) { </del> <ins> func hostCheck() (int, int, int) { </ins> // this is a very rough check. We just want to rule out system that does NOT have // multi-NUMA nodes or at least 4 cores

e2eskipper.Skipf("this test is intended to be run on a system with at least %d cores per socket", minCoreCount) } <del> return numaNodes, coreCount </del> <ins> threadsPerCore := detectThreadPerCore() return numaNodes, coreCount, threadsPerCore </ins> } func checkMetricValueGreaterThan(value interface{}) types.GomegaMatcher {

return coreCount } <ins> func detectThreadPerCore() int { outData, err := exec.Command("/bin/sh", "-c", "lscpu | grep "Thread(s) per core:" | cut -d ":" -f 2").Output() framework.ExpectNoError(err) threadCount, err := strconv.Atoi(strings.TrimSpace(string(outData))) framework.ExpectNoError(err) return threadCount } </ins> func makeContainers(ctnCmd string, ctnAttributes []tmCtnAttribute) (ctns []v1.Container) { for _, ctnAttr := range ctnAttributes { ctn := v1.Container{