Change:
Dual socket server with 96 threads total (2242), ~192G of RAM, cpu & memory manager static policy, topologyManagerPolicy best-effort, 10Gi of RAM reserved on NUMA node 0, 1 core (2 threads) reserved on NUMA node 0 kubeadm config: If I try to allocate 2 Guaranteed pod with 85Gi of RAM each, one pod is admitted, and the second one fails with even if it would fit using memory of both NUMA nodes. As I'm using deployments a new pod is recreated right away and you end up with tons of failed pod in state. Error in the logs is: Either the pod becomes pending, or memory from both NUMA nodes is used Have some memory reserved on NUMA 0, launch 2 identical pods with memory limits really close to the max so 1 pod fit on NUMA 1 but the second doesn't fit on NUMA 0 (the sum fits on the server) Here the If I launch 1 pod with 46 CPUs / 170Gi RAM, I get CPUs from 1 NUMA nodes, and memory from both NUMA nodes <details</detailsIf I launch a pod with 46 CPUs / 85Gi RAM, it's properly placed on NUMA 1 <details</detailsIf I try to launch 2 pods with 46 CPUs / 85Gi RAM, one fails with If I try to launch 2 pods with 46 CPUs / 80Gi RAM, everything works 26.8 NONE / bare-metal Alma 8.8 base + rpm-ostree kubeadm containerd 1.7.5 No response
/sig node
/cc Hi! Just to clarify, the memory manager can't keep the pod pending, this is a scheduler feature and builtin kube scheduler (= any of its plugin) is not NUMA-aware atm. So in this case memory manager should extend the pod to both NUMA zones, and failing to do so is likely the issue
Hello Is it possible that you are hitting the exact scenario describe at the KEP: ? Please check the first two steps.
I was going to say I'm hitting the expected unexpected error, but I'm not sure the KEP doc 100% match the current memoryManager behavior (at least when using ). test 1: the "reject pod2 from the KEP" actually works: start pod 60 cpus / 120Gi RAM wait start pod 20 cpus / 20Gi RAM <details</detailstest 2 (start a 1 NUMA node container then one than spans 2 NUMA node) also works: start pod 20 cpus / 20Gi RAM wait start pod 60 cpus / 120Gi RAM <details</detailsWhat doesn't work is when you get too close to the limits it seems
Interesting those pods are guaranteed I presume? And what about the total node memory capacity? When you check the node status do you have enough memory left to fulfill the second pod request?
click on details you will have the content of yes they are guaranteed else they would not show up in memorymanagerstate it's less then the total node capacity else pod would be pending
/assign
assigned to me for bookkeeping reason but is doing the real work here
Hello so far I didn't manage to reproduce the issue you're describing. In my case I manage to ran both pods, or end up with the second one . However, I managed to find something similar (different affinity error) but with topology manager policy and not as you described which I'm going to open a separate issue for. Could you please share the journal logs while you're reproducing the issue again? Basically on the node where the pod is about to land, share the output
One more thing please share output of the node before and after you reproduce the issue
/triage accepted /priority important-longterm
off until Tuesday, I'll try to reproduce with 1.28.2 then and share the logs next week
NP. I won't be here next week either, so no rush
Hello did you manage to reproduce the issue?
yes, reproduced on my dev machine in 1.28.2, tell me what else you need / if you want me to rebuild kubelet with some debug that's also possible <details</details
Hi I did mange to reproduce the issue, but following the code it seems exactly like I mentioned in my previous I'll try to elaborate: When the first pod gets admitted it placed on NUMA 0 (in my case). When the second pod gets admitted, there is not enough memory on each NUMA individually (NUMA 0 or NUMA1) to satisfy the memory request, so memory manager tries to span the memory allocation through NUMA 0 and NUMA 1. But it failed short in this line: because NUMA node 0 is now used for single-NUMA memory allocation which align with the KEP . In such case, the pod cannot move to because is done at the scheduler level, while from the scheduler's POV there is enough memory on the node to satisfy the request. Now, about your comment here: wait start pod 20 cpus / 20Gi RAM Make sense because both NUMA 0 and NUMA 1 are already serves mulit-NUMA allocation, since the first pod spans across the two NUMAs, the second pod can squeezed into one of them with no problems. wait start pod 60 cpus / 120Gi RAM This case is more interesting and I didn't manage to reproduce it - I always got which is align with the KEP. Could you please increase Kubelet logs verbosity, reproduce test 2 and attached the following information: logs file before 1 manifest file after pod1 2 manifest file after pod2 Thanks
Hi this is a different server with a bit less CPUs, but test 2 still works for me for verbosity I've used v=99 instead of v=2 Here part of my KubeletConfiguration The logs: And the commands: <details</details
I think I pinpointed a bit more the issue For my test2 pod the cpu manager generated a hint with both NUMA nodes. For test2 pod, if CPU request span both NUMA nodes, then the memory allocation will work whether it requires to use 1 or 2 NUMA nodes If CPU request only requires 1 NUMA node, then it'll fail if memory allocation need to extend to 2 NUMA node.
Hello , After carefully going through the code and the KEP I think that the behavior you're expecting: "Either the pod becomes pending, or memory from both NUMA nodes is used" is not aligned with the architecture. Let's break it down: "The pod becomes pending" - From the default scheduler POV, the node have enough resources (because the scheduler is not NUMA aware) so the fact that the pod scheduled on this node is align with the expected behavior. Sidenote: if you want to enable NUMA aware scheduling you can take a look at "memory from both NUMA nodes is used" - Because NUMA 0 is already serves as single NUMA memory allocation, it cannot serves as multi NUMA memory allocation as well, otherwise the memory pinning is not guaranteed since linux kernel doesn't support multi-NUMA allocation (pinning). What it can do is grouping NUMA 0 with NUMA 1 into a single group but again this is against the original design: Please check and look for "No spare capacity, reject Pod 7" (last example) And you were right, when the hint generated by cpu manager, spans for both NUMAs the pod gets admitted, and that's exactly where actual bug is - the pod should never get admitted. IOW you were suppose get the error: So the fix here should be preventing the pod admission even when the CPU manager hint spans across both NUMAs. Now, in your case, if you still want and need both pods to get admitted you can change the from to because as I mentioned, the memory pinning in your scenario is not guaranteed anyway.
Hello I still think the bug is that memoryManager doesn't respect topologyManagerPolicy: best-effort. For now I'm exploring using a simple NRI plugin to set the to the nodes of
, The best-effort means that even if the affinity is not preferred, the pod would get admit anyway. But here we're talking about respecting the NUMA node grouping done by memory manager. This is an architectural design we should refer to the original creator memory manager or at least to topology maintainers folks. do you know who can answer on such questions?
some additional info about the limitation i mentioned above

Discussion:
bestHint = extendedHint } <ins> // the best hint might violate the NUMA allocation rule on which // NUMA node cannot have both single and cross NUMA node allocations // https://kubernetes.io/blog/2021/08/11/kubernetes-1-22-feature-memory-manager-moves-to-beta/#single-vs-cross-numa-node-allocation if isAffinityViolatingNUMAAllocations(machineState, bestHint.NUMANodeAffinity) { return fmt.Errorf("[memorymanager] preferred hint violates NUMA node allocation") } </ins> var containerBlocks []state.Block maskBits := bestHint.NUMANodeAffinity.GetBits() for resourceName, requestedSize := range requestedResources {

return bitMask1.IsEqual(bitMask2) } <ins> func isAffinityViolatingNUMAAllocations(machineState state.NUMANodeMap, mask bitmask.BitMask) bool { maskBits := mask.GetBits() singleNUMAHint := len(maskBits) == 1 for _, nodeID := range mask.GetBits() { // the node was never used for the memory allocation if machineState[nodeID].NumberOfAssignments == 0 { continue } if singleNUMAHint { continue } // the node used for the single NUMA memory allocation, it cannot be used for the multi NUMA node allocation if len(machineState[nodeID].Cells) == 1 { return true } // the node already used with a different group of nodes, it cannot be used within the current hint if !areGroupsEqual(machineState[nodeID].Cells, maskBits) { return true } } return false } </ins>

pod:          getPod("pod1", "container1", requirementsGuaranteed), topologyHint: &topologymanager.TopologyHint{Preferred: true}, }, <ins> { description: "should validate NUMA node can not have both single and cross NUMA node memory allocations", assignments: state.ContainerMemoryAssignments{ "pod1": map[string][]state.Block{ "container1": { { NUMAAffinity: []int{0}, Type:         v1.ResourceMemory, Size:         1024 * mb, }, }, }, }, expectedAssignments: state.ContainerMemoryAssignments{ "pod1": map[string][]state.Block{ "container1": { { NUMAAffinity: []int{0}, Type:         v1.ResourceMemory, Size:         1024 * mb, }, }, }, }, machineState: state.NUMANodeMap{ 0: &state.NUMANodeState{ MemoryMap: map[v1.ResourceName]*state.MemoryTable{ v1.ResourceMemory: { Allocatable:    1536 * mb, Free:           512 * mb, Reserved:       1024 * mb, SystemReserved: 512 * mb, TotalMemSize:   2176 * mb, }, hugepages1Gi: { Allocatable:    gb, Free:           gb, Reserved:       0, SystemReserved: 0, TotalMemSize:   gb, }, }, Cells:               []int{0}, NumberOfAssignments: 1, }, 1: &state.NUMANodeState{ MemoryMap: map[v1.ResourceName]*state.MemoryTable{ v1.ResourceMemory: { Allocatable:    512 * mb, Free:           512 * mb, Reserved:       0, SystemReserved: 512 * mb, TotalMemSize:   2176 * mb, }, hugepages1Gi: { Allocatable:    gb, Free:           gb, Reserved:       0, SystemReserved: 0, TotalMemSize:   gb, }, }, Cells:               []int{1}, NumberOfAssignments: 0, }, }, expectedMachineState: state.NUMANodeMap{ 0: &state.NUMANodeState{ MemoryMap: map[v1.ResourceName]*state.MemoryTable{ v1.ResourceMemory: { Allocatable:    1536 * mb, Free:           512 * mb, Reserved:       1024 * mb, SystemReserved: 512 * mb, TotalMemSize:   2176 * mb, }, hugepages1Gi: { Allocatable:    gb, Free:           gb, Reserved:       0, SystemReserved: 0, TotalMemSize:   gb, }, }, Cells:               []int{0}, NumberOfAssignments: 1, }, 1: &state.NUMANodeState{ MemoryMap: map[v1.ResourceName]*state.MemoryTable{ v1.ResourceMemory: { Allocatable:    512 * mb, Free:           512 * mb, Reserved:       0, SystemReserved: 512 * mb, TotalMemSize:   2176 * mb, }, hugepages1Gi: { Allocatable:    gb, Free:           gb, Reserved:       0, SystemReserved: 0, TotalMemSize:   gb, }, }, Cells:               []int{1}, NumberOfAssignments: 0, }, }, systemReserved: systemReservedMemory{ 0: map[v1.ResourceName]uint64{ v1.ResourceMemory: 512 * mb, }, 1: map[v1.ResourceName]uint64{ v1.ResourceMemory: 512 * mb, }, }, pod:           getPod("pod2", "container1", requirementsGuaranteed), topologyHint:  &topologymanager.TopologyHint{NUMANodeAffinity: newNUMAAffinity(0, 1), Preferred: true}, expectedError: fmt.Errorf("[memorymanager] preferred hint violates NUMA node allocation"), }, </ins> } for _, testCase := range testCases {

}) } } <ins> func Test_isAffinityViolatingNUMAAllocations(t *testing.T) { testsCases := []struct { description         string machineState        map[int]*state.NUMANodeState topologyHint        *topologymanager.TopologyHint isViolationExpected bool }{ { description: "violating NUMA allocations because given affinity asks for NUMA ID 1 which is on different cells group", machineState: map[int]*state.NUMANodeState{ 0: { NumberOfAssignments: 1, Cells:               []int{0, 1}, }, 1: { NumberOfAssignments: 1, Cells:               []int{0, 1}, }, 2: { NumberOfAssignments: 1, Cells:               []int{2}, }, 3: { NumberOfAssignments: 0, Cells:               []int{3}, }, }, topologyHint: &topologymanager.TopologyHint{ NUMANodeAffinity: newNUMAAffinity(1, 2), }, isViolationExpected: true, }, { description: "violating NUMA allocations because given affinity with multiple nodes asks for NUMA ID 1 which is used for a single NUMA node memory allocation", machineState: map[int]*state.NUMANodeState{ 0: { NumberOfAssignments: 0, Cells:               []int{0, 1}, }, 1: { NumberOfAssignments: 1, Cells:               []int{1}, }, }, topologyHint: &topologymanager.TopologyHint{ NUMANodeAffinity: newNUMAAffinity(0, 1), }, isViolationExpected: true, }, { description: "valid affinity, no prior assignments", machineState: map[int]*state.NUMANodeState{ 0: { NumberOfAssignments: 0, Cells:               []int{0}, }, 1: { NumberOfAssignments: 0, Cells:               []int{1}, }, }, topologyHint: &topologymanager.TopologyHint{ NUMANodeAffinity: newNUMAAffinity(0, 1), }, isViolationExpected: false, }, } for _, tc := range testsCases { t.Run(tc.description, func(t *testing.T) { if isAffinityViolatingNUMAAllocations(tc.machineState, tc.topologyHint.NUMANodeAffinity) != tc.isViolationExpected { t.Errorf("isAffinityViolatingNUMAAllocations with affinity %v expected to return %t, got %t", tc.topologyHint.NUMANodeAffinity.GetBits(), tc.isViolationExpected, !tc.isViolationExpected) } }) } } </ins>