Change:
What happened: I think I'm seeing a problem with the topology manager. I have two numa nodes, 22 cores per node. Kubelet has "--cpu-manager-policy=static --topology-manager-policy=single-numa-node ", and when I try to create a simple pod with 3 cpus and 500Mi of RAM in the guaranteed class it fails. Describing the pod gives "Resources cannot be allocated with Topology Locality", but I know there's lots of CPU and RAM available on both NUMA nodes. I'm instrumenting the topology manager kubelet code, currently there's not enough debug logs to actually be able to tell what's going wrong. What you expected to happen: I expect the pod container to be created on one or the other numa node and get allocated exclusive CPUs (via the static policy) on that NUMA node, and be affined to those CPUs. How to reproduce it (as minimally and precisely as possible): On a kubernetes node with multiple NUMA nodes, enable kubelet with static cpu manager policy and single-numa-node topology manager policy. Try to create a basic pod with a single container in the guaranteed QoS class. Anything else we need to know?: Environment: Kubernetes version (use ): sysadmin chris(keystoneadmin)]$ kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"archive", BuildDate:"2019-10-09T05:41:44Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: wildcat pass hardware, 22 cores per NUMA node, two NUMA nodes. 64GB RAM per NUMA node. OS (e.g: ): CentOS 7 Kernel (e.g. ): 3.10.0-
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
oops, I thought the "close" meant the comment, not the whole issue. :)
I was able to reproduce this issue. The following (new) uint test in fails unexpectedly: With any policy other than the this test passes. This is due to 3 factors: 1) The CPUManager is the only hint provider passing actual valid hints 2) The devicemanager passes back as its hints, resulting in a single (i.e. don’t care) hint 3) We a special casing for the in the device manager that essentially turns the devicemanager's hint into a : Because the single hint from the devicemanager now has a preference, there is no chance to have it combine with any of the CPUManager hints to result in a successful hint. This should ultimately be addressed as part of the following in 1.17 But we should put a one-off patch into the 1.16.x branch to fix this in 1.16.
ptal
I've just updated with my observations. can you have a look and comment ?

Discussion:
// If hints is nil, insert a single, preferred any-numa hint into allProviderHints. if len(hints) == 0 { klog.Infof("[topologymanager] Hint Provider has no preference for NUMA affinity with any resource") <del> allProviderHints = append(allProviderHints, []TopologyHint{{defaultAffinity, true}}) </del> <ins> allProviderHints = append(allProviderHints, []TopologyHint{{nil, true}}) </ins> continue }

for resource := range hints { if hints[resource] == nil { klog.Infof("[topologymanager] Hint Provider has no preference for NUMA affinity with resource '%s'", resource) <del> allProviderHints = append(allProviderHints, []TopologyHint{{defaultAffinity, true}}) </del> <ins> allProviderHints = append(allProviderHints, []TopologyHint{{nil, true}}) </ins> continue } if len(hints[resource]) == 0 { klog.Infof("[topologymanager] Hint Provider has no possible NUMA affinities for resource '%s'", resource) <del> allProviderHints = append(allProviderHints, []TopologyHint{{defaultAffinity, false}}) </del> <ins> allProviderHints = append(allProviderHints, []TopologyHint{{nil, false}}) </ins> continue }

preferred := true var numaAffinities []bitmask.BitMask for _, hint := range permutation { <del> // Only consider hints that have an actual NUMANodeAffinity set. if hint.NUMANodeAffinity != nil { if !hint.Preferred { preferred = false } // Special case PolicySingleNumaNode to only prefer hints where // all providers have a single NUMA affinity set. if m.policy != nil && m.policy.Name() == PolicySingleNumaNode && hint.NUMANodeAffinity.Count() > 1 { preferred = false } </del> <ins> if hint.NUMANodeAffinity == nil { numaAffinities = append(numaAffinities, defaultAffinity) } else { </ins> numaAffinities = append(numaAffinities, hint.NUMANodeAffinity) } <ins> if !hint.Preferred { preferred = false } // Special case PolicySingleNumaNode to only prefer hints where // all providers have a single NUMA affinity set. if m.policy != nil && m.policy.Name() == PolicySingleNumaNode && hint.NUMANodeAffinity != nil && hint.NUMANodeAffinity.Count() > 1 { preferred = false } </ins> } // Merge the affinities using a bitwise-and operation.

}, expected: TopologyHint{ NUMANodeAffinity: NewTestBitMask(numaNodes...), <del> Preferred:        true, </del> <ins> Preferred:        false, </ins> }, }, {

Preferred:        false, }, }, <ins> { name:   "Special cased PolicySingleNumaNode with one no-preference provider", policy: NewSingleNumaNodePolicy(), hp: []HintProvider{ &mockHintProvider{ map[string][]TopologyHint{ "resource1": { { NUMANodeAffinity: NewTestBitMask(0), Preferred:        true, }, { NUMANodeAffinity: NewTestBitMask(1), Preferred:        true, }, { NUMANodeAffinity: NewTestBitMask(0, 1), Preferred:        false, }, }, }, }, &mockHintProvider{ nil, }, }, expected: TopologyHint{ NUMANodeAffinity: NewTestBitMask(0), Preferred:        true, }, }, </ins> } for _, tc := range tcases {