Change:
kubeletcertificate "k8s.io/kubernetes/pkg/kubelet/certificate" "k8s.io/kubernetes/pkg/kubelet/certificate/bootstrap" "k8s.io/kubernetes/pkg/kubelet/cm" <ins> "k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/topology" </ins> "k8s.io/kubernetes/pkg/kubelet/cm/cpuset" "k8s.io/kubernetes/pkg/kubelet/config" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: On a node that has a topology of more than 1 numa node, If some or all of the CPUs of a numa node are turned off (mostly for testing purposes) a valid set can cause the Kubelet service to fail on: This failure has occurred on a node that has 2 numa nodes with hyperthreading enabled. What you expected to happen: When setting a valid set, the Kubelet service should pass validations and function properly even if some CPU cores are turned off but the reserved cpus are still under a valid range. How to reproduce it (as minimally and precisely as possible): minimal cluster should have 1 master and 1 worker node. worker node should have 2 Numa nodes and hyperthreading enabled(BIOS setting) Important: The CPUs divided between the numa nodes cannot be sequential in order to reproduce the issue that are under Numa node 0 only (under ) Kubelet service off all CPUs that belong to numa node 1 () Kubelet service systemctl status kubelet.service and journal logs that will present: Anything else we need to know?: The scenario of split ranges of Numa nodes might occur only with Hyperthreading enabled. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ): (real time kernel) Install tools:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a real example where the error occurred: For the following CPU layout (after turning off numa node 1 CPUs) using : will result in: CPUs 0-2 and 40-42 are all under numa node 0 and are active.

The validation fails here: In this scenario we have 40 active CPUs (all from numa node 0) but the range check will fail on the last item in reserved CPUs: 42 (last time) = 40 (NumCores)
/cc
/sig node
This is probably not just strictly related to NUMA. The same behaviour could happen when offlining CPUs on any multi-core system, for example: disabling HT on a single numa node may end up with a very similar scenario (depends on the CPU specifics). The kubelet validation is too naive; it needs to be aware (and act accordingly to) the online CPU set. The emerging hidden assumption in kubelet is all the physical cpus are always online.
I wasn't able to recreate with all cpus on numa 1 offline, On-line CPU(s) list: 0-11,24-35 Off-line CPU(s) list: 12-23,36-47 with HT enabled, --reserved-cpus=0-3, kubelet restart fine but as mentioned by this option kubelet validation didn't consider the situation where some cpus are offline.
Ok I was able to reproduce when I used --reserved-cpus=28-35.
Just a FYI, even with the validation update, if you take certain CPUs offline, you need to delete /var/lib/kubelet/cpumanagerstate before restart kubelet. This state check is done by cpumanager, see
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale