Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: On a node that has a topology of more than 1 numa node, If some or all of the CPUs of a numa node are turned off (mostly for testing purposes) a valid set can cause the Kubelet service to fail on: This failure has occurred on a node that has 2 numa nodes with hyperthreading enabled. What you expected to happen: When setting a valid set, the Kubelet service should pass validations and function properly even if some CPU cores are turned off but the reserved cpus are still under a valid range. How to reproduce it (as minimally and precisely as possible): minimal cluster should have 1 master and 1 worker node. worker node should have 2 Numa nodes and hyperthreading enabled(BIOS setting) Important: The CPUs divided between the numa nodes cannot be sequential in order to reproduce the issue that are under Numa node 0 only (under ) Kubelet service off all CPUs that belong to numa node 1 () Kubelet service systemctl status kubelet.service and journal logs that will present: Anything else we need to know?: The scenario of split ranges of Numa nodes might occur only with Hyperthreading enabled. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ): (real time kernel) Install tools:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a real example where the error occurred: For the following CPU layout (after turning off numa node 1 CPUs) using : will result in: CPUs 0-2 and 40-42 are all under numa node 0 and are active. The validation fails here: In this scenario we have 40 active CPUs (all from numa node 0) but the range check will fail on the last item in reserved CPUs: 42 (last time) = 40 (NumCores)
/cc
/sig node
This is probably not just strictly related to NUMA. The same behaviour could happen when offlining CPUs on any multi-core system, for example: disabling HT on a single numa node may end up with a very similar scenario (depends on the CPU specifics). The kubelet validation is too naive; it needs to be aware (and act accordingly to) the online CPU set. The emerging hidden assumption in kubelet is all the physical cpus are always online.
I wasn't able to recreate with all cpus on numa 1 offline, On-line CPU(s) list: 0-11,24-35 Off-line CPU(s) list: 12-23,36-47 with HT enabled, --reserved-cpus=0-3, kubelet restart fine but as mentioned by this option kubelet validation didn't consider the situation where some cpus are offline.
Ok I was able to reproduce when I used --reserved-cpus=28-35.
Just a FYI, even with the validation update, if you take certain CPUs offline, you need to delete /var/lib/kubelet/cpumanagerstate before restart kubelet. This state check is done by cpumanager, see
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale

Discussion:
"k8s.io/klog/v2" "k8s.io/mount-utils" <ins> cadvisorapi "github.com/google/cadvisor/info/v1" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

kubeletcertificate "k8s.io/kubernetes/pkg/kubelet/certificate" "k8s.io/kubernetes/pkg/kubelet/certificate/bootstrap" "k8s.io/kubernetes/pkg/kubelet/cm" <ins> "k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/topology" </ins> "k8s.io/kubernetes/pkg/kubelet/cm/cpuset" "k8s.io/kubernetes/pkg/kubelet/config" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"

} } <ins> func getReservedCPUs(machineInfo *cadvisorapi.MachineInfo, cpus string) (cpuset.CPUSet, error) { emptyCPUSet := cpuset.NewCPUSet() if cpus == "" { return emptyCPUSet, nil } topo, err := topology.Discover(machineInfo) if err != nil { return emptyCPUSet, fmt.Errorf("Unable to discover CPU topology info: %s", err) } reservedCPUSet, err := cpuset.Parse(cpus) if err != nil { return emptyCPUSet, fmt.Errorf("Unable to parse reserved-cpus list: %s", err) } allCPUSet := topo.CPUDetails.CPUs() if !reservedCPUSet.IsSubsetOf(allCPUSet) { return emptyCPUSet, fmt.Errorf("reserved-cpus: %s is not a subset of online-cpus: %s", cpus, allCPUSet.String()) } return reservedCPUSet, nil } </ins> func run(ctx context.Context, s *options.KubeletServer, kubeDeps *kubelet.Dependencies, featureGate featuregate.FeatureGate) (err error) { // Set global feature gates based on the value on the initial KubeletServer err = utilfeature.DefaultMutableFeatureGate.SetFromMap(s.KubeletConfiguration.FeatureGates)

s.CgroupRoot = "/" } <del> var reservedSystemCPUs cpuset.CPUSet if s.ReservedSystemCPUs != "" { // is it safe do use CAdvisor here ?? machineInfo, err := kubeDeps.CAdvisorInterface.MachineInfo() if err != nil { // if can't use CAdvisor here, fall back to non-explicit cpu list behavior klog.InfoS("Failed to get MachineInfo, set reservedSystemCPUs to empty") reservedSystemCPUs = cpuset.NewCPUSet() } else { var errParse error reservedSystemCPUs, errParse = cpuset.Parse(s.ReservedSystemCPUs) if errParse != nil { // invalid cpu list is provided, set reservedSystemCPUs to empty, so it won't overwrite kubeReserved/systemReserved klog.InfoS("Invalid ReservedSystemCPUs", "systemReservedCPUs", s.ReservedSystemCPUs) return errParse } reservedList := reservedSystemCPUs.ToSlice() first := reservedList[0] last := reservedList[len(reservedList)-1] if first < 0 || last >= machineInfo.NumCores { // the specified cpuset is outside of the range of what the machine has klog.InfoS("Invalid cpuset specified by --reserved-cpus") return fmt.Errorf("Invalid cpuset %q specified by --reserved-cpus", s.ReservedSystemCPUs) } } } else { reservedSystemCPUs = cpuset.NewCPUSet() </del> <ins> machineInfo, err := kubeDeps.CAdvisorInterface.MachineInfo() if err != nil { return err } reservedSystemCPUs, err := getReservedCPUs(machineInfo, s.ReservedSystemCPUs) if err != nil { return err </ins> } if reservedSystemCPUs.Size() > 0 { // at cmd option validation phase it is tested either --system-reserved-cgroup or --kube-reserved-cgroup is specified, so overwrite should be ok klog.InfoS("Option --reserved-cpus is specified, it will overwrite the cpu setting in KubeReserved and SystemReserved", "kubeReservedCPUs", s.KubeReserved, "systemReservedCPUs", s.SystemReserved)