Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to create kubepods cgroup on larger machine with 512 cpus, fails with the following error: What you expected to happen: Run kubelet properly without any error How to reproduce it (as minimally and precisely as possible): Start the kubelet on a machine which has a more processors Anything else we need to know?: Environment: Kubernetes version (use ): 18.5 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
When I changed CgroupDriver to cgroupfs, kubelet is running is fine:
And here is the sample code which is creating this issue:
so this is essentially a runc/libcontainer problem or a problem with the host configuration - e.g. the host having less than the requested ? this line: hints that your snippet is for an older version of runc. rc8 renamed it to "Manager": is that the version of the kubelet too? k/k 1.18 is using rc10: so ideally the match test for kubelet 1.18.5 should be against rc10 of runc. and current master of k/k is at rc90: are you able to reproduce the same with rc90?
I tried running with master kublet which does use rc90 build and kubelet fails with the same error: complete log: I think this is happening because we are trying to create a cgroup more than a supported range for the Linux kernel: 2^18 = is the max value we can use for the CPUShare, so wondering whether can add logic into the kubelet to address this issue when we run on bigger machines(more than 256 processors)
/cc
interesting snippet in moby: ! where
i think we have a misconception here that CPU shares == # of CPU. "cpu shares" is a weighting proportion to give a container more resources in comparison to other running containers: the reason for 2^18 in the kernel is to have a larger integer scale to work with. unlike docker that has (nanocpu / fractional cpus) (): does not have such an option. but it does expose (CpusetCpus) which accepts a such as . i have a wild guess that this same problem is not present if you use dockershim. to see more on what is happening you could: run the kubelet with --v=4 or more. patch the kubelet to add debug code.
sorry for the late reply. I think the issue is not in libcontainer, but in the Kubelet asking for a out of range value for cpu shares. I am not sure how it makes sense to set the number of shares in relation to the number of available CPUs, since it is a weight and not a limit. The error you are seeing is coming straight from systemd: Since there is already a check for the minimum value, I think it makes sense to add also a check for the maximum allowed (while we figure out why cpu shares are used this way). Could you give a try to the current patch? I've nothing close to 512 cores to validate it really solves the issue you've seen :smile: I'll open a PR once we are sure it solves the problem
You are right, this is coming from the systemd library which is getting used in the libcontainer With the above patch, I see kubelet is running fine but I don't think this is just good enough because while requesting the cpus in the pod is in terms of milicores which is 1000th of 1 cpu which is getting converted into and assigned to the pod's cgroup. e.g: if the user requests a cpu of which 1024/ = ~2 cpu share out 512 which brakes the current logic in the code. I think we need to add an additional logic for allocation based on the number of cpus present in the machine. Simple math should work fine if cpus are more than 256 in the system: e.g: if the request is 1024m 1024/(()/) = 512 shares
the value doesn't really limit the cpu time a process gets. It is a weight used for the cgroups at the same level. It looks to me that the Kubelet is using this mechanism simply to maintain some sort of priorities among the different pods (i.e. if pod A has double millicpus than pod B, then A gets double weight than B). Before adding more complexity, I'd make sure why it is done this way, I think it is just wrong to put in relation to the number of cores. The cpu shares assigned to each pod though are not dependent on the number of cores, but they are converted directly from the spec file. You've reported the issue on systemd, and it happens since systemd adds another check that is not present with cgroupfs or the kernel. The kernel does the same clamping I've proposed above: so the patch is just to make sure the systemd cgroup driver behaves in the same way as cgroupfs; and I think it doesn't have any impact on the pods
I've opened a PR here:
Thanks for the PR, my only worry about - how k8s will prioritize when we have two pods requesting more than the MaxShares how this logic will prioritize it!
for now it works in the same way as in the cgroupfs case. Pod requests are not in relationship with the # of cores, that AFAICS happens only when setting up the node cgroups. I think a static allocation would work better for such cgroups, as anyway is a weight.

Related discussion:
text: ) const ( <del> // Taken from lmctfy https://github.com/google/lmctfy/blob/master/lmctfy/controllers/cpu_controller.cc MinShares     = 2 </del> <ins> // These limits are defined in the kernel: // https://github.com/torvalds/linux/blob/0bddd227f3dc55975e2b8dfa7fc6f959b062a2c7/kernel/sched/sched.h#L427-L428 MinShares = 2 MaxShares = 262144 </ins> SharesPerCPU  = 1024 MilliCPUToCPU = 1000