Change:
Deployment of a POD onto a machine with 488 cores fails if the POD gets deployed with QoS Guaranteed or Burstable when cpu request is larger than 256. Kubelet Error Trace: RunContainerError: failed to start container "4ad8bf2f988c9c387d8136ac664caf4af80648ac436e92dac6ecd1f9aece9144": Error response from daemon: OCI runtime create failed: starting container process caused " container init caused " setting cgroup config for procHooks process caused "The maximum allowed cpu-shares is """: unknown Normal start of the container with the right number of cpu-shares set yo the cgroup the POD runs in. Install kubernetes on a machine with more than 300 cores and deploy a POD with QoS Guaranteed or Burstable where the cpu-request value is set to more than 256 cpus. Deployment without CPU request and limits works. Deployments with cpu-request <= 256 works as well. I found this change: Unfortunately the changed code does not get used in all cases - is not used in my case. Kubernetes version (use ): Kubernetes v1.19.8 Cloud provider or hardware configuration: AWS / EC2 u- (6 TB memory / 488 vCPUS) OS (e.g: ): NAME="SLES" VERSION="15-SP1" VERSIONID="15.1" PRETTYNAME="SUSE Linux Enterprise Server 15 SP1" ID="sles" IDLIKE="suse" ANSICOLOR="0;32" CPENAME="cpe:/o:suse:sles:15:sp1" VARIANT_ID="chost" Kernel (e.g. ): Linux ip-10-250-0- 4.12.14-197.72-default Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig/node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig/node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Hi thanks for raising the issue. Can you share the pod spec(maybe the part that includes CPU shares) that caused the issue? Also which container runtime are you using? And its version?
I think might be related to the error message you're seeing. That repo as you can see is archived so I think you should try upgrading your container runtime. Also in my understanding, CPU share is a relative value that tells kernel scheduler how to schedule the cpu time slice to each task. In my understanding we should expect a large cpu quota here, rather than a large CPU share. Just thinking loud.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
This looks like a kernel limitation to me, not kubelet bug per se. Thanks for linking the bug above, I agree it's related. Kernel code where max limits are set: /triage accepted /remove-lifecycle stale It seems we are only clamping for stuff under , I don't know that this is being applied to all pods. /cc I think there's two things we should do: this limit more clearly the clamping to ensure we don't run into this error /kind documentation /help Help requested with improving the docs.
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : This looks like a kernel limitation to me, not kubelet bug per se. Thanks for linking the bug above, I agree it's related. /remove-lifecycle stale Unfortunately the changed code does not get used in all cases - is not used in my case. Update the clamping to ensure we don't run into this error /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think the issue is in the Kubelet. The kernel uses the range for , but the kubelet can generate and request values outside of the valid range. When using cgroupfs, the kernel automatically clamps the value. Instead systemd has an additional check and refuses to use an invalid value (even if the kernel would accept and clamp it). The clamping is just to make sure the systemd driver works in the same way as cgroupfs, but now any value set above the valid range will get assigned the same number of . would you mind opening a PR to add the clamping also for your case?
I would like to take this one by opening a PR for the clamping issue.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
have you had a chance to look at this issue?
I just created the PR but not sure if I it to the right place. Could you take a look?
/remove-lifecycle stale
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale

Discussion:
package kuberuntime const ( <del> // Taken from lmctfy https://github.com/google/lmctfy/blob/master/lmctfy/controllers/cpu_controller.cc minShares     = 2 sharesPerCPU  = 1024 </del> milliCPUToCPU = 1000 // 100000 is equivalent to 100ms

minQuotaPeriod = 1000 ) <del> // milliCPUToShares converts milliCPU to CPU shares func milliCPUToShares(milliCPU int64) int64 { if milliCPU == 0 { // Return 2 here to really match kernel default for zero milliCPU. return minShares } // Conceptually (milliCPU / milliCPUToCPU) * sharesPerCPU, but factored to improve rounding. shares := (milliCPU * sharesPerCPU) / milliCPUToCPU if shares < minShares { return minShares } return shares } </del> // milliCPUToQuota converts milliCPU to CFS quota and period values func milliCPUToQuota(milliCPU int64, period int64) (quota int64) { // CFS quota is measured in two values:

*/ package kuberuntime <del> // milliCPUToShares converts milliCPU to CPU shares func milliCPUToShares(milliCPU int64) int64 { return 0 } </del>

// API server does this for new containers, but we repeat this logic in Kubelet // for containers running on existing Kubernetes clusters. if cpuRequest.IsZero() && !cpuLimit.IsZero() { <del> cpuShares = milliCPUToShares(cpuLimit.MilliValue()) </del> <ins> cpuShares = int64(cm.MilliCPUToShares(cpuLimit.MilliValue())) </ins> } else { <del> // if cpuRequest.Amount is nil, then milliCPUToShares will return the minimal number </del> <ins> // if cpuRequest.Amount is nil, then MilliCPUToShares will return the minimal number </ins> // of CPU shares. <del> cpuShares = milliCPUToShares(cpuRequest.MilliValue()) </del> <ins> cpuShares = int64(cm.MilliCPUToShares(cpuRequest.MilliValue())) </ins> } resources.CpuShares = cpuShares if memLimit != 0 {