Repository: kubernetes/kubernetes (language: go)

Issue:
Disable cpu quota(use only cpuset) for pod Guaranteed
<!-- Please only use this template for submitting enhancement requests -->

**What would you like to be added**:

Disable cpu quota when we use cpuset for guaranteed pods.

**Why is this needed**:

For now, kubelet default adds cpu quota and share for all cases when `cpuCFSQuota` is open, even if *cpuset* will be set in guaranteed pod case. 

However, this kind of use(cpuset with quota) may sometimes bring us cpu throttle time which we don't expect.

My question is why not make a `if` statement when we add cpuQuota config, if it's *guaranteed* case, just set `cpuQuota=-1`?

```
	if m.cpuCFSQuota {
		// if cpuLimit.Amount is nil, then the appropriate default value is returned
		// to allow full usage of cpu resource.
		cpuQuota, cpuPeriod := milliCPUToQuota(cpuLimit.MilliValue())
		lc.Resources.CpuQuota = cpuQuota
		lc.Resources.CpuPeriod = cpuPeriod
	}
``` 



<!-- DO NOT EDIT BELOW THIS LINE -->
/kind feature


Discussion / hints:
/sig kubelet
/sig architecture
/sig node
Hello,

We are using cpu manager in k8s `v1.10.9` and experiencing the same issue.

https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/ states:


> As Guaranteed pods whose containers fit the requirements for being statically assigned are scheduled to the node, CPUs are removed from the shared pool and placed in the cpuset for the container. CFS quota is not used to bound the CPU usage of these containers as their usage is bound by the scheduling domain itself. In others words, the number of CPUs in the container cpuset is equal to the integer CPU limit specified in the pod spec. This static assignment increases CPU affinity and decreases context switches due to throttling for the CPU-bound workload.


However `CFS quota is not used to bound the CPU usage of these containers as their usage is bound by the scheduling domain itself. ` portion of the document does not work in our case.

So, the setup looks as follows:
1. We have cpu manager enabled and running with `static` policy
2. I have a pod with two containers
3. The first container has limits/requests set to 2 whole cpus
4. The second container has limits/requests set to 0.25 cpu

k8s schedules the pod to a node. When I `exec` into the first container, I see that it has cpu set  defined correctly:

```
/sys/fs/cgroup/cpu,cpuacct # cat /proc/1/status | grep Cpu
Cpus_allowed:    00000000,00000000,00000000,00000202
Cpus_allowed_list:    1,9
```

However, for some reason (I presume that this is a bug), the container has CFS quota defined too, which should not be the case. The pod is restricted to the two cpus already and should be able to use 100% of them, without quota:

```
/sys/fs/cgroup/cpu,cpuacct # cat cpu.cfs_*
100000
200000
```

When I check cpu stats, I see that the container is severely throttled:

```
/sys/fs/cgroup/cpu,cpuacct # cat cpu.stat
nr_periods 2686183
nr_throttled 363111
throttled_time 27984636050628
```

Moreover, when I check cgroup set of the whole pod, I see that k8s put CFS quota to the whole pod too:

```
oot@ip-10-33-99-26:/sys/fs/cgroup/cpu,cpuacct/kubepods/pod62d886b5-ee75-11e8-8f1c-0a4ce21644ea# cat cpu.cfs_*
100000
225000
root@ip-10-33-99-26:/sys/fs/cgroup/cpu,cpuacct/kubepods/pod62d886b5-ee75-11e8-8f1c-0a4ce21644ea# cat fafd0f38288710446b53402716d11449c4b046a09c26266d2725eae2ac7f0352/cpu.cfs_*
100000
200000
```

My understanding is that if the pod is in guaranteed QoS class and requests integral number of cpus, it should not be subject to CFS quota (as stated in the docs).
So in this case:
1. the pod should not have CFS quota defined.
2. The first container should have cpuset defined, but no cfs quota (as the docs imply)
3. The second container should have CFS quota defined as 25000 µs of 100000 µs period
cc: @ipuustin @ConnorDoyle might be of interest to you
Same issue with CPU manager here. We're not realizing any performance benefits due to this issue. In fact we're getting far worse latency due to the CFS throttling bug. Was it intentional to add CFS quota to pods that are CPU pinned? The documentation makes it seem like having a CFS quota on those pods is a bug.
@Wenfeng-GAO it seems that this should be labelled as bug and not a feature :smile: 

as mentioned by @mariusgrigoriu and @zaa, CPUManager's documentation suggests that Pods with pinned CPUs should **not** be bound to CPU quotas, however it is quite clear that they are, which seems wrong.

We keep getting hit by cpu throttling on Guaranteed QoS Pods with pinned CPUs, which is totally counter-productive :disappointed: 


/kind bug
/remove-kind feature
FYI @ConnorDoyle 

I came across your comment here https://github.com/kubernetes/kubernetes/issues/51135#issuecomment-345768456 (from a while back) and it seems that this contradicts what you've mentioned in that post.

Figured you'd may be interested to weigh in.
In theory setting CPU CFS quota for CPU pinned pods should not affect performance, but in practice it does cause performance degradation due to the (buggy) way that CFS quota are implemented in the Linux kernel.

I think the best fix would be to unset any CPU CFS quota from the pod in the CPU manager when reserved CPUs (CPU sets) are assigned.
Thanks @praseodym!

Totally agree, unseting the quota for Pods with cpusets seems like the best solution.

Regarding the Kernel, it seems that a recent patch (included in 4.18) should solve the issue --> https://github.com/kubernetes/kubernetes/issues/67577#issuecomment-466609030

OTOH, this comment --> https://github.com/kubernetes/kubernetes/issues/67577#issuecomment-476228770 is a bit confusing ¯\_(ツ)_/¯

We are going to upgrade to latest CoreOS (2023.5.0) that comes with Kernel 4.19.25 and see how things look.
512ac99 may resolve the issue in this thread, and assuming things are all correctly pinned and cgrouped the issue I discuss in the other bug due to 512ac99 should not be in play.
That's good to know!

This issue will still need some work to align the implemented behaviour with the documentation ("CFS quota is not used to bound the CPU usage of these containers as their usage is bound by the scheduling domain itself.") or vice versa.
Thanks @chiluk, just read your comment on the other thread (https://github.com/kubernetes/kubernetes/issues/67577#issuecomment-476333869) and to be honest this seems a bit worrying as we might actually hit this if we go for >4.18 kernels.

We are using 48-core bare-metals for our worker servers and also deploy java (and php) workloads in various sizes.

So, although this might help in the case of the Pods with pinned CPUs it would actually make things worse for the rest of the workload?
> So, although this might help in the case of the Pods with pinned CPUs it would actually make things worse for the rest of the workload?

If you're pinning a container to a CPU with cpu-manager it should have full use of that CPU. The rest of the workload *should* be excluded. The rest of the system should behave normally, but CFS quota should be disabled for the cgroups that cpuset have been used on. That's my understanding anyway.
@blakebarnett Yes, that's accurate - however it's not the case now, and hopefully will be fixed by https://github.com/kubernetes/kubernetes/pull/75682

What concerns me is that using the latest kernel (>4.18) may degrade performance for the pods that are not pinned to specific cpusets (the majority) as indicated by @chiluk's tests...
/remove-sig architecture 
@liggitt: Those labels are not set on the issue: `sig/`

<details>

In response to [this](https://github.com/kubernetes/kubernetes/issues/70585#issuecomment-489286689):

>/remove-sig architecture 


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
+1
I agree we should be able to disable the use of cfs quota for pods in Guaranteed QoS class when static cpu manager is enabled.

/priority important-soon
/cc @draveness
/assign
+1 @praseodym are you still working on https://github.com/kubernetes/kubernetes/pull/75682?
We (airbnb) would like to test the fix on our clusters
@evansheng I am, but was stuck on a couple of things (see questions in PR comments) — any help would be greatly appreciated!
Issues go stale after 90d of inactivity.
Mark the issue as fresh with `/remove-lifecycle stale`.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with `/close`.

Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta).
/lifecycle stale
/remove-lifecycle stale
AFAICT this inadvertant throttling is likely resolved by fixes 512ac999 and de53fd7aedb in the kernel.  Complain to your distros to get these fixes backported.
> AFAICT this inadvertant throttling is likely resolved by fixes 512ac999 and de53fd7aedb in the kernel. Complain to your distros to get these fixes backported.


That would be great, could you help to link these commits to this issue?



https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=512ac999
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=de53fd7aedb
and probably for completeness
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=763a9ec06c4

YW.  Say hi at Kubecon.
I think this can probably be closed.  If people start hitting throttling again, this can probably be re-opened/revisited.
> I think this can probably be closed. If people start hitting throttling again, this can probably be re-opened/revisited.

👍 

/close
@draveness: Closing this issue.

<details>

In response to [this](https://github.com/kubernetes/kubernetes/issues/70585#issuecomment-552189706):

>> I think this can probably be closed. If people start hitting throttling again, this can probably be re-opened/revisited.
>
>👍 
>
>/close


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
@chiluk @draveness 
I am running the ubuntu 20.04 Focal AMI 

```
/sys/fs/cgroup/cpu,cpuacct # uname -a
Linux [REDACTED] 5.4.0-1018-aws #18-Ubuntu SMP Wed Jun 24 01:15:00 UTC 2020 x86_64 Linux
```

The Kubernetes version is: `1.14.10`

I have configured the CPU manager static policy `--cpu-manager-policy=static` as kubelet parameter (using Kops)
The node has 4 CPU cores and CPU Kube reservation and system reservations are set to 100m each. 

Here is my pod resource configuration:

Limits:
	cpu:     1
	memory:  1Gi
Requests:
	cpu:      1
	memory:   1Gi

When I ssh into the pod I can see the CPU throttling stats and 

```
sys/fs/cgroup/cpu,cpuacct # cat cpu.cfs_*
100000
100000
/sys/fs/cgroup/cpu,cpuacct # cat cpu.stat
nr_periods 1575
nr_throttled 26
throttled_time 303875035
```

At the same time the kubelet.service logs on the nodes shows this for the given pod:

```
Jul 15 14:23:23 [ip] kubelet[6201]: I0715 14:23:23.675164    6201 policy_static.go:175] [cpumanager] static policy: AddContainer (pod: [POD-NAME-REDACTED], container: [CONTAINER-NAME-REDACTED], container id: CONTAINER-ID-REDACTED)
Jul 15 14:23:23 ip kubelet[6201]: I0715 14:23:23.675205    6201 policy_static.go:205] [cpumanager] allocateCpus: (numCPUs: 1)
Jul 15 14:23:23 ip kubelet[6201]: I0715 14:23:23.675278    6201 state_mem.go:84] [cpumanager] updated default cpuset: "0-1,3"
Jul 15 14:23:23 ip kubelet[6201]: I0715 14:23:23.678063    6201 policy_static.go:213] [cpumanager] allocateCPUs: returning "2"
Jul 15 14:23:23 ip kubelet[6201]: I0715 14:23:23.678112    6201 state_mem.go:76] [cpumanager] updated desired cpuset (container id: CONTAINER-ID-REDACTED, cpuset: "2")
```
I am not sure whether the CPU Manager is working and allocating the CPU or not. Let me know if the bug still exists or something is wrong with my configuration. 
The cpu.cfs settings look correct, even though I've never ssh'd into the pod to check. It looks like the cpuset was assigned to CPU 2, which also is likely correct.

I'm not sure how you then hit cfs "throttling" though.  You most definitely are hitting a performance limit because your application could use more cores, but you've only allowed it to use one.  There's a chance the cfs scheduler may be incrementing nr_throttled whenever you have used your whole timeslice *(100ms of quota), but still have runnable tasks in the period.  You'd have to check the code to be sure.

I do know there was talk of disabling cfs for guaranteed resource request pods, but that is definitely in a later version of kubernetes than what you are running.
> I do know there was talk of disabling cfs for guaranteed resource request pods, but that is definitely in a later version of kubernetes than what you are running.

#75682 was never merged, so afaik this is not yet fixed in any Kubernetes version.
@prashantkalkar  @chiluk 
I have the same observation as @prashantkalkar on Ubuntu 19_10 and Ubuntu_20_04 for a Pod in guaranteed QoS class (without cpu-managent-policy=static in kubelet). 

A single guaranteed Pod, which is modestly loaded on an otherwise unloaded machine, is throttled in an exaggerated fashion. The parent container of this Pod is started with a `period=100000, quota of 200000 and cpushares=2048`.
When I just run a Docker container with the same resource allocation values for the period, quota and cpushares parameters of the docker run command, but without the `--cgroup-parent value pointing to the parent container of the Pod`, there is still throttling but much less (9 times less).  When I remove all Pod resource declarations and thus make it belo`ng to the best-effort QoS class, the Pod is not throttled at all because the quota is set to -1.  

I am using the latest kubeadm version in the kubernetes-xenial base on top of Ubuntu distros `5.3.0-40-generic` and `5.4.0-42-generic` , which should have the CFS patch included. 

```
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:10:16Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Server: Docker Engine - Community
 Engine:
  Version:          19.03.6
  API version:      1.40
```

Our experiment is a single Cassandra:2.0 Pod with the following  resource declarations: 
```
 resources:
          limits:
            cpu: "2"
            memory: "3Gi"
          requests:`  
            cpu: "2"
            memory: "3Gi"    
```

We have exposed the Cassandra Pod to a Write-only workload using the YCSB Workload A Load phase. Cassandra is known to be optimized for write-heavy workloads due to use of memtables and therefore this workload is CPU intensive

Here is the YCSB command:

```
python ./bin/ycsb load cassandra2-cql -P workloads/workloada -p recordcount=1500000 -p operationcount=1500000 -p measurementtype=raw -p cassandra.connecttimeoutmillis=60000 -p cassandra.readtimeoutmillis=60000 -target 1500 -threads 20  -p hosts=[redacted: PodIP]
```

This is the kind of throttling observed for the Pod of Guaranteed Class
```
ubuntu@kube-ub19-1/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-pod029a8016_dd52_
11ea_8e53_fa163e52b692.slice# cat cpu.stat
nr_periods 60786
nr_throttled 11359
throttled_time 833585279531
```
Thus is the `docker run` command and the associated throttling, which is a factor of 9 times less, is shown below
```
sudo docker run --name cassandra2 -v /data/cassandra2:/var/lib/cassandra --memory=3221225472 --cpu-shares="2048" --cpu-period="100000" --cpu-quota="200000"  --cap-add="IPC_LOCK" --security-opt="seccomp=unconfined" --net=host decomads/cassandra:2.2.16
```

```
ubuntu@kube-ub19-1:/sys/fs/cgroup/cpu,cpuacct/docker/a9e29898b9a7b32c9959946e37ee779328f4d657837c8920c81338e9e3e9386b$ more cpu.stat
nr_periods 14210
nr_throttled 399
throttled_time 28042320998
```

As stated above, when I finally remove all Pod resource declarations and thus make it belong to the best-effort QoS class, the Pod is not throttled at all: 

```
/sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod8dfb5d99-b80f-4b7d-8bbd-d2c70c988f11$ more cpu.stat
nr_periods 0
nr_throttled 0
throttled_time 0
```

Note that the exagerated throttling has an impact on YCSB benchmark metrics for Cassandra as reported [here](https://github.com/kubernetes/kubernetes/issues/90133) for ubuntu 16.04 without the CFS patch and with kubeadm 14.10
In comparison to this older issue , we have observed a performance improvement for both the Docker container and the Kubernetes Pod for the distros with the CFS patch, but there is still a difference of the same order of magnitude between Docker and Kubernetes.  The cause of the difference can be traced back to to use of the `--cgroup-parent` setting in Kubernetes that has been introduced as part of the following design proposal: 
  * Buddha Prakash and Vishnu Kannan, Pod level resource management in Kubelet,  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/pod-resource-management.md
  * https://github.com/kubernetes/kubernetes/issues/5671
  * https://github.com/kubernetes/kubernetes/issues/27204






I can confirm we still see throttling on workloads that don't use a lot of cpu on these guaranteed pods with static cpu manager policy on. Disabling the quota would make lots of sense IMO, since there's no real reason why it should get used in the first place.
Please forgive my tardy response, but I'm just now returning from paternity leave.  Yes it's rare, but kernel hackers do occasionally procreate.

@eddytruyen 
> A single guaranteed Pod, which is modestly loaded on an otherwise unloaded machine, is throttled in an exaggerated fashion. The parent container of this Pod is started with a `period=10000, quota of 20000 and cpushares=2048`.

To be apples to apples I believe you need perioed=100000 and quota 200000, as I'm pretty sure those values are in microseconds.  The recomendations to lower period are misguided in 99% of cases.

By all means, if you don't want to see throttling turn off cpu limits.  The entire purpose of cpu limits is to throttle applications so they don't adversely affect other applications on the same box. If you do this, make sure you understand not only your singular workload, but all other workloads going on in the cluster.  Turning off cpu limits and saying your application runs faster is like saying you moved your application from a 2 core box and onto a 128 core box and it ran faster.  You have to understand what you are doing.
@chiluk 
Thank you for your reply, I am also taking 20% paternity leave at the moment ;)

> To be apples to apples I believe you need perioed=100000 and quota 200000, as I'm pretty sure those values are in microseconds. The recomendations to lower period are misguided in 99% of cases.

I am sorry for the mistake. I checked it and yes you are right:
```
ubuntu@kube-ub19-1:/sys/fs/cgroup/cpu/kubepods/pod0035397c-173a-40b1-9d32-af1f7fad21be$ more cpu.cfs_quota_us
200000
ubuntu@kube-ub19-1:/sys/fs/cgroup/cpu/kubepods/pod0035397c-173a-40b1-9d32-af1f7fad21be$ more cpu.cfs_period_us
100000
````
My extensive comment on 16 Aug in two sentences summarized: On a Kubernetes worker node with 4 CPU cores, with only one Pod (with one Cassandra container) of the guaranteed QoS class (its cpu request==cpu limit==2) in the default Namespace of Kubernetes, this Pod is substantially more throttled than  a Cassandra container with cpu-shares=2048 and cpu-quota=200000 on that node.The cause of this is the fact that the container of the Pod has set a Cgroup-Parent, whereas the Docker container does not.  However, the Cgroup-Parent of the Pod does not impose a stricter cpu limit and there is no other child container besides the Cassandra container and the pause container (with cpu.cfs_quota=-1).  

So the Docker container has the same cpu limit as the container inside the Pod, but the latter is much more throttled than the former. 
I consider this a serious bug. What am i missing? 
  
FYI in case it was not clear I am seeing this still on k8s 1.17.8, kernel version 5.4.38. I think it needs to get reopened
/reopening to investigate recent messages :microscope: 
/reopen
@alejandrox1: Reopened this issue.

<details>

In response to [this](https://github.com/kubernetes/kubernetes/issues/70585#issuecomment-705246569):

>/reopening to investigate recent messages :microscope: 
>/reopen


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
/unassign @draveness 
/triage accepted
Issues go stale after 90d of inactivity.
Mark the issue as fresh with `/remove-lifecycle stale`.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with `/close`.

Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta).
/lifecycle stale
/remove-lifecycle stale

still very much an issue, causes lots of pain for us
I just thought it would be nice to gather some information about how people are experiencing this issues, and see if we are talking about the same thing. I agree that we should make it possible to deactivate cfs for pods bound to a given set of cpus, even tho. the reason for the throttling of theses pods is imo. a bug in the kernel/a feature not necessary intended. Pods may have a high `nr_throttled/nr_periods` ratio, but the `throttled_time/cpuacct.usage` ratio would be extremely slow (unless there are other bugs in the kernel).

> @2rs2ts: I can confirm we still see throttling on workloads that don't use a lot of cpu on these guaranteed pods with static cpu manager policy on. Disabling the quota would make lots of sense IMO, since there's no real reason why it should get used in the first place.

Are you referring to guaranteed pods with exclusive cpusets being throttled, or are you talking about pods without exclusive cpusets? 

Also, it would be nice to see the output of `cpu.{cfs_period_us,cfs_quota_us,stat}`, `cpuacct.usage_percpu`, `cgroup.procs` and `cpuset.cpus` to get more information about the processes being throttled.
>Are you referring to guaranteed pods with exclusive cpusets being throttled, or are you talking about pods without exclusive cpusets?

The former. I have the same problem as described here. https://github.com/kubernetes/kubernetes/issues/70585#issuecomment-442445507

Throttling happens even if the process is not brushing up close to its limit because of many factors, not the least of which are the host processes that are scheduling on that same vCPU and eating up CPU time. Is it less bad than without the static CPU manager turned on and without giving these containers an integer number of CPUs? Yes, of course. Is it still bad enough that it's caused nasty latency spikes? Also yes.
> The former. I have the same problem as described here. #70585 (comment)

> Throttling happens even if the process is not brushing up close to its limit because of many factors, not the least of which are the host processes that are scheduling on that same vCPU and eating up CPU time. Is it less bad than without the static CPU manager turned on and without giving these containers an integer number of CPUs? Yes, of course. Is it still bad enough that it's caused nasty latency spikes? Also yes.

Thanks for the extra information @2rs2ts, that makes the case more clear to me, and makes it look even more important to get a proper fix to it!

So the reason might be because the non-pinned containers are using more than their quota on the container cgroup and getting throttled, resulting in using more than "their" share of quota on the pod cgroup. This happens because a runqueue for the container ends up with a negative _remaining runtime_, gets throttled, and has to wait until the next period to be able to pay back that "debt". On the container cgroup however, it can pay that in the same period as used, resulting in a lower available quota for the pinned container in that period. There may/will also be left (max) 1ms of quota in the run-queue on the pod cgroup on a cpu that the pinned container cannot use, resulting in even less available quota. That might then result in the pinned container also getting throttled. Can you verify the `nr_throttled` and `throttled_time` on the container cgroups and the pod cgroup in order to check that theory?

A temporary fix/mitigation that would be nice if you could test is to add a small amount of extra cpu to the pod via [https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/) (eg. 100m, 250m or 500m to test, depending on your cfs_period) to see if that fixes the problem.

I must honestly agree that I more and more think that disabling the pod level cfs quota is an option, maybe behind a config flag, since the throttling of a pod can cause another one to throttle as well, without (imo.) giving any real benefit for most cases. The only place (as far as I can come up with now) is for container runtimes like kata where you run VMs instead of containers directly in a children cgroup.
>So the reason might be because the non-pinned containers are using more than their quota on the container cgroup and getting throttled, resulting in using more than "their" share of quota on the pod cgroup.

We've analyzed pod and container metrics really thoroughly and found that we would get throttling even when all containers in a pod were using far less than what each was allocated. Like, in some cases, it was as bad as a container using around 10-20% of what it was allocated was still getting throttled. We were able to identify some mitigation strategies, such as reducing the number of threads in our processes to 1 thread per core, increasing reserved resources in kubelet settings several times over, and of course, using the static CPU manager, but at the end of the day I think CPU quotas in Linux are just not designed for using on latency-sensitive workloads in multi-tenanted environments 🤷‍♂️

>Can you verify the `nr_throttled` and `throttled_time` on the container cgroups and the pod cgroup in order to check that theory?

Probably, though this is quite the moving target in a very large system. Do these happen to correlate to some of the kubelet metrics? We've got them [collected via datadog](https://docs.datadoghq.com/agent/kubernetes/data_collected/#kubelet) at my job, and I can really easily pull up the timeseries for `kubernetes.cpu.cfs.throttled.periods` and `kubernetes.cpu.cfs.throttled.seconds` and get historical data. But if you need the raw information from the cgroups then that will take me a while to get.

>A temporary fix/mitigation that would be nice if you could test is to add a small amount of extra cpu to the pod via https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/ (eg. 100m, 250m or 500m to test, depending on your cfs_period) to see if that fixes the problem.

Does this mess with the static CPU manager setting? For example: if I gave every pod an extra 250m, would a pod that's currently requesting 1 cpu and getting its own cpuset as a result, stop getting that cpuset? Feels like the answer would obviously be yes, and if that's the case then it's a no-go, it'd be far too complicated for us to adjust all the pods that are making use of the static CPU manager.
> We've analyzed pod and container metrics really thoroughly and found that we would get throttling even when all containers in a pod were using far less than what each was allocated. Like, in some cases, it was as bad as a container using around 10-20% of what it was allocated was still getting throttled. We were able to identify some mitigation strategies, such as reducing the number of threads in our processes to 1 thread per core, increasing reserved resources in kubelet settings several times over, and of course, using the static CPU manager, but at the end of the day I think CPU quotas in Linux are just not designed for using on latency-sensitive workloads in multi-tenanted environments 🤷‍♂️

This is the last time I will mention it and than I will shut up:
CPU quotas are not problem, cgroup hierarchies are. A container is throttled when its quota **or the quota of its parent** is surpassed.
Since Pods can aggregate multiple containers, the cgroup-parent parameter is always set in Docker  for example.  The experiment with Cassandra as I described  [here](https://github.com/kubernetes/kubernetes/issues/70585#issuecomment-674537406) demonstrates that the cgroup-parent setting in Docker causes the exaggerated throttling.


>A container is throttled when its quota or the quota of its parent is surpassed.
Since Pods can aggregate multiple containers, the cgroup-parent parameter is always set in Docker for example.

I understand this. I said that we checked _all_ containers in a pod. Pods where not a single container is using up more than a sliver of its respective quota will still be throttled. So it is not a problem of container A using too many resources and that causing container B to get throttled, at least not in my case. Or are you referring to higher-level cgroup parents like the ones for the QoS class in general?
> Or are you referring to higher-level cgroup parents like the ones for the QoS class in general

Yes and no. There's nothing higher-level about the cgroup parents for the guaranteed QoS class. For each Pod of the guaranteed QoS class, there is a separate cgroup directly stored under the kubepods slice. This cgroup has a cpu-quota that equals the sum of the cpu-quotas of the containers in the Pod. 

But I guess, the exaggerated throttling also happens for Pods of the burstable class whose cgroup-parent is stored under kubepods/burstable

The following snippet shows the cgroup-structure for a Pod with 2 containers and their cpu limit set to 1 and 2 respectively (note: in this particular example the cgroupfs driver is used instead of systemd, but the same reasoning applies to systemd as well). The third container is the pause container btw.
```
ubuntu@4:/sys/fs/cgroup/cpu/kubepods$ for i in 'quota'; do for j in `find . | grep $i`; do echo $j: `cat $j`; done; done | more
./pod64427488-aaec-49e8-995a-48e5366b1787/0e34ff7bc3e537c2e79680e9ef3c90fa73bf9106ea6fd63f3a7dc211658ab772/cpu.cfs_quota_us: -1
./pod64427488-aaec-49e8-995a-48e5366b1787/cpu.cfs_quota_us: 300000
./pod64427488-aaec-49e8-995a-48e5366b1787/c810b92253115f1474febfad54edcd30af0d45cd8e183b81fae1c68f566dced0/cpu.cfs_quota_us: 100000
./pod64427488-aaec-49e8-995a-48e5366b1787/ef8780f721e814cf3af37edf00b8016fbd6f695ebd73c9556a4f3f505cf88c73/cpu.cfs_quota_us: 200000
````

Somehow, this configuration causes the exaggerated throttling in the Cassandra experiment described [here](https://github.com/kubernetes/kubernetes/issues/70585#issuecomment-674537406). A workaround could be as simple as letting the kubelet set `./pod64427488-aaec-49e8-995a-48e5366b1787/cpu.cfs_quota_us` to -1 while setting the containers's quota at 100000 and 200000


Thanks again @2rs2ts!

> We've analyzed pod and container metrics really thoroughly and found that we would get throttling even when all containers in a pod were using far less than what each was allocated. Like, in some cases, it was as bad as a container using around 10-20% of what it was allocated was still getting throttled. We were able to identify some mitigation strategies, such as reducing the number of threads in our processes to 1 thread per core, increasing reserved resources in kubelet settings several times over, and of course, using the static CPU manager, but at the end of the day I think CPU quotas in Linux are just not designed for using on latency-sensitive workloads in multi-tenanted environments man_shrugging

The avg. load doesn't matter for cfs tho., it is per period. So if your container is not limited by a cpuset, that sounds like the intended behavior. And, if the container is _not_ constraint by a cpuset, reducing the threads to the same as or smaller than the `quota/period` ratio will definitely help.

> Probably, though this is quite the moving target in a very large system. Do these happen to correlate to some of the kubelet metrics? We've got them collected via datadog at my job, and I can really easily pull up the timeseries for kubernetes.cpu.cfs.throttled.periods and kubernetes.cpu.cfs.throttled.seconds and get historical data. But if you need the raw information from the cgroups then that will take me a while to get.

Ahh, I see. I do still think i understand why it happen tho. The raw information isn't _that_ important, but it looks like your metrics only look at throttling on the container cgroups and not the pod level one.

> Does this mess with the static CPU manager setting? For example: if I gave every pod an extra 250m, would a pod that's currently requesting 1 cpu and getting its own cpuset as a result, stop getting that cpuset? Feels like the answer would obviously be yes, and if that's the case then it's a no-go, it'd be far too complicated for us to adjust all the pods that are making use of the static CPU manager.

It will not mess with CPU manager, since it only affects the `memory.max` and `cpu.max`/cfs values on the *pod* level cgroup, while CPU Manager only looks at "one container at a time". Did double check myself, and it is the case. So it would be interesting to see if that mitigates your issue. I think a relatively small amount like `10m` to `100m` should do quite a lot. Definitely not a long term solution, but it might help mitigate the issues.


> Pods where not a single container is using up more than a sliver of its respective quota will still be throttled. So it is not a problem of container A using too many resources and that causing container B to get throttled, at least not in my case. Or are you referring to higher-level cgroup parents like the ones for the QoS class in general?

Did you verify that non of the pods were not throttled (via nr_throttled)? Also, due to how cfs works, a fair amount of the quota could be distributed in a way so that the pod cgroup could be throttled, even though non of the containers were using above their quota. The chance for this increases when `nr_cpus_cfs_quota_equivalent/nr_cpus_on_macine` decreases. That is due to the nature of how cfs works.

Edit: Also, one more thing: You say that both the container with a cpu set and the one without is throttled. Is the ratio between nr_throttled between the containers equal to ~1? If so, there is definitely some kind of correlation.  
We should probably split this into two separate issues: One with `cpuManagerPolicy: static` and one with `cpuManagerPolicy: none`...

> @eddytruyen: CPU quotas are not problem, cgroup hierarchies are. A container is throttled when its quota or the quota of its parent is surpassed.

Yes, and no I guess. The biggest "problem" here is the hierarchy, and I do kinda agree about that. That does however not explain the issues for the people running a single container in a pod where the container is limited by a cpuset, but I do think the majority of throttling is due to how the hierarchy works. Therefore I think one way to go forward is to make it possible to disable the pod level cgroup via a config flag, since some workloads depend on having a pod level quota as well.

> Since Pods can aggregate multiple containers, the cgroup-parent parameter is always set in Docker for example. The experiment with Cassandra as I described here demonstrates that the cgroup-parent setting in Docker causes the exaggerated throttling.

Not sure if I understand those results, since you say that they were executed without the newest cfs patches. I would however expect a benchmark with 20 threads and a cfs quota of the equivalent of 2 cpus to have a ratio of `nr_throttled/nr_periods` equal to almost 1. 

> As stated above, when I finally remove all Pod resource declarations and thus make it belong to the best-effort QoS class, the Pod is not throttled at all:

Do you mean you still had cfs limits on the container level? If so, how much was that being throttled?

> [...]
> Somehow, this configuration causes the exaggerated throttling in the Cassandra experiment described here. A workaround could be as simple as letting the kubelet set ./pod64427488-aaec-49e8-995a-48e5366b1787/cpu.cfs_quota_us to -1 while setting the containers's quota at 100000 and 200000

I would expect some throttling when the pod level cgroup has this quota, but not sure how much would be expected, but it would ofc. depend on how the applications run. The fact that we have multiple containers that can "steal" more quota from the pod cgroup than they have in their own cfs quota is on of the reason, the other one is that the quota periods are not synced, so they operate on their own.

But yeah, after all, it looks like there should be a way to disable the cfs quota on the pod cgroup... :sweat_smile: 
> We should probably split this into two separate issues: One with `cpuManagerPolicy: static` and one with `cpuManagerPolicy: none`...

@odinuge Yes, with static cpu management polciies, cpu quota's are set, but they shouldn't be according to the documentation.

> 
> > @eddytruyen: CPU quotas are not problem, cgroup hierarchies are. A container is throttled when its quota or the quota of its parent is surpassed.
> 
> [..] That does however not explain the issues for the people running a single container in a pod where the container is limited by a cpuset, [..] 

Actually with the static cpu manager policy, cpusets as well as cpu quotas are set at the same time. I  even think cpu-shares are set as well, but I am not sure. 

> Therefore I think one way to go forward is to make it possible to disable the pod level cgroup via a config flag, since some workloads depend on having a pod level quota as well.
> 

 Yes, but that is a work-around and not a solution of the cgroups hierachy problem, that is hopefully solved in cgroupsv2. 

> > Since Pods can aggregate multiple containers, the cgroup-parent parameter is always set in Docker for example. The experiment with Cassandra as I described here demonstrates that the cgroup-parent setting in Docker causes the exaggerated throttling.
> 
> Not sure if I understand those results, since you say that they were executed without the newest cfs patches. I would however expect a benchmark with 20 threads and a cfs quota of the equivalent of 2 cpus to have a ratio of `nr_throttled/nr_periods` equal to almost 1.

No, they were executed **with** the newest cfs patches, but I guess the above is a typo. Maybe I should clarify that the Cassandra experiment exercised a moderate workload that does not intend to fully use the 2 cpus, but only +/-1 cpu. 

> 
> > As stated above, when I finally remove all Pod resource declarations and thus make it belong to the best-effort QoS class, the Pod is not throttled at all:
> 
> Do you mean you still had cfs limits on the container level? If so, how much was that being throttled?

No there are no cfs limits for the best-effort class, so throttling is not applied.

> 
> > [...]
> > Somehow, this configuration causes the exaggerated throttling in the Cassandra experiment described here. A workaround could be as simple as letting the kubelet set ./pod64427488-aaec-49e8-995a-48e5366b1787/cpu.cfs_quota_us to -1 while setting the containers's quota at 100000 and 200000
> 
> I would expect some throttling when the pod level cgroup has this quota, but not sure how much would be expected, but it would ofc. depend on how the applications run. The fact that we have multiple containers that can "steal" more quota from the pod cgroup than they have in their own cfs quota is on of the reason, the other one is that the quota periods are not synced, so they operate on their own.

This does not apply in the Cassandra experiment as we never hit the 2 cpu quota limit for 20 threads.

> 
> But yeah, after all, it looks like there should be a way to disable the cfs quota on the pod cgroup... 😅

Certainly for cpu management policies.

For guaranteed and burstable Pods, without CPU management, I propose the following steps:

1. It should be first investigated if [my finding with Cassandra](https://github.com/kubernetes/kubernetes/issues/70585#issuecomment-674537406) can be reproduced in in other workloads. I can already state that the problem does not appear for the [fibonacci](https://github.com/indeedeng/fibtest) test used for diagnosing and solving the cfs patch. 

2. Probably the [sig-node group](https://github.com/kubernetes/community/tree/master/sig-node) is the best group to consult about this change since they have been adding the [pod overhead feature](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/). Logically, it seems the  right place to discuss adding such flag. All in all, I think it is a non-breaking change as a Pod's cgroup's cpu.quota is not used for scheduling nor pod eviction and the Pod's actual cpu limit is managed by the API server/etcd.




> @odinuge Yes, with static cpu management polciies, cpu quota's are set, but they shouldn't be according to the documentation.

Yeah, kinda. When it comes to container level the docs are a bit ambiguous about this, ref. `CFS quota is not used to bound the CPU usage of these containers as their usage is bound by the scheduling domain itself`.

> Actually with the static cpu manager policy, cpusets as well as cpu quotas are set at the same time. I even think cpu-shares are set as well, but I am not sure.

The pod level limit is "still required" since that is a guarantee we gives to the users in a way. But yeah, there should be a way to opt out. The CPU shares would essentially do nothing if two containers in a pod have different cpusets, or if the cpuset used by a pod is exclusive.

> Yes, but that is a work-around and not a solution of the cgroups hierachy problem, that is hopefully solved in cgroupsv2.

Not sure I understand what you mean by that? Are we talking about the same issue, or are you saying there is a bug in the kernel causing this?

> No, they were executed with the newest cfs patches, but I guess the above is a typo. Maybe I should clarify that the Cassandra experiment exercised a moderate workload that does not intend to fully use the 2 cpus, but only +/-1 cpu.

Hmm, yeah, ok. But if you start 20 threads on a eg. 60 core machine, the cgroup will be throttled even tho the wall time used is lower than the quota. That is a known issue/feature with CFS on machines with many cores. If a cgroup is throttled it is because one CPU looks for more available runtime for the current period, and if there is none, it will be throttled. It is more or less impossible to know how much cpu time a set of processes on a set of cores _will_ use, so cfs keep reserving a share of the quota/runtime for a period, and throttles when it cannot get a new share.

> No there are no cfs limits for the best-effort class, so throttling is not applied.

Ahh, I thought you set the cfs quota to `-1` on the pod level cgroup while keeping the quota on the container level cgroups. But yeah, if you are in best-effort, you will never be throttled, but in the same time you could "loose" runtime due to CPU pressure made by higher priority cgroups.

> This does not apply in the Cassandra experiment as we never hit the 2 cpu quota limit for 20 threads.

Not sure if I understand this... You say you get throttled, and that means that you *do* hit that limit. Since there is no more runtime left to distribute in a period, and a CPU wants more, it will be throttled. You could in theory with a short cfs period get throttled while essentially not using any CPU time at all.

> For guaranteed and burstable Pods, without CPU management, I propose the following steps:
> 
>   1. It should be first investigated if my finding with Cassandra can be reproduced in in other workloads. I can already state that the problem does not appear for the fibonacci test used for diagnosing and solving the cfs patch.

Yes! Having a reliable and simple way to reproduce would be awesome!

>   2. Probably the sig-node group is the best group to consult about this change since they have been adding the pod overhead feature. Logically, it seems the right place to discuss adding such flag. All in all, I think it is a non-breaking change as a Pod's cgroup's cpu.quota is not used for scheduling nor pod eviction and the Pod's actual cpu limit is managed by the API server/etcd.

Yup, sig-node is definitely the way to go! I do think disabling it by default can be a bad idea, since people might rely on the pod level limits, but a flag/way to disable it looks like the way to go.



On another note, this KEP can be interesting for you, since it is somewhat related to this issue: https://github.com/kubernetes/enhancements/pull/1592
@odinuge I assume there is a bug in the kernel or either the cpu usage of Kubernetes code at the pod level is too high
> @odinuge I assume there is a bug in the kernel or either the cpu usage of Kubernetes code at the pod level is too high

Interesting! Kubernetes should not run processes in those cgroups by default, so it shouldn't be the problem. Health Checks might however use some quota though. Exec probes run inside the container, and http/rpc probes "consume" resources in the way that the container has to answer. I don't think other stuff should interact with that cgroup...
> Yes! Having a reliable and simple way to reproduce would be awesome!

Starting up a container that mimics a pod as much as possible but without the cgroup parent parameter could easily be scripted. I can give it a go at such script that also grabs cpu.stats. from the cgroups. The delopers who reported to still have problems could run this script and run their (proprietary) workload to see differences in throttling and performance metrics between the container and the pod.


@odinuge 

>Did you verify that non of the pods were not throttled (via nr_throttled)?

No, I'll have to look into adding some automated collection of that, if it's not the same as the container level metrics I had for cfs throttled time/periods.

>You say that both the container with a cpu set and the one without is throttled.

Did I say that? In my case we see throttling when all containers have integer CPU requests/limits. And with single container pods too. It literally just always happens no matter what.

re: the conversation you both are having, I think it should be obvious that Kubernetes should disable cpu quota on pods that are given exclusive cpusets by the CPU manager. Even if the docs are kind of vague about it now, I do not see an upside to having quotas in that case. Literally just wastes time and electricity.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with `/remove-lifecycle stale`.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with `/close`.

Send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).
/lifecycle stale
/remove-lifecycle stale
> 
> 
> > @odinuge I assume there is a bug in the kernel or either the cpu usage of Kubernetes code at the pod level is too high
> 
> Interesting! Kubernetes should not run processes in those cgroups by default, so it shouldn't be the problem. Health Checks might however use some quota though. Exec probes run inside the container, and http/rpc probes "consume" resources in the way that the container has to answer. I don't think other stuff should interact with that cgroup...

@odinuge We have looked into the possibility that throttling increases due to the readiness probes. Yes it did! 
The execution of the  readiness probes are separate processes that show up in the cgroup.procs file of the cassandra container's cgroup, not the higher-level pod's cgroup. 

Below you find a comparison of the number and percentage of throttled periods for 3 setups. This shows that the readiness probes are the ones being throttled. 

```
1 pod with docker engine as cr, cpu-quota of pod turned on, cpu-quota of cassandra container turned on
# periods 13737
# throttled 6664
% throttled 49%

1 pod in docker engine as cr, cpu-quota of pod and cassandra container turned on, probes turned off
# periods 14289
# throttled 697
% throttled 5%


1 container in docker engine directly, no kubernetes, cpu-quota of cassandra container turned on
# periods 12454
# throttled 712
% throttled 6% 
```

The following image shows the impact on latency of these probes.  

![image](https://user-images.githubusercontent.com/11139984/121353212-e5243a00-c92d-11eb-8249-a1397075bb5b.png)

It seems there is a high system load/cpu issue with Kubernetes because it constantly spawns runc for running the readiness/liveliness checks: https://github.com/kubernetes/kubernetes/issues/82440#.  We also see that with  the readiness probe of cassandra. The  following picture shows the decrease in cpu utilization when removing the readiness probe:
![image](https://user-images.githubusercontent.com/11139984/121353790-88754f00-c92e-11eb-93d1-0a980dea4508.png)

We also looked what is the impact of turning off the cpu_quota of the cgroup of the Pod as follows:
```
cd /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/
kubepods-burstable.slice/kubepods-burstable-pod<pod-id>.slice/
echo -1 > cpu.cfs_quota_us
```
We did see a small improvement in latency and throttling, but not an as large improvement as turning off probes.
```
1 pod in docker, cpu-quota of pod turned off, cpu-quota of cassandra container turned on
# periods 14348
# throttled 6104
% throttled 43%

```

![image](https://user-images.githubusercontent.com/11139984/121356152-e5720480-c930-11eb-856a-7a2d82d742a9.png)

Here's the configuration and code of the readiness probe
```
readinessProbe:
exec:
command:
- /bin/bash
- -c
- /ready-probe.sh
failureThreshold: 3
initialDelaySeconds: 15
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
```

```
ready_probe.sh:
if [[ $(nodetool status | grep $POD_IP) == *"UN"* ]]; then
  if [[ $DEBUG ]]; then
    echo "UN";
  fi
  exit 0;
else
  if [[ $DEBUG ]]; then
    echo "Not Up";
  fi
  exit 1;
fi
```

> @odinuge We have looked into the possibility that throttling increases due to the readiness probes. Yes it did!
The execution of the readiness probes are separate processes that show up in the cgroup.procs file of the cassandra container's cgroup, not the higher-level pod's cgroup.

Yes, by design, exec-probes run inside the container, and the resources they use are accounted. That is expected behavior. And yes, they are executed in the _container_ control group, _not_ the pod level one. And yes, exec probes are just plain linux processes being executed inside the container "sandbox", there is not anything special about them.


> Below you find a comparison of the number and percentage of throttled periods for 3 setups. This shows that the readiness probes are the ones being throttled.

Those numbers look reasonable, no doubt there.

> It seems there is a high system load/cpu issue with Kubernetes because it constantly spawns runc for running the readiness/liveliness checks: #82440. We also see that with the readiness probe of cassandra. The following picture shows the decrease in cpu utilization when removing the readiness probe:

Not 100% updated on this, but the overhead of constantly using runc for exec probes is definitely not zero.  In your example, I am not sure how much CPU time is used by runc compared to the scripts tho. You can try to use crun instead of runc and see if there is any difference. Due to the nature of exec-probes, they act very spiky on the CPU. Eg. they use 300ms to run, and that translates to one ~one CPU on three CFS bandwidth periods. If your application is already close to the limit, that will certainly cause throttling...

> We also looked what is the impact of turning off the cpu_quota of the cgroup of the Pod as follows:

That also sounds expected. If you are being throttled in the container control group, you will not see much difference by "disabling" cfs bandwidth on the pod control group. But yeah, there might be a small amount of change.

But, overall, the metric of `throttled periods` divided by `periods` is not the best metric here. Looking at the time spent throttled is at least as interesting. It is possible to get a ratio of more than `90%` without seeing any increase in latency, all depends on the workload and configuration.

---

Also, looking at more than only the 95% percentile would be interesting. Something like https://hdrhistogram.github.io/HdrHistogram/ are cool for looking at the latency effects.

edit: new paragraph
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue or PR as fresh with `/remove-lifecycle stale`
- Mark this issue or PR as rotten with `/lifecycle rotten`
- Close this issue or PR with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-lifecycle stale
/assign @cynepco3hahue 
We are also seeing a small, but unexpected amount of throttling with static cpu policy set and fully guaranteed QoS (with integer core count) on containers in AKS.
> We are also seeing a small, but unexpected amount of throttling with static cpu policy set and fully guaranteed QoS (with integer core count) on containers in AKS.

We have exactly the same with an on-prem Rancher Cluster.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue or PR as fresh with `/remove-lifecycle stale`
- Mark this issue or PR as rotten with `/lifecycle rotten`
- Close this issue or PR with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue or PR as fresh with `/remove-lifecycle stale`
- Mark this issue or PR as rotten with `/lifecycle rotten`
- Close this issue or PR with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue or PR as fresh with `/remove-lifecycle stale`
- Mark this issue or PR as rotten with `/lifecycle rotten`
- Close this issue or PR with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-lifecycle stale
We also cause the same problem. The `cpu.shares` can lead some performance lost for guaranteed pod(use only cpuset). When move the pid from container's cpu cgroup to `/sys/fs/cgroup/cpu/cgroup.procs`, the process performance works as expected.
/cc @ffromani 

/priority backlog

it is active for so long. Changing priority to reflect this
This issue is labeled with `priority/important-soon` but has not been updated in over 90 days, and should be re-triaged.
Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release.

You can:
- Confirm that this issue is still relevant with `/triage accepted` (org members only)
- Deprioritize it with `/priority important-longterm` or `/priority backlog`
- Close this issue with `/close`

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted
/triage accepted
/remove-priority important-soon
any progress?
https://github.com/kubernetes/kubernetes/pull/127525

looks ready and aiming for a early 1.33 merge
copying sig-node feedback after mtg Oct 8, 2024 and updating
- The fix must be in effect iff there are CPU exclusively assigned to pod and the cpumanager policy is static (implied by exclusive cpu assignment)
- to address the concerns about unexpected side effects, the SIG agrees we can use a Feature Gate. (see: https://kubernetes.slack.com/archives/C5P3FE08M/p1727072265347319)
- the Feature Gate will start enabled, because we will merge a fix we are confident about.
- the Feature Gate will be time-bound at +4 releases. We introduce the FG in 1.33 locked to enabled, we remove in 1.33+4=1.37
- if we get issues related to the fix, we will iterate over the fix; affected users can use the Feature Gate to restore the old behavior
- if no issues reported in the allotted time, the Feature Gate will be locked to on and removed.
- the preferred fix is #127525
handling init containers (xref: https://github.com/kubernetes/kubernetes/pull/127525#discussion_r1851913894)
all the following assuming Guaranteed QoS with cpumanager static policy in use:

1. if app containers have exclusive CPU assignment, disable CPU quota. Rationale: this is what the bug is about :)
2. if sidecar containers  have exclusive CPU assignment, disable CPU quota (treat as app containers). Rationale: sidecar containers have a lifespan like the app containers. It is unlikely users want sidecars with exclusive CPUs and app containers without, yet this is legal
3. if init containers have have exclusive CPU assignment, *and app or sidecontainer do*,  disable CPU quota. Rationale: falls in the same bucket as 1.
4. if init containers have have exclusive CPU assignment, *and app or sidecontainer do NOT*,  **keep CPU quota enabled**. Rationale: Init container lifetime should be little compared to app container. The issue affects containers running at steady state. Disabling quota because init containers can cause problems to app containers which would expect their quota set. All in all, this seems a corner case which should be treated separately, if ever becomes a real concern.

Would that work? @odinuge @SergeyKanzhelev @scott-grimes @MarSik 
hmm 
1 + 2 + 3 are covered with the current logic
4 is trickier. if we wish to enable this logic we can no longer rely on `podContainerManager.HasExclusiveCPUs(pod)` to determine if a given pod should have pod-level cpu quota enforced because it'd be possible that a pod has exclusive cpus but the pod-level cfs enforcement is still enabled

# Current Logic 

[link](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/cpumanager/policy_static.go#L482-L511)

In `policy_static.go` `podGuaranteedCPUs` takes into account both init containers and app containers when returning the number of guaranteed cpu pod-wide 

# Proposed Logic

Keep #127525 as-is. This introduces the edge case noted above under (4) where a pod with guaranteed init container and no other guaranteed application containers would have its pod-level cpu quota removed. I concur this is an edge case we could treat separately if it becomes a real concern 

I'm getting somewhat confused about the exact intention of https://github.com/kubernetes/kubernetes/pull/127525 after giving it a second pass. Can you please add some more high level context on the PR to describe exactly what the intended behavior of this PR is? As right now it seems to disable all (?) cpu limits for all containers as long as a single container has an exclusive cpuset. That behavior does not sound fully correct to me, but I might be missing something.

Something like this would be very useful;
- pod level cgroup: We remove the cpu limit when X is true.
- container level cgroup for normal container: We remove the cpu limit when X is true.
- container level cgroup for init container: We remove the cpu limit when X is true.
- container level cgroup for sidecar container: We remove the cpu limit when X is true.

Correct me if I'm wrong here, but I'm also pretty sure we never set the cpuset values on the _pod_ level cgroup, so we cannot really disable all cpu limits on both pod and container level for pods without an exclusive cpuset without changing the end user behavior. 
this is the proposed functionality:

1) pod level cgroup: We remove the pod-level cpu limit when `any container in the pod has a dedicated CPU`
2) container level cgroup: We remove the container-level cpu limit per container when `the container has a dedicated CPU`

a container within a pod is allocated a dedicated CPU when all of the following are true per https://github.com/kubernetes/kubernetes/blob/02fd991c5b3d1b705658633a93f8d8aae2c694bb/pkg/kubelet/cm/cpumanager/policy_static.go#L457-L478 

* cpumanager policy is static
* pod has qos guaranteed
* container has integer cpu request

the `HasExclusiveCPUs` method as it exists now is insufficient to accomplish the above proposed functionality so I've split this into two methods. Both of these methods rely on cpumanager's `GetExclusiveCPUs` function:

1) `PodHasExclusiveCPUs`: iterates over the pods init and normal containers to determine (1)
2) `ContainerHasExclusiveCPUs`: directly determines (2) for the given pod & container 

I've updated the PR https://github.com/kubernetes/kubernetes/pull/127525 to test both the pod-level and container-level cfs quotas 

---

note the above proposal does NOT account for the following edge case:

> if init containers have have exclusive CPU assignment, and app or sidecontainer do NOT, [we should] CPU quota enabled. 

as it stands with the PR the cpu quota for the pod would be disabled. as https://github.com/kubernetes/kubernetes/issues/70585#issuecomment-2491031661 mentioned this is a very unusual edge case and should be dealt with separately 
@scott-grimes thanks for the updates. At glance your latest upload LGTM, will have a later pass later but I see no obvious problems. I'd probably add an extra comment to be really explicit why we need to disable the sandbox quota: the reason was explored in the issue conversation, but is buried quite deep in the series of comments/review comments.
@ffromani amended the PR body and feature gate in-line comments, swapped out feature gate to 1.33 and re-pushed
thanks @scott-grimes, will have another pass ASAP. Please make sure to address @odinuge comments. From my perspective we were quite LGTM already.

Fix patch:
diff --git a/pkg/features/kube_features.go b/pkg/features/kube_features.go
index e0d37c5bfb7d0..8e2833a6af2ab 100644
--- a/pkg/features/kube_features.go
+++ b/pkg/features/kube_features.go
@@ -862,6 +862,22 @@ const (
 	//
 	// Enables specifying resources at pod-level.
 	PodLevelResources featuregate.Feature = "PodLevelResources"
+
+	// owner: @ffromani
+	// beta: v1.33
+	//
+	// Disables CPU Quota for containers which have exclusive CPUs allocated.
+	// Disables pod-Level CPU Quota for pods containing at least one container with exclusive CPUs allocated
+	// Exclusive CPUs for a container (init, application, sidecar) are allocated when:
+	// (1) cpumanager policy is static,
+	// (2) the pod has QoS Guaranteed,
+	// (3) the container has integer cpu request.
+	// The expected behavior is that CPU Quota for containers having exclusive CPUs allocated is disabled.
+	// Because this fix changes a long-established (but incorrect) behavior, users observing
+	// any regressions can use the DisableCPUQuotaWithExclusiveCPUs feature gate (default on) to
+	// restore the old behavior. Please file issues if you hit issues and have to use this Feature Gate.
+	// The Feature Gate will be locked to true and then removed in +2 releases (1.35) if there are no bug reported
+	DisableCPUQuotaWithExclusiveCPUs featuregate.Feature = "DisableCPUQuotaWithExclusiveCPUs"
 )
 
 func init() {
diff --git a/pkg/features/versioned_kube_features.go b/pkg/features/versioned_kube_features.go
index 8158e71c8b95e..e07b9b7a8b5e0 100644
--- a/pkg/features/versioned_kube_features.go
+++ b/pkg/features/versioned_kube_features.go
@@ -828,4 +828,7 @@ var defaultVersionedKubernetesFeatureGates = map[featuregate.Feature]featuregate
 	zpagesfeatures.ComponentStatusz: {
 		{Version: version.MustParse("1.32"), Default: false, PreRelease: featuregate.Alpha},
 	},
+	DisableCPUQuotaWithExclusiveCPUs: {
+		{Version: version.MustParse("1.33"), Default: true, PreRelease: featuregate.Beta},
+	},
 }
diff --git a/pkg/kubelet/cm/container_manager.go b/pkg/kubelet/cm/container_manager.go
index 5f4b53dd0d2f2..0f4a20807531b 100644
--- a/pkg/kubelet/cm/container_manager.go
+++ b/pkg/kubelet/cm/container_manager.go
@@ -31,6 +31,7 @@ import (
 	v1 "k8s.io/api/core/v1"
 	"k8s.io/apiserver/pkg/server/healthz"
 	internalapi "k8s.io/cri-api/pkg/apis"
+	"k8s.io/klog/v2"
 	podresourcesapi "k8s.io/kubelet/pkg/apis/podresources/v1"
 	kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config"
 	"k8s.io/kubernetes/pkg/kubelet/apis/podresources"
@@ -154,6 +155,13 @@ type ContainerManager interface {
 	// Updates returns a channel that receives an Update when the device changed its status.
 	Updates() <-chan resourceupdates.Update
 
+	// PodHasExclusiveCPUs returns true if the provided pod has containers with exclusive CPUs,
+	// This means that at least one sidecar container or one app container has exclusive CPUs allocated.
+	PodHasExclusiveCPUs(pod *v1.Pod) bool
+
+	// ContainerHasExclusiveCPUs returns true if the provided container in the pod has exclusive cpu
+	ContainerHasExclusiveCPUs(pod *v1.Pod, container *v1.Container) bool
+
 	// Implements the PodResources Provider API
 	podresources.CPUsProvider
 	podresources.DevicesProvider
@@ -161,6 +169,10 @@ type ContainerManager interface {
 	podresources.DynamicResourcesProvider
 }
 
+type cpuAllocationReader interface {
+	GetExclusiveCPUs(podUID, containerName string) cpuset.CPUSet
+}
+
 type NodeConfig struct {
 	NodeName              types.NodeName
 	RuntimeCgroupsName    string
@@ -212,6 +224,30 @@ func int64Slice(in []int) []int64 {
 	return out
 }
 
+func podHasExclusiveCPUs(cr cpuAllocationReader, pod *v1.Pod) bool {
+	for _, container := range pod.Spec.InitContainers {
+		if containerHasExclusiveCPUs(cr, pod, &container) {
+			return true
+		}
+	}
+	for _, container := range pod.Spec.Containers {
+		if containerHasExclusiveCPUs(cr, pod, &container) {
+			return true
+		}
+	}
+	klog.V(4).InfoS("Pod contains no container with pinned cpus", "podName", pod.Name)
+	return false
+}
+
+func containerHasExclusiveCPUs(cr cpuAllocationReader, pod *v1.Pod, container *v1.Container) bool {
+	exclusiveCPUs := cr.GetExclusiveCPUs(string(pod.UID), container.Name)
+	if !exclusiveCPUs.IsEmpty() {
+		klog.V(4).InfoS("Container has pinned cpus", "podName", pod.Name, "containerName", container.Name)
+		return true
+	}
+	return false
+}
+
 // parsePercentage parses the percentage string to numeric value.
 func parsePercentage(v string) (int64, error) {
 	if !strings.HasSuffix(v, "%") {
diff --git a/pkg/kubelet/cm/container_manager_linux.go b/pkg/kubelet/cm/container_manager_linux.go
index 93d9acfc4b864..434323421eac2 100644
--- a/pkg/kubelet/cm/container_manager_linux.go
+++ b/pkg/kubelet/cm/container_manager_linux.go
@@ -365,7 +365,8 @@ func (cm *containerManagerImpl) NewPodContainerManager() PodContainerManager {
 			enforceCPULimits:  cm.EnforceCPULimits,
 			// cpuCFSQuotaPeriod is in microseconds. NodeConfig.CPUCFSQuotaPeriod is time.Duration (measured in nano seconds).
 			// Convert (cm.CPUCFSQuotaPeriod) [nanoseconds] / time.Microsecond (1000) to get cpuCFSQuotaPeriod in microseconds.
-			cpuCFSQuotaPeriod: uint64(cm.CPUCFSQuotaPeriod / time.Microsecond),
+			cpuCFSQuotaPeriod:   uint64(cm.CPUCFSQuotaPeriod / time.Microsecond),
+			podContainerManager: cm,
 		}
 	}
 	return &podContainerManagerNoop{
@@ -373,6 +374,14 @@ func (cm *containerManagerImpl) NewPodContainerManager() PodContainerManager {
 	}
 }
 
+func (cm *containerManagerImpl) PodHasExclusiveCPUs(pod *v1.Pod) bool {
+	return podHasExclusiveCPUs(cm.cpuManager, pod)
+}
+
+func (cm *containerManagerImpl) ContainerHasExclusiveCPUs(pod *v1.Pod, container *v1.Container) bool {
+	return containerHasExclusiveCPUs(cm.cpuManager, pod, container)
+}
+
 func (cm *containerManagerImpl) InternalContainerLifecycle() InternalContainerLifecycle {
 	return &internalContainerLifecycleImpl{cm.cpuManager, cm.memoryManager, cm.topologyManager}
 }
diff --git a/pkg/kubelet/cm/container_manager_stub.go b/pkg/kubelet/cm/container_manager_stub.go
index e117a7377d787..429ccf0b495d3 100644
--- a/pkg/kubelet/cm/container_manager_stub.go
+++ b/pkg/kubelet/cm/container_manager_stub.go
@@ -195,6 +195,14 @@ func (cm *containerManagerStub) Updates() <-chan resourceupdates.Update {
 	return nil
 }
 
+func (cm *containerManagerStub) PodHasExclusiveCPUs(pod *v1.Pod) bool {
+	return false
+}
+
+func (cm *containerManagerStub) ContainerHasExclusiveCPUs(pod *v1.Pod, container *v1.Container) bool {
+	return false
+}
+
 func NewStubContainerManager() ContainerManager {
 	return &containerManagerStub{shouldResetExtendedResourceCapacity: false}
 }
diff --git a/pkg/kubelet/cm/container_manager_windows.go b/pkg/kubelet/cm/container_manager_windows.go
index 524ba4c259820..d3e080614fd42 100644
--- a/pkg/kubelet/cm/container_manager_windows.go
+++ b/pkg/kubelet/cm/container_manager_windows.go
@@ -369,3 +369,11 @@ func (cm *containerManagerImpl) UnprepareDynamicResources(ctx context.Context, p
 func (cm *containerManagerImpl) PodMightNeedToUnprepareResources(UID types.UID) bool {
 	return false
 }
+
+func (cm *containerManagerImpl) PodHasExclusiveCPUs(pod *v1.Pod) bool {
+	return podHasExclusiveCPUs(cm.cpuManager, pod)
+}
+
+func (cm *containerManagerImpl) ContainerHasExclusiveCPUs(pod *v1.Pod, container *v1.Container) bool {
+	return containerHasExclusiveCPUs(cm.cpuManager, pod, container)
+}
diff --git a/pkg/kubelet/cm/fake_container_manager.go b/pkg/kubelet/cm/fake_container_manager.go
index 63f974c10cec3..dee9ccc22fe62 100644
--- a/pkg/kubelet/cm/fake_container_manager.go
+++ b/pkg/kubelet/cm/fake_container_manager.go
@@ -268,3 +268,11 @@ func (cm *FakeContainerManager) UpdateAllocatedResourcesStatus(pod *v1.Pod, stat
 func (cm *FakeContainerManager) Updates() <-chan resourceupdates.Update {
 	return nil
 }
+
+func (cm *FakeContainerManager) PodHasExclusiveCPUs(pod *v1.Pod) bool {
+	return false
+}
+
+func (cm *FakeContainerManager) ContainerHasExclusiveCPUs(pod *v1.Pod, container *v1.Container) bool {
+	return false
+}
diff --git a/pkg/kubelet/cm/pod_container_manager_linux.go b/pkg/kubelet/cm/pod_container_manager_linux.go
index c67f0cd4842c9..d4c344c712632 100644
--- a/pkg/kubelet/cm/pod_container_manager_linux.go
+++ b/pkg/kubelet/cm/pod_container_manager_linux.go
@@ -55,6 +55,8 @@ type podContainerManagerImpl struct {
 	// cpuCFSQuotaPeriod is the cfs period value, cfs_period_us, setting per
 	// node for all containers in usec
 	cpuCFSQuotaPeriod uint64
+	// podContainerManager is the ContainerManager running on the machine
+	podContainerManager ContainerManager
 }
 
 // Make sure that podContainerManagerImpl implements the PodContainerManager interface
@@ -73,6 +75,11 @@ func (m *podContainerManagerImpl) EnsureExists(pod *v1.Pod) error {
 	// check if container already exist
 	alreadyExists := m.Exists(pod)
 	if !alreadyExists {
+		enforceCPULimits := m.enforceCPULimits
+		if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.DisableCPUQuotaWithExclusiveCPUs) && m.podContainerManager.PodHasExclusiveCPUs(pod) {
+			klog.V(2).InfoS("Disabled CFS quota", "pod", klog.KObj(pod))
+			enforceCPULimits = false
+		}
 		enforceMemoryQoS := false
 		if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.MemoryQoS) &&
 			libcontainercgroups.IsCgroup2UnifiedMode() {
@@ -82,7 +89,7 @@ func (m *podContainerManagerImpl) EnsureExists(pod *v1.Pod) error {
 		podContainerName, _ := m.GetPodContainerName(pod)
 		containerConfig := &CgroupConfig{
 			Name:               podContainerName,
-			ResourceParameters: ResourceConfigForPod(pod, m.enforceCPULimits, m.cpuCFSQuotaPeriod, enforceMemoryQoS),
+			ResourceParameters: ResourceConfigForPod(pod, enforceCPULimits, m.cpuCFSQuotaPeriod, enforceMemoryQoS),
 		}
 		if m.podPidsLimit > 0 {
 			containerConfig.ResourceParameters.PidsLimit = &m.podPidsLimit
diff --git a/pkg/kubelet/kuberuntime/fake_kuberuntime_manager.go b/pkg/kubelet/kuberuntime/fake_kuberuntime_manager.go
index 80e8e53832f34..5c125f175724b 100644
--- a/pkg/kubelet/kuberuntime/fake_kuberuntime_manager.go
+++ b/pkg/kubelet/kuberuntime/fake_kuberuntime_manager.go
@@ -108,6 +108,7 @@ func newFakeKubeRuntimeManager(runtimeService internalapi.RuntimeService, imageS
 		startupManager:         proberesults.NewManager(),
 		machineInfo:            machineInfo,
 		osInterface:            osInterface,
+		containerManager:       cm.NewFakeContainerManager(),
 		runtimeHelper:          runtimeHelper,
 		runtimeService:         runtimeService,
 		imageService:           imageService,
diff --git a/pkg/kubelet/kuberuntime/kuberuntime_container_linux.go b/pkg/kubelet/kuberuntime/kuberuntime_container_linux.go
index 20d7985f365a4..de32e864a5211 100644
--- a/pkg/kubelet/kuberuntime/kuberuntime_container_linux.go
+++ b/pkg/kubelet/kuberuntime/kuberuntime_container_linux.go
@@ -133,7 +133,12 @@ func (m *kubeGenericRuntimeManager) generateLinuxContainerResources(pod *v1.Pod,
 
 	memoryLimit := getMemoryLimit(pod, container)
 	cpuLimit := getCPULimit(pod, container)
-	lcr := m.calculateLinuxResources(cpuRequest, cpuLimit, memoryLimit)
+
+	// If pod has exclusive cpu and the container in question has integer cpu requests
+	// the cfs quota will not be enforced
+	disableCPUQuota := utilfeature.DefaultFeatureGate.Enabled(kubefeatures.DisableCPUQuotaWithExclusiveCPUs) && m.containerManager.ContainerHasExclusiveCPUs(pod, container)
+	klog.V(2).InfoS("Enforcing CFS quota", "pod", klog.KObj(pod), "unlimited", disableCPUQuota)
+	lcr := m.calculateLinuxResources(cpuRequest, cpuLimit, memoryLimit, disableCPUQuota)
 
 	lcr.OomScoreAdj = int64(qos.GetContainerOOMScoreAdjust(pod, container,
 		int64(m.machineInfo.MemoryCapacity)))
@@ -244,7 +249,7 @@ func (m *kubeGenericRuntimeManager) generateContainerResources(pod *v1.Pod, cont
 }
 
 // calculateLinuxResources will create the linuxContainerResources type based on the provided CPU and memory resource requests, limits
-func (m *kubeGenericRuntimeManager) calculateLinuxResources(cpuRequest, cpuLimit, memoryLimit *resource.Quantity) *runtimeapi.LinuxContainerResources {
+func (m *kubeGenericRuntimeManager) calculateLinuxResources(cpuRequest, cpuLimit, memoryLimit *resource.Quantity, disableCPUQuota bool) *runtimeapi.LinuxContainerResources {
 	resources := runtimeapi.LinuxContainerResources{}
 	var cpuShares int64
 
@@ -276,6 +281,9 @@ func (m *kubeGenericRuntimeManager) calculateLinuxResources(cpuRequest, cpuLimit
 		}
 		cpuQuota := milliCPUToQuota(cpuLimit.MilliValue(), cpuPeriod)
 		resources.CpuQuota = cpuQuota
+		if disableCPUQuota {
+			resources.CpuQuota = int64(-1)
+		}
 		resources.CpuPeriod = cpuPeriod
 	}
 
diff --git a/pkg/kubelet/kuberuntime/kuberuntime_manager.go b/pkg/kubelet/kuberuntime/kuberuntime_manager.go
index 8111dd5d72d38..3f9ecb6c1bed3 100644
--- a/pkg/kubelet/kuberuntime/kuberuntime_manager.go
+++ b/pkg/kubelet/kuberuntime/kuberuntime_manager.go
@@ -675,7 +675,12 @@ func (m *kubeGenericRuntimeManager) computePodResizeAction(pod *v1.Pod, containe
 func (m *kubeGenericRuntimeManager) doPodResizeAction(pod *v1.Pod, podContainerChanges podActions, result *kubecontainer.PodSyncResult) {
 	pcm := m.containerManager.NewPodContainerManager()
 	//TODO(vinaykul,InPlacePodVerticalScaling): Figure out best way to get enforceMemoryQoS value (parameter #4 below) in platform-agnostic way
-	podResources := cm.ResourceConfigForPod(pod, m.cpuCFSQuota, uint64((m.cpuCFSQuotaPeriod.Duration)/time.Microsecond), false)
+	enforceCPULimits := m.cpuCFSQuota
+	if utilfeature.DefaultFeatureGate.Enabled(features.DisableCPUQuotaWithExclusiveCPUs) && m.containerManager.PodHasExclusiveCPUs(pod) {
+		enforceCPULimits = false
+		klog.V(2).InfoS("Disabled CFS quota", "pod", klog.KObj(pod))
+	}
+	podResources := cm.ResourceConfigForPod(pod, enforceCPULimits, uint64((m.cpuCFSQuotaPeriod.Duration)/time.Microsecond), false)
 	if podResources == nil {
 		klog.ErrorS(nil, "Unable to get resource configuration", "pod", pod.Name)
 		result.Fail(fmt.Errorf("unable to get resource configuration processing resize for pod %s", pod.Name))
diff --git a/pkg/kubelet/kuberuntime/kuberuntime_sandbox_linux.go b/pkg/kubelet/kuberuntime/kuberuntime_sandbox_linux.go
index 6345e3af0a086..ebf8d4e6204db 100644
--- a/pkg/kubelet/kuberuntime/kuberuntime_sandbox_linux.go
+++ b/pkg/kubelet/kuberuntime/kuberuntime_sandbox_linux.go
@@ -24,6 +24,7 @@ import (
 	"k8s.io/apimachinery/pkg/api/resource"
 	utilfeature "k8s.io/apiserver/pkg/util/feature"
 	runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1"
+	"k8s.io/klog/v2"
 	"k8s.io/kubernetes/pkg/features"
 
 	resourcehelper "k8s.io/component-helpers/resource"
@@ -37,7 +38,7 @@ func (m *kubeGenericRuntimeManager) convertOverheadToLinuxResources(pod *v1.Pod)
 
 		// For overhead, we do not differentiate between requests and limits. Treat this overhead
 		// as "guaranteed", with requests == limits
-		resources = m.calculateLinuxResources(cpu, cpu, memory)
+		resources = m.calculateLinuxResources(cpu, cpu, memory, false)
 	}
 
 	return resources
@@ -55,7 +56,12 @@ func (m *kubeGenericRuntimeManager) calculateSandboxResources(pod *v1.Pod) *runt
 	if _, cpuRequestExists := req[v1.ResourceCPU]; cpuRequestExists {
 		cpuRequest = req.Cpu()
 	}
-	return m.calculateLinuxResources(cpuRequest, lim.Cpu(), lim.Memory())
+
+	// If pod has exclusive cpu the sandbox will not have cfs quote enforced
+	disableCPUQuota := utilfeature.DefaultFeatureGate.Enabled(features.DisableCPUQuotaWithExclusiveCPUs) && m.containerManager.PodHasExclusiveCPUs(pod)
+	klog.V(2).InfoS("Enforcing CFS quota", "pod", klog.KObj(pod), "unlimited", disableCPUQuota)
+
+	return m.calculateLinuxResources(cpuRequest, lim.Cpu(), lim.Memory(), disableCPUQuota)
 }
 
 func (m *kubeGenericRuntimeManager) applySandboxResources(pod *v1.Pod, config *runtimeapi.PodSandboxConfig) error {


Test patch:
diff --git a/pkg/kubelet/cm/testing/mock_container_manager.go b/pkg/kubelet/cm/testing/mock_container_manager.go
index ef93f5df41c14..03fc5a4ca4258 100644
--- a/pkg/kubelet/cm/testing/mock_container_manager.go
+++ b/pkg/kubelet/cm/testing/mock_container_manager.go
@@ -28,8 +28,6 @@ import (
 
 	context "context"
 
-	corev1 "k8s.io/api/core/v1"
-
 	cri "k8s.io/cri-api/pkg/apis"
 
 	framework "k8s.io/kubernetes/pkg/scheduler/framework"
@@ -40,13 +38,15 @@ import (
 
 	mock "github.com/stretchr/testify/mock"
 
+	podresourcesv1 "k8s.io/kubelet/pkg/apis/podresources/v1"
+
 	resourceupdates "k8s.io/kubernetes/pkg/kubelet/cm/resourceupdates"
 
 	status "k8s.io/kubernetes/pkg/kubelet/status"
 
 	types "k8s.io/apimachinery/pkg/types"
 
-	v1 "k8s.io/kubelet/pkg/apis/podresources/v1"
+	v1 "k8s.io/api/core/v1"
 )
 
 // MockContainerManager is an autogenerated mock type for the ContainerManager type
@@ -62,6 +62,53 @@ func (_m *MockContainerManager) EXPECT() *MockContainerManager_Expecter {
 	return &MockContainerManager_Expecter{mock: &_m.Mock}
 }
 
+// ContainerHasExclusiveCPUs provides a mock function with given fields: pod, _a1
+func (_m *MockContainerManager) ContainerHasExclusiveCPUs(pod *v1.Pod, _a1 *v1.Container) bool {
+	ret := _m.Called(pod, _a1)
+
+	if len(ret) == 0 {
+		panic("no return value specified for ContainerHasExclusiveCPUs")
+	}
+
+	var r0 bool
+	if rf, ok := ret.Get(0).(func(*v1.Pod, *v1.Container) bool); ok {
+		r0 = rf(pod, _a1)
+	} else {
+		r0 = ret.Get(0).(bool)
+	}
+
+	return r0
+}
+
+// MockContainerManager_ContainerHasExclusiveCPUs_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ContainerHasExclusiveCPUs'
+type MockContainerManager_ContainerHasExclusiveCPUs_Call struct {
+	*mock.Call
+}
+
+// ContainerHasExclusiveCPUs is a helper method to define mock.On call
+//   - pod *v1.Pod
+//   - _a1 *v1.Container
+func (_e *MockContainerManager_Expecter) ContainerHasExclusiveCPUs(pod interface{}, _a1 interface{}) *MockContainerManager_ContainerHasExclusiveCPUs_Call {
+	return &MockContainerManager_ContainerHasExclusiveCPUs_Call{Call: _e.mock.On("ContainerHasExclusiveCPUs", pod, _a1)}
+}
+
+func (_c *MockContainerManager_ContainerHasExclusiveCPUs_Call) Run(run func(pod *v1.Pod, _a1 *v1.Container)) *MockContainerManager_ContainerHasExclusiveCPUs_Call {
+	_c.Call.Run(func(args mock.Arguments) {
+		run(args[0].(*v1.Pod), args[1].(*v1.Container))
+	})
+	return _c
+}
+
+func (_c *MockContainerManager_ContainerHasExclusiveCPUs_Call) Return(_a0 bool) *MockContainerManager_ContainerHasExclusiveCPUs_Call {
+	_c.Call.Return(_a0)
+	return _c
+}
+
+func (_c *MockContainerManager_ContainerHasExclusiveCPUs_Call) RunAndReturn(run func(*v1.Pod, *v1.Container) bool) *MockContainerManager_ContainerHasExclusiveCPUs_Call {
+	_c.Call.Return(run)
+	return _c
+}
+
 // GetAllocatableCPUs provides a mock function with given fields:
 func (_m *MockContainerManager) GetAllocatableCPUs() []int64 {
 	ret := _m.Called()
@@ -110,19 +157,19 @@ func (_c *MockContainerManager_GetAllocatableCPUs_Call) RunAndReturn(run func()
 }
 
 // GetAllocatableDevices provides a mock function with given fields:
-func (_m *MockContainerManager) GetAllocatableDevices() []*v1.ContainerDevices {
+func (_m *MockContainerManager) GetAllocatableDevices() []*podresourcesv1.ContainerDevices {
 	ret := _m.Called()
 
 	if len(ret) == 0 {
 		panic("no return value specified for GetAllocatableDevices")
 	}
 
-	var r0 []*v1.ContainerDevices
-	if rf, ok := ret.Get(0).(func() []*v1.ContainerDevices); ok {
+	var r0 []*podresourcesv1.ContainerDevices
+	if rf, ok := ret.Get(0).(func() []*podresourcesv1.ContainerDevices); ok {
 		r0 = rf()
 	} else {
 		if ret.Get(0) != nil {
-			r0 = ret.Get(0).([]*v1.ContainerDevices)
+			r0 = ret.Get(0).([]*podresourcesv1.ContainerDevices)
 		}
 	}
 
@@ -146,30 +193,30 @@ func (_c *MockContainerManager_GetAllocatableDevices_Call) Run(run func()) *Mock
 	return _c
 }
 
-func (_c *MockContainerManager_GetAllocatableDevices_Call) Return(_a0 []*v1.ContainerDevices) *MockContainerManager_GetAllocatableDevices_Call {
+func (_c *MockContainerManager_GetAllocatableDevices_Call) Return(_a0 []*podresourcesv1.ContainerDevices) *MockContainerManager_GetAllocatableDevices_Call {
 	_c.Call.Return(_a0)
 	return _c
 }
 
-func (_c *MockContainerManager_GetAllocatableDevices_Call) RunAndReturn(run func() []*v1.ContainerDevices) *MockContainerManager_GetAllocatableDevices_Call {
+func (_c *MockContainerManager_GetAllocatableDevices_Call) RunAndReturn(run func() []*podresourcesv1.ContainerDevices) *MockContainerManager_GetAllocatableDevices_Call {
 	_c.Call.Return(run)
 	return _c
 }
 
 // GetAllocatableMemory provides a mock function with given fields:
-func (_m *MockContainerManager) GetAllocatableMemory() []*v1.ContainerMemory {
+func (_m *MockContainerManager) GetAllocatableMemory() []*podresourcesv1.ContainerMemory {
 	ret := _m.Called()
 
 	if len(ret) == 0 {
 		panic("no return value specified for GetAllocatableMemory")
 	}
 
-	var r0 []*v1.ContainerMemory
-	if rf, ok := ret.Get(0).(func() []*v1.ContainerMemory); ok {
+	var r0 []*podresourcesv1.ContainerMemory
+	if rf, ok := ret.Get(0).(func() []*podresourcesv1.ContainerMemory); ok {
 		r0 = rf()
 	} else {
 		if ret.Get(0) != nil {
-			r0 = ret.Get(0).([]*v1.ContainerMemory)
+			r0 = ret.Get(0).([]*podresourcesv1.ContainerMemory)
 		}
 	}
 
@@ -193,12 +240,12 @@ func (_c *MockContainerManager_GetAllocatableMemory_Call) Run(run func()) *MockC
 	return _c
 }
 
-func (_c *MockContainerManager_GetAllocatableMemory_Call) Return(_a0 []*v1.ContainerMemory) *MockContainerManager_GetAllocatableMemory_Call {
+func (_c *MockContainerManager_GetAllocatableMemory_Call) Return(_a0 []*podresourcesv1.ContainerMemory) *MockContainerManager_GetAllocatableMemory_Call {
 	_c.Call.Return(_a0)
 	return _c
 }
 
-func (_c *MockContainerManager_GetAllocatableMemory_Call) RunAndReturn(run func() []*v1.ContainerMemory) *MockContainerManager_GetAllocatableMemory_Call {
+func (_c *MockContainerManager_GetAllocatableMemory_Call) RunAndReturn(run func() []*podresourcesv1.ContainerMemory) *MockContainerManager_GetAllocatableMemory_Call {
 	_c.Call.Return(run)
 	return _c
 }
@@ -300,19 +347,19 @@ func (_c *MockContainerManager_GetCPUs_Call) RunAndReturn(run func(string, strin
 }
 
 // GetCapacity provides a mock function with given fields: localStorageCapacityIsolation
-func (_m *MockContainerManager) GetCapacity(localStorageCapacityIsolation bool) corev1.ResourceList {
+func (_m *MockContainerManager) GetCapacity(localStorageCapacityIsolation bool) v1.ResourceList {
 	ret := _m.Called(localStorageCapacityIsolation)
 
 	if len(ret) == 0 {
 		panic("no return value specified for GetCapacity")
 	}
 
-	var r0 corev1.ResourceList
-	if rf, ok := ret.Get(0).(func(bool) corev1.ResourceList); ok {
+	var r0 v1.ResourceList
+	if rf, ok := ret.Get(0).(func(bool) v1.ResourceList); ok {
 		r0 = rf(localStorageCapacityIsolation)
 	} else {
 		if ret.Get(0) != nil {
-			r0 = ret.Get(0).(corev1.ResourceList)
+			r0 = ret.Get(0).(v1.ResourceList)
 		}
 	}
 
@@ -337,43 +384,43 @@ func (_c *MockContainerManager_GetCapacity_Call) Run(run func(localStorageCapaci
 	return _c
 }
 
-func (_c *MockContainerManager_GetCapacity_Call) Return(_a0 corev1.ResourceList) *MockContainerManager_GetCapacity_Call {
+func (_c *MockContainerManager_GetCapacity_Call) Return(_a0 v1.ResourceList) *MockContainerManager_GetCapacity_Call {
 	_c.Call.Return(_a0)
 	return _c
 }
 
-func (_c *MockContainerManager_GetCapacity_Call) RunAndReturn(run func(bool) corev1.ResourceList) *MockContainerManager_GetCapacity_Call {
+func (_c *MockContainerManager_GetCapacity_Call) RunAndReturn(run func(bool) v1.ResourceList) *MockContainerManager_GetCapacity_Call {
 	_c.Call.Return(run)
 	return _c
 }
 
 // GetDevicePluginResourceCapacity provides a mock function with given fields:
-func (_m *MockContainerManager) GetDevicePluginResourceCapacity() (corev1.ResourceList, corev1.ResourceList, []string) {
+func (_m *MockContainerManager) GetDevicePluginResourceCapacity() (v1.ResourceList, v1.ResourceList, []string) {
 	ret := _m.Called()
 
 	if len(ret) == 0 {
 		panic("no return value specified for GetDevicePluginResourceCapacity")
 	}
 
-	var r0 corev1.ResourceList
-	var r1 corev1.ResourceList
+	var r0 v1.ResourceList
+	var r1 v1.ResourceList
 	var r2 []string
-	if rf, ok := ret.Get(0).(func() (corev1.ResourceList, corev1.ResourceList, []string)); ok {
+	if rf, ok := ret.Get(0).(func() (v1.ResourceList, v1.ResourceList, []string)); ok {
 		return rf()
 	}
-	if rf, ok := ret.Get(0).(func() corev1.ResourceList); ok {
+	if rf, ok := ret.Get(0).(func() v1.ResourceList); ok {
 		r0 = rf()
 	} else {
 		if ret.Get(0) != nil {
-			r0 = ret.Get(0).(corev1.ResourceList)
+			r0 = ret.Get(0).(v1.ResourceList)
 		}
 	}
 
-	if rf, ok := ret.Get(1).(func() corev1.ResourceList); ok {
+	if rf, ok := ret.Get(1).(func() v1.ResourceList); ok {
 		r1 = rf()
 	} else {
 		if ret.Get(1) != nil {
-			r1 = ret.Get(1).(corev1.ResourceList)
+			r1 = ret.Get(1).(v1.ResourceList)
 		}
 	}
 
@@ -405,30 +452,30 @@ func (_c *MockContainerManager_GetDevicePluginResourceCapacity_Call) Run(run fun
 	return _c
 }
 
-func (_c *MockContainerManager_GetDevicePluginResourceCapacity_Call) Return(_a0 corev1.ResourceList, _a1 corev1.ResourceList, _a2 []string) *MockContainerManager_GetDevicePluginResourceCapacity_Call {
+func (_c *MockContainerManager_GetDevicePluginResourceCapacity_Call) Return(_a0 v1.ResourceList, _a1 v1.ResourceList, _a2 []string) *MockContainerManager_GetDevicePluginResourceCapacity_Call {
 	_c.Call.Return(_a0, _a1, _a2)
 	return _c
 }
 
-func (_c *MockContainerManager_GetDevicePluginResourceCapacity_Call) RunAndReturn(run func() (corev1.ResourceList, corev1.ResourceList, []string)) *MockContainerManager_GetDevicePluginResourceCapacity_Call {
+func (_c *MockContainerManager_GetDevicePluginResourceCapacity_Call) RunAndReturn(run func() (v1.ResourceList, v1.ResourceList, []string)) *MockContainerManager_GetDevicePluginResourceCapacity_Call {
 	_c.Call.Return(run)
 	return _c
 }
 
 // GetDevices provides a mock function with given fields: podUID, containerName
-func (_m *MockContainerManager) GetDevices(podUID string, containerName string) []*v1.ContainerDevices {
+func (_m *MockContainerManager) GetDevices(podUID string, containerName string) []*podresourcesv1.ContainerDevices {
 	ret := _m.Called(podUID, containerName)
 
 	if len(ret) == 0 {
 		panic("no return value specified for GetDevices")
 	}
 
-	var r0 []*v1.ContainerDevices
-	if rf, ok := ret.Get(0).(func(string, string) []*v1.ContainerDevices); ok {
+	var r0 []*podresourcesv1.ContainerDevices
+	if rf, ok := ret.Get(0).(func(string, string) []*podresourcesv1.ContainerDevices); ok {
 		r0 = rf(podUID, containerName)
 	} else {
 		if ret.Get(0) != nil {
-			r0 = ret.Get(0).([]*v1.ContainerDevices)
+			r0 = ret.Get(0).([]*podresourcesv1.ContainerDevices)
 		}
 	}
 
@@ -454,30 +501,30 @@ func (_c *MockContainerManager_GetDevices_Call) Run(run func(podUID string, cont
 	return _c
 }
 
-func (_c *MockContainerManager_GetDevices_Call) Return(_a0 []*v1.ContainerDevices) *MockContainerManager_GetDevices_Call {
+func (_c *MockContainerManager_GetDevices_Call) Return(_a0 []*podresourcesv1.ContainerDevices) *MockContainerManager_GetDevices_Call {
 	_c.Call.Return(_a0)
 	return _c
 }
 
-func (_c *MockContainerManager_GetDevices_Call) RunAndReturn(run func(string, string) []*v1.ContainerDevices) *MockContainerManager_GetDevices_Call {
+func (_c *MockContainerManager_GetDevices_Call) RunAndReturn(run func(string, string) []*podresourcesv1.ContainerDevices) *MockContainerManager_GetDevices_Call {
 	_c.Call.Return(run)
 	return _c
 }
 
 // GetDynamicResources provides a mock function with given fields: pod, _a1
-func (_m *MockContainerManager) GetDynamicResources(pod *corev1.Pod, _a1 *corev1.Container) []*v1.DynamicResource {
+func (_m *MockContainerManager) GetDynamicResources(pod *v1.Pod, _a1 *v1.Container) []*podresourcesv1.DynamicResource {
 	ret := _m.Called(pod, _a1)
 
 	if len(ret) == 0 {
 		panic("no return value specified for GetDynamicResources")
 	}
 
-	var r0 []*v1.DynamicResource
-	if rf, ok := ret.Get(0).(func(*corev1.Pod, *corev1.Container) []*v1.DynamicResource); ok {
+	var r0 []*podresourcesv1.DynamicResource
+	if rf, ok := ret.Get(0).(func(*v1.Pod, *v1.Container) []*podresourcesv1.DynamicResource); ok {
 		r0 = rf(pod, _a1)
 	} else {
 		if ret.Get(0) != nil {
-			r0 = ret.Get(0).([]*v1.DynamicResource)
+			r0 = ret.Get(0).([]*podresourcesv1.DynamicResource)
 		}
 	}
 
@@ -490,25 +537,25 @@ type MockContainerManager_GetDynamicResources_Call struct {
 }
 
 // GetDynamicResources is a helper method to define mock.On call
-//   - pod *corev1.Pod
-//   - _a1 *corev1.Container
+//   - pod *v1.Pod
+//   - _a1 *v1.Container
 func (_e *MockContainerManager_Expecter) GetDynamicResources(pod interface{}, _a1 interface{}) *MockContainerManager_GetDynamicResources_Call {
 	return &MockContainerManager_GetDynamicResources_Call{Call: _e.mock.On("GetDynamicResources", pod, _a1)}
 }
 
-func (_c *MockContainerManager_GetDynamicResources_Call) Run(run func(pod *corev1.Pod, _a1 *corev1.Container)) *MockContainerManager_GetDynamicResources_Call {
+func (_c *MockContainerManager_GetDynamicResources_Call) Run(run func(pod *v1.Pod, _a1 *v1.Container)) *MockContainerManager_GetDynamicResources_Call {
 	_c.Call.Run(func(args mock.Arguments) {
-		run(args[0].(*corev1.Pod), args[1].(*corev1.Container))
+		run(args[0].(*v1.Pod), args[1].(*v1.Container))
 	})
 	return _c
 }
 
-func (_c *MockContainerManager_GetDynamicResources_Call) Return(_a0 []*v1.DynamicResource) *MockContainerManager_GetDynamicResources_Call {
+func (_c *MockContainerManager_GetDynamicResources_Call) Return(_a0 []*podresourcesv1.DynamicResource) *MockContainerManager_GetDynamicResources_Call {
 	_c.Call.Return(_a0)
 	return _c
 }
 
-func (_c *MockContainerManager_GetDynamicResources_Call) RunAndReturn(run func(*corev1.Pod, *corev1.Container) []*v1.DynamicResource) *MockContainerManager_GetDynamicResources_Call {
+func (_c *MockContainerManager_GetDynamicResources_Call) RunAndReturn(run func(*v1.Pod, *v1.Container) []*podresourcesv1.DynamicResource) *MockContainerManager_GetDynamicResources_Call {
 	_c.Call.Return(run)
 	return _c
 }
@@ -561,19 +608,19 @@ func (_c *MockContainerManager_GetHealthCheckers_Call) RunAndReturn(run func() [
 }
 
 // GetMemory provides a mock function with given fields: podUID, containerName
-func (_m *MockContainerManager) GetMemory(podUID string, containerName string) []*v1.ContainerMemory {
+func (_m *MockContainerManager) GetMemory(podUID string, containerName string) []*podresourcesv1.ContainerMemory {
 	ret := _m.Called(podUID, containerName)
 
 	if len(ret) == 0 {
 		panic("no return value specified for GetMemory")
 	}
 
-	var r0 []*v1.ContainerMemory
-	if rf, ok := ret.Get(0).(func(string, string) []*v1.ContainerMemory); ok {
+	var r0 []*podresourcesv1.ContainerMemory
+	if rf, ok := ret.Get(0).(func(string, string) []*podresourcesv1.ContainerMemory); ok {
 		r0 = rf(podUID, containerName)
 	} else {
 		if ret.Get(0) != nil {
-			r0 = ret.Get(0).([]*v1.ContainerMemory)
+			r0 = ret.Get(0).([]*podresourcesv1.ContainerMemory)
 		}
 	}
 
@@ -599,12 +646,12 @@ func (_c *MockContainerManager_GetMemory_Call) Run(run func(podUID string, conta
 	return _c
 }
 
-func (_c *MockContainerManager_GetMemory_Call) Return(_a0 []*v1.ContainerMemory) *MockContainerManager_GetMemory_Call {
+func (_c *MockContainerManager_GetMemory_Call) Return(_a0 []*podresourcesv1.ContainerMemory) *MockContainerManager_GetMemory_Call {
 	_c.Call.Return(_a0)
 	return _c
 }
 
-func (_c *MockContainerManager_GetMemory_Call) RunAndReturn(run func(string, string) []*v1.ContainerMemory) *MockContainerManager_GetMemory_Call {
+func (_c *MockContainerManager_GetMemory_Call) RunAndReturn(run func(string, string) []*podresourcesv1.ContainerMemory) *MockContainerManager_GetMemory_Call {
 	_c.Call.Return(run)
 	return _c
 }
@@ -657,19 +704,19 @@ func (_c *MockContainerManager_GetMountedSubsystems_Call) RunAndReturn(run func(
 }
 
 // GetNodeAllocatableAbsolute provides a mock function with given fields:
-func (_m *MockContainerManager) GetNodeAllocatableAbsolute() corev1.ResourceList {
+func (_m *MockContainerManager) GetNodeAllocatableAbsolute() v1.ResourceList {
 	ret := _m.Called()
 
 	if len(ret) == 0 {
 		panic("no return value specified for GetNodeAllocatableAbsolute")
 	}
 
-	var r0 corev1.ResourceList
-	if rf, ok := ret.Get(0).(func() corev1.ResourceList); ok {
+	var r0 v1.ResourceList
+	if rf, ok := ret.Get(0).(func() v1.ResourceList); ok {
 		r0 = rf()
 	} else {
 		if ret.Get(0) != nil {
-			r0 = ret.Get(0).(corev1.ResourceList)
+			r0 = ret.Get(0).(v1.ResourceList)
 		}
 	}
 
@@ -693,30 +740,30 @@ func (_c *MockContainerManager_GetNodeAllocatableAbsolute_Call) Run(run func())
 	return _c
 }
 
-func (_c *MockContainerManager_GetNodeAllocatableAbsolute_Call) Return(_a0 corev1.ResourceList) *MockContainerManager_GetNodeAllocatableAbsolute_Call {
+func (_c *MockContainerManager_GetNodeAllocatableAbsolute_Call) Return(_a0 v1.ResourceList) *MockContainerManager_GetNodeAllocatableAbsolute_Call {
 	_c.Call.Return(_a0)
 	return _c
 }
 
-func (_c *MockContainerManager_GetNodeAllocatableAbsolute_Call) RunAndReturn(run func() corev1.ResourceList) *MockContainerManager_GetNodeAllocatableAbsolute_Call {
+func (_c *MockContainerManager_GetNodeAllocatableAbsolute_Call) RunAndReturn(run func() v1.ResourceList) *MockContainerManager_GetNodeAllocatableAbsolute_Call {
 	_c.Call.Return(run)
 	return _c
 }
 
 // GetNodeAllocatableReservation provides a mock function with given fields:
-func (_m *MockContainerManager) GetNodeAllocatableReservation() corev1.ResourceList {
+func (_m *MockContainerManager) GetNodeAllocatableReservation() v1.ResourceList {
 	ret := _m.Called()
 
 	if len(ret) == 0 {
 		panic("no return value specified for GetNodeAllocatableReservation")
 	}
 
-	var r0 corev1.ResourceList
-	if rf, ok := ret.Get(0).(func() corev1.ResourceList); ok {
+	var r0 v1.ResourceList
+	if rf, ok := ret.Get(0).(func() v1.ResourceList); ok {
 		r0 = rf()
 	} else {
 		if ret.Get(0) != nil {
-			r0 = ret.Get(0).(corev1.ResourceList)
+			r0 = ret.Get(0).(v1.ResourceList)
 		}
 	}
 
@@ -740,12 +787,12 @@ func (_c *MockContainerManager_GetNodeAllocatableReservation_Call) Run(run func(
 	return _c
 }
 
-func (_c *MockContainerManager_GetNodeAllocatableReservation_Call) Return(_a0 corev1.ResourceList) *MockContainerManager_GetNodeAllocatableReservation_Call {
+func (_c *MockContainerManager_GetNodeAllocatableReservation_Call) Return(_a0 v1.ResourceList) *MockContainerManager_GetNodeAllocatableReservation_Call {
 	_c.Call.Return(_a0)
 	return _c
 }
 
-func (_c *MockContainerManager_GetNodeAllocatableReservation_Call) RunAndReturn(run func() corev1.ResourceList) *MockContainerManager_GetNodeAllocatableReservation_Call {
+func (_c *MockContainerManager_GetNodeAllocatableReservation_Call) RunAndReturn(run func() v1.ResourceList) *MockContainerManager_GetNodeAllocatableReservation_Call {
 	_c.Call.Return(run)
 	return _c
 }
@@ -933,7 +980,7 @@ func (_c *MockContainerManager_GetQOSContainersInfo_Call) RunAndReturn(run func(
 }
 
 // GetResources provides a mock function with given fields: ctx, pod, _a2
-func (_m *MockContainerManager) GetResources(ctx context.Context, pod *corev1.Pod, _a2 *corev1.Container) (*container.RunContainerOptions, error) {
+func (_m *MockContainerManager) GetResources(ctx context.Context, pod *v1.Pod, _a2 *v1.Container) (*container.RunContainerOptions, error) {
 	ret := _m.Called(ctx, pod, _a2)
 
 	if len(ret) == 0 {
@@ -942,10 +989,10 @@ func (_m *MockContainerManager) GetResources(ctx context.Context, pod *corev1.Po
 
 	var r0 *container.RunContainerOptions
 	var r1 error
-	if rf, ok := ret.Get(0).(func(context.Context, *corev1.Pod, *corev1.Container) (*container.RunContainerOptions, error)); ok {
+	if rf, ok := ret.Get(0).(func(context.Context, *v1.Pod, *v1.Container) (*container.RunContainerOptions, error)); ok {
 		return rf(ctx, pod, _a2)
 	}
-	if rf, ok := ret.Get(0).(func(context.Context, *corev1.Pod, *corev1.Container) *container.RunContainerOptions); ok {
+	if rf, ok := ret.Get(0).(func(context.Context, *v1.Pod, *v1.Container) *container.RunContainerOptions); ok {
 		r0 = rf(ctx, pod, _a2)
 	} else {
 		if ret.Get(0) != nil {
@@ -953,7 +1000,7 @@ func (_m *MockContainerManager) GetResources(ctx context.Context, pod *corev1.Po
 		}
 	}
 
-	if rf, ok := ret.Get(1).(func(context.Context, *corev1.Pod, *corev1.Container) error); ok {
+	if rf, ok := ret.Get(1).(func(context.Context, *v1.Pod, *v1.Container) error); ok {
 		r1 = rf(ctx, pod, _a2)
 	} else {
 		r1 = ret.Error(1)
@@ -969,15 +1016,15 @@ type MockContainerManager_GetResources_Call struct {
 
 // GetResources is a helper method to define mock.On call
 //   - ctx context.Context
-//   - pod *corev1.Pod
-//   - _a2 *corev1.Container
+//   - pod *v1.Pod
+//   - _a2 *v1.Container
 func (_e *MockContainerManager_Expecter) GetResources(ctx interface{}, pod interface{}, _a2 interface{}) *MockContainerManager_GetResources_Call {
 	return &MockContainerManager_GetResources_Call{Call: _e.mock.On("GetResources", ctx, pod, _a2)}
 }
 
-func (_c *MockContainerManager_GetResources_Call) Run(run func(ctx context.Context, pod *corev1.Pod, _a2 *corev1.Container)) *MockContainerManager_GetResources_Call {
+func (_c *MockContainerManager_GetResources_Call) Run(run func(ctx context.Context, pod *v1.Pod, _a2 *v1.Container)) *MockContainerManager_GetResources_Call {
 	_c.Call.Run(func(args mock.Arguments) {
-		run(args[0].(context.Context), args[1].(*corev1.Pod), args[2].(*corev1.Container))
+		run(args[0].(context.Context), args[1].(*v1.Pod), args[2].(*v1.Container))
 	})
 	return _c
 }
@@ -987,7 +1034,7 @@ func (_c *MockContainerManager_GetResources_Call) Return(_a0 *container.RunConta
 	return _c
 }
 
-func (_c *MockContainerManager_GetResources_Call) RunAndReturn(run func(context.Context, *corev1.Pod, *corev1.Container) (*container.RunContainerOptions, error)) *MockContainerManager_GetResources_Call {
+func (_c *MockContainerManager_GetResources_Call) RunAndReturn(run func(context.Context, *v1.Pod, *v1.Container) (*container.RunContainerOptions, error)) *MockContainerManager_GetResources_Call {
 	_c.Call.Return(run)
 	return _c
 }
@@ -1086,6 +1133,52 @@ func (_c *MockContainerManager_NewPodContainerManager_Call) RunAndReturn(run fun
 	return _c
 }
 
+// PodHasExclusiveCPUs provides a mock function with given fields: pod
+func (_m *MockContainerManager) PodHasExclusiveCPUs(pod *v1.Pod) bool {
+	ret := _m.Called(pod)
+
+	if len(ret) == 0 {
+		panic("no return value specified for PodHasExclusiveCPUs")
+	}
+
+	var r0 bool
+	if rf, ok := ret.Get(0).(func(*v1.Pod) bool); ok {
+		r0 = rf(pod)
+	} else {
+		r0 = ret.Get(0).(bool)
+	}
+
+	return r0
+}
+
+// MockContainerManager_PodHasExclusiveCPUs_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'PodHasExclusiveCPUs'
+type MockContainerManager_PodHasExclusiveCPUs_Call struct {
+	*mock.Call
+}
+
+// PodHasExclusiveCPUs is a helper method to define mock.On call
+//   - pod *v1.Pod
+func (_e *MockContainerManager_Expecter) PodHasExclusiveCPUs(pod interface{}) *MockContainerManager_PodHasExclusiveCPUs_Call {
+	return &MockContainerManager_PodHasExclusiveCPUs_Call{Call: _e.mock.On("PodHasExclusiveCPUs", pod)}
+}
+
+func (_c *MockContainerManager_PodHasExclusiveCPUs_Call) Run(run func(pod *v1.Pod)) *MockContainerManager_PodHasExclusiveCPUs_Call {
+	_c.Call.Run(func(args mock.Arguments) {
+		run(args[0].(*v1.Pod))
+	})
+	return _c
+}
+
+func (_c *MockContainerManager_PodHasExclusiveCPUs_Call) Return(_a0 bool) *MockContainerManager_PodHasExclusiveCPUs_Call {
+	_c.Call.Return(_a0)
+	return _c
+}
+
+func (_c *MockContainerManager_PodHasExclusiveCPUs_Call) RunAndReturn(run func(*v1.Pod) bool) *MockContainerManager_PodHasExclusiveCPUs_Call {
+	_c.Call.Return(run)
+	return _c
+}
+
 // PodMightNeedToUnprepareResources provides a mock function with given fields: UID
 func (_m *MockContainerManager) PodMightNeedToUnprepareResources(UID types.UID) bool {
 	ret := _m.Called(UID)
@@ -1133,7 +1226,7 @@ func (_c *MockContainerManager_PodMightNeedToUnprepareResources_Call) RunAndRetu
 }
 
 // PrepareDynamicResources provides a mock function with given fields: _a0, _a1
-func (_m *MockContainerManager) PrepareDynamicResources(_a0 context.Context, _a1 *corev1.Pod) error {
+func (_m *MockContainerManager) PrepareDynamicResources(_a0 context.Context, _a1 *v1.Pod) error {
 	ret := _m.Called(_a0, _a1)
 
 	if len(ret) == 0 {
@@ -1141,7 +1234,7 @@ func (_m *MockContainerManager) PrepareDynamicResources(_a0 context.Context, _a1
 	}
 
 	var r0 error
-	if rf, ok := ret.Get(0).(func(context.Context, *corev1.Pod) error); ok {
+	if rf, ok := ret.Get(0).(func(context.Context, *v1.Pod) error); ok {
 		r0 = rf(_a0, _a1)
 	} else {
 		r0 = ret.Error(0)
@@ -1157,14 +1250,14 @@ type MockContainerManager_PrepareDynamicResources_Call struct {
 
 // PrepareDynamicResources is a helper method to define mock.On call
 //   - _a0 context.Context
-//   - _a1 *corev1.Pod
+//   - _a1 *v1.Pod
 func (_e *MockContainerManager_Expecter) PrepareDynamicResources(_a0 interface{}, _a1 interface{}) *MockContainerManager_PrepareDynamicResources_Call {
 	return &MockContainerManager_PrepareDynamicResources_Call{Call: _e.mock.On("PrepareDynamicResources", _a0, _a1)}
 }
 
-func (_c *MockContainerManager_PrepareDynamicResources_Call) Run(run func(_a0 context.Context, _a1 *corev1.Pod)) *MockContainerManager_PrepareDynamicResources_Call {
+func (_c *MockContainerManager_PrepareDynamicResources_Call) Run(run func(_a0 context.Context, _a1 *v1.Pod)) *MockContainerManager_PrepareDynamicResources_Call {
 	_c.Call.Run(func(args mock.Arguments) {
-		run(args[0].(context.Context), args[1].(*corev1.Pod))
+		run(args[0].(context.Context), args[1].(*v1.Pod))
 	})
 	return _c
 }
@@ -1174,7 +1267,7 @@ func (_c *MockContainerManager_PrepareDynamicResources_Call) Return(_a0 error) *
 	return _c
 }
 
-func (_c *MockContainerManager_PrepareDynamicResources_Call) RunAndReturn(run func(context.Context, *corev1.Pod) error) *MockContainerManager_PrepareDynamicResources_Call {
+func (_c *MockContainerManager_PrepareDynamicResources_Call) RunAndReturn(run func(context.Context, *v1.Pod) error) *MockContainerManager_PrepareDynamicResources_Call {
 	_c.Call.Return(run)
 	return _c
 }
@@ -1225,7 +1318,7 @@ func (_c *MockContainerManager_ShouldResetExtendedResourceCapacity_Call) RunAndR
 }
 
 // Start provides a mock function with given fields: _a0, _a1, _a2, _a3, _a4, _a5, _a6, _a7
-func (_m *MockContainerManager) Start(_a0 context.Context, _a1 *corev1.Node, _a2 cm.ActivePodsFunc, _a3 cm.GetNodeFunc, _a4 config.SourcesReady, _a5 status.PodStatusProvider, _a6 cri.RuntimeService, _a7 bool) error {
+func (_m *MockContainerManager) Start(_a0 context.Context, _a1 *v1.Node, _a2 cm.ActivePodsFunc, _a3 cm.GetNodeFunc, _a4 config.SourcesReady, _a5 status.PodStatusProvider, _a6 cri.RuntimeService, _a7 bool) error {
 	ret := _m.Called(_a0, _a1, _a2, _a3, _a4, _a5, _a6, _a7)
 
 	if len(ret) == 0 {
@@ -1233,7 +1326,7 @@ func (_m *MockContainerManager) Start(_a0 context.Context, _a1 *corev1.Node, _a2
 	}
 
 	var r0 error
-	if rf, ok := ret.Get(0).(func(context.Context, *corev1.Node, cm.ActivePodsFunc, cm.GetNodeFunc, config.SourcesReady, status.PodStatusProvider, cri.RuntimeService, bool) error); ok {
+	if rf, ok := ret.Get(0).(func(context.Context, *v1.Node, cm.ActivePodsFunc, cm.GetNodeFunc, config.SourcesReady, status.PodStatusProvider, cri.RuntimeService, bool) error); ok {
 		r0 = rf(_a0, _a1, _a2, _a3, _a4, _a5, _a6, _a7)
 	} else {
 		r0 = ret.Error(0)
@@ -1249,7 +1342,7 @@ type MockContainerManager_Start_Call struct {
 
 // Start is a helper method to define mock.On call
 //   - _a0 context.Context
-//   - _a1 *corev1.Node
+//   - _a1 *v1.Node
 //   - _a2 cm.ActivePodsFunc
 //   - _a3 cm.GetNodeFunc
 //   - _a4 config.SourcesReady
@@ -1260,9 +1353,9 @@ func (_e *MockContainerManager_Expecter) Start(_a0 interface{}, _a1 interface{},
 	return &MockContainerManager_Start_Call{Call: _e.mock.On("Start", _a0, _a1, _a2, _a3, _a4, _a5, _a6, _a7)}
 }
 
-func (_c *MockContainerManager_Start_Call) Run(run func(_a0 context.Context, _a1 *corev1.Node, _a2 cm.ActivePodsFunc, _a3 cm.GetNodeFunc, _a4 config.SourcesReady, _a5 status.PodStatusProvider, _a6 cri.RuntimeService, _a7 bool)) *MockContainerManager_Start_Call {
+func (_c *MockContainerManager_Start_Call) Run(run func(_a0 context.Context, _a1 *v1.Node, _a2 cm.ActivePodsFunc, _a3 cm.GetNodeFunc, _a4 config.SourcesReady, _a5 status.PodStatusProvider, _a6 cri.RuntimeService, _a7 bool)) *MockContainerManager_Start_Call {
 	_c.Call.Run(func(args mock.Arguments) {
-		run(args[0].(context.Context), args[1].(*corev1.Node), args[2].(cm.ActivePodsFunc), args[3].(cm.GetNodeFunc), args[4].(config.SourcesReady), args[5].(status.PodStatusProvider), args[6].(cri.RuntimeService), args[7].(bool))
+		run(args[0].(context.Context), args[1].(*v1.Node), args[2].(cm.ActivePodsFunc), args[3].(cm.GetNodeFunc), args[4].(config.SourcesReady), args[5].(status.PodStatusProvider), args[6].(cri.RuntimeService), args[7].(bool))
 	})
 	return _c
 }
@@ -1272,7 +1365,7 @@ func (_c *MockContainerManager_Start_Call) Return(_a0 error) *MockContainerManag
 	return _c
 }
 
-func (_c *MockContainerManager_Start_Call) RunAndReturn(run func(context.Context, *corev1.Node, cm.ActivePodsFunc, cm.GetNodeFunc, config.SourcesReady, status.PodStatusProvider, cri.RuntimeService, bool) error) *MockContainerManager_Start_Call {
+func (_c *MockContainerManager_Start_Call) RunAndReturn(run func(context.Context, *v1.Node, cm.ActivePodsFunc, cm.GetNodeFunc, config.SourcesReady, status.PodStatusProvider, cri.RuntimeService, bool) error) *MockContainerManager_Start_Call {
 	_c.Call.Return(run)
 	return _c
 }
@@ -1323,19 +1416,19 @@ func (_c *MockContainerManager_Status_Call) RunAndReturn(run func() cm.Status) *
 }
 
 // SystemCgroupsLimit provides a mock function with given fields:
-func (_m *MockContainerManager) SystemCgroupsLimit() corev1.ResourceList {
+func (_m *MockContainerManager) SystemCgroupsLimit() v1.ResourceList {
 	ret := _m.Called()
 
 	if len(ret) == 0 {
 		panic("no return value specified for SystemCgroupsLimit")
 	}
 
-	var r0 corev1.ResourceList
-	if rf, ok := ret.Get(0).(func() corev1.ResourceList); ok {
+	var r0 v1.ResourceList
+	if rf, ok := ret.Get(0).(func() v1.ResourceList); ok {
 		r0 = rf()
 	} else {
 		if ret.Get(0) != nil {
-			r0 = ret.Get(0).(corev1.ResourceList)
+			r0 = ret.Get(0).(v1.ResourceList)
 		}
 	}
 
@@ -1359,18 +1452,18 @@ func (_c *MockContainerManager_SystemCgroupsLimit_Call) Run(run func()) *MockCon
 	return _c
 }
 
-func (_c *MockContainerManager_SystemCgroupsLimit_Call) Return(_a0 corev1.ResourceList) *MockContainerManager_SystemCgroupsLimit_Call {
+func (_c *MockContainerManager_SystemCgroupsLimit_Call) Return(_a0 v1.ResourceList) *MockContainerManager_SystemCgroupsLimit_Call {
 	_c.Call.Return(_a0)
 	return _c
 }
 
-func (_c *MockContainerManager_SystemCgroupsLimit_Call) RunAndReturn(run func() corev1.ResourceList) *MockContainerManager_SystemCgroupsLimit_Call {
+func (_c *MockContainerManager_SystemCgroupsLimit_Call) RunAndReturn(run func() v1.ResourceList) *MockContainerManager_SystemCgroupsLimit_Call {
 	_c.Call.Return(run)
 	return _c
 }
 
 // UnprepareDynamicResources provides a mock function with given fields: _a0, _a1
-func (_m *MockContainerManager) UnprepareDynamicResources(_a0 context.Context, _a1 *corev1.Pod) error {
+func (_m *MockContainerManager) UnprepareDynamicResources(_a0 context.Context, _a1 *v1.Pod) error {
 	ret := _m.Called(_a0, _a1)
 
 	if len(ret) == 0 {
@@ -1378,7 +1471,7 @@ func (_m *MockContainerManager) UnprepareDynamicResources(_a0 context.Context, _
 	}
 
 	var r0 error
-	if rf, ok := ret.Get(0).(func(context.Context, *corev1.Pod) error); ok {
+	if rf, ok := ret.Get(0).(func(context.Context, *v1.Pod) error); ok {
 		r0 = rf(_a0, _a1)
 	} else {
 		r0 = ret.Error(0)
@@ -1394,14 +1487,14 @@ type MockContainerManager_UnprepareDynamicResources_Call struct {
 
 // UnprepareDynamicResources is a helper method to define mock.On call
 //   - _a0 context.Context
-//   - _a1 *corev1.Pod
+//   - _a1 *v1.Pod
 func (_e *MockContainerManager_Expecter) UnprepareDynamicResources(_a0 interface{}, _a1 interface{}) *MockContainerManager_UnprepareDynamicResources_Call {
 	return &MockContainerManager_UnprepareDynamicResources_Call{Call: _e.mock.On("UnprepareDynamicResources", _a0, _a1)}
 }
 
-func (_c *MockContainerManager_UnprepareDynamicResources_Call) Run(run func(_a0 context.Context, _a1 *corev1.Pod)) *MockContainerManager_UnprepareDynamicResources_Call {
+func (_c *MockContainerManager_UnprepareDynamicResources_Call) Run(run func(_a0 context.Context, _a1 *v1.Pod)) *MockContainerManager_UnprepareDynamicResources_Call {
 	_c.Call.Run(func(args mock.Arguments) {
-		run(args[0].(context.Context), args[1].(*corev1.Pod))
+		run(args[0].(context.Context), args[1].(*v1.Pod))
 	})
 	return _c
 }
@@ -1411,7 +1504,7 @@ func (_c *MockContainerManager_UnprepareDynamicResources_Call) Return(_a0 error)
 	return _c
 }
 
-func (_c *MockContainerManager_UnprepareDynamicResources_Call) RunAndReturn(run func(context.Context, *corev1.Pod) error) *MockContainerManager_UnprepareDynamicResources_Call {
+func (_c *MockContainerManager_UnprepareDynamicResources_Call) RunAndReturn(run func(context.Context, *v1.Pod) error) *MockContainerManager_UnprepareDynamicResources_Call {
 	_c.Call.Return(run)
 	return _c
 }
@@ -1449,7 +1542,7 @@ func (_c *MockContainerManager_UpdateAllocatedDevices_Call) RunAndReturn(run fun
 }
 
 // UpdateAllocatedResourcesStatus provides a mock function with given fields: pod, _a1
-func (_m *MockContainerManager) UpdateAllocatedResourcesStatus(pod *corev1.Pod, _a1 *corev1.PodStatus) {
+func (_m *MockContainerManager) UpdateAllocatedResourcesStatus(pod *v1.Pod, _a1 *v1.PodStatus) {
 	_m.Called(pod, _a1)
 }
 
@@ -1459,15 +1552,15 @@ type MockContainerManager_UpdateAllocatedResourcesStatus_Call struct {
 }
 
 // UpdateAllocatedResourcesStatus is a helper method to define mock.On call
-//   - pod *corev1.Pod
-//   - _a1 *corev1.PodStatus
+//   - pod *v1.Pod
+//   - _a1 *v1.PodStatus
 func (_e *MockContainerManager_Expecter) UpdateAllocatedResourcesStatus(pod interface{}, _a1 interface{}) *MockContainerManager_UpdateAllocatedResourcesStatus_Call {
 	return &MockContainerManager_UpdateAllocatedResourcesStatus_Call{Call: _e.mock.On("UpdateAllocatedResourcesStatus", pod, _a1)}
 }
 
-func (_c *MockContainerManager_UpdateAllocatedResourcesStatus_Call) Run(run func(pod *corev1.Pod, _a1 *corev1.PodStatus)) *MockContainerManager_UpdateAllocatedResourcesStatus_Call {
+func (_c *MockContainerManager_UpdateAllocatedResourcesStatus_Call) Run(run func(pod *v1.Pod, _a1 *v1.PodStatus)) *MockContainerManager_UpdateAllocatedResourcesStatus_Call {
 	_c.Call.Run(func(args mock.Arguments) {
-		run(args[0].(*corev1.Pod), args[1].(*corev1.PodStatus))
+		run(args[0].(*v1.Pod), args[1].(*v1.PodStatus))
 	})
 	return _c
 }
@@ -1477,7 +1570,7 @@ func (_c *MockContainerManager_UpdateAllocatedResourcesStatus_Call) Return() *Mo
 	return _c
 }
 
-func (_c *MockContainerManager_UpdateAllocatedResourcesStatus_Call) RunAndReturn(run func(*corev1.Pod, *corev1.PodStatus)) *MockContainerManager_UpdateAllocatedResourcesStatus_Call {
+func (_c *MockContainerManager_UpdateAllocatedResourcesStatus_Call) RunAndReturn(run func(*v1.Pod, *v1.PodStatus)) *MockContainerManager_UpdateAllocatedResourcesStatus_Call {
 	_c.Call.Return(run)
 	return _c
 }
diff --git a/pkg/kubelet/kuberuntime/kuberuntime_container_linux_test.go b/pkg/kubelet/kuberuntime/kuberuntime_container_linux_test.go
index 69960b72d288e..6bb6f606966b3 100644
--- a/pkg/kubelet/kuberuntime/kuberuntime_container_linux_test.go
+++ b/pkg/kubelet/kuberuntime/kuberuntime_container_linux_test.go
@@ -408,7 +408,7 @@ func TestCalculateLinuxResources(t *testing.T) {
 	for _, test := range tests {
 		setCgroupVersionDuringTest(test.cgroupVersion)
 		m.singleProcessOOMKill = ptr.To(test.singleProcessOOMKill)
-		linuxContainerResources := m.calculateLinuxResources(test.cpuReq, test.cpuLim, test.memLim)
+		linuxContainerResources := m.calculateLinuxResources(test.cpuReq, test.cpuLim, test.memLim, false)
 		assert.Equal(t, test.expected, linuxContainerResources)
 	}
 }
diff --git a/pkg/kubelet/kuberuntime/kuberuntime_manager_test.go b/pkg/kubelet/kuberuntime/kuberuntime_manager_test.go
index 17cf6806d9d56..9d7de2907e9ae 100644
--- a/pkg/kubelet/kuberuntime/kuberuntime_manager_test.go
+++ b/pkg/kubelet/kuberuntime/kuberuntime_manager_test.go
@@ -2951,6 +2951,8 @@ func TestDoPodResizeAction(t *testing.T) {
 	} {
 		t.Run(tc.testName, func(t *testing.T) {
 			mockCM := cmtesting.NewMockContainerManager(t)
+			mockCM.EXPECT().PodHasExclusiveCPUs(mock.Anything).Return(false).Maybe()
+			mockCM.EXPECT().ContainerHasExclusiveCPUs(mock.Anything, mock.Anything).Return(false).Maybe()
 			m.containerManager = mockCM
 			mockPCM := cmtesting.NewMockPodContainerManager(t)
 			mockCM.EXPECT().NewPodContainerManager().Return(mockPCM)
diff --git a/test/e2e_node/cpu_manager_test.go b/test/e2e_node/cpu_manager_test.go
index 4e58e06117296..00fa990269fec 100644
--- a/test/e2e_node/cpu_manager_test.go
+++ b/test/e2e_node/cpu_manager_test.go
@@ -70,6 +70,16 @@ func makeCPUManagerPod(podName string, ctnAttributes []ctnAttribute) *v1.Pod {
 				},
 			},
 			Command: []string{"sh", "-c", cpusetCmd},
+			VolumeMounts: []v1.VolumeMount{
+				{
+					Name:      "sysfscgroup",
+					MountPath: "/sysfscgroup",
+				},
+				{
+					Name:      "podinfo",
+					MountPath: "/podinfo",
+				},
+			},
 		}
 		containers = append(containers, ctn)
 	}
@@ -81,6 +91,30 @@ func makeCPUManagerPod(podName string, ctnAttributes []ctnAttribute) *v1.Pod {
 		Spec: v1.PodSpec{
 			RestartPolicy: v1.RestartPolicyNever,
 			Containers:    containers,
+			Volumes: []v1.Volume{
+				{
+					Name: "sysfscgroup",
+					VolumeSource: v1.VolumeSource{
+						HostPath: &v1.HostPathVolumeSource{Path: "/sys/fs/cgroup"},
+					},
+				},
+				{
+					Name: "podinfo",
+					VolumeSource: v1.VolumeSource{
+						DownwardAPI: &v1.DownwardAPIVolumeSource{
+							Items: []v1.DownwardAPIVolumeFile{
+								{
+									Path: "uid",
+									FieldRef: &v1.ObjectFieldSelector{
+										APIVersion: "v1",
+										FieldPath:  "metadata.uid",
+									},
+								},
+							},
+						},
+					},
+				},
+			},
 		},
 	}
 }
@@ -232,10 +266,11 @@ func getCoreSiblingList(cpuRes int64) string {
 }
 
 type cpuManagerKubeletArguments struct {
-	policyName              string
-	enableCPUManagerOptions bool
-	reservedSystemCPUs      cpuset.CPUSet
-	options                 map[string]string
+	policyName                       string
+	enableCPUManagerOptions          bool
+	disableCPUQuotaWithExclusiveCPUs bool
+	reservedSystemCPUs               cpuset.CPUSet
+	options                          map[string]string
 }
 
 func configureCPUManagerInKubelet(oldCfg *kubeletconfig.KubeletConfiguration, kubeletArguments *cpuManagerKubeletArguments) *kubeletconfig.KubeletConfiguration {
@@ -247,6 +282,7 @@ func configureCPUManagerInKubelet(oldCfg *kubeletconfig.KubeletConfiguration, ku
 	newCfg.FeatureGates["CPUManagerPolicyOptions"] = kubeletArguments.enableCPUManagerOptions
 	newCfg.FeatureGates["CPUManagerPolicyBetaOptions"] = kubeletArguments.enableCPUManagerOptions
 	newCfg.FeatureGates["CPUManagerPolicyAlphaOptions"] = kubeletArguments.enableCPUManagerOptions
+	newCfg.FeatureGates["DisableCPUQuotaWithExclusiveCPUs"] = kubeletArguments.disableCPUQuotaWithExclusiveCPUs
 
 	newCfg.CPUManagerPolicy = kubeletArguments.policyName
 	newCfg.CPUManagerReconcilePeriod = metav1.Duration{Duration: 1 * time.Second}
@@ -556,6 +592,178 @@ func runMultipleCPUContainersGuPod(ctx context.Context, f *framework.Framework)
 	waitForContainerRemoval(ctx, pod.Spec.Containers[1].Name, pod.Name, pod.Namespace)
 }
 
+func runCfsQuotaGuPods(ctx context.Context, f *framework.Framework, disabledCPUQuotaWithExclusiveCPUs bool) {
+	var err error
+	var ctnAttrs []ctnAttribute
+	var pod1, pod2, pod3 *v1.Pod
+	var cleanupPods []*v1.Pod
+	ginkgo.DeferCleanup(func() {
+		// waitForContainerRemoval takes "long" to complete; if we use the parent ctx we get a
+		// 'deadline expired' message and the cleanup aborts, which we don't want.
+		ctx2 := context.TODO()
+		ginkgo.By("by deleting the pods and waiting for container removal")
+		for _, cleanupPod := range cleanupPods {
+			framework.Logf("deleting pod: %s/%s", cleanupPod.Namespace, cleanupPod.Name)
+			deletePodSyncByName(ctx2, f, cleanupPod.Name)
+			waitForContainerRemoval(ctx2, cleanupPod.Spec.Containers[0].Name, cleanupPod.Name, cleanupPod.Namespace)
+			framework.Logf("deleted pod: %s/%s", cleanupPod.Namespace, cleanupPod.Name)
+		}
+	})
+
+	cfsCheckCommand := []string{"sh", "-c", "cat /sys/fs/cgroup/cpu.max && sleep 1d"}
+	defaultPeriod := "100000"
+
+	ctnAttrs = []ctnAttribute{
+		{
+			ctnName:    "gu-container-cfsquota-disabled",
+			cpuRequest: "1",
+			cpuLimit:   "1",
+		},
+	}
+	pod1 = makeCPUManagerPod("gu-pod1", ctnAttrs)
+	pod1.Spec.Containers[0].Command = cfsCheckCommand
+	pod1 = e2epod.NewPodClient(f).CreateSync(ctx, pod1)
+	cleanupPods = append(cleanupPods, pod1)
+
+	ginkgo.By("checking if the expected cfs quota was assigned (GU pod, exclusive CPUs, unlimited)")
+
+	expectedQuota := "100000"
+	if disabledCPUQuotaWithExclusiveCPUs {
+		expectedQuota = "max"
+	}
+	expCFSQuotaRegex := fmt.Sprintf("^%s %s\n$", expectedQuota, defaultPeriod)
+	err = e2epod.NewPodClient(f).MatchContainerOutput(ctx, pod1.Name, pod1.Spec.Containers[0].Name, expCFSQuotaRegex)
+	framework.ExpectNoError(err, "expected log not found in container [%s] of pod [%s]",
+		pod1.Spec.Containers[0].Name, pod1.Name)
+
+	ctnAttrs = []ctnAttribute{
+		{
+			ctnName:    "gu-container-cfsquota-enabled",
+			cpuRequest: "500m",
+			cpuLimit:   "500m",
+		},
+	}
+	pod2 = makeCPUManagerPod("gu-pod2", ctnAttrs)
+	pod2.Spec.Containers[0].Command = cfsCheckCommand
+	pod2 = e2epod.NewPodClient(f).CreateSync(ctx, pod2)
+	cleanupPods = append(cleanupPods, pod2)
+
+	ginkgo.By("checking if the expected cfs quota was assigned (GU pod, limited)")
+
+	expectedQuota = "50000"
+	expCFSQuotaRegex = fmt.Sprintf("^%s %s\n$", expectedQuota, defaultPeriod)
+	err = e2epod.NewPodClient(f).MatchContainerOutput(ctx, pod2.Name, pod2.Spec.Containers[0].Name, expCFSQuotaRegex)
+	framework.ExpectNoError(err, "expected log not found in container [%s] of pod [%s]",
+		pod2.Spec.Containers[0].Name, pod2.Name)
+
+	ctnAttrs = []ctnAttribute{
+		{
+			ctnName:    "non-gu-container",
+			cpuRequest: "100m",
+			cpuLimit:   "500m",
+		},
+	}
+	pod3 = makeCPUManagerPod("non-gu-pod3", ctnAttrs)
+	pod3.Spec.Containers[0].Command = cfsCheckCommand
+	pod3 = e2epod.NewPodClient(f).CreateSync(ctx, pod3)
+	cleanupPods = append(cleanupPods, pod3)
+
+	ginkgo.By("checking if the expected cfs quota was assigned (BU pod, limited)")
+
+	expectedQuota = "50000"
+	expCFSQuotaRegex = fmt.Sprintf("^%s %s\n$", expectedQuota, defaultPeriod)
+	err = e2epod.NewPodClient(f).MatchContainerOutput(ctx, pod3.Name, pod3.Spec.Containers[0].Name, expCFSQuotaRegex)
+	framework.ExpectNoError(err, "expected log not found in container [%s] of pod [%s]",
+		pod3.Spec.Containers[0].Name, pod3.Name)
+
+	ctnAttrs = []ctnAttribute{
+		{
+			ctnName:    "gu-container-non-int-values",
+			cpuRequest: "500m",
+			cpuLimit:   "500m",
+		},
+		{
+			ctnName:    "gu-container-int-values",
+			cpuRequest: "1",
+			cpuLimit:   "1",
+		},
+	}
+	pod4 := makeCPUManagerPod("gu-pod4", ctnAttrs)
+	pod4.Spec.Containers[0].Command = cfsCheckCommand
+	pod4.Spec.Containers[1].Command = cfsCheckCommand
+	pod4 = e2epod.NewPodClient(f).CreateSync(ctx, pod4)
+	cleanupPods = append(cleanupPods, pod4)
+
+	ginkgo.By("checking if the expected cfs quota was assigned (GU pod, container 0 exclusive CPUs unlimited, container 1 limited)")
+
+	expectedQuota = "50000"
+	expCFSQuotaRegex = fmt.Sprintf("^%s %s\n$", expectedQuota, defaultPeriod)
+	err = e2epod.NewPodClient(f).MatchContainerOutput(ctx, pod4.Name, pod4.Spec.Containers[0].Name, expCFSQuotaRegex)
+	framework.ExpectNoError(err, "expected log not found in container [%s] of pod [%s]",
+		pod4.Spec.Containers[0].Name, pod4.Name)
+	expectedQuota = "100000"
+	if disabledCPUQuotaWithExclusiveCPUs {
+		expectedQuota = "max"
+	}
+	expCFSQuotaRegex = fmt.Sprintf("^%s %s\n$", expectedQuota, defaultPeriod)
+	err = e2epod.NewPodClient(f).MatchContainerOutput(ctx, pod4.Name, pod4.Spec.Containers[1].Name, expCFSQuotaRegex)
+	framework.ExpectNoError(err, "expected log not found in container [%s] of pod [%s]",
+		pod4.Spec.Containers[1].Name, pod4.Name)
+
+	ctnAttrs = []ctnAttribute{
+		{
+			ctnName:    "gu-container-non-int-values",
+			cpuRequest: "500m",
+			cpuLimit:   "500m",
+		},
+		{
+			ctnName:    "gu-container-int-values",
+			cpuRequest: "1",
+			cpuLimit:   "1",
+		},
+	}
+
+	podCFSCheckCommand := []string{"sh", "-c", `cat $(find /sysfscgroup | grep "$(cat /podinfo/uid | sed 's/-/_/g').slice/cpu.max$") && sleep 1d`}
+
+	pod5 := makeCPUManagerPod("gu-pod5", ctnAttrs)
+	pod5.Spec.Containers[0].Command = podCFSCheckCommand
+	pod5 = e2epod.NewPodClient(f).CreateSync(ctx, pod5)
+	cleanupPods = append(cleanupPods, pod5)
+	ginkgo.By("checking if the expected cfs quota was assigned to pod (GU pod, unlimited)")
+
+	expectedQuota = "150000"
+
+	if disabledCPUQuotaWithExclusiveCPUs {
+		expectedQuota = "max"
+	}
+
+	expCFSQuotaRegex = fmt.Sprintf("^%s %s\n$", expectedQuota, defaultPeriod)
+
+	err = e2epod.NewPodClient(f).MatchContainerOutput(ctx, pod5.Name, pod5.Spec.Containers[0].Name, expCFSQuotaRegex)
+	framework.ExpectNoError(err, "expected log not found in container [%s] of pod [%s]", pod5.Spec.Containers[0].Name, pod5.Name)
+
+	ctnAttrs = []ctnAttribute{
+		{
+			ctnName:    "gu-container",
+			cpuRequest: "100m",
+			cpuLimit:   "100m",
+		},
+	}
+
+	pod6 := makeCPUManagerPod("gu-pod6", ctnAttrs)
+	pod6.Spec.Containers[0].Command = podCFSCheckCommand
+	pod6 = e2epod.NewPodClient(f).CreateSync(ctx, pod6)
+	cleanupPods = append(cleanupPods, pod6)
+
+	ginkgo.By("checking if the expected cfs quota was assigned to pod (GU pod, limited)")
+
+	expectedQuota = "10000"
+	expCFSQuotaRegex = fmt.Sprintf("^%s %s\n$", expectedQuota, defaultPeriod)
+	err = e2epod.NewPodClient(f).MatchContainerOutput(ctx, pod6.Name, pod6.Spec.Containers[0].Name, expCFSQuotaRegex)
+	framework.ExpectNoError(err, "expected log not found in container [%s] of pod [%s]", pod6.Spec.Containers[0].Name, pod6.Name)
+
+}
+
 func runMultipleGuPods(ctx context.Context, f *framework.Framework) {
 	var expAllowedCPUsListRegex string
 	var cpuList []int
@@ -709,6 +917,37 @@ func runCPUManagerTests(f *framework.Framework) {
 		runSMTAlignmentPositiveTests(ctx, f, smtLevel)
 	})
 
+	ginkgo.It("should not enforce CFS quota for containers with static CPUs assigned", func(ctx context.Context) {
+		if !IsCgroup2UnifiedMode() {
+			e2eskipper.Skipf("Skipping since CgroupV2 not used")
+		}
+		newCfg := configureCPUManagerInKubelet(oldCfg,
+			&cpuManagerKubeletArguments{
+				policyName:                       string(cpumanager.PolicyStatic),
+				reservedSystemCPUs:               cpuset.New(0),
+				disableCPUQuotaWithExclusiveCPUs: true,
+			},
+		)
+		updateKubeletConfig(ctx, f, newCfg, true)
+		runCfsQuotaGuPods(ctx, f, true)
+	})
+
+	ginkgo.It("should keep enforcing the CFS quota for containers with static CPUs assigned and feature gate disabled", func(ctx context.Context) {
+		if !IsCgroup2UnifiedMode() {
+			e2eskipper.Skipf("Skipping since CgroupV2 not used")
+		}
+		newCfg := configureCPUManagerInKubelet(oldCfg,
+			&cpuManagerKubeletArguments{
+				policyName:                       string(cpumanager.PolicyStatic),
+				reservedSystemCPUs:               cpuset.New(0),
+				disableCPUQuotaWithExclusiveCPUs: false,
+			},
+		)
+
+		updateKubeletConfig(ctx, f, newCfg, true)
+		runCfsQuotaGuPods(ctx, f, false)
+	})
+
 	f.It("should not reuse CPUs of restartable init containers", feature.SidecarContainers, func(ctx context.Context) {
 		cpuCap, cpuAlloc, _ = getLocalNodeCPUDetails(ctx, f)
 
diff --git a/test/featuregates_linter/test_data/versioned_feature_list.yaml b/test/featuregates_linter/test_data/versioned_feature_list.yaml
index 30db7939e983e..3a33b8928736e 100644
--- a/test/featuregates_linter/test_data/versioned_feature_list.yaml
+++ b/test/featuregates_linter/test_data/versioned_feature_list.yaml
@@ -416,6 +416,12 @@
     lockToDefault: true
     preRelease: GA
     version: "1.31"
+- name: DisableCPUQuotaWithExclusiveCPUs
+  versionedSpecs:
+  - default: true
+    lockToDefault: false
+    preRelease: Beta
+    version: "1.33"
 - name: DisableKubeletCloudCredentialProviders
   versionedSpecs:
   - default: false