Repository: kubernetes-sigs/kueue

Issue:
Flaky perf scheduling test
/kind flake

**What happened**:

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4845/pull-kueue-test-scheduling-perf-main/1906781402901254144

**What you expected to happen**:

no failures

**How to reproduce it (as minimally and precisely as possible)**:

ci

**Anything else we need to know?**:

```
{Failed  === RUN   TestScalability/CommandStats
    checker_test.go:87: Average CPU usage 538mCpu is greater than maximum expected 535mCPU
--- FAIL: TestScalability/CommandStats (0.00s)
}
```

Discussion:
@mimowo: 
This might be related to https://github.com/kubernetes-sigs/kueue/pull/4813 as it is using reclaimWithinCohort: Any. 

cc @mbobrovskyi ptal, maybe the cpu got higher recently, but the execution time dropped? If this is the case, I think it is WAI and I'm ok to bump the thresholds to make it stable.
@mimowo: Another occurrence: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4837/pull-kueue-test-scheduling-perf-main/1906685554880155648@mimowo: cc @gabesaba as this might be related to the recent scheduling changes@mimowo: another occurrence https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4929/pull-kueue-test-scheduling-perf-main/1912139065210179584@mimowo: I would suggest to just bump the CPU to 550m. I don't like the CPU metric anyway and would be ok with dropping the threshold completely, as discussed before when introducing it. 

WDYT @tenzen-y @gabesaba @mbobrovskyi ?@mimowo: I would speculate the CPU consumption increased after the fix by @gabesaba of reclamation, and this is an expected change, but it is hard to confirm now@gabesaba: I would either: 

1) increase the threshold with the goal of replacing it very soon
2) delete it now

I lean towards option 2, and open a bug to track creating a better metric

> I don't like the CPU metric anyway and would be ok with dropping the threshold completely, as discussed before when introducing it.

+1


@mimowo: > I lean towards option 2, and open a bug to track creating a better metric

+1, but we already have [maxWallMs](https://github.com/kubernetes-sigs/kueue/blob/d68df3a996c1ef1f6dc4abeb156cc0f934141def/test/performance/scheduler/default_rangespec.yaml#L9) which is the total execution time. Do you have anything else on your mind?@tenzen-y: What metrics are used in kube-scheduler perf-test? Doesn't it use CPU threshold?
@mimowo: It currently does use [mCPU: 535](https://github.com/kubernetes-sigs/kueue/blob/d68df3a996c1ef1f6dc4abeb156cc0f934141def/test/performance/scheduler/default_rangespec.yaml#L12C3-L12C12), and this is the reason for the flakes, because occasionally this is exceeded as in this case it was 538mCpu.

Note that 535 is not the original value which was 500 (iirc), but later bumped. @tenzen-y: I meant kube-scheduler perf-test instead of kueue scheduler perf-test@mimowo: cc @macsko who is likely to know@macsko: > What metrics are used in kube-scheduler perf-test? Doesn't it use CPU threshold?

kube-scheduler perf tests ([scheduler_perf](https://github.com/kubernetes/kubernetes/tree/master/test/integration/scheduler_perf)) verify only the scheduling throughput, without checking cpu/memory usage. Code that calculates the throughput is [here](https://github.com/kubernetes/kubernetes/blob/44c230bf5c321056e8bc89300b37c497f464f113/test/integration/scheduler_perf/util.go#L468-L615). It is done using [event handler subscribed on Pod/Add and Pod/Update events](https://github.com/kubernetes/kubernetes/blob/44c230bf5c321056e8bc89300b37c497f464f113/test/integration/scheduler_perf/util.go#L486-L511). For statistical purposes, and not for calculating throughput itself, we use the [`scheduler_schedule_attempts_total` metric](https://github.com/kubernetes/kubernetes/blob/44c230bf5c321056e8bc89300b37c497f464f113/test/integration/scheduler_perf/util.go#L597).@mimowo: > kube-scheduler perf tests ([scheduler_perf](https://github.com/kubernetes/kubernetes/tree/master/test/integration/scheduler_perf)) verify only the scheduling throughput, without checking cpu/memory usage.

So, I think we don't need it either.

> Code that calculates the throughput is [here](https://github.com/kubernetes/kubernetes/blob/44c230bf5c321056e8bc89300b37c497f464f113/test/integration/scheduler_perf/util.go#L468-L615). It is done using [event handler subscribed on Pod/Add and Pod/Update events](https://github.com/kubernetes/kubernetes/blob/44c230bf5c321056e8bc89300b37c497f464f113/test/integration/scheduler_perf/util.go#L486-L511). For statistical purposes, and not for calculating throughput itself, we use the [scheduler_schedule_attempts_total metric](https://github.com/kubernetes/kubernetes/blob/44c230bf5c321056e8bc89300b37c497f464f113/test/integration/scheduler_perf/util.go#L597).

I would say that we also measure throughput in our tests, where ` maxWallMs` is a proxy, and more granual is `wlClassesMaxAvgTimeToAdmissionMs`. 

Probably we also don't need then cmd.maxrss and `clusterQueueClassesMinUsage` (though we haven't had flakes with them yet. WDYT @tenzen-y ? @tenzen-y: > > kube-scheduler perf tests ([scheduler_perf](https://github.com/kubernetes/kubernetes/tree/master/test/integration/scheduler_perf)) verify only the scheduling throughput, without checking cpu/memory usage.
> 
> So, I think we don't need it either.
> 
> > Code that calculates the throughput is [here](https://github.com/kubernetes/kubernetes/blob/44c230bf5c321056e8bc89300b37c497f464f113/test/integration/scheduler_perf/util.go#L468-L615). It is done using [event handler subscribed on Pod/Add and Pod/Update events](https://github.com/kubernetes/kubernetes/blob/44c230bf5c321056e8bc89300b37c497f464f113/test/integration/scheduler_perf/util.go#L486-L511). For statistical purposes, and not for calculating throughput itself, we use the [scheduler_schedule_attempts_total metric](https://github.com/kubernetes/kubernetes/blob/44c230bf5c321056e8bc89300b37c497f464f113/test/integration/scheduler_perf/util.go#L597).
> 
> I would say that we also measure throughput in our tests, where ` maxWallMs` is a proxy, and more granual is `wlClassesMaxAvgTimeToAdmissionMs`.
> 
> Probably we also don't need then cmd.maxrss and `clusterQueueClassesMinUsage` (though we haven't had flakes with them yet. WDYT [@tenzen-y](https://github.com/tenzen-y) ?

I synced with @mimowo offline. In conclusion, we agree with dropping the assertion based on the threshold for CPU and memory usage.
However, we keep collecting those metrics during CI and push the result to artifacts since those are still useful to evaluate Kueue project.
@IrvingMg: /assign@mimowo: new occurrence: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4989/pull-kueue-test-scheduling-perf-main/1915345602112131072

Solution patch:
diff --git a/Makefile-test.mk b/Makefile-test.mk
index 4d84b7fb2..0c06298e5 100644
--- a/Makefile-test.mk
+++ b/Makefile-test.mk
@@ -47,11 +47,6 @@ INTEGRATION_TARGET_MULTIKUEUE ?= ./test/integration/multikueue/...
 # Verbosity level for apiserver logging.
 # The logging is disabled if 0.
 INTEGRATION_API_LOG_LEVEL ?= 0
-# Integration filters
-INTEGRATION_RUN_ALL?=true
-ifneq ($(INTEGRATION_RUN_ALL),true) 
-	INTEGRATION_FILTERS= --label-filter="!slow && !redundant"
-endif
 
 # Folder where the e2e tests are located.
 E2E_TARGET ?= ./test/e2e/...
@@ -99,6 +94,14 @@ test-integration: gomod-download envtest ginkgo dep-crds kueuectl ginkgo-top ##
 	$(GINKGO) $(INTEGRATION_FILTERS) $(GINKGO_ARGS) $(GOFLAGS) -procs=$(INTEGRATION_NPROCS) --race --junit-report=junit.xml --json-report=integration.json --output-dir=$(ARTIFACTS) -v $(INTEGRATION_TARGET)
 	$(PROJECT_DIR)/bin/ginkgo-top -i $(ARTIFACTS)/integration.json > $(ARTIFACTS)/integration-top.yaml
 
+.PHONY: test-integration-baseline
+test-integration-baseline: INTEGRATION_FILTERS= --label-filter="!slow && !redundant"
+test-integration-baseline: test-integration ## Run baseline integration tests for singlecluster suites.
+
+.PHONY: test-integration-extended
+test-integration-extended: INTEGRATION_FILTERS= --label-filter="slow || redundant"
+test-integration-extended: test-integration ## Run extended integration tests for singlecluster suites.
+
 .PHONY: test-multikueue-integration
 test-multikueue-integration: gomod-download envtest ginkgo dep-crds ginkgo-top ## Run integration tests for MultiKueue suite.
 	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) -p path)" \
diff --git a/charts/kueue/templates/_helpers.tpl b/charts/kueue/templates/_helpers.tpl
index 9b41bb6a8..cb9a404cf 100644
--- a/charts/kueue/templates/_helpers.tpl
+++ b/charts/kueue/templates/_helpers.tpl
@@ -51,6 +51,14 @@ app.kubernetes.io/instance: {{ .Release.Name }}
 control-plane: controller-manager
 {{- end }}
 
+{{/*
+Labels for metrics service
+*/}}
+{{- define "kueue.metricsService.labels" -}}
+{{ include "kueue.labels" . }}
+app.kubernetes.io/component: metrics
+{{- end }}
+
 {{/*
 Create the name of the service account to use
 */}}
diff --git a/charts/kueue/templates/manager/auth_proxy_service.yaml b/charts/kueue/templates/manager/auth_proxy_service.yaml
index 02942cab0..e3eb2f72a 100644
--- a/charts/kueue/templates/manager/auth_proxy_service.yaml
+++ b/charts/kueue/templates/manager/auth_proxy_service.yaml
@@ -4,7 +4,7 @@ metadata:
   name: {{ include "kueue.fullname" . }}-controller-manager-metrics-service
   namespace: '{{ .Release.Namespace }}'
   labels:
-  {{- include "kueue.labels" . | nindent 4 }}
+  {{- include "kueue.metricsService.labels" . | nindent 4 }}
   {{- if .Values.metricsService.annotations }}
   annotations:
   {{- toYaml .Values.metricsService.annotations | nindent 4 }}
diff --git a/charts/kueue/templates/prometheus/monitor.yaml b/charts/kueue/templates/prometheus/monitor.yaml
index 7ce986f29..553e643c8 100644
--- a/charts/kueue/templates/prometheus/monitor.yaml
+++ b/charts/kueue/templates/prometheus/monitor.yaml
@@ -15,5 +15,5 @@ spec:
     tlsConfig: {{- toYaml .Values.metrics.serviceMonitor.tlsConfig | nindent 6 }}
   selector:
     matchLabels:
-    {{- include "kueue.labels" . | nindent 6 }}
+    {{- include "kueue.metricsService.labels" . | nindent 6 }}
 {{- end }}
diff --git a/keps/4136-admission-fair-sharing/README.md b/keps/4136-admission-fair-sharing/README.md
new file mode 100644
index 000000000..6435d0f87
--- /dev/null
+++ b/keps/4136-admission-fair-sharing/README.md
@@ -0,0 +1,255 @@
+# KEP-4136: Admission Fair Sharing
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories (Optional)](#user-stories-optional)
+    - [Story 1](#story-1)
+    - [Story 2](#story-2)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [Test Plan](#test-plan)
+      - [Prerequisite testing updates](#prerequisite-testing-updates)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP describes the mechanism for fair admission of workloads coming from a group of
+different sources (like Cluster and Local Queues) based on the source shared resource usage.
+Workloads from sources that use less are admitted before workloads coming from sources that use more. 
+
+## Motivation
+
+Currently Kueue has a Fair Sharing mechanism that enforces fair-sharing of unused resources
+via preemption. If one Cluster Queue is using much more of the resources than the other one
+that is in need, some workloads from the first one may be preempted to allow more “fair” distribution. 
+
+This model has multiple assumptions:
+
+* Workloads can be preempted.
+* Users get a proper quota to do their regular business.
+* Users come from multiple teams/organizations and they would rather have a strict policy but 
+fair policy than show understanding to those consuming all shared resources. 
+
+However, these assumptions are not universal. Sometimes:
+
+* Workloads should not be preempted.
+* Users operate on a shared but bigger quota.
+* Fairness should not win over getting the workloads eventually completed.
+
+In that case the existing mode doesn’t work and a different one needs to be employed.
+
+### Goals
+
+* Establish a method for how shared resource usage is calculated and recorded and how users can fine tune the mechanism.
+* Allow to specify a fair admission scope at either individual Cluster Queue or Cohort scope.
+* Allow to specify the relative importance of LocalQueues targeting the same ClusterQueue.
+* Amend the admission mechanism to work on admission scopes instead of only on ClusterQueues. 
+* Select the appropriate admission candidates for each of the admission scopes and admit them according to the selected queueing policy.
+* Make the new mechanism complementary to the existing preemption-based fair sharing
+
+### Non-Goals
+
+* Store time series data inside K8S.
+* Provide precise shared resource usage accounting or billing.
+
+## Proposal
+
+* Modify CQ’s FairSharing struct with 
+
+```go
+type FairSharing struct {
+	// Weight denotes how important the given queue when competing against other queues 
+// for unused shared resources. The exact impact of the weight in fair share calculations
+// depends on the fair share algorithm used. Default = 1.
+	Weight *resource.Quantity `json:"weight,omitempty"`
+}
+```
+
+* Expand LQ spec with the same FairSharing struct (Cohort will be expanded with FairSharing 
+as a part of the Preemptive FS implementation in hierarchical structure).
+
+* Modify FairSharingStatus struct with 
+
+```go
+type FairSharingStatus struct {
+	// WeightedShare represents the usage above nominal quota, with the weight applied in.
+	// The bigger the value is the more shared resources has been allocated and the less 
+	// entitled the queue is for more shared resources. 
+	// The exact details and the interpretation of the value depends on 
+    // the fair sharing algorithm used.
+	WeightedShare int64 `json:"weightedShare"`
+
+	
+	// ConsumedResources represents the aggregated usage of resources over time,
+	// with decaying function applied. 
+	// The value is populated if usage consumption functionality is enabled in Kueue config.
+	ConsumedResources corev1.ResourceList `json:"consumedResources,omitempty"`
+
+	// LastUpdate is the time when share and consumed resources were updated.
+	LastUpdate metav1.Time `json:"lastUpdate,omitempty"`
+}
+```
+
+* Add FairSharingStatus to LocalQueue (and Cohort).
+
+* Create a new struct AdmissionScope and make it an optional field for CQ and Cohort Spec. If
+not provided, CQ or Cohort is not considered an AdmissionScope and is not a subject for new
+admission logic. If there are two AdmissionScopes on the path from CQ/Cohort to the top of
+the hierarchy tree, the higher one is used.
+
+```go
+const (
+	// FairSharing based on usage, with QueuingStrategy as defined in CQ.
+	UsageBasedAdmissionFairSharing AdmissionMode = UsageBasedFairSharing
+
+    NoAdmissionFairSharing AdmissionMode = NoAdmissionFairSharing
+)
+
+type AdmissionScope struct {
+    AdmissionMode AdmissionMode
+}
+```
+
+* When selecting candidates for admission groups all workloads from LQ to CQ or the
+topmost Cohort that is marked with AdmissionScope. Then sort them using criterias:
+
+- Usage vector  built from ConsumedResources from TopCohort to LQ
+- Priority
+- Timestamp
+
+The usage vector will be sum of the `ConsumedResources` weighted according to `resourceWeights`
+(mentioned later in the KEP)
+
+Let’s look at a couple scenarios:
+
+1. AdmissionScope at CQ, CQ queueing policy is FIFO, 3 LQ pointing to CQ. Kueue considers
+all CQ resources and potentially borrowed resources as “shared” resources and fair sharing
+is applied to all workloads.
+Kueue sorts the workloads by their LQ usage (if mode is `UsageBasedFairSharing`), priority and
+timestamp and tries to admit the first one from the list. Other workloads are not attempted
+until the first one is not admitted.
+
+2.  AdmissionScope at CQ, CQ queueing policy is BestEffort, 3 LQ pointing to CQ. 
+Kueue considers all CQ resources and potentially borrowed resources as “shared” resources and
+fair sharing is applied to all workloads.
+
+Kueue sorts the workloads by their LQ usage (if mode is `UsageBasedFairSharing`), priority and
+timestamp and tries to admit the first one from the list. If it fails and the second, third
+or following is possible then that workload is admitted, under condition that it might get preempted.
+
+3. AdmissionScope at Cohort level - Kueue operates in a mixed mode. Inside CQ workloads are
+selected according to their AdmissionMode (if specified). If a workload fits entirely into
+nominal quota, then it is admitted immediately, if not it goes into cohort-level fair sharing.
+For Cohort we select all the “sticking out” workloads, and sort them by their CQ usage, priority
+and timestamp. Kueue attempts to admit the first workload from the list of sticking-out
+(workload + current_usage > resources), just like if it was one big strict FIFO queue. 
+For multi-level hierarchy under one AdmissionScope we would treat underlying Cohorts as Fifo CQs.
+
+
+* Additionally there will be the resource usage calculation loop. The frequency of the
+calculation will be controlled globally in Kueue’s config file. Accounting would be done
+using something like geometric average:
+
+usage_sum = (1-A) * previous_usage_sum + A * current_usage.  
+
+The value will be stored in FairSharingStatus for all LQ, CQ, and Cohorts. The value will not be zeroed
+after Kueue restart or after brief period of downtime. However if the period is longer,
+the value should be automatically zeroed.
+
+
+The user will be able to configure the decaying factor A in Kueue’s config file by specifying
+the half life decay time - after what time the current shared usage will decay to half of its original value.
+
+A = 1 - 0.5 ^ (sampling/half_life_decay)
+
+* Configuration will sit in FairSharing stuct in Kueue config. There will be the following modifications:
+
+  - usageHalfLifeDecayTime - half life decay time of usage, as described above.
+  - usageSamplingInterval - how often usage is calculated.
+  - resourceWeights - how much consumption of individual resources is important when comparing usage.
+  - resetInactivityPeriod - if Kueue has not updated the value for this period then the value should be zeroed.
+ 
+If the user doesn't want any preemptions while fair sharing, preemptionStrategies should be left empty.
+
+* If preemptionStrategies is non empty Kueue attempts to combine two fair sharings at the same time. 
+For each of the Admission Scopes Kueue selects one workload to be attempted. And then these pre-selected 
+workloads are sorted based on their Preemption-based fair share value. If some of them don't fit,
+fair sharing preemption may be executed. So admission-based fair sharing only reshuffles workloads
+within AdmissionScope and then other mechanisms are applied as usual.
+
+### User Stories (Optional)
+#### Story 1
+
+I have multiple users using the same ClusterQueue. Each has its own namespace and LocalQueue
+through which they submit workloads. I want to fairly admit their workloads so that one active
+user doesn’t block the cluster too much.
+
+#### Story 2
+
+I have multiple teams that may be sending workloads of various sizes. I want to give each team
+some guaranteed capacity and at the same time, allow them to fairly share some bigger pool of resources. 
+
+### Risks and Mitigations
+
+* Having 2 fair sharing mechanisms and confusion between preemption-based fair sharing and admission time fair sharing.
+
+* Increased complexity of the project.
+
+## Design Details
+
+Covered in Proposal.
+
+### Test Plan
+[x] I/we understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+##### Prerequisite testing updates
+
+#### Unit Tests
+The code will be thoroughly covered with unit tests.
+
+In particular:
+* CQ level fair sharing between LQ for both strict FIFO and best effort.
+* Cohort with owned resources, and CQ with guaranteed quota.
+
+
+#### Integration tests
+
+They will mainly focus on larger scope scheduling (involving multiple cohorts/cqs) and
+interactions with preemptive fair sharing outside admission scope.
+
+### Graduation Criteria
+
+The implementation will be split into 2 subfeatures:
+
+* CQ + LQ level support
+* Multi-level Cohort+CQ+LQ support
+
+Obviously, the second depends on the first to some extent. The first however may reach
+Beta/GA without starting the second.
+
+The graduation criterias are quite standard:
+
+* Beta - positive feedback from Alpha, api seems reasonable.
+* GA - positive feedback, no bugs, no api changes needed.
+
+We hope to have CQ+LQ in alpha for the next Kueue release (0.12).
+
+## Drawbacks
+* Adds additional complexity to the system.
+* Creates yet another fair sharing mechanism.
+
+## Alternatives
+
+* Not having the feature.
+* Modifying/replacing the existing preemptive fair sharing algorithm.
+
diff --git a/keps/4136-admission-fair-sharing/kep.yaml b/keps/4136-admission-fair-sharing/kep.yaml
new file mode 100644
index 000000000..a06a2e831
--- /dev/null
+++ b/keps/4136-admission-fair-sharing/kep.yaml
@@ -0,0 +1,36 @@
+title: Admission fair sharing
+kep-number: 4136
+authors:
+  - "@mwielgus"
+status: draft
+creation-date: 2025-02-03
+reviewers:
+  - "@mimowo"
+  - "@gabesaba"
+  - "@pbundyra"
+approvers:
+  - "@mimowo"
+  - "@tenzen-y"
+sea-also:
+  - "KEP-1714"
+
+# The target maturity stage in the current dev cycle for this KEP.
+stage: alpha
+
+# The most recent milestone for which work toward delivery of this KEP has been
+# done. This can be the current (upcoming) milestone, if it is being actively
+# worked on.
+latest-milestone: "v0.12"
+
+# The milestone at which this feature was, or is targeted to be, at each stage.
+milestone:
+  alpha: "v0.12"
+
+# The following PRR answers are required at alpha release
+# List the feature gate name and the components for which it must be enabled
+disable-supported: true
+
+# The following PRR answers are required at beta release
+# metrics:
+#   - my_feature_metric
+



Test patch:
diff --git a/test/performance/scheduler/checker/checker_test.go b/test/performance/scheduler/checker/checker_test.go
index 641552da1..336c6ee98 100644
--- a/test/performance/scheduler/checker/checker_test.go
+++ b/test/performance/scheduler/checker/checker_test.go
@@ -35,9 +35,7 @@ var (
 
 type RangeSpec struct {
 	Cmd struct {
-		MaxWallMs int64  `json:"maxWallMs"`
-		MCPU      int64  `json:"mCPU"`
-		Maxrss    uint64 `json:"maxrss"`
+		MaxWallMs int64 `json:"maxWallMs"`
 	} `json:"cmd"`
 	ClusterQueueClassesMinUsage      map[string]float64 `json:"clusterQueueClassesMinUsage"`
 	WlClassesMaxAvgTimeToAdmissionMs map[string]int64   `json:"wlClassesMaxAvgTimeToAdmissionMs"`
@@ -82,13 +80,6 @@ func TestScalability(t *testing.T) {
 		if cmdStats.WallMs > rangeSpec.Cmd.MaxWallMs {
 			t.Errorf("Wall time %dms is greater than maximum expected %dms", cmdStats.WallMs, rangeSpec.Cmd.MaxWallMs)
 		}
-		mCPUUsed := (cmdStats.SysMs + cmdStats.UserMs) * 1000 / cmdStats.WallMs
-		if mCPUUsed > rangeSpec.Cmd.MCPU {
-			t.Errorf("Average CPU usage %dmCpu is greater than maximum expected %dmCPU", mCPUUsed, rangeSpec.Cmd.MCPU)
-		}
-		if cmdStats.Maxrss > int64(rangeSpec.Cmd.Maxrss) {
-			t.Errorf("Maxrss %dKib is greater than maximum expected %dKib", cmdStats.Maxrss, rangeSpec.Cmd.Maxrss)
-		}
 	})
 
 	t.Run("ClusterQueueClasses", func(t *testing.T) {
diff --git a/test/performance/scheduler/default_rangespec.yaml b/test/performance/scheduler/default_rangespec.yaml
index ee689c59f..57c95f8d3 100644
--- a/test/performance/scheduler/default_rangespec.yaml
+++ b/test/performance/scheduler/default_rangespec.yaml
@@ -8,12 +8,6 @@ cmd:
   # Average value 351116.4 (+/- 0.9%), setting at +20%
   maxWallMs: 425_000
 
-  # Average value 396 mCPU (+/- 8%), setting at +35%
-  mCPU: 535
-  
-  # Average value 445012 (+/- 0.3%), setting at +5%
-  maxrss: 468_000
-
 clusterQueueClassesMinUsage:
   # Average value 58.7 (+/- 1.2%), setting at -7%
   cq: 55 #%