Repository: kubernetes-sigs/kueue
Pull request #5259: [Feature] Create a metric to track the number of workloads evicted due to unique reasons
Author: login: mszadkow
Labels: name: kind/feature
description: Categorizes issue or PR as related to a new feature.
name: release-note
description: Denotes a PR that will be considered when it comes time to generate release notes.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: size/XL
description: Denotes a PR that changes 500-999 lines, ignoring generated files.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
Created: 2025-05-15T15:45:51Z | Merged: 2025-05-20T09:23:16Z

<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind feature
<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

#### What this PR does / why we need it:
Create a metric to track the number of workloads evicted due to unique reasons.
This approach is more generic than limit ourselves to reasons related to PodsReady timeouts.

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #5071 

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".
-->
```release-note
evicted_workloads_once_total - counts number of unique workload evictions.
```

Linked issue #5071: Create a metric to track the number of workloads evicted due to PodsReady timeouts
**What would you like to be added**:

A metric which will count the number of workloads evicted due to PodsReady timeout

```golang
WaitForPodsReadyWorkloadEvictionTotal  = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Subsystem: constants.KueueName,
            Name:      "wait_for_pods_ready_workload_eviction_total",
            Help:      "The number of workloads evicted due to hitting one of the WaitForPodsReady timeouts at least once",
        }, []string{"reason”}, // RecoveryTimeout, StartupTimeout
    )
```

This probably requires extending the API to track the number of evictions each timeout specifically.

**Why is this needed**:

For alerting, to track the number of workloads evicted due to hitting of the timeouts.

**Completion requirements**:

This enhancement requires the following artifacts:

- [ ] Design doc
- [ ] API change
- [ ] Docs update

The artifacts should be linked in subsequent comments.

Linked issue comments: user: login: mimowo
created_at: 2025-04-23T10:51:10Z
body: cc @mwielgus @mwysokin @mbobrovskyi 
author_association: CONTRIBUTOR
user: login: mbobrovskyi
created_at: 2025-04-23T11:06:04Z
body: /assign
author_association: CONTRIBUTOR

Diff:
diff --git a/apis/kueue/v1beta1/workload_types.go b/apis/kueue/v1beta1/workload_types.go
index ca6be0e15d1..8e60a53c675 100644
--- a/apis/kueue/v1beta1/workload_types.go
+++ b/apis/kueue/v1beta1/workload_types.go
@@ -387,6 +387,48 @@ type WorkloadStatus struct {
 	//
 	// +optional
 	AccumulatedPastExexcutionTimeSeconds *int32 `json:"accumulatedPastExexcutionTimeSeconds,omitempty"`
+
+	// schedulingStats tracks scheduling statistics
+	//
+	// +optional
+	SchedulingStats *SchedulingStats `json:"schedulingStats,omitempty"`
+}
+
+type SchedulingStats struct {
+	// evictions tracks eviction statistics by reason and underlyingCause.
+	//
+	// +optional
+	// +listType=map
+	// +listMapKey=reason
+	// +listMapKey=underlyingCause
+	// +patchStrategy=merge
+	// +patchMergeKey=reason
+	// +patchMergeKey=underlyingCause
+	Evictions []WorkloadSchedulingStatsEvicition `json:"evictions,omitempty"`
+}
+
+type WorkloadSchedulingStatsEvicition struct {
+	// reason specifies the programmatic identifier for the eviction cause.
+	//
+	// +required
+	// +kubebuilder:validation:Required
+	// +kubebuilder:validation:MaxLength=316
+	Reason string `json:"reason"`
+
+	// underlyingCause specifies a finer-grained explanation that complements the eviction reason.
+	// This may be an empty string.
+	//
+	// +required
+	// +kubebuilder:validation:Required
+	// +kubebuilder:validation:MaxLength=316
+	UnderlyingCause string `json:"underlyingCause"`
+
+	// count tracks the number of evictions for this reason and detailed reason.
+	//
+	// +required
+	// +kubebuilder:validation:Required
+	// +kubebuilder:validation:Minimum=0
+	Count int32 `json:"count"`
 }
 
 type RequeueState struct {
diff --git a/apis/kueue/v1beta1/zz_generated.deepcopy.go b/apis/kueue/v1beta1/zz_generated.deepcopy.go
index 62e454e71ab..45c869df9e0 100644
--- a/apis/kueue/v1beta1/zz_generated.deepcopy.go
+++ b/apis/kueue/v1beta1/zz_generated.deepcopy.go
@@ -1561,6 +1561,26 @@ func (in *ResourceUsage) DeepCopy() *ResourceUsage {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *SchedulingStats) DeepCopyInto(out *SchedulingStats) {
+	*out = *in
+	if in.Evictions != nil {
+		in, out := &in.Evictions, &out.Evictions
+		*out = make([]WorkloadSchedulingStatsEvicition, len(*in))
+		copy(*out, *in)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SchedulingStats.
+func (in *SchedulingStats) DeepCopy() *SchedulingStats {
+	if in == nil {
+		return nil
+	}
+	out := new(SchedulingStats)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *TopologyAssignment) DeepCopyInto(out *TopologyAssignment) {
 	*out = *in
@@ -1744,6 +1764,21 @@ func (in *WorkloadPriorityClassList) DeepCopyObject() runtime.Object {
 	return nil
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *WorkloadSchedulingStatsEvicition) DeepCopyInto(out *WorkloadSchedulingStatsEvicition) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new WorkloadSchedulingStatsEvicition.
+func (in *WorkloadSchedulingStatsEvicition) DeepCopy() *WorkloadSchedulingStatsEvicition {
+	if in == nil {
+		return nil
+	}
+	out := new(WorkloadSchedulingStatsEvicition)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *WorkloadSpec) DeepCopyInto(out *WorkloadSpec) {
 	*out = *in
@@ -1825,6 +1860,11 @@ func (in *WorkloadStatus) DeepCopyInto(out *WorkloadStatus) {
 		*out = new(int32)
 		**out = **in
 	}
+	if in.SchedulingStats != nil {
+		in, out := &in.SchedulingStats, &out.SchedulingStats
+		*out = new(SchedulingStats)
+		(*in).DeepCopyInto(*out)
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new WorkloadStatus.
diff --git a/charts/kueue/templates/crd/kueue.x-k8s.io_workloads.yaml b/charts/kueue/templates/crd/kueue.x-k8s.io_workloads.yaml
index 29cf2a1f0b9..c4c9b25d88e 100644
--- a/charts/kueue/templates/crd/kueue.x-k8s.io_workloads.yaml
+++ b/charts/kueue/templates/crd/kueue.x-k8s.io_workloads.yaml
@@ -8355,6 +8355,39 @@ spec:
                   x-kubernetes-list-map-keys:
                     - name
                   x-kubernetes-list-type: map
+                schedulingStats:
+                  description: schedulingStats tracks scheduling statistics
+                  properties:
+                    evictions:
+                      description: evictions tracks eviction statistics by reason and underlyingCause.
+                      items:
+                        properties:
+                          count:
+                            description: count tracks the number of evictions for this reason and detailed reason.
+                            format: int32
+                            minimum: 0
+                            type: integer
+                          reason:
+                            description: reason specifies the programmatic identifier for the eviction cause.
+                            maxLength: 316
+                            type: string
+                          underlyingCause:
+                            description: |-
+                              underlyingCause specifies a finer-grained explanation that complements the eviction reason.
+                              This may be an empty string.
+                            maxLength: 316
+                            type: string
+                        required:
+                          - count
+                          - reason
+                          - underlyingCause
+                        type: object
+                      type: array
+                      x-kubernetes-list-map-keys:
+                        - reason
+                        - underlyingCause
+                      x-kubernetes-list-type: map
+                  type: object
               type: object
           type: object
           x-kubernetes-validations:
diff --git a/client-go/applyconfiguration/kueue/v1beta1/schedulingstats.go b/client-go/applyconfiguration/kueue/v1beta1/schedulingstats.go
new file mode 100644
index 00000000000..6e22ba21719
--- /dev/null
+++ b/client-go/applyconfiguration/kueue/v1beta1/schedulingstats.go
@@ -0,0 +1,43 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+// Code generated by applyconfiguration-gen. DO NOT EDIT.
+
+package v1beta1
+
+// SchedulingStatsApplyConfiguration represents a declarative configuration of the SchedulingStats type for use
+// with apply.
+type SchedulingStatsApplyConfiguration struct {
+	Evictions []WorkloadSchedulingStatsEvicitionApplyConfiguration `json:"evictions,omitempty"`
+}
+
+// SchedulingStatsApplyConfiguration constructs a declarative configuration of the SchedulingStats type for use with
+// apply.
+func SchedulingStats() *SchedulingStatsApplyConfiguration {
+	return &SchedulingStatsApplyConfiguration{}
+}
+
+// WithEvictions adds the given value to the Evictions field in the declarative configuration
+// and returns the receiver, so that objects can be build by chaining "With" function invocations.
+// If called multiple times, values provided by each call will be appended to the Evictions field.
+func (b *SchedulingStatsApplyConfiguration) WithEvictions(values ...*WorkloadSchedulingStatsEvicitionApplyConfiguration) *SchedulingStatsApplyConfiguration {
+	for i := range values {
+		if values[i] == nil {
+			panic("nil value passed to WithEvictions")
+		}
+		b.Evictions = append(b.Evictions, *values[i])
+	}
+	return b
+}
diff --git a/client-go/applyconfiguration/kueue/v1beta1/workloadschedulingstatsevicition.go b/client-go/applyconfiguration/kueue/v1beta1/workloadschedulingstatsevicition.go
new file mode 100644
index 00000000000..5dab31c3aed
--- /dev/null
+++ b/client-go/applyconfiguration/kueue/v1beta1/workloadschedulingstatsevicition.go
@@ -0,0 +1,56 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+// Code generated by applyconfiguration-gen. DO NOT EDIT.
+
+package v1beta1
+
+// WorkloadSchedulingStatsEvicitionApplyConfiguration represents a declarative configuration of the WorkloadSchedulingStatsEvicition type for use
+// with apply.
+type WorkloadSchedulingStatsEvicitionApplyConfiguration struct {
+	Reason          *string `json:"reason,omitempty"`
+	UnderlyingCause *string `json:"underlyingCause,omitempty"`
+	Count           *int32  `json:"count,omitempty"`
+}
+
+// WorkloadSchedulingStatsEvicitionApplyConfiguration constructs a declarative configuration of the WorkloadSchedulingStatsEvicition type for use with
+// apply.
+func WorkloadSchedulingStatsEvicition() *WorkloadSchedulingStatsEvicitionApplyConfiguration {
+	return &WorkloadSchedulingStatsEvicitionApplyConfiguration{}
+}
+
+// WithReason sets the Reason field in the declarative configuration to the given value
+// and returns the receiver, so that objects can be built by chaining "With" function invocations.
+// If called multiple times, the Reason field is set to the value of the last call.
+func (b *WorkloadSchedulingStatsEvicitionApplyConfiguration) WithReason(value string) *WorkloadSchedulingStatsEvicitionApplyConfiguration {
+	b.Reason = &value
+	return b
+}
+
+// WithUnderlyingCause sets the UnderlyingCause field in the declarative configuration to the given value
+// and returns the receiver, so that objects can be built by chaining "With" function invocations.
+// If called multiple times, the UnderlyingCause field is set to the value of the last call.
+func (b *WorkloadSchedulingStatsEvicitionApplyConfiguration) WithUnderlyingCause(value string) *WorkloadSchedulingStatsEvicitionApplyConfiguration {
+	b.UnderlyingCause = &value
+	return b
+}
+
+// WithCount sets the Count field in the declarative configuration to the given value
+// and returns the receiver, so that objects can be built by chaining "With" function invocations.
+// If called multiple times, the Count field is set to the value of the last call.
+func (b *WorkloadSchedulingStatsEvicitionApplyConfiguration) WithCount(value int32) *WorkloadSchedulingStatsEvicitionApplyConfiguration {
+	b.Count = &value
+	return b
+}
diff --git a/client-go/applyconfiguration/kueue/v1beta1/workloadstatus.go b/client-go/applyconfiguration/kueue/v1beta1/workloadstatus.go
index e27ce1a1505..1d1239d8e04 100644
--- a/client-go/applyconfiguration/kueue/v1beta1/workloadstatus.go
+++ b/client-go/applyconfiguration/kueue/v1beta1/workloadstatus.go
@@ -31,6 +31,7 @@ type WorkloadStatusApplyConfiguration struct {
 	AdmissionChecks                      []AdmissionCheckStateApplyConfiguration `json:"admissionChecks,omitempty"`
 	ResourceRequests                     []PodSetRequestApplyConfiguration       `json:"resourceRequests,omitempty"`
 	AccumulatedPastExexcutionTimeSeconds *int32                                  `json:"accumulatedPastExexcutionTimeSeconds,omitempty"`
+	SchedulingStats                      *SchedulingStatsApplyConfiguration      `json:"schedulingStats,omitempty"`
 }
 
 // WorkloadStatusApplyConfiguration constructs a declarative configuration of the WorkloadStatus type for use with
@@ -114,3 +115,11 @@ func (b *WorkloadStatusApplyConfiguration) WithAccumulatedPastExexcutionTimeSeco
 	b.AccumulatedPastExexcutionTimeSeconds = &value
 	return b
 }
+
+// WithSchedulingStats sets the SchedulingStats field in the declarative configuration to the given value
+// and returns the receiver, so that objects can be built by chaining "With" function invocations.
+// If called multiple times, the SchedulingStats field is set to the value of the last call.
+func (b *WorkloadStatusApplyConfiguration) WithSchedulingStats(value *SchedulingStatsApplyConfiguration) *WorkloadStatusApplyConfiguration {
+	b.SchedulingStats = value
+	return b
+}
diff --git a/client-go/applyconfiguration/utils.go b/client-go/applyconfiguration/utils.go
index a8ad2fb9e6a..fbe711c4d50 100644
--- a/client-go/applyconfiguration/utils.go
+++ b/client-go/applyconfiguration/utils.go
@@ -145,6 +145,8 @@ func ForKind(kind schema.GroupVersionKind) interface{} {
 		return &kueuev1beta1.ResourceQuotaApplyConfiguration{}
 	case v1beta1.SchemeGroupVersion.WithKind("ResourceUsage"):
 		return &kueuev1beta1.ResourceUsageApplyConfiguration{}
+	case v1beta1.SchemeGroupVersion.WithKind("SchedulingStats"):
+		return &kueuev1beta1.SchedulingStatsApplyConfiguration{}
 	case v1beta1.SchemeGroupVersion.WithKind("TopologyAssignment"):
 		return &kueuev1beta1.TopologyAssignmentApplyConfiguration{}
 	case v1beta1.SchemeGroupVersion.WithKind("TopologyDomainAssignment"):
@@ -155,6 +157,8 @@ func ForKind(kind schema.GroupVersionKind) interface{} {
 		return &kueuev1beta1.WorkloadApplyConfiguration{}
 	case v1beta1.SchemeGroupVersion.WithKind("WorkloadPriorityClass"):
 		return &kueuev1beta1.WorkloadPriorityClassApplyConfiguration{}
+	case v1beta1.SchemeGroupVersion.WithKind("WorkloadSchedulingStatsEvicition"):
+		return &kueuev1beta1.WorkloadSchedulingStatsEvicitionApplyConfiguration{}
 	case v1beta1.SchemeGroupVersion.WithKind("WorkloadSpec"):
 		return &kueuev1beta1.WorkloadSpecApplyConfiguration{}
 	case v1beta1.SchemeGroupVersion.WithKind("WorkloadStatus"):
diff --git a/config/components/crd/bases/kueue.x-k8s.io_workloads.yaml b/config/components/crd/bases/kueue.x-k8s.io_workloads.yaml
index e36b2efbbba..f9d578f5c50 100644
--- a/config/components/crd/bases/kueue.x-k8s.io_workloads.yaml
+++ b/config/components/crd/bases/kueue.x-k8s.io_workloads.yaml
@@ -8841,6 +8841,42 @@ spec:
                 x-kubernetes-list-map-keys:
                 - name
                 x-kubernetes-list-type: map
+              schedulingStats:
+                description: schedulingStats tracks scheduling statistics
+                properties:
+                  evictions:
+                    description: evictions tracks eviction statistics by reason and
+                      underlyingCause.
+                    items:
+                      properties:
+                        count:
+                          description: count tracks the number of evictions for this
+                            reason and detailed reason.
+                          format: int32
+                          minimum: 0
+                          type: integer
+                        reason:
+                          description: reason specifies the programmatic identifier
+                            for the eviction cause.
+                          maxLength: 316
+                          type: string
+                        underlyingCause:
+                          description: |-
+                            underlyingCause specifies a finer-grained explanation that complements the eviction reason.
+                            This may be an empty string.
+                          maxLength: 316
+                          type: string
+                      required:
+                      - count
+                      - reason
+                      - underlyingCause
+                      type: object
+                    type: array
+                    x-kubernetes-list-map-keys:
+                    - reason
+                    - underlyingCause
+                    x-kubernetes-list-type: map
+                type: object
             type: object
         type: object
         x-kubernetes-validations:
diff --git a/pkg/controller/core/workload_controller.go b/pkg/controller/core/workload_controller.go
index d07ecbf161a..047e0c8234d 100644
--- a/pkg/controller/core/workload_controller.go
+++ b/pkg/controller/core/workload_controller.go
@@ -245,6 +245,7 @@ func (r *WorkloadReconciler) Reconcile(ctx context.Context, req ctrl.Request) (c
 			wl.Status.RequeueState = nil
 			updated = true
 		}
+		reportWorkloadEvictedOnce := workload.WorkloadEvictionStateInc(&wl, kueue.WorkloadDeactivated, "")
 		updated = workload.ResetChecksOnEviction(&wl, r.clock.Now()) || updated
 		if updated {
 			if err := workload.ApplyAdmissionStatus(ctx, r.client, &wl, true, r.clock); err != nil {
@@ -252,6 +253,9 @@ func (r *WorkloadReconciler) Reconcile(ctx context.Context, req ctrl.Request) (c
 			}
 			if evicted && wl.Status.Admission != nil {
 				workload.ReportEvictedWorkload(r.recorder, &wl, wl.Status.Admission.ClusterQueue, reason, message)
+				if reportWorkloadEvictedOnce {
+					metrics.ReportEvictedWorkloadsOnce(wl.Status.Admission.ClusterQueue, kueue.WorkloadDeactivated, "")
+				}
 			}
 			return ctrl.Result{}, nil
 		}
@@ -432,11 +436,15 @@ func (r *WorkloadReconciler) reconcileCheckBasedEviction(ctx context.Context, wl
 	message := "At least one admission check is false"
 	workload.SetEvictedCondition(wl, kueue.WorkloadEvictedByAdmissionCheck, message)
 	workload.ResetChecksOnEviction(wl, r.clock.Now())
+	reportWorkloadEvictedOnce := workload.WorkloadEvictionStateInc(wl, kueue.WorkloadEvictedByAdmissionCheck, "")
 	if err := workload.ApplyAdmissionStatus(ctx, r.client, wl, true, r.clock); err != nil {
 		return false, client.IgnoreNotFound(err)
 	}
 	cqName, _ := r.queues.ClusterQueueForWorkload(wl)
 	workload.ReportEvictedWorkload(r.recorder, wl, cqName, kueue.WorkloadEvictedByAdmissionCheck, message)
+	if reportWorkloadEvictedOnce {
+		metrics.ReportEvictedWorkloadsOnce(wl.Status.Admission.ClusterQueue, kueue.WorkloadEvictedByAdmissionCheck, "")
+	}
 	return true, nil
 }
 
@@ -519,10 +527,14 @@ func (r *WorkloadReconciler) reconcileOnClusterQueueActiveState(ctx context.Cont
 		log.V(3).Info("Workload is evicted because the ClusterQueue is stopped", "clusterQueue", klog.KRef("", string(cqName)))
 		message := "The ClusterQueue is stopped"
 		workload.SetEvictedCondition(wl, kueue.WorkloadEvictedByClusterQueueStopped, message)
+		reportWorkloadEvictedOnce := workload.WorkloadEvictionStateInc(wl, kueue.WorkloadEvictedByClusterQueueStopped, "")
 		workload.ResetChecksOnEviction(wl, r.clock.Now())
 		err := workload.ApplyAdmissionStatus(ctx, r.client, wl, true, r.clock)
 		if err == nil {
 			workload.ReportEvictedWorkload(r.recorder, wl, cqName, kueue.WorkloadEvictedByClusterQueueStopped, message)
+			if reportWorkloadEvictedOnce {
+				metrics.ReportEvictedWorkloadsOnce(wl.Status.Admission.ClusterQueue, kueue.WorkloadEvictedByClusterQueueStopped, "")
+			}
 		}
 		return true, client.IgnoreNotFound(err)
 	}
@@ -584,8 +596,9 @@ func (r *WorkloadReconciler) reconcileNotReadyTimeout(ctx context.Context, req c
 		// the workload has already been evicted by the PodsReadyTimeout or been deactivated.
 		return 0, nil
 	}
-	countingTowardsTimeout, recheckAfter := r.admittedNotReadyWorkload(wl)
-	if !countingTowardsTimeout {
+
+	underlyingCause, recheckAfter := r.admittedNotReadyWorkload(wl)
+	if underlyingCause == "" {
 		return 0, nil
 	}
 	if recheckAfter > 0 {
@@ -599,10 +612,14 @@ func (r *WorkloadReconciler) reconcileNotReadyTimeout(ctx context.Context, req c
 	message := fmt.Sprintf("Exceeded the PodsReady timeout %s", req.String())
 	workload.SetEvictedCondition(wl, kueue.WorkloadEvictedByPodsReadyTimeout, message)
 	workload.ResetChecksOnEviction(wl, r.clock.Now())
+	reportWorkloadEvictedOnce := workload.WorkloadEvictionStateInc(wl, kueue.WorkloadEvictedByPodsReadyTimeout, underlyingCause)
 	err := workload.ApplyAdmissionStatus(ctx, r.client, wl, true, r.clock)
 	if err == nil {
 		cqName, _ := r.queues.ClusterQueueForWorkload(wl)
 		workload.ReportEvictedWorkload(r.recorder, wl, cqName, kueue.WorkloadEvictedByPodsReadyTimeout, message)
+		if reportWorkloadEvictedOnce {
+			metrics.ReportEvictedWorkloadsOnce(cqName, kueue.WorkloadEvictedByPodsReadyTimeout, underlyingCause)
+		}
 	}
 	return 0, client.IgnoreNotFound(err)
 }
@@ -836,37 +853,44 @@ func (r *WorkloadReconciler) SetupWithManager(mgr ctrl.Manager, cfg *config.Conf
 		Complete(WithLeadingManager(mgr, r, &kueue.Workload{}, cfg))
 }
 
-// admittedNotReadyWorkload returns as a pair of values. The first boolean determines
-// if the workload is currently counting towards the timeout for PodsReady, i.e.
-// it has the Admitted condition True and the PodsReady condition not equal
-// True (False or not set). The second value is the remaining time to exceed the
-// specified timeout counted since max of the LastTransitionTime's for the
-// Admitted and PodsReady conditions.
-func (r *WorkloadReconciler) admittedNotReadyWorkload(wl *kueue.Workload) (bool, time.Duration) {
+// admittedNotReadyWorkload checks if a workload counts toward the PodsReady timeout
+// and calculates the remaining timeout duration.
+//
+// It returns two values:
+//  1. A underlyingCause that complements informarion carried by kueue.WorkloadEvictedByPodsReadyTimeout
+//
+// (e.g., WaitForStart, WaitForRecovery, or empty if not applicable).
+//
+//  2. The remaining time (in seconds) until the timeout is exceeded, based on
+//     the maximum of the LastTransitionTime for the Admitted and PodsReady conditions.
+//
+// If the workload is not admitted, PodsReady is true, or no timeout is configured,
+// it returns an empty underlyingCause and zero duration.
+func (r *WorkloadReconciler) admittedNotReadyWorkload(wl *kueue.Workload) (string, time.Duration) {
 	if r.waitForPodsReady == nil {
 		// the timeout is not configured for the workload controller
-		return false, 0
+		return "", 0
 	}
 	if !workload.IsAdmitted(wl) {
 		// the workload is not admitted so there is no need to time it out
-		return false, 0
+		return "", 0
 	}
 
 	podsReadyCond := apimeta.FindStatusCondition(wl.Status.Conditions, kueue.WorkloadPodsReady)
 	if podsReadyCond != nil && podsReadyCond.Status == metav1.ConditionTrue {
-		return false, 0
+		return "", 0
 	}
 
 	if podsReadyCond == nil || podsReadyCond.Reason == kueue.WorkloadWaitForStart || podsReadyCond.Reason == "PodsReady" {
 		admittedCond := apimeta.FindStatusCondition(wl.Status.Conditions, kueue.WorkloadAdmitted)
 		elapsedTime := r.clock.Since(admittedCond.LastTransitionTime.Time)
-		return true, max(r.waitForPodsReady.timeout-elapsedTime, 0)
+		return kueue.WorkloadWaitForStart, max(r.waitForPodsReady.timeout-elapsedTime, 0)
 	} else if podsReadyCond.Reason == kueue.WorkloadWaitForRecovery && r.waitForPodsReady.recoveryTimeout != nil {
 		// A pod has failed and the workload is waiting for recovery
 		elapsedTime := r.clock.Since(podsReadyCond.LastTransitionTime.Time)
-		return true, max(*r.waitForPodsReady.recoveryTimeout-elapsedTime, 0)
+		return kueue.WorkloadWaitForRecovery, max(*r.waitForPodsReady.recoveryTimeout-elapsedTime, 0)
 	}
-	return false, 0
+	return "", 0
 }
 
 type resourceUpdatesHandler struct {
diff --git a/pkg/controller/core/workload_controller_test.go b/pkg/controller/core/workload_controller_test.go
index 9c17962325e..58a3bd04264 100644
--- a/pkg/controller/core/workload_controller_test.go
+++ b/pkg/controller/core/workload_controller_test.go
@@ -50,10 +50,10 @@ func TestAdmittedNotReadyWorkload(t *testing.T) {
 	fakeClock := testingclock.NewFakeClock(now)
 
 	testCases := map[string]struct {
-		workload                   kueue.Workload
-		waitForPodsReady           *waitForPodsReadyConfig
-		wantCountingTowardsTimeout bool
-		wantRecheckAfter           time.Duration
+		workload            kueue.Workload
+		waitForPodsReady    *waitForPodsReadyConfig
+		wantUnderlyingCause string
+		wantRecheckAfter    time.Duration
 	}{
 		"workload without Admitted condition; not counting": {
 			workload: kueue.Workload{},
@@ -71,9 +71,9 @@ func TestAdmittedNotReadyWorkload(t *testing.T) {
 					},
 				},
 			},
-			waitForPodsReady:           &waitForPodsReadyConfig{timeout: 5 * time.Minute},
-			wantCountingTowardsTimeout: true,
-			wantRecheckAfter:           4 * time.Minute,
+			waitForPodsReady:    &waitForPodsReadyConfig{timeout: 5 * time.Minute},
+			wantUnderlyingCause: kueue.WorkloadWaitForStart,
+			wantRecheckAfter:    4 * time.Minute,
 		},
 		"workload with Admitted=True, no PodsReady, but no timeout configured; not counting": {
 			workload: kueue.Workload{
@@ -102,9 +102,9 @@ func TestAdmittedNotReadyWorkload(t *testing.T) {
 					},
 				},
 			},
-			waitForPodsReady:           &waitForPodsReadyConfig{timeout: 5 * time.Minute},
-			wantCountingTowardsTimeout: true,
-			wantRecheckAfter:           0,
+			waitForPodsReady:    &waitForPodsReadyConfig{timeout: 5 * time.Minute},
+			wantUnderlyingCause: kueue.WorkloadWaitForStart,
+			wantRecheckAfter:    0,
 		},
 		"with reason WorkloadWaitForPodsReadyStart; workload with Admitted=True, PodsReady=False; counting since admitted.LastTransitionTime": {
 			workload: kueue.Workload{
@@ -125,9 +125,9 @@ func TestAdmittedNotReadyWorkload(t *testing.T) {
 					},
 				},
 			},
-			waitForPodsReady:           &waitForPodsReadyConfig{timeout: 5 * time.Minute},
-			wantCountingTowardsTimeout: true,
-			wantRecheckAfter:           4 * time.Minute,
+			waitForPodsReady:    &waitForPodsReadyConfig{timeout: 5 * time.Minute},
+			wantUnderlyingCause: kueue.WorkloadWaitForStart,
+			wantRecheckAfter:    4 * time.Minute,
 		},
 		"with reason PodsReady; workload with Admitted=True, PodsReady=False; counting since admitted.LastTransitionTime": {
 			workload: kueue.Workload{
@@ -148,9 +148,9 @@ func TestAdmittedNotReadyWorkload(t *testing.T) {
 					},
 				},
 			},
-			waitForPodsReady:           &waitForPodsReadyConfig{timeout: 5 * time.Minute},
-			wantCountingTowardsTimeout: true,
-			wantRecheckAfter:           4 * time.Minute,
+			waitForPodsReady:    &waitForPodsReadyConfig{timeout: 5 * time.Minute},
+			wantUnderlyingCause: kueue.WorkloadWaitForStart,
+			wantRecheckAfter:    4 * time.Minute,
 		},
 		"workload with Admitted=True, PodsReady=False, Reason=WorkloadWaitForPodsReadyRecovery": {
 			workload: kueue.Workload{
@@ -171,9 +171,9 @@ func TestAdmittedNotReadyWorkload(t *testing.T) {
 					},
 				},
 			},
-			waitForPodsReady:           &waitForPodsReadyConfig{recoveryTimeout: ptr.To(3 * time.Minute)},
-			wantCountingTowardsTimeout: true,
-			wantRecheckAfter:           3 * time.Minute,
+			waitForPodsReady:    &waitForPodsReadyConfig{recoveryTimeout: ptr.To(3 * time.Minute)},
+			wantUnderlyingCause: kueue.WorkloadWaitForRecovery,
+			wantRecheckAfter:    3 * time.Minute,
 		},
 		"workload with Admitted=True, PodsReady=False, Reason=WorkloadWaitForPodsReadyRecovery, recoveryTimeout not configured": {
 			workload: kueue.Workload{
@@ -194,8 +194,7 @@ func TestAdmittedNotReadyWorkload(t *testing.T) {
 					},
 				},
 			},
-			waitForPodsReady:           &waitForPodsReadyConfig{recoveryTimeout: nil},
-			wantCountingTowardsTimeout: false,
+			waitForPodsReady: &waitForPodsReadyConfig{recoveryTimeout: nil},
 		},
 		"workload with Admitted=Unknown; not counting": {
 			workload: kueue.Workload{
@@ -252,14 +251,14 @@ func TestAdmittedNotReadyWorkload(t *testing.T) {
 	for name, tc := range testCases {
 		t.Run(name, func(t *testing.T) {
 			wRec := WorkloadReconciler{waitForPodsReady: tc.waitForPodsReady, clock: fakeClock}
-			countingTowardsTimeout, recheckAfter := wRec.admittedNotReadyWorkload(&tc.workload)
+			underlyingCause, recheckAfter := wRec.admittedNotReadyWorkload(&tc.workload)
 
-			if tc.wantCountingTowardsTimeout != countingTowardsTimeout {
-				t.Errorf("Unexpected countingTowardsTimeout, want=%v, got=%v", tc.wantCountingTowardsTimeout, countingTowardsTimeout)
-			}
 			if tc.wantRecheckAfter != recheckAfter {
 				t.Errorf("Unexpected recheckAfter, want=%v, got=%v", tc.wantRecheckAfter, recheckAfter)
 			}
+			if tc.wantUnderlyingCause != underlyingCause {
+				t.Errorf("Unexpected underlyingCause, want=%v, got=%v", tc.wantUnderlyingCause, underlyingCause)
+			}
 		})
 	}
 }
@@ -676,6 +675,12 @@ func TestReconcile(t *testing.T) {
 						Message: "Admission check(s): check-1, were rejected",
 					},
 				).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason: kueue.WorkloadDeactivated,
+						Count:  1,
+					},
+				).
 				Obj(),
 			wantEvents: []utiltesting.EventRecord{
 				{
@@ -716,6 +721,12 @@ func TestReconcile(t *testing.T) {
 					Reason:  "AdmissionCheck",
 					Message: "At least one admission check is false",
 				}).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason: kueue.WorkloadEvictedByAdmissionCheck,
+						Count:  1,
+					},
+				).
 				Obj(),
 			wantEvents: []utiltesting.EventRecord{
 				{
@@ -751,6 +762,20 @@ func TestReconcile(t *testing.T) {
 				}).
 				Admitted(true).
 				RequeueState(ptr.To[int32](3), nil).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason:          kueue.WorkloadEvictedByPodsReadyTimeout,
+						UnderlyingCause: kueue.WorkloadWaitForRecovery,
+						Count:           1,
+					},
+				).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason:          kueue.WorkloadEvictedByPodsReadyTimeout,
+						UnderlyingCause: kueue.WorkloadWaitForStart,
+						Count:           1,
+					},
+				).
 				Generation(1).
 				Obj(),
 			wantWorkload: utiltesting.MakeWorkload("wl", "ns").
@@ -771,6 +796,21 @@ func TestReconcile(t *testing.T) {
 				}).
 				// 10s * 2^(4-1) = 80s
 				RequeueState(ptr.To[int32](4), ptr.To(metav1.NewTime(testStartTime.Add(80*time.Second).Truncate(time.Second)))).
+				// check EvictionState mergeStrategy
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason:          kueue.WorkloadEvictedByPodsReadyTimeout,
+						UnderlyingCause: kueue.WorkloadWaitForRecovery,
+						Count:           1,
+					},
+				).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason:          kueue.WorkloadEvictedByPodsReadyTimeout,
+						UnderlyingCause: kueue.WorkloadWaitForStart,
+						Count:           2,
+					},
+				).
 				Obj(),
 			wantEvents: []utiltesting.EventRecord{
 				{
@@ -866,6 +906,89 @@ func TestReconcile(t *testing.T) {
 				}).
 				//  10s * 2^(11-1) = 10240s > requeuingBackoffMaxSeconds; then wait time should be limited to requeuingBackoffMaxSeconds
 				RequeueState(ptr.To[int32](11), ptr.To(metav1.NewTime(testStartTime.Add(7200*time.Second).Truncate(time.Second)))).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason:          kueue.WorkloadEvictedByPodsReadyTimeout,
+						UnderlyingCause: kueue.WorkloadWaitForStart,
+						Count:           1,
+					},
+				).
+				Obj(),
+			wantEvents: []utiltesting.EventRecord{
+				{
+					Key:       types.NamespacedName{Name: "wl", Namespace: "ns"},
+					EventType: corev1.EventTypeNormal,
+					Reason:    "EvictedDueToPodsReadyTimeout",
+					Message:   "Exceeded the PodsReady timeout ns/wl",
+				},
+			},
+		},
+		"recovery time should be limited to recoveryTimeout": {
+			reconcilerOpts: []Option{
+				WithWaitForPodsReady(&waitForPodsReadyConfig{
+					timeout:                     5 * time.Minute,
+					recoveryTimeout:             ptr.To(3 * time.Second),
+					requeuingBackoffLimitCount:  ptr.To[int32](100),
+					requeuingBackoffBaseSeconds: 10,
+					requeuingBackoffJitter:      0,
+					requeuingBackoffMaxDuration: time.Duration(7200) * time.Second,
+				}),
+			},
+			workload: utiltesting.MakeWorkload("wl", "ns").
+				ReserveQuota(utiltesting.MakeAdmission("q1").Obj()).
+				AdmissionCheck(kueue.AdmissionCheckState{
+					Name:  "check",
+					State: kueue.CheckStateReady,
+				}).
+				Generation(1).
+				Condition(metav1.Condition{ // Override LastTransitionTime
+					Type:               kueue.WorkloadAdmitted,
+					Status:             metav1.ConditionTrue,
+					LastTransitionTime: metav1.NewTime(testStartTime.Add(-5 * time.Minute)),
+					Reason:             "ByTest",
+					Message:            "Admitted by ClusterQueue q1",
+				}).
+				Condition(metav1.Condition{
+					Type:               kueue.WorkloadPodsReady,
+					Status:             metav1.ConditionFalse,
+					LastTransitionTime: metav1.NewTime(testStartTime.Add(-3 * time.Minute)),
+					Reason:             kueue.WorkloadWaitForRecovery,
+					Message:            "At least one pod has failed, waiting for recovery",
+				}).
+				Admitted(true).
+				Obj(),
+			wantWorkload: utiltesting.MakeWorkload("wl", "ns").
+				ReserveQuota(utiltesting.MakeAdmission("q1").Obj()).
+				Admitted(true).
+				AdmissionCheck(kueue.AdmissionCheckState{
+					Name:    "check",
+					State:   kueue.CheckStatePending,
+					Message: "Reset to Pending after eviction. Previously: Ready",
+				}).
+				Generation(1).
+				Condition(metav1.Condition{
+					Type:               kueue.WorkloadEvicted,
+					Status:             metav1.ConditionTrue,
+					Reason:             kueue.WorkloadEvictedByPodsReadyTimeout,
+					Message:            "Exceeded the PodsReady timeout ns/wl",
+					ObservedGeneration: 1,
+				}).
+				Condition(metav1.Condition{
+					Type:               kueue.WorkloadPodsReady,
+					Status:             metav1.ConditionFalse,
+					LastTransitionTime: metav1.NewTime(testStartTime),
+					Reason:             kueue.WorkloadWaitForRecovery,
+					Message:            "At least one pod has failed, waiting for recovery",
+				}).
+				//  10s * 2^(11-1) = 10240s > requeuingBackoffMaxSeconds; then wait time should be limited to requeuingBackoffMaxSeconds
+				RequeueState(ptr.To[int32](1), ptr.To(metav1.NewTime(testStartTime.Add(7200*time.Second).Truncate(time.Second)))).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason:          kueue.WorkloadEvictedByPodsReadyTimeout,
+						UnderlyingCause: kueue.WorkloadWaitForRecovery,
+						Count:           1,
+					},
+				).
 				Obj(),
 			wantEvents: []utiltesting.EventRecord{
 				{
@@ -1078,6 +1201,12 @@ func TestReconcile(t *testing.T) {
 					Reason:  kueue.WorkloadDeactivated,
 					Message: "The workload is deactivated",
 				}).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason: kueue.WorkloadDeactivated,
+						Count:  1,
+					},
+				).
 				Obj(),
 		},
 		"should set the Evicted condition with Deactivated reason when the .spec.active=False and Admitted": {
@@ -1096,6 +1225,12 @@ func TestReconcile(t *testing.T) {
 					Reason:  kueue.WorkloadDeactivated,
 					Message: "The workload is deactivated",
 				}).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason: kueue.WorkloadDeactivated,
+						Count:  1,
+					},
+				).
 				Obj(),
 			wantEvents: []utiltesting.EventRecord{
 				{
@@ -1117,6 +1252,13 @@ func TestReconcile(t *testing.T) {
 					Reason:  kueue.WorkloadEvictedByPodsReadyTimeout,
 					Message: "Exceeded the PodsReady timeout ns",
 				}).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason:          kueue.WorkloadEvictedByPodsReadyTimeout,
+						UnderlyingCause: kueue.WorkloadWaitForStart,
+						Count:           1,
+					},
+				).
 				Obj(),
 			wantWorkload: utiltesting.MakeWorkload("wl", "ns").
 				Active(false).
@@ -1128,6 +1270,19 @@ func TestReconcile(t *testing.T) {
 					Reason:  kueue.WorkloadDeactivated,
 					Message: "The workload is deactivated",
 				}).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason:          kueue.WorkloadEvictedByPodsReadyTimeout,
+						UnderlyingCause: kueue.WorkloadWaitForStart,
+						Count:           1,
+					},
+				).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason: kueue.WorkloadDeactivated,
+						Count:  1,
+					},
+				).
 				Obj(),
 			wantEvents: []utiltesting.EventRecord{
 				{
@@ -1188,6 +1343,9 @@ func TestReconcile(t *testing.T) {
 					Reason:  kueue.WorkloadRequeuingLimitExceeded,
 					Message: "exceeding the maximum number of re-queuing retries",
 				}).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{Reason: kueue.WorkloadDeactivated, Count: 1},
+				).
 				Obj(),
 			wantEvents: []utiltesting.EventRecord{
 				{
@@ -1248,6 +1406,12 @@ func TestReconcile(t *testing.T) {
 					Reason:  kueue.WorkloadRequeuingLimitExceeded,
 					Message: "exceeding the maximum number of re-queuing retries",
 				}).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason: kueue.WorkloadDeactivated,
+						Count:  1,
+					},
+				).
 				Obj(),
 			wantEvents: []utiltesting.EventRecord{
 				{
@@ -1311,6 +1475,12 @@ func TestReconcile(t *testing.T) {
 				}).
 				// The requeueState should be reset in the real cluster, but the fake client doesn't allow us to do it.
 				RequeueState(ptr.To[int32](100), nil).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason: kueue.WorkloadDeactivated,
+						Count:  1,
+					},
+				).
 				Obj(),
 			wantEvents: []utiltesting.EventRecord{
 				{
@@ -1374,6 +1544,12 @@ func TestReconcile(t *testing.T) {
 				}).
 				// The requeueState should be reset in the real cluster, but the fake client doesn't allow us to do it.
 				RequeueState(ptr.To[int32](100), nil).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason: kueue.WorkloadDeactivated,
+						Count:  1,
+					},
+				).
 				Obj(),
 			wantEvents: []utiltesting.EventRecord{
 				{
@@ -1428,6 +1604,12 @@ func TestReconcile(t *testing.T) {
 					Reason:  kueue.WorkloadEvictedByClusterQueueStopped,
 					Message: "The ClusterQueue is stopped",
 				}).
+				SchedulingStatsEviction(
+					kueue.WorkloadSchedulingStatsEvicition{
+						Reason: kueue.WorkloadEvictedByClusterQueueStopped,
+						Count:  1,
+					},
+				).
 				Obj(),
 			wantEvents: []utiltesting.EventRecord{
 				{
diff --git a/pkg/metrics/metrics.go b/pkg/metrics/metrics.go
index e7664718188..1b059599a1d 100644
--- a/pkg/metrics/metrics.go
+++ b/pkg/metrics/metrics.go
@@ -269,6 +269,20 @@ The label 'reason' can have the following values:
 		}, []string{"name", "namespace", "reason"},
 	)
 
+	EvictedWorkloadsOnceTotal = prometheus.NewCounterVec(
+		prometheus.CounterOpts{
+			Subsystem: constants.KueueName,
+			Name:      "evicted_workloads_once_total",
+			Help: `The number of unique workload evictions per 'cluster_queue',
+The label 'reason' can have the following values:
+- "Preempted" means that the workload was evicted in order to free resources for a workload with a higher priority or reclamation of nominal quota.
+- "PodsReadyTimeout" means that the eviction took place due to a PodsReady timeout.
+- "AdmissionCheck" means that the workload was evicted because at least one admission check transitioned to False.
+- "ClusterQueueStopped" means that the workload was evicted because the ClusterQueue is stopped.
+- "Deactivated" means that the workload was evicted because spec.active is set to false`,
+		}, []string{"cluster_queue", "reason", "detailed_reason"},
+	)
+
 	PreemptedWorkloadsTotal = prometheus.NewCounterVec(
 		prometheus.CounterOpts{
 			Subsystem: constants.KueueName,
@@ -490,6 +504,10 @@ func ReportLocalQueueEvictedWorkloads(lq LocalQueueReference, reason string) {
 	LocalQueueEvictedWorkloadsTotal.WithLabelValues(string(lq.Name), lq.Namespace, reason).Inc()
 }
 
+func ReportEvictedWorkloadsOnce(cqName kueue.ClusterQueueReference, reason, underlyingCause string) {
+	EvictedWorkloadsOnceTotal.WithLabelValues(string(cqName), reason, underlyingCause).Inc()
+}
+
 func ReportPreemption(preemptingCqName kueue.ClusterQueueReference, preemptingReason string, targetCqName kueue.ClusterQueueReference) {
 	PreemptedWorkloadsTotal.WithLabelValues(string(preemptingCqName), preemptingReason).Inc()
 	ReportEvictedWorkloads(targetCqName, kueue.WorkloadEvictedByPreemption)
@@ -514,6 +532,7 @@ func ClearClusterQueueMetrics(cqName string) {
 	queuedUntilReadyWaitTime.DeleteLabelValues(cqName)
 	admittedUntilReadyWaitTime.DeleteLabelValues(cqName)
 	EvictedWorkloadsTotal.DeletePartialMatch(prometheus.Labels{"cluster_queue": cqName})
+	EvictedWorkloadsOnceTotal.DeletePartialMatch(prometheus.Labels{"cluster_queue": cqName})
 	PreemptedWorkloadsTotal.DeletePartialMatch(prometheus.Labels{"preempting_cluster_queue": cqName})
 }
 
@@ -679,6 +698,7 @@ func Register() {
 		quotaReservedWaitTime,
 		AdmittedWorkloadsTotal,
 		EvictedWorkloadsTotal,
+		EvictedWorkloadsOnceTotal,
 		PreemptedWorkloadsTotal,
 		admissionWaitTime,
 		admissionChecksWaitTime,
diff --git a/pkg/scheduler/preemption/preemption.go b/pkg/scheduler/preemption/preemption.go
index d67ea31ed9c..b87c1d20096 100644
--- a/pkg/scheduler/preemption/preemption.go
+++ b/pkg/scheduler/preemption/preemption.go
@@ -186,7 +186,11 @@ func (p *Preemptor) applyPreemptionWithSSA(ctx context.Context, w *kueue.Workloa
 	w = w.DeepCopy()
 	workload.SetEvictedCondition(w, kueue.WorkloadEvictedByPreemption, message)
 	workload.ResetChecksOnEviction(w, p.clock.Now())
+	reportWorkloadEvictedOnce := workload.WorkloadEvictionStateInc(w, kueue.WorkloadEvictedByPreemption, "")
 	workload.SetPreemptedCondition(w, reason, message)
+	if reportWorkloadEvictedOnce {
+		metrics.ReportEvictedWorkloadsOnce(w.Status.Admission.ClusterQueue, kueue.WorkloadEvictedByPreemption, "")
+	}
 	return workload.ApplyAdmissionStatus(ctx, p.client, w, true, p.clock)
 }
 
diff --git a/pkg/util/testing/wrappers.go b/pkg/util/testing/wrappers.go
index bb1777c0411..11382bfab03 100644
--- a/pkg/util/testing/wrappers.go
+++ b/pkg/util/testing/wrappers.go
@@ -373,6 +373,14 @@ func (w *WorkloadWrapper) PastAdmittedTime(v int32) *WorkloadWrapper {
 	return w
 }
 
+func (w *WorkloadWrapper) SchedulingStatsEviction(evictionState kueue.WorkloadSchedulingStatsEvicition) *WorkloadWrapper {
+	if w.Status.SchedulingStats == nil {
+		w.Status.SchedulingStats = &kueue.SchedulingStats{}
+	}
+	w.Status.SchedulingStats.Evictions = append(w.Status.SchedulingStats.Evictions, evictionState)
+	return w
+}
+
 type PodSetWrapper struct{ kueue.PodSet }
 
 func MakePodSet(name kueue.PodSetReference, count int) *PodSetWrapper {
diff --git a/pkg/workload/workload.go b/pkg/workload/workload.go
index 2ae3d9d7afd..6d8bfd1fccb 100644
--- a/pkg/workload/workload.go
+++ b/pkg/workload/workload.go
@@ -746,6 +746,12 @@ func AdmissionStatusPatch(w *kueue.Workload, wlCopy *kueue.Workload, strict bool
 		wlCopy.ResourceVersion = w.ResourceVersion
 	}
 	wlCopy.Status.AccumulatedPastExexcutionTimeSeconds = w.Status.AccumulatedPastExexcutionTimeSeconds
+	if w.Status.SchedulingStats != nil {
+		if wlCopy.Status.SchedulingStats == nil {
+			wlCopy.Status.SchedulingStats = &kueue.SchedulingStats{}
+		}
+		wlCopy.Status.SchedulingStats.Evictions = append(wlCopy.Status.SchedulingStats.Evictions, w.Status.SchedulingStats.Evictions...)
+	}
 }
 
 func AdmissionChecksStatusPatch(w *kueue.Workload, wlCopy *kueue.Workload, c clock.Clock) {
@@ -969,3 +975,44 @@ func References(wls []*Info) []klog.ObjectRef {
 	}
 	return keys
 }
+
+func WorkloadEvictionStateInc(wl *kueue.Workload, reason, underlyingCause string) bool {
+	evictionState := FindSchedulingStatsEvictionByReason(wl, reason, underlyingCause)
+	if evictionState == nil {
+		evictionState = &kueue.WorkloadSchedulingStatsEvicition{
+			Reason:          reason,
+			UnderlyingCause: underlyingCause,
+		}
+	}
+	report := evictionState.Count == 0
+	evictionState.Count++
+	SetSchedulingStatsEviction(wl, *evictionState)
+	return report
+}
+
+func FindSchedulingStatsEvictionByReason(wl *kueue.Workload, reason, underlyingCause string) *kueue.WorkloadSchedulingStatsEvicition {
+	if wl.Status.SchedulingStats != nil {
+		for i := range wl.Status.SchedulingStats.Evictions {
+			if wl.Status.SchedulingStats.Evictions[i].Reason == reason && wl.Status.SchedulingStats.Evictions[i].UnderlyingCause == underlyingCause {
+				return &wl.Status.SchedulingStats.Evictions[i]
+			}
+		}
+	}
+	return nil
+}
+
+func SetSchedulingStatsEviction(wl *kueue.Workload, newEvictionState kueue.WorkloadSchedulingStatsEvicition) bool {
+	if wl.Status.SchedulingStats == nil {
+		wl.Status.SchedulingStats = &kueue.SchedulingStats{}
+	}
+	evictionState := FindSchedulingStatsEvictionByReason(wl, newEvictionState.Reason, newEvictionState.UnderlyingCause)
+	if evictionState == nil {
+		wl.Status.SchedulingStats.Evictions = append(wl.Status.SchedulingStats.Evictions, newEvictionState)
+		return true
+	}
+	if evictionState.Count != newEvictionState.Count {
+		evictionState.Count = newEvictionState.Count
+		return true
+	}
+	return false
+}
diff --git a/site/content/en/docs/reference/kueue.v1beta1.md b/site/content/en/docs/reference/kueue.v1beta1.md
index 701be7e4edd..60fb1993f83 100644
--- a/site/content/en/docs/reference/kueue.v1beta1.md
+++ b/site/content/en/docs/reference/kueue.v1beta1.md
@@ -2613,6 +2613,30 @@ words, it's the used quota that is over the nominalQuota.</p>
 </tbody>
 </table>
 
+## `SchedulingStats`     {#kueue-x-k8s-io-v1beta1-SchedulingStats}
+    
+
+**Appears in:**
+
+- [WorkloadStatus](#kueue-x-k8s-io-v1beta1-WorkloadStatus)
+
+
+
+<table class="table">
+<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
+<tbody>
+    
+  
+<tr><td><code>evictions</code><br/>
+<a href="#kueue-x-k8s-io-v1beta1-WorkloadSchedulingStatsEvicition"><code>[]WorkloadSchedulingStatsEvicition</code></a>
+</td>
+<td>
+   <p>evictions tracks eviction statistics by reason and underlyingCause.</p>
+</td>
+</tr>
+</tbody>
+</table>
+
 ## `StopPolicy`     {#kueue-x-k8s-io-v1beta1-StopPolicy}
     
 (Alias of `string`)
@@ -2742,6 +2766,45 @@ domain indicated by the values field.</p>
 
 
 
+## `WorkloadSchedulingStatsEvicition`     {#kueue-x-k8s-io-v1beta1-WorkloadSchedulingStatsEvicition}
+    
+
+**Appears in:**
+
+- [SchedulingStats](#kueue-x-k8s-io-v1beta1-SchedulingStats)
+
+
+
+<table class="table">
+<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
+<tbody>
+    
+  
+<tr><td><code>reason</code> <B>[Required]</B><br/>
+<code>string</code>
+</td>
+<td>
+   <p>reason specifies the programmatic identifier for the eviction cause.</p>
+</td>
+</tr>
+<tr><td><code>underlyingCause</code> <B>[Required]</B><br/>
+<code>string</code>
+</td>
+<td>
+   <p>underlyingCause specifies a finer-grained explanation that complements the eviction reason.
+This may be an empty string.</p>
+</td>
+</tr>
+<tr><td><code>count</code> <B>[Required]</B><br/>
+<code>int32</code>
+</td>
+<td>
+   <p>count tracks the number of evictions for this reason and detailed reason.</p>
+</td>
+</tr>
+</tbody>
+</table>
+
 ## `WorkloadSpec`     {#kueue-x-k8s-io-v1beta1-WorkloadSpec}
     
 
@@ -2915,6 +2978,13 @@ admission.resourceUsage contains the detailed information.</p>
 in Admitted state, in the previous <code>Admit</code> - <code>Evict</code> cycles.</p>
 </td>
 </tr>
+<tr><td><code>schedulingStats</code><br/>
+<a href="#kueue-x-k8s-io-v1beta1-SchedulingStats"><code>SchedulingStats</code></a>
+</td>
+<td>
+   <p>schedulingStats tracks scheduling statistics</p>
+</td>
+</tr>
 </tbody>
 </table>
   
\ No newline at end of file
diff --git a/site/content/en/docs/reference/metrics.md b/site/content/en/docs/reference/metrics.md
index 447d4d84818..b29f9b1bc93 100644
--- a/site/content/en/docs/reference/metrics.md
+++ b/site/content/en/docs/reference/metrics.md
@@ -31,6 +31,7 @@ Use the following metrics to monitor the status of your ClusterQueues:
 | `kueue_quota_reserved_wait_time_seconds`   | Histogram | The time between a workload was created or requeued until it got quota reservation. | `cluster_queue`: the name of the ClusterQueue                                                                                                                                                          |
 | `kueue_admitted_workloads_total`           | Counter   | The total number of admitted workloads.                                             | `cluster_queue`: the name of the ClusterQueue                                                                                                                                                          |
 | `kueue_evicted_workloads_total`            | Counter   | The total number of evicted workloads.                                              | `cluster_queue`: the name of the ClusterQueue<br> `reason`: Possible values are `Preempted`, `PodsReadyTimeout`, `AdmissionCheck`, `ClusterQueueStopped` or `Deactivated`                              |
+| `kueue_evicted_workloads_once_total`            | Counter   | The number of unique workload evictions per 'cluster_queue'                                              | `cluster_queue`: the name of the ClusterQueue<br> `reason`: Possible values are `Preempted`, `PodsReadyTimeout`, `AdmissionCheck`, `ClusterQueueStopped` or `Deactivated`<br> `detailedReason`: specifies a finer-grained explanation that complements the eviction cause                              |
 | `kueue_admission_wait_time_seconds`        | Histogram | The time between a workload was created or requeued until admission.                | `cluster_queue`: the name of the ClusterQueue                                                                                                                                                          |
 | `kueue_admission_checks_wait_time_seconds` | Histogram | The time from when a workload got the quota reservation until admission.            | `cluster_queue`: the name of the ClusterQueue                                                                                                                                                          |
 | `kueue_admitted_active_workloads`          | Gauge     | The number of admitted Workloads that are active (unsuspended and not finished)     | `cluster_queue`: the name of the ClusterQueue                                                                                                                                                          |
diff --git a/test/e2e/customconfigs/waitforpodsready_test.go b/test/e2e/customconfigs/waitforpodsready_test.go
index 51c5ca49ef1..730b4406bca 100644
--- a/test/e2e/customconfigs/waitforpodsready_test.go
+++ b/test/e2e/customconfigs/waitforpodsready_test.go
@@ -160,9 +160,22 @@ var _ = ginkgo.Describe("WaitForPodsReady Job Controller E2E", ginkgo.Ordered, f
 					g.Expect(k8sClient.Get(ctx, wlKey, &wl)).Should(gomega.Succeed())
 					g.Expect(wl.Status.Conditions).To(testing.HaveConditionStatusFalseAndReason(kueue.WorkloadPodsReady, kueue.WorkloadWaitForStart))
 					g.Expect(wl.Status.Conditions).To(testing.HaveConditionStatusTrueAndReason(kueue.WorkloadEvicted, kueue.WorkloadEvictedByPodsReadyTimeout))
+					g.Expect(wl.Status.SchedulingStats.Evictions).To(
+						gomega.BeComparableTo([]kueue.WorkloadSchedulingStatsEvicition{{
+							Reason:          kueue.WorkloadEvictedByPodsReadyTimeout,
+							UnderlyingCause: kueue.WorkloadWaitForStart,
+							Count:           1,
+						}}),
+					)
 				}, util.Timeout, util.Interval).Should(gomega.Succeed())
 			})
 
+			ginkgo.By("verifying that the metric is updated", func() {
+				util.ExpectMetricsToBeAvailable(ctx, cfg, restClient, curlPod.Name, curlContainerName, [][]string{
+					{"kueue_evicted_workloads_once_total", cq.Name, string(kueue.WorkloadEvictedByPodsReadyTimeout), kueue.WorkloadWaitForStart, "1"},
+				})
+			})
+
 			ginkgo.By("verifying that the job is suspended", func() {
 				gomega.Eventually(func(g gomega.Gomega) {
 					g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(job), job)).Should(gomega.Succeed())
@@ -242,8 +255,22 @@ var _ = ginkgo.Describe("WaitForPodsReady Job Controller E2E", ginkgo.Ordered, f
 					g.Expect(wl.Status.Conditions).To(testing.HaveConditionStatusTrueAndReason(kueue.WorkloadPodsReady, kueue.WorkloadStarted))
 					g.Expect(wl.Status.Conditions).To(testing.HaveConditionStatusFalse(kueue.WorkloadEvicted))
 					g.Expect(wl.Status.Conditions).To(testing.HaveConditionStatusTrue(kueue.WorkloadRequeued))
+
+					g.Expect(wl.Status.SchedulingStats.Evictions).To(
+						gomega.BeComparableTo([]kueue.WorkloadSchedulingStatsEvicition{{
+							Reason:          kueue.WorkloadEvictedByPodsReadyTimeout,
+							UnderlyingCause: kueue.WorkloadWaitForRecovery,
+							Count:           1,
+						}}),
+					)
 				}, util.LongTimeout, util.Interval).Should(gomega.Succeed())
 			})
+
+			ginkgo.By("verifying that the metric is updated", func() {
+				util.ExpectMetricsToBeAvailable(ctx, cfg, restClient, curlPod.Name, curlContainerName, [][]string{
+					{"kueue_evicted_workloads_once_total", cq.Name, string(kueue.WorkloadEvictedByPodsReadyTimeout), kueue.WorkloadWaitForRecovery, "1"},
+				})
+			})
 		})
 	})
 
@@ -308,6 +335,12 @@ var _ = ginkgo.Describe("WaitForPodsReady Job Controller E2E", ginkgo.Ordered, f
 					g.Expect(wl.Status.Conditions).To(testing.HaveConditionStatusTrueAndReason(kueue.WorkloadPodsReady, kueue.WorkloadRecovered))
 				}, util.LongTimeout, util.Interval).Should(gomega.Succeed())
 			})
+
+			ginkgo.By("verifying that the metric is not updated", func() {
+				util.ExpectMetricsNotToBeAvailable(ctx, cfg, restClient, curlPod.Name, curlContainerName, [][]string{
+					{"kueue_evicted_workloads_once_total", ns.Name},
+				})
+			})
 		})
 	})
 })
diff --git a/test/e2e/singlecluster/metrics_test.go b/test/e2e/singlecluster/metrics_test.go
index f0f066e4512..307125952a0 100644
--- a/test/e2e/singlecluster/metrics_test.go
+++ b/test/e2e/singlecluster/metrics_test.go
@@ -521,6 +521,7 @@ var _ = ginkgo.Describe("Metrics", func() {
 			metrics := [][]string{
 				{"kueue_admission_cycle_preemption_skips"},
 				{"kueue_evicted_workloads_total"},
+				{"kueue_evicted_workloads_once_total"},
 				{"kueue_preempted_workloads_total"},
 
 				{"kueue_local_queue_evicted_workloads_total"},
diff --git a/test/integration/singlecluster/controller/core/workload_controller_test.go b/test/integration/singlecluster/controller/core/workload_controller_test.go
index 5c02a53df53..0b6c06f10e3 100644
--- a/test/integration/singlecluster/controller/core/workload_controller_test.go
+++ b/test/integration/singlecluster/controller/core/workload_controller_test.go
@@ -311,6 +311,7 @@ var _ = ginkgo.Describe("Workload controller", ginkgo.Ordered, ginkgo.ContinueOn
 					g.Expect(k8sClient.Get(ctx, wlKey, updatedWl)).To(gomega.Succeed())
 					g.Expect(workload.IsEvictedByDeactivation(updatedWl)).To(gomega.BeTrue())
 					util.ExpectEvictedWorkloadsTotalMetric(clusterQueue.Name, "DeactivatedDueToAdmissionCheck", 1)
+					util.ExpectEvictedWorkloadsOnceTotalMetric(clusterQueue.Name, kueue.WorkloadDeactivated, "", 1)
 				}, util.Timeout, util.Interval).Should(gomega.Succeed())
 			})
 		})
@@ -395,6 +396,7 @@ var _ = ginkgo.Describe("Workload controller", ginkgo.Ordered, ginkgo.ContinueOn
 
 					g.Expect(workload.IsEvictedByDeactivation(updatedWl)).To(gomega.BeTrue())
 					util.ExpectEvictedWorkloadsTotalMetric(clusterQueue.Name, "DeactivatedDueToAdmissionCheck", 1)
+					util.ExpectEvictedWorkloadsOnceTotalMetric(clusterQueue.Name, kueue.WorkloadDeactivated, "", 1)
 				}, util.Timeout, util.Interval).Should(gomega.Succeed())
 			})
 		})
diff --git a/test/integration/singlecluster/scheduler/podsready/scheduler_test.go b/test/integration/singlecluster/scheduler/podsready/scheduler_test.go
index 375779488b8..f535f7cd442 100644
--- a/test/integration/singlecluster/scheduler/podsready/scheduler_test.go
+++ b/test/integration/singlecluster/scheduler/podsready/scheduler_test.go
@@ -255,6 +255,7 @@ var _ = ginkgo.Describe("SchedulerWithWaitForPodsReady", func() {
 
 			util.FinishEvictionForWorkloads(ctx, k8sClient, prodWl1)
 			util.ExpectEvictedWorkloadsTotalMetric(prodClusterQ.Name, kueue.WorkloadEvictedByPodsReadyTimeout, 1)
+			util.ExpectEvictedWorkloadsOnceTotalMetric(prodClusterQ.Name, kueue.WorkloadEvictedByPodsReadyTimeout, kueue.WorkloadWaitForStart, 1)
 
 			ginkgo.By("verify the 'prod2' workload gets admitted and the 'prod1' is pending by backoff")
 			util.ExpectWorkloadsToHaveQuotaReservation(ctx, k8sClient, prodClusterQ.Name, prodWl2)
@@ -306,6 +307,7 @@ var _ = ginkgo.Describe("SchedulerWithWaitForPodsReady", func() {
 			util.FinishEvictionForWorkloads(ctx, k8sClient, prodWl)
 			// should observe a metrics of WorkloadEvictedByDeactivation
 			util.ExpectEvictedWorkloadsTotalMetric(prodClusterQ.Name, "DeactivatedDueToRequeuingLimitExceeded", 1)
+			util.ExpectEvictedWorkloadsOnceTotalMetric(prodClusterQ.Name, kueue.WorkloadEvictedByPodsReadyTimeout, kueue.WorkloadWaitForStart, 1)
 
 			ginkgo.By("the reactivated workload should not be deactivated by the scheduler unless exceeding the backoffLimitCount")
 			gomega.Eventually(func(g gomega.Gomega) {
diff --git a/test/integration/singlecluster/scheduler/preemption_test.go b/test/integration/singlecluster/scheduler/preemption_test.go
index 373f0b42388..ee59a84c2ec 100644
--- a/test/integration/singlecluster/scheduler/preemption_test.go
+++ b/test/integration/singlecluster/scheduler/preemption_test.go
@@ -135,6 +135,7 @@ var _ = ginkgo.Describe("Preemption", func() {
 			util.FinishEvictionForWorkloads(ctx, k8sClient, lowWl1, lowWl2)
 			util.ExpectEvictedWorkloadsTotalMetric(cq.Name, kueue.WorkloadEvictedByPreemption, 2)
 			util.ExpectPreemptedWorkloadsTotalMetric(cq.Name, kueue.InClusterQueueReason, 2)
+			util.ExpectEvictedWorkloadsOnceTotalMetric(cq.Name, kueue.WorkloadEvictedByPreemption, "", 2)
 
 			util.ExpectWorkloadsToHaveQuotaReservation(ctx, k8sClient, cq.Name, highWl2)
 			util.ExpectWorkloadsToBePending(ctx, k8sClient, lowWl1, lowWl2)
diff --git a/test/integration/singlecluster/scheduler/scheduler_test.go b/test/integration/singlecluster/scheduler/scheduler_test.go
index f52bc13d514..f414247098a 100644
--- a/test/integration/singlecluster/scheduler/scheduler_test.go
+++ b/test/integration/singlecluster/scheduler/scheduler_test.go
@@ -1985,6 +1985,7 @@ var _ = ginkgo.Describe("Scheduler", func() {
 
 			util.ExpectClusterQueueStatusMetric(cq, metrics.CQStatusPending)
 			util.ExpectEvictedWorkloadsTotalMetric(clusterQueue.Name, kueue.WorkloadEvictedByClusterQueueStopped, 1)
+			util.ExpectEvictedWorkloadsOnceTotalMetric(cq.Name, kueue.WorkloadEvictedByClusterQueueStopped, "", 1)
 
 			ginkgo.By("Checking the condition of workload is evicted", func() {
 				createdWl := kueue.Workload{}
diff --git a/test/util/util.go b/test/util/util.go
index 4764918225c..1b30d01c728 100644
--- a/test/util/util.go
+++ b/test/util/util.go
@@ -567,6 +567,11 @@ func ExpectEvictedWorkloadsTotalMetric(cqName, reason string, v int) {
 	expectCounterMetric(metric, v)
 }
 
+func ExpectEvictedWorkloadsOnceTotalMetric(cqName string, reason, underlyingCause string, v int) {
+	metric := metrics.EvictedWorkloadsOnceTotal.WithLabelValues(cqName, reason, underlyingCause)
+	expectCounterMetric(metric, v)
+}
+
 func ExpectPreemptedWorkloadsTotalMetric(preemptorCqName, reason string, v int) {
 	metric := metrics.PreemptedWorkloadsTotal.WithLabelValues(preemptorCqName, reason)
 	expectCounterMetric(metric, v)


Reviews:
user: login: mykysha
body: Overall looks great, one minor comment and a question whether it would be possible to add the metric to the `test/e2e/singlecluster/metrics_test.go`?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T12:52:43Z
user: login: mszadkow
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T12:56:05Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T12:00:18Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T12:03:24Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T13:37:24Z
user: login: mszadkow
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T15:11:20Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T15:48:17Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T06:57:11Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T06:59:14Z
user: login: mszadkow
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T06:59:27Z
user: login: mszadkow
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T06:59:39Z
user: login: mszadkow
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T07:01:17Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T07:08:17Z
user: login: mszadkow
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T07:21:10Z
user: login: mszadkow
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T07:30:55Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T08:01:10Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T08:01:42Z
user: login: mszadkow
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T08:55:22Z
user: login: mszadkow
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T08:55:36Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T14:32:20Z

Review comments:
diff_hunk: @@ -1656,21 +1760,21 @@ func TestReconcile(t *testing.T) {
 				Obj(),
 		},
 
-		"admitted workload with max execution time": {
-			workload: utiltesting.MakeWorkload("wl", "ns").
-				ReserveQuota(utiltesting.MakeAdmission("q1").Obj()).
-				MaximumExecutionTimeSeconds(120).
-				AdmittedAt(true, testStartTime.Add(-time.Minute)).
-				ControllerReference(batchv1.SchemeGroupVersion.WithKind("Job"), "ownername", "owneruid").
-				Obj(),
-			wantWorkload: utiltesting.MakeWorkload("wl", "ns").
-				ReserveQuota(utiltesting.MakeAdmission("q1").Obj()).
-				MaximumExecutionTimeSeconds(120).
-				AdmittedAt(true, testStartTime.Add(-time.Minute)).
-				ControllerReference(batchv1.SchemeGroupVersion.WithKind("Job"), "ownername", "owneruid").
-				Obj(),
-			wantResult: reconcile.Result{RequeueAfter: time.Minute},
-		},
+		// "admitted workload with max execution time": {
path: pkg/controller/core/workload_controller_test.go
user: login: mykysha
body: Do we need this commented-out block?
created_at: 2025-05-16T12:50:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1656,21 +1760,21 @@ func TestReconcile(t *testing.T) {
 				Obj(),
 		},
 
-		"admitted workload with max execution time": {
-			workload: utiltesting.MakeWorkload("wl", "ns").
-				ReserveQuota(utiltesting.MakeAdmission("q1").Obj()).
-				MaximumExecutionTimeSeconds(120).
-				AdmittedAt(true, testStartTime.Add(-time.Minute)).
-				ControllerReference(batchv1.SchemeGroupVersion.WithKind("Job"), "ownername", "owneruid").
-				Obj(),
-			wantWorkload: utiltesting.MakeWorkload("wl", "ns").
-				ReserveQuota(utiltesting.MakeAdmission("q1").Obj()).
-				MaximumExecutionTimeSeconds(120).
-				AdmittedAt(true, testStartTime.Add(-time.Minute)).
-				ControllerReference(batchv1.SchemeGroupVersion.WithKind("Job"), "ownername", "owneruid").
-				Obj(),
-			wantResult: reconcile.Result{RequeueAfter: time.Minute},
-		},
+		// "admitted workload with max execution time": {
path: pkg/controller/core/workload_controller_test.go
user: login: mszadkow
body: ups, good catch
created_at: 2025-05-16T12:56:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -341,7 +341,7 @@ func GetKueueMetrics(ctx context.Context, cfg *rest.Config, restClient *rest.RES
 	metricsOutput, _, err := KExecute(ctx, cfg, restClient, configapi.DefaultNamespace, curlPodName, curlContainerName, []string{
 		"/bin/sh", "-c",
 		fmt.Sprintf(
-			"curl -s -k -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://%s.%s.svc.cluster.local:8443/metrics",
+			"curl --connect-timeout 10 --max-time 30 -s -k -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://%s.%s.svc.cluster.local:8443/metrics",
path: test/util/e2e.go
user: login: mimowo
body: Please move changes in the file to a dedicated PR as we may need to cherry-pick them in the future.
created_at: 2025-05-19T12:00:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -536,6 +578,17 @@ const (
 	WorkloadDeactivationTarget = "DeactivationTarget"
 )
 
+// Detailed Reasons for the WorkloadEvicted condition due to PodsReadyTimeout.
+const (
+	// StartupTimeoutReason is the exceedance of the maximum time allowed for a workload
+	// to transition from admission to pods being ready.
+	StartupTimeoutReason string = "StartupTimeout"
path: apis/kueue/v1beta1/workload_types.go
user: login: mimowo
body: I'm wondering about reusing the same reasons as "reasons" of the PodsReady condition: WaitForStart & WaitForRecovery. Would it be more or less work?
created_at: 2025-05-19T12:03:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -387,6 +387,48 @@ type WorkloadStatus struct {
 	//
 	// +optional
 	AccumulatedPastExexcutionTimeSeconds *int32 `json:"accumulatedPastExexcutionTimeSeconds,omitempty"`
+
+	// schedulingStats tracks scheduling statistics (e.g. evictionStates)
path: apis/kueue/v1beta1/workload_types.go
user: login: mimowo
body: ```suggestion
	// schedulingStats tracks scheduling statistics
```
created_at: 2025-05-19T13:37:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -536,6 +578,17 @@ const (
 	WorkloadDeactivationTarget = "DeactivationTarget"
 )
 
+// Detailed Reasons for the WorkloadEvicted condition due to PodsReadyTimeout.
+const (
+	// StartupTimeoutReason is the exceedance of the maximum time allowed for a workload
+	// to transition from admission to pods being ready.
+	StartupTimeoutReason string = "StartupTimeout"
path: apis/kueue/v1beta1/workload_types.go
user: login: mszadkow
body: I think they mean the same, we can use it easily.
created_at: 2025-05-19T15:11:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -536,6 +578,17 @@ const (
 	WorkloadDeactivationTarget = "DeactivationTarget"
 )
 
+// Detailed Reasons for the WorkloadEvicted condition due to PodsReadyTimeout.
+const (
+	// StartupTimeoutReason is the exceedance of the maximum time allowed for a workload
+	// to transition from admission to pods being ready.
+	StartupTimeoutReason string = "StartupTimeout"
path: apis/kueue/v1beta1/workload_types.go
user: login: mimowo
body: wdyt @tenzen-y , I would probably suggest to use the same values as in the reason, to avoid the unnecessary mapping 
created_at: 2025-05-19T15:48:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -387,6 +387,48 @@ type WorkloadStatus struct {
 	//
 	// +optional
 	AccumulatedPastExexcutionTimeSeconds *int32 `json:"accumulatedPastExexcutionTimeSeconds,omitempty"`
+
+	// schedulingStats tracks scheduling statistics (e.g. evictionStates)
path: apis/kueue/v1beta1/workload_types.go
user: login: mimowo
body: I suggested to remove because the field is no longer called evictionStates. However, feel free to adjust the comment according to recent API changes. For example e.g. eviction.
created_at: 2025-05-20T06:57:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -341,7 +341,7 @@ func GetKueueMetrics(ctx context.Context, cfg *rest.Config, restClient *rest.RES
 	metricsOutput, _, err := KExecute(ctx, cfg, restClient, configapi.DefaultNamespace, curlPodName, curlContainerName, []string{
 		"/bin/sh", "-c",
 		fmt.Sprintf(
-			"curl -s -k -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://%s.%s.svc.cluster.local:8443/metrics",
+			"curl --connect-timeout 10 --max-time 30 -s -k -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://%s.%s.svc.cluster.local:8443/metrics",
path: test/util/e2e.go
user: login: mimowo
body: I proposed separate PR because I assumed this is a fix for flakes, but please clarify what this is for. If indeed for flakes, then we merge separately and cherrypick. 
created_at: 2025-05-20T06:59:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -387,6 +387,48 @@ type WorkloadStatus struct {
 	//
 	// +optional
 	AccumulatedPastExexcutionTimeSeconds *int32 `json:"accumulatedPastExexcutionTimeSeconds,omitempty"`
+
+	// schedulingStats tracks scheduling statistics (e.g. evictionStates)
path: apis/kueue/v1beta1/workload_types.go
user: login: mszadkow
body: Removed
created_at: 2025-05-20T06:59:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -536,6 +578,17 @@ const (
 	WorkloadDeactivationTarget = "DeactivationTarget"
 )
 
+// Detailed Reasons for the WorkloadEvicted condition due to PodsReadyTimeout.
+const (
+	// StartupTimeoutReason is the exceedance of the maximum time allowed for a workload
+	// to transition from admission to pods being ready.
+	StartupTimeoutReason string = "StartupTimeout"
path: apis/kueue/v1beta1/workload_types.go
user: login: mszadkow
body: done
created_at: 2025-05-20T06:59:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -341,7 +341,7 @@ func GetKueueMetrics(ctx context.Context, cfg *rest.Config, restClient *rest.RES
 	metricsOutput, _, err := KExecute(ctx, cfg, restClient, configapi.DefaultNamespace, curlPodName, curlContainerName, []string{
 		"/bin/sh", "-c",
 		fmt.Sprintf(
-			"curl -s -k -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://%s.%s.svc.cluster.local:8443/metrics",
+			"curl --connect-timeout 10 --max-time 30 -s -k -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://%s.%s.svc.cluster.local:8443/metrics",
path: test/util/e2e.go
user: login: mszadkow
body: In case of timeout we don't wait 2 minutes rather 30 seconds.
I will add it in separate PR
created_at: 2025-05-20T07:01:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -354,7 +354,7 @@ func ExpectMetricsToBeAvailable(ctx context.Context, cfg *rest.Config, restClien
 		metricsOutput, err := GetKueueMetrics(ctx, cfg, restClient, curlPodName, curlContainerName)
 		g.Expect(err).NotTo(gomega.HaveOccurred())
 		g.Expect(metricsOutput).Should(utiltesting.ContainMetrics(metrics))
-	}, Timeout).Should(gomega.Succeed())
+	}, LongTimeout).Should(gomega.Succeed())
path: test/util/e2e.go
user: login: mimowo
body: Is this a fix for flake? If so please move to a separate PR (same with the other change in the file).
created_at: 2025-05-20T07:08:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -354,7 +354,7 @@ func ExpectMetricsToBeAvailable(ctx context.Context, cfg *rest.Config, restClien
 		metricsOutput, err := GetKueueMetrics(ctx, cfg, restClient, curlPodName, curlContainerName)
 		g.Expect(err).NotTo(gomega.HaveOccurred())
 		g.Expect(metricsOutput).Should(utiltesting.ContainMetrics(metrics))
-	}, Timeout).Should(gomega.Succeed())
+	}, LongTimeout).Should(gomega.Succeed())
path: test/util/e2e.go
user: login: mszadkow
body: ok, I am fine with that
created_at: 2025-05-20T07:21:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -354,7 +354,7 @@ func ExpectMetricsToBeAvailable(ctx context.Context, cfg *rest.Config, restClien
 		metricsOutput, err := GetKueueMetrics(ctx, cfg, restClient, curlPodName, curlContainerName)
 		g.Expect(err).NotTo(gomega.HaveOccurred())
 		g.Expect(metricsOutput).Should(utiltesting.ContainMetrics(metrics))
-	}, Timeout).Should(gomega.Succeed())
+	}, LongTimeout).Should(gomega.Succeed())
path: test/util/e2e.go
user: login: mszadkow
body: moved to separate PR

created_at: 2025-05-20T07:30:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -836,37 +853,40 @@ func (r *WorkloadReconciler) SetupWithManager(mgr ctrl.Manager, cfg *config.Conf
 		Complete(WithLeadingManager(mgr, r, &kueue.Workload{}, cfg))
 }
 
-// admittedNotReadyWorkload returns as a pair of values. The first boolean determines
-// if the workload is currently counting towards the timeout for PodsReady, i.e.
-// it has the Admitted condition True and the PodsReady condition not equal
-// True (False or not set). The second value is the remaining time to exceed the
-// specified timeout counted since max of the LastTransitionTime's for the
-// Admitted and PodsReady conditions.
-func (r *WorkloadReconciler) admittedNotReadyWorkload(wl *kueue.Workload) (bool, time.Duration) {
-	if r.waitForPodsReady == nil {
-		// the timeout is not configured for the workload controller
-		return false, 0
-	}
-	if !workload.IsAdmitted(wl) {
-		// the workload is not admitted so there is no need to time it out
-		return false, 0
+// admittedNotReadyWorkload checks if a workload counts toward the PodsReady timeout
+// and calculates the remaining timeout duration.
+//
+// It returns two values:
+//  1. A underlyingCause that complements informarion carried by kueue.WorkloadEvictedByPodsReadyTimeout
+//
+// (e.g., WaitForStart, WaitForRecovery, or empty if not applicable).
+//
+//  2. The remaining time (in seconds) until the timeout is exceeded, based on
+//     the maximum of the LastTransitionTime for the Admitted and PodsReady conditions.
+//
+// If the workload is not admitted, PodsReady is true, or no timeout is configured,
+// it returns an empty underlyingCause and zero duration.
+func (r *WorkloadReconciler) admittedNotReadyWorkload(wl *kueue.Workload) (string, time.Duration) {
+	// Return early if timeout is not configured or workload is not admitted
+	if r.waitForPodsReady == nil || !workload.IsAdmitted(wl) {
+		return "", 0
path: pkg/controller/core/workload_controller.go
user: login: mimowo
body: This looks like a useful optimization, but is this necessary for the PR? If all tests continue to pass I would suggest to drop the optimization. Let's focus on the metric, making the diff smaller will give us more confidence into correctness. If this was a bug it warrants a separate PR which we can cherry-pick.
created_at: 2025-05-20T08:01:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -836,37 +853,40 @@ func (r *WorkloadReconciler) SetupWithManager(mgr ctrl.Manager, cfg *config.Conf
 		Complete(WithLeadingManager(mgr, r, &kueue.Workload{}, cfg))
 }
 
-// admittedNotReadyWorkload returns as a pair of values. The first boolean determines
-// if the workload is currently counting towards the timeout for PodsReady, i.e.
-// it has the Admitted condition True and the PodsReady condition not equal
-// True (False or not set). The second value is the remaining time to exceed the
-// specified timeout counted since max of the LastTransitionTime's for the
-// Admitted and PodsReady conditions.
-func (r *WorkloadReconciler) admittedNotReadyWorkload(wl *kueue.Workload) (bool, time.Duration) {
-	if r.waitForPodsReady == nil {
-		// the timeout is not configured for the workload controller
-		return false, 0
-	}
-	if !workload.IsAdmitted(wl) {
-		// the workload is not admitted so there is no need to time it out
-		return false, 0
+// admittedNotReadyWorkload checks if a workload counts toward the PodsReady timeout
+// and calculates the remaining timeout duration.
+//
+// It returns two values:
+//  1. A underlyingCause that complements informarion carried by kueue.WorkloadEvictedByPodsReadyTimeout
+//
+// (e.g., WaitForStart, WaitForRecovery, or empty if not applicable).
+//
+//  2. The remaining time (in seconds) until the timeout is exceeded, based on
+//     the maximum of the LastTransitionTime for the Admitted and PodsReady conditions.
+//
+// If the workload is not admitted, PodsReady is true, or no timeout is configured,
+// it returns an empty underlyingCause and zero duration.
+func (r *WorkloadReconciler) admittedNotReadyWorkload(wl *kueue.Workload) (string, time.Duration) {
+	// Return early if timeout is not configured or workload is not admitted
+	if r.waitForPodsReady == nil || !workload.IsAdmitted(wl) {
+		return "", 0
 	}
 
 	podsReadyCond := apimeta.FindStatusCondition(wl.Status.Conditions, kueue.WorkloadPodsReady)
 	if podsReadyCond != nil && podsReadyCond.Status == metav1.ConditionTrue {
-		return false, 0
+		return "", 0
 	}
 
-	if podsReadyCond == nil || podsReadyCond.Reason == kueue.WorkloadWaitForStart || podsReadyCond.Reason == "PodsReady" {
+	if podsReadyCond == nil || podsReadyCond.Reason == kueue.WorkloadWaitForStart || podsReadyCond.Reason == kueue.WorkloadPodsReady {
path: pkg/controller/core/workload_controller.go
user: login: mimowo
body: I suggest to revert any drive-by cleanups and focus on the metric. We can have a separate PR for this.
created_at: 2025-05-20T08:01:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -836,37 +853,40 @@ func (r *WorkloadReconciler) SetupWithManager(mgr ctrl.Manager, cfg *config.Conf
 		Complete(WithLeadingManager(mgr, r, &kueue.Workload{}, cfg))
 }
 
-// admittedNotReadyWorkload returns as a pair of values. The first boolean determines
-// if the workload is currently counting towards the timeout for PodsReady, i.e.
-// it has the Admitted condition True and the PodsReady condition not equal
-// True (False or not set). The second value is the remaining time to exceed the
-// specified timeout counted since max of the LastTransitionTime's for the
-// Admitted and PodsReady conditions.
-func (r *WorkloadReconciler) admittedNotReadyWorkload(wl *kueue.Workload) (bool, time.Duration) {
-	if r.waitForPodsReady == nil {
-		// the timeout is not configured for the workload controller
-		return false, 0
-	}
-	if !workload.IsAdmitted(wl) {
-		// the workload is not admitted so there is no need to time it out
-		return false, 0
+// admittedNotReadyWorkload checks if a workload counts toward the PodsReady timeout
+// and calculates the remaining timeout duration.
+//
+// It returns two values:
+//  1. A underlyingCause that complements informarion carried by kueue.WorkloadEvictedByPodsReadyTimeout
+//
+// (e.g., WaitForStart, WaitForRecovery, or empty if not applicable).
+//
+//  2. The remaining time (in seconds) until the timeout is exceeded, based on
+//     the maximum of the LastTransitionTime for the Admitted and PodsReady conditions.
+//
+// If the workload is not admitted, PodsReady is true, or no timeout is configured,
+// it returns an empty underlyingCause and zero duration.
+func (r *WorkloadReconciler) admittedNotReadyWorkload(wl *kueue.Workload) (string, time.Duration) {
+	// Return early if timeout is not configured or workload is not admitted
+	if r.waitForPodsReady == nil || !workload.IsAdmitted(wl) {
+		return "", 0
 	}
 
 	podsReadyCond := apimeta.FindStatusCondition(wl.Status.Conditions, kueue.WorkloadPodsReady)
 	if podsReadyCond != nil && podsReadyCond.Status == metav1.ConditionTrue {
-		return false, 0
+		return "", 0
 	}
 
-	if podsReadyCond == nil || podsReadyCond.Reason == kueue.WorkloadWaitForStart || podsReadyCond.Reason == "PodsReady" {
+	if podsReadyCond == nil || podsReadyCond.Reason == kueue.WorkloadWaitForStart || podsReadyCond.Reason == kueue.WorkloadPodsReady {
path: pkg/controller/core/workload_controller.go
user: login: mszadkow
body: ok, I accept that
created_at: 2025-05-20T08:55:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -836,37 +853,40 @@ func (r *WorkloadReconciler) SetupWithManager(mgr ctrl.Manager, cfg *config.Conf
 		Complete(WithLeadingManager(mgr, r, &kueue.Workload{}, cfg))
 }
 
-// admittedNotReadyWorkload returns as a pair of values. The first boolean determines
-// if the workload is currently counting towards the timeout for PodsReady, i.e.
-// it has the Admitted condition True and the PodsReady condition not equal
-// True (False or not set). The second value is the remaining time to exceed the
-// specified timeout counted since max of the LastTransitionTime's for the
-// Admitted and PodsReady conditions.
-func (r *WorkloadReconciler) admittedNotReadyWorkload(wl *kueue.Workload) (bool, time.Duration) {
-	if r.waitForPodsReady == nil {
-		// the timeout is not configured for the workload controller
-		return false, 0
-	}
-	if !workload.IsAdmitted(wl) {
-		// the workload is not admitted so there is no need to time it out
-		return false, 0
+// admittedNotReadyWorkload checks if a workload counts toward the PodsReady timeout
+// and calculates the remaining timeout duration.
+//
+// It returns two values:
+//  1. A underlyingCause that complements informarion carried by kueue.WorkloadEvictedByPodsReadyTimeout
+//
+// (e.g., WaitForStart, WaitForRecovery, or empty if not applicable).
+//
+//  2. The remaining time (in seconds) until the timeout is exceeded, based on
+//     the maximum of the LastTransitionTime for the Admitted and PodsReady conditions.
+//
+// If the workload is not admitted, PodsReady is true, or no timeout is configured,
+// it returns an empty underlyingCause and zero duration.
+func (r *WorkloadReconciler) admittedNotReadyWorkload(wl *kueue.Workload) (string, time.Duration) {
+	// Return early if timeout is not configured or workload is not admitted
+	if r.waitForPodsReady == nil || !workload.IsAdmitted(wl) {
+		return "", 0
path: pkg/controller/core/workload_controller.go
user: login: mszadkow
body: np, done
created_at: 2025-05-20T08:55:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -536,6 +578,17 @@ const (
 	WorkloadDeactivationTarget = "DeactivationTarget"
 )
 
+// Detailed Reasons for the WorkloadEvicted condition due to PodsReadyTimeout.
+const (
+	// StartupTimeoutReason is the exceedance of the maximum time allowed for a workload
+	// to transition from admission to pods being ready.
+	StartupTimeoutReason string = "StartupTimeout"
path: apis/kueue/v1beta1/workload_types.go
user: login: tenzen-y
body: > wdyt @tenzen-y , I would probably suggest to use the same values as in the reason, to avoid the unnecessary mapping

+1, I want to avoid the additional cross mappings.

created_at: 2025-05-20T14:32:20Z
author_association: MEMBER

PR comments:
user: login: netlify[bot]
created_at: 2025-05-15T15:45:58Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *kubernetes-sigs-kueue* canceled.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 9dc4ef770f3df7199382e1956bb3b4dea9037645 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/682c4365e258de00084620ab |
author_association: NONE
user: login: mszadkow
created_at: 2025-05-16T11:26:12Z
body: /cc @mimowo 
author_association: CONTRIBUTOR
user: login: mszadkow
created_at: 2025-05-16T11:26:31Z
body: I am still fighting with flaky - pull-kueue-test-e2e-customconfigs-main

/test pull-kueue-test-e2e-customconfigs-main
author_association: CONTRIBUTOR
user: login: mszadkow
created_at: 2025-05-16T11:26:52Z
body: /cc @PBundyra 
author_association: CONTRIBUTOR
user: login: mszadkow
created_at: 2025-05-16T11:27:09Z
body: related to https://github.com/kubernetes-sigs/kueue/pull/5238
author_association: CONTRIBUTOR
user: login: mszadkow
created_at: 2025-05-16T13:17:36Z
body: > Overall looks great, one minor comment and a question whether it would be possible to add the metric to the `test/e2e/singlecluster/metrics_test.go`?

done
author_association: CONTRIBUTOR
user: login: mykysha
created_at: 2025-05-16T13:22:37Z
body: Thanks!
/lgtm
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-05-16T13:22:46Z
body: LGTM label has been added.  <details>Git tree hash: 886ec870cf72d9f0ec03debc564811aa8d2093ac</details>
author_association: CONTRIBUTOR
user: login: mszadkow
created_at: 2025-05-16T15:22:19Z
body: Will be updated based on https://github.com/kubernetes-sigs/kueue/pull/5238
author_association: CONTRIBUTOR
user: login: mimowo
created_at: 2025-05-20T08:02:29Z
body: LGTM, only small-ish comments. 

cc @tenzen-y I think this is almost ready to go.
author_association: CONTRIBUTOR
user: login: mimowo
created_at: 2025-05-20T09:21:40Z
body: /lgtm
/approve
Thanks 👍 
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-05-20T09:21:46Z
body: LGTM label has been added.  <details>Git tree hash: 9bf99cf87321d72b612e281cd707b254b2911ee3</details>
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-05-20T09:21:47Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/kueue/pull/5259#issuecomment-2893615859" title="Approved">mimowo</a>*, *<a href="https://github.com/kubernetes-sigs/kueue/pull/5259#" title="Author self-approved">mszadkow</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkueue).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/kueue/blob/main/OWNERS)~~ [mimowo]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR