Repository: kubernetes-sigs/kueue

Issue:
Adding observability for admitted pods
<!-- Please only use this template for submitting enhancement requests -->

**What would you like to be added**:

In our dashboard, we would like the ability to differentiate between different types of pending pods.
With Kueue, a pod can be pending for two main reasons:

1. It has not yet been admitted and still contains a scheduling gate.
2. It has been admitted, but no available nodes have the capacity to accommodate it.

To achieve this, we propose:

- A metric (potentially limited to pods managed by Kueue) that indicates whether a pod has a scheduling gate.
- Alternatively, a label added to the pod once it has been admitted.

This metric will typically be used when joined with other pod metrics to provide a clearer understanding of the pod’s current state in the cluster.
If adding a label to the pod is chosen as the approach, the same information can be obtained using the already existing kube_pod_labels metric.

We have implemented an in-house exporter that exposes this data for us, and the metric looks like this:

```
# Admitted pod - without scheduling gates
pod_scheduling_gates{condition="false", endpoint="http", instance="x.x.x.x:9469", job="scheduling-gated-pods-exporter", namespace="default", pod="example-pod-1", service="scheduling-gated-pods-exporter"} = 1  
pod_scheduling_gates{condition="true", endpoint="http", instance="x.x.x.x:9469", job="scheduling-gated-pods-exporter", namespace="default", pod="example-pod-1", service="scheduling-gated-pods-exporter"} = 0

# Non-admitted pod - with scheduling gates
pod_scheduling_gates{condition="false", endpoint="http", instance="x.x.x.x:9469", job="scheduling-gated-pods-exporter", namespace="default", pod="example-pod-2", service="scheduling-gated-pods-exporter"} = 0  
pod_scheduling_gates{condition="true", endpoint="http", instance="x.x.x.x:9469", job="scheduling-gated-pods-exporter", namespace="default", pod="example-pod-2", service="scheduling-gated-pods-exporter"} = 1  
```

**Why is this needed**:

Understanding why a pod is pending is crucial for observability. Since Kueue does not provide a one-size-fits-all mechanism for capacity checks during admission, we need a way to distinguish between pods that are waiting for admission and those that are unschedulable due to resource constraints. This will improve monitoring, and decision-making for workload management.

**Completion requirements**:

This enhancement requires the following artifacts:

- [ ] Design doc
- [ ] API change
- [ ] Docs update

The artifacts should be linked in subsequent comments.


Hints from issue comments:
@kannon92: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness/#observability

Can you use this metric to determine pods that are gated?@yuvalaz99: > https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness/#observability
> 
> Can you use this metric to determine pods that are gated?

This metric is a Gauge which counts the number of gated pods. 
We need to know this information about individual pods :) @mimowo: > This metric is a Gauge which counts the number of gated pods.
> We need to know this information about individual pods :)

Right, but this metric can be used to have a "coarse-grained" understanding of what is going on in the cluster, similarly to the one which is proposed in the issue. I'm not sure we need more detailed metric in Kueue. Note that Kueue by default does not track Pods, so it would need to probably be an opt-in metric in configuration.

Also, if you need to know if a specific pod is gates or not, then this information is in `.spec.schedulingGates`, for example by `kubectl get pod/<podname> -oyaml`.

> we need a way to distinguish between pods that are waiting for admission and those that are unschedulable due to resource constraints.

Right, I think in the majority of cases this could be determined by `.spec.schedulingGates`. Maybe what is needed is an extension to the troubleshooting guide for Pods: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_pods/@yuvalaz99: > > This metric is a Gauge which counts the number of gated pods.
> > We need to know this information about individual pods :)
> 
> Right, but this metric can be used to have a "coarse-grained" understanding of what is going on in the cluster, similarly to the one which is proposed in the issue. I'm not sure we need more detailed metric in Kueue. Note that Kueue by default does not track Pods, so it would need to probably be an opt-in metric in configuration.
> 
> Also, if you need to know if a specific pod is gates or not, then this information is in `.spec.schedulingGates`, for example by `kubectl get pod/<podname> -oyaml`.
> 
> > we need a way to distinguish between pods that are waiting for admission and those that are unschedulable due to resource constraints.
> 
> Right, I think in the majority of cases this could be determined by `.spec.schedulingGates`. Maybe what is needed is an extension to the troubleshooting guide for Pods: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_pods/

Thank you for the detailed answer :) 
So this observability knowledge is required to understand the day-to-day state of the system, and not for troubleshooting purposes, which can be used in a lot of places. for example:
1. Helping developers quickly understand why their workload is pending. We don’t want them to manually troubleshoot each workload using specific commands.
2. Enabling administrators to estimate fragmentation in the cluster (e.g., workloads that cannot fit on any node due to capacity constraints), which helps identify wasted GPU resources.

I agree that this information can be determined from scheduling gates, but currently, there is no metric that represents it at the per-pod level.@mimowo: > I agree that this information can be determined from scheduling gates, but currently, there is no metric that represents it at the per-pod level.

I see, generally in Kubernetes metrics with such [high cardinality](https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/#metric-cardinality-enforcement) are problematic. On some clusters exposing the metric means 100k (for such a number of pods) lines, probably around 100Mi of output just for the metric. PTAL also at the [blog post](https://kubernetes.io/blog/2021/04/13/kube-state-metrics-v-2-0/#what-is-new-in-v2-0) saying "We also removed some metrics that caused unnecessarily high cardinality in Prometheus!".

Moreover, if this metric is "counter" type then it will keep information about all past pods.

So, I think this would need to be opt-in, and guage, probably something like this:

```golang
	KueueSchedulingGatedPods = prometheus.NewGaugeVec(
		prometheus.GaugeOpts{
			Subsystem: constants.KueueName,
			Name:      "scheduling_gated_pods",
			Help:      "The current number of scheduling gated pods managed by Kueue which are scheduling gated",
		}, []string{"name", "namespace"},
	)
```

I imagine there is a small opt-in controller (probably enabled by the global Kueue config) which monitors all events for pods (add, delete, update), and based on the events it increments or decrements the Pods. 

I assume these are "Kueue"-managed Pods only. Otherwise it seems to be more generic observability tool than just for Kueue.

A potentially useful modification of the metric is:

```golang
	KueuePendingPods = prometheus.NewGaugeVec(
		prometheus.GaugeOpts{
			Subsystem: constants.KueueName,
			Name:      "pending_pods",
			Help:      "The current number of pending pods managed by Kueue which by the reason",
		}, []string{"name", "namespace", "reason"},
// reason could be 
// "scheduling gated" - has at least one scheduling gate
//  "not scheduled" - does not have scheduling gates, but does not have "spec.nodeName"
// "starting" - has "spec.nodeName"
	)
```
Let me know your thoughts @yuvalaz99.
@mimowo: Exploring the label idea:
> Alternatively, a label added to the pod once it has been admitted.

We already have this label which is added to every Pod admitted by Kueue: https://github.com/kubernetes-sigs/kueue/blob/9d2919ea7b682a9e22b185347d286fe7e82591d4/apis/kueue/v1alpha1/topology_types.go#L60-L63

This label in conjunction with the pod `status.Phase` could be probably used for fast filtering to identify Pods which are admitted by not Running.

This label was introduced for a different purpose (TopologyAwareScheduling), but seems to match the requirements so I would prefer to go with the label rather than introduce yet another. So, the MVP would be extract setting of the label out of the TopologyAwareScheduling feature gate.@mimowo: I consulted this with @mwysokin who collaborates with the user, and we believe the label is what is needed. 

In that case, I suggest we go ahead and extract the label from the TAS feature gate.

cc @mbobrovskyi @mszadkow @tenzen-y @mimowo: /assign @mszadkow 

Solution patch:
diff --git a/apis/kueue/v1alpha1/topology_types.go b/apis/kueue/v1alpha1/topology_types.go
index a24fb02d8..b45872633 100644
--- a/apis/kueue/v1alpha1/topology_types.go
+++ b/apis/kueue/v1alpha1/topology_types.go
@@ -57,11 +57,6 @@ const (
 	// annotation is set when starting the Job, and removed on stopping the Job.
 	WorkloadAnnotation = "kueue.x-k8s.io/workload"
 
-	// PodSetLabel is a label set on the Job's PodTemplate to indicate the name
-	// of the PodSet of the admitted Workload corresponding to the PodTemplate.
-	// The label is set when starting the Job, and removed on stopping the Job.
-	PodSetLabel = "kueue.x-k8s.io/podset"
-
 	// TASLabel is a label set on the Job's PodTemplate to indicate that the
 	// PodSet is admitted using TopologyAwareScheduling, and all Pods created
 	// from the Job's PodTemplate also have the label. For the Pod-based
diff --git a/keps/2724-topology-aware-scheduling/README.md b/keps/2724-topology-aware-scheduling/README.md
index 0b44eae13..67990061f 100644
--- a/keps/2724-topology-aware-scheduling/README.md
+++ b/keps/2724-topology-aware-scheduling/README.md
@@ -649,11 +649,6 @@ const (
   // annotation is set when starting the Job, and removed on stopping the Job.
   WorkloadAnnotation = "kueue.x-k8s.io/workload"
 
-  // PodSetLabel is a label set on the Job's PodTemplate to indicate the name
-  // of the PodSet of the admitted Workload corresponding to the PodTemplate.
-  // The label is set when starting the Job, and removed on stopping the Job.
-  PodSetLabel = "kueue.x-k8s.io/podset"
-
   // TASLabel is a label set on the Job's PodTemplate to indicate that the
   // PodSet is admitted using TopologyAwareScheduling, and all Pods created
   // from the Job's PodTemplate also have the label.
@@ -752,10 +747,14 @@ reconciler which lists all pods for a given TAS PodSet, and ensures that the
 pods in the expected number are un-gated to a given value.
 
 Along with the scheduling gate, to each pod template the
-`kueue.x-k8s.io/workload` and `kueue.x-k8s.io/podset` labels / annotations are
+`kueue.x-k8s.io/workload` label / annotation is
 added to facilitate quick lookup (List request) for all pods corresponding to
 the workload (and more specifically PodSetAssignment) by `TopologyAssigner`.
 
+The `kueue.x-k8s.io/podset` is the label that was extracted and moved to general usage regardless of TAS feature.
+This label serves as a link between the Job's PodTemplate and the name of the PodSet associated with the admitted Workload.
+Also it is indication of the Job in progress as it is added when it starts and it is removed when it's stopped.
+
 The TopologyUngater watches for pod events which trigger the reconciliation
 loop. The reconciliations are batched by 1s periods to make sure multiple
 created pods at the similar time don't trigger too many reconciliations.
diff --git a/keps/976-plain-pods/README.md b/keps/976-plain-pods/README.md
index 2af74af47..9b37e82a8 100644
--- a/keps/976-plain-pods/README.md
+++ b/keps/976-plain-pods/README.md
@@ -390,6 +390,10 @@ their finalizers are removed), the Workload will be automatically cleaned up.
 If individual Pods in the group fail and a replacement Pod comes in, the replacement Pod will be
 added as an owner of the Workload as well.
 
+When the user's workload (such as Pod, Job, etc.) is started, its PodTemplate gets a label `kueue.x-k8s.io/podset`.
+This label helps identify which PodSet is associated with the Workload being executed by the Job.
+The label is automatically removed when the Job stops.
+
 #### Single Pods
 
 The simplest case we want to support is single Pod jobs. These Pods only have the label
diff --git a/pkg/controller/constants/constants.go b/pkg/controller/constants/constants.go
index 3e88ec196..a5246afd3 100644
--- a/pkg/controller/constants/constants.go
+++ b/pkg/controller/constants/constants.go
@@ -48,4 +48,9 @@ const (
 
 	// MaxExecTimeSecondsLabel is the label key in the job that holds the maximum execution time.
 	MaxExecTimeSecondsLabel = `kueue.x-k8s.io/max-exec-time-seconds`
+
+	// PodSetLabel is a label set on the Job's PodTemplate to indicate the name
+	// of the PodSet of the admitted Workload corresponding to the PodTemplate.
+	// The label is set when starting the Job, and removed on stopping the Job.
+	PodSetLabel = "kueue.x-k8s.io/podset"
 )
diff --git a/pkg/controller/jobframework/reconciler.go b/pkg/controller/jobframework/reconciler.go
index 44835a4a7..4d05ac04c 100644
--- a/pkg/controller/jobframework/reconciler.go
+++ b/pkg/controller/jobframework/reconciler.go
@@ -575,6 +575,15 @@ func (r *JobReconciler) ReconcileGenericJob(ctx context.Context, req ctrl.Reques
 			}
 			return ctrl.Result{}, err
 		}
+		// update workload priority if job's label changed
+		if wl.Spec.PriorityClassName != "" && WorkloadPriorityClassName(object) != wl.Spec.PriorityClassName {
+			log.V(2).Info("Job changed priority, updating workload", "oldPriority", wl.Spec.PriorityClassName, "newPriority", WorkloadPriorityClassName(object))
+			if _, err = r.updateWorkloadToMatchJob(ctx, job, object, wl); err != nil {
+				log.Error(err, "Updating workload priority")
+				return ctrl.Result{}, err
+			}
+			return ctrl.Result{}, nil
+		}
 		log.V(3).Info("Job is suspended and workload not yet admitted by a clusterQueue, nothing to do")
 		return ctrl.Result{}, nil
 	}
@@ -1188,9 +1197,11 @@ func getPodSetsInfoFromStatus(ctx context.Context, c client.Client, w *kueue.Wor
 			return nil, err
 		}
 		if features.Enabled(features.TopologyAwareScheduling) {
-			info.Labels[kueuealpha.PodSetLabel] = string(psAssignment.Name)
 			info.Annotations[kueuealpha.WorkloadAnnotation] = w.Name
 		}
+
+		info.Labels[controllerconsts.PodSetLabel] = string(psAssignment.Name)
+
 		for _, admissionCheck := range w.Status.AdmissionChecks {
 			for _, podSetUpdate := range admissionCheck.PodSetUpdates {
 				if podSetUpdate.Name == info.Name {
diff --git a/pkg/controller/jobframework/validation.go b/pkg/controller/jobframework/validation.go
index e6664bef4..a25e7d38f 100644
--- a/pkg/controller/jobframework/validation.go
+++ b/pkg/controller/jobframework/validation.go
@@ -73,8 +73,10 @@ func ValidateJobOnCreate(job GenericJob) field.ErrorList {
 func ValidateJobOnUpdate(oldJob, newJob GenericJob) field.ErrorList {
 	allErrs := validateUpdateForQueueName(oldJob, newJob)
 	allErrs = append(allErrs, validateUpdateForPrebuiltWorkload(oldJob, newJob)...)
-	allErrs = append(allErrs, ValidateUpdateForWorkloadPriorityClassName(oldJob.Object(), newJob.Object())...)
 	allErrs = append(allErrs, validateUpdateForMaxExecTime(oldJob, newJob)...)
+	if !newJob.IsSuspended() {
+		allErrs = append(allErrs, ValidateUpdateForWorkloadPriorityClassName(oldJob.Object(), newJob.Object())...)
+	}
 	return allErrs
 }
 
diff --git a/pkg/controller/jobs/deployment/deployment_webhook.go b/pkg/controller/jobs/deployment/deployment_webhook.go
index caa667260..c7dbd1394 100644
--- a/pkg/controller/jobs/deployment/deployment_webhook.go
+++ b/pkg/controller/jobs/deployment/deployment_webhook.go
@@ -126,14 +126,16 @@ func (wh *Webhook) ValidateUpdate(ctx context.Context, oldObj, newObj runtime.Ob
 	newQueueName := jobframework.QueueNameForObject(newDeployment.Object())
 
 	allErrs := jobframework.ValidateQueueName(newDeployment.Object())
-	allErrs = append(allErrs, jobframework.ValidateUpdateForWorkloadPriorityClassName(oldDeployment.Object(), newDeployment.Object())...)
 
 	// Prevents updating the queue-name if at least one Pod is not suspended
 	// or if the queue-name has been deleted.
-	if oldDeployment.Status.ReadyReplicas > 0 || newQueueName == "" {
+	isSuspended := oldDeployment.Status.ReadyReplicas == 0
+	if !isSuspended || newQueueName == "" {
 		allErrs = append(allErrs, apivalidation.ValidateImmutableField(oldQueueName, newQueueName, queueNameLabelPath)...)
 	}
-
+	if !isSuspended {
+		allErrs = append(allErrs, jobframework.ValidateUpdateForWorkloadPriorityClassName(oldDeployment.Object(), newDeployment.Object())...)
+	}
 	return warnings, allErrs.ToAggregate()
 }
 
diff --git a/pkg/controller/jobs/leaderworkerset/leaderworkerset_webhook.go b/pkg/controller/jobs/leaderworkerset/leaderworkerset_webhook.go
index bfcb0ba2d..85959c1c8 100644
--- a/pkg/controller/jobs/leaderworkerset/leaderworkerset_webhook.go
+++ b/pkg/controller/jobs/leaderworkerset/leaderworkerset_webhook.go
@@ -106,7 +106,6 @@ var _ webhook.CustomValidator = &Webhook{}
 var (
 	labelsPath               = field.NewPath("metadata", "labels")
 	queueNameLabelPath       = labelsPath.Key(constants.QueueLabel)
-	priorityClassNamePath    = labelsPath.Key(constants.WorkloadPriorityClassLabel)
 	specPath                 = field.NewPath("spec")
 	leaderWorkerTemplatePath = specPath.Child("leaderWorkerTemplate")
 	leaderTemplatePath       = leaderWorkerTemplatePath.Child("leaderTemplate")
@@ -140,10 +139,14 @@ func (wh *Webhook) ValidateUpdate(ctx context.Context, oldObj, newObj runtime.Ob
 		jobframework.QueueNameForObject(oldLeaderWorkerSet.Object()),
 		queueNameLabelPath,
 	)...)
-	allErrs = append(allErrs, jobframework.ValidateUpdateForWorkloadPriorityClassName(
-		newLeaderWorkerSet.Object(),
-		oldLeaderWorkerSet.Object(),
-	)...)
+
+	isSuspended := oldLeaderWorkerSet.Status.ReadyReplicas == 0
+	if !isSuspended {
+		allErrs = append(allErrs, jobframework.ValidateUpdateForWorkloadPriorityClassName(
+			newLeaderWorkerSet.Object(),
+			oldLeaderWorkerSet.Object(),
+		)...)
+	}
 
 	suspend, err := jobframework.WorkloadShouldBeSuspended(ctx, newLeaderWorkerSet.Object(), wh.client, wh.manageJobsWithoutQueueName, wh.managedJobsNamespaceSelector)
 	if err != nil {
diff --git a/pkg/controller/jobs/statefulset/statefulset_webhook.go b/pkg/controller/jobs/statefulset/statefulset_webhook.go
index aca9901d6..9b052a177 100644
--- a/pkg/controller/jobs/statefulset/statefulset_webhook.go
+++ b/pkg/controller/jobs/statefulset/statefulset_webhook.go
@@ -140,13 +140,16 @@ func (wh *Webhook) ValidateUpdate(ctx context.Context, oldObj, newObj runtime.Ob
 
 	// Prevents updating the queue-name if at least one Pod is not suspended
 	// or if the queue-name has been deleted.
-	if oldStatefulSet.Status.ReadyReplicas > 0 || newQueueName == "" {
+	isSuspended := oldStatefulSet.Status.ReadyReplicas == 0
+	if !isSuspended || newQueueName == "" {
 		allErrs = append(allErrs, apivalidation.ValidateImmutableField(oldQueueName, newQueueName, queueNameLabelPath)...)
 	}
-	allErrs = append(allErrs, jobframework.ValidateUpdateForWorkloadPriorityClassName(
-		oldStatefulSet.Object(),
-		newStatefulSet.Object(),
-	)...)
+	if !isSuspended {
+		allErrs = append(allErrs, jobframework.ValidateUpdateForWorkloadPriorityClassName(
+			oldStatefulSet.Object(),
+			newStatefulSet.Object(),
+		)...)
+	}
 
 	suspend, err := jobframework.WorkloadShouldBeSuspended(ctx, newStatefulSet.Object(), wh.client, wh.manageJobsWithoutQueueName, wh.managedJobsNamespaceSelector)
 	if err != nil {
diff --git a/pkg/controller/tas/topology_ungater.go b/pkg/controller/tas/topology_ungater.go
index 6b91e2fdc..5b2504115 100644
--- a/pkg/controller/tas/topology_ungater.go
+++ b/pkg/controller/tas/topology_ungater.go
@@ -42,6 +42,7 @@ import (
 	kueuealpha "sigs.k8s.io/kueue/apis/kueue/v1alpha1"
 	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta1"
 	"sigs.k8s.io/kueue/pkg/constants"
+	controllerconsts "sigs.k8s.io/kueue/pkg/controller/constants"
 	"sigs.k8s.io/kueue/pkg/controller/core"
 	"sigs.k8s.io/kueue/pkg/controller/tas/indexer"
 	utilclient "sigs.k8s.io/kueue/pkg/util/client"
@@ -247,7 +248,7 @@ func (r *topologyUngater) Generic(event.TypedGenericEvent[*kueue.Workload]) bool
 func (r *topologyUngater) podsForPodSet(ctx context.Context, ns, wlName string, psName kueue.PodSetReference) ([]*corev1.Pod, error) {
 	var pods corev1.PodList
 	if err := r.client.List(ctx, &pods, client.InNamespace(ns), client.MatchingLabels{
-		kueuealpha.PodSetLabel: string(psName),
+		controllerconsts.PodSetLabel: string(psName),
 	}, client.MatchingFields{
 		indexer.WorkloadNameKey: wlName,
 	}); err != nil {



Test patch:
diff --git a/pkg/controller/jobs/deployment/deployment_webhook_test.go b/pkg/controller/jobs/deployment/deployment_webhook_test.go
index b56259f92..7c233c20a 100644
--- a/pkg/controller/jobs/deployment/deployment_webhook_test.go
+++ b/pkg/controller/jobs/deployment/deployment_webhook_test.go
@@ -291,7 +291,7 @@ func TestValidateUpdate(t *testing.T) {
 				},
 			}.ToAggregate(),
 		},
-		"update priority-class": {
+		"update priority-class when suspended": {
 			oldDeployment: testingdeployment.MakeDeployment("test-pod", "").
 				Queue("test-queue").
 				Label(constants.WorkloadPriorityClassLabel, "test").
@@ -300,6 +300,18 @@ func TestValidateUpdate(t *testing.T) {
 				Queue("test-queue").
 				Label(constants.WorkloadPriorityClassLabel, "new-test").
 				Obj(),
+		},
+		"update priority-class when replicas ready": {
+			oldDeployment: testingdeployment.MakeDeployment("test-pod", "").
+				Queue("test-queue").
+				Label(constants.WorkloadPriorityClassLabel, "test").
+				ReadyReplicas(int32(1)).
+				Obj(),
+			newDeployment: testingdeployment.MakeDeployment("test-pod", "").
+				Queue("test-queue").
+				Label(constants.WorkloadPriorityClassLabel, "new-test").
+				ReadyReplicas(int32(1)).
+				Obj(),
 			wantErr: field.ErrorList{
 				&field.Error{
 					Type:  field.ErrorTypeInvalid,
diff --git a/pkg/controller/jobs/job/job_controller_test.go b/pkg/controller/jobs/job/job_controller_test.go
index 7204cd3d4..f5180589f 100644
--- a/pkg/controller/jobs/job/job_controller_test.go
+++ b/pkg/controller/jobs/job/job_controller_test.go
@@ -496,6 +496,9 @@ func TestReconciler(t *testing.T) {
 	baseWPCWrapper := utiltesting.MakeWorkloadPriorityClass("test-wpc").
 		PriorityValue(100)
 
+	highWPCWrapper := utiltesting.MakeWorkloadPriorityClass("test-wpc-high").
+		PriorityValue(200)
+
 	basePCWrapper := utiltesting.MakePriorityClass("test-pc").
 		PriorityValue(200)
 
@@ -843,7 +846,7 @@ func TestReconciler(t *testing.T) {
 			job: *baseJobWrapper.DeepCopy(),
 			wantJob: *baseJobWrapper.Clone().
 				Suspend(false).
-				PodLabel(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+				PodLabel(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 				PodAnnotation(kueuealpha.WorkloadAnnotation, "wl").
 				Obj(),
 			workloads: []kueue.Workload{
@@ -938,6 +941,7 @@ func TestReconciler(t *testing.T) {
 			wantJob: *baseJobWrapper.Clone().
 				Suspend(false).
 				PodLabel("ac-key", "ac-value").
+				PodLabel(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 				Obj(),
 			workloads: []kueue.Workload{
 				*baseWorkloadWrapper.Clone().
@@ -2242,6 +2246,7 @@ func TestReconciler(t *testing.T) {
 				PodAnnotation("annotation-key1", "common-value").
 				PodAnnotation("annotation-key2", "only-in-check1").
 				PodLabel("label-key1", "common-value").
+				PodLabel(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 				NodeSelector("node-selector-key1", "common-value").
 				NodeSelector("node-selector-key2", "only-in-check2").
 				Obj(),
@@ -2348,6 +2353,7 @@ func TestReconciler(t *testing.T) {
 			job: *baseJobWrapper.DeepCopy(),
 			wantJob: *baseJobWrapper.Clone().
 				Suspend(false).
+				PodLabel(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 				Obj(),
 			workloads: []kueue.Workload{
 				*baseWorkloadWrapper.Clone().
@@ -2434,6 +2440,7 @@ func TestReconciler(t *testing.T) {
 				SetAnnotation(JobMinParallelismAnnotation, "5").
 				Suspend(false).
 				Parallelism(8).
+				PodLabel(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 				Obj(),
 			workloads: []kueue.Workload{
 				*utiltesting.MakeWorkload("a", "ns").
@@ -2583,19 +2590,25 @@ func TestReconciler(t *testing.T) {
 			job: *baseJobWrapper.
 				Clone().
 				Suspend(true).
+				WorkloadPriorityClass(highWPCWrapper.Name).
 				UID("test-uid").
 				Obj(),
 			wantJob: *baseJobWrapper.
 				Clone().
+				WorkloadPriorityClass(highWPCWrapper.Name).
 				UID("test-uid").
 				Obj(),
+			priorityClasses: []client.Object{
+				baseWPCWrapper.Obj(), highWPCWrapper.Obj(),
+			},
 			workloads: []kueue.Workload{
 				*utiltesting.MakeWorkload("job", "ns").
 					Finalizers(kueue.ResourceInUseFinalizerName).
 					PodSets(*utiltesting.MakePodSet(kueue.DefaultPodSetName, 10).Request(corev1.ResourceCPU, "1").Obj()).
 					Queue("foo").
-					Priority(0).
-					PriorityClass("new-priority-class").
+					Priority(baseWPCWrapper.Value).
+					PriorityClassSource(constants.WorkloadPriorityClassSource).
+					PriorityClass(baseWPCWrapper.Name).
 					Labels(map[string]string{
 						controllerconsts.JobUIDLabel: "test-uid",
 					}).
@@ -2606,13 +2619,22 @@ func TestReconciler(t *testing.T) {
 					Finalizers(kueue.ResourceInUseFinalizerName).
 					PodSets(*utiltesting.MakePodSet(kueue.DefaultPodSetName, 10).Request(corev1.ResourceCPU, "1").Obj()).
 					Queue("foo").
-					Priority(0).
-					PriorityClass("new-priority-class").
+					Priority(highWPCWrapper.Value).
+					PriorityClassSource(constants.WorkloadPriorityClassSource).
+					PriorityClass(highWPCWrapper.Name).
 					Labels(map[string]string{
 						controllerconsts.JobUIDLabel: "test-uid",
 					}).
 					Obj(),
 			},
+			wantEvents: []utiltesting.EventRecord{
+				{
+					Key:       types.NamespacedName{Name: "job", Namespace: "ns"},
+					EventType: "Normal",
+					Reason:    "UpdatedWorkload",
+					Message:   "Updated not matching Workload for suspended job: ns/job",
+				},
+			},
 		},
 		"the workload without uid label is created when job's uid is longer than 63 characters": {
 			job: *baseJobWrapper.
diff --git a/pkg/controller/jobs/job/job_webhook_test.go b/pkg/controller/jobs/job/job_webhook_test.go
index b0219ef86..c00cb1b0a 100644
--- a/pkg/controller/jobs/job/job_webhook_test.go
+++ b/pkg/controller/jobs/job/job_webhook_test.go
@@ -407,9 +407,14 @@ func TestValidateUpdate(t *testing.T) {
 			wantErr: nil,
 		},
 		{
-			name:   "workloadPriorityClassName is immutable",
+			name:   "workloadPriorityClassName is mutable when job is suspended",
 			oldJob: testingutil.MakeJob("job", "default").WorkloadPriorityClass("test-1").Obj(),
 			newJob: testingutil.MakeJob("job", "default").WorkloadPriorityClass("test-2").Obj(),
+		},
+		{
+			name:   "workloadPriorityClassName is immutable when job is running",
+			oldJob: testingutil.MakeJob("job", "default").WorkloadPriorityClass("test-1").Suspend(false).Obj(),
+			newJob: testingutil.MakeJob("job", "default").WorkloadPriorityClass("test-2").Suspend(false).Obj(),
 			wantErr: field.ErrorList{
 				field.Invalid(workloadPriorityClassNamePath, "test-2", apivalidation.FieldImmutableErrorMsg),
 			},
diff --git a/pkg/controller/jobs/leaderworkerset/leaderworkerset_webhook_test.go b/pkg/controller/jobs/leaderworkerset/leaderworkerset_webhook_test.go
index 42bc3ebbc..2b81c4d78 100644
--- a/pkg/controller/jobs/leaderworkerset/leaderworkerset_webhook_test.go
+++ b/pkg/controller/jobs/leaderworkerset/leaderworkerset_webhook_test.go
@@ -285,7 +285,7 @@ func TestValidateUpdate(t *testing.T) {
 				},
 			}.ToAggregate(),
 		},
-		"change priority class": {
+		"change priority class when suspended": {
 			oldObj: testingleaderworkerset.MakeLeaderWorkerSet("test-lws", "").
 				LeaderTemplate(corev1.PodTemplateSpec{}).
 				Queue("test-queue").
@@ -296,10 +296,24 @@ func TestValidateUpdate(t *testing.T) {
 				Queue("test-queue").
 				Label(constants.WorkloadPriorityClassLabel, "new-test").
 				Obj(),
+		},
+		"change priority class when replicas ready": {
+			oldObj: testingleaderworkerset.MakeLeaderWorkerSet("test-lws", "").
+				LeaderTemplate(corev1.PodTemplateSpec{}).
+				Queue("test-queue").
+				Label(constants.WorkloadPriorityClassLabel, "test").
+				ReadyReplicas(int32(1)).
+				Obj(),
+			newObj: testingleaderworkerset.MakeLeaderWorkerSet("test-lws", "").
+				LeaderTemplate(corev1.PodTemplateSpec{}).
+				Queue("test-queue").
+				Label(constants.WorkloadPriorityClassLabel, "new-test").
+				ReadyReplicas(int32(1)).
+				Obj(),
 			wantErr: field.ErrorList{
 				&field.Error{
 					Type:  field.ErrorTypeInvalid,
-					Field: priorityClassNamePath.String(),
+					Field: "metadata.labels[kueue.x-k8s.io/priority-class]",
 				},
 			}.ToAggregate(),
 		},
diff --git a/pkg/controller/jobs/pod/pod_controller_test.go b/pkg/controller/jobs/pod/pod_controller_test.go
index 0279fb1b5..d74804b1f 100644
--- a/pkg/controller/jobs/pod/pod_controller_test.go
+++ b/pkg/controller/jobs/pod/pod_controller_test.go
@@ -294,6 +294,7 @@ func TestReconciler(t *testing.T) {
 				ManagedByKueueLabel().
 				NodeSelector(corev1.LabelArchStable, "arm64").
 				KueueFinalizer().
+				Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 				Obj()},
 			workloads: []kueue.Workload{
 				*utiltesting.MakeWorkload("unit-test", "ns").Finalizers(kueue.ResourceInUseFinalizerName).
@@ -1044,6 +1045,7 @@ func TestReconciler(t *testing.T) {
 					Group("test-group").
 					GroupTotalCount("2").
 					NodeSelector(corev1.LabelArchStable, "arm64").
+					Label(controllerconsts.PodSetLabel, podUID).
 					Obj(),
 				*basePodWrapper.
 					Clone().
@@ -1053,6 +1055,7 @@ func TestReconciler(t *testing.T) {
 					Group("test-group").
 					GroupTotalCount("2").
 					NodeSelector(corev1.LabelArchStable, "arm64").
+					Label(controllerconsts.PodSetLabel, podUID).
 					Obj(),
 			},
 			workloads: []kueue.Workload{
@@ -1598,6 +1601,7 @@ func TestReconciler(t *testing.T) {
 					KueueFinalizer().
 					Group("test-group").
 					GroupTotalCount("1").
+					Label(controllerconsts.PodSetLabel, podUID).
 					Obj(),
 			},
 			workloads: []kueue.Workload{
@@ -1658,6 +1662,7 @@ func TestReconciler(t *testing.T) {
 					KueueFinalizer().
 					Group("test-group").
 					GroupTotalCount("3").
+					Label(controllerconsts.PodSetLabel, podUID).
 					StatusPhase(corev1.PodRunning).
 					Obj(),
 				*basePodWrapper.
@@ -1711,6 +1716,7 @@ func TestReconciler(t *testing.T) {
 					KueueFinalizer().
 					Group("test-group").
 					GroupTotalCount("3").
+					Label(controllerconsts.PodSetLabel, podUID).
 					StatusPhase(corev1.PodRunning).
 					Obj(),
 				*basePodWrapper.
@@ -1737,6 +1743,7 @@ func TestReconciler(t *testing.T) {
 					KueueFinalizer().
 					Group("test-group").
 					GroupTotalCount("3").
+					Label(controllerconsts.PodSetLabel, podUID).
 					Obj(),
 			},
 			wantWorkloads: []kueue.Workload{
@@ -4430,6 +4437,7 @@ func TestReconciler(t *testing.T) {
 					StatusPhase(corev1.PodRunning).
 					Group("test-group").
 					GroupTotalCount("2").
+					Label(controllerconsts.PodSetLabel, podUID).
 					CreationTimestamp(now.Add(-time.Hour)).
 					Obj(),
 				*basePodWrapper.
@@ -4479,6 +4487,7 @@ func TestReconciler(t *testing.T) {
 					StatusPhase(corev1.PodRunning).
 					Group("test-group").
 					GroupTotalCount("2").
+					Label(controllerconsts.PodSetLabel, podUID).
 					CreationTimestamp(now.Add(-time.Hour)).
 					Obj(),
 				*basePodWrapper.
@@ -4488,6 +4497,7 @@ func TestReconciler(t *testing.T) {
 					KueueFinalizer().
 					Group("test-group").
 					GroupTotalCount("2").
+					Label(controllerconsts.PodSetLabel, podUID).
 					CreationTimestamp(now).
 					Obj(),
 			},
diff --git a/pkg/controller/jobs/raycluster/raycluster_controller_test.go b/pkg/controller/jobs/raycluster/raycluster_controller_test.go
index 5165d955d..4d2c5fd90 100644
--- a/pkg/controller/jobs/raycluster/raycluster_controller_test.go
+++ b/pkg/controller/jobs/raycluster/raycluster_controller_test.go
@@ -35,6 +35,7 @@ import (
 
 	kueuealpha "sigs.k8s.io/kueue/apis/kueue/v1alpha1"
 	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta1"
+	controllerconsts "sigs.k8s.io/kueue/pkg/controller/constants"
 	"sigs.k8s.io/kueue/pkg/controller/jobframework"
 	"sigs.k8s.io/kueue/pkg/features"
 	"sigs.k8s.io/kueue/pkg/podset"
@@ -325,6 +326,8 @@ func TestReconciler(t *testing.T) {
 			wantJob: *baseJobWrapper.Clone().
 				Suspend(false).
 				NodeSelectorHeadGroup(corev1.LabelArchStable, "arm64").
+				NodeLabel(rayv1.HeadNode, controllerconsts.PodSetLabel, "head").
+				NodeLabel(rayv1.WorkerNode, controllerconsts.PodSetLabel, "workers-group-0").
 				Obj(),
 			workloads: []kueue.Workload{
 				*utiltesting.MakeWorkload("test", "ns").
@@ -387,6 +390,7 @@ func TestReconciler(t *testing.T) {
 										Obj(),
 								},
 							}).
+							Labels(map[string]string{controllerconsts.PodSetLabel: "head"}).
 							Obj(),
 						*utiltesting.MakePodSet("workers-group-0", 1).
 							PodSpec(corev1.PodSpec{
@@ -397,6 +401,7 @@ func TestReconciler(t *testing.T) {
 										Obj(),
 								},
 							}).
+							Labels(map[string]string{controllerconsts.PodSetLabel: "workers-group-0"}).
 							Obj(),
 					).
 					ReserveQuota(
@@ -541,6 +546,8 @@ func TestReconciler(t *testing.T) {
 				Suspend(false).
 				NodeSelectorHeadGroup(corev1.LabelArchStable, "arm64").
 				WithNumOfHosts("workers-group-0", 2).
+				NodeLabel(rayv1.HeadNode, controllerconsts.PodSetLabel, "head").
+				NodeLabel(rayv1.WorkerNode, controllerconsts.PodSetLabel, "workers-group-0").
 				Obj(),
 			workloads: []kueue.Workload{
 				*utiltesting.MakeWorkload("test", "ns").
diff --git a/pkg/controller/jobs/raycluster/raycluster_webhook_test.go b/pkg/controller/jobs/raycluster/raycluster_webhook_test.go
index c0e7153ac..4edac8f1d 100644
--- a/pkg/controller/jobs/raycluster/raycluster_webhook_test.go
+++ b/pkg/controller/jobs/raycluster/raycluster_webhook_test.go
@@ -38,11 +38,6 @@ import (
 	testingrayutil "sigs.k8s.io/kueue/pkg/util/testingjobs/raycluster"
 )
 
-var (
-	labelsPath                    = field.NewPath("metadata", "labels")
-	workloadPriorityClassNamePath = labelsPath.Key(constants.WorkloadPriorityClassLabel)
-)
-
 func TestValidateDefault(t *testing.T) {
 	testcases := map[string]struct {
 		oldJob               *rayv1.RayCluster
@@ -303,7 +298,7 @@ func TestValidateUpdate(t *testing.T) {
 				Obj(),
 			wantErr: nil,
 		},
-		"priorityClassName is immutable": {
+		"priorityClassName is mutable": {
 			oldJob: testingrayutil.MakeCluster("job", "ns").
 				Queue("queue").
 				WorkloadPriorityClass("test-1").
@@ -312,9 +307,6 @@ func TestValidateUpdate(t *testing.T) {
 				Queue("queue").
 				WorkloadPriorityClass("test-2").
 				Obj(),
-			wantErr: field.ErrorList{
-				field.Invalid(workloadPriorityClassNamePath, "test-2", apivalidation.FieldImmutableErrorMsg),
-			}.ToAggregate(),
 		},
 	}
 
diff --git a/pkg/controller/jobs/rayjob/rayjob_webhook_test.go b/pkg/controller/jobs/rayjob/rayjob_webhook_test.go
index 01ca3fd02..42ca93151 100644
--- a/pkg/controller/jobs/rayjob/rayjob_webhook_test.go
+++ b/pkg/controller/jobs/rayjob/rayjob_webhook_test.go
@@ -38,11 +38,6 @@ import (
 	testingrayutil "sigs.k8s.io/kueue/pkg/util/testingjobs/rayjob"
 )
 
-var (
-	labelsPath                    = field.NewPath("metadata", "labels")
-	workloadPriorityClassNamePath = labelsPath.Key(constants.WorkloadPriorityClassLabel)
-)
-
 func TestDefault(t *testing.T) {
 	testcases := map[string]struct {
 		oldJob               *rayv1.RayJob
@@ -379,9 +374,6 @@ func TestValidateUpdate(t *testing.T) {
 				Queue("queue").
 				WorkloadPriorityClass("test-2").
 				Obj(),
-			wantErr: field.ErrorList{
-				field.Invalid(workloadPriorityClassNamePath, "test-2", apivalidation.FieldImmutableErrorMsg),
-			}.ToAggregate(),
 		},
 	}
 
diff --git a/pkg/controller/jobs/statefulset/statefulset_webhook_test.go b/pkg/controller/jobs/statefulset/statefulset_webhook_test.go
index cedef8e8e..617ffc061 100644
--- a/pkg/controller/jobs/statefulset/statefulset_webhook_test.go
+++ b/pkg/controller/jobs/statefulset/statefulset_webhook_test.go
@@ -337,7 +337,7 @@ func TestValidateUpdate(t *testing.T) {
 				},
 			}.ToAggregate(),
 		},
-		"change in priority class label": {
+		"change in priority class label when suspended": {
 			oldObj: &appsv1.StatefulSet{
 				ObjectMeta: metav1.ObjectMeta{
 					Labels: map[string]string{
@@ -354,6 +354,30 @@ func TestValidateUpdate(t *testing.T) {
 					},
 				},
 			},
+		},
+		"change in priority class label when replicas ready": {
+			oldObj: &appsv1.StatefulSet{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						constants.QueueLabel:                 "queue1",
+						constants.WorkloadPriorityClassLabel: "priority1",
+					},
+				},
+				Status: appsv1.StatefulSetStatus{
+					ReadyReplicas: int32(1),
+				},
+			},
+			newObj: &appsv1.StatefulSet{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						constants.QueueLabel:                 "queue1",
+						constants.WorkloadPriorityClassLabel: "priority2",
+					},
+				},
+				Status: appsv1.StatefulSetStatus{
+					ReadyReplicas: int32(1),
+				},
+			},
 			wantErr: field.ErrorList{
 				&field.Error{
 					Type:  field.ErrorTypeInvalid,
diff --git a/pkg/controller/tas/topology_ungater_test.go b/pkg/controller/tas/topology_ungater_test.go
index 3fa1de8c3..16385692e 100644
--- a/pkg/controller/tas/topology_ungater_test.go
+++ b/pkg/controller/tas/topology_ungater_test.go
@@ -37,6 +37,7 @@ import (
 
 	kueuealpha "sigs.k8s.io/kueue/apis/kueue/v1alpha1"
 	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta1"
+	controllerconsts "sigs.k8s.io/kueue/pkg/controller/constants"
 	"sigs.k8s.io/kueue/pkg/controller/tas/indexer"
 	utilpod "sigs.k8s.io/kueue/pkg/util/pod"
 	utiltesting "sigs.k8s.io/kueue/pkg/util/testing"
@@ -143,14 +144,14 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					Obj(),
 			},
 			wantCounts: []counts{
@@ -191,23 +192,23 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("pod2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					Obj(),
 				*testingpod.MakePod("pod2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					Obj(),
 			},
 			wantCounts: []counts{
@@ -255,23 +256,23 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("pod2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					Obj(),
 				*testingpod.MakePod("pod2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					Obj(),
 			},
 			wantCounts: []counts{
@@ -300,14 +301,14 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -328,14 +329,14 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -368,14 +369,14 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -407,13 +408,13 @@ func TestReconcile(t *testing.T) {
 			},
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod", "ns").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod", "ns").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -484,7 +485,7 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Gate("example.com/gate").
@@ -493,7 +494,7 @@ func TestReconcile(t *testing.T) {
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Gate("example.com/gate").
@@ -537,7 +538,7 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Gate("example.com/gate").
 					Obj(),
@@ -545,7 +546,7 @@ func TestReconcile(t *testing.T) {
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Gate("example.com/gate").
@@ -589,28 +590,28 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod-already-running", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					StatusPhase(corev1.PodRunning).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("pod-gated", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod-already-running", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					StatusPhase(corev1.PodRunning).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("pod-gated", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -652,28 +653,28 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod-already-running", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					StatusPhase(corev1.PodFailed).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("pod-gated", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod-already-running", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					StatusPhase(corev1.PodFailed).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("pod-gated", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -716,26 +717,26 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").UID("x").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("pod2", "ns").UID("y").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").UID("x").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("pod2", "ns").UID("y").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -778,28 +779,28 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod-already-succeeded", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					StatusPhase(corev1.PodSucceeded).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("pod-gated", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod-already-succeeded", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					StatusPhase(corev1.PodSucceeded).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("pod-gated", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					Obj(),
 			},
 			wantCounts: []counts{
@@ -841,26 +842,26 @@ func TestReconcile(t *testing.T) {
 			pods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").UID("x").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("pod2", "ns").UID("y").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
 			wantPods: []corev1.Pod{
 				*testingpod.MakePod("pod1", "ns").UID("x").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("pod2", "ns").UID("y").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -921,33 +922,33 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("p0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("p1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p3", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "3").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b2").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("p4", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "4").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -956,35 +957,35 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("p0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("p1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("p2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r2").
 					Obj(),
 				*testingpod.MakePod("p3", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "3").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b2").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("p4", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "4").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b2").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1059,31 +1060,31 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("p0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p3", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "3").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p4", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "4").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -1092,35 +1093,35 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("p0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("p1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("p2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r2").
 					Obj(),
 				*testingpod.MakePod("p3", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "3").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b2").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("p4", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "4").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b2").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1196,25 +1197,25 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("p0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p3", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -1223,22 +1224,22 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("p0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					Obj(),
 				*testingpod.MakePod("p1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					Obj(),
 				*testingpod.MakePod("p2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					Obj(),
 				*testingpod.MakePod("p3", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					Obj(),
 			},
 			wantCounts: []counts{
@@ -1296,7 +1297,7 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("p0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					StatusPhase(corev1.PodFailed).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
@@ -1304,7 +1305,7 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("p1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -1313,7 +1314,7 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("p0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					StatusPhase(corev1.PodFailed).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
@@ -1321,7 +1322,7 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("p1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1386,7 +1387,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
 					Label(jobset.JobIndexKey, "0").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p1", "ns").
@@ -1394,7 +1395,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
 					Label(jobset.JobIndexKey, "0").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p2", "ns").
@@ -1402,7 +1403,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
 					Label(jobset.JobIndexKey, "1").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p3", "ns").
@@ -1410,7 +1411,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
 					Label(jobset.JobIndexKey, "1").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -1421,7 +1422,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
 					Label(jobset.JobIndexKey, "0").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1430,7 +1431,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
 					Label(jobset.JobIndexKey, "0").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1439,7 +1440,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
 					Label(jobset.JobIndexKey, "1").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r2").
 					Obj(),
@@ -1448,7 +1449,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
 					Label(jobset.JobIndexKey, "1").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b2").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1527,7 +1528,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
 					Label(jobset.JobIndexKey, "0").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p1", "ns").
@@ -1535,7 +1536,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
 					Label(jobset.JobIndexKey, "0").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1544,7 +1545,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
 					Label(jobset.JobIndexKey, "1").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r2").
 					Obj(),
@@ -1553,7 +1554,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
 					Label(jobset.JobIndexKey, "1").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -1564,7 +1565,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
 					Label(jobset.JobIndexKey, "0").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1573,7 +1574,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
 					Label(jobset.JobIndexKey, "0").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1582,7 +1583,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "0").
 					Label(jobset.JobIndexKey, "1").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r2").
 					Obj(),
@@ -1591,7 +1592,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
 					Label(jobset.JobIndexKey, "1").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b2").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1670,7 +1671,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
 					Label(jobset.JobIndexKey, "0").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("p3", "ns").
@@ -1678,7 +1679,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
 					Label(jobset.JobIndexKey, "1").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -1689,7 +1690,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
 					Label(jobset.JobIndexKey, "0").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1698,7 +1699,7 @@ func TestReconcile(t *testing.T) {
 					Label(batchv1.JobCompletionIndexAnnotation, "1").
 					Label(jobset.JobIndexKey, "1").
 					Label(jobset.ReplicatedJobReplicas, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b2").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1766,28 +1767,28 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("l0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "launcher").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("w0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "worker").
 					Label(kftraining.ReplicaIndexLabel, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("w1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "worker").
 					Label(kftraining.ReplicaIndexLabel, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("w2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "worker").
 					Label(kftraining.ReplicaIndexLabel, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -1796,7 +1797,7 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("l0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "launcher").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1804,7 +1805,7 @@ func TestReconcile(t *testing.T) {
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "worker").
 					Label(kftraining.ReplicaIndexLabel, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1812,7 +1813,7 @@ func TestReconcile(t *testing.T) {
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "worker").
 					Label(kftraining.ReplicaIndexLabel, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r2").
 					Obj(),
@@ -1820,7 +1821,7 @@ func TestReconcile(t *testing.T) {
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "worker").
 					Label(kftraining.ReplicaIndexLabel, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b2").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1895,21 +1896,21 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("l0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "launcher").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("w0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "worker").
 					Label(kftraining.ReplicaIndexLabel, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("w1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "worker").
 					Label(kftraining.ReplicaIndexLabel, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1917,7 +1918,7 @@ func TestReconcile(t *testing.T) {
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "worker").
 					Label(kftraining.ReplicaIndexLabel, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r2").
 					Obj(),
@@ -1927,7 +1928,7 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("l0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "launcher").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1935,7 +1936,7 @@ func TestReconcile(t *testing.T) {
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "worker").
 					Label(kftraining.ReplicaIndexLabel, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b2").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1943,7 +1944,7 @@ func TestReconcile(t *testing.T) {
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "worker").
 					Label(kftraining.ReplicaIndexLabel, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -1951,7 +1952,7 @@ func TestReconcile(t *testing.T) {
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kftraining.JobRoleLabel, "worker").
 					Label(kftraining.ReplicaIndexLabel, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r2").
 					Obj(),
@@ -2026,25 +2027,25 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("w0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("w1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("w2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("w3", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "3").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 			},
@@ -2053,28 +2054,28 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("w0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("w1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("w2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r2").
 					Obj(),
 				*testingpod.MakePod("w3", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "3").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b2").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
@@ -2149,26 +2150,26 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("w0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("w1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("w2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					TopologySchedulingGate().
 					Obj(),
 				*testingpod.MakePod("w3", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "3").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r2").
 					Obj(),
@@ -2178,28 +2179,28 @@ func TestReconcile(t *testing.T) {
 				*testingpod.MakePod("w0", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "0").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("w1", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "1").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
 				*testingpod.MakePod("w2", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "2").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b1").
 					NodeSelector(tasRackLabel, "r2").
 					Obj(),
 				*testingpod.MakePod("w3", "ns").
 					Annotation(kueuealpha.WorkloadAnnotation, "unit-test").
 					Label(kueuealpha.PodGroupPodIndexLabel, "3").
-					Label(kueuealpha.PodSetLabel, string(kueue.DefaultPodSetName)).
+					Label(controllerconsts.PodSetLabel, string(kueue.DefaultPodSetName)).
 					NodeSelector(tasBlockLabel, "b2").
 					NodeSelector(tasRackLabel, "r1").
 					Obj(),
diff --git a/pkg/util/testingjobs/leaderworkerset/wrappers.go b/pkg/util/testingjobs/leaderworkerset/wrappers.go
index d27730ad4..59380ca20 100644
--- a/pkg/util/testingjobs/leaderworkerset/wrappers.go
+++ b/pkg/util/testingjobs/leaderworkerset/wrappers.go
@@ -149,6 +149,12 @@ func (w *LeaderWorkerSetWrapper) Replicas(n int32) *LeaderWorkerSetWrapper {
 	return w
 }
 
+// Replicas sets the number of ready replicas of the LeaderWorkerSet.
+func (w *LeaderWorkerSetWrapper) ReadyReplicas(n int32) *LeaderWorkerSetWrapper {
+	w.Status.ReadyReplicas = n
+	return w
+}
+
 // Size sets the size of the LeaderWorkerSet.
 func (w *LeaderWorkerSetWrapper) Size(n int32) *LeaderWorkerSetWrapper {
 	w.Spec.LeaderWorkerTemplate.Size = ptr.To[int32](n)
diff --git a/pkg/util/testingjobs/raycluster/wrappers.go b/pkg/util/testingjobs/raycluster/wrappers.go
index 118a6e195..2edc0b679 100644
--- a/pkg/util/testingjobs/raycluster/wrappers.go
+++ b/pkg/util/testingjobs/raycluster/wrappers.go
@@ -190,6 +190,23 @@ func (j *ClusterWrapper) Label(key, value string) *ClusterWrapper {
 	return j
 }
 
+// NodeLabel sets the label key and value for specific RayNodeType
+func (j *ClusterWrapper) NodeLabel(rayType rayv1.RayNodeType, key, value string) *ClusterWrapper {
+	switch rayType {
+	case rayv1.HeadNode:
+		if j.Spec.HeadGroupSpec.Template.Labels == nil {
+			j.Spec.HeadGroupSpec.Template.Labels = make(map[string]string)
+		}
+		j.Spec.HeadGroupSpec.Template.Labels[key] = value
+	case rayv1.WorkerNode:
+		if j.Spec.WorkerGroupSpecs[0].Template.Labels == nil {
+			j.Spec.WorkerGroupSpecs[0].Template.Labels = make(map[string]string)
+		}
+		j.Spec.WorkerGroupSpecs[0].Template.Labels[key] = value
+	}
+	return j
+}
+
 // StatusConditions adds a condition
 func (j *ClusterWrapper) StatusConditions(c metav1.Condition) *ClusterWrapper {
 	j.Status.Conditions = append(j.Status.Conditions, c)
diff --git a/test/e2e/singlecluster/e2e_test.go b/test/e2e/singlecluster/e2e_test.go
index 4f41c5c42..291a62e2f 100644
--- a/test/e2e/singlecluster/e2e_test.go
+++ b/test/e2e/singlecluster/e2e_test.go
@@ -308,6 +308,101 @@ var _ = ginkgo.Describe("Kueue", func() {
 				}, util.LongTimeout, util.Interval).Should(gomega.Succeed())
 			})
 		})
+
+		ginkgo.It("Should allow updating the workload's priority through the job", func() {
+			lowPriority := "low-priority"
+			lowPriorityClass := testing.MakeWorkloadPriorityClass(lowPriority).PriorityValue(100).Obj()
+			util.MustCreate(ctx, k8sClient, lowPriorityClass)
+			ginkgo.DeferCleanup(func() {
+				gomega.Expect(k8sClient.Delete(ctx, lowPriorityClass)).To(gomega.Succeed())
+			})
+
+			midPriority := "mid-priority"
+			midPriorityClass := testing.MakeWorkloadPriorityClass(midPriority).PriorityValue(200).Obj()
+			util.MustCreate(ctx, k8sClient, midPriorityClass)
+			ginkgo.DeferCleanup(func() {
+				gomega.Expect(k8sClient.Delete(ctx, midPriorityClass)).To(gomega.Succeed())
+			})
+
+			highPriority := "high-priority"
+			highPriorityClass := testing.MakeWorkloadPriorityClass(highPriority).PriorityValue(300).Obj()
+			util.MustCreate(ctx, k8sClient, highPriorityClass)
+			ginkgo.DeferCleanup(func() {
+				gomega.Expect(k8sClient.Delete(ctx, highPriorityClass)).To(gomega.Succeed())
+			})
+
+			ginkgo.By("Create job-one with mid priority", func() {
+				sampleJob = (&testingjob.JobWrapper{Job: *sampleJob}).
+					WorkloadPriorityClass(midPriority).
+					Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
+					NodeSelector("instance-type", "on-demand").
+					Obj()
+				util.MustCreate(ctx, k8sClient, sampleJob)
+			})
+
+			ginkgo.By("Verify the job-one is running", func() {
+				util.ExpectJobUnsuspendedWithNodeSelectors(ctx, k8sClient, jobKey, map[string]string{
+					"instance-type": "on-demand",
+				})
+			})
+
+			var sampleJob2 *batchv1.Job
+			ginkgo.By("Create job-two with low priority", func() {
+				sampleJob2 = testingjob.MakeJob("test-job-2", ns.Name).
+					Queue("main").
+					RequestAndLimit("cpu", "1").
+					RequestAndLimit("memory", "20Mi").
+					WorkloadPriorityClass(lowPriority).
+					Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
+					NodeSelector("instance-type", "on-demand").
+					Obj()
+				util.MustCreate(ctx, k8sClient, sampleJob2)
+			})
+
+			ginkgo.By("Verify workload with low priority is not admitted", func() {
+				createdJob := &batchv1.Job{}
+				jobKey = client.ObjectKeyFromObject(sampleJob2)
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, jobKey, createdJob)).Should(gomega.Succeed())
+					g.Expect(*createdJob.Spec.Suspend).Should(gomega.BeTrue())
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+
+				wlLookupKey := types.NamespacedName{Name: workloadjob.GetWorkloadNameForJob(sampleJob2.Name, sampleJob2.UID), Namespace: ns.Name}
+				createdWorkload := &kueue.Workload{}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, wlLookupKey, createdWorkload)).Should(gomega.Succeed())
+					g.Expect(workload.HasQuotaReservation(createdWorkload)).Should(gomega.BeFalse())
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("Increase job-two priority", func() {
+				createdJob := &batchv1.Job{}
+				jobKey = client.ObjectKeyFromObject(sampleJob2)
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, jobKey, createdJob)).Should(gomega.Succeed())
+					createdJob.Labels[constants.WorkloadPriorityClassLabel] = highPriority
+					g.Expect(k8sClient.Update(ctx, createdJob)).Should(gomega.Succeed())
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("Verify workload priority was updated", func() {
+				wlLookupKey := types.NamespacedName{Name: workloadjob.GetWorkloadNameForJob(sampleJob2.Name, sampleJob2.UID), Namespace: ns.Name}
+				createdWorkload := &kueue.Workload{}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, wlLookupKey, createdWorkload)).Should(gomega.Succeed())
+					g.Expect(ptr.Deref(createdWorkload.Spec.Priority, -1)).Should(gomega.Equal(highPriorityClass.Value))
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("Verify job-two is running", func() {
+				createdJob := &batchv1.Job{}
+				jobKey = client.ObjectKeyFromObject(sampleJob2)
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, jobKey, createdJob)).Should(gomega.Succeed())
+					g.Expect(*createdJob.Spec.Suspend).Should(gomega.BeFalse())
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+		})
 	})
 
 	ginkgo.When("Creating a Job In a Twostepadmission Queue", func() {
diff --git a/test/e2e/singlecluster/leaderworkerset_test.go b/test/e2e/singlecluster/leaderworkerset_test.go
index d441ba0ff..a10c58511 100644
--- a/test/e2e/singlecluster/leaderworkerset_test.go
+++ b/test/e2e/singlecluster/leaderworkerset_test.go
@@ -21,6 +21,7 @@ import (
 	"github.com/onsi/gomega"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/utils/ptr"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -28,6 +29,7 @@ import (
 
 	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta1"
 	"sigs.k8s.io/kueue/pkg/constants"
+	ctrlconstants "sigs.k8s.io/kueue/pkg/controller/constants"
 	"sigs.k8s.io/kueue/pkg/controller/jobs/leaderworkerset"
 	podconstants "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants"
 	"sigs.k8s.io/kueue/pkg/util/testing"
@@ -797,6 +799,119 @@ var _ = ginkgo.Describe("LeaderWorkerSet integration", func() {
 				}, util.Timeout, util.Interval).Should(gomega.Succeed())
 			})
 		})
+
+		ginkgo.It("should allow to update the workload priority in LeaderWorkerSet", func() {
+			lowPriorityLWS := leaderworkersettesting.MakeLeaderWorkerSet("low-priority", ns.Name).
+				Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
+				Size(3).
+				Replicas(1).
+				RequestAndLimit(corev1.ResourceCPU, "1").
+				TerminationGracePeriod(1).
+				Queue(lq.Name).
+				WorkloadPriorityClass(lowPriorityWPC.Name).
+				Obj()
+			ginkgo.By("Create a low priority LeaderWorkerSet", func() {
+				util.MustCreate(ctx, k8sClient, lowPriorityLWS)
+			})
+
+			ginkgo.By("Waiting for replicas is ready in low priority LeaderWorkerSet", func() {
+				createdLowPriorityLWS := &leaderworkersetv1.LeaderWorkerSet{}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(lowPriorityLWS), createdLowPriorityLWS)).To(gomega.Succeed())
+					g.Expect(createdLowPriorityLWS.Status.ReadyReplicas).To(gomega.Equal(int32(1)))
+					g.Expect(createdLowPriorityLWS.Status.Conditions).To(gomega.ContainElement(
+						gomega.BeComparableTo(metav1.Condition{
+							Type:   "Available",
+							Status: metav1.ConditionTrue,
+							Reason: "AllGroupsReady",
+						}, util.IgnoreConditionTimestampsAndObservedGeneration, util.IgnoreConditionMessage),
+					))
+				}, util.LongTimeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			createdLowPriorityWl := &kueue.Workload{}
+			lowPriorityWlLookupKey := types.NamespacedName{
+				Name:      leaderworkerset.GetWorkloadName(lowPriorityLWS.UID, lowPriorityLWS.Name, "0"),
+				Namespace: ns.Name,
+			}
+			ginkgo.By("Check the low priority Workload is created and admitted", func() {
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, lowPriorityWlLookupKey, createdLowPriorityWl)).To(gomega.Succeed())
+					g.Expect(createdLowPriorityWl.Spec.PriorityClassSource).To(gomega.Equal(constants.WorkloadPriorityClassSource))
+					g.Expect(createdLowPriorityWl.Spec.PriorityClassName).To(gomega.Equal(lowPriorityWPC.Name))
+					g.Expect(createdLowPriorityWl.Status.Conditions).To(testing.HaveConditionStatusTrue(kueue.WorkloadAdmitted))
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			updatablePriorityLWS := leaderworkersettesting.MakeLeaderWorkerSet("updatable-priority", ns.Name).
+				Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
+				Size(3).
+				Replicas(1).
+				RequestAndLimit(corev1.ResourceCPU, "1").
+				TerminationGracePeriod(1).
+				Queue(lq.Name).
+				WorkloadPriorityClass(lowPriorityWPC.Name).
+				Obj()
+			ginkgo.By("Create another low priority LeaderWorkerSet", func() {
+				util.MustCreate(ctx, k8sClient, updatablePriorityLWS)
+			})
+
+			createdUpdatablePriorityWl := &kueue.Workload{}
+			updatablePriorityWlLookupKey := types.NamespacedName{
+				Name:      leaderworkerset.GetWorkloadName(updatablePriorityLWS.UID, updatablePriorityLWS.Name, "0"),
+				Namespace: ns.Name,
+			}
+
+			ginkgo.By("Check the new Workload is created with low priority", func() {
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, updatablePriorityWlLookupKey, createdUpdatablePriorityWl)).To(gomega.Succeed())
+					g.Expect(createdUpdatablePriorityWl.Spec.PriorityClassName).To(gomega.Equal(lowPriorityWPC.Name))
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("Updating priority of second LeaderWorkerSet to high-priority", func() {
+				createdHighPriorityLWS := &leaderworkersetv1.LeaderWorkerSet{}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(updatablePriorityLWS), createdHighPriorityLWS)).To(gomega.Succeed())
+					createdHighPriorityLWS.Labels[ctrlconstants.WorkloadPriorityClassLabel] = highPriorityWPC.Name
+					g.Expect(k8sClient.Update(ctx, createdHighPriorityLWS)).Should(gomega.Succeed())
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("Checking the Workload priority is updated", func() {
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, updatablePriorityWlLookupKey, createdUpdatablePriorityWl)).To(gomega.Succeed())
+					g.Expect(createdUpdatablePriorityWl.Spec.PriorityClassName).To(gomega.Equal(highPriorityWPC.Name))
+				}, util.LongTimeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("Await for the low-priority LeaderWorkerSet to be preempted", func() {
+				createdLowPriorityLWS := &leaderworkersetv1.LeaderWorkerSet{}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(lowPriorityLWS), createdLowPriorityLWS)).To(gomega.Succeed())
+					g.Expect(createdLowPriorityLWS.Status.ReadyReplicas).To(gomega.Equal(int32(0)))
+					g.Expect(createdLowPriorityLWS.Status.Conditions).To(gomega.ContainElements(
+						gomega.BeComparableTo(metav1.Condition{
+							Type:   "Progressing",
+							Status: metav1.ConditionTrue,
+							Reason: "GroupsProgressing",
+						}, util.IgnoreConditionTimestampsAndObservedGeneration, util.IgnoreConditionMessage),
+						gomega.BeComparableTo(metav1.Condition{
+							Type:   "Available",
+							Status: metav1.ConditionFalse,
+							Reason: "AllGroupsReady",
+						}, util.IgnoreConditionTimestampsAndObservedGeneration, util.IgnoreConditionMessage),
+					))
+				}, util.LongTimeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("Await for the high priority Workload to be admitted", func() {
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, updatablePriorityWlLookupKey, createdUpdatablePriorityWl)).To(gomega.Succeed())
+					g.Expect(createdUpdatablePriorityWl.Status.Conditions).To(testing.HaveConditionStatusTrue(kueue.WorkloadAdmitted))
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+		})
 	})
 
 	ginkgo.When("Workload deactivated", func() {
diff --git a/test/integration/singlecluster/controller/jobs/job/job_controller_test.go b/test/integration/singlecluster/controller/jobs/job/job_controller_test.go
index 688310f6e..14ec9698f 100644
--- a/test/integration/singlecluster/controller/jobs/job/job_controller_test.go
+++ b/test/integration/singlecluster/controller/jobs/job/job_controller_test.go
@@ -52,13 +52,15 @@ import (
 )
 
 const (
-	parallelism       = 4
-	jobName           = "test-job"
-	instanceKey       = "cloud.provider.com/instance"
-	priorityClassName = "test-priority-class"
-	priorityValue     = 10
-	parentJobName     = jobName + "-parent"
-	childJobName      = jobName + "-child"
+	parallelism           = 4
+	jobName               = "test-job"
+	instanceKey           = "cloud.provider.com/instance"
+	priorityClassName     = "test-priority-class"
+	priorityValue         = 10
+	highPriorityClassName = "high-priority-class"
+	highPriorityValue     = 20
+	parentJobName         = jobName + "-parent"
+	childJobName          = jobName + "-child"
 )
 
 var _ = ginkgo.Describe("Job controller", ginkgo.Ordered, ginkgo.ContinueOnFailure, func() {
@@ -285,6 +287,53 @@ var _ = ginkgo.Describe("Job controller", ginkgo.Ordered, ginkgo.ContinueOnFailu
 		}, util.Timeout, util.Interval).Should(gomega.Succeed())
 	})
 
+	ginkgo.It("Should sync workload priority when job priority label changes", func() {
+		priorityClass := testing.MakeWorkloadPriorityClass(priorityClassName).
+			PriorityValue(int32(priorityValue)).Obj()
+		util.MustCreate(ctx, k8sClient, priorityClass)
+		ginkgo.DeferCleanup(func() {
+			gomega.Expect(k8sClient.Delete(ctx, priorityClass)).To(gomega.Succeed())
+		})
+
+		highPriorityClass := testing.MakeWorkloadPriorityClass(highPriorityClassName).
+			PriorityValue(int32(highPriorityValue)).Obj()
+		util.MustCreate(ctx, k8sClient, highPriorityClass)
+		ginkgo.DeferCleanup(func() {
+			gomega.Expect(k8sClient.Delete(ctx, highPriorityClass)).To(gomega.Succeed())
+		})
+
+		ginkgo.By("creating job with priority")
+		job := testingjob.MakeJob(jobName, ns.Name).
+			WorkloadPriorityClass(priorityClassName).
+			Obj()
+		util.MustCreate(ctx, k8sClient, job)
+		lookupKey := types.NamespacedName{Name: jobName, Namespace: ns.Name}
+		createdJob := &batchv1.Job{}
+		gomega.Eventually(func(g gomega.Gomega) {
+			g.Expect(k8sClient.Get(ctx, lookupKey, createdJob)).Should(gomega.Succeed())
+			g.Expect(createdJob.Spec.Suspend).Should(gomega.Equal(ptr.To(true)))
+		}, util.Timeout, util.Interval).Should(gomega.Succeed())
+
+		ginkgo.By("checking the workload is created with priority, priorityName")
+		createdWorkload := &kueue.Workload{}
+		wlLookupKey := types.NamespacedName{Name: workloadjob.GetWorkloadNameForJob(job.Name, job.UID), Namespace: ns.Name}
+		gomega.Eventually(func(g gomega.Gomega) {
+			g.Expect(k8sClient.Get(ctx, wlLookupKey, createdWorkload)).Should(gomega.Succeed())
+		}, util.Timeout, util.Interval).Should(gomega.Succeed())
+		gomega.Expect(createdWorkload.Spec.PriorityClassName).Should(gomega.Equal(priorityClassName))
+		gomega.Expect(*createdWorkload.Spec.Priority).Should(gomega.Equal(int32(priorityValue)))
+
+		ginkgo.By("checking the workload priority is updated when the job priority label changes")
+		gomega.Expect(k8sClient.Get(ctx, lookupKey, createdJob)).Should(gomega.Succeed())
+		createdJob.Labels[constants.WorkloadPriorityClassLabel] = highPriorityClassName
+		gomega.Expect(k8sClient.Update(ctx, createdJob)).Should(gomega.Succeed())
+		gomega.Eventually(func(g gomega.Gomega) {
+			g.Expect(k8sClient.Get(ctx, wlLookupKey, createdWorkload)).Should(gomega.Succeed())
+			g.Expect(createdWorkload.Spec.PriorityClassName).Should(gomega.Equal(highPriorityClassName))
+			g.Expect(*createdWorkload.Spec.Priority).Should(gomega.Equal(int32(highPriorityValue)))
+		}, util.Timeout, util.Interval).Should(gomega.Succeed())
+	})
+
 	ginkgo.It("Should reconcile job when queueName set by annotation (deprecated)", func() {
 		ginkgo.By("checking the workload is created with correct queue name assigned")
 		var jobQueueName kueue.LocalQueueName = "test-queue"