Repository: kubernetes-sigs/kueue (Go)

Issue:
QueueName in Workload sometimes is removed
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


**What happened**:
After I deployed two MPIJobs and the kueue-controller created Workload with QueueName, the kueue-controller sometimes removes `.spec.queueName` in Workload.

```shell
$ kubectl get workloads.kueue.x-k8s.io 
NAME                QUEUE        ADMITTED BY     AGE
mpijob-pi-0-1881b   user-queue   cluster-queue   54s
mpijob-pi-1-ef222   user-queue                   54s
$
$ kubectl get workloads.kueue.x-k8s.io 
NAME                QUEUE        ADMITTED BY     AGE
mpijob-pi-0-1881b   user-queue   cluster-queue   55s
mpijob-pi-1-ef222                                55s
```

**What you expected to happen**:
QueueName in Workload isn't removed.

**How to reproduce it (as minimally and precisely as possible)**:

- manifests for kueue

```yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: "default-flavor"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: "cluster-queue"
spec:
  namespaceSelector: {} # match all.
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 4
      - name: "memory"
        nominalQuota: 5Gi
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  namespace: "default"
  name: "user-queue"
spec:
  clusterQueue: "cluster-queue"
```

- MPIJobs

```yaml
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: pi-0
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  slotsPerWorker: 1
  runPolicy:
    suspend: true
    cleanPodPolicy: Running
    ttlSecondsAfterFinished: 60
  sshAuthMountPath: /home/mpiuser/.ssh
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
          - image: mpioperator/mpi-pi:openmpi
            name: mpi-launcher
            securityContext:
              runAsUser: 1000
            command:
            - mpirun
            args:
            - -n
            - "2"
            - /home/mpiuser/pi
            resources:
              limits:
                cpu: 1
                memory: 1Gi
    Worker:
      replicas: 2
      template:
        spec:
          containers:
          - image: mpioperator/mpi-pi:openmpi
            name: mpi-worker
            securityContext:
              runAsUser: 1000
            command:
            - /usr/sbin/sshd
            args:
            - -De
            - -f
            - /home/mpiuser/.sshd_config
            resources:
              limits:
                cpu: 1
                memory: 1Gi
---
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: pi-1
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  slotsPerWorker: 1
  runPolicy:
    suspend: true
    cleanPodPolicy: Running
    ttlSecondsAfterFinished: 60
  sshAuthMountPath: /home/mpiuser/.ssh
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
          - image: mpioperator/mpi-pi:openmpi
            name: mpi-launcher
            securityContext:
              runAsUser: 1000
            command:
            - mpirun
            args:
            - -n
            - "2"
            - /home/mpiuser/pi
            resources:
              limits:
                cpu: 1
                memory: 1Gi
    Worker:
      replicas: 2
      template:
        spec:
          containers:
          - image: mpioperator/mpi-pi:openmpi
            name: mpi-worker
            securityContext:
              runAsUser: 1000
            command:
            - /usr/sbin/sshd
            args:
            - -De
            - -f
            - /home/mpiuser/.sshd_config
            resources:
              limits:
                cpu: 1
                memory: 1Gi
```

**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`): v1.25.5
- Kueue version (use `git describe --tags --dirty --always`): v0.4.0
- Cloud provider or hardware configuration: On-Prem
- OS (e.g: `cat /etc/os-release`): 
```
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
```
- Kernel (e.g. `uname -a`): `Linux 5.4.0-136-generic #153-Ubuntu SMP Thu Nov 24 15:56:58 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux`
- Install tools:
- Others:


Discussion: /assign

Please include a change for the changelog when you work on the fix. You probably need to create a new file for 0.4. Once approved we can cherry pick.
Thanks for reporting!
> Please include a change for the changelog when you work on the fix. You probably need to create a new file for 0.4. Once approved we can cherry pick. Thanks for reporting!

Sure!

Through more investigation, the below process updates the queue name with an empty:

https://github.com/kubernetes-sigs/kueue/blob/9ca57c86cf06c11a94a2d5b7badf60233a51a2f2/pkg/controller/jobframework/reconciler.go#L181-L191

When I commented on the above place, the queue name wasn't updated with an empty.

Oh, I could reach the root cause :)
It seems that the batch/v1 Job integration controller updates the field.




Fix (patch):
diff --git a/CHANGELOG/CHANGELOG-0.4.md b/CHANGELOG/CHANGELOG-0.4.md
new file mode 100644
index 00000000000..e2311da28dc
--- /dev/null
+++ b/CHANGELOG/CHANGELOG-0.4.md
@@ -0,0 +1,7 @@
+## v0.4.0
+
+Changes since `v0.3.0`:
+
+### Bug fixes
+
+- Fix a bug that updates a queue name in workloads with an empty value when using framework jobs that use batch/job internally, such as MPIJob. #713 
diff --git a/pkg/controller/jobframework/reconciler.go b/pkg/controller/jobframework/reconciler.go
index dc59aaaa034..bf0e3b55cf4 100644
--- a/pkg/controller/jobframework/reconciler.go
+++ b/pkg/controller/jobframework/reconciler.go
@@ -180,7 +180,7 @@ func (r *JobReconciler) ReconcileGenericJob(ctx context.Context, req ctrl.Reques
 
 		// update queue name if changed.
 		q := QueueName(job)
-		if wl.Spec.QueueName != q {
+		if wl.Spec.QueueName != q && isStandaloneJob {
 			log.V(2).Info("Job changed queues, updating workload")
 			wl.Spec.QueueName = q
 			err := r.client.Update(ctx, wl)