Repository: kubernetes-sigs/kueue

Issue:
Add option to disable strict pod spec validation
<!-- Please only use this template for submitting enhancement requests -->
**What would you like to be added**:

We would need to influence strictness level of Pod spec validation.

**Why is this needed**:

It turned out Kueue validates Pod spec differently compared to plain K8s.
In our case workloads have been blocked due to env vars being duplicated in Pod spec.
Because there are many components involved regarding pod spec creation and manipulation
we cannot guarantee env var keys to be unique.

Plain K8s wouldn't complain about duplicated env vars. This is why we think it would be helpful
to offer more control on Kueue's validation mechanism.

This could also be relevant to resource types other than Pods.

Log of kueue controller `v0.8.1`:
``` json
 {
    "level": "error",
    "ts": "2024-11-14T17:56:56.563529963Z",
    "caller": "controller/controller.go:329",
    "msg": "Reconcilererror",
    "controller": "v1_pod",
    "namespace": "",
    "name": "",
    "reconcileID": "",
    "error": "Workload.kueue.x-k8s.io\"pod-reproduce-env-bug-o5ymx1-n0-0-24899\"isinvalid:spec.podSets[0].template.spec.containers[0].env[2]:Duplicatevalue:map[string]interface{},{\"name\":\"DUPLICATE_ENV\"},",
    "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"
  }
```

**Completion requirements**:

This enhancement requires the following artifacts:

- [ ] Design doc
- [x] API change
- [x] Docs update

The artifacts should be linked in subsequent comments.


<sub>Lucas Zanella [lucas.zanella@mercedes-benz.com](mailto:lucas.zanella@mercedes-benz.com), Mercedes-Benz Tech Innovation GmbH, [imprint](https://github.com/mercedes-benz/foss/blob/master/PROVIDER_INFORMATION.md)</sub>


Hints from issue comments:
@mimowo: I believe this is not about Kueue validation but the underlying bug in SSA which prevents updates when fields are duplicated. For reference: https://github.com/kubernetes/kubernetes/issues/113482.

Given the bug for SSA exists for 2 years now I think it is unlikely this will be resolved soon, so I would suggest to consider withdrawing from SSA in Kueue. This is what we did for PodFailurePolicy in the core k8s: https://github.com/kubernetes/kubernetes/pull/121103.

I guess we could maintain two modes in Kueue behind a feature gate like `UseSSAForWorkload`, in case someone needs it. WDYT @tenzen-y ?@mbobrovskyi: > I believe this is not about Kueue validation but the underlying bug in SSA which prevents updates when fields are duplicated. For reference: [kubernetes/kubernetes#113482](https://github.com/kubernetes/kubernetes/issues/113482).
> 
> Given the bug for SSA exists for 2 years now I think it is unlikely this will be resolved soon, so I would suggest to consider withdrawing from SSA in Kueue. This is what we did for PodFailurePolicy in the core k8s: [kubernetes/kubernetes#121103](https://github.com/kubernetes/kubernetes/pull/121103).
> 
> I guess we could maintain two modes in Kueue behind a feature gate like `UseSSAForWorkload`, in case someone needs it. WDYT @tenzen-y ?

The problem occurs when creating the workload, but not when patching it.

https://github.com/kubernetes-sigs/kueue/blob/30da29361a6fc95ade0a4350b97494394d30b0b9/pkg/controller/jobframework/reconciler.go#L1050-L1052@mimowo: @mbobrovskyi can you check which layer rejects the creation? Is this validation in Kueue? If validation if Kueue I suppose it might have been added to prevent later failures due to SSA for patches.@mbobrovskyi: > @mbobrovskyi can you check which layer rejects the creation? Is this validation in Kueue?

No, it looks like k8s validation.@mimowo: Interesting, the error message in the description is from `controller/controller.go:329`, and the message contains `Workload.kueue.x-k8s.io\"pod-reproduce-env-bug-o5ymx1-n0-0-24899`, which suggests that the object already exists.@mbobrovskyi: I tried with creation. But I believe we have the same issue with update.

```
{"level":"Level(-2)","ts":"2024-11-18T05:35:30.888806634Z","caller":"jobframework/reconciler.go:332","msg":"Reconciling Job","controller":"v1_pod","namespace":"default","name":"kueue-sleep-bswbh","reconcileID":"541c2f22-019a-4285-a710-5a7f28c0c0ba","job":"default/kueue-sleep-bswbh","gvk":"/v1, Kind=Pod"}
{"level":"error","ts":"2024-11-18T05:35:30.902655051Z","caller":"jobframework/reconciler.go:410","msg":"Handling job with no workload","controller":"v1_pod","namespace":"default","name":"kueue-sleep-bswbh","reconcileID":"541c2f22-019a-4285-a710-5a7f28c0c0ba","job":"default/kueue-sleep-bswbh","gvk":"/v1, Kind=Pod","error":"Workload.kueue.x-k8s.io \"pod-kueue-sleep-bswbh-cb334\" is invalid: spec.podSets[0].template.spec.containers[0].env[1]: Duplicate value: map[string]interface {}{\"name\":\"DUPLICATE\"}","stacktrace":"sigs.k8s.io/kueue/pkg/controller/jobframework.(*JobReconciler).ReconcileGenericJob\n\t/workspace/pkg/controller/jobframework/reconciler.go:410\nsigs.k8s.io/kueue/pkg/controller/jobs/pod.(*Reconciler).Reconcile\n\t/workspace/pkg/controller/jobs/pod/pod_controller.go:123\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2024-11-18T05:35:30.902756843Z","caller":"controller/controller.go:316","msg":"Reconciler error","controller":"v1_pod","namespace":"default","name":"kueue-sleep-bswbh","reconcileID":"541c2f22-019a-4285-a710-5a7f28c0c0ba","error":"Workload.kueue.x-k8s.io \"pod-kueue-sleep-bswbh-cb334\" is invalid: spec.podSets[0].template.spec.containers[0].env[1]: Duplicate value: map[string]interface {}{\"name\":\"DUPLICATE\"}","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224"}
```@hy00nc: Hi @mimowo , do we have any updates on this? I'm facing the same issue on pod creation and this is blocking Kueue from creating a Workload from a pod making it stuck in scheduling gated without any related events.

It is hard to notice why the pod is not scheduled unless users are familiar with Kueue and look into controller logs. I'd really appreciate if we can get some enhancement on this part.@tenzen-y: > I believe this is not about Kueue validation but the underlying bug in SSA which prevents updates when fields are duplicated. For reference: [kubernetes/kubernetes#113482](https://github.com/kubernetes/kubernetes/issues/113482).
> 
> Given the bug for SSA exists for 2 years now I think it is unlikely this will be resolved soon, so I would suggest to consider withdrawing from SSA in Kueue. This is what we did for PodFailurePolicy in the core k8s: [kubernetes/kubernetes#121103](https://github.com/kubernetes/kubernetes/pull/121103).
> 
> I guess we could maintain two modes in Kueue behind a feature gate like `UseSSAForWorkload`, in case someone needs it. WDYT [@tenzen-y](https://github.com/tenzen-y) ?

As I confirmed the duplicated map keys (ports, envs, and so on), I still face the same issue in v1.32 cluster as well.
But, as we can see https://github.com/kubernetes/kubernetes/issues/113482, this issue seems not to be resolved.

So, I agree with introducing `UseSSAForWorkload` feature gate. After the root cause is fixed in core kube, we can delete the `UseSSAForWorkload`. I honestly think we should not use the duplicated keys in those fields.

But, I think `UseSSAForWorkoad` could be helpful to mitigate this issue immediately, then provide the improvement period in their production cluster. @mimowo Any thoughts?

Note that `UseSSAForWorkload` should always be enabled by default.@k8s-triage-robot: The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle stale`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/@k8s-triage-robot: The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle rotten`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle rotten

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/@mimowo: /remove-lifecycle rotten@mimowo: /kind bug
I think this can be considered a bug in Kueue. Users can create workloads which execute of without Kueue, but fail to execute when using Kueue. Yes, these are corner cases, but we've seen such workloads in practice.@mimowo: > But, I think UseSSAForWorkoad could be helpful to mitigate this issue immediately, then provide the improvement period in their production cluster. @mimowo Any thoughts?

I'm ok with the feature gate, but given the amount of issues SSA creates, I would just drop it altogether from Kueue, see https://github.com/kubernetes-sigs/kueue/issues/6158.

I'm not clear what are the benefits of using SSA in Kueue, especially as the upstream bug is not fixed for 3 years it does not seem as fixed any time soon. Now we have in Kueue a weird mixture of Patches and SSA which makes reasoning about code hard.

> Note that UseSSAForWorkload should always be enabled by default.

TBH, I would propose to have feature gate `UsaPatchForWorkload` and graduate it (see above).@mimowo: @tenzen-y do you see any reason to continue using SSA in Kueue?@mwysokin: @LucasZanellaMBTI I just noticed that the progress on the issue is not clearly visible here. We've decided to implement an alternative strategy to SSA using PATCH and the work is in-flight. We're planning to release it in September. The issue where you can track the work is here: https://github.com/kubernetes-sigs/kueue/issues/6158@LucasZanellaMBTI: @mwysokin we are glad to read this. Thanks for the update here.@bd-hhause: Just ran into this today with Buildkite adding a duplicate key to `spec.podSets[0].template.spec.containers[0].env`. Also happy to see that a fix is actively being worked on.@mwysokin: @LucasZanellaMBTI @bd-hhause FYI The feature was implemented and is available in releases 0.14.0 and 0.14.1
The docs: https://kueue.sigs.k8s.io/docs/concepts/workload/#workload-updates-by-kueue
@bd-hhause: Awesome, we'll try out the `WorkloadRequestUseMergePatch` soon. Where is this mentioned in the release notes?@mwysokin: I think the contributor and the reviewers might've missed adding the proper release notes block because the feature was scattered across multiple PRs or only adding a switch was worthy of a release notes item because I think I see it as the last item in the features section for release 0.14.0


If you're interested in how it was solved here's the umbrella issue (https://github.com/kubernetes-sigs/kueue/issues/6158). 

CC @mimowo @mszadkow @mimowo: > Awesome, we'll try out the WorkloadRequestUseMergePatch soon. Where is this mentioned in the release notes?

Great, I can see it is listed under [0.14.0](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.14.0), the notes for patch releases are only incremental. So, when you look for notes for "features" they are all under 0.x.0 release.

> I think the contributor and the reviewers might've missed adding the proper release notes block because the feature was scattered across multiple PRs or only adding a switch was worthy of a release notes

Indeed, when a feature is backed by multiple PRs we only report once. This is not unique to that feature by any means.

While this is implemented, note that the feature remains Alpha. One "known issue" is that on rare occasions the conditions set by a Kueue internal controller might be overitten by a patch sent by Kueue scheduler. This should be safe in all cases I know about - the controller will be re-triggered and will re-add the condition. However, I'm still considering this as an area for improvement before graduation to Beta: https://github.com/kubernetes-sigs/kueue/issues/7035

/close
Let's use  https://github.com/kubernetes-sigs/kueue/issues/7035 to track the remaining work.@k8s-ci-robot: @mimowo: Closing this issue.

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/3540#issuecomment-3384426804):

>> Awesome, we'll try out the WorkloadRequestUseMergePatch soon. Where is this mentioned in the release notes?
>
>Great, I can see it is listed under 0.14.0, the notes for patch releases are only incremental. So, when you look for notes for "features" they are all under 0.x.0 release.
>
>> I think the contributor and the reviewers might've missed adding the proper release notes block because the feature was scattered across multiple PRs or only adding a switch was worthy of a release notes
>
>Indeed, when a feature is backed by multiple PRs we only report once. This is not unique to that feature by any means.
>
>While this is implemented, note that the feature remains Alpha. One "known issue" is that on rare occasions the conditions set by a Kueue internal controller might be overitten by a patch sent by Kueue scheduler. This should be safe in all cases I know about - the controller will be re-triggered and will re-add the condition. However, I'm still considering this as an area for improvement before graduation to Beta: https://github.com/kubernetes-sigs/kueue/issues/7035
>
>/close
>Let's use  https://github.com/kubernetes-sigs/kueue/issues/7035 to track the remaining work.


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@LucasZanellaMBTI: @mimowo @mwysokin thank you so much for implementing this. We we'll give it a try asap.@mimowo: @LucasZanellaMBTI we just released 0.14.2 which contains a small fix for the mechanism. 

It would be great if you can share results of your testing@bd-hhause: @mimowo I tried enabling the feature flag a couple of days ago and am still experiencing the following error in the Kueue controller: 

```
"error":"Workload.kueue.x-k8s.io \"job-buildkite-0199151e-0f3b-4dd8-b28b-8790dbf06e9e-08da5\" is invalid: spec.podSets[0].template.spec.containers[0].env[14]: Duplicate value: map[string]interface {}{\"name\":\"BUILDKITE_HOOKS_PATH\"}"
```

This error occurs when the workload first is created at `handleJobWithNoWorkload` in the jobFramework reconciler. I am also exploring options to remove the duplicate ENV VAR from the other controller, since it doesn't seem like it should be Kueue's responsibility to handle this case.

Does this case fall outside of the scope of changes in this PR? @mimowo: > @mimowo I tried enabling the feature flag a couple of days ago and am still experiencing the following error in the Kueue controller:

This is quite surprising, because this error is SSA specific, whilst with this feature gate SSA should not be used.

There might be an omission though, we should test this more end-to-end. I will update the thread.@bd-hhause: Thanks, let me know if you'd like me to share a minimal example to reproduce. @mimowo: Do you maybe the logs to include the specific log line which might be sending the problematic patch? @mimowo: /reopen
I reproduced the issue quite easily (on 0.14.2), the log line :
```
{"level":"error","ts":"2025-10-23T14:19:18.259276575Z","caller":"jobframework/reconciler.go:481","msg":"Handling job with no workload","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"sample-jobcfcfp","namespace":"default"},"namespace":"default","name":"sample-jobcfcfp","reconcileID":"0d7f5d12-403b-4f75-be17-15b842e563e7","job":"default/sample-jobcfcfp","gvk":"batch/v1, Kind=Job","error":"Workload.kueue.x-k8s.io \"job-sample-jobcfcfp-b9d77\" is invalid: spec.podSets[0].template.spec.containers[0].env[1]: Duplicate value: {\"name\":\"DEMO\"}","stacktrace":"sigs.k8s.io/kueue/pkg/controller/jobframework.(*JobReconciler).ReconcileGenericJob\n\t/workspace/pkg/controller/jobframework/reconciler.go:481\nsigs.k8s.io/kueue/pkg/controller/jobframework.(*genericReconciler).Reconcile\n\t/workspace/pkg/controller/jobframework/reconciler.go:1485\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296"}
{"level":"error","ts":"2025-10-23T14:19:18.259355125Z","caller":"controller/controller.go:474","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"sample-jobcfcfp","namespace":"default"},"namespace":"default","name":"sample-jobcfcfp","reconcileID":"0d7f5d12-403b-4f75-be17-15b842e563e7","error":"Workload.kueue.x-k8s.io \"job-sample-jobcfcfp-b9d77\" is invalid: spec.podSets[0].template.spec.containers[0].env[1]: Duplicate value: {\"name\":\"DEMO\"}","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296"}
```
with the Job:
```yaml
apiVersion: batch/v1
kind: Job
metadata:
  generateName: sample-job
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  parallelism: 3
  completions: 3
  completionMode: Indexed
  suspend: true
  template:
    spec:
      containers:
      - name: dummy-job
        image: gcr.io/k8s-staging-perf-tests/sleep:v0.0.3
        args: ["300s"]
        env:
        - name: DEMO
          value: "Hello from the environment"
        - name: DEMO
          value: "Such a sweet sorrow"
        resources:
          requests:
            cpu: "1"
            memory: "1Mi"
      restartPolicy: Never
```@k8s-ci-robot: @mimowo: Reopened this issue.

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/3540#issuecomment-3437305129):

>/reopen
>I reproduced the issue quite easily, the log line :
>```
>{"level":"error","ts":"2025-10-23T14:19:18.259276575Z","caller":"jobframework/reconciler.go:481","msg":"Handling job with no workload","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"sample-jobcfcfp","namespace":"default"},"namespace":"default","name":"sample-jobcfcfp","reconcileID":"0d7f5d12-403b-4f75-be17-15b842e563e7","job":"default/sample-jobcfcfp","gvk":"batch/v1, Kind=Job","error":"Workload.kueue.x-k8s.io \"job-sample-jobcfcfp-b9d77\" is invalid: spec.podSets[0].template.spec.containers[0].env[1]: Duplicate value: {\"name\":\"DEMO\"}","stacktrace":"sigs.k8s.io/kueue/pkg/controller/jobframework.(*JobReconciler).ReconcileGenericJob\n\t/workspace/pkg/controller/jobframework/reconciler.go:481\nsigs.k8s.io/kueue/pkg/controller/jobframework.(*genericReconciler).Reconcile\n\t/workspace/pkg/controller/jobframework/reconciler.go:1485\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296"}
>{"level":"error","ts":"2025-10-23T14:19:18.259355125Z","caller":"controller/controller.go:474","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"sample-jobcfcfp","namespace":"default"},"namespace":"default","name":"sample-jobcfcfp","reconcileID":"0d7f5d12-403b-4f75-be17-15b842e563e7","error":"Workload.kueue.x-k8s.io \"job-sample-jobcfcfp-b9d77\" is invalid: spec.podSets[0].template.spec.containers[0].env[1]: Duplicate value: {\"name\":\"DEMO\"}","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296"}
>```
>with the Job:
>```yaml
>apiVersion: batch/v1
>kind: Job
>metadata:
>  generateName: sample-job
>  namespace: default
>  labels:
>    kueue.x-k8s.io/queue-name: user-queue
>spec:
>  parallelism: 3
>  completions: 3
>  completionMode: Indexed
>  suspend: true
>  template:
>    spec:
>      containers:
>      - name: dummy-job
>        image: gcr.io/k8s-staging-perf-tests/sleep:v0.0.3
>        args: ["300s"]
>        env:
>        - name: DEMO
>          value: "Hello from the environment"
>        - name: DEMO
>          value: "Such a sweet sorrow"
>        resources:
>          requests:
>            cpu: "1"
>            memory: "1Mi"
>      restartPolicy: Never
>```


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@mimowo: cc @mszadkow @mbobrovskyi could you check if maybe we still are using SSA in some places?@mbobrovskyi: /assign@bd-hhause: @mimowo Sure, here's a quick example: 

```
apiVersion: batch/v1
kind: Job
metadata:
  name: test-duplicate-env
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: busybox
        image: busybox:1.35
        command: ["sh", "-c", "env | grep TEST_VAR; sleep 30"]
        env:
        - name: TEST_VAR
          value: "first_value"
        - name: TEST_VAR
          value: "second_value"
        - name: OTHER_VAR
          value: "normal_var"
```

`k apply -f test_job.yaml`

View error: 
`k logs -n kueue-system kueue-controller-manager-7499d78568-jb7rv | grep test-duplicate-env`

```
{
  "level": "error",
  "ts": "2025-10-23T14:28:08.106613615Z",
  "caller": "controller/controller.go:474",
  "msg": "Reconciler error",
  "controller": "job",
  "controllerGroup": "batch",
  "controllerKind": "Job",
  "Job": {
    "name": "test-duplicate-env",
  },
  "namespace": "default",
  "name": "test-duplicate-env",
  "reconcileID": "c2f78148-ad97-4f69-a75e-f775eb91bc8d",
  "error": "Workload.kueue.x-k8s.io \"job-test-duplicate-env-0ee7d\" is invalid: spec.podSets[0].template.spec.containers[0].env[1]: Duplicate value: map[string]interface {}{\"name\":\"TEST_VAR\"}",
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296"
}
```

Thanks for the quick response. Happy to provide further details if necessary!@mbobrovskyi: I found easier solution how to catch it:

```yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: Workload
metadata:
  name: sample-job
spec:
  active: true
  queueName: team-a-queue
  podSets:
    - count: 3
      name: main
      template:
        spec:
          containers:
            - name: container
              image: registry.k8s.io/e2e-test-images/agnhost:latest
              args: ["pause"]
              imagePullPolicy: Always
              resources:
                requests:
                  cpu: "1"
                  memory: 200Mi
              env:
                - name: DEMO
                  value: "Hello from the environment"
                - name: DEMO
                  value: "Such a sweet sorrow"
          restartPolicy: Never
```

```
The Workload "sample-job" is invalid: spec.podSets[0].template.spec.containers[0].env[1]: Duplicate value: {"name":"DEMO"}
```

The problem is caused by a duplicated environment variable. We can’t create the workload because of it, and an error occurs in the job framework reconciler. 

https://github.com/kubernetes-sigs/kueue/blob/178155c7eadd36393cecb5da349995c946918b78/pkg/controller/jobframework/reconciler.go#L1378-L1380

```
{"level":"error","ts":"2025-10-23T14:42:57.846072599Z","caller":"jobframework/reconciler.go:481","msg":"Handling job with no workload","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"sample-job","namespace":"default"},"namespace":"default","name":"sample-job","reconcileID":"511efb60-60bc-44ac-8780-d371117e320c","job":"default/sample-job","gvk":"batch/v1, Kind=Job","error":"Workload.kueue.x-k8s.io \"job-sample-job-ef531\" is invalid: spec.podSets[0].template.spec.containers[0].env[1]: Duplicate value: {\"name\":\"DEMO\"}","stacktrace":"sigs.k8s.io/kueue/pkg/controller/jobframework.(*JobReconciler).ReconcileGenericJob\n\t/workspace/pkg/controller/jobframework/reconciler.go:481\nsigs.k8s.io/kueue/pkg/controller/jobframework.(*genericReconciler).Reconcile\n\t/workspace/pkg/controller/jobframework/reconciler.go:1485\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296"}
{"level":"error","ts":"2025-10-23T14:42:57.846366808Z","caller":"controller/controller.go:474","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"sample-job","namespace":"default"},"namespace":"default","name":"sample-job","reconcileID":"511efb60-60bc-44ac-8780-d371117e320c","error":"Workload.kueue.x-k8s.io \"job-sample-job-ef531\" is invalid: spec.podSets[0].template.spec.containers[0].env[1]: Duplicate value: {\"name\":\"DEMO\"}","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296"}
```

It might not be the best solution, but it could work if we remove duplicated environment variables in the mutation webhook and keep only the last one.@bd-hhause: @mbobrovskyi I'm also looking at removing the duplicate env var from the controller creating the spec. My question is: is it even Kueue's responsibility to handle this? Is the duplicate key something that should be handled by all controllers? I couldn't find any good answer on this. @mbobrovskyi: Yeah, this should also work, but I think it’s better to add it here:

https://github.com/kubernetes-sigs/kueue/blob/178155c7eadd36393cecb5da349995c946918b78/pkg/webhooks/workload_webhook.go#L60-L73@LucasZanellaMBTI: @mimowo I was off and therefore couldn't participate recent discussion, sorry for that. Does it make sense to postpone our evaluation until open points got fixed/implmented?@mimowo: > Yeah, this should also work, but I think it’s better to add it here:

Workload webhook is one possible place, but probably even better in the code which prepares the workload to be created.

The good thing about this change is also that it should work fine both with patch and SSA.

This should work I think. We may also investigate the "Restore" code to try not overriding the original Job created by the user.

>  Does it make sense to postpone our evaluation until open points got fixed/implmented?

Yes, I think this is not going to fly currently, because the Workload creation is rejected early due to the CRD validation generated for the embedded PodTemplateSpec.
@mimowo: The fix is already merged but not released yet, it will be part of 0.14.3 and 0.13.8 (tentatively planned today):
- https://github.com/kubernetes-sigs/kueue/issues/7435
- https://github.com/kubernetes-sigs/kueue/issues/7436@mwysokin: @LucasZanellaMBTI @bd-hhause I tested it today E2E with Kueue 0.14.3 and it worked correctly. I used the following yaml:

```yaml
apiVersion: batch/v1
kind: Job 
metadata:
  generateName: sleep-job-
  labels:
    kueue.x-k8s.io/queue-name: lq
spec:
  parallelism: 1
  completions: 1
  completionMode: Indexed
  template:
    spec:
      containers:
      - name: dummy-job
        image: ubuntu
        command: ["sleep"]
        args: ["600s"]
        env:
        - name: "A"
          value: "1"
        - name: "B"
          value: "1"
        - name: "A"
          value: "2"
        resources:
          requests:
            cpu: "0.1" 
          limits:
            cpu: "0.1"
      restartPolicy: Never
```

Result:

```bash
$ kubectl exec sleep-job-5qwcw-0-kglvt -- env | grep "A="
A=2
```@vladmirtxrx: Issue still affects pods with duplicate port numbers (kueue 0.14.4):
```
 ports:                                                                                                                                                                                                                                                                             
        - containerPort: 8000                                                                                                                                                                                                                                                              
          name: http                                                                                                                                                                                                                                                                       
          protocol: TCP                                                                                                                                                                                                                                                                    
        - containerPort: 8000                                                                                                                                                                                                                                                              
          name: metrics                                                                                                                                                                                                                                                                    
          protocol: TCP                                                                                                                                                                                                                                                                    
        - containerPort: 5001                                                                                                                                                                                                                                                              
          name: grpc                                                                                                                                                                                                                                                                       
          protocol: TCP          
```
```
{"level":"error","ts":"2025-11-18T11:38:24.97681862Z","caller":"jobframework/reconciler.go:481","msg":"Handling job with no workload","controller":"v1_pod","namespace":"app-cvtryout","name":"modelwgpu-dogsftriton-0-modelwgpu-5768674df6-pd56m","reconcileID":"432ca6dc-ee63-441a-92c2-efab7f2d4775","job":"app-cvtryout/modelwgpu-dogsftriton-0-modelwgpu-5768674df6-pd56m","gvk":"/v1, Kind=Pod","error":"Workload.kueue.x-k8s.io "pod-modelwgpu-dogsftriton-0-modelwgpu-5768674df6-pd56m-27c03" is invalid: spec.podSets[0].template.spec.containers[1].ports[1]: Duplicate value: map[string]interface {}{"containerPort":8000, "protocol":"TCP"}","stacktrace":"sigs.k8s.io/kueue/pkg/controller/jobframework.(*JobReconciler).ReconcileGenericJob
	/workspace/pkg/controller/jobframework/reconciler.go:481
sigs.k8s.io/kueue/pkg/controller/jobs/pod.(*Reconciler).Reconcile
	/workspace/pkg/controller/jobs/pod/pod_controller.go:117
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:461
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296"}      
```

Solution patch:
diff --git a/pkg/controller/jobframework/reconciler.go b/pkg/controller/jobframework/reconciler.go
index e1bbd1cb4..dd417e7b7 100644
--- a/pkg/controller/jobframework/reconciler.go
+++ b/pkg/controller/jobframework/reconciler.go
@@ -827,7 +827,7 @@ func (r *JobReconciler) ensureOneWorkload(ctx context.Context, job GenericJob, o
 
 	// If workload slicing is enabled for this job, use the slice-based processing path.
 	if workloadSliceEnabled(job) {
-		podSets, err := job.PodSets(ctx)
+		podSets, err := JobPodSets(ctx, job)
 		if err != nil {
 			return nil, fmt.Errorf("failed to retrieve pod sets from job: %w", err)
 		}
@@ -1068,7 +1068,7 @@ func EquivalentToWorkload(ctx context.Context, c client.Client, job GenericJob,
 		return false, nil
 	}
 
-	getPodSets, err := job.PodSets(ctx)
+	getPodSets, err := JobPodSets(ctx, job)
 	if err != nil {
 		return false, err
 	}
@@ -1215,7 +1215,7 @@ func ConstructWorkload(ctx context.Context, c client.Client, job GenericJob, lab
 	log := ctrl.LoggerFrom(ctx)
 	object := job.Object()
 
-	podSets, err := job.PodSets(ctx)
+	podSets, err := JobPodSets(ctx, job)
 	if err != nil {
 		return nil, err
 	}
diff --git a/pkg/controller/jobframework/utils.go b/pkg/controller/jobframework/utils.go
new file mode 100644
index 000000000..91497066d
--- /dev/null
+++ b/pkg/controller/jobframework/utils.go
@@ -0,0 +1,66 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package jobframework
+
+import (
+	"context"
+
+	corev1 "k8s.io/api/core/v1"
+
+	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2"
+	"sigs.k8s.io/kueue/pkg/features"
+	"sigs.k8s.io/kueue/pkg/util/orderedgroups"
+)
+
+// JobPodSets retrieves the pod sets from a GenericJob and applies environment variable
+// deduplication if the SanitizePodSets feature gate is enabled.
+func JobPodSets(ctx context.Context, job GenericJob) ([]kueue.PodSet, error) {
+	podSets, err := job.PodSets(ctx)
+	if err != nil {
+		return nil, err
+	}
+	SanitizePodSets(podSets)
+	return podSets, nil
+}
+
+// SanitizePodSets sanitizes all PodSets in the given slice by removing duplicate
+// environment variables from each container when the SanitizePodSets
+// feature is enabled. This function modifies the podSets slice in place.
+func SanitizePodSets(podSets []kueue.PodSet) {
+	for podSetIndex := range podSets {
+		SanitizePodSet(&podSets[podSetIndex])
+	}
+}
+
+// SanitizePodSet sanitizes a single PodSet by removing duplicate environment
+// variables from all containers in its pod template, but only if the
+// SanitizePodSets feature gate is enabled.
+func SanitizePodSet(podSet *kueue.PodSet) {
+	if features.Enabled(features.SanitizePodSets) {
+		for containerIndex := range podSet.Template.Spec.Containers {
+			container := &podSet.Template.Spec.Containers[containerIndex]
+			envVarGroups := orderedgroups.NewOrderedGroups[string, corev1.EnvVar]()
+			for _, envVar := range container.Env {
+				envVarGroups.Insert(envVar.Name, envVar)
+			}
+			container.Env = make([]corev1.EnvVar, 0, len(container.Env))
+			for _, envVars := range envVarGroups.InOrder {
+				container.Env = append(container.Env, envVars[len(envVars)-1])
+			}
+		}
+	}
+}
diff --git a/pkg/controller/jobs/job/job_webhook.go b/pkg/controller/jobs/job/job_webhook.go
index b437ca3c1..0b9bdee24 100644
--- a/pkg/controller/jobs/job/job_webhook.go
+++ b/pkg/controller/jobs/job/job_webhook.go
@@ -227,7 +227,7 @@ func (w *JobWebhook) validateTopologyRequest(ctx context.Context, job *Job) (fie
 		return validationErrs, nil
 	}
 
-	podSets, err := job.PodSets(ctx)
+	podSets, err := jobframework.JobPodSets(ctx, job)
 	if err != nil {
 		return nil, err
 	}
diff --git a/pkg/controller/jobs/jobset/jobset_webhook.go b/pkg/controller/jobs/jobset/jobset_webhook.go
index 0c20b90e9..483b66244 100644
--- a/pkg/controller/jobs/jobset/jobset_webhook.go
+++ b/pkg/controller/jobs/jobset/jobset_webhook.go
@@ -142,7 +142,7 @@ func (w *JobSetWebhook) validateCreate(ctx context.Context, jobSet *JobSet) (fie
 func (w *JobSetWebhook) validateTopologyRequest(ctx context.Context, jobSet *JobSet) (field.ErrorList, error) {
 	var allErrs field.ErrorList
 
-	podSets, podSetsErr := jobSet.PodSets(ctx)
+	podSets, podSetsErr := jobframework.JobPodSets(ctx, jobSet)
 
 	if podSetsErr == nil {
 		allErrs = append(allErrs, jobframework.ValidatePodSetGroupingTopology(podSets, buildPodSetAnnotationsPathByNameMap(jobSet))...)
diff --git a/pkg/controller/jobs/kubeflow/kubeflowjob/kubeflowjob_controller.go b/pkg/controller/jobs/kubeflow/kubeflowjob/kubeflowjob_controller.go
index bb7e3795d..52d3b922e 100644
--- a/pkg/controller/jobs/kubeflow/kubeflowjob/kubeflowjob_controller.go
+++ b/pkg/controller/jobs/kubeflow/kubeflowjob/kubeflowjob_controller.go
@@ -187,7 +187,7 @@ func (j *KubeflowJob) ValidateOnCreate(ctx context.Context) (field.ErrorList, er
 		return nil, nil
 	}
 
-	podSets, podSetsErr := j.PodSets(ctx)
+	podSets, podSetsErr := jobframework.JobPodSets(ctx, j)
 
 	var allErrs field.ErrorList
 	replicaTypes := j.OrderedReplicaTypes()
diff --git a/pkg/controller/jobs/leaderworkerset/leaderworkerset_reconciler.go b/pkg/controller/jobs/leaderworkerset/leaderworkerset_reconciler.go
index 615dff331..8646588fa 100644
--- a/pkg/controller/jobs/leaderworkerset/leaderworkerset_reconciler.go
+++ b/pkg/controller/jobs/leaderworkerset/leaderworkerset_reconciler.go
@@ -200,17 +200,23 @@ func (r *Reconciler) constructWorkload(lws *leaderworkersetv1.LeaderWorkerSet, w
 	return createdWorkload, nil
 }
 
+func newPodSet(name kueue.PodSetReference, count int32, template *corev1.PodTemplateSpec) kueue.PodSet {
+	podSet := kueue.PodSet{
+		Name:  name,
+		Count: count,
+		Template: corev1.PodTemplateSpec{
+			Spec: *template.Spec.DeepCopy(),
+		},
+	}
+	jobframework.SanitizePodSet(&podSet)
+	return podSet
+}
+
 func podSets(lws *leaderworkersetv1.LeaderWorkerSet) ([]kueue.PodSet, error) {
 	podSets := make([]kueue.PodSet, 0, 2)
 
 	if lws.Spec.LeaderWorkerTemplate.LeaderTemplate != nil {
-		podSet := kueue.PodSet{
-			Name:  leaderPodSetName,
-			Count: 1,
-			Template: corev1.PodTemplateSpec{
-				Spec: *lws.Spec.LeaderWorkerTemplate.LeaderTemplate.Spec.DeepCopy(),
-			},
-		}
+		podSet := newPodSet(leaderPodSetName, 1, lws.Spec.LeaderWorkerTemplate.LeaderTemplate)
 		if features.Enabled(features.TopologyAwareScheduling) {
 			topologyRequest, err := jobframework.NewPodSetTopologyRequest(
 				&lws.Spec.LeaderWorkerTemplate.LeaderTemplate.ObjectMeta).Build()
@@ -232,14 +238,7 @@ func podSets(lws *leaderworkersetv1.LeaderWorkerSet) ([]kueue.PodSet, error) {
 		defaultPodSetCount--
 	}
 
-	podSet := kueue.PodSet{
-		Name:  defaultPodSetName,
-		Count: defaultPodSetCount,
-		Template: corev1.PodTemplateSpec{
-			Spec: *lws.Spec.LeaderWorkerTemplate.WorkerTemplate.Spec.DeepCopy(),
-		},
-	}
-
+	podSet := newPodSet(defaultPodSetName, defaultPodSetCount, &lws.Spec.LeaderWorkerTemplate.WorkerTemplate)
 	if features.Enabled(features.TopologyAwareScheduling) {
 		topologyRequest, err := jobframework.NewPodSetTopologyRequest(
 			&lws.Spec.LeaderWorkerTemplate.WorkerTemplate.ObjectMeta).PodIndexLabel(
diff --git a/pkg/controller/jobs/mpijob/mpijob_webhook.go b/pkg/controller/jobs/mpijob/mpijob_webhook.go
index bfac8731e..03a09d6d0 100644
--- a/pkg/controller/jobs/mpijob/mpijob_webhook.go
+++ b/pkg/controller/jobs/mpijob/mpijob_webhook.go
@@ -155,7 +155,7 @@ func (w *MpiJobWebhook) validateCommon(ctx context.Context, mpiJob *MPIJob) (fie
 func (w *MpiJobWebhook) validateTopologyRequest(ctx context.Context, mpiJob *MPIJob) (field.ErrorList, error) {
 	var allErrs field.ErrorList
 
-	podSets, podSetsErr := mpiJob.PodSets(ctx)
+	podSets, podSetsErr := jobframework.JobPodSets(ctx, mpiJob)
 
 	if podSetsErr == nil {
 		allErrs = append(allErrs, jobframework.ValidatePodSetGroupingTopology(podSets, podSetAnnotationsPathByName)...)
diff --git a/pkg/controller/jobs/pod/pod_controller.go b/pkg/controller/jobs/pod/pod_controller.go
index 693d29610..622748917 100644
--- a/pkg/controller/jobs/pod/pod_controller.go
+++ b/pkg/controller/jobs/pod/pod_controller.go
@@ -1063,7 +1063,7 @@ func (p *Pod) ConstructComposableWorkload(ctx context.Context, c client.Client,
 		p.list.Items = activePods[:len(activePods)-excessPodsCount]
 	}
 
-	podSets, err := p.PodSets(ctx)
+	podSets, err := jobframework.JobPodSets(ctx, p)
 	if err != nil {
 		if jobframework.IsUnretryableError(err) {
 			r.Eventf(p.Object(), corev1.EventTypeWarning, jobframework.ReasonErrWorkloadCompose, err.Error())
diff --git a/pkg/controller/jobs/raycluster/raycluster_webhook.go b/pkg/controller/jobs/raycluster/raycluster_webhook.go
index e31ac1e20..607c4d776 100644
--- a/pkg/controller/jobs/raycluster/raycluster_webhook.go
+++ b/pkg/controller/jobs/raycluster/raycluster_webhook.go
@@ -196,7 +196,7 @@ func validateElasticJob(job *rayv1.RayCluster) field.ErrorList {
 func (w *RayClusterWebhook) validateTopologyRequest(ctx context.Context, rayJob *RayCluster) (field.ErrorList, error) {
 	var allErrs field.ErrorList
 
-	podSets, podSetsErr := rayJob.PodSets(ctx)
+	podSets, podSetsErr := jobframework.JobPodSets(ctx, rayJob)
 
 	allErrs = append(allErrs, jobframework.ValidateTASPodSetRequest(headGroupMetaPath, &rayJob.Spec.HeadGroupSpec.Template.ObjectMeta)...)
 
diff --git a/pkg/controller/jobs/rayjob/rayjob_webhook.go b/pkg/controller/jobs/rayjob/rayjob_webhook.go
index 98cbcbf19..dc87d0bca 100644
--- a/pkg/controller/jobs/rayjob/rayjob_webhook.go
+++ b/pkg/controller/jobs/rayjob/rayjob_webhook.go
@@ -177,7 +177,7 @@ func (w *RayJobWebhook) validateTopologyRequest(ctx context.Context, rayJob *ray
 		return allErrs, nil
 	}
 
-	podSets, podSetsErr := (*RayJob)(rayJob).PodSets(ctx)
+	podSets, podSetsErr := jobframework.JobPodSets(ctx, (*RayJob)(rayJob))
 
 	allErrs = append(allErrs, jobframework.ValidateTASPodSetRequest(headGroupMetaPath, &rayJob.Spec.RayClusterSpec.HeadGroupSpec.Template.ObjectMeta)...)
 
diff --git a/pkg/features/kube_features.go b/pkg/features/kube_features.go
index 6056a64e5..d36f55d6b 100644
--- a/pkg/features/kube_features.go
+++ b/pkg/features/kube_features.go
@@ -198,6 +198,15 @@ const (
 	//
 	// Enable all updates to Workload objects to use Patch Merge instead of Patch Apply.
 	WorkloadRequestUseMergePatch featuregate.Feature = "WorkloadRequestUseMergePatch"
+
+	// owner: @mbobrovskyi
+	//
+	// SanitizePodSets enables automatic sanitization of PodSets when creating the Workload object.
+	// The main use case it deduplication of environment variables
+	// in PodSet templates within Workload objects. When enabled, duplicate env var entries
+	// are resolved by keeping only the last occurrence, allowing workload creation to succeed
+	// even when duplicates are present in the spec.
+	SanitizePodSets featuregate.Feature = "SanitizePodSets"
 )
 
 func init() {
@@ -313,6 +322,9 @@ var defaultVersionedFeatureGates = map[featuregate.Feature]featuregate.Versioned
 	WorkloadRequestUseMergePatch: {
 		{Version: version.MustParse("0.14"), Default: false, PreRelease: featuregate.Alpha},
 	},
+	SanitizePodSets: {
+		{Version: version.MustParse("0.15"), Default: true, PreRelease: featuregate.Beta},
+	},
 }
 
 func SetFeatureGateDuringTest(tb testing.TB, f featuregate.Feature, value bool) {
diff --git a/site/content/en/docs/installation/_index.md b/site/content/en/docs/installation/_index.md
index 529662891..a7b4966c7 100644
--- a/site/content/en/docs/installation/_index.md
+++ b/site/content/en/docs/installation/_index.md
@@ -296,6 +296,7 @@ spec:
 | `ManagedJobsNamespaceSelectorAlwaysRespected` | `false` | Alpha | 0.13  |       |
 | `FlavorFungibilityImplicitPreferenceDefault`  | `false` | Alpha | 0.13  |       |
 | `WorkloadRequestUseMergePatch`                | `false` | Alpha | 0.14  |       |
+| `SanitizePodSets`                             | `true`  | Beta  | 0.15  |       |
 
 ### Feature gates for graduated or deprecated features
 
diff --git a/site/content/zh-CN/docs/installation/_index.md b/site/content/zh-CN/docs/installation/_index.md
index 8ad673b03..ffeb3341f 100644
--- a/site/content/zh-CN/docs/installation/_index.md
+++ b/site/content/zh-CN/docs/installation/_index.md
@@ -291,6 +291,7 @@ spec:
 | `ManagedJobsNamespaceSelectorAlwaysRespected` | `false` | Alpha | 0.13     |          |
 | `FlavorFungibilityImplicitPreferenceDefault`  | `false` | Alpha | 0.13     |          |
 | `WorkloadRequestUseMergePatch`                | `false` | Alpha | 0.14     |          |
+| `SanitizePodSets`                             | `true`  | Beta | 0.15     |          |
 
 ### 已毕业或已弃用特性的特性门控 {#feature-gates-for-graduated-or-deprecated-features}
 



Test patch:
diff --git a/pkg/util/testing/wrappers.go b/pkg/util/testing/wrappers.go
index 97fb37217..ef342b468 100644
--- a/pkg/util/testing/wrappers.go
+++ b/pkg/util/testing/wrappers.go
@@ -162,6 +162,12 @@ func (c *ContainerWrapper) Name(name string) *ContainerWrapper {
 	return c
 }
 
+// Image sets the image of the container.
+func (c *ContainerWrapper) Image(image string) *ContainerWrapper {
+	c.Container.Image = image
+	return c
+}
+
 // WithResourceReq appends a resource request to the container.
 func (c *ContainerWrapper) WithResourceReq(resourceName corev1.ResourceName, quantity string) *ContainerWrapper {
 	requests := utilResource.MergeResourceListKeepFirst(c.Resources.Requests, corev1.ResourceList{
@@ -182,6 +188,12 @@ func (c *ContainerWrapper) WithResourceLimit(resourceName corev1.ResourceName, q
 	return c
 }
 
+// WithEnvVar appends a env variable to the container.
+func (c *ContainerWrapper) WithEnvVar(envVar corev1.EnvVar) *ContainerWrapper {
+	c.Env = append(c.Env, envVar)
+	return c
+}
+
 // AsSidecar makes the container a sidecar when used as an Init Container.
 func (c *ContainerWrapper) AsSidecar() *ContainerWrapper {
 	c.RestartPolicy = ptr.To(corev1.ContainerRestartPolicyAlways)
diff --git a/test/e2e/singlecluster/e2e_test.go b/test/e2e/singlecluster/e2e_test.go
index 9dd1deab9..563e34e8f 100644
--- a/test/e2e/singlecluster/e2e_test.go
+++ b/test/e2e/singlecluster/e2e_test.go
@@ -31,6 +31,7 @@ import (
 
 	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2"
 	"sigs.k8s.io/kueue/pkg/controller/constants"
+	"sigs.k8s.io/kueue/pkg/controller/jobframework"
 	workloadjob "sigs.k8s.io/kueue/pkg/controller/jobs/job"
 	"sigs.k8s.io/kueue/pkg/util/slices"
 	"sigs.k8s.io/kueue/pkg/util/testing"
@@ -495,6 +496,104 @@ var _ = ginkgo.Describe("Kueue", func() {
 			})
 		})
 
+		ginkgo.It("Should deduplicate env variables", func() {
+			highPriorityClass := testing.MakePriorityClass("high").PriorityValue(100).Obj()
+			util.MustCreate(ctx, k8sClient, highPriorityClass)
+			ginkgo.DeferCleanup(func() {
+				gomega.Expect(k8sClient.Delete(ctx, highPriorityClass)).To(gomega.Succeed())
+			})
+
+			lowJob := testingjob.MakeJob("low", ns.Name).
+				Queue(kueue.LocalQueueName(localQueue.Name)).
+				Parallelism(1).
+				NodeSelector("instance-type", "on-demand").
+				Containers(
+					*testing.MakeContainer().
+						Name("c").
+						Image("sleep").
+						WithResourceReq(corev1.ResourceCPU, "1").
+						WithEnvVar(corev1.EnvVar{Name: "TEST_ENV", Value: "test1"}).
+						WithEnvVar(corev1.EnvVar{Name: "TEST_ENV", Value: "test2"}).
+						Obj(),
+				).
+				Obj()
+
+			ginkgo.By("Creating a low-priority job with duplicated environment variables", func() {
+				util.MustCreate(ctx, k8sClient, lowJob)
+			})
+
+			lowCreatedWorkload := &kueue.Workload{}
+			lowWlLookupKey := types.NamespacedName{Name: workloadjob.GetWorkloadNameForJob(lowJob.Name, lowJob.UID), Namespace: ns.Name}
+
+			ginkgo.By("Checking that the low-priority workload is created with deduplicated environment variables", func() {
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, lowWlLookupKey, lowCreatedWorkload)).Should(gomega.Succeed())
+					g.Expect(lowCreatedWorkload.Spec.PodSets).Should(gomega.HaveLen(1))
+					g.Expect(lowCreatedWorkload.Spec.PodSets[0].Template.Spec.Containers).Should(gomega.HaveLen(1))
+					g.Expect(lowCreatedWorkload.Spec.PodSets[0].Template.Spec.Containers[0].Env).Should(gomega.BeComparableTo(
+						[]corev1.EnvVar{{Name: "TEST_ENV", Value: "test2"}},
+					))
+					g.Expect(workload.IsAdmitted(lowCreatedWorkload)).Should(gomega.BeTrue())
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			highJob := testingjob.MakeJob("high", ns.Name).
+				Queue(kueue.LocalQueueName(localQueue.Name)).
+				Parallelism(1).
+				PriorityClass(highPriorityClass.Name).
+				Request(corev1.ResourceCPU, "1").
+				NodeSelector("instance-type", "on-demand").
+				Obj()
+
+			ginkgo.By("Creating a high-priority job", func() {
+				util.MustCreate(ctx, k8sClient, highJob)
+			})
+
+			highCreatedWorkload := &kueue.Workload{}
+			highWlLookupKey := types.NamespacedName{Name: workloadjob.GetWorkloadNameForJob(lowJob.Name, lowJob.UID), Namespace: ns.Name}
+
+			ginkgo.By("Checking that the high-priority workload is created and admitted", func() {
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, highWlLookupKey, highCreatedWorkload)).Should(gomega.Succeed())
+					g.Expect(workload.IsAdmitted(highCreatedWorkload)).Should(gomega.BeTrue())
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("Checking that the low-priority workload is successfully preempted", func() {
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, lowWlLookupKey, lowCreatedWorkload)).Should(gomega.Succeed())
+					g.Expect(workload.IsEvicted(lowCreatedWorkload)).Should(gomega.BeTrue())
+					g.Expect(lowCreatedWorkload.Status.Conditions).Should(testing.HaveConditionStatusTrue(kueue.WorkloadPreempted))
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("Checking that the low-priority job still has duplication of environment variables", func() {
+				createdLowJob := &batchv1.Job{}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(lowJob), createdLowJob)).Should(gomega.Succeed())
+					g.Expect(createdLowJob.Spec.Template.Spec.Containers).Should(gomega.HaveLen(1))
+					g.Expect(createdLowJob.Spec.Template.Spec.Containers[0].Env).Should(gomega.BeComparableTo(
+						[]corev1.EnvVar{
+							{Name: "TEST_ENV", Value: "test1"},
+							{Name: "TEST_ENV", Value: "test2"},
+						},
+					))
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			// Verify that the workload is not being continuously reconciled and updated
+			// due to a misbehaving EquivalentToWorkload implementation.
+			ginkgo.By("Use events to observe the workload is not updated", func() {
+				eventList := &corev1.EventList{}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.List(ctx, eventList, &client.ListOptions{Namespace: ns.Name})).To(gomega.Succeed())
+					for _, event := range eventList.Items {
+						g.Expect(event.Reason).ShouldNot(gomega.Equal(jobframework.ReasonUpdatedWorkload))
+					}
+				}, util.ConsistentDuration, util.ShortInterval).Should(gomega.Succeed())
+			})
+		})
+
 		ginkgo.It("Should not allow removing the workload's priority through the job", func() {
 			samplePriority := "sample-priority"
 			samplePriorityClass := utiltestingapi.MakeWorkloadPriorityClass(samplePriority).PriorityValue(100).Obj()