Repository: kubernetes-sigs/kueue

Issue:
Promote the LocalQueueDefaulting feature to Beta in 0.12
<!-- Please only use this template for submitting enhancement requests -->

**What would you like to be added**:

The LocalQueueDefaulting feature was introduced at the Alpha level in 0.10.  
I would like to promote to Beta and enable by default in 0.12.

**Why is this needed**:

Promoting to Beta and enabling by default is the best way to get user feedback on the feature.

The feature provides a simpler and potentially less surprising mechanism than 
`manageJobsWithoutQueueName` for allowing batch admins to enable quota enforcement in
selected user-namespaces.  As such, I expect there would be user demand for the feature.

**Completion requirements**:

This enhancement requires the following artifacts:

- [ ] Design doc
- [ ] API change
- [ ] Docs update

The artifacts should be linked in subsequent comments.


Discussion:
@dgrove-oss: I think this should be fairly straightforward to enable.

If we agree this should be done, I think I can find time to make the code changes before a 0.12 release in mid-May.@mimowo: Awesome. We need some e2e tests. cc @tenzen-y @mbobrovskyi @tenzen-y: SGTM@dgrove-oss: /assign@MaysaMacedo: Hello @dgrove-oss, do you mind if I contribute to this issue?
Thanks in advance.@dgrove-oss: Hi @MaysaMacedo that would be great; I will assign to you.  Happy to answer questions if you have any along the way.
/assign @MaysaMacedo @kannon92: As we enable this for beta which I assume means feature gate on, I think we should consider a configuration for this. I think as we promote this to beta, we will probably need a configuration to opt in or opt out of this feature.

The configuration could be a simple configuration for LocalQueueDefault and if it’s empty, then we assume feature is disabled. If it’s specified then we can assume localQueueDefaulting is set.

I also think we can maybe validate that LocalQueurDefsulting and ManagedJobWithoitQueueName cannot both be set.@dgrove-oss: > As we enable this for beta which I assume means feature gate on, I think we should consider a configuration for this. I think as we promote this to beta, we will probably need a configuration to opt in or opt out of this feature.
> 

I'm not convinced we need this.  A batch-admin opts into LocalQueueDefaulting on a namespace-by-namespace basis by creating a local queue named `default` in a namespace.  We need to document this clearly and highlight in the release note, but I don't see the need for adding an operator-level config.

> I also think we can maybe validate that LocalQueurDefsulting and ManagedJobWithoitQueueName cannot both be set.

These can interoperate just fine.  If a namespace has a localqueue named `default` then LocalQueueDefaulting kicks in.  If a namespace doesn't have such a local queue then `ManangeJobsWithoutQueueName` can come into play (if the ManagedJobsNamespaceSelector matches).  It's a bit of an odd configuration choice, but it is not invalid.@kannon92: I see @dgrove-oss. Yea I think documentation may be sufficient here.

Solution patch:
diff --git a/pkg/features/kube_features.go b/pkg/features/kube_features.go
index 2c9ca6c1b..ad9821c24 100644
--- a/pkg/features/kube_features.go
+++ b/pkg/features/kube_features.go
@@ -230,6 +230,7 @@ var defaultVersionedFeatureGates = map[featuregate.Feature]featuregate.Versioned
 	},
 	LocalQueueDefaulting: {
 		{Version: version.MustParse("0.10"), Default: false, PreRelease: featuregate.Alpha},
+		{Version: version.MustParse("0.12"), Default: true, PreRelease: featuregate.Beta},
 	},
 	TASProfileMostFreeCapacity: {
 		{Version: version.MustParse("0.11"), Default: false, PreRelease: featuregate.Deprecated},
diff --git a/site/content/en/docs/installation/_index.md b/site/content/en/docs/installation/_index.md
index 07657d2d5..0b5cfdbf6 100644
--- a/site/content/en/docs/installation/_index.md
+++ b/site/content/en/docs/installation/_index.md
@@ -285,7 +285,8 @@ spec:
 | `WorkloadResourceRequestsSummary`     | `false` | Alpha      | 0.9   | 0.9   |
 | `WorkloadResourceRequestsSummary`     | `true`  | Beta       | 0.10  | 0.10  |
 | `ManagedJobsNamespaceSelector`        | `true`  | Beta       | 0.10  |       |
-| `LocalQueueDefaulting`                | `false` | Alpha      | 0.10  |       |
+| `LocalQueueDefaulting`                | `false` | Alpha      | 0.10  | 0.11  |
+| `LocalQueueDefaulting`                | `true`  | Beta       | 0.12  |       |
 | `LocalQueueMetrics`                   | `false` | Alpha      | 0.10  |       |
 
 ### Feature gates for graduated or deprecated features
diff --git a/site/content/en/docs/tasks/manage/enforce_job_management/setup_default_local_queue.md b/site/content/en/docs/tasks/manage/enforce_job_management/setup_default_local_queue.md
index c4eb49912..b1d8aa20e 100644
--- a/site/content/en/docs/tasks/manage/enforce_job_management/setup_default_local_queue.md
+++ b/site/content/en/docs/tasks/manage/enforce_job_management/setup_default_local_queue.md
@@ -6,19 +6,24 @@ description: >
   Setup default LocalQueue to fullfil a queue label on jobs that submited without queue label.
 ---
 
-{{< feature-state state="alpha" for_version="v0.10" >}}
+{{< feature-state state="beta" for_version="v0.12" >}}
+
+{{% alert title="Note" color="primary" %}}
+
+`LocalQueueDefaulting` is a Beta feature that is enabled by default.
+
+You can disable it by setting the `LocalQueueDefaulting` feature gate. Check the [Installation](/docs/installation/#change-the-feature-gates-configuration) guide for details on feature gate configuration.
+{{% /alert %}}
 
 This page describes how to setup default LocalQueue to ensure that all Workloads submitted to a specific namespace are managed by Kueue,
 even if the `kueue.x-k8s.io/queue-name` label is not specified explicitly.
 
 ## Setup default LocalQueue
 
-LocalQueueDefaulting is an Alpha feature that allows the use of a LocalQueue with name `default` as the default LocalQueue
+LocalQueueDefaulting is an Beta feature that allows the use of a LocalQueue with name `default` as the default LocalQueue
 for workloads in the same namespace that do not have the `kueue.x-k8s.io/queue-name` label.
-The feature is gated by the `LocalQueueDefaulting` feature gate, and is disabled by default. To use this feature:
+The feature is gated by the `LocalQueueDefaulting` feature gate, and is enabled by default. To use this feature:
 
-- Enable the LocalQueueDefaulting feature gate. Refer to the [feature gates configuration](/docs/installation/#change-the-feature-gates-configuration)
-guide for details.
 - create a LocalQueue with the name `default` in a namespace.
 
 That's all! Now, to test the feature, create a Job in the same namespace. Observe that the Job is updated with the `kueue.x-k8s.io/queue-name: default` label.



Test patch:
diff --git a/test/e2e/config/default/controller_manager_config.yaml b/test/e2e/config/default/controller_manager_config.yaml
index 8295a62a7..bf08f2d89 100644
--- a/test/e2e/config/default/controller_manager_config.yaml
+++ b/test/e2e/config/default/controller_manager_config.yaml
@@ -33,3 +33,7 @@ integrations:
   - "deployment"
   - "statefulset"
   - "leaderworkerset.x-k8s.io/leaderworkerset"
+featureGates:
+  MultiKueueBatchJobWithManagedBy: true
+  TopologyAwareScheduling: true
+  LocalQueueMetrics: true
diff --git a/test/e2e/config/default/manager_e2e_patch.yaml b/test/e2e/config/default/manager_e2e_patch.yaml
index 007f65bcc..8cdf47c02 100644
--- a/test/e2e/config/default/manager_e2e_patch.yaml
+++ b/test/e2e/config/default/manager_e2e_patch.yaml
@@ -1,6 +1,3 @@
 - op: replace
   path: /spec/template/spec/containers/0/imagePullPolicy
   value: IfNotPresent
-- op: add
-  path: /spec/template/spec/containers/0/args/-
-  value: --feature-gates=MultiKueueBatchJobWithManagedBy=true,TopologyAwareScheduling=true,LocalQueueMetrics=true
diff --git a/test/e2e/customconfigs/managejobswithoutqueuename_test.go b/test/e2e/customconfigs/managejobswithoutqueuename_test.go
index ca8ad7206..1c6e06ba2 100644
--- a/test/e2e/customconfigs/managejobswithoutqueuename_test.go
+++ b/test/e2e/customconfigs/managejobswithoutqueuename_test.go
@@ -84,6 +84,103 @@ var _ = ginkgo.Describe("ManageJobsWithoutQueueName", ginkgo.Ordered, func() {
 		util.ExpectObjectToBeDeletedWithTimeout(ctx, k8sClient, defaultRf, true, util.LongTimeout)
 		util.ExpectAllPodsInNamespaceDeleted(ctx, k8sClient, ns)
 	})
+
+	ginkgo.When("manageJobsWithoutQueueName=true and LocalQueueDefaulting=false", func() {
+		ginkgo.It("should suspend a job", func() {
+			var testJob, createdJob *batchv1.Job
+			var jobLookupKey types.NamespacedName
+
+			ginkgo.By("setting local queue defulting as false", func() {
+				updateKueueConfiguration(func(cfg *config.Configuration) {
+					cfg.FeatureGates = map[string]bool{"LocalQueueDefaulting": false}
+					cfg.ManageJobsWithoutQueueName = true
+				})
+			})
+
+			ginkgo.By("creating a default LocalQueue", func() {
+				localQueue = testing.MakeLocalQueue("default", ns.Name).ClusterQueue("cluster-queue").Obj()
+				util.MustCreate(ctx, k8sClient, localQueue)
+			})
+
+			ginkgo.By("creating an unsuspended job without a queue name", func() {
+				testJob = testingjob.MakeJob("test-job", ns.Name).
+					Suspend(false).
+					Image(util.E2eTestAgnHostImage, util.BehaviorExitFast).
+					Obj()
+				util.MustCreate(ctx, k8sClient, testJob)
+			})
+
+			ginkgo.By("verifying that the job does get suspended", func() {
+				createdJob = &batchv1.Job{}
+				jobLookupKey = types.NamespacedName{Name: testJob.Name, Namespace: ns.Name}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, jobLookupKey, createdJob)).Should(gomega.Succeed())
+					g.Expect(ptr.Deref(createdJob.Spec.Suspend, false)).To(gomega.BeTrue())
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("verifying no existence of kueue labels", func() {
+				jobLookupKey = types.NamespacedName{Name: testJob.Name, Namespace: ns.Name}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, jobLookupKey, createdJob)).Should(gomega.Succeed())
+					g.Expect(createdJob.Labels).ShouldNot(gomega.HaveKeyWithValue(controllerconstants.QueueLabel, "default"))
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+			ginkgo.By("setting feature gates back to its original state", func() {
+				updateKueueConfiguration(func(cfg *config.Configuration) {
+					cfg.ManageJobsWithoutQueueName = true
+				})
+			})
+		})
+	})
+
+	ginkgo.When("manageJobsWithoutQueueName=true and LocalQueueDefaulting=true", func() {
+		ginkgo.It("should not suspend a job", func() {
+			var testJob, createdJob *batchv1.Job
+			var jobLookupKey types.NamespacedName
+
+			ginkgo.By("creating a default LocalQueue", func() {
+				localQueue = testing.MakeLocalQueue("default", ns.Name).ClusterQueue("cluster-queue").Obj()
+				util.MustCreate(ctx, k8sClient, localQueue)
+			})
+
+			ginkgo.By("creating an unsuspended job without a queue name", func() {
+				testJob = testingjob.MakeJob("test-job", ns.Name).
+					Suspend(false).
+					Image(util.E2eTestAgnHostImage, util.BehaviorExitFast).
+					Obj()
+				util.MustCreate(ctx, k8sClient, testJob)
+			})
+
+			ginkgo.By("verifying that the job does not get suspended", func() {
+				createdJob = &batchv1.Job{}
+				jobLookupKey = types.NamespacedName{Name: testJob.Name, Namespace: ns.Name}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, jobLookupKey, createdJob)).Should(gomega.Succeed())
+					g.Expect(ptr.Deref(createdJob.Spec.Suspend, false)).To(gomega.BeFalse())
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("verifying existence of kueue labels", func() {
+				jobLookupKey = types.NamespacedName{Name: testJob.Name, Namespace: ns.Name}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, jobLookupKey, createdJob)).Should(gomega.Succeed())
+					g.Expect(createdJob.Labels).Should(gomega.HaveKeyWithValue(controllerconstants.QueueLabel, "default"))
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("verifying that the job has been admitted", func() {
+				wlLookupKey := types.NamespacedName{Name: workloadjob.GetWorkloadNameForJob(testJob.Name, testJob.UID), Namespace: ns.Name}
+				createdWorkload := &kueue.Workload{}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, wlLookupKey, createdWorkload)).Should(gomega.Succeed())
+					g.Expect(createdWorkload.Status.Admission).ShouldNot(gomega.BeNil())
+					util.ExpectWorkloadsToBeAdmitted(ctx, k8sClient, createdWorkload)
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+		})
+	})
+
 	ginkgo.When("manageJobsWithoutQueueName=true", func() {
 		ginkgo.It("should suspend a job", func() {
 			var testJob, createdJob *batchv1.Job