Repository: kubernetes/kubernetes (language: go)

Issue:
Elastic Indexed Job
### What would you like to be added?

Currently we validate that `spec.completions` is set when `completionMode=Indexed` for job. I suggest we relax this validation. 

This will require the following changes:

1) We need to update the job controller to look at `spec.parallelism` instead of `spec.completions` when calculating the range of indexes.
2) When reducing parallelism, we should remove the higher indexes first (otherwise the controller will eventually do that anyways to ensure that the pods have a continuous range).

both changes only apply when `spec.completions=nil`

### Why is this needed?

The use case is jobs with workers that require a form of autoscaling, like Ray, Spark or Horovord. By modeling these as jobs with `spec.completion=nil`, `spec.parallelism` can be used to scale the job.

While such jobs could be modeled as a StatefulSet, the job API offers batch-specific features that makes it a better fit (like [retriable and non retriable failures](https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/3329-retriable-and-non-retriable-failures))

/sig apps
@soltysh @alculquicondor wdyt?


Discussion / hints:
@ahg-g: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the `triage/accepted` label and provide further guidance.

The `triage/accepted` label can be added by org members by writing `/triage accepted` in a comment.


<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
+1
The only thing to note is that if we do the controller changes in release X, then the API validation changes need to go in release X+1
/wg batch
/assign @danielvegamyhre
@ahg-g: GitHub didn't allow me to assign the following users: danielvegamyhre.

Note that only [kubernetes members](https://github.com/orgs/kubernetes/people) with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see [the contributor guide](https://git.k8s.io/community/contributors/guide/first-contribution.md#issue-assignment-in-github)

<details>

In response to [this](https://github.com/kubernetes/kubernetes/issues/114862#issuecomment-1372896041):

>/assign @danielvegamyhre


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
/assign danielvegamyhre
If we make this change, what are the criteria for considering the Job as complete?
Probably worth bringing this up to the next SIG meeting, to make sure we are not missing anything.

> If we make this change, what are the criteria for considering the Job as complete?

We should use the same behavior that NonIndexed jobs have: once one pod finishes successfully, no new pods are created.
> The Job is considered _complete_ when there is one successfully completed Pod for each index

&mdash; https://kubernetes.io/docs/concepts/workloads/controllers/job/#completion-mode

So it's a change we should communicate carefully. I think this wants, if not a KEP, then a couple of user stories so that we're agreeing on the end user experience and how that changes from today.
:thought_balloon: How about using `Indexed` to describe the completion mode we already know, and `IndexedFirstToComplete` for the new behavior? Something like that.
We are not changing existing behavior, because `completions=nil` is just disallowed during validation. Adding a new completionMode sounds like an overkill.
My worry: someone might watch for a Job and not write code to check `completions`, because they see that the mode is `"Indexed"`. Their scheduler incorrectly ignores the Job because they think `completions: null` is guaranteed by the API, based on their reading of the docs (even if we never made such a guarantee in terms of API versioning).
What can help: draft the change to https://kubernetes.io/docs/concepts/workloads/controllers/job/#completion-mode _first_, and then once we're happy that it's feasible to explain, start on the code.
> My worry: someone might watch for a Job and not write code to check completions, because they see that the mode is "Indexed". Their scheduler incorrectly ignores the Job because they think completions: null is guaranteed by the API, based on their reading of the docs (even if we never made such a guarantee in terms of API versioning).

Thanks @sftim for the feedback, relaxing validation in general is allowed. As to this specific example, the change we are proposing doesn't break this hypothetical scheduler, it will continue to operate as expected from that scheduler's perspective.

> What can help: draft the change to https://kubernetes.io/docs/concepts/workloads/controllers/job/#completion-mode first, and then once we're happy that it's feasible to explain, start on the code.

 `completions=nil` is historically a permitted mode of operation and have clear and documented semantics, and this proposal doesn't change those semantics.
But let me draft one on this issue, I will post in a moment
### Completion mode

Jobs can have the following modes of operation specified in `.spec.completionMode`:

- `NonIndexed` (default): the Pods of a Job are considered homologous of each other. When `spec.completions` is set, the Job is considered complete when there have been `.spec.completions` successfully completed Pods.
- `Indexed`: the Pods of a Job get an associated completion index from 0 to
  `$(count)-1`, where count is .spec.completions if set, .spec.parallelism if not. The index is available 
     through three mechanisms:
  - The Pod annotation `batch.kubernetes.io/job-completion-index`.
  - As part of the Pod hostname, following the pattern `$(job-name)-$(index)`.
    When you use an Indexed Job in combination with a
    {{< glossary_tooltip term_id="Service" >}}, Pods within the Job can use
    the deterministic hostnames to address each other via DNS. For more information about
    how to configure this, see [Job with Pod-to-Pod Communication](/docs/tasks/job/job-with-pod-to-pod-communication/).
  - From the containerized task, in the environment variable `JOB_COMPLETION_INDEX`.
  - When scaling `spec.parallelism` down, the higher active indexes will be removed first.
  
  When spec.completions is set, and indexed Job is considered complete when there is one successfully completed Pod
  for each index. For more information about how to use this mode, see
  [Indexed Job for Parallel Processing with Static Work Assignment](/docs/tasks/job/indexed-parallel-processing-static/).
  Note that, although rare, more than one Pod could be started for the same
  index, but only one of them will count towards the completion count.

Finally, in both modes, when `spec.completions` is not set, then the job is considered successful when at least one pod successfully completes. 

I think we need to explain that if `completions` isn't set, indexes can be reused. At least, that's what the phrase
> an associated completion index from 0 to `$(count)-1`, where count is `.spec.completions` if set, `.spec.parallelism` if not

implies.

If we didn't want to reuse the indices, then let's explain (and, by implication, consider the design details for) what happens if we create more Pods than we have parallelism.

I'd like this to have KEP: having an enhancement proposal makes it easier for SIG Release to track the work around updating the docs.
What do you mean by indexes can be reused?


In the Kubernetes blog, [Introducing Indexed Jobs](https://kubernetes.io/blog/2021/04/19/introducing-indexed-jobs/) suggests that
> each worker Pod can have a statically assigned partition of the data based on the index

and someone may have made a container image that relies on that assumption. Does this change create the risk of confusing people who do assume this?

Let's say I have a work queue that will turn out to hold 99 items, and I want to use a parallelism of 10. I don't know the size of the work queue at the time I define the Job. I set this new Job to be indexed because all my other (fixed-size) Jobs are indexed.

Some people will want to have an index that matches the ordinal of running Pods like with a StatefulSet, but other people might use the index for something like, say, interpolating into the filename of the Job's output.

If we can handle both of the cases I've sketched here, great. If not, let's document which case we're supporting and add a [callout](https://kubernetes.io/docs/contribute/style/style-guide/#caution) to alert the people who were expecting the opposite behavior.

(This is why I think it's good to draft the docs first - if we're making a design that requires warning users, we might want to reconsider it).
> and someone may have made a container image that relies on that assumption. Does this change create the risk of confusing people who do assume this?

Note that he proposed change does not impact existing workloads. We are adding new semantics, not changing existing ones.

> Some people will want to have an index that matches the ordinal of running Pods like with a StatefulSet, but other people might use the index for something like, say, interpolating into the filename of the Job's output.

The proposed semantics supports a StatefulSet like behavior, the indexes will cover the number of replicas (parallelism). The second use case is an interesting one, but it will not be enabled by the proposed change, and I don't know why one would think that if we are clearly stating that the indexes will range from 0 to spec.parallelism-1 and that success is defined as any pod finishing successfully. I will send out a KEP.

> but other people might use the index for something like, say, interpolating into the filename of the Job's output.

I haven't seen a batch API with these semantics, and to address this use case (number of indexes is not defined before hand) the queue can associate the item with an index, not the job. The work queue pattern by design aims to reuse the pods and treat them as stateless because the state is in the queue. 

Slightly related, we are thinking about adding a `completionPolicy` API to Job (e.g., the job should be declared successful if 80% of the pods successfully completes);  so, if we set `spec.completions` to some maximum number and come up with a completion policy that says "exit if the queue is empty", then we could support the case you mention via job, but I am not really convinced that this is the right approach because I believe indexed job isn't the right mode for this use case as I mentioned in the paragraph above.
> Finally, in both modes, when spec.completions is not set, then the job is considered successful when at least one pod successfully completes.

Is it more accurate to say "Finally, in both modes, when spec.completions is not set, then the job is considered successful when at least `N` pod successfully completes where N is the `spec.parallelism`"?
> Is it more accurate to say "Finally, in both modes, when spec.completions is not set, then the job is considered successful when at least N pod successfully completes where N is the spec.parallelism"?

No, that is not the current semantics of `spec.completions=nil`; as explained in  https://kubernetes.io/docs/concepts/workloads/controllers/job/#parallel-jobs: 
```
once any Pod has exited with success, no other Pod should still be doing any work for this task or writing any output. They should all be in the process of exiting.
```

The behavior you are asking about can be achieved by setting `spec.completions=spec.parallelism`
@ahg-g ,
I created a job with the following spec:

```
apiVersion: batch/v1
kind: Job
metadata:
  name: backoff-exp
spec:
  backoffLimit: 4
  parallelism: 10
  template:
    spec:
      containers:
        - name: die
          image: myubuntu:1
          command: ["/bin/sh","-c"]
          args: ["/bin/sleep `rand -M 300` && /bin/true"]
          imagePullPolicy: Never
      restartPolicy: Never
```

Each pod sleeps `random` seconds and exits successfully. In this case, the job was marked completed only after all the 10 pods finished.

#### kubectl get pods
```
NAME                READY   STATUS      RESTARTS   AGE
backoff-exp-5mb6m   0/1     Completed   0          4m41s
backoff-exp-6j76z   0/1     Completed   0          4m41s
backoff-exp-6vv88   0/1     Completed   0          4m41s
backoff-exp-7vp8c   0/1     Completed   0          4m41s
backoff-exp-gtz7n   0/1     Completed   0          4m41s
backoff-exp-q9kvt   0/1     Completed   0          4m41s
backoff-exp-vnw7m   0/1     Completed   0          4m41s
backoff-exp-wngkf   0/1     Completed   0          4m41s
backoff-exp-xhq9x   0/1     Completed   0          4m41s
backoff-exp-z4cc9   0/1     Completed   0          4m41s
``` 

#### Start and Finish time for all the above pods
```
2023-01-09T03:56:35Z | 2023-01-09T03:59:55Z
2023-01-09T03:56:35Z | 2023-01-09T03:59:55Z
2023-01-09T03:56:36Z | 2023-01-09T03:59:56Z
2023-01-09T03:56:36Z | 2023-01-09T04:01:14Z
2023-01-09T03:56:36Z | 2023-01-09T04:01:14Z
2023-01-09T03:56:36Z | 2023-01-09T04:01:14Z
2023-01-09T03:56:36Z | 2023-01-09T04:01:14Z
2023-01-09T03:56:37Z | 2023-01-09T03:59:44Z
2023-01-09T03:56:37Z | 2023-01-09T03:59:44Z
2023-01-09T03:56:37Z | 2023-01-09T03:59:44Z
```

#### Status of the completed job
```
  status:
    completionTime: "2023-01-09T04:01:17Z"
    conditions:
    - lastProbeTime: "2023-01-09T04:01:17Z"
      lastTransitionTime: "2023-01-09T04:01:17Z"
      status: "True"
      type: Complete
    ready: 0
    startTime: "2023-01-09T03:56:34Z"
    succeeded: 10
    uncountedTerminatedPods: {}
```
The job is marked complete only after the last pod has finished. Is the above behavior correct? 

Please let me know your thoughts on this!
Any active pods will not be deleted and will be allowed to run to completion (fail or success), but the job will be marked as **successful** if one pod finishes successfully. 
I have this one on today's (Jan 9th) sig-apps meeting
Meeting outcome: we will write a KEP, but there seems to be an agreement on the general direction as proposed in the initial post.
Curious to know If `spec.completions=nil` then will a failed pod will ever be retried?
> Curious to know If `spec.completions=nil` then will a failed pod will ever be retried?

Yes, just like it is today, it will be retried while respecting `backoffLimit` 
PR for Elastic Index Job docs (in progress): https://github.com/kubernetes/website/pull/39999
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle stale`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-lifecycle stale the work is underway!
/remove-lifecycle stale
hey folks! I wanted to check in on the state of this - it looks like the elastic index job docs PR was merged https://github.com/kubernetes/website/pull/39999 what comes next?
Since this was completed:
/close

What would you like to see next regarding this topic? However, we probably should open new issue(s)
@alculquicondor: Closing this issue.

<details>

In response to [this](https://github.com/kubernetes/kubernetes/issues/114862#issuecomment-1653512219):

>Since this was completed:
>/close
>
>What would you like to see next regarding this topic? However, we probably should open new issue(s)


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
> Since this was completed:

I thought that might be the case!

> What would you like to see next regarding this topic? However, we probably should open new issue(s)

We are using this for experiments to scale / reduce our Flux MiniClusters and it's working great! I don't have further comments here but will definitely open new issues or bring up discussion as needed. 

Fix patch:
diff --git a/pkg/apis/batch/validation/validation.go b/pkg/apis/batch/validation/validation.go
index e8612e022d4a3..21ecefdf96dfc 100644
--- a/pkg/apis/batch/validation/validation.go
+++ b/pkg/apis/batch/validation/validation.go
@@ -838,13 +838,7 @@ func ValidateJobTemplateSpec(spec *batch.JobTemplateSpec, fldPath *field.Path, o
 }
 
 func validateCompletions(spec, oldSpec batch.JobSpec, fldPath *field.Path, opts JobValidationOptions) field.ErrorList {
-	if !opts.AllowElasticIndexedJobs {
-		return apivalidation.ValidateImmutableField(spec.Completions, oldSpec.Completions, fldPath)
-	}
-
-	// Completions is immutable for non-indexed jobs.
-	// For Indexed Jobs, if ElasticIndexedJob feature gate is not enabled,
-	// fall back to validating that spec.Completions is always immutable.
+	// Completions is immutable for non-indexed jobs, but mutable for Indexed Jobs.
 	isIndexedJob := spec.CompletionMode != nil && *spec.CompletionMode == batch.IndexedCompletion
 	if !isIndexedJob {
 		return apivalidation.ValidateImmutableField(spec.Completions, oldSpec.Completions, fldPath)
@@ -998,8 +992,6 @@ type JobValidationOptions struct {
 	apivalidation.PodValidationOptions
 	// Allow mutable node affinity, selector and tolerations of the template
 	AllowMutableSchedulingDirectives bool
-	// Allow elastic indexed jobs
-	AllowElasticIndexedJobs bool
 	// Require Job to have the label on batch.kubernetes.io/job-name and batch.kubernetes.io/controller-uid
 	RequirePrefixedLabels bool
 }
diff --git a/pkg/features/kube_features.go b/pkg/features/kube_features.go
index b19102c297afa..9e443b915249d 100644
--- a/pkg/features/kube_features.go
+++ b/pkg/features/kube_features.go
@@ -1131,7 +1131,7 @@ var defaultKubernetesFeatureGates = map[featuregate.Feature]featuregate.FeatureS
 
 	RuntimeClassInImageCriAPI: {Default: false, PreRelease: featuregate.Alpha},
 
-	ElasticIndexedJob: {Default: true, PreRelease: featuregate.Beta},
+	ElasticIndexedJob: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // GA in 1.31, remove in 1.32
 
 	SchedulerQueueingHints: {Default: false, PreRelease: featuregate.Beta},
 
diff --git a/pkg/registry/batch/job/strategy.go b/pkg/registry/batch/job/strategy.go
index 498c4d998f32d..e39679a545a10 100644
--- a/pkg/registry/batch/job/strategy.go
+++ b/pkg/registry/batch/job/strategy.go
@@ -187,9 +187,8 @@ func validationOptionsForJob(newJob, oldJob *batch.Job) batchvalidation.JobValid
 		oldPodTemplate = &oldJob.Spec.Template
 	}
 	opts := batchvalidation.JobValidationOptions{
-		PodValidationOptions:    pod.GetValidationOptionsFromPodTemplate(newPodTemplate, oldPodTemplate),
-		AllowElasticIndexedJobs: utilfeature.DefaultFeatureGate.Enabled(features.ElasticIndexedJob),
-		RequirePrefixedLabels:   true,
+		PodValidationOptions:  pod.GetValidationOptionsFromPodTemplate(newPodTemplate, oldPodTemplate),
+		RequirePrefixedLabels: true,
 	}
 	if oldJob != nil {
 		opts.AllowInvalidLabelValueInSelector = opts.AllowInvalidLabelValueInSelector || metav1validation.LabelSelectorHasInvalidLabelValue(oldJob.Spec.Selector)


Test patch:
diff --git a/pkg/apis/batch/validation/validation_test.go b/pkg/apis/batch/validation/validation_test.go
index 9a4166c386176..aded56c5ff449 100644
--- a/pkg/apis/batch/validation/validation_test.go
+++ b/pkg/apis/batch/validation/validation_test.go
@@ -1609,22 +1609,6 @@ func TestValidateJobUpdate(t *testing.T) {
 				Field: "spec.completions",
 			},
 		},
-		"immutable completions for indexed job when AllowElasticIndexedJobs is false": {
-			old: batch.Job{
-				ObjectMeta: metav1.ObjectMeta{Name: "abc", Namespace: metav1.NamespaceDefault},
-				Spec: batch.JobSpec{
-					Selector: validGeneratedSelector,
-					Template: validPodTemplateSpecForGenerated,
-				},
-			},
-			update: func(job *batch.Job) {
-				job.Spec.Completions = pointer.Int32(1)
-			},
-			err: &field.Error{
-				Type:  field.ErrorTypeInvalid,
-				Field: "spec.completions",
-			},
-		},
 		"immutable selector": {
 			old: batch.Job{
 				ObjectMeta: metav1.ObjectMeta{Name: "abc", Namespace: metav1.NamespaceDefault},
@@ -1930,7 +1914,7 @@ func TestValidateJobUpdate(t *testing.T) {
 				Field: "spec.completionMode",
 			},
 		},
-		"immutable completions for non-indexed job when AllowElasticIndexedJobs is true": {
+		"immutable completions for non-indexed job": {
 			old: batch.Job{
 				ObjectMeta: metav1.ObjectMeta{Name: "abc", Namespace: metav1.NamespaceDefault},
 				Spec: batch.JobSpec{
@@ -1947,7 +1931,6 @@ func TestValidateJobUpdate(t *testing.T) {
 				Type:  field.ErrorTypeInvalid,
 				Field: "spec.completions",
 			},
-			opts: JobValidationOptions{AllowElasticIndexedJobs: true},
 		},
 
 		"immutable node affinity": {
@@ -2200,9 +2183,6 @@ func TestValidateJobUpdate(t *testing.T) {
 				job.Spec.Completions = pointer.Int32(2)
 				job.Spec.Parallelism = pointer.Int32(2)
 			},
-			opts: JobValidationOptions{
-				AllowElasticIndexedJobs: true,
-			},
 		},
 		"previous parallelism != previous completions, new parallelism == new completions": {
 			old: batch.Job{
@@ -2219,9 +2199,6 @@ func TestValidateJobUpdate(t *testing.T) {
 				job.Spec.Completions = pointer.Int32(3)
 				job.Spec.Parallelism = pointer.Int32(3)
 			},
-			opts: JobValidationOptions{
-				AllowElasticIndexedJobs: true,
-			},
 		},
 		"indexed job updating completions and parallelism to different values is invalid": {
 			old: batch.Job{
@@ -2238,9 +2215,6 @@ func TestValidateJobUpdate(t *testing.T) {
 				job.Spec.Completions = pointer.Int32(2)
 				job.Spec.Parallelism = pointer.Int32(3)
 			},
-			opts: JobValidationOptions{
-				AllowElasticIndexedJobs: true,
-			},
 			err: &field.Error{
 				Type:  field.ErrorTypeInvalid,
 				Field: "spec.completions",
@@ -2261,9 +2235,6 @@ func TestValidateJobUpdate(t *testing.T) {
 				job.Spec.Completions = nil
 				job.Spec.Parallelism = pointer.Int32(3)
 			},
-			opts: JobValidationOptions{
-				AllowElasticIndexedJobs: true,
-			},
 			err: &field.Error{
 				Type:  field.ErrorTypeRequired,
 				Field: "spec.completions",
@@ -2284,9 +2255,6 @@ func TestValidateJobUpdate(t *testing.T) {
 				job.Spec.Completions = pointer.Int32(2)
 				job.Spec.Parallelism = pointer.Int32(1)
 			},
-			opts: JobValidationOptions{
-				AllowElasticIndexedJobs: true,
-			},
 		},
 		"indexed job with completions unchanged, parallelism increased higher than completions": {
 			old: batch.Job{
@@ -2303,9 +2271,6 @@ func TestValidateJobUpdate(t *testing.T) {
 				job.Spec.Completions = pointer.Int32(2)
 				job.Spec.Parallelism = pointer.Int32(3)
 			},
-			opts: JobValidationOptions{
-				AllowElasticIndexedJobs: true,
-			},
 		},
 	}
 	ignoreValueAndDetail := cmpopts.IgnoreFields(field.Error{}, "BadValue", "Detail")
diff --git a/test/integration/job/job_test.go b/test/integration/job/job_test.go
index 427e62b466c34..bb692c577bcb5 100644
--- a/test/integration/job/job_test.go
+++ b/test/integration/job/job_test.go
@@ -34,10 +34,8 @@ import (
 	eventsv1 "k8s.io/api/events/v1"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/sets"
-	"k8s.io/apimachinery/pkg/util/validation/field"
 	"k8s.io/apimachinery/pkg/util/wait"
 	"k8s.io/apimachinery/pkg/watch"
 	"k8s.io/apiserver/pkg/util/feature"
@@ -2756,25 +2754,10 @@ func TestElasticIndexedJob(t *testing.T) {
 		wantTerminating      *int32
 	}
 	cases := map[string]struct {
-		featureGate bool
-		jobUpdates  []jobUpdate
-		wantErr     *apierrors.StatusError
+		jobUpdates []jobUpdate
+		wantErr    *apierrors.StatusError
 	}{
-		"feature flag off, mutation not allowed": {
-			jobUpdates: []jobUpdate{
-				{
-					completions:     ptr.To[int32](4),
-					wantTerminating: ptr.To[int32](0),
-				},
-			},
-			wantErr: apierrors.NewInvalid(
-				schema.GroupKind{Group: "batch", Kind: "Job"},
-				"test-job",
-				field.ErrorList{field.Invalid(field.NewPath("spec", "completions"), 4, "field is immutable")},
-			),
-		},
 		"scale up": {
-			featureGate: true,
 			jobUpdates: []jobUpdate{
 				{
 					// Scale up completions 3->4 then succeed indexes 0-3
@@ -2786,7 +2769,6 @@ func TestElasticIndexedJob(t *testing.T) {
 			},
 		},
 		"scale down": {
-			featureGate: true,
 			jobUpdates: []jobUpdate{
 				// First succeed index 1 and fail index 2 while completions is still original value (3).
 				{
@@ -2810,7 +2792,6 @@ func TestElasticIndexedJob(t *testing.T) {
 			},
 		},
 		"index finishes successfully, scale down, scale up": {
-			featureGate: true,
 			jobUpdates: []jobUpdate{
 				// First succeed index 2 while completions is still original value (3).
 				{
@@ -2837,7 +2818,6 @@ func TestElasticIndexedJob(t *testing.T) {
 			},
 		},
 		"scale down to 0, verify that the job succeeds": {
-			featureGate: true,
 			jobUpdates: []jobUpdate{
 				{
 					completions:     ptr.To[int32](0),
@@ -2850,7 +2830,6 @@ func TestElasticIndexedJob(t *testing.T) {
 	for name, tc := range cases {
 		tc := tc
 		t.Run(name, func(t *testing.T) {
-			featuregatetesting.SetFeatureGateDuringTest(t, feature.DefaultFeatureGate, features.ElasticIndexedJob, tc.featureGate)
 			closeFn, restConfig, clientSet, ns := setup(t, "indexed")
 			defer closeFn()
 			ctx, cancel := startJobControllerAndWaitForCaches(t, restConfig)