Change:
Make node affinity and tolerations mutable for Jobs. In we a new flag to allow suspending jobs to control when the Pods of a Job get created by controller-manager. We proposed this as a primitive that higher-level controllers can use to support job queuing, Allowing to mutate node affinity and tolerations allows a higher-level queue controller to influence job placement by injecting node affinity or tolerations before un-suspending a job. The implementation is basically relaxing the validation on job updates to allow for mutating the node affinity and tolerations of the pod template for jobs. /sig scheduling /sig apps /kind feature
/cc
Two questions: "nodeAffinity", I suppose it means - with volume affinity excluded. Given it's a pod field, this would allow arbitrary workload (not only Job) to mutate it, is it intentional, or you want to restrict to Job API only? mentioned it's intended for suspended Job, so are you going to limit the scope to pending Pods (with empty ) only? - mutating (on purpose or by mistake) a running Pod's nodeAffinity may cause unexpected behavior.
Yeah, I am referring to nodeAffinity constraints in the pod template of the job. It is restricted to the job api only at this point. The proposal is not to allow mutating nodeAffinity of pods, only the pod template of a suspended job. So when the job is unsuspended, the pods that will be created by the controller will already be carrying the modified constraints.
Thanks for the clarification. The limited scope is fine with me. I will leave technical feasibility and to API experts. I suppose this is based on the current semantics of "suspended job" - when a Job is suspended, it deletes all pods it already spawned. So what if the semantics change in the future to "literally only spawn new pods and keep created pods intact"?
I don't think this is an acceptable change of existing behavior of the feature; if that to happen, it will need to be done using a new API field.
Letting the scheduling portions of the pod template (node selector, node affinity, anti-affinity, tolerations) be mutable until the job is unsuspended could make sense for late-binding scheduling purposes. Could the mutation be limited only be allowed before the job is unsuspended? Ideally we could avoid this scenario: an unsuspended job starts creating pods the job the pod template the job handling does not fully delete pods created in step 2 (controller was down, or events arrived quickly and job controller sync handler was busy with other jobs) continues creating pods from new template The result of that is that you have a mix of pods for the same job with different specs/scheduling directives
Perhaps we can avoid that scenario by accepting the mutation only if the the job is suspended and the number of is 0?
does the controller set active pods before creating new pods and after deleting pods for a suspended job? if not, there's a race
I believe so, can you confirm?
Actually it seems that in both creating and deleting the status is updated after. But I think the create part isn't really related to our case, we only care about the delete case.
Yes, first the pods are created/deleted, and then the status field is updated if there were no errors. why do we need to restrict the change to suspended jobs? If the user changes the node affinity in a way that the job stops working, that's user error. But generally, the job would continue working. The pods just might start going to different zones. In fact, I can imagine a use case beyond suspended jobs: I have a long running job (high number of completions and relatively low parallelism) that ended up with affinity to certain zone. Then, they announce that the zone is going to go on maintenance. I want the new pods to start in a different zone, but I want to let the old pods finish successfully. I didn't suspend the job at any point.
if you don't restrict it to suspended jobs, in your case (updating an unsuspended job's template), isn't the running pods' spec gets updated and then kubelet may reject them? (by kubelet's regular admission check)
No, pods that have already been created aren't impacted by the update to node affinity in the template. I don't have a strong opinion whether or not we restrict this to suspended jobs.
It is related, imagine: we unsuspend the job we create the pod (but they job is not yet updated) we suspend again we change the node-affinity And we have pods with different affinities. I think that if we can prevent the race, I would be fine with it, but if not I agree with what Jordan suggested.
I updated the issue to include tolerations. I feel that both node affinity and tolerations are node related scheduling directives, while pod affinity and topology spread are more workload specific that ideally should be declared by the workload itself.
I guess it depends on the user's expectations... if we let them change scheduling bits on the pod template, will it surprise them if there are pending unschedulable pods for the job with the old scheduling bits? Makes sense to me. I'd also include nodeSelector in the set of node-related scheduling directives that should be mutable.
You can't prevent the race condition. Elaborating on scenario: we unsuspend the job the job controller creates the pod, but the job status update fails (so active remains at zero). we suspend again we change the node-affinity (note that this needs to happen in apiserver, where we wouldn't be listing pods during the update validation) I think it's better to not restrict the change to suspended jobs and be clear that existing pods are not re-created.
Aldo pointed one race. But even without failing job status there is a race. Creation of a pod and Job status update aren't synchronous, and creation of a pod happens first. So you can in theory always put: "suspend job + change node selector + unsuspend" API calls in the middle.
It's a cleaner semantic, but it might be surprising/confusing to users, I'm afraid.
I guess the only way to ensure no surprises is if we allow mutating affinity only if the job started as suspended and was never unsuspended when making the update, I don't think we have something in the job status indicating that though. But that also might be too restrictive. are we ok then with suspended+active=0 condition while acknowledging the potential race? I am also on the fence with regards to allowing it with no restrictions, but perhaps it should be fine with proper documentation emphasizing best practices on how/when to update those directives?
I personally prefer "no restrictions", than a racy one of "suspended + active=0"
I agree. I don't think we should make the spec API consumer poll for status before allowing them to set spec fields... if we want to allow scheduling fields to be mutable throughout the job lifetime, I think the controller should be made to deal with that. The clearest scenario is the progression. I don't have a good sense for how common retargeting an already running job is to know if it's worth the extra work to make the job controller handle that reliably.
Right, and that would be the best practice we could highlight in the docs. Sounds good then, I will update the KEP to remove the restriction and add node selector.
hmm, I misread your comment, it seems you still prefer we check for suspended+active=0 during validation, correct? Note that the job controller is not involved here, it should simply react to the new state we allowed into the api-server.
not quite... I think we should either: allow mutation for jobs which have never been unsuspended mutation in all cases and make the job controller handle mutated scheduling fields in the template in the least surprising way possible, and make the guaranteed behavior super clear in the API docs (I'd be surprised if I changed scheduling fields in the template and my job got stuck with one unschedulable pending pod with the old scheduling values) If we make active==0 part of the required state to be allowed to mutate these fields, it is involved (you'd have to suspend, wait for job controller to observe, delete, update status to active=0 before being allowed to mutate)
This feels too restrictive. I still want to support the case of preempting jobs and send them somewhere else. The least surprising way with guaranteed behavior is one of two options: 1) the job-controller re-creates all pods, so basically a change to the template is implicitly a suspend. If we do this, then we are not enabling the re-targeting use case while allowing existing pods to complete. 2) a slight enhancement of the one above, we only re-create pods that are not scheduled yet. ok, to achieve this the job-controller will need to mark the job as "reconciling" whenever the is not yet matched with the actual status of the job or update the status in general before creating pods, which is not going to be ideal in cases where in the same cycle the controller is deleting some pods and adding others (not common, but possible). I am torn here, it feels like an overkill to be honest. But perhaps re-creating the pods (all or only unscheduled ones) might be the best option to move forward here if the option of simply documenting the behavior is ruled out.
When I was initially thinking about it I didn't consider the race (pods with different scheduling directives) being a problem, although now I question my assumption. I guess it will depend on the particular use-case, the job's pod will do whatever it's told no matter where it lands, or in the worst case it'll fail and will be re-created (if specialized hardware was required and no such scheduling directive was initially presented, for example). I'm curious to learn how many such situations we're talking about before we invest effort into dealing with that race, which will unnecessarily complicate the controller. Is it worth the effort? Personally, I'm leaning towards best effort approach which proposed, ie. suspended + active=0 as being sufficient, assuming it will be properly documented.
At scale such races surely will manifest itself. My worry is that to prevent this race condition and guarantee that the job is suspended we will not only need to complicate the job controller, but also very likely incur at least one additional API call per job, which I would very much like to avoid for such a feature that may not be used frequently. I expect the most common use case is indeed to update affinity before a job has ever been unsuspended; note that a race would still exist even if we try implement such a constraint because it also involves checking the job status's StartTime, and so the same race conditions discussed above apply. However, I think this is a race that is less likely to occur unintentionally. So I suggest we do this in two phases: Phase 1: allow the update for suspended jobs with StartTime=nil (meaning the job was never started before), acknowledging the potential race if someone intentionally try to trigger it. Phase 2 (if there is demand): allow the update with no constraints while offering guarantees on behavior via re-creating pods (potentially re-creating only the not yet scheduled ones). This is backward compatible with phase 1. The reason I don't want to pursue phase 2 now is that the job controller has become complicated and it is due for a refactor, which will naturally happen when we graduate to GA. What do you folks think?
I like this - let's do that and figure out later if we really need more.
:+1: I totally agree with simpler approach and I'm similarly worried about additional complication of the controller. Thank you very much
/triage accepted /assign
As a followup, we also want to make annotations and labels of the pod template mutable. For better or worse, annotations are sometimes used as scheduling directives; also both labels and annotations allow higher-level job controllers to inject metadata required for managing the pods of a job after they are created. Updated KEP:

Discussion:
// owner: @ahg // beta: v1.23 <ins> // stable: v1.27 </ins> // // Allow updating node scheduling directives in the pod template of jobs. Specifically, // node affinity, selector and tolerations. This is allowed only for suspended jobs

JobPodFailurePolicy: {Default: true, PreRelease: featuregate.Beta}, <del> JobMutableNodeSchedulingDirectives: {Default: true, PreRelease: featuregate.Beta}, </del> <ins> JobMutableNodeSchedulingDirectives: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.29 </ins> JobReadyPods: {Default: true, PreRelease: featuregate.Beta},

// only for suspended jobs that never started before. suspended := oldJob.Spec.Suspend != nil && *oldJob.Spec.Suspend notStarted := oldJob.Status.StartTime == nil <del> opts.AllowMutableSchedulingDirectives = utilfeature.DefaultFeatureGate.Enabled(features.JobMutableNodeSchedulingDirectives) && suspended && notStarted </del> <ins> opts.AllowMutableSchedulingDirectives = suspended && notStarted </ins> } return opts }

} now := metav1.Now() cases := map[string]struct { <del> job                                *batch.Job update                             func(*batch.Job) wantErrs                           field.ErrorList mutableSchedulingDirectivesEnabled bool </del> <ins> job      *batch.Job update   func(*batch.Job) wantErrs field.ErrorList </ins> }{ "update parallelism": { job: &batch.Job{

wantErrs: field.ErrorList{ {Type: field.ErrorTypeInvalid, Field: "spec.template"}, }, <del> mutableSchedulingDirectivesEnabled: true, </del> }, "updating node selector for suspended but previously started job disallowed": { job: &batch.Job{

wantErrs: field.ErrorList{ {Type: field.ErrorTypeInvalid, Field: "spec.template"}, }, <del> mutableSchedulingDirectivesEnabled: true, </del> }, "updating node selector for suspended and not previously started job allowed": { job: &batch.Job{

update: func(job *batch.Job) { job.Spec.Template.Spec.NodeSelector = map[string]string{"foo": "bar"} }, <del> mutableSchedulingDirectivesEnabled: true, }, "updating node selector whilte gate disabled disallowed": { job: &batch.Job{ ObjectMeta: metav1.ObjectMeta{ Name:            "myjob", Namespace:       metav1.NamespaceDefault, ResourceVersion: "0", Annotations:     map[string]string{"foo": "bar"}, }, Spec: batch.JobSpec{ Selector:       validSelector, Template:       validPodTemplateSpec, ManualSelector: pointer.BoolPtr(true), Parallelism:    pointer.Int32Ptr(1), Suspend:        pointer.BoolPtr(true), }, }, update: func(job *batch.Job) { job.Spec.Template.Spec.NodeSelector = map[string]string{"foo": "bar"} }, wantErrs: field.ErrorList{ {Type: field.ErrorTypeInvalid, Field: "spec.template"}, }, mutableSchedulingDirectivesEnabled: false, </del> }, "invalid label selector": { job: &batch.Job{

} for name, tc := range cases { t.Run(name, func(t *testing.T) { <del> defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.JobMutableNodeSchedulingDirectives, tc.mutableSchedulingDirectivesEnabled)() </del> newJob := tc.job.DeepCopy() tc.update(newJob) errs := Strategy.ValidateUpdate(ctx, newJob, tc.job)

} func TestNodeSelectorUpdate(t *testing.T) { <del> for name, featureGate := range map[string]bool{ "feature gate disabled": false, "feature gate enabled":  true, } { t.Run(name, func(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, feature.DefaultFeatureGate, features.JobMutableNodeSchedulingDirectives, featureGate)() </del> <ins> closeFn, restConfig, clientSet, ns := setup(t, "suspend") defer closeFn() ctx, cancel := startJobControllerAndWaitForCaches(restConfig) defer cancel() </ins> <del> closeFn, restConfig, clientSet, ns := setup(t, "suspend") defer closeFn() ctx, cancel := startJobControllerAndWaitForCaches(restConfig) defer cancel() </del> <ins> job, err := createJobWithDefaults(ctx, clientSet, ns.Name, &batchv1.Job{Spec: batchv1.JobSpec{ Parallelism: pointer.Int32(1), Suspend:     pointer.BoolPtr(true), }}) if err != nil { t.Fatalf("Failed to create Job: %v", err) } jobName := job.Name jobNamespace := job.Namespace jobClient := clientSet.BatchV1().Jobs(jobNamespace) </ins> <del> job, err := createJobWithDefaults(ctx, clientSet, ns.Name, &batchv1.Job{Spec: batchv1.JobSpec{ Parallelism: pointer.Int32(1), Suspend:     pointer.BoolPtr(true), }}) if err != nil { t.Fatalf("Failed to create Job: %v", err) } jobName := job.Name jobNamespace := job.Namespace jobClient := clientSet.BatchV1().Jobs(jobNamespace) // (1) Unsuspend and set node selector in the same update. nodeSelector := map[string]string{"foo": "bar"} _, err = updateJob(ctx, jobClient, jobName, func(j *batchv1.Job) { j.Spec.Template.Spec.NodeSelector = nodeSelector j.Spec.Suspend = pointer.BoolPtr(false) }) if !featureGate { if err == nil || !strings.Contains(err.Error(), "spec.template: Invalid value") { t.Errorf("Expected "spec.template: Invalid value" error, got: %v", err) } } else if featureGate && err != nil { t.Errorf("Unexpected error: %v", err) } </del> <ins> // (1) Unsuspend and set node selector in the same update. nodeSelector := map[string]string{"foo": "bar"} if _, err := updateJob(ctx, jobClient, jobName, func(j *batchv1.Job) { j.Spec.Template.Spec.NodeSelector = nodeSelector j.Spec.Suspend = pointer.BoolPtr(false) }); err != nil { t.Errorf("Unexpected error: %v", err) } </ins> <del> // (2) Check that the pod was created using the expected node selector. if featureGate { var pod *v1.Pod if err := wait.PollImmediate(waitInterval, wait.ForeverTestTimeout, func() (bool, error) { pods, err := clientSet.CoreV1().Pods(jobNamespace).List(ctx, metav1.ListOptions{}) if err != nil { t.Fatalf("Failed to list Job Pods: %v", err) } if len(pods.Items) == 0 { return false, nil } pod = &pods.Items[0] return true, nil }); err != nil || pod == nil { t.Fatalf("pod not found: %v", err) } </del> <ins> // (2) Check that the pod was created using the expected node selector. </ins> <del> // if the feature gate is enabled, then the job should now be unsuspended and // the pod has the node selector. if diff := cmp.Diff(nodeSelector, pod.Spec.NodeSelector); diff != "" { t.Errorf("Unexpected nodeSelector (-want,+got):n%s", diff) } } </del> <ins> var pod *v1.Pod if err := wait.PollImmediate(waitInterval, wait.ForeverTestTimeout, func() (bool, error) { pods, err := clientSet.CoreV1().Pods(jobNamespace).List(ctx, metav1.ListOptions{}) if err != nil { t.Fatalf("Failed to list Job Pods: %v", err) } if len(pods.Items) == 0 { return false, nil } pod = &pods.Items[0] return true, nil }); err != nil || pod == nil { t.Fatalf("pod not found: %v", err) } </ins> <del> // (3) Update node selector again. It should fail since the job is unsuspended. _, err = updateJob(ctx, jobClient, jobName, func(j *batchv1.Job) { j.Spec.Template.Spec.NodeSelector = map[string]string{"foo": "baz"} }) </del> <ins> // if the feature gate is enabled, then the job should now be unsuspended and // the pod has the node selector. if diff := cmp.Diff(nodeSelector, pod.Spec.NodeSelector); diff != "" { t.Errorf("Unexpected nodeSelector (-want,+got):n%s", diff) } </ins> <del> if err == nil || !strings.Contains(err.Error(), "spec.template: Invalid value") { t.Errorf("Expected "spec.template: Invalid value" error, got: %v", err) } </del> <ins> // (3) Update node selector again. It should fail since the job is unsuspended. _, err = updateJob(ctx, jobClient, jobName, func(j *batchv1.Job) { j.Spec.Template.Spec.NodeSelector = map[string]string{"foo": "baz"} }) </ins> <del> }) </del> <ins> if err == nil || !strings.Contains(err.Error(), "spec.template: Invalid value") { t.Errorf("Expected "spec.template: Invalid value" error, got: %v", err) </ins> } } type podsByStatus struct {