Repository: kubernetes-sigs/kueue (Go)

Issue:
A mechanism to suspend a running Job without requeueing
<!-- Please only use this template for submitting enhancement requests -->

**What would you like to be added**:

A field in the Workload API that would allow an administrator to permanently suspend a running job (no requeueing). The operation can be reversed by the administrator.

**Why is this needed**:

Some systems need the ability to "terminate" arbitrary jobs (because it's running for too long or other policies) without deleting the Job object, so it can be debugged.

**Completion requirements**:

This enhancement requires the following artifacts:

- [x] Design doc
- [x] API change
- [x] Docs update

The artifacts should be linked in subsequent comments.



Discussion: I think we can keep it generic, suspend/unsuspend workload. This is more useful that permanently suspend the job and terminating it.
updated the comment.
**Automatically**(judged by running time etc.) or **Manually**, what do you expect here?
Manually, by an administrator (or another controller)
I am curious about the load on the API server then job logs get very large, do we know any performance implications and if the job logs get rolled over after a certain size?
IIUC, log rotation is independent of the existence of the job object.

Note that this feature would be opt-in. Administrators should use it at their own risk, probably having some form of garbage collection in place.
/assign

I'd like to work on this :) 
There is a open question about permissions:
Should the job owner have the ability to suspend a job, or just the administrator?
One approach I've been thinking about is to add a `requeuePolicy` field to wworkload spec, with initial values of `Always` (default) or `Never`. This seems slightly cleaner than a field to directly suspend the job since we're not introducing new mechanisms to suspend workloads. Thoughts?
This reminds me when designing the `Suspend`, maybe it should be string rather than boolean for expandability, then we can support like `pause` or something like that.

> Should the job owner have the ability to suspend a job, or just the administrator?

If we focus on the job itself, like debugging, I doubt that the administrator will dig oneself.
> One approach I've been thinking about is to add a `requeuePolicy` field to wworkload spec, with initial values of `Always` (default) or `Never`. This seems slightly cleaner than a field to directly suspend the job since we're not introducing new mechanisms to suspend workloads. Thoughts?

Make sense to me, somehow likes the restartPolicy. But how to populate the field `Never` to workload, it's spawned, another annotation?
> But how to populate the field Never to workload, it's spawned, another annotation?

I might be misunderstanding your question. I was thinking with a new field `workload.spec.requeuePolicy`, you can set the value to `Never` even while the workload is running. It wouldn't suspend the workload itself, but next time you suspend the workload it will ensure it won't be requeued. So taking Job API as an example, it would take 2 steps to suspend a running Job:
1. Set workload.spec.requeuePolicy = true
2. Set job.spec.suspend = true 
I'm not sure this API is unambiguous.

Imagine that it is possible to create a Workload with requeuePolicy=Never. What does that mean? Should the Workload not be queued and admitted in this case?

Another sceneario: the workload just got admitted, and an administrator manages to set requeuePolicy=Never right before the controller sets suspend=false. What should happen to the job?

Maybe this can be fixed by changing the steps to suspend a running job:
- set the requeuePolicy in Workload to Never
- set the evicted condition in the Workload status

maybe requeuePolicy is not the best name, as it doesn't explain what happens when the Workload was never admitted. Maybe just `queueing: Enabled|Disabled`?
> Imagine that it is possible to create a Workload with requeuePolicy=Never. What does that mean? Should the Workload not be queued and admitted in this case?

> Another sceneario: the workload just got admitted, and an administrator manages to set requeuePolicy=Never right before the controller sets suspend=false. What should happen to the job?

We could fix this with a 3rd policy, `AdmitOnce` or similar. But I don't have enough experience here to say this is a valid use-case.

> maybe requeuePolicy is not the best name, as it doesn't explain what happens when the Workload was never admitted. Maybe just queueing: Enabled|Disabled?

This seems fine to me, but we can probably just make it a bool instead then? Agree the corner cases around requeuePolicy can be misleading with jobs that were never previously admitted. Will start with `queueing` (or similar) and iterate from there. 
a boolean should be fine. I think it's clear that this will always be only 2 modes.
> I might be misunderstanding your question.

What I mean is manually set the `Never` sounds dumb, but if we want it automated, since workload is spawned by job, then we should add a knob to the job. But if this is only for debugging, then there's no problem and we can enhance this if necessary in the future, it's easy.


> a boolean should be fine. I think it's clear that this will always be only 2 modes.

If we want to make the requeueing automatically oneday as I mentioned above, then **queueing** is not a good choice because the workload will never enter into the scheduling queue semantically.

With `requeuePolicy`, we can achieve the same goal but still retain the extensibilities, like requeue for once or several times as we do for `restartPolicy`, yes, **this is not demanded** right now, but maybe possible in the future. 🤔
That would mean that `requeuePolicy: Never` implies that we need to queue at least once. So we need to keep a record of that. Not sure if it's worth the complexity.
> That would mean that requeuePolicy: Never implies that we need to queue at least once. So we need to keep a record of that. Not sure if it's worth the complexity.

I agree that `requeuePolicy: Never` can be misleading because the term "requeue" implies it is still admitted once and the policy only applies on "requeue" (after the first queue).

What about `queuingPolicy: AdmitAlways|AdmitOnce|Suspend`? I agree with @kerthcet on the extensibility is better since there _could_ be more than 2 modes down the road.

> So we need to keep a record of that.

For the "AdmitOnce" case can we just check the status of the workload to check if it previously ran or is there limitations to this approach?
As a side note ... _if_ we did add a `queuingPolicy` or `requeuePolicy` field, would it make sense to allow defaulting this policy at the LocalQueue or ClusterQueue level? 
> What about `queuingPolicy: AdmitAlways|AdmitOnce|Suspend`? I agree with Kante Yin on the extensibility is better since there _could_ be more than 2 modes down the road.

But only implement `AdmitAlways | Suspend` for now?

> For the "AdmitOnce" case can we just check the status of the workload to check if it previously ran or is there limitations to this approach?

We don't keep that information. There might be some hints, like the `Evicted` condition, or the fact that the `Admitted` condition has a transition timestamp much older than the creationTimestamp.
> But only implement AdmitAlways | Suspend for now?

Sure we can defer the `AdmitOnce` policy since we don't even have a mechanism for tracking this state as you mentioned
By the way, in what condition job will requeue as it's condition is running, if it runs too long, we'll mark it as failed.
The job would only be requeued if it's ever suspended, not if it's marked as Failed (for example, if it reaches the ActiveDeadline)



Fix (patch):
diff --git a/apis/kueue/v1beta1/workload_types.go b/apis/kueue/v1beta1/workload_types.go
index 798cb55b5c3..fca411c02de 100644
--- a/apis/kueue/v1beta1/workload_types.go
+++ b/apis/kueue/v1beta1/workload_types.go
@@ -59,6 +59,17 @@ type WorkloadSpec struct {
 	// +kubebuilder:default=""
 	// +kubebuilder:validation:Enum=kueue.x-k8s.io/workloadpriorityclass;scheduling.k8s.io/priorityclass;""
 	PriorityClassSource string `json:"priorityClassSource,omitempty"`
+
+	// Active determines if a workload can be admitted into a queue.
+	// Changing active from true to false will evict any running workloads.
+	// Possible values are:
+	//
+	//   - false: indicates that a workload should never be admitted and evicts running workloads
+	//   - true: indicates that a workload can be evaluated for admission into it's respective queue.
+	//
+	// Defaults to true
+	// +kubebuilder:default=true
+	Active *bool `json:"active,omitempty"`
 }
 
 type Admission struct {
@@ -267,6 +278,10 @@ const (
 	// WorkloadEvictedByAdmissionCheck indicates that the workload was evicted
 	// beacuse at least one admission check transitioned to False.
 	WorkloadEvictedByAdmissionCheck = "AdmissionCheck"
+
+	// WorkloadEvictedByDeactivation indicates that the workload was evicted
+	// because spec.active is set to false.
+	WorkloadEvictedByDeactivation = "InactiveWorkload"
 )
 
 // +genclient
diff --git a/apis/kueue/v1beta1/zz_generated.deepcopy.go b/apis/kueue/v1beta1/zz_generated.deepcopy.go
index 18eb9579fdb..a0ad6a529a1 100644
--- a/apis/kueue/v1beta1/zz_generated.deepcopy.go
+++ b/apis/kueue/v1beta1/zz_generated.deepcopy.go
@@ -1081,6 +1081,11 @@ func (in *WorkloadSpec) DeepCopyInto(out *WorkloadSpec) {
 		*out = new(int32)
 		**out = **in
 	}
+	if in.Active != nil {
+		in, out := &in.Active, &out.Active
+		*out = new(bool)
+		**out = **in
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new WorkloadSpec.
diff --git a/charts/kueue/templates/crd/kueue.x-k8s.io_workloads.yaml b/charts/kueue/templates/crd/kueue.x-k8s.io_workloads.yaml
index 158f697309f..25f791af833 100644
--- a/charts/kueue/templates/crd/kueue.x-k8s.io_workloads.yaml
+++ b/charts/kueue/templates/crd/kueue.x-k8s.io_workloads.yaml
@@ -62,6 +62,15 @@ spec:
           spec:
             description: WorkloadSpec defines the desired state of Workload
             properties:
+              active:
+                default: true
+                description: "Active determines if a workload can be admitted into
+                  a queue. Changing active from true to false will evict any running
+                  workloads. Possible values are: \n - false: indicates that a workload
+                  should never be admitted and evicts running workloads - true: indicates
+                  that a workload can be evaluated for admission into it's respective
+                  queue. \n Defaults to true"
+                type: boolean
               podSets:
                 description: podSets is a list of sets of homogeneous pods, each described
                   by a Pod spec and a count. There must be at least one element and
diff --git a/client-go/applyconfiguration/kueue/v1beta1/workloadspec.go b/client-go/applyconfiguration/kueue/v1beta1/workloadspec.go
index 04a855b67c7..685da6adc46 100644
--- a/client-go/applyconfiguration/kueue/v1beta1/workloadspec.go
+++ b/client-go/applyconfiguration/kueue/v1beta1/workloadspec.go
@@ -25,6 +25,7 @@ type WorkloadSpecApplyConfiguration struct {
 	PriorityClassName   *string                    `json:"priorityClassName,omitempty"`
 	Priority            *int32                     `json:"priority,omitempty"`
 	PriorityClassSource *string                    `json:"priorityClassSource,omitempty"`
+	Active              *bool                      `json:"active,omitempty"`
 }
 
 // WorkloadSpecApplyConfiguration constructs an declarative configuration of the WorkloadSpec type for use with
@@ -77,3 +78,11 @@ func (b *WorkloadSpecApplyConfiguration) WithPriorityClassSource(value string) *
 	b.PriorityClassSource = &value
 	return b
 }
+
+// WithActive sets the Active field in the declarative configuration to the given value
+// and returns the receiver, so that objects can be built by chaining "With" function invocations.
+// If called multiple times, the Active field is set to the value of the last call.
+func (b *WorkloadSpecApplyConfiguration) WithActive(value bool) *WorkloadSpecApplyConfiguration {
+	b.Active = &value
+	return b
+}
diff --git a/config/components/crd/bases/kueue.x-k8s.io_workloads.yaml b/config/components/crd/bases/kueue.x-k8s.io_workloads.yaml
index ba84762856a..0d8e04cbebc 100644
--- a/config/components/crd/bases/kueue.x-k8s.io_workloads.yaml
+++ b/config/components/crd/bases/kueue.x-k8s.io_workloads.yaml
@@ -49,6 +49,15 @@ spec:
           spec:
             description: WorkloadSpec defines the desired state of Workload
             properties:
+              active:
+                default: true
+                description: "Active determines if a workload can be admitted into
+                  a queue. Changing active from true to false will evict any running
+                  workloads. Possible values are: \n - false: indicates that a workload
+                  should never be admitted and evicts running workloads - true: indicates
+                  that a workload can be evaluated for admission into it's respective
+                  queue. \n Defaults to true"
+                type: boolean
               podSets:
                 description: podSets is a list of sets of homogeneous pods, each described
                   by a Pod spec and a count. There must be at least one element and
diff --git a/pkg/controller/core/workload_controller.go b/pkg/controller/core/workload_controller.go
index 0ea87583ee3..949d7d663a2 100644
--- a/pkg/controller/core/workload_controller.go
+++ b/pkg/controller/core/workload_controller.go
@@ -32,6 +32,7 @@ import (
 	"k8s.io/client-go/util/workqueue"
 	"k8s.io/klog/v2"
 	"k8s.io/utils/clock"
+	"k8s.io/utils/ptr"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/event"
@@ -376,6 +377,7 @@ func (r *WorkloadReconciler) Update(e event.UpdateEvent) bool {
 	status := workloadStatus(wl)
 	log := r.log.WithValues("workload", klog.KObj(wl), "queue", wl.Spec.QueueName, "status", status)
 	ctx := ctrl.LoggerInto(context.Background(), log)
+	active := ptr.Deref(wl.Spec.Active, true)
 
 	prevQueue := oldWl.Spec.QueueName
 	if prevQueue != wl.Spec.QueueName {
@@ -398,7 +400,10 @@ func (r *WorkloadReconciler) Update(e event.UpdateEvent) bool {
 	workload.AdjustResources(ctrl.LoggerInto(ctx, log), r.client, wlCopy)
 
 	switch {
-	case status == finished:
+	case status == finished || !active:
+		if !active {
+			log.V(2).Info("Workload will not be queued because the workload is not active", "workload", klog.KObj(wl))
+		}
 		// The workload could have been in the queues if we missed an event.
 		r.queues.DeleteWorkload(wl)
 
diff --git a/pkg/controller/jobframework/reconciler.go b/pkg/controller/jobframework/reconciler.go
index c1ac22312b2..4a6e71d3082 100644
--- a/pkg/controller/jobframework/reconciler.go
+++ b/pkg/controller/jobframework/reconciler.go
@@ -363,7 +363,17 @@ func (r *JobReconciler) ReconcileGenericJob(ctx context.Context, req ctrl.Reques
 		return ctrl.Result{}, nil
 	}
 
-	// 8. handle job is unsuspended.
+	// 8. handle workload is deactivated.
+	if !ptr.Deref(wl.Spec.Active, true) {
+		workload.SetEvictedCondition(wl, kueue.WorkloadEvictedByDeactivation, "The workload is deactivated")
+		err := workload.ApplyAdmissionStatus(ctx, r.client, wl, true)
+		if err != nil {
+			return ctrl.Result{}, fmt.Errorf("setting eviction: %w", err)
+		}
+		return ctrl.Result{}, nil
+	}
+
+	// 9. handle job is unsuspended.
 	if !workload.IsAdmitted(wl) {
 		// the job must be suspended if the workload is not yet admitted.
 		log.V(2).Info("Running job is not admitted by a cluster queue, suspending")