Repository: kubernetes-sigs/kueue
Pull request #2742: KEP 1618: Optional garbage collection of finished Workloads
Author: login: mwysokin
Labels: name: kind/feature
description: Categorizes issue or PR as related to a new feature.
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: release-note-none
description: Denotes a PR that doesn't merit a release note.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: ok-to-test
description: Indicates a non-member PR verified by an org member that is safe to test.
name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
Created: 2024-08-01T11:34:57Z | Merged: 2025-05-08T12:05:15Z

<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

/kind documentation
/kind api-change

#### What this PR does / why we need it:

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
KEP for #1618

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".
-->
```release-note
NONE
```


Diff:
diff --git a/keps/1618-optional-gc-of-workloads/README.md b/keps/1618-optional-gc-of-workloads/README.md
new file mode 100644
index 00000000000..012eed5ea14
--- /dev/null
+++ b/keps/1618-optional-gc-of-workloads/README.md
@@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for Workloads.
+    // A nil value disables automatic deletion of Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines the policies for when Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivatedByKueue is the duration to wait after *any* Kueue-managed Workload
+    // (such as a Job, JobSet, or other custom workload types) has been marked
+    // as deactivated by Kueue before automatically deleting it.
+    // Deletion of deactivated workloads may cascade to objects not created by
+    // Kueue, since deleting the parent Workload owner (e.g. JobSet) can trigger
+    // garbage-collection of dependent resources.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivatedByKueue *metav1.Duration `json:"afterDeactivatedByKueue,omitempty"`
+}
+```
+
+### Behavior
+The new behavior is as follows:
+
+1. During Kueue's initial reconciliation loop, all previously finished and deactivated Workloads
+   will be evaluated against the Workload retention policy. They will either be
+   deleted immediately if they are already expired or requeued for reconciliation
+   after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated
+   using the same approach as in step 1. However, the evaluation will occur
+   during the next reconciliation loop after the Workload was declared finished or deactivated.
+   Based on the evaluation result against the retention policy, the Workload will
+   either be requeued for later reconciliation or deleted.
+
+### Test Plan
+
+[X] I understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+#### Unit Tests
+
+The following new unit tests or modifications to existing ones were added to test 
+the new functionality:
+
+- `apis/config/v1beta1/defaults_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` which sets 
+  `FinishedWorkloadRetention` to nil which ensures that the default 
+  scenario maintains backward compatibility and no retention settings are set.
+  - new unit test: `set object retention policy for finished workloads` which
+  checks if provided can configuration overrides default values.
+- `pkg/config/config_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` set to nil 
+  which tests if for existing tests the default behavior maintains backward
+  compatibility and results in no retention settings.
+  - new unit test: `objectRetentionPolicies config` which sets some example 
+  retention policy and then checks if it can be retrieved.
+- `pkg/controller/core/workload_controller_test.go`
+  - added following new tests:
+    - `shouldn't delete the workload because, object retention not configured`
+    - `shouldn't try to delete the workload (no event emitted) because it is already being deleted by kubernetes, object retention configured`
+    - `shouldn't try to delete the workload because the retention period hasn't elapsed yet, object retention configured`
+    - `should delete the workload because the retention period has elapsed, object retention configured`
+    - `should delete the workload immediately after it completed because the retention period is set to 0s`
+
+#### Integration tests
+
+- Added behavior `Workload controller with resource retention` 
+in `test/integration/controller/core/workload_controller_test.go`
+
+
+### Graduation Criteria
+
+## Implementation History
+
+- Initial PR created for the feature pre-KEP [#2686](https://github.com/kubernetes-sigs/kueue/pull/2686).
+
+## Drawbacks
+
+- Deleting objects is inherently complex, and garbage collection (GC) behavior can often be 
+surprising for end-users who did not configure the service themselves. Even though the 
+mechanism proposed in this KEP is relatively simple and must be explicitly enabled, 
+it is a global mechanism applied at the controller level. The author has added some 
+observability to the controller itself (logging and emitting deletion events), 
+but Kubernetes users and developers accustomed to tracking object lifecycles 
+using per-object policies might be surprised by this behavior.
+
+
+## Alternatives
+
+- Leave the current state of things as is, not implementing any garbage 
+collection mechanism for finished Workloads.
+- Initially, a different approach was hypothesized where the deletion logic
+  would not be integrated into the `Reconcile` function. Instead, a
+  dedicated handler and watcher pair would manage the cleanup of finished Workloads. This approach had the following advantages and disadvantages::
+  - Cons:
+    - Kueue's internal state management for Workloads happens within the `WorkloadReconciler`. Moving the deletion logic elsewhere would split up code meant to handle a single resource's lifecycle.
+    - Implementing a watcher specifically for expired Workloads turned out to be unexpectedly complex, leading to this approach being abandoned.
+  - Pros:
+    - A separate handler/watcher could offload the deletion logic, making the already large and intricate Reconcile function more focused and potentially easier to maintain.
diff --git a/keps/1618-optional-gc-of-workloads/kep.yaml b/keps/1618-optional-gc-of-workloads/kep.yaml
new file mode 100644
index 00000000000..cbb30730aac
--- /dev/null
+++ b/keps/1618-optional-gc-of-workloads/kep.yaml
@@ -0,0 +1,34 @@
+title: Optional garbage collection of finished and deactivated Workloads
+kep-number: 1618
+authors:
+  - "@mwysokin"
+  - "@mykysha"
+status: implementable
+creation-date: 2024-07-31
+reviewers:
+  - "@denkensk"
+  - "@mimowo"
+  - "@PBundyra"
+  - "@tenzen-y"
+  - "@mbobrovskyi"
+approvers:
+  - "@mimowo"
+  - "@tenzen-y"
+
+# The target maturity stage in the current dev cycle for this KEP.
+stage: alpha
+
+# The following PRR answers are required at alpha release
+# List the feature gate name and the components for which it must be enabled
+feature-gates:
+  - name: ObjectRetentionPolicies
+disable-supported: true
+
+# The most recent milestone for which work toward delivery of this KEP has been
+# done. This can be the current (upcoming) milestone, if it is being actively
+# worked on.
+latest-milestone: "v0.12"
+
+# The milestone at which this feature was, or is targeted to be, at each stage.
+milestone:
+  alpha: "v0.12"


Reviews:
user: login: mimowo
body: First pass, it lgtm overall. Left some comments to improve it.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-01T12:03:41Z
user: login: PBundyra
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-01T13:47:02Z
user: login: PBundyra
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-01T14:01:26Z
user: login: PBundyra
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-01T14:08:30Z
user: login: PBundyra
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-01T14:55:32Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-07T09:51:32Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-07T09:54:30Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-07T09:54:50Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-07T09:58:12Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-07T10:04:20Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-07T10:07:22Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-07T10:47:40Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-07T10:48:51Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-08T07:21:50Z
user: login: mimowo
body: LGTM, one remaining question from me: https://github.com/kubernetes-sigs/kueue/pull/2742/files#r1708818068. @tenzen-y would you like to give it a pass?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-08T08:27:45Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-08T09:37:52Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-08T11:24:16Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-08T11:35:44Z
user: login: mimowo
body: /lgtm
/approve
/hold
To wait for feedback from @tenzen-y as another pair of eyes.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-08T11:53:09Z
user: login: PBundyra
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-08T12:26:26Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2024-08-08T15:44:28Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-09T05:59:44Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-09T12:35:57Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-09T12:49:44Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-09T13:07:20Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-09T15:36:53Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2024-08-14T14:17:59Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-14T18:08:41Z
user: login: mwysokin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-14T18:09:00Z
user: login: alculquicondor
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-16T15:35:09Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2024-08-16T18:23:02Z
user: login: alculquicondor
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-16T18:35:54Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2024-08-16T18:38:27Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2024-08-16T18:48:16Z
user: login: alculquicondor
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-08-16T18:55:31Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2024-08-16T18:57:40Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T12:21:49Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T12:23:31Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T12:24:02Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T12:24:19Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T12:25:23Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T12:25:50Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T12:26:06Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T11:25:48Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T11:28:55Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T11:29:40Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T11:33:56Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T08:48:13Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T08:48:44Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T08:49:14Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T09:06:28Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T09:16:59Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T09:46:08Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T09:46:50Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T11:25:07Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T11:40:45Z
user: login: mimowo
body: LGTM, I would just add a feature gate which is a pattern we use for other features based on beta API directly.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T11:41:40Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T11:42:02Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T14:22:45Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T14:36:47Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T14:48:34Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T15:01:28Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T15:34:14Z
user: login: mimowo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T15:41:11Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T22:10:56Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T08:55:33Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T08:55:51Z
user: login: tenzen-y
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T09:30:15Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T11:47:33Z
user: login: mykysha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T11:48:14Z

Review comments:
diff_hunk: @@ -0,0 +1,441 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: I think this goes a bit too deep into the implementation details.

I believe there is only one story (maybe 2) from the user (admin in this case) perspective:
1. "I would like to minimize the memory footprint by deleting finished Workloads as soon as possible"

This admin will set 0.

2. "I would like to minimize the memory footprint by deleting finished Workloads, but keep them for a while"

This admin will set 30min

So that I can do live debugging it might be helpful to have the objects, rather than just rely on logs.

The things like "I want Kueue to fail if I misconfigured it" and "I would like the mechanism to be backwards compatible", if needed would rather belong to Notes section.
created_at: 2024-08-01T11:59:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,441 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
+
+If the finished Workloads policy is not configured correctly 
+(the provided value is not a valid time.Duration), kueue will fail to start 
+during configuration parsing.
+
+#### Story 1 - Retention policy for finished Workloads is not configured (default)
+
+**This story ensures backward compatibility.**
+
+If the object retention policy section is not configured, 
+or finished Workloads retention is either not configured or set to `null`, 
+Workload objects will not be deleted by kueue.
+
+#### Story 2 - Retention policy for finished Workloads is configured and set to a value > 0s
+
+If the object retention policy is configured to a value greater than 0 seconds, 
+the finished Workload retention policy will be evaluated against every 
+completed Workload. The flow is as follows:
+
+1. During kueue's initial reconciliation loop, all previously finished Workloads 
+will be evaluated against the Workload retention policy. They will either be 
+deleted immediately if they are already expired or requeued for reconciliation 
+after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated 
+using the same approach as in step 1. However, the evaluation will occur 
+during the next reconciliation loop after the Workload was declared finished. 
+Based on the evaluation result against the retention policy, the Workload will 
+either be requeued for later reconciliation or deleted.
+
+#### Story 3 - Retention policy for finished Workloads is configured and set to 0s
+
+When the object retention policy is set to 0 seconds, finished Workloads will be 
+deleted immediately during the first reconciliation loop after they are declared finished.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of kueue 
+(Kubernetes resources, physical resources, etc.).
+- A different implementation was proposed initially. The deletion logic was not 
+supposed to be part of the `Reconcile` function but rather was supposed to have 
+its own handler/watcher pair. During implementation, this proved challenging 
+without substantial changes to the code. Also, creating a watcher for 
+completed Workloads proved to be very complex.
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+        //...
+
+	// ObjectRetentionPolicies provides configuration options for retention of kueue owned
+	// resources.
+	ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+//...
+
+type ObjectRetentionPolicies struct {
+// FinishedWorkloadRetention is the duration to retain finished Workloads.
+// A duration of 0 will delete finished Workloads immediately.
+// A nil value will disable automatic deletion.
+// The value is represented using the metav1.Duration format, allowing for flexible
+// specification of time units (e.g., "24h", "1h30m", "30s").
+//
+// Defaults to null.
+// +optional
+FinishedWorkloadRetention *metav1.Duration `json:"finishedWorkloadRetention"`
+}
+```
+
+### Test Plan
+
+<!--
+**Note:** *Not required until targeted at a release.*
+The goal is to ensure that we don't accept enhancements with inadequate testing.
+
+All code is expected to have adequate tests (eventually with coverage
+expectations). Please adhere to the [Kubernetes testing guidelines][testing-guidelines]
+when drafting this test plan.
+
+[testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
+-->
+
+[X] I understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+[//]: # (##### Prerequisite testing updates)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # (Based on reviewers feedback describe what additional tests need to be added prior)
+
+[//]: # (implementing this enhancement to ensure the enhancements have also solid foundations.)
+
+[//]: # (-->)
+
+#### Unit Tests
+
+<!--
+In principle every added code should have complete unit test coverage, so providing
+the exact set of tests will not bring additional value.
+However, if complete unit test coverage is not possible, explain the reason of it
+together with explanation why this is acceptable.
+-->
+
+<!--
+Additionally, try to enumerate the core package you will be touching
+to implement this enhancement and provide the current unit coverage for those
+in the form of:
+- <package>: <date> - <current test coverage>
+
+This can inform certain test coverage improvements that we want to do before
+extending the production code to implement this enhancement.
+-->
+
+The following new unit tests or modifications to existing ones were added to test 
+the new functionality:
+
+- `apis/config/v1beta1/defaults_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` which sets 
+  `FinishedWorkloadRetention` to nil which ensures that the default 
+  scenario maintains backward compatibility and no retention settings are set.
+  - new unit test: `set object retention policy for finished workloads` which
+  checks if provided can configuration overrides default values.
+- `pkg/config/config_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` set to nil 
+  which tests if for existing tests the default behavior maintains backward
+  compatibility and results in no retention settings.
+  - new unit test: `objectRetentionPolicies config` which sets some example 
+  retention policy and then checks if it can be retrieved.
+- `pkg/controller/core/workload_controller_test.go`
+  - added following new tests:
+    - `shouldn't delete the workload because, object retention not configured`
+    - `shouldn't try to delete the workload (no event emitted) because it is already being deleted by kubernetes, object retention configured`
+    - `shouldn't try to delete the workload because the retention period hasn't elapsed yet, object retention configured`
+    - `should delete the workload because the retention period has elapsed, object retention configured`
+    - `should delete the workload immediately after it completed because the retention period is set to 0s`
+
+#### Integration tests
+
+<!--
+Describe what tests will be added to ensure proper quality of the enhancement.
+
+After the implementation PR is merged, add the names of the tests here.
+-->
+
+- TBA
+
+
+[//]: # (### Graduation Criteria)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # ()
+[//]: # (Clearly define what it means for the feature to be implemented and)
+
+[//]: # (considered stable.)
+
+[//]: # ()
+[//]: # (If the feature you are introducing has high complexity, consider adding graduation)
+
+[//]: # (milestones with these graduation criteria:)
+
+[//]: # (- [Maturity levels &#40;`alpha`, `beta`, `stable`&#41;][maturity-levels])
+
+[//]: # (- [Feature gate][feature gate] lifecycle)
+
+[//]: # (- [Deprecation policy][deprecation-policy])
+
+[//]: # ()
+[//]: # ([feature gate]: https://git.k8s.io/community/contributors/devel/sig-architecture/feature-gates.md)
+
+[//]: # ([maturity-levels]: https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions)
+
+[//]: # ([deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/)
+
+[//]: # (-->)
+
+## Implementation History
+
+<!--
+Major milestones in the lifecycle of a KEP should be tracked in this section.
+Major milestones might include:
+- the `Summary` and `Motivation` sections being merged, signaling SIG acceptance
+- the `Proposal` section being merged, signaling agreement on a proposed design
+- the date implementation started
+- the first Kubernetes release where an initial version of the KEP was available
+- the version of Kubernetes where the KEP graduated to general availability
+- when the KEP was retired or superseded
+-->
+
+- Initial PR created for the feature pre-KEP [#2686](https://github.com/kubernetes-sigs/kueue/pull/2686).
+
+## Drawbacks
+
+<!--
+Why should this KEP _not_ be implemented?
+-->
+
+- Deleting objects is inherently complex, and garbage collection (GC) behavior can often be 
+surprising for end-users who did not configure the service themselves. Even though the 
+mechanism proposed in this KEP is relatively simple and must be explicitly enabled, 
+it is a global mechanism applied at the controller level. The author has added some 
+observability to the controller itself (logging and emitting deletion events), 
+but Kubernetes users and developers accustomed to tracking object lifecycles 
+using per-object policies might be surprised by this behavior.
+
+
+## Alternatives
+
+<!--
+What other approaches did you consider, and why did you rule them out? These do
+not need to be as detailed as the proposal, but should include enough
+information to express the idea and why it was not acceptable.
+-->
+
+- Leave the current state of things as is, not implementing any garbage 
+collection mechanism for finished Workloads.
+- Implement retention policies attached per object (like `.spec.ttlSecondsAfterFinished`) instead of having 
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: Also, workloads are objects internal to Kueue and created by Kueue, so it is not clear how users could set `.spec.ttlSecondsAfterFinished` on a workload.
created_at: 2024-08-01T12:00:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,441 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
+
+If the finished Workloads policy is not configured correctly 
+(the provided value is not a valid time.Duration), kueue will fail to start 
+during configuration parsing.
+
+#### Story 1 - Retention policy for finished Workloads is not configured (default)
+
+**This story ensures backward compatibility.**
+
+If the object retention policy section is not configured, 
+or finished Workloads retention is either not configured or set to `null`, 
+Workload objects will not be deleted by kueue.
+
+#### Story 2 - Retention policy for finished Workloads is configured and set to a value > 0s
+
+If the object retention policy is configured to a value greater than 0 seconds, 
+the finished Workload retention policy will be evaluated against every 
+completed Workload. The flow is as follows:
+
+1. During kueue's initial reconciliation loop, all previously finished Workloads 
+will be evaluated against the Workload retention policy. They will either be 
+deleted immediately if they are already expired or requeued for reconciliation 
+after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated 
+using the same approach as in step 1. However, the evaluation will occur 
+during the next reconciliation loop after the Workload was declared finished. 
+Based on the evaluation result against the retention policy, the Workload will 
+either be requeued for later reconciliation or deleted.
+
+#### Story 3 - Retention policy for finished Workloads is configured and set to 0s
+
+When the object retention policy is set to 0 seconds, finished Workloads will be 
+deleted immediately during the first reconciliation loop after they are declared finished.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of kueue 
+(Kubernetes resources, physical resources, etc.).
+- A different implementation was proposed initially. The deletion logic was not 
+supposed to be part of the `Reconcile` function but rather was supposed to have 
+its own handler/watcher pair. During implementation, this proved challenging 
+without substantial changes to the code. Also, creating a watcher for 
+completed Workloads proved to be very complex.
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+        //...
+
+	// ObjectRetentionPolicies provides configuration options for retention of kueue owned
+	// resources.
+	ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+//...
+
+type ObjectRetentionPolicies struct {
+// FinishedWorkloadRetention is the duration to retain finished Workloads.
+// A duration of 0 will delete finished Workloads immediately.
+// A nil value will disable automatic deletion.
+// The value is represented using the metav1.Duration format, allowing for flexible
+// specification of time units (e.g., "24h", "1h30m", "30s").
+//
+// Defaults to null.
+// +optional
+FinishedWorkloadRetention *metav1.Duration `json:"finishedWorkloadRetention"`
+}
+```
+
+### Test Plan
+
+<!--
+**Note:** *Not required until targeted at a release.*
+The goal is to ensure that we don't accept enhancements with inadequate testing.
+
+All code is expected to have adequate tests (eventually with coverage
+expectations). Please adhere to the [Kubernetes testing guidelines][testing-guidelines]
+when drafting this test plan.
+
+[testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
+-->
+
+[X] I understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+[//]: # (##### Prerequisite testing updates)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # (Based on reviewers feedback describe what additional tests need to be added prior)
+
+[//]: # (implementing this enhancement to ensure the enhancements have also solid foundations.)
+
+[//]: # (-->)
+
+#### Unit Tests
+
+<!--
+In principle every added code should have complete unit test coverage, so providing
+the exact set of tests will not bring additional value.
+However, if complete unit test coverage is not possible, explain the reason of it
+together with explanation why this is acceptable.
+-->
+
+<!--
+Additionally, try to enumerate the core package you will be touching
+to implement this enhancement and provide the current unit coverage for those
+in the form of:
+- <package>: <date> - <current test coverage>
+
+This can inform certain test coverage improvements that we want to do before
+extending the production code to implement this enhancement.
+-->
+
+The following new unit tests or modifications to existing ones were added to test 
+the new functionality:
+
+- `apis/config/v1beta1/defaults_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` which sets 
+  `FinishedWorkloadRetention` to nil which ensures that the default 
+  scenario maintains backward compatibility and no retention settings are set.
+  - new unit test: `set object retention policy for finished workloads` which
+  checks if provided can configuration overrides default values.
+- `pkg/config/config_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` set to nil 
+  which tests if for existing tests the default behavior maintains backward
+  compatibility and results in no retention settings.
+  - new unit test: `objectRetentionPolicies config` which sets some example 
+  retention policy and then checks if it can be retrieved.
+- `pkg/controller/core/workload_controller_test.go`
+  - added following new tests:
+    - `shouldn't delete the workload because, object retention not configured`
+    - `shouldn't try to delete the workload (no event emitted) because it is already being deleted by kubernetes, object retention configured`
+    - `shouldn't try to delete the workload because the retention period hasn't elapsed yet, object retention configured`
+    - `should delete the workload because the retention period has elapsed, object retention configured`
+    - `should delete the workload immediately after it completed because the retention period is set to 0s`
+
+#### Integration tests
+
+<!--
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: I think a single scenario to verify that workload is deleted for a finished Job when the retention is configured would be very helpful. (maybe there are more scenarios, but I believe they could be figured out during implementation).
created_at: 2024-08-01T12:01:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,440 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
+
+If the finished Workloads policy is not configured correctly 
+(the provided value is not a valid `time.Duration`), kueue will fail to start 
+during configuration parsing.
+
+#### Story 1 - Retention policy for finished Workloads is not configured (default)
+
+**This story ensures backward compatibility.**
+
+If the object retention policy section is not configured, 
+or finished Workloads retention is either not configured or set to `null`, 
+Workload objects will not be deleted by kueue.
+
+#### Story 2 - Retention policy for finished Workloads is configured and set to a value > 0s
+
+If the object retention policy is configured to a value greater than 0 seconds, 
+the finished Workload retention policy will be evaluated against every 
+completed Workload. The flow is as follows:
+
+1. During kueue's initial reconciliation loop, all previously finished Workloads 
+will be evaluated against the Workload retention policy. They will either be 
+deleted immediately if they are already expired or requeued for reconciliation 
+after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated 
+using the same approach as in step 1. However, the evaluation will occur 
+during the next reconciliation loop after the Workload was declared finished. 
+Based on the evaluation result against the retention policy, the Workload will 
+either be requeued for later reconciliation or deleted.
+
+#### Story 3 - Retention policy for finished Workloads is configured and set to 0s
+
+When the object retention policy is set to 0 seconds, finished Workloads will be 
+deleted immediately during the first reconciliation loop after they are declared finished.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of kueue 
+(Kubernetes resources, physical resources, etc.).
+- A different implementation was proposed initially. The deletion logic was not 
+supposed to be part of the `Reconcile` function but rather was supposed to have 
+its own handler/watcher pair. During implementation, this proved challenging 
+without substantial changes to the code. Also, creating a watcher for 
+completed Workloads proved to be very complex.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: PBundyra
body: Could we move this point to the Alternatives section, and list its pros and cons?
created_at: 2024-08-01T13:47:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,440 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
+
+If the finished Workloads policy is not configured correctly 
+(the provided value is not a valid `time.Duration`), kueue will fail to start 
+during configuration parsing.
+
+#### Story 1 - Retention policy for finished Workloads is not configured (default)
+
+**This story ensures backward compatibility.**
+
+If the object retention policy section is not configured, 
+or finished Workloads retention is either not configured or set to `null`, 
+Workload objects will not be deleted by kueue.
+
+#### Story 2 - Retention policy for finished Workloads is configured and set to a value > 0s
+
+If the object retention policy is configured to a value greater than 0 seconds, 
+the finished Workload retention policy will be evaluated against every 
+completed Workload. The flow is as follows:
+
+1. During kueue's initial reconciliation loop, all previously finished Workloads 
+will be evaluated against the Workload retention policy. They will either be 
+deleted immediately if they are already expired or requeued for reconciliation 
+after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated 
+using the same approach as in step 1. However, the evaluation will occur 
+during the next reconciliation loop after the Workload was declared finished. 
+Based on the evaluation result against the retention policy, the Workload will 
+either be requeued for later reconciliation or deleted.
+
+#### Story 3 - Retention policy for finished Workloads is configured and set to 0s
+
+When the object retention policy is set to 0 seconds, finished Workloads will be 
+deleted immediately during the first reconciliation loop after they are declared finished.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of kueue 
+(Kubernetes resources, physical resources, etc.).
+- A different implementation was proposed initially. The deletion logic was not 
+supposed to be part of the `Reconcile` function but rather was supposed to have 
+its own handler/watcher pair. During implementation, this proved challenging 
+without substantial changes to the code. Also, creating a watcher for 
+completed Workloads proved to be very complex.
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    //...
+    // ObjectRetentionPolicies provides configuration options for retention of kueue owned
+    // resources.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: PBundyra
body: Maybe we could also add comment that states that a nil value disables deletion?
created_at: 2024-08-01T14:01:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,440 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
+
+If the finished Workloads policy is not configured correctly 
+(the provided value is not a valid `time.Duration`), kueue will fail to start 
+during configuration parsing.
+
+#### Story 1 - Retention policy for finished Workloads is not configured (default)
+
+**This story ensures backward compatibility.**
+
+If the object retention policy section is not configured, 
+or finished Workloads retention is either not configured or set to `null`, 
+Workload objects will not be deleted by kueue.
+
+#### Story 2 - Retention policy for finished Workloads is configured and set to a value > 0s
+
+If the object retention policy is configured to a value greater than 0 seconds, 
+the finished Workload retention policy will be evaluated against every 
+completed Workload. The flow is as follows:
+
+1. During kueue's initial reconciliation loop, all previously finished Workloads 
+will be evaluated against the Workload retention policy. They will either be 
+deleted immediately if they are already expired or requeued for reconciliation 
+after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated 
+using the same approach as in step 1. However, the evaluation will occur 
+during the next reconciliation loop after the Workload was declared finished. 
+Based on the evaluation result against the retention policy, the Workload will 
+either be requeued for later reconciliation or deleted.
+
+#### Story 3 - Retention policy for finished Workloads is configured and set to 0s
+
+When the object retention policy is set to 0 seconds, finished Workloads will be 
+deleted immediately during the first reconciliation loop after they are declared finished.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of kueue 
+(Kubernetes resources, physical resources, etc.).
+- A different implementation was proposed initially. The deletion logic was not 
+supposed to be part of the `Reconcile` function but rather was supposed to have 
+its own handler/watcher pair. During implementation, this proved challenging 
+without substantial changes to the code. Also, creating a watcher for 
+completed Workloads proved to be very complex.
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    //...
+    // ObjectRetentionPolicies provides configuration options for retention of kueue owned
+    // resources.
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+//...
+
+type ObjectRetentionPolicies struct {
+    // FinishedWorkloadRetention is the duration to retain finished Workloads.
+    // A duration of 0 will delete finished Workloads immediately.
+    // A nil value will disable automatic deletion.
+    // The value is represented using the metav1.Duration format, allowing for flexible
+    // specification of time units (e.g., "24h", "1h30m", "30s").
+    //
+    // Defaults to null.
+    // +optional
+    FinishedWorkloadRetention *metav1.Duration `json:"finishedWorkloadRetention"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: PBundyra
body: Could this name suggest the field is related to time e.g. `FinishedWorkloadRetentionTime`
created_at: 2024-08-01T14:08:30Z
author_association: CONTRIBUTOR
path: keps/1618-optional-gc-of-workloads/README.md
user: login: PBundyra
body: Just a nit - could we use `Kueue` with capitalized K instead of `kueue` to be consistent with other artifacts?
created_at: 2024-08-01T14:55:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,441 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: I moved the misconfiguration and default scenario to notes as suggested. I squeezed all stories into one more generic and moved the description of the behavior to its own section Behavior in Design Details.
created_at: 2024-08-07T09:51:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,441 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
+
+If the finished Workloads policy is not configured correctly 
+(the provided value is not a valid time.Duration), kueue will fail to start 
+during configuration parsing.
+
+#### Story 1 - Retention policy for finished Workloads is not configured (default)
+
+**This story ensures backward compatibility.**
+
+If the object retention policy section is not configured, 
+or finished Workloads retention is either not configured or set to `null`, 
+Workload objects will not be deleted by kueue.
+
+#### Story 2 - Retention policy for finished Workloads is configured and set to a value > 0s
+
+If the object retention policy is configured to a value greater than 0 seconds, 
+the finished Workload retention policy will be evaluated against every 
+completed Workload. The flow is as follows:
+
+1. During kueue's initial reconciliation loop, all previously finished Workloads 
+will be evaluated against the Workload retention policy. They will either be 
+deleted immediately if they are already expired or requeued for reconciliation 
+after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated 
+using the same approach as in step 1. However, the evaluation will occur 
+during the next reconciliation loop after the Workload was declared finished. 
+Based on the evaluation result against the retention policy, the Workload will 
+either be requeued for later reconciliation or deleted.
+
+#### Story 3 - Retention policy for finished Workloads is configured and set to 0s
+
+When the object retention policy is set to 0 seconds, finished Workloads will be 
+deleted immediately during the first reconciliation loop after they are declared finished.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of kueue 
+(Kubernetes resources, physical resources, etc.).
+- A different implementation was proposed initially. The deletion logic was not 
+supposed to be part of the `Reconcile` function but rather was supposed to have 
+its own handler/watcher pair. During implementation, this proved challenging 
+without substantial changes to the code. Also, creating a watcher for 
+completed Workloads proved to be very complex.
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+        //...
+
+	// ObjectRetentionPolicies provides configuration options for retention of kueue owned
+	// resources.
+	ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+//...
+
+type ObjectRetentionPolicies struct {
+// FinishedWorkloadRetention is the duration to retain finished Workloads.
+// A duration of 0 will delete finished Workloads immediately.
+// A nil value will disable automatic deletion.
+// The value is represented using the metav1.Duration format, allowing for flexible
+// specification of time units (e.g., "24h", "1h30m", "30s").
+//
+// Defaults to null.
+// +optional
+FinishedWorkloadRetention *metav1.Duration `json:"finishedWorkloadRetention"`
+}
+```
+
+### Test Plan
+
+<!--
+**Note:** *Not required until targeted at a release.*
+The goal is to ensure that we don't accept enhancements with inadequate testing.
+
+All code is expected to have adequate tests (eventually with coverage
+expectations). Please adhere to the [Kubernetes testing guidelines][testing-guidelines]
+when drafting this test plan.
+
+[testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
+-->
+
+[X] I understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+[//]: # (##### Prerequisite testing updates)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # (Based on reviewers feedback describe what additional tests need to be added prior)
+
+[//]: # (implementing this enhancement to ensure the enhancements have also solid foundations.)
+
+[//]: # (-->)
+
+#### Unit Tests
+
+<!--
+In principle every added code should have complete unit test coverage, so providing
+the exact set of tests will not bring additional value.
+However, if complete unit test coverage is not possible, explain the reason of it
+together with explanation why this is acceptable.
+-->
+
+<!--
+Additionally, try to enumerate the core package you will be touching
+to implement this enhancement and provide the current unit coverage for those
+in the form of:
+- <package>: <date> - <current test coverage>
+
+This can inform certain test coverage improvements that we want to do before
+extending the production code to implement this enhancement.
+-->
+
+The following new unit tests or modifications to existing ones were added to test 
+the new functionality:
+
+- `apis/config/v1beta1/defaults_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` which sets 
+  `FinishedWorkloadRetention` to nil which ensures that the default 
+  scenario maintains backward compatibility and no retention settings are set.
+  - new unit test: `set object retention policy for finished workloads` which
+  checks if provided can configuration overrides default values.
+- `pkg/config/config_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` set to nil 
+  which tests if for existing tests the default behavior maintains backward
+  compatibility and results in no retention settings.
+  - new unit test: `objectRetentionPolicies config` which sets some example 
+  retention policy and then checks if it can be retrieved.
+- `pkg/controller/core/workload_controller_test.go`
+  - added following new tests:
+    - `shouldn't delete the workload because, object retention not configured`
+    - `shouldn't try to delete the workload (no event emitted) because it is already being deleted by kubernetes, object retention configured`
+    - `shouldn't try to delete the workload because the retention period hasn't elapsed yet, object retention configured`
+    - `should delete the workload because the retention period has elapsed, object retention configured`
+    - `should delete the workload immediately after it completed because the retention period is set to 0s`
+
+#### Integration tests
+
+<!--
+Describe what tests will be added to ensure proper quality of the enhancement.
+
+After the implementation PR is merged, add the names of the tests here.
+-->
+
+- TBA
+
+
+[//]: # (### Graduation Criteria)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # ()
+[//]: # (Clearly define what it means for the feature to be implemented and)
+
+[//]: # (considered stable.)
+
+[//]: # ()
+[//]: # (If the feature you are introducing has high complexity, consider adding graduation)
+
+[//]: # (milestones with these graduation criteria:)
+
+[//]: # (- [Maturity levels &#40;`alpha`, `beta`, `stable`&#41;][maturity-levels])
+
+[//]: # (- [Feature gate][feature gate] lifecycle)
+
+[//]: # (- [Deprecation policy][deprecation-policy])
+
+[//]: # ()
+[//]: # ([feature gate]: https://git.k8s.io/community/contributors/devel/sig-architecture/feature-gates.md)
+
+[//]: # ([maturity-levels]: https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions)
+
+[//]: # ([deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/)
+
+[//]: # (-->)
+
+## Implementation History
+
+<!--
+Major milestones in the lifecycle of a KEP should be tracked in this section.
+Major milestones might include:
+- the `Summary` and `Motivation` sections being merged, signaling SIG acceptance
+- the `Proposal` section being merged, signaling agreement on a proposed design
+- the date implementation started
+- the first Kubernetes release where an initial version of the KEP was available
+- the version of Kubernetes where the KEP graduated to general availability
+- when the KEP was retired or superseded
+-->
+
+- Initial PR created for the feature pre-KEP [#2686](https://github.com/kubernetes-sigs/kueue/pull/2686).
+
+## Drawbacks
+
+<!--
+Why should this KEP _not_ be implemented?
+-->
+
+- Deleting objects is inherently complex, and garbage collection (GC) behavior can often be 
+surprising for end-users who did not configure the service themselves. Even though the 
+mechanism proposed in this KEP is relatively simple and must be explicitly enabled, 
+it is a global mechanism applied at the controller level. The author has added some 
+observability to the controller itself (logging and emitting deletion events), 
+but Kubernetes users and developers accustomed to tracking object lifecycles 
+using per-object policies might be surprised by this behavior.
+
+
+## Alternatives
+
+<!--
+What other approaches did you consider, and why did you rule them out? These do
+not need to be as detailed as the proposal, but should include enough
+information to express the idea and why it was not acceptable.
+-->
+
+- Leave the current state of things as is, not implementing any garbage 
+collection mechanism for finished Workloads.
+- Implement retention policies attached per object (like `.spec.ttlSecondsAfterFinished`) instead of having 
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: Good point. I forgot that the spec section is not populated by Kueue but by the user. I removed references to that field.
created_at: 2024-08-07T09:54:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,441 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
+
+If the finished Workloads policy is not configured correctly 
+(the provided value is not a valid time.Duration), kueue will fail to start 
+during configuration parsing.
+
+#### Story 1 - Retention policy for finished Workloads is not configured (default)
+
+**This story ensures backward compatibility.**
+
+If the object retention policy section is not configured, 
+or finished Workloads retention is either not configured or set to `null`, 
+Workload objects will not be deleted by kueue.
+
+#### Story 2 - Retention policy for finished Workloads is configured and set to a value > 0s
+
+If the object retention policy is configured to a value greater than 0 seconds, 
+the finished Workload retention policy will be evaluated against every 
+completed Workload. The flow is as follows:
+
+1. During kueue's initial reconciliation loop, all previously finished Workloads 
+will be evaluated against the Workload retention policy. They will either be 
+deleted immediately if they are already expired or requeued for reconciliation 
+after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated 
+using the same approach as in step 1. However, the evaluation will occur 
+during the next reconciliation loop after the Workload was declared finished. 
+Based on the evaluation result against the retention policy, the Workload will 
+either be requeued for later reconciliation or deleted.
+
+#### Story 3 - Retention policy for finished Workloads is configured and set to 0s
+
+When the object retention policy is set to 0 seconds, finished Workloads will be 
+deleted immediately during the first reconciliation loop after they are declared finished.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of kueue 
+(Kubernetes resources, physical resources, etc.).
+- A different implementation was proposed initially. The deletion logic was not 
+supposed to be part of the `Reconcile` function but rather was supposed to have 
+its own handler/watcher pair. During implementation, this proved challenging 
+without substantial changes to the code. Also, creating a watcher for 
+completed Workloads proved to be very complex.
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+        //...
+
+	// ObjectRetentionPolicies provides configuration options for retention of kueue owned
+	// resources.
+	ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+//...
+
+type ObjectRetentionPolicies struct {
+// FinishedWorkloadRetention is the duration to retain finished Workloads.
+// A duration of 0 will delete finished Workloads immediately.
+// A nil value will disable automatic deletion.
+// The value is represented using the metav1.Duration format, allowing for flexible
+// specification of time units (e.g., "24h", "1h30m", "30s").
+//
+// Defaults to null.
+// +optional
+FinishedWorkloadRetention *metav1.Duration `json:"finishedWorkloadRetention"`
+}
+```
+
+### Test Plan
+
+<!--
+**Note:** *Not required until targeted at a release.*
+The goal is to ensure that we don't accept enhancements with inadequate testing.
+
+All code is expected to have adequate tests (eventually with coverage
+expectations). Please adhere to the [Kubernetes testing guidelines][testing-guidelines]
+when drafting this test plan.
+
+[testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
+-->
+
+[X] I understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+[//]: # (##### Prerequisite testing updates)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # (Based on reviewers feedback describe what additional tests need to be added prior)
+
+[//]: # (implementing this enhancement to ensure the enhancements have also solid foundations.)
+
+[//]: # (-->)
+
+#### Unit Tests
+
+<!--
+In principle every added code should have complete unit test coverage, so providing
+the exact set of tests will not bring additional value.
+However, if complete unit test coverage is not possible, explain the reason of it
+together with explanation why this is acceptable.
+-->
+
+<!--
+Additionally, try to enumerate the core package you will be touching
+to implement this enhancement and provide the current unit coverage for those
+in the form of:
+- <package>: <date> - <current test coverage>
+
+This can inform certain test coverage improvements that we want to do before
+extending the production code to implement this enhancement.
+-->
+
+The following new unit tests or modifications to existing ones were added to test 
+the new functionality:
+
+- `apis/config/v1beta1/defaults_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` which sets 
+  `FinishedWorkloadRetention` to nil which ensures that the default 
+  scenario maintains backward compatibility and no retention settings are set.
+  - new unit test: `set object retention policy for finished workloads` which
+  checks if provided can configuration overrides default values.
+- `pkg/config/config_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` set to nil 
+  which tests if for existing tests the default behavior maintains backward
+  compatibility and results in no retention settings.
+  - new unit test: `objectRetentionPolicies config` which sets some example 
+  retention policy and then checks if it can be retrieved.
+- `pkg/controller/core/workload_controller_test.go`
+  - added following new tests:
+    - `shouldn't delete the workload because, object retention not configured`
+    - `shouldn't try to delete the workload (no event emitted) because it is already being deleted by kubernetes, object retention configured`
+    - `shouldn't try to delete the workload because the retention period hasn't elapsed yet, object retention configured`
+    - `should delete the workload because the retention period has elapsed, object retention configured`
+    - `should delete the workload immediately after it completed because the retention period is set to 0s`
+
+#### Integration tests
+
+<!--
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: Added the test to the original PR and mentioned it here.
created_at: 2024-08-07T09:54:50Z
author_association: CONTRIBUTOR
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: Done
created_at: 2024-08-07T09:58:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,440 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
+
+If the finished Workloads policy is not configured correctly 
+(the provided value is not a valid `time.Duration`), kueue will fail to start 
+during configuration parsing.
+
+#### Story 1 - Retention policy for finished Workloads is not configured (default)
+
+**This story ensures backward compatibility.**
+
+If the object retention policy section is not configured, 
+or finished Workloads retention is either not configured or set to `null`, 
+Workload objects will not be deleted by kueue.
+
+#### Story 2 - Retention policy for finished Workloads is configured and set to a value > 0s
+
+If the object retention policy is configured to a value greater than 0 seconds, 
+the finished Workload retention policy will be evaluated against every 
+completed Workload. The flow is as follows:
+
+1. During kueue's initial reconciliation loop, all previously finished Workloads 
+will be evaluated against the Workload retention policy. They will either be 
+deleted immediately if they are already expired or requeued for reconciliation 
+after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated 
+using the same approach as in step 1. However, the evaluation will occur 
+during the next reconciliation loop after the Workload was declared finished. 
+Based on the evaluation result against the retention policy, the Workload will 
+either be requeued for later reconciliation or deleted.
+
+#### Story 3 - Retention policy for finished Workloads is configured and set to 0s
+
+When the object retention policy is set to 0 seconds, finished Workloads will be 
+deleted immediately during the first reconciliation loop after they are declared finished.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of kueue 
+(Kubernetes resources, physical resources, etc.).
+- A different implementation was proposed initially. The deletion logic was not 
+supposed to be part of the `Reconcile` function but rather was supposed to have 
+its own handler/watcher pair. During implementation, this proved challenging 
+without substantial changes to the code. Also, creating a watcher for 
+completed Workloads proved to be very complex.
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    //...
+    // ObjectRetentionPolicies provides configuration options for retention of kueue owned
+    // resources.
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+//...
+
+type ObjectRetentionPolicies struct {
+    // FinishedWorkloadRetention is the duration to retain finished Workloads.
+    // A duration of 0 will delete finished Workloads immediately.
+    // A nil value will disable automatic deletion.
+    // The value is represented using the metav1.Duration format, allowing for flexible
+    // specification of time units (e.g., "24h", "1h30m", "30s").
+    //
+    // Defaults to null.
+    // +optional
+    FinishedWorkloadRetention *metav1.Duration `json:"finishedWorkloadRetention"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: It could and if you feel strongly about it I'm happy to do it. Personally I think it'd a bit excessive and redundant since retention in this context could only be associated with time. But as I said, if you think it'd improve the clarity of the code and if in your opinion the association isn't obvious and could be mistaken with something else I'll make the change. WDYT?
created_at: 2024-08-07T10:04:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,440 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
+
+If the finished Workloads policy is not configured correctly 
+(the provided value is not a valid `time.Duration`), kueue will fail to start 
+during configuration parsing.
+
+#### Story 1 - Retention policy for finished Workloads is not configured (default)
+
+**This story ensures backward compatibility.**
+
+If the object retention policy section is not configured, 
+or finished Workloads retention is either not configured or set to `null`, 
+Workload objects will not be deleted by kueue.
+
+#### Story 2 - Retention policy for finished Workloads is configured and set to a value > 0s
+
+If the object retention policy is configured to a value greater than 0 seconds, 
+the finished Workload retention policy will be evaluated against every 
+completed Workload. The flow is as follows:
+
+1. During kueue's initial reconciliation loop, all previously finished Workloads 
+will be evaluated against the Workload retention policy. They will either be 
+deleted immediately if they are already expired or requeued for reconciliation 
+after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated 
+using the same approach as in step 1. However, the evaluation will occur 
+during the next reconciliation loop after the Workload was declared finished. 
+Based on the evaluation result against the retention policy, the Workload will 
+either be requeued for later reconciliation or deleted.
+
+#### Story 3 - Retention policy for finished Workloads is configured and set to 0s
+
+When the object retention policy is set to 0 seconds, finished Workloads will be 
+deleted immediately during the first reconciliation loop after they are declared finished.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of kueue 
+(Kubernetes resources, physical resources, etc.).
+- A different implementation was proposed initially. The deletion logic was not 
+supposed to be part of the `Reconcile` function but rather was supposed to have 
+its own handler/watcher pair. During implementation, this proved challenging 
+without substantial changes to the code. Also, creating a watcher for 
+completed Workloads proved to be very complex.
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    //...
+    // ObjectRetentionPolicies provides configuration options for retention of kueue owned
+    // resources.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: Good point. Added!
created_at: 2024-08-07T10:07:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,429 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: nit: drop it
created_at: 2024-08-07T10:47:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,441 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: sounds good, feel free to resolve the comment. Followed up with just minor tweaks.
created_at: 2024-08-07T10:48:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,429 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: QQ: I'm wondering if the KEP and the initial implementation tackle this issue, because the workloads in the context of this issue are not necessarily finished. It might still be ok to mention in motivation for the generic KEP on workload cleanups, but I would like to clarify if the first version will cover it.
created_at: 2024-08-08T07:21:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,440 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
+
+If the finished Workloads policy is not configured correctly 
+(the provided value is not a valid `time.Duration`), kueue will fail to start 
+during configuration parsing.
+
+#### Story 1 - Retention policy for finished Workloads is not configured (default)
+
+**This story ensures backward compatibility.**
+
+If the object retention policy section is not configured, 
+or finished Workloads retention is either not configured or set to `null`, 
+Workload objects will not be deleted by kueue.
+
+#### Story 2 - Retention policy for finished Workloads is configured and set to a value > 0s
+
+If the object retention policy is configured to a value greater than 0 seconds, 
+the finished Workload retention policy will be evaluated against every 
+completed Workload. The flow is as follows:
+
+1. During kueue's initial reconciliation loop, all previously finished Workloads 
+will be evaluated against the Workload retention policy. They will either be 
+deleted immediately if they are already expired or requeued for reconciliation 
+after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated 
+using the same approach as in step 1. However, the evaluation will occur 
+during the next reconciliation loop after the Workload was declared finished. 
+Based on the evaluation result against the retention policy, the Workload will 
+either be requeued for later reconciliation or deleted.
+
+#### Story 3 - Retention policy for finished Workloads is configured and set to 0s
+
+When the object retention policy is set to 0 seconds, finished Workloads will be 
+deleted immediately during the first reconciliation loop after they are declared finished.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of kueue 
+(Kubernetes resources, physical resources, etc.).
+- A different implementation was proposed initially. The deletion logic was not 
+supposed to be part of the `Reconcile` function but rather was supposed to have 
+its own handler/watcher pair. During implementation, this proved challenging 
+without substantial changes to the code. Also, creating a watcher for 
+completed Workloads proved to be very complex.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: Done in: https://github.com/kubernetes-sigs/kueue/pull/2742/commits/2d59922d3a0a32aaea36e70c14d8a2d9f55fd0bc
created_at: 2024-08-08T09:37:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,429 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: I checked and even though an extension to the code would be minimal I'd rather keep it as a separate task not to pollute and further delay this feature. I will update the KEP once I have something more polished then just a draft version of how the removal of orphaned workloads would look like. For now I'm going to remove the orphaned Workloads issue from the KEP. WDYT?
created_at: 2024-08-08T11:24:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,429 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: Thank you for checking I think removing them from the KEP makes sense, because:
- they are not finished
- there is no point in keeping such workloads for >30min (or whatever is configured).
So, I consider https://github.com/kubernetes-sigs/kueue/issues/1789 to be just a bug which does not require any API changes or KEP,. 
created_at: 2024-08-08T11:35:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,440 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 0 - Retention policy for finished Workloads is misconfigured](#story-0---retention-policy-for-finished-workloads-is-misconfigured)
+    - [Story 1 - Retention policy for finished Workloads is not configured (default)](#story-1---retention-policy-for-finished-workloads-is-not-configured-default)
+    - [Story 2 - Retention policy for finished Workloads is configured and set to a value &gt; 0s](#story-2---retention-policy-for-finished-workloads-is-configured-and-set-to-a-value--0s)
+    - [Story 3 - Retention policy for finished Workloads is configured and set to 0s](#story-3---retention-policy-for-finished-workloads-is-configured-and-set-to-0s)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in kueue. 
+Currently, kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing kueue memory footprint**: Finished Workloads are loaded during kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any kueue-authored Kubernetes object other than finished Workloads.
+- Configure retention policies (like `.spec.ttlSecondsAfterFinished`) that are attached to 
+individual Kubernetes objects.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+Based on how the retention policy is configured for finished Workloads, 
+several user stories are proposed below.
+
+#### Story 0 - Retention policy for finished Workloads is misconfigured
+
+If the finished Workloads policy is not configured correctly 
+(the provided value is not a valid `time.Duration`), kueue will fail to start 
+during configuration parsing.
+
+#### Story 1 - Retention policy for finished Workloads is not configured (default)
+
+**This story ensures backward compatibility.**
+
+If the object retention policy section is not configured, 
+or finished Workloads retention is either not configured or set to `null`, 
+Workload objects will not be deleted by kueue.
+
+#### Story 2 - Retention policy for finished Workloads is configured and set to a value > 0s
+
+If the object retention policy is configured to a value greater than 0 seconds, 
+the finished Workload retention policy will be evaluated against every 
+completed Workload. The flow is as follows:
+
+1. During kueue's initial reconciliation loop, all previously finished Workloads 
+will be evaluated against the Workload retention policy. They will either be 
+deleted immediately if they are already expired or requeued for reconciliation 
+after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated 
+using the same approach as in step 1. However, the evaluation will occur 
+during the next reconciliation loop after the Workload was declared finished. 
+Based on the evaluation result against the retention policy, the Workload will 
+either be requeued for later reconciliation or deleted.
+
+#### Story 3 - Retention policy for finished Workloads is configured and set to 0s
+
+When the object retention policy is set to 0 seconds, finished Workloads will be 
+deleted immediately during the first reconciliation loop after they are declared finished.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of kueue 
+(Kubernetes resources, physical resources, etc.).
+- A different implementation was proposed initially. The deletion logic was not 
+supposed to be part of the `Reconcile` function but rather was supposed to have 
+its own handler/watcher pair. During implementation, this proved challenging 
+without substantial changes to the code. Also, creating a watcher for 
+completed Workloads proved to be very complex.
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    //...
+    // ObjectRetentionPolicies provides configuration options for retention of kueue owned
+    // resources.
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+//...
+
+type ObjectRetentionPolicies struct {
+    // FinishedWorkloadRetention is the duration to retain finished Workloads.
+    // A duration of 0 will delete finished Workloads immediately.
+    // A nil value will disable automatic deletion.
+    // The value is represented using the metav1.Duration format, allowing for flexible
+    // specification of time units (e.g., "24h", "1h30m", "30s").
+    //
+    // Defaults to null.
+    // +optional
+    FinishedWorkloadRetention *metav1.Duration `json:"finishedWorkloadRetention"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: PBundyra
body: What I had in mind were fields such as `TTLSecondsAfterFinished` in Job or `TerminationGracePeriodSeconds` in Pod, but indeed the context of these fields is different - they are configured in Object's spec. Given the context of this field - as it's inside the `ObjectRetentionPolicies` in the global config,  I'm fine with leaving it as it is. 
created_at: 2024-08-08T12:26:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: Does this mean that you will work on the deletion of other objects except the finished Workloads in the next iteration?

But, you mention that ''Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.'' in the Non-Goals section.

created_at: 2024-08-08T15:09:43Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: I believe that we can mitigate this risk by setting the strictly qps only when admins enable this feature.
Once we limit the client qps and burst in the KueueConfiguration, we can mitigate the burst kube-apiserver load.

https://github.com/kubernetes-sigs/kueue/blob/c6f181ba7c2f0954accba777accd50bed4d56edb/config/components/manager/controller_manager_config.yaml#L21-L23

So, I would propose the mentioning limiting client qps when the admins enable this feature in the existing cluster.
Hence, could you mention this mitigation here?

created_at: 2024-08-08T15:20:36Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: TBH, I feel that this API looks like a half-bake to me.
Because when we add new policies to garbage collect other resources like integrations Jobs, we need to hard cord the API name here like `FinishedRayClusterRetention`. But, the Kueue integration Jobs are the extensible interface for external Jobs like AppWrapper or in-house one. So, I guess that it is challenging to maintain retention policies for all integrations like:

```yaml
[...]
objectRetentionPolicies:
  finiishedWorkloadRetention: 10s
  failedRayClusterRetention: 30s
  # How to specify in-house or other OSS resource policy, here?
  # In the upstream Kueue, we can not find all integrations 
  # since the integration framework could be implemented in other places regardless of private or public repositories.
[...]
```
From another perspective, even if we restrict the objective objects to specify retention duration, this API is a little bit extensible because in spite of that the API says that this API is used for retention policies for "object", but in this case, actually, we can specify only Workload retention policies.

```yaml
objectRetentionPolicies:
 finishedWorkloadRetention: 10s
 succeededWorkloadRetention: 30s 
```

In conclusion, I would ask which objects will you support in this feature through all iterations?
It's only Workload? or It's arbitrary objects?

created_at: 2024-08-08T15:33:37Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    //...
+    // ObjectRetentionPolicies provides configuration options for retention of Kueue owned
+    // objects. A nil value will disable automatic deletion for all objects.
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+//...
+
+type ObjectRetentionPolicies struct {
+    // FinishedWorkloadRetention is the duration to retain finished Workloads.
+    // A duration of 0 will delete finished Workloads immediately.
+    // A nil value will disable automatic deletion.
+    // The value is represented using the metav1.Duration format, allowing for flexible
+    // specification of time units (e.g., "24h", "1h30m", "30s").
+    //
+    // Defaults to null.
+    // +optional
+    FinishedWorkloadRetention *metav1.Duration `json:"finishedWorkloadRetention"`
+}
+```
+
+### Behavior
+The new behavior is as follows:
+
+1. During Kueue's initial reconciliation loop, all previously finished Workloads
+   will be evaluated against the Workload retention policy. They will either be
+   deleted immediately if they are already expired or requeued for reconciliation
+   after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated
+   using the same approach as in step 1. However, the evaluation will occur
+   during the next reconciliation loop after the Workload was declared finished.
+   Based on the evaluation result against the retention policy, the Workload will
+   either be requeued for later reconciliation or deleted.
+
+### Test Plan
+
+<!--
+**Note:** *Not required until targeted at a release.*
+The goal is to ensure that we don't accept enhancements with inadequate testing.
+
+All code is expected to have adequate tests (eventually with coverage
+expectations). Please adhere to the [Kubernetes testing guidelines][testing-guidelines]
+when drafting this test plan.
+
+[testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
+-->
+
+[X] I understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+[//]: # (##### Prerequisite testing updates)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # (Based on reviewers feedback describe what additional tests need to be added prior)
+
+[//]: # (implementing this enhancement to ensure the enhancements have also solid foundations.)
+
+[//]: # (-->)
+
+#### Unit Tests
+
+<!--
+In principle every added code should have complete unit test coverage, so providing
+the exact set of tests will not bring additional value.
+However, if complete unit test coverage is not possible, explain the reason of it
+together with explanation why this is acceptable.
+-->
+
+<!--
+Additionally, try to enumerate the core package you will be touching
+to implement this enhancement and provide the current unit coverage for those
+in the form of:
+- <package>: <date> - <current test coverage>
+
+This can inform certain test coverage improvements that we want to do before
+extending the production code to implement this enhancement.
+-->
+
+The following new unit tests or modifications to existing ones were added to test 
+the new functionality:
+
+- `apis/config/v1beta1/defaults_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` which sets 
+  `FinishedWorkloadRetention` to nil which ensures that the default 
+  scenario maintains backward compatibility and no retention settings are set.
+  - new unit test: `set object retention policy for finished workloads` which
+  checks if provided can configuration overrides default values.
+- `pkg/config/config_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` set to nil 
+  which tests if for existing tests the default behavior maintains backward
+  compatibility and results in no retention settings.
+  - new unit test: `objectRetentionPolicies config` which sets some example 
+  retention policy and then checks if it can be retrieved.
+- `pkg/controller/core/workload_controller_test.go`
+  - added following new tests:
+    - `shouldn't delete the workload because, object retention not configured`
+    - `shouldn't try to delete the workload (no event emitted) because it is already being deleted by kubernetes, object retention configured`
+    - `shouldn't try to delete the workload because the retention period hasn't elapsed yet, object retention configured`
+    - `should delete the workload because the retention period has elapsed, object retention configured`
+    - `should delete the workload immediately after it completed because the retention period is set to 0s`
+
+#### Integration tests
+
+<!--
+Describe what tests will be added to ensure proper quality of the enhancement.
+
+After the implementation PR is merged, add the names of the tests here.
+-->
+
+- Added behavior `Workload controller with resource retention` 
+in `test/integration/controller/core/workload_controller_test.go`
+
+
+[//]: # (### Graduation Criteria)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # ()
+[//]: # (Clearly define what it means for the feature to be implemented and)
+
+[//]: # (considered stable.)
+
+[//]: # ()
+[//]: # (If the feature you are introducing has high complexity, consider adding graduation)
+
+[//]: # (milestones with these graduation criteria:)
+
+[//]: # (- [Maturity levels &#40;`alpha`, `beta`, `stable`&#41;][maturity-levels])
+
+[//]: # (- [Feature gate][feature gate] lifecycle)
+
+[//]: # (- [Deprecation policy][deprecation-policy])
+
+[//]: # ()
+[//]: # ([feature gate]: https://git.k8s.io/community/contributors/devel/sig-architecture/feature-gates.md)
+
+[//]: # ([maturity-levels]: https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions)
+
+[//]: # ([deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/)
+
+[//]: # (-->)
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: Could you revert these comment-outs since we need to revisit here when we want to graduate this feature to Beta/Stable?
created_at: 2024-08-08T15:44:02Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,429 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: Actually, on the second thought. I think it is better to honor user's choice of using "orphan" mode. So, I would suggest to just transition them to finished state. This way we could again link the issue and the KEP. 
created_at: 2024-08-09T05:59:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: The goal was to include certain extensibility into the design so that other objects could be added without modifying root lvl API (`Jobs`, etc.). That's the main reason why the API key for `Workloads` is nested. I haven't seen any issues for removal of other completed object (other than orphaned `Workloads` #1789) but if there's any interest in extending that feature I'd be happy to work on it but figuring out how to remove every object that Kueue owns wasn't the goal here, that's where the items in goals/non-goals came from. I'm also happy to work on any updates to this KEP if we decide to extend the cleanup process to other objects.
created_at: 2024-08-09T12:35:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    //...
+    // ObjectRetentionPolicies provides configuration options for retention of Kueue owned
+    // objects. A nil value will disable automatic deletion for all objects.
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+//...
+
+type ObjectRetentionPolicies struct {
+    // FinishedWorkloadRetention is the duration to retain finished Workloads.
+    // A duration of 0 will delete finished Workloads immediately.
+    // A nil value will disable automatic deletion.
+    // The value is represented using the metav1.Duration format, allowing for flexible
+    // specification of time units (e.g., "24h", "1h30m", "30s").
+    //
+    // Defaults to null.
+    // +optional
+    FinishedWorkloadRetention *metav1.Duration `json:"finishedWorkloadRetention"`
+}
+```
+
+### Behavior
+The new behavior is as follows:
+
+1. During Kueue's initial reconciliation loop, all previously finished Workloads
+   will be evaluated against the Workload retention policy. They will either be
+   deleted immediately if they are already expired or requeued for reconciliation
+   after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated
+   using the same approach as in step 1. However, the evaluation will occur
+   during the next reconciliation loop after the Workload was declared finished.
+   Based on the evaluation result against the retention policy, the Workload will
+   either be requeued for later reconciliation or deleted.
+
+### Test Plan
+
+<!--
+**Note:** *Not required until targeted at a release.*
+The goal is to ensure that we don't accept enhancements with inadequate testing.
+
+All code is expected to have adequate tests (eventually with coverage
+expectations). Please adhere to the [Kubernetes testing guidelines][testing-guidelines]
+when drafting this test plan.
+
+[testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
+-->
+
+[X] I understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+[//]: # (##### Prerequisite testing updates)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # (Based on reviewers feedback describe what additional tests need to be added prior)
+
+[//]: # (implementing this enhancement to ensure the enhancements have also solid foundations.)
+
+[//]: # (-->)
+
+#### Unit Tests
+
+<!--
+In principle every added code should have complete unit test coverage, so providing
+the exact set of tests will not bring additional value.
+However, if complete unit test coverage is not possible, explain the reason of it
+together with explanation why this is acceptable.
+-->
+
+<!--
+Additionally, try to enumerate the core package you will be touching
+to implement this enhancement and provide the current unit coverage for those
+in the form of:
+- <package>: <date> - <current test coverage>
+
+This can inform certain test coverage improvements that we want to do before
+extending the production code to implement this enhancement.
+-->
+
+The following new unit tests or modifications to existing ones were added to test 
+the new functionality:
+
+- `apis/config/v1beta1/defaults_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` which sets 
+  `FinishedWorkloadRetention` to nil which ensures that the default 
+  scenario maintains backward compatibility and no retention settings are set.
+  - new unit test: `set object retention policy for finished workloads` which
+  checks if provided can configuration overrides default values.
+- `pkg/config/config_test.go`
+  - updated existing tests with `defaultObjectRetentionPolicies` set to nil 
+  which tests if for existing tests the default behavior maintains backward
+  compatibility and results in no retention settings.
+  - new unit test: `objectRetentionPolicies config` which sets some example 
+  retention policy and then checks if it can be retrieved.
+- `pkg/controller/core/workload_controller_test.go`
+  - added following new tests:
+    - `shouldn't delete the workload because, object retention not configured`
+    - `shouldn't try to delete the workload (no event emitted) because it is already being deleted by kubernetes, object retention configured`
+    - `shouldn't try to delete the workload because the retention period hasn't elapsed yet, object retention configured`
+    - `should delete the workload because the retention period has elapsed, object retention configured`
+    - `should delete the workload immediately after it completed because the retention period is set to 0s`
+
+#### Integration tests
+
+<!--
+Describe what tests will be added to ensure proper quality of the enhancement.
+
+After the implementation PR is merged, add the names of the tests here.
+-->
+
+- Added behavior `Workload controller with resource retention` 
+in `test/integration/controller/core/workload_controller_test.go`
+
+
+[//]: # (### Graduation Criteria)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # ()
+[//]: # (Clearly define what it means for the feature to be implemented and)
+
+[//]: # (considered stable.)
+
+[//]: # ()
+[//]: # (If the feature you are introducing has high complexity, consider adding graduation)
+
+[//]: # (milestones with these graduation criteria:)
+
+[//]: # (- [Maturity levels &#40;`alpha`, `beta`, `stable`&#41;][maturity-levels])
+
+[//]: # (- [Feature gate][feature gate] lifecycle)
+
+[//]: # (- [Deprecation policy][deprecation-policy])
+
+[//]: # ()
+[//]: # ([feature gate]: https://git.k8s.io/community/contributors/devel/sig-architecture/feature-gates.md)
+
+[//]: # ([maturity-levels]: https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions)
+
+[//]: # ([deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/)
+
+[//]: # (-->)
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: I explicitly commented it out instead of removing it so it could be revisited later but maybe it needs to be there from the beginning? It's my first KEP ever so I'm learning as I go and I'll follow your guidance 🙏 
created_at: 2024-08-09T12:49:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: TBH I started working on this KEP without too much vision but rather I wanted to implement #1618 and I wanted to make the API flexible enough so that it could be extended with other object types in the future. After reading what you said it indeed makes sense to come up with a consistent approach for all objects so that user's could benefit from Kueue's extensibility. 

I don't have much experience with KEPs so I'm not sure what should happen now. For sure I'd like to ship the feature so that it doesn't slowly fades away in the in-review stage but at the same time if you feel that it's half-baked maybe I need to make a step back and come up with a more flexible approach. All feedback and guidance is really welcome 🙇 

I'll do some research over the weekend and I'll try to propose something early next week.
created_at: 2024-08-09T13:07:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: There are a couple of questions / aspects I see touched here:
1. should we make it generic to cover Job CRDs?
2. should we nest the API inside `objectRetentionPolicies`?

Re 1: There is an intrinsic asymmetry between workloads and Job CRDs. Job CRDs are created by users who can specify `spec.ttlSecondsAfterFinished`. Workloads are created by Kueue, and so users don't have this control. So, I think it is justified to start from Workloads only and have the API field specific to them.

Another API maintained by Kueue that comes to my mind is ProvisioningRequest. I think having `finishedProvisioningRequestRetention` will make sense too (but we can leave it for later as currently we delete them eagerly anyway).

Re 2: I think it makes sense to nest the API inside `objectRetentionPolicies` to be future-proof in case we have some other objects with lifetime maintained by Kueue (like provisioning requests, or different policies for finished workloads depending on success  /failure). 
created_at: 2024-08-09T15:36:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: > Re 1: There is an intrinsic asymmetry between workloads and Job CRDs. Job CRDs are created by users who can specify spec.ttlSecondsAfterFinished. Workloads are created by Kueue, and so users don't have this control. So, I think it is justified to start from Workloads only and have the API field specific to them.

Yes, I agree with you. TBH, I don't prefer to implement the feature to do garbage collection of Jobs since I'm thinking that deletion jobs are responsible for Job's own controller like `ttlSecondsAfterFinished`. But this proposed API is extensible for Job API. So, I asked, "which objects will you support in this feature".

> Another API maintained by Kueue that comes to my mind is ProvisioningRequest. I think having finishedProvisioningRequestRetention will make sense too (but we can leave it for later as currently we delete them eagerly anyway).

I agree with you. I believe that the separate and dedicated CRDs for this feature are overkill. So, I would recommend not introducing the dedicated CRD.

> Re 2: I think it makes sense to nest the API inside objectRetentionPolicies to be future-proof in case we have some other objects with lifetime maintained by Kueue (like provisioning requests, or different policies for finished workloads depending on success /failure).

If we want to support additional Kueue-managed objects in the future,
could we add a dedicated field for each object something like this?
I guess that the criterion for garbage collection depends on the object’s kind. 

```yaml
workloadRetentionPolicy:
  onConditions:
    [metav1.Condition]
admissionCheckRetentionsPolicy:
  onConditions:
    [metav1.Condition]
  onResourceFlavor:
    name: xxx
    ...
```

created_at: 2024-08-14T14:09:55Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: I would recommend limiting the scope for this KEP to Kueue managed objects, not only to Kueue Workload objects since actually, we aim to extend the API for the future extensibility for the other Kueue managed resources like AdmissionCheck.

The v1beta1 means that we can not bring breaking changes to APIs without bumping the API version.
So, I would like to take care of future API possibilities in this KEP.

So, my recommendation is to remove "starting with finished Workloads." from Goal, and move the statement to the proposal. And then, mention that we do not aim to add APIs for the Jobs like batch/job, RayJob as we discussed in another thread...
I'm wondering if we need to add another mechanisms for the Jobs retention in the future when we find the actual use cases.

created_at: 2024-08-14T14:17:53Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: This looks a bit more complex than I originally anticipated but I agree that it's quite extensible. TBH I lack expertise when it comes evaluating whether this would be a consistent and a good way for Kueue to support that so I'll defer to other maintainers (@mimowo, @alculquicondor). Once we reach the consensus I'll be happy to implement it.
created_at: 2024-08-14T18:08:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mwysokin
body: Got it. Let me propose another set of changes before the end of the week.
created_at: 2024-08-14T18:09:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: alculquicondor
body: > Introduce configuration of global retention policies for Kueue-authored Kubernetes objects

This sentence matches `Workload` and `ProvisioiningRequest`. We don't need to clean up provisioning requests explicitly, because they are children of the `Workloads`.

So, for now, we only need to take care of `Workloads`. But I agree we should keep the API extensible for supporting the cleanup of finished jobs, but that can be addressed in a separate KEP.
created_at: 2024-08-16T15:19:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,416 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: alculquicondor
body: ```suggestion
- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
```
created_at: 2024-08-16T15:20:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,416 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
path: keps/1618-optional-gc-of-workloads/README.md
user: login: alculquicondor
body: Please remove the comments
created_at: 2024-08-16T15:20:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,416 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
path: keps/1618-optional-gc-of-workloads/README.md
user: login: alculquicondor
body: ```suggestion
Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
```
created_at: 2024-08-16T15:21:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,416 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: alculquicondor
body: ```suggestion
with options for other Kueue-managed objects.
```
created_at: 2024-08-16T15:22:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,416 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
path: keps/1618-optional-gc-of-workloads/README.md
user: login: alculquicondor
body: we don't usually keep the names of the people who suggested the changes.
created_at: 2024-08-16T15:24:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: alculquicondor
body: I think that API looks too complex. For the most part, we will be happy just cleaning up finished Workloads.

Let me propose the following:

```yaml
objectRetentionPolicies:
  workloads:
    afterFinished: 10m
```

It leaves us the possibility to extend to other objects and to even add conditions, but I prefer we don't give that flexibility yet.
created_at: 2024-08-16T15:32:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,416 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    //...
+    // ObjectRetentionPolicies provides configuration options for retention of Kueue owned
+    // objects. A nil value will disable automatic deletion for all objects.
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+//...
+
+type ObjectRetentionPolicies struct {
+    // FinishedWorkloadRetention is the duration to retain finished Workloads.
+    // A duration of 0 will delete finished Workloads immediately.
+    // A nil value will disable automatic deletion.
+    // The value is represented using the metav1.Duration format, allowing for flexible
+    // specification of time units (e.g., "24h", "1h30m", "30s").
+    //
+    // Defaults to null.
+    // +optional
+    FinishedWorkloadRetention *metav1.Duration `json:"finishedWorkloadRetention"`
+}
+```
+
+### Behavior
+The new behavior is as follows:
+
+1. During Kueue's initial reconciliation loop, all previously finished Workloads
+   will be evaluated against the Workload retention policy. They will either be
+   deleted immediately if they are already expired or requeued for reconciliation
+   after their retention period has expired.
+
+2. During subsequent reconciliation loops, each Workload will be evaluated
+   using the same approach as in step 1. However, the evaluation will occur
+   during the next reconciliation loop after the Workload was declared finished.
+   Based on the evaluation result against the retention policy, the Workload will
+   either be requeued for later reconciliation or deleted.
+
+### Test Plan
+
+<!--
+**Note:** *Not required until targeted at a release.*
+The goal is to ensure that we don't accept enhancements with inadequate testing.
+
+All code is expected to have adequate tests (eventually with coverage
+expectations). Please adhere to the [Kubernetes testing guidelines][testing-guidelines]
+when drafting this test plan.
+
+[testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
+-->
+
+[X] I understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+[//]: # (##### Prerequisite testing updates)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # (Based on reviewers feedback describe what additional tests need to be added prior)
+
+[//]: # (implementing this enhancement to ensure the enhancements have also solid foundations.)
+
+[//]: # (-->)
path: keps/1618-optional-gc-of-workloads/README.md
user: login: alculquicondor
body: You can remove this if there aren't any requirements.
created_at: 2024-08-16T15:34:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: > but I prefer we don't give that flexibility yet.

I understand this motivation. But, on the other hand, the proposed API still says that it can be extensible somehow in the future.

For the current API design, I am concerned that we extend the API something like below because the API doesn't restrict the target object to Workload.

```yaml
[...]
objectRetentionPolicies:
  finiishedWorkloadRetention: 10s
  acceptedAdmissionCheckRetention: 30s
[...]
```

So, I would like to select either 1. restricting the target object to Workload or 2. leaving the possibility for the other objects like this:

1

```yaml
[...]
workloadRetentionPolicies:
  afterfFinished: 10s
[...]
```

2

```yaml
objectRetentionPolicies:
  workloads:
    afterFinished: 10m
```

@alculquicondor WDYT?

created_at: 2024-08-16T18:23:02Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: alculquicondor
body: I prefer 2. It looks more future-proof.
created_at: 2024-08-16T18:35:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: That sounds good to me.

created_at: 2024-08-16T18:38:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: This is a side note, although. To evolve API more easily, I want to find a way to guard the new API field :)

IIUC, k/k has the feature gate guard for the new API fields like here: https://github.com/kubernetes/kubernetes/blob/1c4f540669397d031d464877ec3d2b8ac6be9b2f/pkg/apis/core/types.go#L543-L548

We can not break the new API field without bumping the API version in the current API evolving steps. I think that a new API without a guard makes API evolution harder since we need to consider future compatibility and extensibility in the first implementation.
created_at: 2024-08-16T18:48:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: alculquicondor
body: The feature gate only implies that the field will be ignored when disabled. It doesn't mean that we can change the API in a future version of kubernetes, without changing the API version.
created_at: 2024-08-16T18:55:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: Oh, I see. That was not my expected behavior.
Thanks for letting me know.

created_at: 2024-08-16T18:57:40Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Have added a mention about limiting the qps
created_at: 2025-04-28T12:21:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,428 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation by `@mimowo`, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy for finished Workloads is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or finished Workloads retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+<!--
+What are the risks of this proposal, and how do we mitigate? Think broadly.
+For example, consider both security and how this will impact the larger
+Kubernetes ecosystem.
+
+How will security be reviewed, and by whom?
+
+How will UX be reviewed, and by whom?
+
+Consider including folks who also work outside the SIG or subproject.
+-->
+
+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+
+### API
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Have updated the API, introducing a section that should reflect changes for #4471 
created_at: 2025-04-28T12:23:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,416 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Changed this part
created_at: 2025-04-28T12:24:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,416 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Done
created_at: 2025-04-28T12:24:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,416 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Done
created_at: 2025-04-28T12:25:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,416 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Done
created_at: 2025-04-28T12:25:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,416 @@
+# KEP-1618: Optional garbage collection of finished Workloads
+
+<!--
+This is the title of your KEP. Keep it short, simple, and descriptive. A good
+title can help communicate what the KEP is and should be considered as part of
+any review.
+-->
+
+<!--
+A table of contents is helpful for quickly jumping to sections of a KEP and for
+highlighting any additional information provided beyond the standard KEP
+template.
+
+Ensure the TOC is wrapped with
+  <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code>
+tags, and then generate with `hack/update-toc.sh`.
+-->
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+<!--
+This section is incredibly important for producing high-quality, user-focused
+documentation such as release notes or a development roadmap. It should be
+possible to collect this information before implementation begins, in order to
+avoid requiring implementors to split their attention between writing release
+notes and implementing the feature itself. KEP editors and SIG Docs
+should help to ensure that the tone and content of the `Summary` section is
+useful for a wide audience.
+
+A good summary is probably at least a paragraph in length.
+
+Both in this section and below, follow the guidelines of the [documentation
+style guide]. In particular, wrap lines to a reasonable length, to make it
+easier for reviewers to cite specific portions, and to minimize diff churn on
+updates.
+
+[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
+-->
+
+This KEP proposes a mechanism for garbage collection of finished Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on finished Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing finished Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+<!--
+This section is for explicitly listing the motivation, goals, and non-goals of
+this KEP.  Describe why the change is important and the benefits to users. The
+motivation section can optionally provide links to [experience reports] to
+demonstrate the interest in a KEP within the wider Kubernetes community.
+
+[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
+-->
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
+
+## Proposal
+
+<!--
+This is where we get down to the specifics of what the proposal actually is.
+This should have enough detail that reviewers can understand exactly what
+you're proposing, but should not include things like API designs or
+implementation. What is the desired outcome and how do we measure success?.
+The "Design Details" section below is for the real
+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.
+
+
+### User Stories
+
+<!--
+Detail the things that people will be able to do if this KEP is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+-->
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+<!--
+What are the caveats to the proposal?
+What are some important details that didn't come across above?
+Go in to as much detail as necessary here.
+This might be a good place to talk about core concepts and how they relate.
+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Removed these references

created_at: 2025-04-28T12:26:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished and deactivated Workloads
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: I would this as separate user stories, since they originated from two different issues.
created_at: 2025-05-05T11:25:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: Actually, in case of deleting workloads due to deactivation we need to delete the actual Job object. See the script in https://github.com/kubernetes-sigs/kueue/issues/4471#issuecomment-2699341853, it says `delete_jobset`
created_at: 2025-05-05T11:28:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished and deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished and deactivated Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion or deactivation.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for finished Workloads.
+    // A nil value disables automatic deletion of finished Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines when finished Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after a Workload has been deactivated
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: We need to describe and warn users this may actually delete objects not created by Kueue directly. 
created_at: 2025-05-05T11:29:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished and deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished and deactivated Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion or deactivation.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for finished Workloads.
+    // A nil value disables automatic deletion of finished Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines when finished Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after a Workload has been deactivated
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: Since this is an intrusive operation (may delete Jobs not created by Kueue0 I would consider API which day one filters the workloads by the reason for deactivation. For example:
```golang
afterDeactivated:
  onReasons:
  - reason:  RequeuingLimitExceeded
     ttl: 10min
  - reason:  MaximumExecutionTimeExceeded
     ttl: 20min
  defaultTTL: 1h # optional
```
wdyt @tenzen-y @mwysokin ?
created_at: 2025-05-05T11:33:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished and deactivated Workloads
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Have split this in two user stories
created_at: 2025-05-06T08:48:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Have removed this point
created_at: 2025-05-06T08:48:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished and deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished and deactivated Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion or deactivation.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for finished Workloads.
+    // A nil value disables automatic deletion of finished Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines when finished Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after a Workload has been deactivated
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Described deletion of jobsets and how that might delete objects not created by Kueue
created_at: 2025-05-06T08:49:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished and deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished and deactivated Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion or deactivation.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for finished Workloads.
+    // A nil value disables automatic deletion of finished Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines when finished Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after a Workload has been deactivated
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: This is not just about JobSet, in general any  workload objects (like Job or JobSet) which are manged by Kueue. 

EDIT: we could introduce API to filter only JobSets by kind, etc. I don't think this is needed day one, but we probably need more flexibility that just timeout field.
created_at: 2025-05-06T09:06:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished and deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished and deactivated Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion or deactivation.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for finished Workloads.
+    // A nil value disables automatic deletion of finished Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines when finished Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after a Workload has been deactivated
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: The workload condition reasons are not documented, and we do not clear when the workloads get those reasons. Additionally, we do not guarantee using the reason name and reachable criteria. So, it's challenging to specify the exact reasons here for users.

So, I would keep the current `afterFinished` and `afterDeactivated`.

created_at: 2025-05-06T09:16:59Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished and deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished and deactivated Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion or deactivation.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for finished Workloads.
+    // A nil value disables automatic deletion of finished Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines when finished Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after a Workload has been deactivated
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: > The workload condition reasons are not documented

The workload condition reasons are not documented, and we do not clear when the workloads get those reasons.

>  So, I would keep the current afterFinished and afterDeactivated.

Ok, we can start here.

created_at: 2025-05-06T09:46:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished and deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished and deactivated Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion or deactivation.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for finished Workloads.
+    // A nil value disables automatic deletion of finished Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines when finished Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after a Workload has been deactivated
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: Ok, we can start with just timeout as suggested in https://github.com/kubernetes-sigs/kueue/pull/2742#discussion_r2075066376, but the comment should emphasize this is not just for JobSet.
created_at: 2025-05-06T09:46:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished and deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished and deactivated Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion or deactivation.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for finished Workloads.
+    // A nil value disables automatic deletion of finished Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines when finished Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after a Workload has been deactivated
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Have changed the description, this one should reflect the functionality better
created_at: 2025-05-07T11:25:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,28 @@
+title: Optional garbage collection of finished and deactivated Workloads
+kep-number: 1618
+authors:
+  - "@mwysokin"
+  - "@mykysha"
+status: implementable
+creation-date: 2024-07-31
+reviewers:
+  - "@denkensk"
+  - "@mimowo"
+  - "@PBundyra"
+  - "@tenzen-y"
+  - "@mbobrovskyi"
+approvers:
+  - "@mimowo"
+  - "@tenzen-y"
+
+# The target maturity stage in the current dev cycle for this KEP.
+stage: alpha
path: keps/1618-optional-gc-of-workloads/kep.yaml
user: login: mimowo
body: Since the API is beta we need to either call it beta, or introduce a feature gate in alpha first. I would suggest a feature gate.

cc @mwysokin @tenzen-y wdyt?
created_at: 2025-05-07T11:40:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,252 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished and deactivated Workloads](#story-1---configurable-retention-for-finished-and-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished and deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished and deactivated Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete Workloads after a 
+specified duration or to immediately delete them upon completion or deactivation.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for finished Workloads.
+    // A nil value disables automatic deletion of finished Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines when finished Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after a Workload has been deactivated
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: sgtm
created_at: 2025-05-07T11:42:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: ```suggestion
time is very long for that initial run until all expired objects are deleted.
```

nit
created_at: 2025-05-07T14:18:23Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for Workloads.
+    // A nil value disables automatic deletion of Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines the policies for when Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after *any* Kueue-managed Workload
+    // (such as a Job, JobSet, or other custom workload types) has been marked
+    // as deactivated before automatically deleting it.
+    // Deletion of deactivated workloads may cascade to objects not created by
+    // Kueue, since deleting the parent Workload owner (e.g. JobSet) can trigger
+    // garbage-collection of dependent resources.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: How to distinguish between manual deactivation and deactivation by Kueue?

created_at: 2025-05-07T14:21:35Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,28 @@
+title: Optional garbage collection of finished and deactivated Workloads
+kep-number: 1618
+authors:
+  - "@mwysokin"
+  - "@mykysha"
+status: implementable
+creation-date: 2024-07-31
+reviewers:
+  - "@denkensk"
+  - "@mimowo"
+  - "@PBundyra"
+  - "@tenzen-y"
+  - "@mbobrovskyi"
+approvers:
+  - "@mimowo"
+  - "@tenzen-y"
+
+# The target maturity stage in the current dev cycle for this KEP.
+stage: alpha
path: keps/1618-optional-gc-of-workloads/kep.yaml
user: login: tenzen-y
body: +1 on feature gate.

created_at: 2025-05-07T14:22:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for Workloads.
+    // A nil value disables automatic deletion of Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines the policies for when Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after *any* Kueue-managed Workload
+    // (such as a Job, JobSet, or other custom workload types) has been marked
+    // as deactivated before automatically deleting it.
+    // Deletion of deactivated workloads may cascade to objects not created by
+    // Kueue, since deleting the parent Workload owner (e.g. JobSet) can trigger
+    // garbage-collection of dependent resources.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: I think this is not possible at the moment in a clean API. I think when Kueue deactivates we use the DeactivationTarget condition: https://github.com/kubernetes-sigs/kueue/blob/6d4ef2bd6bbefde82d02d68adecc534608cb58f7/pkg/controller/core/workload_controller.go#L200

```golang
		reason := kueue.WorkloadDeactivated
		message := "The workload is deactivated"
		dtCond := apimeta.FindStatusCondition(wl.Status.Conditions, kueue.WorkloadDeactivationTarget)
		if !apimeta.IsStatusConditionTrue(wl.Status.Conditions, kueue.WorkloadEvicted) {
			if dtCond != nil {
				reason = fmt.Sprintf("%sDueTo%s", reason, dtCond.Reason)
				message = fmt.Sprintf("%s due to %s", message, dtCond.Message)
			}
			workload.SetEvictedCondition(&wl, reason, message)
			updated = true
			evicted = true
		}
```
So we could say that if deactivated by Kueue then it starts with "DeactivatedDueTo".

Would you like to only delete those deactivated by Kueue? I think this makes sense. cc @mwysokin if you have some ideas.
created_at: 2025-05-07T14:36:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for Workloads.
+    // A nil value disables automatic deletion of Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines the policies for when Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after *any* Kueue-managed Workload
+    // (such as a Job, JobSet, or other custom workload types) has been marked
+    // as deactivated before automatically deleting it.
+    // Deletion of deactivated workloads may cascade to objects not created by
+    // Kueue, since deleting the parent Workload owner (e.g. JobSet) can trigger
+    // garbage-collection of dependent resources.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: > Would you like to only delete those deactivated by Kueue? I think this makes sense.

I think so too. Manual deactivation has semantics based on users or in-house controllers. In that case, they should manually delete the Workload object by themselves. IIUC, this feature aims to automatically delete the Workload that is automatically preserved by the Kueue deactivation mechanism.

created_at: 2025-05-07T14:48:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for Workloads.
+    // A nil value disables automatic deletion of Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines the policies for when Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after *any* Kueue-managed Workload
+    // (such as a Job, JobSet, or other custom workload types) has been marked
+    // as deactivated before automatically deleting it.
+    // Deletion of deactivated workloads may cascade to objects not created by
+    // Kueue, since deleting the parent Workload owner (e.g. JobSet) can trigger
+    // garbage-collection of dependent resources.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: `AfterDeactivatedByKueue` should we call it then, and could recognize by the "DeactivatedDueTo" prefix?
created_at: 2025-05-07T15:01:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for Workloads.
+    // A nil value disables automatic deletion of Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines the policies for when Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after *any* Kueue-managed Workload
+    // (such as a Job, JobSet, or other custom workload types) has been marked
+    // as deactivated before automatically deleting it.
+    // Deletion of deactivated workloads may cascade to objects not created by
+    // Kueue, since deleting the parent Workload owner (e.g. JobSet) can trigger
+    // garbage-collection of dependent resources.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: I think we have 3 selectable options

1. Renaming the field name to `AfterDeactivatedByKueue`.
2. Adding a knob under the `afterDeactivated` field whether or not all deactivated workload should be removed automatically.
3. Adding active field to status sub resource to distinguish between those

For v1beta1 API, we can select (1, and after we figure out the use case for deleting manual deactivated workload, we can add `AfterDeactivatedByManual`. For v1beta2, we might be able to improve those by status field.

created_at: 2025-05-07T15:34:14Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for Workloads.
+    // A nil value disables automatic deletion of Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines the policies for when Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after *any* Kueue-managed Workload
+    // (such as a Job, JobSet, or other custom workload types) has been marked
+    // as deactivated before automatically deleting it.
+    // Deletion of deactivated workloads may cascade to objects not created by
+    // Kueue, since deleting the parent Workload owner (e.g. JobSet) can trigger
+    // garbage-collection of dependent resources.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mimowo
body: we will probably need more knobs for the feature indeed. Im fine to start with 1.for v1beta1 and add more knobs as we collect feedback.
created_at: 2025-05-07T15:41:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for Workloads.
+    // A nil value disables automatic deletion of Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines the policies for when Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after *any* Kueue-managed Workload
+    // (such as a Job, JobSet, or other custom workload types) has been marked
+    // as deactivated before automatically deleting it.
+    // Deletion of deactivated workloads may cascade to objects not created by
+    // Kueue, since deleting the parent Workload owner (e.g. JobSet) can trigger
+    // garbage-collection of dependent resources.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: SGTM
created_at: 2025-05-07T22:10:55Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for Workloads.
+    // A nil value disables automatic deletion of Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines the policies for when Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivated is the duration to wait after *any* Kueue-managed Workload
+    // (such as a Job, JobSet, or other custom workload types) has been marked
+    // as deactivated before automatically deleting it.
+    // Deletion of deactivated workloads may cascade to objects not created by
+    // Kueue, since deleting the parent Workload owner (e.g. JobSet) can trigger
+    // garbage-collection of dependent resources.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivated *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Have renamed the field to be `AfterDeactivatedByKueue`
created_at: 2025-05-08T08:55:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Thanks, have removed it
created_at: 2025-05-08T08:55:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for Workloads.
+    // A nil value disables automatic deletion of Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines the policies for when Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivatedByKueue is the duration to wait after *any* Kueue-managed Workload
+    // (such as a Job, JobSet, or other custom workload types) has been marked
+    // as deactivated by Kueue before automatically deleting it.
+    // Deletion of deactivated workloads may cascade to objects not created by
+    // Kueue, since deleting the parent Workload owner (e.g. JobSet) can trigger
+    // garbage-collection of dependent resources.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivatedByKueue *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: tenzen-y
body: ```suggestion
    AfterDeactivatedByKueue *metav1.Duration `json:"afterDeactivatedByKueue,omitempty"`
```

Let's update YAML as well
created_at: 2025-05-08T09:30:15Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,264 @@
+# KEP-1618: Optional garbage collection of finished and deactivated Workloads
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+    - [Story 1 - Configurable retention for finished Workloads](#story-1---configurable-retention-for-finished-workloads)
+    - [Story 2 - Configurable retention for deactivated Workloads](#story-2---configurable-retention-for-deactivated-workloads)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [API](#api)
+  - [Behavior](#behavior)
+  - [Test Plan](#test-plan)
+    - [Unit Tests](#unit-tests)
+    - [Integration tests](#integration-tests)
+  - [Graduation Criteria](#graduation-criteria)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+<!-- /toc -->
+
+## Summary
+
+This KEP proposes a mechanism for garbage collection of finished and deactivated Workloads in Kueue. 
+Currently, Kueue does not delete its own Kubernetes objects, leading to potential accumulation 
+and unnecessary resource consumption. The proposed solution involves adding a new 
+API section called `objectRetentionPolicies` to allow users to specify retention 
+policies for finished and deactivated Workloads. This can be set to a specific duration or 
+disabled entirely to maintain backward compatibility.
+
+Benefits of implementing this KEP include the deletion of superfluous information, 
+decreased memory footprint for Kueue, and freeing up etcd storage. The proposed changes 
+are designed to be explicitly enabled and configured by users, ensuring backward compatibility. 
+While the primary focus is on Workloads, the API section can be extended to include 
+other Kueue-authored objects in the future. The implementation involves incorporating the 
+deletion logic into the reconciliation loop and includes considerations for 
+potential risks, such as handling a large number of existing Workloads. 
+Overall, this KEP aims to enhance Kueue's resource management and provide users with 
+more control over object retention.
+
+## Motivation
+
+Currently, Kueue does not delete its own Kubernetes objects, leaving it to Kubernetes' 
+garbage collector (GC) to manage the retention of objects created by Kueue, such as Pods, Jobs, and Workloads. 
+We do not set any expiration on these objects, so by default, they are kept indefinitely. Based on usage patterns, 
+including the amount of jobs created, their complexity, and frequency over time, these objects can accumulate, 
+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618), a mechanism for garbage collection of finished Workloads 
+is being proposed.
+
+With this mechanism in place, deleting finished and deactivated Workloads has several benefits:
+- **Deletion of superfluous information**: Finished Workloads do not store any useful runtime-related information 
+(which is stored by Jobs), nor are they useful for debugging. They could possibly be useful for auditing purposes, 
+but this is not a primary concern.
+- **Decreasing Kueue memory footprint**: Finished and deactivated Workloads are loaded during Kueue's initialization and kept in memory, 
+consuming resources unnecessarily.
+- **Freeing up etcd storage**: Finished and deactivated Workloads are stored in etcd memory until Kubernetes' built-in GC mechanism 
+collects them, but by default, Custom Resources are stored indefinitely.
+
+### Goals
+
+- Support the deletion of finished and deactivated Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+- Maintain backward compatibility (the feature must be explicitly enabled and configured).
+
+### Non-Goals
+
+## Proposal
+
+Add a new field called `objectRetentionPolicies` to the Kueue Configuration API, 
+which will enable specifying a retention policy for finished and deactivated Workloads first under a
+`WorkloadRetentionPolicy` section. This API would be extended in the future 
+with options for other Kueue-managed objects.
+
+
+### User Stories
+
+#### Story 1 - Configurable retention for finished Workloads
+
+As a Kueue administrator, I want to control the retention of 
+finished Workloads to minimize the memory footprint and optimize 
+storage usage in my Kubernetes cluster. I want the flexibility to 
+configure a retention period to automatically delete finished Workloads after a 
+specified duration or to immediately delete them upon completion.
+
+#### Story 2 - Configurable retention for deactivated Workloads
+
+As a Kueue administrator, I want to control the retention of 
+Workloads that have been deactivated due to exceeding 
+the backoff limit to minimize the memory footprint and optimize
+storage usage in my Kubernetes cluster. I want the flexibility to
+configure a retention period to automatically delete failed Workloads after a
+specified duration or to immediately delete them upon exceeding the backoff limit.
+
+**Note:** Immediate deletion can be configured by setting the retention period to 0 seconds.
+
+### Notes/Constraints/Caveats
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback, it was changed 
+because "Resources" bears too many meanings in the context of Kueue 
+(Kubernetes resources, physical resources, etc.).
+- The deletion mechanism assumes that finished Workload objects do not have 
+any finalizers attached to them. After an investigation, 
+the [exact place in the code](https://github.com/kubernetes-sigs/kueue/blob/a128ae3354a56670ffa58695fef1eca270fe3a47/pkg/controller/jobframework/reconciler.go#L332-L340) 
+was found that removes the resource-in-use 
+finalizer when the Workload state transitions to finished. If that behavior ever 
+changes, or if there is an external source of a finalizer being attached 
+to the Workload, it will be marked for deletion by the Kubernetes client 
+but will not actually be deleted until the finalizer is removed.
+- If the retention policy is misconfigured 
+(the provided value is not a valid time.Duration), Kueue will fail to start 
+during configuration parsing.
+- In the default scenario, where the object retention policy section 
+is not configured or Workload retention is either not configured 
+or set to null, the existing behavior is maintained for backward compatibility. 
+This means that Workload objects will not be deleted by Kueue, 
+aligning with the behavior before the introduction of this feature.
+
+### Risks and Mitigations
+
+- **R**: In clusters with a large number of existing finished and deactivated Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.
+**M**: Unfortunately, there is no easy way to directly mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.
+Another currently available option would be to limit client qps and burst when the feature is enabled,
+allowing to mitigate the burst kube-apiserver load.
+
+## Design Details
+
+### API
+```go
+// Configuration is the Schema for the kueueconfigurations API
+type Configuration struct {
+    // …
+    // ObjectRetentionPolicies provides configuration options for automatic deletion
+    // of Kueue-managed objects. A nil value disables all automatic deletions.
+    // +optional
+    ObjectRetentionPolicies *ObjectRetentionPolicies `json:"objectRetentionPolicies,omitempty"`
+}
+
+// ObjectRetentionPolicies holds retention settings for different object types.
+type ObjectRetentionPolicies struct {
+    // Workloads configures retention for Workloads.
+    // A nil value disables automatic deletion of Workloads.
+    // +optional
+    Workloads *WorkloadRetentionPolicy `json:"workloads,omitempty"`
+}
+
+// WorkloadRetentionPolicy defines the policies for when Workloads should be deleted.
+type WorkloadRetentionPolicy struct {
+    // AfterFinished is the duration to wait after a Workload finishes
+    // before deleting it.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterFinished *metav1.Duration `json:"afterFinished,omitempty"`
+    // AfterDeactivatedByKueue is the duration to wait after *any* Kueue-managed Workload
+    // (such as a Job, JobSet, or other custom workload types) has been marked
+    // as deactivated by Kueue before automatically deleting it.
+    // Deletion of deactivated workloads may cascade to objects not created by
+    // Kueue, since deleting the parent Workload owner (e.g. JobSet) can trigger
+    // garbage-collection of dependent resources.
+    // A duration of 0 will delete immediately.
+    // A nil value disables automatic deletion.
+    // Represented using metav1.Duration (e.g. "10m", "1h30m").
+    // +optional
+    AfterDeactivatedByKueue *metav1.Duration `json:"afterDeactivated,omitempty"`
path: keps/1618-optional-gc-of-workloads/README.md
user: login: mykysha
body: Done
created_at: 2025-05-08T11:47:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,28 @@
+title: Optional garbage collection of finished and deactivated Workloads
+kep-number: 1618
+authors:
+  - "@mwysokin"
+  - "@mykysha"
+status: implementable
+creation-date: 2024-07-31
+reviewers:
+  - "@denkensk"
+  - "@mimowo"
+  - "@PBundyra"
+  - "@tenzen-y"
+  - "@mbobrovskyi"
+approvers:
+  - "@mimowo"
+  - "@tenzen-y"
+
+# The target maturity stage in the current dev cycle for this KEP.
+stage: alpha
path: keps/1618-optional-gc-of-workloads/kep.yaml
user: login: mykysha
body: Have added a feature-gate here
created_at: 2025-05-08T11:48:13Z
author_association: CONTRIBUTOR

PR comments:
user: login: k8s-ci-robot
created_at: 2024-08-01T11:35:06Z
body: Hi @mwysokin. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkueue).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

author_association: CONTRIBUTOR
user: login: netlify[bot]
created_at: 2024-08-01T11:35:14Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *kubernetes-sigs-kueue* canceled.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | b21a063601194bc79fcf8b06e0a094997ad79777 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/681c99ad7878a00008660a77 |
author_association: NONE
user: login: mimowo
created_at: 2024-08-01T11:40:07Z
body: /cc @tenzen-y @PBundyra 
author_association: CONTRIBUTOR
user: login: mimowo
created_at: 2024-08-01T11:50:10Z
body: nit, please say "KEP for https://github.com/kubernetes-sigs/kueue/issues/1618" in the PR description instead of "Fixes". Otherwise the robot will close the issue after merging the PR :) 
author_association: CONTRIBUTOR
user: login: mwysokin
created_at: 2024-08-01T11:56:06Z
body: /assign
author_association: CONTRIBUTOR
user: login: mimowo
created_at: 2024-08-02T08:06:04Z
body: /ok-to-test
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2024-08-08T11:53:15Z
body: LGTM label has been added.  <details>Git tree hash: 4b4f2f1108238ec7b0edef46b807fe0cc471a4d8</details>
author_association: CONTRIBUTOR
user: login: PBundyra
created_at: 2024-08-08T12:33:03Z
body: LGTM from my side as well
author_association: CONTRIBUTOR
user: login: tenzen-y
created_at: 2024-08-08T14:59:52Z
body: > /lgtm /approve /hold To wait for feedback from @tenzen-y as another pair of eyes.

ACK
author_association: MEMBER
user: login: tenzen-y
created_at: 2024-08-08T15:00:29Z
body: /remove-kind api-change

author_association: MEMBER
user: login: alculquicondor
created_at: 2024-08-13T17:28:25Z
body: /assign
author_association: CONTRIBUTOR
user: login: mwysokin
created_at: 2024-08-26T05:33:18Z
body: FYI I just wanted to let everyone know that I didn't abandon the work. I've been on vacation last week and I'll be back in September. Sorry for the delay 🙇 
author_association: CONTRIBUTOR
user: login: tenzen-y
created_at: 2024-08-26T22:52:15Z
body: > FYI I just wanted to let everyone know that I didn't abandon the work. I've been on vacation last week and I'll be back in September. Sorry for the delay 🙇

No worries. I'm looking forward to moving this discussion after you come back here!

author_association: MEMBER
user: login: mwysokin
created_at: 2024-09-27T08:59:09Z
body: I'm really sorry it's been dragging. I'm coming back to it starting Monday.
author_association: CONTRIBUTOR
user: login: linux-foundation-easycla[bot]
created_at: 2025-04-28T09:03:37Z
body: <a href="https://easycla.lfx.linuxfoundation.org/#/?version=2"><img src="https://s3.amazonaws.com/cla-project-logo-prod/cla-signed.svg" alt="CLA Signed" align="left" height="28" width="328" ></a><br/><br />The committers listed above are authorized under a signed CLA.<ul><li>:white_check_mark: login: mykysha / name: Mykyta Derhunov  (d1e4e76905b765fe0397c68ae30233e67cea1a84, 82e7be11a5cd9a715d73e2aa3753fba1f0870415, b21a063601194bc79fcf8b06e0a094997ad79777, ac8d003b44c4df2ebf8a088929a3475ec2492a92)</li><li>:white_check_mark: login: mwysokin / name: Michal Wysokinski  (181714a38147f854a54f73733f8dbb53611d4c50, 17df6131af0ef9702bd487cbacaeecf9a8969927, 5c39c83020669a7c9d74396d38f91ac29ac13bbb, c8d89750ad68ba072672a6cfff6b86de9d70f75f, 2490db6ee9c68468e8fdbb903763f1f7441f478c, cfd2b5b4d0c5edf64a50b89543c6ea66b50d4805, 7b48ab0d0ded9fd5e3741ac96ed2f7e529f164bd, 6ab3c67cf75d1dd2a702a862344847e750d5cea8, 682eae96658356dffe906dc453188f442d75265e, 39aa6ca365075d7cf2d47e493a907f9300d23414, c728b55c9232604a15ec9554d17262a6426ebaee, 5da02bc9544a1e999b37ce59102396dc8140b608, 2281c1f23db1009ad34a4535319c14269e9c1d33, 306e10465ea9d348251e50f47b8b35f337069692, a49d479c03fed977983d0d99448ec1d12053ffc7)</li></ul><!-- Date Modified: 2025-05-08 11:46:55.865036 -->
author_association: NONE
user: login: mykysha
created_at: 2025-04-28T09:18:34Z
body: /retest
author_association: CONTRIBUTOR
user: login: mbobrovskyi
created_at: 2025-05-02T08:34:59Z
body: @tenzen-y @mimowo could you please take a look?
author_association: CONTRIBUTOR
user: login: mykysha
created_at: 2025-05-05T08:32:04Z
body: Hi @tenzen-y , could you please take a look at the changes here? Should have fixed existing comments, as well as added API changes in regards to the #4471 
author_association: CONTRIBUTOR
user: login: mimowo
created_at: 2025-05-07T11:55:45Z
body: /remove-kind api-change
the kep by itself does not change api

author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-05-07T11:55:48Z
body: @mimowo: Those labels are not set on the issue: `kind/api-change`

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/pull/2742#issuecomment-2858299692):

>/remove-kind api-change
>the kep by itself does not change api
>


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: CONTRIBUTOR
user: login: mimowo
created_at: 2025-05-08T11:57:39Z
body: /lgtm
/approve
I believe all remarks are addressed, thank you
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-05-08T11:57:45Z
body: LGTM label has been added.  <details>Git tree hash: 647912b515c32db1db1776973046a29339e87672</details>
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-05-08T11:57:48Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/kueue/pull/2742#issuecomment-2862779759" title="Approved">mimowo</a>*, *<a href="https://github.com/kubernetes-sigs/kueue/pull/2742#" title="Author self-approved">mwysokin</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkueue).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/kueue/blob/main/OWNERS)~~ [mimowo]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: mbobrovskyi
created_at: 2025-05-08T12:02:24Z
body: /unhold
author_association: CONTRIBUTOR