Repository: kubernetes-sigs/kueue

Issue:
Merge the FlowSchema and PriorityLevelConfiguration v1 manifests into the installation manifests
<!-- Please only use this template for submitting enhancement requests -->

**What would you like to be added**:

As we discussed [here](https://github.com/kubernetes-sigs/kueue/pull/3043#discussion_r1764680475) and [here](https://github.com/kubernetes-sigs/kueue/pull/3043#discussion_r1764915102) we should merge the FlowSchema and PriorityLevelConfiguration v1 manifests into the installation manifests and remove v1beta3 manifests once the latest Kubernetes version (1.28) no longer supports v1beta3.

**Why is this needed**:

 - During the second iteration of [Graduation criteria](https://github.com/kubernetes-sigs/kueue/tree/3a428360818bab0f1618edcbdea26dd94b4ca6e5/keps/168-2-pending-workloads-visibility#beta) of visibility we need to merge manifests.

 - Due to flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema is deprecated in v1.29+ and unavailable in v1.32+ we don't need to support v1beta3 anymore.

**Completion requirements**:

This enhancement requires the following artifacts:

- [ ] Design doc
- [ ] API change
- [ ] Docs update

The artifacts should be linked in subsequent comments.


Discussion:
@mimowo: Question regarding the support: https://github.com/kubernetes-sigs/kueue/pull/3043#discussion_r1760626660. I'm wondering what happens if 1.32 is released, so we release 0.10 Kueue according to the plan. IIUC this will complicate deployment of the latest Kueue on 1.31 which will be still used by majority of clusters at that point. @tenzen-y: @mbobrovskyi Could you update this issue to align with the latest decision?
@k8s-triage-robot: The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle stale`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/@mimowo: /remove-lifecycle stale@mimowo: As discussed in https://github.com/kubernetes-sigs/kueue/pull/3983#discussion_r1925026803 I prefer to keep the APF configuration as opt-in for a while longer after the last EOL release of Kubernetes. 

I'm ok some features of Kueue not working ideally on older versions, but not being to install at all is a flag,  because we claim Kueue can be at least installed on older versions (as in the main README: `A Kubernetes cluster with version 1.25 or newer is running. Learn how to [install the Kubernetes tools](https://kubernetes.io/docs/tasks/tools/).`. I would prefer to keep it that way, as the additional release will be very helpful for some users to migrate away for a version that just became EOL.

I propose to merge the configuration only when 1.30 gets EOL (+1 release). Then we update the README to claim Kueue installation is supported on 1.29+. 

Unless there are other ideas, or other factors at play.

Solution patch:
diff --git a/Makefile b/Makefile
index e093513a7..1f45d8394 100644
--- a/Makefile
+++ b/Makefile
@@ -266,8 +266,7 @@ artifacts: kustomize yq helm ## Generate release artifacts.
 	$(KUSTOMIZE) build config/dev -o artifacts/manifests-dev.yaml
 	$(KUSTOMIZE) build config/alpha-enabled -o artifacts/manifests-alpha-enabled.yaml
 	$(KUSTOMIZE) build config/prometheus -o artifacts/prometheus.yaml
-	$(KUSTOMIZE) build config/visibility-apf/default -o artifacts/visibility-apf.yaml
-	$(KUSTOMIZE) build config/visibility-apf/1_28 -o artifacts/visibility-apf-1-28.yaml
+	$(KUSTOMIZE) build config/visibility-apf -o artifacts/visibility-apf.yaml
 	@$(call clean-manifests)
 	# Update the image tag and policy
 	$(YQ)  e  '.controllerManager.manager.image.repository = "$(IMAGE_REPO)" | .controllerManager.manager.image.tag = "$(GIT_TAG)" | .controllerManager.manager.image.pullPolicy = "IfNotPresent"' -i charts/kueue/values.yaml
diff --git a/charts/kueue/templates/visibility-apf/flowschema.yaml b/charts/kueue/templates/visibility-apf/flowschema.yaml
index 7b5e90e28..d43a31a72 100644
--- a/charts/kueue/templates/visibility-apf/flowschema.yaml
+++ b/charts/kueue/templates/visibility-apf/flowschema.yaml
@@ -1,5 +1,5 @@
 {{- if .Values.enableVisibilityAPF }}
-apiVersion: flowcontrol.apiserver.k8s.io/{{ and (eq .Capabilities.KubeVersion.Major "1") (eq .Capabilities.KubeVersion.Minor "28") | ternary "v1beta3" "v1" }}
+apiVersion: flowcontrol.apiserver.k8s.io/v1
 kind: FlowSchema
 metadata:
   labels:
diff --git a/charts/kueue/templates/visibility-apf/prioritylevelconfigurations.yaml b/charts/kueue/templates/visibility-apf/prioritylevelconfigurations.yaml
index b855863c6..1175cae2f 100644
--- a/charts/kueue/templates/visibility-apf/prioritylevelconfigurations.yaml
+++ b/charts/kueue/templates/visibility-apf/prioritylevelconfigurations.yaml
@@ -1,5 +1,5 @@
 {{- if .Values.enableVisibilityAPF }}
-apiVersion: flowcontrol.apiserver.k8s.io/{{ and (eq .Capabilities.KubeVersion.Major "1") (eq .Capabilities.KubeVersion.Minor "28") | ternary "v1beta3" "v1" }}
+apiVersion: flowcontrol.apiserver.k8s.io/v1
 kind: PriorityLevelConfiguration
 metadata:
   labels:
diff --git a/config/visibility-apf/1_28/kustomization.yaml b/config/visibility-apf/1_28/kustomization.yaml
deleted file mode 100644
index 23832e5e1..000000000
--- a/config/visibility-apf/1_28/kustomization.yaml
+++ /dev/null
@@ -1,19 +0,0 @@
-# This overlay builds the APF components for Kubernetes version 1.28 to be used in combination
-# with other overlays.
-
-resources:
-- ../default/
-
-patches:
-  - target:
-      kind: FlowSchema
-    patch: |
-      - op: replace
-        path: /apiVersion
-        value: flowcontrol.apiserver.k8s.io/v1beta3
-  - target:
-      kind: PriorityLevelConfiguration
-    patch: |
-      - op: replace
-        path: /apiVersion
-        value: flowcontrol.apiserver.k8s.io/v1beta3
\ No newline at end of file
diff --git a/config/visibility-apf/default/kustomization.yaml b/config/visibility-apf/kustomization.yaml
similarity index 88%
rename from config/visibility-apf/default/kustomization.yaml
rename to config/visibility-apf/kustomization.yaml
index 914c81bdf..847f776a9 100644
--- a/config/visibility-apf/default/kustomization.yaml
+++ b/config/visibility-apf/kustomization.yaml
@@ -11,4 +11,4 @@ labels:
       control-plane: controller-manager
 
 resources:
-  - ../../components/visibility-apf/
\ No newline at end of file
+  - ../components/visibility-apf/
\ No newline at end of file
diff --git a/hack/e2e-common.sh b/hack/e2e-common.sh
index 1af6ba6f9..cf9bdc486 100644
--- a/hack/e2e-common.sh
+++ b/hack/e2e-common.sh
@@ -143,11 +143,7 @@ function cluster_kind_load_image {
 # $1 cluster
 function cluster_kueue_deploy {
     kubectl config use-context "kind-${1}"
-    if [ "${KIND_VERSION%.*}" = "1.28" ]; then
-        kubectl apply --server-side -k test/e2e/config/1_28
-    else
-        kubectl apply --server-side -k test/e2e/config/default
-    fi
+    kubectl apply --server-side -k test/e2e/config/default
 }
 
 #$1 - cluster name
diff --git a/hack/update-helm.sh b/hack/update-helm.sh
index 781102857..d557a1c7f 100755
--- a/hack/update-helm.sh
+++ b/hack/update-helm.sh
@@ -350,7 +350,6 @@ done
 
 # Replace flowcontrol version on visibility-apf directory
 for output_file in "${DEST_VISIBILITY_APF_DIR}"/*.yaml; do
-  $YQ -N -i '.apiVersion = "flowcontrol.apiserver.k8s.io/{{ and (eq .Capabilities.KubeVersion.Major \"1\") (eq .Capabilities.KubeVersion.Minor \"28\") | ternary \"v1beta3\" \"v1\" }}"' "$output_file"
   $YQ -N -i '.metadata.name |= "{{ include \"kueue.fullname\" . }}-" + .' "$output_file"
   $YQ -N -i '.metadata.namespace = "{{ .Release.Namespace }}"' "$output_file"
   $SED -i '/^metadata:.*/a\  labels:\n  {{- include "kueue.labels" . | nindent 4 }}' "$output_file"
diff --git a/site/content/en/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand.md b/site/content/en/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand.md
index 34369a3fd..434b05f9f 100644
--- a/site/content/en/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand.md
+++ b/site/content/en/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand.md
@@ -24,12 +24,11 @@ Make sure the following conditions are met:
 
 ### Configure API Priority and Fairness:
 
-To install the [API Priority and Fairness](https://kubernetes.io/docs/concepts/cluster-administration/flow-control/) configuration for the visibility API apply one of the manifests, depending on your Kubernetes version:
+To install the [API Priority and Fairness](https://kubernetes.io/docs/concepts/cluster-administration/flow-control/) configuration for the visibility API apply manifests:
 
-{{< tabpane lang="shell" persist=disabled >}}
-{{< tab header="Kubernetes 1.29 or newer" >}} kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/{{< param "version" >}}/visibility-apf.yaml {{< /tab >}}
-{{< tab header="Kubernetes 1.28" >}} kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/{{< param "version" >}}/visibility-apf-1-28.yaml {{< /tab >}}
-{{< /tabpane >}}
+```shell
+kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/{{< param "version" >}}/visibility-apf.yaml
+```
 
 ### Directly accessing the Visibility API
 



Test patch:
diff --git a/test/e2e/config/1_28/kustomization.yaml b/test/e2e/config/1_28/kustomization.yaml
deleted file mode 100644
index 90bbf9daa..000000000
--- a/test/e2e/config/1_28/kustomization.yaml
+++ /dev/null
@@ -1,3 +0,0 @@
-resources:
-- ../common
-- ../../../../config/visibility-apf/1_28
\ No newline at end of file
diff --git a/test/e2e/config/common/kustomization.yaml b/test/e2e/config/common/kustomization.yaml
deleted file mode 100644
index dcf86970a..000000000
--- a/test/e2e/config/common/kustomization.yaml
+++ /dev/null
@@ -1,25 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-
-resources:
-- ../../../../config/dev
-
-replicas:
-- name: kueue-controller-manager
-  count: 2
-
-patches:
-- path: manager_e2e_patch.yaml
-  target:
-    group: apps
-    version: v1
-    kind: Deployment
-    name: kueue-controller-manager
-    namespace: kueue-system
-
-configMapGenerator:
-- behavior: merge
-  files:
-  - controller_manager_config.yaml
-  name: kueue-manager-config
-  namespace: kueue-system
diff --git a/test/e2e/config/common/controller_manager_config.yaml b/test/e2e/config/default/controller_manager_config.yaml
similarity index 100%
rename from test/e2e/config/common/controller_manager_config.yaml
rename to test/e2e/config/default/controller_manager_config.yaml
diff --git a/test/e2e/config/default/kustomization.yaml b/test/e2e/config/default/kustomization.yaml
index 3c6314485..11a455055 100644
--- a/test/e2e/config/default/kustomization.yaml
+++ b/test/e2e/config/default/kustomization.yaml
@@ -1,3 +1,26 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
 resources:
-- ../common
-- ../../../../config/visibility-apf/default
\ No newline at end of file
+- ../../../../config/dev
+- ../../../../config/visibility-apf
+
+replicas:
+- name: kueue-controller-manager
+  count: 2
+
+patches:
+- path: manager_e2e_patch.yaml
+  target:
+    group: apps
+    version: v1
+    kind: Deployment
+    name: kueue-controller-manager
+    namespace: kueue-system
+
+configMapGenerator:
+- behavior: merge
+  files:
+  - controller_manager_config.yaml
+  name: kueue-manager-config
+  namespace: kueue-system
diff --git a/test/e2e/config/common/manager_e2e_patch.yaml b/test/e2e/config/default/manager_e2e_patch.yaml
similarity index 100%
rename from test/e2e/config/common/manager_e2e_patch.yaml
rename to test/e2e/config/default/manager_e2e_patch.yaml