Repository: kubernetes-sigs/kueue

Issue:
Unable to install kueue v0.12.0 in GKE
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


**What happened**:

When installing or upgrading to kueue v0.12.0 in GKE standard or autopilot, the following error is returned

```
Error from server (Forbidden): admission webhook "flowcontrol-guardrails.common-webhooks.networking.gke.io" denied the request: can't recognize the priority level configuration "kueue-visibility" used within flow-schema "kueue-visibility
```

**What you expected to happen**:

No error should happen

**How to reproduce it (as minimally and precisely as possible)**:

- Create a GKE cluster (standard or autopilot)
- Obtain the credentials for the newly created GKE cluster
- `kubectl apply -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.12.0/manifests.yaml`

**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`): 1.31.x
- Kueue version (use `git describe --tags --dirty --always`): v0.12.0
- Cloud provider or hardware configuration: GKE
- OS (e.g: `cat /etc/os-release`):
- Kernel (e.g. `uname -a`):
- Install tools:
- Others:

Discussion:
@mimowo: cc @mwysokin @mbobrovskyi @tenzen-y 
Some details are on the slack discussion https://kubernetes.slack.com/archives/C032ZE66A2X/p1748328823580939@mimowo: I think if this is problematic to fix then I would suggest to make the configuration opt-in,  basically revert https://github.com/kubernetes-sigs/kueue/pull/5043@mimowo: I have also reproduced this on my cluster@mimowo: cc @gabesaba @tenzen-y: > I think if this is problematic to fix then I would suggest to make the configuration opt-in, basically revert [#5043](https://github.com/kubernetes-sigs/kueue/pull/5043)

I discussed with @mimowo offline.
In conclusion, we agreed to revert https://github.com/kubernetes-sigs/kueue/pull/5043.
However, we should mention the recommendation to install APF manifests in our documentation when they want to enable Visibility On-Demand.
If they do not install them, the visibility requests could occupy the API Server global seats, which could lead to cluster disruption in some situations.


Solution patch:
diff --git a/Makefile b/Makefile
index 99fe4343d..61d2b8d79 100644
--- a/Makefile
+++ b/Makefile
@@ -313,6 +313,7 @@ artifacts: kustomize yq helm ## Generate release artifacts.
 	$(KUSTOMIZE) build config/dev -o artifacts/manifests-dev.yaml
 	$(KUSTOMIZE) build config/alpha-enabled -o artifacts/manifests-alpha-enabled.yaml
 	$(KUSTOMIZE) build config/prometheus -o artifacts/prometheus.yaml
+	$(KUSTOMIZE) build config/visibility-apf -o artifacts/visibility-apf.yaml
 	$(KUSTOMIZE) build config/kueueviz -o artifacts/kueueviz.yaml
 	@$(call clean-manifests)
 	@$(call clean-kueueviz-manifests)
diff --git a/charts/kueue/README.md b/charts/kueue/README.md
index 2c4719dea..e25ce3997 100644
--- a/charts/kueue/README.md
+++ b/charts/kueue/README.md
@@ -73,6 +73,7 @@ The following table lists the configurable parameters of the kueue chart and the
 | `fullnameOverride`                                     | override the resource name                             | ``                                          |
 | `enablePrometheus`                                     | enable Prometheus                                      | `false`                                     |
 | `enableCertManager`                                    | enable CertManager                                     | `false`                                     |
+| `enableVisibilityAPF`                                  | enable APF for the visibility API                      | `false`                                     |
 | `enableKueueViz`                                       | enable KueueViz dashboard                              | `false`                                     |
 | `KueueViz.backend.image`                               | KueueViz dashboard backend image                       | `us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueueviz-backend:main` |
 | `KueueViz.frontend.image`                              | KueueViz dashboard frontend image                      | `us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueueviz-frontend:main` |
diff --git a/charts/kueue/templates/visibility/flowschema.yaml b/charts/kueue/templates/visibility-apf/flowschema.yaml
similarity index 93%
rename from charts/kueue/templates/visibility/flowschema.yaml
rename to charts/kueue/templates/visibility-apf/flowschema.yaml
index 18b2e8010..d43a31a72 100644
--- a/charts/kueue/templates/visibility/flowschema.yaml
+++ b/charts/kueue/templates/visibility-apf/flowschema.yaml
@@ -1,3 +1,4 @@
+{{- if .Values.enableVisibilityAPF }}
 apiVersion: flowcontrol.apiserver.k8s.io/v1
 kind: FlowSchema
 metadata:
@@ -29,3 +30,4 @@ spec:
         - group:
             name: system:authenticated
           kind: Group
+{{- end }}
diff --git a/charts/kueue/templates/visibility/prioritylevelconfigurations.yaml b/charts/kueue/templates/visibility-apf/prioritylevelconfigurations.yaml
similarity index 90%
rename from charts/kueue/templates/visibility/prioritylevelconfigurations.yaml
rename to charts/kueue/templates/visibility-apf/prioritylevelconfigurations.yaml
index 970c73dc6..1175cae2f 100644
--- a/charts/kueue/templates/visibility/prioritylevelconfigurations.yaml
+++ b/charts/kueue/templates/visibility-apf/prioritylevelconfigurations.yaml
@@ -1,3 +1,4 @@
+{{- if .Values.enableVisibilityAPF }}
 apiVersion: flowcontrol.apiserver.k8s.io/v1
 kind: PriorityLevelConfiguration
 metadata:
@@ -16,3 +17,4 @@ spec:
       type: Queue
     nominalConcurrencyShares: 10
   type: Limited
+{{- end }}
diff --git a/charts/kueue/values.yaml b/charts/kueue/values.yaml
index 688009bc3..e570fe503 100644
--- a/charts/kueue/values.yaml
+++ b/charts/kueue/values.yaml
@@ -7,6 +7,8 @@ fullnameOverride: ""
 enablePrometheus: false
 # Enable x509 automated certificate management using cert-manager (cert-manager.io)
 enableCertManager: false
+# Enable API Priority and Fairness configuration for the visibility API
+enableVisibilityAPF: false
 # Customize controllerManager
 controllerManager:
   #featureGates:
diff --git a/config/components/visibility/flowschema.yaml b/config/components/visibility-apf/flowschema.yaml
similarity index 100%
rename from config/components/visibility/flowschema.yaml
rename to config/components/visibility-apf/flowschema.yaml
diff --git a/config/components/visibility-apf/kustomization.yaml b/config/components/visibility-apf/kustomization.yaml
new file mode 100644
index 000000000..2ab67922c
--- /dev/null
+++ b/config/components/visibility-apf/kustomization.yaml
@@ -0,0 +1,3 @@
+resources:
+- prioritylevelconfigurations.yaml
+- flowschema.yaml
diff --git a/config/components/visibility/prioritylevelconfigurations.yaml b/config/components/visibility-apf/prioritylevelconfigurations.yaml
similarity index 100%
rename from config/components/visibility/prioritylevelconfigurations.yaml
rename to config/components/visibility-apf/prioritylevelconfigurations.yaml
diff --git a/config/components/visibility/kustomization.yaml b/config/components/visibility/kustomization.yaml
index 23c787354..cd29a9935 100644
--- a/config/components/visibility/kustomization.yaml
+++ b/config/components/visibility/kustomization.yaml
@@ -2,5 +2,3 @@ resources:
 - apiservice_v1beta1.yaml
 - role_binding.yaml
 - service.yaml
-- flowschema.yaml
-- prioritylevelconfigurations.yaml
diff --git a/config/visibility-apf/kustomization.yaml b/config/visibility-apf/kustomization.yaml
new file mode 100644
index 000000000..58f2aade5
--- /dev/null
+++ b/config/visibility-apf/kustomization.yaml
@@ -0,0 +1,14 @@
+# This overlay builds the APF components to be used in combination
+# with other overlays.
+
+namespace: kueue-system
+namePrefix: kueue-
+
+labels:
+  - pairs:
+      app.kubernetes.io/name: kueue
+      app.kubernetes.io/component: controller
+      control-plane: controller-manager
+
+resources:
+  - ../components/visibility-apf/
diff --git a/hack/processing-plan.yaml b/hack/processing-plan.yaml
index 7260558ca..089643d5b 100644
--- a/hack/processing-plan.yaml
+++ b/hack/processing-plan.yaml
@@ -103,6 +103,44 @@ files:
           {{- include "kueue.selectorLabels" . | nindent 4 }}
         indentation: 2
         onFileCondition: '.kind == "Service"'
+  - path: ./config/components/visibility-apf/*.yaml
+    outputDir: ./charts/kueue/templates/visibility-apf
+    excludes:
+      - kustomization.yaml
+    continueOnError: true
+    operations:
+      - type: APPEND
+        key: .metadata.name
+        value: '"{{ include \"kueue.fullname\" . }}-"'
+        onFileCondition: '.metadata.name != "v1beta*"'
+      - type: UPDATE
+        key: .metadata.namespace
+        value: '"{{ .Release.Namespace }}"'
+        onFileCondition: '.metadata.namespace != "kube-system"'
+      - type: APPEND
+        key: .spec.service.name
+        value: '"{{ include \"kueue.fullname\" . }}-"'
+      - type: UPDATE
+        key: .spec.service.namespace
+        value: '"{{ .Release.Namespace }}"'
+      - type: UPDATE
+        key: .subjects.[].namespace
+        value: '"{{ .Release.Namespace }}"'
+    postOperations:
+      - type: INSERT_TEXT
+        key: .metadata
+        value: |
+          labels:
+          {{- include "kueue.labels" . | nindent 4 }}
+        indentation: 2
+      - type: INSERT_TEXT
+        position: START
+        value: |
+          {{- if .Values.enableVisibilityAPF }}
+      - type: INSERT_TEXT
+        position: END
+        value: |
+          {{- end }}
   - path: ./config/components/webhook/service.yaml
     outputDir: ./charts/kueue/templates/webhook
     continueOnError: true
diff --git a/site/content/en/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand.md b/site/content/en/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand.md
index e635e60f5..434b05f9f 100644
--- a/site/content/en/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand.md
+++ b/site/content/en/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand.md
@@ -22,6 +22,14 @@ Make sure the following conditions are met:
 - The kubectl command-line tool has communication with your cluster.
 - [Kueue is installed](/docs/installation) in version v0.6.0 or later.
 
+### Configure API Priority and Fairness:
+
+To install the [API Priority and Fairness](https://kubernetes.io/docs/concepts/cluster-administration/flow-control/) configuration for the visibility API apply manifests:
+
+```shell
+kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/{{< param "version" >}}/visibility-apf.yaml
+```
+
 ### Directly accessing the Visibility API
 
 If you want to directly access the Visibility API with a http client like 



Test patch:
diff --git a/test/e2e/config/default/kustomization.yaml b/test/e2e/config/default/kustomization.yaml
index dcf86970a..11a455055 100644
--- a/test/e2e/config/default/kustomization.yaml
+++ b/test/e2e/config/default/kustomization.yaml
@@ -3,6 +3,7 @@ kind: Kustomization
 
 resources:
 - ../../../../config/dev
+- ../../../../config/visibility-apf
 
 replicas:
 - name: kueue-controller-manager