Repository: kubernetes-sigs/kueue

Issue:
Flaky E2E test: MultiKueue when Creating a multikueue admission check Should run a RayCluster on worker if admitted
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


**What happened**:
End To End MultiKueue Suite: kindest/node:v1.32.3: [It] MultiKueue when Creating a multikueue admission check Should run a RayCluster on worker if admitted

```
{Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/multikueue/e2e_test.go:828 with:
Expected
    <int32>: 0
to equal
    <int32>: 1 failed [FAILED] Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/multikueue/e2e_test.go:828 with:
Expected
    <int32>: 0
to equal
    <int32>: 1
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/multikueue/e2e_test.go:830 @ 04/01/25 07:23:48.388
}
```

**What you expected to happen**:
No errors

**How to reproduce it (as minimally and precisely as possible)**:
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4846/pull-kueue-test-e2e-multikueue-main/1906965243951583232

**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`):
- Kueue version (use `git describe --tags --dirty --always`):
- Cloud provider or hardware configuration:
- OS (e.g: `cat /etc/os-release`):
- Kernel (e.g. `uname -a`):
- Install tools:
- Others:

Hints from issue comments:
@mbobrovskyi: /kind flake@mbobrovskyi: cc: @mszadkow @KPostOffice: The `Readiness` probe succeeds [here](https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4846/pull-kueue-test-e2e-multikueue-main/1906965243951583232/artifacts/run-test-multikueue-e2e-1.32.3/kind-worker1-control-plane/kubelet.log#:~:text=Apr%2001%2007%3A23%3A47%20kind%2Dworker1%2Dcontrol%2Dplane%20kubelet%5B775%5D%3A%20I0401%2007%3A23%3A47.744698%20%20%20%20%20775%20prober.go%3A116%5D%20%22Probe%20succeeded%22%20probeType%3D%22Readiness%22%20pod%3D%22multikueue%2D2db8t/raycluster1%2Dworkers%2Dgroup%2D0%2Dworker%2Dt94gd%22%20podUID%3D%221debb19f%2D6977%2D4287%2Da787%2Db9f3f2cf7ef2%22%20containerName%3D%22worker%2Dcontainer%22) in the Kubelet logs@mimowo: Would waiting a bit longer help? Maybe we have a case for increasing the CPU to 12 actually. I think we set 10 before introducing Ray and AppWrapper operators, but it would be good to see it we are CPU constrained by the dashboard@mbobrovskyi: It happens again https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5013/pull-kueue-test-e2e-multikueue-main/1912548758457946112.@mszadkow: I found one more failed run here: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4917/pull-kueue-test-e2e-multikueue-main/1912169469686321152

I will investigate
/assign@mszadkow: 1. That was with the `rayproject/ray:2.9.0`. 
Now it's `ray-mini`, which allows to get up the ray cluster worker pods much faster. 
2. I would still increase timeout here from `LongTimeout` to `VeryLongTimeout` as only this one doesn't have extended timeout like other test.
Which makes perfect sense as the full ray image is still be used in `periodic` tests.


Solution patch:
diff --git a/charts/kueue/templates/rbac/topology_editor_role.yaml b/charts/kueue/templates/rbac/topology_editor_role.yaml
new file mode 100644
index 000000000..a24c6343f
--- /dev/null
+++ b/charts/kueue/templates/rbac/topology_editor_role.yaml
@@ -0,0 +1,21 @@
+# permissions for end users to edit resourceflavors.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: '{{ include "kueue.fullname" . }}-topology-editor-role'
+  labels:
+  {{- include "kueue.labels" . | nindent 4 }}
+    rbac.kueue.x-k8s.io/batch-admin: "true"
+rules:
+  - apiGroups:
+      - kueue.x-k8s.io
+    resources:
+      - topology
+    verbs:
+      - create
+      - delete
+      - get
+      - list
+      - patch
+      - update
+      - watch
diff --git a/charts/kueue/templates/rbac/topology_viewer_role.yaml b/charts/kueue/templates/rbac/topology_viewer_role.yaml
new file mode 100644
index 000000000..818660e76
--- /dev/null
+++ b/charts/kueue/templates/rbac/topology_viewer_role.yaml
@@ -0,0 +1,16 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: '{{ include "kueue.fullname" . }}-topology-viewer-role'
+  labels:
+  {{- include "kueue.labels" . | nindent 4 }}
+    rbac.kueue.x-k8s.io/batch-admin: "true"
+rules:
+  - apiGroups:
+      - kueue.x-k8s.io
+    resources:
+      - topology
+    verbs:
+      - get
+      - list
+      - watch
diff --git a/config/alpha-enabled/kustomization.yaml b/config/alpha-enabled/kustomization.yaml
index 21e4b163d..1babb5144 100644
--- a/config/alpha-enabled/kustomization.yaml
+++ b/config/alpha-enabled/kustomization.yaml
@@ -3,6 +3,8 @@
 # Use default settings as a base.
 resources:
 - ../default
+- topology_editor_role.yaml
+- topology_viewer_role.yaml
 
 patches:
 # Modify feature gates to enable AllAlpha=true
diff --git a/config/components/rbac/kustomization.yaml b/config/components/rbac/kustomization.yaml
index 5c17be6f0..9fec0f16e 100644
--- a/config/components/rbac/kustomization.yaml
+++ b/config/components/rbac/kustomization.yaml
@@ -29,6 +29,8 @@ resources:
 - resourceflavor_viewer_role.yaml
 - pending_workloads_cq_viewer_role.yaml
 - pending_workloads_lq_viewer_role.yaml
+- topology_editor_role.yaml
+- topology_viewer_role.yaml
 - workload_editor_role.yaml
 - workload_viewer_role.yaml
 
diff --git a/config/components/rbac/topology_editor_role.yaml b/config/components/rbac/topology_editor_role.yaml
new file mode 100644
index 000000000..7315e5a2a
--- /dev/null
+++ b/config/components/rbac/topology_editor_role.yaml
@@ -0,0 +1,20 @@
+# permissions for end users to edit resourceflavors.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: topology-editor-role
+  labels:
+    rbac.kueue.x-k8s.io/batch-admin: "true"
+rules:
+- apiGroups:
+  - kueue.x-k8s.io
+  resources:
+  - topology
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
diff --git a/config/components/rbac/topology_viewer_role.yaml b/config/components/rbac/topology_viewer_role.yaml
new file mode 100644
index 000000000..50d2a72fd
--- /dev/null
+++ b/config/components/rbac/topology_viewer_role.yaml
@@ -0,0 +1,15 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: topology-viewer-role
+  labels:
+    rbac.kueue.x-k8s.io/batch-admin: "true"
+rules:
+- apiGroups:
+  - kueue.x-k8s.io
+  resources:
+  - topology
+  verbs:
+  - get
+  - list
+  - watch
diff --git a/hack/e2e-common.sh b/hack/e2e-common.sh
index ef9b01ef3..e5703b165 100644
--- a/hack/e2e-common.sh
+++ b/hack/e2e-common.sh
@@ -134,7 +134,16 @@ function cluster_kind_load {
 # $1 cluster
 # $2 image
 function cluster_kind_load_image {
-    $KIND load docker-image "$2" --name "$1"
+    # check if the command to get worker nodes could succeeded
+    if ! $KIND get nodes --name "$1" > /dev/null 2>&1; then
+        echo "Failed to retrieve nodes for cluster '$1'."
+        return 1
+    fi
+    # filter out 'control-plane' node, use only worker nodes to load image
+    worker_nodes=$($KIND get nodes --name "$1" | grep -v 'control-plane' | paste -sd "," -)
+    if [[ -n "$worker_nodes" ]]; then
+        $KIND load docker-image "$2" --name "$1" --nodes "$worker_nodes"
+    fi
 }
 
 # Wait until all cert-manager deployments are available.
diff --git a/hack/multikueue/manager-cluster.kind.yaml b/hack/multikueue/manager-cluster.kind.yaml
index 3a21a4af1..8962e9364 100644
--- a/hack/multikueue/manager-cluster.kind.yaml
+++ b/hack/multikueue/manager-cluster.kind.yaml
@@ -18,6 +18,16 @@ nodes:
         v: "3"
   - |
     kind: InitConfiguration
+    nodeRegistration:
+      kubeletExtraArgs:
+        v: "3"
+- role: worker
+  labels:
+    instance-type: on-demand
+
+kubeadmConfigPatches:
+  - |
+    kind: JoinConfiguration
     nodeRegistration:
       kubeletExtraArgs:
         v: "3"
\ No newline at end of file
diff --git a/hack/multikueue/worker-cluster.kind.yaml b/hack/multikueue/worker-cluster.kind.yaml
index 3a21a4af1..8962e9364 100644
--- a/hack/multikueue/worker-cluster.kind.yaml
+++ b/hack/multikueue/worker-cluster.kind.yaml
@@ -18,6 +18,16 @@ nodes:
         v: "3"
   - |
     kind: InitConfiguration
+    nodeRegistration:
+      kubeletExtraArgs:
+        v: "3"
+- role: worker
+  labels:
+    instance-type: on-demand
+
+kubeadmConfigPatches:
+  - |
+    kind: JoinConfiguration
     nodeRegistration:
       kubeletExtraArgs:
         v: "3"
\ No newline at end of file


Test patch:
diff --git a/test/e2e/customconfigs/skipjobswithoutqueuename_test.go b/test/e2e/customconfigs/skipjobswithoutqueuename_test.go
index 013abc0b0..9109a9b80 100644
--- a/test/e2e/customconfigs/skipjobswithoutqueuename_test.go
+++ b/test/e2e/customconfigs/skipjobswithoutqueuename_test.go
@@ -17,8 +17,6 @@ limitations under the License.
 package queuename
 
 import (
-	"time"
-
 	"github.com/onsi/ginkgo/v2"
 	"github.com/onsi/gomega"
 	awv1beta2 "github.com/project-codeflare/appwrapper/api/v1beta2"
@@ -29,6 +27,7 @@ import (
 	"k8s.io/utils/ptr"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
+	config "sigs.k8s.io/kueue/apis/config/v1beta1"
 	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta1"
 	"sigs.k8s.io/kueue/pkg/constants"
 	workloadjob "sigs.k8s.io/kueue/pkg/controller/jobs/job"
@@ -52,12 +51,9 @@ var _ = ginkgo.Describe("ManageJobsWithoutQueueName", ginkgo.Ordered, func() {
 	)
 
 	ginkgo.BeforeAll(func() {
-		configurationUpdate := time.Now()
-		config := defaultKueueCfg.DeepCopy()
-		config.ManageJobsWithoutQueueName = true
-		util.ApplyKueueConfiguration(ctx, k8sClient, config)
-		util.RestartKueueController(ctx, k8sClient)
-		ginkgo.GinkgoLogr.Info("Kueue configuration updated", "took", time.Since(configurationUpdate))
+		updateKueueConfiguration(func(cfg *config.Configuration) {
+			cfg.ManageJobsWithoutQueueName = true
+		})
 	})
 
 	ginkgo.BeforeEach(func() {
diff --git a/test/e2e/customconfigs/suite_test.go b/test/e2e/customconfigs/suite_test.go
index 1d84cb7a8..ed1fd2406 100644
--- a/test/e2e/customconfigs/suite_test.go
+++ b/test/e2e/customconfigs/suite_test.go
@@ -67,3 +67,12 @@ var _ = ginkgo.AfterSuite(func() {
 	util.RestartKueueController(ctx, k8sClient)
 	ginkgo.GinkgoLogr.Info("Default Kueue configuration restored")
 })
+
+func updateKueueConfiguration(applyChanges func(cfg *v1beta1.Configuration)) {
+	configurationUpdate := time.Now()
+	config := defaultKueueCfg.DeepCopy()
+	applyChanges(config)
+	util.ApplyKueueConfiguration(ctx, k8sClient, config)
+	util.RestartKueueController(ctx, k8sClient)
+	ginkgo.GinkgoLogr.Info("Kueue configuration updated", "took", time.Since(configurationUpdate))
+}