Repository: kubernetes-sigs/kueue

Issue:
[flaky integration test] Preemption is enabled within ClusterQueue should preempt the low and mid priority workloads to fit the high-priority workload
/kind flake 

**What happened**:

failure on unrelated branch https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5394/pull-kueue-test-integration-baseline-main/1927649254336106496

**What you expected to happen**:
no failure
**How to reproduce it (as minimally and precisely as possible)**:
ci
**Anything else we need to know?**:

```
TopologyAwareScheduling Suite: [It] Topology Aware Scheduling when Single TAS Resource Flavor when Preemption is enabled within ClusterQueue should preempt the low and mid priority workloads to fit the high-priority workload expand_less	13s
{Timed out after 10.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/util/util.go:365 with:
Not enough workloads are admitted
Expected
    <int>: 1
to equal
    <int>: 2 failed [FAILED] Timed out after 10.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/util/util.go:365 with:
Not enough workloads are admitted
Expected
    <int>: 1
to equal
    <int>: 2
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/integration/singlecluster/tas/tas_test.go:1374 @ 05/28/25 09:06:47.971
}
```

Hints from issue comments:
@mimowo: cc @mbobrovskyi @mykysha ptal@tenzen-y: We observed the same failure on #5416, again.@mimowo: cc @mszadkow @mszadkow: /assign@mszadkow: Something very strange happened in that test run.
`wl1` got admitted and later it was verified, when `wl2` was evaluated for admission `wl1` disappeared without a trace...
What I mean is there was no `delete` or `preemption` recorded, maybe just this attempt to preempt `wl1` which meanwhile disappear.
 
```
2025-05-28T09:06:37.97242884Z	LEVEL(-2)	scheduler	scheduler/scheduler.go:226	Attempting to schedule workload	{"schedulingCycle": 38, "workload": {"name":"wl2","namespace":"tas-94dv5"}, "clusterQueue": {"name":"cluster-queue"}}
  2025-05-28T09:06:37.975342209Z	LEVEL(-2)	scheduler	scheduler/scheduler.go:572	Workload not admitted because it was deleted	{"schedulingCycle": 37, "workload": {"name":"wl1","namespace":"tas-7vrc8"}, "clusterQueue": {"name":"cluster-queue"}}
  2025-05-28T09:06:37.978364299Z	ERROR	scheduler	scheduler/scheduler.go:268	Failed to preempt workloads	{"schedulingCycle": 38, "workload": {"name":"wl2","namespace":"tas-94dv5"}, "clusterQueue": {"name":"cluster-queue"}, "error": "workloads.kueue.x-k8s.io \"wl1\" not found"}
  sigs.k8s.io/kueue/pkg/scheduler.(*Scheduler).schedule
  	/home/prow/go/src/sigs.k8s.io/kueue/pkg/scheduler/scheduler.go:268
  sigs.k8s.io/kueue/pkg/util/wait.untilWithBackoff.func1
  	/home/prow/go/src/sigs.k8s.io/kueue/pkg/util/wait/backoff.go:43
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
  	/home/prow/go/src/sigs.k8s.io/kueue/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233
  k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1
  	/home/prow/go/src/sigs.k8s.io/kueue/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:255
  k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext
  	/home/prow/go/src/sigs.k8s.io/kueue/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:256
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil
  	/home/prow/go/src/sigs.k8s.io/kueue/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233
  sigs.k8s.io/kueue/pkg/util/wait.untilWithBackoff
  	/home/prow/go/src/sigs.k8s.io/kueue/pkg/util/wait/backoff.go:42
  sigs.k8s.io/kueue/pkg/util/wait.UntilWithBackoff
  	/home/prow/go/src/sigs.k8s.io/kueue/pkg/util/wait/backoff.go:34
  2025-05-28T09:06:37.978562602Z	LEVEL(-3)	scheduler	scheduler/logging.go:43	Workload evaluated for admission	{"schedulingCycle": 38, "workload": {"name":"wl2","namespace":"tas-94dv5"}, "clusterQueue": {"name":"cluster-queue"}, "status": "", "reason": "couldn't assign flavors to pod set worker: topology \"default\" doesn't allow to fit any of 1 pod(s)"}
```@mimowo: Could you try to repro locally by looping the test under some load? Maybe we could capture it locally.@mszadkow: I have tried already, nothing happens, works smoothly.
But I will keep trying, I just wanted to share this "weird" thing that `wl1` namespace somehow got changed.
From `tas-94dv5` to `tas-7vrc8`, which is from the previous test case...

```
  2025-05-28T09:06:37.710423748Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:747	Workload update event	{"workload": {"name":"wl1","namespace":"tas-94dv5"}, "queue": "local-queue", "status": "admitted", "prevStatus": "pending", "clusterQueue": "cluster-queue"}
...
2025-05-28T09:06:37.975342209Z	LEVEL(-2)	scheduler	scheduler/scheduler.go:572	Workload not admitted because it was deleted	{"schedulingCycle": 37, "workload": {"name":"wl1","namespace":"tas-7vrc8"}, "clusterQueue": {"name":"cluster-queue"}}
```
My suspect is that we share lq and cq as pointers between tests...
But I would like to get repro anyway@mimowo: Thank you for investigating that. Indeed, I don't exclude the possibility of cross-test interaction, this would be hard to capture. Even if you are out of ideas for now, please think if we could improve the logging (either in tests) or in code which could be helpful next time.@mimowo: another recent occurence: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5433/pull-kueue-test-integration-baseline-release-0-12/1929436649175912448@mszadkow: Yes, I believe the fix is ready to be merged - https://github.com/kubernetes-sigs/kueue/pull/5423@mbobrovskyi: Looks like we still have an issue https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5443/pull-kueue-test-integration-baseline-main/1929489390048907264.@mbobrovskyi: /reopen@k8s-ci-robot: @mbobrovskyi: Reopened this issue.

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/5395#issuecomment-2930073887):

>/reopen


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@mbobrovskyi: One more

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5225/pull-kueue-test-integration-baseline-main/1929477214840033280@mszadkow: ack, looking into logs@mimowo: same here: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5440/pull-kueue-test-integration-baseline-main/1929495861822230528@mszadkow: The next suspect is the `queue manager` as it seems that `wl1` from the previous test is considered for preemption.
While it should be `wl1` from the current test and `wl1` from previous test should be removed from the queue already.
Investigating there...although still it's not possible to reproduce locally.@mimowo: Hmm, this might be a bug related to this code: https://github.com/kubernetes-sigs/kueue/blob/e3bbc66fa7c95df49668a33934e9ae9c97d9b65a/pkg/queue/manager.go#L793-L795

It might be the goroutine is spawned to add the workload after 1s and it does not consider the context is already cancelled when test ended.

Another fix might be to do not enqueue if the CQ no longer exists. 

To test this hypothesis locally maybe you could extend 1s to 2s.@tenzen-y: We observed this again.

https://github.com/kubernetes-sigs/kueue/pull/5432
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5432/pull-kueue-test-integration-baseline-main/1930110922383167488@tenzen-y: Once more:

- https://github.com/kubernetes-sigs/kueue/pull/5491
- https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5491/pull-kueue-test-integration-baseline-release-0-12/1930119799250096128

Solution patch:
diff --git a/.github/workflows/sync-dependabot.yaml b/.github/workflows/sync-dependabot.yaml
index b486eb5cc..41c3008e8 100644
--- a/.github/workflows/sync-dependabot.yaml
+++ b/.github/workflows/sync-dependabot.yaml
@@ -18,18 +18,29 @@ jobs:
             - uses: actions/checkout@v4
               with: 
                 ref: ${{github.head_ref}}
-            - name: sync hugo version
+            - name: Set git user from last commit
+              run: |
+                git config user.name "$(git log -1 --pretty=%an)"
+                git config user.email "$(git log -1 --pretty=%ae)"
+            - name: Sync hugo version
               if: ${{ contains(github.event.pull_request.title, 'hugo') }}
               run: |
-                  make sync-hugo-version
-            - name: commit and push the changes
+                make sync-hugo-version
+                if ! git diff --exit-code; then
+                  git add .
+                  git commit -m "Sync hugo version"
+                  git push
+                else
+                  echo "No changes to commit"
+                fi
+            - name: Sync controller-tools version
+              if: ${{ contains(github.event.pull_request.title, 'sigs.k8s.io/controller-tools') }}
               run: |
+                make manifests
                 if ! git diff --exit-code; then
-                      git config user.name "$(git log -1 --pretty=%an)"
-                      git config user.email "$(git log -1 --pretty=%ae)"
-                      git add .
-                      git commit -m "Sync hugo version"
-                      git push
+                  git add .
+                  git commit -m "Sync controller-tools version"
+                  git push
                 else
-                      echo "No changes to commit"
+                  echo "No changes to commit"
                 fi
diff --git a/Makefile b/Makefile
index 53e0f4bfb..c0ce35846 100644
--- a/Makefile
+++ b/Makefile
@@ -68,6 +68,8 @@ BASE_IMAGE ?= gcr.io/distroless/static:nonroot
 BUILDER_IMAGE ?= golang:$(GO_VERSION)
 CGO_ENABLED ?= 0
 
+YAML_PROCESSOR_LOG_LEVEL ?= info
+
 # Setting SHELL to bash allows bash commands to be executed by recipes.
 # This is a requirement for 'setup-envtest.sh' in the test target.
 # Options are set to exit when a recipe line exits non-zero or a piped command fails.
@@ -134,7 +136,7 @@ manifests: controller-gen ## Generate WebhookConfiguration, ClusterRole and Cust
 
 .PHONY: update-helm
 update-helm: manifests yq yaml-processor
-	$(PROJECT_DIR)/bin/yaml-processor hack/processing-plan.yaml
+	$(PROJECT_DIR)/bin/yaml-processor -zap-log-level=$(YAML_PROCESSOR_LOG_LEVEL) hack/processing-plan.yaml
 
 .PHONY: generate
 generate: gomod-download generate-apiref generate-code generate-kueuectl-docs
diff --git a/charts/kueue/templates/rbac/cohort_editor_role.yaml b/charts/kueue/templates/rbac/cohort_editor_role.yaml
new file mode 100644
index 000000000..91d6f9410
--- /dev/null
+++ b/charts/kueue/templates/rbac/cohort_editor_role.yaml
@@ -0,0 +1,21 @@
+# permissions for end users to edit cohorts.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: '{{ include "kueue.fullname" . }}-cohort-editor-role'
+  labels:
+  {{- include "kueue.labels" . | nindent 4 }}
+    rbac.kueue.x-k8s.io/batch-admin: "true"
+rules:
+  - apiGroups:
+      - kueue.x-k8s.io
+    resources:
+      - cohorts
+    verbs:
+      - create
+      - delete
+      - get
+      - list
+      - patch
+      - update
+      - watch
diff --git a/charts/kueue/templates/rbac/cohort_viewer_role.yaml b/charts/kueue/templates/rbac/cohort_viewer_role.yaml
new file mode 100644
index 000000000..745956508
--- /dev/null
+++ b/charts/kueue/templates/rbac/cohort_viewer_role.yaml
@@ -0,0 +1,17 @@
+# permissions for end users to view cohorts.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: '{{ include "kueue.fullname" . }}-cohort-viewer-role'
+  labels:
+  {{- include "kueue.labels" . | nindent 4 }}
+    rbac.kueue.x-k8s.io/batch-admin: "true"
+rules:
+  - apiGroups:
+      - kueue.x-k8s.io
+    resources:
+      - cohorts
+    verbs:
+      - get
+      - list
+      - watch
diff --git a/config/components/rbac/cohort_editor_role.yaml b/config/components/rbac/cohort_editor_role.yaml
new file mode 100644
index 000000000..0cf124b10
--- /dev/null
+++ b/config/components/rbac/cohort_editor_role.yaml
@@ -0,0 +1,20 @@
+# permissions for end users to edit cohorts.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: cohort-editor-role
+  labels:
+    rbac.kueue.x-k8s.io/batch-admin: "true"
+rules:
+- apiGroups:
+  - kueue.x-k8s.io
+  resources:
+  - cohorts
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
diff --git a/config/components/rbac/cohort_viewer_role.yaml b/config/components/rbac/cohort_viewer_role.yaml
new file mode 100644
index 000000000..08f5b29ab
--- /dev/null
+++ b/config/components/rbac/cohort_viewer_role.yaml
@@ -0,0 +1,16 @@
+# permissions for end users to view cohorts.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: cohort-viewer-role
+  labels:
+    rbac.kueue.x-k8s.io/batch-admin: "true"
+rules:
+- apiGroups:
+  - kueue.x-k8s.io
+  resources:
+  - cohorts
+  verbs:
+  - get
+  - list
+  - watch
diff --git a/config/components/rbac/kustomization.yaml b/config/components/rbac/kustomization.yaml
index edfdd05d7..14ec21bbb 100644
--- a/config/components/rbac/kustomization.yaml
+++ b/config/components/rbac/kustomization.yaml
@@ -33,6 +33,8 @@ resources:
 - topology_viewer_role.yaml
 - workload_editor_role.yaml
 - workload_viewer_role.yaml
+- cohort_editor_role.yaml
+- cohort_viewer_role.yaml
 
 # ClusterRoles for Kueue integrations
 - job_editor_role.yaml
diff --git a/hack/internal/tools/yaml-processor/main.go b/hack/internal/tools/yaml-processor/main.go
index fec97e19d..c32ea2740 100644
--- a/hack/internal/tools/yaml-processor/main.go
+++ b/hack/internal/tools/yaml-processor/main.go
@@ -18,6 +18,8 @@ package main
 
 import (
 	"errors"
+	"flag"
+	"fmt"
 	"log"
 	"os"
 	"syscall"
@@ -28,27 +30,30 @@ import (
 	"sigs.k8s.io/kueue/internal/tools/yaml-processor/yamlproc"
 )
 
+type options struct {
+	LogLevel       string
+	ProcessingPlan string
+}
+
 func main() {
-	logger, err := newLogger()
+	opts := parseOptions()
+
+	logger, err := newLogger(opts.LogLevel)
 	if err != nil {
 		log.Fatalf("Failed to initialize logger: %v", err)
 	}
 	defer func() {
 		// Sync logger and handle errors, ignoring specific non-critical errors for console outputs.
-		// Ignores syscall.ENOTTY (MacOS: "inappropriate ioctl for device") and syscall.EINVAL (Linux: "invalid argument")
-		// when syncing to non-file descriptors like /dev/stderr or /dev/stdout.
-		if err := logger.Sync(); err != nil && !errors.Is(err, syscall.ENOTTY) && !errors.Is(err, syscall.EINVAL) {
-			log.Printf("Error syncing logger: %v", err)
+		// Ignores syscall.ENOTTY (MacOS: "inappropriate ioctl for device"), syscall.EBADF (MacOS: "bad file descriptor")
+		// and syscall.EINVAL (Linux: "invalid argument") when syncing to non-file descriptors like
+		// /dev/stderr or /dev/stdout.
+		if err := logger.Sync(); err != nil && !errors.Is(err, syscall.ENOTTY) && !errors.Is(err, syscall.EBADF) && !errors.Is(err, syscall.EINVAL) {
+			log.Fatalf("Error syncing logger: %v", err)
 		}
 	}()
 	yamlproc.SetLogger(logger)
 
-	if len(os.Args) < 2 {
-		logger.Fatal("Usage: ./yaml-processor <processing-plan.yaml>")
-	}
-
-	planPath := os.Args[1]
-	processingPlan, err := yamlproc.LoadProcessingPlan(planPath)
+	processingPlan, err := yamlproc.LoadProcessingPlan(opts.ProcessingPlan)
 	if err != nil {
 		logger.Fatal("Failed to load processing plan", zap.Error(err))
 	}
@@ -60,8 +65,34 @@ func main() {
 	fileProcessor.ProcessPlan(*processingPlan)
 }
 
-func newLogger() (*zap.Logger, error) {
+func parseOptions() *options {
+	opts := &options{}
+
+	flag.StringVar(&opts.LogLevel, "zap-log-level", "info", "Minimum enabled logging level")
+	flag.Usage = func() {
+		fmt.Fprintf(flag.CommandLine.Output(), "Usage: yaml-processor <processing-plan.yaml>\n")
+		flag.PrintDefaults()
+	}
+
+	flag.Parse()
+	if flag.NArg() != 1 {
+		flag.Usage()
+		os.Exit(1)
+	}
+
+	opts.ProcessingPlan = flag.Arg(0)
+
+	return opts
+}
+
+func newLogger(logLevel string) (*zap.Logger, error) {
+	atomicLevel, err := zap.ParseAtomicLevel(logLevel)
+	if err != nil {
+		return nil, err
+	}
+
 	loggerConfig := zap.NewProductionConfig()
+	loggerConfig.Level = atomicLevel
 	loggerConfig.Encoding = "console"
 	loggerConfig.EncoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
 
diff --git a/hack/internal/tools/yaml-processor/yamlproc/processor.go b/hack/internal/tools/yaml-processor/yamlproc/processor.go
index 141457bb2..7ec7c4e7c 100644
--- a/hack/internal/tools/yaml-processor/yamlproc/processor.go
+++ b/hack/internal/tools/yaml-processor/yamlproc/processor.go
@@ -89,10 +89,15 @@ func (fp *FileProcessor) ProcessPlan(plan ProcessingPlan) {
 		}
 
 		if len(matchedFiles) == 0 {
-			logger.Warn("No files matched for pattern", zap.String("pattern", file.Path))
+			logger.Debug("No files matched for pattern", zap.String("pattern", file.Path))
 		}
 
-		for _, filePath := range filterFiles(matchedFiles, file.Excludes) {
+		filteredFiles, err := filterFiles(matchedFiles, file.Excludes)
+		if err != nil {
+			logger.Fatal("Failed to filter files", zap.Strings("excludes", file.Excludes), zap.Error(err))
+		}
+
+		for _, filePath := range filteredFiles {
 			file.Path = filePath
 			err := fp.ProcessFile(file)
 			if err != nil && !file.ContinueOnError {
@@ -102,30 +107,34 @@ func (fp *FileProcessor) ProcessPlan(plan ProcessingPlan) {
 	}
 }
 
-func filterFiles(files, excludes []string) []string {
+func filterFiles(files, excludes []string) ([]string, error) {
 	var filtered []string
 	for _, file := range files {
-		if !isExcluded(file, excludes) {
+		excluded, err := isExcluded(file, excludes)
+		if err != nil {
+			return nil, err
+		}
+		if !excluded {
 			filtered = append(filtered, file)
 		}
 	}
-	return filtered
+	return filtered, nil
 }
 
-func isExcluded(file string, excludes []string) bool {
+func isExcluded(file string, excludes []string) (bool, error) {
 	base := filepath.Base(file)
 	for _, pattern := range excludes {
 		matched, err := filepath.Match(pattern, base)
 		if err != nil {
-			logger.Warn("Invalid exclude pattern", zap.String("pattern", file), zap.Error(err))
-			continue
+			logger.Debug("Invalid exclude pattern", zap.String("pattern", file), zap.Error(err))
+			return false, err
 		}
 
 		if matched {
-			return true
+			return true, nil
 		}
 	}
-	return false
+	return false, nil
 }
 
 func (fp *FileProcessor) ProcessFile(fileOps FileOperations) error {
@@ -192,7 +201,7 @@ func (fp *FileProcessor) processRegularOperations(data []byte, fileOps FileOpera
 
 	for _, op := range fileOps.Operations {
 		if err := fp.validateOperation(op, data); err != nil {
-			logger.Warn("Skipping operation", zap.String("operation", op.Type), zap.String("file", fileOps.Path), zap.Error(err))
+			logger.Debug("Skipping operation", zap.String("operation", op.Type), zap.String("file", fileOps.Path), zap.Error(err))
 			opErrors = append(opErrors, err)
 			continue
 		}
@@ -208,13 +217,13 @@ func (fp *FileProcessor) processRegularOperations(data []byte, fileOps FileOpera
 		case Delete:
 			data, err = fp.yq.DeleteKey(data, op.Key, op.OnItemCondition)
 		default:
-			logger.Warn("Unknown operation type", zap.String("operation", op.Type), zap.String("file", fileOps.Path))
+			logger.Debug("Unknown operation type", zap.String("operation", op.Type), zap.String("file", fileOps.Path))
 			opErrors = append(opErrors, fmt.Errorf("unknown operation type: %s", op.Type))
 			continue
 		}
 
 		if err != nil {
-			logger.Warn("Cannot apply operation", zap.String("operation", op.Type), zap.Error(err))
+			logger.Debug("Cannot apply operation", zap.String("operation", op.Type), zap.Error(err))
 			opErrors = append(opErrors, err)
 		}
 	}
@@ -229,7 +238,7 @@ func (fp *FileProcessor) processPostOperations(data []byte, fileOps FileOperatio
 	// Validate all post operations beforehand, as INSERT_TEXT operations can potentially lead to invalid YAML.
 	for _, op := range fileOps.PostOperations {
 		if err := fp.validatePostOperation(op, data); err != nil {
-			logger.Warn("Skipping post operation", zap.String("operation", op.Type), zap.String("file", fileOps.Path), zap.Error(err))
+			logger.Debug("Skipping post operation", zap.String("operation", op.Type), zap.String("file", fileOps.Path), zap.Error(err))
 			opErrors = append(opErrors, err)
 			continue
 		}
@@ -248,13 +257,13 @@ func (fp *FileProcessor) processPostOperations(data []byte, fileOps FileOperatio
 				OnItemCondition: op.OnItemCondition,
 			})
 		default:
-			logger.Warn("Unknown post operation type", zap.String("operation", op.Type), zap.String("file", fileOps.Path))
+			logger.Debug("Unknown post operation type", zap.String("operation", op.Type), zap.String("file", fileOps.Path))
 			opErrors = append(opErrors, fmt.Errorf("unknown post operation type: %s", op.Type))
 			continue
 		}
 
 		if err != nil {
-			logger.Warn("Cannot apply post operation", zap.String("operation", op.Type), zap.Error(err))
+			logger.Debug("Cannot apply post operation", zap.String("operation", op.Type), zap.Error(err))
 			opErrors = append(opErrors, err)
 		}
 	}
@@ -277,11 +286,21 @@ func (fp *FileProcessor) prepareFile(fileOps FileOperations) ([]byte, error) {
 }
 
 func (fp *FileProcessor) validateOperation(op Operation, data []byte) error {
-	if op.OnFileCondition != "" && !fp.yq.EvaluateCondition(data, op.OnFileCondition) {
+	met, err := fp.yq.EvaluateCondition(data, op.OnFileCondition)
+	if err != nil {
+		return err
+	}
+
+	if op.OnFileCondition != "" && !met {
 		return fmt.Errorf("condition '%s' not met", op.OnFileCondition)
 	}
 
-	if !fp.yq.HasKey(data, op.Key) && !op.AddKeyIfMissing {
+	exist, err := fp.yq.HasKey(data, op.Key)
+	if err != nil {
+		return err
+	}
+
+	if !exist && !op.AddKeyIfMissing {
 		return fmt.Errorf("key '%s' does not exist", op.Key)
 	}
 
@@ -289,7 +308,12 @@ func (fp *FileProcessor) validateOperation(op Operation, data []byte) error {
 }
 
 func (fp *FileProcessor) validatePostOperation(op Operation, data []byte) error {
-	if op.OnFileCondition != "" && !fp.yq.EvaluateCondition(data, op.OnFileCondition) {
+	met, err := fp.yq.EvaluateCondition(data, op.OnFileCondition)
+	if err != nil {
+		return err
+	}
+
+	if op.OnFileCondition != "" && !met {
 		return fmt.Errorf("condition '%s' not met", op.OnFileCondition)
 	}
 
diff --git a/hack/internal/tools/yaml-processor/yamlproc/yq.go b/hack/internal/tools/yaml-processor/yamlproc/yq.go
index 6c5a3b352..ed6545bea 100644
--- a/hack/internal/tools/yaml-processor/yamlproc/yq.go
+++ b/hack/internal/tools/yaml-processor/yamlproc/yq.go
@@ -96,7 +96,7 @@ func (yq *YQClient) buildSelectExpression(key, condition string) string {
 	return fmt.Sprintf("%s | select(%s) | %s", baseArrayKey, condition, remainingKey)
 }
 
-func (yq *YQClient) HasKey(data []byte, key string) bool {
+func (yq *YQClient) HasKey(data []byte, key string) (bool, error) {
 	if !strings.HasPrefix(key, ".") {
 		key = "." + key
 	}
@@ -104,13 +104,13 @@ func (yq *YQClient) HasKey(data []byte, key string) bool {
 	expression := yq.buildHasExpression(key)
 	out, err := yq.Evaluate(data, expression)
 	if err != nil {
-		logger.Warn("Cannot run yq expression", zap.String("expression", expression), zap.Error(err))
-		return false
+		logger.Debug("Cannot run yq expression", zap.String("expression", expression), zap.Error(err))
+		return false, err
 	}
 
 	val := strings.TrimSpace(string(out))
 
-	return slices.Contains(strings.Split(val, "\n"), "true")
+	return slices.Contains(strings.Split(val, "\n"), "true"), nil
 }
 
 func (yq *YQClient) buildHasExpression(key string) string {
@@ -127,14 +127,14 @@ func (yq *YQClient) buildHasExpression(key string) string {
 	return fmt.Sprintf(".%s | has(\"%s\")", prefix, lastElement)
 }
 
-func (yq *YQClient) EvaluateCondition(data []byte, condition string) bool {
+func (yq *YQClient) EvaluateCondition(data []byte, condition string) (bool, error) {
 	result, err := yq.Evaluate(data, condition)
 	if err != nil {
-		logger.Warn("Cannot evaluate condition", zap.String("condition", condition), zap.Error(err))
-		return false
+		logger.Debug("Cannot evaluate condition", zap.String("condition", condition), zap.Error(err))
+		return false, err
 	}
 
-	return strings.TrimSpace(string(result)) == "true"
+	return strings.TrimSpace(string(result)) == "true", nil
 }
 
 // FindKeyLines returns the zero-based line numbers where the specified key is located in the YAML.
diff --git a/pkg/controller/core/workload_controller.go b/pkg/controller/core/workload_controller.go
index e099c3635..9f45ca351 100644
--- a/pkg/controller/core/workload_controller.go
+++ b/pkg/controller/core/workload_controller.go
@@ -427,7 +427,8 @@ func (r *WorkloadReconciler) reconcileMaxExecutionTime(ctx context.Context, wl *
 	return 0, nil
 }
 
-// reconcileCheckBasedEviction returns true if Workload has been deactivated or evicted
+// reconcileCheckBasedEviction evicts or deactivates the given Workload if any admission checks have failed.
+// Returns true if the Workload was rejected or deactivated, and false otherwise.
 func (r *WorkloadReconciler) reconcileCheckBasedEviction(ctx context.Context, wl *kueue.Workload) (bool, error) {
 	if apimeta.IsStatusConditionTrue(wl.Status.Conditions, kueue.WorkloadEvicted) || (!workload.HasRetryChecks(wl) && !workload.HasRejectedChecks(wl)) {
 		return false, nil



Test patch:
diff --git a/test/integration/singlecluster/tas/tas_test.go b/test/integration/singlecluster/tas/tas_test.go
index 235182425..22d5efc88 100644
--- a/test/integration/singlecluster/tas/tas_test.go
+++ b/test/integration/singlecluster/tas/tas_test.go
@@ -66,6 +66,7 @@ var _ = ginkgo.Describe("Topology Aware Scheduling", ginkgo.Ordered, func() {
 	})
 
 	ginkgo.AfterEach(func() {
+		gomega.Expect(util.DeleteNamespace(ctx, k8sClient, ns)).To(gomega.Succeed())
 		_ = features.SetEnable(features.TopologyAwareScheduling, false)
 	})
 
@@ -1318,8 +1319,8 @@ var _ = ginkgo.Describe("Topology Aware Scheduling", ginkgo.Ordered, func() {
 					Preemption(kueue.ClusterQueuePreemption{
 						WithinClusterQueue: kueue.PreemptionPolicyLowerPriority,
 					}).
-					// We set quota above what is physically available to hit
-					// the topology requirements rather quota.
+					// We set the quota above what is physically available to meet
+					// the topology requirements rather than the quota.
 					ResourceGroup(*testing.MakeFlavorQuotas(tasFlavor.Name).
 						Resource(corev1.ResourceCPU, "20").
 						Resource(corev1.ResourceMemory, "20Gi").Obj()).
@@ -1502,9 +1503,9 @@ var _ = ginkgo.Describe("Topology Aware Scheduling", ginkgo.Ordered, func() {
 				})
 
 				ginkgo.By("creating a workload in CQB which reclaims its quota", func() {
-					// Note that workload wl2 would still fit within regular quota
-					// without preempting wl1. However, there is only 1CPU left
-					// on both nodes, and so wl1 needs to be preempted  to allow
+					// Note that workload wl2 would still fit within the regular quota
+					// without preempting wl1. However, there is only 1 CPU left
+					// on both nodes, so wl1 needs to be preempted to allow
 					// scheduling of wl2.
 					wl2 = testing.MakeWorkload("wl2", ns.Name).
 						Priority(2).
@@ -1531,8 +1532,6 @@ var _ = ginkgo.Describe("Topology Aware Scheduling", ginkgo.Ordered, func() {
 			)
 
 			ginkgo.BeforeEach(func() {
-				ns = util.CreateNamespaceFromPrefixWithLog(ctx, k8sClient, "tas-")
-
 				topology = testing.MakeDefaultTwoLevelTopology("default")
 				util.MustCreate(ctx, k8sClient, topology)
 
@@ -1612,8 +1611,6 @@ var _ = ginkgo.Describe("Topology Aware Scheduling", ginkgo.Ordered, func() {
 			)
 
 			ginkgo.BeforeEach(func() {
-				ns = util.CreateNamespaceFromPrefixWithLog(ctx, k8sClient, "tas-")
-
 				topology = testing.MakeDefaultThreeLevelTopology("default")
 				util.MustCreate(ctx, k8sClient, topology)
 
@@ -1779,8 +1776,6 @@ var _ = ginkgo.Describe("Topology Aware Scheduling", ginkgo.Ordered, func() {
 			)
 
 			ginkgo.BeforeEach(func() {
-				ns = util.CreateNamespaceFromPrefixWithLog(ctx, k8sClient, "tas-")
-
 				topology = testing.MakeDefaultThreeLevelTopology("default")
 				util.MustCreate(ctx, k8sClient, topology)