Repository: kubernetes-sigs/kueue

Issue:
[flaky test] MultiKueue Should run a TrainJob on worker if admitted
/kind flake

**What happened**:

example: 
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/7077/pull-kueue-test-integration-multikueue-main/1973001871303905280

**What you expected to happen**:
no failure
**How to reproduce it (as minimally and precisely as possible)**:
ci
**Anything else we need to know?**:

I think this started after re-enabling the test in https://github.com/kubernetes-sigs/kueue/pull/7068

unfortunately no more details:

```
/home/prow/go/src/sigs.k8s.io/kueue/test/integration/framework/framework.go:145
------------------------------
[DeferCleanup (Suite)] PASSED [0.001 seconds]
[DeferCleanup (Suite)] 
/home/prow/go/src/sigs.k8s.io/kueue/test/integration/framework/framework.go:145
------------------------------
[ReportAfterSuite] PASSED [2.849 seconds]
[ReportAfterSuite] Autogenerated ReportAfterSuite for --json-report --junit-report
autogenerated by Ginkgo
------------------------------
Ran 31 of 31 Specs in 169.067 seconds
SUCCESS! -- 31 Passed | 0 Failed | 0 Pending | 0 Skipped
Ginkgo ran 1 suite in 4m40.050654998s
Test Suite Failed
make: *** [Makefile-test.mk:90: test-multikueue-integration] Error 1

```

Hints from issue comments:
@mimowo: cc @ichekrygin @ichekrygin: Is this intermitted or more/less consistent repro?

Hoping, I can flush it out during https://github.com/kubernetes-sigs/kueue/pull/7095@mimowo: Unfortunately no easy repro for now. What is worse is that when it fails, then it does not say any specific reason.

We had a similar issue in the past where running with INTEGRATION_NPROC>1 would not output the DATA RACE of panics sometimes, so I opened https://github.com/kubernetes-sigs/kueue/pull/7085 but no luck so far@mimowo: @mszadkow I seem to remember you were investigating similarly failing multikueue integration tests (also without a clear indication of the reason). And for that reason we decreased INTEGRATION_NPROCS_MULTIKUEUE from 4 to 3. 

Do you remember maybe what was the underlying issue? 

I seem to remember this was after we added the Ray CRDs to the suite, maybe not this is happening after adding the Kubeflow Trainer v2 to the suite. An additional CRD may put the machines under stress more.

So, maybe this has nothing to do with the ElasticJobs which is just red herring here.@mimowo: Ok, I did a bit of archeology and for context we had this similar issue here: https://github.com/kubernetes-sigs/kueue/pull/3986, and as mitigation steps the proposal was summarized here: https://github.com/kubernetes-sigs/kueue/issues/4040:

```
For the experiments in the context of [Kuberay multikueue integrations tests](https://github.com/kubernetes-sigs/kueue/pull/3986) we learned that:
- increasing resources for the job for integration tests helped a little bit, but didn't fully solve the problem with running Ray (see [comment](https://github.com/kubernetes-sigs/kueue/pull/3986#issuecomment-2609155189))
- we know that setting INTEGRATION_NPROC=3 makes the tests usually pass, but it means lower parallelism enforced on the lightweight tests, which is not ideal
```
However, we have never fully understood why the issue was occurring, just found the above mitigates the issue.

I'm not yet totally sure what is the best way to move forward. I suppose the issue was triggered again by including the TrainerJob CRDs, but I would like to understand what is going on. Notably with decreased parallelism I wasn't able to repro here: https://github.com/kubernetes-sigs/kueue/pull/7085

If the issue continues and we cannot understand still maybe we need to mitigate again by decreasing parallelism (not sure yet, brainstorming).@mszadkow: I have been able to reproduce the issue, here is the log with race condition
`ginkgo --race -r --procs=4 --repeat=10`

```
All tests passed...
This was attempt 2 of 11.
Running Suite: MultiKueue Suite - /Users/michal_szadkowski/workspace/kueue/test/integration/multikueue
======================================================================================================
Random Seed: 1759314779

Will run 31 of 31 specs
Running in parallel across 4 processes
•••••••••••••••••••••••••••••
------------------------------
• [4.323 seconds]
MultiKueue Should run a TrainJob on worker if admitted
/Users/michal_szadkowski/workspace/kueue/test/integration/multikueue/jobs_test.go:1564

  Captured StdOut/StdErr Output >>
  ==================
  WARNING: DATA RACE
  Write at 0x000104cbc7a8 by goroutine 3233:
    github.com/kubeflow/trainer/v2/pkg/runtime/core.(*TrainingRuntime).newRuntimeInfo.WithPodSetSyncer.func6()
        /Users/michal_szadkowski/workspace/kueue/vendor/github.com/kubeflow/trainer/v2/pkg/runtime/runtime.go:173 +0x30
    github.com/kubeflow/trainer/v2/pkg/runtime.NewInfo()
        /Users/michal_szadkowski/workspace/kueue/vendor/github.com/kubeflow/trainer/v2/pkg/runtime/runtime.go:180 +0xf8
    github.com/kubeflow/trainer/v2/pkg/runtime/core.(*TrainingRuntime).newRuntimeInfo()
        /Users/michal_szadkowski/workspace/kueue/vendor/github.com/kubeflow/trainer/v2/pkg/runtime/core/trainingruntime.go:187 +0xe80
    github.com/kubeflow/trainer/v2/pkg/runtime/core.(*TrainingRuntime).RuntimeInfo()
        /Users/michal_szadkowski/workspace/kueue/vendor/github.com/kubeflow/trainer/v2/pkg/runtime/core/trainingruntime.go:103 +0xd8
    github.com/kubeflow/trainer/v2/pkg/runtime/core.(*ClusterTrainingRuntime).RuntimeInfo()
        /Users/michal_szadkowski/workspace/kueue/vendor/github.com/kubeflow/trainer/v2/pkg/runtime/core/clustertrainingruntime.go:71 +0x60
    sigs.k8s.io/kueue/pkg/controller/jobs/trainjob.getChildJobSet()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobs/trainjob/trainjob_controller.go:175 +0x1a8
    sigs.k8s.io/kueue/pkg/controller/jobs/trainjob.(*TrainJob).PodSets()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobs/trainjob/trainjob_controller.go:234 +0x28
    sigs.k8s.io/kueue/pkg/controller/jobframework.EquivalentToWorkload()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobframework/reconciler.go:1047 +0x144
    sigs.k8s.io/kueue/pkg/controller/jobframework.FindMatchingWorkloads()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobframework/reconciler.go:940 +0x308
    sigs.k8s.io/kueue/pkg/controller/jobframework.(*JobReconciler).ensureOneWorkload()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobframework/reconciler.go:859 +0x6c8
    sigs.k8s.io/kueue/pkg/controller/jobframework.(*JobReconciler).ReconcileGenericJob()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobframework/reconciler.go:397 +0xf2c
    sigs.k8s.io/kueue/pkg/controller/jobs/trainjob.(*trainJobReconciler).Reconcile()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobs/trainjob/trainjob_controller.go:106 +0xd0
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[go.shape.struct { k8s.io/apimachinery/pkg/types.NamespacedName }]).Reconcile()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216 +0x164
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[go.shape.struct { k8s.io/apimachinery/pkg/types.NamespacedName }]).reconcileHandler()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:461 +0x374
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[go.shape.struct { k8s.io/apimachinery/pkg/types.NamespacedName }]).processNextWorkItem()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421 +0x23c
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[go.shape.struct { k8s.io/apimachinery/pkg/types.NamespacedName }]).Start.func1.1()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296 +0xb8

  Previous write at 0x000104cbc7a8 by goroutine 3133:
    github.com/kubeflow/trainer/v2/pkg/runtime/core.(*TrainingRuntime).newRuntimeInfo.WithPodSetSyncer.func6()
        /Users/michal_szadkowski/workspace/kueue/vendor/github.com/kubeflow/trainer/v2/pkg/runtime/runtime.go:173 +0x30
    github.com/kubeflow/trainer/v2/pkg/runtime.NewInfo()
        /Users/michal_szadkowski/workspace/kueue/vendor/github.com/kubeflow/trainer/v2/pkg/runtime/runtime.go:180 +0xf8
    github.com/kubeflow/trainer/v2/pkg/runtime/core.(*TrainingRuntime).newRuntimeInfo()
        /Users/michal_szadkowski/workspace/kueue/vendor/github.com/kubeflow/trainer/v2/pkg/runtime/core/trainingruntime.go:187 +0xe80
    github.com/kubeflow/trainer/v2/pkg/runtime/core.(*TrainingRuntime).RuntimeInfo()
        /Users/michal_szadkowski/workspace/kueue/vendor/github.com/kubeflow/trainer/v2/pkg/runtime/core/trainingruntime.go:103 +0xd8
    github.com/kubeflow/trainer/v2/pkg/runtime/core.(*ClusterTrainingRuntime).RuntimeInfo()
        /Users/michal_szadkowski/workspace/kueue/vendor/github.com/kubeflow/trainer/v2/pkg/runtime/core/clustertrainingruntime.go:71 +0x60
    sigs.k8s.io/kueue/pkg/controller/jobs/trainjob.getChildJobSet()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobs/trainjob/trainjob_controller.go:175 +0x1a8
    sigs.k8s.io/kueue/pkg/controller/jobs/trainjob.(*TrainJob).PodSets()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobs/trainjob/trainjob_controller.go:234 +0x28
    sigs.k8s.io/kueue/pkg/controller/jobframework.EquivalentToWorkload()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobframework/reconciler.go:1047 +0x144
    sigs.k8s.io/kueue/pkg/controller/jobframework.FindMatchingWorkloads()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobframework/reconciler.go:940 +0x308
    sigs.k8s.io/kueue/pkg/controller/jobframework.(*JobReconciler).ensureOneWorkload()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobframework/reconciler.go:859 +0x6c8
    sigs.k8s.io/kueue/pkg/controller/jobframework.(*JobReconciler).ReconcileGenericJob()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobframework/reconciler.go:397 +0xf2c
    sigs.k8s.io/kueue/pkg/controller/jobs/trainjob.(*trainJobReconciler).Reconcile()
        /Users/michal_szadkowski/workspace/kueue/pkg/controller/jobs/trainjob/trainjob_controller.go:106 +0xd0
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[go.shape.struct { k8s.io/apimachinery/pkg/types.NamespacedName }]).Reconcile()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216 +0x164
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[go.shape.struct { k8s.io/apimachinery/pkg/types.NamespacedName }]).reconcileHandler()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:461 +0x374
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[go.shape.struct { k8s.io/apimachinery/pkg/types.NamespacedName }]).processNextWorkItem()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421 +0x23c
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[go.shape.struct { k8s.io/apimachinery/pkg/types.NamespacedName }]).Start.func1.1()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296 +0xb8

  Goroutine 3233 (running) created at:
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[go.shape.struct { k8s.io/apimachinery/pkg/types.NamespacedName }]).Start.func1()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:292 +0x2d8
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[go.shape.struct { k8s.io/apimachinery/pkg/types.NamespacedName }]).Start()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303 +0x14c
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[sigs.k8s.io/controller-runtime/pkg/reconcile.Request]).Start()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:259 +0x48
    sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/manager/runnable_group.go:260 +0x158
    sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.gowrap1()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/manager/runnable_group.go:283 +0x40

  Goroutine 3133 (running) created at:
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[go.shape.struct { k8s.io/apimachinery/pkg/types.NamespacedName }]).Start.func1()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:292 +0x2d8
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[go.shape.struct { k8s.io/apimachinery/pkg/types.NamespacedName }]).Start()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303 +0x14c
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[sigs.k8s.io/controller-runtime/pkg/reconcile.Request]).Start()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:259 +0x48
    sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/manager/runnable_group.go:260 +0x158
    sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.gowrap1()
        /Users/michal_szadkowski/workspace/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/manager/runnable_group.go:283 +0x40
  ==================
  << Captured StdOut/StdErr Output
------------------------------
•

Ran 31 of 31 Specs in 147.607 seconds
SUCCESS! -- 31 Passed | 0 Failed | 0 Pending | 0 Skipped


Tests failed on attempt #3


Ginkgo ran 1 suite in 7m49.903828292s

Test Suite Failed
(prow-artifacts) [12:35:27 @michal_szadkowski:~/workspace/kueue/test/integration/multikueue]
```@mszadkow: Well this is unfortunate that race conditions are not visible when we run tests with Makefile.
However I do believe this is different from https://github.com/kubernetes-sigs/kueue/issues/3987 
There we had a panic and test suite stopped working all together.@mimowo: nice, thank you for the repro. I see the TrainJob on the stracktrace which is a new card indeed. Do you know which specific It (test case) triggers the race?

cc @kaisoz @mszadkow: I think this is this one:
```
MultiKueue Should run a TrainJob on worker if admitted
kueue/test/integration/multikueue/jobs_test.go:1564
```@mimowo: I see, so this looks like two threads are modifying the internal state of the `runtime` somewhere here: https://github.com/kubernetes-sigs/kueue/blob/81113ca514a6b20dce3e0b3fecc6d6fe14b75109/pkg/controller/jobs/trainjob/trainjob_controller.go#L176C15-L176C43

I guess we may need to DeepCopy the runtime before calling RuntimeInfo. Or avoid storing "runtimes" as state, seems like premature optimization (potentially). If you have a reliable repro then worth checking if DeepCopy or getting new runtime each time helps.

EDIT: if that does not work we may need to take a mutex on the getChildJobSet function (or even just around calling RuntimeInfo).@mszadkow: /assign@mimowo: /retitle [flaky test] MultiKueue Should run a TrainJob on worker if admitted@mimowo: cc @tenzen-y @andreyvelich I'm also wondering if TrainerJob could / should internally avoid mutating the state by calling RuntimeInfo.@mszadkow: Actually if we talk about the first idea, what it could be, I learnt that is quite easy to reproduce, by increasing number of processes
When we run: `ginkgo --race -r --procs=6 --repeat=10`
We got old issue with CRD everytime:
```
  [FAILED] in [BeforeSuite] - /Users/michal_szadkowski/workspace/kueue/vendor/github.com/onsi/ginkgo/v2/internal/suite.go:328 @ 10/01/25 15:14:37.505
  << Timeline

  [FAILED] Unexpected error:
      <*fmt.wrapError | 0xc004dcfbc0>: 
      unable to install CRDs onto control plane: something went wrong waiting for CRDs to appear as API resources: context deadline exceeded
      {
          msg: "unable to install CRDs onto control plane: something went wrong waiting for CRDs to appear as API resources: context deadline exceeded",
          err: <*fmt.wrapError | 0xc004dcfb80>{
              msg: "something went wrong waiting for CRDs to appear as API resources: context deadline exceeded",
              err: <context.deadlineExceededError>{},
          },
      }
  occurred
  In [BeforeSuite] at: /Users/michal_szadkowski/workspace/kueue/vendor/github.com/onsi/ginkgo/v2/internal/suite.go:328 @ 10/01/25 15:14:37.505
------------------------------
[AfterSuite] [PANICKED] [0.006 seconds]
[AfterSuite] 
/Users/michal_szadkowski/workspace/kueue/test/integration/multikueue/suite_test.go:396

  Timeline >>
  STEP: tearing down the test environment @ 10/01/25 15:14:37.816
  [PANICKED] in [AfterSuite] - /Users/michal_szadkowski/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/src/runtime/panic.go:262 @ 10/01/25 15:14:37.817
  << Timeline

  [PANICKED] Test Panicked
  In [AfterSuite] at: /Users/michal_szadkowski/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/src/runtime/panic.go:262 @ 10/01/25 15:14:37.817

  runtime error: invalid memory address or nil pointer dereference

  Full Stack Trace
    sigs.k8s.io/kueue/test/integration/framework.(*Framework).Teardown(0x0)
        /Users/michal_szadkowski/workspace/kueue/test/integration/framework/framework.go:218 +0x44
    sigs.k8s.io/kueue/test/integration/multikueue.init.func10()
        /Users/michal_szadkowski/workspace/kueue/test/integration/multikueue/suite_test.go:397 +0x38
------------------------------
•••••••••••••••••••••••••••••••

Summarizing 2 Failures:
  [FAIL] [BeforeSuite] 
  /Users/michal_szadkowski/workspace/kueue/vendor/github.com/onsi/ginkgo/v2/internal/suite.go:328
  [PANICKED!] [AfterSuite] 
  /Users/michal_szadkowski/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.darwin-arm64/src/runtime/panic.go:262

Ran 31 of 31 Specs in 170.093 seconds
FAIL! -- 31 Passed | 0 Failed | 0 Pending | 0 Skipped
```
@mimowo: nice! @kaisoz: > ...Or avoid storing "runtimes" as state, seems like premature optimization (potentially). If you have a reliable repro then worth checking if DeepCopy or getting new runtime each time helps.

Unfortunately the runtimes can ony be created once at this moment (I've tried it already)

The call to the runtimes [New](https://github.com/kubernetes-sigs/kueue/blob/81113ca514a6b20dce3e0b3fecc6d6fe14b75109/pkg/controller/jobs/trainjob/trainjob_controller.go#L93) function  (implementation [here](https://github.com/kubeflow/trainer/blob/b918411a6b4d5dfc4e68a328139cdb7d821040f2/pkg/runtime/core/core.go#L31)) ends up calling each runtime constructor, which in turn [register the field in the indexer](https://github.com/kubeflow/trainer/blob/b918411a6b4d5dfc4e68a328139cdb7d821040f2/pkg/runtime/core/trainingruntime.go#L65). This registration **can only be done once**

@mszadkow: > > ...Or avoid storing "runtimes" as state, seems like premature optimization (potentially). If you have a reliable repro then worth checking if DeepCopy or getting new runtime each time helps.
> 
> Unfortunately the runtimes can ony be created once at this moment (I've tried it already)
> 
> The call to the runtimes [New](https://github.com/kubernetes-sigs/kueue/blob/81113ca514a6b20dce3e0b3fecc6d6fe14b75109/pkg/controller/jobs/trainjob/trainjob_controller.go#L93) function (implementation [here](https://github.com/kubeflow/trainer/blob/b918411a6b4d5dfc4e68a328139cdb7d821040f2/pkg/runtime/core/core.go#L31)) ends up calling each runtime constructor, which in turn [register the field in the indexer](https://github.com/kubeflow/trainer/blob/b918411a6b4d5dfc4e68a328139cdb7d821040f2/pkg/runtime/core/trainingruntime.go#L65). This registration **can only be done once**

I guess lock on `trainjob.getChildJobSet` is the best option here@mimowo: > I guess lock on trainjob.getChildJobSet is the best option here

I think so, at least as a quick fix. We can report in the Kubeflow Trainer to make it thread safe. 
Also, I think you don't need to get mutex for entire getChildJobSet, just calling RuntimeInfo (could be extracted to another helper wrapped in mutex like), runtimeInfo.

Solution patch:
diff --git a/hack/releasing/sync-notes.sh b/hack/releasing/sync-notes.sh
index 93b3e6a74..bafd2a997 100755
--- a/hack/releasing/sync-notes.sh
+++ b/hack/releasing/sync-notes.sh
@@ -151,7 +151,7 @@ fi
 RELEASE_ISSUE_NAME="Release ${RELEASE_VERSION}"
 declare RELEASE_ISSUE_NAME
 
-RELEASE_ISSUE_NUMBER=$(gh issue list --repo="${MAIN_REPO_ORG}/${MAIN_REPO_NAME}" | grep "${RELEASE_ISSUE_NAME}" | awk '{print $1}' || true)
+RELEASE_ISSUE_NUMBER=$(gh issue list --repo="${MAIN_REPO_ORG}/${MAIN_REPO_NAME}" --search "in:title ${RELEASE_ISSUE_NAME}" | awk '{print $1}' || true)
 if [ -z "$RELEASE_ISSUE_NUMBER" ]; then
   echo "!!! No release issue found for version ${RELEASE_VERSION}. Please create 'Release ${RELEASE_VERSION}' issue first."
   exit 1
diff --git a/pkg/cache/scheduler/clusterqueue_snapshot.go b/pkg/cache/scheduler/clusterqueue_snapshot.go
index 9036df94a..8c132c6bb 100644
--- a/pkg/cache/scheduler/clusterqueue_snapshot.go
+++ b/pkg/cache/scheduler/clusterqueue_snapshot.go
@@ -72,24 +72,6 @@ func (c *ClusterQueueSnapshot) RGByResource(resource corev1.ResourceName) *Resou
 	return nil
 }
 
-// SimulateWorkloadRemoval modifies the snapshot by removing the usage
-// corresponding to the list of workloads. It returns a function which
-// can be used to restore the usage.
-func (c *ClusterQueueSnapshot) SimulateWorkloadRemoval(workloads []*workload.Info) func() {
-	usage := make([]workload.Usage, 0, len(workloads))
-	for _, w := range workloads {
-		usage = append(usage, w.Usage())
-	}
-	for _, u := range usage {
-		c.RemoveUsage(u)
-	}
-	return func() {
-		for _, u := range usage {
-			c.AddUsage(u)
-		}
-	}
-}
-
 // SimulateUsageAddition modifies the snapshot by adding usage, and
 // returns a function used to restore the usage.
 func (c *ClusterQueueSnapshot) SimulateUsageAddition(usage workload.Usage) func() {
diff --git a/pkg/cache/scheduler/snapshot.go b/pkg/cache/scheduler/snapshot.go
index f0968ec2e..8aa0af873 100644
--- a/pkg/cache/scheduler/snapshot.go
+++ b/pkg/cache/scheduler/snapshot.go
@@ -52,7 +52,7 @@ func (s *Snapshot) RemoveWorkload(wl *workload.Info) {
 	cq.RemoveUsage(wl.Usage())
 }
 
-// AddWorkload adds a workload from its corresponding ClusterQueue and
+// AddWorkload adds a workload to its corresponding ClusterQueue and
 // updates resource usage.
 func (s *Snapshot) AddWorkload(wl *workload.Info) {
 	cq := s.ClusterQueue(wl.ClusterQueue)
@@ -60,6 +60,29 @@ func (s *Snapshot) AddWorkload(wl *workload.Info) {
 	cq.AddUsage(wl.Usage())
 }
 
+// SimulateWorkloadRemoval modifies the snapshot by removing the usage
+// corresponding to the list of workloads from workloads' respective
+// ClusterQueues. It returns a function which can be used to restore
+// this usage.
+func (s *Snapshot) SimulateWorkloadRemoval(workloads []*workload.Info) func() {
+	type cqUsage struct {
+		cq    kueue.ClusterQueueReference
+		usage workload.Usage
+	}
+	cqUsages := make([]cqUsage, 0, len(workloads))
+	for _, w := range workloads {
+		cqUsages = append(cqUsages, cqUsage{cq: w.ClusterQueue, usage: w.Usage()})
+	}
+	for _, cqUsage := range cqUsages {
+		s.ClusterQueue(cqUsage.cq).RemoveUsage(cqUsage.usage)
+	}
+	return func() {
+		for _, cqUsage := range cqUsages {
+			s.ClusterQueue(cqUsage.cq).AddUsage(cqUsage.usage)
+		}
+	}
+}
+
 func (s *Snapshot) Log(log logr.Logger) {
 	for name, cq := range s.ClusterQueues() {
 		cohortName := "<none>"
diff --git a/pkg/scheduler/preemption/preemption_oracle.go b/pkg/scheduler/preemption/preemption_oracle.go
index 084b030ce..0524409f1 100644
--- a/pkg/scheduler/preemption/preemption_oracle.go
+++ b/pkg/scheduler/preemption/preemption_oracle.go
@@ -57,7 +57,7 @@ func (p *PreemptionOracle) SimulatePreemption(log logr.Logger, cq *schdcache.Clu
 	for i, c := range candidates {
 		workloadsToPreempt[i] = c.WorkloadInfo
 	}
-	revertRemoval := cq.SimulateWorkloadRemoval(workloadsToPreempt)
+	revertRemoval := p.snapshot.SimulateWorkloadRemoval(workloadsToPreempt)
 	borrowAfterPreemptions, _ := classical.FindHeightOfLowestSubtreeThatFits(cq, fr, quantity)
 	revertRemoval()
 
diff --git a/pkg/scheduler/scheduler.go b/pkg/scheduler/scheduler.go
index 12e463340..a9909d4da 100644
--- a/pkg/scheduler/scheduler.go
+++ b/pkg/scheduler/scheduler.go
@@ -280,7 +280,7 @@ func (s *Scheduler) schedule(ctx context.Context) wait.SpeedSignal {
 		}
 
 		usage := e.assignmentUsage()
-		if !fits(cq, &usage, preemptedWorkloads, e.preemptionTargets) {
+		if !fits(snapshot, cq, &usage, preemptedWorkloads, e.preemptionTargets) {
 			setSkipped(e, "Workload no longer fits after processing another workload")
 			if mode == flavorassigner.Preempt {
 				skippedPreemptions[cq.Name]++
@@ -451,12 +451,12 @@ func (s *Scheduler) nominate(ctx context.Context, workloads []workload.Info, sna
 	return entries, inadmissibleEntries
 }
 
-func fits(cq *schdcache.ClusterQueueSnapshot, usage *workload.Usage, preemptedWorkloads preemption.PreemptedWorkloads, newTargets []*preemption.Target) bool {
+func fits(snapshot *schdcache.Snapshot, cq *schdcache.ClusterQueueSnapshot, usage *workload.Usage, preemptedWorkloads preemption.PreemptedWorkloads, newTargets []*preemption.Target) bool {
 	workloads := slices.Collect(maps.Values(preemptedWorkloads))
 	for _, target := range newTargets {
 		workloads = append(workloads, target.WorkloadInfo)
 	}
-	revertUsage := cq.SimulateWorkloadRemoval(workloads)
+	revertUsage := snapshot.SimulateWorkloadRemoval(workloads)
 	defer revertUsage()
 	return cq.Fits(*usage)
 }
@@ -506,7 +506,7 @@ type partialAssignment struct {
 func (s *Scheduler) getAssignments(log logr.Logger, wl *workload.Info, snap *schdcache.Snapshot) (flavorassigner.Assignment, []*preemption.Target) {
 	assignment, targets := s.getInitialAssignments(log, wl, snap)
 	cq := snap.ClusterQueue(wl.ClusterQueue)
-	updateAssignmentForTAS(cq, wl, &assignment, targets)
+	updateAssignmentForTAS(snap, cq, wl, &assignment, targets)
 	return assignment, targets
 }
 
@@ -584,7 +584,7 @@ func (s *Scheduler) evictWorkloadAfterFailedTASReplacement(ctx context.Context,
 	return nil
 }
 
-func updateAssignmentForTAS(cq *schdcache.ClusterQueueSnapshot, wl *workload.Info, assignment *flavorassigner.Assignment, targets []*preemption.Target) {
+func updateAssignmentForTAS(snapshot *schdcache.Snapshot, cq *schdcache.ClusterQueueSnapshot, wl *workload.Info, assignment *flavorassigner.Assignment, targets []*preemption.Target) {
 	if features.Enabled(features.TopologyAwareScheduling) && assignment.RepresentativeMode() == flavorassigner.Preempt &&
 		(workload.IsExplicitlyRequestingTAS(wl.Obj.Spec.PodSets...) || cq.IsTASOnly()) && !workload.HasTopologyAssignmentWithUnhealthyNode(wl.Obj) {
 		tasRequests := assignment.WorkloadsTopologyRequests(wl, cq)
@@ -594,7 +594,7 @@ func updateAssignmentForTAS(cq *schdcache.ClusterQueueSnapshot, wl *workload.Inf
 			for _, target := range targets {
 				targetWorkloads = append(targetWorkloads, target.WorkloadInfo)
 			}
-			revertUsage := cq.SimulateWorkloadRemoval(targetWorkloads)
+			revertUsage := snapshot.SimulateWorkloadRemoval(targetWorkloads)
 			tasResult = cq.FindTopologyAssignmentsForWorkload(tasRequests)
 			revertUsage()
 		} else {



Test patch:
diff --git a/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go b/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go
index c45469a2f..852c46227 100644
--- a/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go
+++ b/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go
@@ -493,6 +493,7 @@ var _ = ginkgo.Describe("Scheduler", ginkgo.Ordered, ginkgo.ContinueOnFailure, f
 	ginkgo.When("using hierarchical cohorts with several flavors", func() {
 		var (
 			cqp1 *kueue.ClusterQueue
+			cqp5 *kueue.ClusterQueue
 		)
 		ginkgo.BeforeEach(func() {
 			createCohort(testing.MakeCohort("root-cohort").Obj())
@@ -566,7 +567,7 @@ var _ = ginkgo.Describe("Scheduler", ginkgo.Ordered, ginkgo.ContinueOnFailure, f
 				Preemption(preemption).
 				Obj())
 
-			createQueue(testing.MakeClusterQueue("cq-p5").
+			cqp5 = createQueue(testing.MakeClusterQueue("cq-p5").
 				Cohort("cohort-b").
 				FairWeight(resource.MustParse("1")).
 				ResourceGroup(
@@ -614,6 +615,38 @@ var _ = ginkgo.Describe("Scheduler", ginkgo.Ordered, ginkgo.ContinueOnFailure, f
 			util.ExpectClusterQueueWeightedShareMetric(cqp1, 600)
 			expectCohortWeightedShare("cohort-a", 0)
 		})
+
+		// scenario from Kueue#7015
+		// WeightedShare(CohortA) = 0/20 * 1000 = 0
+		// WeightedShare(cq-p1)  = 12/20 * 1000 = 600
+		// WeightedShare(CohortB) = 0/20 * 1000 = 0
+		// WeightedShare(cq-p5)   = 2/20 * 1000 = 100
+		ginkgo.It("CohortB preempts and schedules in flavor which has guarantees", func() {
+			_ = features.SetEnable(features.FlavorFungibilityImplicitPreferenceDefault, true)
+			ginkgo.By("Create workload which saturate all cohort resources")
+			for range 20 {
+				createWorkload("cq-p1", "1")
+			}
+			util.ExpectReservingActiveWorkloadsMetric(cqp1, 20)
+			expectCohortWeightedShare("cohort-a", 100)
+
+			ginkgo.By("Create workloads in CohortB which will preempt CohortA")
+			createWorkload("cq-p5", "1")
+			createWorkload("cq-p5", "1")
+
+			ginkgo.By("Finish Preemption of Workloads")
+			util.FinishEvictionOfWorkloadsInCQ(ctx, k8sClient, cqp1, 2)
+
+			ginkgo.By("Check expected workloads active")
+			util.ExpectReservingActiveWorkloadsMetric(cqp1, 18)
+			util.ExpectReservingActiveWorkloadsMetric(cqp5, 2)
+
+			ginkgo.By("Expected Weighted Shares")
+			util.ExpectClusterQueueWeightedShareMetric(cqp1, 600)
+			util.ExpectClusterQueueWeightedShareMetric(cqp5, 100)
+			expectCohortWeightedShare("cohort-a", 0)
+			expectCohortWeightedShare("cohort-b", 0)
+		})
 	})
 
 	ginkgo.When("Using AdmissionFairSharing at ClusterQueue level", func() {