Repository: kubernetes/kubernetes (language: go)

Issue:
Pods gated by custom PreEnqueue plugins don't go through backoffQ even in case they ought to
### What happened?

With the current implementation, all gated Pods are always regarded as **not** backing off.

https://github.com/kubernetes/kubernetes/blob/ef9965ebc66dafda37800bb04f5e284535bbba10/pkg/scheduler/internal/queue/scheduling_queue.go#L658-L661

That is correct for a vanilla scheduler because all Pods gated by SchedulingGates are not backing off for sure,
but, not correct for a custom scheduler with out-of-tree PreEnqueue plugins,
which could gate Pods that already have gone through scheduling cycles (and got unschedulable).

### What did you expect to happen?

Pods which have gone through scheduling cycles, but now gated should be put into backoffQ. (or activeQ, if it's in the unschedQ for enough time already)

### How can we reproduce it (as minimally and precisely as possible)?

1. Create a Pod that cannot be scheduled.
2. After a Pod got unschedulable, custom PreEnqueue plugin gates the Pod.
3. Trigger a cluster event which could make the Pod schedulable, and with this event, remove a gate.
4. The Pod directly goes to activeQ, not going through backoffQ, while it's once gone through a scheduling cycle.

### Anything else we need to know?

_No response_

### Kubernetes version

master

### Cloud provider

<details>
n/a
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>



Discussion / hints:
/sig scheduling
This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the `triage/accepted` label and provide further guidance.

The `triage/accepted` label can be added by org members by writing `/triage accepted` in a comment.


<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
/assign
I don't consider this a bug, as PreEnqueue was not initially designed for Pods that went through the scheduling cycle already.

/remove-kind bug
/kind feature

In any case, I'm happy to review a reasonable proposal.
Current design is reasonable to me, so again ping @Huang-Wei 
@kerthcet moving the discussion here:

To reply to your last comment
> I don't know if we have such case, but if a pod is gated again, it means it's not ready, maybe it should restart the scheduling lifecycle.. my two cents. Let's track this under your issue then.
https://github.com/kubernetes/kubernetes/pull/125527#discussion_r1645373474

At least, we have a usecsae here -
https://github.com/pfnet/scheduler-plugins/tree/master/plugins/gang
(wow hello again 😅 I've brought it into several discussion)

What it does with PreEnqueue is-
> We also use PreEnqueue in https://github.com/pfnet/scheduler-plugins/tree/master/plugins/gang (gang scheduling); PreEnqueue allows Pods to enter scheduling cycles iff all Pods in the group are now ready to retry scheduling.
e.g., When gangA-pod1 is rejected by plugin-1 and when gangA-pod2 is rejected by plugin-2, we want to requeue all gangA Pods iff the scheduling queue observes some cluster events that could change the scheduling results of gangA-pod1 and gangA-pod2. (i.e., when QHints of plugin-1 and plugin-2 returns Queue for those Pods)
https://github.com/kubernetes/kubernetes/issues/125223#issuecomment-2174787844

So, Pods once got un-gated could get gated again.

To clarify, from less-subjective view, we have two options here:
1. Add `once PreEnqueue allows a Pod to enter a scheduling cycle, PreEnqueue shouldn't gate the Pod again` to PreEnqueue specification.
2. Allow `even after PreEnqueue allows a Pod to enter a scheduling cycle once, PreEnqueue could gate the Pod again`. Fix a backoff behaviour like the issue description at the top argues in `What did you expect to happen?`.

As @alculquicondor said, I admit the flow `gated` -> `un-gated` -> (experience scheduling cycles) -> `gated` again isn't considered officially until now since the scheduling gate, which doesn't accept the flow, is the first motivation to establish PreEnqueue.

So, the current situation is we don't say anything for that case, and people like us _technically_ can implement that flow with almost no issue, but only has this minor issue.

> Current design is reasonable to me
https://github.com/kubernetes/kubernetes/issues/125538#issuecomment-2177587995

(as you know) I'm tend to the second option since I don't see any big reason/concern that we have to limit the usage of PreEnqueue, like the first one said.

I'm curious what reason the current behaviour looks reasonable to you for.
I can break this into two parts:
- A single Pod
- A group of Pods

I mostly pointed to the single Pod, I feel a re-gated Pod should restart the whole lifecycle of scheduling, backoffQ is designed for not blocking the queue however gated Pods will not enter into the queue.

For a group of Pods, it's much more complex, like if a Pod failed in scheduling, it should be marked as unschedulable or gated again, maybe unschedulable, then it's already under backoff. And they should be re-scheduled first next time because they're the main reason blocking the whole group. When scheduling timeout, how to handle these Pods, assumedPods should be gated again maybe, unscheduledPod should still be unscheduledPods, backoffing in the background?

So the total workflow is still unclear to me, but that's what I'm working for right now. Always open to new ideas :)
Okay, so the first thing not a direct answer to your last comment (sorry), but let me confirm one thing as I feel like we're not on the same base page.

So, this issue does not depend on which kind of Pods they are. The point that I cannot justify the current behaviour is more general.

Reviewing the concept of backoff (just in case), backoff time is a [penalty](https://github.com/kubernetes/kubernetes/blob/09472b673d9e34b866fbd41c544887ba60d7ca7b/pkg/scheduler/framework/interface.go#L93-L95) that we impose on Pods when they consume a scheduling cycle, but they didn't get scheduled and came back to the queue.
So, regardless of whether a Pod is gated or not, they are supposed to get a penalty if they wasted some scheduling cycles before. It's the law in the scheduler; it's their obligation that they must meet before retrying a schedule again.
 
However, the current situation this issue highlights is, if they get gated again, they somehow get immune from this penalty.
Let's say Pods have failed at scheduling 1k times before, but if they were being gated in the queue even during a moment, they bypass backoff, which is illegal. That's my point that I want to clarify with this PR.

---

Then, answering your questions,
> I feel a re-gated Pod should restart the whole lifecycle of scheduling, backoffQ is designed for not blocking the queue however gated Pods will not enter into the queue.

Why is it better to restart a whole lifecycle of scheduling? 

My point is, again -
> Let's say Pods have failed at scheduling 1k times before, but if they were being gated in the queue even during a moment, they bypass backoff, which is illegal.

So, I believe Pod should not restart a whole lifecycle, but the scheduling queue should remember this guy has failed at scheduling 1k times, and then impose a backoff penalty on them.

Usecase-wise, DRA's PreEnqueue, which is not a plugin schedules a group of Pods, could gate a Pod once it allows. (e.g., when a claim for Pod-A is deleted somehow after Pod-A's scheduling failure, it gates Pod-A until the claim is recreated.)

> For a group of Pods

Sorry, I don't quite understand what you have in your mind. 

But, let me elaborate our gang plugin's behaviour around retrying:

1. Let's say Pod1, Pod2 and Pod3 are in the same group and they're just created, going to scheduling cycles.
2. Pod1 and Pod2 goes thru Filter phase, and waiting for Pod3 (waitOnPermit).
3. Pod3, unfortunately, is rejected by noderesourcefit. 
4. Pod3 goes back to unschedQ.
5. Also, we bring back Pod1 and Pod2 to unschedQ too, without waiting for timeout they have. The reason is for "waiting Pods reserve too large space in a cluster https://github.com/kubernetes/enhancements/issues/4671#issuecomment-2141307013"
6. Now, Pod1 and Pod2 have `unschedPlugin: gang` and Pod3 has `unschedPlugin: noderesourcefit`. 
7. This group won't be schedulable until Pod3's failure is resolved. So, Pod1 and Pod2 will be gated until Pod3's failure is solved.
8. And, when the scheduler receives an event that is registered in noderesourcefit ([if QHint enabled] + when QHint returns `Queue` to the event ), Pod3 is moved from unschedQ to activeQ or backoffQ because Pod3 _may_ be schedulable now. 
9. Then, move Pod1 and Pod2 to activeQ/backoffQ too so that all Pods in this group are again proceeding to scheduling cycles.
10. Then, if you're lucky, all Pods are scheduled successfully. If you're not, some of them might be unschedulable, and we're doing the same process of requeueing again.

Fix patch:
diff --git a/pkg/scheduler/backend/queue/scheduling_queue.go b/pkg/scheduler/backend/queue/scheduling_queue.go
index 56d4ec20bab63..24d05e6219f11 100644
--- a/pkg/scheduler/backend/queue/scheduling_queue.go
+++ b/pkg/scheduler/backend/queue/scheduling_queue.go
@@ -627,9 +627,6 @@ func (p *PriorityQueue) activate(logger klog.Logger, pod *v1.Pod) bool {
 // isPodBackingoff returns true if a pod is still waiting for its backoff timer.
 // If this returns true, the pod should not be re-tried.
 func (p *PriorityQueue) isPodBackingoff(podInfo *framework.QueuedPodInfo) bool {
-	if podInfo.Gated {
-		return false
-	}
 	boTime := p.getBackoffTime(podInfo)
 	return boTime.After(p.clock.Now())
 }
@@ -1240,6 +1237,12 @@ func (p *PriorityQueue) getBackoffTime(podInfo *framework.QueuedPodInfo) time.Ti
 // calculateBackoffDuration is a helper function for calculating the backoffDuration
 // based on the number of attempts the pod has made.
 func (p *PriorityQueue) calculateBackoffDuration(podInfo *framework.QueuedPodInfo) time.Duration {
+	if podInfo.Attempts == 0 {
+		// When the Pod hasn't experienced any scheduling attempts,
+		// they aren't obliged to get a backoff penalty at all.
+		return 0
+	}
+
 	duration := p.podInitialBackoffDuration
 	for i := 1; i < podInfo.Attempts; i++ {
 		// Use subtraction instead of addition or multiplication to avoid overflow.
diff --git a/pkg/scheduler/framework/types.go b/pkg/scheduler/framework/types.go
index 4743306ca635c..bf334033794eb 100644
--- a/pkg/scheduler/framework/types.go
+++ b/pkg/scheduler/framework/types.go
@@ -232,7 +232,7 @@ type QueuedPodInfo struct {
 	// The time pod added to the scheduling queue.
 	Timestamp time.Time
 	// Number of schedule attempts before successfully scheduled.
-	// It's used to record the # attempts metric.
+	// It's used to record the # attempts metric and calculate the backoff time this Pod is obliged to get before retrying.
 	Attempts int
 	// The time when the pod is added to the queue for the first time. The pod may be added
 	// back to the queue multiple times before it's successfully scheduled.


Test patch:
diff --git a/pkg/scheduler/backend/queue/scheduling_queue_test.go b/pkg/scheduler/backend/queue/scheduling_queue_test.go
index 566fe6aa5b05a..bcc2f54455525 100644
--- a/pkg/scheduler/backend/queue/scheduling_queue_test.go
+++ b/pkg/scheduler/backend/queue/scheduling_queue_test.go
@@ -697,7 +697,7 @@ func Test_InFlightPods(t *testing.T) {
 				case action.eventHappens != nil:
 					q.MoveAllToActiveOrBackoffQueue(logger, *action.eventHappens, nil, nil, nil)
 				case action.podEnqueued != nil:
-					err := q.AddUnschedulableIfNotPresent(logger, action.podEnqueued, q.SchedulingCycle())
+					err := q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(action.podEnqueued), q.SchedulingCycle())
 					if err != nil {
 						t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 					}
@@ -1028,7 +1028,7 @@ func TestPriorityQueue_Update(t *testing.T) {
 			name:  "when updating a pod which is in unschedulable queue and is backing off, it will be moved to backoff queue",
 			wantQ: backoffQ,
 			prepareFunc: func(t *testing.T, logger klog.Logger, q *PriorityQueue) (oldPod, newPod *v1.Pod) {
-				q.unschedulablePods.addOrUpdate(q.newQueuedPodInfo(medPriorityPodInfo.Pod, queuePlugin))
+				q.unschedulablePods.addOrUpdate(attemptQueuedPodInfo(q.newQueuedPodInfo(medPriorityPodInfo.Pod, queuePlugin)))
 				updatedPod := medPriorityPodInfo.Pod.DeepCopy()
 				updatedPod.Annotations["foo"] = "test"
 				return medPriorityPodInfo.Pod, updatedPod
@@ -1039,7 +1039,7 @@ func TestPriorityQueue_Update(t *testing.T) {
 			name:  "when updating a pod which is in unschedulable queue and is not backing off, it will be moved to active queue",
 			wantQ: activeQ,
 			prepareFunc: func(t *testing.T, logger klog.Logger, q *PriorityQueue) (oldPod, newPod *v1.Pod) {
-				q.unschedulablePods.addOrUpdate(q.newQueuedPodInfo(medPriorityPodInfo.Pod, queuePlugin))
+				q.unschedulablePods.addOrUpdate(attemptQueuedPodInfo(q.newQueuedPodInfo(medPriorityPodInfo.Pod, queuePlugin)))
 				updatedPod := medPriorityPodInfo.Pod.DeepCopy()
 				updatedPod.Annotations["foo"] = "test1"
 				// Move clock by podInitialBackoffDuration, so that pods in the unschedulablePods would pass the backing off,
@@ -1175,7 +1175,7 @@ func TestPriorityQueue_UpdateWhenInflight(t *testing.T) {
 	// test-pod got rejected by fakePlugin,
 	// but the update event that it just got may change this scheduling result,
 	// and hence we should put this pod to activeQ/backoffQ.
-	err := q.AddUnschedulableIfNotPresent(logger, newQueuedPodInfoForLookup(updatedPod, "fakePlugin"), q.SchedulingCycle())
+	err := q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(newQueuedPodInfoForLookup(updatedPod, "fakePlugin")), q.SchedulingCycle())
 	if err != nil {
 		t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 	}
@@ -1519,7 +1519,7 @@ func TestPriorityQueue_MoveAllToActiveOrBackoffQueueWithQueueingHint(t *testing.
 		},
 		{
 			name:      "Queue queues pod to backoffQ if Pod is backing off",
-			podInfo:   &framework.QueuedPodInfo{PodInfo: mustNewPodInfo(p), UnschedulablePlugins: sets.New("foo")},
+			podInfo:   &framework.QueuedPodInfo{PodInfo: mustNewPodInfo(p), Attempts: 1, UnschedulablePlugins: sets.New("foo")},
 			hint:      queueHintReturnQueue,
 			expectedQ: backoffQ,
 		},
@@ -1550,6 +1550,12 @@ func TestPriorityQueue_MoveAllToActiveOrBackoffQueueWithQueueingHint(t *testing.
 			hint:      queueHintReturnQueue,
 			expectedQ: activeQ,
 		},
+		{
+			name:      "Pod that experienced a scheduling failure before should be queued to backoffQ after un-gated",
+			podInfo:   setQueuedPodInfoGated(&framework.QueuedPodInfo{PodInfo: mustNewPodInfo(p), Attempts: 1, UnschedulablePlugins: sets.New("foo")}),
+			hint:      queueHintReturnQueue,
+			expectedQ: backoffQ,
+		},
 	}
 
 	for _, test := range tests {
@@ -1618,11 +1624,11 @@ func TestPriorityQueue_MoveAllToActiveOrBackoffQueue(t *testing.T) {
 		t.Errorf("Expected: %v after Pop, but got: %v", highPriorityPodInfo.Pod.Name, p.Pod.Name)
 	}
 	expectInFlightPods(t, q, unschedulablePodInfo.Pod.UID, highPriorityPodInfo.Pod.UID)
-	err := q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(unschedulablePodInfo.Pod, "fooPlugin"), q.SchedulingCycle())
+	err := q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(q.newQueuedPodInfo(unschedulablePodInfo.Pod, "fooPlugin")), q.SchedulingCycle())
 	if err != nil {
 		t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 	}
-	err = q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(highPriorityPodInfo.Pod, "fooPlugin"), q.SchedulingCycle())
+	err = q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(q.newQueuedPodInfo(highPriorityPodInfo.Pod, "fooPlugin")), q.SchedulingCycle())
 	if err != nil {
 		t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 	}
@@ -1635,7 +1641,7 @@ func TestPriorityQueue_MoveAllToActiveOrBackoffQueue(t *testing.T) {
 	}
 	expectInFlightPods(t, q, hpp1.UID)
 	// This Pod will go to backoffQ because no failure plugin is associated with it.
-	err = q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(hpp1), q.SchedulingCycle())
+	err = q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(q.newQueuedPodInfo(hpp1)), q.SchedulingCycle())
 	if err != nil {
 		t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 	}
@@ -1648,7 +1654,7 @@ func TestPriorityQueue_MoveAllToActiveOrBackoffQueue(t *testing.T) {
 	}
 	expectInFlightPods(t, q, hpp2.UID)
 	// This Pod will go to the unschedulable Pod pool.
-	err = q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(hpp2, "barPlugin"), q.SchedulingCycle())
+	err = q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(q.newQueuedPodInfo(hpp2, "barPlugin")), q.SchedulingCycle())
 	if err != nil {
 		t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 	}
@@ -1691,9 +1697,9 @@ func TestPriorityQueue_MoveAllToActiveOrBackoffQueue(t *testing.T) {
 	if p, err := q.Pop(logger); err != nil || p.Pod != hpp1 {
 		t.Errorf("Expected: %v after Pop, but got: %v", hpp1, p.Pod.Name)
 	}
-	unschedulableQueuedPodInfo := q.newQueuedPodInfo(unschedulablePodInfo.Pod, "fooPlugin")
-	highPriorityQueuedPodInfo := q.newQueuedPodInfo(highPriorityPodInfo.Pod, "fooPlugin")
-	hpp1QueuedPodInfo := q.newQueuedPodInfo(hpp1)
+	unschedulableQueuedPodInfo := attemptQueuedPodInfo(q.newQueuedPodInfo(unschedulablePodInfo.Pod, "fooPlugin"))
+	highPriorityQueuedPodInfo := attemptQueuedPodInfo(q.newQueuedPodInfo(highPriorityPodInfo.Pod, "fooPlugin"))
+	hpp1QueuedPodInfo := attemptQueuedPodInfo(q.newQueuedPodInfo(hpp1))
 	expectInFlightPods(t, q, medPriorityPodInfo.Pod.UID, unschedulablePodInfo.Pod.UID, highPriorityPodInfo.Pod.UID, hpp1.UID)
 	err = q.AddUnschedulableIfNotPresent(logger, unschedulableQueuedPodInfo, q.SchedulingCycle())
 	if err != nil {
@@ -1757,25 +1763,25 @@ func TestPriorityQueue_MoveAllToActiveOrBackoffQueueWithOutQueueingHint(t *testi
 	// To simulate the pod is failed in scheduling in the real world, Pop() the pod from activeQ before AddUnschedulableIfNotPresent()s below.
 	q.Add(logger, medPriorityPodInfo.Pod)
 
-	err := q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(unschedulablePodInfo.Pod, "fooPlugin"), q.SchedulingCycle())
+	err := q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(q.newQueuedPodInfo(unschedulablePodInfo.Pod, "fooPlugin")), q.SchedulingCycle())
 	if err != nil {
 		t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 	}
-	err = q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(highPriorityPodInfo.Pod, "fooPlugin"), q.SchedulingCycle())
+	err = q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(q.newQueuedPodInfo(highPriorityPodInfo.Pod, "fooPlugin")), q.SchedulingCycle())
 	if err != nil {
 		t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 	}
 	// Construct a Pod, but don't associate its scheduler failure to any plugin
 	hpp1 := clonePod(highPriorityPodInfo.Pod, "hpp1")
 	// This Pod will go to backoffQ because no failure plugin is associated with it.
-	err = q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(hpp1), q.SchedulingCycle())
+	err = q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(q.newQueuedPodInfo(hpp1)), q.SchedulingCycle())
 	if err != nil {
 		t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 	}
 	// Construct another Pod, and associate its scheduler failure to plugin "barPlugin".
 	hpp2 := clonePod(highPriorityPodInfo.Pod, "hpp2")
 	// This Pod will go to the unschedulable Pod pool.
-	err = q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(hpp2, "barPlugin"), q.SchedulingCycle())
+	err = q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(q.newQueuedPodInfo(hpp2, "barPlugin")), q.SchedulingCycle())
 	if err != nil {
 		t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 	}
@@ -1803,9 +1809,9 @@ func TestPriorityQueue_MoveAllToActiveOrBackoffQueueWithOutQueueingHint(t *testi
 		}
 	}
 
-	unschedulableQueuedPodInfo := q.newQueuedPodInfo(unschedulablePodInfo.Pod, "fooPlugin")
-	highPriorityQueuedPodInfo := q.newQueuedPodInfo(highPriorityPodInfo.Pod, "fooPlugin")
-	hpp1QueuedPodInfo := q.newQueuedPodInfo(hpp1)
+	unschedulableQueuedPodInfo := attemptQueuedPodInfo(q.newQueuedPodInfo(unschedulablePodInfo.Pod, "fooPlugin"))
+	highPriorityQueuedPodInfo := attemptQueuedPodInfo(q.newQueuedPodInfo(highPriorityPodInfo.Pod, "fooPlugin"))
+	hpp1QueuedPodInfo := attemptQueuedPodInfo(q.newQueuedPodInfo(hpp1))
 	err = q.AddUnschedulableIfNotPresent(logger, unschedulableQueuedPodInfo, q.SchedulingCycle())
 	if err != nil {
 		t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
@@ -2071,11 +2077,11 @@ func TestPriorityQueue_PendingPods(t *testing.T) {
 		t.Errorf("Expected: %v after Pop, but got: %v", highPriorityPodInfo.Pod.Name, p.Pod.Name)
 	}
 	q.Add(logger, medPriorityPodInfo.Pod)
-	err := q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(unschedulablePodInfo.Pod, "plugin"), q.SchedulingCycle())
+	err := q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(q.newQueuedPodInfo(unschedulablePodInfo.Pod, "plugin")), q.SchedulingCycle())
 	if err != nil {
 		t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 	}
-	err = q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(highPriorityPodInfo.Pod, "plugin"), q.SchedulingCycle())
+	err = q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(q.newQueuedPodInfo(highPriorityPodInfo.Pod, "plugin")), q.SchedulingCycle())
 	if err != nil {
 		t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 	}
@@ -2720,6 +2726,7 @@ var (
 				Reason:  v1.PodReasonUnschedulable,
 				Message: "fake scheduling failure",
 			})
+			pInfo = attemptQueuedPodInfo(pInfo)
 		}
 		queue.unschedulablePods.addOrUpdate(pInfo)
 	}
@@ -2859,6 +2866,16 @@ func TestPendingPodsMetric(t *testing.T) {
 	totalWithDelay := 20
 	pInfosWithDelay := makeQueuedPodInfos(totalWithDelay, "z", queueable, timestamp.Add(2*time.Second))
 
+	resetPodInfos := func() {
+		// reset PodInfo's Attempts because they influence the backoff time calculation.
+		for i := range pInfos {
+			pInfos[i].Attempts = 0
+		}
+		for i := range pInfosWithDelay {
+			pInfosWithDelay[i].Attempts = 0
+		}
+	}
+
 	tests := []struct {
 		name                       string
 		operations                 []operation
@@ -3093,6 +3110,7 @@ scheduler_plugin_execution_duration_seconds_count{extension_point="PreEnqueue",p
 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
 			resetMetrics()
+			resetPodInfos()
 			logger, ctx := ktesting.NewTestContext(t)
 			ctx, cancel := context.WithCancel(ctx)
 			defer cancel()
@@ -3465,7 +3483,7 @@ func TestMoveAllToActiveOrBackoffQueue_PreEnqueueChecks(t *testing.T) {
 					t.Errorf("Expected: %v after Pop, but got: %v", podInfo.Pod.Name, p.Pod.Name)
 				}
 				podInfo.UnschedulablePlugins = sets.New("plugin")
-				err := q.AddUnschedulableIfNotPresent(logger, podInfo, q.activeQ.schedulingCycle())
+				err := q.AddUnschedulableIfNotPresent(logger, attemptQueuedPodInfo(podInfo), q.activeQ.schedulingCycle())
 				if err != nil {
 					t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err)
 				}
@@ -3870,3 +3888,8 @@ func Test_queuedPodInfo_gatedSetUponCreationAndUnsetUponUpdate(t *testing.T) {
 		t.Error("Expected pod to be ungated")
 	}
 }
+
+func attemptQueuedPodInfo(podInfo *framework.QueuedPodInfo) *framework.QueuedPodInfo {
+	podInfo.Attempts++
+	return podInfo
+}