Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In large clusters, events that trigger rescheduling of the unschedulable pods are frequent. For example, in a cluster with 1000 nodes, node updates, which are often just heartbeats, may happen at an average rate of 100/s. These events trigger rescheduling of unschedulable pods. The scheduler moves unschedulable pods to its active scheduling queue when such events are observed. There are two particularly problematic scenarios: are high-priority unschedulable pods and preemption does not help make them schedulable. For example, they may have node affinity that cannot be satisfied, or they may have pod affinity to other pods that do not exist. Such high priority pods may go to the head of the active scheduling queue and block everything behind them. are many pending pods with the same priority, but a number of them are unschedulable. When these unschedulable pods are moved to the active queue, they are expected to go behind other pods in the queue, but since our queue sorts pods by priority and these pods have the same priority, their order is unspecified. The Pop() algorithm of the heap swaps the head of the queue with the tail, effectively causing the heap to act similar to a LIFO. So, pods which are tried recently, go to the head with a high probability and are retried. This starves pods behind them in the queue. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig scheduling /assign
For case 1: How about add an field to record how many times scheduler have tried to schedule it? The bigger it is, the latter we should schedule it. Then we could consider to add it as a factor for sorting pods.
Yes, we basically need a . Another approach is to . We will pursue both of these approaches.
Well differentiating optionsounds better
Agree. On the one hand we need to reduce the frequency of unnecessary rescheduling, on the other hand we need to optimize the sorting algorithm. And the would help to reduce the frequency. But I think "differentiate between node heartbeats vs. actual node updates" is a better approach.
Yes its a better approach indeed but what if we use the recursion there
We should have all three changes to address all known scenarios efficiently. The three changes are: back off mechanism fair sorting of pods in the scheduling queue smarter processing of node updates Node updates are not the only trigger for the scheduler to retry unschedulable pods. The scheduler also retries when running pods terminate, among a couple of other events. So, without having backoff mechanism a high priority unschedulable pod could still block head of the queue in clusters with high churn, for example, in clusters where many pods terminate frequently. Back off mechanism by itself addresses the issue in all scenarios, but the other two changes make the scheduling queue smarter and more efficient.
The three prs above are quite comprehensive : ).
All of the above items are done already.

Discussion:
if condition == nil { return &pod.CreationTimestamp } <del> return &condition.LastTransitionTime </del> <ins> if condition.LastProbeTime.IsZero() { return &condition.LastTransitionTime } return &condition.LastProbeTime </ins> } // activeQComp is the function used by the activeQ heap algorithm to sort pods.

} // Update pod condition to unschedulable. podutil.UpdatePodCondition(&p1.Status, &v1.PodCondition{ <del> Type:    v1.PodScheduled, Status:  v1.ConditionFalse, Reason:  v1.PodReasonUnschedulable, Message: "fake scheduling failure", </del> <ins> Type:          v1.PodScheduled, Status:        v1.ConditionFalse, Reason:        v1.PodReasonUnschedulable, Message:       "fake scheduling failure", LastProbeTime: metav1.Now(), </ins> }) // Put in the unschedulable queue. q.AddUnschedulableIfNotPresent(p1)

} } <ins> // TestPodFailedSchedulingMultipleTimesDoesNotBlockNewerPod tests // that a pod determined as unschedulable multiple times doesn't block any newer pod. // This behavior ensures that an unschedulable pod does not block head of the queue when there // are frequent events that move pods to the active queue. func TestPodFailedSchedulingMultipleTimesDoesNotBlockNewerPod(t *testing.T) { q := NewPriorityQueue(nil) // Add an unschedulable pod to a priority queue. // This makes a situation that the pod was tried to schedule // and had been determined unschedulable so far. unschedulablePod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "test-pod-unscheduled", Namespace: "ns1", UID:       "tp001", }, Spec: v1.PodSpec{ Priority: &highPriority, }, Status: v1.PodStatus{ NominatedNodeName: "node1", }, } // Update pod condition to unschedulable. podutil.UpdatePodCondition(&unschedulablePod.Status, &v1.PodCondition{ Type:          v1.PodScheduled, Status:        v1.ConditionFalse, Reason:        v1.PodReasonUnschedulable, Message:       "fake scheduling failure", LastProbeTime: metav1.Now(), }) // Put in the unschedulable queue q.AddUnschedulableIfNotPresent(&unschedulablePod) // Clear its backoff to simulate backoff its expiration q.clearPodBackoff(&unschedulablePod) // Move all unschedulable pods to the active queue. q.MoveAllToActiveQueue() // Simulate a pod being popped by the scheduler, // At this time, unschedulable pod should be popped. p1, err := q.Pop() if err != nil { t.Errorf("Error while popping the head of the queue: %v", err) } if p1 != &unschedulablePod { t.Errorf("Expected that test-pod-unscheduled was popped, got %v", p1.Name) } // Assume newer pod was added just after unschedulable pod // being popped and before being pushed back to the queue. newerPod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:              "test-newer-pod", Namespace:         "ns1", UID:               "tp002", CreationTimestamp: metav1.Now(), }, Spec: v1.PodSpec{ Priority: &highPriority, }, Status: v1.PodStatus{ NominatedNodeName: "node1", }, } q.Add(&newerPod) // And then unschedulablePod was determined as unschedulable AGAIN. podutil.UpdatePodCondition(&unschedulablePod.Status, &v1.PodCondition{ Type:          v1.PodScheduled, Status:        v1.ConditionFalse, Reason:        v1.PodReasonUnschedulable, Message:       "fake scheduling failure", LastProbeTime: metav1.Now(), }) // And then, put unschedulable pod to the unschedulable queue q.AddUnschedulableIfNotPresent(&unschedulablePod) // Clear its backoff to simulate its backoff expiration q.clearPodBackoff(&unschedulablePod) // Move all unschedulable pods to the active queue. q.MoveAllToActiveQueue() // At this time, newerPod should be popped // because it is the oldest tried pod. p2, err2 := q.Pop() if err2 != nil { t.Errorf("Error while popping the head of the queue: %v", err2) } if p2 != &newerPod { t.Errorf("Expected that test-newer-pod was popped, got %v", p2.Name) } } </ins> // TestHighPriorityBackoff tests that a high priority pod does not block // other pods if it is unschedulable func TestHighProirotyBackoff(t *testing.T) {

sched.config.Error(pod, err) sched.config.Recorder.Event(pod, v1.EventTypeWarning, "FailedScheduling", message) sched.config.PodConditionUpdater.Update(pod, &v1.PodCondition{ <del> Type:    v1.PodScheduled, Status:  v1.ConditionFalse, Reason:  reason, Message: err.Error(), </del> <ins> Type:          v1.PodScheduled, Status:        v1.ConditionFalse, LastProbeTime: metav1.Now(), Reason:        reason, Message:       err.Error(), </ins> }) }