Change:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : The scheduler places unschedulable pods in "unschedulabe" queue and retries them only when certain events happen that could potentially make them schedulable. This logic works well in almost all scenarios, but inevitable race condition in large distributed systems, could potentially cause some events to be seen before pods are to the unschedulable queue. If this happens, pods may be left in the unschedulable queue and not be retried. Such scenarios should be rare and even if they occur, usually there are other events that trigger a retry and cover them. However, if such scenarios happen in smaller and low churn clusters, other events may not be seen for a while and pods may be stuck in the unschedulable queue for a long time. The scheduling queue () has a method that is triggered every second to move backed off pods to the active queue. We should add another go routine that is triggered every 30 seconds and moves pods whose last schedule time is older than 1 minute from "unschedulble" queue to the active queue. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug I don't know if this should be marked as a bug. I don't have any report of such issue. It is more of a precaution to prevent issues. /sig scheduling
I am worried if this will lead to continuous repeated scheduling failures?
What is the problem with repeated scheduling failures? The scheduler had a FIFO until recently. It used to keep retrying unschedulable pods all the time.
can I take this?
Sure, thanks!
GitHub didn't allow me to assign the following users: denkensk. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It increases the schedule possibility of unschedulble pods and the chances of scheduling failures. Should we also consider latter?

Discussion:
queueClosed = "scheduling queue is closed" ) <ins> // If the pod stays in unschedulableQ longer than the unschedulableQTimeInterval, // the pod will be moved from unschedulableQ to activeQ. const unschedulableQTimeInterval = 60 * time.Second </ins> // SchedulingQueue is an interface for a queue to store pods waiting to be scheduled. // The interface follows a pattern similar to cache.FIFO and cache.Heap and // makes it easy to use those data structures as a SchedulingQueue.

// run starts the goroutine to pump from podBackoffQ to activeQ func (p *PriorityQueue) run() { go wait.Until(p.flushBackoffQCompleted, 1.0*time.Second, p.stop) <ins> go wait.Until(p.flushUnschedulableQLeftover, 30*time.Second, p.stop) </ins> } // Add adds a pod to the active queue. It should be called only when a new pod

} } <ins> // flushUnschedulableQLeftover moves pod which stays in unschedulableQ longer than the durationStayUnschedulableQ // to activeQ. func (p *PriorityQueue) flushUnschedulableQLeftover() { p.lock.Lock() defer p.lock.Unlock() var podsToMove []*v1.Pod currentTime := p.clock.Now() for _, pod := range p.unschedulableQ.pods { lastScheduleTime := podTimestamp(pod) if !lastScheduleTime.IsZero() && currentTime.Sub(lastScheduleTime.Time) > unschedulableQTimeInterval { podsToMove = append(podsToMove, pod) } } if len(podsToMove) > 0 { p.movePodsToActiveQueue(podsToMove) } } </ins> // Pop removes the head of the active queue and returns it. It blocks if the // activeQ is empty and waits until a new item is added to the queue. It also // clears receivedMoveRequest to mark the beginning of a new scheduling cycle.

"reflect" "sync" "testing" <ins> "time" </ins> "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

t.Errorf("Expected to get mid prority pod, got: %v", p) } } <ins> // TestHighProirotyFlushUnschedulableQLeftover tests that pods will be moved to // activeQ after one minutes if it is in unschedulableQ func TestHighProirotyFlushUnschedulableQLeftover(t *testing.T) { q := NewPriorityQueue(nil) midPod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "test-midpod", Namespace: "ns1", UID:       types.UID("tp-mid"), }, Spec: v1.PodSpec{ Priority: &midPriority, }, Status: v1.PodStatus{ NominatedNodeName: "node1", }, } highPod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "test-highpod", Namespace: "ns1", UID:       types.UID("tp-high"), }, Spec: v1.PodSpec{ Priority: &highPriority, }, Status: v1.PodStatus{ NominatedNodeName: "node1", }, } q.unschedulableQ.addOrUpdate(&highPod) q.unschedulableQ.addOrUpdate(&midPod) // Update pod condition to highPod. podutil.UpdatePodCondition(&highPod.Status, &v1.PodCondition{ Type:          v1.PodScheduled, Status:        v1.ConditionFalse, Reason:        v1.PodReasonUnschedulable, Message:       "fake scheduling failure", LastProbeTime: metav1.Time{Time: time.Now().Add(-1 * unschedulableQTimeInterval)}, }) // Update pod condition to midPod. podutil.UpdatePodCondition(&midPod.Status, &v1.PodCondition{ Type:          v1.PodScheduled, Status:        v1.ConditionFalse, Reason:        v1.PodReasonUnschedulable, Message:       "fake scheduling failure", LastProbeTime: metav1.Time{Time: time.Now().Add(-1 * unschedulableQTimeInterval)}, }) if p, err := q.Pop(); err != nil || p != &highPod { t.Errorf("Expected: %v after Pop, but got: %v", highPriorityPod.Name, p.Name) } if p, err := q.Pop(); err != nil || p != &midPod { t.Errorf("Expected: %v after Pop, but got: %v", medPriorityPod.Name, p.Name) } } </ins>