Change:
When a pod requires preempting more than one pod, this happens sequentially. It should be done in parallel. /sig scheduling /sig scalability Preemption happens in the scheduling cycle, meaning that no other scheduling attempts happen while preemption is ongoing, hurting throughput. An alternative is to the deletes in a separate routine, like we do for binding.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I'm a newbie, may I have a try at this proposal?
Yes, this is a good candidate for new contributors. However, I would like to hear from SIG scalability whether there are any concerns about this.
Parallelizing the loops looks like the easier option to implement, but deleting in a separate routine will likely offer the better throughput for the average case.
Parallelizing loops seems fine from SIG Scalability perspective - we already do something similar for example in KCM when creating pods or in scheduler each binding is done by separate goroutine. One question though, is there a limit of pods that we want to preempt here? 100 parallel calls will be probably fine, but 1000 wouldn't be most likely.
We only preempt pods from a single node, so that's a natural threshold. And we have a parallelizing mini-library that has 16 workers by default.
Hi, do you still want to work on this one?
YesIt's still under discussion.
So we can use the parallelizing mini-library to do this?
You can use this parallelizer we defined in scheduler framework directly.
feel free to take it by writing on a comment.
/assign

Discussion:
apipod "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/scheduler/framework" <ins> "k8s.io/kubernetes/pkg/scheduler/framework/parallelize" </ins> "k8s.io/kubernetes/pkg/scheduler/metrics" "k8s.io/kubernetes/pkg/scheduler/util" )

func (ev *Evaluator) prepareCandidate(ctx context.Context, c Candidate, pod *v1.Pod, pluginName string) *framework.Status { fh := ev.Handler cs := ev.Handler.ClientSet() <del> for _, victim := range c.Victims().Pods { </del> <ins> ctx, cancel := context.WithCancel(ctx) defer cancel() errCh := parallelize.NewErrorChannel() preemptPod := func(index int) { victim := c.Victims().Pods[index] </ins> // If the victim is a WaitingPod, send a reject message to the PermitPlugin. // Otherwise we should delete the victim. if waitingPod := fh.GetWaitingPod(victim.UID); waitingPod != nil {

if apipod.UpdatePodCondition(newStatus, condition) { if err := util.PatchPodStatus(ctx, cs, victim, newStatus); err != nil { klog.ErrorS(err, "Preparing pod preemption", "pod", klog.KObj(victim), "preemptor", klog.KObj(pod)) <del> return framework.AsStatus(err) </del> <ins> errCh.SendErrorWithCancel(err, cancel) return </ins> } } } if err := util.DeletePod(ctx, cs, victim); err != nil { klog.ErrorS(err, "Preempting pod", "pod", klog.KObj(victim), "preemptor", klog.KObj(pod)) <del> return framework.AsStatus(err) </del> <ins> errCh.SendErrorWithCancel(err, cancel) return </ins> } } fh.EventRecorder().Eventf(victim, pod, v1.EventTypeNormal, "Preempted", "Preempting", "Preempted by %v/%v on node %v", pod.Namespace, pod.Name, c.Name()) } <ins> fh.Parallelizer().Until(ctx, len(c.Victims().Pods), preemptPod, ev.PluginName) if err := errCh.ReceiveError(); err != nil { return framework.AsStatus(err) } </ins> metrics.PreemptionVictims.Observe(float64(len(c.Victims().Pods))) // Lower priority pods nominated to run on this node, may no longer fit on