Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- We may delete podS though they may be waiting in Do not delete waiting pods if they are not scheduled. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc WDYT ?
It sounds to me that you prefer to prioritize waiting Pod over other assigned Pods. In that case, you may want to exclude waiting Pods in the earlier Filtering/Sorting Victims step, instead of simply disregard the deletion in PrepareCandidate(). In general, when Permit plugins work along with PostFilter (preemption), the logic could be complicated. There may not be a general preemption implementation (probably in the future we can exercise pluginArgs to provide different options). So for now the best bet is to implement your own PostFilter plugin.
Got it! But for the default preemption plugin, we delete the victims to free resources. If pod has assigned, we should delete it. If pod has not assigned, we just need reject it in WaitingPod because the pod just exists in the cache.
Oh, your point was to verify if it's a waiting pod first, and then delete it? If so, then it makes sense to me. Feel free to raise a PR. /assign
Yes, I will open a PR to do this

Discussion:
// - Clear the low-priority pods' nominatedNodeName status if needed func PrepareCandidate(c Candidate, fh framework.Handle, cs kubernetes.Interface, pod *v1.Pod, pluginName string) *framework.Status { for _, victim := range c.Victims().Pods { <del> if err := util.DeletePod(cs, victim); err != nil { klog.ErrorS(err, "preempting pod", "pod", klog.KObj(victim)) return framework.AsStatus(err) } // If the victim is a WaitingPod, send a reject message to the PermitPlugin </del> <ins> // If the victim is a WaitingPod, send a reject message to the PermitPlugin. // Otherwise we should delete the victim. </ins> if waitingPod := fh.GetWaitingPod(victim.UID); waitingPod != nil { waitingPod.Reject(pluginName, "preempted") <ins> } else if err := util.DeletePod(cs, victim); err != nil { klog.ErrorS(err, "Preempting pod", "pod", klog.KObj(victim), "preemptor", klog.KObj(pod)) return framework.AsStatus(err) </ins> } fh.EventRecorder().Eventf(victim, pod, v1.EventTypeNormal, "Preempted", "Preempting", "Preempted by %v/%v on node %v", pod.Namespace, pod.Name, c.Name())

"time" v1 "k8s.io/api/core/v1" <del> apierrors "k8s.io/apimachinery/pkg/api/errors" </del> <ins> "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime"

permitPlugin.timeoutPermit = false permitPlugin.waitAndRejectPermit = false permitPlugin.waitAndAllowPermit = true <ins> permitPlugin.waitingPod = "waiting-pod" </ins> lowPriority, highPriority := int32(100), int32(300) resourceRequest := v1.ResourceRequirements{Requests: v1.ResourceList{ <ins> v1.ResourceCPU:    *resource.NewMilliQuantity(200, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(200, resource.DecimalSI)}, } preemptorResourceRequest := v1.ResourceRequirements{Requests: v1.ResourceList{ </ins> v1.ResourceCPU:    *resource.NewMilliQuantity(400, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(400, resource.DecimalSI)}, } <del> // First pod will go waiting. </del> <ins> // First pod will go running. runningPod := initPausePod(&pausePodConfig{Name: "running-pod", Namespace: testCtx.NS.Name, Priority: &lowPriority, Resources: &resourceRequest}) runningPod.Spec.TerminationGracePeriodSeconds = new(int64) runningPod, err = createPausePod(testCtx.ClientSet, runningPod) if err != nil { t.Errorf("Error while creating the waiting pod: %v", err) } // Wait until the pod scheduled, then create a preemptor pod to preempt it. wait.Poll(100*time.Millisecond, 30*time.Second, podScheduled(testCtx.ClientSet, runningPod.Name, runningPod.Namespace)) // Second pod will go waiting. </ins> waitingPod := initPausePod(&pausePodConfig{Name: "waiting-pod", Namespace: testCtx.NS.Name, Priority: &lowPriority, Resources: &resourceRequest}) waitingPod.Spec.TerminationGracePeriodSeconds = new(int64) waitingPod, err = createPausePod(testCtx.ClientSet, waitingPod)

return w, nil }) <del> // Create second pod which should preempt first pod. </del> <ins> // Create third pod which should preempt other pods. </ins> preemptorPod, err := createPausePod(testCtx.ClientSet, <del> initPausePod(&pausePodConfig{Name: "preemptor-pod", Namespace: testCtx.NS.Name, Priority: &highPriority, Resources: &resourceRequest})) </del> <ins> initPausePod(&pausePodConfig{Name: "preemptor-pod", Namespace: testCtx.NS.Name, Priority: &highPriority, Resources: &preemptorResourceRequest})) </ins> if err != nil { t.Errorf("Error while creating the preemptor pod: %v", err) }

// } if err := wait.Poll(200*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { <del> _, err := getPod(testCtx.ClientSet, waitingPod.Name, waitingPod.Namespace) return apierrors.IsNotFound(err), nil </del> <ins> w := false permitPlugin.fh.IterateOverWaitingPods(func(wp framework.WaitingPod) { w = true }) return !w, nil </ins> }); err != nil { <del> t.Error("Expected the waiting pod to get preempted and deleted") </del> <ins> t.Error("Expected the waiting pod to get preempted") } // Expect the waitingPod to be still present. if _, err := getPod(testCtx.ClientSet, waitingPod.Name, waitingPod.Namespace); err != nil { t.Error("Get waiting pod in waiting pod failed.") } // Expect the runningPod to be deleted physically. _, err = getPod(testCtx.ClientSet, runningPod.Name, runningPod.Namespace) if err != nil && !errors.IsNotFound(err) { t.Error("Get running pod failed.") } if err == nil { t.Error("Running pod still exist.") </ins> } if permitPlugin.numPermitCalled == 0 { t.Errorf("Expected the permit plugin to be called.") } permitPlugin.reset() <del> testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{waitingPod, preemptorPod}) </del> <ins> testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{waitingPod, runningPod, preemptorPod}) </ins> } func initTestSchedulerForFrameworkTest(t *testing.T, testCtx *testutils.TestContext, nodeCount int, opts ...scheduler.Option) *testutils.TestContext {