Change:
In k8s 1.27, scheduler introduced a new "short circuit" semantics in PreFilter to bypass a Pod quickly. A change introduced here is it relies on that plugin to be stored in , which makes sense as what it was explained . However, the info, although is populated upon each plugin's PreFilter invocation, is not injected into when some PreFilter returns unschedulable: So think about this workflow: pluginA returns Skip in its PreFilter pluginB return Unschedulable in its PreFilter note: pluginA is not included in due to aforementioned bug a Pod is claimed as unschedulable (failed by pluginB) during PostFilter, it's supposed to bypass pluginA's PreEnqueueExtension's AddPod/RemovePod, but it doesn't calling a "skipped" Plugin's AddPod/RemovePod during preemption, sometimes will abort the whole preemption as the logic tries to read info from cycleState, and some plugin's "skip" version doesn't fill cycleState properly, e.g., InterPodAffinity (skip immediately w/o write preFilterStateKey). Short circuiting using Skip is fine, but should injected properly at all cases. Described in the workflow above. It's firstly captured in an out-of-tree PostFilter plugin: , but can also happen in in-tree plugins. No response v1.27 <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind regression /sig scheduling
cc
Among in-tree plugins, 3 plugins employ non-nil and may return Skip in their PreFilter() implementation w/o writing to cycleState, while return error immediately if is missing in their Add/RemovePod() functions. So technically, as long as a scheduling flow hit the following criteria, it would hit the bug (preemption doesn't function): plugin 1/2/3 return Skip in PreFilter arbitrary PreFilter plugin after the "skipped" plugin returns Unschedulable in its PreFilter
/cc
/cc
Cherry-pick PRs: 1.28: 1.27:

Discussion:
// If a non-success status is returned, then the scheduling cycle is aborted. func (f *frameworkImpl) RunPreFilterPlugins(ctx context.Context, state *framework.CycleState, pod *v1.Pod) (_ *framework.PreFilterResult, status *framework.Status) { startTime := time.Now() <ins> skipPlugins := sets.New[string]() </ins> defer func() { <ins> state.SkipFilterPlugins = skipPlugins </ins> metrics.FrameworkExtensionPointDuration.WithLabelValues(metrics.PreFilter, status.Code().String(), f.profileName).Observe(metrics.SinceInSeconds(startTime)) }() var result *framework.PreFilterResult var pluginsWithNodes []string <del> skipPlugins := sets.New[string]() </del> logger := klog.FromContext(ctx) logger = klog.LoggerWithName(logger, "PreFilter") // TODO(knelasevero): Remove duplicated keys from log entry calls

return nil, framework.NewStatus(framework.Unschedulable, msg) } } <del> state.SkipFilterPlugins = skipPlugins </del> return result, nil }

inj:  injectedResult{PreFilterStatus: int(framework.Error)}, }, }, <del> wantPreFilterResult: nil, wantStatusCode:      framework.Error, </del> <ins> wantSkippedPlugins: sets.New("skip"), wantStatusCode:     framework.Error, </ins> }, { name: "all PreFilter plugins returned skip",

const tokenFilterName = "token-filter" <ins> // tokenFilter is a fake plugin that implements PreFilter and Filter. // `Token` simulates the allowed pods number a cluster can host. // If `EnablePreFilter` is set to false or `Token` is positive, PreFilter passes; otherwise returns Unschedulable // For each Filter() call, `Token` is decreased by one. When `Token` is positive, Filter passes; otherwise return // Unschedulable or UnschedulableAndUnresolvable (when `Unresolvable` is set to true) // AddPod()/RemovePod() adds/removes one token to the cluster to simulate the dryrun preemption </ins> type tokenFilter struct { <del> Tokens       int Unresolvable bool </del> <ins> Tokens          int Unresolvable    bool EnablePreFilter bool </ins> } // Name returns name of the plugin.

} func (fp *tokenFilter) PreFilter(ctx context.Context, state *framework.CycleState, pod *v1.Pod) (*framework.PreFilterResult, *framework.Status) { <del> return nil, nil </del> <ins> if !fp.EnablePreFilter || fp.Tokens > 0 { return nil, nil } return nil, framework.NewStatus(framework.Unschedulable) </ins> } func (fp *tokenFilter) AddPod(ctx context.Context, state *framework.CycleState, podToSchedule *v1.Pod,

existingPods                  []*v1.Pod pod                           *v1.Pod initTokens                    int <ins> enablePreFilter               bool </ins> unresolvable                  bool preemptedPodIndexes           map[int]struct{} enablePodDisruptionConditions bool

}), preemptedPodIndexes: map[int]struct{}{0: {}}, }, <ins> // This is identical with previous subtest except for setting enablePreFilter to true. // With this fake plugin returning Unschedulable in PreFilter, it's able to exercise the path // that in-tree plugins return Skip in PreFilter and their AddPod/RemovePod functions are also // skipped properly upon preemption. { name:            "basic pod preemption with preFilter", initTokens:      1, enablePreFilter: true, existingPods: []*v1.Pod{ initPausePod(&testutils.PausePodConfig{ Name:      "victim-pod", Namespace: testCtx.NS.Name, Priority:  &lowPriority, Resources: &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(200, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(200, resource.DecimalSI)}, }, }), }, pod: initPausePod(&testutils.PausePodConfig{ Name:      "preemptor-pod", Namespace: testCtx.NS.Name, Priority:  &highPriority, Resources: &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(200, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(200, resource.DecimalSI)}, }, }), preemptedPodIndexes: map[int]struct{}{0: {}}, }, </ins> { // same as the previous test, but the filter is unresolvable. name:         "basic pod preemption with unresolvable filter",

t.Run(test.name, func(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, feature.DefaultFeatureGate, features.PodDisruptionConditions, test.enablePodDisruptionConditions)() filter.Tokens = test.initTokens <ins> filter.EnablePreFilter = test.enablePreFilter </ins> filter.Unresolvable = test.unresolvable pods := make([]*v1.Pod, len(test.existingPods)) // Create and run existingPods.