Repository: kubernetes/kubernetes (language: go)

Issue:
Expand UnschedulablePlugins/PendingPlugins to include PreBind plugins
Currently, the scheduler honors Unschedulable(AndUnresolvable) or Pending **_only_** from PreFilter, Filter, Reserve, and Permit (WaitOnPermit).
https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/framework/types.go#L210-L214

It doesn't expect other extension points to return those statuses and just regard all non-success status as errors, that is, immediately retrying scheduling of Pods.

Actually [DRA](https://github.com/kubernetes/kubernetes/blob/e67f889edc4ab278028f6cffd2501bc90a0defcf/pkg/scheduler/framework/plugins/dynamicresources/dynamicresources.go#L1613) returns Pending at PreBind, which is currently being ignored as described.
So, we should expand UnschedulablePlugins/PendingPlugins to include PreBind plugins. (or maybe all extension points too?)

/cc @pohly 
/sig scheduling
/kind feature


Discussion / hints:
This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the `triage/accepted` label and provide further guidance.

The `triage/accepted` label can be added by org members by writing `/triage accepted` in a comment.


<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
> So, we should expand UnschedulablePlugins/PendingPlugins to include PreBind plugins. (or maybe all extension points too?)

Both seems reasonable to me.


Because `Pending` is not handled for `PreBind`, the following error is logged:

```
E0604 15:45:50.980929  306340 schedule_one.go:1048] "Error scheduling pod; retrying" err="waiting for resource driver" pod="test/test-draqld28"
```

This patch gets rid of the error. I am uncertain whether it has other implications? Performance was the same with and without it.

```patch
diff --git a/pkg/scheduler/schedule_one.go b/pkg/scheduler/schedule_one.go
index 6b6992095d8..8c7868fedec 100644
--- a/pkg/scheduler/schedule_one.go
+++ b/pkg/scheduler/schedule_one.go
@@ -292,6 +292,17 @@ func (sched *Scheduler) bindingCycle(
 
        // Run "prebind" plugins.
        if status := fwk.RunPreBindPlugins(ctx, state, assumedPod, scheduleResult.SuggestedHost); !status.IsSuccess() {
+               if status.IsRejected() {
+                       fitErr := &framework.FitError{
+                               NumAllNodes: 1,
+                               Pod:         assumedPodInfo.Pod,
+                               Diagnosis: framework.Diagnosis{
+                                       NodeToStatusMap:      framework.NodeToStatusMap{scheduleResult.SuggestedHost: status},
+                                       UnschedulablePlugins: sets.New(status.Plugin()),
+                               },
+                       }
+                       return framework.NewStatus(status.Code()).WithError(fitErr)
+               }
                return status
        }
 
```
What I imagined is like that. It shouldn't have any other impact because volumebinding, which is the only PreBind plugin apart from DRA in in-tree, only returns success or error status. 
https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/framework/plugins/volumebinding/volume_binding.go#L355-L378

We have to mention the change in the release note for custom plugin developers though.

Fix patch:
diff --git a/pkg/scheduler/framework/types.go b/pkg/scheduler/framework/types.go
index 687143e520afd..9b4c0b874c43d 100644
--- a/pkg/scheduler/framework/types.go
+++ b/pkg/scheduler/framework/types.go
@@ -208,7 +208,7 @@ type QueuedPodInfo struct {
 	// latency for a pod.
 	InitialAttemptTimestamp *time.Time
 	// UnschedulablePlugins records the plugin names that the Pod failed with Unschedulable or UnschedulableAndUnresolvable status.
-	// It's registered only when the Pod is rejected in PreFilter, Filter, Reserve, or Permit (WaitOnPermit).
+	// It's registered only when the Pod is rejected in PreFilter, Filter, Reserve, PreBind or Permit (WaitOnPermit).
 	UnschedulablePlugins sets.Set[string]
 	// PendingPlugins records the plugin names that the Pod failed with Pending status.
 	PendingPlugins sets.Set[string]
diff --git a/pkg/scheduler/schedule_one.go b/pkg/scheduler/schedule_one.go
index 33f68468d2c2f..9324a7f1e120e 100644
--- a/pkg/scheduler/schedule_one.go
+++ b/pkg/scheduler/schedule_one.go
@@ -292,6 +292,17 @@ func (sched *Scheduler) bindingCycle(
 
 	// Run "prebind" plugins.
 	if status := fwk.RunPreBindPlugins(ctx, state, assumedPod, scheduleResult.SuggestedHost); !status.IsSuccess() {
+		if status.IsRejected() {
+			fitErr := &framework.FitError{
+				NumAllNodes: 1,
+				Pod:         assumedPodInfo.Pod,
+				Diagnosis: framework.Diagnosis{
+					NodeToStatusMap:      framework.NodeToStatusMap{scheduleResult.SuggestedHost: status},
+					UnschedulablePlugins: sets.New(status.Plugin()),
+				},
+			}
+			return framework.NewStatus(status.Code()).WithError(fitErr)
+		}
 		return status
 	}
 


Test patch: