Repository: kubernetes/kubernetes
Pull request #130622: DRA: Prioritized Alternatives in Device Requests, II
Author: login: pohly
Labels: name: area/test
name: priority/important-soon
description: Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
name: kind/cleanup
description: Categorizes issue or PR as related to cleaning up code, process, or technical debt.
name: sig/scheduling
description: Categorizes an issue or PR as relevant to SIG Scheduling.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: sig/node
description: Categorizes an issue or PR as relevant to SIG Node.
name: sig/api-machinery
description: Categorizes an issue or PR as relevant to SIG API Machinery.
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
name: release-note-none
description: Denotes a PR that doesn't merit a release note.
name: sig/apps
description: Categorizes an issue or PR as relevant to SIG Apps.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: sig/testing
description: Categorizes an issue or PR as relevant to SIG Testing.
name: area/code-generation
name: triage/accepted
description: Indicates an issue or PR is ready to be actively worked on.
name: wg/device-management
description: Categorizes an issue or PR as relevant to WG Device Management.
Created: 2025-03-06T18:59:00Z | Merged: 2025-03-10T13:51:51Z

#### What type of PR is this?

/kind cleanup

#### What this PR does / why we need it:

This is a follow-up to address some review comments in https://github.com/kubernetes/kubernetes/pull/128586:
- Abort scheduling of a pod sooner in PreFilter when it depends on the feature and the feature is disabled (https://github.com/kubernetes/kubernetes/pull/128586#discussion_r1982767989).
- Add some integration tests (https://github.com/kubernetes/kubernetes/pull/128586#discussion_r1982759300).

#### Which issue(s) this PR fixes:

Related-to: https://github.com/kubernetes/enhancements/issues/4816

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?

NONE because this is not a functional changed compared to 1.32. Listing the feature once is enough.

```release-note
NONE
```



Diff:
diff --git a/pkg/scheduler/framework/plugins/dynamicresources/dynamicresources.go b/pkg/scheduler/framework/plugins/dynamicresources/dynamicresources.go
index 171e17e7c6869..3b792d0bd1775 100644
--- a/pkg/scheduler/framework/plugins/dynamicresources/dynamicresources.go
+++ b/pkg/scheduler/framework/plugins/dynamicresources/dynamicresources.go
@@ -415,6 +415,9 @@ func (pl *DynamicResources) PreFilter(ctx context.Context, state *framework.Cycl
 						return nil, status
 					}
 				} else {
+					if !pl.enablePrioritizedList {
+						return nil, statusUnschedulable(logger, fmt.Sprintf("resource claim %s, request %s: has subrequests, but the DRAPrioritizedList feature is disabled", klog.KObj(claim), request.Name))
+					}
 					for _, subRequest := range request.FirstAvailable {
 						qualRequestName := strings.Join([]string{request.Name, subRequest.Name}, "/")
 						if status := pl.validateDeviceClass(logger, subRequest.DeviceClassName, qualRequestName); status != nil {
diff --git a/pkg/scheduler/framework/plugins/dynamicresources/dynamicresources_test.go b/pkg/scheduler/framework/plugins/dynamicresources/dynamicresources_test.go
index dd79e6dcacabf..0b85ac77aacfa 100644
--- a/pkg/scheduler/framework/plugins/dynamicresources/dynamicresources_test.go
+++ b/pkg/scheduler/framework/plugins/dynamicresources/dynamicresources_test.go
@@ -861,10 +861,11 @@ func TestPlugin(t *testing.T) {
 			claims:                   []*resourceapi.ResourceClaim{claimWithPrioritzedList},
 			classes:                  []*resourceapi.DeviceClass{deviceClass},
 			want: want{
-				filter: perNodeResult{
-					workerNode.Name: {
-						status: framework.NewStatus(framework.UnschedulableAndUnresolvable, `claim default/my-pod-my-resource, request req-1: has subrequests, but the feature is disabled`),
-					},
+				prefilter: result{
+					status: framework.NewStatus(framework.UnschedulableAndUnresolvable, `claim default/my-pod-my-resource, request req-1: has subrequests, but the DRAPrioritizedList feature is disabled`),
+				},
+				postfilter: result{
+					status: framework.NewStatus(framework.Unschedulable, `no new claims to deallocate`),
 				},
 			},
 		},
diff --git a/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator.go b/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator.go
index 0068a0e5e7953..4a4b5add94bff 100644
--- a/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator.go
+++ b/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator.go
@@ -171,7 +171,7 @@ func (a *Allocator) Allocate(ctx context.Context, node *v1.Node) (finalResult []
 			// Error out if the prioritizedList feature is not enabled and the request
 			// has subrequests. This is to avoid surprising behavior for users.
 			if !a.prioritizedListEnabled && hasSubRequests {
-				return nil, fmt.Errorf("claim %s, request %s: has subrequests, but the feature is disabled", klog.KObj(claim), request.Name)
+				return nil, fmt.Errorf("claim %s, request %s: has subrequests, but the DRAPrioritizedList feature is disabled", klog.KObj(claim), request.Name)
 			}
 
 			if hasSubRequests {
diff --git a/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator_test.go b/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator_test.go
index 52b1f6dc8ed05..992bf04690682 100644
--- a/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator_test.go
+++ b/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator_test.go
@@ -1923,7 +1923,7 @@ func TestAllocator(t *testing.T) {
 			node:    node(node1, region1),
 
 			expectResults: nil,
-			expectError:   gomega.MatchError(gomega.ContainSubstring("claim claim-0, request req-0: has subrequests, but the feature is disabled")),
+			expectError:   gomega.MatchError(gomega.ContainSubstring("claim claim-0, request req-0: has subrequests, but the DRAPrioritizedList feature is disabled")),
 		},
 		"prioritized-list-multi-request": {
 			prioritizedList: true,
diff --git a/test/integration/dra/dra_test.go b/test/integration/dra/dra_test.go
index 9b57883295e6f..eea35f5a6202f 100644
--- a/test/integration/dra/dra_test.go
+++ b/test/integration/dra/dra_test.go
@@ -17,13 +17,19 @@ limitations under the License.
 package dra
 
 import (
+	"context"
+	"errors"
 	"fmt"
 	"regexp"
 	"sort"
 	"strings"
 	"testing"
+	"time"
 
+	"github.com/onsi/gomega"
+	"github.com/onsi/gomega/gstruct"
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 
 	v1 "k8s.io/api/core/v1"
 	resourcealphaapi "k8s.io/api/resource/v1alpha3"
@@ -34,10 +40,15 @@ import (
 	utilfeature "k8s.io/apiserver/pkg/util/feature"
 	"k8s.io/component-base/featuregate"
 	featuregatetesting "k8s.io/component-base/featuregate/testing"
+	"k8s.io/klog/v2"
+	kubeschedulerconfigv1 "k8s.io/kube-scheduler/config/v1"
 	kubeapiservertesting "k8s.io/kubernetes/cmd/kube-apiserver/app/testing"
 	"k8s.io/kubernetes/pkg/features"
+	"k8s.io/kubernetes/pkg/scheduler/apis/config"
+	kubeschedulerscheme "k8s.io/kubernetes/pkg/scheduler/apis/config/scheme"
 	st "k8s.io/kubernetes/pkg/scheduler/testing"
 	"k8s.io/kubernetes/test/integration/framework"
+	"k8s.io/kubernetes/test/integration/util"
 	"k8s.io/kubernetes/test/utils/ktesting"
 	"k8s.io/utils/ptr"
 )
@@ -54,11 +65,21 @@ var (
 				Container("my-container").
 				PodResourceClaims(v1.PodResourceClaim{Name: resourceName, ResourceClaimName: &claimName}).
 				Obj()
+	class = &resourceapi.DeviceClass{
+		ObjectMeta: metav1.ObjectMeta{
+			Name: className,
+		},
+	}
 	claim = st.MakeResourceClaim().
 		Name(claimName).
 		Namespace(namespace).
 		Request(className).
 		Obj()
+	claimPrioritizedList = st.MakeResourceClaim().
+				Name(claimName).
+				Namespace(namespace).
+				RequestWithPrioritizedList(className).
+				Obj()
 )
 
 // createTestNamespace creates a namespace with a name that is derived from the
@@ -106,6 +127,7 @@ func TestDRA(t *testing.T) {
 			features: map[featuregate.Feature]bool{features.DynamicResourceAllocation: true},
 			f: func(tCtx ktesting.TContext) {
 				tCtx.Run("AdminAccess", func(tCtx ktesting.TContext) { testAdminAccess(tCtx, false) })
+				tCtx.Run("PrioritizedList", func(tCtx ktesting.TContext) { testPrioritizedList(tCtx, false) })
 				tCtx.Run("Pod", func(tCtx ktesting.TContext) { testPod(tCtx, true) })
 			},
 		},
@@ -119,11 +141,13 @@ func TestDRA(t *testing.T) {
 				// Additional DRA feature gates go here,
 				// in alphabetical order,
 				// as needed by tests for them.
-				features.DRAAdminAccess: true,
+				features.DRAAdminAccess:     true,
+				features.DRAPrioritizedList: true,
 			},
 			f: func(tCtx ktesting.TContext) {
 				tCtx.Run("AdminAccess", func(tCtx ktesting.TContext) { testAdminAccess(tCtx, true) })
 				tCtx.Run("Convert", testConvert)
+				tCtx.Run("PrioritizedList", func(tCtx ktesting.TContext) { testPrioritizedList(tCtx, true) })
 			},
 		},
 	} {
@@ -146,21 +170,43 @@ func TestDRA(t *testing.T) {
 			etcdOptions := framework.SharedEtcd()
 			apiServerOptions := kubeapiservertesting.NewDefaultTestServerOptions()
 			apiServerFlags := framework.DefaultTestServerFlags()
-			// Default kube-apiserver behavior, must be requested explicitly for test server.
-			runtimeConfigs := []string{"api/alpha=false", "api/beta=false"}
+			var runtimeConfigs []string
 			for key, value := range tc.apis {
 				runtimeConfigs = append(runtimeConfigs, fmt.Sprintf("%s=%t", key, value))
 			}
 			apiServerFlags = append(apiServerFlags, "--runtime-config="+strings.Join(runtimeConfigs, ","))
 			server := kubeapiservertesting.StartTestServerOrDie(t, apiServerOptions, apiServerFlags, etcdOptions)
 			tCtx.Cleanup(server.TearDownFn)
-
 			tCtx = ktesting.WithRESTConfig(tCtx, server.ClientConfig)
+
 			tc.f(tCtx)
 		})
 	}
 }
 
+func startScheduler(tCtx ktesting.TContext) {
+	// Run scheduler with default configuration.
+	tCtx.Log("Scheduler starting...")
+	schedulerCtx := klog.NewContext(tCtx, klog.LoggerWithName(tCtx.Logger(), "scheduler"))
+	schedulerCtx, cancel := context.WithCancelCause(schedulerCtx)
+	_, informerFactory := util.StartScheduler(schedulerCtx, tCtx.Client(), tCtx.RESTConfig(), newDefaultSchedulerComponentConfig(tCtx), nil)
+	// Stop clients of the apiserver before stopping the apiserver itself,
+	// otherwise it delays its shutdown.
+	tCtx.Cleanup(informerFactory.Shutdown)
+	tCtx.Cleanup(func() {
+		tCtx.Log("Stoping scheduler...")
+		cancel(errors.New("test is done"))
+	})
+}
+
+func newDefaultSchedulerComponentConfig(tCtx ktesting.TContext) *config.KubeSchedulerConfiguration {
+	gvk := kubeschedulerconfigv1.SchemeGroupVersion.WithKind("KubeSchedulerConfiguration")
+	cfg := config.KubeSchedulerConfiguration{}
+	_, _, err := kubeschedulerscheme.Codecs.UniversalDecoder().Decode(nil, &gvk, &cfg)
+	tCtx.ExpectNoError(err, "decode default scheduler configuration")
+	return &cfg
+}
+
 // testPod creates a pod with a resource claim reference and then checks
 // whether that field is or isn't getting dropped.
 func testPod(tCtx ktesting.TContext, draEnabled bool) {
@@ -220,3 +266,45 @@ func testAdminAccess(tCtx ktesting.TContext, adminAccessEnabled bool) {
 		}
 	}
 }
+
+func testPrioritizedList(tCtx ktesting.TContext, enabled bool) {
+	tCtx.Parallel()
+	_, err := tCtx.Client().ResourceV1beta1().DeviceClasses().Create(tCtx, class, metav1.CreateOptions{})
+	tCtx.ExpectNoError(err, "create class")
+	namespace := createTestNamespace(tCtx)
+	claim := claimPrioritizedList.DeepCopy()
+	claim.Namespace = namespace
+	claim, err = tCtx.Client().ResourceV1beta1().ResourceClaims(namespace).Create(tCtx, claim, metav1.CreateOptions{})
+
+	if !enabled {
+		require.Error(tCtx, err, "claim should have become invalid after dropping FirstAvailable")
+		return
+	}
+
+	require.NotEmpty(tCtx, claim.Spec.Devices.Requests[0].FirstAvailable, "should store FirstAvailable")
+	tCtx.Run("scheduler", func(tCtx ktesting.TContext) {
+		startScheduler(tCtx)
+
+		// The fake cluster configuration is not complete enough to actually schedule pods.
+		// That is covered over in test/integration/scheduler_perf.
+		// Here we only test that we get to the point where it notices that, without failing
+		// during PreFilter because of FirstAvailable.
+		pod := podWithClaimName.DeepCopy()
+		pod.Namespace = namespace
+		_, err := tCtx.Client().CoreV1().Pods(namespace).Create(tCtx, pod, metav1.CreateOptions{})
+		tCtx.ExpectNoError(err, "create pod")
+		schedulingAttempted := gomega.HaveField("Status.Conditions", gomega.ContainElement(
+			gstruct.MatchFields(gstruct.IgnoreExtras, gstruct.Fields{
+				"Type":    gomega.Equal(v1.PodScheduled),
+				"Status":  gomega.Equal(v1.ConditionFalse),
+				"Reason":  gomega.Equal("Unschedulable"),
+				"Message": gomega.Equal("no nodes available to schedule pods"),
+			}),
+		))
+		ktesting.Eventually(tCtx, func(tCtx ktesting.TContext) *v1.Pod {
+			pod, err := tCtx.Client().CoreV1().Pods(namespace).Get(tCtx, pod.Name, metav1.GetOptions{})
+			tCtx.ExpectNoError(err, "get pod")
+			return pod
+		}).WithTimeout(time.Minute).WithPolling(time.Second).Should(schedulingAttempted)
+	})
+}
diff --git a/test/integration/scheduler_perf/dra/performance-config.yaml b/test/integration/scheduler_perf/dra/performance-config.yaml
index 35bea6a2c003d..77badbaa57f4b 100644
--- a/test/integration/scheduler_perf/dra/performance-config.yaml
+++ b/test/integration/scheduler_perf/dra/performance-config.yaml
@@ -294,6 +294,66 @@
       maxClaimsPerNode: 10
       duration: 10s
 
+# SteadyStateResourceClaimTemplateFirstAvailable is a variant of SteadyStateResourceClaimTemplate
+# with a claim template that uses the "firstAvailable" subrequests, aka DRAPrioritizedList.
+- name: SteadyStateClusterResourceClaimTemplateFirstAvailable
+  featureGates:
+    DynamicResourceAllocation: true
+    DRAPrioritizedList: true
+  workloadTemplate:
+  - opcode: createNodes
+    countParam: $nodesWithoutDRA
+  - opcode: createNodes
+    nodeTemplatePath: templates/node-with-dra-test-driver.yaml
+    countParam: $nodesWithDRA
+  - opcode: createResourceDriver
+    driverName: test-driver.cdi.k8s.io
+    nodes: scheduler-perf-dra-*
+    maxClaimsPerNodeParam: $maxClaimsPerNode
+  - opcode: createAny
+    templatePath: templates/deviceclass.yaml
+  - opcode: createAny
+    templatePath: templates/resourceclaim.yaml
+    countParam: $initClaims
+    namespace: init
+  - opcode: allocResourceClaims
+    namespace: init
+  - opcode: createAny
+    templatePath: templates/resourceclaimtemplate-first-available.yaml
+    namespace: test
+  - opcode: createPods
+    namespace: test
+    count: 10
+    steadyState: true
+    durationParam: $duration
+    podTemplatePath: templates/pod-with-claim-template.yaml
+    collectMetrics: true
+  workloads:
+  - name: fast
+    featureGates:
+      SchedulerQueueingHints: false
+    labels: [integration-test, short]
+    params:
+      # This testcase runs through all code paths without
+      # taking too long overall.
+      nodesWithDRA: 1
+      nodesWithoutDRA: 1
+      initClaims: 0
+      maxClaimsPerNode: 10
+      duration: 2s
+  - name: fast_QueueingHintsEnabled
+    featureGates:
+      SchedulerQueueingHints: true
+    labels: [integration-test, short]
+    params:
+      # This testcase runs through all code paths without
+      # taking too long overall.
+      nodesWithDRA: 1
+      nodesWithoutDRA: 1
+      initClaims: 0
+      maxClaimsPerNode: 10
+      duration: 2s
+ 
 # SchedulingWithResourceClaimTemplate uses ResourceClaims
 # with deterministic names that are shared between pods.
 # There is a fixed ratio of 1:5 between claims and pods.
diff --git a/test/integration/scheduler_perf/dra/templates/resourceclaimtemplate-first-available.yaml b/test/integration/scheduler_perf/dra/templates/resourceclaimtemplate-first-available.yaml
new file mode 100644
index 0000000000000..b343ee13df561
--- /dev/null
+++ b/test/integration/scheduler_perf/dra/templates/resourceclaimtemplate-first-available.yaml
@@ -0,0 +1,14 @@
+apiVersion: resource.k8s.io/v1alpha3
+kind: ResourceClaimTemplate
+metadata:
+  name: test-claim-template
+spec:
+  spec:
+    devices:
+      requests:
+      - name: req-0
+        firstAvailable:
+        - name: sub-0
+          deviceClassName: no-such-class
+        - name: sub-1
+          deviceClassName: test-class


Reviews:
user: login: pohly
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T07:48:54Z
user: login: pohly
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T07:56:54Z
user: login: sanposhiho
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T11:17:39Z
user: login: dom4ha
body: /lgtm
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T13:17:49Z
user: login: pohly
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-09T08:25:14Z
user: login: pohly
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-09T08:25:19Z
user: login: pohly
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-09T08:25:25Z
user: login: sanposhiho
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-09T09:52:02Z
user: login: sanposhiho
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-09T09:57:52Z
user: login: sanposhiho
body: /approve

Leave `/lgtm` for others. @macsko @dom4ha 

state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-09T09:59:25Z
user: login: pohly
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-09T20:09:09Z
user: login: pohly
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T07:22:15Z
user: login: pohly
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T07:23:16Z
user: login: macsko
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-10T08:44:01Z
user: login: pohly
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T10:39:04Z
user: login: dom4ha
body: /lgtm
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-10T11:15:33Z
user: login: sanposhiho
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-10T14:56:38Z
user: login: pohly
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T16:35:18Z

Review comments:
diff_hunk: @@ -146,21 +170,42 @@ func TestDRA(t *testing.T) {
 			etcdOptions := framework.SharedEtcd()
 			apiServerOptions := kubeapiservertesting.NewDefaultTestServerOptions()
 			apiServerFlags := framework.DefaultTestServerFlags()
-			// Default kube-apiserver behavior, must be requested explicitly for test server.
-			runtimeConfigs := []string{"api/alpha=false", "api/beta=false"}
+			var runtimeConfigs []string
path: test/integration/dra/dra_test.go
user: login: pohly
body: Drive-by change: I had changed the defaults of the test server in a different PR.

created_at: 2025-03-07T07:48:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,21 +170,42 @@ func TestDRA(t *testing.T) {
 			etcdOptions := framework.SharedEtcd()
 			apiServerOptions := kubeapiservertesting.NewDefaultTestServerOptions()
 			apiServerFlags := framework.DefaultTestServerFlags()
-			// Default kube-apiserver behavior, must be requested explicitly for test server.
-			runtimeConfigs := []string{"api/alpha=false", "api/beta=false"}
+			var runtimeConfigs []string
 			for key, value := range tc.apis {
 				runtimeConfigs = append(runtimeConfigs, fmt.Sprintf("%s=%t", key, value))
 			}
 			apiServerFlags = append(apiServerFlags, "--runtime-config="+strings.Join(runtimeConfigs, ","))
 			server := kubeapiservertesting.StartTestServerOrDie(t, apiServerOptions, apiServerFlags, etcdOptions)
 			tCtx.Cleanup(server.TearDownFn)
-
 			tCtx = ktesting.WithRESTConfig(tCtx, server.ClientConfig)
+
 			tc.f(tCtx)
 		})
 	}
 }
 
+func startScheduler(tCtx ktesting.TContext) {
path: test/integration/dra/dra_test.go
user: login: pohly
body: We have two choices:
- Put integration tests for DRA involving the scheduler under test/integration/scheduler.
- Put them here and bring up the scheduler if needed.

I was looking at the first option but wasn't sure where that would go. @sanposhiho: where do other scheduler plugins run their integration tests?

I thought it was test/integration/scheduler/plugins, but that's not true: that one covers plugin support in the scheduler with mock plugins, not testing of real plugins.

There is scheduler_perf, which is useful for some scenarios, but a bit too limited when it comes to really special one-of scenarios that require custom Go code for their setup.

I think a dedicated test/integration/dra is the better choice. Bringing up additional cluster components as needed, like the scheduler here, is fine. The main advantage is that it is much easier to review and run DRA-related integration tests.

test/integration/resourceclaim tests should be moved here, too.

created_at: 2025-03-07T07:56:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,21 +170,42 @@ func TestDRA(t *testing.T) {
 			etcdOptions := framework.SharedEtcd()
 			apiServerOptions := kubeapiservertesting.NewDefaultTestServerOptions()
 			apiServerFlags := framework.DefaultTestServerFlags()
-			// Default kube-apiserver behavior, must be requested explicitly for test server.
-			runtimeConfigs := []string{"api/alpha=false", "api/beta=false"}
+			var runtimeConfigs []string
 			for key, value := range tc.apis {
 				runtimeConfigs = append(runtimeConfigs, fmt.Sprintf("%s=%t", key, value))
 			}
 			apiServerFlags = append(apiServerFlags, "--runtime-config="+strings.Join(runtimeConfigs, ","))
 			server := kubeapiservertesting.StartTestServerOrDie(t, apiServerOptions, apiServerFlags, etcdOptions)
 			tCtx.Cleanup(server.TearDownFn)
-
 			tCtx = ktesting.WithRESTConfig(tCtx, server.ClientConfig)
+
 			tc.f(tCtx)
 		})
 	}
 }
 
+func startScheduler(tCtx ktesting.TContext) {
path: test/integration/dra/dra_test.go
user: login: sanposhiho
body: > where do other scheduler plugins run their integration tests?

It's mostly in:
- https://github.com/kubernetes/kubernetes/blob/master/test/integration/scheduler/filters/filters_test.go
- https://github.com/kubernetes/kubernetes/blob/master/test/integration/scheduler/scoring/priorities_test.go

I don't have a strong preference though, having test/integration/dra and starting necessary component(s) there sounds good to me as well.
created_at: 2025-03-07T11:17:39Z
author_association: MEMBER
diff_hunk: @@ -146,21 +170,42 @@ func TestDRA(t *testing.T) {
 			etcdOptions := framework.SharedEtcd()
 			apiServerOptions := kubeapiservertesting.NewDefaultTestServerOptions()
 			apiServerFlags := framework.DefaultTestServerFlags()
-			// Default kube-apiserver behavior, must be requested explicitly for test server.
-			runtimeConfigs := []string{"api/alpha=false", "api/beta=false"}
+			var runtimeConfigs []string
 			for key, value := range tc.apis {
 				runtimeConfigs = append(runtimeConfigs, fmt.Sprintf("%s=%t", key, value))
 			}
 			apiServerFlags = append(apiServerFlags, "--runtime-config="+strings.Join(runtimeConfigs, ","))
 			server := kubeapiservertesting.StartTestServerOrDie(t, apiServerOptions, apiServerFlags, etcdOptions)
 			tCtx.Cleanup(server.TearDownFn)
-
 			tCtx = ktesting.WithRESTConfig(tCtx, server.ClientConfig)
+
 			tc.f(tCtx)
 		})
 	}
 }
 
+func startScheduler(tCtx ktesting.TContext) {
+	// Run scheduler with default configuration.
+	tCtx.Log("Scheduler starting...")
+	schedulerCtx := klog.NewContext(tCtx, klog.LoggerWithName(tCtx.Logger(), "scheduler"))
+	schedulerCtx, cancel := context.WithCancelCause(schedulerCtx)
+	_, informerFactory := util.StartScheduler(schedulerCtx, tCtx.Client(), tCtx.RESTConfig(), newDefaultSchedulerComponentConfig(tCtx), nil)
+	// Stop clients of the apiserver before stopping the apiserver itself,
+	// otherwise it delays its shutdown.
+	tCtx.Cleanup(informerFactory.Shutdown)
+	tCtx.Cleanup(func() {
+		cancel(errors.New("test is done"))
+	})
path: test/integration/dra/dra_test.go
user: login: dom4ha
body: 
```suggestion
    tCtx.Log("Stoping scheduler...")
	})
```
created_at: 2025-03-07T12:50:32Z
author_association: MEMBER
diff_hunk: @@ -294,6 +294,66 @@
       maxClaimsPerNode: 10
       duration: 10s
 
+# SteadyStateResourceClaimTemplateFirstAvailable is a variant of SteadyStateResourceClaimTemplate
+# with a claim template that uses the "firstAvailable" subrequests, aka DRAPrioritizedList.
+- name: SteadyStateClusterResourceClaimTemplateFirstAvailable
+  featureGates:
+    DynamicResourceAllocation: true
+    DRAPrioritizedList: true
+  workloadTemplate:
+  - opcode: createNodes
+    countParam: $nodesWithoutDRA
+  - opcode: createNodes
+    nodeTemplatePath: templates/node-with-dra-test-driver.yaml
+    countParam: $nodesWithDRA
+  - opcode: createResourceDriver
+    driverName: test-driver.cdi.k8s.io
+    nodes: scheduler-perf-dra-*
+    maxClaimsPerNodeParam: $maxClaimsPerNode
+  - opcode: createAny
+    templatePath: templates/deviceclass.yaml
+  - opcode: createAny
+    templatePath: templates/resourceclaim.yaml
+    countParam: $initClaims
path: test/integration/scheduler_perf/dra/performance-config.yaml
user: login: dom4ha
body: Do we need this since it's currently always 0? 
created_at: 2025-03-07T13:15:55Z
author_association: MEMBER
diff_hunk: @@ -220,3 +265,43 @@ func testAdminAccess(tCtx ktesting.TContext, adminAccessEnabled bool) {
 		}
 	}
 }
+
+func testPrioritizedList(tCtx ktesting.TContext, enabled bool) {
+	tCtx.Parallel()
+	_, err := tCtx.Client().ResourceV1beta1().DeviceClasses().Create(tCtx, class, metav1.CreateOptions{})
+	tCtx.ExpectNoError(err, "create class")
+	namespace := createTestNamespace(tCtx)
+	claim := claimPrioritizedList.DeepCopy()
+	claim.Namespace = namespace
+	claim, err = tCtx.Client().ResourceV1beta1().ResourceClaims(namespace).Create(tCtx, claim, metav1.CreateOptions{})
+	if enabled {
path: test/integration/dra/dra_test.go
user: login: macsko
body: Maybe reduce the indentation by flipping the logic:
```suggestion
	if !enabled {
		require.Error(tCtx, err, "claim should have become invalid after dropping FirstAvailable")
		return
	}
```
created_at: 2025-03-07T13:50:17Z
author_association: MEMBER
diff_hunk: @@ -294,6 +294,66 @@
       maxClaimsPerNode: 10
       duration: 10s
 
+# SteadyStateResourceClaimTemplateFirstAvailable is a variant of SteadyStateResourceClaimTemplate
+# with a claim template that uses the "firstAvailable" subrequests, aka DRAPrioritizedList.
+- name: SteadyStateClusterResourceClaimTemplateFirstAvailable
+  featureGates:
+    DynamicResourceAllocation: true
+    DRAPrioritizedList: true
+  workloadTemplate:
+  - opcode: createNodes
+    countParam: $nodesWithoutDRA
+  - opcode: createNodes
+    nodeTemplatePath: templates/node-with-dra-test-driver.yaml
+    countParam: $nodesWithDRA
+  - opcode: createResourceDriver
+    driverName: test-driver.cdi.k8s.io
+    nodes: scheduler-perf-dra-*
+    maxClaimsPerNodeParam: $maxClaimsPerNode
+  - opcode: createAny
+    templatePath: templates/deviceclass.yaml
+  - opcode: createAny
+    templatePath: templates/resourceclaim.yaml
+    countParam: $initClaims
path: test/integration/scheduler_perf/dra/performance-config.yaml
user: login: pohly
body: Not really. I wanted to keep this the same as in the other variants, in case that we want to add a test case which needs it.
created_at: 2025-03-09T08:25:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,21 +170,42 @@ func TestDRA(t *testing.T) {
 			etcdOptions := framework.SharedEtcd()
 			apiServerOptions := kubeapiservertesting.NewDefaultTestServerOptions()
 			apiServerFlags := framework.DefaultTestServerFlags()
-			// Default kube-apiserver behavior, must be requested explicitly for test server.
-			runtimeConfigs := []string{"api/alpha=false", "api/beta=false"}
+			var runtimeConfigs []string
 			for key, value := range tc.apis {
 				runtimeConfigs = append(runtimeConfigs, fmt.Sprintf("%s=%t", key, value))
 			}
 			apiServerFlags = append(apiServerFlags, "--runtime-config="+strings.Join(runtimeConfigs, ","))
 			server := kubeapiservertesting.StartTestServerOrDie(t, apiServerOptions, apiServerFlags, etcdOptions)
 			tCtx.Cleanup(server.TearDownFn)
-
 			tCtx = ktesting.WithRESTConfig(tCtx, server.ClientConfig)
+
 			tc.f(tCtx)
 		})
 	}
 }
 
+func startScheduler(tCtx ktesting.TContext) {
+	// Run scheduler with default configuration.
+	tCtx.Log("Scheduler starting...")
+	schedulerCtx := klog.NewContext(tCtx, klog.LoggerWithName(tCtx.Logger(), "scheduler"))
+	schedulerCtx, cancel := context.WithCancelCause(schedulerCtx)
+	_, informerFactory := util.StartScheduler(schedulerCtx, tCtx.Client(), tCtx.RESTConfig(), newDefaultSchedulerComponentConfig(tCtx), nil)
+	// Stop clients of the apiserver before stopping the apiserver itself,
+	// otherwise it delays its shutdown.
+	tCtx.Cleanup(informerFactory.Shutdown)
+	tCtx.Cleanup(func() {
+		cancel(errors.New("test is done"))
+	})
path: test/integration/dra/dra_test.go
user: login: pohly
body: Ack, will add.

created_at: 2025-03-09T08:25:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,21 +170,42 @@ func TestDRA(t *testing.T) {
 			etcdOptions := framework.SharedEtcd()
 			apiServerOptions := kubeapiservertesting.NewDefaultTestServerOptions()
 			apiServerFlags := framework.DefaultTestServerFlags()
-			// Default kube-apiserver behavior, must be requested explicitly for test server.
-			runtimeConfigs := []string{"api/alpha=false", "api/beta=false"}
+			var runtimeConfigs []string
 			for key, value := range tc.apis {
 				runtimeConfigs = append(runtimeConfigs, fmt.Sprintf("%s=%t", key, value))
 			}
 			apiServerFlags = append(apiServerFlags, "--runtime-config="+strings.Join(runtimeConfigs, ","))
 			server := kubeapiservertesting.StartTestServerOrDie(t, apiServerOptions, apiServerFlags, etcdOptions)
 			tCtx.Cleanup(server.TearDownFn)
-
 			tCtx = ktesting.WithRESTConfig(tCtx, server.ClientConfig)
+
 			tc.f(tCtx)
 		})
 	}
 }
 
+func startScheduler(tCtx ktesting.TContext) {
+	// Run scheduler with default configuration.
+	tCtx.Log("Scheduler starting...")
+	schedulerCtx := klog.NewContext(tCtx, klog.LoggerWithName(tCtx.Logger(), "scheduler"))
+	schedulerCtx, cancel := context.WithCancelCause(schedulerCtx)
+	_, informerFactory := util.StartScheduler(schedulerCtx, tCtx.Client(), tCtx.RESTConfig(), newDefaultSchedulerComponentConfig(tCtx), nil)
+	// Stop clients of the apiserver before stopping the apiserver itself,
+	// otherwise it delays its shutdown.
+	tCtx.Cleanup(informerFactory.Shutdown)
+	tCtx.Cleanup(func() {
+		cancel(errors.New("test is done"))
+	})
path: test/integration/dra/dra_test.go
user: login: pohly
body: /hold

created_at: 2025-03-09T08:25:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -415,6 +415,9 @@ func (pl *DynamicResources) PreFilter(ctx context.Context, state *framework.Cycl
 						return nil, status
 					}
 				} else {
+					if !pl.enablePrioritizedList {
path: pkg/scheduler/framework/plugins/dynamicresources/dynamicresources.go
user: login: sanposhiho
body: Should we reject pods with `FirstAvailable` at PreEnqueue additionally when PrioritizedList is disabled? which would reduce unnecessary scheduling cycles consumed for those pods.
created_at: 2025-03-09T09:51:38Z
author_association: MEMBER
diff_hunk: @@ -415,6 +415,9 @@ func (pl *DynamicResources) PreFilter(ctx context.Context, state *framework.Cycl
 						return nil, status
 					}
 				} else {
+					if !pl.enablePrioritizedList {
path: pkg/scheduler/framework/plugins/dynamicresources/dynamicresources.go
user: login: sanposhiho
body: Ah, that being said, I just remembered the discussion at https://github.com/kubernetes/kubernetes/issues/129698. 
If we reject pods at PreEnqueue, no error message would be shown up for users. Hmm, so looks like this is not a great idea, at least for now, until we've got a solution for that problem.
created_at: 2025-03-09T09:57:52Z
author_association: MEMBER
diff_hunk: @@ -415,6 +415,9 @@ func (pl *DynamicResources) PreFilter(ctx context.Context, state *framework.Cycl
 						return nil, status
 					}
 				} else {
+					if !pl.enablePrioritizedList {
path: pkg/scheduler/framework/plugins/dynamicresources/dynamicresources.go
user: login: pohly
body: That would be even better. Let me check that.

created_at: 2025-03-09T20:09:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,21 +170,42 @@ func TestDRA(t *testing.T) {
 			etcdOptions := framework.SharedEtcd()
 			apiServerOptions := kubeapiservertesting.NewDefaultTestServerOptions()
 			apiServerFlags := framework.DefaultTestServerFlags()
-			// Default kube-apiserver behavior, must be requested explicitly for test server.
-			runtimeConfigs := []string{"api/alpha=false", "api/beta=false"}
+			var runtimeConfigs []string
 			for key, value := range tc.apis {
 				runtimeConfigs = append(runtimeConfigs, fmt.Sprintf("%s=%t", key, value))
 			}
 			apiServerFlags = append(apiServerFlags, "--runtime-config="+strings.Join(runtimeConfigs, ","))
 			server := kubeapiservertesting.StartTestServerOrDie(t, apiServerOptions, apiServerFlags, etcdOptions)
 			tCtx.Cleanup(server.TearDownFn)
-
 			tCtx = ktesting.WithRESTConfig(tCtx, server.ClientConfig)
+
 			tc.f(tCtx)
 		})
 	}
 }
 
+func startScheduler(tCtx ktesting.TContext) {
+	// Run scheduler with default configuration.
+	tCtx.Log("Scheduler starting...")
+	schedulerCtx := klog.NewContext(tCtx, klog.LoggerWithName(tCtx.Logger(), "scheduler"))
+	schedulerCtx, cancel := context.WithCancelCause(schedulerCtx)
+	_, informerFactory := util.StartScheduler(schedulerCtx, tCtx.Client(), tCtx.RESTConfig(), newDefaultSchedulerComponentConfig(tCtx), nil)
+	// Stop clients of the apiserver before stopping the apiserver itself,
+	// otherwise it delays its shutdown.
+	tCtx.Cleanup(informerFactory.Shutdown)
+	tCtx.Cleanup(func() {
+		cancel(errors.New("test is done"))
+	})
path: test/integration/dra/dra_test.go
user: login: pohly
body: /hold cancel

Done.
created_at: 2025-03-10T07:22:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -415,6 +415,9 @@ func (pl *DynamicResources) PreFilter(ctx context.Context, state *framework.Cycl
 						return nil, status
 					}
 				} else {
+					if !pl.enablePrioritizedList {
path: pkg/scheduler/framework/plugins/dynamicresources/dynamicresources.go
user: login: pohly
body: Right, there was a reason (hadn't seen your second message when writing my initial reply).

created_at: 2025-03-10T07:23:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -220,3 +265,43 @@ func testAdminAccess(tCtx ktesting.TContext, adminAccessEnabled bool) {
 		}
 	}
 }
+
+func testPrioritizedList(tCtx ktesting.TContext, enabled bool) {
+	tCtx.Parallel()
+	_, err := tCtx.Client().ResourceV1beta1().DeviceClasses().Create(tCtx, class, metav1.CreateOptions{})
+	tCtx.ExpectNoError(err, "create class")
+	namespace := createTestNamespace(tCtx)
+	claim := claimPrioritizedList.DeepCopy()
+	claim.Namespace = namespace
+	claim, err = tCtx.Client().ResourceV1beta1().ResourceClaims(namespace).Create(tCtx, claim, metav1.CreateOptions{})
+	if enabled {
path: test/integration/dra/dra_test.go
user: login: pohly
body: Okay. Fixed, and also the unit test failure in the allocation that was caused by the modified error text.
created_at: 2025-03-10T10:39:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -415,6 +415,9 @@ func (pl *DynamicResources) PreFilter(ctx context.Context, state *framework.Cycl
 						return nil, status
 					}
 				} else {
+					if !pl.enablePrioritizedList {
path: pkg/scheduler/framework/plugins/dynamicresources/dynamicresources.go
user: login: sanposhiho
body: Yeah, it would be nice to be in PreEnqueue performance-wise though. Anyways, we can revisit all those PreEnqueue stuff later.
created_at: 2025-03-10T14:56:37Z
author_association: MEMBER
diff_hunk: @@ -415,6 +415,9 @@ func (pl *DynamicResources) PreFilter(ctx context.Context, state *framework.Cycl
 						return nil, status
 					}
 				} else {
+					if !pl.enablePrioritizedList {
path: pkg/scheduler/framework/plugins/dynamicresources/dynamicresources.go
user: login: pohly
body: This situation is going to be extremely rare (enable feature, create claim, disable feature) and temporary (will disappear as the feature matures, hopefully in a few releases).  It's good to catch it when it's just one if check in some existing code, but I am not sure whether it would be worth more complex changes in PreEnqueue.

I was about to write that when I saw that you had already retracted the suggestion. :smile: 



created_at: 2025-03-10T16:35:18Z
author_association: CONTRIBUTOR

PR comments:
user: login: k8s-ci-robot
created_at: 2025-03-06T18:59:03Z
body: Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with `/test all`
author_association: CONTRIBUTOR
user: login: pohly
created_at: 2025-03-07T07:59:08Z
body: /assign @sanposhiho 
/priority important-soon

The PreFilter change is nice to have for 1.33, but not required.

author_association: CONTRIBUTOR
user: login: k8s-triage-robot
created_at: 2025-03-07T08:08:58Z
body: This PR [may require API review](https://git.k8s.io/community/sig-architecture/api-review-process.md#what-apis-need-to-be-reviewed).

If so, when the changes are ready, [complete the pre-review checklist and request an API review](https://git.k8s.io/community/sig-architecture/api-review-process.md#mechanics).

Status of requested reviews is tracked in the [API Review project](https://github.com/orgs/kubernetes/projects/169).
author_association: NONE
user: login: sanposhiho
created_at: 2025-03-07T11:17:48Z
body: /cc @dom4ha @macsko 
author_association: MEMBER
user: login: sanposhiho
created_at: 2025-03-07T11:17:57Z
body: /cc @AxeZhan 
author_association: MEMBER
user: login: k8s-ci-robot
created_at: 2025-03-07T13:17:56Z
body: LGTM label has been added.  <details>Git tree hash: e8e9d4deafbcc0548ea2084415b2e077093f4f0c</details>
author_association: CONTRIBUTOR
user: login: sanposhiho
created_at: 2025-03-09T09:59:37Z
body: /retest
author_association: MEMBER
user: login: k8s-ci-robot
created_at: 2025-03-09T09:59:38Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes/kubernetes/pull/130622#" title="Author self-approved">pohly</a>*, *<a href="https://github.com/kubernetes/kubernetes/pull/130622#pullrequestreview-2669382137" title="Approved">sanposhiho</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fkubernetes).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[pkg/scheduler/OWNERS](https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/OWNERS)~~ [sanposhiho]
- ~~[staging/src/k8s.io/dynamic-resource-allocation/OWNERS](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/dynamic-resource-allocation/OWNERS)~~ [pohly]
- ~~[test/integration/dra/OWNERS](https://github.com/kubernetes/kubernetes/blob/master/test/integration/dra/OWNERS)~~ [pohly]
- ~~[test/integration/scheduler_perf/OWNERS](https://github.com/kubernetes/kubernetes/blob/master/test/integration/scheduler_perf/OWNERS)~~ [pohly,sanposhiho]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: sanposhiho
created_at: 2025-03-09T09:59:53Z
body: /remove-kind api-change
I guess?
author_association: MEMBER
user: login: pohly
created_at: 2025-03-10T07:24:09Z
body: > /remove-kind api-change
> I guess?

Yes. That was a leftover from having included the full implementation earlier before rebasing.

@dom4ha LGTM?

author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-03-10T11:15:39Z
body: LGTM label has been added.  <details>Git tree hash: e0070d417ffb22b670c09ae3109164e3248c7e74</details>
author_association: CONTRIBUTOR
user: login: macsko
created_at: 2025-03-10T11:21:58Z
body: /lgtm
author_association: MEMBER
user: login: pohly
created_at: 2025-03-10T12:13:26Z
body: /test pull-kubernetes-e2e-kind

Pod scheduling timeout.

author_association: CONTRIBUTOR
user: login: aaron-prindle
created_at: 2025-03-13T20:05:53Z
body: /triage accepted
author_association: CONTRIBUTOR