Repository: kubernetes-sigs/gateway-api-inference-extension
Pull request #862: Multi cycle scheduler
Author: login: nirrozenbaum
Labels: name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: size/XXL
description: Denotes a PR that changes 1000+ lines, ignoring generated files.
Created: 2025-05-22T06:47:20Z | Merged: 2025-05-23T05:12:35Z

This is an initial PR that lays the ground for a multi cycle scheduler.
we have an ongoing open discussions in #845 and on google docs and this is not intended to serve as a replacement for the discussions.

This PR is not covering all the planned changes that were discussed, but takes a big step towards the end goal.

### in the scope of this PR:
- restructuring and moving some parts (like plugins, etc) under `/scheduling/framework`.
- renaming Pre/Post Schedule to Pre/Post Cycle to avoid confusion when doing a multi cycle scheduling.
- renaming the previously called `SchedulerConfig` to `SchedulerProfile`. the Profile includes the extensions.
- introducing profiles-picker extension. upon receiving a new schedule request, the profile-picker will pick iteratively which profiles to run. it may pick all or just a subset. this PR includes the simplest form of profile picking and implements a plugin that always selects all profiles.
- introducing a new `SchedulerConfig` that now includes the registered profiles (map from profile name(string) to the profile config) and the profiles-picker. 


### out of scope:
- any struct rename other than Pre/Post Cycle.
- Pre/Post Cycle should be removed as agreed in other threads. atm the only plugin that uses those is prefix. should first change prefix implementation not to use these extension points and than it can be removed.
- this PR doesn't handle the PostResponse (Or my preferred name PostDispatch) part. therefore it's still part of the scheduler, but it should absolutely be removed and moved to a higher layer.
- does not implement any results aggregation. I think this is an open discussion on where this should be handled (inside or outside of the scheduler). current PR returns map from profile -> result and therefore keep the existing behavior. this can be addressed in follow up PRs.

### summary:
additional changes may be needed as we continue to shape how we want the scheduler to look like.  
as a first step, this implements many of the points we already agree on, enables the removal of P/D scheduler from llm-d, and allows configuring all llm-d scheduler related behavior (including conditional profile picking) through plugins only.


Diff:
diff --git a/cmd/epp/main.go b/cmd/epp/main.go
index 44ad9f0d12..5437b1561b 100644
--- a/cmd/epp/main.go
+++ b/cmd/epp/main.go
@@ -44,10 +44,12 @@ import (
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics/collectors"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins/filter"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins/multi/prefix"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins/picker"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins/scorer"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/filter"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/multi/prefix"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/picker"
+	profilepicker "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile-picker"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/scorer"
 	runserver "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/server"
 	envutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/env"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
@@ -196,20 +198,21 @@ func run() error {
 		queueScorerWeight := envutil.GetEnvInt("QUEUE_SCORE_WEIGHT", scorer.DefaultQueueScorerWeight, setupLog)
 		kvCacheScorerWeight := envutil.GetEnvInt("KV_CACHE_SCORE_WEIGHT", scorer.DefaultKVCacheScorerWeight, setupLog)
 
-		schedulerConfig := scheduling.NewSchedulerConfig().
+		schedulerProfile := framework.NewSchedulerProfile().
 			WithFilters(filter.NewSheddableCapacityFilter()).
-			WithScorers(scorer.NewWeightedScorer(&scorer.QueueScorer{}, queueScorerWeight),
-				scorer.NewWeightedScorer(&scorer.KVCacheScorer{}, kvCacheScorerWeight)).
+			WithScorers(framework.NewWeightedScorer(&scorer.QueueScorer{}, queueScorerWeight),
+				framework.NewWeightedScorer(&scorer.KVCacheScorer{}, kvCacheScorerWeight)).
 			WithPicker(picker.NewMaxScorePicker())
 
 		if prefixCacheScheduling == "true" {
 			prefixScorerWeight := envutil.GetEnvInt("PREFIX_CACHE_SCORE_WEIGHT", prefix.DefaultScorerWeight, setupLog)
-			if err := schedulerConfig.AddPlugins(scorer.NewWeightedScorer(prefix.New(loadPrefixCacheConfig()), prefixScorerWeight)); err != nil {
+			if err := schedulerProfile.AddPlugins(framework.NewWeightedScorer(prefix.New(loadPrefixCacheConfig()), prefixScorerWeight)); err != nil {
 				setupLog.Error(err, "Failed to register scheduler plugins")
 				return err
 			}
 		}
 
+		schedulerConfig := scheduling.NewSchedulerConfig(profilepicker.NewAllProfilesPicker(), map[string]*framework.SchedulerProfile{"schedulerv2": schedulerProfile})
 		scheduler = scheduling.NewSchedulerWithConfig(datastore, schedulerConfig)
 	}
 	serverRunner := &runserver.ExtProcServerRunner{
diff --git a/pkg/epp/requestcontrol/director.go b/pkg/epp/requestcontrol/director.go
index 85c8ee34ff..78daf0d920 100644
--- a/pkg/epp/requestcontrol/director.go
+++ b/pkg/epp/requestcontrol/director.go
@@ -35,7 +35,7 @@ import (
 )
 
 type Scheduler interface {
-	Schedule(ctx context.Context, b *schedulingtypes.LLMRequest) (result *schedulingtypes.Result, err error)
+	Schedule(ctx context.Context, b *schedulingtypes.LLMRequest) (result map[string]*schedulingtypes.Result, err error)
 	OnResponse(ctx context.Context, resp *schedulingtypes.LLMResponse, targetPodName string)
 }
 
@@ -108,23 +108,27 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 }
 
 // Dispatch runs one or many scheduling cycles.
-func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) ([]*schedulingtypes.Result, error) {
+func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) (map[string]*schedulingtypes.Result, error) {
 	var err error
 	res, err := d.scheduler.Schedule(ctx, llmReq)
 	if err != nil {
 		return nil, errutil.Error{Code: errutil.InferencePoolResourceExhausted, Msg: fmt.Errorf("failed to find target pod: %w", err).Error()}
 	}
 
-	return []*schedulingtypes.Result{res}, nil
+	return res, nil // TODO handle multi cycle result after defining the PostDispatch extension point
 }
 
-func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results []*schedulingtypes.Result) (*handlers.RequestContext, error) {
+func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results map[string]*schedulingtypes.Result) (*handlers.RequestContext, error) {
 	logger := log.FromContext(ctx)
 	// currently only get a single result. Will refactor to pluggably implement the PostSchedule
 	if len(results) == 0 {
 		return reqCtx, errutil.Error{Code: errutil.Internal, Msg: "results must be greater than zero"}
 	}
-	targetPod := results[0].TargetPod.GetPod()
+	var targetPod *backend.Pod
+	// TODO should handle multi cycle results, this should be pluggable logic
+	for _, result := range results {
+		targetPod = result.TargetPod.GetPod()
+	}
 
 	pool, err := d.datastore.PoolGet()
 	if err != nil {
diff --git a/pkg/epp/scheduling/plugins/plugins.go b/pkg/epp/scheduling/framework/plugins.go
similarity index 66%
rename from pkg/epp/scheduling/plugins/plugins.go
rename to pkg/epp/scheduling/framework/plugins.go
index f3412ab727..99c8389fa5 100644
--- a/pkg/epp/scheduling/plugins/plugins.go
+++ b/pkg/epp/scheduling/framework/plugins.go
@@ -14,18 +14,19 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-package plugins
+package framework
 
 import (
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
 const (
-	PreSchedulerPluginType = "PreSchedule"
+	ProfilePickerType      = "ProfilePicker"
+	PreCyclePluginType     = "PreCycle"
 	FilterPluginType       = "Filter"
 	ScorerPluginType       = "Scorer"
-	PostSchedulePluginType = "PostSchedule"
 	PickerPluginType       = "Picker"
+	PostCyclePluginType    = "PostCycle"
 	PostResponsePluginType = "PostResponse"
 )
 
@@ -36,11 +37,18 @@ type Plugin interface {
 	Name() string
 }
 
-// PreSchedule is called when the scheduler receives a new request. It can be used for various
-// initialization work.
-type PreSchedule interface {
+// ProfilePicker selects the SchedulingProfiles to run from a list of candidate profiles, while taking into consideration the request properties
+// and the previously executed SchedluderProfile cycles along with their results.
+type ProfilePicker interface {
 	Plugin
-	PreSchedule(ctx *types.SchedulingContext)
+	Pick(request *types.LLMRequest, profiles map[string]*SchedulerProfile, executionResults map[string]*types.Result) map[string]*SchedulerProfile
+}
+
+// PreCycle is called when the scheduler receives a new request and invokes a SchedulerProfile cycle.
+// It can be used for various initialization work.
+type PreCycle interface {
+	Plugin
+	PreCycle(ctx *types.SchedulingContext)
 }
 
 // Filter defines the interface for filtering a list of pods based on context.
@@ -62,10 +70,10 @@ type Picker interface {
 	Pick(ctx *types.SchedulingContext, scoredPods []*types.ScoredPod) *types.Result
 }
 
-// PostSchedule is called by the scheduler after it selects a targetPod for the request.
-type PostSchedule interface {
+// PostCycle is called by the scheduler after it selects a targetPod for the request in the SchedulerProfile cycle.
+type PostCycle interface {
 	Plugin
-	PostSchedule(ctx *types.SchedulingContext, res *types.Result)
+	PostCycle(ctx *types.SchedulingContext, res *types.Result)
 }
 
 // PostResponse is called by the scheduler after a successful response was sent.
diff --git a/pkg/epp/scheduling/framework/plugins/README.md b/pkg/epp/scheduling/framework/plugins/README.md
new file mode 100644
index 0000000000..56ca315e61
--- /dev/null
+++ b/pkg/epp/scheduling/framework/plugins/README.md
@@ -0,0 +1,15 @@
+# Scheduling Plugins
+
+This package contains the scheduling plugin implementations.
+
+Plugins are organized by the following rule. Follow this rule when adding a new
+plugin.
+
+```
+plugins/
+|__ filter/(Plugins that implement the Filter interface only.)
+|__ scorer/ (Plugins that implement the Scorer interface only.)
+|__ picker/(Plugins that implement the Picker interface only.)
+|__ multi/ (Plugins that implement multiple plugin interfaces.)
+|____prefix/ (Prefix cache aware scheduling plugin.)
+```
diff --git a/pkg/epp/scheduling/plugins/filter/decision_tree_filter.go b/pkg/epp/scheduling/framework/plugins/filter/decision_tree_filter.go
similarity index 93%
rename from pkg/epp/scheduling/plugins/filter/decision_tree_filter.go
rename to pkg/epp/scheduling/framework/plugins/filter/decision_tree_filter.go
index 066a90d69b..9baec0907e 100644
--- a/pkg/epp/scheduling/plugins/filter/decision_tree_filter.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/decision_tree_filter.go
@@ -17,31 +17,31 @@ limitations under the License.
 package filter
 
 import (
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
 )
 
-// compile-time type validation
-var _ plugins.Filter = &DecisionTreeFilter{}
+// compile-time type assertion
+var _ framework.Filter = &DecisionTreeFilter{}
 
 // DecisionTreeFilter applies current fitler, and then recursively applies next filters
 // depending success or failure of the current filter.
 // It can be used to construct a flow chart algorithm.
 type DecisionTreeFilter struct {
-	Current plugins.Filter
+	Current framework.Filter
 	// NextOnSuccess filter will be applied after successfully applying the current filter.
 	// The filtered results will be passed to the next filter.
-	NextOnSuccess plugins.Filter
+	NextOnSuccess framework.Filter
 	// NextOnFailure filter will be applied if current filter results in no pods.
 	// The original input will be passed to the next filter.
-	NextOnFailure plugins.Filter
+	NextOnFailure framework.Filter
 	// NextOnSuccessOrFailure is a convenience field to configure the next filter regardless of the
 	// success or failure of the current filter.
 	// NOTE: When using NextOnSuccessOrFailure, both nextOnSuccess and nextOnFailure SHOULD be nil.
 	// However if that's not the case, nextOnSuccess and nextOnFailure will be used, instead of
 	// NextOnSuccessOrFailure, in the success and failure scenarios, respectively.
-	NextOnSuccessOrFailure plugins.Filter
+	NextOnSuccessOrFailure framework.Filter
 }
 
 // Name returns the name of the filter.
diff --git a/pkg/epp/scheduling/plugins/filter/filter_test.go b/pkg/epp/scheduling/framework/plugins/filter/filter_test.go
similarity index 98%
rename from pkg/epp/scheduling/plugins/filter/filter_test.go
rename to pkg/epp/scheduling/framework/plugins/filter/filter_test.go
index 3f844740a0..e9064b5ec9 100644
--- a/pkg/epp/scheduling/plugins/filter/filter_test.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/filter_test.go
@@ -26,10 +26,13 @@ import (
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend"
 	backendmetrics "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend/metrics"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/config"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
+// compile-time type assertion
+var _ framework.Filter = &filterAll{}
+
 type filterAll struct{}
 
 func (f *filterAll) Name() string {
@@ -44,7 +47,7 @@ func TestFilter(t *testing.T) {
 	tests := []struct {
 		name   string
 		req    *types.LLMRequest
-		filter plugins.Filter
+		filter framework.Filter
 		input  []types.Pod
 		output []types.Pod
 	}{
diff --git a/pkg/epp/scheduling/plugins/filter/least_kvcache_filter.go b/pkg/epp/scheduling/framework/plugins/filter/least_kvcache_filter.go
similarity index 96%
rename from pkg/epp/scheduling/plugins/filter/least_kvcache_filter.go
rename to pkg/epp/scheduling/framework/plugins/filter/least_kvcache_filter.go
index ee7d824869..32942272e2 100644
--- a/pkg/epp/scheduling/plugins/filter/least_kvcache_filter.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/least_kvcache_filter.go
@@ -19,12 +19,12 @@ package filter
 import (
 	"math"
 
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
-// compile-time type validation
-var _ plugins.Filter = &LeastKVCacheFilter{}
+// compile-time type assertion
+var _ framework.Filter = &LeastKVCacheFilter{}
 
 // NewLeastKVCacheFilter initializes a new LeastKVCacheFilter and returns its pointer.
 func NewLeastKVCacheFilter() *LeastKVCacheFilter {
diff --git a/pkg/epp/scheduling/plugins/filter/least_queue_filter.go b/pkg/epp/scheduling/framework/plugins/filter/least_queue_filter.go
similarity index 96%
rename from pkg/epp/scheduling/plugins/filter/least_queue_filter.go
rename to pkg/epp/scheduling/framework/plugins/filter/least_queue_filter.go
index 7e0bfc32a9..1f24d194e3 100644
--- a/pkg/epp/scheduling/plugins/filter/least_queue_filter.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/least_queue_filter.go
@@ -19,12 +19,12 @@ package filter
 import (
 	"math"
 
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
-// compile-time type validation
-var _ plugins.Filter = &LeastQueueFilter{}
+// compile-time type assertion
+var _ framework.Filter = &LeastQueueFilter{}
 
 // NewLeastQueueFilter initializes a new LeastQueueFilter and returns its pointer.
 func NewLeastQueueFilter() *LeastQueueFilter {
diff --git a/pkg/epp/scheduling/plugins/filter/lora_affinity_filter.go b/pkg/epp/scheduling/framework/plugins/filter/lora_affinity_filter.go
similarity index 97%
rename from pkg/epp/scheduling/plugins/filter/lora_affinity_filter.go
rename to pkg/epp/scheduling/framework/plugins/filter/lora_affinity_filter.go
index bc1e55b032..d6110d09d3 100644
--- a/pkg/epp/scheduling/plugins/filter/lora_affinity_filter.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/lora_affinity_filter.go
@@ -21,12 +21,12 @@ import (
 	"time"
 
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/config"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
-// compile-time type validation
-var _ plugins.Filter = &LoraAffinityFilter{}
+// compile-time type assertion
+var _ framework.Filter = &LoraAffinityFilter{}
 
 // NewLoraAffinityFilter initializes a new LoraAffinityFilter and returns its pointer.
 func NewLoraAffinityFilter() *LoraAffinityFilter {
diff --git a/pkg/epp/scheduling/plugins/filter/low_queue_filter.go b/pkg/epp/scheduling/framework/plugins/filter/low_queue_filter.go
similarity index 95%
rename from pkg/epp/scheduling/plugins/filter/low_queue_filter.go
rename to pkg/epp/scheduling/framework/plugins/filter/low_queue_filter.go
index ce7d3523ae..fdff41f261 100644
--- a/pkg/epp/scheduling/plugins/filter/low_queue_filter.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/low_queue_filter.go
@@ -18,12 +18,12 @@ package filter
 
 import (
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/config"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
-// compile-time type validation
-var _ plugins.Filter = &LowQueueFilter{}
+// compile-time type assertion
+var _ framework.Filter = &LowQueueFilter{}
 
 // NewLowQueueFilter initializes a new LowQueueFilter and returns its pointer.
 func NewLowQueueFilter() *LowQueueFilter {
diff --git a/pkg/epp/scheduling/plugins/filter/sheddable_capacity_filter.go b/pkg/epp/scheduling/framework/plugins/filter/sheddable_capacity_filter.go
similarity index 95%
rename from pkg/epp/scheduling/plugins/filter/sheddable_capacity_filter.go
rename to pkg/epp/scheduling/framework/plugins/filter/sheddable_capacity_filter.go
index cdc3355fd9..555de2c0e7 100644
--- a/pkg/epp/scheduling/plugins/filter/sheddable_capacity_filter.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/sheddable_capacity_filter.go
@@ -18,12 +18,12 @@ package filter
 
 import (
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/config"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
-// compile-time type validation
-var _ plugins.Filter = &SheddableCapacityFilter{}
+// compile-time type assertion
+var _ framework.Filter = &SheddableCapacityFilter{}
 
 // NewSheddableCapacityFilter initializes a new SheddableCapacityFilter and returns its pointer.
 func NewSheddableCapacityFilter() *SheddableCapacityFilter {
diff --git a/pkg/epp/scheduling/plugins/multi/prefix/indexer.go b/pkg/epp/scheduling/framework/plugins/multi/prefix/indexer.go
similarity index 100%
rename from pkg/epp/scheduling/plugins/multi/prefix/indexer.go
rename to pkg/epp/scheduling/framework/plugins/multi/prefix/indexer.go
diff --git a/pkg/epp/scheduling/plugins/multi/prefix/indexer_test.go b/pkg/epp/scheduling/framework/plugins/multi/prefix/indexer_test.go
similarity index 100%
rename from pkg/epp/scheduling/plugins/multi/prefix/indexer_test.go
rename to pkg/epp/scheduling/framework/plugins/multi/prefix/indexer_test.go
diff --git a/pkg/epp/scheduling/plugins/multi/prefix/plugin.go b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go
similarity index 93%
rename from pkg/epp/scheduling/plugins/multi/prefix/plugin.go
rename to pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go
index b07f085db1..c3a28592ee 100644
--- a/pkg/epp/scheduling/plugins/multi/prefix/plugin.go
+++ b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go
@@ -23,7 +23,7 @@ import (
 	"github.com/cespare/xxhash/v2"
 	k8stypes "k8s.io/apimachinery/pkg/types"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
 )
@@ -88,7 +88,7 @@ func (s ServerID) String() string {
 	return k8stypes.NamespacedName(s).String()
 }
 
-// compile-time type validation
+// compile-time type assertion
 var _ types.StateData = &schedulingContextState{}
 
 // This is the state of this plugin to be used during a scheduling cycle.
@@ -113,10 +113,10 @@ func (s *schedulingContextState) Clone() types.StateData {
 	}
 }
 
-// compile-time type validation
-var _ plugins.PreSchedule = &Plugin{}
-var _ plugins.Scorer = &Plugin{}
-var _ plugins.PostSchedule = &Plugin{}
+// compile-time type assertion
+var _ framework.PreCycle = &Plugin{}
+var _ framework.Scorer = &Plugin{}
+var _ framework.PostCycle = &Plugin{}
 
 // New initializes a new prefix Plugin and returns its pointer.
 func New(config Config) *Plugin {
@@ -132,8 +132,8 @@ func (m *Plugin) Name() string {
 	return "prefix-cache"
 }
 
-// PreSchedule initializes the prefix plugin state for the current scheduling cycle.
-func (m *Plugin) PreSchedule(ctx *types.SchedulingContext) {
+// PreCycle initializes the prefix plugin state for the current scheduling cycle.
+func (m *Plugin) PreCycle(ctx *types.SchedulingContext) {
 	hashes := hashPrompt(ctx, m.HashBlockSize, m.MaxPrefixBlocksToMatch)
 	state := &schedulingContextState{
 		PrefixHashes:       hashes,
@@ -141,11 +141,11 @@ func (m *Plugin) PreSchedule(ctx *types.SchedulingContext) {
 	}
 
 	ctx.CycleState.Write(types.StateKey(m.Name()), state)
-	ctx.Logger.V(logutil.TRACE).Info(fmt.Sprintf("PreSchedule, cached servers: %+v", state.PrefixCacheServers), "hashes", state.PrefixHashes)
+	ctx.Logger.V(logutil.TRACE).Info(fmt.Sprintf("PreCycle, cached servers: %+v", state.PrefixCacheServers), "hashes", state.PrefixHashes)
 }
 
-// PostSchedule records in the plugin cache the result of the scheduling selection.
-func (m *Plugin) PostSchedule(ctx *types.SchedulingContext, res *types.Result) {
+// PostCycle records in the plugin cache the result of the scheduling selection.
+func (m *Plugin) PostCycle(ctx *types.SchedulingContext, res *types.Result) {
 	targetPod := res.TargetPod.GetPod()
 	state, err := m.getPrefixState(ctx.CycleState)
 	if err != nil {
diff --git a/pkg/epp/scheduling/plugins/multi/prefix/plugin_test.go b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go
similarity index 92%
rename from pkg/epp/scheduling/plugins/multi/prefix/plugin_test.go
rename to pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go
index 7e4e218ec0..65e324c996 100644
--- a/pkg/epp/scheduling/plugins/multi/prefix/plugin_test.go
+++ b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go
@@ -28,7 +28,7 @@ func TestPrefixPlugin(t *testing.T) {
 		Prompt:      "aaaaaa",
 	}
 	ctx := types.NewSchedulingContext(context.Background(), req1, nil, pods)
-	plugin.PreSchedule(ctx)
+	plugin.PreCycle(ctx)
 	state, err := plugin.getPrefixState(ctx.CycleState)
 	assert.NoError(t, err)
 	t.Logf("Hashes %+v, cached servers: %+v", state.PrefixHashes, state.PrefixCacheServers)
@@ -43,7 +43,7 @@ func TestPrefixPlugin(t *testing.T) {
 	assert.Equal(t, float64(0), scores[pod2], "score for pod2")
 
 	// Simulate pod1 was picked.
-	plugin.PostSchedule(ctx, &types.Result{TargetPod: pod1})
+	plugin.PostCycle(ctx, &types.Result{TargetPod: pod1})
 
 	// Second request doesn't share any prefix with first one. It should be added to the cache but
 	// the pod score should be 0.
@@ -52,7 +52,7 @@ func TestPrefixPlugin(t *testing.T) {
 		Prompt:      "bbbbbb",
 	}
 	ctx = types.NewSchedulingContext(context.Background(), req2, nil, pods)
-	plugin.PreSchedule(ctx)
+	plugin.PreCycle(ctx)
 	state, err = plugin.getPrefixState(ctx.CycleState)
 	assert.NoError(t, err)
 	t.Logf("Hashes %+v, cached servers: %+v", state.PrefixHashes, state.PrefixCacheServers)
@@ -67,7 +67,7 @@ func TestPrefixPlugin(t *testing.T) {
 	assert.Equal(t, float64(0), scores[pod2], "score for pod2")
 
 	// Simulate pod2 was picked.
-	plugin.PostSchedule(ctx, &types.Result{TargetPod: pod2})
+	plugin.PostCycle(ctx, &types.Result{TargetPod: pod2})
 
 	// Third request shares partial prefix with first one.
 	req3 := &types.LLMRequest{
@@ -75,7 +75,7 @@ func TestPrefixPlugin(t *testing.T) {
 		Prompt:      "aaaabbbb",
 	}
 	ctx = types.NewSchedulingContext(context.Background(), req3, nil, pods)
-	plugin.PreSchedule(ctx)
+	plugin.PreCycle(ctx)
 	state, err = plugin.getPrefixState(ctx.CycleState)
 	assert.NoError(t, err)
 	t.Logf("Hashes %+v, cached servers: %+v", state.PrefixHashes, state.PrefixCacheServers)
@@ -89,7 +89,7 @@ func TestPrefixPlugin(t *testing.T) {
 	assert.Equal(t, float64(2)/float64(3), scores[pod1], "score should be 2/3 - the model and the first prefix block match")
 	assert.Equal(t, float64(0), scores[pod2], "score for pod2")
 
-	plugin.PostSchedule(ctx, &types.Result{TargetPod: pod1})
+	plugin.PostCycle(ctx, &types.Result{TargetPod: pod1})
 
 	// 4th request is same as req3 except the model is different, still no match.
 	req4 := &types.LLMRequest{
@@ -97,7 +97,7 @@ func TestPrefixPlugin(t *testing.T) {
 		Prompt:      "aaaabbbb",
 	}
 	ctx = types.NewSchedulingContext(context.Background(), req4, nil, pods)
-	plugin.PreSchedule(ctx)
+	plugin.PreCycle(ctx)
 	state, err = plugin.getPrefixState(ctx.CycleState)
 	assert.NoError(t, err)
 	t.Logf("Hashes %+v, cached servers: %+v", state.PrefixHashes, state.PrefixCacheServers)
@@ -111,7 +111,7 @@ func TestPrefixPlugin(t *testing.T) {
 	assert.Equal(t, float64(0), scores[pod1], "score for pod1")
 	assert.Equal(t, float64(0), scores[pod2], "score for pod2")
 
-	plugin.PostSchedule(ctx, &types.Result{TargetPod: pod1})
+	plugin.PostCycle(ctx, &types.Result{TargetPod: pod1})
 
 	// 5th request shares partial prefix with 3rd one.
 	req5 := &types.LLMRequest{
@@ -119,7 +119,7 @@ func TestPrefixPlugin(t *testing.T) {
 		Prompt:      "aaaabbbbcccc",
 	}
 	ctx = types.NewSchedulingContext(context.Background(), req5, nil, pods)
-	plugin.PreSchedule(ctx)
+	plugin.PreCycle(ctx)
 	state, err = plugin.getPrefixState(ctx.CycleState)
 	assert.NoError(t, err)
 	t.Logf("Hashes %+v, cached servers: %+v", state.PrefixHashes, state.PrefixCacheServers)
@@ -133,5 +133,5 @@ func TestPrefixPlugin(t *testing.T) {
 	assert.Equal(t, 0.75, scores[pod1], "score should be 0.75 - the model and the first 2 prefix blocks match")
 	assert.Equal(t, float64(0), scores[pod2], "score for pod2")
 
-	plugin.PostSchedule(ctx, &types.Result{TargetPod: pod1})
+	plugin.PostCycle(ctx, &types.Result{TargetPod: pod1})
 }
diff --git a/pkg/epp/scheduling/plugins/picker/max_score_picker.go b/pkg/epp/scheduling/framework/plugins/picker/max_score_picker.go
similarity index 96%
rename from pkg/epp/scheduling/plugins/picker/max_score_picker.go
rename to pkg/epp/scheduling/framework/plugins/picker/max_score_picker.go
index e85aee5fe3..f206956c13 100644
--- a/pkg/epp/scheduling/plugins/picker/max_score_picker.go
+++ b/pkg/epp/scheduling/framework/plugins/picker/max_score_picker.go
@@ -19,13 +19,13 @@ package picker
 import (
 	"fmt"
 
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
 )
 
-// compile-time type validation
-var _ plugins.Picker = &MaxScorePicker{}
+// compile-time type assertion
+var _ framework.Picker = &MaxScorePicker{}
 
 // NewMaxScorePicker initializes a new MaxScorePicker and returns its pointer.
 func NewMaxScorePicker() *MaxScorePicker {
diff --git a/pkg/epp/scheduling/plugins/picker/random_picker.go b/pkg/epp/scheduling/framework/plugins/picker/random_picker.go
similarity index 94%
rename from pkg/epp/scheduling/plugins/picker/random_picker.go
rename to pkg/epp/scheduling/framework/plugins/picker/random_picker.go
index 1d12198ceb..6a50da0f00 100644
--- a/pkg/epp/scheduling/plugins/picker/random_picker.go
+++ b/pkg/epp/scheduling/framework/plugins/picker/random_picker.go
@@ -20,13 +20,13 @@ import (
 	"fmt"
 	"math/rand"
 
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
 )
 
-// compile-time type validation
-var _ plugins.Picker = &RandomPicker{}
+// compile-time type assertion
+var _ framework.Picker = &RandomPicker{}
 
 // NewRandomPicker initializes a new RandomPicker and returns its pointer.
 func NewRandomPicker() *RandomPicker {
diff --git a/pkg/epp/scheduling/framework/plugins/profile-picker/all_profiles_picker.go b/pkg/epp/scheduling/framework/plugins/profile-picker/all_profiles_picker.go
new file mode 100644
index 0000000000..eb9c741037
--- /dev/null
+++ b/pkg/epp/scheduling/framework/plugins/profile-picker/all_profiles_picker.go
@@ -0,0 +1,48 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package profilepicker
+
+import (
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+// compile-time type assertion
+var _ framework.ProfilePicker = &AllProfilesPicker{}
+
+// NewAllProfilesPicker initializes a new AllProfilesPicker and returns its pointer.
+func NewAllProfilesPicker() *AllProfilesPicker {
+	return &AllProfilesPicker{}
+}
+
+// AllProfilesPicker picks all profiles always.
+type AllProfilesPicker struct{}
+
+// Name returns the name of the Profiles Picker.
+func (p *AllProfilesPicker) Name() string {
+	return "all-profiles"
+}
+
+// Pick selects the SchedulingProfiles to run from the list of candidate profiles, while taking into consideration the request properties and the
+// previously executed cycles along with their results.
+func (p *AllProfilesPicker) Pick(request *types.LLMRequest, profiles map[string]*framework.SchedulerProfile, executionResults map[string]*types.Result) map[string]*framework.SchedulerProfile {
+	if len(profiles) == len(executionResults) { // all profiles have been executed already in previous call
+		return map[string]*framework.SchedulerProfile{}
+	}
+	// return all profiles
+	return profiles
+}
diff --git a/pkg/epp/scheduling/plugins/scorer/kvcache.go b/pkg/epp/scheduling/framework/plugins/scorer/kvcache.go
similarity index 93%
rename from pkg/epp/scheduling/plugins/scorer/kvcache.go
rename to pkg/epp/scheduling/framework/plugins/scorer/kvcache.go
index de6f87b0e8..916de3d153 100644
--- a/pkg/epp/scheduling/plugins/scorer/kvcache.go
+++ b/pkg/epp/scheduling/framework/plugins/scorer/kvcache.go
@@ -17,7 +17,7 @@ limitations under the License.
 package scorer
 
 import (
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
@@ -25,8 +25,8 @@ const (
 	DefaultKVCacheScorerWeight = 1
 )
 
-// compile-time type validation
-var _ plugins.Scorer = &KVCacheScorer{}
+// compile-time type assertion
+var _ framework.Scorer = &KVCacheScorer{}
 
 // KVCacheScorer scores list of candidate pods based on KV cache utilization.
 type KVCacheScorer struct{}
diff --git a/pkg/epp/scheduling/plugins/scorer/kvcache_test.go b/pkg/epp/scheduling/framework/plugins/scorer/kvcache_test.go
similarity index 100%
rename from pkg/epp/scheduling/plugins/scorer/kvcache_test.go
rename to pkg/epp/scheduling/framework/plugins/scorer/kvcache_test.go
diff --git a/pkg/epp/scheduling/plugins/scorer/queue.go b/pkg/epp/scheduling/framework/plugins/scorer/queue.go
similarity index 96%
rename from pkg/epp/scheduling/plugins/scorer/queue.go
rename to pkg/epp/scheduling/framework/plugins/scorer/queue.go
index ac87f0b9e1..93f0a4d4c0 100644
--- a/pkg/epp/scheduling/plugins/scorer/queue.go
+++ b/pkg/epp/scheduling/framework/plugins/scorer/queue.go
@@ -19,7 +19,7 @@ package scorer
 import (
 	"math"
 
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
@@ -27,8 +27,8 @@ const (
 	DefaultQueueScorerWeight = 1
 )
 
-// compile-time type validation
-var _ plugins.Scorer = &QueueScorer{}
+// compile-time type assertion
+var _ framework.Scorer = &QueueScorer{}
 
 // QueueScorer scores list of candidate pods based on the pod's waiting queue size.
 // the less waiting queue size the pod has, the higher score it will get (since it's more available to serve new request).
diff --git a/pkg/epp/scheduling/plugins/scorer/queue_test.go b/pkg/epp/scheduling/framework/plugins/scorer/queue_test.go
similarity index 100%
rename from pkg/epp/scheduling/plugins/scorer/queue_test.go
rename to pkg/epp/scheduling/framework/plugins/scorer/queue_test.go
diff --git a/pkg/epp/scheduling/framework/scheduler_profile.go b/pkg/epp/scheduling/framework/scheduler_profile.go
new file mode 100644
index 0000000000..6809624fc9
--- /dev/null
+++ b/pkg/epp/scheduling/framework/scheduler_profile.go
@@ -0,0 +1,218 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package framework
+
+import (
+	"fmt"
+	"time"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+	errutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/error"
+	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
+)
+
+// NewSchedulerProfile creates a new SchedulerProfile object and returns its pointer.
+func NewSchedulerProfile() *SchedulerProfile {
+	return &SchedulerProfile{
+		preCyclePlugins:     []PreCycle{},
+		filters:             []Filter{},
+		scorers:             []*WeightedScorer{},
+		postCyclePlugins:    []PostCycle{},
+		PostResponsePlugins: []PostResponse{},
+		// picker remains nil since profile doesn't support multiple pickers
+	}
+}
+
+// SchedulerProfile provides a profile configuration for the scheduler which influence routing decisions.
+type SchedulerProfile struct {
+	preCyclePlugins     []PreCycle
+	filters             []Filter
+	scorers             []*WeightedScorer
+	picker              Picker
+	postCyclePlugins    []PostCycle
+	PostResponsePlugins []PostResponse // TODO this field should get out of the scheduler
+}
+
+// WithPreCyclePlugins sets the given plugins as the PreCycle plugins.
+// If the SchedulerProfile has PreCycle plugins, this call replaces the existing plugins with the given ones.
+func (p *SchedulerProfile) WithPreCyclePlugins(plugins ...PreCycle) *SchedulerProfile {
+	p.preCyclePlugins = plugins
+	return p
+}
+
+// WithFilters sets the given filter plugins as the Filter plugins.
+// if the SchedulerProfile has Filter plugins, this call replaces the existing plugins with the given ones.
+func (p *SchedulerProfile) WithFilters(filters ...Filter) *SchedulerProfile {
+	p.filters = filters
+	return p
+}
+
+// WithScorers sets the given scorer plugins as the Scorer plugins.
+// if the SchedulerProfile has Scorer plugins, this call replaces the existing plugins with the given ones.
+func (p *SchedulerProfile) WithScorers(scorers ...*WeightedScorer) *SchedulerProfile {
+	p.scorers = scorers
+	return p
+}
+
+// WithPicker sets the given picker plugins as the Picker plugin.
+// if the SchedulerProfile has Picker plugin, this call replaces the existing plugin with the given one.
+func (p *SchedulerProfile) WithPicker(picker Picker) *SchedulerProfile {
+	p.picker = picker
+	return p
+}
+
+// WithPostCyclePlugins sets the given plugins as the PostCycle plugins.
+// If the SchedulerProfile has PostCycle plugins, this call replaces the existing plugins with the given ones.
+func (p *SchedulerProfile) WithPostCyclePlugins(plugins ...PostCycle) *SchedulerProfile {
+	p.postCyclePlugins = plugins
+	return p
+}
+
+// AddPlugins adds the given plugins to all scheduler plugins according to the interfaces each plugin implements.
+// A plugin may implement more than one scheduler plugin interface.
+// Special Case: In order to add a scorer, one must use the scorer.NewWeightedScorer function in order to provide a weight.
+// if a scorer implements more than one interface, supplying a WeightedScorer is sufficient. The function will take the internal
+// scorer object and register it to all interfaces it implements.
+func (p *SchedulerProfile) AddPlugins(pluginObjects ...Plugin) error {
+	for _, plugin := range pluginObjects {
+		if weightedScorer, ok := plugin.(*WeightedScorer); ok {
+			p.scorers = append(p.scorers, weightedScorer)
+			plugin = weightedScorer.Scorer // if we got WeightedScorer, unwrap the plugin
+		} else if scorer, ok := plugin.(Scorer); ok { // if we got a Scorer instead of WeightedScorer that's an error.
+			return fmt.Errorf("failed to register scorer '%s' without a weight. follow function documentation to register a scorer", scorer.Name())
+		}
+		if preCyclePlugin, ok := plugin.(PreCycle); ok {
+			p.preCyclePlugins = append(p.preCyclePlugins, preCyclePlugin)
+		}
+		if filter, ok := plugin.(Filter); ok {
+			p.filters = append(p.filters, filter)
+		}
+		if picker, ok := plugin.(Picker); ok {
+			if p.picker != nil {
+				return fmt.Errorf("failed to set '%s' as picker, already have a registered picker plugin '%s'", picker.Name(), p.picker.Name())
+			}
+			p.picker = picker
+		}
+		if postCyclePlugin, ok := plugin.(PostCycle); ok {
+			p.postCyclePlugins = append(p.postCyclePlugins, postCyclePlugin)
+		}
+		if postResponsePlugin, ok := plugin.(PostResponse); ok {
+			p.PostResponsePlugins = append(p.PostResponsePlugins, postResponsePlugin)
+		}
+	}
+	return nil
+}
+
+// RunCycle runs a SchedulerProfile cycle. In other words, it invokes all the SchedulerProfile plugins in this
+// order - PreCyclePlugins, Filters, Scorers, Picker, PostCyclePlugins. After completing all, it returns the result.
+func (p *SchedulerProfile) RunCycle(ctx *types.SchedulingContext) (*types.Result, error) {
+	p.runPreCyclePlugins(ctx)
+
+	pods := p.runFilterPlugins(ctx)
+	if len(pods) == 0 {
+		return nil, errutil.Error{Code: errutil.Internal, Msg: "no pods available for the given request"}
+	}
+	// if we got here, there is at least one pod to score
+	weightedScorePerPod := p.runScorerPlugins(ctx, pods)
+
+	result := p.runPickerPlugin(ctx, weightedScorePerPod)
+
+	p.runPostCyclePlugins(ctx, result)
+
+	return result, nil
+}
+
+func (p *SchedulerProfile) runPreCyclePlugins(ctx *types.SchedulingContext) {
+	for _, plugin := range p.preCyclePlugins {
+		ctx.Logger.V(logutil.DEBUG).Info("Running pre-cycle plugin", "plugin", plugin.Name())
+		before := time.Now()
+		plugin.PreCycle(ctx)
+		metrics.RecordSchedulerPluginProcessingLatency(PreCyclePluginType, plugin.Name(), time.Since(before))
+	}
+}
+
+func (p *SchedulerProfile) runFilterPlugins(ctx *types.SchedulingContext) []types.Pod {
+	loggerDebug := ctx.Logger.V(logutil.DEBUG)
+	filteredPods := ctx.PodsSnapshot
+	loggerDebug.Info("Before running filter plugins", "pods", filteredPods)
+
+	for _, filter := range p.filters {
+		loggerDebug.Info("Running filter plugin", "plugin", filter.Name())
+		before := time.Now()
+		filteredPods = filter.Filter(ctx, filteredPods)
+		metrics.RecordSchedulerPluginProcessingLatency(FilterPluginType, filter.Name(), time.Since(before))
+		loggerDebug.Info("Filter plugin result", "plugin", filter.Name(), "pods", filteredPods)
+		if len(filteredPods) == 0 {
+			break
+		}
+	}
+	loggerDebug.Info("After running filter plugins")
+
+	return filteredPods
+}
+
+func (p *SchedulerProfile) runScorerPlugins(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	loggerDebug := ctx.Logger.V(logutil.DEBUG)
+	loggerDebug.Info("Before running scorer plugins", "pods", pods)
+
+	weightedScorePerPod := make(map[types.Pod]float64, len(pods))
+	for _, pod := range pods {
+		weightedScorePerPod[pod] = float64(0) // initialize weighted score per pod with 0 value
+	}
+	// Iterate through each scorer in the chain and accumulate the weighted scores.
+	for _, scorer := range p.scorers {
+		loggerDebug.Info("Running scorer", "scorer", scorer.Name())
+		before := time.Now()
+		scores := scorer.Score(ctx, pods)
+		metrics.RecordSchedulerPluginProcessingLatency(ScorerPluginType, scorer.Name(), time.Since(before))
+		for pod, score := range scores { // weight is relative to the sum of weights
+			weightedScorePerPod[pod] += score * float64(scorer.Weight())
+		}
+		loggerDebug.Info("After running scorer", "scorer", scorer.Name())
+	}
+	loggerDebug.Info("After running scorer plugins")
+
+	return weightedScorePerPod
+}
+
+func (p *SchedulerProfile) runPickerPlugin(ctx *types.SchedulingContext, weightedScorePerPod map[types.Pod]float64) *types.Result {
+	loggerDebug := ctx.Logger.V(logutil.DEBUG)
+	scoredPods := make([]*types.ScoredPod, len(weightedScorePerPod))
+	i := 0
+	for pod, score := range weightedScorePerPod {
+		scoredPods[i] = &types.ScoredPod{Pod: pod, Score: score}
+		i++
+	}
+
+	loggerDebug.Info("Before running picker plugin", "pods weighted score", fmt.Sprint(weightedScorePerPod))
+	before := time.Now()
+	result := p.picker.Pick(ctx, scoredPods)
+	metrics.RecordSchedulerPluginProcessingLatency(PickerPluginType, p.picker.Name(), time.Since(before))
+	loggerDebug.Info("After running picker plugin", "result", result)
+
+	return result
+}
+
+func (p *SchedulerProfile) runPostCyclePlugins(ctx *types.SchedulingContext, res *types.Result) {
+	for _, plugin := range p.postCyclePlugins {
+		ctx.Logger.V(logutil.DEBUG).Info("Running post-cycle plugin", "plugin", plugin.Name())
+		before := time.Now()
+		plugin.PostCycle(ctx, res)
+		metrics.RecordSchedulerPluginProcessingLatency(PostCyclePluginType, plugin.Name(), time.Since(before))
+	}
+}
diff --git a/pkg/epp/scheduling/framework/scheduler_profile_test.go b/pkg/epp/scheduling/framework/scheduler_profile_test.go
new file mode 100644
index 0000000000..d212f60408
--- /dev/null
+++ b/pkg/epp/scheduling/framework/scheduler_profile_test.go
@@ -0,0 +1,284 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package framework
+
+import (
+	"context"
+	"testing"
+
+	"github.com/google/go-cmp/cmp"
+	"github.com/google/uuid"
+	k8stypes "k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend"
+	backendmetrics "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend/metrics" // Import config for thresholds
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+func TestSchedulePlugins(t *testing.T) {
+	tp1 := &testPlugin{
+		NameRes:   "test1",
+		ScoreRes:  0.3,
+		FilterRes: []k8stypes.NamespacedName{{Name: "pod1"}, {Name: "pod2"}, {Name: "pod3"}},
+	}
+	tp2 := &testPlugin{
+		NameRes:   "test2",
+		ScoreRes:  0.8,
+		FilterRes: []k8stypes.NamespacedName{{Name: "pod1"}, {Name: "pod2"}},
+	}
+	tp_filterAll := &testPlugin{
+		NameRes:   "filter all",
+		FilterRes: []k8stypes.NamespacedName{},
+	}
+	pickerPlugin := &testPlugin{
+		NameRes: "picker",
+		PickRes: k8stypes.NamespacedName{Name: "pod1"},
+	}
+
+	tests := []struct {
+		name           string
+		profile        *SchedulerProfile
+		input          []backendmetrics.PodMetrics
+		wantTargetPod  k8stypes.NamespacedName
+		targetPodScore float64
+		// Number of expected pods to score (after filter)
+		numPodsToScore int
+		err            bool
+	}{
+		{
+			name: "all plugins executed successfully, all scorers with same weight",
+			profile: NewSchedulerProfile().
+				WithPreCyclePlugins(tp1, tp2).
+				WithFilters(tp1, tp2).
+				WithScorers(NewWeightedScorer(tp1, 1), NewWeightedScorer(tp2, 1)).
+				WithPicker(pickerPlugin).
+				WithPostCyclePlugins(tp1, tp2),
+			input: []backendmetrics.PodMetrics{
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}}},
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}}},
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod3"}}},
+			},
+			wantTargetPod:  k8stypes.NamespacedName{Name: "pod1"},
+			targetPodScore: 1.1,
+			numPodsToScore: 2,
+			err:            false,
+		},
+		{
+			name: "all plugins executed successfully, different scorers weights",
+			profile: NewSchedulerProfile().
+				WithPreCyclePlugins(tp1, tp2).
+				WithFilters(tp1, tp2).
+				WithScorers(NewWeightedScorer(tp1, 60), NewWeightedScorer(tp2, 40)).
+				WithPicker(pickerPlugin).
+				WithPostCyclePlugins(tp1, tp2),
+			input: []backendmetrics.PodMetrics{
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}}},
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}}},
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod3"}}},
+			},
+			wantTargetPod:  k8stypes.NamespacedName{Name: "pod1"},
+			targetPodScore: 50,
+			numPodsToScore: 2,
+			err:            false,
+		},
+		{
+			name: "filter all",
+			profile: NewSchedulerProfile().
+				WithPreCyclePlugins(tp1, tp2).
+				WithFilters(tp1, tp_filterAll).
+				WithScorers(NewWeightedScorer(tp1, 1), NewWeightedScorer(tp2, 1)).
+				WithPicker(pickerPlugin).
+				WithPostCyclePlugins(tp1, tp2),
+			input: []backendmetrics.PodMetrics{
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}}},
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}}},
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod3"}}},
+			},
+			numPodsToScore: 0,
+			err:            true, // no available pods to server after filter all
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			// Reset all plugins before each new test case.
+			for _, plugin := range test.profile.preCyclePlugins {
+				plugin.(*testPlugin).reset()
+			}
+			for _, plugin := range test.profile.filters {
+				plugin.(*testPlugin).reset()
+			}
+			for _, plugin := range test.profile.scorers {
+				plugin.Scorer.(*testPlugin).reset()
+			}
+			test.profile.picker.(*testPlugin).reset()
+			for _, plugin := range test.profile.postCyclePlugins {
+				plugin.(*testPlugin).reset()
+			}
+
+			// Initialize the scheduling context
+			req := &types.LLMRequest{
+				TargetModel: "test-model",
+				RequestId:   uuid.NewString(),
+			}
+			schedulingContext := types.NewSchedulingContext(context.Background(), req, nil, types.ToSchedulerPodMetrics(test.input))
+			// Run profile cycle
+			got, err := test.profile.RunCycle(schedulingContext)
+
+			// Validate error state
+			if test.err != (err != nil) {
+				t.Fatalf("Unexpected error, got %v, want %v", err, test.err)
+			}
+
+			if err != nil {
+				return
+			}
+
+			// Validate output
+			wantPod := &types.PodMetrics{
+				Pod: &backend.Pod{NamespacedName: test.wantTargetPod, Labels: make(map[string]string)},
+			}
+			wantRes := &types.Result{
+				TargetPod: wantPod,
+			}
+
+			if diff := cmp.Diff(wantRes, got); diff != "" {
+				t.Errorf("Unexpected output (-want +got): %v", diff)
+			}
+			// Validate plugin execution counts dynamically
+			for _, plugin := range test.profile.preCyclePlugins {
+				tp, _ := plugin.(*testPlugin)
+				if tp.PreScheduleCallCount != 1 {
+					t.Errorf("Plugin %s PreSchedule() called %d times, expected 1", plugin.Name(), tp.PreScheduleCallCount)
+				}
+			}
+			for _, plugin := range test.profile.filters {
+				tp, _ := plugin.(*testPlugin)
+				if tp.FilterCallCount != 1 {
+					t.Errorf("Plugin %s Filter() called %d times, expected 1", plugin.Name(), tp.FilterCallCount)
+				}
+			}
+			for _, plugin := range test.profile.scorers {
+				tp, _ := plugin.Scorer.(*testPlugin)
+				if tp.ScoreCallCount != 1 {
+					t.Errorf("Plugin %s Score() called %d times, expected 1", plugin.Name(), tp.ScoreCallCount)
+				}
+				if test.numPodsToScore != tp.NumOfScoredPods {
+					t.Errorf("Plugin %s Score() called with %d pods, expected %d", plugin.Name(), tp.NumOfScoredPods, test.numPodsToScore)
+				}
+			}
+			tp, _ := test.profile.picker.(*testPlugin)
+			if tp.NumOfPickerCandidates != test.numPodsToScore {
+				t.Errorf("Picker plugin %s Pick() called with %d candidates, expected %d", tp.Name(), tp.NumOfPickerCandidates, tp.NumOfScoredPods)
+			}
+			if tp.PickCallCount != 1 {
+				t.Errorf("Picker plugin %s Pick() called %d times, expected 1", tp.Name(), tp.PickCallCount)
+			}
+			if tp.WinnderPodScore != test.targetPodScore {
+				t.Errorf("winnder pod score %v, expected %v", tp.WinnderPodScore, test.targetPodScore)
+			}
+			for _, plugin := range test.profile.postCyclePlugins {
+				tp, _ := plugin.(*testPlugin)
+				if tp.PostScheduleCallCount != 1 {
+					t.Errorf("Plugin %s PostSchedule() called %d times, expected 1", plugin.Name(), tp.PostScheduleCallCount)
+				}
+			}
+		})
+	}
+}
+
+// testPlugin is an implementation useful in unit tests.
+type testPlugin struct {
+	NameRes               string
+	ScoreCallCount        int
+	NumOfScoredPods       int
+	ScoreRes              float64
+	FilterCallCount       int
+	FilterRes             []k8stypes.NamespacedName
+	PreScheduleCallCount  int
+	PostScheduleCallCount int
+	PickCallCount         int
+	NumOfPickerCandidates int
+	PickRes               k8stypes.NamespacedName
+	WinnderPodScore       float64
+}
+
+func (tp *testPlugin) Name() string { return tp.NameRes }
+
+func (tp *testPlugin) PreCycle(ctx *types.SchedulingContext) {
+	tp.PreScheduleCallCount++
+}
+
+func (tp *testPlugin) Filter(ctx *types.SchedulingContext, pods []types.Pod) []types.Pod {
+	tp.FilterCallCount++
+	return findPods(ctx, tp.FilterRes...)
+
+}
+
+func (tp *testPlugin) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	tp.ScoreCallCount++
+	scoredPods := make(map[types.Pod]float64, len(pods))
+	for _, pod := range pods {
+		scoredPods[pod] += tp.ScoreRes
+	}
+	tp.NumOfScoredPods = len(scoredPods)
+	return scoredPods
+}
+
+func (tp *testPlugin) Pick(ctx *types.SchedulingContext, scoredPods []*types.ScoredPod) *types.Result {
+	tp.PickCallCount++
+	tp.NumOfPickerCandidates = len(scoredPods)
+	pod := findPods(ctx, tp.PickRes)[0]
+	tp.WinnderPodScore = getPodScore(scoredPods, pod)
+	return &types.Result{TargetPod: pod}
+}
+
+func (tp *testPlugin) PostCycle(ctx *types.SchedulingContext, res *types.Result) {
+	tp.PostScheduleCallCount++
+}
+
+func (tp *testPlugin) reset() {
+	tp.PreScheduleCallCount = 0
+	tp.FilterCallCount = 0
+	tp.ScoreCallCount = 0
+	tp.NumOfScoredPods = 0
+	tp.PostScheduleCallCount = 0
+	tp.PickCallCount = 0
+	tp.NumOfPickerCandidates = 0
+}
+
+func findPods(ctx *types.SchedulingContext, names ...k8stypes.NamespacedName) []types.Pod {
+	res := []types.Pod{}
+	for _, pod := range ctx.PodsSnapshot {
+		for _, name := range names {
+			if pod.GetPod().NamespacedName.String() == name.String() {
+				res = append(res, pod)
+			}
+		}
+	}
+	return res
+}
+
+func getPodScore(scoredPods []*types.ScoredPod, selectedPod types.Pod) float64 {
+	finalScore := 0.0
+	for _, scoredPod := range scoredPods {
+		if scoredPod.GetPod().NamespacedName.String() == selectedPod.GetPod().NamespacedName.String() {
+			finalScore = scoredPod.Score
+			break
+		}
+	}
+	return finalScore
+}
diff --git a/pkg/epp/scheduling/plugins/scorer/weighted_scorer.go b/pkg/epp/scheduling/framework/weighted_scorer.go
similarity index 82%
rename from pkg/epp/scheduling/plugins/scorer/weighted_scorer.go
rename to pkg/epp/scheduling/framework/weighted_scorer.go
index 1e83b6c834..3b8d80a42e 100644
--- a/pkg/epp/scheduling/plugins/scorer/weighted_scorer.go
+++ b/pkg/epp/scheduling/framework/weighted_scorer.go
@@ -14,14 +14,10 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-package scorer
-
-import (
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
-)
+package framework
 
 // NewWeightedScorer initializes a new WeightedScorer and returns its pointer.
-func NewWeightedScorer(scorer plugins.Scorer, weight int) *WeightedScorer {
+func NewWeightedScorer(scorer Scorer, weight int) *WeightedScorer {
 	return &WeightedScorer{
 		Scorer: scorer,
 		weight: weight,
@@ -30,7 +26,7 @@ func NewWeightedScorer(scorer plugins.Scorer, weight int) *WeightedScorer {
 
 // WeightedScorer is a struct that encapsulates a scorer with its weight.
 type WeightedScorer struct {
-	plugins.Scorer
+	Scorer
 	weight int
 }
 
diff --git a/pkg/epp/scheduling/plugins/README.md b/pkg/epp/scheduling/plugins/README.md
deleted file mode 100644
index 80118ddd94..0000000000
--- a/pkg/epp/scheduling/plugins/README.md
+++ /dev/null
@@ -1,16 +0,0 @@
-# Scheduling Plugins
-
-This package contains the scheduling plugin interface definitions and
-implementations.
-
-Plugins are organized by the following rule. Follow this rule when adding a new
-plugin.
-
-```
-plugins/
-|__ filter/(Plugins that only implement the Filter interface.)
-|__ scorer/ (Plugins that only implement the Scorer interface.)
-|__ picker/(Plugins that only implement the Picker interface.)
-|__ multi/ (Plugins that implement multiple plugin interfaces.)
-|____prefix/ (Prefix cache aware scheduling plugin.)
-```
diff --git a/pkg/epp/scheduling/scheduler.go b/pkg/epp/scheduling/scheduler.go
index 87591b2bc7..f2ac743fcf 100644
--- a/pkg/epp/scheduling/scheduler.go
+++ b/pkg/epp/scheduling/scheduler.go
@@ -25,12 +25,11 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/log"
 	backendmetrics "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend/metrics"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins/filter"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins/picker"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins/scorer"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/filter"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/picker"
+	profilepicker "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile-picker"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
-	errutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/error"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
 )
 
@@ -65,34 +64,28 @@ func NewScheduler(datastore Datastore) *Scheduler {
 		},
 	}
 
-	defaultConfig := NewSchedulerConfig().
+	defaultProfile := framework.NewSchedulerProfile().
 		WithFilters(filter.NewSheddableCapacityFilter(), lowLatencyFilter).
 		WithPicker(&picker.RandomPicker{})
 
-	return NewSchedulerWithConfig(datastore, defaultConfig)
+	profilePicker := profilepicker.NewAllProfilesPicker()
+
+	return NewSchedulerWithConfig(datastore, NewSchedulerConfig(profilePicker, map[string]*framework.SchedulerProfile{"default": defaultProfile}))
 }
 
 // NewSchedulerWithConfig returns a new scheduler with the given scheduler plugins configuration.
 func NewSchedulerWithConfig(datastore Datastore, config *SchedulerConfig) *Scheduler {
 	return &Scheduler{
-		datastore:           datastore,
-		preSchedulePlugins:  config.preSchedulePlugins,
-		filters:             config.filters,
-		scorers:             config.scorers,
-		picker:              config.picker,
-		postSchedulePlugins: config.postSchedulePlugins,
-		postResponsePlugins: config.postResponsePlugins,
+		datastore:     datastore,
+		profilePicker: config.profilePicker,
+		profiles:      config.profiles,
 	}
 }
 
 type Scheduler struct {
-	datastore           Datastore
-	preSchedulePlugins  []plugins.PreSchedule
-	filters             []plugins.Filter
-	scorers             []*scorer.WeightedScorer
-	picker              plugins.Picker
-	postSchedulePlugins []plugins.PostSchedule
-	postResponsePlugins []plugins.PostResponse
+	datastore     Datastore
+	profilePicker framework.ProfilePicker
+	profiles      map[string]*framework.SchedulerProfile
 }
 
 type Datastore interface {
@@ -100,7 +93,7 @@ type Datastore interface {
 }
 
 // Schedule finds the target pod based on metrics and the requested lora adapter.
-func (s *Scheduler) Schedule(ctx context.Context, req *types.LLMRequest) (*types.Result, error) {
+func (s *Scheduler) Schedule(ctx context.Context, req *types.LLMRequest) (map[string]*types.Result, error) {
 	logger := log.FromContext(ctx).WithValues("request", req)
 	loggerDebug := logger.V(logutil.DEBUG)
 
@@ -111,104 +104,36 @@ func (s *Scheduler) Schedule(ctx context.Context, req *types.LLMRequest) (*types
 
 	// Snapshot pod metrics from the datastore to:
 	// 1. Reduce concurrent access to the datastore.
-	// 2. Ensure consistent data during the scheduling operation of a request.
+	// 2. Ensure consistent data during the scheduling operation of a request between all scheduling cycles.
 	sCtx := types.NewSchedulingContext(ctx, req, nil, types.ToSchedulerPodMetrics(s.datastore.PodGetAll()))
 	loggerDebug.Info(fmt.Sprintf("Scheduling a request, Metrics: %+v", sCtx.PodsSnapshot))
 
-	s.runPreSchedulePlugins(sCtx)
-
-	pods := s.runFilterPlugins(sCtx)
-	if len(pods) == 0 {
-		return nil, errutil.Error{Code: errutil.Internal, Msg: "no pods available for the given request"}
-	}
-	// if we got here, there is at least one pod to score
-	weightedScorePerPod := s.runScorerPlugins(sCtx, pods)
-
-	result := s.runPickerPlugin(sCtx, weightedScorePerPod)
-
-	s.runPostSchedulePlugins(sCtx, result)
-
-	return result, nil
-}
+	profileExecutionResults := map[string]*types.Result{}
 
-func (s *Scheduler) runPreSchedulePlugins(ctx *types.SchedulingContext) {
-	for _, plugin := range s.preSchedulePlugins {
-		ctx.Logger.V(logutil.DEBUG).Info("Running pre-schedule plugin", "plugin", plugin.Name())
+	for { // get the next set of profiles to run iteratively based on the request and the previous execution results
 		before := time.Now()
-		plugin.PreSchedule(ctx)
-		metrics.RecordSchedulerPluginProcessingLatency(plugins.PreSchedulerPluginType, plugin.Name(), time.Since(before))
-	}
-}
-
-func (s *Scheduler) runFilterPlugins(ctx *types.SchedulingContext) []types.Pod {
-	loggerDebug := ctx.Logger.V(logutil.DEBUG)
-	filteredPods := ctx.PodsSnapshot
-	loggerDebug.Info("Before running filter plugins", "pods", filteredPods)
-
-	for _, filter := range s.filters {
-		loggerDebug.Info("Running filter plugin", "plugin", filter.Name())
-		before := time.Now()
-		filteredPods = filter.Filter(ctx, filteredPods)
-		metrics.RecordSchedulerPluginProcessingLatency(plugins.FilterPluginType, filter.Name(), time.Since(before))
-		loggerDebug.Info("Filter plugin result", "plugin", filter.Name(), "pods", filteredPods)
-		if len(filteredPods) == 0 {
+		profiles := s.profilePicker.Pick(req, s.profiles, profileExecutionResults)
+		metrics.RecordSchedulerPluginProcessingLatency(framework.ProfilePickerType, s.profilePicker.Name(), time.Since(before))
+		if len(profiles) == 0 { // profile picker didn't pick any profile to run
 			break
 		}
-	}
-	loggerDebug.Info("After running filter plugins")
 
-	return filteredPods
-}
-
-func (s *Scheduler) runScorerPlugins(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
-	loggerDebug := ctx.Logger.V(logutil.DEBUG)
-	loggerDebug.Info("Before running scorer plugins", "pods", pods)
+		for name, profile := range profiles {
+			// run the selected profiles and collect results (current code runs all profiles)
+			profileExecutionResult, err := profile.RunCycle(sCtx)
+			if err != nil {
+				return nil, fmt.Errorf("failed to run all required scheduling profiles - %w", err)
+			}
 
-	weightedScorePerPod := make(map[types.Pod]float64, len(pods))
-	for _, pod := range pods {
-		weightedScorePerPod[pod] = float64(0) // initialize weighted score per pod with 0 value
-	}
-	// Iterate through each scorer in the chain and accumulate the weighted scores.
-	for _, weightedScorer := range s.scorers {
-		loggerDebug.Info("Running scorer", "scorer", weightedScorer.Name())
-		before := time.Now()
-		scores := weightedScorer.Score(ctx, pods)
-		metrics.RecordSchedulerPluginProcessingLatency(plugins.ScorerPluginType, weightedScorer.Name(), time.Since(before))
-		for pod, score := range scores { // weight is relative to the sum of weights
-			weightedScorePerPod[pod] += score * float64(weightedScorer.Weight())
+			profileExecutionResults[name] = profileExecutionResult
 		}
-		loggerDebug.Info("After running scorer", "scorer", weightedScorer.Name())
 	}
-	loggerDebug.Info("After running scorer plugins")
-
-	return weightedScorePerPod
-}
 
-func (s *Scheduler) runPickerPlugin(ctx *types.SchedulingContext, weightedScorePerPod map[types.Pod]float64) *types.Result {
-	loggerDebug := ctx.Logger.V(logutil.DEBUG)
-	scoredPods := make([]*types.ScoredPod, len(weightedScorePerPod))
-	i := 0
-	for pod, score := range weightedScorePerPod {
-		scoredPods[i] = &types.ScoredPod{Pod: pod, Score: score}
-		i++
+	if len(profileExecutionResults) == 0 {
+		return nil, fmt.Errorf("failed to run any SchedulingProfile for the request - %s", req)
 	}
 
-	loggerDebug.Info("Before running picker plugin", "pods weighted score", fmt.Sprint(weightedScorePerPod))
-	before := time.Now()
-	result := s.picker.Pick(ctx, scoredPods)
-	metrics.RecordSchedulerPluginProcessingLatency(plugins.PickerPluginType, s.picker.Name(), time.Since(before))
-	loggerDebug.Info("After running picker plugin", "result", result)
-
-	return result
-}
-
-func (s *Scheduler) runPostSchedulePlugins(ctx *types.SchedulingContext, res *types.Result) {
-	for _, plugin := range s.postSchedulePlugins {
-		ctx.Logger.V(logutil.DEBUG).Info("Running post-schedule plugin", "plugin", plugin.Name())
-		before := time.Now()
-		plugin.PostSchedule(ctx, res)
-		metrics.RecordSchedulerPluginProcessingLatency(plugins.PostSchedulePluginType, plugin.Name(), time.Since(before))
-	}
+	return profileExecutionResults, nil
 }
 
 // OnResponse is invoked during the processing of a response from an inference pod. It will invoke
@@ -228,14 +153,19 @@ func (s *Scheduler) OnResponse(ctx context.Context, resp *types.LLMResponse, tar
 
 	sCtx := types.NewSchedulingContext(ctx, nil, resp, pods)
 
-	s.runPostResponsePlugins(sCtx, targetPod)
+	// WORKAROUND until PostResponse is out of Scheduler
+	profileExecutionResults := map[string]*types.Result{}
+	profiles := s.profilePicker.Pick(nil, s.profiles, profileExecutionResults) // all profiles
+	for _, profile := range profiles {
+		s.runPostResponsePlugins(sCtx, targetPod, profile)
+	}
 }
 
-func (s *Scheduler) runPostResponsePlugins(ctx *types.SchedulingContext, targetPod types.Pod) {
-	for _, plugin := range s.postResponsePlugins {
+func (s *Scheduler) runPostResponsePlugins(ctx *types.SchedulingContext, targetPod types.Pod, profile *framework.SchedulerProfile) {
+	for _, plugin := range profile.PostResponsePlugins {
 		ctx.Logger.V(logutil.DEBUG).Info("Running post-response plugin", "plugin", plugin.Name())
 		before := time.Now()
 		plugin.PostResponse(ctx, targetPod)
-		metrics.RecordSchedulerPluginProcessingLatency(plugins.PostResponsePluginType, plugin.Name(), time.Since(before))
+		metrics.RecordSchedulerPluginProcessingLatency(framework.PostResponsePluginType, plugin.Name(), time.Since(before))
 	}
 }
diff --git a/pkg/epp/scheduling/scheduler_config.go b/pkg/epp/scheduling/scheduler_config.go
index 06a23f4695..4008b7002a 100644
--- a/pkg/epp/scheduling/scheduler_config.go
+++ b/pkg/epp/scheduling/scheduler_config.go
@@ -16,108 +16,18 @@ limitations under the License.
 
 package scheduling
 
-import (
-	"fmt"
-
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins/scorer"
-)
+import "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 
 // NewSchedulerConfig creates a new SchedulerConfig object and returns its pointer.
-func NewSchedulerConfig() *SchedulerConfig {
+func NewSchedulerConfig(profilePicker framework.ProfilePicker, profiles map[string]*framework.SchedulerProfile) *SchedulerConfig {
 	return &SchedulerConfig{
-		preSchedulePlugins:  []plugins.PreSchedule{},
-		filters:             []plugins.Filter{},
-		scorers:             []*scorer.WeightedScorer{},
-		postSchedulePlugins: []plugins.PostSchedule{},
-		postResponsePlugins: []plugins.PostResponse{},
-		// picker remains nil since config doesn't support multiple pickers
+		profilePicker: profilePicker,
+		profiles:      profiles,
 	}
 }
 
 // SchedulerConfig provides a configuration for the scheduler which influence routing decisions.
 type SchedulerConfig struct {
-	preSchedulePlugins  []plugins.PreSchedule
-	filters             []plugins.Filter
-	scorers             []*scorer.WeightedScorer
-	picker              plugins.Picker
-	postSchedulePlugins []plugins.PostSchedule
-	postResponsePlugins []plugins.PostResponse
-}
-
-// WithPreSchedulePlugins sets the given plugins as the PreSchedule plugins.
-// If the SchedulerConfig has PreSchedule plugins, this call replaces the existing plugins with the given ones.
-func (c *SchedulerConfig) WithPreSchedulePlugins(plugins ...plugins.PreSchedule) *SchedulerConfig {
-	c.preSchedulePlugins = plugins
-	return c
-}
-
-// WithFilters sets the given filter plugins as the Filter plugins.
-// if the SchedulerConfig has Filter plugins, this call replaces the existing plugins with the given ones.
-func (c *SchedulerConfig) WithFilters(filters ...plugins.Filter) *SchedulerConfig {
-	c.filters = filters
-	return c
-}
-
-// WithScorers sets the given scorer plugins as the Scorer plugins.
-// if the SchedulerConfig has Scorer plugins, this call replaces the existing plugins with the given ones.
-func (c *SchedulerConfig) WithScorers(scorers ...*scorer.WeightedScorer) *SchedulerConfig {
-	c.scorers = scorers
-	return c
-}
-
-// WithPicker sets the given picker plugins as the Picker plugin.
-// if the SchedulerConfig has Picker plugin, this call replaces the existing plugin with the given one.
-func (c *SchedulerConfig) WithPicker(picker plugins.Picker) *SchedulerConfig {
-	c.picker = picker
-	return c
-}
-
-// WithPostSchedulePlugins sets the given plugins as the PostSchedule plugins.
-// If the SchedulerConfig has PostSchedule plugins, this call replaces the existing plugins with the given ones.
-func (c *SchedulerConfig) WithPostSchedulePlugins(plugins ...plugins.PostSchedule) *SchedulerConfig {
-	c.postSchedulePlugins = plugins
-	return c
-}
-
-// WithPostResponsePlugins sets the given plugins as the PostResponse plugins.
-// If the SchedulerConfig has PostResponse plugins, this call replaces the existing plugins with the given ones.
-func (c *SchedulerConfig) WithPostResponsePlugins(plugins ...plugins.PostResponse) *SchedulerConfig {
-	c.postResponsePlugins = plugins
-	return c
-}
-
-// AddPlugins adds the given plugins to all scheduler plugins according to the interfaces each plugin implements.
-// A plugin may implement more than one scheduler plugin interface.
-// Special Case: In order to add a scorer, one must use the scorer.NewWeightedScorer function in order to provide a weight.
-// if a scorer implements more than one interface, supplying a WeightedScorer is sufficient. The function will take the internal
-// scorer object and register it to all interfaces it implements.
-func (c *SchedulerConfig) AddPlugins(pluginObjects ...plugins.Plugin) error {
-	for _, plugin := range pluginObjects {
-		if weightedScorer, ok := plugin.(*scorer.WeightedScorer); ok {
-			c.scorers = append(c.scorers, weightedScorer)
-			plugin = weightedScorer.Scorer // if we got WeightedScorer, unwrap the plugin
-		} else if scorer, ok := plugin.(plugins.Scorer); ok { // if we got a Scorer instead of WeightedScorer that's an error.
-			return fmt.Errorf("failed to register scorer '%s' without a weight. follow function documentation to register a scorer", scorer.Name())
-		}
-		if preSchedulePlugin, ok := plugin.(plugins.PreSchedule); ok {
-			c.preSchedulePlugins = append(c.preSchedulePlugins, preSchedulePlugin)
-		}
-		if filter, ok := plugin.(plugins.Filter); ok {
-			c.filters = append(c.filters, filter)
-		}
-		if picker, ok := plugin.(plugins.Picker); ok {
-			if c.picker != nil {
-				return fmt.Errorf("failed to set '%s' as picker, already have a registered picker plugin '%s'", picker.Name(), c.picker.Name())
-			}
-			c.picker = picker
-		}
-		if postSchedulePlugin, ok := plugin.(plugins.PostSchedule); ok {
-			c.postSchedulePlugins = append(c.postSchedulePlugins, postSchedulePlugin)
-		}
-		if postResponsePlugin, ok := plugin.(plugins.PostResponse); ok {
-			c.postResponsePlugins = append(c.postResponsePlugins, postResponsePlugin)
-		}
-	}
-	return nil
+	profilePicker framework.ProfilePicker
+	profiles      map[string]*framework.SchedulerProfile
 }
diff --git a/pkg/epp/scheduling/scheduler_test.go b/pkg/epp/scheduling/scheduler_test.go
index 1c6e6495c0..bb3647277b 100644
--- a/pkg/epp/scheduling/scheduler_test.go
+++ b/pkg/epp/scheduling/scheduler_test.go
@@ -25,8 +25,8 @@ import (
 	k8stypes "k8s.io/apimachinery/pkg/types"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend"
 	backendmetrics "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend/metrics" // Import config for thresholds
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/plugins/scorer"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
+	profilepicker "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile-picker"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
@@ -36,7 +36,7 @@ func TestSchedule(t *testing.T) {
 		name    string
 		req     *types.LLMRequest
 		input   []*backendmetrics.FakePodMetrics
-		wantRes *types.Result
+		wantRes map[string]*types.Result
 		err     bool
 	}{
 		{
@@ -46,8 +46,9 @@ func TestSchedule(t *testing.T) {
 				RequestId:   uuid.NewString(),
 				Critical:    true,
 			},
-			input: []*backendmetrics.FakePodMetrics{},
-			err:   true,
+			input:   []*backendmetrics.FakePodMetrics{},
+			wantRes: nil,
+			err:     true,
 		},
 		{
 			name: "critical request",
@@ -95,19 +96,21 @@ func TestSchedule(t *testing.T) {
 					},
 				},
 			},
-			wantRes: &types.Result{
-				TargetPod: &types.ScoredPod{
-					Pod: &types.PodMetrics{
-						Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}, Labels: make(map[string]string)},
-						MetricsState: &backendmetrics.MetricsState{
-							WaitingQueueSize:    3,
-							KVCacheUsagePercent: 0.1,
-							MaxActiveModels:     2,
-							ActiveModels: map[string]int{
-								"foo":      1,
-								"critical": 1,
+			wantRes: map[string]*types.Result{
+				"default": {
+					TargetPod: &types.ScoredPod{
+						Pod: &types.PodMetrics{
+							Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}, Labels: make(map[string]string)},
+							MetricsState: &backendmetrics.MetricsState{
+								WaitingQueueSize:    3,
+								KVCacheUsagePercent: 0.1,
+								MaxActiveModels:     2,
+								ActiveModels: map[string]int{
+									"foo":      1,
+									"critical": 1,
+								},
+								WaitingModels: map[string]int{},
 							},
-							WaitingModels: map[string]int{},
 						},
 					},
 				},
@@ -158,19 +161,21 @@ func TestSchedule(t *testing.T) {
 					},
 				},
 			},
-			wantRes: &types.Result{
-				TargetPod: &types.ScoredPod{
-					Pod: &types.PodMetrics{
-						Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}, Labels: make(map[string]string)},
-						MetricsState: &backendmetrics.MetricsState{
-							WaitingQueueSize:    0,
-							KVCacheUsagePercent: 0.2,
-							MaxActiveModels:     2,
-							ActiveModels: map[string]int{
-								"foo": 1,
-								"bar": 1,
+			wantRes: map[string]*types.Result{
+				"default": {
+					TargetPod: &types.ScoredPod{
+						Pod: &types.PodMetrics{
+							Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}, Labels: make(map[string]string)},
+							MetricsState: &backendmetrics.MetricsState{
+								WaitingQueueSize:    0,
+								KVCacheUsagePercent: 0.2,
+								MaxActiveModels:     2,
+								ActiveModels: map[string]int{
+									"foo": 1,
+									"bar": 1,
+								},
+								WaitingModels: map[string]int{},
 							},
-							WaitingModels: map[string]int{},
 						},
 					},
 				},
@@ -242,192 +247,6 @@ func TestSchedule(t *testing.T) {
 	}
 }
 
-func TestSchedulePlugins(t *testing.T) {
-	tp1 := &TestPlugin{
-		NameRes:   "test1",
-		ScoreRes:  0.3,
-		FilterRes: []k8stypes.NamespacedName{{Name: "pod1"}, {Name: "pod2"}, {Name: "pod3"}},
-	}
-	tp2 := &TestPlugin{
-		NameRes:   "test2",
-		ScoreRes:  0.8,
-		FilterRes: []k8stypes.NamespacedName{{Name: "pod1"}, {Name: "pod2"}},
-	}
-	tp_filterAll := &TestPlugin{
-		NameRes:   "filter all",
-		FilterRes: []k8stypes.NamespacedName{},
-	}
-	pickerPlugin := &TestPlugin{
-		NameRes: "picker",
-		PickRes: k8stypes.NamespacedName{Name: "pod1"},
-	}
-
-	tests := []struct {
-		name           string
-		config         SchedulerConfig
-		input          []*backendmetrics.FakePodMetrics
-		wantTargetPod  k8stypes.NamespacedName
-		targetPodScore float64
-		// Number of expected pods to score (after filter)
-		numPodsToScore int
-		err            bool
-	}{
-		{
-			name: "all plugins executed successfully, all scorers with same weight",
-			config: SchedulerConfig{
-				preSchedulePlugins: []plugins.PreSchedule{tp1, tp2},
-				filters:            []plugins.Filter{tp1, tp2},
-				scorers: []*scorer.WeightedScorer{
-					scorer.NewWeightedScorer(tp1, 1),
-					scorer.NewWeightedScorer(tp2, 1),
-				},
-				picker:              pickerPlugin,
-				postSchedulePlugins: []plugins.PostSchedule{tp1, tp2},
-			},
-			input: []*backendmetrics.FakePodMetrics{
-				{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}}},
-				{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}}},
-				{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod3"}}},
-			},
-			wantTargetPod:  k8stypes.NamespacedName{Name: "pod1"},
-			targetPodScore: 1.1,
-			numPodsToScore: 2,
-			err:            false,
-		},
-		{
-			name: "all plugins executed successfully, different scorers weights",
-			config: SchedulerConfig{
-				preSchedulePlugins: []plugins.PreSchedule{tp1, tp2},
-				filters:            []plugins.Filter{tp1, tp2},
-				scorers: []*scorer.WeightedScorer{
-					scorer.NewWeightedScorer(tp1, 60),
-					scorer.NewWeightedScorer(tp2, 40),
-				},
-				picker:              pickerPlugin,
-				postSchedulePlugins: []plugins.PostSchedule{tp1, tp2},
-			},
-			input: []*backendmetrics.FakePodMetrics{
-				{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}}},
-				{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}}},
-				{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod3"}}},
-			},
-			wantTargetPod:  k8stypes.NamespacedName{Name: "pod1"},
-			targetPodScore: 50,
-			numPodsToScore: 2,
-			err:            false,
-		},
-		{
-			name: "filter all",
-			config: SchedulerConfig{
-				preSchedulePlugins: []plugins.PreSchedule{tp1, tp2},
-				filters:            []plugins.Filter{tp1, tp_filterAll},
-				scorers: []*scorer.WeightedScorer{
-					scorer.NewWeightedScorer(tp1, 1),
-					scorer.NewWeightedScorer(tp2, 1),
-				},
-				picker:              pickerPlugin,
-				postSchedulePlugins: []plugins.PostSchedule{tp1, tp2},
-			},
-			input: []*backendmetrics.FakePodMetrics{
-				{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}}},
-				{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}}},
-				{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod3"}}},
-			},
-			numPodsToScore: 0,
-			err:            true, // no available pods to server after filter all
-		},
-	}
-
-	for _, test := range tests {
-		t.Run(test.name, func(t *testing.T) {
-			// Reset all plugins before each new test case.
-			for _, plugin := range test.config.preSchedulePlugins {
-				plugin.(*TestPlugin).reset()
-			}
-			for _, plugin := range test.config.filters {
-				plugin.(*TestPlugin).reset()
-			}
-			for _, plugin := range test.config.scorers {
-				plugin.Scorer.(*TestPlugin).reset()
-			}
-			test.config.picker.(*TestPlugin).reset()
-			for _, plugin := range test.config.postSchedulePlugins {
-				plugin.(*TestPlugin).reset()
-			}
-
-			// Initialize the scheduler
-			scheduler := NewSchedulerWithConfig(&fakeDataStore{pods: test.input}, &test.config)
-
-			req := &types.LLMRequest{
-				TargetModel: "test-model",
-				RequestId:   uuid.NewString(),
-			}
-			got, err := scheduler.Schedule(context.Background(), req)
-
-			// Validate error state
-			if test.err != (err != nil) {
-				t.Fatalf("Unexpected error, got %v, want %v", err, test.err)
-			}
-
-			if err != nil {
-				return
-			}
-
-			// Validate output
-			wantPod := &types.PodMetrics{
-				Pod: &backend.Pod{NamespacedName: test.wantTargetPod, Labels: make(map[string]string)},
-			}
-			wantRes := &types.Result{TargetPod: wantPod}
-			if diff := cmp.Diff(wantRes, got); diff != "" {
-				t.Errorf("Unexpected output (-want +got): %v", diff)
-			}
-
-			// Validate plugin execution counts dynamically
-			for _, plugin := range test.config.preSchedulePlugins {
-				tp, _ := plugin.(*TestPlugin)
-				if tp.PreScheduleCallCount != 1 {
-					t.Errorf("Plugin %s PreSchedule() called %d times, expected 1", plugin.Name(), tp.PreScheduleCallCount)
-				}
-			}
-
-			for _, plugin := range test.config.filters {
-				tp, _ := plugin.(*TestPlugin)
-				if tp.FilterCallCount != 1 {
-					t.Errorf("Plugin %s Filter() called %d times, expected 1", plugin.Name(), tp.FilterCallCount)
-				}
-			}
-
-			for _, plugin := range test.config.scorers {
-				tp, _ := plugin.Scorer.(*TestPlugin)
-				if tp.ScoreCallCount != 1 {
-					t.Errorf("Plugin %s Score() called %d times, expected 1", plugin.Name(), tp.ScoreCallCount)
-				}
-				if test.numPodsToScore != tp.NumOfScoredPods {
-					t.Errorf("Plugin %s Score() called with %d pods, expected %d", plugin.Name(), tp.NumOfScoredPods, test.numPodsToScore)
-				}
-			}
-
-			tp, _ := test.config.picker.(*TestPlugin)
-			if tp.NumOfPickerCandidates != test.numPodsToScore {
-				t.Errorf("Picker plugin %s Pick() called with %d candidates, expected %d", tp.Name(), tp.NumOfPickerCandidates, tp.NumOfScoredPods)
-			}
-			if tp.PickCallCount != 1 {
-				t.Errorf("Picker plugin %s Pick() called %d times, expected 1", tp.Name(), tp.PickCallCount)
-			}
-			if tp.WinnderPodScore != test.targetPodScore {
-				t.Errorf("winnder pod score %v, expected %v", tp.WinnderPodScore, test.targetPodScore)
-			}
-
-			for _, plugin := range test.config.postSchedulePlugins {
-				tp, _ := plugin.(*TestPlugin)
-				if tp.PostScheduleCallCount != 1 {
-					t.Errorf("Plugin %s PostSchedule() called %d times, expected 1", plugin.Name(), tp.PostScheduleCallCount)
-				}
-			}
-		})
-	}
-}
-
 func TestPostResponse(t *testing.T) {
 	pr1 := &testPostResponse{
 		NameRes:                 "pr1",
@@ -439,15 +258,15 @@ func TestPostResponse(t *testing.T) {
 
 	tests := []struct {
 		name               string
-		config             SchedulerConfig
+		config             *framework.SchedulerProfile
 		input              []*backendmetrics.FakePodMetrics
 		responseHeaders    map[string]string
 		wantUpdatedHeaders map[string]string
 	}{
 		{
 			name: "Simple postResponse test",
-			config: SchedulerConfig{
-				postResponsePlugins: []plugins.PostResponse{pr1},
+			config: &framework.SchedulerProfile{
+				PostResponsePlugins: []framework.PostResponse{pr1},
 			},
 			input: []*backendmetrics.FakePodMetrics{
 				{Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}}},
@@ -459,7 +278,8 @@ func TestPostResponse(t *testing.T) {
 	}
 
 	for _, test := range tests {
-		scheduler := NewSchedulerWithConfig(&fakeDataStore{pods: test.input}, &test.config)
+		schedulerConfig := NewSchedulerConfig(profilepicker.NewAllProfilesPicker(), map[string]*framework.SchedulerProfile{"default": test.config})
+		scheduler := NewSchedulerWithConfig(&fakeDataStore{pods: test.input}, schedulerConfig)
 
 		headers := map[string]string{}
 		for k, v := range test.responseHeaders {
@@ -493,66 +313,6 @@ func (fds *fakeDataStore) PodGetAll() []backendmetrics.PodMetrics {
 	return pm
 }
 
-// TestPlugin is an implementation useful in unit tests.
-type TestPlugin struct {
-	NameRes               string
-	ScoreCallCount        int
-	NumOfScoredPods       int
-	ScoreRes              float64
-	FilterCallCount       int
-	FilterRes             []k8stypes.NamespacedName
-	PreScheduleCallCount  int
-	PostScheduleCallCount int
-	PickCallCount         int
-	NumOfPickerCandidates int
-	PickRes               k8stypes.NamespacedName
-	WinnderPodScore       float64
-}
-
-func (tp *TestPlugin) Name() string { return tp.NameRes }
-
-func (tp *TestPlugin) PreSchedule(ctx *types.SchedulingContext) {
-	tp.PreScheduleCallCount++
-}
-
-func (tp *TestPlugin) Filter(ctx *types.SchedulingContext, pods []types.Pod) []types.Pod {
-	tp.FilterCallCount++
-	return findPods(ctx, tp.FilterRes...)
-
-}
-
-func (tp *TestPlugin) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
-	tp.ScoreCallCount++
-	scoredPods := make(map[types.Pod]float64, len(pods))
-	for _, pod := range pods {
-		scoredPods[pod] += tp.ScoreRes
-	}
-	tp.NumOfScoredPods = len(scoredPods)
-	return scoredPods
-}
-
-func (tp *TestPlugin) Pick(ctx *types.SchedulingContext, scoredPods []*types.ScoredPod) *types.Result {
-	tp.PickCallCount++
-	tp.NumOfPickerCandidates = len(scoredPods)
-	pod := findPods(ctx, tp.PickRes)[0]
-	tp.WinnderPodScore = getPodScore(scoredPods, pod)
-	return &types.Result{TargetPod: pod}
-}
-
-func (tp *TestPlugin) PostSchedule(ctx *types.SchedulingContext, res *types.Result) {
-	tp.PostScheduleCallCount++
-}
-
-func (tp *TestPlugin) reset() {
-	tp.PreScheduleCallCount = 0
-	tp.FilterCallCount = 0
-	tp.ScoreCallCount = 0
-	tp.NumOfScoredPods = 0
-	tp.PostScheduleCallCount = 0
-	tp.PickCallCount = 0
-	tp.NumOfPickerCandidates = 0
-}
-
 type testPostResponse struct {
 	NameRes                 string
 	ReceivedResponseHeaders map[string]string
@@ -569,26 +329,3 @@ func (pr *testPostResponse) PostResponse(ctx *types.SchedulingContext, pod types
 		ctx.Resp.Headers[key] = value
 	}
 }
-
-func findPods(ctx *types.SchedulingContext, names ...k8stypes.NamespacedName) []types.Pod {
-	res := []types.Pod{}
-	for _, pod := range ctx.PodsSnapshot {
-		for _, name := range names {
-			if pod.GetPod().NamespacedName.String() == name.String() {
-				res = append(res, pod)
-			}
-		}
-	}
-	return res
-}
-
-func getPodScore(scoredPods []*types.ScoredPod, selectedPod types.Pod) float64 {
-	finalScore := 0.0
-	for _, scoredPod := range scoredPods {
-		if scoredPod.GetPod().NamespacedName.String() == selectedPod.GetPod().NamespacedName.String() {
-			finalScore = scoredPod.Score
-			break
-		}
-	}
-	return finalScore
-}
diff --git a/test/integration/epp/hermetic_test.go b/test/integration/epp/hermetic_test.go
index a9d54fa4e0..f570f93076 100644
--- a/test/integration/epp/hermetic_test.go
+++ b/test/integration/epp/hermetic_test.go
@@ -475,7 +475,7 @@ func TestFullDuplexStreamed_KubeInferenceModelRequest(t *testing.T) {
 							Status: &envoyTypePb.HttpStatus{
 								Code: envoyTypePb.StatusCode_TooManyRequests,
 							},
-							Body: []byte("inference gateway: InferencePoolResourceExhausted - failed to find target pod: inference gateway: Internal - no pods available for the given request"),
+							Body: []byte("inference gateway: InferencePoolResourceExhausted - failed to find target pod: failed to run all required scheduling profiles - inference gateway: Internal - no pods available for the given request"),
 						},
 					},
 				},


Reviews:
user: login: kfswain
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-22T20:08:07Z
user: login: nirrozenbaum
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T20:25:34Z
user: login: kfswain
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-22T20:29:04Z
user: login: nirrozenbaum
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T20:32:27Z
user: login: nirrozenbaum
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T20:36:41Z
user: login: kfswain
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-23T04:24:08Z
user: login: kfswain
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-23T04:25:38Z
user: login: nirrozenbaum
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T04:53:34Z

Review comments:
diff_hunk: @@ -108,23 +108,27 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 }
 
 // Dispatch runs one or many scheduling cycles.
-func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) ([]*schedulingtypes.Result, error) {
+func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) (map[string]*schedulingtypes.Result, error) {
 	var err error
 	res, err := d.scheduler.Schedule(ctx, llmReq)
 	if err != nil {
 		return nil, errutil.Error{Code: errutil.InferencePoolResourceExhausted, Msg: fmt.Errorf("failed to find target pod: %w", err).Error()}
 	}
 
-	return []*schedulingtypes.Result{res}, nil
+	return res, nil // TODO handle multi cycle result after defining the PostDispatch extension point
 }
 
-func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results []*schedulingtypes.Result) (*handlers.RequestContext, error) {
+func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results map[string]*schedulingtypes.Result) (*handlers.RequestContext, error) {
 	logger := log.FromContext(ctx)
 	// currently only get a single result. Will refactor to pluggably implement the PostSchedule
 	if len(results) == 0 {
 		return reqCtx, errutil.Error{Code: errutil.Internal, Msg: "results must be greater than zero"}
 	}
-	targetPod := results[0].TargetPod.GetPod()
+	var targetPod *backend.Pod
+	// TODO should handle multi cycle results, this should be pluggable logic
+	for _, result := range results {
path: pkg/epp/requestcontrol/director.go
user: login: kfswain
body: this will just set the final targetPod. Can we instead just index on the key & get the result that way? We can leave the todo to indicate this is a transitory state.
created_at: 2025-05-22T19:43:00Z
author_association: COLLABORATOR
diff_hunk: @@ -108,23 +108,27 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 }
 
 // Dispatch runs one or many scheduling cycles.
-func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) ([]*schedulingtypes.Result, error) {
+func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) (map[string]*schedulingtypes.Result, error) {
 	var err error
 	res, err := d.scheduler.Schedule(ctx, llmReq)
 	if err != nil {
 		return nil, errutil.Error{Code: errutil.InferencePoolResourceExhausted, Msg: fmt.Errorf("failed to find target pod: %w", err).Error()}
 	}
 
-	return []*schedulingtypes.Result{res}, nil
+	return res, nil // TODO handle multi cycle result after defining the PostDispatch extension point
 }
 
-func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results []*schedulingtypes.Result) (*handlers.RequestContext, error) {
+func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results map[string]*schedulingtypes.Result) (*handlers.RequestContext, error) {
 	logger := log.FromContext(ctx)
 	// currently only get a single result. Will refactor to pluggably implement the PostSchedule
 	if len(results) == 0 {
 		return reqCtx, errutil.Error{Code: errutil.Internal, Msg: "results must be greater than zero"}
 	}
-	targetPod := results[0].TargetPod.GetPod()
+	var targetPod *backend.Pod
+	// TODO should handle multi cycle results, this should be pluggable logic
+	for _, result := range results {
path: pkg/epp/requestcontrol/director.go
user: login: nirrozenbaum
body: you mean doing this?
```go
targetPod = results[key].TargetPod.GetPod()
```
instead of:
```go
targetPod = result.TargetPod.GetPod()
```

not sure I got the intention. we have here a map from profile-name -> result. 
this is a transitionary stage where only one profile is used, so only one result. 
but since it's a map I must use the range loop.
created_at: 2025-05-22T20:25:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,23 +108,27 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 }
 
 // Dispatch runs one or many scheduling cycles.
-func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) ([]*schedulingtypes.Result, error) {
+func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) (map[string]*schedulingtypes.Result, error) {
 	var err error
 	res, err := d.scheduler.Schedule(ctx, llmReq)
 	if err != nil {
 		return nil, errutil.Error{Code: errutil.InferencePoolResourceExhausted, Msg: fmt.Errorf("failed to find target pod: %w", err).Error()}
 	}
 
-	return []*schedulingtypes.Result{res}, nil
+	return res, nil // TODO handle multi cycle result after defining the PostDispatch extension point
 }
 
-func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results []*schedulingtypes.Result) (*handlers.RequestContext, error) {
+func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results map[string]*schedulingtypes.Result) (*handlers.RequestContext, error) {
 	logger := log.FromContext(ctx)
 	// currently only get a single result. Will refactor to pluggably implement the PostSchedule
 	if len(results) == 0 {
 		return reqCtx, errutil.Error{Code: errutil.Internal, Msg: "results must be greater than zero"}
 	}
-	targetPod := results[0].TargetPod.GetPod()
+	var targetPod *backend.Pod
+	// TODO should handle multi cycle results, this should be pluggable logic
+	for _, result := range results {
path: pkg/epp/requestcontrol/director.go
user: login: kfswain
body: Yeah, just do `targetPod = results[key].TargetPod.GetPod()` and drop the for loop.

Since we only end up with one result as is even if there are multiple profiles, it reads more obvious if we just use the key, I think. 
created_at: 2025-05-22T20:29:04Z
author_association: COLLABORATOR
diff_hunk: @@ -108,23 +108,27 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 }
 
 // Dispatch runs one or many scheduling cycles.
-func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) ([]*schedulingtypes.Result, error) {
+func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) (map[string]*schedulingtypes.Result, error) {
 	var err error
 	res, err := d.scheduler.Schedule(ctx, llmReq)
 	if err != nil {
 		return nil, errutil.Error{Code: errutil.InferencePoolResourceExhausted, Msg: fmt.Errorf("failed to find target pod: %w", err).Error()}
 	}
 
-	return []*schedulingtypes.Result{res}, nil
+	return res, nil // TODO handle multi cycle result after defining the PostDispatch extension point
 }
 
-func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results []*schedulingtypes.Result) (*handlers.RequestContext, error) {
+func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results map[string]*schedulingtypes.Result) (*handlers.RequestContext, error) {
 	logger := log.FromContext(ctx)
 	// currently only get a single result. Will refactor to pluggably implement the PostSchedule
 	if len(results) == 0 {
 		return reqCtx, errutil.Error{Code: errutil.Internal, Msg: "results must be greater than zero"}
 	}
-	targetPod := results[0].TargetPod.GetPod()
+	var targetPod *backend.Pod
+	// TODO should handle multi cycle results, this should be pluggable logic
+	for _, result := range results {
path: pkg/epp/requestcontrol/director.go
user: login: nirrozenbaum
body: so now the next question, what is the value of `key` in this line 
```go
targetPod = results[key].TargetPod.GetPod()?
```
which `key` value to use?
if we used default configuration it's just "default", but if we used schedulerv2, it's a different profile.
since we don't know the key, I'm using the loop...
created_at: 2025-05-22T20:32:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,23 +108,27 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 }
 
 // Dispatch runs one or many scheduling cycles.
-func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) ([]*schedulingtypes.Result, error) {
+func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) (map[string]*schedulingtypes.Result, error) {
 	var err error
 	res, err := d.scheduler.Schedule(ctx, llmReq)
 	if err != nil {
 		return nil, errutil.Error{Code: errutil.InferencePoolResourceExhausted, Msg: fmt.Errorf("failed to find target pod: %w", err).Error()}
 	}
 
-	return []*schedulingtypes.Result{res}, nil
+	return res, nil // TODO handle multi cycle result after defining the PostDispatch extension point
 }
 
-func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results []*schedulingtypes.Result) (*handlers.RequestContext, error) {
+func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results map[string]*schedulingtypes.Result) (*handlers.RequestContext, error) {
 	logger := log.FromContext(ctx)
 	// currently only get a single result. Will refactor to pluggably implement the PostSchedule
 	if len(results) == 0 {
 		return reqCtx, errutil.Error{Code: errutil.Internal, Msg: "results must be greater than zero"}
 	}
-	targetPod := results[0].TargetPod.GetPod()
+	var targetPod *backend.Pod
+	// TODO should handle multi cycle results, this should be pluggable logic
+	for _, result := range results {
path: pkg/epp/requestcontrol/director.go
user: login: nirrozenbaum
body: I expect this to be solved as soon as we implement the extension point
created_at: 2025-05-22T20:36:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,23 +108,27 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 }
 
 // Dispatch runs one or many scheduling cycles.
-func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) ([]*schedulingtypes.Result, error) {
+func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) (map[string]*schedulingtypes.Result, error) {
 	var err error
 	res, err := d.scheduler.Schedule(ctx, llmReq)
 	if err != nil {
 		return nil, errutil.Error{Code: errutil.InferencePoolResourceExhausted, Msg: fmt.Errorf("failed to find target pod: %w", err).Error()}
 	}
 
-	return []*schedulingtypes.Result{res}, nil
+	return res, nil // TODO handle multi cycle result after defining the PostDispatch extension point
 }
 
-func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results []*schedulingtypes.Result) (*handlers.RequestContext, error) {
+func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results map[string]*schedulingtypes.Result) (*handlers.RequestContext, error) {
 	logger := log.FromContext(ctx)
 	// currently only get a single result. Will refactor to pluggably implement the PostSchedule
 	if len(results) == 0 {
 		return reqCtx, errutil.Error{Code: errutil.Internal, Msg: "results must be greater than zero"}
 	}
-	targetPod := results[0].TargetPod.GetPod()
+	var targetPod *backend.Pod
+	// TODO should handle multi cycle results, this should be pluggable logic
+	for _, result := range results {
path: pkg/epp/requestcontrol/director.go
user: login: kfswain
body: The only one we set is `schedulerv2` right? 
line 215 in the main file of this PR
created_at: 2025-05-23T04:24:08Z
author_association: COLLABORATOR
diff_hunk: @@ -108,23 +108,27 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 }
 
 // Dispatch runs one or many scheduling cycles.
-func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) ([]*schedulingtypes.Result, error) {
+func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) (map[string]*schedulingtypes.Result, error) {
 	var err error
 	res, err := d.scheduler.Schedule(ctx, llmReq)
 	if err != nil {
 		return nil, errutil.Error{Code: errutil.InferencePoolResourceExhausted, Msg: fmt.Errorf("failed to find target pod: %w", err).Error()}
 	}
 
-	return []*schedulingtypes.Result{res}, nil
+	return res, nil // TODO handle multi cycle result after defining the PostDispatch extension point
 }
 
-func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results []*schedulingtypes.Result) (*handlers.RequestContext, error) {
+func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results map[string]*schedulingtypes.Result) (*handlers.RequestContext, error) {
 	logger := log.FromContext(ctx)
 	// currently only get a single result. Will refactor to pluggably implement the PostSchedule
 	if len(results) == 0 {
 		return reqCtx, errutil.Error{Code: errutil.Internal, Msg: "results must be greater than zero"}
 	}
-	targetPod := results[0].TargetPod.GetPod()
+	var targetPod *backend.Pod
+	// TODO should handle multi cycle results, this should be pluggable logic
+	for _, result := range results {
path: pkg/epp/requestcontrol/director.go
user: login: kfswain
body: I agree that this may be a nit, but the for loop would look out of place since the iteration isn't quite being used 
created_at: 2025-05-23T04:25:38Z
author_association: COLLABORATOR
diff_hunk: @@ -108,23 +108,27 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 }
 
 // Dispatch runs one or many scheduling cycles.
-func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) ([]*schedulingtypes.Result, error) {
+func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) (map[string]*schedulingtypes.Result, error) {
 	var err error
 	res, err := d.scheduler.Schedule(ctx, llmReq)
 	if err != nil {
 		return nil, errutil.Error{Code: errutil.InferencePoolResourceExhausted, Msg: fmt.Errorf("failed to find target pod: %w", err).Error()}
 	}
 
-	return []*schedulingtypes.Result{res}, nil
+	return res, nil // TODO handle multi cycle result after defining the PostDispatch extension point
 }
 
-func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results []*schedulingtypes.Result) (*handlers.RequestContext, error) {
+func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results map[string]*schedulingtypes.Result) (*handlers.RequestContext, error) {
 	logger := log.FromContext(ctx)
 	// currently only get a single result. Will refactor to pluggably implement the PostSchedule
 	if len(results) == 0 {
 		return reqCtx, errutil.Error{Code: errutil.Internal, Msg: "results must be greater than zero"}
 	}
-	targetPod := results[0].TargetPod.GetPod()
+	var targetPod *backend.Pod
+	// TODO should handle multi cycle results, this should be pluggable logic
+	for _, result := range results {
path: pkg/epp/requestcontrol/director.go
user: login: nirrozenbaum
body: if we used the env var to enable "schedulerv2" than the key is "schedulerv2".
if not and we use default configuration, the key is "default" (in NewScheduler func):
https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/epp/scheduling/scheduler.go#L73

this is why I didn't put here a const.
created_at: 2025-05-23T04:53:34Z
author_association: CONTRIBUTOR

PR comments:
user: login: netlify[bot]
created_at: 2025-05-22T06:47:25Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *gateway-api-inference-extension* ready!


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | d658430e3b290e2aa9282799e8158aaffeda46f8 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/gateway-api-inference-extension/deploys/682ee20a9f28620008adec17 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-862--gateway-api-inference-extension.netlify.app](https://deploy-preview-862--gateway-api-inference-extension.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br />![QR Code](https://app.netlify.com/qr-code/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1cmwiOiJodHRwczovL2RlcGxveS1wcmV2aWV3LTg2Mi0tZ2F0ZXdheS1hcGktaW5mZXJlbmNlLWV4dGVuc2lvbi5uZXRsaWZ5LmFwcCJ9.bFpKQU-eE7wgMUmOPp2-jiIbpXNfcJvyZKOyFZg5v-4)<br /><br />_Use your smartphone camera to open QR code link._</details> |
---
<!-- [gateway-api-inference-extension Preview](https://deploy-preview-862--gateway-api-inference-extension.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/gateway-api-inference-extension/configuration/notifications#deploy-notifications)._
author_association: NONE
user: login: nirrozenbaum
created_at: 2025-05-22T06:48:01Z
body: cc @ahg-g @kfswain
author_association: CONTRIBUTOR
user: login: kfswain
created_at: 2025-05-22T12:03:04Z
body: /hold

I was tempted to close this. But I will hold for now. Scheduler refactors should not move forward until: https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/845/files is merged
author_association: COLLABORATOR
user: login: kfswain
created_at: 2025-05-22T20:09:37Z
body: mostly I think this is fine, a lot of this will be replaced as we most closely align with: https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/845

small comment to keep the director clean for now.
author_association: COLLABORATOR
user: login: nirrozenbaum
created_at: 2025-05-22T20:27:59Z
body: > mostly I think this is fine, a lot of this will be replaced as we most closely align with: #845
> 
> small comment to keep the director clean for now.

yes, agreed. 
this PR that takes a step towards the end goal, but is subject to change based on the discussion 
(hopefully it would be mostly interface changes and adaptations).
author_association: CONTRIBUTOR
user: login: kfswain
created_at: 2025-05-23T04:26:20Z
body: /lgtm
/approve
/hold

Holding submission for author to choose to address comments
author_association: COLLABORATOR
user: login: k8s-ci-robot
created_at: 2025-05-23T04:26:27Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/862#issuecomment-2903211330" title="Approved">kfswain</a>*, *<a href="https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/862#" title="Author self-approved">nirrozenbaum</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fgateway-api-inference-extension).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/OWNERS)~~ [kfswain]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: nirrozenbaum
created_at: 2025-05-23T04:54:43Z
body: unholding to avoid conflicts with other PRs since this PR touches 30 files.
we can iterate over the last nit in a tightly scoped follow up PR.

/unhold
author_association: CONTRIBUTOR