Repository: kubernetes/autoscaler (Go)

Issue:
Cluster autoscaler to support custom schedulers' Filter logic
<!--
Thanks for taking the time to raise a feature request! Please answer these questions as best you can before submitting.
-->

**Which component are you using?**: 
Cluster autoscaler (CA)

**Is your feature request designed to solve a problem? If so describe the problem this feature should solve.**:

When a Pod is being scheduled with a custom scheduler (i.e. non default-scheduler) that uses the Filter API, it can be marked as Unschedulable. When CA polls for unschedulable pods, it uses a different Filter API (I believe the non default-scheduler) to determine whether it can be scheduled on existing nodes ([from this block](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/core/filteroutschedulable/filter_out_schedulable.go#L49-L63)). This difference in logic can cause a pod to be stuck in Pending given that differing logic is used to determine its placement.  

**Describe the solution you'd like.**:

To allow CA to be extended to use the same Filter API as the scheduler used for the Pod that is currently in Unscheduled state, so that it can scale up the node that is required.

**Describe any alternative solutions you've considered.**:
N/A

**Additional context.**:

I am doing testing with the placement-policy scheduler plugin for this.

- https://github.com/Azure/placement-policy-scheduler-plugins/

I am also open to any other workaround to allow CA to play nicely with other schedulers, of which I am probably unaware of.

Thanks for taking the time to read this!



Discussion: The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue or PR as fresh with `/remove-lifecycle stale`
- Mark this issue or PR as rotten with `/lifecycle rotten`
- Close this issue or PR with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle stale`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle rotten`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle rotten

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Reopen this issue with `/reopen`
- Mark this issue as fresh with `/remove-lifecycle rotten`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/close not-planned

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/
@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

<details>

In response to [this](https://github.com/kubernetes/autoscaler/issues/5106#issuecomment-1517939692):

>The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
>
>This bot triages issues according to the following rules:
>- After 90d of inactivity, `lifecycle/stale` is applied
>- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
>- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed
>
>You can:
>- Reopen this issue with `/reopen`
>- Mark this issue as fresh with `/remove-lifecycle rotten`
>- Offer to help out with [Issue Triage][1]
>
>Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).
>
>/close not-planned
>
>[1]: https://www.kubernetes.dev/docs/guide/issue-triage/


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
There are 2 options here:
1. Import your own scheduler code into CA, re-compile and use the new CA image
2. Convert your filter plugin into an extender ([design proposal](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md), [official docs reference](https://kubernetes.io/docs/reference/config-api/kube-scheduler-config.v1/#kubescheduler-config-k8s-io-v1-Extender), [discussion around extenders in #sig-scheduling](https://kubernetes.slack.com/archives/C09TP78DV/p1681325345961669))
/reopen
@vadasambar: You can't reopen an issue/PR unless you authored it or you are a collaborator.

<details>

In response to [this](https://github.com/kubernetes/autoscaler/issues/5106#issuecomment-1521227510):

>/reopen


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
/remove-lifecycle rotten
Looks like I can't re-open the issue. @jdheyburn you might have to re-open the issue. 
/remove-lifecycle rotten
/reopen
@jdheyburn: Reopened this issue.

<details>

In response to [this](https://github.com/kubernetes/autoscaler/issues/5106#issuecomment-1521535896):

>/reopen


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
@vadasambar Thanks for picking this up, although I'm not in a position to test it out, I'll look to have a play around with it once it is released



Fix (patch):
diff --git a/cluster-autoscaler/config/autoscaling_options.go b/cluster-autoscaler/config/autoscaling_options.go
index 59092452369f..ae1bc09b673e 100644
--- a/cluster-autoscaler/config/autoscaling_options.go
+++ b/cluster-autoscaler/config/autoscaling_options.go
@@ -18,6 +18,8 @@ package config
 
 import (
 	"time"
+
+	scheduler_config "k8s.io/kubernetes/pkg/scheduler/apis/config"
 )
 
 // GpuLimits define lower and upper bound on GPU instances of given type in cluster
@@ -166,6 +168,9 @@ type AutoscalingOptions struct {
 	// ScaleDownSimulationTimeout defines the maximum time that can be
 	// spent on scale down simulation.
 	ScaleDownSimulationTimeout time.Duration
+	// SchedulerConfig allows changing configuration of in-tree
+	// scheduler plugins acting on PreFilter and Filter extension points
+	SchedulerConfig *scheduler_config.KubeSchedulerConfiguration
 	// NodeDeletionDelayTimeout is maximum time CA waits for removing delay-deletion.cluster-autoscaler.kubernetes.io/ annotations before deleting the node.
 	NodeDeletionDelayTimeout time.Duration
 	// WriteStatusConfigMap tells if the status information should be written to a ConfigMap
diff --git a/cluster-autoscaler/config/const.go b/cluster-autoscaler/config/const.go
index b87c7638ebcb..1025ac9059d7 100644
--- a/cluster-autoscaler/config/const.go
+++ b/cluster-autoscaler/config/const.go
@@ -19,6 +19,10 @@ package config
 import "time"
 
 const (
+	// SchedulerConfigFileFlag is the name of the flag
+	// for passing in custom scheduler config for in-tree scheduelr plugins
+	SchedulerConfigFileFlag = "scheduler-config-file"
+
 	// DefaultMaxClusterCores is the default maximum number of cores in the cluster.
 	DefaultMaxClusterCores = 5000 * 64
 	// DefaultMaxClusterMemory is the default maximum number of gigabytes of memory in cluster.
diff --git a/cluster-autoscaler/main.go b/cluster-autoscaler/main.go
index 274908d4f6b0..225b94743c0e 100644
--- a/cluster-autoscaler/main.go
+++ b/cluster-autoscaler/main.go
@@ -56,6 +56,7 @@ import (
 	"k8s.io/autoscaler/cluster-autoscaler/simulator/clustersnapshot"
 	"k8s.io/autoscaler/cluster-autoscaler/utils/errors"
 	kube_util "k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes"
+	scheduler_util "k8s.io/autoscaler/cluster-autoscaler/utils/scheduler"
 	"k8s.io/autoscaler/cluster-autoscaler/utils/units"
 	"k8s.io/autoscaler/cluster-autoscaler/version"
 	kube_client "k8s.io/client-go/kubernetes"
@@ -68,6 +69,7 @@ import (
 	"k8s.io/component-base/config/options"
 	"k8s.io/component-base/metrics/legacyregistry"
 	"k8s.io/klog/v2"
+	scheduler_config "k8s.io/kubernetes/pkg/scheduler/apis/config"
 )
 
 // MultiStringFlag is a flag for passing multiple parameters using same flag
@@ -133,6 +135,7 @@ var (
 			"for scale down when some candidates from previous iteration are no longer valid."+
 			"When calculating the pool size for additional candidates we take"+
 			"max(#nodes * scale-down-candidates-pool-ratio, scale-down-candidates-pool-min-count).")
+	schedulerConfigFile         = flag.String(config.SchedulerConfigFileFlag, "", "scheduler-config allows changing configuration of in-tree scheduler plugins acting on PreFilter and Filter extension points")
 	nodeDeletionDelayTimeout    = flag.Duration("node-deletion-delay-timeout", 2*time.Minute, "Maximum time CA waits for removing delay-deletion.cluster-autoscaler.kubernetes.io/ annotations before deleting the node.")
 	nodeDeletionBatcherInterval = flag.Duration("node-deletion-batcher-interval", 0*time.Second, "How long CA ScaleDown gather nodes to delete them in batch.")
 	scanInterval                = flag.Duration("scan-interval", 10*time.Second, "How often cluster is reevaluated for scale up or down")
@@ -274,6 +277,15 @@ func createAutoscalingOptions() config.AutoscalingOptions {
 		*maxEmptyBulkDeleteFlag = *maxScaleDownParallelismFlag
 	}
 
+	var parsedSchedConfig *scheduler_config.KubeSchedulerConfiguration
+	// if scheduler config flag was set by the user
+	if pflag.CommandLine.Changed(config.SchedulerConfigFileFlag) {
+		parsedSchedConfig, err = scheduler_util.ConfigFromPath(*schedulerConfigFile)
+	}
+	if err != nil {
+		klog.Fatalf("Failed to get scheduler config: %v", err)
+	}
+
 	return config.AutoscalingOptions{
 		NodeGroupDefaults: config.NodeGroupAutoscalingOptions{
 			ScaleDownUtilizationThreshold:    *scaleDownUtilizationThreshold,
@@ -316,6 +328,7 @@ func createAutoscalingOptions() config.AutoscalingOptions {
 		ScaleDownNonEmptyCandidatesCount: *scaleDownNonEmptyCandidatesCount,
 		ScaleDownCandidatesPoolRatio:     *scaleDownCandidatesPoolRatio,
 		ScaleDownCandidatesPoolMinCount:  *scaleDownCandidatesPoolMinCount,
+		SchedulerConfig:                  parsedSchedConfig,
 		WriteStatusConfigMap:             *writeStatusConfigMapFlag,
 		StatusConfigMapName:              *statusConfigMapName,
 		BalanceSimilarNodeGroups:         *balanceSimilarNodeGroupsFlag,
@@ -422,7 +435,8 @@ func buildAutoscaler(debuggingSnapshotter debuggingsnapshot.DebuggingSnapshotter
 
 	eventsKubeClient := createKubeClient(getKubeConfig())
 
-	predicateChecker, err := predicatechecker.NewSchedulerBasedPredicateChecker(kubeClient, make(chan struct{}))
+	predicateChecker, err := predicatechecker.NewSchedulerBasedPredicateChecker(kubeClient,
+		autoscalingOptions.SchedulerConfig, make(chan struct{}))
 	if err != nil {
 		return nil, err
 	}
diff --git a/cluster-autoscaler/simulator/predicatechecker/schedulerbased.go b/cluster-autoscaler/simulator/predicatechecker/schedulerbased.go
index fde1d2758a78..9b7075323c14 100644
--- a/cluster-autoscaler/simulator/predicatechecker/schedulerbased.go
+++ b/cluster-autoscaler/simulator/predicatechecker/schedulerbased.go
@@ -27,6 +27,7 @@ import (
 	kube_client "k8s.io/client-go/kubernetes"
 	v1listers "k8s.io/client-go/listers/core/v1"
 	klog "k8s.io/klog/v2"
+	"k8s.io/kubernetes/pkg/scheduler/apis/config"
 	scheduler_config "k8s.io/kubernetes/pkg/scheduler/apis/config/latest"
 	schedulerframework "k8s.io/kubernetes/pkg/scheduler/framework"
 	scheduler_plugins "k8s.io/kubernetes/pkg/scheduler/framework/plugins"
@@ -44,21 +45,26 @@ type SchedulerBasedPredicateChecker struct {
 }
 
 // NewSchedulerBasedPredicateChecker builds scheduler based PredicateChecker.
-func NewSchedulerBasedPredicateChecker(kubeClient kube_client.Interface, stop <-chan struct{}) (*SchedulerBasedPredicateChecker, error) {
+func NewSchedulerBasedPredicateChecker(kubeClient kube_client.Interface, schedConfig *config.KubeSchedulerConfiguration, stop <-chan struct{}) (*SchedulerBasedPredicateChecker, error) {
 	informerFactory := informers.NewSharedInformerFactory(kubeClient, 0)
-	config, err := scheduler_config.Default()
-	if err != nil {
-		return nil, fmt.Errorf("couldn't create scheduler config: %v", err)
+
+	if schedConfig == nil {
+		var err error
+		schedConfig, err = scheduler_config.Default()
+		if err != nil {
+			return nil, fmt.Errorf("couldn't create scheduler config: %v", err)
+		}
 	}
-	if len(config.Profiles) != 1 || config.Profiles[0].SchedulerName != apiv1.DefaultSchedulerName {
-		return nil, fmt.Errorf("unexpected scheduler config: expected default scheduler profile only (found %d profiles)", len(config.Profiles))
+
+	if len(schedConfig.Profiles) != 1 {
+		return nil, fmt.Errorf("unexpected scheduler config: expected one scheduler profile only (found %d profiles)", len(schedConfig.Profiles))
 	}
 	sharedLister := NewDelegatingSchedulerSharedLister()
 
 	framework, err := schedulerframeworkruntime.NewFramework(
 		context.TODO(),
 		scheduler_plugins.NewInTreeRegistry(),
-		&config.Profiles[0],
+		&schedConfig.Profiles[0],
 		schedulerframeworkruntime.WithInformerFactory(informerFactory),
 		schedulerframeworkruntime.WithSnapshotSharedLister(sharedLister),
 	)
@@ -181,6 +187,7 @@ func (p *SchedulerBasedPredicateChecker) CheckPredicates(clusterSnapshot cluster
 			filterReasons,
 			p.buildDebugInfo(filterName, nodeInfo))
 	}
+
 	return nil
 }
 
diff --git a/cluster-autoscaler/utils/scheduler/scheduler.go b/cluster-autoscaler/utils/scheduler/scheduler.go
index 1b9e61259e2d..59d008db64d8 100644
--- a/cluster-autoscaler/utils/scheduler/scheduler.go
+++ b/cluster-autoscaler/utils/scheduler/scheduler.go
@@ -18,14 +18,25 @@ package scheduler
 
 import (
 	"fmt"
+	"os"
 	"strings"
 
 	apiv1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
 	"k8s.io/apimachinery/pkg/util/uuid"
+	scheduler_config "k8s.io/kubernetes/pkg/scheduler/apis/config"
+	scheduler_scheme "k8s.io/kubernetes/pkg/scheduler/apis/config/scheme"
+	scheduler_validation "k8s.io/kubernetes/pkg/scheduler/apis/config/validation"
 	schedulerframework "k8s.io/kubernetes/pkg/scheduler/framework"
 )
 
+const (
+	schedulerConfigDecodeErr   = "couldn't decode scheduler config"
+	schedulerConfigLoadErr     = "couldn't load scheduler config"
+	schedulerConfigTypeCastErr = "couldn't assert type as KubeSchedulerConfiguration"
+	schedulerConfigInvalidErr  = "invalid KubeSchedulerConfiguration"
+)
+
 // CreateNodeNameToInfoMap obtains a list of pods and pivots that list into a map where the keys are node names
 // and the values are the aggregated information for that node. Pods waiting lower priority pods preemption
 // (pod.Status.NominatedNodeName is set) are also added to list of pods for a node.
@@ -106,3 +117,33 @@ func ResourceToResourceList(r *schedulerframework.Resource) apiv1.ResourceList {
 	}
 	return result
 }
+
+// ConfigFromPath loads scheduler config from a path.
+// TODO(vadasambar): replace code to parse scheduler config with upstream function
+// once https://github.com/kubernetes/kubernetes/pull/119057 is merged
+func ConfigFromPath(path string) (*scheduler_config.KubeSchedulerConfiguration, error) {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return nil, fmt.Errorf("%s: %v", schedulerConfigLoadErr, err)
+	}
+
+	obj, gvk, err := scheduler_scheme.Codecs.UniversalDecoder().Decode(data, nil, nil)
+	if err != nil {
+		return nil, fmt.Errorf("%s: %v", schedulerConfigDecodeErr, err)
+	}
+
+	cfgObj, ok := obj.(*scheduler_config.KubeSchedulerConfiguration)
+	if !ok {
+		return nil, fmt.Errorf("%s, gvk: %s", schedulerConfigTypeCastErr, gvk)
+	}
+
+	// this needs to be set explicitly because config's api version is empty after decoding
+	// check kubernetes/cmd/kube-scheduler/app/options/configfile.go for more info
+	cfgObj.TypeMeta.APIVersion = gvk.GroupVersion().String()
+
+	if err := scheduler_validation.ValidateKubeSchedulerConfiguration(cfgObj); err != nil {
+		return nil, fmt.Errorf("%s: %v", schedulerConfigInvalidErr, err)
+	}
+
+	return cfgObj, nil
+}