Repository: kubernetes-sigs/kube-scheduler-simulator
Pull request #393 (2024-11-12 04:52:26)

feat: support the wasm extension in the debuggable scheduler
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?

<!--
Add related area tag(s):
/area web
/area simulator
/area scenario

Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

/area simulator
/kind feature

#### What this PR does / why we need it:

This PR adds [kube-scheduler-wasm-extension](https://github.com/kubernetes-sigs/kube-scheduler-wasm-extension) support into the simulator via the debuggable scheduler. This allows easy testing of WASM plugins by simply adding the required entries to the scheduler config.

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->

#### Special notes for your reviewer:


/label tide/merge-method-squash


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/kube-scheduler-simulator/pull/393#" title="Author self-approved">jgiannuzzi</a>*
**Once this PR has been reviewed and has the lgtm label**, please assign [sanposhiho](https://github.com/sanposhiho) for approval. For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkube-scheduler-simulator).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/kube-scheduler-simulator/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["sanposhiho"]} -->
k8s-ci-robot: Hi @jgiannuzzi. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkube-scheduler-simulator).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

jgiannuzzi: /cc @sanposhiho
k8s-ci-robot: @jgiannuzzi: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kube-scheduler-simulator-backend-lint | a77a3c7da294fc9f94cf189fb76b7871c246b695 | [link](https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kube-scheduler-simulator/393/pull-kube-scheduler-simulator-backend-lint/1856386851661680640) | true | `/test pull-kube-scheduler-simulator-backend-lint`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kube-scheduler-simulator&pr=393). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ajgiannuzzi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kube-scheduler-simulator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @jgiannuzzi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kube-scheduler-simulator-backend-lint | a77a3c7da294fc9f94cf189fb76b7871c246b695 | [link](https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kube-scheduler-simulator/393/pull-kube-scheduler-simulator-backend-lint/1856386851661680640) | true | `/test pull-kube-scheduler-simulator-backend-lint`
pull-kube-scheduler-simulator-backend-unit-test | a77a3c7da294fc9f94cf189fb76b7871c246b695 | [link](https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kube-scheduler-simulator/393/pull-kube-scheduler-simulator-backend-unit-test/1856386851762343936) | true | `/test pull-kube-scheduler-simulator-backend-unit-test`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kube-scheduler-simulator&pr=393). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ajgiannuzzi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kube-scheduler-simulator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jgiannuzzi: Thanks for your guidance @sanposhiho. The code is now yet even simpler and feels more right 😄 
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/kube-scheduler-simulator/pull/393#" title="Author self-approved">jgiannuzzi</a>*, *<a href="https://github.com/kubernetes-sigs/kube-scheduler-simulator/pull/393#pullrequestreview-2431009548" title="Approved">sanposhiho</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkube-scheduler-simulator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/kube-scheduler-simulator/blob/master/OWNERS)~~ [sanposhiho]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->

Patch:
diff --git a/simulator/go.mod b/simulator/go.mod
index bd62f8e7..06c4fe4b 100644
--- a/simulator/go.mod
+++ b/simulator/go.mod
@@ -43,6 +43,7 @@ require (
 	github.com/spf13/cobra v1.7.0
 	github.com/stretchr/testify v1.9.0
 	go.etcd.io/etcd/client/v3 v3.5.10
+	golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e
 	golang.org/x/sync v0.8.0
 	golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2
 	gopkg.in/yaml.v2 v2.4.0
@@ -56,6 +57,7 @@ require (
 	k8s.io/kube-scheduler v0.30.4
 	k8s.io/kubernetes v1.30.4
 	k8s.io/utils v0.0.0-20230726121419-3b25d923346b
+	sigs.k8s.io/kube-scheduler-wasm-extension/scheduler v0.0.0-20241111155045-2ce7072312c8
 )
 
 require (
@@ -121,6 +123,7 @@ require (
 	github.com/prometheus/procfs v0.10.1 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
 	github.com/stoewer/go-strcase v1.2.0 // indirect
+	github.com/tetratelabs/wazero v1.7.2 // indirect
 	github.com/valyala/bytebufferpool v1.0.0 // indirect
 	github.com/valyala/fasttemplate v1.2.1 // indirect
 	go.etcd.io/etcd/api/v3 v3.5.10 // indirect
@@ -138,7 +141,6 @@ require (
 	go.uber.org/multierr v1.11.0 // indirect
 	go.uber.org/zap v1.26.0 // indirect
 	golang.org/x/crypto v0.26.0 // indirect
-	golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e // indirect
 	golang.org/x/mod v0.17.0 // indirect
 	golang.org/x/net v0.28.0 // indirect
 	golang.org/x/oauth2 v0.22.0 // indirect
diff --git a/simulator/go.sum b/simulator/go.sum
index 9dcb15bb..c7c086fa 100644
--- a/simulator/go.sum
+++ b/simulator/go.sum
@@ -220,6 +220,8 @@ github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO
 github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
 github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
 github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+github.com/tetratelabs/wazero v1.7.2 h1:1+z5nXJNwMLPAWaTePFi49SSTL0IMx/i3Fg8Yc25GDc=
+github.com/tetratelabs/wazero v1.7.2/go.mod h1:ytl6Zuh20R/eROuyDaGPkp82O9C/DJfXAwJfQ3X6/7Y=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 h1:6fotK7otjonDflCTK0BCfls4SPy3NcCVb5dqqmbRknE=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75/go.mod h1:KO6IkyS8Y3j8OdNO85qEYBsRPuteD+YciPomcXdrMnk=
 github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
@@ -430,6 +432,8 @@ k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7F
 k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340/go.mod h1:yD4MZYeKMBwQKVht279WycxKyM84kkAx2DPrTXaeb98=
 k8s.io/kube-scheduler v0.30.4 h1:6H+NfNuJ4RvUdSD1dtnJBXuYGAp+ETGZaxFk4sLGPdU=
 k8s.io/kube-scheduler v0.30.4/go.mod h1:7MMICziKySYQQ7rOFGBsSkn1PE3urPwsxlp6uwU+NhU=
+k8s.io/kubectl v0.30.4 h1:jU+6WSvcSgzs07H624RA2L79FnqppyV6BKI+dHWDzyo=
+k8s.io/kubectl v0.30.4/go.mod h1:4KnGCshO4fFxd/tncWcbKH3Nj9wtoFYwMYPj8CUnduE=
 k8s.io/kubelet v0.30.4 h1:2TP59RVxuWuKpD58gQ6qow1Oy2Ys2uOH4hfSD/qv5EQ=
 k8s.io/kubelet v0.30.4/go.mod h1:v0lRl+1y2NNId5OlFiJ1rhjXc9D8Tp7PqvQYJS7W/L0=
 k8s.io/kubernetes v1.30.4 h1:LfWX7JNmT9Hp8uFVHsB9gQCZesjcWTQ02PHwMz6dGqk=
@@ -442,6 +446,8 @@ sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.29.0 h1:/U5vjBbQn3RCh
 sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.29.0/go.mod h1:z7+wmGM2dfIiLRfrC6jb5kV2Mq/sK1ZP303cxzkV5Y4=
 sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo=
 sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0=
+sigs.k8s.io/kube-scheduler-wasm-extension/scheduler v0.0.0-20241111155045-2ce7072312c8 h1:jOyiY8zbvXtuMb7ImqNcSFtVOmnXhWQ8pxhECz95PKk=
+sigs.k8s.io/kube-scheduler-wasm-extension/scheduler v0.0.0-20241111155045-2ce7072312c8/go.mod h1:N0BrfQ58AMkBbmjeGupvhUv/FFErpuYYs8D1YPPyY0Q=
 sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4=
 sigs.k8s.io/structured-merge-diff/v4 v4.4.1/go.mod h1:N8hJocpFajUSSeSJ9bOZ77VzejKZaXsTtZo4/u7Io08=
 sigs.k8s.io/yaml v1.3.0 h1:a2VclLzOGrwOHDiV8EfBGhvjHvP46CtW5j6POvhYGGo=
diff --git a/simulator/pkg/debuggablescheduler/debuggable_scheduler.go b/simulator/pkg/debuggablescheduler/debuggable_scheduler.go
index 2343fbee..433a007f 100644
--- a/simulator/pkg/debuggablescheduler/debuggable_scheduler.go
+++ b/simulator/pkg/debuggablescheduler/debuggable_scheduler.go
@@ -57,6 +57,12 @@ func NewConfigs() (Configs, error) {
 		return Configs{}, xerrors.Errorf("load scheduler config: %w", err)
 	}
 
+	// Register wasm plugins to the wasm registry.
+	// This _needs_ to happen before the scheduler configuration is converted.
+	if err := simulatorschedulerconfig.RegisterWasmPlugins(versionedcfg); err != nil {
+		return Configs{}, xerrors.Errorf("register wasm plugins: %w", err)
+	}
+
 	versioned, err := scheduler.ConvertConfigurationForSimulator(versionedcfg)
 	if err != nil {
 		return Configs{}, xerrors.Errorf("convert scheduler config to apply: %w", err)
diff --git a/simulator/scheduler/config/wasm.go b/simulator/scheduler/config/wasm.go
new file mode 100644
index 00000000..8ab97257
--- /dev/null
+++ b/simulator/scheduler/config/wasm.go
@@ -0,0 +1,58 @@
+package config
+
+import (
+	"golang.org/x/xerrors"
+	"k8s.io/apimachinery/pkg/util/sets"
+	configv1 "k8s.io/kube-scheduler/config/v1"
+	"k8s.io/kubernetes/pkg/scheduler/apis/config"
+	"k8s.io/kubernetes/pkg/scheduler/apis/config/scheme"
+	"k8s.io/kubernetes/pkg/scheduler/framework/runtime"
+	wasm "sigs.k8s.io/kube-scheduler-wasm-extension/scheduler/plugin"
+)
+
+// RegisterWasmPlugins registers wasm plugins from the given configuration.
+func RegisterWasmPlugins(versionedCfg *configv1.KubeSchedulerConfiguration) error {
+	cfg := config.KubeSchedulerConfiguration{}
+	if err := scheme.Scheme.Convert(versionedCfg, &cfg, nil); err != nil {
+		return xerrors.Errorf("convert configuration: %w", err)
+	}
+
+	registry, err := getWasmRegistryFromUnversionedConfig(&cfg)
+	if err != nil {
+		return err
+	}
+
+	SetOutOfTreeRegistries(registry)
+
+	return nil
+}
+
+// getWasmRegistryFromUnversionedConfig registers wasm plugins from the given unversioned configuration.
+func getWasmRegistryFromUnversionedConfig(cfg *config.KubeSchedulerConfiguration) (runtime.Registry, error) {
+	registry := runtime.Registry{}
+
+	for _, profile := range cfg.Profiles {
+		wasmplugins := sets.New[string]()
+		// look for the wasm plugin in the plugin config.
+		for _, config := range profile.PluginConfig {
+			if err := runtime.DecodeInto(config.Args, &wasm.PluginConfig{}); err != nil {
+				// not wasm plugin.
+				continue
+			}
+
+			wasmplugins.Insert(config.Name)
+		}
+
+		// look for the wasm plugin in the enabled plugins.
+		// (assuming that the wasm plugin is specified as a multi-point plugin.)
+		for _, plugin := range profile.Plugins.MultiPoint.Enabled {
+			if wasmplugins.Has(plugin.Name) {
+				if err := registry.Register(plugin.Name, wasm.PluginFactory(plugin.Name)); err != nil {
+					return nil, xerrors.Errorf("register plugin %s: %w", plugin.Name, err)
+				}
+			}
+		}
+	}
+
+	return registry, nil
+}


Test patch:
diff --git a/simulator/scheduler/config/wasm_test.go b/simulator/scheduler/config/wasm_test.go
new file mode 100644
index 00000000..8a2fd01e
--- /dev/null
+++ b/simulator/scheduler/config/wasm_test.go
@@ -0,0 +1,121 @@
+package config
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/kubernetes/pkg/scheduler/apis/config"
+)
+
+func TestGetWasmRegistryFromUnversionedConfig(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name     string
+		cfg      *config.KubeSchedulerConfiguration
+		expected int
+	}{
+		{
+			name:     "no profiles",
+			cfg:      &config.KubeSchedulerConfiguration{},
+			expected: 0,
+		},
+		{
+			name: "no wasm plugins",
+			cfg: &config.KubeSchedulerConfiguration{
+				Profiles: []config.KubeSchedulerProfile{
+					{
+						PluginConfig: []config.PluginConfig{
+							{
+								Name: "DefaultPreemption",
+								Args: &config.DefaultPreemptionArgs{},
+							},
+						},
+						Plugins: &config.Plugins{
+							MultiPoint: config.PluginSet{
+								Enabled: []config.Plugin{
+									{Name: "DefaultPreemption"},
+								},
+							},
+						},
+					},
+				},
+			},
+			expected: 0,
+		},
+		{
+			name: "one wasm plugin",
+			cfg: &config.KubeSchedulerConfiguration{
+				Profiles: []config.KubeSchedulerProfile{
+					{
+						PluginConfig: []config.PluginConfig{
+							{
+								Name: "DefaultPreemption",
+								Args: &config.DefaultPreemptionArgs{},
+							},
+							{Name: "wasmPlugin", Args: &runtime.Unknown{
+								ContentType: runtime.ContentTypeJSON,
+								Raw:         []byte(`{"guestURL":"http://example.com/plugin.wasm"}`),
+							}},
+						},
+						Plugins: &config.Plugins{
+							MultiPoint: config.PluginSet{
+								Enabled: []config.Plugin{
+									{Name: "DefaultPreemption"},
+									{Name: "wasmPlugin"},
+								},
+							},
+						},
+					},
+				},
+			},
+			expected: 1,
+		},
+		{
+			name: "multiple wasm plugins",
+			cfg: &config.KubeSchedulerConfiguration{
+				Profiles: []config.KubeSchedulerProfile{
+					{
+						PluginConfig: []config.PluginConfig{
+							{
+								Name: "DefaultPreemption",
+								Args: &config.DefaultPreemptionArgs{},
+							},
+							{Name: "wasmPlugin1", Args: &runtime.Unknown{
+								ContentType: runtime.ContentTypeJSON,
+								Raw:         []byte(`{"guestURL":"http://example.com/plugin1.wasm"}`),
+							}},
+							{Name: "wasmPlugin2", Args: &runtime.Unknown{
+								ContentType: runtime.ContentTypeJSON,
+								Raw:         []byte(`{"guestURL":"http://example.com/plugin2.wasm"}`),
+							}},
+						},
+						Plugins: &config.Plugins{
+							MultiPoint: config.PluginSet{
+								Enabled: []config.Plugin{
+									{Name: "DefaultPreemption"},
+									{Name: "wasmPlugin1"},
+									{Name: "wasmPlugin2"},
+								},
+							},
+						},
+					},
+				},
+			},
+			expected: 2,
+		},
+	}
+
+	for _, tt := range tests {
+		tt := tt
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			registry, err := getWasmRegistryFromUnversionedConfig(tt.cfg)
+			require.NoError(t, err, "check error")
+			if len(registry) != tt.expected {
+				t.Errorf("expected %d plugins, got %d", tt.expected, len(registry))
+			}
+		})
+	}
+}