Repository: koordinator-sh/koordinator

Issue:
[BUG] cpuPolicy: coreSched not working
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!-->

**What happened**:
Following the steps in https://koordinator.sh/zh-Hans/docs/user-manuals/cpu-qos#%E4%BD%BF%E7%94%A8cpu-qos%E5%9F%BA%E4%BA%8Ecore-scheduling

- Patched the coreSched configmap to Koordinator.
- Deployed the ls-pod-demo.yaml
- Executed `curl localhost:9316/metrics | grep koordlet_container_core_sched_cookie | grep ls-pod-demo` got nothing. 
- Ran ` curl localhost:9316/metrics | grep ls-pod-demo`, got 
```
koordlet_container_scaled_cfs_burst_us{container_id="docker://94a84566354f46fd31f0824d1a414205bca85df9ac25cea1815d09689e8b6ae6",container_name="nginx",node="minikube",pod_name="ls-pod-demo",pod_namespace="default"}
```
logs:
```shell
docker@minikube:~$ docker ps  | grep koord | awk '{ print $1 }' | xargs -I{} docker logs {} 2>&1  | grep -i 'CoreSched'  -C 5
I0904 07:10:09.564027  256325 callback_runner.go:76] states informer callback runtime-hooks-reconciler has registered for type RegisterTypeAllPods
I0904 07:10:09.564034  256325 callback_runner.go:76] states informer callback host-app-reconciler has registered for type RegisterTypeNodeSLOSpec
I0904 07:10:09.564036  256325 runtimehooks.go:177] start register plugins for runtime hook
I0904 07:10:09.564039  256325 runtimehooks.go:183] runtime hook plugin CoreSched enable false
I0904 07:10:09.564042  256325 runtimehooks.go:183] runtime hook plugin TerwayQoS enable false
I0904 07:10:09.564049  256325 bvt.go:54] register hook GroupIdentity
I0904 07:10:09.564064  256325 hooks.go:52] hook GroupIdentity is registered
I0904 07:10:09.564108  256325 bvt.go:78] update system supported info to false for plugin GroupIdentity, supported msg file not exist
I0904 07:10:09.564113  256325 rule.go:59] new rule GroupIdentity has registered
```

```shell
docker@minikube:~$ docker ps  | grep koord | awk '{ print $1 }' | xargs -I{} docker logs {} 2>&1  | grep -i 'register hook'  -C 5
I0904 07:10:09.564027  256325 callback_runner.go:76] states informer callback runtime-hooks-reconciler has registered for type RegisterTypeAllPods
I0904 07:10:09.564034  256325 callback_runner.go:76] states informer callback host-app-reconciler has registered for type RegisterTypeNodeSLOSpec
I0904 07:10:09.564036  256325 runtimehooks.go:177] start register plugins for runtime hook
I0904 07:10:09.564039  256325 runtimehooks.go:183] runtime hook plugin CoreSched enable false
I0904 07:10:09.564042  256325 runtimehooks.go:183] runtime hook plugin TerwayQoS enable false
I0904 07:10:09.564049  256325 bvt.go:54] register hook GroupIdentity
I0904 07:10:09.564064  256325 hooks.go:52] hook GroupIdentity is registered
I0904 07:10:09.564108  256325 bvt.go:78] update system supported info to false for plugin GroupIdentity, supported msg file not exist
I0904 07:10:09.564113  256325 rule.go:59] new rule GroupIdentity has registered
I0904 07:10:09.564121  256325 reconciler.go:210] register reconcile function reconcile pod level cpu bvt value finished, info: level=pod, resourceType=cpu.bvt_warp_ns, filter=none, conditions=[]
I0904 07:10:09.564131  256325 reconciler.go:210] register reconcile function reconcile kubeqos level cpu bvt value finished, info: level=kubeqos, resourceType=cpu.bvt_warp_ns, filter=none, conditions=[]
I0904 07:10:09.564134  256325 runtimehooks.go:183] runtime hook plugin GroupIdentity enable true
I0904 07:10:09.564139  256325 cpuset.go:55] register hook CPUSetAllocator
I0904 07:10:09.564145  256325 hooks.go:52] hook CPUSetAllocator is registered
I0904 07:10:09.564147  256325 hooks.go:52] hook CPUSetAllocator is registered
I0904 07:10:09.564148  256325 hooks.go:52] hook CPUSetAllocator is registered
I0904 07:10:09.564151  256325 rule.go:59] new rule CPUSetAllocator has registered
I0904 07:10:09.564154  256325 reconciler.go:210] register reconcile function set container cpuset and unset container cpu quota if needed for cpuset pod finished, info: level=container, resourceType=cpuset.cpus, filter=podQOS, conditions=[LSE LSR]
--
I0904 07:10:09.564179  256325 reconciler.go:210] register reconcile function unset pod cpu quota if needed for cpuset pod finished, info: level=pod, resourceType=cpu.cfs_quota_us, filter=podQOS, conditions=[LSE LSR]
I0904 07:10:09.564183  256325 reconciler.go:170] register reconcile function set container cpuset for cpushare pod is specified finished, info: level=container, resourceType=cpuset.cpus, add conditions=[LS BE SYSTEM ]
I0904 07:10:09.564189  256325 reconciler.go:170] register reconcile function set sandbox container cpuset for cpushare pod is specified finished, info: level=sandbox, resourceType=cpuset.cpus, add conditions=[LS BE SYSTEM ]
I0904 07:10:09.564192  256325 runtimehooks.go:183] runtime hook plugin CPUSetAllocator enable true
I0904 07:10:09.564194  256325 runtimehooks.go:183] runtime hook plugin GPUEnvInject enable false
I0904 07:10:09.564197  256325 batch_resource.go:55] register hook BatchResource
I0904 07:10:09.564202  256325 rule.go:59] new rule BatchResource (nodeSLO) has registered
I0904 07:10:09.564205  256325 rule.go:59] new rule BatchResource (nodeMeta) has registered
I0904 07:10:09.564206  256325 hooks.go:52] hook BatchResource is registered
I0904 07:10:09.564210  256325 hooks.go:52] hook BatchResource is registered
I0904 07:10:09.564212  256325 hooks.go:52] hook BatchResource is registered
```

**What you expected to happen**:
Core Scheduling works and `curl localhost:9316/metrics | grep koordlet_container_core_sched_cookie | grep ls-pod-demo` show core_sched_cookie.

**How to reproduce it (as minimally and precisely as possible)**:
1. Install minikube, and start with flags --driver=docker --force
2. Install koordinator by helm: `helm install koordinator koordinator-sh/koordinator --version 1.6.0` (Version 1.5.0 was also tested; the logs provided above are from the 1.5.0 installation )
3. `kubectl patch cm -n koordinator-system slo-controller-config --patch "$(cat configmap.yaml)"` from https://koordinator.sh/zh-Hans/docs/user-manuals/cpu-qos#%E4%BD%BF%E7%94%A8cpu-qos%E5%9F%BA%E4%BA%8Ecore-scheduling
4. `kubectl apply -f ls-pod-demo.yaml`
5. `curl localhost:9316/metrics | grep koordlet_container_core_sched_cookie | grep ls-pod-demo`

**Anything else we need to know?**:

**Environment**:
- App version:1.6.0
- Kubernetes version (use `kubectl version`): Client Version: v1.33.1 Kustomize Version: v5.6.0 Server Version: v1.33.1
- Install details (e.g. helm install args):  helm install koordinator koordinator-sh/koordinator --version 1.6 --set scheduler.log.level=15 --set koordlet.log.level=15 --set manager.log.level=15
- Node environment (for koordlet/runtime-proxy issue):
  - Containerd/Docker version:  28.3.3
  - OS version: Ubuntu Server 24.04.3 LTS
  - Kernal version: 6.8.0-78-generic
  - Cgroup driver: ative.cgroupdriver=cgroupfs
- Others: **cgroup v1, with (systemd.unified_cgroup_hierarchy=0)**



Discussion:
@saintube: @hwenwur The prerequisite of this feature is that the Linux kernel supports Core Scheduling. See [this](https://koordinator.sh/zh-Hans/docs/user-manuals/cpu-qos/#%E5%89%8D%E6%8F%90%E6%9D%A1%E4%BB%B6). We've tested the support on the [Anolis OS](https://github.com/alibaba/cloud-kernel), but I'm afraid Ubuntu 24.04 may not support it or use different interfaces. Could you show us the detailed log related to the core sched or sysctl? It should indicate why the core sched feature does not work.@hwenwur: @saintube 
After some investigation, I've identified the root cause: the func `IsCoreSchedSupported() (bool, string)` function incorrectly returns false when running on a mainline kernel.
Unlike Anolis OS, mainline kernels have Core Scheduling always enabled and do not provide an interface for dynamic enable/disable. I tested modifying the code to skip this support check, and Core Scheduling worked perfectly.
Would it be feasible to adjust the logic to skip the support check specifically when a mainline kernel is detected?

https://github.com/koordinator-sh/koordinator/blob/98d1bd9b3f15d8a2d6cdf021c8cc2ecdc02973ed/pkg/koordlet/util/system/core_sched.go#L277-L295@saintube: > [@saintube](https://github.com/saintube) After some investigation, I've identified the root cause: the func `IsCoreSchedSupported() (bool, string)` function incorrectly returns false when running on a mainline kernel. Unlike Anolis OS, mainline kernels have Core Scheduling always enabled and do not provide an interface for dynamic enable/disable. I tested modifying the code to skip this support check, and Core Scheduling worked perfectly. Would it be feasible to adjust the logic to skip the support check specifically when a mainline kernel is detected?
> 
> [koordinator/pkg/koordlet/util/system/core_sched.go](https://github.com/koordinator-sh/koordinator/blob/98d1bd9b3f15d8a2d6cdf021c8cc2ecdc02973ed/pkg/koordlet/util/system/core_sched.go#L277-L295)
> 
> Lines 277 to 295 in [98d1bd9](/koordinator-sh/koordinator/commit/98d1bd9b3f15d8a2d6cdf021c8cc2ecdc02973ed)
> 
>  func IsCoreSchedSupported() (bool, string) { 
>  	// kernel supports if: 
>  	// a) sysctl has sched_core, 
>  	// b) sched_features has SCHED_CORE/NO_SCHED_CORE 
>  	if IsCoreSchedSysctlSupported() { 
>  		return true, "sysctl supported" 
>  	} 
>   
>  	isSchedFeaturesSuppported, msg := SchedFeatures.IsSupported("") 
>  	if !isSchedFeaturesSuppported { // sched_features unavailable 
>  		return false, msg 
>  	} 
>  	_, err := IsCoreSchedFeatureEnabled() 
>  	if err == nil { 
>  		return true, "sched_features supported" 
>  	} 
>   
>  	return false, "not supported neither by sysctl nor by sched_features" 
>  }

Yes, we should support a mainline kernel. To avoid unnecessary file writes and err logs, I suppose we should keep the support check but adopt the stable interface for the upstream. Welcome to contribute~@hwenwur: Hi, I'm trying to create a PR but have encountered an issue. The mainline kernel doesn't have either /proc/sys/kernel/sched_core or /sys/kernel/debug/sched_features, yet it still supports core scheduling. This behavior is causing some unit tests to fail (for example: https://github.com/koordinator-sh/koordinator/blob/45685d10f00855a9db31cfc92b21dcb50ea8a46f/pkg/koordlet/runtimehooks/hooks/coresched/core_sched_test.go#L79).
Can I simply delete those tests? Or is there an API to distinguish between the mainline kernel and the Anolis OS kernel?@saintube: > Hi, I'm trying to create a PR but have encountered an issue. The mainline kernel doesn't have either /proc/sys/kernel/sched_core or /sys/kernel/debug/sched_features, yet it still supports core scheduling. This behavior is causing some unit tests to fail (for example:
> 
> [koordinator/pkg/koordlet/runtimehooks/hooks/coresched/core_sched_test.go](https://github.com/koordinator-sh/koordinator/blob/45685d10f00855a9db31cfc92b21dcb50ea8a46f/pkg/koordlet/runtimehooks/hooks/coresched/core_sched_test.go#L79)
> 
> Line 79 in [45685d1](/koordinator-sh/koordinator/commit/45685d10f00855a9db31cfc92b21dcb50ea8a46f)
> 
>  wants: wants{ 
> ).
> Can I simply delete those tests? Or is there an API to distinguish between the mainline kernel and the Anolis OS kernel?

This check is for Anolis OS where the lower version does not support the core scheduling. Perhaps we can use the global variable `HostSystemInfo` to differentiate whether the daemon is on Anolis OS.@hwenwur: > > Hi, I'm trying to create a PR but have encountered an issue. The mainline kernel doesn't have either /proc/sys/kernel/sched_core or /sys/kernel/debug/sched_features, yet it still supports core scheduling. This behavior is causing some unit tests to fail (for example:
> > [koordinator/pkg/koordlet/runtimehooks/hooks/coresched/core_sched_test.go](https://github.com/koordinator-sh/koordinator/blob/45685d10f00855a9db31cfc92b21dcb50ea8a46f/pkg/koordlet/runtimehooks/hooks/coresched/core_sched_test.go#L79)
> > Line 79 in [45685d1](/koordinator-sh/koordinator/commit/45685d10f00855a9db31cfc92b21dcb50ea8a46f)
> > wants: wants{
> > ).
> > Can I simply delete those tests? Or is there an API to distinguish between the mainline kernel and the Anolis OS kernel?
> 
> This check is for Anolis OS where the lower version does not support the core scheduling. Perhaps we can use the global variable `HostSystemInfo` to differentiate whether the daemon is on Anolis OS.

Maybe we can make use of the information provided by [uname()](https://man7.org/linux/man-pages/man2/uname.2.html), but I don't have access to Anolis OS or Alibaba Cloud Linux machines at hand. Do you know any sample outputs of the `uname -a `command on these machines?@saintube: @hwenwur There is a flag in the `HostSystemInfo ` indicating whether the OS is Anolis.

Solution patch:
diff --git a/apis/extension/device_share.go b/apis/extension/device_share.go
index 690dc612..8060a75b 100644
--- a/apis/extension/device_share.go
+++ b/apis/extension/device_share.go
@@ -69,8 +69,9 @@ const (
 )
 
 const (
-	GPUVendorNVIDIA = "nvidia"
-	GPUVendorHuawei = "huawei"
+	GPUVendorNVIDIA    = "nvidia"
+	GPUVendorHuawei    = "huawei"
+	GPUVendorCambricon = "cambricon"
 )
 
 // DeviceAllocations would be injected into Pod as form of annotation during Pre-bind stage.
diff --git a/apis/thirdparty/scheduler-plugins/pkg/util/podgroup.go b/apis/thirdparty/scheduler-plugins/pkg/util/podgroup.go
deleted file mode 100644
index b74e634b..00000000
--- a/apis/thirdparty/scheduler-plugins/pkg/util/podgroup.go
+++ /dev/null
@@ -1,76 +0,0 @@
-/*
-Copyright 2022 The Koordinator Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package util
-
-import (
-	"encoding/json"
-	"fmt"
-	"time"
-
-	v1 "k8s.io/api/core/v1"
-	"k8s.io/apimachinery/pkg/util/strategicpatch"
-
-	"github.com/koordinator-sh/koordinator/apis/thirdparty/scheduler-plugins/pkg/apis/scheduling/v1alpha1"
-)
-
-// DefaultWaitTime is 60s if ScheduleTimeoutSeconds is not specified.
-const DefaultWaitTime = 60 * time.Second
-
-// CreateMergePatch return patch generated from original and new interfaces
-func CreateMergePatch(original, new interface{}) ([]byte, error) {
-	pvByte, err := json.Marshal(original)
-	if err != nil {
-		return nil, err
-	}
-	cloneByte, err := json.Marshal(new)
-	if err != nil {
-		return nil, err
-	}
-	patch, err := strategicpatch.CreateTwoWayMergePatch(pvByte, cloneByte, original)
-	if err != nil {
-		return nil, err
-	}
-	return patch, nil
-}
-
-// GetPodGroupLabel get pod group from pod annotations
-func GetPodGroupLabel(pod *v1.Pod) string {
-	return pod.Labels[v1alpha1.PodGroupLabel]
-}
-
-// GetPodGroupFullName get namespaced group name from pod annotations
-func GetPodGroupFullName(pod *v1.Pod) string {
-	pgName := GetPodGroupLabel(pod)
-	if len(pgName) == 0 {
-		return ""
-	}
-	return fmt.Sprintf("%v/%v", pod.Namespace, pgName)
-}
-
-// GetWaitTimeDuration returns a wait timeout based on the following precedences:
-// 1. spec.scheduleTimeoutSeconds of the given pg, if specified
-// 2. given scheduleTimeout, if not nil
-// 3. fall back to DefaultWaitTime
-func GetWaitTimeDuration(pg *v1alpha1.PodGroup, scheduleTimeout *time.Duration) time.Duration {
-	if pg != nil && pg.Spec.ScheduleTimeoutSeconds != nil {
-		return time.Duration(*pg.Spec.ScheduleTimeoutSeconds) * time.Second
-	}
-	if scheduleTimeout != nil && *scheduleTimeout != 0 {
-		return *scheduleTimeout
-	}
-	return DefaultWaitTime
-}
diff --git a/apis/thirdparty/scheduler-plugins/pkg/util/resource.go b/apis/thirdparty/scheduler-plugins/pkg/util/resource.go
deleted file mode 100644
index 2c9f0a3b..00000000
--- a/apis/thirdparty/scheduler-plugins/pkg/util/resource.go
+++ /dev/null
@@ -1,77 +0,0 @@
-/*
-Copyright 2022 The Koordinator Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package util
-
-import (
-	"k8s.io/api/core/v1"
-	"k8s.io/apimachinery/pkg/api/resource"
-	v1helper "k8s.io/kubernetes/pkg/apis/core/v1/helper"
-	"k8s.io/kubernetes/pkg/scheduler/framework"
-)
-
-// ResourceList returns a resource list of this resource.
-// Note: this code used to exist in k/k, but removed in k/k#101465.
-func ResourceList(r *framework.Resource) v1.ResourceList {
-	result := v1.ResourceList{
-		v1.ResourceCPU:              *resource.NewMilliQuantity(r.MilliCPU, resource.DecimalSI),
-		v1.ResourceMemory:           *resource.NewQuantity(r.Memory, resource.BinarySI),
-		v1.ResourcePods:             *resource.NewQuantity(int64(r.AllowedPodNumber), resource.BinarySI),
-		v1.ResourceEphemeralStorage: *resource.NewQuantity(r.EphemeralStorage, resource.BinarySI),
-	}
-	for rName, rQuant := range r.ScalarResources {
-		if v1helper.IsHugePageResourceName(rName) {
-			result[rName] = *resource.NewQuantity(rQuant, resource.BinarySI)
-		} else {
-			result[rName] = *resource.NewQuantity(rQuant, resource.DecimalSI)
-		}
-	}
-	return result
-}
-
-// GetPodEffectiveRequest gets the effective request resource of a pod to the origin resource.
-// The Pod's effective request is the higher of:
-// - the sum of all app containers(spec.Containers) request for a resource.
-// - the effective init containers(spec.InitContainers) request for a resource.
-// The effective init containers request is the highest request on all init containers.
-func GetPodEffectiveRequest(pod *v1.Pod) v1.ResourceList {
-	initResources := make(v1.ResourceList)
-	resources := make(v1.ResourceList)
-
-	for _, container := range pod.Spec.InitContainers {
-		for name, quantity := range container.Resources.Requests {
-			if q, ok := initResources[name]; ok && quantity.Cmp(q) <= 0 {
-				continue
-			}
-			initResources[name] = quantity
-		}
-	}
-	for _, container := range pod.Spec.Containers {
-		for name, quantity := range container.Resources.Requests {
-			if q, ok := resources[name]; ok {
-				quantity.Add(q)
-			}
-			resources[name] = quantity
-		}
-	}
-	for name, quantity := range initResources {
-		if q, ok := resources[name]; ok && quantity.Cmp(q) <= 0 {
-			continue
-		}
-		resources[name] = quantity
-	}
-	return resources
-}
diff --git a/pkg/koordlet/util/system/core_sched.go b/pkg/koordlet/util/system/core_sched.go
index 319f92fa..18bf4a7f 100644
--- a/pkg/koordlet/util/system/core_sched.go
+++ b/pkg/koordlet/util/system/core_sched.go
@@ -20,7 +20,9 @@ import (
 	"fmt"
 	"os"
 	"strings"
+	"unsafe"
 
+	"golang.org/x/sys/unix"
 	"k8s.io/klog/v2"
 )
 
@@ -272,12 +274,37 @@ func IsCoreSchedSysctlSupported() bool {
 	return FileExists(GetProcSysFilePath(KernelSchedCore))
 }
 
+// ProbeCoreSchedIfEnabled checks if the MAINLINE kernel support the core scheduling.
+// Since there's no direct API, we probe by calling prctl and checking its return value.
+func ProbeCoreSchedIfEnabled() bool {
+	cookie := uint64(0)
+	cookiePtr := &cookie
+	ret, err := unix.PrctlRetInt(unix.PR_SCHED_CORE, unix.PR_SCHED_CORE_GET, uintptr(0), uintptr(CoreSchedScopeThread), uintptr(unsafe.Pointer(cookiePtr)))
+	if err != nil {
+		klog.V(4).Infof("failed to probe core sched status via prctl, err: %s", err)
+		return false
+	}
+	return ret == 0
+}
+
 // IsCoreSchedSupported checks if the kernel supports the core scheduling.
-// Currently, it relies on the interfaces provided by the Anolis OS.
+// Currently, it supports both Anolis OS and mainline kernel.
 func IsCoreSchedSupported() (bool, string) {
-	// kernel supports if:
+	// Anolis OS supports if:
 	// a) sysctl has sched_core,
 	// b) sched_features has SCHED_CORE/NO_SCHED_CORE
+	// Mainline kernel supports if(only 5.14 and later):
+	// a) CONFIG_SCHED_CORE config option enabled
+
+	if !HostSystemInfo.IsAnolisOS {
+		// For mainline kernel
+		if ProbeCoreSchedIfEnabled() {
+			return true, "prctl supported"
+		} else {
+			return false, "not unsupported by prctl"
+		}
+	}
+
 	if IsCoreSchedSysctlSupported() {
 		return true, "sysctl supported"
 	}
@@ -296,12 +323,22 @@ func IsCoreSchedSupported() (bool, string) {
 
 // EnableCoreSchedIfSupported checks if the core scheduling feature is enabled in the kernel sched_features.
 // If kernel supported (available in the latest Anolis OS), it tries to enable the core scheduling feature.
+// For Anolis OS
 // The core sched's kernel feature is known set in two places, if both of them are not found, the system is considered
 // unsupported for the core scheduling:
 //  1. In `/proc/sys/kernel/sched_core`, the value `1` means the feature is enabled while `0` means disabled.
 //  2. (Older kernel) In `/sys/kernel/debug/sched_features`, the field `CORE_SCHED` means the feature is enabled while `NO_CORE_SCHED`
 //     means it is disabled.
+//
+// For mainline kernel
+// Core scheduling support is enabled via the CONFIG_SCHED_CORE config option. And can not be turn on/off dynamically.
 func EnableCoreSchedIfSupported() (bool, string) {
+	if !HostSystemInfo.IsAnolisOS {
+		// For mainline kernel
+		if ProbeCoreSchedIfEnabled() {
+			return true, ""
+		}
+	}
 	// 1. try sysctl
 	isSysctlSupported, err := GetSchedCore()
 	if err == nil && isSysctlSupported {
diff --git a/pkg/scheduler/frameworkext/errorhandler_dispatcher.go b/pkg/scheduler/frameworkext/errorhandler_dispatcher.go
index 1484b57d..da5625a4 100644
--- a/pkg/scheduler/frameworkext/errorhandler_dispatcher.go
+++ b/pkg/scheduler/frameworkext/errorhandler_dispatcher.go
@@ -36,9 +36,7 @@ type errorHandlerDispatcher struct {
 }
 
 func newErrorHandlerDispatcher() *errorHandlerDispatcher {
-	return &errorHandlerDispatcher{
-		preHandlerFilters: []PreErrorHandlerFilter{TakeoverNominatingInfo},
-	}
+	return &errorHandlerDispatcher{}
 }
 
 func (d *errorHandlerDispatcher) setDefaultHandler(handler scheduler.FailureHandlerFn) {
@@ -63,6 +61,9 @@ func (d *errorHandlerDispatcher) Error(ctx context.Context, fwk framework.Framew
 		}
 	}()
 
+	// FIXME here, we replace nominatingInfo to avoid modifying clearNominatedNode
+	nominatingInfo = TakeoverNominatingInfo(ctx, fwk, podInfo, status, nominatingInfo, start)
+
 	for _, handlerFilter := range d.preHandlerFilters {
 		if handlerFilter(ctx, fwk, podInfo, status, nominatingInfo, start) {
 			return
diff --git a/pkg/scheduler/frameworkext/framework_extender.go b/pkg/scheduler/frameworkext/framework_extender.go
index f9ef05a9..a0284079 100644
--- a/pkg/scheduler/frameworkext/framework_extender.go
+++ b/pkg/scheduler/frameworkext/framework_extender.go
@@ -273,7 +273,14 @@ func (ext *frameworkExtenderImpl) RunFilterPluginsWithNominatedPods(ctx context.
 			nodeInfo = newNodeInfo
 		}
 	}
-	status := ext.Framework.RunFilterPluginsWithNominatedPods(ctx, cycleState, pod, nodeInfo)
+
+	nominatedPodsOfSameJob := getNominatedPodsOfTheSameJob(cycleState)
+	var status *framework.Status
+	if len(nominatedPodsOfSameJob) != 0 {
+		status = ext.runFilterPluginsWithNominatedPodsIgnoreSameJob(ctx, cycleState, pod, nodeInfo, nominatedPodsOfSameJob)
+	} else {
+		status = ext.Framework.RunFilterPluginsWithNominatedPods(ctx, cycleState, pod, nodeInfo)
+	}
 	if !status.IsSuccess() && debugFilterFailure {
 		klog.Infof("Failed to filter for Pod %q on Node %q, failedPlugin: %s, schedulingPhaseBeingInvoked: %s, reason: %s", klog.KObj(pod), klog.KObj(nodeInfo.Node()), status.FailedPlugin(), schedulingphase.GetExtensionPointBeingExecuted(cycleState), status.Message())
 	}
diff --git a/pkg/scheduler/frameworkext/job_nominated_pods.go b/pkg/scheduler/frameworkext/job_nominated_pods.go
new file mode 100644
index 00000000..f27c269b
--- /dev/null
+++ b/pkg/scheduler/frameworkext/job_nominated_pods.go
@@ -0,0 +1,152 @@
+/*
+Copyright 2022 The Koordinator Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package frameworkext
+
+import (
+	"context"
+
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/util/sets"
+	corev1helper "k8s.io/component-helpers/scheduling/corev1"
+	"k8s.io/klog/v2"
+	"k8s.io/kubernetes/pkg/scheduler/framework"
+
+	"github.com/koordinator-sh/koordinator/apis/extension"
+)
+
+const (
+	nominatedPodsOfTheSameJob = extension.SchedulingDomainPrefix + "/nominated-pods-to-ignore"
+)
+
+type NominatedPodsOfTheSameJob struct {
+	UIDs sets.Set[string]
+}
+
+func (s *NominatedPodsOfTheSameJob) Clone() framework.StateData {
+	return s
+}
+
+func MakeNominatedPodsOfTheSameJob(cycleState *framework.CycleState, uids []string) {
+	cycleState.Write(nominatedPodsOfTheSameJob, &NominatedPodsOfTheSameJob{
+		UIDs: sets.New[string](uids...),
+	})
+}
+
+func getNominatedPodsOfTheSameJob(cycleState *framework.CycleState) sets.Set[string] {
+	s, err := cycleState.Read(nominatedPodsOfTheSameJob)
+	if err != nil || s == nil {
+		return nil
+	}
+	return s.(*NominatedPodsOfTheSameJob).UIDs
+
+}
+
+func (ext *frameworkExtenderImpl) runFilterPluginsWithNominatedPodsIgnoreSameJob(ctx context.Context, state *framework.CycleState, pod *corev1.Pod, info *framework.NodeInfo, podsOfSameJob sets.Set[string]) *framework.Status {
+	var status *framework.Status
+
+	podsAdded := false
+	// We run filters twice in some cases. If the node has greater or equal priority
+	// nominated pods, we run them when those pods are added to PreFilter state and nodeInfo.
+	// If all filters succeed in this pass, we run them again when these
+	// nominated pods are not added. This second pass is necessary because some
+	// filters such as inter-pod affinity may not pass without the nominated pods.
+	// If there are no nominated pods for the node or if the first run of the
+	// filters fail, we don't run the second pass.
+	// We consider only equal or higher priority pods in the first pass, because
+	// those are the current "pod" must yield to them and not take a space opened
+	// for running them. It is ok if the current "pod" take resources freed for
+	// lower priority pods.
+	// Requiring that the new pod is schedulable in both circumstances ensures that
+	// we are making a conservative decision: filters like resources and inter-pod
+	// anti-affinity are more likely to fail when the nominated pods are treated
+	// as running, while filters like pod affinity are more likely to fail when
+	// the nominated pods are treated as not running. We can't just assume the
+	// nominated pods are running because they are not running right now and in fact,
+	// they may end up getting scheduled to a different node.
+	logger := klog.FromContext(ctx)
+	logger = klog.LoggerWithName(logger, "FilterWithNominatedPods")
+	ctx = klog.NewContext(ctx, logger)
+	for i := 0; i < 2; i++ {
+		stateToUse := state
+		nodeInfoToUse := info
+		var addedPods []string
+		if i == 0 {
+			var err error
+			podsAdded, stateToUse, nodeInfoToUse, err, addedPods = addNominatedPods(ctx, ext, pod, state, info, podsOfSameJob)
+			if err != nil {
+				return framework.AsStatus(err)
+			}
+		} else if !podsAdded || !status.IsSuccess() {
+			break
+		}
+
+		status = ext.RunFilterPlugins(ctx, stateToUse, pod, nodeInfoToUse)
+		if !status.IsSuccess() && pod.Status.NominatedNodeName == nodeInfoToUse.Node().Name && klog.V(4).Enabled() {
+			existingPods := make([]string, 0, len(nodeInfoToUse.Pods))
+			for _, podInfo := range nodeInfoToUse.Pods {
+				if len(podInfo.Pod.OwnerReferences) != 0 && podInfo.Pod.OwnerReferences[0].Kind == "DaemonSet" {
+					// Normally, the daemonset on the node does not occupy resources, so when collecting the existing Pods, the daemonset Pods are ignored to reduce the number of logs.
+					continue
+				}
+				existingPods = append(existingPods, framework.GetNamespacedName(podInfo.Pod.Namespace, podInfo.Pod.Name))
+			}
+			klog.V(4).Infof("Pod %s/%s is nominated to run on node %q, but failed to scheduling cause some existingPods: %+v, addedPods: %+v, status: %+v", pod.Namespace, pod.Name, nodeInfoToUse.Node().Name, existingPods, addedPods, status)
+		}
+		if !status.IsSuccess() && !status.IsUnschedulable() {
+			return status
+		}
+
+	}
+
+	return status
+}
+
+// addNominatedPods adds pods with equal or greater priority which are nominated
+// to run on the node. It returns 1) whether any pod was added, 2) augmented cycleState,
+// 3) augmented nodeInfo.
+func addNominatedPods(ctx context.Context, fh framework.Handle, pod *corev1.Pod, state *framework.CycleState, nodeInfo *framework.NodeInfo, podsOfSameJob sets.Set[string]) (bool, *framework.CycleState, *framework.NodeInfo, error, []string) {
+	if fh == nil {
+		// This may happen only in tests.
+		return false, state, nodeInfo, nil, nil
+	}
+	nominatedPodInfos := fh.NominatedPodsForNode(nodeInfo.Node().Name)
+	if len(nominatedPodInfos) == 0 {
+		return false, state, nodeInfo, nil, nil
+	}
+	nodeInfoOut := nodeInfo.Clone()
+	stateOut := state.Clone()
+	podsAdded := false
+	var addedPods []string
+
+	for _, pi := range nominatedPodInfos {
+		if corev1helper.PodPriority(pi.Pod) >= corev1helper.PodPriority(pod) && pi.Pod.UID != pod.UID && !podsOfSameJob.Has(string(pi.Pod.UID)) {
+			nodeInfoOut.AddPodInfo(pi)
+			if klog.V(5).Enabled() || pod.Status.NominatedNodeName == nodeInfo.Node().Name {
+				addedPods = append(addedPods, framework.GetNamespacedName(pi.Pod.Namespace, pi.Pod.Name))
+			}
+			status := fh.RunPreFilterExtensionAddPod(ctx, stateOut, pod, pi, nodeInfoOut)
+			if !status.IsSuccess() {
+				return false, state, nodeInfo, status.AsError(), nil
+			}
+			podsAdded = true
+		}
+	}
+	if len(addedPods) > 0 {
+		klog.V(5).Infof("Added %v pods with equal or higher priority to the node %q when schedule pod %s/%s", addedPods, nodeInfo.Node().Name, pod.Namespace, pod.Name)
+	}
+	return podsAdded, stateOut, nodeInfoOut, nil, addedPods
+}
diff --git a/pkg/scheduler/frameworkext/nominating_info_takeover.go b/pkg/scheduler/frameworkext/nominating_info_takeover.go
index 16a525c5..0ab7ebfd 100644
--- a/pkg/scheduler/frameworkext/nominating_info_takeover.go
+++ b/pkg/scheduler/frameworkext/nominating_info_takeover.go
@@ -25,31 +25,36 @@ import (
 )
 
 var (
-	// JobPreemptionRejectPlugin If the waitingPod is rejected by the Job preemption plug-in, the assumedNodeName is the nominatedNodeName.
-	JobPreemptionRejectPlugin = "JobPreemptionRejectPlugin"
+	// JobPreemptionSuccessPlugin If the waitingPod is rejected by the Job preemption plugin, the assumedNodeName is the nominatedNodeName.
+	JobPreemptionSuccessPlugin = "JobPreemptionSuccessPlugin"
+	// JobPreemptionFailurePlugin If Job preemption fails, the nominatedNodeName of the WaitingPod should be erased.
+	JobPreemptionFailurePlugin = "JobPreemptionFailurePlugin"
 	// JobRejectPlugin If the waitingPod is only recycled by the JobRejectPlugin, the NominatedNodeName should not be modified.
 	JobRejectPlugin = "JobRejectPlugin"
 )
 
-func TakeoverNominatingInfo(
-	ctx context.Context,
-	fwk framework.Framework,
-	podInfo *framework.QueuedPodInfo,
-	status *framework.Status,
-	nominatingInfo *framework.NominatingInfo,
-	start time.Time) bool {
+func TakeoverNominatingInfo(ctx context.Context, fwk framework.Framework, podInfo *framework.QueuedPodInfo, status *framework.Status, nominatingInfo *framework.NominatingInfo, start time.Time) *framework.NominatingInfo {
 	if status == nil || status.Code() != framework.Unschedulable {
-		return false
+		return nominatingInfo
 	}
 	rejecterPlugin := getRejecterPlugin(status)
-	if rejecterPlugin == JobPreemptionRejectPlugin {
-		nominatingInfo.NominatingMode = framework.ModeOverride
-		nominatingInfo.NominatedNodeName = podInfo.Pod.Spec.NodeName
+	if rejecterPlugin == JobPreemptionSuccessPlugin {
+		return &framework.NominatingInfo{
+			NominatingMode:    framework.ModeOverride,
+			NominatedNodeName: podInfo.Pod.Spec.NodeName,
+		}
 	} else if status.FailedPlugin() == JobRejectPlugin {
-		nominatingInfo.NominatingMode = framework.ModeNoop
-		nominatingInfo.NominatedNodeName = ""
+		return &framework.NominatingInfo{
+			NominatingMode:    framework.ModeNoop,
+			NominatedNodeName: "",
+		}
+	} else if status.FailedPlugin() == JobPreemptionFailurePlugin {
+		return &framework.NominatingInfo{
+			NominatingMode:    framework.ModeOverride,
+			NominatedNodeName: "",
+		}
 	}
-	return false
+	return nominatingInfo
 }
 
 func getRejecterPlugin(status *framework.Status) string {
diff --git a/pkg/scheduler/plugins/deviceshare/device_plugin_adapter.go b/pkg/scheduler/plugins/deviceshare/device_plugin_adapter.go
index 61009d52..63b81063 100644
--- a/pkg/scheduler/plugins/deviceshare/device_plugin_adapter.go
+++ b/pkg/scheduler/plugins/deviceshare/device_plugin_adapter.go
@@ -17,17 +17,22 @@ limitations under the License.
 package deviceshare
 
 import (
+	"context"
 	"fmt"
 	"strconv"
 	"strings"
+	"time"
 
+	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/client-go/kubernetes"
 	"k8s.io/klog/v2"
 	"k8s.io/utils/clock"
 
 	apiext "github.com/koordinator-sh/koordinator/apis/extension"
 	schedulingv1alpha1 "github.com/koordinator-sh/koordinator/apis/scheduling/v1alpha1"
 	"github.com/koordinator-sh/koordinator/pkg/scheduler/frameworkext"
+	"github.com/koordinator-sh/koordinator/pkg/util"
 )
 
 const (
@@ -51,34 +56,62 @@ const (
 	// AnnotationHuaweiNPUCore represents the NPU/vNPU allocation result for the pod which is used by
 	// Huawei NPU device plugins to allocate NPU(s)/vNPU for pod.
 	AnnotationHuaweiNPUCore = "huawei.com/npu-core"
+
+	// AnnotationCambriconDsmluAssigned is used by Cambricon dynamic sMLU device plugins to determine pod.
+	AnnotationCambriconDsmluAssigned = "CAMBRICON_DSMLU_ASSIGHED"
+	// AnnotationCambriconDsmluProfile represents the dynamic sMLU allocation result for the pod which is used by
+	// Cambricon dynamic sMLU device plugins to allocate sMLU for pod.
+	AnnotationCambriconDsmluProfile = "CAMBRICON_DSMLU_PROFILE"
+	// AnnotationCambriconDsmluLockTime is annotated to node by scheduler to prevent scheduling multiple pods
+	// with dynamic sMLU to the same node which would make Cambricon device plugin unable to determine pod.
+	// Cambricon dynamic sMLU device plugin will automatically remove it after allocation of a pod.
+	AnnotationCambriconDsmluLockTime = "cambricon.com/dsmlu.lock"
+	// cambriconVMemoryUnit is the minial virtual memory unit of Cambricon dynamic sMLU currently supported.
+	// This should always match the min-dsmlu-unit arg of device plugin.
+	cambriconVMemoryUnit = 256 * 1024 * 1024
 )
 
 // DevicePluginAdapter adapt koord-scheduler's device allocation result to the format that third party vendor's
 // device plugin recognizes, so that user can directly use them as allocators with koord-scheduler without modification.
 // This is especially useful when third party device plugin natively supports fine-grained device allocation or virtualization.
 type DevicePluginAdapter interface {
-	Adapt(object metav1.Object, allocation []*apiext.DeviceAllocation) error
+	Adapt(ctx *DevicePluginAdaptContext, object metav1.Object, allocation []*apiext.DeviceAllocation) error
+}
+
+type DevicePluginAdaptContext struct {
+	context.Context
+	client kubernetes.Interface
+	node   *corev1.Node
 }
 
 var (
-	defaultDevicePluginAdapter = &generalDevicePluginAdapter{
+	defaultDevicePluginAdapter DevicePluginAdapter = &generalDevicePluginAdapter{
 		clock: clock.RealClock{},
 	}
-	defaultGPUDevicePluginAdapter = &generalGPUDevicePluginAdapter{}
-	gpuDevicePluginAdapterMap     = map[string]DevicePluginAdapter{
+	defaultGPUDevicePluginAdapter DevicePluginAdapter = &generalGPUDevicePluginAdapter{}
+	gpuDevicePluginAdapterMap                         = map[string]DevicePluginAdapter{
 		apiext.GPUVendorHuawei: &huaweiGPUDevicePluginAdapter{
 			clock: clock.RealClock{},
 		},
+		apiext.GPUVendorCambricon: &cambriconGPUDevicePluginAdapter{
+			clock: clock.RealClock{},
+		},
 	}
 )
 
-func (p *Plugin) adaptForDevicePlugin(object metav1.Object, allocationResult apiext.DeviceAllocations, nodeName string) error {
-	if err := defaultDevicePluginAdapter.Adapt(object, nil); err != nil {
+func (p *Plugin) adaptForDevicePlugin(ctx context.Context, object metav1.Object, allocationResult apiext.DeviceAllocations, nodeName string) error {
+	if err := defaultDevicePluginAdapter.Adapt(nil, object, nil); err != nil {
 		return err
 	}
 
 	if gpuAllocation, ok := allocationResult[schedulingv1alpha1.GPU]; ok {
-		if err := defaultGPUDevicePluginAdapter.Adapt(object, gpuAllocation); err != nil {
+		if err := defaultGPUDevicePluginAdapter.Adapt(nil, object, gpuAllocation); err != nil {
+			return err
+		}
+
+		nodeInfo, err := p.handle.SnapshotSharedLister().NodeInfos().Get(nodeName)
+		if err != nil {
+			klog.ErrorS(err, "Failed to get NodeInfo", "node", nodeName)
 			return err
 		}
 
@@ -93,9 +126,14 @@ func (p *Plugin) adaptForDevicePlugin(object metav1.Object, allocationResult api
 			return err
 		}
 
+		adaptCtx := &DevicePluginAdaptContext{
+			Context: ctx,
+			client:  p.handle.ClientSet(),
+			node:    nodeInfo.Node(),
+		}
 		vendor := device.Labels[apiext.LabelGPUVendor]
 		if adapter, ok := gpuDevicePluginAdapterMap[vendor]; ok {
-			if err := adapter.Adapt(object, gpuAllocation); err != nil {
+			if err := adapter.Adapt(adaptCtx, object, gpuAllocation); err != nil {
 				return fmt.Errorf("failed to adapt for GPU device plugin of vendor %q: %w", vendor, err)
 			}
 		}
@@ -110,7 +148,7 @@ type generalDevicePluginAdapter struct {
 	clock clock.Clock
 }
 
-func (a *generalDevicePluginAdapter) Adapt(object metav1.Object, _ []*apiext.DeviceAllocation) error {
+func (a *generalDevicePluginAdapter) Adapt(_ *DevicePluginAdaptContext, object metav1.Object, _ []*apiext.DeviceAllocation) error {
 	object.GetAnnotations()[AnnotationBindTimestamp] = strconv.FormatInt(a.clock.Now().UnixNano(), 10)
 	return nil
 }
@@ -120,7 +158,7 @@ func (a *generalDevicePluginAdapter) Adapt(object metav1.Object, _ []*apiext.Dev
 type generalGPUDevicePluginAdapter struct {
 }
 
-func (a *generalGPUDevicePluginAdapter) Adapt(object metav1.Object, allocation []*apiext.DeviceAllocation) error {
+func (a *generalGPUDevicePluginAdapter) Adapt(_ *DevicePluginAdaptContext, object metav1.Object, allocation []*apiext.DeviceAllocation) error {
 	object.GetAnnotations()[AnnotationGPUMinors] = buildGPUMinorsStr(allocation)
 	return nil
 }
@@ -129,7 +167,7 @@ type huaweiGPUDevicePluginAdapter struct {
 	clock clock.Clock
 }
 
-func (a *huaweiGPUDevicePluginAdapter) Adapt(object metav1.Object, allocation []*apiext.DeviceAllocation) error {
+func (a *huaweiGPUDevicePluginAdapter) Adapt(_ *DevicePluginAdaptContext, object metav1.Object, allocation []*apiext.DeviceAllocation) error {
 	object.GetAnnotations()[AnnotationPredicateTime] = strconv.FormatInt(a.clock.Now().UnixNano(), 10)
 	if allocation[0].Extension != nil && allocation[0].Extension.GPUSharedResourceTemplate != "" {
 		// vNPU
@@ -141,6 +179,74 @@ func (a *huaweiGPUDevicePluginAdapter) Adapt(object metav1.Object, allocation []
 	return nil
 }
 
+type cambriconGPUDevicePluginAdapter struct {
+	clock clock.Clock
+}
+
+func (a *cambriconGPUDevicePluginAdapter) Adapt(ctx *DevicePluginAdaptContext, object metav1.Object, allocation []*apiext.DeviceAllocation) error {
+	if len(allocation) > 1 {
+		return fmt.Errorf("multiple gpu share is not supported on device side")
+	}
+
+	core, ok := allocation[0].Resources[apiext.ResourceGPUCore]
+	if !ok {
+		return fmt.Errorf("gpu core resource is required")
+	}
+	vcore := core.Value()
+
+	memory := allocation[0].Resources[apiext.ResourceGPUMemory]
+	if memory.Value() < cambriconVMemoryUnit {
+		return fmt.Errorf("gpu memory must not be less than %d bytes", cambriconVMemoryUnit)
+	}
+	vmemory := memory.Value() / cambriconVMemoryUnit
+
+	if err := a.lockNode(ctx, ctx.client, ctx.node); err != nil {
+		return fmt.Errorf("failed to lock node: %v", err)
+	}
+
+	object.GetAnnotations()[AnnotationCambriconDsmluAssigned] = "false"
+	object.GetAnnotations()[AnnotationCambriconDsmluProfile] = fmt.Sprintf("%d_%d_%d", allocation[0].Minor, vcore, vmemory)
+	return nil
+}
+
+func (a *cambriconGPUDevicePluginAdapter) lockNode(ctx context.Context, client kubernetes.Interface, node *corev1.Node) error {
+	if val, ok := node.Annotations[AnnotationCambriconDsmluLockTime]; ok {
+		lockTime, err := time.Parse(time.RFC3339, val)
+		if err != nil {
+			return err
+		}
+		if time.Since(lockTime) > 2*time.Minute {
+			// this should never happen in normal cases, and we don't want to lock the node forever
+			if err := a.unlockNode(ctx, client, node); err != nil {
+				return err
+			}
+		} else {
+			return fmt.Errorf("node %q has been locked", node.Name)
+		}
+	}
+	newNode := node.DeepCopy()
+	if newNode.Annotations == nil {
+		newNode.Annotations = map[string]string{}
+	}
+	newNode.Annotations[AnnotationCambriconDsmluLockTime] = a.clock.Now().Format(time.RFC3339)
+	if _, err := util.PatchNode(ctx, client, node, newNode); err != nil {
+		return fmt.Errorf("failed to patch node %q: %v", node.Name, err)
+	}
+	return nil
+}
+
+func (a *cambriconGPUDevicePluginAdapter) unlockNode(ctx context.Context, client kubernetes.Interface, node *corev1.Node) error {
+	if _, ok := node.Annotations[AnnotationCambriconDsmluLockTime]; !ok {
+		return nil
+	}
+	newNode := node.DeepCopy()
+	delete(newNode.Annotations, AnnotationCambriconDsmluLockTime)
+	if _, err := util.PatchNode(ctx, client, node, newNode); err != nil {
+		return fmt.Errorf("failed to patch node %q: %v", node.Name, err)
+	}
+	return nil
+}
+
 func buildGPUMinorsStr(allocation []*apiext.DeviceAllocation) string {
 	minors := make([]string, 0, len(allocation))
 	for _, alloc := range allocation {
diff --git a/pkg/scheduler/plugins/deviceshare/plugin.go b/pkg/scheduler/plugins/deviceshare/plugin.go
index b562861a..c1617c21 100644
--- a/pkg/scheduler/plugins/deviceshare/plugin.go
+++ b/pkg/scheduler/plugins/deviceshare/plugin.go
@@ -623,7 +623,7 @@ func (p *Plugin) preBindObject(ctx context.Context, cycleState *framework.CycleS
 	}
 
 	if k8sfeature.DefaultMutableFeatureGate.Enabled(features.DevicePluginAdaption) {
-		if err := p.adaptForDevicePlugin(object, state.allocationResult, nodeName); err != nil {
+		if err := p.adaptForDevicePlugin(ctx, object, state.allocationResult, nodeName); err != nil {
 			return framework.AsStatus(err)
 		}
 	}
diff --git a/pkg/scheduler/plugins/elasticquota/controller.go b/pkg/scheduler/plugins/elasticquota/controller.go
index 31b8e8cf..8cb26740 100644
--- a/pkg/scheduler/plugins/elasticquota/controller.go
+++ b/pkg/scheduler/plugins/elasticquota/controller.go
@@ -28,13 +28,13 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/apimachinery/pkg/util/strategicpatch"
 	"k8s.io/apimachinery/pkg/util/wait"
 	quotav1 "k8s.io/apiserver/pkg/quota/v1"
 	"k8s.io/klog/v2"
 
 	"github.com/koordinator-sh/koordinator/apis/extension"
 	"github.com/koordinator-sh/koordinator/apis/thirdparty/scheduler-plugins/pkg/apis/scheduling/v1alpha1"
-	"github.com/koordinator-sh/koordinator/apis/thirdparty/scheduler-plugins/pkg/util"
 	"github.com/koordinator-sh/koordinator/pkg/scheduler/plugins/elasticquota/core"
 	koordutil "github.com/koordinator-sh/koordinator/pkg/util"
 )
@@ -128,7 +128,7 @@ func (ctrl *Controller) syncElasticQuotaStatus(eq *v1alpha1.ElasticQuota) {
 		klog.InfoS("Try updating elasticQuota since it has changed", "elasticQuota", eq.Name)
 	}
 
-	patch, err := util.CreateMergePatch(eq, newEQ)
+	patch, err := createMergePatch(eq, newEQ)
 	if err != nil {
 		klog.ErrorS(err, "Failed to create mergePatch", "elasticQuota", eq.Name)
 		return
@@ -153,6 +153,23 @@ func (ctrl *Controller) syncElasticQuotaStatus(eq *v1alpha1.ElasticQuota) {
 	}
 }
 
+// createMergePatch return patch generated from original and new interfaces
+func createMergePatch(original, new interface{}) ([]byte, error) {
+	pvByte, err := json.Marshal(original)
+	if err != nil {
+		return nil, err
+	}
+	cloneByte, err := json.Marshal(new)
+	if err != nil {
+		return nil, err
+	}
+	patch, err := strategicpatch.CreateTwoWayMergePatch(pvByte, cloneByte, original)
+	if err != nil {
+		return nil, err
+	}
+	return patch, nil
+}
+
 var resourceDecorators []func(quota *v1alpha1.ElasticQuota, resource v1.ResourceList)
 
 func decorateResource(quota *v1alpha1.ElasticQuota, resource v1.ResourceList) {
diff --git a/pkg/scheduler/plugins/reservation/transformer.go b/pkg/scheduler/plugins/reservation/transformer.go
index 8f72ec72..d63c142e 100644
--- a/pkg/scheduler/plugins/reservation/transformer.go
+++ b/pkg/scheduler/plugins/reservation/transformer.go
@@ -77,8 +77,9 @@ func (pl *Plugin) AfterPreFilter(ctx context.Context, cycleState *framework.Cycl
 	parallelCtx, cancel := context.WithCancel(ctx)
 	defer cancel()
 	errCh := parallelize.NewErrorChannel()
+	nodeInfoLister := pl.handle.SnapshotSharedLister().NodeInfos()
 	checkNode := func(i int) {
-		nodeInfo, err := pl.handle.SnapshotSharedLister().NodeInfos().Get(allNodes[i])
+		nodeInfo, err := nodeInfoLister.Get(allNodes[i])
 		if err != nil {
 			klog.Warningf("Failed to get NodeInfo of %s during reservation's AfterPreFilter for pod %v, err: %s", allNodes[i], klog.KObj(pod), err)
 			return
diff --git a/pkg/util/utils.go b/pkg/util/utils.go
index 3103185d..8ddf3bc7 100644
--- a/pkg/util/utils.go
+++ b/pkg/util/utils.go
@@ -259,6 +259,45 @@ func PatchReservationSafe(ctx context.Context, clientset koordinatorclientset.In
 	return patched, nil
 }
 
+func GenerateNodePatch(oldNode, newNode *corev1.Node) ([]byte, error) {
+	oldData, err := json.Marshal(oldNode)
+	if err != nil {
+		return nil, err
+	}
+
+	newData, err := json.Marshal(newNode)
+	if err != nil {
+		return nil, err
+	}
+	return strategicpatch.CreateTwoWayMergePatch(oldData, newData, &corev1.Pod{})
+}
+
+func PatchNode(ctx context.Context, clientset clientset.Interface, oldNode, newNode *corev1.Node) (*corev1.Node, error) {
+	if reflect.DeepEqual(oldNode, newNode) {
+		return oldNode, nil
+	}
+
+	// generate patch bytes for the update
+	patchBytes, err := GenerateNodePatch(oldNode, newNode)
+	if err != nil {
+		klog.V(5).InfoS("failed to generate node patch", "node", klog.KObj(oldNode), "err", err)
+		return nil, err
+	}
+	if string(patchBytes) == "{}" { // nothing to patch
+		return oldNode, nil
+	}
+
+	// patch with node client
+	patched, err := clientset.CoreV1().Nodes().
+		Patch(ctx, oldNode.Name, apimachinerytypes.StrategicMergePatchType, patchBytes, metav1.PatchOptions{})
+	if err != nil {
+		klog.V(5).InfoS("failed to patch node", "node", klog.KObj(oldNode), "patch", string(patchBytes), "err", err)
+		return nil, err
+	}
+	klog.V(6).InfoS("successfully patch node", "node", klog.KObj(oldNode), "patch", string(patchBytes))
+	return patched, nil
+}
+
 func GetNamespacedName(namespace, name string) string {
 	return fmt.Sprintf("%s/%s", namespace, name)
 }



Test patch:
diff --git a/pkg/koordlet/runtimehooks/hooks/coresched/core_sched_test.go b/pkg/koordlet/runtimehooks/hooks/coresched/core_sched_test.go
index e544b0c5..3c1b8fce 100644
--- a/pkg/koordlet/runtimehooks/hooks/coresched/core_sched_test.go
+++ b/pkg/koordlet/runtimehooks/hooks/coresched/core_sched_test.go
@@ -35,6 +35,7 @@ import (
 	"github.com/koordinator-sh/koordinator/pkg/koordlet/runtimehooks/protocol"
 	"github.com/koordinator-sh/koordinator/pkg/koordlet/statesinformer"
 	"github.com/koordinator-sh/koordinator/pkg/koordlet/util"
+	"github.com/koordinator-sh/koordinator/pkg/koordlet/util/system"
 	sysutil "github.com/koordinator-sh/koordinator/pkg/koordlet/util/system"
 )
 
@@ -50,7 +51,8 @@ func TestPlugin(t *testing.T) {
 
 func TestPluginSystemSupported(t *testing.T) {
 	type fields struct {
-		prepareFn func(helper *sysutil.FileTestUtil)
+		prepareFn   func(helper *sysutil.FileTestUtil)
+		notAnolisOS bool
 	}
 	type wants struct {
 		systemSupported bool
@@ -134,6 +136,7 @@ func TestPluginSystemSupported(t *testing.T) {
 				Reader:   resourceexecutor.NewCgroupReader(),
 				Executor: resourceexecutor.NewTestResourceExecutor(),
 			})
+			system.HostSystemInfo.IsAnolisOS = !tt.fields.notAnolisOS
 			sysSupported := p.SystemSupported()
 			assert.Equal(t, tt.wants.systemSupported, sysSupported)
 			assert.Equal(t, tt.wants.supportMsg, p.supportedMsg)
@@ -145,6 +148,7 @@ func TestPlugin_initSystem(t *testing.T) {
 	type fields struct {
 		prepareFn   func(helper *sysutil.FileTestUtil)
 		giSupported *bool
+		notAnolisOS bool
 	}
 	tests := []struct {
 		name      string
@@ -281,6 +285,7 @@ func TestPlugin_initSystem(t *testing.T) {
 			if tt.fields.prepareFn != nil {
 				tt.fields.prepareFn(helper)
 			}
+			system.HostSystemInfo.IsAnolisOS = !tt.fields.notAnolisOS
 
 			p := newPlugin()
 			p.Setup(hooks.Options{})
@@ -302,6 +307,7 @@ func TestPlugin_SetContainerCookie(t *testing.T) {
 		preparePluginFn func(p *Plugin)
 		cse             sysutil.CoreSchedExtendedInterface
 		groupID         string
+		notAnolisOS     bool
 	}
 	type wantFields struct {
 		cookieToPIDs  map[uint64][]uint32
@@ -1245,6 +1251,7 @@ func TestPlugin_SetContainerCookie(t *testing.T) {
 			if tt.fields.prepareFn != nil {
 				tt.fields.prepareFn(helper)
 			}
+			system.HostSystemInfo.IsAnolisOS = !tt.fields.notAnolisOS
 			p := tt.fields.plugin
 			if tt.fields.cse != nil {
 				p.cse = tt.fields.cse
diff --git a/pkg/koordlet/runtimehooks/hooks/coresched/rule_test.go b/pkg/koordlet/runtimehooks/hooks/coresched/rule_test.go
index 60f9eeb9..0df66daf 100644
--- a/pkg/koordlet/runtimehooks/hooks/coresched/rule_test.go
+++ b/pkg/koordlet/runtimehooks/hooks/coresched/rule_test.go
@@ -30,6 +30,7 @@ import (
 	"github.com/koordinator-sh/koordinator/pkg/koordlet/resourceexecutor"
 	"github.com/koordinator-sh/koordinator/pkg/koordlet/statesinformer"
 	"github.com/koordinator-sh/koordinator/pkg/koordlet/util"
+	"github.com/koordinator-sh/koordinator/pkg/koordlet/util/system"
 	sysutil "github.com/koordinator-sh/koordinator/pkg/koordlet/util/system"
 	"github.com/koordinator-sh/koordinator/pkg/util/sloconfig"
 )
@@ -496,6 +497,7 @@ func Test_ruleUpdateCb(t *testing.T) {
 		plugin          *Plugin
 		preparePluginFn func(p *Plugin)
 		cse             sysutil.CoreSchedExtendedInterface
+		notAnolisOS     bool
 	}
 	type wantFields struct {
 		rule               *Rule
@@ -1663,7 +1665,7 @@ func Test_ruleUpdateCb(t *testing.T) {
 				defer close(stopCh)
 				p.executor.Run(stopCh)
 			}
-
+			system.HostSystemInfo.IsAnolisOS = !tt.fields.notAnolisOS
 			gotErr := p.ruleUpdateCb(tt.arg)
 			assert.Equal(t, tt.wantErr, gotErr != nil, gotErr)
 			assert.Equal(t, tt.wantFields.rule, p.rule)
diff --git a/pkg/koordlet/util/system/core_sched_test.go b/pkg/koordlet/util/system/core_sched_test.go
index ea47f137..2c9af669 100644
--- a/pkg/koordlet/util/system/core_sched_test.go
+++ b/pkg/koordlet/util/system/core_sched_test.go
@@ -174,7 +174,8 @@ func TestFakeCoreSchedExtended(t *testing.T) {
 
 func TestEnableCoreSchedIfSupported(t *testing.T) {
 	type fields struct {
-		prepareFn func(helper *FileTestUtil)
+		prepareFn   func(helper *FileTestUtil)
+		notAnolisOS bool
 	}
 	tests := []struct {
 		name   string
@@ -283,6 +284,7 @@ func TestEnableCoreSchedIfSupported(t *testing.T) {
 			if tt.fields.prepareFn != nil {
 				tt.fields.prepareFn(helper)
 			}
+			HostSystemInfo.IsAnolisOS = !tt.fields.notAnolisOS
 			got, got1 := EnableCoreSchedIfSupported()
 			assert.Equal(t, tt.want, got)
 			assert.Equal(t, tt.want1, got1)
diff --git a/pkg/scheduler/frameworkext/job_nominated_pods_test.go b/pkg/scheduler/frameworkext/job_nominated_pods_test.go
new file mode 100644
index 00000000..e377bc98
--- /dev/null
+++ b/pkg/scheduler/frameworkext/job_nominated_pods_test.go
@@ -0,0 +1,314 @@
+/*
+Copyright 2022 The Koordinator Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package frameworkext
+
+import (
+	"context"
+	"testing"
+
+	"github.com/gin-gonic/gin"
+	"github.com/stretchr/testify/assert"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/util/sets"
+	"k8s.io/client-go/informers"
+	kubefake "k8s.io/client-go/kubernetes/fake"
+	"k8s.io/klog/v2"
+	"k8s.io/kubernetes/pkg/scheduler/framework"
+	frameworkfake "k8s.io/kubernetes/pkg/scheduler/framework/fake"
+	"k8s.io/kubernetes/pkg/scheduler/framework/plugins/defaultbinder"
+	"k8s.io/kubernetes/pkg/scheduler/framework/plugins/noderesources"
+	"k8s.io/kubernetes/pkg/scheduler/framework/plugins/queuesort"
+	frameworkruntime "k8s.io/kubernetes/pkg/scheduler/framework/runtime"
+	schedulertesting "k8s.io/kubernetes/pkg/scheduler/testing"
+
+	koordfake "github.com/koordinator-sh/koordinator/pkg/client/clientset/versioned/fake"
+	koordinatorinformers "github.com/koordinator-sh/koordinator/pkg/client/informers/externalversions"
+	"github.com/koordinator-sh/koordinator/pkg/scheduler/frameworkext/services"
+)
+
+type FakeFitPlugin struct {
+}
+
+func (f *FakeFitPlugin) Filter(ctx context.Context, state *framework.CycleState, pod *corev1.Pod, nodeInfo *framework.NodeInfo) *framework.Status {
+	if insufficientResources := noderesources.Fits(pod, nodeInfo); len(insufficientResources) != 0 {
+		var reasons []string
+		for _, insufficientResource := range insufficientResources {
+			reasons = append(reasons, insufficientResource.Reason)
+		}
+		return framework.NewStatus(framework.Unschedulable, reasons...)
+	}
+	return nil
+}
+func (f *FakeFitPlugin) Name() string {
+	return "FakeFitPlugin"
+}
+
+func Test_frameworkExtenderImpl_RunFilterPluginsWithNominatedIgnoreSameJob(t *testing.T) {
+	tests := []struct {
+		name                  string
+		pod                   *corev1.Pod
+		nominatedPods         []*corev1.Pod
+		node                  *corev1.Node
+		nominatedPodsToRemove sets.Set[string]
+		want                  *framework.Status
+	}{
+		{
+			name: "insufficient resource cause nominated pod",
+			pod: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "test-pod-1",
+					Namespace: "default",
+				},
+				Spec: corev1.PodSpec{
+					Containers: []corev1.Container{
+						{
+							Resources: corev1.ResourceRequirements{
+								Requests: corev1.ResourceList{
+									corev1.ResourceCPU: resource.MustParse("16"),
+								},
+							},
+						},
+					},
+				},
+			},
+			nominatedPods: []*corev1.Pod{
+				{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "nominated-pod-1",
+						Namespace: "default",
+						UID:       "nominated-pod-1",
+					},
+					Spec: corev1.PodSpec{
+						Containers: []corev1.Container{
+							{
+								Resources: corev1.ResourceRequirements{
+									Requests: corev1.ResourceList{
+										corev1.ResourceCPU: resource.MustParse("16"),
+									},
+								},
+							},
+						},
+					},
+					Status: corev1.PodStatus{
+						NominatedNodeName: "test-node",
+					},
+				},
+			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+				},
+				Status: corev1.NodeStatus{
+					Allocatable: corev1.ResourceList{
+						corev1.ResourceCPU:  resource.MustParse("16"),
+						corev1.ResourcePods: resource.MustParse("110"),
+					},
+				},
+			},
+			want: framework.NewStatus(framework.Unschedulable, "Insufficient cpu").WithFailedPlugin("FakeFitPlugin"),
+		},
+		{
+			name: "ignore nominated pod",
+			pod: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "test-pod-1",
+					Namespace: "default",
+				},
+				Spec: corev1.PodSpec{
+					Containers: []corev1.Container{
+						{
+							Resources: corev1.ResourceRequirements{
+								Requests: corev1.ResourceList{
+									corev1.ResourceCPU: resource.MustParse("16"),
+								},
+							},
+						},
+					},
+				},
+			},
+			nominatedPods: []*corev1.Pod{
+				{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "nominated-pod-1",
+						Namespace: "default",
+						UID:       "nominated-pod-1",
+					},
+					Spec: corev1.PodSpec{
+						Containers: []corev1.Container{
+							{
+								Resources: corev1.ResourceRequirements{
+									Requests: corev1.ResourceList{
+										corev1.ResourceCPU: resource.MustParse("16"),
+									},
+								},
+							},
+						},
+					},
+					Status: corev1.PodStatus{
+						NominatedNodeName: "test-node",
+					},
+				},
+			},
+			nominatedPodsToRemove: sets.New[string]("nominated-pod-1"),
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+				},
+				Status: corev1.NodeStatus{
+					Allocatable: corev1.ResourceList{
+						corev1.ResourceCPU:  resource.MustParse("16"),
+						corev1.ResourcePods: resource.MustParse("110"),
+					},
+				},
+			},
+			want: nil,
+		},
+		{
+			name: "insufficient resource though ignore nominated pod",
+			pod: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "test-pod-1",
+					Namespace: "default",
+				},
+				Spec: corev1.PodSpec{
+					Containers: []corev1.Container{
+						{
+							Resources: corev1.ResourceRequirements{
+								Requests: corev1.ResourceList{
+									corev1.ResourceCPU: resource.MustParse("16"),
+								},
+							},
+						},
+					},
+				},
+			},
+			nominatedPods: []*corev1.Pod{
+				{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "nominated-pod-1",
+						Namespace: "default",
+						UID:       "nominated-pod-1",
+					},
+					Spec: corev1.PodSpec{
+						Containers: []corev1.Container{
+							{
+								Resources: corev1.ResourceRequirements{
+									Requests: corev1.ResourceList{
+										corev1.ResourceCPU: resource.MustParse("16"),
+									},
+								},
+							},
+						},
+					},
+					Status: corev1.PodStatus{
+						NominatedNodeName: "test-node",
+					},
+				},
+				{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "nominated-pod-2",
+						Namespace: "default",
+						UID:       "nominated-pod-2",
+					},
+					Spec: corev1.PodSpec{
+						Containers: []corev1.Container{
+							{
+								Resources: corev1.ResourceRequirements{
+									Requests: corev1.ResourceList{
+										corev1.ResourceCPU: resource.MustParse("16"),
+									},
+								},
+							},
+						},
+					},
+					Status: corev1.PodStatus{
+						NominatedNodeName: "test-node",
+					},
+				},
+			},
+			nominatedPodsToRemove: sets.New[string]("nominated-pod-1"),
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+				},
+				Status: corev1.NodeStatus{
+					Allocatable: corev1.ResourceList{
+						corev1.ResourceCPU:  resource.MustParse("16"),
+						corev1.ResourcePods: resource.MustParse("110"),
+					},
+				},
+			},
+			want: framework.NewStatus(framework.Unschedulable, "Insufficient cpu").WithFailedPlugin("FakeFitPlugin"),
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			koordClientSet := koordfake.NewSimpleClientset()
+			koordSharedInformerFactory := koordinatorinformers.NewSharedInformerFactory(koordClientSet, 0)
+			extenderFactory, err := NewFrameworkExtenderFactory(
+				WithServicesEngine(services.NewEngine(gin.New())),
+				WithKoordinatorClientSet(koordClientSet),
+				WithKoordinatorSharedInformerFactory(koordSharedInformerFactory),
+			)
+			assert.NoError(t, err)
+			assert.NotNil(t, extenderFactory)
+			assert.Equal(t, koordClientSet, extenderFactory.KoordinatorClientSet())
+			assert.Equal(t, koordSharedInformerFactory, extenderFactory.KoordinatorSharedInformerFactory())
+			registeredPlugins := []schedulertesting.RegisterPluginFunc{
+				schedulertesting.RegisterBindPlugin(defaultbinder.Name, defaultbinder.New),
+				schedulertesting.RegisterQueueSortPlugin(queuesort.Name, queuesort.New),
+				schedulertesting.RegisterFilterPlugin("FakeFitPlugin", func(configuration runtime.Object, f framework.Handle) (framework.Plugin, error) {
+					return &FakeFitPlugin{}, nil
+				}),
+			}
+			fakeClient := kubefake.NewSimpleClientset()
+			sharedInformerFactory := informers.NewSharedInformerFactory(fakeClient, 0)
+
+			nodeInfo := framework.NewNodeInfo()
+			nodeInfo.SetNode(tt.node)
+			fh, err := schedulertesting.NewFramework(
+				context.TODO(),
+				registeredPlugins,
+				"koord-scheduler",
+				frameworkruntime.WithSnapshotSharedLister(fakeNodeInfoLister{NodeInfoLister: frameworkfake.NodeInfoLister{nodeInfo}}),
+				frameworkruntime.WithClientSet(fakeClient),
+				frameworkruntime.WithInformerFactory(sharedInformerFactory),
+				frameworkruntime.WithPodNominator(NewPodNominator()),
+			)
+			ctx := context.TODO()
+			logger := klog.FromContext(ctx)
+			for i := range tt.nominatedPods {
+				nominatedPodInfo, _ := framework.NewPodInfo(tt.nominatedPods[i])
+				fh.AddNominatedPod(logger, nominatedPodInfo, &framework.NominatingInfo{
+					NominatedNodeName: tt.nominatedPods[i].Status.NominatedNodeName,
+					NominatingMode:    framework.ModeOverride,
+				})
+			}
+			assert.NoError(t, err)
+			frameworkExtender := extenderFactory.NewFrameworkExtender(fh)
+			frameworkExtender.SetConfiguredPlugins(fh.ListPlugins())
+			cycleState := framework.NewCycleState()
+			if tt.nominatedPodsToRemove.Len() != 0 {
+				MakeNominatedPodsOfTheSameJob(cycleState, tt.nominatedPodsToRemove.UnsortedList())
+			}
+			assert.Equal(t, tt.want, frameworkExtender.RunFilterPluginsWithNominatedPods(context.TODO(), cycleState, tt.pod, nodeInfo))
+		})
+	}
+}
diff --git a/pkg/scheduler/frameworkext/nominating_info_takeover_test.go b/pkg/scheduler/frameworkext/nominating_info_takeover_test.go
index 77b32f7d..30f00e44 100644
--- a/pkg/scheduler/frameworkext/nominating_info_takeover_test.go
+++ b/pkg/scheduler/frameworkext/nominating_info_takeover_test.go
@@ -38,10 +38,9 @@ func TestTakeoverNominatingInfo(t *testing.T) {
 		name               string
 		args               args
 		wantNominatingInfo *framework.NominatingInfo
-		want               bool
 	}{
 		{
-			name: "job preemption reject waitingPod",
+			name: "job preemption success reject waitingPod",
 			args: args{
 				nominatingInfo: &framework.NominatingInfo{NominatingMode: framework.ModeOverride, NominatedNodeName: ""},
 				podInfo: &framework.QueuedPodInfo{
@@ -56,10 +55,9 @@ func TestTakeoverNominatingInfo(t *testing.T) {
 						},
 					},
 				},
-				status: (framework.NewStatus(framework.Unschedulable)).WithFailedPlugin(JobPreemptionRejectPlugin),
+				status: (framework.NewStatus(framework.Unschedulable)).WithFailedPlugin(JobPreemptionSuccessPlugin),
 			},
 			wantNominatingInfo: &framework.NominatingInfo{NominatingMode: framework.ModeOverride, NominatedNodeName: "test-node"},
-			want:               false,
 		},
 		{
 			name: "job reject waitingPod",
@@ -80,10 +78,29 @@ func TestTakeoverNominatingInfo(t *testing.T) {
 				status: (framework.NewStatus(framework.Unschedulable)).WithFailedPlugin(JobRejectPlugin),
 			},
 			wantNominatingInfo: &framework.NominatingInfo{NominatingMode: framework.ModeNoop},
-			want:               false,
 		},
 		{
-			name: "other plugin",
+			name: "job preemption failure reject waitingPod",
+			args: args{
+				nominatingInfo: &framework.NominatingInfo{NominatingMode: framework.ModeOverride, NominatedNodeName: ""},
+				podInfo: &framework.QueuedPodInfo{
+					PodInfo: &framework.PodInfo{
+						Pod: &corev1.Pod{
+							ObjectMeta: metav1.ObjectMeta{
+								Name: "test-pod",
+							},
+							Spec: corev1.PodSpec{
+								NodeName: "test-node",
+							},
+						},
+					},
+				},
+				status: (framework.NewStatus(framework.Unschedulable)).WithFailedPlugin(JobPreemptionFailurePlugin),
+			},
+			wantNominatingInfo: &framework.NominatingInfo{NominatingMode: framework.ModeOverride, NominatedNodeName: ""},
+		},
+		{
+			name: "other plugin failure",
 			args: args{
 				nominatingInfo: &framework.NominatingInfo{NominatingMode: framework.ModeOverride, NominatedNodeName: ""},
 				podInfo: &framework.QueuedPodInfo{
@@ -101,10 +118,9 @@ func TestTakeoverNominatingInfo(t *testing.T) {
 				status: (framework.NewStatus(framework.Unschedulable)).WithFailedPlugin("other-plugin"),
 			},
 			wantNominatingInfo: &framework.NominatingInfo{NominatingMode: framework.ModeOverride, NominatedNodeName: ""},
-			want:               false,
 		},
 		{
-			name: "other plugin",
+			name: "success",
 			args: args{
 				nominatingInfo: &framework.NominatingInfo{NominatingMode: framework.ModeOverride, NominatedNodeName: ""},
 				podInfo: &framework.QueuedPodInfo{
@@ -122,13 +138,12 @@ func TestTakeoverNominatingInfo(t *testing.T) {
 				status: &framework.Status{},
 			},
 			wantNominatingInfo: &framework.NominatingInfo{NominatingMode: framework.ModeOverride, NominatedNodeName: ""},
-			want:               false,
 		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			assert.Equal(t, tt.want, TakeoverNominatingInfo(context.TODO(), nil, tt.args.podInfo, tt.args.status, tt.args.nominatingInfo, time.Now()))
-			assert.Equal(t, tt.wantNominatingInfo, tt.args.nominatingInfo)
+			nominatingInfo := TakeoverNominatingInfo(context.TODO(), nil, tt.args.podInfo, tt.args.status, tt.args.nominatingInfo, time.Now())
+			assert.Equal(t, tt.wantNominatingInfo, nominatingInfo)
 		})
 	}
 }
diff --git a/pkg/scheduler/plugins/deviceshare/device_plugin_adapter_test.go b/pkg/scheduler/plugins/deviceshare/device_plugin_adapter_test.go
index bb3c7ea5..58809b37 100644
--- a/pkg/scheduler/plugins/deviceshare/device_plugin_adapter_test.go
+++ b/pkg/scheduler/plugins/deviceshare/device_plugin_adapter_test.go
@@ -24,7 +24,9 @@ import (
 
 	"github.com/stretchr/testify/assert"
 	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	kubefake "k8s.io/client-go/kubernetes/fake"
 	fakeclock "k8s.io/utils/clock/testing"
 
 	apiext "github.com/koordinator-sh/koordinator/apis/extension"
@@ -68,7 +70,7 @@ func TestGeneralDevicePluginAdapter_Adapt(t *testing.T) {
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			err := adapter.Adapt(tt.args.object, nil)
+			err := adapter.Adapt(nil, tt.args.object, nil)
 			assert.Equal(t, tt.wantErr, err != nil, err)
 			assert.Equal(t, tt.wantObject, tt.args.object)
 		})
@@ -113,7 +115,7 @@ func TestGeneralGPUDevicePluginAdapter_Adapt(t *testing.T) {
 	adapter := &generalGPUDevicePluginAdapter{}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			err := adapter.Adapt(tt.args.object, tt.args.allocation)
+			err := adapter.Adapt(nil, tt.args.object, tt.args.allocation)
 			assert.Equal(t, tt.wantErr, err != nil, err)
 			assert.Equal(t, tt.wantObject, tt.args.object)
 		})
@@ -190,13 +192,286 @@ func TestHuaweiGPUDevicePluginAdapter_Adapt(t *testing.T) {
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			err := adapter.Adapt(tt.args.object, tt.args.allocation)
+			err := adapter.Adapt(nil, tt.args.object, tt.args.allocation)
 			assert.Equal(t, tt.wantErr, err != nil, err)
 			assert.Equal(t, tt.wantObject, tt.args.object)
 		})
 	}
 }
 
+func TestCambriconGPUDevicePluginAdapter_Adapt(t *testing.T) {
+	now := time.Now()
+
+	type args struct {
+		object     metav1.Object
+		allocation []*apiext.DeviceAllocation
+	}
+	tests := []struct {
+		name       string
+		args       args
+		node       *corev1.Node
+		wantErr    bool
+		wantObject metav1.Object
+		wantNode   *corev1.Node
+	}{
+		{
+			name: "normal case",
+			args: args{
+				object: &corev1.Pod{
+					ObjectMeta: metav1.ObjectMeta{
+						Annotations: map[string]string{},
+					},
+				},
+				allocation: []*apiext.DeviceAllocation{
+					{
+						Minor: 0,
+						Resources: corev1.ResourceList{
+							apiext.ResourceGPUCore:   resource.MustParse("5"),
+							apiext.ResourceGPUMemory: resource.MustParse("1Gi"),
+						},
+					},
+				},
+			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+				},
+			},
+			wantErr: false,
+			wantObject: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						AnnotationCambriconDsmluAssigned: "false",
+						AnnotationCambriconDsmluProfile:  "0_5_4",
+					},
+				},
+			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+					Annotations: map[string]string{
+						AnnotationCambriconDsmluLockTime: now.Format(time.RFC3339),
+					},
+				},
+			},
+		},
+		{
+			name: "multiple gpu share",
+			args: args{
+				object: &corev1.Pod{
+					ObjectMeta: metav1.ObjectMeta{
+						Annotations: map[string]string{},
+					},
+				},
+				allocation: []*apiext.DeviceAllocation{
+					{
+						Minor: 0,
+						Resources: corev1.ResourceList{
+							apiext.ResourceGPUCore:   resource.MustParse("5"),
+							apiext.ResourceGPUMemory: resource.MustParse("1Gi"),
+						},
+					},
+					{
+						Minor: 1,
+						Resources: corev1.ResourceList{
+							apiext.ResourceGPUCore:   resource.MustParse("5"),
+							apiext.ResourceGPUMemory: resource.MustParse("1Gi"),
+						},
+					},
+				},
+			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+				},
+			},
+			wantErr: true,
+			wantObject: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{},
+				},
+			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+				},
+			},
+		},
+		{
+			name: "missing gpu core",
+			args: args{
+				object: &corev1.Pod{
+					ObjectMeta: metav1.ObjectMeta{
+						Annotations: map[string]string{},
+					},
+				},
+				allocation: []*apiext.DeviceAllocation{
+					{
+						Minor: 0,
+						Resources: corev1.ResourceList{
+							apiext.ResourceGPUMemory: resource.MustParse("1Gi"),
+						},
+					},
+				},
+			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+				},
+			},
+			wantErr: true,
+			wantObject: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{},
+				},
+			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+				},
+			},
+		},
+		{
+			name: "too small gpu memory",
+			args: args{
+				object: &corev1.Pod{
+					ObjectMeta: metav1.ObjectMeta{
+						Annotations: map[string]string{},
+					},
+				},
+				allocation: []*apiext.DeviceAllocation{
+					{
+						Minor: 0,
+						Resources: corev1.ResourceList{
+							apiext.ResourceGPUCore:   resource.MustParse("5"),
+							apiext.ResourceGPUMemory: resource.MustParse("128Mi"),
+						},
+					},
+				},
+			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+				},
+			},
+			wantErr: true,
+			wantObject: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{},
+				},
+			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+				},
+			},
+		},
+		{
+			name: "node locked",
+			args: args{
+				object: &corev1.Pod{
+					ObjectMeta: metav1.ObjectMeta{
+						Annotations: map[string]string{},
+					},
+				},
+				allocation: []*apiext.DeviceAllocation{
+					{
+						Minor: 0,
+						Resources: corev1.ResourceList{
+							apiext.ResourceGPUCore:   resource.MustParse("5"),
+							apiext.ResourceGPUMemory: resource.MustParse("1Gi"),
+						},
+					},
+				},
+			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+					Annotations: map[string]string{
+						AnnotationCambriconDsmluLockTime: now.Add(-time.Minute).Format(time.RFC3339),
+					},
+				},
+			},
+			wantErr: true,
+			wantObject: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{},
+				},
+			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+					Annotations: map[string]string{
+						AnnotationCambriconDsmluLockTime: now.Add(-time.Minute).Format(time.RFC3339),
+					},
+				},
+			},
+		},
+		{
+			name: "node lock expired",
+			args: args{
+				object: &corev1.Pod{
+					ObjectMeta: metav1.ObjectMeta{
+						Annotations: map[string]string{},
+					},
+				},
+				allocation: []*apiext.DeviceAllocation{
+					{
+						Minor: 0,
+						Resources: corev1.ResourceList{
+							apiext.ResourceGPUCore:   resource.MustParse("5"),
+							apiext.ResourceGPUMemory: resource.MustParse("1Gi"),
+						},
+					},
+				},
+			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+					Annotations: map[string]string{
+						AnnotationCambriconDsmluLockTime: now.Add(-3 * time.Minute).Format(time.RFC3339),
+					},
+				},
+			},
+			wantErr: false,
+			wantObject: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						AnnotationCambriconDsmluAssigned: "false",
+						AnnotationCambriconDsmluProfile:  "0_5_4",
+					},
+				},
+			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+					Annotations: map[string]string{
+						AnnotationCambriconDsmluLockTime: now.Format(time.RFC3339),
+					},
+				},
+			},
+		},
+	}
+
+	adapter := &cambriconGPUDevicePluginAdapter{
+		clock: fakeclock.NewFakeClock(now),
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			client := kubefake.NewSimpleClientset(tt.node)
+			err := adapter.Adapt(&DevicePluginAdaptContext{
+				Context: context.TODO(),
+				client:  client,
+				node:    tt.node,
+			}, tt.args.object, tt.args.allocation)
+			assert.Equal(t, tt.wantErr, err != nil, err)
+			assert.Equal(t, tt.wantObject, tt.args.object)
+			node, _ := client.CoreV1().Nodes().Get(context.TODO(), tt.node.Name, metav1.GetOptions{})
+			assert.Equal(t, tt.wantNode, node)
+		})
+	}
+}
+
 func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 	now := time.Now()
 	defaultDevicePluginAdapter = &generalDevicePluginAdapter{
@@ -206,6 +481,9 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 		apiext.GPUVendorHuawei: &huaweiGPUDevicePluginAdapter{
 			clock: fakeclock.NewFakeClock(now),
 		},
+		apiext.GPUVendorCambricon: &cambriconGPUDevicePluginAdapter{
+			clock: fakeclock.NewFakeClock(now),
+		},
 	}
 
 	type args struct {
@@ -217,8 +495,10 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 		name       string
 		args       args
 		device     *schedulingv1alpha1.Device
+		node       *corev1.Node
 		wantErr    bool
 		wantObject metav1.Object
+		wantNode   *corev1.Node
 	}{
 		{
 			name: "nvidia",
@@ -243,6 +523,11 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 					},
 				},
 			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "nvidia",
+				},
+			},
 			wantErr: false,
 			wantObject: &corev1.Pod{
 				ObjectMeta: metav1.ObjectMeta{
@@ -252,6 +537,11 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 					},
 				},
 			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "nvidia",
+				},
+			},
 		},
 		{
 			name: "huawei",
@@ -276,6 +566,11 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 					},
 				},
 			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "huawei",
+				},
+			},
 			wantErr: false,
 			wantObject: &corev1.Pod{
 				ObjectMeta: metav1.ObjectMeta{
@@ -287,6 +582,65 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 					},
 				},
 			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "huawei",
+				},
+			},
+		},
+		{
+			name: "cambricon",
+			args: args{
+				object: &corev1.Pod{
+					ObjectMeta: metav1.ObjectMeta{
+						Annotations: map[string]string{},
+					},
+				},
+				allocationResult: apiext.DeviceAllocations{
+					schedulingv1alpha1.GPU: []*apiext.DeviceAllocation{
+						{
+							Minor: 0,
+							Resources: corev1.ResourceList{
+								apiext.ResourceGPUCore:   resource.MustParse("5"),
+								apiext.ResourceGPUMemory: resource.MustParse("1Gi"),
+							},
+						},
+					},
+				},
+				nodeName: "cambricon",
+			},
+			device: &schedulingv1alpha1.Device{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "cambricon",
+					Labels: map[string]string{
+						apiext.LabelGPUVendor: apiext.GPUVendorCambricon,
+					},
+				},
+			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "cambricon",
+				},
+			},
+			wantErr: false,
+			wantObject: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						AnnotationBindTimestamp:          strconv.FormatInt(now.UnixNano(), 10),
+						AnnotationGPUMinors:              "0",
+						AnnotationCambriconDsmluAssigned: "false",
+						AnnotationCambriconDsmluProfile:  "0_5_4",
+					},
+				},
+			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "cambricon",
+					Annotations: map[string]string{
+						AnnotationCambriconDsmluLockTime: now.Format(time.RFC3339),
+					},
+				},
+			},
 		},
 		{
 			name: "unknown vendor",
@@ -311,6 +665,11 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 					},
 				},
 			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test",
+				},
+			},
 			wantErr: false,
 			wantObject: &corev1.Pod{
 				ObjectMeta: metav1.ObjectMeta{
@@ -320,6 +679,11 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 					},
 				},
 			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test",
+				},
+			},
 		},
 		{
 			name: "non-gpu device allocation",
@@ -344,6 +708,11 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 					},
 				},
 			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "nvidia",
+				},
+			},
 			wantErr: false,
 			wantObject: &corev1.Pod{
 				ObjectMeta: metav1.ObjectMeta{
@@ -352,6 +721,11 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 					},
 				},
 			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "nvidia",
+				},
+			},
 		},
 		{
 			name: "device not found",
@@ -376,6 +750,54 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 					},
 				},
 			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test",
+				},
+			},
+			wantErr: true,
+			wantObject: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						AnnotationBindTimestamp: strconv.FormatInt(now.UnixNano(), 10),
+						AnnotationGPUMinors:     "0",
+					},
+				},
+			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test",
+				},
+			},
+		},
+		{
+			name: "node not found",
+			args: args{
+				object: &corev1.Pod{
+					ObjectMeta: metav1.ObjectMeta{
+						Annotations: map[string]string{},
+					},
+				},
+				allocationResult: apiext.DeviceAllocations{
+					schedulingv1alpha1.GPU: []*apiext.DeviceAllocation{
+						{Minor: 0},
+					},
+				},
+				nodeName: "nvidia",
+			},
+			device: &schedulingv1alpha1.Device{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "nvidia",
+					Labels: map[string]string{
+						apiext.LabelGPUVendor: apiext.GPUVendorNVIDIA,
+					},
+				},
+			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test",
+				},
+			},
 			wantErr: true,
 			wantObject: &corev1.Pod{
 				ObjectMeta: metav1.ObjectMeta{
@@ -385,12 +807,17 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 					},
 				},
 			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test",
+				},
+			},
 		},
 	}
 
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			suit := newPluginTestSuit(t, nil)
+			suit := newPluginTestSuit(t, []*corev1.Node{tt.node})
 			if tt.device != nil {
 				_, err := suit.koordClientSet.SchedulingV1alpha1().Devices().Create(context.TODO(), tt.device, metav1.CreateOptions{})
 				assert.NoError(t, err)
@@ -404,9 +831,11 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 			suit.Framework.SharedInformerFactory().WaitForCacheSync(nil)
 			suit.koordinatorSharedInformerFactory.WaitForCacheSync(nil)
 
-			err = pl.(*Plugin).adaptForDevicePlugin(tt.args.object, tt.args.allocationResult, tt.args.nodeName)
+			err = pl.(*Plugin).adaptForDevicePlugin(context.TODO(), tt.args.object, tt.args.allocationResult, tt.args.nodeName)
 			assert.Equal(t, tt.wantErr, err != nil, err)
 			assert.Equal(t, tt.wantObject, tt.args.object)
+			node, _ := suit.ClientSet().CoreV1().Nodes().Get(context.TODO(), tt.node.Name, metav1.GetOptions{})
+			assert.Equal(t, tt.wantNode, node)
 		})
 	}
 }
diff --git a/pkg/scheduler/plugins/deviceshare/plugin_test.go b/pkg/scheduler/plugins/deviceshare/plugin_test.go
index 90b6c2d1..f85a70a6 100644
--- a/pkg/scheduler/plugins/deviceshare/plugin_test.go
+++ b/pkg/scheduler/plugins/deviceshare/plugin_test.go
@@ -128,7 +128,11 @@ func (f *testSharedLister) HavePodsWithRequiredAntiAffinityList() ([]*framework.
 }
 
 func (f *testSharedLister) Get(nodeName string) (*framework.NodeInfo, error) {
-	return f.nodeInfoMap[nodeName], nil
+	nodeInfo, ok := f.nodeInfoMap[nodeName]
+	if !ok {
+		return nil, fmt.Errorf("unable to find node: %s", nodeName)
+	}
+	return nodeInfo, nil
 }
 
 type pluginTestSuit struct {
@@ -4766,7 +4770,7 @@ func Test_Plugin_PreBind(t *testing.T) {
 		t.Run(tt.name, func(t *testing.T) {
 			defer utilfeature.SetFeatureGateDuringTest(t, k8sfeature.DefaultMutableFeatureGate, koordfeatures.DevicePluginAdaption, tt.devicePluginAdaptionFeatureEnabled)()
 
-			suit := newPluginTestSuit(t, nil)
+			suit := newPluginTestSuit(t, []*corev1.Node{{ObjectMeta: metav1.ObjectMeta{Name: "test-node-1"}}})
 			_, err := suit.ClientSet().CoreV1().Pods(testPod.Namespace).Create(context.TODO(), testPod, metav1.CreateOptions{})
 			assert.NoError(t, err)
 			if tt.deviceCR != nil {
diff --git a/pkg/util/utils_test.go b/pkg/util/utils_test.go
index 8d37fb65..313613d6 100644
--- a/pkg/util/utils_test.go
+++ b/pkg/util/utils_test.go
@@ -395,7 +395,7 @@ func Test_GenerateReservationPatchWithUID(t *testing.T) {
 	}
 }
 
-func TestPatch(t *testing.T) {
+func TestPatchPod(t *testing.T) {
 	tests := []struct {
 		name        string
 		originalObj metav1.Object
@@ -705,7 +705,7 @@ func TestPatch(t *testing.T) {
 	}
 }
 
-func TestPatchSafe(t *testing.T) {
+func TestPatchPodSafe(t *testing.T) {
 	tests := []struct {
 		name        string
 		originalObj metav1.Object
@@ -1017,6 +1017,107 @@ func TestPatchSafe(t *testing.T) {
 	}
 }
 
+func Test_GenerateNodePatch(t *testing.T) {
+	node1 := &corev1.Node{
+		ObjectMeta: metav1.ObjectMeta{
+			Name: "test-node-1",
+			UID:  "xxx",
+		},
+	}
+	patchAnnotation := map[string]string{"test_case": "Test_GenerateNodePatch"}
+	node2 := node1.DeepCopy()
+	node2.SetAnnotations(patchAnnotation)
+	patchBytes, err := GenerateNodePatch(node1, node2)
+	if err != nil {
+		t.Errorf("error creating patch bytes %v", err)
+	}
+	var patchMap map[string]interface{}
+	err = json.Unmarshal(patchBytes, &patchMap)
+	if err != nil {
+		t.Errorf("error unmarshalling json patch : %v", err)
+	}
+	metadata, ok := patchMap["metadata"].(map[string]interface{})
+	if !ok {
+		t.Errorf("error converting metadata to version map")
+	}
+	annotation, _ := metadata["annotations"].(map[string]interface{})
+	if fmt.Sprint(annotation) != fmt.Sprint(patchAnnotation) {
+		t.Errorf("expect patchBytes: %q, got: %q", patchAnnotation, annotation)
+	}
+}
+
+func TestPatchNode(t *testing.T) {
+	tests := []struct {
+		name         string
+		originalNode *corev1.Node
+		modifiedNode *corev1.Node
+		wantErr      bool
+	}{
+		{
+			name: "patch node",
+			originalNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+				},
+			},
+			modifiedNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+					Annotations: map[string]string{
+						"testAnnotation": "1",
+					},
+					Labels: map[string]string{
+						"testLabel": "2",
+					},
+				},
+			},
+		},
+		{
+			name: "skipped to patch node",
+			originalNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+					UID:  "xxxxxx",
+					Annotations: map[string]string{
+						"testAnnotation": "1",
+					},
+					Labels: map[string]string{
+						"testLabel": "2",
+					},
+				},
+			},
+			modifiedNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-node",
+					UID:  "xxxxxx",
+					Annotations: map[string]string{
+						"testAnnotation": "1",
+					},
+					Labels: map[string]string{
+						"testLabel": "2",
+					},
+				},
+			},
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			clientSet := kubefake.NewSimpleClientset()
+
+			_, err := clientSet.CoreV1().Nodes().Create(context.TODO(), tt.originalNode, metav1.CreateOptions{})
+			assert.NoError(t, err)
+
+			original, modified := tt.originalNode, tt.modifiedNode
+			_, err = PatchNode(context.TODO(), clientSet, original, modified)
+			assert.Equal(t, tt.wantErr, err != nil, err)
+
+			got, err := clientSet.CoreV1().Nodes().Get(context.TODO(), tt.originalNode.Name, metav1.GetOptions{})
+			assert.NoError(t, err)
+			assert.Equal(t, modified, got)
+		})
+	}
+}
+
 func TestMinFloat64(t *testing.T) {
 	big := 2.0
 	small := 1.0