Repository: koordinator-sh/koordinator

Issue:
[proposal] Provides GPU allocation and utilization metrics
<!-- Please only use this template for submitting proposal -->

**What is your proposal**:

Provides GPU allocation and utilization metrics

**Why is this needed**:

As a cluster administrator, you want a holistic view of the cluster's GPU allocation rate and the GPU utilization of nodes and pods on a single node.
1. Currently, the scheduler writes pod allocation results to pod annotations. To collect GPU allocation statistics, users must understand Koordinator's DeviceAllocations protocol and convert it into Prometheus metrics; this is not available out of the box.
2. Koordlet previously provided a PodDevices Proxy, allowing components like DCGM to access Koordinator's device allocation results through the kubelet's standard API, allowing for pod-level GPU aggregation. However, this is ineffective for GPU sharding scenarios, as DCGM does not expose sharded GPU metrics. Currently, Koordinator primarily integrates HAMi sharding, and it is necessary to provide pod sharding metrics for users who utilize both Koordinator and HAMi sharding.

**Is there a suggested solution, if so, please add it**:

1. The scheduler exposes cache information to Metrics.
2. Can we move HAMi's vgpu-monitor sidecar container to koordlet? we need to discuss a solution with HAMi.


Discussion:
@ZiMengSheng: After some discussion with HAMi, I learned that integrating HAMi's GPU sharding card scheduling capabilities requires the following:
1. When launching the GPU container in NRI, Koordlet mounts a dedicated host monitoring directory and environment variables, instructing the GPU container using HAMi-Core to expose usage information to this directory.
2. HAMi-core distribute integrates the vgpu-monitor sidecar, which mounts the directory set in step 1, collects usage information, and exposes it as metrics.

References:
1. vGPU monitor: https://github.com/Project-HAMi/HAMi/blob/master/charts/hami/templates/device-plugin/daemonsetnvidia.yaml#L100
2. envoronment variable: 

<img width="1144" height="480" alt="Image" src="https://github.com/user-attachments/assets/93e74f00-4411-48a5-a3b4-12abb1f15fe9" />@qinfustu:  The following work needs to be done after testing:
1. The host directory /usr/local/vgpu/containers must be mounted into the koordlet container. This enables koordlet to create per-container directories on the host during Pod startup, following the pattern: /usr/local/vgpu/containers/{podUID}_{containerName}.
2. Add the label hami.io/vgpu-node={nodeName} to Pods. This allows ​​vGPUmonitor​​ to filter and collect metrics exclusively from Pods using vGPU resources.

<img width="1235" height="379" alt="Image" src="https://github.com/user-attachments/assets/3618c889-5a5f-44c9-ad7e-0fd2a0ec2092" />@ZiMengSheng: 1. After specifying the environment variables, will HAMi-core not automatically create the corresponding directory?
2. OK, we can do it in the scheduler PreBind.@qinfustu: > 1. After specifying the environment variables, will HAMi-core not automatically create the corresponding directory?

In the HAMi project, hami-core functions solely as a runtime enhancement — it activates only during Pod Running state when GPU calls occur, with no container configuration logic involved. Directory creation is handled exclusively by hami-device-plugin.@ZiMengSheng: > > 1. After specifying the environment variables, will HAMi-core not automatically create the corresponding directory?
> 
> In the HAMi project, hami-core functions solely as a runtime enhancement — it activates only during Pod Running state when GPU calls occur, with no container configuration logic involved. Directory creation is handled exclusively by hami-device-plugin.

OK@ZiMengSheng: @qinfustu Will you achieve it?@qinfustu: > @qinfustu Will you achieve it?​​

Yes, I will.​​

Solution patch:
diff --git a/apis/extension/device_share.go b/apis/extension/device_share.go
index 8060a75b..53873a7a 100644
--- a/apis/extension/device_share.go
+++ b/apis/extension/device_share.go
@@ -66,6 +66,7 @@ const (
 	LabelSecondaryDeviceWellPlanned string = NodeDomainPrefix + "/secondary-device-well-planned"
 
 	LabelGPUIsolationProvider = DomainPrefix + "gpu-isolation-provider"
+	LabelHAMIVGPUNodeName     = "hami.io/vgpu-node"
 )
 
 const (
diff --git a/config/manager/koordlet.yaml b/config/manager/koordlet.yaml
index 09c92b5e..770d8344 100644
--- a/config/manager/koordlet.yaml
+++ b/config/manager/koordlet.yaml
@@ -95,6 +95,8 @@ spec:
             - mountPath: /var/run/koordlet/xpu-device-infos/
               mountPropagation: Bidirectional
               name: xpu-device-infos
+            - mountPath: /usr/local/vgpu/containers
+              name: host-vgpu-containers
       hostNetwork: true
       hostPID: true
       restartPolicy: Always
@@ -162,3 +164,7 @@ spec:
             path: /var/run/koordlet/xpu-device-infos/
             type: DirectoryOrCreate
           name: xpu-device-infos
+        - hostPath:
+            path: /usr/local/vgpu/containers
+            type: DirectoryOrCreate
+          name: host-vgpu-containers
\ No newline at end of file
diff --git a/pkg/features/koordlet_features.go b/pkg/features/koordlet_features.go
index de88eed0..c64f946e 100644
--- a/pkg/features/koordlet_features.go
+++ b/pkg/features/koordlet_features.go
@@ -160,6 +160,12 @@ const (
 	// PodResourcesProxy enabled hooked podResources of kubelet provided by koordlet.
 	// It provides a grpc service to enable discovery of pod resources allocated by koordinator system.
 	PodResourcesProxy featuregate.Feature = "PodResourcesProxy"
+
+	// owner: @qinfustu
+	// alpha v1.7
+	//
+	// HamiCoreVGPUMonitor enables the vGPU monitoring for HAMi-core.
+	HamiCoreVGPUMonitor featuregate.Feature = "HamiCoreVGPUMonitor"
 )
 
 func init() {
@@ -192,6 +198,7 @@ var (
 		ColdPageCollector:      {Default: false, PreRelease: featuregate.Alpha},
 		HugePageReport:         {Default: false, PreRelease: featuregate.Alpha},
 		PodResourcesProxy:      {Default: false, PreRelease: featuregate.Alpha},
+		HamiCoreVGPUMonitor:    {Default: false, PreRelease: featuregate.Alpha},
 	}
 )
 
diff --git a/pkg/koordlet/runtimehooks/hooks/gpu/gpu.go b/pkg/koordlet/runtimehooks/hooks/gpu/gpu.go
index 25c2839f..e7a498a8 100644
--- a/pkg/koordlet/runtimehooks/hooks/gpu/gpu.go
+++ b/pkg/koordlet/runtimehooks/hooks/gpu/gpu.go
@@ -18,12 +18,15 @@ package gpu
 
 import (
 	"fmt"
+	"os"
+	"path/filepath"
 	"strings"
 
 	"k8s.io/klog/v2"
 
 	ext "github.com/koordinator-sh/koordinator/apis/extension"
 	schedulingv1alpha1 "github.com/koordinator-sh/koordinator/apis/scheduling/v1alpha1"
+	"github.com/koordinator-sh/koordinator/pkg/features"
 	"github.com/koordinator-sh/koordinator/pkg/koordlet/runtimehooks/hooks"
 	"github.com/koordinator-sh/koordinator/pkg/koordlet/runtimehooks/protocol"
 	"github.com/koordinator-sh/koordinator/pkg/koordlet/util/system"
@@ -88,6 +91,7 @@ func (p *gpuPlugin) InjectContainerGPUEnv(proto protocol.HooksProtocol) error {
 				containerCtx.Response.AddContainerEnvs["CUDA_DEVICE_SM_LIMIT"] = fmt.Sprintf("%d", gpuCore.Value())
 			}
 			containerCtx.Response.AddContainerEnvs["LD_PRELOAD"] = system.Conf.HAMICoreLibraryDirectoryPath
+
 			containerCtx.Response.AddContainerMounts = append(containerCtx.Response.AddContainerMounts,
 				&protocol.Mount{
 					Destination: system.Conf.HAMICoreLibraryDirectoryPath,
@@ -103,6 +107,25 @@ func (p *gpuPlugin) InjectContainerGPUEnv(proto protocol.HooksProtocol) error {
 					Options:     []string{"rbind"},
 				},
 			)
+
+			if features.DefaultKoordletFeatureGate.Enabled(features.HamiCoreVGPUMonitor) {
+				hamiDirPath := filepath.Dir(system.Conf.HAMICoreLibraryDirectoryPath)
+				containerCtx.Response.AddContainerEnvs["CUDA_DEVICE_MEMORY_SHARED_CACHE"] = fmt.Sprintf("%s/%s_%s.cache", hamiDirPath, containerReq.PodMeta.UID, containerReq.ContainerMeta.Name)
+				cacheFileHostDirectory := fmt.Sprintf("%s/containers/%s_%s", hamiDirPath, containerReq.PodMeta.UID, containerReq.ContainerMeta.Name)
+				// TODO: Move this operation into the pkg resource-executor.​
+				klog.V(5).Infof("​​create a vgpu monitoring data directory [%s] and grant it 0777 permissions", cacheFileHostDirectory)
+				os.RemoveAll(cacheFileHostDirectory)
+				os.MkdirAll(cacheFileHostDirectory, 0777)
+				os.Chmod(cacheFileHostDirectory, 0777)
+				containerCtx.Response.AddContainerMounts = append(containerCtx.Response.AddContainerMounts,
+					&protocol.Mount{
+						Destination: hamiDirPath,
+						Type:        "bind",
+						Source:      cacheFileHostDirectory,
+						Options:     []string{"rbind"},
+					},
+				)
+			}
 		}
 	}
 
diff --git a/pkg/scheduler/plugins/deviceshare/device_plugin_adapter.go b/pkg/scheduler/plugins/deviceshare/device_plugin_adapter.go
index 63b81063..dfcad970 100644
--- a/pkg/scheduler/plugins/deviceshare/device_plugin_adapter.go
+++ b/pkg/scheduler/plugins/deviceshare/device_plugin_adapter.go
@@ -105,16 +105,22 @@ func (p *Plugin) adaptForDevicePlugin(ctx context.Context, object metav1.Object,
 	}
 
 	if gpuAllocation, ok := allocationResult[schedulingv1alpha1.GPU]; ok {
-		if err := defaultGPUDevicePluginAdapter.Adapt(nil, object, gpuAllocation); err != nil {
-			return err
-		}
-
 		nodeInfo, err := p.handle.SnapshotSharedLister().NodeInfos().Get(nodeName)
 		if err != nil {
 			klog.ErrorS(err, "Failed to get NodeInfo", "node", nodeName)
 			return err
 		}
 
+		adaptCtx := &DevicePluginAdaptContext{
+			Context: ctx,
+			client:  p.handle.ClientSet(),
+			node:    nodeInfo.Node(),
+		}
+
+		if err := defaultGPUDevicePluginAdapter.Adapt(adaptCtx, object, gpuAllocation); err != nil {
+			return err
+		}
+
 		extendedHandle, ok := p.handle.(frameworkext.ExtendedHandle)
 		if !ok {
 			return fmt.Errorf("expect handle to be type frameworkext.ExtendedHandle, got %T", p.handle)
@@ -126,11 +132,6 @@ func (p *Plugin) adaptForDevicePlugin(ctx context.Context, object metav1.Object,
 			return err
 		}
 
-		adaptCtx := &DevicePluginAdaptContext{
-			Context: ctx,
-			client:  p.handle.ClientSet(),
-			node:    nodeInfo.Node(),
-		}
 		vendor := device.Labels[apiext.LabelGPUVendor]
 		if adapter, ok := gpuDevicePluginAdapterMap[vendor]; ok {
 			if err := adapter.Adapt(adaptCtx, object, gpuAllocation); err != nil {
@@ -158,8 +159,11 @@ func (a *generalDevicePluginAdapter) Adapt(_ *DevicePluginAdaptContext, object m
 type generalGPUDevicePluginAdapter struct {
 }
 
-func (a *generalGPUDevicePluginAdapter) Adapt(_ *DevicePluginAdaptContext, object metav1.Object, allocation []*apiext.DeviceAllocation) error {
+func (a *generalGPUDevicePluginAdapter) Adapt(ctx *DevicePluginAdaptContext, object metav1.Object, allocation []*apiext.DeviceAllocation) error {
 	object.GetAnnotations()[AnnotationGPUMinors] = buildGPUMinorsStr(allocation)
+	if object.GetLabels()[apiext.LabelGPUIsolationProvider] == string(apiext.GPUIsolationProviderHAMICore) {
+		object.GetLabels()[apiext.LabelHAMIVGPUNodeName] = ctx.node.Name
+	}
 	return nil
 }
 



Test patch:
diff --git a/pkg/koordlet/runtimehooks/hooks/gpu/gpu_test.go b/pkg/koordlet/runtimehooks/hooks/gpu/gpu_test.go
index e1ae5e53..38780bed 100644
--- a/pkg/koordlet/runtimehooks/hooks/gpu/gpu_test.go
+++ b/pkg/koordlet/runtimehooks/hooks/gpu/gpu_test.go
@@ -22,6 +22,7 @@ import (
 	"github.com/stretchr/testify/assert"
 
 	ext "github.com/koordinator-sh/koordinator/apis/extension"
+	"github.com/koordinator-sh/koordinator/pkg/features"
 	"github.com/koordinator-sh/koordinator/pkg/koordlet/runtimehooks/protocol"
 )
 
@@ -71,6 +72,12 @@ func Test_InjectContainerGPUEnv(t *testing.T) {
 			expectedError:    false,
 			proto: &protocol.ContainerContext{
 				Request: protocol.ContainerRequest{
+					PodMeta: protocol.PodMeta{
+						UID: "pod-uid",
+					},
+					ContainerMeta: protocol.ContainerMeta{
+						Name: "container-name",
+					},
 					PodLabels: map[string]string{
 						ext.LabelGPUIsolationProvider: string(ext.GPUIsolationProviderHAMICore),
 					},
@@ -80,10 +87,11 @@ func Test_InjectContainerGPUEnv(t *testing.T) {
 				},
 			},
 			expectedEnvs: map[string]string{
-				GpuAllocEnv:                "1",
-				"CUDA_DEVICE_MEMORY_LIMIT": "17179869184",
-				"CUDA_DEVICE_SM_LIMIT":     "50",
-				"LD_PRELOAD":               "/usr/local/vgpu/libvgpu.so",
+				GpuAllocEnv:                       "1",
+				"CUDA_DEVICE_MEMORY_LIMIT":        "17179869184",
+				"CUDA_DEVICE_SM_LIMIT":            "50",
+				"LD_PRELOAD":                      "/usr/local/vgpu/libvgpu.so",
+				"CUDA_DEVICE_MEMORY_SHARED_CACHE": "/usr/local/vgpu/pod-uid_container-name.cache",
 			},
 			expectedMounts: []*protocol.Mount{
 				{
@@ -98,17 +106,31 @@ func Test_InjectContainerGPUEnv(t *testing.T) {
 					Source:      "/tmp/vgpulock",
 					Options:     []string{"rbind"},
 				},
+				{
+					Destination: "/usr/local/vgpu",
+					Type:        "bind",
+					Source:      "/usr/local/vgpu/containers/pod-uid_container-name",
+					Options:     []string{"rbind"},
+				},
 			},
 		},
 	}
 	plugin := gpuPlugin{}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
+			testFeatureGates := map[string]bool{string(features.HamiCoreVGPUMonitor): true}
+			err := features.DefaultMutableKoordletFeatureGate.SetFromMap(testFeatureGates)
+			assert.NoError(t, err)
+			defer func() {
+				err = features.DefaultMutableKoordletFeatureGate.SetFromMap(testFeatureGates)
+				assert.NoError(t, err)
+			}()
+
 			var containerCtx *protocol.ContainerContext
 			if tt.proto != nil {
 				containerCtx = tt.proto.(*protocol.ContainerContext)
 			}
-			err := plugin.InjectContainerGPUEnv(containerCtx)
+			err = plugin.InjectContainerGPUEnv(containerCtx)
 			assert.Equal(t, tt.expectedError, err != nil, tt.name)
 			if tt.proto != nil {
 				containerCtx := tt.proto.(*protocol.ContainerContext)
diff --git a/pkg/scheduler/plugins/deviceshare/device_plugin_adapter_test.go b/pkg/scheduler/plugins/deviceshare/device_plugin_adapter_test.go
index 58809b37..0b465d3e 100644
--- a/pkg/scheduler/plugins/deviceshare/device_plugin_adapter_test.go
+++ b/pkg/scheduler/plugins/deviceshare/device_plugin_adapter_test.go
@@ -803,7 +803,6 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 				ObjectMeta: metav1.ObjectMeta{
 					Annotations: map[string]string{
 						AnnotationBindTimestamp: strconv.FormatInt(now.UnixNano(), 10),
-						AnnotationGPUMinors:     "0",
 					},
 				},
 			},
@@ -813,6 +812,56 @@ func TestPlugin_adaptForDevicePlugin(t *testing.T) {
 				},
 			},
 		},
+		{
+			name: "hami-core",
+			args: args{
+				object: &corev1.Pod{
+					ObjectMeta: metav1.ObjectMeta{
+						Labels: map[string]string{
+							apiext.LabelGPUIsolationProvider: string(apiext.GPUIsolationProviderHAMICore),
+						},
+						Annotations: map[string]string{},
+					},
+				},
+				allocationResult: apiext.DeviceAllocations{
+					schedulingv1alpha1.GPU: []*apiext.DeviceAllocation{
+						{Minor: 0},
+					},
+				},
+				nodeName: "hami-core",
+			},
+			device: &schedulingv1alpha1.Device{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "hami-core",
+					Labels: map[string]string{
+						apiext.LabelGPUVendor: apiext.GPUVendorNVIDIA,
+					},
+				},
+			},
+			node: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "hami-core",
+				},
+			},
+			wantErr: false,
+			wantObject: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						apiext.LabelGPUIsolationProvider: string(apiext.GPUIsolationProviderHAMICore),
+						apiext.LabelHAMIVGPUNodeName:     "hami-core",
+					},
+					Annotations: map[string]string{
+						AnnotationBindTimestamp: strconv.FormatInt(now.UnixNano(), 10),
+						AnnotationGPUMinors:     "0",
+					},
+				},
+			},
+			wantNode: &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "hami-core",
+				},
+			},
+		},
 	}
 
 	for _, tt := range tests {
diff --git a/pkg/scheduler/plugins/deviceshare/plugin_test.go b/pkg/scheduler/plugins/deviceshare/plugin_test.go
index f85a70a6..b2cecea9 100644
--- a/pkg/scheduler/plugins/deviceshare/plugin_test.go
+++ b/pkg/scheduler/plugins/deviceshare/plugin_test.go
@@ -4765,6 +4765,42 @@ func Test_Plugin_PreBind(t *testing.T) {
 				},
 			},
 		},
+		{
+			name: "has the LabelGPUIsolationProvider label",
+			args: args{
+				pod: &corev1.Pod{
+					ObjectMeta: metav1.ObjectMeta{
+						Labels: map[string]string{
+							apiext.LabelGPUIsolationProvider: string(apiext.GPUIsolationProviderHAMICore),
+						},
+					},
+				},
+
+				state: &preFilterState{
+					skip: false,
+					allocationResult: apiext.DeviceAllocations{
+						schedulingv1alpha1.GPU: {
+							{Minor: 0},
+						},
+					},
+				},
+			},
+			deviceCR:                           fakeDeviceCR,
+			devicePluginAdaptionFeatureEnabled: true,
+			wantPod: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						apiext.LabelHAMIVGPUNodeName:     "test-node-1",
+						apiext.LabelGPUIsolationProvider: string(apiext.GPUIsolationProviderHAMICore),
+					},
+					Annotations: map[string]string{
+						apiext.AnnotationDeviceAllocated: `{"gpu":[{"minor":0,"resources":null,"id":"GPU-8c25ea37-2909-6e62-b7bf-e2fcadebea8d"}]}`,
+						AnnotationBindTimestamp:          strconv.FormatInt(now.UnixNano(), 10),
+						AnnotationGPUMinors:              "0",
+					},
+				},
+			},
+		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {