Repository: kserve/kserve
Pull request #4303: Refactor: Consolidate duplicate custom gpu type checking logic
Author: login: Jooho
Created: 2025-03-11T23:52:23Z | Merged: 2025-03-25T18:43:01Z

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://www.kubeflow.org/docs/about/contributing/ and developer guide https://github.com/kserve/kserve/blob/master/docs/DEVELOPER_GUIDE.md
2. Before raising a PR, please run `make go-lint` and `make py-fmt` to check the code style.
3. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
4. Follow the instructions for writing a release note: https://git.k8s.io/community/contributors/guide/release-notes.md
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

**What this PR does / why we need it**:

- Add 'multiNode' Field to inferenceService-config:
  - A new field, multiNode, has been added to the inferenceService-config ConfigMap. This field is intended to support multi-node setups, providing more flexibility in GPU resource management within multi-node runtime pods.
- Refactor: Consolidate Duplicate Custom GPU Type Checking Logic:
  - The previous implementation involved repetitive code for checking custom GPU resource types. In this update, this duplicate logic has been refactored into a reusable function.
  - The refactor improves code readability and reduces redundancy.
  - Additionally, unit tests have been added to verify the functionality of the newly introduced functions an updated existing unit tests.


**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes # part of https://github.com/kserve/kserve/issues/4228

**Type of changes**
Please delete options that are not relevant.

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] This change requires a documentation update

**Feature/Issue validation/testing**:

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

Setting
- Deploy Kserve
- Create a test namespace "vllm-multinode"

- [X] Test A -1(unit test included)

custom GPU type without annotation
```
cat <<EOF| oc apply -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    serving.kserve.io/autoscalerClass: external
    serving.kserve.io/deploymentMode: RawDeployment
  name: vllm-llama3-8b
  namespace: vllm-multinode
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      name: ""
      resources: 
        requests:
	  custom.com/gpu: 1         
      runtime: kserve-huggingfaceserver-multinode
      storageUri: pvc://llama-3-8b-pvc/hf/8b_instruction_tuned
    workerSpec: 
      pipelineParallelSize: 2
      tensorParallelSize: 1
EOF
```
**Expected result: denied to create isvc by webhook**

- [X] Test A-2 (unit test included)
custom GPU type with annotation
```
cat <<EOF| kubectl apply -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    serving.kserve.io/autoscalerClass: external
    serving.kserve.io/deploymentMode: RawDeployment
    serving.kserve.io//gpu-resource-types: '["custom.com/gpu"]
  name: vllm-llama3-8b
  namespace: vllm-multinode
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      name: ""
      resources: 
        requests:
	  custom.com/gpu: 1         
      runtime: kserve-huggingfaceserver-multinode
      storageUri: pvc://llama-3-8b-pvc/hf/8b_instruction_tuned
    workerSpec: 
      pipelineParallelSize: 2
      tensorParallelSize: 1
EOF
```
**Expected result: allowed to create isvc by webhook**

clean up
```
kubectl delete isvc --all -n vllm-multinode
```

- [X] Test B - with inferenceservice-config configmap (unit test included)

update configmap
```
kubectl edit  cm inferenceservice-config -n kserve
..
   multiNode: |-
      {
        "customGPUResourceTypeList": [
          "custom.com/gpu"
        ]
      }
..

cat <<EOF| kubectl apply -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    serving.kserve.io/autoscalerClass: external
    serving.kserve.io/deploymentMode: RawDeployment
  name: vllm-llama3-8b
  namespace: vllm-multinode
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      name: ""
      resources: 
        requests:
	  custom.com/gpu: 1         
      runtime: kserve-huggingfaceserver-multinode
      storageUri: pvc://llama-3-8b-pvc/hf/8b_instruction_tuned
    workerSpec: 
      pipelineParallelSize: 2
      tensorParallelSize: 1
EOF
```
**Expected result: allowed to create isvc by webhook**

- Logs

**Special notes for your reviewer**:

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

**Checklist**:

- [ ] Have you added unit/e2e tests that prove your fix is effective or that this feature works?
- [ ] Has code been commented, particularly in hard-to-understand areas?
- [ ] Have you made corresponding changes to the documentation?

**Release note**:
<!--  Write your release note:
1. Enter your extended release note in the below block. If the PR requires additional action from users switching to the new release, include the string "action required".
2. If no release note is required, just write "NONE".
-->
```release-note

```

**Re-running failed tests**

- `/rerun-all` - rerun all failed workflows.
- `/rerun-workflow <workflow name>` - rerun a specific failed workflow. Only one workflow name can be specified. Multiple /rerun-workflow commands are allowed per comment.


Diff:
diff --git a/cmd/manager/main.go b/cmd/manager/main.go
index 15b1cbff059..5dbfa2f997b 100644
--- a/cmd/manager/main.go
+++ b/cmd/manager/main.go
@@ -167,6 +167,13 @@ func main() {
 		os.Exit(1)
 	}
 
+	// Update Global GPU Resource Type List when custom GPU resource types are provided
+	_, err = v1beta1.NewMultiNodeConfig(isvcConfigMap)
+	if err != nil {
+		setupLog.Error(err, "unable to get multiNode config.")
+		os.Exit(1)
+	}
+
 	ksvcFound, ksvcCheckErr := utils.IsCrdAvailable(cfg, knservingv1.SchemeGroupVersion.String(), constants.KnativeServiceKind)
 	if ksvcCheckErr != nil {
 		setupLog.Error(ksvcCheckErr, "error when checking if Knative Service kind is available")
diff --git a/config/configmap/inferenceservice.yaml b/config/configmap/inferenceservice.yaml
index 1c9cde1ffcf..3d368b31e90 100644
--- a/config/configmap/inferenceservice.yaml
+++ b/config/configmap/inferenceservice.yaml
@@ -73,6 +73,24 @@ data:
           "my.custom.label.1"  
         ]
       }  
+    # ====================================== MultiNode CONFIGURATION ======================================
+    # Example   
+    multiNode: |-
+      {
+        "customGPUResourceTypeList": [
+          "custom.com/gpu"
+        ]
+      }
+    # Example of multinode configuration
+    multiNode: |-
+      {      
+        # CustomGPUResourceTypeList is a list of custom GPU resource types intended to identify the GPU type of a resource,
+        # not to restrict the user from using a specific GPU type.
+        # The MultiNode runtime pod will dynamically add GPU resources based on the registered GPU types.
+        "customGPUResourceTypeList": [
+          "custom.com/gpu"
+        ]
+      }  
      # ====================================== STORAGE INITIALIZER CONFIGURATION ======================================
      # Example
      storageInitializer: |-
@@ -116,7 +134,7 @@ data:
            # if pvc volume is provided in storageuri then the pvc volume is directly mounted to /mnt/models in the user container.
            # rather than symlink it to a shared volume. For more info see https://github.com/kserve/kserve/issues/2737
            "enableDirectPvcVolumeMount": true,
-    
+
            # enableModelcar enabled allows you to directly access an OCI container image by
            # using a source URL with an "oci://" schema.
            "enableModelcar": false,
@@ -482,7 +500,7 @@ data:
          # ModelMesh https://kserve.github.io/website/master/admin/modelmesh/
          "defaultDeploymentMode": "Serverless"
        }
-    
+
      # ====================================== SERVICE CONFIGURATION ======================================
      # Example
      service: |-
@@ -686,12 +704,12 @@ data:
       "defaultJobImage" : "kserve/storage-initializer:latest",
       "fsGroup": 1000
     }
-  
+
   security: |-
     {
       "autoMountServiceAccountToken": true
     }
-    
+
   inferenceService: |-
     {
       "resource": {
diff --git a/hack/violation_exceptions.list b/hack/violation_exceptions.list
index dae70f3ec0b..2a6a869f148 100644
--- a/hack/violation_exceptions.list
+++ b/hack/violation_exceptions.list
@@ -18,6 +18,7 @@ API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/
 API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,InferenceServicesConfig,ServiceAnnotationDisallowedList
 API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,InferenceServicesConfig,ServiceLabelDisallowedList
 API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,LoggerSpec,MetadataHeaders
+API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,MultiNodeConfig,CustomGPUResourceTypeList
 API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,PodSpec,Containers
 API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,PodSpec,EphemeralContainers
 API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,PodSpec,HostAliases
diff --git a/pkg/apis/serving/v1beta1/configmap.go b/pkg/apis/serving/v1beta1/configmap.go
index 7ea92433da3..f22d7621197 100644
--- a/pkg/apis/serving/v1beta1/configmap.go
+++ b/pkg/apis/serving/v1beta1/configmap.go
@@ -30,6 +30,7 @@ import (
 	"k8s.io/client-go/kubernetes"
 
 	"github.com/kserve/kserve/pkg/constants"
+	"github.com/kserve/kserve/pkg/utils"
 )
 
 // ConfigMap Keys
@@ -42,6 +43,7 @@ const (
 	SecurityConfigName            = "security"
 	ServiceConfigName             = "service"
 	ResourceConfigName            = "resource"
+	MultiNodeConfigKeyName        = "multiNode"
 )
 
 const (
@@ -84,6 +86,12 @@ type InferenceServicesConfig struct {
 	Resource ResourceConfig `json:"resource,omitempty"`
 }
 
+// +kubebuilder:object:generate=false
+type MultiNodeConfig struct {
+	// CustomGPUResourceTypeList is a list of custom GPU resource types that are allowed to be used in the ServingRuntime and inferenceService
+	CustomGPUResourceTypeList []string `json:"customGPUResourceTypeList,omitempty"`
+}
+
 // +kubebuilder:object:generate=false
 type IngressConfig struct {
 	EnableGatewayAPI           bool      `json:"enableGatewayApi,omitempty"`
@@ -181,6 +189,25 @@ func NewInferenceServicesConfig(isvcConfigMap *corev1.ConfigMap) (*InferenceServ
 	return icfg, nil
 }
 
+func NewMultiNodeConfig(isvcConfigMap *corev1.ConfigMap) (*MultiNodeConfig, error) {
+	mncfg := &MultiNodeConfig{}
+	for _, err := range []error{
+		getComponentConfig(MultiNodeConfigKeyName, isvcConfigMap, &mncfg),
+	} {
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	if mncfg.CustomGPUResourceTypeList == nil {
+		mncfg.CustomGPUResourceTypeList = []string{}
+	}
+
+	// update global GPU resource type list
+	utils.UpdateGlobalGPUResourceTypeList(append(mncfg.CustomGPUResourceTypeList, constants.DefaultGPUResourceTypeList...))
+	return mncfg, nil
+}
+
 func validateIngressGateway(ingressConfig *IngressConfig) error {
 	if ingressConfig.KserveIngressGateway == "" {
 		return errors.New(ErrKserveIngressGatewayRequired)
diff --git a/pkg/apis/serving/v1beta1/configmap_test.go b/pkg/apis/serving/v1beta1/configmap_test.go
index 942ad3c2410..2b535f9783e 100644
--- a/pkg/apis/serving/v1beta1/configmap_test.go
+++ b/pkg/apis/serving/v1beta1/configmap_test.go
@@ -64,6 +64,14 @@ var (
 		"serviceAnnotationDisallowedList": %s,
 		"serviceLabelDisallowedList": %s
 	}`, []string{}, []string{})
+
+	MultiNodeConfigData = `{
+		"customGPUResourceTypeList": [
+			"custom.com/gpu-1",
+			"custom.com/gpu-2"
+		]
+	}`
+	MultiNodeConfigNoData = `{}`
 )
 
 func TestNewInferenceServiceConfig(t *testing.T) {
@@ -78,6 +86,61 @@ func TestNewInferenceServiceConfig(t *testing.T) {
 	g.Expect(isvcConfig).ShouldNot(gomega.BeNil())
 }
 
+func TestNewMultiNodeConfigWithNoData(t *testing.T) {
+	g := gomega.NewGomegaWithT(t)
+	clientset := fakeclientset.NewSimpleClientset(&corev1.ConfigMap{
+		ObjectMeta: metav1.ObjectMeta{Name: constants.InferenceServiceConfigMapName, Namespace: constants.KServeNamespace},
+		Data: map[string]string{
+			MultiNodeConfigKeyName: MultiNodeConfigNoData,
+		},
+	})
+
+	configMap, err := GetInferenceServiceConfigMap(context.Background(), clientset)
+	g.Expect(err).ShouldNot(gomega.HaveOccurred())
+
+	multiNodeCfg, err := NewMultiNodeConfig(configMap)
+	g.Expect(err).ShouldNot(gomega.HaveOccurred())
+	g.Expect(multiNodeCfg).ShouldNot(gomega.BeNil())
+	g.Expect(multiNodeCfg.CustomGPUResourceTypeList).To(gomega.Equal([]string{}))
+	g.Expect(constants.DefaultGPUResourceTypeList).To(gomega.Equal([]string{"nvidia.com/gpu", "amd.com/gpu", "intel.com/gpu", "habana.ai/gaudi"}))
+}
+
+func TestNewMultiNodeConfigWithoutData(t *testing.T) {
+	g := gomega.NewGomegaWithT(t)
+	clientset := fakeclientset.NewSimpleClientset(&corev1.ConfigMap{
+		ObjectMeta: metav1.ObjectMeta{Name: constants.InferenceServiceConfigMapName, Namespace: constants.KServeNamespace},
+		Data:       map[string]string{},
+	})
+
+	configMap, err := GetInferenceServiceConfigMap(context.Background(), clientset)
+	g.Expect(err).ShouldNot(gomega.HaveOccurred())
+
+	multiNodeCfg, err := NewMultiNodeConfig(configMap)
+	g.Expect(err).ShouldNot(gomega.HaveOccurred())
+	g.Expect(multiNodeCfg).ShouldNot(gomega.BeNil())
+	g.Expect(multiNodeCfg.CustomGPUResourceTypeList).To(gomega.Equal([]string{}))
+	g.Expect(constants.DefaultGPUResourceTypeList).To(gomega.Equal([]string{"nvidia.com/gpu", "amd.com/gpu", "intel.com/gpu", "habana.ai/gaudi"}))
+}
+
+func TestNewMultiNodeConfigWithData(t *testing.T) {
+	g := gomega.NewGomegaWithT(t)
+	clientset := fakeclientset.NewSimpleClientset(&corev1.ConfigMap{
+		ObjectMeta: metav1.ObjectMeta{Name: constants.InferenceServiceConfigMapName, Namespace: constants.KServeNamespace},
+		Data: map[string]string{
+			MultiNodeConfigKeyName: MultiNodeConfigData,
+		},
+	})
+
+	configMap, err := GetInferenceServiceConfigMap(context.Background(), clientset)
+	g.Expect(err).ShouldNot(gomega.HaveOccurred())
+
+	multiNodeCfg, err := NewMultiNodeConfig(configMap)
+	g.Expect(err).ShouldNot(gomega.HaveOccurred())
+	g.Expect(multiNodeCfg).ShouldNot(gomega.BeNil())
+	g.Expect(multiNodeCfg.CustomGPUResourceTypeList).To(gomega.Equal([]string{"custom.com/gpu-1", "custom.com/gpu-2"}))
+	g.Expect(constants.DefaultGPUResourceTypeList).To(gomega.Equal([]string{"nvidia.com/gpu", "amd.com/gpu", "intel.com/gpu", "habana.ai/gaudi", "custom.com/gpu-1", "custom.com/gpu-2"}))
+}
+
 func TestNewIngressConfig(t *testing.T) {
 	g := gomega.NewGomegaWithT(t)
 	clientset := fakeclientset.NewSimpleClientset(&corev1.ConfigMap{
diff --git a/pkg/apis/serving/v1beta1/inference_service_validation.go b/pkg/apis/serving/v1beta1/inference_service_validation.go
index e2d36bc1714..2106bd1cee1 100644
--- a/pkg/apis/serving/v1beta1/inference_service_validation.go
+++ b/pkg/apis/serving/v1beta1/inference_service_validation.go
@@ -156,14 +156,9 @@ func validateMultiNodeVariables(isvc *InferenceService) error {
 				return fmt.Errorf(DisallowedWorkerSpecTensorParallelSizeEnvError, isvc.Name)
 			}
 
-			customGPUResourceTypes := isvc.GetAnnotations()[constants.CustomGPUResourceTypesAnnotationKey]
-			if customGPUResourceTypes != "" {
-				if !utils.IsValidCustomGPUArray(customGPUResourceTypes) {
-					return fmt.Errorf(InvalidCustomGPUTypesAnnotationFormatError, isvc.Name, constants.CustomGPUResourceTypesAnnotationKey)
-				}
-			}
-
-			if utils.IsUnknownGpuResourceType(isvc.Spec.Predictor.Model.Resources, customGPUResourceTypes) {
+			if isUnknownGPUType, err := utils.IsUnknownGpuResourceType(isvc.Spec.Predictor.Model.Resources, isvc.Annotations); err != nil {
+				return err
+			} else if isUnknownGPUType {
 				return fmt.Errorf(InvalidUnknownGPUTypeError, isvc.Name)
 			}
 
@@ -192,7 +187,9 @@ func validateMultiNodeVariables(isvc *InferenceService) error {
 
 		if isvc.Spec.Predictor.WorkerSpec.Containers != nil {
 			for _, container := range isvc.Spec.Predictor.WorkerSpec.Containers {
-				if utils.IsUnknownGpuResourceType(container.Resources, isvc.GetAnnotations()[constants.CustomGPUResourceTypesAnnotationKey]) {
+				if isUnknownGPUType, err := utils.IsUnknownGpuResourceType(container.Resources, isvc.Annotations); err != nil {
+					return err
+				} else if isUnknownGPUType {
 					return fmt.Errorf(InvalidUnknownGPUTypeError, isvc.Name)
 				}
 			}
diff --git a/pkg/constants/constants.go b/pkg/constants/constants.go
index 21df310019c..e72fc7ddda0 100644
--- a/pkg/constants/constants.go
+++ b/pkg/constants/constants.go
@@ -257,7 +257,7 @@ const (
 
 var CustomGPUResourceTypesAnnotationKey = KServeAPIGroupName + "/gpu-resource-types"
 
-var GPUResourceTypeList = []string{
+var DefaultGPUResourceTypeList = []string{
 	NvidiaGPUResourceType,
 	AmdGPUResourceType,
 	IntelGPUResourceType,
diff --git a/pkg/controller/v1beta1/inferenceservice/components/predictor.go b/pkg/controller/v1beta1/inferenceservice/components/predictor.go
index 16b1f70223d..6b2afe01e81 100644
--- a/pkg/controller/v1beta1/inferenceservice/components/predictor.go
+++ b/pkg/controller/v1beta1/inferenceservice/components/predictor.go
@@ -427,10 +427,6 @@ func (p *Predictor) reconcileWorker(sRuntime v1alpha1.ServingRuntimeSpec, isvc *
 
 	sRuntimeWorkerAnnotations := sRuntime.WorkerSpec.Annotations
 	sRuntimeWorkerLabels := sRuntime.WorkerSpec.ServingRuntimePodSpec.Labels
-	// If CustomGPUResourceTypeAnnotationKey is set, the specified custom GPU resource will be added to the available GPUResourceTypeList.
-	if isvc.GetAnnotations()[constants.CustomGPUResourceTypesAnnotationKey] != "" {
-		sRuntimeWorkerAnnotations[constants.CustomGPUResourceTypesAnnotationKey] = isvc.GetAnnotations()[constants.CustomGPUResourceTypesAnnotationKey]
-	}
 
 	if workerPodSpec, err = multiNodeProcess(sRuntime, isvc, podSpec, annotations, isvcGeneration); err != nil {
 		return workerObjectMeta, workerPodSpec, err
diff --git a/pkg/controller/v1beta1/inferenceservice/reconcilers/deployment/deployment_reconciler.go b/pkg/controller/v1beta1/inferenceservice/reconcilers/deployment/deployment_reconciler.go
index 7f7dc23b37b..2810a63fe2e 100644
--- a/pkg/controller/v1beta1/inferenceservice/reconcilers/deployment/deployment_reconciler.go
+++ b/pkg/controller/v1beta1/inferenceservice/reconcilers/deployment/deployment_reconciler.go
@@ -330,11 +330,11 @@ func addGPUResourceToDeployment(deployment *appsv1.Deployment, targetContainerNa
 	// If CustomGPUResourceTypeAnnotationKey is set, the specified custom GPU resource will be added to the available GPUResourceTypeList.
 	customGPUResourceTypes := deployment.GetAnnotations()[constants.CustomGPUResourceTypesAnnotationKey]
 	if customGPUResourceTypes != "" {
-		constants.GPUResourceTypeList = append(constants.GPUResourceTypeList, strings.Split(customGPUResourceTypes, ",")...)
+		constants.DefaultGPUResourceTypeList = append(constants.DefaultGPUResourceTypeList, strings.Split(customGPUResourceTypes, ",")...)
 	}
 	for i, container := range deployment.Spec.Template.Spec.Containers {
 		if container.Name == targetContainerName {
-			for _, gpuType := range constants.GPUResourceTypeList {
+			for _, gpuType := range constants.DefaultGPUResourceTypeList {
 				resourceName := corev1.ResourceName(gpuType)
 				if qty, exists := deployment.Spec.Template.Spec.Containers[i].Resources.Limits[resourceName]; exists && !qty.IsZero() {
 					gpuResourceType = resourceName
diff --git a/pkg/openapi/openapi_generated.go b/pkg/openapi/openapi_generated.go
index 0efe1ab1fbd..a40806cde82 100644
--- a/pkg/openapi/openapi_generated.go
+++ b/pkg/openapi/openapi_generated.go
@@ -95,6 +95,7 @@ func GetOpenAPIDefinitions(ref common.ReferenceCallback) map[string]common.OpenA
 		"github.com/kserve/kserve/pkg/apis/serving/v1beta1.ModelRevisionStates":          schema_pkg_apis_serving_v1beta1_ModelRevisionStates(ref),
 		"github.com/kserve/kserve/pkg/apis/serving/v1beta1.ModelSpec":                    schema_pkg_apis_serving_v1beta1_ModelSpec(ref),
 		"github.com/kserve/kserve/pkg/apis/serving/v1beta1.ModelStatus":                  schema_pkg_apis_serving_v1beta1_ModelStatus(ref),
+		"github.com/kserve/kserve/pkg/apis/serving/v1beta1.MultiNodeConfig":              schema_pkg_apis_serving_v1beta1_MultiNodeConfig(ref),
 		"github.com/kserve/kserve/pkg/apis/serving/v1beta1.ONNXRuntimeSpec":              schema_pkg_apis_serving_v1beta1_ONNXRuntimeSpec(ref),
 		"github.com/kserve/kserve/pkg/apis/serving/v1beta1.PMMLSpec":                     schema_pkg_apis_serving_v1beta1_PMMLSpec(ref),
 		"github.com/kserve/kserve/pkg/apis/serving/v1beta1.PaddleServerSpec":             schema_pkg_apis_serving_v1beta1_PaddleServerSpec(ref),
@@ -6840,6 +6841,33 @@ func schema_pkg_apis_serving_v1beta1_ModelStatus(ref common.ReferenceCallback) c
 	}
 }
 
+func schema_pkg_apis_serving_v1beta1_MultiNodeConfig(ref common.ReferenceCallback) common.OpenAPIDefinition {
+	return common.OpenAPIDefinition{
+		Schema: spec.Schema{
+			SchemaProps: spec.SchemaProps{
+				Type: []string{"object"},
+				Properties: map[string]spec.Schema{
+					"customGPUResourceTypeList": {
+						SchemaProps: spec.SchemaProps{
+							Description: "CustomGPUResourceTypeList is a list of custom GPU resource types that are allowed to be used in the ServingRuntime and inferenceService",
+							Type:        []string{"array"},
+							Items: &spec.SchemaOrArray{
+								Schema: &spec.Schema{
+									SchemaProps: spec.SchemaProps{
+										Default: "",
+										Type:    []string{"string"},
+										Format:  "",
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+		},
+	}
+}
+
 func schema_pkg_apis_serving_v1beta1_ONNXRuntimeSpec(ref common.ReferenceCallback) common.OpenAPIDefinition {
 	return common.OpenAPIDefinition{
 		Schema: spec.Schema{
diff --git a/pkg/openapi/swagger.json b/pkg/openapi/swagger.json
index ced3bd1b66a..718e4277653 100644
--- a/pkg/openapi/swagger.json
+++ b/pkg/openapi/swagger.json
@@ -3792,6 +3792,19 @@
         }
       }
     },
+    "v1beta1.MultiNodeConfig": {
+      "type": "object",
+      "properties": {
+        "customGPUResourceTypeList": {
+          "description": "CustomGPUResourceTypeList is a list of custom GPU resource types that are allowed to be used in the ServingRuntime and inferenceService",
+          "type": "array",
+          "items": {
+            "type": "string",
+            "default": ""
+          }
+        }
+      }
+    },
     "v1beta1.ONNXRuntimeSpec": {
       "description": "ONNXRuntimeSpec defines arguments for configuring ONNX model serving.",
       "type": "object",
diff --git a/pkg/utils/utils.go b/pkg/utils/utils.go
index dda827bc3c0..15133cad7ee 100644
--- a/pkg/utils/utils.go
+++ b/pkg/utils/utils.go
@@ -18,6 +18,7 @@ package utils
 
 import (
 	"encoding/json"
+	"fmt"
 	"strconv"
 	"strings"
 
@@ -253,7 +254,7 @@ func GetEnvVarValue(envVars []corev1.EnvVar, key string) (string, bool) {
 }
 
 // IsUnknownGpuResourceType check if the provided gpu resource type is unknown one
-func IsUnknownGpuResourceType(resources corev1.ResourceRequirements, customGpuResourceTypes string) bool {
+func IsUnknownGpuResourceType(resources corev1.ResourceRequirements, annotations map[string]string) (bool, error) {
 	basicResourceTypes := map[corev1.ResourceName]struct{}{
 		corev1.ResourceCPU:              {},
 		corev1.ResourceMemory:           {},
@@ -276,46 +277,49 @@ func IsUnknownGpuResourceType(resources corev1.ResourceRequirements, customGpuRe
 	addNonBasicResources(resources.Limits)
 	addNonBasicResources(resources.Requests)
 
-	// Validate GPU resource types
-	// If CustomGPUResourceTypesAnnotationKey is set, the specified custom GPU resource will be added to the available GPUResourceTypeList.
-	if customGpuResourceTypes != "" {
-		constants.GPUResourceTypeList = append(constants.GPUResourceTypeList, strings.Split(customGpuResourceTypes, ",")...)
+	// Update GPU resource type list
+	newGPUResourceTypeList, err := UpdateGPUResourceTypeListByAnnotation(annotations)
+	if err != nil {
+		return false, err
 	}
 
-	for _, gpuType := range constants.GPUResourceTypeList {
+	// Validate GPU resource types
+	for _, gpuType := range newGPUResourceTypeList {
 		allowedGPUResourceName := corev1.ResourceName(gpuType)
 		delete(possibleGPUResourceType, allowedGPUResourceName) // Remove allowed GPU resource if exists
 	}
 
 	// Return true if there are unknown GPU resources
-	return len(possibleGPUResourceType) > 0
+	return len(possibleGPUResourceType) > 0, nil
 }
 
 // IsValidCustomGPUArray checks if the input string is a valid JSON array of strings.
 // It returns false if the array is empty, contains empty strings, or any non-string elements.
-func IsValidCustomGPUArray(s string) bool {
+// Otherwise, it returns true and the list of custom GPU types.
+func IsValidCustomGPUArray(s string) ([]string, bool) {
 	// Check if the input string is a valid JSON array
 	var arr []interface{}
 	if err := json.Unmarshal([]byte(s), &arr); err != nil {
-		return false // Not a valid JSON array
+		return nil, false // Not a valid JSON array
 	}
 
 	// Check if the array is empty
 	if len(arr) == 0 {
-		return false
+		return nil, false
 	}
-
+	customGPUTypes := []string{}
 	// Check each element to ensure they are all strings
 	for _, item := range arr {
 		if _, ok := item.(string); !ok {
-			return false // Found a non-string element
+			return nil, false // Found a non-string element
 		}
 		if item.(string) == "" {
-			return false // Found an empty string
+			return nil, false // Found an empty string
 		}
+		customGPUTypes = append(customGPUTypes, item.(string))
 	}
 
-	return true
+	return customGPUTypes, true
 }
 
 // StringToInt32 converts a given integer to int32. If the number exceeds the int32 limit, it returns an error.
@@ -326,3 +330,51 @@ func StringToInt32(number string) (int32, error) {
 	}
 	return int32(converted), err
 }
+
+// UpdateGPUResourceTypeListByAnnotation updates the GPU resource type list
+// by combining the global GPU resource types from inferenceservice-config with custom GPU resource types specified in the annotations.
+func UpdateGPUResourceTypeListByAnnotation(isvcAnnotations map[string]string) ([]string, error) {
+	// Deep copy
+	updatedGPUResourceTypes := append([]string{}, constants.DefaultGPUResourceTypeList...)
+
+	if customGPUResourceTypes := isvcAnnotations[constants.CustomGPUResourceTypesAnnotationKey]; customGPUResourceTypes != "" {
+		newGPUResourceTypesFromAnnotation, isValid := IsValidCustomGPUArray(customGPUResourceTypes)
+		if !isValid {
+			return nil, fmt.Errorf("invalid GPU format(%s) for %s annotation: must be a valid JSON array", customGPUResourceTypes, constants.CustomGPUResourceTypesAnnotationKey)
+		}
+
+		// Use a map to avoid duplicates
+		existingTypes := make(map[string]struct{}, len(constants.DefaultGPUResourceTypeList))
+		for _, t := range constants.DefaultGPUResourceTypeList {
+			existingTypes[t] = struct{}{}
+		}
+
+		// Add only unique GPU resource types
+		for _, t := range newGPUResourceTypesFromAnnotation {
+			if _, exists := existingTypes[t]; !exists {
+				updatedGPUResourceTypes = append(updatedGPUResourceTypes, t)
+				existingTypes[t] = struct{}{}
+			}
+		}
+	}
+	return updatedGPUResourceTypes, nil
+}
+
+// UpdateGlobalGPUResourceTypeList adds new GPU resource types from inferenceservice-config to constants.GPUResourceTypeList.
+func UpdateGlobalGPUResourceTypeList(newGPUResourceTypes []string) error {
+	// Use a map to avoid duplicates
+	existingTypes := make(map[string]struct{}, len(constants.DefaultGPUResourceTypeList))
+	for _, t := range constants.DefaultGPUResourceTypeList {
+		existingTypes[t] = struct{}{}
+	}
+
+	// Add only unique GPU resource types
+	for _, t := range newGPUResourceTypes {
+		if _, exists := existingTypes[t]; !exists {
+			constants.DefaultGPUResourceTypeList = append(constants.DefaultGPUResourceTypeList, t)
+			existingTypes[t] = struct{}{}
+		}
+	}
+
+	return nil
+}
diff --git a/pkg/utils/utils_test.go b/pkg/utils/utils_test.go
index 5a156c81f4f..8d860da2737 100644
--- a/pkg/utils/utils_test.go
+++ b/pkg/utils/utils_test.go
@@ -18,6 +18,7 @@ package utils
 
 import (
 	"errors"
+	"reflect"
 	"strconv"
 	"testing"
 
@@ -598,6 +599,7 @@ func TestIsUnknownGpuResourceType(t *testing.T) {
 
 	scenarios := map[string]struct {
 		resources       corev1.ResourceRequirements
+		annotations     map[string]string
 		expectedUnknown bool
 	}{
 		"OnlyBasicResources": {
@@ -611,6 +613,7 @@ func TestIsUnknownGpuResourceType(t *testing.T) {
 					corev1.ResourceMemory: resource.MustParse("1Gi"),
 				},
 			},
+			annotations:     map[string]string{},
 			expectedUnknown: false,
 		},
 		"ValidGpuResource": {
@@ -626,6 +629,7 @@ func TestIsUnknownGpuResourceType(t *testing.T) {
 					corev1.ResourceName("nvidia.com/gpu"): resource.MustParse("1"),
 				},
 			},
+			annotations:     map[string]string{},
 			expectedUnknown: false,
 		},
 		"UnknownGpuResource": {
@@ -641,8 +645,25 @@ func TestIsUnknownGpuResourceType(t *testing.T) {
 					corev1.ResourceName("unknown.com/gpu"): resource.MustParse("1"),
 				},
 			},
+			annotations:     map[string]string{},
 			expectedUnknown: true,
 		},
+		"UnknownGpuResourceWithAnnotation": {
+			resources: corev1.ResourceRequirements{
+				Limits: corev1.ResourceList{
+					corev1.ResourceCPU:                       resource.MustParse("1"),
+					corev1.ResourceMemory:                    resource.MustParse("1Gi"),
+					corev1.ResourceName("unknown.com-1/gpu"): resource.MustParse("1"),
+				},
+				Requests: corev1.ResourceList{
+					corev1.ResourceCPU:                       resource.MustParse("1"),
+					corev1.ResourceMemory:                    resource.MustParse("1Gi"),
+					corev1.ResourceName("unknown.com-1/gpu"): resource.MustParse("1"),
+				},
+			},
+			annotations:     map[string]string{constants.CustomGPUResourceTypesAnnotationKey: `["unknown.com-1/gpu"]`},
+			expectedUnknown: false,
+		},
 		"MixedResources": {
 			resources: corev1.ResourceRequirements{
 				Limits: corev1.ResourceList{
@@ -656,20 +677,38 @@ func TestIsUnknownGpuResourceType(t *testing.T) {
 					corev1.ResourceName("unknown.com/gpu"): resource.MustParse("1"),
 				},
 			},
+			annotations:     map[string]string{},
 			expectedUnknown: true,
 		},
+		"MixedResourcesWithAnnotation": {
+			resources: corev1.ResourceRequirements{
+				Limits: corev1.ResourceList{
+					corev1.ResourceCPU:                    resource.MustParse("1"),
+					corev1.ResourceMemory:                 resource.MustParse("1Gi"),
+					corev1.ResourceName("nvidia.com/gpu"): resource.MustParse("1"),
+				},
+				Requests: corev1.ResourceList{
+					corev1.ResourceCPU:                       resource.MustParse("1"),
+					corev1.ResourceMemory:                    resource.MustParse("1Gi"),
+					corev1.ResourceName("unknown.com-2/gpu"): resource.MustParse("1"),
+				},
+			},
+			annotations:     map[string]string{constants.CustomGPUResourceTypesAnnotationKey: `["unknown.com-2/gpu"]`},
+			expectedUnknown: false,
+		},
 		"EmptyResources": {
 			resources: corev1.ResourceRequirements{
 				Limits:   corev1.ResourceList{},
 				Requests: corev1.ResourceList{},
 			},
+			annotations:     map[string]string{},
 			expectedUnknown: false,
 		},
 	}
 
 	for name, scenario := range scenarios {
 		t.Run(name, func(t *testing.T) {
-			result := IsUnknownGpuResourceType(scenario.resources, "")
+			result, _ := IsUnknownGpuResourceType(scenario.resources, scenario.annotations)
 			g.Expect(result).Should(gomega.Equal(scenario.expectedUnknown))
 		})
 	}
@@ -677,30 +716,35 @@ func TestIsUnknownGpuResourceType(t *testing.T) {
 
 func TestIsValidCustomGPUArray(t *testing.T) {
 	tests := []struct {
-		input    string
-		expected bool
+		input                       string
+		expected                    bool
+		expectedNewGPUResourceTypes []string
 	}{
-		{"[]", false},
-		{"[\"item1\", \"item2\"]", true},
-		{"[\"item1\", \"item2\", \"item3\"]", true},
-		{"[\"item1\", \"item2\", \"\"]", false},
-		{"[\"item1\", 42]", false},
-		{"[\"item1\", \"item2\",]", false},
-		{"[\"item1\", \"item2\", \"item3\"", false},
-		{"[item1, item2]", false},
-		{"[\"item1\", \"item2\" \"item3\"]", false},
-		{"[\"item1\", null]", false},
-		{"[\"item1\", true]", false},
-		{"[\"item1\", false]", false},
-		{"[\"item1\", \"item2\", 42]", false},
-		{"[\"item1\", \"item2\", \"item3\", \"\"]", false},
+		{"[\"custom-gpu-1.com/gpu\"]", true, []string{"custom-gpu-1.com/gpu"}},
+		{"[\"custom-gpu-1.com/gpu\",\"custom-gpu-2.com/gpu\"]", true, []string{"custom-gpu-1.com/gpu", "custom-gpu-2.com/gpu"}},
+		{"[\"custom-gpu-1.com/gpu\",\"custom-gpu-2.com/gpu\",\"custom-gpu-3.com/gpu\"]", true, []string{"custom-gpu-1.com/gpu", "custom-gpu-2.com/gpu", "custom-gpu-3.com/gpu"}},
+		{"[]", false, nil},
+		{"[\"\"custom-gpu-1.com/gpu\"\", \"\"custom-gpu-2.com/gpu\"\", \"\"]", false, nil},
+		{"[\"\"custom-gpu-1.com/gpu\"\", 42]", false, nil},
+		{"[\"\"custom-gpu-1.com/gpu\"\", \"\"custom-gpu-2.com/gpu\"\",]", false, nil},
+		{"[\"\"custom-gpu-1.com/gpu\"\", \"\"custom-gpu-2.com/gpu\"\", \"\"custom-gpu-3.com/gpu\"\"", false, nil},
+		{"[custom-gpu-1.com/gpu, custom-gpu-2.com/gpu]", false, nil},
+		{"[\"\"custom-gpu-1.com/gpu\"\", \"\"custom-gpu-2.com/gpu\"\" \"\"custom-gpu-3.com/gpu\"\"]", false, nil},
+		{"[\"\"custom-gpu-1.com/gpu\"\", null]", false, nil},
+		{"[\"\"custom-gpu-1.com/gpu\"\", true]", false, nil},
+		{"[\"\"custom-gpu-1.com/gpu\"\", false]", false, nil},
+		{"[\"\"custom-gpu-1.com/gpu\"\", \"\"custom-gpu-2.com/gpu\"\", 42]", false, nil},
+		{"[\"\"custom-gpu-1.com/gpu\"\", \"\"custom-gpu-2.com/gpu\"\", \"\"custom-gpu-3.com/gpu\"\", \"\"]", false, nil},
 	}
 
 	for _, test := range tests {
 		t.Run(test.input, func(t *testing.T) {
-			result := IsValidCustomGPUArray(test.input)
-			if result != test.expected {
-				t.Errorf("expected %v, got %v", test.expected, result)
+			newGPUResourceTypes, isValid := IsValidCustomGPUArray(test.input)
+			if isValid != test.expected {
+				t.Errorf("expected %v, got %v", test.expected, isValid)
+			}
+			if !reflect.DeepEqual(newGPUResourceTypes, test.expectedNewGPUResourceTypes) {
+				t.Errorf("expected %v, got %v", test.expectedNewGPUResourceTypes, newGPUResourceTypes)
 			}
 		})
 	}
@@ -922,3 +966,104 @@ func TestStringToInt32(t *testing.T) {
 		})
 	}
 }
+
+func TestUpdateGPUResourceTypeListByAnnotation(t *testing.T) {
+	g := gomega.NewGomegaWithT(t)
+
+	combineGPU := func(customList []string) []string {
+		combindedList := append([]string{}, constants.DefaultGPUResourceTypeList...)
+		combindedList = append(combindedList, customList...)
+		return combindedList
+	}
+	tests := []struct {
+		name                 string
+		isvcAnnotations      map[string]string
+		expectedResourceList []string
+		expectError          types.GomegaMatcher
+	}{
+		{
+			name: "Valid GPU resources with unique types",
+			isvcAnnotations: map[string]string{
+				constants.CustomGPUResourceTypesAnnotationKey: "[\"custom-gpu-1.com/gpu\", \"custom-gpu-2.com/gpu\"]",
+			},
+			expectedResourceList: combineGPU([]string{"custom-gpu-1.com/gpu", "custom-gpu-2.com/gpu"}),
+			expectError:          gomega.BeNil(),
+		},
+		{
+			name: "Invalid GPU format (invalid JSON)",
+			isvcAnnotations: map[string]string{
+				constants.CustomGPUResourceTypesAnnotationKey: "[custom-gpu-1.com/gpu, custom-gpu-2.com/gpu", // Invalid JSON
+			},
+			expectedResourceList: nil,
+			expectError:          gomega.Not(gomega.BeNil()),
+		},
+		{
+			name: "Existing GPU resources in annotation",
+			isvcAnnotations: map[string]string{
+				constants.CustomGPUResourceTypesAnnotationKey: "[\"custom-gpu-1.com/gpu\"]",
+			},
+			expectedResourceList: combineGPU([]string{"custom-gpu-1.com/gpu"}), // Should not add anything
+			expectError:          gomega.BeNil(),
+		},
+	}
+
+	// Test each case
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			// Run the function to be tested
+			updatedList, err := UpdateGPUResourceTypeListByAnnotation(tt.isvcAnnotations)
+
+			g.Expect(updatedList).Should(gomega.Equal(tt.expectedResourceList))
+			g.Expect(err).Should(tt.expectError)
+		})
+	}
+}
+
+func TestUpdateGlobalGPUResourceTypeList(t *testing.T) {
+	g := gomega.NewWithT(t)
+	originalDefaultGPUResourceTypeList := append([]string{}, constants.DefaultGPUResourceTypeList...)
+	combineGPU := func(customList []string) []string {
+		combindedList := append([]string{}, constants.DefaultGPUResourceTypeList...)
+		combindedList = append(combindedList, customList...)
+		return combindedList
+	}
+
+	tests := []struct {
+		name                string
+		newGPUResourceTypes []string
+		expectedResult      []string
+	}{
+		{
+			name:                "Add new GPU resource types",
+			newGPUResourceTypes: []string{"custom-gpu-1.com/gpu", "custom-gpu-2.com/gpu"},
+			expectedResult:      combineGPU([]string{"custom-gpu-1.com/gpu", "custom-gpu-2.com/gpu"}),
+		},
+		{
+			name:                "Avoid duplicates",
+			newGPUResourceTypes: []string{"custom-gpu-1.com/gpu", constants.DefaultGPUResourceTypeList[0]},
+			expectedResult:      combineGPU([]string{"custom-gpu-1.com/gpu"}),
+		},
+		{
+			name:                "Avoid duplicates - 2",
+			newGPUResourceTypes: []string{"custom-gpu-1.com/gpu", "custom-gpu-1.com/gpu"},
+			expectedResult:      combineGPU([]string{"custom-gpu-1.com/gpu"}),
+		},
+		{
+			name:                "Empty new GPU resource types",
+			newGPUResourceTypes: []string{},
+			expectedResult:      combineGPU([]string{}),
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			// Rest global GPU resource type list for next test case
+			constants.DefaultGPUResourceTypeList = originalDefaultGPUResourceTypeList
+
+			// Reset the constants for each test to ensure isolation
+			err := UpdateGlobalGPUResourceTypeList(tt.newGPUResourceTypes)
+			g.Expect(err).ShouldNot(gomega.HaveOccurred())
+			g.Expect(constants.DefaultGPUResourceTypeList).Should(gomega.Equal(tt.expectedResult))
+		})
+	}
+}
diff --git a/pkg/webhook/admission/servingruntime/servingruntime_webhook.go b/pkg/webhook/admission/servingruntime/servingruntime_webhook.go
index f3659f7c39c..2963d6aa6e3 100644
--- a/pkg/webhook/admission/servingruntime/servingruntime_webhook.go
+++ b/pkg/webhook/admission/servingruntime/servingruntime_webhook.go
@@ -222,14 +222,19 @@ func validateMultiNodeSpec(newSpec *v1alpha1.ServingRuntimeSpec, existingSpec *v
 				if _, exists := utils.GetEnvVarValue(newSpec.Containers[i].Env, constants.TensorParallelSizeEnvName); exists {
 					return errors.New(DisallowedWorkerSpecTensorParallelSizeEnvError)
 				}
-				if utils.IsUnknownGpuResourceType(container.Resources, "") {
+
+				if isUnknownGPUType, err := utils.IsUnknownGpuResourceType(container.Resources, newSpec.Annotations); err != nil {
+					return err
+				} else if isUnknownGPUType {
 					return fmt.Errorf(InvalidUnknownGPUTypeError, constants.InferenceServiceContainerName)
 				}
 			}
 		}
 		workerContainer := newSpec.WorkerSpec.Containers[0]
 		if workerContainer.Name == constants.WorkerContainerName {
-			if utils.IsUnknownGpuResourceType(workerContainer.Resources, "") {
+			if isUnknownGPUType, err := utils.IsUnknownGpuResourceType(workerContainer.Resources, newSpec.Annotations); err != nil {
+				return err
+			} else if isUnknownGPUType {
 				return fmt.Errorf(InvalidUnknownGPUTypeError, constants.WorkerContainerName)
 			}
 		}
diff --git a/python/kserve/docs/V1beta1MultiNodeConfig.md b/python/kserve/docs/V1beta1MultiNodeConfig.md
new file mode 100644
index 00000000000..793d23bd352
--- /dev/null
+++ b/python/kserve/docs/V1beta1MultiNodeConfig.md
@@ -0,0 +1,10 @@
+# V1beta1MultiNodeConfig
+
+## Properties
+Name | Type | Description | Notes
+------------ | ------------- | ------------- | -------------
+**custom_gpu_resource_type_list** | **list[str]** | CustomGPUResourceTypeList is a list of custom GPU resource types that are allowed to be used in the ServingRuntime and inferenceService | [optional] 
+
+[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
+
+
diff --git a/python/kserve/kserve/models/__init__.py b/python/kserve/kserve/models/__init__.py
index faab1152e29..d04ac6f56e7 100644
--- a/python/kserve/kserve/models/__init__.py
+++ b/python/kserve/kserve/models/__init__.py
@@ -94,6 +94,7 @@
 from kserve.models.v1beta1_model_revision_states import V1beta1ModelRevisionStates
 from kserve.models.v1beta1_model_spec import V1beta1ModelSpec
 from kserve.models.v1beta1_model_status import V1beta1ModelStatus
+from kserve.models.v1beta1_multi_node_config import V1beta1MultiNodeConfig
 from kserve.models.v1beta1_onnx_runtime_spec import V1beta1ONNXRuntimeSpec
 from kserve.models.v1beta1_pmml_spec import V1beta1PMMLSpec
 from kserve.models.v1beta1_paddle_server_spec import V1beta1PaddleServerSpec
diff --git a/python/kserve/kserve/models/v1beta1_multi_node_config.py b/python/kserve/kserve/models/v1beta1_multi_node_config.py
new file mode 100644
index 00000000000..54954c35e00
--- /dev/null
+++ b/python/kserve/kserve/models/v1beta1_multi_node_config.py
@@ -0,0 +1,136 @@
+# Copyright 2023 The KServe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# coding: utf-8
+
+"""
+    KServe
+
+    Python SDK for KServe  # noqa: E501
+
+    The version of the OpenAPI document: v0.1
+    Generated by: https://openapi-generator.tech
+"""
+
+
+import pprint
+import re  # noqa: F401
+
+import six
+
+from kserve.configuration import Configuration
+
+
+class V1beta1MultiNodeConfig(object):
+    """NOTE: This class is auto generated by OpenAPI Generator.
+    Ref: https://openapi-generator.tech
+
+    Do not edit the class manually.
+    """
+
+    """
+    Attributes:
+      openapi_types (dict): The key is attribute name
+                            and the value is attribute type.
+      attribute_map (dict): The key is attribute name
+                            and the value is json key in definition.
+    """
+    openapi_types = {
+        'custom_gpu_resource_type_list': 'list[str]'
+    }
+
+    attribute_map = {
+        'custom_gpu_resource_type_list': 'customGPUResourceTypeList'
+    }
+
+    def __init__(self, custom_gpu_resource_type_list=None, local_vars_configuration=None):  # noqa: E501
+        """V1beta1MultiNodeConfig - a model defined in OpenAPI"""  # noqa: E501
+        if local_vars_configuration is None:
+            local_vars_configuration = Configuration()
+        self.local_vars_configuration = local_vars_configuration
+
+        self._custom_gpu_resource_type_list = None
+        self.discriminator = None
+
+        if custom_gpu_resource_type_list is not None:
+            self.custom_gpu_resource_type_list = custom_gpu_resource_type_list
+
+    @property
+    def custom_gpu_resource_type_list(self):
+        """Gets the custom_gpu_resource_type_list of this V1beta1MultiNodeConfig.  # noqa: E501
+
+        CustomGPUResourceTypeList is a list of custom GPU resource types that are allowed to be used in the ServingRuntime and inferenceService  # noqa: E501
+
+        :return: The custom_gpu_resource_type_list of this V1beta1MultiNodeConfig.  # noqa: E501
+        :rtype: list[str]
+        """
+        return self._custom_gpu_resource_type_list
+
+    @custom_gpu_resource_type_list.setter
+    def custom_gpu_resource_type_list(self, custom_gpu_resource_type_list):
+        """Sets the custom_gpu_resource_type_list of this V1beta1MultiNodeConfig.
+
+        CustomGPUResourceTypeList is a list of custom GPU resource types that are allowed to be used in the ServingRuntime and inferenceService  # noqa: E501
+
+        :param custom_gpu_resource_type_list: The custom_gpu_resource_type_list of this V1beta1MultiNodeConfig.  # noqa: E501
+        :type: list[str]
+        """
+
+        self._custom_gpu_resource_type_list = custom_gpu_resource_type_list
+
+    def to_dict(self):
+        """Returns the model properties as a dict"""
+        result = {}
+
+        for attr, _ in six.iteritems(self.openapi_types):
+            value = getattr(self, attr)
+            if isinstance(value, list):
+                result[attr] = list(map(
+                    lambda x: x.to_dict() if hasattr(x, "to_dict") else x,
+                    value
+                ))
+            elif hasattr(value, "to_dict"):
+                result[attr] = value.to_dict()
+            elif isinstance(value, dict):
+                result[attr] = dict(map(
+                    lambda item: (item[0], item[1].to_dict())
+                    if hasattr(item[1], "to_dict") else item,
+                    value.items()
+                ))
+            else:
+                result[attr] = value
+
+        return result
+
+    def to_str(self):
+        """Returns the string representation of the model"""
+        return pprint.pformat(self.to_dict())
+
+    def __repr__(self):
+        """For `print` and `pprint`"""
+        return self.to_str()
+
+    def __eq__(self, other):
+        """Returns true if both objects are equal"""
+        if not isinstance(other, V1beta1MultiNodeConfig):
+            return False
+
+        return self.to_dict() == other.to_dict()
+
+    def __ne__(self, other):
+        """Returns true if both objects are not equal"""
+        if not isinstance(other, V1beta1MultiNodeConfig):
+            return True
+
+        return self.to_dict() != other.to_dict()
diff --git a/python/kserve/test/test_v1beta1_multi_node_config.py b/python/kserve/test/test_v1beta1_multi_node_config.py
new file mode 100644
index 00000000000..c872d9bc09c
--- /dev/null
+++ b/python/kserve/test/test_v1beta1_multi_node_config.py
@@ -0,0 +1,64 @@
+# Copyright 2023 The KServe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# coding: utf-8
+
+"""
+KServe
+
+Python SDK for KServe  # noqa: E501
+
+The version of the OpenAPI document: v0.1
+Generated by: https://openapi-generator.tech
+"""
+
+
+from __future__ import absolute_import
+
+import unittest
+import datetime
+
+import kserve
+from kserve.models.v1beta1_multi_node_config import V1beta1MultiNodeConfig  # noqa: E501
+from kserve.rest import ApiException
+
+
+class TestV1beta1MultiNodeConfig(unittest.TestCase):
+    """V1beta1MultiNodeConfig unit test stubs"""
+
+    def setUp(self):
+        pass
+
+    def tearDown(self):
+        pass
+
+    def make_instance(self, include_optional):
+        """Test V1beta1MultiNodeConfig
+        include_option is a boolean, when False only required
+        params are included, when True both required and
+        optional params are included"""
+        # model = kserve.models.v1beta1_multi_node_config.V1beta1MultiNodeConfig()  # noqa: E501
+        if include_optional:
+            return V1beta1MultiNodeConfig(custom_gpu_resource_type_list=["0"])
+        else:
+            return V1beta1MultiNodeConfig()
+
+    def testV1beta1MultiNodeConfig(self):
+        """Test V1beta1MultiNodeConfig"""
+        inst_req_only = self.make_instance(include_optional=False)
+        inst_req_and_optional = self.make_instance(include_optional=True)
+
+
+if __name__ == "__main__":
+    unittest.main()


Reviews:
user: login: spolti
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T15:19:42Z
user: login: VedantMahabaleshwarkar
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T17:47:47Z
user: login: Jooho
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T22:29:27Z
user: login: spolti
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T13:51:06Z
user: login: spolti
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T13:51:14Z
user: login: spolti
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T13:52:09Z
user: login: Jooho
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T15:57:19Z
user: login: Jooho
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T16:00:53Z
user: login: Jooho
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T16:01:33Z
user: login: terrytangyuan
body: Thanks!

/lgtm
/approve
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-24T18:50:39Z

Review comments:
diff_hunk: @@ -167,6 +167,15 @@ func main() {
 		os.Exit(1)
 	}
 
+	// Update Global GPU Resource Type List when custom GPU resource types are provided
path: cmd/manager/main.go
user: login: spolti
body: Shouldn't this piece of code be in the configmap.go where all the other configs are read?
created_at: 2025-03-20T14:55:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -73,6 +73,24 @@ data:
           "my.custom.label.1"  
         ]
       }  
+    # ====================================== MultiNode CONFIGURATION ======================================
path: config/configmap/inferenceservice.yaml
user: login: spolti
body: I was thinking about, whether this extra config is needed, I mean, if the user uses a custom GPU and sets it in the isvc or the SR, that should be enough, IMHO.
Because, if I understand it correctly, the user will need to add the custom GPU in the config manually, later in the isvc, and this config will just check if the custom GPU is valid.

Despite the default ones, I am not sure if validating custom ones is necessary. It would make sense if the intention is to prevent users from setting any other GPU type than the ones defined here.
I will probably be missing something.
created_at: 2025-03-20T14:58:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -276,46 +277,49 @@ func IsUnknownGpuResourceType(resources corev1.ResourceRequirements, customGpuRe
 	addNonBasicResources(resources.Limits)
 	addNonBasicResources(resources.Requests)
 
-	// Validate GPU resource types
-	// If CustomGPUResourceTypesAnnotationKey is set, the specified custom GPU resource will be added to the available GPUResourceTypeList.
-	if customGpuResourceTypes != "" {
-		constants.GPUResourceTypeList = append(constants.GPUResourceTypeList, strings.Split(customGpuResourceTypes, ",")...)
+	// Update GPU resource type list
+	newGPUResourceTypeList, err := UpdateGPUResourceTypeListByAnnotation(annotations)
+	if err != nil {
+		return false, err
 	}
 
-	for _, gpuType := range constants.GPUResourceTypeList {
+	// Validate GPU resource types
+	for _, gpuType := range newGPUResourceTypeList {
 		allowedGPUResourceName := corev1.ResourceName(gpuType)
 		delete(possibleGPUResourceType, allowedGPUResourceName) // Remove allowed GPU resource if exists
 	}
 
 	// Return true if there are unknown GPU resources
-	return len(possibleGPUResourceType) > 0
+	return len(possibleGPUResourceType) > 0, nil
 }
 
 // IsValidCustomGPUArray checks if the input string is a valid JSON array of strings.
 // It returns false if the array is empty, contains empty strings, or any non-string elements.
-func IsValidCustomGPUArray(s string) bool {
+// Otherwise, it returns true and the list of custom GPU types.
+func IsValidCustomGPUArray(s string) ([]string, bool) {
path: pkg/utils/utils.go
user: login: spolti
body: isn't better to treat it as array of string instead json?
created_at: 2025-03-20T15:02:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -326,3 +330,51 @@ func StringToInt32(number string) (int32, error) {
 	}
 	return int32(converted), err
 }
+
+// UpdateGPUResourceTypeListByAnnotation updates the GPU resource type list
+// by combining the global GPU resource types from inferenceservice-config with custom GPU resource types specified in the annotations.
+func UpdateGPUResourceTypeListByAnnotation(isvcAnnotations map[string]string) ([]string, error) {
+	// Deep copy
+	updatedGPUResourceTypes := append([]string{}, constants.DefaultGPUResourceTypeList...)
+
+	if customGPUResourceTypes := isvcAnnotations[constants.CustomGPUResourceTypesAnnotationKey]; customGPUResourceTypes != "" {
+		newGPUResourceTypesFromAnnotation, isValid := IsValidCustomGPUArray(customGPUResourceTypes)
+		if !isValid {
+			return nil, fmt.Errorf("invalid GPU format(%s) for %s annotation: must be a valid JSON array", customGPUResourceTypes, constants.CustomGPUResourceTypesAnnotationKey)
+		}
+
+		// Use a set to avoid duplicates
+		existingTypes := make(map[string]struct{}, len(constants.DefaultGPUResourceTypeList))
+		for _, t := range constants.DefaultGPUResourceTypeList {
+			existingTypes[t] = struct{}{}
+		}
+
+		// Add only unique GPU resource types
+		for _, t := range newGPUResourceTypesFromAnnotation {
+			if _, exists := existingTypes[t]; !exists {
+				updatedGPUResourceTypes = append(updatedGPUResourceTypes, t)
+				existingTypes[t] = struct{}{}
+			}
+		}
+	}
+	return updatedGPUResourceTypes, nil
+}
+
+// UpdateGlobalGPUResourceTypeList adds new GPU resource types from inferenceservice-config to constants.GPUResourceTypeList.
+func UpdateGlobalGPUResourceTypeList(newGPUResourceTypes []string) error {
+	// Use a set to avoid duplicates
path: pkg/utils/utils.go
user: login: spolti
body: ```suggestion
	// Use a map to avoid duplicates
```
created_at: 2025-03-20T15:13:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -656,51 +677,74 @@ func TestIsUnknownGpuResourceType(t *testing.T) {
 					corev1.ResourceName("unknown.com/gpu"): resource.MustParse("1"),
 				},
 			},
+			annotations:     map[string]string{},
 			expectedUnknown: true,
 		},
+		"MixedResourcesWithAnnotation": {
+			resources: corev1.ResourceRequirements{
+				Limits: corev1.ResourceList{
+					corev1.ResourceCPU:                    resource.MustParse("1"),
+					corev1.ResourceMemory:                 resource.MustParse("1Gi"),
+					corev1.ResourceName("nvidia.com/gpu"): resource.MustParse("1"),
+				},
+				Requests: corev1.ResourceList{
+					corev1.ResourceCPU:                       resource.MustParse("1"),
+					corev1.ResourceMemory:                    resource.MustParse("1Gi"),
+					corev1.ResourceName("unknown.com-2/gpu"): resource.MustParse("1"),
path: pkg/utils/utils_test.go
user: login: spolti
body: does your code detect if in the requests GPU is the same as limits?
created_at: 2025-03-20T15:17:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -156,14 +156,9 @@ func validateMultiNodeVariables(isvc *InferenceService) error {
 				return fmt.Errorf(DisallowedWorkerSpecTensorParallelSizeEnvError, isvc.Name)
 			}
 
-			customGPUResourceTypes := isvc.GetAnnotations()[constants.CustomGPUResourceTypesAnnotationKey]
-			if customGPUResourceTypes != "" {
-				if !utils.IsValidCustomGPUArray(customGPUResourceTypes) {
-					return fmt.Errorf(InvalidCustomGPUTypesAnnotationFormatError, isvc.Name, constants.CustomGPUResourceTypesAnnotationKey)
-				}
-			}
-
-			if utils.IsUnknownGpuResourceType(isvc.Spec.Predictor.Model.Resources, customGPUResourceTypes) {
+			if isUnknownGPUType, err := utils.IsUnknownGpuResourceType(isvc.Spec.Predictor.Model.Resources, isvc.Annotations); err != nil {
path: pkg/apis/serving/v1beta1/inference_service_validation.go
user: login: VedantMahabaleshwarkar
body: [here](https://github.com/kserve/kserve/pull/4303/files#diff-81ddbadfb415ccbb9c7af84f11668d1aa5e53c34025bf86d4702f16b4e42f045R281) the resourceTypeList is updated with the annotation from the isvc (if the annotation is not in the default list). So when will this function ever return false? 
created_at: 2025-03-20T17:40:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -156,14 +156,9 @@ func validateMultiNodeVariables(isvc *InferenceService) error {
 				return fmt.Errorf(DisallowedWorkerSpecTensorParallelSizeEnvError, isvc.Name)
 			}
 
-			customGPUResourceTypes := isvc.GetAnnotations()[constants.CustomGPUResourceTypesAnnotationKey]
-			if customGPUResourceTypes != "" {
-				if !utils.IsValidCustomGPUArray(customGPUResourceTypes) {
-					return fmt.Errorf(InvalidCustomGPUTypesAnnotationFormatError, isvc.Name, constants.CustomGPUResourceTypesAnnotationKey)
-				}
-			}
-
-			if utils.IsUnknownGpuResourceType(isvc.Spec.Predictor.Model.Resources, customGPUResourceTypes) {
+			if isUnknownGPUType, err := utils.IsUnknownGpuResourceType(isvc.Spec.Predictor.Model.Resources, isvc.Annotations); err != nil {
path: pkg/apis/serving/v1beta1/inference_service_validation.go
user: login: VedantMahabaleshwarkar
body: Question : [here](https://github.com/kserve/kserve/pull/4303/files#diff-81ddbadfb415ccbb9c7af84f11668d1aa5e53c34025bf86d4702f16b4e42f045R364) the constants list is updated with any new entries from the configmap, so if the isvc annotations has a new custom gpu annotation that is not in the constants list, is that a valid GPU type or unknown? 
created_at: 2025-03-20T17:45:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -330,11 +330,11 @@ func addGPUResourceToDeployment(deployment *appsv1.Deployment, targetContainerNa
 	// If CustomGPUResourceTypeAnnotationKey is set, the specified custom GPU resource will be added to the available GPUResourceTypeList.
 	customGPUResourceTypes := deployment.GetAnnotations()[constants.CustomGPUResourceTypesAnnotationKey]
 	if customGPUResourceTypes != "" {
-		constants.GPUResourceTypeList = append(constants.GPUResourceTypeList, strings.Split(customGPUResourceTypes, ",")...)
+		constants.DefaultGPUResourceTypeList = append(constants.DefaultGPUResourceTypeList, strings.Split(customGPUResourceTypes, ",")...)
path: pkg/controller/v1beta1/inferenceservice/reconcilers/deployment/deployment_reconciler.go
user: login: VedantMahabaleshwarkar
body: This is already done in https://github.com/kserve/kserve/pull/4303/files#diff-81ddbadfb415ccbb9c7af84f11668d1aa5e53c34025bf86d4702f16b4e42f045R364 right? 
created_at: 2025-03-20T17:47:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -656,51 +677,74 @@ func TestIsUnknownGpuResourceType(t *testing.T) {
 					corev1.ResourceName("unknown.com/gpu"): resource.MustParse("1"),
 				},
 			},
+			annotations:     map[string]string{},
 			expectedUnknown: true,
 		},
+		"MixedResourcesWithAnnotation": {
+			resources: corev1.ResourceRequirements{
+				Limits: corev1.ResourceList{
+					corev1.ResourceCPU:                    resource.MustParse("1"),
+					corev1.ResourceMemory:                 resource.MustParse("1Gi"),
+					corev1.ResourceName("nvidia.com/gpu"): resource.MustParse("1"),
+				},
+				Requests: corev1.ResourceList{
+					corev1.ResourceCPU:                       resource.MustParse("1"),
+					corev1.ResourceMemory:                    resource.MustParse("1Gi"),
+					corev1.ResourceName("unknown.com-2/gpu"): resource.MustParse("1"),
path: pkg/utils/utils_test.go
user: login: Jooho
body: no it does not have. 
created_at: 2025-03-20T21:11:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -276,46 +277,49 @@ func IsUnknownGpuResourceType(resources corev1.ResourceRequirements, customGpuRe
 	addNonBasicResources(resources.Limits)
 	addNonBasicResources(resources.Requests)
 
-	// Validate GPU resource types
-	// If CustomGPUResourceTypesAnnotationKey is set, the specified custom GPU resource will be added to the available GPUResourceTypeList.
-	if customGpuResourceTypes != "" {
-		constants.GPUResourceTypeList = append(constants.GPUResourceTypeList, strings.Split(customGpuResourceTypes, ",")...)
+	// Update GPU resource type list
+	newGPUResourceTypeList, err := UpdateGPUResourceTypeListByAnnotation(annotations)
+	if err != nil {
+		return false, err
 	}
 
-	for _, gpuType := range constants.GPUResourceTypeList {
+	// Validate GPU resource types
+	for _, gpuType := range newGPUResourceTypeList {
 		allowedGPUResourceName := corev1.ResourceName(gpuType)
 		delete(possibleGPUResourceType, allowedGPUResourceName) // Remove allowed GPU resource if exists
 	}
 
 	// Return true if there are unknown GPU resources
-	return len(possibleGPUResourceType) > 0
+	return len(possibleGPUResourceType) > 0, nil
 }
 
 // IsValidCustomGPUArray checks if the input string is a valid JSON array of strings.
 // It returns false if the array is empty, contains empty strings, or any non-string elements.
-func IsValidCustomGPUArray(s string) bool {
+// Otherwise, it returns true and the list of custom GPU types.
+func IsValidCustomGPUArray(s string) ([]string, bool) {
path: pkg/utils/utils.go
user: login: Jooho
body: From my two cents, it is similar. It needs parsing anyway to make a string array. Are there specific benefits if I use a string array?

created_at: 2025-03-20T21:49:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -330,11 +330,11 @@ func addGPUResourceToDeployment(deployment *appsv1.Deployment, targetContainerNa
 	// If CustomGPUResourceTypeAnnotationKey is set, the specified custom GPU resource will be added to the available GPUResourceTypeList.
 	customGPUResourceTypes := deployment.GetAnnotations()[constants.CustomGPUResourceTypesAnnotationKey]
 	if customGPUResourceTypes != "" {
-		constants.GPUResourceTypeList = append(constants.GPUResourceTypeList, strings.Split(customGPUResourceTypes, ",")...)
+		constants.DefaultGPUResourceTypeList = append(constants.DefaultGPUResourceTypeList, strings.Split(customGPUResourceTypes, ",")...)
path: pkg/controller/v1beta1/inferenceservice/reconcilers/deployment/deployment_reconciler.go
user: login: Jooho
body: No it is different. this add the gpuType from annotations to DefaultGPUResourceTypeList. 
The one you pointed out is adding the custom GPU list in the inferenceservice-config to the DefaultGPUResourceTypeList. 
created_at: 2025-03-20T21:55:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -156,14 +156,9 @@ func validateMultiNodeVariables(isvc *InferenceService) error {
 				return fmt.Errorf(DisallowedWorkerSpecTensorParallelSizeEnvError, isvc.Name)
 			}
 
-			customGPUResourceTypes := isvc.GetAnnotations()[constants.CustomGPUResourceTypesAnnotationKey]
-			if customGPUResourceTypes != "" {
-				if !utils.IsValidCustomGPUArray(customGPUResourceTypes) {
-					return fmt.Errorf(InvalidCustomGPUTypesAnnotationFormatError, isvc.Name, constants.CustomGPUResourceTypesAnnotationKey)
-				}
-			}
-
-			if utils.IsUnknownGpuResourceType(isvc.Spec.Predictor.Model.Resources, customGPUResourceTypes) {
+			if isUnknownGPUType, err := utils.IsUnknownGpuResourceType(isvc.Spec.Predictor.Model.Resources, isvc.Annotations); err != nil {
path: pkg/apis/serving/v1beta1/inference_service_validation.go
user: login: Jooho
body: >here the resourceTypeList is updated with the annotation from the isvc (if the annotation is not in the default list). So when will this function ever return false?

By the fault, allowed GPU types are 4. If the user uses a custom GPU type like `custom.com/gpu: 1` and it returns false

> Question : here the constants list is updated with any new entries from the configmap, so if the isvc annotations has a new custom gpu annotation that is not in the constants list, is that a valid GPU type or unknown?

The custom GPU type in annotation will be added as a valid GPU type but it is not added to constants.
It returns [a new string array](https://github.com/kserve/kserve/pull/4303/files/7a6404c238e5747e7f38510b1a8dbe6a46a50747#diff-81ddbadfb415ccbb9c7af84f11668d1aa5e53c34025bf86d4702f16b4e42f045R360)

So it will be valid GPU type
created_at: 2025-03-20T22:04:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -167,6 +167,15 @@ func main() {
 		os.Exit(1)
 	}
 
+	// Update Global GPU Resource Type List when custom GPU resource types are provided
path: cmd/manager/main.go
user: login: Jooho
body: Some configurations also uses [main.go](https://github.com/kserve/kserve/blob/master/cmd/manager/main.go#L154-L168)

The multinode part has only one configuration for now, and it doesn’t need to be called every time—just once to update the constants.  By the way, I already added the multinode part in configmap.go but I missed unit test for it. I will follow up it
created_at: 2025-03-20T22:13:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -73,6 +73,24 @@ data:
           "my.custom.label.1"  
         ]
       }  
+    # ====================================== MultiNode CONFIGURATION ======================================
path: config/configmap/inferenceservice.yaml
user: login: Jooho
body: Multinode in KServe needs to determine the number of GPUs by calculating tensorParallelSize and pipelineParallelSize. To specify GPUs, we need to know the exact GPU resource names.

However, GPU resource names can be arbitrarily defined by vendors, so we cannot determine them in advance. That’s why I believe this approach is the best way to identify which resources in the configuration are intended for GPUs.

Even within NVIDIA GPUs, there can be hundreds of different custom resource names, making it difficult to identify them. While we could use nvidia.com to detect them, as I mentioned earlier, this wouldn’t work for custom GPUs from new vendors. That’s why I think allowing users to define them manually is the most reasonable approach.

To make this process more convenient, I added a new field in inferenceservice-config so that all ISVCs can check GPU types automatically.
created_at: 2025-03-20T22:22:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -276,46 +277,49 @@ func IsUnknownGpuResourceType(resources corev1.ResourceRequirements, customGpuRe
 	addNonBasicResources(resources.Limits)
 	addNonBasicResources(resources.Requests)
 
-	// Validate GPU resource types
-	// If CustomGPUResourceTypesAnnotationKey is set, the specified custom GPU resource will be added to the available GPUResourceTypeList.
-	if customGpuResourceTypes != "" {
-		constants.GPUResourceTypeList = append(constants.GPUResourceTypeList, strings.Split(customGpuResourceTypes, ",")...)
+	// Update GPU resource type list
+	newGPUResourceTypeList, err := UpdateGPUResourceTypeListByAnnotation(annotations)
+	if err != nil {
+		return false, err
 	}
 
-	for _, gpuType := range constants.GPUResourceTypeList {
+	// Validate GPU resource types
+	for _, gpuType := range newGPUResourceTypeList {
 		allowedGPUResourceName := corev1.ResourceName(gpuType)
 		delete(possibleGPUResourceType, allowedGPUResourceName) // Remove allowed GPU resource if exists
 	}
 
 	// Return true if there are unknown GPU resources
-	return len(possibleGPUResourceType) > 0
+	return len(possibleGPUResourceType) > 0, nil
 }
 
 // IsValidCustomGPUArray checks if the input string is a valid JSON array of strings.
 // It returns false if the array is empty, contains empty strings, or any non-string elements.
-func IsValidCustomGPUArray(s string) bool {
+// Otherwise, it returns true and the list of custom GPU types.
+func IsValidCustomGPUArray(s string) ([]string, bool) {
path: pkg/utils/utils.go
user: login: spolti
body: What do you mean by, `it needs parsing anyway`?

created_at: 2025-03-21T13:51:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -73,6 +73,24 @@ data:
           "my.custom.label.1"  
         ]
       }  
+    # ====================================== MultiNode CONFIGURATION ======================================
path: config/configmap/inferenceservice.yaml
user: login: spolti
body: yeah, that makes sense.
created_at: 2025-03-21T13:51:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -656,51 +677,74 @@ func TestIsUnknownGpuResourceType(t *testing.T) {
 					corev1.ResourceName("unknown.com/gpu"): resource.MustParse("1"),
 				},
 			},
+			annotations:     map[string]string{},
 			expectedUnknown: true,
 		},
+		"MixedResourcesWithAnnotation": {
+			resources: corev1.ResourceRequirements{
+				Limits: corev1.ResourceList{
+					corev1.ResourceCPU:                    resource.MustParse("1"),
+					corev1.ResourceMemory:                 resource.MustParse("1Gi"),
+					corev1.ResourceName("nvidia.com/gpu"): resource.MustParse("1"),
+				},
+				Requests: corev1.ResourceList{
+					corev1.ResourceCPU:                       resource.MustParse("1"),
+					corev1.ResourceMemory:                    resource.MustParse("1Gi"),
+					corev1.ResourceName("unknown.com-2/gpu"): resource.MustParse("1"),
path: pkg/utils/utils_test.go
user: login: spolti
body: it might be useful if requests is greater than limits, if that happens, respect the limits to set the tensor.
created_at: 2025-03-21T13:52:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -656,51 +677,74 @@ func TestIsUnknownGpuResourceType(t *testing.T) {
 					corev1.ResourceName("unknown.com/gpu"): resource.MustParse("1"),
 				},
 			},
+			annotations:     map[string]string{},
 			expectedUnknown: true,
 		},
+		"MixedResourcesWithAnnotation": {
+			resources: corev1.ResourceRequirements{
+				Limits: corev1.ResourceList{
+					corev1.ResourceCPU:                    resource.MustParse("1"),
+					corev1.ResourceMemory:                 resource.MustParse("1Gi"),
+					corev1.ResourceName("nvidia.com/gpu"): resource.MustParse("1"),
+				},
+				Requests: corev1.ResourceList{
+					corev1.ResourceCPU:                       resource.MustParse("1"),
+					corev1.ResourceMemory:                    resource.MustParse("1Gi"),
+					corev1.ResourceName("unknown.com-2/gpu"): resource.MustParse("1"),
path: pkg/utils/utils_test.go
user: login: Jooho
body: Your comment adds value, but I feel this feature might be unnecessary at this point. If a real need arises in the future, we can consider adding it then.
created_at: 2025-03-24T15:57:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -276,46 +277,49 @@ func IsUnknownGpuResourceType(resources corev1.ResourceRequirements, customGpuRe
 	addNonBasicResources(resources.Limits)
 	addNonBasicResources(resources.Requests)
 
-	// Validate GPU resource types
-	// If CustomGPUResourceTypesAnnotationKey is set, the specified custom GPU resource will be added to the available GPUResourceTypeList.
-	if customGpuResourceTypes != "" {
-		constants.GPUResourceTypeList = append(constants.GPUResourceTypeList, strings.Split(customGpuResourceTypes, ",")...)
+	// Update GPU resource type list
+	newGPUResourceTypeList, err := UpdateGPUResourceTypeListByAnnotation(annotations)
+	if err != nil {
+		return false, err
 	}
 
-	for _, gpuType := range constants.GPUResourceTypeList {
+	// Validate GPU resource types
+	for _, gpuType := range newGPUResourceTypeList {
 		allowedGPUResourceName := corev1.ResourceName(gpuType)
 		delete(possibleGPUResourceType, allowedGPUResourceName) // Remove allowed GPU resource if exists
 	}
 
 	// Return true if there are unknown GPU resources
-	return len(possibleGPUResourceType) > 0
+	return len(possibleGPUResourceType) > 0, nil
 }
 
 // IsValidCustomGPUArray checks if the input string is a valid JSON array of strings.
 // It returns false if the array is empty, contains empty strings, or any non-string elements.
-func IsValidCustomGPUArray(s string) bool {
+// Otherwise, it returns true and the list of custom GPU types.
+func IsValidCustomGPUArray(s string) ([]string, bool) {
path: pkg/utils/utils.go
user: login: Jooho
body: Annotations in Kubernetes can only store strings, so even if we input something like ["a","b","c"], it still needs to be parsed into a string array.

In the end, whether we store it as a JSON string or a plain string array, parsing is required either way. Are there specific advantages to treating it as an array in this case?
created_at: 2025-03-24T16:00:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -167,6 +167,15 @@ func main() {
 		os.Exit(1)
 	}
 
+	// Update Global GPU Resource Type List when custom GPU resource types are provided
path: cmd/manager/main.go
user: login: Jooho
body: updated
created_at: 2025-03-24T16:01:32Z
author_association: CONTRIBUTOR

PR comments:
user: login: Jooho
created_at: 2025-03-13T22:08:42Z
body: /rerun-all
author_association: CONTRIBUTOR
user: login: Jooho
created_at: 2025-03-14T13:33:26Z
body: /rerun-all
author_association: CONTRIBUTOR
user: login: Jooho
created_at: 2025-03-18T18:04:53Z
body: /rerun-all
author_association: CONTRIBUTOR
user: login: Jooho
created_at: 2025-03-21T13:13:41Z
body: /rerun-all
author_association: CONTRIBUTOR
user: login: VedantMahabaleshwarkar
created_at: 2025-03-21T17:16:08Z
body: /lgtm
author_association: CONTRIBUTOR
user: login: Jooho
created_at: 2025-03-24T16:35:39Z
body: /rerun-all
author_association: CONTRIBUTOR
user: login: Jooho
created_at: 2025-03-24T18:26:21Z
body: /rerun-all
author_association: CONTRIBUTOR
user: login: Jooho
created_at: 2025-03-25T14:28:58Z
body: /rerun-all
author_association: CONTRIBUTOR
user: login: Jooho
created_at: 2025-03-25T18:16:24Z
body: /rerun-all
author_association: CONTRIBUTOR