Repository: kubernetes-sigs/aws-ebs-csi-driver
Pull request #2108 (2024-08-12 15:17:54)

Fix gpus not being considered when counting allocatables
**Is this a bug fix or adding new feature?**
bug fix to address part of gh [#2105](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/2105) gpus not being considered when counting allocatables.

**What is this PR about? / Why do we need it?**
We need this PR in order to fix the above referenced bug

**What testing is done?** 
Compared manually to [list of gpu instances](https://aws.amazon.com/ec2/instance-types/)
and created unit tests for instances mentioned in ticket and for a gpu instance with 4 gpus.

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
**Once this PR has been reviewed and has the lgtm label**, please assign [leakingtapan](https://github.com/leakingtapan) for approval. For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Faws-ebs-csi-driver).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["leakingtapan"]} -->
torredil: /ok-to-test
rdpsin: Should we do some manual testing by setting up a cluster with GPU instances?
ElijahQuinones: > Should we do some manual testing by setting up a cluster with GPU instances?

I have manually tested the limit outside of Kubernettes entirely by just spinning up a g4ad.xlarge instance and mannualy attaching ebs volumes until I hit the limit where all volumes afterwards are stuck in the attaching state this test concluded that 24 volumes could be attached to the g4ad.xlarge which complies with what the documentation provides.

starts at 28
-1 eni
-1 root volume
-1 gpu 
-1 instance store 
we also ens up with 24 

Additionally the unit tests for a g4ad.xlarge return the expected value of 24 slots.

however if we feel it would cover more bases I can run the test with a g4ad.xlarge cluster just to make sure there is no gotchas.

Patch:
diff --git a/charts/aws-ebs-csi-driver/templates/clusterrole-attacher.yaml b/charts/aws-ebs-csi-driver/templates/clusterrole-attacher.yaml
index bff6577b..76022228 100644
--- a/charts/aws-ebs-csi-driver/templates/clusterrole-attacher.yaml
+++ b/charts/aws-ebs-csi-driver/templates/clusterrole-attacher.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 ---
 kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
@@ -24,3 +25,4 @@ rules:
   {{- with .Values.sidecars.attacher.additionalClusterRoleRules }}
     {{- . | toYaml | nindent 2 }}
   {{- end }}
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/clusterrole-provisioner.yaml b/charts/aws-ebs-csi-driver/templates/clusterrole-provisioner.yaml
index cb69cc8d..74337903 100644
--- a/charts/aws-ebs-csi-driver/templates/clusterrole-provisioner.yaml
+++ b/charts/aws-ebs-csi-driver/templates/clusterrole-provisioner.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 ---
 kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
@@ -39,3 +40,4 @@ rules:
   {{- with .Values.sidecars.provisioner.additionalClusterRoleRules }}
     {{- . | toYaml | nindent 2 }}
   {{- end }}
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/clusterrole-resizer.yaml b/charts/aws-ebs-csi-driver/templates/clusterrole-resizer.yaml
index 81858af3..48b5421a 100644
--- a/charts/aws-ebs-csi-driver/templates/clusterrole-resizer.yaml
+++ b/charts/aws-ebs-csi-driver/templates/clusterrole-resizer.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 ---
 kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
@@ -35,3 +36,4 @@ rules:
   {{- with .Values.sidecars.resizer.additionalClusterRoleRules }}
     {{- . | toYaml | nindent 2 }}
   {{- end }}
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/clusterrole-snapshotter.yaml b/charts/aws-ebs-csi-driver/templates/clusterrole-snapshotter.yaml
index 697e818d..8af9d993 100644
--- a/charts/aws-ebs-csi-driver/templates/clusterrole-snapshotter.yaml
+++ b/charts/aws-ebs-csi-driver/templates/clusterrole-snapshotter.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 ---
 kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
@@ -28,3 +29,4 @@ rules:
   {{- with .Values.sidecars.snapshotter.additionalClusterRoleRules }}
     {{- . | toYaml | nindent 2 }}
   {{- end }}
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/clusterrolebinding-attacher.yaml b/charts/aws-ebs-csi-driver/templates/clusterrolebinding-attacher.yaml
index bb23044b..32c41961 100644
--- a/charts/aws-ebs-csi-driver/templates/clusterrolebinding-attacher.yaml
+++ b/charts/aws-ebs-csi-driver/templates/clusterrolebinding-attacher.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 ---
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
@@ -13,3 +14,4 @@ roleRef:
   kind: ClusterRole
   name: ebs-external-attacher-role
   apiGroup: rbac.authorization.k8s.io
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/clusterrolebinding-provisioner.yaml b/charts/aws-ebs-csi-driver/templates/clusterrolebinding-provisioner.yaml
index 9d2749ac..32008486 100644
--- a/charts/aws-ebs-csi-driver/templates/clusterrolebinding-provisioner.yaml
+++ b/charts/aws-ebs-csi-driver/templates/clusterrolebinding-provisioner.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 ---
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
@@ -13,3 +14,4 @@ roleRef:
   kind: ClusterRole
   name: ebs-external-provisioner-role
   apiGroup: rbac.authorization.k8s.io
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/clusterrolebinding-resizer.yaml b/charts/aws-ebs-csi-driver/templates/clusterrolebinding-resizer.yaml
index 88cb47de..f0c694fe 100644
--- a/charts/aws-ebs-csi-driver/templates/clusterrolebinding-resizer.yaml
+++ b/charts/aws-ebs-csi-driver/templates/clusterrolebinding-resizer.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 ---
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
@@ -13,3 +14,4 @@ roleRef:
   kind: ClusterRole
   name: ebs-external-resizer-role
   apiGroup: rbac.authorization.k8s.io
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/clusterrolebinding-snapshotter.yaml b/charts/aws-ebs-csi-driver/templates/clusterrolebinding-snapshotter.yaml
index 2d429054..4c349efb 100644
--- a/charts/aws-ebs-csi-driver/templates/clusterrolebinding-snapshotter.yaml
+++ b/charts/aws-ebs-csi-driver/templates/clusterrolebinding-snapshotter.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 ---
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
@@ -13,3 +14,4 @@ roleRef:
   kind: ClusterRole
   name: ebs-external-snapshotter-role
   apiGroup: rbac.authorization.k8s.io
+{{- end -}}
diff --git a/charts/aws-ebs-csi-driver/templates/controller.yaml b/charts/aws-ebs-csi-driver/templates/controller.yaml
index 738d927b..4053ee18 100644
--- a/charts/aws-ebs-csi-driver/templates/controller.yaml
+++ b/charts/aws-ebs-csi-driver/templates/controller.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 # Controller Service
 kind: Deployment
 apiVersion: apps/v1
@@ -518,3 +519,4 @@ spec:
       dnsConfig:
         {{- toYaml .Values.controller.dnsConfig | nindent 8 }}
       {{- end }}
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/csidriver.yaml b/charts/aws-ebs-csi-driver/templates/csidriver.yaml
index a46d4b50..a78eb58d 100644
--- a/charts/aws-ebs-csi-driver/templates/csidriver.yaml
+++ b/charts/aws-ebs-csi-driver/templates/csidriver.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 apiVersion: {{ ternary "storage.k8s.io/v1" "storage.k8s.io/v1beta1" (semverCompare ">=1.18.0-0" .Capabilities.KubeVersion.Version) }}
 kind: CSIDriver
 metadata:
@@ -10,3 +11,4 @@ spec:
   {{- if not .Values.useOldCSIDriver }}
   fsGroupPolicy: File
   {{- end }}
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/ebs-csi-default-sc.yaml b/charts/aws-ebs-csi-driver/templates/ebs-csi-default-sc.yaml
index a5859572..95d74387 100644
--- a/charts/aws-ebs-csi-driver/templates/ebs-csi-default-sc.yaml
+++ b/charts/aws-ebs-csi-driver/templates/ebs-csi-default-sc.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 {{- if .Values.defaultStorageClass.enabled }}
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
@@ -9,3 +10,4 @@ provisioner: ebs.csi.aws.com
 volumeBindingMode: WaitForFirstConsumer
 allowVolumeExpansion: true
 {{- end }}
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/metrics.yaml b/charts/aws-ebs-csi-driver/templates/metrics.yaml
index d68bd7ab..9792af4e 100644
--- a/charts/aws-ebs-csi-driver/templates/metrics.yaml
+++ b/charts/aws-ebs-csi-driver/templates/metrics.yaml
@@ -1,4 +1,4 @@
-{{- if .Values.controller.enableMetrics -}}
+{{- if and .Values.controller.enableMetrics (not .Values.nodeComponentOnly) -}}
 ---
 apiVersion: v1
 kind: Service
diff --git a/charts/aws-ebs-csi-driver/templates/poddisruptionbudget-controller.yaml b/charts/aws-ebs-csi-driver/templates/poddisruptionbudget-controller.yaml
index 0a1e97cc..36fed9bc 100644
--- a/charts/aws-ebs-csi-driver/templates/poddisruptionbudget-controller.yaml
+++ b/charts/aws-ebs-csi-driver/templates/poddisruptionbudget-controller.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 apiVersion: policy/v1
 kind: PodDisruptionBudget
 metadata:
@@ -15,3 +16,4 @@ spec:
   {{- else }}
   minAvailable: 2
   {{- end }}
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/role-leases.yaml b/charts/aws-ebs-csi-driver/templates/role-leases.yaml
index 1ec62bb4..f1260c0a 100644
--- a/charts/aws-ebs-csi-driver/templates/role-leases.yaml
+++ b/charts/aws-ebs-csi-driver/templates/role-leases.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 kind: Role
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
@@ -9,3 +10,4 @@ rules:
 - apiGroups: ["coordination.k8s.io"]
   resources: ["leases"]
   verbs: ["get", "watch", "list", "delete", "update", "create"]
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/rolebinding-leases.yaml b/charts/aws-ebs-csi-driver/templates/rolebinding-leases.yaml
index 88fded8a..f2826cbc 100644
--- a/charts/aws-ebs-csi-driver/templates/rolebinding-leases.yaml
+++ b/charts/aws-ebs-csi-driver/templates/rolebinding-leases.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.nodeComponentOnly -}}
 kind: RoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
@@ -13,3 +14,4 @@ roleRef:
   kind: Role
   name: ebs-csi-leases-role
   apiGroup: rbac.authorization.k8s.io
+{{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/serviceaccount-csi-controller.yaml b/charts/aws-ebs-csi-driver/templates/serviceaccount-csi-controller.yaml
index d819f549..bfb8c4bc 100644
--- a/charts/aws-ebs-csi-driver/templates/serviceaccount-csi-controller.yaml
+++ b/charts/aws-ebs-csi-driver/templates/serviceaccount-csi-controller.yaml
@@ -1,4 +1,4 @@
-{{- if .Values.controller.serviceAccount.create -}}
+{{- if and .Values.controller.serviceAccount.create (not .Values.nodeComponentOnly) -}}
 apiVersion: v1
 kind: ServiceAccount
 metadata:
diff --git a/charts/aws-ebs-csi-driver/values.yaml b/charts/aws-ebs-csi-driver/values.yaml
index 75a44574..6089d835 100644
--- a/charts/aws-ebs-csi-driver/values.yaml
+++ b/charts/aws-ebs-csi-driver/values.yaml
@@ -476,6 +476,8 @@ volumeSnapshotClasses: []
 # Intended for use with older clusters that cannot easily replace the CSIDriver object
 # This parameter should always be false for new installations
 useOldCSIDriver: false
+# Deploy EBS CSI Driver without controller and associated resources
+nodeComponentOnly: false
 helmTester:
   enabled: true
   # Supply a custom image to the ebs-csi-driver-test pod in helm-tester.yaml
diff --git a/hack/generate-gpu-count-table.sh b/hack/generate-gpu-count-table.sh
new file mode 100755
index 00000000..00ae0e4c
--- /dev/null
+++ b/hack/generate-gpu-count-table.sh
@@ -0,0 +1,37 @@
+# Copyright 2024 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the 'License');
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an 'AS IS' BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Generates gpu table for `pkg/cloud/volume_limits.go` from the AWS API
+# Ensure you are opted into all opt-in regions before running
+# Ensure your account isn't in any private instance type betas before running
+
+set -euo pipefail
+
+BIN="$(dirname "$(realpath "${BASH_SOURCE[0]}")")/../bin"
+
+function get_gpus_for_region() {
+  REGION="${1}"
+  echo "Getting gpu counts for ${REGION}..." >&2
+  "${BIN}/aws" ec2 describe-instance-types --region "${REGION}" --query "InstanceTypes[?GpuInfo!=null].[InstanceType, GpuInfo]" |
+    jq -r 'map("\"" + .[0] + "\": " + (.[1].Gpus | map(.Count) | add | tostring) + ",") | .[]'
+}
+
+function get_all_gpus() {
+  "${BIN}/aws" account list-regions --max-results 50 | jq -r '.Regions | map(.RegionName) | .[]' | while read REGION; do
+    sleep 1
+    get_gpus_for_region $REGION &
+  done
+}
+
+get_all_gpus | sort | uniq
diff --git a/hack/generate-instance-store-table.sh b/hack/generate-instance-store-table.sh
index b4b4417e..f94e3cba 100755
--- a/hack/generate-instance-store-table.sh
+++ b/hack/generate-instance-store-table.sh
@@ -24,14 +24,15 @@ BIN="$(dirname "$(realpath "${BASH_SOURCE[0]}")")/../bin"
 
 function get_instance_stores_for_region() {
   REGION="${1}"
-  echo "Getting limits for ${REGION}..." >&2
+  echo "Getting instance store limits for ${REGION}..." >&2
   "${BIN}/aws" ec2 describe-instance-types --region "${REGION}" --filters "Name=instance-storage-supported,Values=true" --query "InstanceTypes[].[InstanceType, InstanceStorageInfo]" |
     jq -r 'map("\"" + .[0] + "\": " + (.[1].Disks | map(.Count) | add | tostring) + ",") | .[]'
 }
 
 function get_all_instance_stores() {
   "${BIN}/aws" account list-regions --max-results 50 | jq -r '.Regions | map(.RegionName) | .[]' | while read REGION; do
-    get_instance_stores_for_region $REGION
+    sleep 1
+    get_instance_stores_for_region $REGION &
   done
 }
 
diff --git a/pkg/cloud/volume_limits.go b/pkg/cloud/volume_limits.go
index 901a449b..b925e5bd 100644
--- a/pkg/cloud/volume_limits.go
+++ b/pkg/cloud/volume_limits.go
@@ -138,11 +138,18 @@ func GetDedicatedLimitForInstanceType(it string) int {
 	}
 }
 
-func GetNVMeInstanceStoreVolumesForInstanceType(it string) int {
-	if v, ok := nvmeInstanceStoreVolumes[it]; ok {
-		return v
+// GetReservedSlotsForInstanceType calculates how many attachment slots are already used up by other devices on shared EBS volume limit instances.
+func GetReservedSlotsForInstanceType(it string) int {
+	total := 0
+	nvmeInstanceStoreVolumes, ok := nvmeInstanceStoreVolumes[it]
+	if ok {
+		total += nvmeInstanceStoreVolumes
 	}
-	return 0
+	gpus, ok := gpuInstanceGpus[it]
+	if ok {
+		total += gpus
+	}
+	return total
 }
 
 // / https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-store-volumes.html
@@ -489,3 +496,58 @@ var nvmeInstanceStoreVolumes = map[string]int{
 	"z1d.metal":       2,
 	"z1d.xlarge":      1,
 }
+
+// / https://aws.amazon.com/ec2/instance-types
+var gpuInstanceGpus = map[string]int{
+	"dl1.24xlarge":  8,
+	"g3.16xlarge":   4,
+	"g3.4xlarge":    1,
+	"g3.8xlarge":    2,
+	"g3s.xlarge":    1,
+	"g4ad.16xlarge": 4,
+	"g4ad.2xlarge":  1,
+	"g4ad.4xlarge":  1,
+	"g4ad.8xlarge":  2,
+	"g4ad.xlarge":   1,
+	"g4dn.12xlarge": 4,
+	"g4dn.16xlarge": 1,
+	"g4dn.2xlarge":  1,
+	"g4dn.4xlarge":  1,
+	"g4dn.8xlarge":  1,
+	"g4dn.metal":    8,
+	"g4dn.xlarge":   1,
+	"g5.12xlarge":   4,
+	"g5.16xlarge":   1,
+	"g5.24xlarge":   4,
+	"g5.2xlarge":    1,
+	"g5.48xlarge":   8,
+	"g5.4xlarge":    1,
+	"g5.8xlarge":    1,
+	"g5g.16xlarge":  2,
+	"g5g.2xlarge":   1,
+	"g5g.4xlarge":   1,
+	"g5g.8xlarge":   1,
+	"g5g.metal":     2,
+	"g5g.xlarge":    1,
+	"g5.xlarge":     1,
+	"g6.12xlarge":   4,
+	"g6.16xlarge":   1,
+	"g6.24xlarge":   4,
+	"g6.2xlarge":    1,
+	"g6.48xlarge":   8,
+	"g6.4xlarge":    1,
+	"g6.8xlarge":    1,
+	"g6.xlarge":     1,
+	"gr6.4xlarge":   1,
+	"gr6.8xlarge":   1,
+	"p2.16xlarge":   16,
+	"p2.8xlarge":    8,
+	"p2.xlarge":     1,
+	"p3.16xlarge":   8,
+	"p3.2xlarge":    1,
+	"p3.8xlarge":    4,
+	"p3dn.24xlarge": 8,
+	"p4d.24xlarge":  8,
+	"p4de.24xlarge": 8,
+	"p5.48xlarge":   8,
+}
diff --git a/pkg/driver/node.go b/pkg/driver/node.go
index ca634418..80979c04 100644
--- a/pkg/driver/node.go
+++ b/pkg/driver/node.go
@@ -791,7 +791,7 @@ func (d *NodeService) getVolumesLimit() int64 {
 		availableAttachments = dedicatedLimit
 	} else if isNitro {
 		enis := d.metadata.GetNumAttachedENIs()
-		nvmeInstanceStoreVolumes := cloud.GetNVMeInstanceStoreVolumesForInstanceType(instanceType)
+		nvmeInstanceStoreVolumes := cloud.GetReservedSlotsForInstanceType(instanceType)
 		availableAttachments = availableAttachments - enis - nvmeInstanceStoreVolumes
 	}
 	availableAttachments = availableAttachments - reservedVolumeAttachments


Test patch:
diff --git a/charts/aws-ebs-csi-driver/templates/tests/helm-tester.yaml b/charts/aws-ebs-csi-driver/templates/tests/helm-tester.yaml
index 08113b7c..6c618b0a 100644
--- a/charts/aws-ebs-csi-driver/templates/tests/helm-tester.yaml
+++ b/charts/aws-ebs-csi-driver/templates/tests/helm-tester.yaml
@@ -1,4 +1,4 @@
-{{- if .Values.helmTester.enabled -}}
+{{- if and .Values.helmTester.enabled (not .Values.nodeComponentOnly) -}}
 ---
 apiVersion: v1
 kind: ServiceAccount
diff --git a/pkg/driver/node_test.go b/pkg/driver/node_test.go
index d7536e33..1abe1c65 100644
--- a/pkg/driver/node_test.go
+++ b/pkg/driver/node_test.go
@@ -1183,6 +1183,56 @@ func TestGetVolumesLimit(t *testing.T) {
 				return m
 			},
 		},
+		{
+			name: "g4dn.xlarge_volume_attach_limit (1 GPU 1 InstanceStoreVolume)",
+			options: &Options{
+				VolumeAttachLimit:         -1,
+				ReservedVolumeAttachments: -1,
+			},
+			expectedVal: 24,
+			metadataMock: func(ctrl *gomock.Controller) *metadata.MockMetadataService {
+				m := metadata.NewMockMetadataService(ctrl)
+				m.EXPECT().GetRegion().Return("us-west-2")
+				m.EXPECT().GetInstanceType().Return("g4dn.xlarge")
+				m.EXPECT().GetNumBlockDeviceMappings().Return(0)
+				m.EXPECT().GetNumAttachedENIs().Return(1)
+				return m
+			},
+		},
+		// 1 gpu
+		{
+			name: "g4ad.xlarge_volume_attach_limit (1 GPU 1 InstanceStoreVolume)",
+			options: &Options{
+				VolumeAttachLimit:         -1,
+				ReservedVolumeAttachments: -1,
+			},
+			expectedVal: 24,
+			metadataMock: func(ctrl *gomock.Controller) *metadata.MockMetadataService {
+				m := metadata.NewMockMetadataService(ctrl)
+				m.EXPECT().GetRegion().Return("us-west-2")
+				m.EXPECT().GetInstanceType().Return("g4ad.xlarge")
+				m.EXPECT().GetNumBlockDeviceMappings().Return(0)
+				m.EXPECT().GetNumAttachedENIs().Return(1)
+				return m
+			},
+		},
+		// 4 gpus
+		{
+			name: "g4dn.12xlarge_volume_attach_limit (4 GPUS, 1 InstanceStoreVolume)",
+			options: &Options{
+				VolumeAttachLimit:         -1,
+				ReservedVolumeAttachments: -1,
+			},
+			expectedVal: 21,
+			metadataMock: func(ctrl *gomock.Controller) *metadata.MockMetadataService {
+				m := metadata.NewMockMetadataService(ctrl)
+				m.EXPECT().GetRegion().Return("us-west-2")
+				m.EXPECT().GetInstanceType().Return("g4dn.12xlarge")
+				m.EXPECT().GetNumBlockDeviceMappings().Return(0)
+				m.EXPECT().GetNumAttachedENIs().Return(1)
+				return m
+			},
+		},
 	}
 
 	for _, tc := range testCases {