Repository: kubernetes-sigs/aws-ebs-csi-driver

Issue:
Volume Attachment Capacity Issue: regression in v1.49.0
/kind bug


**What happened?**

After upgrading the AWS EBS CSI Driver from `v1.47.0` to `v1.50.2` in our clusters, we observed a [Volume Attachment Capacity Issue (FAQ)](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/faq.md#volume-attachment-capacity-issues) resulting in pod creation failures. Specifically, stateful pods were scheduled to nodes that did not have a free slot for EBS volume attachment, causing the pods to remain in a pending state until a volume was detached from the node.

Upon reviewing recent source code changes, we identified a potential **regression** introduced in version `v1.49.0` with the PR:

- #2660

cc @ConnorJC3 to raise awareness of this issue.

**What did you expect to happen?**

The driver should correctly set the volume attachment capacity limit for all VM instance types, in alignment with AWS documentation and actual instance capabilities.

**How to reproduce it (as minimally and precisely as possible)?**

We have not attempted to reproduce the issue in a minimal environment. However, the problem was consistently observed in our production setup, and our source code review provides sufficient evidence to report this issue.


**Anything else we need to know?**

- PR #2660 introduced a utility to generate a volume limits table: see [hack/generate-volume-limits-table/main.go](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/74ac0e5afdb8d05a7424d3485d8fa156b843da82/hack/generate-volume-limits-table/main.go).
	- The `volumeLimits` map contains instance types with non-standard volume limits. Non-nitro instances, and nitro instances with the default limit (**shared 27**) are excluded.
	- Instance types with the default limit of **27** slots are therefore not explicitly listed in the `volumeLimits` map (see [main.go#L57-L63](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/74ac0e5afdb8d05a7424d3485d8fa156b843da82/hack/generate-volume-limits-table/main.go#L57-L63), [main.go#L182](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/74ac0e5afdb8d05a7424d3485d8fa156b843da82/hack/generate-volume-limits-table/main.go#L182), and [volume_limits_table.go#L93-L95](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/74ac0e5afdb8d05a7424d3485d8fa156b843da82/pkg/cloud/volume_limits_table.go#L93-L95)).
	- The value **27** is consistent with AWS documentation ([gp_storage-ebs](https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html#gp_storage-ebs)), which lists "Up to 27 (Shared limit)" for most instance types.

However, in [pkg/cloud/volume_limits.go#L54](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/74ac0e5afdb8d05a7424d3485d8fa156b843da82/pkg/cloud/volume_limits.go#L54), the `GetVolumeLimits(instanceType string)` function, after checking the `volumeLimits` map, falls back to returning **28** as the default limit.

This fallback is inconsistent with both the generated volume limits table and the AWS documentation. The default should be **27**, not **28**.

As a result, the AWS EBS CSI Driver currently assumes a limit that is one higher than the actual value for all instance types with the default limit of 27 slots. This leads to the volume attachment capacity misalignment described above.

The fix appears straightforward: change the default in [pkg/cloud/volume_limits.go#L54](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/74ac0e5afdb8d05a7424d3485d8fa156b843da82/pkg/cloud/volume_limits.go#L54) from `28` to `27`. We have not submitted a PR immediately, as testing a local build of the driver was not feasible within our current time frame.


**Additional notes:**

We believe the source of confusion is as follows:

The general description of EBS volume attachment limits in the [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/volume_limits.html#shared-limit) states:

> A volume attachment limit is **shared** between Amazon EBS volumes, network interfaces, and NVMe instance store volumes.
> You can attach any number of Amazon EBS volumes up to that limit, **less the number of attached network interfaces** and NVMe instance store volumes.
>
> Most Nitro instances support a maximum of **28** attachments.
> Keep in mind that every instance **must have at least one network interface**.
> Example: With an m5.xlarge instance with only the primary network interface, you can attach 27 EBS volumes. **28** volumes **- 1 network interface** = **27**

Thus, while the general limit is **28**, every instance has a mandatory primary network interface that counts towards this limit, resulting in an effective EBS volume attachment limit of **27**.

On the [instance type limits page](https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html#gp_storage-ebs), the default limit is consistently listed as **27**, with no mention of the general shared limit of 28. It appears AWS has already subtracted 1 for the mandatory network interface in these tables and in the metadata service, which is reflected in the generated volume limits table in the CSI driver codebase.

This implicit deduction is consistent with the implementation in [pkg/driver/node.go#L793-L798](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/74ac0e5afdb8d05a7424d3485d8fa156b843da82/pkg/driver/node.go#L793-L798), where the available attachments are calculated by subtracting one less than the number of network interfaces, since the primary interface is already accounted for in the default limit of 27.

cc @dimityrmirchev @dguendisch @kon-angelo @hebelsan

**Environment**
- Kubernetes version (use `kubectl version`): `1.33.4`
- Driver version: `v1.49.0` (the regression was likely introduced in `v1.49.0` and affects all subsequent versions)

Discussion:
@gmauleon: I wonder if this is really fixed, we still have the issue here, and there is a thread over at https://github.com/kubernetes/autoscaler/issues/8241 as well.@gmauleon: Ohh
> Today, CSI plugins report node attachment capacity only once, at startup, via the `NodeGetInfo` RPC.

So node need to be rolled out in order for the change to take effect?@ConnorJC3: @gmauleon I can confirm the above bug is fixed in the latest patch versions of the EBS CSI Driver on all minor releases. Generally speaking restarting the EBS CSI node pod for the update should cause Kubernetes to re-query the device count, but to be certain you could reboot/recreate the nodes.

If you're dynamically attaching additional ENIs past those present at startup (which is unrelated to this issue), you need a Kubernetes installation with the `MutableCSINodeAllocatableCount` feature enabled, and an EBS CSI Driver installation using the IMDS metadata source. More details and specific requirements are available in our docs: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/faq.md#mutablecsinodeallocatablecount-kubernetes-feature

Solution patch:
diff --git a/charts/aws-ebs-csi-driver/CHANGELOG.md b/charts/aws-ebs-csi-driver/CHANGELOG.md
index ffc9bbf2..3fc8e2b8 100644
--- a/charts/aws-ebs-csi-driver/CHANGELOG.md
+++ b/charts/aws-ebs-csi-driver/CHANGELOG.md
@@ -5,6 +5,7 @@
 ### Feature
 
 - Bump driver version to `v1.52.0`.
+- Add Helm parameter `node.serviceAccount.disableMutation` to disable mutating RBAC permissions to the `ebs-csi-node` service account. When enabled, driver features such as taint removal may not function. ([#2723](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/2723), [@ConnorJC3](https://github.com/ConnorJC3))
 - Add ALPHA metadata-labeler sidecar and metadata source ([#2591](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/2591), [@sylviah23](https://github.com/sylviah23))
 
 ## 2.51.1
diff --git a/pkg/cloud/volume_limits.go b/pkg/cloud/volume_limits.go
index 0b934c21..2cdca227 100644
--- a/pkg/cloud/volume_limits.go
+++ b/pkg/cloud/volume_limits.go
@@ -50,8 +50,8 @@ func GetVolumeLimits(instanceType string) (int, string) {
 		return limit.maxAttachments, limit.attachmentType
 	}
 
-	// Default to shared limit of 28
-	return 28, util.AttachmentShared
+	// Default to shared limit of 27
+	return 27, util.AttachmentShared
 }
 
 // KnownInstanceTypes returns all known instance types from the limits table.



Test patch:
diff --git a/pkg/driver/node_test.go b/pkg/driver/node_test.go
index 0288f081..d67d5f34 100644
--- a/pkg/driver/node_test.go
+++ b/pkg/driver/node_test.go
@@ -1073,6 +1073,21 @@ func TestGetVolumesLimit(t *testing.T) {
 				return m
 			},
 		},
+		{
+			name: "m5.large_volume_attach_limit",
+			options: &Options{
+				VolumeAttachLimit:         -1,
+				ReservedVolumeAttachments: -1,
+			},
+			expectedVal: 27,
+			metadataMock: func(ctrl *gomock.Controller) *metadata.MockMetadataService {
+				m := metadata.NewMockMetadataService(ctrl)
+				m.EXPECT().GetNumBlockDeviceMappings().Return(0)
+				m.EXPECT().GetInstanceType().Return("m5.large")
+				m.EXPECT().GetNumAttachedENIs().Return(0)
+				return m
+			},
+		},
 		{
 			name: "ReservedVolumeAttachments_specified",
 			options: &Options{
diff --git a/tests/e2e/metadata-labeler.go b/tests/e2e/metadata-labeler.go
index 7c2c8e27..923cea54 100644
--- a/tests/e2e/metadata-labeler.go
+++ b/tests/e2e/metadata-labeler.go
@@ -171,7 +171,7 @@ var _ = framework.Describe("[ebs-csi-e2e] [Disruptive] Metadata Labeler Sidecar"
 
 // getAllocatableCount returns the limit of volumes that the node supports.
 func getAllocatableCount(volumes, enis int) int32 {
-	return int32(28 - volumes - enis)
+	return int32(27 - volumes - enis)
 }
 
 // getVolENIs gets the expected metadata of each instance from the ec2 API