Repository: kubernetes-sigs/aws-ebs-csi-driver
Pull request #2121 (2024-08-19 18:14:21)

Add node.legacyXFS parameter to fix XFS on nodes with kernel version ≤ 5.4
**Is this a bug fix or adding new feature?**

Feature

**What is this PR about? / Why do we need it?**

Adds workaround for customers using XFS on nodes with linux kernel version ≤ 5.4.

Warning: This option will be removed in a future release. It is a temporary workaround for users unable to immediately migrate off of older kernel versions.

Adds `legacy-xfs` node driver option and and `node.legacyXFS` helm chart parameter which formats XFS volumes with `bigtime=0,inobtcount=0,reflink=0`, so that they can be mounted onto nodes with linux kernel ≤ 5.4. Volumes formatted with this option may experience issues after 2038, and will be unable to use some XFS features (for example, reflinks).

**What testing is done?** 

Created node with Linux ≤ v5.4, attempted to run stateful pod with and without `legacy-xfs` parameter. Volume only mounted when legacy-xfs=true.


k8s-ci-robot: Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with `/test all`
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
**Once this PR has been reviewed and has the lgtm label**, please ask for approval from [andrewsirenko](https://github.com/andrewsirenko). For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Faws-ebs-csi-driver).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["andrewsirenko"]} -->
AndrewSirenko: /hold

Will test on linux 5.4 today. 

But still ready for review. 
AndrewSirenko: /unhold

Customer tested with their custom AMI, driver was functional. 

Patch:
diff --git a/charts/aws-ebs-csi-driver/templates/_node-windows.tpl b/charts/aws-ebs-csi-driver/templates/_node-windows.tpl
index 25a42897..ee758cb7 100644
--- a/charts/aws-ebs-csi-driver/templates/_node-windows.tpl
+++ b/charts/aws-ebs-csi-driver/templates/_node-windows.tpl
@@ -68,12 +68,12 @@ spec:
           args:
             - node
             - --endpoint=$(CSI_ENDPOINT)
-            {{- with .Values.node.reservedVolumeAttachments }}
-            - --reserved-volume-attachments={{ . }}
-            {{- end }}
             {{- with .Values.node.volumeAttachLimit }}
             - --volume-attach-limit={{ . }}
             {{- end }}
+            {{- if .Values.node.legacyXFS }}
+            - --legacy-xfs=true
+            {{- end}}
             {{- with .Values.node.loggingFormat }}
             - --logging-format={{ . }}
             {{- end }}
diff --git a/charts/aws-ebs-csi-driver/templates/_node.tpl b/charts/aws-ebs-csi-driver/templates/_node.tpl
index c0c48143..a91b1b5e 100644
--- a/charts/aws-ebs-csi-driver/templates/_node.tpl
+++ b/charts/aws-ebs-csi-driver/templates/_node.tpl
@@ -74,6 +74,9 @@ spec:
             {{- with .Values.node.volumeAttachLimit }}
             - --volume-attach-limit={{ . }}
             {{- end }}
+            {{- if .Values.node.legacyXFS }}
+            - --legacy-xfs=true
+            {{- end}}
             {{- with .Values.node.loggingFormat }}
             - --logging-format={{ . }}
             {{- end }}
diff --git a/charts/aws-ebs-csi-driver/values.yaml b/charts/aws-ebs-csi-driver/values.yaml
index 242d4d9b..0d65ba3d 100644
--- a/charts/aws-ebs-csi-driver/values.yaml
+++ b/charts/aws-ebs-csi-driver/values.yaml
@@ -392,6 +392,10 @@ node:
   # Enable the linux daemonset creation
   enableLinux: true
   enableWindows: false
+  # Warning: This option will be removed in a future release. It is a temporary workaround for users unable to immediately migrate off of older kernel versions.
+  # Formats XFS volumes with bigtime=0,inobtcount=0,reflink=0, for mounting onto nodes with linux kernel version <= 5.4.
+  # Note that XFS volumes formatted with this option will only have timestamp records until 2038.
+  legacyXFS: false
   # The number of attachment slots to reserve for system use (and not to be used for CSI volumes)
   # When this parameter is not specified (or set to -1), the EBS CSI Driver will attempt to determine the number of reserved slots via heuristic
   # Cannot be specified at the same time as `node.volumeAttachLimit`
diff --git a/docs/snapshot.md b/docs/fast-snapshot-restores.md
similarity index 61%
rename from docs/snapshot.md
rename to docs/fast-snapshot-restores.md
index 5643cdf3..fe45a04f 100644
--- a/docs/snapshot.md
+++ b/docs/fast-snapshot-restores.md
@@ -1,23 +1,11 @@
-## Supported Parameters
-| Parameter                          | Description of value                                      |
-|------------------------------------|-----------------------------------------------------------|
-| fastSnapshotRestoreAvailabilityZones | Comma separated list of availability zones                |
-| outpostArn                         | Arn of the outpost you wish to have the snapshot saved to | 
-
-The AWS EBS CSI Driver supports [tagging](tagging.md) through `VolumeSnapshotClass.parameters` (in v1.6.0 and later). 
-## Prerequisites
-
-- Install the [Kubernetes Volume Snapshot CRDs](https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd) and external-snapshotter sidecar. For installation instructions, see [CSI Snapshotter Usage](https://github.com/kubernetes-csi/external-snapshotter#usage).
-
 # Fast Snapshot Restores
 
 The EBS CSI Driver provides support for [Fast Snapshot Restores(FSR)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-fast-snapshot-restore.html) via `VolumeSnapshotClass.parameters.fastSnapshotRestoreAvailabilityZones`.
 
+Amazon EBS fast snapshot restore (FSR) enables you to create a volume from a snapshot that is fully initialized at creation. This eliminates the latency of I/O operations on a block when it is accessed for the first time. Volumes that are created using fast snapshot restore instantly deliver all of their provisioned performance.
 
 Availability zones are specified as a comma separated list.
 
-- The EBS CSI Driver must be given permission to access the [`EnableFastSnapshotRestores` EC2 API](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_EnableFastSnapshotRestores.html). This example snippet can be used in an IAM policy to grant access to `EnableFastSnapshotRestores`:
-
 **Example**
 ```
 apiVersion: snapshot.storage.k8s.io/v1
@@ -30,6 +18,12 @@ parameters:
   fastSnapshotRestoreAvailabilityZones: "us-east-1a, us-east-1b"
 ```
 
+## Prerequisites
+
+- Install the [Kubernetes Volume Snapshot CRDs](https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd) and external-snapshotter sidecar. For installation instructions, see [CSI Snapshotter Usage](https://github.com/kubernetes-csi/external-snapshotter#usage).
+
+- The EBS CSI Driver must be given permission to access the [`EnableFastSnapshotRestores` EC2 API](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_EnableFastSnapshotRestores.html). This example snippet can be used in an IAM policy to grant access to `EnableFastSnapshotRestores`:
+
 ```json
 {
   "Effect": "Allow",
@@ -40,25 +34,6 @@ parameters:
 }
 ```
 
-## Failure Mode 
+## Failure Mode
 
 The driver will attempt to check if the availability zones provided are supported for fast snapshot restore before attempting to create the snapshot. If the `EnableFastSnapshotRestores` API call fails, the driver will hard-fail the request and delete the snapshot. This is to ensure that the snapshot is not left in an inconsistent state.
-
-
-# Amazon EBS Local Snapshots on Outposts
-
-The EBS CSI Driver provides support for [Amazon EBS local snapshots on Outposts](https://docs.aws.amazon.com/ebs/latest/userguide/snapshots-outposts.html) via `VolumeSnapshotClass.parameters.outpostArn`.
-
-
-
-**Example**
-```
-apiVersion: snapshot.storage.k8s.io/v1
-kind: VolumeSnapshotClass
-metadata:
-  name: csi-aws-vsc
-driver: ebs.csi.aws.com
-deletionPolicy: Delete
-parameters:
-  outpostarn: {arn of your outpost}
-```
diff --git a/docs/options.md b/docs/options.md
index 05951f2a..71ccb185 100644
--- a/docs/options.md
+++ b/docs/options.md
@@ -2,20 +2,21 @@
 
 There are a couple of driver options that can be passed as arguments when starting the driver container.
 
-| Option argument             | value sample                                      | default                                             | Description         |
-|-----------------------------|---------------------------------------------------|-----------------------------------------------------|---------------------|
-| endpoint                    | tcp://127.0.0.1:10000/                            | unix:///var/lib/csi/sockets/pluginproxy/csi.sock    | The socket on which the driver will listen for CSI RPCs|
-| http-endpoint               | :8080                                             |                                                     | The TCP network address where the HTTP server for metrics will listen (example: `:8080`). The default is empty string, which means the server is disabled.|
-| metrics-cert-file           | /metrics.crt                                      |                                                     | The path to a certificate to use for serving the metrics server over HTTPS. If the certificate is signed by a certificate authority, this file should be the concatenation of the server's certificate, any intermediates, and the CA's certificate. If this is non-empty, `--http-endpoint` and `--metrics-key-file` MUST also be non-empty.|
-| metrics-key-file            | /metrics.key                                      |                                                     | The path to a key to use for serving the metrics server over HTTPS. If this is non-empty, `--http-endpoint` and `--metrics-cert-file` MUST also be non-empty.|
-| volume-attach-limit         | 1,2,3 ...                                         | -1                                                  | Value for the maximum number of volumes attachable per node. If specified, the limit applies to all nodes. If not specified, the value is approximated from the instance type|
-| extra-tags                  | key1=value1,key2=value2                           |                                                     | Tags attached to each dynamically provisioned resource|
-| k8s-tag-cluster-id          | aws-cluster-id-1                                  |                                                     | ID of the Kubernetes cluster used for tagging provisioned EBS volumes|
-| aws-sdk-debug-log           | true                                              | false                                               | If set to true, the driver will enable the aws sdk debug log level|
-| logging-format              | json                                              | text                                                | Sets the log format. Permitted formats: text, json|
-| user-agent-extra            | csi-ebs                                           | helm                                                | Extra string appended to user agent|
-| enable-otel-tracing         | true                                              | false                                               | If set to true, the driver will enable opentelemetry tracing. Might need [additional env variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration) to export the traces to the right collector|
-| batching                    | true                                              | true                                                | If set to true, the driver will enable batching of API calls. This is especially helpful for improving performance in workloads that are sensitive to EC2 rate limits at the cost of a small increase to worst-case latency|
-| modify-volume-request-handler-timeout | 10s                                     | 2s                                                  | Timeout for the window in which volume modification calls must be received in order for them to coalesce into a single volume modification call to AWS. If changing this, be aware that the ebs-csi-controller's csi-resizer and volumemodifier containers both have timeouts on the calls they make, if this value exceeds those timeouts it will cause them to always fail and fall into a retry loop, so adjust those values accordingly.
-| warn-on-invalid-tag         | true                                              | false                                               | To warn on invalid tags, instead of returning an error|
-|reserved-volume-attachments  | 2                                                 | -1                                                  | Number of volume attachments reserved for system use. Not used when --volume-attach-limit is specified. When -1, the amount of reserved attachments is loaded from instance metadata that captured state at node boot and may include not only system disks but also CSI volumes.|
+| Option argument           | value sample            | default                                          | Description         |
+|---------------------------|-------------------------|--------------------------------------------------|---------------------|
+| endpoint                  | tcp://127.0.0.1:10000/  | unix:///var/lib/csi/sockets/pluginproxy/csi.sock | The socket on which the driver will listen for CSI RPCs|
+| http-endpoint             | :8080                   |                                                  | The TCP network address where the HTTP server for metrics will listen (example: `:8080`). The default is empty string, which means the server is disabled.|
+| metrics-cert-file         | /metrics.crt            |                                                  | The path to a certificate to use for serving the metrics server over HTTPS. If the certificate is signed by a certificate authority, this file should be the concatenation of the server's certificate, any intermediates, and the CA's certificate. If this is non-empty, `--http-endpoint` and `--metrics-key-file` MUST also be non-empty.|
+| metrics-key-file          | /metrics.key            |                                                  | The path to a key to use for serving the metrics server over HTTPS. If this is non-empty, `--http-endpoint` and `--metrics-cert-file` MUST also be non-empty.|
+| volume-attach-limit       | 1,2,3 ...               | -1                                               | Value for the maximum number of volumes attachable per node. If specified, the limit applies to all nodes. If not specified, the value is approximated from the instance type|
+| extra-tags                | key1=value1,key2=value2 |                                                  | Tags attached to each dynamically provisioned resource|
+| k8s-tag-cluster-id        | aws-cluster-id-1        |                                                  | ID of the Kubernetes cluster used for tagging provisioned EBS volumes|
+| aws-sdk-debug-log         | true                    | false                                            | If set to true, the driver will enable the aws sdk debug log level|
+| logging-format            | json                    | text                                             | Sets the log format. Permitted formats: text, json|
+| user-agent-extra          | csi-ebs                 | helm                                             | Extra string appended to user agent|
+| enable-otel-tracing       | true                    | false                                            | If set to true, the driver will enable opentelemetry tracing. Might need [additional env variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration) to export the traces to the right collector|
+| batching                  | true                    | true                                             | If set to true, the driver will enable batching of API calls. This is especially helpful for improving performance in workloads that are sensitive to EC2 rate limits at the cost of a small increase to worst-case latency|
+| modify-volume-request-handler-timeout | 10s                     | 2s                                               | Timeout for the window in which volume modification calls must be received in order for them to coalesce into a single volume modification call to AWS. If changing this, be aware that the ebs-csi-controller's csi-resizer and volumemodifier containers both have timeouts on the calls they make, if this value exceeds those timeouts it will cause them to always fail and fall into a retry loop, so adjust those values accordingly.
+| warn-on-invalid-tag       | true                    | false                                            | To warn on invalid tags, instead of returning an error|
+| reserved-volume-attachments | 2                       | -1                                               | Number of volume attachments reserved for system use. Not used when --volume-attach-limit is specified. When -1, the amount of reserved attachments is loaded from instance metadata that captured state at node boot and may include not only system disks but also CSI volumes.|
+| legacy-xfs                | true                    | false                                            | Warning: This option will be removed in a future release. It is a temporary workaround for users unable to immediately migrate off of older kernel versions. Formats XFS volumes with `bigtime=0,inobtcount=0,reflink=0`, so that they can be mounted onto nodes with linux kernel ≤ v5.4. Volumes formatted with this option may experience issues after 2038, and will be unable to use some XFS features (for example, reflinks).|
diff --git a/pkg/cloud/cloud.go b/pkg/cloud/cloud.go
index da71ef13..4e946008 100644
--- a/pkg/cloud/cloud.go
+++ b/pkg/cloud/cloud.go
@@ -252,8 +252,7 @@ type ListSnapshotsResponse struct {
 
 // SnapshotOptions represents parameters to create an EBS volume
 type SnapshotOptions struct {
-	Tags       map[string]string
-	OutpostArn string
+	Tags map[string]string
 }
 
 // ec2ListSnapshotsResponse is a helper struct returned from the AWS API calling function to the main ListSnapshots function
@@ -1276,7 +1275,6 @@ func (c *cloud) CreateSnapshot(ctx context.Context, volumeID string, snapshotOpt
 	descriptions := "Created by AWS EBS CSI driver for volume " + volumeID
 
 	var tags []types.Tag
-	var request *ec2.CreateSnapshotInput
 	for key, value := range snapshotOptions.Tags {
 		tags = append(tags, types.Tag{Key: aws.String(key), Value: aws.String(value)})
 	}
@@ -1284,14 +1282,12 @@ func (c *cloud) CreateSnapshot(ctx context.Context, volumeID string, snapshotOpt
 		ResourceType: types.ResourceTypeSnapshot,
 		Tags:         tags,
 	}
-	request = &ec2.CreateSnapshotInput{
+	request := &ec2.CreateSnapshotInput{
 		VolumeId:          aws.String(volumeID),
 		TagSpecifications: []types.TagSpecification{tagSpec},
 		Description:       aws.String(descriptions),
 	}
-	if snapshotOptions.OutpostArn != "" {
-		request.OutpostArn = aws.String(snapshotOptions.OutpostArn)
-	}
+
 	res, err := c.ec2.CreateSnapshot(ctx, request, func(o *ec2.Options) {
 		o.Retryer = c.rm.createSnapshotRetryer
 	})
diff --git a/pkg/driver/constants.go b/pkg/driver/constants.go
index 17d6ab69..7775bbbd 100644
--- a/pkg/driver/constants.go
+++ b/pkg/driver/constants.go
@@ -102,9 +102,6 @@ const (
 	// TagKeyPrefix contains the prefix of a volume parameter that designates it as
 	// a tag to be attached to the resource
 	TagKeyPrefix = "tagSpecification"
-
-	// OutpostArn represents key for outpost's arn
-	OutpostArnKey = "outpostarn"
 )
 
 // constants of keys in snapshot parameters
diff --git a/pkg/driver/controller.go b/pkg/driver/controller.go
index 5af42fba..7b460fd6 100644
--- a/pkg/driver/controller.go
+++ b/pkg/driver/controller.go
@@ -669,7 +669,6 @@ func (d *ControllerService) CreateSnapshot(ctx context.Context, req *csi.CreateS
 
 	snapshotName := req.GetName()
 	volumeID := req.GetSourceVolumeId()
-	var outpostArn string
 
 	// check if a request is already in-flight
 	if ok := d.inFlight.Insert(snapshotName); !ok {
@@ -710,12 +709,6 @@ func (d *ControllerService) CreateSnapshot(ctx context.Context, req *csi.CreateS
 		case FastSnapshotRestoreAvailabilityZones:
 			f := strings.ReplaceAll(value, " ", "")
 			fsrAvailabilityZones = strings.Split(f, ",")
-		case OutpostArnKey:
-			if arn.IsARN(value) {
-				outpostArn = value
-			} else {
-				return nil, status.Errorf(codes.InvalidArgument, "Invalid parameter value %s is not a valid arn", value)
-			}
 		default:
 			if strings.HasPrefix(key, TagKeyPrefix) {
 				vscTags = append(vscTags, value)
@@ -748,8 +741,7 @@ func (d *ControllerService) CreateSnapshot(ctx context.Context, req *csi.CreateS
 	}
 
 	opts := &cloud.SnapshotOptions{
-		Tags:       snapshotTags,
-		OutpostArn: outpostArn,
+		Tags: snapshotTags,
 	}
 
 	// Check if the availability zone is supported for fast snapshot restore
diff --git a/pkg/driver/node.go b/pkg/driver/node.go
index c77b8286..750fa6ee 100644
--- a/pkg/driver/node.go
+++ b/pkg/driver/node.go
@@ -274,6 +274,9 @@ func (d *NodeService) NodeStageVolume(ctx context.Context, req *csi.NodeStageVol
 	if len(ext4ClusterSize) > 0 {
 		formatOptions = append(formatOptions, "-C", ext4ClusterSize)
 	}
+	if fsType == FSTypeXfs && d.options.LegacyXFSProgs {
+		formatOptions = append(formatOptions, "-m", "bigtime=0,inobtcount=0,reflink=0")
+	}
 	err = d.mounter.FormatAndMountSensitiveWithFormatOptions(source, target, fsType, mountOptions, nil, formatOptions)
 	if err != nil {
 		msg := fmt.Sprintf("could not format %q and mount it at %q: %v", source, target, err)
diff --git a/pkg/driver/options.go b/pkg/driver/options.go
index 5dc13c4b..fc5017e9 100644
--- a/pkg/driver/options.go
+++ b/pkg/driver/options.go
@@ -87,6 +87,8 @@ type Options struct {
 	ReservedVolumeAttachments int
 	// ALPHA: WindowsHostProcess indicates whether the driver is running in a Windows privileged container
 	WindowsHostProcess bool
+	// LegacyXFSProgs formats XFS volumes with `bigtime=0,inobtcount=0,reflink=0`, so that they can be mounted onto nodes with linux kernel ≤ v5.4. Volumes formatted with this option may experience issues after 2038, and will be unable to use some XFS features (for example, reflinks).
+	LegacyXFSProgs bool
 }
 
 func (o *Options) AddFlags(f *flag.FlagSet) {
@@ -115,6 +117,7 @@ func (o *Options) AddFlags(f *flag.FlagSet) {
 		f.Int64Var(&o.VolumeAttachLimit, "volume-attach-limit", -1, "Value for the maximum number of volumes attachable per node. If specified, the limit applies to all nodes and overrides --reserved-volume-attachments. If not specified, the value is approximated from the instance type.")
 		f.IntVar(&o.ReservedVolumeAttachments, "reserved-volume-attachments", -1, "Number of volume attachments reserved for system use. Not used when --volume-attach-limit is specified. The total amount of volume attachments for a node is computed as: <nr. of attachments for corresponding instance type> - <number of NICs, if relevant to the instance type> - <reserved-volume-attachments value>. When -1, the amount of reserved attachments is loaded from instance metadata that captured state at node boot and may include not only system disks but also CSI volumes.")
 		f.BoolVar(&o.WindowsHostProcess, "windows-host-process", false, "ALPHA: Indicates whether the driver is running in a Windows privileged container")
+		f.BoolVar(&o.LegacyXFSProgs, "legacy-xfs", false, "Warning: This option will be removed in a future version of EBS CSI Driver. Formats XFS volumes with `bigtime=0,inobtcount=0,reflink=0`, so that they can be mounted onto nodes with linux kernel ≤ v5.4. Volumes formatted with this option may experience issues after 2038, and will be unable to use some XFS features (for example, reflinks).")
 	}
 }
 


Test patch:
diff --git a/pkg/cloud/cloud_test.go b/pkg/cloud/cloud_test.go
index 5d3c0611..f7acef7b 100644
--- a/pkg/cloud/cloud_test.go
+++ b/pkg/cloud/cloud_test.go
@@ -2059,25 +2059,6 @@ func TestCreateSnapshot(t *testing.T) {
 			},
 			expErr: nil,
 		},
-		{
-			name:         "success: outpost",
-			snapshotName: "snap-test-name",
-			snapshotOptions: &SnapshotOptions{
-				Tags: map[string]string{
-					SnapshotNameTagKey: "snap-test-name",
-					AwsEbsDriverTagKey: "true",
-					"extra-tag-key":    "extra-tag-value",
-				},
-				OutpostArn: "arn:aws:outposts:us-east-1:222222222222:outpost/aa-aaaaaaaaaaaaaaaaa",
-			},
-			expSnapshot: &Snapshot{
-				SnapshotID:     "snap-test-name",
-				SourceVolumeID: "snap-test-volume",
-				Size:           10,
-				ReadyToUse:     true,
-			},
-			expErr: nil,
-		},
 	}
 
 	for _, tc := range testCases {
diff --git a/pkg/driver/controller_test.go b/pkg/driver/controller_test.go
index 63fc630c..819c17d5 100644
--- a/pkg/driver/controller_test.go
+++ b/pkg/driver/controller_test.go
@@ -2271,49 +2271,6 @@ func TestCreateSnapshot(t *testing.T) {
 				}
 			},
 		},
-		{
-			name: "success outpost",
-			testFunc: func(t *testing.T) {
-				req := &csi.CreateSnapshotRequest{
-					Name: "test-snapshot",
-					Parameters: map[string]string{
-						"outpostArn": "arn:aws:outposts:us-east-1:222222222222:outpost/aa-aaaaaaaaaaaaaaaaa",
-					},
-					SourceVolumeId: "vol-test",
-				}
-				expSnapshot := &csi.Snapshot{
-					ReadyToUse: true,
-				}
-
-				ctx := context.Background()
-				mockSnapshot := &cloud.Snapshot{
-					SnapshotID:     fmt.Sprintf("snapshot-%d", rand.New(rand.NewSource(time.Now().UnixNano())).Uint64()),
-					SourceVolumeID: req.GetSourceVolumeId(),
-					Size:           1,
-					CreationTime:   time.Now(),
-				}
-				mockCtl := gomock.NewController(t)
-				defer mockCtl.Finish()
-
-				mockCloud := cloud.NewMockCloud(mockCtl)
-				mockCloud.EXPECT().CreateSnapshot(gomock.Eq(ctx), gomock.Eq(req.GetSourceVolumeId()), gomock.Any()).Return(mockSnapshot, nil)
-				mockCloud.EXPECT().GetSnapshotByName(gomock.Eq(ctx), gomock.Eq(req.GetName())).Return(nil, cloud.ErrNotFound)
-
-				awsDriver := ControllerService{
-					cloud:    mockCloud,
-					inFlight: internal.NewInFlight(),
-					options:  &Options{},
-				}
-				resp, err := awsDriver.CreateSnapshot(context.Background(), req)
-				if err != nil {
-					t.Fatalf("Unexpected error: %v", err)
-				}
-
-				if snap := resp.GetSnapshot(); snap == nil {
-					t.Fatalf("Expected snapshot %v, got nil", expSnapshot)
-				}
-			},
-		},
 		{
 			name: "success with cluster-id",
 			testFunc: func(t *testing.T) {
@@ -2461,35 +2418,6 @@ func TestCreateSnapshot(t *testing.T) {
 				}
 			},
 		},
-		{
-			name: "fail outpost arn not valid",
-			testFunc: func(t *testing.T) {
-				req := &csi.CreateSnapshotRequest{
-					Name: "test-snapshot",
-					Parameters: map[string]string{
-						"outpostArn": "notAnArn",
-					},
-					SourceVolumeId: "vol-test",
-				}
-
-				ctx := context.Background()
-
-				mockCtl := gomock.NewController(t)
-				defer mockCtl.Finish()
-
-				mockCloud := cloud.NewMockCloud(mockCtl)
-				mockCloud.EXPECT().GetSnapshotByName(gomock.Eq(ctx), gomock.Eq(req.GetName())).Return(nil, cloud.ErrNotFound)
-
-				awsDriver := ControllerService{
-					cloud:    mockCloud,
-					inFlight: internal.NewInFlight(),
-					options:  &Options{},
-				}
-				_, err := awsDriver.CreateSnapshot(context.Background(), req)
-				checkExpectedErrorCode(t, err, codes.InvalidArgument)
-
-			},
-		},
 		{
 			name: "fail same name different volume ID",
 			testFunc: func(t *testing.T) {
diff --git a/pkg/driver/node_test.go b/pkg/driver/node_test.go
index 92e44134..a4da70b3 100644
--- a/pkg/driver/node_test.go
+++ b/pkg/driver/node_test.go
@@ -84,6 +84,7 @@ func TestNodeStageVolume(t *testing.T) {
 		req          *csi.NodeStageVolumeRequest
 		mounterMock  func(ctrl *gomock.Controller) *mounter.MockMounter
 		metadataMock func(ctrl *gomock.Controller) *metadata.MockMetadataService
+		options      *Options
 		expectedErr  error
 		inflight     bool
 	}{
@@ -966,6 +967,45 @@ func TestNodeStageVolume(t *testing.T) {
 			},
 			expectedErr: nil,
 		},
+		{
+			name: "format_options_xfs_legacy",
+			req: &csi.NodeStageVolumeRequest{
+				VolumeId:          "vol-test",
+				StagingTargetPath: "/staging/path",
+				VolumeCapability: &csi.VolumeCapability{
+					AccessType: &csi.VolumeCapability_Mount{
+						Mount: &csi.VolumeCapability_MountVolume{
+							FsType: "xfs",
+						},
+					},
+					AccessMode: &csi.VolumeCapability_AccessMode{
+						Mode: csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER,
+					},
+				},
+				VolumeContext: map[string]string{
+					InodeSizeKey: "512",
+				},
+				PublishContext: map[string]string{
+					DevicePathKey: "/dev/xvdba",
+				},
+			},
+			mounterMock: func(ctrl *gomock.Controller) *mounter.MockMounter {
+				m := mounter.NewMockMounter(ctrl)
+				m.EXPECT().FindDevicePath(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return("/dev/xvdba", nil)
+				m.EXPECT().PathExists(gomock.Eq("/staging/path")).Return(true, nil)
+				m.EXPECT().GetDeviceNameFromMount(gomock.Eq("/staging/path")).Return("", 1, nil)
+				m.EXPECT().FormatAndMountSensitiveWithFormatOptions(gomock.Eq("/dev/xvdba"), gomock.Eq("/staging/path"), gomock.Eq("xfs"), gomock.Any(), gomock.Any(), gomock.Eq([]string{"-i", "size=512", "-m", "bigtime=0,inobtcount=0,reflink=0"})).Return(nil)
+				m.EXPECT().NeedResize(gomock.Eq("/dev/xvdba"), gomock.Eq("/staging/path")).Return(false, nil)
+				return m
+			},
+			metadataMock: func(ctrl *gomock.Controller) *metadata.MockMetadataService {
+				m := metadata.NewMockMetadataService(ctrl)
+				m.EXPECT().GetRegion().Return("us-west-2")
+				return m
+			},
+			options:     &Options{LegacyXFSProgs: true},
+			expectedErr: nil,
+		},
 	}
 
 	for _, tc := range testCases {
@@ -983,9 +1023,15 @@ func TestNodeStageVolume(t *testing.T) {
 				metadata = tc.metadataMock(ctrl)
 			}
 
+			options := tc.options
+			if options == nil {
+				options = &Options{} // Initialize struct to avoid nil pointer dereference
+			}
+
 			driver := &NodeService{
 				metadata: metadata,
 				mounter:  mounter,
+				options:  options,
 				inFlight: internal.NewInFlight(),
 			}
 
diff --git a/pkg/driver/options_test.go b/pkg/driver/options_test.go
index 21b4fe4a..38944b7c 100644
--- a/pkg/driver/options_test.go
+++ b/pkg/driver/options_test.go
@@ -70,6 +70,9 @@ func TestAddFlags(t *testing.T) {
 	if err := f.Set("reserved-volume-attachments", "5"); err != nil {
 		t.Errorf("error setting reserved-volume-attachments: %v", err)
 	}
+	if err := f.Set("legacy-xfs", "true"); err != nil {
+		t.Errorf("error setting legacy-xfs: %v", err)
+	}
 
 	if o.Endpoint != "custom-endpoint" {
 		t.Errorf("unexpected Endpoint: got %s, want custom-endpoint", o.Endpoint)
@@ -107,6 +110,9 @@ func TestAddFlags(t *testing.T) {
 	if o.ReservedVolumeAttachments != 5 {
 		t.Errorf("unexpected ReservedVolumeAttachments: got %d, want 5", o.ReservedVolumeAttachments)
 	}
+	if !o.LegacyXFSProgs {
+		t.Errorf("unexpected LegacyXFSProgs: got false, want true")
+	}
 }
 
 func TestValidateAttachmentLimits(t *testing.T) {