Question: "How do I restore, resize, or create an EBS persistent storage snapshot in Amazon EKS for disaster recovery or when the EBS modification rate is exceeded?"

I want to use an Amazon Elastic Block Store (Amazon EBS) persistent storage snapshot in Amazon Elastic Kubernetes Service (Amazon EKS) for disaster recovery. How do I create, resize, or restore such a snapshot? Or, I exceeded my Amazon EBS modification rate. But I still need to resize, restore, or create a snapshot of my Amazon EBS persistent storage in Amazon EKS.

Short descriptionYou're modifying your Amazon EBS persistent storage in Amazon EKS, and you receive the following error:errorCode: Client.VolumeModificationRateExceedederrorMessage: You've reached the maximum modification rate per volume limit. Wait at least 6 hours between modifications per EBS volumeAfter you modify a volume, you must wait at least six hours before you can continue to modify the volume. Make sure that the volume is in the in-use or available state before you modify it again.Your organization might have a Disaster Recovery (DR) objective with a Recovery Time Objective (RTO) that's less than six hours. For RTOs that are less than six hours, create a snapshot and restore your volume using the Amazon EBS Container Storage Interface (CSI) driver.

ResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.Use the Amazon EBS CSI driver and external snapshotter to do the following:Create an Amazon EBS snapshot of the PersistentVolumeClaim.Restore the PersistentVolumeClaim.Bind the PersistentVolumeClaim to the workload.Pre-requisites:An existing Amazon EKS cluster with worker nodes. If you don't have one, then create your Amazon EKS cluster.The latest versions of AWS CLI, eksctl, and kubectl.An Amazon EBS CSI driver AWS Identity and Access Management (IAM) role for service accounts.Install the Amazon EBS CSI driver with the external snapshotter1.    Check if you have an existing IAM OpenID Connect (OIDC) provider for your cluster:% cluster_name=ebs% oidc_id=$(aws eks describe-cluster --name cluster_name --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)% aws iam list-open-id-connect-providers | grep $oidc_idNote: Replace cluster_name with your cluster's name.Example output:"Arn": "arn:aws:iam::XXXXXXXXXX:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/B7E2BC2980D17C9A5A3889998CB22B23"Note: If you don't have an IAM OIDC provider, then create one for your cluster.2.    Install the external snapshotter.Note: You must install the external snapshotter before you install the Amazon EBS CSI add-on. Also, you must install the external snapshotter components in the following order:CustomResourceDefinition (CRD) for volumesnapshotclasses, volumesnapshots, and volumesnapshotcontentsmkdir crdcd crdwget https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/kustomization.yamlwget https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yamlwget https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yamlwget https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yamlkubectl apply -k ../crdRBAC, such as ClusterRole, and ClusterRoleBindingkubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yamlController deploymentkubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml3.    Create your Amazon EBS CSI plugin IAM role using eksctl:eksctl create iamserviceaccount \  --name ebs-csi-controller-sa \  --namespace kube-system \  --cluster cluster_name \  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \  --approve \  --role-only \  --role-name AmazonEKS_EBS_CSI_DriverRole4.    Add the Amazon EBS CSI add-on using eksctl:eksctl create addon --name aws-ebs-csi-driver --cluster cluster_name --service-account-role-arn arn:aws:iam::account_id:role/AmazonEKS_EBS_CSI_DriverRole --forceNote: Replace account_id with your AWS account ID.5.    Confirm that the Amazon EBS CSI driver and external snapshotter pods are running:% kubectl get pods -A | egrep "csi|snapshot"Create a StatefulSet with Amazon EBS persistent storage1.    Download the manifests from the GitHub website.2.    Create the StorageClass and VolumeSnapshotClass:% kubectl apply -f manifests/classes/Example output:volumesnapshotclass.snapshot.storage.k8s.io/csi-aws-vsc createdstorageclass.storage.k8s.io/ebs-sc created3.    Deploy the StatefulSet on your cluster along with the PersistentVolumeClaim:% kubectl apply -f manifests/app/Example output:service/cassandra createdStatefulSet.apps/cassandra created4.    Verify that the pods are in Running status:% kubectl get podsExample output:NAME         READY  STATUS   RESTARTS  AGEcassandra-0  1/1    Running  0         33mcassandra-1  1/1    Running  0         32mcassandra-2  1/1    Running  0         30m5.    Verify that the PersistenVolumeClaim is bound to your PersisentVolume:% kubectl get pvcExample output:NAME                                              STATUS  VOLUME                                    CAPACITY ACCESS MODES STORAGECLASS  AGEpersistentvolumeclaim/cassandra-data-cassandra-0  Bound   pvc-b3ab4971-37dd-48d8-9f59-8c64bb65b2c8  2Gi      RWO          ebs-sc        28mpersistentvolumeclaim/cassandra-data-cassandra-1  Bound   pvc-6d68170e-2e51-40f4-be52-430790684e51  2Gi      RWO          ebs-sc        28mpersistentvolumeclaim/cassandra-data-cassandra-2  Bound   pvc-d0403adb-1a6f-44b0-9e7f-a367ad7b7353  2Gi      RWO          ebs-sc        26m...Note: Note the names of each PersistentVolumeClaim to compare to the PersistentVolumeClaim names in the snapshot manifest.6.    To test the StatefulSet, write content to the PersistentVolumeClaim:for i in {0..2}; do kubectl exec "cassandra-$i" -- sh -c 'echo "$(hostname)" > /cassandra_data/data/file.txt'; doneCreate a snapshotThe persistentVolumeClaimName in the snapshot manifest must match the name of the PersistentVolumeClaim that you created for each pod in the StatefulSet. For example:apiVersion: snapshot.storage.k8s.io/v1kind: VolumeSnapshotmetadata:  name: cassandra-data-snapshot-0spec:  volumeSnapshotClassName: csi-aws-vsc  source:    persistentVolumeClaimName: cassandra-data-cassandra-01.    Create a snapshot from each PersistenVolumeClaim:% kubectl apply -f manifests/snapshot/Example output:volumesnapshot.snapshot.storage.k8s.io/cassandra-data-snapshot-0 createdvolumesnapshot.snapshot.storage.k8s.io/cassandra-data-snapshot-1 createdvolumesnapshot.snapshot.storage.k8s.io/cassandra-data-snapshot-2 created2.    After the state is completed, verify that the snapshots are available on the Amazon Elastic Compute Cloud (Amazon EC2) console:aws ec2 describe-snapshots --filters "Name=tag-key,Values=*ebs*" --query 'Snapshots[*].{VOL_ID:VolumeId,SnapshotID:SnapshotId,State:State,Size:VolumeSize,Name:[Tags[?Key==`Name`].Value] [0][0]}' --output table---------------------------------------------------------------------------------------------------------------------------------------|                                                          DescribeSnapshots                                                          |+------------------------------------------------------------+-------+-------------------------+------------+-------------------------+|                            Name                            | Size  |       SnapshotID        |   State    |         VOL_ID          |+------------------------------------------------------------+-------+-------------------------+------------+-------------------------+|  ebs-dynamic-snapshot-c6c9cb3c-2dab-4833-9124-40a0abde170d |  2    |  snap-057c5e2de3957d855 |  pending   |  vol-01edf53ee26a615f5  ||  ebs-dynamic-snapshot-1c1ad0c5-a93a-468f-ab54-576db5d630d4 |  2    |  snap-02bf49a3b78ebf194 |  completed |  vol-0289efe54525dca4a  ||  ebs-dynamic-snapshot-760c48e7-33ff-4b83-a6fb-6ef13b8d31b7 |  2    |  snap-0101c3d2efa40af19 |  completed |  vol-0fe68c9ac2f6375a4  |+------------------------------------------------------------+-------+-------------------------+------------+-------------------------+Restore the snapshotYou can restore a PersistentVolumeClaim from a snapshot that's created from an existing PersistentVolumeClaim using the same name of the PersistentVolumeClaim. When you recreate the StatefulSet, the PersistentVolumeClaim dynamically provisions a PersistentVolume and is automatically bound to the StatefulSet pods. The StatefulSet PersistenVolumeClaim name format is: PVC\_TEMPLATE\_NAME-STATEFULSET\_NAME-REPLICA\_INDEX.To restore a snapshot follow these steps:1.    Delete the existing StatefulSet workload:kubectl delete -f manifests/app/Cassandra_statefulset.yamlNote: Deleting the workload also deletes the StatefulSet pods. The snapshot that you created acts as a backup.Example output:statefulset.apps "cassandra" deleted2.    Forcefully delete the PersistentVolumeClaim:for i in {0..2}do  kubectl delete pvc cassandra-data-cassandra-$i --forcedoneNote: Deleting the PersistentVolumeClaim also deletes the PersistentVolume.3.    Restore the PersistentVolumeClaim from the snapshot using the same name of the PersistentVolumeClaim that you created:kubectl apply -f manifests/snapshot-restore/Example output:persistentvolumeclaim/cassandra-data-cassandra-0 createdpersistentvolumeclaim/cassandra-data-cassandra-1 createdpersistentvolumeclaim/cassandra-data-cassandra-2 created4.    Verify that each PersistentVolumeClaim is in Pending status:kubectl get pvcExample output:NAME                         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGEcassandra-data-cassandra-0   Pending                                      ebs-sc         24scassandra-data-cassandra-1   Pending                                      ebs-sc         23scassandra-data-cassandra-2   Pending                                      ebs-sc         22s5.    Recreate the StatefulSet with the original manifest:kubectl apply -f manifests/app-restore/Note: To resize the storage, define the StatefulSet with a new storage size.Example output:StatefulSet.apps/cassandra created6.    Check the content of the Amazon EBS storage to confirm that the snapshot and restore work:for i in {0..2}; do kubectl exec "cassandra-$i" -- sh -c 'cat /cassandra_data/data/file.txt'; donecassandra-0cassandra-1cassandra-2Resize the PersistentVolumeClaimYou can modify the .spec.resources.requests.storage of the PersistentVolumeClaim to automatically reflect the size that you defined in the StatefulSet manifest:for i in {0..2}do  echo "Resizing cassandra-$i"  kubectl patch pvc cassandra-data-cassandra-$i -p '{ "spec": { "resources": { "requests": { "storage": "4Gi" }}}}'doneNote: 4Gi is an example storage size. Define a storage size that's suitable for your use case.Confirm that the new storage size is reflected on the Amazon EC2 console and in the pods:% aws ec2 describe-volumes --filters "Name=tag-key,Values=*pvc*" --query 'Volumes[*].{ID:VolumeId,Size:Size,Name:[Tags[?Key==`Name`].Value] [0][0]}' --output table-------------------------------------------------------------------------------------------|                                     DescribeVolumes                                     |+------------------------+--------------------------------------------------------+-------+|           ID           |                         Name                           | Size  |+------------------------+--------------------------------------------------------+-------+|  vol-01266a5f1f8453e06 |  ebs-dynamic-pvc-359a87f6-b584-49fa-8dd9-e724596b2b43  |  4    ||  vol-01b63a941450342d9 |  ebs-dynamic-pvc-bcc6f2cd-383d-429d-b75f-846e341d6ab2  |  4    ||  vol-041486ec92d4640af |  ebs-dynamic-pvc-fa99a595-84b7-49ad-b9c2-1be296e7f1ce  |  4    |+------------------------+--------------------------------------------------------+-------+% for i in {0..2}do  echo "Inspecting cassandra-$i"  kubectl exec -it cassandra-$i -- lsblk  kubectl exec -it cassandra-$i -- df -hdone...Run the following Kubectl commands to clean up your StatefulSetTo delete the resources that you created for your StatefulSet, run the following kubectl commands:app-restorekubectl delete -f manifests/app-restoresnapshot-restorekubectl delete -f manifests/snapshot-restoresnapshotkubectl delete -f manifests/snapshotclasseskubectl delete -f manifests/classesCassandrakubectl delete -f manifests/app/Cassandra_service.yaml

Related informationModifyVolume