Disaster Recovery on Kubernetes
By: Yitaek Hwang
Tags: Backup, Software Engineering, Disaster Recovery, Programming, Kubernetes

Using VMWare’s Velero to backup and restore, perform disaster recovery, as well as migrate Kubernetes resources.

Photo by Markus Spiske on Unsplash

Although Kubernetes (and especially managed Kubernetes services such as GKE, EKS, and AKS) provide out-of-the-box reliability and resiliency with self-healing and horizontal scaling capabilities, production systems still require disaster recovery solutions to protect against human error (e.g. accidentally deleting a namespace or secret) and infrastructure failures outside of Kubernetes (e.g. persistent volumes). While more companies are embracing multi-region solutions, it is a complicated and potentially expensive option if all you need is a simple backup and restore option. In this post, we’ll look at using Velero to backup and restore Kubernetes resources as well as demonstrating its use as a disaster recovery or migration tool.

Are Backups Still Needed?

A key point that is often lost when running services in high availability (HA) mode is that HA (and thus replication) is not the same as having backups. HA protects against zonal failures, but it will not protect against data corruption or accidental removals. It is very easy to mix up the context or namespaces and accidentally delete or update the wrong Kubernetes resources. This may be a Custom Resource Definition (CRD), a secret, or a namespace. Some may argue that with IaaS tools like Terraform and external solutions to manage some of these Kubernetes resources (e.g. Vault for secrets, ChartMuseum for Helm charts), backups become unnecessary. Still, if you are running a StatefulSet in your cluster (e.g. ELK stack for logging or self-hosting Postgres to install plugins not support on RDS or Cloud SQL), backups are needed to recover from persistent volume failures.

Velero

Velero (formerly known as Ark) is an open-source tool from Heptio (acquired by VMWare) to back up and restore Kubernetes cluster resources and persistent volumes. Velero runs inside the Kubernetes cluster and integrates with various storage providers (e.g. AWS S3, GCP Storage, Minio) as well as restic to take snapshots either on-demand or on a schedule.

Installation

Velero can be installed via Helm or via the CLI tool. In general, it seems like the CLI gets the latest updates and the Helm chart lags behind slightly with compatible Docker images. However, with each release, the Velero team does a great job updating the documentation to patch CRDs and the new Velero container image, so upgrading the Helm chart to the latest isn’t a huge concern.

Configuration

Once you have the server installed, you can configure Velero via CLI or by modifying values.yaml for the Helm chart. The key configuration steps are installing the plugins for the storage provider and defining the Storage Location as well as the Volume Snapshot Location:

configuration:

provider: aws

backupStorageLocation:

name: aws

bucket: <aws-bucket-name>

prefix: velero

config:

kmsKeyId: <my-kms-key>

region: <aws-region>

volumeSnapshotLocation:

name: aws

config:

region: ${region}

logLevel: debug

(Note: There is an issue with CRDs with the latest Helm chart causing backup storage and volume snapshot location to not set the configured values as default. If you decide to name the storage and snapshot location, add --storage-location <name> --volume-snapshot-location name in the following Velero commands)

Creating a Backup

To create a backup, simply apply the backup command to a namespace or select by labels:

$ velero backup create nginx-backup --include-namespaces nginx-example $ velero backup create postgres-backup --selector release=postgres

When the backup command is issued, Velero runs through the following steps:

Call the Kubernetes API to create the Backup CRD Velero BackupController validates the request Once the request is validated, it queries the Kubernetes resources and takes snapshots of disks to back up and creates a tarball Finally, it initiates the upload of the backup objects to the configured storage service

Image Credit: OpenShift

Restoring Data

To list the available backups, first run:

$ velero backup get

Now you can restore from backup by issuing:

$ velero restore create RESTORE_NAME \

--from-backup BACKUP_NAME

Velero also supports restoring objects into a different namespace if you do not wish to override the existing resources (append --namespace-mappings old-ns-1:new-ns-1 to the above command). This is useful if you are experiencing outages and want to diagnose the problem for later while immediately restoring the service.

Velero can change the storage class of persistent volumes during restores. This may be a good way to migrate workloads from HDD to SSD storage or to a smaller disk if you over-provisioning the persistent volume (see the documentation for the configuration).

Finally, you can also selectively restore sub-components of the backup. Inspect the backup tarball by running:

$ velero backup download <backup-name>

From the tarball, you can choose a manifest for a specific resource and individually issue kubectl apply -f . This is useful if you took a snapshot of the entire namespace rather than filtering by labels.

Scheduled Backups

Instead of only creating backups on-demand, you can also configure scheduled backups for critical components:

Via CLI:

$ velero schedule create mysql --schedule="0 2* * *" --include-namespaces mysql

Via Helm values:

schedules:

mysql:

schedule: 0 2 * * *

template:

labelSelector:

matchLabels:

app: mysql

snapshotVolumes: true

ttl: 720h

Notice the ttl configuration, which specifies the time to expire scheduled backups. If you are using a Cloud Storage provider, you can leverage lifecycle policies or control that via Velero as shown above to reduce storage costs.

Other Uses

Besides simply taking backups, Velero can be used as a disaster recovery solution by combining schedules and read-only backup storage locations. Configure Velero to create a daily schedule:

$ velero schedule create <SCHEDULE NAME> --schedule "0 7 * * *"

If you need to recreate resources due to human error or infrastructure outage, change the backup location to be read-only to prevent new backup objects from being created:

$ kubectl patch backupstoragelocation <STORAGE LOCATION NAME> \

--namespace velero \

--type merge \

--patch '{"spec":{"accessMode":"ReadOnly"}}'

Restore from backup in another location:

$ velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

And finally, revert backup to be writable again:

$ kubectl patch backupstoragelocation <STORAGE LOCATION NAME> \

--namespace velero \

--type merge \

--patch '{"spec":{"accessMode":"ReadWrite"}}'

This process works to migrate clusters to a different region (if the provider supports it) or to create the last working version prior to a Kubernetes upgrade. Finally, even if Velero does not natively support migration of persistent volumes across clouds, you can configure restic to make backups at filesystem level and migrate data for a hybrid-cloud back up solution.

Other Solutions

While Velero is very easy to use and configure, it may not fit your specific use case (e.g. cross-cloud backup). As mentioned above, Velero integrates with other solutions such as restic or OpenEBS, but if you are looking for alternatives, the following list provides both open-source and enterprise options: