Project: Velero (Runtime / Cloud Native Storage)
File: cloud-provider-specifics.md

title: "Cloud Provider Specifics" layout: docs NOTE: Documentation may change between releases. See the for links to previous versions of this repository and its docs. To ensure that you are working off a specific release, `git checkout <VERSIONTAG>` where `<VERSIONTAG>` is the appropriate tag for the Ark version you wish to use (e.g. "v0.3.3"). You should `git checkout main` only if you're planning on . While the uses a local storage service to quickly set up Heptio Ark as a demonstration, this document details additional configurations that are required when integrating with the cloud providers below: * * * * To integrate Heptio Ark with AWS, you should follow the instructions below to create an Ark-specific . If you do not have the AWS CLI locally installed, follow the to set it up. Create an IAM user: ```bash aws iam create-user --user-name heptio-ark ``` Attach a policy to give `heptio-ark` the necessary permissions: ```bash aws iam attach-user-policy \ --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess \ --user-name heptio-ark aws iam attach-user-policy \ --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess \ --user-name heptio-ark ``` Create an access key for the user: ```bash aws iam create-access-key --user-name heptio-ark ``` The result should look like: ```json { "AccessKey": { "UserName": "heptio-ark", "Status": "Active", "CreateDate": "2017-07-31T22:24:41.576Z", "SecretAccessKey": <AWSSECRETACCESS_KEY>, "AccessKeyId": <AWSACCESSKEY_ID> } } ``` Using the output from the previous command, create an Ark-specific credentials file (`credentials-ark`) in your local directory that looks like the following: ``` [default] awsaccesskeyid=<AWSACCESSKEYID> awssecretaccesskey=<AWSSECRETACCESSKEY> ``` In the Ark root directory, run the following to first set up namespaces, RBAC, and other scaffolding: ```bash kubectl apply -f examples/common/00-prereqs.yaml ``` Create a Secret, running this command in the local directory of the credentials file you just created: ```bash kubectl create secret generic cloud-credentials \ --namespace heptio-ark \ --from-file cloud=credentials-ark ``` Now that you have your IAM user credentials stored in a Secret, you need to replace some placeholder values in the template files. Specifically, you need to change the following: In file `examples/aws/00-ark-config.yaml`: Replace `<YOURBUCKET>` and `<YOURREGION>`. See the for details. In file `examples/common/10-deployment.yaml`: Make sure that `spec.template.spec.containers[].env.name` is "AWSSHAREDCREDENTIALS_FILE". (Optional) If you are running the Nginx example, in file `examples/nginx-app/with-pv.yaml`: Replace `<YOURSTORAGECLASS_NAME>` with `gp2`. This is AWS's default `StorageClass` name. To integrate Heptio Ark with GCP, you should follow the instructions below to create an Ark-specific . If you do not have the gcloud CLI locally installed, follow the to set it up. View your current config settings: ```bash gcloud config list ``` Store the `project` value from the results in the environment variable `$PROJECT_ID`. Create a service account: ```bash gcloud iam service-accounts create heptio-ark \ --display-name "Heptio Ark service account" ``` Then list all accounts and find the `heptio-ark` account you just created: ```bash gcloud iam service-accounts list ``` Set the `$SERVICEACCOUNTEMAIL` variable to match its `email` value. Attach policies to give `heptio-ark` the necessary permissions to function (replacing placeholders appropriately): ```bash gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$SERVICEACCOUNTEMAIL \ --role roles/compute.storageAdmin gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$SERVICEACCOUNTEMAIL \ --role

``` Create a service account key, specifying an output file (`credentials-ark`) in your local directory: ```bash gcloud iam service-accounts keys create credentials-ark \ --iam-account $SERVICEACCOUNTEMAIL ``` In the Ark root directory, run the following to first set up namespaces, RBAC, and other scaffolding: ```bash kubectl apply -f examples/common/00-prereqs.yaml ``` Create a Secret, running this command in the local directory of the credentials file you just created: ```bash kubectl create secret generic cloud-credentials \ --namespace heptio-ark \ --from-file cloud=credentials-ark ``` Now that you have your Google Cloud credentials stored in a Secret, you need to replace some placeholder values in the template files. Specifically, you need to change the following: In file `examples/gcp/00-ark-config.yaml`: Replace `<YOURBUCKET>` and `<YOURPROJECT>`. See the for details. In file `examples/common/10-deployment.yaml`: Change `spec.template.spec.containers[].env.name` to "GOOGLEAPPLICATIONCREDENTIALS". (Optional) If you are running the Nginx example, in file `examples/nginx-app/with-pv.yaml`: Replace `<YOURSTORAGECLASS_NAME>` with `standard`. This is GCP's default `StorageClass` name. Ensure that the VMs for your agent pool allow Managed Disks. If I/O performance is critical, consider using Premium Managed Disks, as these are SSD backed. To integrate Heptio Ark with Azure, you should follow the instructions below to create an Ark-specific . If you do not have the `az` Azure CLI 2.0 locally installed, follow the to set it up. Once done, run: ```bash az login ``` There are seven environment variables that need to be set for Heptio Ark to work properly. The following steps detail how to acquire these, in the process of setting up the necessary RBAC. Obtain your Azure Account Subscription ID and Tenant ID: ```bash AZURESUBSCRIPTIONID=`az account list --query '[?isDefault].id' -o tsv` AZURETENANTID=`az account list --query '[?isDefault].tenantId' -o tsv` ``` Set the name of the Resource Group that contains your Kubernetes cluster. ```bash AZURERESOURCEGROUP=Kubernetes ``` If you are unsure of the Resource Group name, run the following command to get a list that you can select from. Then set the `AZURERESOURCEGROUP` environment variable to the appropriate value. ```bash az group list --query '[].{ ResourceGroup: name, Location:location }' ``` Get your cluster's Resource Group name from the `ResourceGroup` value in the response, and use it to set `$AZURERESOURCEGROUP`. (Also note the `Location` value in the response -- this is later used in the Azure-specific portion of the Ark Config). Create a service principal with `Contributor` role. This will have subscription-wide access, so protect this credential. You can specify a password or let the `az ad sp create-for-rbac` command create one for you. ```bash AZURECLIENTSECRET=supersecretandhighentropypasswordreplacemewithyourown az ad sp create-for-rbac --name "heptio-ark" --role "Contributor" --password $AZURECLIENTSECRET AZURECLIENTSECRET=`az ad sp create-for-rbac --name "heptio-ark" --role "Contributor" --query 'password' -o tsv` AZURECLIENTID=`az ad sp list --display-name "heptio-ark" --query '[0].appId' -o tsv` ``` Create the storage account and blob container for Ark to store the backups in. The storage account can be created in the same Resource Group as your Kubernetes cluster or separated into its own Resource Group. The example below shows the storage account created in a separate `Ark_Backups` Resource Group. The storage account needs to be created with a globally unique id since this is used for dns. The random function ensures you don't have to come up with a unique name. The storage account is created with encryption at rest capabilities (Microsoft managed keys) and is configured to only allow access via

```bash AZUREBACKUPRESOURCEGROUP=ArkBackups az group create -n $AZUREBACKUPRESOURCE_GROUP --location WestUS AZURESTORAGEACCOUNT_ID="ark`cat /proc/sys/kernel/random/uuid | cut -d '-' -f5`" az storage account create \ --name $AZURESTORAGEACCOUNT_ID \ --resource-group $AZUREBACKUPRESOURCE_GROUP \ --sku Standard_GRS \ --encryption-services blob \ --https-only true \ --kind BlobStorage \ --access-tier Hot az storage container create -n ark --public-access off --account-name $AZURESTORAGEACCOUNT_ID AZURESTORAGEKEY=`az storage account keys list \ --account-name $AZURESTORAGEACCOUNT_ID \ --resource-group $AZUREBACKUPRESOURCE_GROUP \ --query [0].value \ -o tsv` ``` In the Ark root directory, run the following to first set up namespaces, RBAC, and other scaffolding: ```bash kubectl apply -f examples/common/00-prereqs.yaml ``` Now you need to create a Secret that contains all the seven environment variables you just set. The command looks like the following: ```bash kubectl create secret generic cloud-credentials \ --namespace heptio-ark \ --from-literal AZURESUBSCRIPTIONID=${AZURESUBSCRIPTIONID} \ --from-literal AZURETENANTID=${AZURETENANTID} \ --from-literal AZURERESOURCEGROUP=${AZURERESOURCEGROUP} \ --from-literal AZURECLIENTID=${AZURECLIENTID} \ --from-literal AZURECLIENTSECRET=${AZURECLIENTSECRET} \ --from-literal AZURESTORAGEACCOUNTID=${AZURESTORAGEACCOUNTID} \ --from-literal AZURESTORAGEKEY=${AZURESTORAGEKEY} ``` Now that you have your Azure credentials stored in a Secret, you need to replace some placeholder values in the template files. Specifically, you need to change the following: In file `examples/azure/10-ark-config.yaml`: Replace `<YOURBUCKET>`, `<YOURLOCATION>`, and `<YOUR_TIMEOUT>`. See the for details. Here is an example of a completed file. ```yaml apiVersion: ark.heptio.com/v1 kind: Config metadata: namespace: heptio-ark name: default persistentVolumeProvider: name: azure config: location: "West US" apiTimeout: 15m backupStorageProvider: name: azure bucket: ark backupSyncPeriod: 30m gcSyncPeriod: 30m scheduleSyncPeriod: 1m restoreOnlyMode: false ``` You can get a complete list of Azure locations with the following command: ```bash az account list-locations --query "sort([].displayName)" -o tsv ``` Make sure that you have run `kubectl apply -f examples/common/00-prereqs.yaml` first (this command is incorporated in the previous setup instructions because it creates the necessary namespaces). AWS and GCP* Start the Ark server itself, using the Config from the appropriate cloud-provider-specific directory: ```bash kubectl apply -f examples/common/10-deployment.yaml kubectl apply -f examples/<CLOUD-PROVIDER>/ ``` Azure* Because Azure loads its credentials differently (from environment variables rather than a file), you need to instead run: ```bash kubectl apply -f examples/azure/ ``` Start the sample nginx app: ```bash kubectl apply -f examples/nginx-app/base.yaml ``` Now create a backup: ```bash ark backup create nginx-backup --selector app=nginx ``` Simulate a disaster: ```bash kubectl delete namespaces nginx-example ``` Now restore your lost resources: ```bash ark restore create nginx-backup ``` NOTE: For Azure, your Kubernetes cluster needs to be version 1.7.2+ in order to support PV snapshotting of its managed disks. Start the sample nginx app: ```bash kubectl apply -f examples/nginx-app/with-pv.yaml ``` Because Kubernetes does not automatically transfer labels from PVCs to dynamically generated PVs, you need to do so manually: ```bash nginxpvname=$(kubectl get pv -o jsonpath='{.items[?(@.spec.claimRef.name=="nginx-logs")].metadata.name}') kubectl label pv $nginxpvname app=nginx ``` Now create a backup with PV snapshotting: ```bash ark backup create nginx-backup --selector app=nginx ``` Simulate a disaster: ```bash kubectl delete namespaces nginx-example kubectl delete pv $nginxpvname ``` Because the default for dynamically-provisioned PVs is "Delete", the above commands should trigger your cloud provider to delete the disk backing the PV. The deletion process is asynchronous so this may take some time. Before continuing to the next step, check your cloud provider (via dashboard or CLI) to confirm that the disk no longer exists. Now restore your lost resources: ```bash ark restore create nginx-backup ```