veleroでKubernetesのPVが接続されているDeploymentをバックアップする

## velero
- Kubernetesクラスタごとバックアップ可能
- PVをバックアップできる
    - AWS BackupなどでEBSを取得する方法をとった場合、再度IDを指定してデプロイする必要があり、手順が複雑になる
- S3にバックアップされる

## veleroコマンドインストール

- 公式サイト
    - https://www.eksworkshop.com/intermediate/280_backup-and-restore/install_velero/
- Ubuntu20.04でインストール

```
wget https://github.com/vmware-tanzu/velero/releases/download/v1.3.2/velero-v1.3.2-linux-amd64.tar.gz
tar -xvf velero-v1.3.2-linux-amd64.tar.gz -C /tmp
sudo mv /tmp/velero-v1.3.2-linux-amd64/velero /usr/local/bin
velero version
```

## velero初期セットアップ
https://github.com/vmware-tanzu/velero-plugin-for-aws

### バックアップファイル格納用のS3バケットを作成

```
BUCKET=velero-bakcup-vamdemic
REGION=ap-northeast-1
aws s3api create-bucket \
     --bucket $BUCKET \
     --region $REGION \
     --create-bucket-configuration LocationConstraint=$REGION
```

### ユーザー作成

```
aws iam create-user --user-name velero
```

### ユーザーのポリシーを設定

```
cat > velero-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}"
            ]
        }
    ]
}
EOF
```

### ポリシーをアタッチ

```
aws iam put-user-policy \
  --user-name velero \
  --policy-name velero \
  --policy-document file://velero-policy.json
```

### アクセスキーを取得

```
aws iam create-access-key --user-name velero
```

### ファイルにクレデンシャルを埋め込む
- ./credentials-velero

```credencial-velero
[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
```

### veleroをインストール（Podが稼働し始める）
```
velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.1.0 \
    --bucket $BUCKET \
    --backup-location-config region=$REGION \
    --snapshot-location-config region=$REGION \
    --secret-file ./credentials-velero
```

標準出力
- 以下のように出ればOK

```
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: already exists, proceeding
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: already exists, proceeding
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: already exists, proceeding
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: already exists, proceeding
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: already exists, proceeding
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: already exists, proceeding
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: already exists, proceeding
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: already exists, proceeding
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: already exists, proceeding
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: already exists, proceeding
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: already exists, proceeding
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: already exists, proceeding
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: already exists, proceeding
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: already exists, proceeding
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: already exists, proceeding
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: already exists, proceeding
BackupStorageLocation/default: created
VolumeSnapshotLocation/default: attempting to create resource
VolumeSnapshotLocation/default: already exists, proceeding
VolumeSnapshotLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: already exists, proceeding
Deployment/velero: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
```

## 動作確認
### テスト用スペックファイルのダウンロード
```
git clone git@github.com:vmware-tanzu/velero.git
```

### PVが接続されているDepploymentのバックアップ
- `storageClassName: gp2`は自身の環境のStorageClassに合わせて変更する

```examples/nginx-app/with-pv.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: nginx-example
  labels:
    app: nginx

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-logs
  namespace: nginx-example
  labels:
    app: nginx
spec:
  Optional:
  storageClassName: gp2
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Mi

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: nginx-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        pre.hook.backup.velero.io/container: fsfreeze
        pre.hook.backup.velero.io/command: '["/sbin/fsfreeze", "--freeze", "/var/log/nginx"]'
        post.hook.backup.velero.io/container: fsfreeze
        post.hook.backup.velero.io/command: '["/sbin/fsfreeze", "--unfreeze", "/var/log/nginx"]'
    spec:
      volumes:
        - name: nginx-logs
          persistentVolumeClaim:
           claimName: nginx-logs
      containers:
      - image: nginx:1.17.6
        name: nginx
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: "/var/log/nginx"
            name: nginx-logs
            readOnly: false
      - image: ubuntu:bionic
        name: fsfreeze
        securityContext:
          privileged: true
        volumeMounts:
          - mountPath: "/var/log/nginx"
            name: nginx-logs
            readOnly: false
        command:
          - "/bin/bash"
          - "-c"
          - "sleep infinity"

  
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: my-nginx
  namespace: nginx-example
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer
```

### Apply

```
kubectl apply -f examples/nginx-app/with-pv.yaml
```

### テストファイルを配置する
#### ログイン
```
kubectl exec -it nginx-deployment-f96b7fd86-k59ct -n nginx-example /bin/sh
```

####  pvのマウントを確認
```
# df
Filesystem     1K-blocks    Used Available Use% Mounted on
overlay        104845292 6151984  98693308   6% /
tmpfs              65536       0     65536   0% /dev
tmpfs            8117392       0   8117392   0% /sys/fs/cgroup
/dev/nvme0n1p1 104845292 6151984  98693308   6% /etc/hosts
shm                65536       0     65536   0% /dev/shm
/dev/nvme4n1      999320    2568    980368   1% /var/log/nginx
tmpfs            8117392      12   8117380   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            8117392       0   8117392   0% /proc/acpi
tmpfs            8117392       0   8117392   0% /sys/firmware
```

#### テストファイル配置
```
echo aaa > /var/log/nginx/modoru.log
```

出力

```
# ls -l /var/log/nginx/
total 20
-rw-r--r-- 1 root root     0 Jan  8 08:36 access.log
-rw-r--r-- 1 root root     0 Jan  8 08:36 error.log
drwx------ 2 root root 16384 Jan  8 08:36 lost+found
-rw-r--r-- 1 root root     4 Jan  8 08:50 modoru.log
```



### バックアップ
```
velero backup create nginx-backup-with-pv --include-namespaces nginx-example
```

### バックアップ確認
```
velero backup get
```

出力

```
NAME                   STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
nginx-backup-with-pv   Completed   2021-01-08 16:48:10 +0900 JST   29d       default            <none>
```

### Deploymentを削除

```
kubectl delete -f examples/nginx-app/with-pv.yaml
```

### リストア

```
velero restore create --from-backup  nginx-backup-with-pv
```

### リストア確認
- `modoru.log`が復元されている

```
kubectl  -n nginx-example  exec -it nginx-deployment-f96b7fd86-k59ct /bin/sh
ls -l /var/log/nginx/modoru.log
```