How to backup and restore sealed secrets?

# What is sealed secrets?
Sealed secrets are encrypted Kubernetes secrets. The secrets can be encrypted by anyone but can be decrypted only by the sealed secrets controller running in the Kubernetes cluster. The controller decrypts the sealed secrets to generate the Kubernetes secrets and applies it to a cluster

```
This is how sealed secrets looks like:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: sealed-secrets
  namespace: test-namespace
spec:
  encryptedData:
    foo: De1O3T4ErOm3DcGqVb123URIO.....
    bar: chuYOP847NMKleyxp9068NhPP.....
```

Unlike Kubernetes secrets resources which can be decoded easily, sealed secrets can be decrypted only by the controller. This makes it useful to manage secrets along with other Kubernetes configuration in version control system such as git. 

# Why do we need to backup the sealed secrets?
Sealed secrets uses public/private RSA key pair generated by sealed secrets controller to achieve the encryption and decryption. So if the controller is removed due to some cluster failure or any other case then we cannot decrypt the sealed secrets even with newly created sealed secrets controller. We need to again create new sealed secrets which might be sometimes tedious and not feasible.

# How can I backup and restore the sealed secrets?
By backing up the sealed secrets encryption private keys used to encrypt the secrets.

```
$ kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >master.key
```
And to restore we can simply apply the backed up key in new cluster

```
$ kubectl apply -f master.key
```
Remember to apply this key before starting the controller or if the controller has already been started, restart the controller after applying the key


# Demo
## Check the current Kubernetes context
```
$ kubectl config current-context
arn:aws:eks:ap-northeast-1:XXXXXXX:cluster/test-eks-cluster
```

## Apply the sealed secrets controller

```
$ kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.6/controller.yaml
```
It wil applied to `kube-system` namespace

```
$ kubectl get deployment -n kube-system sealed-secrets-controller
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
sealed-secrets-controller   1/1     1            1           3d
```
## Let's create a namespace for testing 
```
$ kubectl create ns backup-restore
namespace/backup-restore created
```
```
$ kubectl get ns backup-restore
NAME             STATUS   AGE
backup-restore   Active   22s
```
## Create a sealed secrets
- Create secrets file using dry-run

   ```
$ kubectl create secret generic test-secret --dry-run -n backup-restore -o yaml --from-literal=test_password1='secrets1' --from-literal=test_password2='secrets2'>test-secret.yaml
   ```
- Fetch the certificate to seal the secrets

```
$ kubeseal --fetch-cert > cert.pem
```
- Create a sealed secrets

```
$ kubeseal -o yaml --cert cert.pem < test-secret.yaml > sealed-secrets.yaml
$ cat sealed-secrets.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: test-secret
  namespace: backup-restore
spec:
  encryptedData:
    test_password1: AgCKzkAeArDQnRrUf2gfZ3MQCYh6+ULC++Uig8ouA7VOUw8WM9DfcTFG/KSIBy+20VuVN4Xp05BEaNUReiR9SoRT/wi+5DUzSdljctUKhuWQCXI/hTvFiDwaeRSvOC9kIVEDPJRelg7kl+6MYREGZMrX/aW4Wgp5ZGbmlqlQ+409/1ehbfVe6j8ld6nwzAQkfj0D+hd5cdVjKHvwuLACYc4bZ+fZ5nEFZ2sDw4iH7zf/c1xNnsj8k1fpsg5uCQVDyaP/8SepYF4D/16LmUCMevMQ9t6zsr8cuLR+l8rajSCG9N0wi9NEWFZup6kkJge+nTiTog4O840GGxAwO6PzuerU8K83xSlgntq0A+VJAC7mPfd9TOwLeEytfbfzxLjbUkbag07ARlRi5lvbAzIQvW22DJXen4OlTFGk6TIdnObS28c0W5V/D23Ws+GWm9cbEURP9Okef0vT+nKRzR/OphJP+vnBlkaAq7syt3RummLBiTfT1aV2QwSjTTgM07bfKcyZv4R2fYVWt5lktl15PKABhdKQnwBX3wGA43c1zsSDnMdpQYjTb9v8kpIeKh8hFOFwqXx3/JuXVOiHvarDOUBIZ+VTeWAOHfe2D5mM4nhWkvie4P77koLvpEz4FPS8wLPWpBRAcoygFkao8WMDxNUuS/zU06Vqf4OR4gqPdO94f3FlLZ4x9oKLTBZETG1swHbpH0YKLOcdRg==
    test_password2: AgAoODKJyL7/exmq55yaeXKZsmrPr9chJthdL335Z++OGajMxpRsUxHgcWRSr7ReOQoZsDNbhVFrx33/+yRt8b4diH50rO3buRBASoAqajjJTdAf7lsUD8B6GmvyXbtosBr/C1GDFlewyLzH0x0/4lPbbHsjRFADCDbVWO1kCVnhx5IHO1ClpHuGbmMtmE8IJa3U5BtsyOzw37a5U54FmAeWrCQx08B6fb28fP4ObHyGzl1H3TdqheERMqQV0GBdpjO0ByK6sMb2dPRDaQcrz2BBUNWpFdwSbjPUG++OPfTC72cpijID5pK5w+cWcf4xQ0UY4G5IDE91kUNYeJWnTaoWgAMC5wcarDJBSzqUQpkztUh9/bVzQO7+PGl7Jww6jL175z2yh2C4m8VvglPRVO/KPHf68obYHvKDxzayq9hfHj2AopoXmapzhVHm7NEuqhwKnjY5RDoja7rW20an7zJMnUMNyHhvpKnpRpQJyg243GR7UFaZqn5DMLqxawRodRYBgq9zQsVwqUX7jThZI++XNjNm/tfEXi39Yo7g4bgSQoBGEg5xdd0VPY2QtzCLNrAhXd8O5wqdm5B5cuJ3u2ucV5PvU4/46tn59R2i72mxyfI4ZuKOb4jl0BClV7wtXD6wy2oSBj1tvuBH7OL+C2C51HLBHzRuoNKNvFPzEFojeNezFzzk3AT7bhyrmqEqhz2YukxzZJ6gRQ==
  template:
    metadata:
      creationTimestamp: null
      name: test-secret
      namespace: backup-restore
```
- Check the existing secrets with label `sealedsecrets.bitnami.com/sealed-secrets-key`

```
$ kubectl get secrets -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key

NAME                       TYPE                DATA   AGE
sealed-secrets-keypjmxr   kubernetes.io/tls   2      10m
```
## Apply the sealed secrets

```
$ kubectl apply -f sealed-secrets.yaml
sealedsecret.bitnami.com/test-secret created

$ kubectl get secrets -n backup-restore
NAME                  TYPE                                  DATA   AGE
default-token-zdz8g   kubernetes.io/service-account-token   3      12h
test-secret           Opaque                                2      6s
```

## Backup the sealed secrets

```
$ kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >backup_sealed_secrets.key
```

## Delete the sealed secrets along with controller for testing 

- delete the secrets

```
$ kubectl delete secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key

secret "sealed-secrets-keypjmxr" deleted

$ kubectl delete -f sealed-secrets.yaml 

sealedsecret.bitnami.com "test-secret" deleted
```
- delete the controller
 
```
$ kubectl delete -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.6/controller.yaml

serviceaccount "sealed-secrets-controller" deleted
deployment.apps "sealed-secrets-controller" deleted
service "sealed-secrets-controller" deleted
role.rbac.authorization.k8s.io "sealed-secrets-service-proxier" deleted
rolebinding.rbac.authorization.k8s.io "sealed-secrets-controller" deleted
clusterrolebinding.rbac.authorization.k8s.io "sealed-secrets-controller" deleted
customresourcedefinition.apiextensions.k8s.io "sealedsecrets.bitnami.com" deleted
rolebinding.rbac.authorization.k8s.io "sealed-secrets-service-proxier" deleted
role.rbac.authorization.k8s.io "sealed-secrets-key-admin" deleted
clusterrole.rbac.authorization.k8s.io "secrets-unsealer" deleted
```

## Test if we can create a secrets using existing sealed secrets in new controller
 
- apply the new sealed secrets controller

```
$ kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.6/controller.yaml

deployment.apps/sealed-secrets-controller created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
clusterrolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
clusterrole.rbac.authorization.k8s.io/secrets-unsealer created
serviceaccount/sealed-secrets-controller created
customresourcedefinition.apiextensions.k8s.io/sealedsecrets.bitnami.com created
service/sealed-secrets-controller created
role.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
role.rbac.authorization.k8s.io/sealed-secrets-key-admin created
```
- apply the sealed secrets

```
$ kubectl apply -f sealed-secrets.yaml
sealedsecret.bitnami.com/test-secret created

$ kubectl get secrets -n backup-restore
NAME                  TYPE                                  DATA   AGE
default-token-zdz8g   kubernetes.io/service-account-token   3      12h
```
Here, we can see the kubernetes secrets is not created.

## Restore the sealed secrets

```
$ kubectl apply -f backup_sealed_secrets.key
secret/sealed-secrets-keypjmxr created
```
But we need to restart the controller

## Delete the controller again before restoring

```
$ kubectl get pods -n kube-system

sealed-secrets-controller-24a9bggd5-t34gn

$ kubectl delete pods sealed-secrets-controller-24a9bggd5-t34gn -n kube-system
```

## Check secrets
```
$ kubectl get secrets -n backup-restore
NAME                  TYPE                                  DATA   AGE
default-token-zdz8g   kubernetes.io/service-account-token   3      12h
test-secret           Opaque                                2      57m
```