Cloud Build + Cloud KMS + kustomizeでGKEへデプロイする


- Cloud Build
- Cloud KMS
- kustomize

を使ってGKEへデプロイするための手順メモ

## Cloud KMS

### KeyrigとKeyの作成

Credential暗号化用のKeyを作っていきます

```bash
$ gcloud kms keyrings create ci-keyring --location=global
$ gcloud kms keyrings list --location global
```

```bash
$ gcloud kms keys create ci-key --keyring ci-keyring --purpose encryption --location global
$ gcloud kms keys list --keyring ci-keyring --location global
```

### 認証情報を暗号化

[Using encrypted resources](https://cloud.google.com/cloud-build/docs/securing-builds/use-encrypted-secrets-credentials)

- ファイルの場合

```bash
$ gcloud kms encrypt --plaintext-file=.env --ciphertext-file=env.encrypted \
                     --keyring=ci-keyring --key=ci-key --location=global
$ git add env.encrypted
$ git commit
```

- 環境変数の場合

```bash
echo -n $MY_SECRET | gcloud kms encrypt \
  --plaintext-file=- \
  --ciphertext-file=- \
  --location=global \
  --keyring=ci-keyring \
  --key=ci-key | base64
```

暗号化したものはcommitします

## Cloud Build

### 権限の付与

- [Using encrypted resources](https://cloud.google.com/cloud-build/docs/securing-builds/use-encrypted-secrets-credentials)
- [Building, testing, and deploying artifacts](https://cloud.google.com/cloud-build/docs/configuring-builds/build-test-deploy-artifacts)

#### Cloud KMSに対する権限を付ける

```bash
$ gcloud kms keys add-iam-policy-binding \
    ci-key --location=global --keyring=ci-keyring \
    --member=serviceAccount:[PROJECT_NUMBER]@cloudbuild.gserviceaccount.com \
    --role=roles/cloudkms.cryptoKeyDecrypter
```

#### GKEへのデプロイ権限の付与

```bash
$ gcloud projects add-iam-policy-binding [PROJECT_ID]  \
    --member=serviceAccount:[PROJECT_NUMBER]@cloudbuild.gserviceaccount.com \
    --role=roles/container.developer
```
## kustomize

`k8s`ディレクトリを切って書いていく

```bash
k8s
├── base
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── service.yaml
└── overlays
    ├── dev
    │   ├── deployment.yaml
    │   └── kustomization.yaml
    └── prod
        ├── deployment.yaml
        └── kustomization.yaml
```

## cloudbuild.yamlの記述

- 最初のステップで暗号化したファイルを復号
- dockerビルド時に `$COMMIT_SHA`をタグとして付ける
- `kustomize edit set image`で[YAML内のimageをCOMMIT_SHA付きのタグに置換](https://github.com/kubernetes-sigs/kustomize/blob/master/examples/image.md)
    - dirは `kustomization.yaml`があるディレクトリを指定する
- secretsで環境変数を展開

```yaml
steps:
- name: gcr.io/cloud-builders/gcloud
  args:
    - kms
    - decrypt
    - --ciphertext-file=ci/dev/env.encrypted
    - --plaintext-file=.env
    - --location=global
    - --keyring=ci-keyring
    - --key=ci-key
- name: 'gcr.io/cloud-builders/docker'
  args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/example:$COMMIT_SHA', '.' ]
- name: 'gcr.io/cloud-builders/docker'
  args: ["push", "gcr.io/$PROJECT_ID/example:$COMMIT_SHA"]
- name: 'gcr.io/$PROJECT_ID/kustomize'
  dir: 'k8s/overlays/dev'
  args:
    - 'edit'
    - 'set'
    - 'image'
    - 'gcr.io/$PROJECT_ID/example=gcr.io/$PROJECT_ID/example:$COMMIT_SHA'
  env:
    - 'CLOUDSDK_COMPUTE_ZONE=asia-northeast1-a'
    - 'CLOUDSDK_CONTAINER_CLUSTER=target-cluster'
# https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/kustomize
- id: deploy
  name: 'gcr.io/$PROJECT_ID/kustomize'
  args:
    - 'build'
    - 'k8s/overlays/dev'
  env:
    - 'APPLY=true'
    - 'CLOUDSDK_COMPUTE_ZONE=asia-northeast1-a'
    - 'CLOUDSDK_CONTAINER_CLUSTER=target-cluster'
secrets:
- kmsKeyName: projects/[PROJECT-ID]/locations/global/keyRings/ci-keyring/cryptoKeys/ci-key
  secretEnv:
    MY_SECRET: <base64-encoded encrypted credential>
```

## 実行

ローカルから実行する場合は、 `COMMIT_SHA`を指定して実行する

```bash
$ gcloud builds submit --config cloudbuild.yaml . \
       --substitutions=COMMIT_SHA=$(git rev-parse HEAD)
```

## References

https://github.com/kubernetes-sigs/kustomize/blob/master/examples/image.md

[Substituting variable values](https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values?hl=ja)

[GKE+Cloud Build+kustomizeのデプロイ](https://qiita.com/maruware/items/a646500795dc0427cadc)