Vault Agent Sidecar Injectorの動作確認

## 1. はじめに

VaultのAgent Sidecar Injectorの動作確認を行った際の手順を記載したもの

## 2. Vault Agent Sidecar Injector

- https://developer.hashicorp.com/vault/docs/platform/k8s/injector
- https://www.hashicorp.com/blog/kubernetes-vault-integration-via-sidecar-agent-injector-vs-csi-provider

### 2.1. 動作概要

![vault.injector.overview.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/6e49eb77-e6ab-077f-165a-53a4933b60e9.png)

1. App Pod起動時に、initContainerやsidecarとして、Agent containerが起動する
2. Agent containerが、app用のService Accountを使ってVaultに認証し、tokenを取得する
3. Agent containerが、tokenを使ってVaultからsecretを取得する
4. Agent containerが取得した Secret情報は、App Container内のshared memory volume(共有ボリューム, /vault/secrets)にファイルとして格納される
5. App Container内のアプリが、共有ボリューム内のファイルを参照し、Secret情報を使用する

### 2.2. 動作に必要なリソース

- App Pod - Secretを利用する側
    - Namespace
    - Service Account
    - Deployment(Pod)
- Vault - Secretを格納する側
    - Secret
    - Policy
    - Auth Method

## 3. 動作確認の流れ

以下の流れで動作確認を実施した。

1. k8s clusterの構築
1. App Podの事前準備(Namespace, Service Account)
1. Vaultの構築
1. Vaultの設定(Secret, Policy, Auth Method)
1. App Podのデプロイ

## 4. 手順詳細

:::note info
動作確認にk8s(minikube)を使用したため、環境がない場合は下記を参考に構築しておく。
- https://qiita.com/n4k4y05h1/items/abff8c110e500ae47437
:::

### 4.1. k8s clusterの構築

検証用のk8s clusterを構築する。

![vault.injector.step1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/f1031fce-6aa0-d0f0-102b-cb5018fadf35.png)

minikubeを起動する。

```bash: コマンド
$ minikube start --memory='4g' --cpus=4
$ alias kubectl="minikube kubectl --"
$ kubectl get pod,svc -A
```

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ minikube start --memory='4g' --cpus=4
😄  minikube v1.33.0 on Ubuntu 24.04 (vbox/amd64)
✨  Automatically selected the docker driver
📌  Using Docker driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.43 ...
🔥  Creating docker container (CPUs=4, Memory=4096MB) ...
🐳  Preparing Kubernetes v1.30.0 on Docker 26.0.1 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ alias kubectl="minikube kubectl --"
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl get pod,svc -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS     AGE
kube-system   pod/coredns-7db6d8ff4d-mccl5           1/1     Running   0            35s
kube-system   pod/etcd-minikube                      1/1     Running   0            48s
kube-system   pod/kube-apiserver-minikube            1/1     Running   0            49s
kube-system   pod/kube-controller-manager-minikube   1/1     Running   0            53s
kube-system   pod/kube-proxy-hfr9j                   1/1     Running   0            35s
kube-system   pod/kube-scheduler-minikube            1/1     Running   0            49s
kube-system   pod/storage-provisioner                1/1     Running   1 (3s ago)   45s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  50s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   50s
ubuntu@ubuntu:~$
```
</details>

### 4.2. App Podの事前準備

Secretを参照するApp Podが使用するリソースを作成する。

:::note info
Vaultの設定でこのパラメータが必要となるため事前に作成したが、
パラメータが決まっていれば、実際にリソースがなくてもVault上で設定は可能。
:::

![vault.injector.step2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/d245697c-5b47-bcae-4cf0-78ec9c66f1f7.png)

NamespaceとService Accountを作成する。

```bash: コマンド
$ kubectl create ns app-ns
$ kubectl create sa app-sa -n app-ns
```

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl create ns app-ns
namespace/app-ns created
ubuntu@ubuntu:~$ kubectl create sa app-sa -n app-ns
serviceaccount/app-sa created
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl get ns
NAME              STATUS   AGE
app-ns            Active   56m
default           Active   57m
kube-node-lease   Active   57m
kube-public       Active   57m
kube-system       Active   57m
vault             Active   52m
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl get sa -n app-ns
NAME      SECRETS   AGE
app-sa    0         56m
default   0         56m
ubuntu@ubuntu:~$
```
</details>

### 4.3. Vault構築

Secretを管理するVaultを構築する。
VaultはApp Podとは別のNamespaceを使用する。

![vault.injector.step3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/a15040aa-d147-417d-f769-f97ddba566f4.png)

#### 4.3.1. Vaultインストール

Helmからインストールする。

```bash: コマンド
$ helm repo add hashicorp https://helm.releases.hashicorp.com
$ helm repo update
$ helm search repo hashicorp/vault
$ helm install vault hashicorp/vault -n vault --create-namespace
```

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ helm search repo hashicorp/vault
NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
hashicorp/vault                         0.28.0          1.16.1          Official HashiCorp Vault Chart
hashicorp/vault-secrets-operator        0.6.0           0.6.0           Official Vault Secrets Operator Chart
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ helm install vault hashicorp/vault -n vault --create-namespace
NAME: vault
LAST DEPLOYED: Sun May  5 07:28:48 2024
NAMESPACE: vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://developer.hashicorp.com/vault/docs


Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault
ubuntu@ubuntu:~$
```
</details>

#### 4.3.2. Vault初期設定

Vaultインストール時点では、下記の状態(READYが0/1)で、初期設定が必要となるため実施する。

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl get pod -n vault
NAME                                       READY   STATUS    RESTARTS   AGE
pod/vault-0                                0/1     Running   0          111s
pod/vault-agent-injector-ff58f5d77-5ggf9   1/1     Running   0          111s
ubuntu@ubuntu:~$
```

下記コマンドを実行する。

```bash: コマンド
$ kubectl exec -it vault-0 -n vault -- vault operator init
$ kubectl exec -it vault-0 -n vault -- vault operator unseal <Unseal Key 1>
$ kubectl exec -it vault-0 -n vault -- vault operator unseal <Unseal Key 2>
$ kubectl exec -it vault-0 -n vault -- vault operator unseal <Unseal Key 3>
```

:::note info
`vault operator init`コマンド実行後に出力されるInitial Root Tokenは後で使用するため控えておく
:::


<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl exec -it vault-0 -n vault -- vault operator init
Unseal Key 1: y1DN06xWAGLZcNPi6T4Ny6qlI2yaMxvZfGodHejuggom
Unseal Key 2: +T0IZltQR1iQOsgN7wSFPaSS9MyQreF+fhzgGpfuiLXs
Unseal Key 3: ESyeZ/iW4S8aws4bxnWQQZIXtEnN/jAvlbBI1rIu0L27
Unseal Key 4: efiHDkOqnrJFkYmqwEqd2JuwE8bWhEugZIXjaHupOFNQ
Unseal Key 5: 7+Noh+DMTpFOKQFEOJ3fdBh4XzqPL82wS+HVeXEA7+ov

Initial Root Token: hvs.322DPaVyEtWIExvLNgUyrbQf

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl exec -it vault-0 -n vault -- vault operator unseal y1DN06xWAGLZcNPi6T4Ny6qlI2yaMxvZfGodHejuggom
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       e13effe0-9c38-2ad1-1d0c-d030d4e6ccf3
Version            1.16.1
Build Date         2024-04-03T12:35:53Z
Storage Type       file
HA Enabled         false
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl exec -it vault-0 -n vault -- vault operator unseal +T0IZltQR1iQOsgN7wSFPaSS9MyQreF+fhzgGpfuiLXs
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       e13effe0-9c38-2ad1-1d0c-d030d4e6ccf3
Version            1.16.1
Build Date         2024-04-03T12:35:53Z
Storage Type       file
HA Enabled         false
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl exec -it vault-0 -n vault -- vault operator unseal ESyeZ/iW4S8aws4bxnWQQZIXtEnN/jAvlbBI1rIu0L27
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.16.1
Build Date      2024-04-03T12:35:53Z
Storage Type    file
Cluster Name    vault-cluster-24d8d5c0
Cluster ID      c7e6ab71-be02-d662-1bfe-2b2b20867e34
HA Enabled      false
ubuntu@ubuntu:~$
```
</details>

初期設定後、Vaultが起動していること(READYが1/1)を確認する。

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl get pod -n vault
NAME                                       READY   STATUS    RESTARTS   AGE
pod/vault-0                                1/1     Running   0          3m38s
pod/vault-agent-injector-ff58f5d77-5ggf9   1/1     Running   0          3m38s
ubuntu@ubuntu:~$
```

### 4.4. Vault設定

App Podが参照するSecret情報や必要となるVaultの設定を行う。

![vault.injector.step4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/e0bd3496-2ab9-26e2-6f7d-89055578f49a.png)

#### 4.4.1. vault login

下記コマンドを実行し、vaultにログインする。
ログイン時にtokenの入力が求められるため、`vault operator init`実行時に表示されたInitial Root Tokenを使用してログインする。

```bash: コマンド
$ kubectl exec -it vault-0 -n vault -- /bin/sh
(vault)$ vault login
```

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl exec -it vault-0 -n vault -- /bin/sh
/ $ vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.322DPaVyEtWIExvLNgUyrbQf
token_accessor       whbdSAeZlpAg40MKnePQtsdl
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
/ $
```
</details>

#### 4.4.2. Secret登録

App Pod内で使用するSecretを登録する。

```bash: コマンド
(vault)$ vault secrets enable -path=app-secret kv-v2
(vault)$ vault kv put app-secret/kv-secret SECRET_KEY1="aaaa" SECRET_KEY2="bbbb"
(vault)$ vault kv list app-secret/
(vault)$ vault kv get app-secret/kv-secret
```

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
/ $ vault secrets enable -path=app-secret kv-v2
Success! Enabled the kv-v2 secrets engine at: app-secret/
/ $ vault kv put app-secret/kv-secret SECRET_KEY1="aaaa" SECRET_KEY2="bbbb"
====== Secret Path ======
app-secret/data/kv-secret

======= Metadata =======
Key                Value
---                -----
created_time       2024-05-05T07:51:10.054349939Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
/ $
/ $
/ $ vault kv list app-secret/
Keys
----
kv-secret
/ $
/ $ vault kv get app-secret/kv-secret
====== Secret Path ======
app-secret/data/kv-secret

======= Metadata =======
Key                Value
---                -----
created_time       2024-05-05T07:51:10.054349939Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

======= Data =======
Key            Value
---            -----
SECRET_KEY1    aaaa
SECRET_KEY2    bbbb
/ $
```
</details>

#### 4.4.3. Policy設定

SecretへのPolicyを設定する。

```bash: コマンド
(vault)$ vault policy write app-secret-all-read-policy - <<EOF
path "app-secret/data/kv-secret" {
  capabilities = ["read"]
}
EOF
(vault)$ vault policy list
(vault)$ vault policy read app-secret-all-read-policy
```

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
/ $ vault policy write app-secret-all-read-policy - <<EOF
> path "app-secret/data/kv-secret" {
>   capabilities = ["read"]
> }
> EOF
Success! Uploaded policy: app-secret-all-read-policy
/ $
/ $ vault policy list
app-secret-all-read-policy
default
root
/ $
/ $ vault policy read app-secret-all-read-policy
path "app-secret/data/kv-secret" {
  capabilities = ["read"]
}
/ $
```
</details>


#### 4.4.4. Auth Method設定

app podがSecretを取得するための認証方法を設定し、ポリシーと紐づける。

```bash: コマンド
(vault)$ vault auth enable --path=kubernetes kubernetes
(vault)$ vault write auth/kubernetes/config kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
(vault)$ vault auth list
(vault)$ vault read auth/kubernetes/config
(vault)$ 
(vault)$ vault write auth/kubernetes/role/app-role \
  bound_service_account_names=app-sa \
  bound_service_account_namespaces=app-ns \
  policies=app-secret-all-read-policy \
  ttl=24h
(vault)$ vault list auth/kubernetes/role/
(vault)$ vault read auth/kubernetes/role/app-role
```

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
/ $ vault auth enable --path=kubernetes kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
/ $
/ $ vault write auth/kubernetes/config kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
Success! Data written to: auth/kubernetes/config
/ $
/ $ vault auth list
Path           Type          Accessor                    Description                Version
----           ----          --------                    -----------                -------
kubernetes/    kubernetes    auth_kubernetes_b514c948    n/a                        n/a
token/         token         auth_token_3da24388         token based credentials    n/a
/ $
/ $ vault read auth/kubernetes/config
Key                                  Value
---                                  -----
disable_iss_validation               true
disable_local_ca_jwt                 false
issuer                               n/a
kubernetes_ca_cert                   n/a
kubernetes_host                      https://10.96.0.1:443
pem_keys                             []
token_reviewer_jwt_set               false
use_annotations_as_alias_metadata    false
/ $
/ $ vault write auth/kubernetes/role/app-role \
>   bound_service_account_names=app-sa \
>   bound_service_account_namespaces=app-ns \
>   policies=app-secret-all-read-policy \
>   ttl=24h
Success! Data written to: auth/kubernetes/role/app-role
/ $
/ $ vault list auth/kubernetes/role/
Keys
----
app-role
/ $
/ $ vault read auth/kubernetes/role/app-role
Key                                         Value
---                                         -----
alias_name_source                           serviceaccount_uid
bound_service_account_names                 [app-sa]
bound_service_account_namespace_selector    n/a
bound_service_account_namespaces            [app-ns]
policies                                    [app-secret-all-read-policy]
token_bound_cidrs                           []
token_explicit_max_ttl                      0s
token_max_ttl                               0s
token_no_default_policy                     false
token_num_uses                              0
token_period                                0s
token_policies                              [app-secret-all-read-policy]
token_ttl                                   24h
token_type                                  default
ttl                                         24h
/ $
```
</details>

#### 4.4.5. Vault Podからログアウト

Vaultの設定は完了したので、Vault Podからexitする。

```bash: 実行結果
/ $ exit
ubuntu@ubuntu:~$
```

### 4.5. App Podのデプロイ

App Podをデプロイし、VaultからSecretが取得できていることを確認する。

![vault.injector.step5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/2d3d71f3-c590-3976-df4a-f768251800a7.png)

App Pod用のmanifestを作成し、applyする。
apply後、App Podが作成されるので、共有ボリューム(/vault/secrets)に、Secret用のファイルが生成されていることを確認する。

```bash: 実行結果
ubuntu@ubuntu:~$ vi app-deployment.yaml
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl apply -f app-deployment.yaml
deployment.apps/app-deployment created
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl get pod -n app-ns
NAME                              READY   STATUS    RESTARTS   AGE
app-deployment-5bbbdd5cb9-q6h8b   2/2     Running   0          11s
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ export POD_NAME=$(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}')
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl exec -ti -n app-ns $POD_NAME -c app -- cat /vault/secrets/kv-secret.txt
{ SECRET_KEY1: "aaaa", SECRET_KEY2: "bbbb" }
ubuntu@ubuntu:~$
```
    
<details><summary>app-deployment.yamlを展開する</summary>

```yaml: app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
  namespace: app-ns
  labels:
    app: app-label
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-label
  template:
    metadata:
      labels:
        app: app-label
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "app-role"
        vault.hashicorp.com/agent-inject-secret-kv-secret.txt: "app-secret/data/kv-secret"
        vault.hashicorp.com/agent-inject-template-kv-secret.txt: |
          {{- with secret "app-secret/data/kv-secret" -}}
            { SECRET_KEY1: "{{ .Data.data.SECRET_KEY1 }}", SECRET_KEY2: "{{ .Data.data.SECRET_KEY2 }}" }
          {{- end -}}
    spec:
      serviceAccountName: app-sa
      containers:
        - name: app
          image: ubuntu:24.04
          command: ["/bin/sh", "-c", "while :; do sleep 10; done"]
```
</details>

## 5. Vault Agent Sidecar Injectorの挙動確認

### 5.1. Vault Injectorが作成するContainer

Vault Injector(annotation)によってApp Pod内に追加されるコンテナーは、vault-agent-initとvault-agentの2つ。

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl describe pod -n app-ns $(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}')
Name:             app-deployment-5bbbdd5cb9-q6h8b
Namespace:        app-ns
Priority:         0
Service Account:  app-sa
Node:             minikube/192.168.49.2
Start Time:       Sun, 05 May 2024 09:03:58 +0000
Labels:           app=app-label
                  pod-template-hash=5bbbdd5cb9
Annotations:      vault.hashicorp.com/agent-inject: true
                  vault.hashicorp.com/agent-inject-secret-kv-secret.txt: app-secret/data/kv-secret
                  vault.hashicorp.com/agent-inject-status: injected
                  vault.hashicorp.com/role: app-role
Status:           Running
IP:               10.244.0.12
IPs:
  IP:           10.244.0.12
Controlled By:  ReplicaSet/app-deployment-5bbbdd5cb9
Init Containers:
  vault-agent-init:
    Container ID:  docker://b25cd0453ababcd37aad895e83d77b143659902cc438ad23ac643fd25c163837
    Image:         hashicorp/vault:1.16.1
    Image ID:      docker-pullable://hashicorp/vault@sha256:5efe43c659d4dc53539a589a95d5ffe77fffb761b407766e0408b0b18c30cad8
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 05 May 2024 09:04:00 +0000
      Finished:     Sun, 05 May 2024 09:04:00 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      NAMESPACE:         app-ns (v1:metadata.namespace)
      HOST_IP:            (v1:status.hostIP)
      POD_IP:             (v1:status.podIP)
      VAULT_LOG_LEVEL:   info
      VAULT_LOG_FORMAT:  standard
      VAULT_CONFIG:      eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoiYXBwLXJvbGUiLCJ0b2tlbl9wYXRoIjoiL3Zhci9ydW4vc2VjcmV0cy9rdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3Rva2VuIn19LCJzaW5rIjpbeyJ0eXBlIjoiZmlsZSIsImNvbmZpZyI6eyJwYXRoIjoiL2hvbWUvdmF1bHQvLnZhdWx0LXRva2VuIn19XX0sImV4aXRfYWZ0ZXJfYXV0aCI6dHJ1ZSwicGlkX2ZpbGUiOiIvaG9tZS92YXVsdC8ucGlkIiwidmF1bHQiOnsiYWRkcmVzcyI6Imh0dHA6Ly92YXVsdC52YXVsdC5zdmM6ODIwMCJ9LCJ0ZW1wbGF0ZSI6W3siZGVzdGluYXRpb24iOiIvdmF1bHQvc2VjcmV0cy9rdi1zZWNyZXQudHh0IiwiY29udGVudHMiOiJ7eyB3aXRoIHNlY3JldCBcImFwcC1zZWNyZXQvZGF0YS9rdi1zZWNyZXRcIiB9fXt7IHJhbmdlICRrLCAkdiA6PSAuRGF0YSB9fXt7ICRrIH19OiB7eyAkdiB9fVxue3sgZW5kIH19e3sgZW5kIH19IiwibGVmdF9kZWxpbWl0ZXIiOiJ7eyIsInJpZ2h0X2RlbGltaXRlciI6In19In1dLCJ0ZW1wbGF0ZV9jb25maWciOnsiZXhpdF9vbl9yZXRyeV9mYWlsdXJlIjp0cnVlfX0=
    Mounts:
      /home/vault from home-init (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lhvrv (ro)
      /vault/secrets from vault-secrets (rw)
Containers:
  app:
    Container ID:  docker://eb08cadde7325f1db8679a57226e4aa802f103db5f43c2d3472f7e19041e3b9b
    Image:         ubuntu:24.04
    Image ID:      docker-pullable://ubuntu@sha256:3f85b7caad41a95462cf5b787d8a04604c8262cdcdf9a472b8c52ef83375fe15
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      while :; do sleep 10; done
    State:          Running
      Started:      Sun, 05 May 2024 09:04:02 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lhvrv (ro)
      /vault/secrets from vault-secrets (rw)
  vault-agent:
    Container ID:  docker://05045e4fa3bd273fe7a1c770ab3c866d6eb970f6205501caee11a68d0ca9c746
    Image:         hashicorp/vault:1.16.1
    Image ID:      docker-pullable://hashicorp/vault@sha256:5efe43c659d4dc53539a589a95d5ffe77fffb761b407766e0408b0b18c30cad8
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Running
      Started:      Sun, 05 May 2024 09:04:02 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      NAMESPACE:         app-ns (v1:metadata.namespace)
      HOST_IP:            (v1:status.hostIP)
      POD_IP:             (v1:status.podIP)
      VAULT_LOG_LEVEL:   info
      VAULT_LOG_FORMAT:  standard
      VAULT_CONFIG:      eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoiYXBwLXJvbGUiLCJ0b2tlbl9wYXRoIjoiL3Zhci9ydW4vc2VjcmV0cy9rdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3Rva2VuIn19LCJzaW5rIjpbeyJ0eXBlIjoiZmlsZSIsImNvbmZpZyI6eyJwYXRoIjoiL2hvbWUvdmF1bHQvLnZhdWx0LXRva2VuIn19XX0sImV4aXRfYWZ0ZXJfYXV0aCI6ZmFsc2UsInBpZF9maWxlIjoiL2hvbWUvdmF1bHQvLnBpZCIsInZhdWx0Ijp7ImFkZHJlc3MiOiJodHRwOi8vdmF1bHQudmF1bHQuc3ZjOjgyMDAifSwidGVtcGxhdGUiOlt7ImRlc3RpbmF0aW9uIjoiL3ZhdWx0L3NlY3JldHMva3Ytc2VjcmV0LnR4dCIsImNvbnRlbnRzIjoie3sgd2l0aCBzZWNyZXQgXCJhcHAtc2VjcmV0L2RhdGEva3Ytc2VjcmV0XCIgfX17eyByYW5nZSAkaywgJHYgOj0gLkRhdGEgfX17eyAkayB9fToge3sgJHYgfX1cbnt7IGVuZCB9fXt7IGVuZCB9fSIsImxlZnRfZGVsaW1pdGVyIjoie3siLCJyaWdodF9kZWxpbWl0ZXIiOiJ9fSJ9XSwidGVtcGxhdGVfY29uZmlnIjp7ImV4aXRfb25fcmV0cnlfZmFpbHVyZSI6dHJ1ZX19
    Mounts:
      /home/vault from home-sidecar (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lhvrv (ro)
      /vault/secrets from vault-secrets (rw)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  kube-api-access-lhvrv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  home-init:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  home-sidecar:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  vault-secrets:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      Memory
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  39s   default-scheduler  Successfully assigned app-ns/app-deployment-5bbbdd5cb9-q6h8b to minikube
  Normal  Pulled     38s   kubelet            Container image "hashicorp/vault:1.16.1" already present on machine
  Normal  Created    37s   kubelet            Created container vault-agent-init
  Normal  Started    37s   kubelet            Started container vault-agent-init
  Normal  Pulled     36s   kubelet            Container image "ubuntu:24.04" already present on machine
  Normal  Created    36s   kubelet            Created container app
  Normal  Started    35s   kubelet            Started container app
  Normal  Pulled     35s   kubelet            Container image "hashicorp/vault:1.16.1" already present on machine
  Normal  Created    35s   kubelet            Created container vault-agent
  Normal  Started    35s   kubelet            Started container vault-agent
ubuntu@ubuntu:~$
```
</details>


### 5.2. agent-inject-templateの役割

Annoationsにあるagent-inject-templateをコメントアウトした場合、data以外にmetadataなどの情報も一緒に格納された。

```yaml: app-deployment.yaml
        annotations:
          vault.hashicorp.com/agent-inject: "true"
          vault.hashicorp.com/role: "app-role"
          vault.hashicorp.com/agent-inject-secret-kv-secret.txt: "app-secret/data/kv-secret"
          # コメントアウトする
          #vault.hashicorp.com/agent-inject-template-kv-secret.txt: |
          #  {{- with secret "app-secret/data/kv-secret" -}}
          #    { SECRET_KEY1: "{{ .Data.data.SECRET_KEY1 }}", SECRET_KEY2: "{{ .Data.data.SECRET_KEY2 }}" }
          #  {{- end -}}
```

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl exec -ti -n app-ns $(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}') -c app -- cat /vault/secrets/kv-secret.txt
data: map[SECRET_KEY1:aaaa SECRET_KEY2:bbbb]
metadata: map[created_time:2024-05-05T07:51:10.054349939Z custom_metadata:<nil> deletion_time: destroyed:false version:1]
ubuntu@ubuntu:~$ 
```

### 5.3. Vault Secret変更時のApp Pod内のSecretの挙動

Vaultに登録しているSecretを変更したところ、App Pod内のSecretも少し待つと自動で更新された。

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
/ $ ## 事前のパラメータ確認
/ $ vault kv get app-secret/kv-secret
====== Secret Path ======
app-secret/data/kv-secret

======= Metadata =======
Key                Value
---                -----
created_time       2024-05-05T07:51:10.054349939Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

======= Data =======
Key            Value
---            -----
SECRET_KEY1    aaaa
SECRET_KEY2    bbbb
/ $
/ $ ## Secret変更
/ $ vault kv put app-secret/kv-secret SECRET_KEY1="cccc" SECRET_KEY2="dddd"
====== Secret Path ======
app-secret/data/kv-secret

======= Metadata =======
Key                Value
---                -----
created_time       2024-05-05T09:20:32.157112084Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2
/ $
/ $ ## Secret変更確認
/ $ vault kv get app-secret/kv-secret
====== Secret Path ======
app-secret/data/kv-secret

======= Metadata =======
Key                Value
---                -----
created_time       2024-05-05T09:20:32.157112084Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

======= Data =======
Key            Value
---            -----
SECRET_KEY1    cccc
SECRET_KEY2    dddd
/ $
/ $
/ $ exit
ubuntu@ubuntu:~$ ## Pod内のSecretを確認
ubuntu@ubuntu:~$ kubectl exec -ti -n app-ns $(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}') -c app -- cat /vault/secrets/kv-secret.txt
{ SECRET_KEY1: "aaaa", SECRET_KEY2: "bbbb" }
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl exec -ti -n app-ns $(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}') -c app -- cat /vault/secrets/kv-secret.txt
{ SECRET_KEY1: "aaaa", SECRET_KEY2: "bbbb" }
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ ## このタイミングでSecretが変更された
ubuntu@ubuntu:~$ kubectl exec -ti -n app-ns $(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}') -c app -- cat /vault/secrets/kv-secret.txt
{ SECRET_KEY1: "cccc", SECRET_KEY2: "dddd" }
ubuntu@ubuntu:~$
```
</details>


vault-agentのログを確認すると、`agent: (runner) rendered "(dynamic)" => "/vault/secrets/kv-secret.txt"`というログが出力していたので、sidecar側で更新を検知して、App Pod内の共有ボリュームの書き換えを実施しているみたい。

```bash: vault-agentのlog
ubuntu@ubuntu:~$ kubectl logs -n app-ns  $(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}') -c vault-agent
==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

           Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.16.1, built 2024-04-03T12:35:53Z
             Version Sha: 6b5986790d7748100de77f7f127119c4a0f78946

2024-05-05T09:16:51.018Z [INFO]  agent.sink.file: creating file sink
2024-05-05T09:16:51.018Z [INFO]  agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2024-05-05T09:16:51.018Z [INFO]  agent.exec.server: starting exec server
2024-05-05T09:16:51.018Z [INFO]  agent.exec.server: no env templates or exec config, exiting
2024-05-05T09:16:51.018Z [INFO]  agent.auth.handler: starting auth handler
2024-05-05T09:16:51.018Z [INFO]  agent.auth.handler: authenticating
2024-05-05T09:16:51.018Z [INFO]  agent.template.server: starting template server
2024-05-05T09:16:51.018Z [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2024-05-05T09:16:51.020Z [INFO]  agent: (runner) creating watcher
2024-05-05T09:16:51.020Z [INFO]  agent.sink.server: starting sink server
2024-05-05T09:16:51.036Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2024-05-05T09:16:51.036Z [INFO]  agent.auth.handler: starting renewal process
2024-05-05T09:16:51.036Z [INFO]  agent.sink.file: token written: path=/home/vault/.vault-token
2024-05-05T09:16:51.036Z [INFO]  agent.template.server: template server received new token
2024-05-05T09:16:51.036Z [INFO]  agent: (runner) stopping
2024-05-05T09:16:51.036Z [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2024-05-05T09:16:51.036Z [INFO]  agent: (runner) creating watcher
2024-05-05T09:16:51.036Z [INFO]  agent: (runner) starting
2024-05-05T09:16:51.052Z [INFO]  agent.auth.handler: renewed auth token
2024-05-05T09:21:26.849Z [INFO]  agent: (runner) rendered "(dynamic)" => "/vault/secrets/kv-secret.txt"
ubuntu@ubuntu:~$
```


### 5.4 Vaultのログ

Vault側のログ確認が必要な場合は、`vault audit enable`を実行する。

```bash: 実行結果
/ $ vault audit enable file file_path=/tmp/audit.log log_raw=true
Success! Enabled the file audit device at: file/
/ $ vault audit list
Path     Type    Description
----     ----    -----------
file/    file    n/a
/ $
/ $ tail -f /tmp/audit.log
## vaultのログが出力する
```

## 6. 環境削除

動作確認に使用したリソースを削除する。

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl delete -f app-deployment.yaml
deployment.apps "app-deployment" deleted
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ rm app-deployment.yaml
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl delete sa app-sa -n app-ns
serviceaccount "app-sa" deleted
ubuntu@ubuntu:~$ kubectl delete ns app-ns
namespace "app-ns" deleted
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ helm uninstall vault -n vault
release "vault" uninstalled
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ helm repo remove hashicorp
"hashicorp" has been removed from your repositories
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ minikube delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /home/ubuntu/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ unalias kubectl
ubuntu@ubuntu:~$
```