Vault Secrets Operatorの動作確認

## 1. はじめに

Vault Secrets Operatorの動作確認を行った際の手順を記載したもの

## 2. Vault Secrets Operator(VSO)

- https://developer.hashicorp.com/vault/docs/platform/k8s/vso
- https://www.hashicorp.com/blog/kubernetes-vault-integration-via-sidecar-agent-injector-vs-csi-provider
- https://developer.hashicorp.com/vault/docs/platform/k8s/vso/sources/vault

### 2.1. 動作概要

![vault.vso.overview.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/a94e9287-699d-6740-cbad-4f83c267cbb4.png)

1. Vault Secrets OperatorがCRDsを確認
1. Vault Secrets OperatorがCRDsをもとに、VaultからSecretを取得
1. Vault Secrets OperatorがCRDsで指定されたパスに、Secretを作成

### 2.2. 動作に必要なリソース

- App Pod
    - Namespace
    - Service Account
    - Secret
- Vault Secrets Operator
    - VaultConnection
    - VaultAuth
    - VaultStaticSecret
- Vault
    - Secret
    - Policy
    - Auth Method

## 3. 動作確認の流れ

以下の流れで動作確認を実施した。

1. k8s clusterの構築
1. App Podの事前準備(Namespace, Service Account)
1. Vaultの構築
1. Vaultの設定(Secret, Policy, Auth Method)
1. VSOの構築
1. VSO CRDsのデプロイ(VaultConnection, VaultAuth, VaultStaticSecret)
1. Secretの確認

## 4. 手順詳細

:::note info
動作確認にk8s(minikube)を使用したため、環境がない場合は下記を参考に構築しておく。
- https://qiita.com/n4k4y05h1/items/abff8c110e500ae47437
:::

### 4.1. k8s clusterの構築

検証用のk8s clusterを構築する。

![vault.vso.step1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/9b251af2-33bf-8f66-93a0-5827f862ad33.png)


[手順はこちらを参照](https://qiita.com/n4k4y05h1/items/a03ede2de60ccd888e9a#41-k8s-cluster%E3%81%AE%E6%A7%8B%E7%AF%89)

### 4.2. App環境の事前準備

Secretが生成される環境を作成する。

![vault.vso.step2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/197e4488-94df-203e-b9a5-a4d3f1f71ff8.png)

[手順はこちらを参照](https://qiita.com/n4k4y05h1/items/a03ede2de60ccd888e9a#42-app-pod%E3%81%AE%E4%BA%8B%E5%89%8D%E6%BA%96%E5%82%99)

### 4.3. Vault構築

Secretを管理するVaultを構築する。
VaultはApp環境とは別のNamespaceを使用する。

![vault.vso.step3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/c6e007c3-7849-76f3-2207-8a747b0b0fdc.png)

[手順はこちらを参照](https://qiita.com/n4k4y05h1/items/a03ede2de60ccd888e9a#43-vault%E6%A7%8B%E7%AF%89)

:::note info
vaultを構築すると、vault-agent-injectorもデプロイされる。
`injector.enabled=false`オプションをつけることでデプロイしないこともできる。
VSOの動作に影響はしないが、injectorが不要な場合はこちらの手順で構築する。

```bash: コマンド
$ helm install vault hashicorp/vault -n vault --create-namespace \
                                     --set 'injector.enabled=false'
```

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ helm install vault hashicorp/vault -n vault --create-namespace --set 'injector.enabled=false'
NAME: vault
LAST DEPLOYED: Thu May  9 05:18:55 2024
NAMESPACE: vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://developer.hashicorp.com/vault/docs


Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl get pod -n vault
NAME      READY   STATUS    RESTARTS   AGE
vault-0   0/1     Running   0          13s
ubuntu@ubuntu:~$
```
</details>
:::



### 4.4. Vault設定

App Podが参照するSecret情報や必要となるVaultの設定を行う。

![vault.vso.step4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/a04da1c3-b706-fcaa-b70b-b6a1d7e1f803.png)

[手順はこちらを参照](https://qiita.com/n4k4y05h1/items/a03ede2de60ccd888e9a#44-vault%E8%A8%AD%E5%AE%9A)

### 4.5. Vault Secrets Operatorの構築

![vault.vso.step5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/6b501fc1-f12a-b597-a541-049e557b9fe2.png)

Vault Secrets Operatorを構築する。

```bash: コマンド
$ helm search repo hashicorp/vault-secrets-operator
$ helm install vault-secrets-operator hashicorp/vault-secrets-operator --version 0.6.0 -n vault-secrets-operator --create-namespace
$ kubectl get all -n vault-secrets-operator
```
    
<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ helm search repo hashicorp/vault-secrets-operator
NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
hashicorp/vault-secrets-operator        0.6.0           0.6.0           Official Vault Secrets Operator Chart
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ helm install vault-secrets-operator hashicorp/vault-secrets-operator --version 0.6.0 -n vault-secrets-operator --create-namespace
NAME: vault-secrets-operator
LAST DEPLOYED: Thu May  9 04:54:29 2024
NAMESPACE: vault-secrets-operator
STATUS: deployed
REVISION: 1
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl get all -n vault-secrets-operator
NAME                                                             READY   STATUS    RESTARTS   AGE
pod/vault-secrets-operator-controller-manager-6954c7fb8c-ntv2q   2/2     Running   0          51s

NAME                                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/vault-secrets-operator-metrics-service   ClusterIP   10.100.252.17   <none>        8443/TCP   51s

NAME                                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-secrets-operator-controller-manager   1/1     1            1           51s

NAME                                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-secrets-operator-controller-manager-6954c7fb8c   1         1         1       51s
ubuntu@ubuntu:~$ 
```
</details>

### 4.6. VSO CRDsのデプロイ

VSOのCRDをデプロイする。
CRDをデプロイすると、VSOがSecretファイルを生成する。

![vault.vso.step6.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3692186/3610e956-fe53-6bb7-228b-dc6b074d5495.png)

#### 4.6.1. VaultConnection

```bash: コマンド
$ kubectl apply -f -<<EOF
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: vaultconnection
  namespace: app-ns
spec:
  address: http://vault.vault.svc.cluster.local:8200
  skipTLSVerify: true
EOF
$ kubectl get VaultConnection -n app-ns
```

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl apply -f -<<EOF
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: vaultconnection
  namespace: app-ns
spec:
  address: http://vault.vault.svc.cluster.local:8200
  skipTLSVerify: true
EOF
vaultconnection.secrets.hashicorp.com/vaultconnection created
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl get VaultConnection -n app-ns
NAME                                                    AGE
vaultconnection.secrets.hashicorp.com/vaultconnection   16s
ubuntu@ubuntu:~$ 
```
</details>

#### 4.6.2. VaultAuth

```bash: コマンド
$ kubectl apply -f -<<EOF
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vaultauth
  namespace: app-ns
spec:
  vaultConnectionRef: vaultconnection
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: app-role
    serviceAccount: app-sa
EOF
$ kubectl get VaultAuth -n app-ns
```

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl apply -f -<<EOF
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vaultauth
  namespace: app-ns
spec:
  vaultConnectionRef: vaultconnection
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: app-role
    serviceAccount: app-sa
EOF
vaultauth.secrets.hashicorp.com/vaultauth created
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl get VaultAuth -n app-ns
NAME                                        AGE
vaultauth.secrets.hashicorp.com/vaultauth   10s
ubuntu@ubuntu:~$ 
```
</details>

#### 4.6.3. VaultStaticSecret

```bash: コマンド
$ kubectl apply -f -<<EOF
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vaultstaticsecret
  namespace: app-ns
spec:
  vaultAuthRef: vaultauth
  type: kv-v2
  mount: app-secret
  path: kv-secret
  refreshAfter: 60s
  destination:
    create: true
    name: app-secret
EOF
$ kubectl get VaultStaticSecret,Secret -n app-ns
```

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl apply -f -<<EOF
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vaultstaticsecret
  namespace: app-ns
spec:
  vaultAuthRef: vaultauth
  type: kv-v2
  mount: app-secret
  path: kv-secret
  refreshAfter: 60s
  destination:
    create: true
    name: app-secret
EOF
vaultstaticsecret.secrets.hashicorp.com/vaultstaticsecret created
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl get VaultStaticSecret -n app-ns
NAME                                                         AGE
vaultstaticsecret.secrets.hashicorp.com/vaultstaticsecret   3s
ubuntu@ubuntu:~$ 
```
</details>

### 4.7. Secret確認

4.6でCRDsをデプロイ後、VSO側でSecretファイルを生成するため確認する。

```bash: コマンド
$ kubectl get secret -n app-ns
$ kubectl get secret -n app-ns -o yaml
```

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl get secret -n app-ns
NAME         TYPE     DATA   AGE
app-secret   Opaque   3      36s
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl get secret -n app-ns -o yaml
apiVersion: v1
items:
- apiVersion: v1
  data:
    _raw: eyJkYXRhIjp7IlNFQ1JFVF9LRVkxIjoiYWFhYSIsIlNFQ1JFVF9LRVkyIjoiYmJiYiJ9LCJtZXRhZGF0YSI6eyJjcmVhdGVkX3RpbWUiOiIyMDI0LTA1LTA5VDA0OjUwOjIzLjgwNTk5MzU2MloiLCJjdXN0b21fbWV0YWRhdGEiOm51bGwsImRlbGV0aW9uX3RpbWUiOiIiLCJkZXN0cm95ZWQiOmZhbHNlLCJ2ZXJzaW9uIjoxfX0=
    SECRET_KEY1: YWFhYQ==
    SECRET_KEY2: YmJiYg==
  kind: Secret
  metadata:
    creationTimestamp: "2024-05-09T04:58:28Z"
    labels:
      app.kubernetes.io/component: secret-sync
      app.kubernetes.io/managed-by: hashicorp-vso
      app.kubernetes.io/name: vault-secrets-operator
      secrets.hashicorp.com/vso-ownerRefUID: 87c01285-0c1a-4011-bd93-b08bd26773f0
    name: app-secret
    namespace: app-ns
    ownerReferences:
    - apiVersion: secrets.hashicorp.com/v1beta1
      kind: VaultStaticSecret
      name: vaultdstaticsecret
      uid: 87c01285-0c1a-4011-bd93-b08bd26773f0
    resourceVersion: "1320"
    uid: a04650f5-62d2-4209-b6c4-c7795b5f4723
  type: Opaque
kind: List
metadata:
  resourceVersion: ""
ubuntu@ubuntu:~$ 
```
</details>

:::note info
Secretはbase64でエンコードされているため、デコードすることで、パラメータを確認できる。

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl get secret app-secret -n app-ns -o jsonpath='{.data.SECRET_KEY1}' |base64 -d
aaaa
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl get secret app-secret -n app-ns -o jsonpath='{.data.SECRET_KEY2}' |base64 -d
bbbb
ubuntu@ubuntu:~$ 
```
:::

## 5. Vault Secrets Operatorの挙動確認


### 5.1. VSOの構成

kube-rbac-proxyとmanagerの2つのコンテナで構成されていた。

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl describe pod -n vault-secrets-operator vault-secrets-operator-controller-manager-6954c7fb8c-ntv
2q
Name:             vault-secrets-operator-controller-manager-6954c7fb8c-ntv2q
Namespace:        vault-secrets-operator
Priority:         0
Service Account:  vault-secrets-operator-controller-manager
Node:             minikube/192.168.49.2
Start Time:       Thu, 09 May 2024 04:54:30 +0000
Labels:           app.kubernetes.io/instance=vault-secrets-operator
                  app.kubernetes.io/name=vault-secrets-operator
                  control-plane=controller-manager
                  pod-template-hash=6954c7fb8c
Annotations:      kubectl.kubernetes.io/default-container: manager
Status:           Running
IP:               10.244.0.5
IPs:
  IP:           10.244.0.5
Controlled By:  ReplicaSet/vault-secrets-operator-controller-manager-6954c7fb8c
Containers:
  kube-rbac-proxy:
    Container ID:  docker://1d71e3fb43e797359630ee253f3fba9606fa8f4b470f93def6a2b9af905d43b6
    Image:         gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0
    Image ID:      docker-pullable://gcr.io/kubebuilder/kube-rbac-proxy@sha256:d8cc6ffb98190e8dd403bfe67ddcb454e6127d32b87acc237b3e5240f70a20fb
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=0
    State:          Running
      Started:      Thu, 09 May 2024 04:54:42 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     5m
      memory:  64Mi
    Environment:
      KUBERNETES_CLUSTER_DOMAIN:  cluster.local
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rz5wg (ro)
  manager:
    Container ID:  docker://f204811b143a94bca95289dc939a09de89ab1715b1739f088692013635f1c7fd
    Image:         hashicorp/vault-secrets-operator:0.6.0
    Image ID:      docker-pullable://hashicorp/vault-secrets-operator@sha256:291286541d7c3052674021003b907c17e7019b9f499790b0859e1fb8442ffc93
    Port:          <none>
    Host Port:     <none>
    Command:
      /vault-secrets-operator
    Args:
      --health-probe-bind-address=:8081
      --metrics-bind-address=127.0.0.1:8080
      --leader-elect
    State:          Running
      Started:      Thu, 09 May 2024 04:54:54 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:      10m
      memory:   64Mi
    Liveness:   http-get http://:8081/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8081/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      OPERATOR_POD_NAME:          vault-secrets-operator-controller-manager-6954c7fb8c-ntv2q (v1:metadata.name)
      OPERATOR_POD_UID:            (v1:metadata.uid)
      KUBERNETES_CLUSTER_DOMAIN:  cluster.local
    Mounts:
      /var/run/podinfo from podinfo (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rz5wg (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.name -> name
      metadata.uid -> uid
  kube-api-access-rz5wg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  10m    default-scheduler  Successfully assigned vault-secrets-operator/vault-secrets-operator-controller-manager-6954c7fb8c-ntv2q to minikube
  Normal  Pulling    10m    kubelet            Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0"
  Normal  Pulled     10m    kubelet            Successfully pulled image "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0" in 10.54s (10.54s including waiting). Image size: 55859514 bytes.
  Normal  Created    10m    kubelet            Created container kube-rbac-proxy
  Normal  Started    10m    kubelet            Started container kube-rbac-proxy
  Normal  Pulling    10m    kubelet            Pulling image "hashicorp/vault-secrets-operator:0.6.0"
  Normal  Pulled     10m    kubelet            Successfully pulled image "hashicorp/vault-secrets-operator:0.6.0" in 10.893s (10.893s including waiting). Image size: 75826538 bytes.
  Normal  Created    9m59s  kubelet            Created container manager
  Normal  Started    9m59s  kubelet            Started container manager
ubuntu@ubuntu:~$
```
</details>


### 5.2. Vault Secret変更時のSecretの挙動

Vaultに登録しているSecretを変更したところ、Secretも少し待つと自動で更新された。

<details><summary>実行結果を展開する</summary>

```bash: 実行結果
/ $ vault kv get app-secret/kv-secret
====== Secret Path ======
app-secret/data/kv-secret

======= Metadata =======
Key                Value
---                -----
created_time       2024-05-09T04:50:23.805993562Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

======= Data =======
Key            Value
---            -----
SECRET_KEY1    aaaa
SECRET_KEY2    bbbb
/ $
/ $ vault kv put app-secret/kv-secret SECRET_KEY1="cccc" SECRET_KEY2="dddd"
====== Secret Path ======
app-secret/data/kv-secret

======= Metadata =======
Key                Value
---                -----
created_time       2024-05-09T05:03:04.669124279Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2
/ $ vault kv get app-secret/kv-secret
====== Secret Path ======
app-secret/data/kv-secret

======= Metadata =======
Key                Value
---                -----
created_time       2024-05-09T05:03:04.669124279Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

======= Data =======
Key            Value
---            -----
SECRET_KEY1    cccc
SECRET_KEY2    dddd
/ $ 
/ $ exit
ubuntu@ubuntu:~$ ## Secretを確認
ubuntu@ubuntu:~$ kubectl get secret app-secret -n app-ns -o jsonpath='{.data.SECRET_KEY1}' |base64 -d
aaaa
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl get secret app-secret -n app-ns -o jsonpath='{.data.SECRET_KEY1}' |base64 -d
aaaa
ubuntu@ubuntu:~$ kubectl get secret app-secret -n app-ns -o jsonpath='{.data.SECRET_KEY1}' |base64 -d
aaaa
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl get secret app-secret -n app-ns -o jsonpath='{.data.SECRET_KEY1}' |base64 -d
cccc
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ 
```
</details>

Secretが更新されたタイミングでVSOのログに下記のログが出力していた。

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl logs -n vault-secrets-operator vault-secrets-operator-controller-manager-6954c7fb8c-ntv2q -f
2024-05-09T05:03:31Z    DEBUG   events  Secret synced   {"type": "Normal", "object": {"kind":"VaultStaticSecret","namespace":"app-ns","name":"vaultdstaticsecret","uid":"87c01285-0c1a-4011-bd93-b08bd26773f0","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"1325"}, "reason": "SecretRotated"}
```
## 6. 環境削除

動作確認に使用したリソースを削除する。

```bash: 実行結果
ubuntu@ubuntu:~$ kubectl delete VaultStaticSecret -n app-ns vaultstaticsecret
vaultstaticsecret.secrets.hashicorp.com "vaultstaticsecret" deleted
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl delete -n app-ns VaultAuth vaultauth
vaultauth.secrets.hashicorp.com "vaultauth" deleted
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl delete -n app-ns  VaultConnection vaultconnection
vaultconnection.secrets.hashicorp.com "vaultconnection" deleted
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ helm uninstall vault-secrets-operator -n vault-secrets-operator
release "vault-secrets-operator" uninstalled
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ helm uninstall vault -n vault
release "vault" uninstalled
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ helm repo remove hashicorp
"hashicorp" has been removed from your repositories
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl delete sa app-sa -n app-ns
serviceaccount "app-sa" deleted
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl delete ns app-ns
namespace "app-ns" deleted
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl delete ns vault-secrets-operator
namespace "vault-secrets-operator" deleted
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ kubectl delete ns vault
namespace "vault" deleted
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ minikube delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /home/ubuntu/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.
ubuntu@ubuntu:~$ 
ubuntu@ubuntu:~$ unalias kubectl
ubuntu@ubuntu:~$ 
```