OpenShift Podのディスク使用量をノードに入って確認する

# はじめに
ROSA（Red Hat OpenShift Service on AWS）の構成を検討する中で、デフォルトで構成されたPV未使用のPrometheusのPodがどれぐらいInfra/Workerノード上のディスクを消費しているのかが気になり確認した際の内容を記載する。
本記事に記載する内容は特にPrometheusに特化した話ではなく、OpenShiftのPodがWorkerノード上のディスクをどの程度使用しているかを確認する方法についての話となる。


# 環境
- ROSA ver.4.10
- STS＆PrivateLinkモード
- Workerノード数：3台
- 複数台のテスト用アプリPodが稼働する環境


# 確認手順
手順の概要は以下のとおり。
1. 確認対象のPodが稼働するWorkerノードの特定
2. Infra/Workerノードに`oc debug node`でアクセス
3. `crictl`コマンドでPodの情報の確認
4. `du`コマンドでディスク使用量の確認

※Podに直接アクセスして確認する方法もあるが、、ここではあえて上記のようにノード上で確認する方法をとってみた。

## 確認対象のPodが稼働するWorkerノードの特定

oc get pod -o wideコマンドにて、OpenShift-Monitoring Project上のPodを確認する。

```
$ oc get pod -n openshift-monitoring -o wide
NAME                                               READY   STATUS      RESTARTS   AGE     IP             NODE
                NOMINATED NODE   READINESS GATES
alertmanager-main-0                                6/6     Running     0          62d     10.128.4.7     ip-10-0-135-107.ec2.internal   <none>           <none>
alertmanager-main-1                                6/6     Running     0          62d     10.131.2.8     ip-10-0-182-241.ec2.internal   <none>           <none>
cluster-monitoring-operator-99fb67bf4-784lp        2/2     Running     0          62d     10.128.0.39    ip-10-0-221-180.ec2.internal   <none>           <none>
configure-alertmanager-operator-56ff578d77-sktsq   1/1     Running     0          9d      10.131.2.88    ip-10-0-182-241.ec2.internal   <none>           <none>
configure-alertmanager-operator-registry-6knhb     1/1     Running     0          56d     10.131.1.67    ip-10-0-215-48.ec2.internal    <none>           <none>
grafana-65dc6dd678-r89dp                           3/3     Running     0          62d     10.128.4.9     ip-10-0-135-107.ec2.internal   <none>           <none>
kube-state-metrics-66bf46c69d-lfxkb                3/3     Running     0          62d     10.128.4.20    ip-10-0-135-107.ec2.internal   <none>           <none>
node-exporter-45hsx                                2/2     Running     2          62d     10.0.182.241   ip-10-0-182-241.ec2.internal   <none>           <none>
node-exporter-5hz6v                                2/2     Running     2          62d     10.0.190.186   ip-10-0-190-186.ec2.internal   <none>           <none>
node-exporter-864s2                                2/2     Running     2          62d     10.0.153.196   ip-10-0-153-196.ec2.internal   <none>           <none>
node-exporter-bmxkn                                2/2     Running     2          62d     10.0.221.180   ip-10-0-221-180.ec2.internal   <none>           <none>
node-exporter-bq7n5                                2/2     Running     2          62d     10.0.135.140   ip-10-0-135-140.ec2.internal   <none>           <none>
node-exporter-jr47c                                2/2     Running     2          62d     10.0.135.107   ip-10-0-135-107.ec2.internal   <none>           <none>
node-exporter-p5xwt                                2/2     Running     2          62d     10.0.199.127   ip-10-0-199-127.ec2.internal   <none>           <none>
node-exporter-szrfv                                2/2     Running     0          49d     10.0.171.183   ip-10-0-171-183.ec2.internal   <none>           <none>
node-exporter-x6g8w                                2/2     Running     2          62d     10.0.215.48    ip-10-0-215-48.ec2.internal    <none>           <none>
openshift-state-metrics-5d5454f44d-mbk8w           3/3     Running     0          62d     10.128.4.10    ip-10-0-135-107.ec2.internal   <none>           <none>
osd-rebalance-infra-nodes-27828240-rnjjn           0/1     Completed   0          40m     10.128.4.136   ip-10-0-135-107.ec2.internal   <none>           <none>
osd-rebalance-infra-nodes-27828255-hrxd9           0/1     Completed   0          25m     10.128.4.140   ip-10-0-135-107.ec2.internal   <none>           <none>
osd-rebalance-infra-nodes-27828270-wfw5s           0/1     Completed   0          10m     10.128.4.143   ip-10-0-135-107.ec2.internal   <none>           <none>
prometheus-adapter-7bd5fcb654-h6x2h                1/1     Running     0          2d14h   10.128.4.205   ip-10-0-135-107.ec2.internal   <none>           <none>
prometheus-adapter-7bd5fcb654-j5pqs                1/1     Running     0          2d14h   10.130.2.38    ip-10-0-199-127.ec2.internal   <none>           <none>
prometheus-k8s-0                                   6/6     Running     0          62d     10.131.2.7     ip-10-0-182-241.ec2.internal   <none>           <none>
prometheus-k8s-1                                   6/6     Running     0          62d     10.130.2.7     ip-10-0-199-127.ec2.internal   <none>           <none>
prometheus-operator-68d866488f-pf7dw               2/2     Running     0          62d     10.128.4.11    ip-10-0-135-107.ec2.internal   <none>           <none>
sre-dns-latency-exporter-6qr68                     1/1     Running     3          139d    10.131.2.5     ip-10-0-182-241.ec2.internal   <none>           <none>
sre-dns-latency-exporter-762sp                     1/1     Running     0          49d     10.129.4.5     ip-10-0-171-183.ec2.internal   <none>           <none>
sre-dns-latency-exporter-8qvvs                     1/1     Running     2          139d    10.129.0.9     ip-10-0-153-196.ec2.internal   <none>           <none>
sre-dns-latency-exporter-cmmr5                     1/1     Running     2          139d    10.128.4.5     ip-10-0-135-107.ec2.internal   <none>           <none>
sre-dns-latency-exporter-f7n8c                     1/1     Running     1          139d    10.130.2.4     ip-10-0-199-127.ec2.internal   <none>           <none>
sre-dns-latency-exporter-fqg68                     1/1     Running     2          139d    10.128.2.2     ip-10-0-135-140.ec2.internal   <none>           <none>
sre-dns-latency-exporter-htjnf                     1/1     Running     2          139d    10.131.0.5     ip-10-0-215-48.ec2.internal    <none>           <none>
sre-dns-latency-exporter-kflb7                     1/1     Running     2          139d    10.128.0.2     ip-10-0-221-180.ec2.internal   <none>           <none>
sre-dns-latency-exporter-phqxg                     1/1     Running     2          139d    10.130.0.12    ip-10-0-190-186.ec2.internal   <none>           <none>
telemeter-client-774cd546df-q6j4b                  3/3     Running     0          62d     10.128.4.19    ip-10-0-135-107.ec2.internal   <none>           <none>
thanos-querier-55dd66b46-fvclz                     6/6     Running     0          62d     10.128.4.17    ip-10-0-135-107.ec2.internal   <none>           <none>
thanos-querier-55dd66b46-gkpsr                     6/6     Running     0          62d     10.130.2.17    ip-10-0-199-127.ec2.internal   <none>           <none>
token-refresher-7f8c567555-qwwn2                   1/1     Running     0          62d     10.130.2.18    ip-10-0-199-127.ec2.internal   <none>           <none>
```

Pod数が多いのでPrometheusだけgrepする。

```
$ oc get pod -n openshift-monitoring -o wide | grep prome
prometheus-adapter-7bd5fcb654-h6x2h                1/1     Running     0          2d14h   10.128.4.205   ip-10-0-135-107.ec2.internal   <none>           <none>
prometheus-adapter-7bd5fcb654-j5pqs                1/1     Running     0          2d14h   10.130.2.38    ip-10-0-199-127.ec2.internal   <none>           <none>
prometheus-k8s-0                                   6/6     Running     0          62d     10.131.2.7     ip-10-0-182-241.ec2.internal   <none>           <none>
prometheus-k8s-1                                   6/6     Running     0          62d     10.130.2.7     ip-10-0-199-127.ec2.internal   <none>           <none>
prometheus-operator-68d866488f-pf7dw               2/2     Running     0          62d     10.128.4.11    ip-10-0-135-107.ec2.internal   <none>           <none>
```

→prometheusは以下のノード上で稼働している。
- prometheus-k8s-0: ip-10-0-182-241.ec2.internal
- prometheus-k8s-1: ip-10-0-199-127.ec2.internal
※今回はprometheusのpodを確認対象とした。

（参考）
上記ノードはRoleとしてはInfra/Workerラベルが付与されている。

```
$ oc get no -o wide
（infra,workerロールのノードのみ抜粋）
NAME                           STATUS   ROLES          AGE    VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                        KERNEL-VERSION                 CONTAINER-RUNTIME
ip-10-0-135-107.ec2.internal   Ready    infra,worker   141d   v1.23.5+012e945   10.0.135.107   <none>        Red Hat Enterprise Linux CoreOS 410.84.202208161501-0 (Ootpa)   4.18.0-305.57.1.el8_4.x86_64   cri-o://1.23.3-15.rhaos4.10.git6af791c.el8
ip-10-0-182-241.ec2.internal   Ready    infra,worker   141d   v1.23.5+012e945   10.0.182.241   <none>        Red Hat Enterprise Linux CoreOS 410.84.202208161501-0 (Ootpa)   4.18.0-305.57.1.el8_4.x86_64   cri-o://1.23.3-15.rhaos4.10.git6af791c.el8ip-
ip-10-0-199-127.ec2.internal   Ready    infra,worker   141d   v1.23.5+012e945   10.0.199.127   <none>        Red Hat Enterprise Linux CoreOS 410.84.202208161501-0 (Ootpa)   4.18.0-305.57.1.el8_4.x86_64   cri-o://1.23.3-15.rhaos4.10.git6af791c.el8
```



## Infra/Workerノードに`oc debug node`でアクセス

prometheus podが稼働するWorkerノードに`oc debug node`コマンドでアクセスする。

```
$ oc debug node/ip-10-0-182-241.ec2.internal
Starting pod/ip-10-0-182-241ec2internal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.182.241
If you don't see a command prompt, try pressing enter.
sh-4.4#
sh-4.4# chroot /host
sh-4.4#
```
コマンド実行するため、chroot /hostを実行しておく。

## `crictl`コマンドでPodの情報の確認

稼働するpodの情報を確認するため、`crictl`コマンドを実行する。

```
sh-4.4# crictl ps
CONTAINER           IMAGE                                                                                                                     CREATED             STATE               NAME                                 ATTEMPT             POD ID
250070a9b431a       a1ce65e6ae985f352a5e339a1392420ac4279eef182d2e20c191b3ba84d04fc7                                                          3 minutes ago       Running             container-00                         0                   88e93156d908a
13a02acb6bb30       quay.io/app-sre/splunk-forwarder@sha256:afc413cd2504b586b6f28c16355db243c8bfdef536cc80e44f61e03c01e12235                  21 hours ago        Running             splunk-uf                            0                   9fe134fcbf6d6
531afe579fa6a       dcfb77e8999b71b39fa062d4f274a0784943a017991cf5d5a4058c08ed9f7554                                                          8 days ago          Running             webhook                              0                   a4c9e48f9316c
535cb9dadc7ae       quay.io/app-sre/addon-operator-manager@sha256:41a7cf65835e4229e4468593a64ab0cdc4dac5a265aab88a379216c5eef688d6            8 days ago          Running             manager                              0                   0b4637b4ee99f
439339a0a779c       19fe8540f8737eec13b6ef65a0acfe4135415299551fed824afaefcb3067f7ad                                                          8 days ago          Running             metrics-relay-server                 0                   0b4637b4ee99f
626378edf135d       quay.io/app-sre/configure-alertmanager-operator@sha256:6c86f39326fb76657d4ed64a5bc0470374574be30507cb67d3ffb1ac78e6f851   9 days ago          Running             configure-alertmanager-operator      0                   13fcca973bbe5
02c392154d4fb       quay.io/app-sre/ocm-agent@sha256:51bafd25d611662abd8937ff3914b9a4ae5ef9361719e38947a56fd38199014d                         12 days ago         Running             ocm-agent                            0                   78d512939824c
370947ae91b06       quay.io/app-sre/ocm-agent-operator@sha256:d697996b5051ee1fdf174a7d23fdd0f0fe612876e8b0d8e94a0c92fee72ea011                13 days ago         Running             ocm-agent-operator                   0                   d4e6016222c39
fbd22d001eef5       quay.io/app-sre/custom-domains-operator@sha256:59158acb6a90927f89d85ffaf448e4831b8e8b352d4539b597b97e5f5ff92f6b           4 weeks ago         Running             custom-domains-operator              0                   c4f5c43d5d5eb
9208b357e7965       quay.io/app-sre/must-gather-operator@sha256:2e9c61f5bb3d7f95805130843b0368853d38934756e892954cdf3e251941ac43              6 weeks ago         Running             must-gather-operator                 0                   27b5ebcdd77e4
54c912fdc02c5       quay.io/app-sre/managed-node-metadata-operator@sha256:81cb5d1fcc8261858727876036c86fcf45b5b3cf9f5129450019c6f107cd11da    7 weeks ago         Running             managed-node-metadata-operator       0                   aa02b2b6f8f3e
4d971e953ad86       410e15456e199925ac531d55aa982db94829abd83b825a1f3c8ef8ad367ec3a8                                                          2 months ago        Running             prom-label-proxy                     0                   32e6aba11b9b3
340cf915b365c       b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa                                                          2 months ago        Running             kube-rbac-proxy-metric               0                   32e6aba11b9b3
6fb5f39097c94       b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa                                                          2 months ago        Running             kube-rbac-proxy                      0                   32e6aba11b9b3
c4ab5a439405d       4a35485847278231eaefeeba227df3b848fcd88251bf387cc75814683c6e1cc9                                                          2 months ago        Running             alertmanager-proxy                   0                   32e6aba11b9b3
44aba6706a156       d7e69e3e6c77da1eb69a016accb5755bd259cdf6b08bbe2505482bdaf191c1a7                                                          2 months ago        Running             config-reloader                      0                   32e6aba11b9b3
43a4c4e9f1aa1       5b7a0f5af01de4e8ccd7d80557c92f25452b8ae54423ebbd93cd6b8cc11bf086                                                          2 months ago        Running             alertmanager                         0                   32e6aba11b9b3
0a26c550f6b59       b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa                                                          2 months ago        Running             kube-rbac-proxy-thanos               0                   fde2304412307
bc28291be733a       b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa                                                          2 months ago        Running             kube-rbac-proxy                      0                   fde2304412307
9d94467cb5722       4a35485847278231eaefeeba227df3b848fcd88251bf387cc75814683c6e1cc9                                                          2 months ago        Running             prometheus-proxy                     0                   fde2304412307
4359fb8b597b6       1cff5886ffcf498a37eb3868762403fb2b0ca2feb4a14c98471fc256b680619e                                                          2 months ago        Running             thanos-sidecar                       0                   fde2304412307
01b339ba5ea1d       d7e69e3e6c77da1eb69a016accb5755bd259cdf6b08bbe2505482bdaf191c1a7                                                          2 months ago        Running             config-reloader                      0                   fde2304412307
27cccaa137e39       d3017e38d93fb39bcad5ae9ee8521f5829c774c389ae7e65e44a00388d67cf1e                                                          2 months ago        Running             prometheus                           0                   fde2304412307
9789aad219d40       b08d2cd9618b330d905aeafe0bb8187a768fbe94c17fe9916d088dbf59acefc3                                                          2 months ago        Running             kube-multus-additional-cni-plugins   1                   923c608753588
d05f57994cae1       46bea5c4f6e46d4112dbbf837186ba68c231c9edd3bb22ccfbf76ad32a329ffd                                                          2 months ago        Running             main                                 3                   6beefa759e932
86c189c7529f6       53ba5113526775eab777eed2bfdd9d6148ae563a3330f609394dcca565283f7b                                                          2 months ago        Running             serve-healthcheck-canary             1                   989fc22ba34bb
fd0a17b4d8ecb       77a2af086aeb1c31914686118f3f20e44f836a251d89f256c5620cda914445a9                                                          2 months ago        Running             network-check-target-container       1                   0139053cefa0d
5ed1426014c35       b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa                                                          2 months ago        Running             kube-rbac-proxy                      1                   f57ba7ed09785
1032c983cbbe0       3e0c06861c106b17273bd4c553ca4be6f34ca3cc226f89b2092aa9d7062a735d                                                          2 months ago        Running             network-metrics-daemon               1                   f57ba7ed09785
c8dc4f438ce2e       b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa                                                          2 months ago        Running             kube-rbac-proxy                      1                   f74f3801db28d
f6453301898b4       fa34f535a918d4d6d07e8b105b9209dfd8b6c9152cee340e10f55934d6b07690                                                          2 months ago        Running             node-exporter                        1                   f74f3801db28d
56010ff9bb875       75ae6c537332567afdc034809cf030f0cfea3cb7ecbc2557be821406abf5adb4                                                          2 months ago        Running             csi-liveness-probe                   1                   fc7868d48269e
360e48e827df6       b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa                                                          2 months ago        Running             kube-rbac-proxy                      1                   aa3699225a6dd
61b3fe4869316       4a35485847278231eaefeeba227df3b848fcd88251bf387cc75814683c6e1cc9                                                          2 months ago        Running             oauth-proxy                          1                   6c0bdff805442
5ca4224aa30f6       97448add286660fbbf4e14abb55c4a728c07162727cf7f7753c014c6540ba9b7                                                          2 months ago        Running             csi-node-driver-registrar            1                   fc7868d48269e
4cc36ee09e831       f44d6c1a2dad7deef542a030cce029d3904da2d69a2a7203094e8e7a15d30883                                                          2 months ago        Running             dns-node-resolver                    1                   3a86f4d95389b
7d70ca3301053       bded42f93595dfe68ce77cac9a401479c0e60180b82947576276e95319fd185c                                                          2 months ago        Running             sdn                                  1                   aa3699225a6dd
8a8fb4dd0b34f       9e085bf5fd76546272ae0de97e127cceb9fa19518128928f4ea1ff19794a420d                                                          2 months ago        Running             csi-driver                           1                   fc7868d48269e
b46475b65fe1b       3e34a83ad92bf15167ec7b450520a61bb5028985ac19ce0bf16aa89d84b58c0f                                                          2 months ago        Running             machine-config-daemon                1                   6c0bdff805442
ab100fa37c49f       8cc9e7e28492bd39a5a0efde92b8a11cbbeb7066e907cb25dab5d7d1ad346477                                                          2 months ago        Running             node-ca                              1                   dfb3974a993f5
8a7b1ea93c0c3       b08d2cd9618b330d905aeafe0bb8187a768fbe94c17fe9916d088dbf59acefc3                                                          2 months ago        Running             kube-multus                          1                   628ede44e698b
3918dd5445192       57d6bfa9aa0faa2b850635e158e56230035e1910361d5412bb939ec7fe409527                                                          2 months ago        Running             tuned                                1                   04b08f03e5865
sh-4.4#
```

pod数が多いので、grep。

```
sh-4.4# crictl ps | grep prometheus
9d94467cb5722       4a35485847278231eaefeeba227df3b848fcd88251bf387cc75814683c6e1cc9                                                          2 months ago        Running             prometheus-proxy                     0                   fde2304412307
27cccaa137e39       d3017e38d93fb39bcad5ae9ee8521f5829c774c389ae7e65e44a00388d67cf1e                                                          2 months ago        Running             prometheus                           0                   fde2304412307
sh-4.4#
```

prometheusのpodの詳細情報を確認するため、`crictl inspect`を実行する。
（長いですが、全量載せておきます）

```
sh-4.4# crictl inspect 27cccaa137e39
{
  "status": {
    "id": "27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084",
    "metadata": {
      "attempt": 0,
      "name": "prometheus"
    },
    "state": "CONTAINER_RUNNING",
    "createdAt": "2022-09-27T08:46:55.100751755Z",
    "startedAt": "2022-09-27T08:46:55.126901515Z",
    "finishedAt": "0001-01-01T00:00:00Z",
    "exitCode": 0,
    "image": {
      "annotations": {},
      "image": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00d156818e2e4264c3268c9ae41de9b561e2811363b64515ca64865c992ec8ea"
    },
    "imageRef": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00d156818e2e4264c3268c9ae41de9b561e2811363b64515ca64865c992ec8ea",
    "reason": "",
    "message": "",
    "labels": {
      "io.kubernetes.container.name": "prometheus",
      "io.kubernetes.pod.name": "prometheus-k8s-0",
      "io.kubernetes.pod.namespace": "openshift-monitoring",
      "io.kubernetes.pod.uid": "a4477163-737d-457c-8698-8688fdd00003"
    },
    "annotations": {
      "io.kubernetes.container.hash": "7ba9b53",
      "io.kubernetes.container.restartCount": "0",
      "io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
      "io.kubernetes.container.terminationMessagePolicy": "FallbackToLogsOnError",
      "io.kubernetes.pod.terminationGracePeriod": "600"
    },
    "mounts": [
      {
        "containerPath": "/prometheus",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/pvc-b17ae945-02d2-43e2-a2d2-23891203b1f3/prometheus/3",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": false,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/dev/termination-log",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers/prometheus/5ab56fc4",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": false,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/hosts",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": false,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/config_out",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~empty-dir/config-out",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/certs",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/tls-assets",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/secrets/prometheus-k8s-thanos-sidecar-tls",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/secrets/kube-etcd-client-certs",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-etcd-client-certs",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/secrets/prometheus-k8s-tls",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/secrets/prometheus-k8s-proxy",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-proxy",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/web_config/web-config.yaml",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/web-config/prometheus/5",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/secrets/kube-rbac-proxy",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-rbac-proxy",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/secrets/metrics-client-certs",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-metrics-client-certs",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/configmaps/serving-certs-ca-bundle",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-serving-certs-ca-bundle",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/configmaps/kubelet-serving-ca-bundle",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-kubelet-serving-ca-bundle",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/configmaps/metrics-client-ca",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-metrics-client-ca",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-k8s-rulefiles-0",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/var/run/secrets/kubernetes.io/serviceaccount",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/kube-api-access-jfzqj",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      },
      {
        "containerPath": "/etc/pki/ca-trust/extracted/pem/",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-trusted-ca-bundle",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": true
      }
    ],
    "logPath": "/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4477163-737d-457c-8698-8688fdd00003/prometheus/0.log"
  },
  "info": {
    "sandboxID": "fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d",
    "pid": 4693,
    "runtimeSpec": {
      "ociVersion": "1.0.2-dev",
      "process": {
        "user": {
          "uid": 65534,
          "gid": 65534,
          "additionalGids": [
            65534
          ]
        },
        "args": [
          "/bin/prometheus",
          "--web.console.templates=/etc/prometheus/consoles",
          "--web.console.libraries=/etc/prometheus/console_libraries",
          "--config.file=/etc/prometheus/config_out/prometheus.env.yaml",
          "--storage.tsdb.path=/prometheus",
          "--storage.tsdb.retention.time=11d",
          "--web.enable-lifecycle",
          "--web.external-url=https://prometheus-k8s-openshift-monitoring.apps.aws-rosa-xxx-xxx.xxx.xxx.openshiftapps.com/",
          "--web.route-prefix=/",
          "--web.listen-address=127.0.0.1:9090",
          "--web.config.file=/etc/prometheus/web_config/web-config.yaml"
        ],
        "env": [
          "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
          "TERM=xterm",
          "HOSTNAME=prometheus-k8s-0",
          "NSS_SDB_USE_CACHE=no",
          "GRAFANA_PORT_3000_TCP_ADDR=172.30.251.5",
          "GRAFANA_PORT_3002_TCP_ADDR=172.30.251.5",
          "THANOS_QUERIER_PORT_9093_TCP_PROTO=tcp",
          "TOKEN_REFRESHER_SERVICE_PORT_HTTP=80",
          "KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1",
          "CONFIGURE_ALERTMANAGER_OPERATOR_SERVICE_PORT=8080",
          "CONFIGURE_ALERTMANAGER_OPERATOR_PORT_8080_TCP_PROTO=tcp",
          "ALERTMANAGER_MAIN_SERVICE_HOST=172.30.199.91",
          "TOKEN_REFRESHER_SERVICE_PORT=80",
          "KUBERNETES_SERVICE_PORT_HTTPS=443",
          "PROMETHEUS_K8S_SERVICE_PORT_WEB=9091",
          "THANOS_QUERIER_PORT_9093_TCP_PORT=9093",
          "PROMETHEUS_K8S_PORT_9091_TCP=tcp://172.30.214.226:9091",
          "ALERTMANAGER_MAIN_PORT_9092_TCP_PROTO=tcp",
          "GRAFANA_PORT_3000_TCP_PROTO=tcp",
          "GRAFANA_PORT_3002_TCP_PORT=3002",
          "PROMETHEUS_K8S_PORT=tcp://172.30.214.226:9091",
          "THANOS_QUERIER_PORT_9093_TCP_ADDR=172.30.64.108",
          "TOKEN_REFRESHER_PORT_80_TCP=tcp://172.30.227.107:80",
          "ALERTMANAGER_MAIN_PORT_9097_TCP_ADDR=172.30.199.91",
          "THANOS_QUERIER_PORT_9094_TCP_PROTO=tcp",
          "CONFIGURE_ALERTMANAGER_OPERATOR_PORT_8080_TCP=tcp://172.30.32.44:8080",
          "GRAFANA_PORT=tcp://172.30.251.5:3000",
          "SRE_DNS_LATENCY_EXPORTER_PORT_80_TCP=tcp://172.30.68.79:80",
          "PROMETHEUS_K8S_PORT_9092_TCP_ADDR=172.30.214.226",
          "CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT_50051_TCP_PORT=50051",
          "ALERTMANAGER_MAIN_PORT_9097_TCP_PORT=9097",
          "KUBERNETES_PORT_443_TCP_PORT=443",
          "PROMETHEUS_K8S_SERVICE_PORT_METRICS=9092",
          "THANOS_QUERIER_PORT_9091_TCP_PORT=9091",
          "THANOS_QUERIER_PORT_9092_TCP=tcp://172.30.64.108:9092",
          "THANOS_QUERIER_PORT_9094_TCP_PORT=9094",
          "ALERTMANAGER_MAIN_PORT_9094_TCP_PROTO=tcp",
          "THANOS_QUERIER_SERVICE_PORT_WEB=9091",
          "CONFIGURE_ALERTMANAGER_OPERATOR_PORT=tcp://172.30.32.44:8080",
          "PROMETHEUS_K8S_PORT_9091_TCP_ADDR=172.30.214.226",
          "ALERTMANAGER_MAIN_PORT_9094_TCP=tcp://172.30.199.91:9094",
          "ALERTMANAGER_MAIN_PORT_9092_TCP_ADDR=172.30.199.91",
          "PROMETHEUS_ADAPTER_PORT_443_TCP=tcp://172.30.48.128:443",
          "GRAFANA_PORT_3000_TCP_PORT=3000",
          "GRAFANA_PORT_3000_TCP=tcp://172.30.251.5:3000",
          "CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_SERVICE_PORT_GRPC=50051",
          "CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT_50051_TCP_PROTO=tcp",
          "ALERTMANAGER_MAIN_SERVICE_PORT=9094",
          "PROMETHEUS_ADAPTER_SERVICE_PORT=443",
          "SRE_DNS_LATENCY_EXPORTER_PORT_80_TCP_ADDR=172.30.68.79",
          "THANOS_QUERIER_SERVICE_PORT_TENANCY=9092",
          "THANOS_QUERIER_PORT=tcp://172.30.64.108:9091",
          "THANOS_QUERIER_PORT_9091_TCP=tcp://172.30.64.108:9091",
          "PROMETHEUS_ADAPTER_PORT_443_TCP_PORT=443",
          "THANOS_QUERIER_PORT_9091_TCP_ADDR=172.30.64.108",
          "THANOS_QUERIER_PORT_9092_TCP_PROTO=tcp",
          "CONFIGURE_ALERTMANAGER_OPERATOR_PORT_8080_TCP_PORT=8080",
          "ALERTMANAGER_MAIN_PORT_9094_TCP_ADDR=172.30.199.91",
          "TOKEN_REFRESHER_PORT=tcp://172.30.227.107:80",
          "GRAFANA_SERVICE_PORT_HTTPS=3000",
          "SRE_DNS_LATENCY_EXPORTER_SERVICE_HOST=172.30.68.79",
          "PROMETHEUS_K8S_PORT_9091_TCP_PORT=9091",
          "THANOS_QUERIER_PORT_9094_TCP_ADDR=172.30.64.108",
          "PROMETHEUS_ADAPTER_SERVICE_HOST=172.30.48.128",
          "THANOS_QUERIER_SERVICE_PORT=9091",
          "THANOS_QUERIER_PORT_9092_TCP_ADDR=172.30.64.108",
          "THANOS_QUERIER_PORT_9093_TCP=tcp://172.30.64.108:9093",
          "TOKEN_REFRESHER_PORT_80_TCP_PORT=80",
          "GRAFANA_PORT_3002_TCP_PROTO=tcp",
          "PROMETHEUS_K8S_PORT_9092_TCP_PROTO=tcp",
          "CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT_50051_TCP_ADDR=172.30.187.125",
          "ALERTMANAGER_MAIN_SERVICE_PORT_WEB=9094",
          "CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT=tcp://172.30.187.125:50051",
          "ALERTMANAGER_MAIN_PORT_9097_TCP=tcp://172.30.199.91:9097",
          "KUBERNETES_SERVICE_PORT=443",
          "GRAFANA_SERVICE_PORT=3000",
          "SRE_DNS_LATENCY_EXPORTER_SERVICE_PORT_HTTP_MAIN=80",
          "PROMETHEUS_K8S_PORT_9092_TCP=tcp://172.30.214.226:9092",
          "PROMETHEUS_K8S_PORT_9091_TCP_PROTO=tcp",
          "ALERTMANAGER_MAIN_SERVICE_PORT_METRICS=9097",
          "ALERTMANAGER_MAIN_PORT_9097_TCP_PROTO=tcp",
          "THANOS_QUERIER_SERVICE_PORT_METRICS=9094",
          "CONFIGURE_ALERTMANAGER_OPERATOR_SERVICE_PORT_METRICS=8080",
          "GRAFANA_SERVICE_HOST=172.30.251.5",
          "GRAFANA_SERVICE_PORT_METRICS=3002",
          "PROMETHEUS_K8S_SERVICE_PORT=9091",
          "CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_SERVICE_PORT=50051",
          "THANOS_QUERIER_SERVICE_HOST=172.30.64.108",
          "CONFIGURE_ALERTMANAGER_OPERATOR_SERVICE_HOST=172.30.32.44",
          "GRAFANA_PORT_3002_TCP=tcp://172.30.251.5:3002",
          "SRE_DNS_LATENCY_EXPORTER_SERVICE_PORT=80",
          "ALERTMANAGER_MAIN_PORT_9092_TCP=tcp://172.30.199.91:9092",
          "ALERTMANAGER_MAIN_PORT_9092_TCP_PORT=9092",
          "THANOS_QUERIER_PORT_9092_TCP_PORT=9092",
          "CONFIGURE_ALERTMANAGER_OPERATOR_PORT_8080_TCP_ADDR=172.30.32.44",
          "THANOS_QUERIER_SERVICE_PORT_TENANCY_RULES=9093",
          "TOKEN_REFRESHER_SERVICE_HOST=172.30.227.107",
          "SRE_DNS_LATENCY_EXPORTER_PORT_80_TCP_PORT=80",
          "PROMETHEUS_ADAPTER_SERVICE_PORT_HTTPS=443",
          "THANOS_QUERIER_PORT_9094_TCP=tcp://172.30.64.108:9094",
          "TOKEN_REFRESHER_PORT_80_TCP_PROTO=tcp",
          "SRE_DNS_LATENCY_EXPORTER_PORT_80_TCP_PROTO=tcp",
          "ALERTMANAGER_MAIN_SERVICE_PORT_TENANCY=9092",
          "ALERTMANAGER_MAIN_PORT_9094_TCP_PORT=9094",
          "PROMETHEUS_ADAPTER_PORT_443_TCP_ADDR=172.30.48.128",
          "THANOS_QUERIER_PORT_9091_TCP_PROTO=tcp",
          "KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443",
          "KUBERNETES_PORT_443_TCP_PROTO=tcp",
          "ALERTMANAGER_MAIN_PORT=tcp://172.30.199.91:9094",
          "KUBERNETES_SERVICE_HOST=172.30.0.1",
          "KUBERNETES_PORT=tcp://172.30.0.1:443",
          "CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT_50051_TCP=tcp://172.30.187.125:50051",
          "PROMETHEUS_ADAPTER_PORT=tcp://172.30.48.128:443",
          "PROMETHEUS_ADAPTER_PORT_443_TCP_PROTO=tcp",
          "CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_SERVICE_HOST=172.30.187.125",
          "TOKEN_REFRESHER_PORT_80_TCP_ADDR=172.30.227.107",
          "SRE_DNS_LATENCY_EXPORTER_PORT=tcp://172.30.68.79:80",
          "PROMETHEUS_K8S_SERVICE_HOST=172.30.214.226",
          "PROMETHEUS_K8S_PORT_9092_TCP_PORT=9092",
          "__doozer=merge",
          "BUILD_RELEASE=202208150436.p0.g3c6cd55.assembly.stream",
          "BUILD_VERSION=v4.10.0",
          "OS_GIT_MAJOR=4",
          "OS_GIT_MINOR=10",
          "OS_GIT_PATCH=0",
          "OS_GIT_TREE_STATE=clean",
          "OS_GIT_VERSION=4.10.0-202208150436.p0.g3c6cd55.assembly.stream-3c6cd55",
          "SOURCE_GIT_TREE_STATE=clean",
          "OS_GIT_COMMIT=3c6cd55",
          "SOURCE_DATE_EPOCH=1640014589",
          "SOURCE_GIT_COMMIT=3c6cd55e3a5e5332e810274ad27fddf662d4f506",
          "SOURCE_GIT_TAG=golang-github-prometheus-prometheus-4.0.0-0.102.0-4305-g3c6cd55e3",
          "SOURCE_GIT_URL=https://github.com/openshift/prometheus",
          "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
          "container=oci"
        ],
        "cwd": "/prometheus",
        "capabilities": {
          "bounding": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FSETID",
            "CAP_FOWNER",
            "CAP_NET_BIND_SERVICE",
            "CAP_SETPCAP"
          ],
          "effective": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FSETID",
            "CAP_FOWNER",
            "CAP_NET_BIND_SERVICE",
            "CAP_SETPCAP"
          ],
          "permitted": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FSETID",
            "CAP_FOWNER",
            "CAP_NET_BIND_SERVICE",
            "CAP_SETPCAP"
          ]
        },
        "oomScoreAdj": 968,
        "selinuxLabel": "system_u:system_r:container_t:s0:c21,c0"
      },
      "root": {
        "path": "/var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged"
      },
      "hostname": "prometheus-k8s-0",
      "mounts": [
        {
          "destination": "/proc",
          "type": "proc",
          "source": "proc",
          "options": [
            "nosuid",
            "noexec",
            "nodev"
          ]
        },
        {
          "destination": "/dev",
          "type": "tmpfs",
          "source": "tmpfs",
          "options": [
            "nosuid",
            "strictatime",
            "mode=755",
            "size=65536k"
          ]
        },
        {
          "destination": "/dev/pts",
          "type": "devpts",
          "source": "devpts",
          "options": [
            "nosuid",
            "noexec",
            "newinstance",
            "ptmxmode=0666",
            "mode=0620",
            "gid=5"
          ]
        },
        {
          "destination": "/dev/mqueue",
          "type": "mqueue",
          "source": "mqueue",
          "options": [
            "nosuid",
            "noexec",
            "nodev"
          ]
        },
        {
          "destination": "/sys",
          "type": "sysfs",
          "source": "sysfs",
          "options": [
            "nosuid",
            "noexec",
            "nodev",
            "ro"
          ]
        },
        {
          "destination": "/sys/fs/cgroup",
          "type": "cgroup",
          "source": "cgroup",
          "options": [
            "nosuid",
            "noexec",
            "nodev",
            "relatime",
            "ro"
          ]
        },
        {
          "destination": "/dev/shm",
          "type": "bind",
          "source": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/shm",
          "options": [
            "rw",
            "bind"
          ]
        },
        {
          "destination": "/etc/resolv.conf",
          "type": "bind",
          "source": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/resolv.conf",
          "options": [
            "rw",
            "bind",
            "nodev",
            "nosuid",
            "noexec"
          ]
        },
        {
          "destination": "/etc/hostname",
          "type": "bind",
          "source": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/hostname",
          "options": [
            "rw",
            "bind"
          ]
        },
        {
          "destination": "/run/.containerenv",
          "type": "bind",
          "source": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/.containerenv",
          "options": [
            "rw",
            "bind"
          ]
        },
        {
          "destination": "/prometheus",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/pvc-b17ae945-02d2-43e2-a2d2-23891203b1f3/prometheus/3",
          "options": [
            "rw",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/dev/termination-log",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers/prometheus/5ab56fc4",
          "options": [
            "rw",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/hosts",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts",
          "options": [
            "rw",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/run/secrets",
          "type": "bind",
          "source": "/run/containers/storage/overlay-containers/27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084/userdata/run/secrets",
          "options": [
            "bind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/config_out",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~empty-dir/config-out",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/certs",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/tls-assets",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/secrets/metrics-client-certs",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-metrics-client-certs",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/secrets/prometheus-k8s-tls",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/secrets/prometheus-k8s-proxy",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-proxy",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/secrets/kube-etcd-client-certs",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-etcd-client-certs",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/secrets/kube-rbac-proxy",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-rbac-proxy",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/secrets/prometheus-k8s-thanos-sidecar-tls",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/configmaps/serving-certs-ca-bundle",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-serving-certs-ca-bundle",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/configmaps/kubelet-serving-ca-bundle",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-kubelet-serving-ca-bundle",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/configmaps/metrics-client-ca",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-metrics-client-ca",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-k8s-rulefiles-0",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/prometheus/web_config/web-config.yaml",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/web-config/prometheus/5",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/var/run/secrets/kubernetes.io/serviceaccount",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/kube-api-access-jfzqj",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        },
        {
          "destination": "/etc/pki/ca-trust/extracted/pem/",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-trusted-ca-bundle",
          "options": [
            "ro",
            "rbind",
            "rprivate",
            "bind"
          ]
        }
      ],
      "annotations": {
        "openshift.io/scc": "nonroot",
        "io.kubernetes.cri-o.Name": "k8s_prometheus_prometheus-k8s-0_openshift-monitoring_a4477163-737d-457c-8698-8688fdd00003_0",
        "io.kubernetes.cri-o.SandboxName": "k8s_prometheus-k8s-0_openshift-monitoring_a4477163-737d-457c-8698-8688fdd00003_0",
        "kubectl.kubernetes.io/default-container": "prometheus",
        "io.kubernetes.cri-o.Stdin": "false",
        "io.kubernetes.cri-o.ResolvPath": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/resolv.conf",
        "io.kubernetes.cri-o.Volumes": "[{\"container_path\":\"/prometheus\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/pvc-b17ae945-02d2-43e2-a2d2-23891203b1f3/prometheus/3\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/dev/termination-log\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers/prometheus/5ab56fc4\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/hosts\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/config_out\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~empty-dir/config-out\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/certs\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/tls-assets\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/prometheus-k8s-thanos-sidecar-tls\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/kube-etcd-client-certs\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-etcd-client-certs\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/prometheus-k8s-tls\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/prometheus-k8s-proxy\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-proxy\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/web_config/web-config.yaml\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/web-config/prometheus/5\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/kube-rbac-proxy\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-rbac-proxy\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/metrics-client-certs\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-metrics-client-certs\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/configmaps/serving-certs-ca-bundle\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-serving-certs-ca-bundle\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/configmaps/kubelet-serving-ca-bundle\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-kubelet-serving-ca-bundle\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/configmaps/metrics-client-ca\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-metrics-client-ca\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-k8s-rulefiles-0\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/var/run/secrets/kubernetes.io/serviceaccount\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/kube-api-access-jfzqj\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/pki/ca-trust/extracted/pem/\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-trusted-ca-bundle\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true}]",
        "io.kubernetes.container.hash": "7ba9b53",
        "io.kubernetes.pod.name": "prometheus-k8s-0",
        "io.kubernetes.cri-o.IP.0": "10.131.2.7",
        "io.kubernetes.cri-o.SeccompProfilePath": "",
        "io.kubernetes.cri-o.Metadata": "{\"name\":\"prometheus\"}",
        "io.kubernetes.cri-o.Labels": "{\"io.kubernetes.container.name\":\"prometheus\",\"io.kubernetes.pod.name\":\"prometheus-k8s-0\",\"io.kubernetes.pod.namespace\":\"openshift-monitoring\",\"io.kubernetes.pod.uid\":\"a4477163-737d-457c-8698-8688fdd00003\"}",
        "io.kubernetes.pod.uid": "a4477163-737d-457c-8698-8688fdd00003",
        "kubernetes.io/config.seen": "2022-09-27T08:46:47.929904784Z",
        "io.kubernetes.cri-o.ImageName": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00d156818e2e4264c3268c9ae41de9b561e2811363b64515ca64865c992ec8ea",
        "io.kubernetes.cri-o.ContainerType": "container",
        "io.kubernetes.cri-o.StdinOnce": "false",
        "io.kubernetes.cri-o.Created": "2022-09-27T08:46:55.0148206Z",
        "io.kubernetes.container.terminationMessagePolicy": "FallbackToLogsOnError",
        "kubernetes.io/config.source": "api",
        "io.kubernetes.cri-o.ImageRef": "d3017e38d93fb39bcad5ae9ee8521f5829c774c389ae7e65e44a00388d67cf1e",
        "io.kubernetes.container.restartCount": "0",
        "org.systemd.property.DefaultDependencies": "true",
        "io.kubernetes.container.name": "prometheus",
        "io.kubernetes.cri-o.LogPath": "/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4477163-737d-457c-8698-8688fdd00003/prometheus/0.log",
        "io.kubernetes.cri-o.TTY": "false",
        "io.container.manager": "cri-o",
        "io.kubernetes.cri-o.MountPoint": "/var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged",
        "io.kubernetes.cri-o.Annotations": "{\"io.kubernetes.container.hash\":\"7ba9b53\",\"io.kubernetes.container.restartCount\":\"0\",\"io.kubernetes.container.terminationMessagePath\":\"/dev/termination-log\",\"io.kubernetes.container.terminationMessagePolicy\":\"FallbackToLogsOnError\",\"io.kubernetes.pod.terminationGracePeriod\":\"600\"}",
        "io.kubernetes.pod.terminationGracePeriod": "600",
        "io.kubernetes.pod.namespace": "openshift-monitoring",
        "io.kubernetes.cri-o.SandboxID": "fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d",
        "io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
        "io.kubernetes.cri-o.Image": "d3017e38d93fb39bcad5ae9ee8521f5829c774c389ae7e65e44a00388d67cf1e",
        "io.kubernetes.cri-o.ContainerID": "27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084",
        "org.systemd.property.TimeoutStopUSec": "uint64 600000000",
        "org.systemd.property.CollectMode": "'inactive-or-failed'",
        "org.systemd.property.After": "['crio.service']"
      },
      "linux": {
        "resources": {
          "devices": [
            {
              "allow": false,
              "access": "rwm"
            }
          ],
          "cpu": {
            "shares": 71,
            "quota": 0,
            "period": 100000
          },
          "pids": {
            "limit": 4096
          },
          "hugepageLimits": [
            {
              "pageSize": "2MB",
              "limit": 0
            },
            {
              "pageSize": "1GB",
              "limit": 0
            }
          ]
        },
        "cgroupsPath": "kubepods-burstable-poda4477163_737d_457c_8698_8688fdd00003.slice:crio:27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084",
        "namespaces": [
          {
            "type": "pid"
          },
          {
            "type": "network",
            "path": "/var/run/netns/5b03e11b-6a0a-4027-b037-cb8aec0048c7"
          },
          {
            "type": "ipc",
            "path": "/var/run/ipcns/5b03e11b-6a0a-4027-b037-cb8aec0048c7"
          },
          {
            "type": "uts",
            "path": "/var/run/utsns/5b03e11b-6a0a-4027-b037-cb8aec0048c7"
          },
          {
            "type": "mount"
          }
        ],
        "maskedPaths": [
          "/proc/acpi",
          "/proc/kcore",
          "/proc/keys",
          "/proc/latency_stats",
          "/proc/timer_list",
          "/proc/timer_stats",
          "/proc/sched_debug",
          "/proc/scsi",
          "/sys/firmware"
        ],
        "readonlyPaths": [
          "/proc/asound",
          "/proc/bus",
          "/proc/fs",
          "/proc/irq",
          "/proc/sys",
          "/proc/sysrq-trigger"
        ],
        "mountLabel": "system_u:object_r:container_file_t:s0:c21,c0"
      }
    },
    "privileged": false
  }
}
sh-4.4#
```

→ Workerノード上のディレクトリパスを確認したいので、`Path`でgrepする。

```
sh-4.4# crictl inspect 27cccaa137e39 | grep Path
      "io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
        "containerPath": "/prometheus",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/pvc-b17ae945-02d2-43e2-a2d2-23891203b1f3/prometheus/3",
        "containerPath": "/dev/termination-log",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers/prometheus/5ab56fc4",
        "containerPath": "/etc/hosts",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts",
        "containerPath": "/etc/prometheus/config_out",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~empty-dir/config-out",
        "containerPath": "/etc/prometheus/certs",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/tls-assets",
        "containerPath": "/etc/prometheus/secrets/prometheus-k8s-thanos-sidecar-tls",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls",
        "containerPath": "/etc/prometheus/secrets/kube-etcd-client-certs",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-etcd-client-certs",
        "containerPath": "/etc/prometheus/secrets/prometheus-k8s-tls",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls",
        "containerPath": "/etc/prometheus/secrets/prometheus-k8s-proxy",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-proxy",
        "containerPath": "/etc/prometheus/web_config/web-config.yaml",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/web-config/prometheus/5",
        "containerPath": "/etc/prometheus/secrets/kube-rbac-proxy",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-rbac-proxy",
        "containerPath": "/etc/prometheus/secrets/metrics-client-certs",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-metrics-client-certs",
        "containerPath": "/etc/prometheus/configmaps/serving-certs-ca-bundle",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-serving-certs-ca-bundle",
        "containerPath": "/etc/prometheus/configmaps/kubelet-serving-ca-bundle",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-kubelet-serving-ca-bundle",
        "containerPath": "/etc/prometheus/configmaps/metrics-client-ca",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-metrics-client-ca",
        "containerPath": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-k8s-rulefiles-0",
        "containerPath": "/var/run/secrets/kubernetes.io/serviceaccount",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/kube-api-access-jfzqj",
        "containerPath": "/etc/pki/ca-trust/extracted/pem/",
        "hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-trusted-ca-bundle",
    "logPath": "/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4477163-737d-457c-8698-8688fdd00003/prometheus/0.log"
        "io.kubernetes.cri-o.Annotations": "{\"io.kubernetes.container.hash\":\"7ba9b53\",\"io.kubernetes.container.restartCount\":\"0\",\"io.kubernetes.container.terminationMessagePath\":\"/dev/termination-log\",\"io.kubernetes.container.terminationMessagePolicy\":\"FallbackToLogsOnError\",\"io.kubernetes.pod.terminationGracePeriod\":\"600\"}",
        "io.kubernetes.cri-o.LogPath": "/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4477163-737d-457c-8698-8688fdd00003/prometheus/0.log",
        "io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
        "io.kubernetes.cri-o.ResolvPath": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/resolv.conf",
        "io.kubernetes.cri-o.SeccompProfilePath": "",
        "cgroupsPath": "kubepods-burstable-poda4477163_737d_457c_8698_8688fdd00003.slice:crio:27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084",
        "maskedPaths": [
        "readonlyPaths": [
sh-4.4#
```

→ホスト（Workerノード）の`/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/`ディレクトリを利用していることが分かる。

```
      "root": {
        "path": "/var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged"
      },
```

他に、上記の部分を確認すると、`/var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged`ディレクトリを利用していることが分かる。



## ディスク使用量の確認

Infra/Workerノード上の当該ディレクトリのサイズを確認する。

```
sh-4.4# du -sh /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/*
0       /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers
4.0K    /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts
4.0K    /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/plugins
14G     /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths
1.1M    /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes
sh-4.4#
```

→ 約14GB 使用。

```
sh-4.4# du -sh /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/*
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/bin
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/boot
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/dev
2.3M    /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/etc
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/home
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/lib
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/lib64
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/lost+found
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/media
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/mnt
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/opt
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/proc
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/prometheus
996K    /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/root
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/run
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/sbin
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/srv
0       /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/sys
8.0K    /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/tmp
370M    /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/usr
24M     /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/var
sh-4.4#
```

→ 約370MB 使用。

合計して、Prometheus 1podが約15GB程度使用していることが確認できた。
※参考：本ROSA環境は検証用の簡単なアプリケーションが数個稼働している程度の環境であり、Prometheusのretantionなどの設定はデフォルトのまま。

# まとめ

Podのディスク使用量を確認するうえで、Podが使用するInfra/Workerノードのディレクトリ情報が必要になり、そのためにcrictlコマンドを利用し確認を行った。
Podのディスク使用量の確認方法については、他にもスマートなやり方がありそうだが、、探せなかったので、ひとまず本記事の方法で確認を行った。

# 参考
本記事ではストレージの利用に関しての説明は省略したが、OpenShiftのPodが利用するストレージについては以下マニュアルを参照のこと。
[Understanding ephemeral storage](https://docs.openshift.com/container-platform/4.10/storage/understanding-ephemeral-storage.html)

# おまけ

ROSAのInfra/Workerノードはデフォルトで300GBのEBSが利用される。
現状では下記の量だけディスクが使用されており、今後他のPodが大量にディスクを消費するようなことがあるならPVの利用も検討すべきか。

```
sh-4.4# df -h /var/lib
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p4  300G   19G  281G   7% /var
```