KubernetesのノードやPodのCPU・メモリリソース使用状況を確認するコマンドを検証する

## はじめに

CPUやメモリ使用量の多いPodをデプロイした検証環境にて、リソース使用状況を確認できる kubectl top コマンドや、kubectl プラグイン の使い勝手を検証してみました。

kubectl プラグインの導入方法やオプションの指定方法の他、かゆいところに手が届くか、リソース使用量の多いノードやPodを把握しやすいかどうかの観点でもお楽しみください。

本記事では、以下コマンドについて紹介していきます。

kubectl 標準のサブコマンド

- kubectl describe node
- kubectl top node
- kubectl top pod

kubectl プラグイン

- kubectl resource-capacity
- kubectl view-allocations
- kubectl ktop

## 1. 検証環境

RHEL8.4 上で [Minikube](https://minikube.sigs.k8s.io/docs/) で Kubernetes `v1.25.3` のクラスタを 用意しました。

```bash
$ uname -a
Linux rhel8 4.18.0-240.1.1.el8_3.x86_64 #1 SMP Fri Oct 16 13:36:46 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.3 (Ootpa)
$ minikube version
minikube version: v1.28.0
commit: 986b1ebd987211ed16f8cc10aed7d2c42fc8392f
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:57:26Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:49:09Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}
$
```

クラスタは 3ノード構成とし、各ノードの cpuは4コア、メモリは16GBを割り当てています。
たった1コマンドでKubernetesクラスタが立ち上がるのはありがたいですね。

```
$ minikube start --driver=kvm2 --cpus 4 --memory 16G --nodes 3
😄  minikube v1.28.0 on Redhat 8.3
✨  Using the kvm2 driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating kvm2 VM (CPUs=4, Memory=16384MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.25.3 on Docker 20.10.20 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass

👍  Starting worker node minikube-m02 in cluster minikube
🔥  Creating kvm2 VM (CPUs=4, Memory=16384MB, Disk=20000MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.39.189
🐳  Preparing Kubernetes v1.25.3 on Docker 20.10.20 ...
    ▪ env NO_PROXY=192.168.39.189
🔎  Verifying Kubernetes components...

👍  Starting worker node minikube-m03 in cluster minikube
🔥  Creating kvm2 VM (CPUs=4, Memory=16384MB, Disk=20000MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.39.189,192.168.39.252
🐳  Preparing Kubernetes v1.25.3 on Docker 20.10.20 ...
    ▪ env NO_PROXY=192.168.39.189
    ▪ env NO_PROXY=192.168.39.189,192.168.39.252
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
$ kubectl cluster-info
Kubernetes control plane is running at https://192.168.39.189:8443
CoreDNS is running at https://192.168.39.189:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$ sudo virsh list --all
 Id   Name           State
-------------------------------
 7    minikube       running
 8    minikube-m02   running
 9    minikube-m03   running
$ kubectl get no
NAME           STATUS   ROLES           AGE     VERSION
minikube       Ready    control-plane   3m6s    v1.25.3
minikube-m02   Ready    <none>          2m25s   v1.25.3
minikube-m03   Ready    <none>          1m47s   v1.25.3
$
```

各コマンドやツールにてCPUやメモリの使用量を取得できるよう、[metrics-server](https://kubernetes.io/ja/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#%E3%83%A1%E3%83%88%E3%83%AA%E3%82%AF%E3%82%B9%E3%82%B5%E3%83%BC%E3%83%90%E3%83%BC) をデプロイしています。
Minikube の場合、`metrics-server` アドオンを有効にすることで簡単にデプロイできます。

```bash
$ minikube addons list
|-----------------------------|----------|--------------|--------------------------------|
|         ADDON NAME          | PROFILE  |    STATUS    |           MAINTAINER           |
|-----------------------------|----------|--------------|--------------------------------|
| ambassador                  | minikube | disabled     | 3rd party (Ambassador)         |
| auto-pause                  | minikube | disabled     | Google                         |
| cloud-spanner               | minikube | disabled     | Google                         |
| csi-hostpath-driver         | minikube | disabled     | Kubernetes                     |
| dashboard                   | minikube | disabled     | Kubernetes                     |
| default-storageclass        | minikube | enabled ✅   | Kubernetes                     |
| efk                         | minikube | disabled     | 3rd party (Elastic)            |
| freshpod                    | minikube | disabled     | Google                         |
| gcp-auth                    | minikube | disabled     | Google                         |
| gvisor                      | minikube | disabled     | Google                         |
| headlamp                    | minikube | disabled     | 3rd party (kinvolk.io)         |
| helm-tiller                 | minikube | disabled     | 3rd party (Helm)               |
| inaccel                     | minikube | disabled     | 3rd party (InAccel             |
|                             |          |              | [info@inaccel.com])            |
| ingress                     | minikube | disabled     | Kubernetes                     |
| ingress-dns                 | minikube | disabled     | Google                         |
| istio                       | minikube | disabled     | 3rd party (Istio)              |
| istio-provisioner           | minikube | disabled     | 3rd party (Istio)              |
| kong                        | minikube | disabled     | 3rd party (Kong HQ)            |
| kubevirt                    | minikube | disabled     | 3rd party (KubeVirt)           |
| logviewer                   | minikube | disabled     | 3rd party (unknown)            |
| metallb                     | minikube | disabled     | 3rd party (MetalLB)            |
| metrics-server              | minikube | disabled     | Kubernetes                     |
| nvidia-driver-installer     | minikube | disabled     | Google                         |
| nvidia-gpu-device-plugin    | minikube | disabled     | 3rd party (Nvidia)             |
| olm                         | minikube | disabled     | 3rd party (Operator Framework) |
| pod-security-policy         | minikube | disabled     | 3rd party (unknown)            |
| portainer                   | minikube | disabled     | 3rd party (Portainer.io)       |
| registry                    | minikube | disabled     | Google                         |
| registry-aliases            | minikube | disabled     | 3rd party (unknown)            |
| registry-creds              | minikube | disabled     | 3rd party (UPMC Enterprises)   |
| storage-provisioner         | minikube | enabled ✅   | Google                         |
| storage-provisioner-gluster | minikube | disabled     | 3rd party (Gluster)            |
| volumesnapshots             | minikube | disabled     | Kubernetes                     |
|-----------------------------|----------|--------------|--------------------------------|
$ minikube addons enable metrics-server
💡  metrics-server is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image k8s.gcr.io/metrics-server/metrics-server:v0.6.1
🌟  The 'metrics-server' addon is enabled
$ minikube addons list
|-----------------------------|----------|--------------|--------------------------------|
|         ADDON NAME          | PROFILE  |    STATUS    |           MAINTAINER           |
|-----------------------------|----------|--------------|--------------------------------|
( 略 )
| metrics-server              | minikube | enabled ✅   | Kubernetes                     |
( 略 )
$ kubectl get po -o wide -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE     IP               NODE           NOMINATED NODE   READINESS GATES
kube-system   coredns-565d847f94-hnn5b           1/1     Running   0          7m39s   10.244.0.2       minikube       <none>           <none>
kube-system   etcd-minikube                      1/1     Running   0          7m50s   192.168.39.189   minikube       <none>           <none>
kube-system   kindnet-b7lfx                      1/1     Running   0          7m10s   192.168.39.252   minikube-m02   <none>           <none>
kube-system   kindnet-g9sfn                      1/1     Running   0          7m39s   192.168.39.189   minikube       <none>           <none>
kube-system   kindnet-xb2rl                      1/1     Running   0          6m31s   192.168.39.139   minikube-m03   <none>           <none>
kube-system   kube-apiserver-minikube            1/1     Running   0          7m50s   192.168.39.189   minikube       <none>           <none>
kube-system   kube-controller-manager-minikube   1/1     Running   0          7m50s   192.168.39.189   minikube       <none>           <none>
kube-system   kube-proxy-55rb9                   1/1     Running   0          6m31s   192.168.39.139   minikube-m03   <none>           <none>
kube-system   kube-proxy-8q8b8                   1/1     Running   0          7m39s   192.168.39.189   minikube       <none>           <none>
kube-system   kube-proxy-c5cfx                   1/1     Running   0          7m10s   192.168.39.252   minikube-m02   <none>           <none>
kube-system   kube-scheduler-minikube            1/1     Running   0          7m50s   192.168.39.189   minikube       <none>           <none>
kube-system   metrics-server-769cd898cd-6cf2f    1/1     Running   0          2m55s   10.244.1.2       minikube-m02   <none>           <none>
kube-system   storage-provisioner                1/1     Running   0          7m48s   192.168.39.189   minikube       <none>           <none>
$
```

### 1.1. 検証用Podの準備 : CPU使用量の多いPod

本記事の検証環境では、CPUリソース使用量の多いPodの例として、2コアを使用する Pod `cpu-demo` を作成しています。
後述のコマンドにより、このPod や このPodが動作しているノードのCPU使用量を把握しやすいかの観点もお楽しみください。

```bash
$ wget https://k8s.io/examples/pods/resource/cpu-request-limit.yaml
$ vim cpu-request-limit.yaml
$ cat cpu-request-limit.yaml
apiVersion: v1
kind: Pod
metadata:
  name: cpu-demo
spec:
  containers:
  - name: cpu-demo-ctr
    image: vish/stress
    resources:
      limits:
        cpu: "3"
      requests:
        cpu: "1"
    args:
    - -cpus
    - "2"
$ kubectl apply -f cpu-request-limit.yaml 
pod/cpu-demo created
$ kubectl get po -o wide
NAME       READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
cpu-demo   1/1     Running   0          5s    10.244.2.5   minikube-m03   <none>           <none>
$
```

参考ドキュメント: コンテナおよびPodへのCPUリソースの割り当て | Kubernetes

https://kubernetes.io/ja/docs/tasks/configure-pod-container/assign-cpu-resource/

### 1.2. 検証用Podの準備 : メモリ使用量の多いPod

メモリを多めに使用するPodの例として 8GBを使用する Pod `memory-demo` を作成しています。
後述のコマンドにより、このPod や このPodが動作しているノードの メモリ使用量を把握しやすいかの観点もお楽しみください。

```bash
$ wget https://k8s.io/examples/pods/resource/memory-request-limit.yaml
$ vim memory-request-limit.yaml
$ cat memory-request-limit.yaml
apiVersion: v1
kind: Pod
metadata:
  name: memory-demo
spec:
  containers:
  - name: memory-demo-ctr
    image: polinux/stress
    resources:
      requests:
        memory: "4Gi"
      limits:
        memory: "12Gi"
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "8G", "--vm-hang", "1"]
$ kubectl apply -f memory-request-limit.yaml
pod/memory-demo created
$ kubectl get po -o wide
NAME          READY   STATUS    RESTARTS   AGE    IP           NODE           NOMINATED NODE   READINESS GATES
cpu-demo      1/1     Running   0          105s   10.244.2.5   minikube-m03   <none>           <none>
memory-demo   1/1     Running   0          24s    10.244.1.3   minikube-m02   <none>           <none>
$ 
```

参考ドキュメント: コンテナおよびPodへのメモリーリソースの割り当て | Kubernetes

https://kubernetes.io/ja/docs/tasks/configure-pod-container/assign-memory-resource/

## 2. kubectl 標準のサブコマンドで確認する

まずは kubectl に標準で備わっているサブコマンドからご紹介していきます。

### 2.1. kubectl describe node <ノード名>

ノードの様々な情報を参照できるコマンドで、当該ノードにデプロイされている Pod の CPUやメモリ の 要求(`Requests`) と 制限(`Limits`) の値も確認できます。

ノード一覧を表示、ノード名を確認しておきます。

```bash
$ kubectl get node
NAME           STATUS   ROLES           AGE   VERSION
minikube       Ready    control-plane   38m   v1.25.3
minikube-m02   Ready    <none>          37m   v1.25.3
minikube-m03   Ready    <none>          36m   v1.25.3
$
```

以下はノード `minikube-m02` 上の Pod について確認する例です。
`Non-terminated Pod` の項目に Pod ごとに表示されます。( 当該項目のみ抜粋しています。)
表示される値はPod内のコンテナの`Requests` と `Limits` の合計値です。

検証用Pod `memory-demo` を作成する際に指定した メモリの `Requests` と `Limits` の値を確かに確認できていますね。

```bash
$ kubectl describe node minikube-m02
Non-terminated Pods: (4 in total)
  Namespace    Name                             CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------    ----                             ------------  ----------  ---------------  -------------  ---
  default      memory-demo                      0 (0%)        0 (0%)      4Gi (25%)        12Gi (76%)     26m
  kube-system  kindnet-b7lfx                    100m (2%)     100m (2%)   50Mi (0%)        50Mi (0%)      50m
  kube-system  kube-proxy-c5cfx                 0 (0%)        0 (0%)      0 (0%)           0 (0%)         50m
  kube-system  metrics-server-769cd898cd-6cf2f  100m (2%)     0 (0%)      200Mi (1%)       0 (0%)         46m
$
```

以下はノード `minikube-m03` 上の Pod について確認する例です。
( `node` は `no` と短く書くこともできます。)

検証用Pod `cpu-demo` を作成する際に指定した CPUの `Requests` と `Limits` の値を確かに確認できていますね。

```log
$ kubectl describe no minikube-m03
Non-terminated Pods:  (3 in total)
  Namespace    Name              CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------    ----              ------------  ----------  ---------------  -------------  ---
  default      cpu-demo          1 (25%)       3 (75%)     0 (0%)           0 (0%)         30m
  kube-system  kindnet-xb2rl     100m (2%)     100m (2%)   50Mi (0%)        50Mi (0%)      52m
  kube-system  kube-proxy-55rb9  0 (0%)        0 (0%)      0 (0%)           0 (0%)         52m
$
```

参考ドキュメント: コンテナのリソース管理 | Kubernetes

https://kubernetes.io/ja/docs/concepts/configuration/manage-resources-containers/#requests-and-limits

### 2.2. kubectl top

CPU や メモリの 使用量・率を確認できるコマンドです。

#### 2.2.1. kubectl top node

ノードごとに CPU や メモリの 使用量・率を確認できるコマンドです。

検証用Pod `memory-demo` がデプロイされているノード `minikube-m02` の メモリ使用量・率や、
検証用Pod `cpu-demo` がデプロイされているノード `minikube-m03` の CPU使用量・率が比較的高いことが分かります。

```bash
$ kubectl top node
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
minikube       84m          2%     1474Mi          9%
minikube-m02   723m         18%    7185Mi          44%
minikube-m03   2020m        50%    863Mi           5%
$
```

`--sort-by` オプションに `cpu` もしくは `memory` を指定してソートできます。
リソース使用率の高いノードを確認する際に便利です。

```bash
$ kubectl top no --sort-by=cpu
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
minikube-m03   2020m        50%    863Mi           5%
minikube-m02   723m         18%    7185Mi          44%
minikube       84m          2%     1474Mi          9%
$ kubectl top no --sort-by=memory
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
minikube-m02   723m         18%    7185Mi          44%
minikube       84m          2%     1474Mi          9%
minikube-m03   2020m        50%    863Mi           5%
$ 
```

#### 2.2.2. kubectl top pod

Pod ごとにCPU や メモリの 使用量・率を確認できるコマンドです。
( `-A` は `--all-namespaces` の指定と同じです。)

```log
$ kubectl top pod -A
NAMESPACE              NAME                                        CPU(cores)   MEMORY(bytes)   
default                cpu-demo                                    2002m        0Mi
default                memory-demo                                 700m         328Mi
kube-system            coredns-565d847f94-hnn5b                    1m           12Mi
kube-system            etcd-minikube                               13m          47Mi
kube-system            kindnet-b7lfx                               1m           8Mi
kube-system            kindnet-g9sfn                               1m           8Mi
kube-system            kindnet-xb2rl                               1m           8Mi
kube-system            kube-apiserver-minikube                     24m          280Mi
kube-system            kube-controller-manager-minikube            8m           41Mi
kube-system            kube-proxy-55rb9                            1m           9Mi
kube-system            kube-proxy-8q8b8                            1m           9Mi
kube-system            kube-proxy-c5cfx                            1m           9Mi
kube-system            kube-scheduler-minikube                     2m           14Mi
kube-system            metrics-server-769cd898cd-6cf2f             2m           15Mi
kube-system            storage-provisioner                         1m           8Mi
kubernetes-dashboard   dashboard-metrics-scraper-b74747df5-ngnnr   1m           10Mi
kubernetes-dashboard   kubernetes-dashboard-57bbdc5f89-qlldd       2m           20Mi
$
```

`--sort-by` オプションに `cpu` もしくは `memory` を指定してソートすることもできます。

```log
$ kubectl top pod -A --sort-by cpu
NAMESPACE              NAME                                        CPU(cores)   MEMORY(bytes)   
default                cpu-demo                                    2002m        0Mi
default                memory-demo                                 703m         5492Mi
kube-system            kube-apiserver-minikube                     23m          280Mi
kube-system            etcd-minikube                               13m          48Mi
kube-system            kube-controller-manager-minikube            8m           41Mi
kube-system            kube-scheduler-minikube                     2m           14Mi
kube-system            metrics-server-769cd898cd-6cf2f             2m           15Mi
kubernetes-dashboard   kubernetes-dashboard-57bbdc5f89-qlldd       2m           21Mi
kube-system            kindnet-xb2rl                               1m           8Mi
kube-system            kindnet-g9sfn                               1m           8Mi
kube-system            kindnet-b7lfx                               1m           8Mi
kube-system            kube-proxy-55rb9                            1m           9Mi
kube-system            kube-proxy-8q8b8                            1m           9Mi
kube-system            kube-proxy-c5cfx                            1m           9Mi
kube-system            coredns-565d847f94-hnn5b                    1m           12Mi
kube-system            storage-provisioner                         1m           9Mi
kubernetes-dashboard   dashboard-metrics-scraper-b74747df5-ngnnr   1m           10Mi
$ kubectl top pod -A --sort-by memory
NAMESPACE              NAME                                        CPU(cores)   MEMORY(bytes)   
default                memory-demo                                 703m         5492Mi
kube-system            kube-apiserver-minikube                     23m          280Mi
kube-system            etcd-minikube                               13m          48Mi
kube-system            kube-controller-manager-minikube            8m           41Mi
kubernetes-dashboard   kubernetes-dashboard-57bbdc5f89-qlldd       2m           21Mi
kube-system            metrics-server-769cd898cd-6cf2f             2m           15Mi
kube-system            kube-scheduler-minikube                     2m           14Mi
kube-system            coredns-565d847f94-hnn5b                    1m           12Mi
kubernetes-dashboard   dashboard-metrics-scraper-b74747df5-ngnnr   1m           10Mi
kube-system            kube-proxy-8q8b8                            1m           9Mi
kube-system            kube-proxy-c5cfx                            1m           9Mi
kube-system            kube-proxy-55rb9                            1m           9Mi
kube-system            storage-provisioner                         1m           9Mi
kube-system            kindnet-g9sfn                               1m           8Mi
kube-system            kindnet-b7lfx                               1m           8Mi
kube-system            kindnet-xb2rl                               1m           8Mi
default                cpu-demo                                    2002m        0Mi
$
```

`watch` コマンドと組み合わせて定期的に表示するのも便利です。

```
$ watch kubectl top pod -A --sort-by cpu
```

他にも `--containers` オプションを付与しコンテナごとの表示としたり、`-l` オプションにラベル(key=val)を指定してPodを絞り込むこともできます。
`kubectl top po --help` やドキュメントをご参照ください。

参考ドキュメント: Kubectl Reference Docs

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#top

## 3. リソースの使用状況を確認できる kubectl プラグイン

kubectl にはプラグイン機構が備わっており、お好きなプラグインを追加してkubectlのサブコマンドとして呼び出すことができるようになっています。
kubectl プラグインについての解説は以下の記事がオススメです。

https://techblog.yahoo.co.jp/entry/2020081830014718/

本章では、様々な kubectl プラグイン の中でも、CPUやメモリの使用状況を確認するのに便利なプラグインをご紹介していきます。

参考ドキュメント：Extend kubectl with plugins | Kubernetes

https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/

### 3.1. Krew を導入し、kubectl のプラグインを管理できるようにする

[Krew](https://krew.sigs.k8s.io/) は kubectl のプラグインを簡単に管理できるツールです。

[Installing · Krew](https://krew.sigs.k8s.io/docs/user-guide/setup/install/) のドキュメントを参考に、動作環境に即した方法でインストールします。
若干長いですが、以下は RHEL8.3 bash 環境にてインストールした際のコマンド実行例です。

余談ですが `${OS}` や `${ARCH}` 文字列 は このようにして生成できるんですね。

```bash
$ (
>   set -x; cd "$(mktemp -d)" &&
>   OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
>   ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
>   KREW="krew-${OS}_${ARCH}" &&
>   curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
>   tar zxvf "${KREW}.tar.gz" &&
>   ./"${KREW}" install krew
> )
++ mktemp -d
+ cd /tmp/tmp.DZxaxnU4CT
++ uname
++ tr '[:upper:]' '[:lower:]'
+ OS=linux
++ uname -m
++ sed -e s/x86_64/amd64/ -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/'
+ ARCH=amd64
+ KREW=krew-linux_amd64
+ curl -fsSLO https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-linux_amd64.tar.gz
+ tar zxvf krew-linux_amd64.tar.gz
./LICENSE
./krew-linux_amd64
+ ./krew-linux_amd64 install krew
Adding "default" plugin index from https://github.com/kubernetes-sigs/krew-index.git.
Updated the local copy of plugin index.
Installing plugin: krew
Installed plugin: krew
\
 | Use this plugin:
 |      kubectl krew
 | Documentation:
 |      https://krew.sigs.k8s.io/
 | Caveats:
 | \
 |  | krew is now installed! To start using kubectl plugins, you need to add
 |  | krew's installation directory to your PATH:
 |  |
 |  |   * macOS/Linux:
 |  |     - Add the following to your ~/.bashrc or ~/.zshrc:
 |  |         export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
 |  |     - Restart your shell.
 |  |
 |  |   * Windows: Add %USERPROFILE%\.krew\bin to your PATH environment variable
 |  |
 |  | To list krew commands and to get help, run:
 |  |   $ kubectl krew
 |  | For a full list of available plugins, run:
 |  |   $ kubectl krew search
 |  |
 |  | You can find documentation at
 |  |   https://krew.sigs.k8s.io/docs/user-guide/quickstart/.
 | /
/
$ vim ~/.bashrc
$ tail -n 1 ~/.bashrc
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
$ 
( いったん exit して再ログインする )
```

インストール後、以下のように krew の ヘルプを表示できればOKです。
krew 自体も kubectl のプラグインであるため、kubectlのサブコマンドとして呼び出すことができます。

```bash
$ kubectl krew --help
krew is the kubectl plugin manager.
You can invoke krew through kubectl: "kubectl krew [command]..."

Usage:
  kubectl krew [command]

Available Commands:
  completion  generate the autocompletion script for the specified shell
  help        Help about any command
  index       Manage custom plugin indexes
  info        Show information about an available plugin
  install     Install kubectl plugins
  list        List installed kubectl plugins
  search      Discover kubectl plugins
  uninstall   Uninstall plugins
  update      Update the local copy of the plugin index
  upgrade     Upgrade installed plugins to newer versions
  version     Show krew version and diagnostics

Flags:
  -h, --help      help for krew
  -v, --v Level   number for the log level verbosity

Use "kubectl krew [command] --help" for more information about a command.
$
```

以下は(かなり長いですが)、Krew を使用してインストールできる kubectl プラグインの一覧です。
本記事執筆時点で Krewは 208個の kubectl プラグインに対応しているようです。多いですね。。

リソース情報を表示できそうなプラグインはいちおうひと通り確認したつもりですが、
本記事でご紹介できていないプラグインがありましたらコメント等でお知らせいただきますと幸いです。

```log
$ kubectl krew search | wc -l
209
$ kubectl krew search
NAME                            DESCRIPTION                                         INSTALLED
access-matrix                   Show an RBAC access matrix for server resources     no
accurate                        Manage Accurate, a multi-tenancy controller         no
advise-policy                   Suggests PodSecurityPolicies and OPA Policies f...  no
advise-psp                      Suggests PodSecurityPolicies for cluster.           no
allctx                          Run commands on contexts in your kubeconfig         no
apparmor-manager                Manage AppArmor profiles for cluster.               no
applier                         Apply 'go text/template' files on k8s.              no
assert                          Assert Kubernetes resources                         no
auth-proxy                      Authentication proxy to a pod or service            no
aws-auth                        Manage aws-auth ConfigMap                           no
azad-proxy                      Generate and handle authentication for azad-kub...  no
bd-xray                         Run Black Duck Image Scans                          no
blame                           Show who edited resource fields.                    no
bulk-action                     Do bulk actions on Kubernetes resources.            no
ca-cert                         Print the PEM CA certificate of the current clu...  no
capture                         Triggers a Sysdig capture to troubleshoot the r...  no
cert-manager                    Manage cert-manager resources inside your cluster   no
change-ns                       View or change the current namespace via kubectl.   no
cilium                          Easily interact with Cilium agents.                 no
cluster-group                   Exec commands across a group of contexts.           no
clusternet                      Wrap multiple kubectl calls to Clusternet           no
cm                              Provides commands for OCM/MCE/ACM.                  no
cnpg                            Manage your CloudNativePG clusters                  no
colorize-applied                Colorize results of apply/dry-run                   no
config-cleanup                  Automatically clean up your kubeconfig              no
config-registry                 Switch between registered kubeconfigs               no
confirm                         Dry-run / diff / confirm before running a command   no
cost                            View cluster cost information                       no
count                           Count resources by kind                             no
creyaml                         Generate custom resource YAML manifest              no
ctx                             Switch between contexts in your kubeconfig          no
custom-cols                     A "kubectl get" replacement with customizable c...  no
cyclonus                        NetworkPolicy analysis tool suite                   no
datadog                         Manage the Datadog Operator                         no
datree                          Scan your cluster resources for misconfigurations   no
dds                             Detect if workloads are mounting the docker socket  no
debug-shell                     Create pod with interactive kube-shell.             no
deprecations                    Checks for deprecated objects in a cluster          no
df-pv                           Show disk usage (like unix df) for persistent v...  no
direct-csi                      CSI driver to manage drives in k8s cluster as v...  no
directpv                        Deploys and manages the lifecycle of DirectPV C...  no
discover                        Find/export kubeconfigs for cloud clusters          no
doctor                          Scans your cluster and reports anomalies.           no
dtlogin                         Login to a cluster via openid-connect               no
duck                            List custom resources with ducktype support         no
edit-status                     Edit /status subresources of CRs                    no
eds                             Easily interact and manage ExtendedDaemonset re...  no
eksporter                       Export resources and removes a pre-defined set ...  no
emit-event                      Emit Kubernetes Events for the requested object     no
evict-pod                       Evicts the given pod                                no
example                         Prints out example manifest YAMLs                   no
exec-as                         Like kubectl exec, but offers a `user` flag to ...  no
exec-cronjob                    Run a CronJob immediately as Job                    no
execws                          kubectl exec using WebSockets                       no
explore                         A better kubectl explain with the fuzzy finder      no
fields                          Grep resources hierarchy by field name              no
flame                           Generate CPU flame graphs from pods                 no
fleet                           Shows config and resources of a fleet of clusters   no
flyte                           Monitor, launch and manage flyte executions         no
foreach                         Run kubectl commands against some/all contexts ...  no
fuzzy                           Fuzzy and partial string search for kubectl         no
gadget                          Gadgets for debugging and introspecting apps        no
get-all                         Like `kubectl get all` but _really_ everything      no
gke-credentials                 Fetch credentials for GKE clusters                  no
gopass                          Imports secrets from gopass                         no
graph                           Visualize Kubernetes resources and relationships.   no
grep                            Filter Kubernetes resources by matching their n...  no
gs                              Handle custom resources with Giant Swarm            no
hlf                             Deploy and manage Hyperledger Fabric components     no
hns                             Manage hierarchical namespaces (part of HNC)        no
htpasswd                        Create nginx-ingress compatible basic-auth secrets  no
ice                             View configuration settings of containers insid...  no
iexec                           Interactive selection tool for `kubectl exec`       no
images                          Show container images used in the cluster.          no
ingress-nginx                   Interact with ingress-nginx                         no
ingress-rule                    Update Ingress rules via command line               no
ipick                           A kubectl wrapper for interactive resource sele...  no
istiolog                        Manipulate istio-proxy logging level without is...  no
janitor                         Lists objects in a problematic state                no
kadalu                          Manage Kadalu Operator, CSI and Storage pods        no
karbon                          Connect to Nutanix Karbon cluster                   no
karmada                         Manage clusters with Karmada federation.            no
kc                              Interactive CRUD operations to manage kubeconfig    no
konfig                          Merge, split or import kubeconfig files             no
krew                            Package manager for kubectl plugins.                yes
kruise                          Easily handle OpenKruise workloads                  no
ks                              Simple management of KubeSphere components          no
ktop                            A top tool to display workload metrics              yes
kubesec-scan                    Scan Kubernetes resources with kubesec.io.          no
kudo                            Declaratively build, install, and run operators...  no
kuota-calc                      Calculate needed quota to perform rolling updates.  no
kurt                            Find what's restarting and why                      no
kuttl                           Declaratively run and test operators                no
kyverno                         Kyverno is a policy engine for kubernetes           no
lineage                         Display all dependent resources or resource dep...  no
linstor                         View and manage LINSTOR storage resources           no
liqo                            Install and manage Liqo on your clusters            no
log2rbac                        Fine-tune your RBAC using log2rbac operator         no
match-name                      Match names of pods and other API objects           no
mc                              Run kubectl commands against multiple clusters ...  no
minio                           Deploy and manage MinIO Operator and Tenant(s)      no
moco                            Interact with MySQL operator MOCO.                  no
modify-secret                   modify secret with implicit base64 translations     no
mtail                           Tail logs from multiple pods matching label sel...  no
multiforward                    Port Forward to multiple Kubernetes Services        no
multinet                        Shows pods' network-status of multi-net-spec        no
neat                            Remove clutter from Kubernetes manifests to mak...  no
net-forward                     Proxy to arbitrary TCP services on a cluster ne...  no
node-admin                      List nodes and run privileged pod with chroot       no
node-restart                    Restart cluster nodes sequentially and gracefully   no
node-shell                      Spawn a root shell on a node via kubectl            no
np-viewer                       Network Policies rules viewer                       no
ns                              Switch between Kubernetes namespaces                no
nsenter                         Run shell command in Pod's namespace on the nod...  no
oidc-login                      Log in to the OpenID Connect provider               no
oomd                            Show recently OOMKilled pods                        no
open-svc                        Open the Kubernetes URL(s) for the specified se...  yes
openebs                         View and debug OpenEBS storage resources            no
operator                        Manage operators with Operator Lifecycle Manager    no
oulogin                         Login to a cluster via OpenUnison                   no
outdated                        Finds outdated container images running in a cl...  no
passman                         Store kubeconfig credentials in keychains or pa...  no
permissions                     Displays and traces service account permissions     no
pexec                           Execute process with privileges in a pod            no
pod-dive                        Shows a pod's workload tree and info inside a node  no
pod-inspect                     Get all of a pod's details at a glance              no
pod-lens                        Show pod-related resources                          no
pod-logs                        Display a list of pods to get logs from             no
pod-shell                       Display a list of pods to execute a shell in        no
podevents                       Show events for pods                                no
popeye                          Scans your clusters for potential resource issues   no
preflight                       Executes application preflight tests in a cluster   no
print-env                       Build config files from k8s environments.           no
profefe                         Gather and manage pprof profiles from running pods  no
promdump                        Dumps the head and persistent blocks of Prometh...  no
prompt                          Prompts for user confirmation when executing co...  no
prune-unused                    Prune unused resources                              no
psp-util                        Manage Pod Security Policy(PSP) and the related...  no
pv-migrate                      Migrate data across persistent volumes              no
pvmigrate                       Migrates PVs between StorageClasses                 no
rabbitmq                        Manage RabbitMQ clusters                            no
rbac-lookup                     Reverse lookup for RBAC                             no
rbac-tool                       Plugin to analyze RBAC permissions and generate...  no
rbac-view                       A tool to visualize your RBAC permissions.          no
realname-diff                   Diffs live and local resources ignoring Kustomi...  no
reap                            Delete unused Kubernetes resources.                 no
relay                           Drop-in "port-forward" replacement with UDP and...  no
reliably                        Surfaces reliability issues in Kubernetes           no
rename-pvc                      Rename a PersistentVolumeClaim (PVC)                no
resource-capacity               Provides an overview of resource requests, limi...  yes
resource-snapshot               Prints a snapshot of nodes, pods and HPAs resou...  no
resource-versions               Print supported API resource versions               no
restart                         Restarts a pod with the given name                  no
rm-standalone-pods              Remove all pods without owner references            no
rolesum                         Summarize RBAC roles for subjects                   no
roll                            Rolling restart of all persistent pods in a nam...  no
rook-ceph                       Rook plugin for Ceph management                     no
safe                            Prompts before running edit commands                no
schemahero                      Declarative database schema migrations via YAML     no
score                           Kubernetes static code analysis.                    no
secretdata                      Viewing decoded Secret data with search flags       no
service-tree                    Status for ingresses, services, and their backends  no
shovel                          Gather diagnostics for .NET Core applications       no
sick-pods                       Find and debug Pods that are "Not Ready"            no
skew                            Find if your cluster/kubectl version is skewed      no
slice                           Split a multi-YAML file into individual files.      no
snap                            Delete half of the pods in a namespace or cluster   no
sniff                           Start a remote packet capture on pods using tcp...  no
socks5-proxy                    SOCKS5 proxy to Services or Pods in the cluster     no
sort-manifests                  Sort manifest files in a proper order by Kind       no
split-yaml                      Split YAML output into one file per resource.       no
spy                             pod debugging tool for kubernetes clusters with...  no
sql                             Query the cluster via pseudo-SQL                    no
ssh-jump                        Access nodes or services using SSH jump Pod         no
sshd                            Run SSH server in a Pod                             no
ssm-secret                      Import/export secrets from/to AWS SSM param store   no
starboard                       Toolkit for finding risks in kubernetes resources   no
status                          Show status details of a given resource.            yes
stern                           Multi pod and container log tailing                 no
strace                          Capture strace logs from a running workload         no
sudo                            Run Kubernetes commands impersonated as group s...  no
support-bundle                  Creates support bundles for off-cluster analysis    no
switch-config                   Switches between kubeconfig files                   no
tail                            Stream logs from multiple pods and containers u...  no
tap                             Interactively proxy Kubernetes Services with ease   no
tmux-exec                       An exec multiplexer using Tmux                      no
topology                        Explore region topology for nodes or pods           no
trace                           Trace Kubernetes pods and nodes with system tools   no
tree                            Show a tree of object hierarchies through owner...  yes
ttsum                           Visualize taints and tolerations                    no
tunnel                          Reverse tunneling between cluster and your machine  no
unused-volumes                  List unused PVCs                                    no
vela                            Easily interact with KubeVela                       no
view-allocations                List allocations per resources, nodes, pods.        yes
view-cert                       View certificate information stored in secrets      no
view-secret                     Decode Kubernetes secrets                           no
view-serviceaccount-kubeconfig  Show a kubeconfig setting to access the apiserv...  no
view-utilization                Shows cluster cpu and memory utilization            no
view-webhook                    Visualize your webhook configurations               no
viewnode                        Displays nodes with their pods and containers a...  no
virt                            Control KubeVirt virtual machines using virtctl     no
volsync                         Manage replication with the VolSync operator        no
vpa-recommendation              Compare VPA recommendations to actual resources...  no
warp                            Sync and execute local files in Pod                 no
whisper-secret                  Create secrets with improved privacy                no
who-can                         Shows who has RBAC permissions to access Kubern...  no
whoami                          Show the subject that's currently authenticated...  no
windows-debug                   Windows node access via kubectl                     no
$
```

以下の公式サイトにて プラグインの一覧や、GitHubリポジトリURL、スター数などを確認できます。
多くのプラグインが提供されていますので、便利なプラグインを探してみるのもおすすめです。

https://krew.sigs.k8s.io/plugins/

それでは、リソースの使用状況を確認できる kubectl プラグイン をご紹介していきます！

### 3.2. kubectl resource-capacity プラグイン

kubectl 標準のサブコマンド `kubectl top` に似ている使い心地ですが、合計値を表示してくれたり、使用量ではなく使用可能な量の表示ができたり、ノードごとのPod一覧表示など、便利な機能が用意されているプラグインです。

https://github.com/robscott/kube-capacity

インストール方法は複数あります。前述した Krew を使用すると以下のように簡単に導入できます。

```bash
$ kubectl krew install resource-capacity
$ kubectl resource-capacity --help
kube-capacity provides an overview of the resource requests, limits, and utilization in a Kubernetes cluster.

Usage:
  kube-capacity [flags]
  kube-capacity [command]

Available Commands:
  help        Help about any command
  version     Print the version number of kube-capacity
( 略 )
$
```

以下はノードごとの表示コマンドです。
`--util` オプションを付与することで `Requests` と `Limits` だけなく、`UTIL` (使用量・率) を表示できます。
合計値も`*`の行に表示してくれます。

```
$ kubectl resource-capacity --util
NODE           CPU REQUESTS   CPU LIMITS    CPU UTIL      MEMORY REQUESTS   MEMORY LIMITS   MEMORY UTIL
*              2150m (17%)    3300m (27%)   2839m (23%)   4616Mi (9%)       12608Mi (26%)   9516Mi (19%)
minikube       850m (21%)     100m (2%)     89m (2%)      220Mi (1%)        220Mi (1%)      1507Mi (9%)
minikube-m02   200m (5%)      100m (2%)     729m (18%)    4346Mi (27%)      12338Mi (77%)   7137Mi (44%)
minikube-m03   1100m (27%)    3100m (77%)   2021m (50%)   50Mi (0%)         50Mi (0%)       873Mi (5%)
$ 
```

`--available` オブションを付与すると、使用可能な量・率を表示できます。
自分で使用量の合計を出したり、全体から引き算しなくて良いのは便利ですね。
各ノードがどれくらい余裕を残しているのか数値として出してくれるのはありがたいです。

```bash
$ kubectl resource-capacity --util --available
NODE           CPU REQUESTS   CPU LIMITS     CPU UTIL       MEMORY REQUESTS   MEMORY LIMITS     MEMORY UTIL
*              9850m/12000m   8700m/12000m   9161m/12000m   43373Mi/47989Mi   35381Mi/47989Mi   38473Mi/47989Mi
minikube       3150m/4000m    3900m/4000m    3911m/4000m    15777Mi/15997Mi   15777Mi/15997Mi   14490Mi/15997Mi
minikube-m02   3800m/4000m    3900m/4000m    3271m/4000m    11651Mi/15997Mi   3659Mi/15997Mi    8860Mi/15997Mi
minikube-m03   2900m/4000m    900m/4000m     1979m/4000m    15947Mi/15997Mi   15947Mi/15997Mi   15124Mi/15997Mi
$ 
```

`--sort` オプションに以下キーを指定してソートも出来ます。

- `cpu.util`
- `cpu.request`
- `cpu.limit`
- `mem.util`
- `mem.request`
- `mem.limit`
- `name`(default)

以下は、CPU、メモリ使用量の多い順にノードを表示する例です。

```log
$ kubectl resource-capacity --util --sort cpu.util
NODE           CPU REQUESTS   CPU LIMITS    CPU UTIL      MEMORY REQUESTS   MEMORY LIMITS   MEMORY UTIL
*              2150m (17%)    3300m (27%)   2841m (23%)   4616Mi (9%)       12608Mi (26%)   7673Mi (15%)
minikube-m03   1100m (27%)    3100m (77%)   2020m (50%)   50Mi (0%)         50Mi (0%)       868Mi (5%)
minikube-m02   200m (5%)      100m (2%)     733m (18%)    4346Mi (27%)      12338Mi (77%)   5296Mi (33%)
minikube       850m (21%)     100m (2%)     89m (2%)      220Mi (1%)        220Mi (1%)      1511Mi (9%)
$ kubectl resource-capacity --util --sort mem.util
NODE           CPU REQUESTS   CPU LIMITS    CPU UTIL      MEMORY REQUESTS   MEMORY LIMITS   MEMORY UTIL
*              2150m (17%)    3300m (27%)   2841m (23%)   4616Mi (9%)       12608Mi (26%)   7673Mi (15%)
minikube-m02   200m (5%)      100m (2%)     733m (18%)    4346Mi (27%)      12338Mi (77%)   5296Mi (33%)
minikube       850m (21%)     100m (2%)     89m (2%)      220Mi (1%)        220Mi (1%)      1511Mi (9%)
minikube-m03   1100m (27%)    3100m (77%)   2020m (50%)   50Mi (0%)         50Mi (0%)       868Mi (5%)
$
```

`--available` オプション付与時もソートできます。親切設計です。
昇順(残り容量の少ない順)表示のようですね。

```log
$ kubectl resource-capacity --util --available --sort cpu.util
NODE           CPU REQUESTS   CPU LIMITS     CPU UTIL       MEMORY REQUESTS   MEMORY LIMITS     MEMORY UTIL
*              9850m/12000m   8700m/12000m   9159m/12000m   43373Mi/47989Mi   35381Mi/47989Mi   40316Mi/47989Mi
minikube-m03   2900m/4000m    900m/4000m     1980m/4000m    15947Mi/15997Mi   15947Mi/15997Mi   15129Mi/15997Mi
minikube-m02   3800m/4000m    3900m/4000m    3267m/4000m    11651Mi/15997Mi   3659Mi/15997Mi    10701Mi/15997Mi
minikube       3150m/4000m    3900m/4000m    3911m/4000m    15777Mi/15997Mi   15777Mi/15997Mi   14486Mi/15997Mi
$ kubectl resource-capacity --util --available --sort mem.util
NODE           CPU REQUESTS   CPU LIMITS     CPU UTIL       MEMORY REQUESTS   MEMORY LIMITS     MEMORY UTIL
*              9850m/12000m   8700m/12000m   9159m/12000m   43373Mi/47989Mi   35381Mi/47989Mi   40316Mi/47989Mi
minikube-m02   3800m/4000m    3900m/4000m    3267m/4000m    11651Mi/15997Mi   3659Mi/15997Mi    10701Mi/15997Mi
minikube       3150m/4000m    3900m/4000m    3911m/4000m    15777Mi/15997Mi   15777Mi/15997Mi   14486Mi/15997Mi
minikube-m03   2900m/4000m    900m/4000m     1980m/4000m    15947Mi/15997Mi   15947Mi/15997Mi   15129Mi/15997Mi
$
```

`--pods` オプションを指定すると、Podごとに表示できます。
kubectl 標準のサブコマンド `kubectl top` とは異なり、ノードごとにPodを表示してくれます。
合計値も`*`の行に表示してくれます。

```
$ kubectl resource-capacity --pods --util
NODE           NAMESPACE              POD                                         CPU REQUESTS   CPU LIMITS    CPU UTIL      MEMORY REQUESTS   MEMORY LIMITS   MEMORY UTIL
*              *                      *                                           2150m (17%)    3300m (27%)   2834m (23%)   4616Mi (9%)       12608Mi (26%)   3547Mi (7%)

minikube       *                      *                                           850m (21%)     100m (2%)     90m (2%)      220Mi (1%)        220Mi (1%)      1512Mi (9%)
minikube       kube-system            coredns-565d847f94-hnn5b                    100m (2%)      0Mi (0%)      1m (0%)       70Mi (0%)         170Mi (1%)      13Mi (0%)
minikube       kubernetes-dashboard   dashboard-metrics-scraper-b74747df5-ngnnr   0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        11Mi (0%)
minikube       kube-system            etcd-minikube                               100m (2%)      0Mi (0%)      13m (0%)      100Mi (0%)        0Mi (0%)        54Mi (0%)
minikube       kube-system            kindnet-g9sfn                               100m (2%)      100m (2%)     1m (0%)       50Mi (0%)         50Mi (0%)       9Mi (0%)
minikube       kube-system            kube-apiserver-minikube                     250m (6%)      0Mi (0%)      27m (0%)      0Mi (0%)          0Mi (0%)        287Mi (1%)
minikube       kube-system            kube-controller-manager-minikube            200m (5%)      0Mi (0%)      8m (0%)       0Mi (0%)          0Mi (0%)        42Mi (0%)
minikube       kube-system            kube-proxy-8q8b8                            0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        10Mi (0%)
minikube       kube-system            kube-scheduler-minikube                     100m (2%)      0Mi (0%)      2m (0%)       0Mi (0%)          0Mi (0%)        15Mi (0%)
minikube       kubernetes-dashboard   kubernetes-dashboard-57bbdc5f89-qlldd       0Mi (0%)       0Mi (0%)      2m (0%)       0Mi (0%)          0Mi (0%)        22Mi (0%)
minikube       kube-system            storage-provisioner                         0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        10Mi (0%)

minikube-m02   *                      *                                           200m (5%)      100m (2%)     726m (18%)    4346Mi (27%)      12338Mi (77%)   1167Mi (7%)
minikube-m02   kube-system            kindnet-b7lfx                               100m (2%)      100m (2%)     1m (0%)       50Mi (0%)         50Mi (0%)       9Mi (0%)
minikube-m02   kube-system            kube-proxy-c5cfx                            0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        10Mi (0%)
minikube-m02   default                memory-demo                                 0Mi (0%)       0Mi (0%)      700m (17%)    4096Mi (25%)      12288Mi (76%)   1539Mi (9%)
minikube-m02   kube-system            metrics-server-769cd898cd-6cf2f             100m (2%)      0Mi (0%)      6m (0%)       200Mi (1%)        0Mi (0%)        17Mi (0%)

minikube-m03   *                      *                                           1100m (27%)    3100m (77%)   2020m (50%)   50Mi (0%)         50Mi (0%)       869Mi (5%)
minikube-m03   default                cpu-demo                                    1000m (25%)    3000m (75%)   2001m (50%)   0Mi (0%)          0Mi (0%)        1Mi (0%)
minikube-m03   kube-system            kindnet-xb2rl                               100m (2%)      100m (2%)     1m (0%)       50Mi (0%)         50Mi (0%)       9Mi (0%)
minikube-m03   kube-system            kube-proxy-55rb9                            0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        10Mi (0%)
$
```

namespace も指定できます。

```
[loft@rhel8 minikube-test]$ kubectl resource-capacity --pods --util -n default
NODE           POD           CPU REQUESTS   CPU LIMITS    CPU UTIL      MEMORY REQUESTS   MEMORY LIMITS   MEMORY UTIL
*              *             1000m (8%)     3000m (25%)   0Mi (0%)      4096Mi (8%)       12288Mi (25%)   0Mi (0%)

minikube       *             0Mi (0%)       0Mi (0%)      0Mi (0%)      0Mi (0%)          0Mi (0%)        0Mi (0%)

minikube-m02   *             0Mi (0%)       0Mi (0%)      703m (17%)    4096Mi (25%)      12288Mi (76%)   6903Mi (43%)
minikube-m02   memory-demo   0Mi (0%)       0Mi (0%)      703m (17%)    4096Mi (25%)      12288Mi (76%)   6903Mi (43%)

minikube-m03   *             1000m (25%)    3000m (75%)   2002m (50%)   0Mi (0%)          0Mi (0%)        2Mi (0%)
minikube-m03   cpu-demo      1000m (25%)    3000m (75%)   2002m (50%)   0Mi (0%)          0Mi (0%)        2Mi (0%)
[loft@rhel8 minikube-test]$ 
```

`--pods` オプション指定時もソートできます。
CPUやメモリ使用量の多い順にPodをノードごとに表示してくれるのはありがたいです。

検証用に準備した `cpu-demo` Pod や `memory-demo` Pod が 1位にランクしており、CPU、メモリを多く使用していることがすぐに把握できます。

```log
$ kubectl resource-capacity --pods --util --sort cpu.util
NODE           NAMESPACE              POD                                         CPU REQUESTS   CPU LIMITS    CPU UTIL      MEMORY REQUESTS   MEMORY LIMITS   MEMORY UTIL
*              *                      *                                           2150m (17%)    3300m (27%)   2843m (23%)   4616Mi (9%)       12608Mi (26%)   11471Mi (23%)

minikube-m03   *                      *                                           1100m (27%)    3100m (77%)   2021m (50%)   50Mi (0%)         50Mi (0%)       871Mi (5%)
minikube-m03   default                cpu-demo                                    1000m (25%)    3000m (75%)   2002m (50%)   0Mi (0%)          0Mi (0%)        1Mi (0%)
minikube-m03   kube-system            kindnet-xb2rl                               100m (2%)      100m (2%)     1m (0%)       50Mi (0%)         50Mi (0%)       9Mi (0%)
minikube-m03   kube-system            kube-proxy-55rb9                            0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        10Mi (0%)

minikube-m02   *                      *                                           200m (5%)      100m (2%)     732m (18%)    4346Mi (27%)      12338Mi (77%)   9086Mi (56%)
minikube-m02   default                memory-demo                                 0Mi (0%)       0Mi (0%)      700m (17%)    4096Mi (25%)      12288Mi (76%)   7485Mi (46%)
minikube-m02   kube-system            metrics-server-769cd898cd-6cf2f             100m (2%)      0Mi (0%)      6m (0%)       200Mi (1%)        0Mi (0%)        17Mi (0%)
minikube-m02   kube-system            kindnet-b7lfx                               100m (2%)      100m (2%)     1m (0%)       50Mi (0%)         50Mi (0%)       9Mi (0%)
minikube-m02   kube-system            kube-proxy-c5cfx                            0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        10Mi (0%)

minikube       *                      *                                           850m (21%)     100m (2%)     91m (2%)      220Mi (1%)        220Mi (1%)      1514Mi (9%)
minikube       kube-system            kube-apiserver-minikube                     250m (6%)      0Mi (0%)      28m (0%)      0Mi (0%)          0Mi (0%)        287Mi (1%)
minikube       kube-system            etcd-minikube                               100m (2%)      0Mi (0%)      13m (0%)      100Mi (0%)        0Mi (0%)        54Mi (0%)
minikube       kube-system            kube-controller-manager-minikube            200m (5%)      0Mi (0%)      8m (0%)       0Mi (0%)          0Mi (0%)        42Mi (0%)
minikube       kube-system            kube-scheduler-minikube                     100m (2%)      0Mi (0%)      2m (0%)       0Mi (0%)          0Mi (0%)        15Mi (0%)
minikube       kubernetes-dashboard   kubernetes-dashboard-57bbdc5f89-qlldd       0Mi (0%)       0Mi (0%)      2m (0%)       0Mi (0%)          0Mi (0%)        22Mi (0%)
minikube       kube-system            storage-provisioner                         0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        10Mi (0%)
minikube       kube-system            kindnet-g9sfn                               100m (2%)      100m (2%)     1m (0%)       50Mi (0%)         50Mi (0%)       9Mi (0%)
minikube       kube-system            kube-proxy-8q8b8                            0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        10Mi (0%)
minikube       kubernetes-dashboard   dashboard-metrics-scraper-b74747df5-ngnnr   0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        11Mi (0%)
minikube       kube-system            coredns-565d847f94-hnn5b                    100m (2%)      0Mi (0%)      1m (0%)       70Mi (0%)         170Mi (1%)      13Mi (0%)
$
$ kubectl resource-capacity --pods --util --sort mem.util
NODE           NAMESPACE              POD                                         CPU REQUESTS   CPU LIMITS    CPU UTIL      MEMORY REQUESTS   MEMORY LIMITS   MEMORY UTIL
*              *                      *                                           2150m (17%)    3300m (27%)   2844m (23%)   4616Mi (9%)       12608Mi (26%)   11470Mi (23%)

minikube-m02   *                      *                                           200m (5%)      100m (2%)     733m (18%)    4346Mi (27%)      12338Mi (77%)   9090Mi (56%)
minikube-m02   default                memory-demo                                 0Mi (0%)       0Mi (0%)      712m (17%)    4096Mi (25%)      12288Mi (76%)   7028Mi (43%)
minikube-m02   kube-system            metrics-server-769cd898cd-6cf2f             100m (2%)      0Mi (0%)      6m (0%)       200Mi (1%)        0Mi (0%)        17Mi (0%)
minikube-m02   kube-system            kube-proxy-c5cfx                            0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        10Mi (0%)
minikube-m02   kube-system            kindnet-b7lfx                               100m (2%)      100m (2%)     1m (0%)       50Mi (0%)         50Mi (0%)       9Mi (0%)

minikube       *                      *                                           850m (21%)     100m (2%)     92m (2%)      220Mi (1%)        220Mi (1%)      1512Mi (9%)
minikube       kube-system            kube-apiserver-minikube                     250m (6%)      0Mi (0%)      29m (0%)      0Mi (0%)          0Mi (0%)        287Mi (1%)
minikube       kube-system            etcd-minikube                               100m (2%)      0Mi (0%)      13m (0%)      100Mi (0%)        0Mi (0%)        54Mi (0%)
minikube       kube-system            kube-controller-manager-minikube            200m (5%)      0Mi (0%)      8m (0%)       0Mi (0%)          0Mi (0%)        42Mi (0%)
minikube       kubernetes-dashboard   kubernetes-dashboard-57bbdc5f89-qlldd       0Mi (0%)       0Mi (0%)      2m (0%)       0Mi (0%)          0Mi (0%)        21Mi (0%)
minikube       kube-system            kube-scheduler-minikube                     100m (2%)      0Mi (0%)      2m (0%)       0Mi (0%)          0Mi (0%)        15Mi (0%)
minikube       kube-system            coredns-565d847f94-hnn5b                    100m (2%)      0Mi (0%)      1m (0%)       70Mi (0%)         170Mi (1%)      13Mi (0%)
minikube       kubernetes-dashboard   dashboard-metrics-scraper-b74747df5-ngnnr   0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        11Mi (0%)
minikube       kube-system            kube-proxy-8q8b8                            0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        10Mi (0%)
minikube       kube-system            storage-provisioner                         0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        9Mi (0%)
minikube       kube-system            kindnet-g9sfn                               100m (2%)      100m (2%)     1m (0%)       50Mi (0%)         50Mi (0%)       9Mi (0%)

minikube-m03   *                      *                                           1100m (27%)    3100m (77%)   2020m (50%)   50Mi (0%)         50Mi (0%)       870Mi (5%)
minikube-m03   kube-system            kube-proxy-55rb9                            0Mi (0%)       0Mi (0%)      1m (0%)       0Mi (0%)          0Mi (0%)        10Mi (0%)
minikube-m03   kube-system            kindnet-xb2rl                               100m (2%)      100m (2%)     1m (0%)       50Mi (0%)         50Mi (0%)       9Mi (0%)
minikube-m03   default                cpu-demo                                    1000m (25%)    3000m (75%)   2002m (50%)   0Mi (0%)          0Mi (0%)        1Mi (0%)
$
```

合計値を表示できること、`--available` オプションがあることや、ソートも簡単にできるのが良いですね。

### 3.3. kubectl view-allocations

`-g` オプションに表示したいグループ( `node`, `namespace`, `pod` など )を指定でき、
複数指定するとそれらの階層構造にして表示してくれます。

https://github.com/davidB/kubectl-view-allocations

krew によるインストールにも対応しています。

```bash
$ kubectl krew install view-allocations
$ kubectl view-allocations --help
kubectl-view-allocations 0.15.1
https://github.com/davidB/kubectl-view-allocations
kubectl plugin to list allocations (cpu, memory, gpu,... X utilization, requested, limit,
allocatable,...)

USAGE:
    kubectl-view_allocations [OPTIONS]

OPTIONS:
        --context <CONTEXT>
            The name of the kubeconfig context to use

    -g, --group-by <GROUP_BY>
            Group information hierarchically (default: -g resource -g node -g pod) [possible values:
            resource, node, pod, namespace]

    -h, --help
            Print help information

    -n, --namespace <NAMESPACE>
            Show only pods from this namespace

    -o, --output <OUTPUT>
            Output format [default: table] [possible values: table, csv]

    -r, --resource-name <RESOURCE_NAME>
            Filter resources shown by name(s), by default all resources are listed

    -u, --utilization
            Force to retrieve utilization (for cpu and memory), require to have metrics-server
            https://github.com/kubernetes-sigs/metrics-server

    -V, --version
            Print version information

    -z, --show-zero
            Show lines with zero requested and zero limit and zero allocatable
$ 
```

`-z` オプションを付与すると、`Requested`, `Limit`, `Allocatable` が0でも表示します。
`-u` オプションを付与すると cpu と memory の `Utilization` (使用率・量) を表示します。
`-r` オプションに表示したいリソース名を指定します。複数指定可能です。( 省略した場合は全て表示します )。

`-g` オプションに表示したいグループを指定できます。以下を指定可能です。

- `node`
- `namespace`
- `pod`
- `resource`
  - `-r` で指定したリソースが採用されるようです。

ノード単位で表示する例:

```bash
$ kubectl view-allocations -z -u -r cpu -r memory -g node
 Resource           Utilization     Requested         Limit  Allocatable    Free 
  cpu                 (23%) 2.8     (18%) 2.1     (28%) 3.3         12.0     8.7
  ├─ minikube        (1%) 43.0m  (21%) 850.0m   (2%) 100.0m          4.0     3.1
  ├─ minikube-m02  (18%) 701.0m   (2%) 100.0m   (2%) 100.0m          4.0     3.9
  └─ minikube-m03     (50%) 2.0     (30%) 1.2     (78%) 3.1          4.0  900.0m
  memory            (12%) 5.6Gi   (10%) 4.5Gi  (26%) 12.3Gi       46.9Gi  34.6Gi
  ├─ minikube      (3%) 432.6Mi  (1%) 220.0Mi  (1%) 220.0Mi       15.6Gi  15.4Gi 
  ├─ minikube-m02   (33%) 5.2Gi   (26%) 4.0Gi  (77%) 12.0Gi       15.6Gi   3.6Gi
  └─ minikube-m03   (0%) 34.6Mi  (2%) 250.0Mi   (0%) 50.0Mi       15.6Gi  15.4Gi
$
```

namespace単位で表示する例:

```bash
$ kubectl view-allocations -z -u -r cpu -r memory -g namespace
 Resource         Utilization    Requested         Limit  Allocatable    Free 
  cpu               (23%) 2.8    (18%) 2.1     (28%) 3.3         12.0     8.7 
  ├─ default              2.7          1.0           3.0           __      __
  └─ kube-system        51.0m          1.1        300.0m           __      __
  memory          (18%) 8.5Gi  (10%) 4.5Gi  (26%) 12.3Gi       46.9Gi  34.6Gi
  ├─ default            8.0Gi        4.0Gi        12.0Gi           __      __
  └─ kube-system      485.0Mi      520.0Mi       320.0Mi           __      __
```

Pod単位で表示する例:

```bash
$ kubectl view-allocations -z -u -r cpu -r memory -g pod
 Resource                              Utilization    Requested         Limit  Allocatable    Free 
  cpu                                    (23%) 2.8    (18%) 2.1     (28%) 3.3         12.0     8.7
  ├─ coredns-565d847f94-k4px2                 1.0m       100.0m            __           __      __
  ├─ cpu-demo                                  2.0          1.0           3.0           __      __ 
  ├─ etcd-minikube                            9.0m       100.0m            __           __      __
  ├─ kindnet-27j48                            1.0m       100.0m        100.0m           __      __
  ├─ kindnet-42jjk                            1.0m       100.0m        100.0m           __      __
  ├─ kindnet-mlrh6                            1.0m       100.0m        100.0m           __      __
  ├─ kube-apiserver-minikube                 23.0m       250.0m            __           __      __
  ├─ kube-controller-manager-minikube         6.0m       200.0m            __           __      __
  ├─ kube-proxy-6k29q                         1.0m           __            __           __      __
  ├─ kube-proxy-ckf4w                         1.0m           __            __           __      __
  ├─ kube-proxy-dt99v                         1.0m           __            __           __      __
  ├─ kube-scheduler-minikube                  1.0m       100.0m            __           __      __
  ├─ memory-demo                            699.0m           __            __           __      __
  ├─ metrics-server-769cd898cd-s24s6          4.0m       100.0m            __           __      __
  └─ storage-provisioner                      1.0m           __            __           __      __
  memory                               (12%) 5.6Gi  (10%) 4.5Gi  (26%) 12.3Gi       46.9Gi  34.6Gi
  ├─ coredns-565d847f94-k4px2               12.9Mi       70.0Mi       170.0Mi           __      __
  ├─ cpu-demo                                1.0Mi           __            __           __      __
  ├─ etcd-minikube                          50.7Mi      100.0Mi            __           __      __
  ├─ kindnet-27j48                           8.4Mi       50.0Mi        50.0Mi           __      __
  ├─ kindnet-42jjk                           8.3Mi       50.0Mi        50.0Mi           __      __
  ├─ kindnet-mlrh6                           8.2Mi       50.0Mi        50.0Mi           __      __
  ├─ kube-apiserver-minikube               286.4Mi           __            __           __      __
  ├─ kube-controller-manager-minikube       40.5Mi           __            __           __      __
  ├─ kube-proxy-6k29q                        9.8Mi           __            __           __      __
  ├─ kube-proxy-ckf4w                        9.5Mi           __            __           __      __
  ├─ kube-proxy-dt99v                        9.5Mi           __            __           __      __
  ├─ kube-scheduler-minikube                15.4Mi           __            __           __      __
  ├─ memory-demo                             5.2Gi        4.0Gi        12.0Gi           __      __
  ├─ metrics-server-769cd898cd-s24s6        15.8Mi      200.0Mi            __           __      __
  └─ storage-provisioner                     8.6Mi           __            __           __      __
$
```

`-g` オプションを複数指定した場合、指定した順番で階層表示します。
ノード - Podの階層表示とする例( `-g node -g pod` を指定 ):

```bash
$ kubectl view-allocations -z -u -r cpu -r memory -g node -g pod
 Resource                                  Utilization     Requested         Limit  Allocatable    Free 
  cpu                                        (23%) 2.8     (18%) 2.1     (28%) 3.3         12.0     8.7
  ├─ minikube                               (1%) 43.0m  (21%) 850.0m   (2%) 100.0m          4.0     3.1
  │  ├─ coredns-565d847f94-k4px2                  1.0m        100.0m            __           __      __ 
  │  ├─ etcd-minikube                             9.0m        100.0m            __           __      __
  │  ├─ kindnet-42jjk                             1.0m        100.0m        100.0m           __      __
  │  ├─ kube-apiserver-minikube                  23.0m        250.0m            __           __      __
  │  ├─ kube-controller-manager-minikube          6.0m        200.0m            __           __      __
  │  ├─ kube-proxy-6k29q                          1.0m            __            __           __      __
  │  ├─ kube-scheduler-minikube                   1.0m        100.0m            __           __      __
  │  └─ storage-provisioner                       1.0m            __            __           __      __
  ├─ minikube-m02                         (18%) 701.0m   (2%) 100.0m   (2%) 100.0m          4.0     3.9
  │  ├─ kindnet-27j48                             1.0m        100.0m        100.0m           __      __
  │  ├─ kube-proxy-dt99v                          1.0m            __            __           __      __
  │  └─ memory-demo                             699.0m            __            __           __      __
  └─ minikube-m03                            (50%) 2.0     (30%) 1.2     (78%) 3.1          4.0  900.0m
     ├─ cpu-demo                                   2.0           1.0           3.0           __      __
     ├─ kindnet-mlrh6                             1.0m        100.0m        100.0m           __      __
     ├─ kube-proxy-ckf4w                          1.0m            __            __           __      __
     └─ metrics-server-769cd898cd-s24s6           4.0m        100.0m            __           __      __
  memory                                   (12%) 5.6Gi   (10%) 4.5Gi  (26%) 12.3Gi       46.9Gi  34.6Gi
  ├─ minikube                             (3%) 431.8Mi  (1%) 220.0Mi  (1%) 220.0Mi       15.6Gi  15.4Gi
  │  ├─ coredns-565d847f94-k4px2                12.9Mi        70.0Mi       170.0Mi           __      __
  │  ├─ etcd-minikube                           50.0Mi       100.0Mi            __           __      __
  │  ├─ kindnet-42jjk                            8.2Mi        50.0Mi        50.0Mi           __      __
  │  ├─ kube-apiserver-minikube                285.7Mi            __            __           __      __
  │  ├─ kube-controller-manager-minikube        40.5Mi            __            __           __      __
  │  ├─ kube-proxy-6k29q                         9.8Mi            __            __           __      __
  │  ├─ kube-scheduler-minikube                 15.5Mi            __            __           __      __
  │  └─ storage-provisioner                      9.1Mi            __            __           __      __
  ├─ minikube-m02                          (33%) 5.1Gi   (26%) 4.0Gi  (77%) 12.0Gi       15.6Gi   3.6Gi
  │  ├─ kindnet-27j48                            8.3Mi        50.0Mi        50.0Mi           __      __
  │  ├─ kube-proxy-dt99v                         9.5Mi            __            __           __      __
  │  └─ memory-demo                              5.1Gi         4.0Gi        12.0Gi           __      __
  └─ minikube-m03                          (0%) 34.5Mi  (2%) 250.0Mi   (0%) 50.0Mi       15.6Gi  15.4Gi
     ├─ cpu-demo                                 1.0Mi            __            __           __      __
     ├─ kindnet-mlrh6                            8.4Mi        50.0Mi        50.0Mi           __      __
     ├─ kube-proxy-ckf4w                         9.5Mi            __            __           __      __
     └─ metrics-server-769cd898cd-s24s6         15.6Mi       200.0Mi            __           __      __
$
```

namespace - Podの階層表示とする例( `-g namespace -g pod` を指定 ):

```bash
$  kubectl view-allocations -z -u -r cpu -r memory -g namespace -g pod
 Resource                                 Utilization    Requested         Limit  Allocatable    Free 
  cpu                                       (23%) 2.8    (18%) 2.1     (28%) 3.3         12.0     8.7
  ├─ default                                      2.7          1.0           3.0           __      __
  │  ├─ cpu-demo                                  2.0          1.0           3.0           __      __
  │  └─ memory-demo                            699.0m           __            __           __      __
  └─ kube-system                                51.0m          1.1        300.0m           __      __
     ├─ coredns-565d847f94-k4px2                 1.0m       100.0m            __           __      __ 
     ├─ etcd-minikube                            9.0m       100.0m            __           __      __
     ├─ kindnet-27j48                            1.0m       100.0m        100.0m           __      __
     ├─ kindnet-42jjk                            1.0m       100.0m        100.0m           __      __
     ├─ kindnet-mlrh6                            1.0m       100.0m        100.0m           __      __
     ├─ kube-apiserver-minikube                 23.0m       250.0m            __           __      __
     ├─ kube-controller-manager-minikube         6.0m       200.0m            __           __      __
     ├─ kube-proxy-6k29q                         1.0m           __            __           __      __
     ├─ kube-proxy-ckf4w                         1.0m           __            __           __      __
     ├─ kube-proxy-dt99v                         1.0m           __            __           __      __
     ├─ kube-scheduler-minikube                  1.0m       100.0m            __           __      __
     ├─ metrics-server-769cd898cd-s24s6          4.0m       100.0m            __           __      __
     └─ storage-provisioner                      1.0m           __            __           __      __
  memory                                  (17%) 7.8Gi  (10%) 4.5Gi  (26%) 12.3Gi       46.9Gi  34.6Gi
  ├─ default                                    7.3Gi        4.0Gi        12.0Gi           __      __
  │  ├─ cpu-demo                                1.0Mi           __            __           __      __
  │  └─ memory-demo                             7.3Gi        4.0Gi        12.0Gi           __      __
  └─ kube-system                              483.0Mi      520.0Mi       320.0Mi           __      __
     ├─ coredns-565d847f94-k4px2               12.7Mi       70.0Mi       170.0Mi           __      __
     ├─ etcd-minikube                          50.4Mi      100.0Mi            __           __      __
     ├─ kindnet-27j48                           8.4Mi       50.0Mi        50.0Mi           __      __
     ├─ kindnet-42jjk                           8.2Mi       50.0Mi        50.0Mi           __      __
     ├─ kindnet-mlrh6                           8.2Mi       50.0Mi        50.0Mi           __      __
     ├─ kube-apiserver-minikube               285.8Mi           __            __           __      __
     ├─ kube-controller-manager-minikube       40.5Mi           __            __           __      __
     ├─ kube-proxy-6k29q                        9.8Mi           __            __           __      __
     ├─ kube-proxy-ckf4w                        9.5Mi           __            __           __      __
     ├─ kube-proxy-dt99v                        9.5Mi           __            __           __      __
     ├─ kube-scheduler-minikube                15.4Mi           __            __           __      __
     ├─ metrics-server-769cd898cd-s24s6        15.6Mi      200.0Mi            __           __      __
     └─ storage-provisioner                     8.9Mi           __            __           __      __
$
```

`-o` オプションに `csv` を指定すると CSVフォーマットで出力します。
他のツールに読み込ませて加工したい場合に便利です。

```bash
$  kubectl view-allocations -z -u -r cpu -r memory -g node -g pod -o csv
Date,Kind,resource,node,pod,Utilization,%Utilization,Requested,%Requested,Limit,%Limit,Allocatable,Free
2022-12-14T14:34:40.680994548+00:00,resource,cpu,,,2.75,23%,2.15,18%,3.30,28%,12.00,8.70
2022-12-14T14:34:40.680994548+00:00,node,cpu,minikube,,0.04,1%,0.85,21%,0.10,2%,4.00,3.15
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube,coredns-565d847f94-k4px2,0.00,,0.10,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube,etcd-minikube,0.01,,0.10,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube,kindnet-42jjk,0.00,,0.10,,0.10,,,
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube,kube-apiserver-minikube,0.02,,0.25,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube,kube-controller-manager-minikube,0.01,,0.20,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube,kube-proxy-6k29q,0.00,,,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube,kube-scheduler-minikube,0.00,,0.10,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube,storage-provisioner,0.00,,,,,,,
2022-12-14T14:34:40.680994548+00:00,node,cpu,minikube-m02,,0.70,17%,0.10,2%,0.10,2%,4.00,3.90
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube-m02,kindnet-27j48,0.00,,0.10,,0.10,,,
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube-m02,kube-proxy-dt99v,0.00,,,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube-m02,memory-demo,0.70,,,,,,,
2022-12-14T14:34:40.680994548+00:00,node,cpu,minikube-m03,,2.01,50%,1.20,30%,3.10,78%,4.00,0.90
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube-m03,cpu-demo,2.00,,1.00,,3.00,,,
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube-m03,kindnet-mlrh6,0.00,,0.10,,0.10,,,
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube-m03,kube-proxy-ckf4w,0.00,,,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,cpu,minikube-m03,metrics-server-769cd898cd-s24s6,0.00,,0.10,,,,,
2022-12-14T14:34:40.680994548+00:00,resource,memory,,,9114296320.00,18%,4840226816.00,10%,13220446208.00,26%,50320109568.00,37099663360.00
2022-12-14T14:34:40.680994548+00:00,node,memory,minikube,,452694016.00,3%,230686720.00,1%,230686720.00,1%,16773369856.00,16542683136.00
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube,coredns-565d847f94-k4px2,13373440.00,,73400320.00,,178257920.00,,,
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube,etcd-minikube,52965376.00,,104857600.00,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube,kindnet-42jjk,8445952.00,,52428800.00,,52428800.00,,,
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube,kube-apiserver-minikube,299843584.00,,,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube,kube-controller-manager-minikube,42496000.00,,,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube,kube-proxy-6k29q,10272768.00,,,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube,kube-scheduler-minikube,16146432.00,,,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube,storage-provisioner,9150464.00,,,,,,,
2022-12-14T14:34:40.680994548+00:00,node,memory,minikube-m02,,8625549312.00,51%,4347396096.00,26%,12937330688.00,77%,16773369856.00,3836039168.00   
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube-m02,kindnet-27j48,8568832.00,,52428800.00,,52428800.00,,,
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube-m02,kube-proxy-dt99v,9932800.00,,,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube-m02,memory-demo,8607047680.00,,4294967296.00,,12884901888.00,,,
2022-12-14T14:34:40.680994548+00:00,node,memory,minikube-m03,,36052992.00,0%,262144000.00,2%,52428800.00,0%,16773369856.00,16511225856.00
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube-m03,cpu-demo,1093632.00,,,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube-m03,kindnet-mlrh6,8445952.00,,52428800.00,,52428800.00,,,
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube-m03,kube-proxy-ckf4w,9986048.00,,,,,,,
2022-12-14T14:34:40.680994548+00:00,pod,memory,minikube-m03,metrics-server-769cd898cd-s24s6,16527360.00,,209715200.00,,,,,
```

`-r` オプションで CPUのみ、メモリのみ、もしくは複数など、表示したいリソースを指定できたり、
`-g` オプションで階層構造を指定して表示できるのも良いですね。

記事執筆時点で、ソートできるオプションは見当たりませんでした。
Issueで会話はされているようです。

https://github.com/davidB/kubectl-view-allocations/issues/177

### 3.4. kubectl ktop プラグイン

CPU や メモリの 使用量・率を グラフィカルに表示してくれる kubectl プラグインです。

https://github.com/vladimirvivien/ktop

表示形式は [htop](https://htop.dev/) コマンドの表示に似ています。

![ktop](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/21106/55b5c3f8-4f02-0c91-db3b-14edfe331eaa.png)

上段にクラスタの情報、中断にノードの情報、下段にPodの情報が表示されます。
リソース使用量が変化すると縦棒のグラフ( `[|||||   ]` )がニョキニョキ動きます。

縦棒のグラフを見ると、ノード `minikube-m02` ( `memory-demo` がデプロイされている ) の メモリ使用量・率や、
ノード `minikube-m03` ( `cpu-demo` がデプロイされている )の CPU使用量・率が比較的高いことが把握できます。

インストール方法はいくつか用意されており、krewでインストールすることもできます。

```bash
$ kubectl krew install ktop
$ kubectl ktop --help
Runs kubectl-ktop as kubectl plugin

Usage:
  kubectl-ktop [flags]

Examples:

# Start ktop using default configuration for the "default" namespace
kubectl-ktop

# Start ktop with default configuration for all accessible namespaces
kubectl-ktop -A

# Start ktop for a specific namespace in current context
kubectl-ktop --namespace <namespace>

# Start ktop for a specific namespace and context
kubectl-ktop --namespace <namespace> --context <context>
( 略 )
$ kubectl ktop -A
```

ポテンシャルを感じるプラグインなのですが、Pod数が多い場合、画面に収まりきれないPodがあります。
`--namespace` オプションを付与するなどして絞り込む必要がありそうです。

記事執筆時点で、ソートできるオプションは見当たりませんでした。
Issueで会話はされているようです。

https://github.com/vladimirvivien/ktop/issues/28

## まとめ

CPUやメモリの要求(`Requests`)、制限(`Limits`)や、使用量・使用率を確認するのに役立つコマンドをご紹介しました。
リソースの使用状況を素早く把握するのにお役に立てましたら幸いです。

ご紹介した中では、`kubectl resource-capacity` プラグイン ( [robscott/kube-capacity](https://github.com/robscott/kube-capacity) ) が便利な印象です。
また、他にも便利な活用方法や、本記事でご紹介出来ていないツールやコマンドもあるかと思います。
お気づきの点がございましたらコメント等で教えていただけますと喜びます。