Openshift 4.6のMaster/Worker Nodeのリソース消費量について

本記事はOpenshift 4.6のMaster/Worker Nodeのリソース消費量について実際のコマンド出力と共に簡易的にまとめたものです。
OpenshiftクラスターをデプロイしOpenshift image registryをManagedに設定した位の状態です。
サイジングする上でCoreOSやシステム系Podが予約・消費しているリソース量について考慮漏れになることがあるため、参考程度に載せておきます。

## 構成

- vSphere UPIデプロイパターン
- Master 3 Node/Worker 2 Node構成

```bash
[root@prov ~]# oc get nodes
NAME      STATUS   ROLES        AGE   VERSION
master1   Ready    master       45d   v1.19.14+fcff70a
master2   Ready    master       45d   v1.19.14+fcff70a
master3   Ready    master       45d   v1.19.14+fcff70a
worker1   Ready    app,worker   45d   v1.19.14+fcff70a
worker2   Ready    app,worker   45d   v1.19.14+fcff70a
[root@prov ~]# oc version
Client Version: 4.7.34
Server Version: 4.6.49
Kubernetes Version: v1.19.14+fcff70a
[root@prov ~]#
```

- 各Master Nodeへのリソース搭載量
    - CPU：8
    - MEM：8GB
    - Disk：100GB

- 各Worker Nodeへのリソース搭載量
    - CPU：8
    - MEM：8GB
    - Disk：100GB

※上記リソース量は適当に割り当てており、管理するノード数やデプロイするOperatorやアプリケーションPod数により大きく変わるため、Red Hatのサイトで適切なリソース量を計算すること

## oc describeコマンド抜粋

- Master Node

```bash
[root@prov ~]# oc describe node master1
Name:               master1
Roles:              master
Capacity:
  cpu:                8
  ephemeral-storage:  104305644Ki
  hugepages-2Mi:      0
  memory:             8160732Ki
  pods:               250
Allocatable:
  cpu:                7500m
  ephemeral-storage:  95054339528
  hugepages-2Mi:      0
  memory:             7009756Ki
  pods:               250
ProviderID:                               vsphere://423d9531-0c8e-1964-5d5d-8f35fc223822
Non-terminated Pods:                      (31 in total)
  Namespace                               Name                                            CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                               ----                                            ------------  ----------  ---------------  -------------  ---
  openshift-apiserver                     apiserver-5648856c64-zlx95                      110m (1%)     0 (0%)      250Mi (3%)       0 (0%)         45d
  openshift-authentication                oauth-openshift-7565c7b94d-2kt7n                10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         45d
  openshift-cluster-node-tuning-operator  tuned-nlfq5                                     10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         45d
  openshift-cluster-samples-operator      cluster-samples-operator-7dbc7958f-6ltt8        20m (0%)      0 (0%)      0 (0%)           0 (0%)         45d
  openshift-cluster-storage-operator      csi-snapshot-controller-5458888996-4zlbd        10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         45d
  openshift-console                       downloads-9776476c4-pd6xm                       10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         45d
  openshift-controller-manager            controller-manager-qdzgz                        100m (1%)     0 (0%)      100Mi (1%)       0 (0%)         47h
  openshift-dns                           dns-default-4tbbp                               65m (0%)      0 (0%)      110Mi (1%)       512Mi (7%)     45d
  openshift-etcd                          etcd-master1                                    430m (5%)     0 (0%)      860Mi (12%)      0 (0%)         45d
  openshift-etcd                          etcd-quorum-guard-78d56fdc5b-lnc77              10m (0%)      0 (0%)      5Mi (0%)         0 (0%)         45d
  openshift-image-registry                node-ca-j6cv5                                   10m (0%)      0 (0%)      10Mi (0%)        0 (0%)         45d
  openshift-kube-apiserver-operator       kube-apiserver-operator-6cc8455974-nldsv        10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         45d
  openshift-kube-apiserver                kube-apiserver-master1                          340m (4%)     0 (0%)      1224Mi (17%)     0 (0%)         47h
  openshift-kube-controller-manager       kube-controller-manager-master1                 100m (1%)     0 (0%)      500Mi (7%)       0 (0%)         45d
  openshift-kube-scheduler                openshift-kube-scheduler-master1                25m (0%)      0 (0%)      150Mi (2%)       0 (0%)         45d
  openshift-machine-api                   cluster-autoscaler-operator-69876758dd-6bhfw    30m (0%)      0 (0%)      70Mi (1%)        0 (0%)         45d
  openshift-machine-api                   machine-api-controllers-654d4df8c9-q8plc        70m (0%)      0 (0%)      140Mi (2%)       0 (0%)         45d
  openshift-machine-api                   machine-api-operator-5ffdfc86c-fk5ct            20m (0%)      0 (0%)      70Mi (1%)        0 (0%)         45d
  openshift-machine-config-operator       machine-config-controller-656f69c9bb-842xs      20m (0%)      0 (0%)      50Mi (0%)        0 (0%)         45d
  openshift-machine-config-operator       machine-config-daemon-qh79g                     40m (0%)      0 (0%)      100Mi (1%)       0 (0%)         45d
  openshift-machine-config-operator       machine-config-server-cz6zv                     20m (0%)      0 (0%)      50Mi (0%)        0 (0%)         45d
  openshift-monitoring                    node-exporter-hlgvc                             9m (0%)       0 (0%)      210Mi (3%)       0 (0%)         45d
  openshift-multus                        multus-5mb6t                                    10m (0%)      0 (0%)      150Mi (2%)       0 (0%)         45d
  openshift-multus                        multus-admission-controller-kn29z               20m (0%)      0 (0%)      20Mi (0%)        0 (0%)         45d
  openshift-multus                        network-metrics-daemon-c8vgz                    20m (0%)      0 (0%)      120Mi (1%)       0 (0%)         45d
  openshift-oauth-apiserver               apiserver-698fdcc744-rqnh5                      150m (2%)     0 (0%)      200Mi (2%)       0 (0%)         45d
  openshift-operator-lifecycle-manager    packageserver-75b9bddfdd-blt8s                  10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         45d
  openshift-sdn                           ovs-fvnpm                                       100m (1%)     0 (0%)      400Mi (5%)       0 (0%)         45d
  openshift-sdn                           sdn-controller-x2csg                            10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         45d
  openshift-sdn                           sdn-d5jvw                                       110m (1%)     0 (0%)      220Mi (3%)       0 (0%)         45d
  openshift-service-ca                    service-ca-5ff99fbc4c-zs9w9                     10m (0%)      0 (0%)      120Mi (1%)       0 (0%)         45d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                1909m (25%)   0 (0%)
  memory             5479Mi (80%)  512Mi (7%)
  ephemeral-storage  0 (0%)        0 (0%)
  hugepages-2Mi      0 (0%)        0 (0%)
```

- Worker Node

```bash
[root@prov ~]# oc describe node worker1
Name:               worker1
Roles:              app,worker
Capacity:
  cpu:                8
  ephemeral-storage:  104305644Ki
  hugepages-2Mi:      0
  memory:             10225132Ki
  pods:               250
Allocatable:
  cpu:                7500m
  ephemeral-storage:  95054339528
  hugepages-2Mi:      0
  memory:             9074156Ki
  pods:               250
ProviderID:                                vsphere://423da5a4-0e4e-35f5-07a5-e2bf8e003d88
Non-terminated Pods:                       (21 in total)
  Namespace                                Name                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                                ----                                        ------------  ----------  ---------------  -------------  ---
  openshift-cluster-node-tuning-operator   tuned-7b2mg                                 10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         45d
  openshift-dns                            dns-default-xftxt                           65m (0%)      0 (0%)      110Mi (1%)       512Mi (5%)     45d
  openshift-image-registry                 image-registry-549fc8689d-j9cmz             100m (1%)     0 (0%)      256Mi (2%)       0 (0%)         28d
  openshift-image-registry                 node-ca-rfxtk                               10m (0%)      0 (0%)      10Mi (0%)        0 (0%)         45d
  openshift-ingress                        router-default-76fd9bbc66-ldqld             100m (1%)     0 (0%)      256Mi (2%)       0 (0%)         28d
  openshift-kube-storage-version-migrator  migrator-68bc7c89f-9wx22                    100m (1%)     0 (0%)      200Mi (2%)       0 (0%)         28d
  openshift-machine-config-operator        machine-config-daemon-ts248                 40m (0%)      0 (0%)      100Mi (1%)       0 (0%)         45d
  openshift-monitoring                     alertmanager-main-0                         8m (0%)       0 (0%)      270Mi (3%)       0 (0%)         47h
  openshift-monitoring                     grafana-575fbc5464-n4dz9                    5m (0%)       0 (0%)      120Mi (1%)       0 (0%)         28d
  openshift-monitoring                     kube-state-metrics-69b4fbb79-n4s5s          4m (0%)       0 (0%)      120Mi (1%)       0 (0%)         28d
  openshift-monitoring                     node-exporter-rs75q                         9m (0%)       0 (0%)      210Mi (2%)       0 (0%)         45d
  openshift-monitoring                     openshift-state-metrics-7fb987f599-4vgxr    3m (0%)       0 (0%)      190Mi (2%)       0 (0%)         28d
  openshift-monitoring                     prometheus-adapter-84dc49f498-8p24m         1m (0%)       0 (0%)      25Mi (0%)        0 (0%)         47h
  openshift-monitoring                     prometheus-k8s-1                            75m (1%)      0 (0%)      1194Mi (13%)     0 (0%)         47h
  openshift-monitoring                     telemeter-client-97449bd78-rp7b4            3m (0%)       0 (0%)      20Mi (0%)        0 (0%)         28d
  openshift-monitoring                     thanos-querier-69875575db-fsc9d             9m (0%)       0 (0%)      92Mi (1%)        0 (0%)         28d
  openshift-monitoring                     thanos-querier-69875575db-hmw7g             9m (0%)       0 (0%)      92Mi (1%)        0 (0%)         28d
  openshift-multus                         multus-hmwxt                                10m (0%)      0 (0%)      150Mi (1%)       0 (0%)         45d
  openshift-multus                         network-metrics-daemon-z6tsz                20m (0%)      0 (0%)      120Mi (1%)       0 (0%)         45d
  openshift-sdn                            ovs-lbfnl                                   100m (1%)     0 (0%)      400Mi (4%)       0 (0%)         45d
  openshift-sdn                            sdn-qxvxk                                   110m (1%)     0 (0%)      220Mi (2%)       0 (0%)         45d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                791m (10%)    0 (0%)
  memory             4205Mi (47%)  512Mi (5%)
  ephemeral-storage  0 (0%)        0 (0%)
  hugepages-2Mi      0 (0%)        0 (0%)
[root@prov ~]#
```

## リソース考慮点

- 各ノード、500mのCPUリソースが予約されている。（恐らくCoreOS分）
    - Openshiftは 1Core = 1000m で計算
    - Allocatable にて 7500m となっている。(8000m - 500m = 7500m)
- 各ノード、約1GBのMEMリソースが予約されている。（恐らくCoreOS分）
- リソースクォータ値について
    - 上記出力結果より、CPUクォータに Master Nodeは1909m、Worker Nodeは791mが設定されている。
    - 上記出力結果より、MEMクォータに Master Nodeは5479Mi、Worker Nodeは4205Miが設定されている。
- デプロイするOperatorによりリソース消費量は大きく変わるため余裕を持ったサイジングが必要
    - Openshift Logging(Elasticsearch/Fluentd/Kibana) Operatorをデプロイすると更にリソース予約・消費量が上昇する。
        - ログ流量によってリソース消費量が大きく異なるため要注意
        - Prometheusについては収集対象の数により異なるためアプリケーションコンテナが多くなる環境は要注意
    - Compliance、File Integrity Operatorなどは全てのNodeでPodが起動するため要注意（インフラノードを用意してもスキャン用Podが起動する）
   
Openshiftには様々なOperatorが存在しており、どれだけのリソースが消費されるかは不明であるため、本番環境への利用を検討されている場合はPoCを実施するのが無難である。


以上