OpenShift 4.4を構築してアプリを動かしてみる - OpenShift構築編 -

bootstrapのVMを起動した後、masterのVMとworkerのVMを順次起動させて、OpenShiftを構築します。

## 1. bootstrapの起動
bootstrapのVMを起動します。
正常に起動すると、 OpenShiftのVersion 4.4とipアドレス:172.16.0.100が表示されます。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/658425/6c208324-60c4-5f5d-4283-ccc1b4434e4e.png)
### SELinux : mount invalid
放置すると、このように大量のSELinuxのメッセージが表示されますが無視して大丈夫です。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/658425/94b2ac3e-70c9-25d8-b84f-0f84b6f700ae.png)

## 2. クラスターの作成
### 2.1 bootstrapにsshログイン
踏み台(bastion)からbootsrapにログインします。
sshのidentityファイルは秘密鍵の`new_rsa`を使用します。
(公開鍵の`new_rsa.pub`は使用しません.)

```
[root@bastion ~]# ssh -i ~/.ssh/new_rsa core@bootstrap
Red Hat Enterprise Linux CoreOS 44.82.202008011133-0
  Part of OpenShift 4.4, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.4/architecture/architecture-rhcos.html

---
This is the bootstrap node; it will be destroyed when the master is fully up.

The primary service is "bootkube.service". To watch its status, run e.g.

  journalctl -b -f -u bootkube.service
```
### 2.2 クラスター作成状況
ssh ログイン後のメッセージに書かれているコマンドをそのまま実行します。
(監視したまま、3.master起動、4.worker起動を実施してください。)

```
[core@bootstrap ~]$ journalctl -b -f -u bootkube.service
```

## 3. master起動
master-0、master-1、master-2を順次起動していきます。

- master-0
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/658425/8419d9ea-2cef-2d8d-95be-565c05a17b9b.png)

- master-1
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/658425/792205d4-428a-735d-e158-7b241d4a7daa.png)

- master-2
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/658425/9bb18341-29d4-3fda-4b37-18196c4457a6.png)

### 起動時のconnection refused
以下の画面のように起動時にconnection refusedと表示され#6で止まる場合がありますが
気長に待っていると起動します。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/658425/02bb7410-559e-dab3-9a2c-43dbeb98955f.png)


## 4. worker起動
worker-0、worker-1を順次起動していきます。

- worker-0
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/658425/f67b3c36-28b3-af45-6a94-0a6f07e95aec.png)

- worker-1
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/658425/646d5460-590d-f723-578a-2c6c834ee11d.png)

### 補足
- bootkube.service complete完了後のworker起動
　数時間たっても、nodesに追加されない場合は、workerのVMの起動が遅かった可能性があります。
　残念ですが、bootstrap、master、workerすべてのVMを削除しbastionの端末でからマニフェストとingitionファイルの再生成からやり直すのが確実です。

- workerの起動が早い場合
workerの初回起動中に500エラーが発生します。
Workerの起動するタイミングは、以下のbootstrapのjournalctlでモニタリングログで `failed to create some manifests:` のメッセージが表示されなくなったタイミングが目安です。  
workerを起動タイミングは、bastionの端末から以下のコマンドで、正確に確認することもできます。

```
[root@bastion #] curl -k https://bootstrap:22623/config/worker
```

```
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma                                                                               ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo                                                                               rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: [#701] failed to create some manifests:
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma                                                                               ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo                                                                               rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:52 bootstrap bootkube.sh[2577]:         Pod Status:openshift-cluster-version/cluster-version-operator        Ready
Feb 26 05:42:52 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-apiserver/kube-apiserver        DoesNotExist
Feb 26 05:42:52 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-scheduler/openshift-kube-scheduler        DoesNotExist
Feb 26 05:42:52 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-controller-manager/kube-controller-manager        DoesNotExist
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: [#702] failed to create some manifests:
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma                                                                               ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:52 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo                                                                               rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:53 bootstrap bootkube.sh[2577]: [#703] failed to create some manifests:
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: [#723] failed to create some manifests:
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma                                                                               ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo                                                                               rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: [#724] failed to create some manifests:
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma                                                                               ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo                                                                               rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: [#725] failed to create some manifests:
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma                                                                               ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo                                                                               rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: [#726] failed to create some manifests:
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-ma                                                                               ster-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-wo                                                                               rker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: Created "99_openshift-machineconfig_99-master-ssh.yaml" machineconfigs.v1.machineconfiguration.openshift.io/99-ma                                                                               ster-ssh -n
Feb 26 05:42:57 bootstrap bootkube.sh[2577]: Created "99_openshift-machineconfig_99-worker-ssh.yaml" machineconfigs.v1.machineconfiguration.openshift.io/99-wo                                                                               rker-ssh -n
Feb 26 05:43:27 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-apiserver/kube-apiserver        DoesNotExist
Feb 26 05:43:27 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-scheduler/openshift-kube-scheduler        Pending
Feb 26 05:43:27 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-controller-manager/kube-controller-manager        DoesNotExist
Feb 26 05:43:27 bootstrap bootkube.sh[2577]:         Pod Status:openshift-cluster-version/cluster-version-operator        Ready
Feb 26 05:43:42 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-controller-manager/kube-controller-manager        DoesNotExist
Feb 26 05:43:42 bootstrap bootkube.sh[2577]:         Pod Status:openshift-cluster-version/cluster-version-operator        Ready
Feb 26 05:43:42 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-apiserver/kube-apiserver        DoesNotExist
Feb 26 05:43:42 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-scheduler/openshift-kube-scheduler        RunningNotReady
Feb 26 05:43:52 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-apiserver/kube-apiserver        DoesNotExist
Feb 26 05:43:52 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-scheduler/openshift-kube-scheduler        Pending
Feb 26 05:43:52 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-controller-manager/kube-controller-manager        DoesNotExist
Feb 26 05:43:52 bootstrap bootkube.sh[2577]:         Pod Status:openshift-cluster-version/cluster-version-operator        Ready
Feb 26 05:43:57 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-apiserver/kube-apiserver        DoesNotExist
Feb 26 05:43:57 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-scheduler/openshift-kube-scheduler        Pending
Feb 26 05:43:57 bootstrap bootkube.sh[2577]:         Pod Status:openshift-kube-controller-manager/kube-controller-manager        Pending

```
## 5. クラスター作成完了
### 5.1. bootkube.service complete
すべてのmasterとworkerを起動後、2.2のbootstrapのbootkube.serviceを確認し、`bootkube.service complete`と表示されればクラスターの作成が完了です。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/658425/e5c8fc2f-575c-a25c-7fe7-4d250de6bace.png)

### 5.2. bootsrapからsshログアウト
bootsrapのsshから`exist`してsshログアウトし、bastionに戻ります。

```
[core@bootstrap ~]$ exit
logout
Connection to bootstrap closed.
[root@bastion ~]#
```

### 5.3. openshift-installer から確認

```
# cd /opt/ocp
# /usr/local/bin/openshift-install --dir=bare-metal wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.4.31
DEBUG Built from commit 0227b5f653786d8d58312cd08a2e924e72ae646f
INFO Waiting up to 20m0s for the Kubernetes API at https://api.lab.openshift.local:6443...
INFO API v1.17.1+40d7dbd up
INFO Waiting up to 40m0s for bootstrapping to complete...
DEBUG Bootstrap status: complete
INFO It is now safe to remove the bootstrap resources
```

## 6. マシンの認証
クラスターが作成された後は、各マシーンのCSRを承認します。

### 6.1 OpenShiftにログイン
踏み台サーバ(bastion)からOpenShiftにログインします。
`system:admin`と表示されれば成功です。

```
# export KUBECONFIG=/opt/ocp/bare-metal/auth/kubeconfig
# /usr/local/bin/oc whoami
system:admin
```

### 6.2. nodesの確認
以下のコマンドで、`master-0`,`master-1`,`master-2`,`worker-0`,`worker-1`の5台が認識されているか確認してください。

```
# /usr/local/bin/oc get nodes -o wide
NAME       STATUS   ROLES    AGE     VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
master-0   Ready    master   10m     v1.17.1+40d7dbd   172.16.0.110   <none>        Red Hat Enterprise Linux CoreOS 44.82.202011202131-0 (Ootpa)   4.18.0-193.29.1.el8_2.x86_64   cri-o://1.17.5-11.rhaos4.4.git7f979af.el8
master-1   Ready    master   9m1s    v1.17.1+40d7dbd   172.16.0.111   <none>        Red Hat Enterprise Linux CoreOS 44.82.202011202131-0 (Ootpa)   4.18.0-193.29.1.el8_2.x86_64   cri-o://1.17.5-11.rhaos4.4.git7f979af.el8
master-2   Ready    master   4m6s    v1.17.1+40d7dbd   172.16.0.112   <none>        Red Hat Enterprise Linux CoreOS 44.82.202011202131-0 (Ootpa)   4.18.0-193.29.1.el8_2.x86_64   cri-o://1.17.5-11.rhaos4.4.git7f979af.el8
worker-0   Ready    worker   5m53s   v1.17.1+40d7dbd   172.16.0.120   <none>        Red Hat Enterprise Linux CoreOS 44.82.202011202131-0 (Ootpa)   4.18.0-193.29.1.el8_2.x86_64   cri-o://1.17.5-11.rhaos4.4.git7f979af.el8
worker-1   Ready    worker   3m46s   v1.17.1+40d7dbd   172.16.0.121   <none>        Red Hat Enterprise Linux CoreOS 44.82.202011202131-0 (Ootpa)   4.18.0-193.29.1.el8_2.x86_64   cri-o://1.17.5-11.rhaos4.4.git7f979af.el8
```

## 7.マシンCSR承認
### 7.1 CSRの確認
`Pending`になっているCSRを確認します。

```
# /usr/local/bin/oc get csr
NAME        AGE   REQUESTOR                                                                   CONDITION
csr-78qq4   24m   system:node:worker-0                                                        Approved,Issued
csr-9dnw9   27m   system:node:master-1                                                        Approved,Issued
csr-bnvd7   29m   system:node:master-0                                                        Approved,Issued
csr-dvcww   28m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-lqc5j   22m   system:node:worker-1                                                        Approved,Issued
csr-mqt2m   22m   system:node:master-2                                                        Approved,Issued
csr-n7s7d   23m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-rhhkt   29m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-t5p5j   25m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-xwfvc   22m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
```

### 7.2 CSRの承認
`pending`になっているCSRのNameを指定して、承認します。

```
# /usr/local/bin/oc adm certificate approve csr-dvcww
certificatesigningrequest.certificates.k8s.io/csr-dvcww approved

# /usr/local/bin/oc adm certificate approve csr-rhhkt
certificatesigningrequest.certificates.k8s.io/csr-rhhkt approved
```
### 7.3 CSRの確認
すべて、`Approved, Issued`になったことを確認します。

```
# /usr/local/bin/oc get csr
NAME        AGE   REQUESTOR                                                                   CONDITION
csr-78qq4   24m   system:node:worker-0                                                        Approved,Issued
csr-9dnw9   27m   system:node:master-1                                                        Approved,Issued
csr-bnvd7   29m   system:node:master-0                                                        Approved,Issued
csr-dvcww   28m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-lqc5j   22m   system:node:worker-1                                                        Approved,Issued
csr-mqt2m   22m   system:node:master-2                                                        Approved,Issued
csr-n7s7d   23m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-rhhkt   29m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-t5p5j   25m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-xwfvc   22m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
```

## 8. Operator の初期設定
### 8.1 Operatorの確認
`image-registry` 以外は `AVAILABLE` `True`になっていることを確認します。

```
# /usr/local/bin/oc get clusteroperator
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.4.31    True        False         False      14m
cloud-credential                           4.4.31    True        False         False      36m
cluster-autoscaler                         4.4.31    True        False         False      23m
console                                    4.4.31    True        False         False      21m
csi-snapshot-controller                    4.4.31    True        False         False      26m
dns                                        4.4.31    True        False         False      33m
etcd                                       4.4.31    True        False         False      29m
image-registry                                       False       False         True       26m
ingress                                    4.4.31    True        False         False      24m
insights                                   4.4.31    True        False         False      26m
kube-apiserver                             4.4.31    True        False         False      28m
kube-controller-manager                    4.4.31    True        False         False      32m
kube-scheduler                             4.4.31    True        False         False      32m
kube-storage-version-migrator              4.4.31    True        False         False      30m
machine-api                                4.4.31    True        False         False      26m
machine-config                             4.4.31    True        False         False      27m
marketplace                                4.4.31    True        False         False      26m
monitoring                                 4.4.31    True        False         False      22m
network                                    4.4.31    True        False         False      33m
node-tuning                                4.4.31    True        False         False      34m
openshift-apiserver                        4.4.31    True        False         False      25m
openshift-controller-manager               4.4.31    True        False         False      25m
openshift-samples                          4.4.31    True        False         False      22m
operator-lifecycle-manager                 4.4.31    True        False         False      33m
operator-lifecycle-manager-catalog         4.4.31    True        False         False      33m
operator-lifecycle-manager-packageserver   4.4.31    True        False         False      25m
service-ca                                 4.4.31    True        False         False      34m
service-catalog-apiserver                  4.4.31    True        False         False      34m
service-catalog-controller-manager         4.4.31    True        False         False      34m
storage                                    4.4.31    True        False         False      26m
``` 

### 8.2 image-registryのパッチ適用
`storage` に`emptyDir:{}`のパラメータを追加します。

```
# /usr/local/bin/oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
config.imageregistry.operator.openshift.io/cluster patched

# /usr/local/bin/oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'
config.imageregistry.operator.openshift.io/cluster patched
```
```yaml

# /usr/local/bin/oc get configs.imageregistry.operator.openshift.io -o yaml
apiVersion: v1
items:
- apiVersion: imageregistry.operator.openshift.io/v1
  kind: Config
  metadata:
    creationTimestamp: "2021-02-26T05:51:03Z"
    finalizers:
    - imageregistry.operator.openshift.io/finalizer
    generation: 2
    name: cluster
    resourceVersion: "30551"
    selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
    uid: fd18386a-774a-401d-924e-65101804229f
  spec:
    httpSecret: 93134baefb52122548a08f508fbde6c1253c9c9fa1ba10da00a2549922a52d17b5503502495b8730bd18f28cef86b40aaac3377f9e3c45463548048518968fff
    logging: 2
    managementState: Managed
    proxy: {}
    replicas: 1
    requests:
      read:
        maxWaitInQueue: 0s
      write:
        maxWaitInQueue: 0s
    rolloutStrategy: RollingUpdate
    storage:
      emptyDir: {}
  status:
    conditions:
    - lastTransitionTime: "2021-02-26T06:27:07Z"
      message: All registry resources are removed
      reason: Removed
      status: "False"
      type: Progressing
    - lastTransitionTime: "2021-02-26T05:51:03Z"
      message: The registry is removed
      reason: Removed
      status: "True"
      type: Available
    - lastTransitionTime: "2021-02-26T05:51:03Z"
      status: "False"
      type: Degraded
    - lastTransitionTime: "2021-02-26T05:51:03Z"
      message: The registry is removed
      reason: Removed
      status: "True"
      type: Removed
    observedGeneration: 2
    readyReplicas: 0
    storage: {}
    storageManaged: false
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
```
### 8.3 修正後のOperatorの確認
すべて `AVAILABLE` `True`になっていることを確認します。

```
# /usr/local/bin/oc get clusteroperator
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.4.31    True        False         False      26m
cloud-credential                           4.4.31    True        False         False      48m
cluster-autoscaler                         4.4.31    True        False         False      35m
console                                    4.4.31    True        False         False      33m
csi-snapshot-controller                    4.4.31    True        False         False      39m
dns                                        4.4.31    True        False         False      45m
etcd                                       4.4.31    True        False         False      41m
image-registry                             4.4.31    True        False         False      38m
ingress                                    4.4.31    True        False         False      36m
insights                                   4.4.31    True        False         False      38m
kube-apiserver                             4.4.31    True        False         False      40m
kube-controller-manager                    4.4.31    True        False         False      44m
kube-scheduler                             4.4.31    True        False         False      44m
kube-storage-version-migrator              4.4.31    True        False         False      42m
machine-api                                4.4.31    True        False         False      38m
machine-config                             4.4.31    True        False         False      39m
marketplace                                4.4.31    True        False         False      38m
monitoring                                 4.4.31    True        False         False      34m
network                                    4.4.31    True        False         False      46m
node-tuning                                4.4.31    True        False         False      46m
openshift-apiserver                        4.4.31    True        False         False      37m
openshift-controller-manager               4.4.31    True        False         False      37m
openshift-samples                          4.4.31    True        False         False      35m
operator-lifecycle-manager                 4.4.31    True        False         False      45m
operator-lifecycle-manager-catalog         4.4.31    True        False         False      46m
operator-lifecycle-manager-packageserver   4.4.31    True        False         False      37m
service-ca                                 4.4.31    True        False         False      46m
service-catalog-apiserver                  4.4.31    True        False         False      46m
service-catalog-controller-manager         4.4.31    True        False         False      46m
storage                                    4.4.31    True        False         False      38m
``` 

# 9. インストール完了の確認
Install complete!と表示され、パスワード等の情報が表示されればOKです。

```
# /usr/local/bin/openshift-install --dir=bare-metal wait-for install-complete --log-level=debug
DEBUG OpenShift Installer 4.4.31
DEBUG Built from commit 0227b5f653786d8d58312cd08a2e924e72ae646f
DEBUG Fetching Install Config...
DEBUG Loading Install Config...
DEBUG   Loading SSH Key...
DEBUG   Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Cluster Name...
DEBUG     Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Pull Secret...
DEBUG   Loading Platform...
DEBUG Using Install Config loaded from state file
DEBUG Reusing previously-fetched Install Config
INFO Waiting up to 30m0s for the cluster at https://api.lab.openshift.local:6443 to initialize...
DEBUG Cluster is initialized
INFO Waiting up to 10m0s for the openshift-console route to be created...
DEBUG Route found in openshift-console namespace: console
DEBUG Route found in openshift-console namespace: downloads
DEBUG OpenShift console route is created
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/opt/ocp/bare-metal/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.lab.openshift.local
INFO Login to the console with user: kubeadmin, password: XXXXX-XXXXX-XXXXX-XXXXX
```

お疲れ様でした