OpenShift 4.7 UPI install on bare metal

## 概要

おうちKubernetes・OpenShift勢の皆さん、こんにちは！！
OpenShift 4.7 のベアメタルへのUPIインストールを実施しましたので、ポイントをご紹介します。

4.6をインストールした時と同じ構成、手順でインストールができましたので、以下記事もご参照ください。

https://qiita.com/loftkun/items/044b6302d734d7f94587

OpenShift 4.11 以降については以下記事もご参照ください。

https://qiita.com/loftkun/items/03ca887ace88ef10496f

`4.7.4`インストール時にバグを踏んでしまいましたので、本記事では、その回避策について記載します。

なお、その後 `4.7.16`をインストールした際には、この事象は発生しませんでしたので、bugfixされている可能性があります。

## 現象

`authentication`, `console`, `monitoring`など、複数のclusteroperatorsが立ち上がりませんでした。

```sh
(抜粋)
[root@bastion ocp]# ./oc get clusteroperators
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.7.4     False       True          True       71m

monitoring                                           False       False         True       64m

openshift-apiserver                        4.7.4     False       False         False      28m

operator-lifecycle-manager-packageserver   4.7.4     False       True          False      2m51s
[root@bastion ocp]#
```

`Multiple errors are preventing progress` と表示されています。

```
root@bastion ocp]# ./openshift-install --dir=bare-metal wait-for install-complete --log-level=debug
DEBUG OpenShift Installer 4.7.4
DEBUG Built from commit 7d4efe10b441e9cb3dda33f81c62fd0eaeb3d6e6
DEBUG Loading Install Config...
DEBUG   Loading SSH Key...
DEBUG   Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Cluster Name...
DEBUG     Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Networking...
DEBUG     Loading Platform...
DEBUG   Loading Pull Secret...
DEBUG   Loading Platform...
DEBUG Using Install Config loaded from state file
INFO Waiting up to 40m0s for the cluster at https://api.test.example.local:6443 to initialize...
DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
DEBUG * Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (565 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (584 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-node-tuning-operator/node-tuning-operator" (341 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (362 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 668)
DEBUG * Could not update prometheusrule "openshift-dns-operator/dns" (609 of 668)
DEBUG * Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (290 of 668)
DEBUG * Could not update prometheusrule "openshift-ingress-operator/ingress-operator" (616 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-apiserver-operator/kube-apiserver-operator" (620 of 668): the server is reporting an internal error
DEBUG * Could not update prometheusrule "openshift-kube-controller-manager-operator/kube-controller-manager-operator" (632 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-scheduler-operator/kube-scheduler-operator" (636 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (253 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/machine-api-operator-prometheus-rules" (644 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-config-operator/machine-config-daemon" (646 of 668)
DEBUG * Could not update prometheusrule "openshift-operator-lifecycle-manager/olm-alert-rules" (651 of 668)
DEBUG * Could not update route "openshift-console/downloads" (441 of 668)
DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
DEBUG * Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (565 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (584 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-node-tuning-operator/node-tuning-operator" (341 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (362 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 668)
DEBUG * Could not update prometheusrule "openshift-dns-operator/dns" (609 of 668)
DEBUG * Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (290 of 668)
DEBUG * Could not update prometheusrule "openshift-ingress-operator/ingress-operator" (616 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-apiserver-operator/kube-apiserver-operator" (620 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-controller-manager-operator/kube-controller-manager-operator" (632 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-scheduler-operator/kube-scheduler-operator" (636 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (253 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/machine-api-operator-prometheus-rules" (644 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-config-operator/machine-config-daemon" (646 of 668)
DEBUG * Could not update prometheusrule "openshift-operator-lifecycle-manager/olm-alert-rules" (651 of 668)
DEBUG * Could not update route "openshift-console/downloads" (441 of 668): the server does not recognize this resource, check extension API servers
DEBUG Still waiting for the cluster to initialize: Working towards 4.7.4: 583 of 668 done (87% complete)
DEBUG Still waiting for the cluster to initialize: Working towards 4.7.4: 584 of 668 done (87% complete)
DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
DEBUG * Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (565 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (584 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-node-tuning-operator/node-tuning-operator" (341 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (362 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 668)
DEBUG * Could not update prometheusrule "openshift-dns-operator/dns" (609 of 668)
DEBUG * Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (290 of 668)
DEBUG * Could not update prometheusrule "openshift-ingress-operator/ingress-operator" (616 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-apiserver-operator/kube-apiserver-operator" (620 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-controller-manager-operator/kube-controller-manager-operator" (632 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-scheduler-operator/kube-scheduler-operator" (636 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (253 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/machine-api-operator-prometheus-rules" (644 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-config-operator/machine-config-daemon" (646 of 668)
DEBUG * Could not update prometheusrule "openshift-operator-lifecycle-manager/olm-alert-rules" (651 of 668)
DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
DEBUG * Could not update oauthclient "console" (405 of 668)
DEBUG * Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (565 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (584 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-node-tuning-operator/node-tuning-operator" (341 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (362 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 668)
DEBUG * Could not update prometheusrule "openshift-dns-operator/dns" (609 of 668)
DEBUG * Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (290 of 668)
DEBUG * Could not update prometheusrule "openshift-ingress-operator/ingress-operator" (616 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-apiserver-operator/kube-apiserver-operator" (620 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-controller-manager-operator/kube-controller-manager-operator" (632 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-scheduler-operator/kube-scheduler-operator" (636 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (253 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/machine-api-operator-prometheus-rules" (644 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-config-operator/machine-config-daemon" (646 of 668)
DEBUG * Could not update prometheusrule "openshift-operator-lifecycle-manager/olm-alert-rules" (651 of 668)
DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
DEBUG * Could not update oauthclient "console" (405 of 668): the server is down or not responding
DEBUG * Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (565 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (584 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-node-tuning-operator/node-tuning-operator" (341 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (362 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 668)
DEBUG * Could not update prometheusrule "openshift-dns-operator/dns" (609 of 668)
DEBUG * Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (290 of 668)
DEBUG * Could not update prometheusrule "openshift-ingress-operator/ingress-operator" (616 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-apiserver-operator/kube-apiserver-operator" (620 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-controller-manager-operator/kube-controller-manager-operator" (632 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-scheduler-operator/kube-scheduler-operator" (636 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (253 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/machine-api-operator-prometheus-rules" (644 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-config-operator/machine-config-daemon" (646 of 668)
DEBUG * Could not update prometheusrule "openshift-operator-lifecycle-manager/olm-alert-rules" (651 of 668)
DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
DEBUG * Could not update console "cluster" (20 of 668)
DEBUG * Could not update credentialsrequest "openshift-cloud-credential-operator/manila-csi-driver-operator" (393 of 668)
DEBUG * Could not update credentialsrequest "openshift-cloud-credential-operator/openshift-image-registry" (279 of 668)
DEBUG * Could not update credentialsrequest "openshift-cloud-credential-operator/openshift-machine-api-gcp" (132 of 668)
DEBUG * Could not update flowschema "openshift-etcd-operator" (74 of 668)
DEBUG * Could not update flowschema "openshift-kube-apiserver-operator" (100 of 668)
DEBUG * Could not update oauthclient "console" (405 of 668)
DEBUG * Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (565 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (584 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-node-tuning-operator/node-tuning-operator" (341 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (362 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 668)
DEBUG * Could not update prometheusrule "openshift-dns-operator/dns" (609 of 668)
DEBUG * Could not update prometheusrule "openshift-ingress-operator/ingress-operator" (616 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-apiserver-operator/kube-apiserver-operator" (620 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-controller-manager-operator/kube-controller-manager-operator" (632 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-scheduler-operator/kube-scheduler-operator" (636 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (253 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/machine-api-operator-prometheus-rules" (644 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-config-operator/machine-config-daemon" (646 of 668)
DEBUG * Could not update prometheusrule "openshift-operator-lifecycle-manager/olm-alert-rules" (651 of 668)
DEBUG Still waiting for the cluster to initialize: Working towards 4.7.4: 544 of 668 done (81% complete)
DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
DEBUG * Could not update oauthclient "console" (405 of 668)
DEBUG * Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (565 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (584 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-node-tuning-operator/node-tuning-operator" (341 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (362 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 668)
DEBUG * Could not update prometheusrule "openshift-dns-operator/dns" (609 of 668)
DEBUG * Could not update prometheusrule "openshift-image-registry/image-registry-operator-alerts" (290 of 668)
DEBUG * Could not update prometheusrule "openshift-ingress-operator/ingress-operator" (616 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-apiserver-operator/kube-apiserver-operator" (620 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-controller-manager-operator/kube-controller-manager-operator" (632 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-scheduler-operator/kube-scheduler-operator" (636 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (253 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/machine-api-operator-prometheus-rules" (644 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-config-operator/machine-config-daemon" (646 of 668)
DEBUG * Could not update prometheusrule "openshift-operator-lifecycle-manager/olm-alert-rules" (651 of 668)
DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
DEBUG * Could not update console "cluster" (20 of 668)
DEBUG * Could not update credentialsrequest "openshift-cloud-credential-operator/manila-csi-driver-operator" (393 of 668)
DEBUG * Could not update credentialsrequest "openshift-cloud-credential-operator/openshift-image-registry" (279 of 668)
DEBUG * Could not update credentialsrequest "openshift-cloud-credential-operator/openshift-machine-api-gcp" (132 of 668)
DEBUG * Could not update flowschema "openshift-etcd-operator" (74 of 668)
DEBUG * Could not update flowschema "openshift-kube-apiserver-operator" (100 of 668)
DEBUG * Could not update oauthclient "console" (405 of 668)
DEBUG * Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (565 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (584 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-node-tuning-operator/node-tuning-operator" (341 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (362 of 668)
DEBUG * Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 668)
DEBUG * Could not update prometheusrule "openshift-dns-operator/dns" (609 of 668)
DEBUG * Could not update prometheusrule "openshift-ingress-operator/ingress-operator" (616 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-apiserver-operator/kube-apiserver-operator" (620 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-controller-manager-operator/kube-controller-manager-operator" (632 of 668)
DEBUG * Could not update prometheusrule "openshift-kube-scheduler-operator/kube-scheduler-operator" (636 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (253 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-api/machine-api-operator-prometheus-rules" (644 of 668)
DEBUG * Could not update prometheusrule "openshift-machine-config-operator/machine-config-daemon" (646 of 668)
DEBUG * Could not update prometheusrule "openshift-operator-lifecycle-manager/olm-alert-rules" (651 of 668)
ERROR Cluster operator authentication Degraded is True with IngressStateEndpoints_MissingSubsets::OAuthRouteCheckEndpointAccessibleController_SyncError::OAuthServerDeployment_GetFailed::OAuthServiceCheckEndpointAccessibleController_SyncError::OAuthServiceEndpointsCheckEndpointAccessibleController_SyncError::OAuthVersionDeployment_GetFailed::Route_FailedCreate: OAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready
ERROR OAuthServiceCheckEndpointAccessibleControllerDegraded: Get "https://172.30.105.234:443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
ERROR IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server
ERROR OAuthRouteCheckEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.test.example.local/healthz": EOF
ERROR OAuthServerDeploymentDegraded: Unable to get "openshift-browser-client" bootstrapped OAuth client: the server is currently unable to handle the request (post oauthclients.oauth.openshift.io)
ERROR OAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps "oauth-openshift" not found
ERROR RouteDegraded: Unable to get or create required route openshift-authentication/oauth-openshift: the server is currently unable to handle the request (get routes.route.openshift.io oauth-openshift)
INFO Cluster operator authentication Available is False with APIServices_Error::OAuthRouteCheckEndpointAccessibleController_EndpointUnavailable::OAuthServiceCheckEndpointAccessibleController_EndpointUnavailable::OAuthServiceEndpointsCheckEndpointAccessibleController_EndpointUnavailable::OAuthVersionDeployment_MissingDeployment: OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints
INFO APIServicesAvailable: "oauth.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
INFO APIServicesAvailable: "user.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
INFO OAuthServiceCheckEndpointAccessibleControllerAvailable: Get "https://172.30.105.234:443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
INFO OAuthRouteCheckEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.test.example.local/healthz": EOF
INFO Cluster operator baremetal Disabled is True with UnsupportedPlatform: Nothing to do on this Platform
ERROR Cluster operator console Degraded is True with ConfigMapSync_FailedGetOAuthClient::CustomRouteSync_FailedDeleteCustomRoutes::DefaultRouteSync_FailedDefaultRouteApply::OAuthClientSync_FailedGet::SyncLoopRefresh_InProgress: SyncLoopRefreshDegraded: the server is currently unable to handle the request (get routes.route.openshift.io console)
ERROR CustomRouteSyncDegraded: the server is currently unable to handle the request (delete routes.route.openshift.io console-custom)
ERROR DefaultRouteSyncDegraded: the server is currently unable to handle the request (get routes.route.openshift.io console)
ERROR ConfigMapSyncDegraded: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console)
ERROR OAuthClientSyncDegraded: oauth client for console does not exist and cannot be created (the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console))
INFO Cluster operator console Available is Unknown with NoData:
INFO Cluster operator insights Disabled is False with AsExpected:
INFO Cluster operator monitoring Available is False with :
ERROR Cluster operator monitoring Degraded is True with UpdatingconfigurationsharingFailed: Failed to rollout the stack. Error: running task Updating configuration sharing failed: failed to retrieve Prometheus host: getting Route object failed: the server is currently unable to handle the request (get routes.route.openshift.io prometheus-k8s)
INFO Cluster operator network ManagementStateDegraded is False with :
INFO Cluster operator openshift-apiserver Available is False with APIServices_Error: APIServicesAvailable: "build.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
INFO APIServicesAvailable: "project.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
INFO APIServicesAvailable: "security.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
INFO APIServicesAvailable: "template.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
INFO Cluster operator openshift-samples Available is False with :
INFO Cluster operator operator-lifecycle-manager-packageserver Available is False with :
INFO Cluster operator operator-lifecycle-manager-packageserver Progressing is True with : Working toward 0.17.0
ERROR Cluster initialization failed because one or more operators are not functioning properly.
ERROR The cluster should be accessible for troubleshooting as detailed in the documentation linked below,
ERROR https://docs.openshift.com/container-platform/latest/support/troubleshooting/troubleshooting-installations.html
ERROR The 'wait-for install-complete' subcommand can then be used to continue the installation
FATAL failed to initialize the cluster: Multiple errors are preventing progress:
FATAL * Could not update console "cluster" (20 of 668)
FATAL * Could not update credentialsrequest "openshift-cloud-credential-operator/manila-csi-driver-operator" (393 of 668)
FATAL * Could not update credentialsrequest "openshift-cloud-credential-operator/openshift-image-registry" (279 of 668)
FATAL * Could not update credentialsrequest "openshift-cloud-credential-operator/openshift-machine-api-gcp" (132 of 668)
FATAL * Could not update flowschema "openshift-etcd-operator" (74 of 668)
FATAL * Could not update flowschema "openshift-kube-apiserver-operator" (100 of 668)
FATAL * Could not update oauthclient "console" (405 of 668)
FATAL * Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (565 of 668)
FATAL * Could not update prometheusrule "openshift-cluster-machine-approver/machineapprover-rules" (584 of 668)
FATAL * Could not update prometheusrule "openshift-cluster-node-tuning-operator/node-tuning-operator" (341 of 668)
FATAL * Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (362 of 668)
FATAL * Could not update prometheusrule "openshift-cluster-version/cluster-version-operator" (9 of 668)
FATAL * Could not update prometheusrule "openshift-dns-operator/dns" (609 of 668)
FATAL * Could not update prometheusrule "openshift-ingress-operator/ingress-operator" (616 of 668)
FATAL * Could not update prometheusrule "openshift-kube-apiserver-operator/kube-apiserver-operator" (620 of 668)
FATAL * Could not update prometheusrule "openshift-kube-controller-manager-operator/kube-controller-manager-operator" (632 of 668)
FATAL * Could not update prometheusrule "openshift-kube-scheduler-operator/kube-scheduler-operator" (636 of 668)
FATAL * Could not update prometheusrule "openshift-machine-api/cluster-autoscaler-operator-rules" (253 of 668)
FATAL * Could not update prometheusrule "openshift-machine-api/machine-api-operator-prometheus-rules" (644 of 668)
FATAL * Could not update prometheusrule "openshift-machine-config-operator/machine-config-daemon" (646 of 668)
FATAL * Could not update prometheusrule "openshift-operator-lifecycle-manager/olm-alert-rules" (651 of 668)
[root@bastion ocp]#
```

## Issue

以下のIssueがありました。

- [Router connectivity issue with OpenShiftSDN in 4.6 · Issue #430 · openshift/okd](https://github.com/openshift/okd/issues/430)
- [4.7 2021-03-21 UPI does not finish cluster installation · Issue #571 · openshift/okd](https://github.com/openshift/okd/issues/571)

以下のような報告が寄せられています。

- 仮想マシンのハードウェア互換性が`13`より大きい場合に発生する。`13`にすると解消した。
- `networkType` を `OVNKubernetes` にすると解消する

## 回避策

私はVMのハードウェア互換性を`13`に下げることで`4.7.4`をインストールすることができることを確認しました。

vmware の場合は以下をご参照ください。VM作成時に`互換性`に`ESXi 6.5 仮想マシン`を指定すると良いです。
[仮想マシンのハードウェア バージョン (1003746)](https://kb.vmware.com/s/article/1003746?lang=ja)

ただ、この方法はVM自体を作り直す必要もあるので個人的にはおすすめしません。
以下のRed Hat 公開情報にWORKAROUNDな回避策がいくつか挙げられていますので、公式情報としては以下を参照するのが良いと思います。

[[Bug] Openshift-apiserver cluster-operator goes unavailable during cluster upgrade from RHOCP 4.6 to 4.7 - Red Hat Customer Portal](https://access.redhat.com/solutions/5896081)

また、UPI/IPIインストール界隈で情報発信されているRed Hat Solutions Architectの田中さんより、本件は`4.7.5`で修正されていると教えていただきましたので、可能であれば`4.7.5`以降を使用するのが良いと思います(田中さんありがとうございます！)。

その際の会話については、以下ツイートに続くリプライをご参照ください。

https://twitter.com/loftkun/status/1378338056311242752

https://twitter.com/tnk4on/status/1393291877584031745

OpenShift 4.7のインストールのポイントについてご参考になりましたら幸いです。