OpenShift - oc-mirror - mirror operator images

#### oc-mirror

OpenShift をインターネット非接続環境に導入 (disconnected installation) する際に、oc-mirror plugin が利用できます。

https://docs.redhat.com/ja/documentation/openshift_container_platform/4.16/html/installing/installing-mirroring-disconnected#installation-oc-mirror-about_installing-mirroring-disconnected

> oc-mirror OpenShift CLI(oc) プラグインを使用すると、単一のツールを使用して、必要なすべての OpenShift Container Platform コンテンツおよびその他のイメージをミラーレジストリーにミラーリングできます。次の機能を提供します。
> - OpenShift Container Platform のリリース、Operator、ヘルムチャート、およびその他のイメージをミラーリングするための一元化された方法を提供します。
> - OpenShift Container Platform および Operator の更新パスを維持します。
> - 宣言型イメージセット設定ファイルを使用して、クラスターに必要な OpenShift Container Platform リリース、Operator、およびイメージのみを含めます。
> - 将来のイメージセットのサイズを縮小するインクリメンタルミラーリングを実行します。
> - 前回の実行以降にイメージセット設定から除外されたターゲットミラーレジストリーからのイメージをプルーニングします。
> - オプションで、OpenShift Update Service (OSUS) を使用する際のサポートアーティファクトを生成します。 

ここでは、oc-mirror を使用して Operator image を Private Registry に登録する方法をご紹介します。


#### Create ImageSetConfiguration
以下を参考にして `ImageSetConfiguration` を作成します。

https://docs.redhat.com/ja/documentation/openshift_container_platform/4.16/html/installing/installing-mirroring-disconnected#oc-mirror-creating-image-set-config_installing-mirroring-disconnected

```shell
$ oc mirror init > ./ODF_Operator_Images.yaml
$ cat ./ODF_Operator_Images.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
storageConfig:
  local:
    path: ./ODF_Operator_Images
mirror:
  platform:
    channels:
    - name: stable-4.16
      type: ocp
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.17
    packages:
    - name: serverless-operator
      channels:
      - name: stable
  additionalImages:
  - name: registry.redhat.io/ubi8/ubi:latest
  helm: {}
```
例として、OpenShift Data Foundation (ODF) の Operator image を取得するために、以下を参考にして修正します。

https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.16/html/planning_your_deployment/disconnected-environment_rhodf#disconnected-environment_rhodf

https://github.com/openshift/oc-mirror/tree/main/docs/examples

```shell
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
storageConfig:
  local:
    path: ./ODF_Operator_Images
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16
    packages:
    - name: ocs-operator
      channels:
      - name: stable-4.16
    - name: odf-operator
      channels:
      - name: stable-4.16
    - name: mcg-operator
      channels:
      - name: stable-4.16
    - name: odf-csi-addons-operator
      channels:
      - name: stable-4.16
    - name: odr-cluster-operator
      channels:
      - name: stable-4.16
    - name: odr-hub-operator
      channels:
      - name: stable-4.16
    - name: local-storage-operator
      channels:
      - name: stable
    - name: odf-multicluster-orchestrator
      channels:
      - name: stable-4.16
  additionalImages:
  - name: registry.redhat.io/ubi8/ubi:latest
  helm: {}
```

#### Mirror Operator image
以下の要領で、Operator image を取得します。
```shell
$ oc mirror --config=./ODF_Operator_Images.yaml file://./ODF_Operator_Images
Creating directory: ODF_Operator_Images/oc-mirror-workspace/src/publish
Creating directory: ODF_Operator_Images/oc-mirror-workspace/src/v2
Creating directory: ODF_Operator_Images/oc-mirror-workspace/src/charts
Creating directory: ODF_Operator_Images/oc-mirror-workspace/src/release-signatures
No metadata detected, creating new workspace
 :
  stats: shared=55 unique=319 size=10.53GiB ratio=0.71
 :
sha256:162f35a85baabd9055c53717463c908247769a9bf617a4b457adf398d2e972be file://odf4/odf-csi-addons-sidecar-rhel9
sha256:41064d2db3c4c1e651f2df89d6cd4a19027c6930edc6ab66b5535a2cd7f39f49 file://odf4/odf-csi-addons-sidecar-rhel9
sha256:272f038ebe7fd778c6c24dcc27524f3aa807465576177681b8b80a6adb32be99 file://odf4/odf-csi-addons-sidecar-rhel9
sha256:859301df4f104e3d36efc2446929c86805e586c3c5ac07548d0277442d5df6ca file://odf4/odf-csi-addons-sidecar-rhel9
sha256:0e565039b003b02e08f6cd80d0f603a568ae772ce1ba8784a3c54bf178b56164 file://odf4/odf-csi-addons-sidecar-rhel9:653aaeb8
sha256:8512465ad8828593d0f4b6d4208a6c60a0f297f70ef26a0c4c874f6ddf4798d2 file://odf4/odf-cosi-sidecar-rhel9
sha256:8b0583cf72b39299e6769f9c9fdc4321b7e384405aec8d1519d4ea5ac060fd05 file://odf4/odf-cosi-sidecar-rhel9
sha256:43334d92af32e97a11691d0fa7069746d59d6f73a0a2c0b97bbbfdc7937f117a file://odf4/odf-cosi-sidecar-rhel9
sha256:e9a6153f70ff785e0c1a7ce2f850f844797753d5d7a1f49c84cf274350f5c6d9 file://odf4/odf-cosi-sidecar-rhel9:f504c52b
sha256:d8d1f88909603e1c5c04d1efa5fc9dd4b978f90835e69b2f731def5c95dea44a file://rhel9/postgresql-15
sha256:06c3a72632d07e5a10dfd249f75ba59ba1107f389843937b0733406cd4b8d991 file://rhel9/postgresql-15
sha256:6ca45c001d94700d4b53ba924cc4fc717210f0d99d412f38fded64749028a335 file://rhel9/postgresql-15
sha256:8ed5093543df7e9e835158759b22d7757e86782d2d856249c3cd2421c8e5125c file://rhel9/postgresql-15
sha256:5c4cad6de1b8e2537c845ef43b588a11347a3297bfab5ea611c032f866a1cb4e file://rhel9/postgresql-15:74ccaef8
sha256:3dc59c38e0cde2c9974d3fa5a7ce1c9f98589a5aeb486fdc8a8e7717656522fe file://odf4/odf-must-gather-rhel9
sha256:0ee210a535f1f11de40e1a0ee55a10855ce13c09dab9791ee4d35c59949ca934 file://odf4/odf-must-gather-rhel9
sha256:44c6bb264bbbd0ad67978dd2b4e01c530e653baf14b15f8a1eddff675fa760bd file://odf4/odf-must-gather-rhel9
sha256:631643362d4345fdefb3fb34416ca09626b634c92fde970e1494a95fe5bdfc87 file://odf4/odf-must-gather-rhel9
sha256:899c34b813d9703d5c000224a64b51cdade2bfd28a3c4bd5ba2d51be7fe0ab57 file://odf4/odf-must-gather-rhel9:9d4bf5d6
info: Mirroring completed in 17m39.64s (10.67MB/s)
Creating archive /ODF_Operator_Images/mirror_seq1_000000.tar
```
完了時点のディスク使用量は、以下のようになっています。
```shell
$ du -csh ODF_Operator_Images/*
13G     ODF_Operator_Images/mirror_seq1_000000.tar
5.3G    ODF_Operator_Images/oc-mirror-workspace
228K    ODF_Operator_Images/publish
18G     total
```
#### Push Operator image to Private Registry
以下の要領で、Private Registry に Operator image を Push します。ここでは、Registry URL に `myhost.com:5000` を使用します。
```shell
$ oc mirror --from=mirror_seq1_000000.tar docker://myhost.com:5000
Checking push permissions for myhost.com:5000
Publishing image set from archive "mirror_seq1_000000.tar" to registry "myhost.com:5000"
myhost.com:5000/
  openshift4/ose-csi-external-provisioner
    blobs:
      file://openshift4/ose-csi-external-provisioner sha256:146e61ec436f8da9758eb3498120b98c6a6513e6e7bc830836c382d154949772 21.98KiB
      file://openshift4/ose-csi-external-provisioner sha256:98084964f3888d97f02b3f27281a53b8a901978b8a32f0d04b88c6963157095a 21.99KiB
      file://openshift4/ose-csi-external-provisioner sha256:183c1144a40720088d81b68981d1b8b6b618d48a3a0039e5eec488c16b51f376 21.99KiB
      file://openshift4/ose-csi-external-provisioner sha256:0678336eb5d28c49cf7983080a2986952bd73c7c0454373100a643fb2b91630d 22KiB
      file://openshift4/ose-csi-external-provisioner sha256:55271fb207f645d4d0a1e30131ae3bb6e9d273af5308f53a0026ab1284977a00 11.14MiB
      file://openshift4/ose-csi-external-provisioner sha256:ae8d7fccfbfba7bfeaf6895f1b8c129facadcf3953db4f79ee7f5fb7bd431e1d 11.27MiB
      file://openshift4/ose-csi-external-provisioner sha256:53cbc85ff8acf89c9706dee739d8101ea5393fbe1ba79b91c414a6efc8cd42a4 11.29MiB
      file://openshift4/ose-csi-external-provisioner sha256:c2b429ad118065c6e3e55a55ad4b552659c0ef257a672ab1ede894d387c54499 11.99MiB
      file://openshift4/ose-csi-external-provisioner sha256:15c3647f00c9291b4348c202fdf5b60a9edaf10bd3911c4a8e9f04dfbb0d8b2e 17.25MiB
      file://openshift4/ose-csi-external-provisioner sha256:e91a2b5c9293335b9807510b4e6b4e7b6e6f88bc97e2e071e490be4de780059c 17.38MiB
      file://openshift4/ose-csi-external-provisioner sha256:3b7cc3123fd83fc00c0753a35660b07f0fcdb0661e4ccc7f2425d3a4034038b8 17.66MiB
      file://openshift4/ose-csi-external-provisioner sha256:54db7a11559bc11d1e077801b892b7795fd26d0994e1039d5eba4c12bbaed168 18.75MiB
      file://openshift4/ose-csi-external-provisioner sha256:d560746f4016c03a8e6a0daea91dcdf37e304ee95b66d1bcf9478e8a77b86af1 23.6MiB
      file://openshift4/ose-csi-external-provisioner sha256:63b0bf9094b989b686c970ede7bb2f9c3ec819c0bdbf233e7f4b83c62c6464ad 23.92MiB
      file://openshift4/ose-csi-external-provisioner sha256:3c189c77d9a637751787329eda984f9d0b9325e1921512ed5db176962f118716 25.73MiB
      file://openshift4/ose-csi-external-provisioner sha256:65045d6772b2d1814ea771846373055def67fc7dbce6a69df5d39164793be8d4 26MiB
      file://openshift4/ose-csi-external-provisioner sha256:5b92631cae6df8d9aef617e5a07532336e5c2a09d7c5418592ea6188a7e10a0c 72.73MiB
      file://openshift4/ose-csi-external-provisioner sha256:abb828c6dd68e53a22f7a11904b72f9d5a7802fb5e7c380b3a22b18a02e9a463 73.31MiB
      file://openshift4/ose-csi-external-provisioner sha256:c7965aa7086045a59bdb113a1fb8a19d7ccf7af4133e59af8ecefd39cda8e0b1 75.31MiB
      file://openshift4/ose-csi-external-provisioner sha256:8bad510b7a13af38e38c6a4b66c8beaac49b44c2f1bac22068b1001bae266f3e 81.92MiB
    manifests:
      sha256:53fb4d4cc05c77af143882adbe7ee28d9db8d4f89f674d521f43263e3dc73f7e
      sha256:580e5b7c58c27405559e54fbd981d1565133a275f0ad2d056f0ef8ca4291f985
      sha256:6f27707a853cb7e0f7f371a7ad13e57b59853dc00338f45b40a57a19b040f5bb
      sha256:9dbdc618cce9e7a1b82abe3d0bf11e072e30d9861dd02905b00b94e1df80e565
      sha256:b68d255606e021b5208ed54edec1f7aadddbd67e72133de4e76f1617b0f37b6d
      sha256:580e5b7c58c27405559e54fbd981d1565133a275f0ad2d056f0ef8ca4291f985 -> bb8cce5
  stats: shared=0 unique=20 size=519.4MiB ratio=1.00
   :
Wrote release signatures to oc-mirror-workspace/results-1722991087
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/include-config.gob
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/index/index.json
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/1279376fba16a0b9b69915c3f0bd3a91f262c9fe98d9f324d19530826e00c733
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/1a45ff480335b45a402df42965ce5f1e27a53139fcf4a718756bc882e6d94596
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/3059f6068401e6b82194cb486bf75573bd18356796e1010a73d575704723cc31
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/31b36112561f9aa75a1aebcbb1fbb834b2757055e7ed5ad3a2fd4025f03715bb
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/347bcf45895bca4358461d025bcb9af0c5ce4ca7f017827794ea7292bbc92850
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/47457964061600b86d26f095f56149e9a395fdd4bc868a1110b3e32e19a0b8c5
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/56e4e8a26d67d86e3c3c1741245857747ae2d8f970ec9ec442334345c392b1e5
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/7082f02cc356245ebfcfeae2c489bb614e7fa6e96a45ef9ab92e41a75ccaadaf
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/77265b8ce7197462215724b8a7aa5280a5bafb37b8265c70fc620e19489d88b7
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/7ae6a6457c8e3432165bd85861a9ffff55326837d188012194cb19cc08a34979
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/7dc941f37734495d75e0e027d69e634257d8715b5acf8fc66b7e3fb07d7be2e0
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/8e7399dda7ddc3c70f9e3a11ef8e33e5c12d62043cb4c7ab92a5a3075996bfd6
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/9a0e86afac7b17960e2411a55e234d639aead65defac982719003eaa242da807
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/9d7fb64dd93101e976267a16e82cc1dd65152171abb72a1c9d5b825a5abb046b
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/a5177f2115c2a9ad5ce7c96eae255635e832b1ce1ed8288953308a3003d3abae
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/ac07b53f4089b423560ff60f7f5b5466b9bd9339fd52ca12c7a83ba68ea4700d
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/afeed291f3507a1dfb39ff6a8ccb0485345b3228e183a0304c1cc9fd2fcd83fc
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/b0894a3f87fc6ba1599959e1197f3de09cab9cf98931f22f18499bb016e5561b
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/b1cd49c7533a0adeb2833aa28ac5aecdfeb683595dc7c00c40e5c8ed722ca697
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/b8f01a14da09788840a6772fb7516029a2f1bbf7faac17aba8031d619c02ee5b
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/b958cf13e572cbd92209b5c01a8e65203cf88c4b017258ec8ff9bfd5d722b715
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/bf8da036594f4fd6a962fd8890aa1c43c718c159c774921de016d33f1cd5ed32
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/d5910d4e93ada0cd1e17d903d54db2ff0f0a232733ecd170d3778fc0d2cdaf81
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/dbd7c88ef29e10a479b19a860da3750a07ab736db805ebf8dc35539d298daf00
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/dfc6a1062f5a20e9461eeb4a8c0aa60841841ec7d585db8a94430aab157331e8
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/e14fb1d0434d5b4a532b7c150abac7565487b49d050d09ce2727c2378a0ca3c1
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/e510c81b08c61bd9987731d0d9e715998e4452113cfa1ed14334c536599af14e
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/blobs/sha256/fb179fb6455274fba93536de2c9c76007b32194fc11310d15f5fa9145d012bcc
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/index.json
catalogs/registry.redhat.io/redhat/redhat-operator-index/v4.16/layout/oci-layout
Rendering catalog image "myhost.com:5000/redhat/redhat-operator-index:v4.16"
Writing image mapping to oc-mirror-workspace/results-1722991087/mapping.txt
Writing CatalogSource manifests to oc-mirror-workspace/results-1722991087
Writing ICSP manifests to oc-mirror-workspace/results-1722991087
```
`oc-mirror-workspace/results-1722991087` サブディレクトリーに各種情報が作成されたことが分かります。また、Private Registry を確認すると、ODF Operator iamge が登録されたことが分かります。
```shell
$ curl -s https://myhost.com:5000/v2/_catalog | jq -r .
{
  "repositories": [
    "odf4/cephcsi-rhel9",
    "odf4/mcg-core-rhel9",
    "odf4/mcg-operator-bundle",
    "odf4/mcg-rhel9-operator",
    "odf4/ocs-client-console-rhel9",
    "odf4/ocs-client-operator-bundle",
    "odf4/ocs-client-rhel9-operator",
    "odf4/ocs-metrics-exporter-rhel9",
    "odf4/ocs-operator-bundle",
    "odf4/ocs-rhel9-operator",
    "odf4/odf-console-rhel9",
    "odf4/odf-cosi-sidecar-rhel9",
    "odf4/odf-csi-addons-operator-bundle",
    "odf4/odf-csi-addons-rhel9-operator",
    "odf4/odf-csi-addons-sidecar-rhel9",
    "odf4/odf-multicluster-console-rhel9",
    "odf4/odf-multicluster-operator-bundle",
    "odf4/odf-multicluster-rhel9-operator",
    "odf4/odf-must-gather-rhel9",
    "odf4/odf-operator-bundle",
    "odf4/odf-prometheus-operator-bundle",
    "odf4/odf-rhel9-operator",
    "odf4/odr-cluster-operator-bundle",
    "odf4/odr-hub-operator-bundle",
    "odf4/odr-recipe-operator-bundle",
    "odf4/odr-rhel9-operator",
    "odf4/rook-ceph-operator-bundle",
    "odf4/rook-ceph-rhel9-operator",
    "openshift4/ose-csi-external-attacher-rhel9",
    "openshift4/ose-csi-external-provisioner",
    "openshift4/ose-csi-external-provisioner-rhel9",
    "openshift4/ose-csi-external-resizer",
    "openshift4/ose-csi-external-resizer-rhel9",
    "openshift4/ose-csi-external-snapshotter-rhel9",
    "openshift4/ose-csi-node-driver-registrar",
    "openshift4/ose-csi-node-driver-registrar-rhel9",
    "openshift4/ose-kube-rbac-proxy-rhel9",
    "openshift4/ose-local-storage-diskmaker-rhel9",
    "openshift4/ose-local-storage-mustgather-rhel9",
    "openshift4/ose-local-storage-operator-bundle",
    "openshift4/ose-local-storage-rhel9-operator",
    "openshift4/ose-oauth-proxy-rhel9",
    "openshift4/ose-prometheus-alertmanager-rhel9",
    "openshift4/ose-prometheus-config-reloader-rhel9",
    "openshift4/ose-prometheus-rhel9",
    "openshift4/ose-prometheus-rhel9-operator",
    "redhat/redhat-operator-index",
    "rhceph/rhceph-7-rhel9",
    "rhel9/postgresql-15",
    "ubi8/ubi"
  ]
}
```
完了時点の Private Registry ディスク使用量は、以下のようになっています。
```shell
$ du -csh /opt/registry/data/
13G     /opt/registry/data/
13G     total
```
これで、Private Registry への Operator image の登録は完了です。この後、以下の要領に従って OCP Cluster に設定を反映して完了です。

2.4.8. oc-mirror が生成したリソースを使用するためのクラスター設定

https://docs.redhat.com/ja/documentation/openshift_container_platform/4.16/html/installing/installing-mirroring-disconnected#oc-mirror-updating-cluster-manifests_installing-mirroring-disconnected

```shell
$ oc apply -f ./oc-mirror-workspace/results-1722991087
```