Image Registry - Multi-Architecture Image (push)

Image Registry では、`Image Manifest V 2, Schema 2` から `Multi-Architecture Image` の使用が可能になりました。これによって、単一の Image Repository で複数 Architecture Image を扱う事ができます。

ここでは、Red Hat Enterprise Linux 8 (RHEL8) 環境で `buildah` (ビルダー）で作成した `Multi-Architecture Image` を Image Registry に登録（push）する方法をご紹介します。

登録先の Image Registry には、IBM Cloud Container Registry (ICR) を使用します。
__Getting started with IBM Cloud Container Registry__
https://cloud.ibm.com/docs/Registry?topic=Registry-getting-started&locale=en

#### ibmcloud CLI の導入
`ibmcloud` CLI は以下の手順で導入します。
__Getting started with the IBM Cloud CLI__
https://cloud.ibm.com/docs/cli?topic=cli-getting-started

ICR 管理用に Plugin も導入します。
__Installing IBM Cloud Container Registry CLI plug-in__
https://cloud.ibm.com/docs/cli?topic=cli-install-devtools-manually#idt-install-container-registry-cli-plugin

ここでは、以下のバージョンを使用します。
```shell
$ ibmcloud --version
ibmcloud version 2.9.0+6abd6f8-2022-07-05T14:46:06+00:00

$ ibmcloud plugin list
Listing installed plug-ins...

Plugin Name          Version   Status   Private endpoints supported
container-registry   0.1.571            true
```
#### ICR の準備
ICR の Registry URL を確認します。`jp.icr.io` が Registry URL になります。
```shell
$ ibmcloud cr info

Plugin version                    0.1.571
Container Registry                jp.icr.io
Container Registry API endpoint   https://jp.icr.io/api
IBM Cloud API endpoint            https://cloud.ibm.com
IBM Cloud account details         My Account (**************************)
```
ICR に login します。ここでは podman/buildah を使用しているため、`--client podman` を指定します。
```shell
$ ibmcloud cr login --client podman
Logging 'podman' in to 'jp.icr.io'...
Logged in to 'jp.icr.io'.

OK

$ podman login --get-login jp.icr.io
iamrefresh
```
Container Image を push するために Namespace を作成します。ここでは `multi-arch` を作成してみます。

```shell
$ ibmcloud cr namespace-add multi-arch
No resource group is targeted. Therefore, the default resource group for the account ('Default') is targeted.

Adding namespace 'multi-arch' in resource group 'Default' for account My Account in registry jp.icr.io...

Successfully added namespace 'multi-arch'

OK

$ ibmcloud cr namespace-list
Listing namespaces for account 'My Account' in registry 'jp.icr.io'...

Namespace
multi-arch

OK
```
#### Container Image の登録（push）
以下で作成した `Multi-Architecture Image` を push します。
__Image Registry - Multi-Architecture Image (build)__
https://qiita.com/qiita2021user/items/a86cf0ddb89c5684d991
```shell
$ podman image ls
REPOSITORY                TAG         IMAGE ID      CREATED       SIZE
<none>                    <none>      7bbb6158020b  26 hours ago  5.86 MB
<none>                    <none>      75b399a47312  26 hours ago  5.86 MB
<none>                    <none>      e4a11a4a273e  26 hours ago  5.86 MB
<none>                    <none>      c3d510a1d552  26 hours ago  5.86 MB
<none>                    <none>      ef8bf7982aac  26 hours ago  5.86 MB
<none>                    <none>      bd2d524da383  26 hours ago  5.86 MB
<none>                    <none>      0bc3e2b24ac7  26 hours ago  5.86 MB
localhost/buildah-test    1.0         0e17508509df  26 hours ago  3.4 kB
docker.io/library/alpine  3.15        0ac33e5f5afa  3 months ago  5.86 MB
```
`buildah manifest push --all` コマンドで Manifest List 内の全ての Container Image を push することができます。
ここでは、Image Registry の Repository URL に `jp.icr.io/multi-arch/buildah-test:1.0` を使用してみます。
```shell
$ buildah manifest push --all localhost/buildah-test:1.0 docker://jp.icr.io/multi-arch/buildah-test:1.0
Getting image list signatures
Copying 7 of 7 images in list
Copying image sha256:1def9f292fe6c71942019875c56dded13fc467f8094176bfc24f3a4e6a19402c (1/7)
Getting image source signatures
Copying blob 1856f948c4bf done
Copying blob 4fc242d58285 done
Copying config 0bc3e2b24a done
Writing manifest to image destination
Storing signatures
Copying image sha256:ac8f1f8fb9169c403633c8caeeed882d6b92a6b4250514e1db8201145815ddaf (2/7)
Getting image source signatures
Copying blob b9a243c34cc6 done
Copying blob 4fc242d58285 skipped: already exists
Copying config bd2d524da3 done
Writing manifest to image destination
Storing signatures
Copying image sha256:9b71ab565a7beed5ba7d5b11ecac66c158bfa6cf95ee6f70cb150a380a93d90d (3/7)
Getting image source signatures
Copying blob 2242870f94ee done
Copying blob 4fc242d58285 skipped: already exists
Copying config ef8bf7982a done
Writing manifest to image destination
Storing signatures
Copying image sha256:70093b577d1be1ed69566e652c21ae06c6411844d855c92e669ba2729d10a48e (4/7)
Getting image source signatures
Copying blob 116ebe7e0830 done
Copying blob 4fc242d58285 skipped: already exists
Copying config c3d510a1d5 done
Writing manifest to image destination
Storing signatures
Copying image sha256:d5b46aadbf0a670f18c147ff064ed1d1a8bebad4b5fc07bb6ac63fd35157ec62 (5/7)
Getting image source signatures
Copying blob 9b3308c1e38e done
Copying blob 4fc242d58285 skipped: already exists
Copying config e4a11a4a27 done
Writing manifest to image destination
Storing signatures
Copying image sha256:f8e04bb902f5030962499b25615419f9b63d497cf1ad7a3cf7af0b672742b058 (6/7)
Getting image source signatures
Copying blob 04665ad84dbc done
Copying blob 4fc242d58285 skipped: already exists
Copying config 75b399a473 done
Writing manifest to image destination
Storing signatures
Copying image sha256:85e206c5a2c071bed43a5caad263cbee58a0bd1a3a49203b00cda229a45abc59 (7/7)
Getting image source signatures
Copying blob 5dec57fd26af done
Copying blob 4fc242d58285 skipped: already exists
Copying config 7bbb615802 done
Writing manifest to image destination
Storing signatures
Writing manifest list to image destination
Storing list signatures
```
表示されるログから、Manifest List にある以下の7個の Container Image が処理されたことがわかります。
```shell
$ podman manifest inspect localhost/buildah-test:1.0 | jq -r '.manifests[].digest'
sha256:1def9f292fe6c71942019875c56dded13fc467f8094176bfc24f3a4e6a19402c
sha256:ac8f1f8fb9169c403633c8caeeed882d6b92a6b4250514e1db8201145815ddaf
sha256:9b71ab565a7beed5ba7d5b11ecac66c158bfa6cf95ee6f70cb150a380a93d90d
sha256:70093b577d1be1ed69566e652c21ae06c6411844d855c92e669ba2729d10a48e
sha256:d5b46aadbf0a670f18c147ff064ed1d1a8bebad4b5fc07bb6ac63fd35157ec62
sha256:f8e04bb902f5030962499b25615419f9b63d497cf1ad7a3cf7af0b672742b058
sha256:85e206c5a2c071bed43a5caad263cbee58a0bd1a3a49203b00cda229a45abc59
``` 
#### 実行結果の確認
##### ICR 側での確認
ICR に登録された `Multi-Architecture-Image` を確認してみます。
```shell
$ ibmcloud cr image-list --show-type --no-trunc
Listing images...

Repository                          Tag   Digest                                                                    Namespace    Created   Size     Security status   Type
jp.icr.io/multi-arch/buildah-test   1.0   sha256:92251cd314782a2ef92a73b8ac8c9f2462337f6004a87ef9bd92ff9adad7f067   multi-arch   -         1.5 kB   -                 OCI Image Index v1

OK

$ ibmcloud cr manifest-inspect -q jp.icr.io/multi-arch/buildah-test:1.0 | jq -r .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:387922dcf0e66c7c611e913db52eb5e1545b80105824b8d0d2894837d6312001",
      "size": 758,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:542dbf4b1d6a14194dd89f9c0e445664f5607d0346866beb5d74d88f05af41f2",
      "size": 758,
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:71fb3164a697aa866a95cedb4516939caddd510ef1105d221dacf95a3b818a51",
      "size": 758,
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v6"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:73d1d586be1b6f79626fb5604793fb6d2fe234320074e373ff3f301e8bef10b3",
      "size": 758,
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:81f84c10b4ba1aaf5deb23cafe946d932e1298a0efacf058e9f566fe8f6dffb5",
      "size": 758,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:3b3cd654b1fbb481dadb25c98e889cc4edc97f5581ef114c3bce2aaed1075b74",
      "size": 758,
      "platform": {
        "architecture": "386",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:304a6bdfcaad365a31abbb7aba205b37611cd19934901cdcf9f812bbc70b3e04",
      "size": 758,
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      }
    }
  ]
}

$ ibmcloud cr manifest-inspect -q jp.icr.io/multi-arch/buildah-test:1.0 | jq -r '.manifests[].digest'
sha256:387922dcf0e66c7c611e913db52eb5e1545b80105824b8d0d2894837d6312001
sha256:542dbf4b1d6a14194dd89f9c0e445664f5607d0346866beb5d74d88f05af41f2
sha256:71fb3164a697aa866a95cedb4516939caddd510ef1105d221dacf95a3b818a51
sha256:73d1d586be1b6f79626fb5604793fb6d2fe234320074e373ff3f301e8bef10b3
sha256:81f84c10b4ba1aaf5deb23cafe946d932e1298a0efacf058e9f566fe8f6dffb5
sha256:3b3cd654b1fbb481dadb25c98e889cc4edc97f5581ef114c3bce2aaed1075b74
sha256:304a6bdfcaad365a31abbb7aba205b37611cd19934901cdcf9f812bbc70b3e04
```
Manifest List に7個の Container Image が登録されていることが分かります。

##### RHEL8 側での確認
以下の要領で `Multi-Architecture Image` を確認してみます。
__Image Registry - Multi-Architecture Image (inspect / pull)__
https://qiita.com/qiita2021user/items/515dae5612dedbffde4a

```shell
$ podman manifest inspect jp.icr.io/multi-arch/buildah-test:1.0
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 758,
            "digest": "sha256:387922dcf0e66c7c611e913db52eb5e1545b80105824b8d0d2894837d6312001",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 758,
            "digest": "sha256:542dbf4b1d6a14194dd89f9c0e445664f5607d0346866beb5d74d88f05af41f2",
            "platform": {
                "architecture": "s390x",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 758,
            "digest": "sha256:71fb3164a697aa866a95cedb4516939caddd510ef1105d221dacf95a3b818a51",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v6"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 758,
            "digest": "sha256:73d1d586be1b6f79626fb5604793fb6d2fe234320074e373ff3f301e8bef10b3",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v7"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 758,
            "digest": "sha256:81f84c10b4ba1aaf5deb23cafe946d932e1298a0efacf058e9f566fe8f6dffb5",
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 758,
            "digest": "sha256:3b3cd654b1fbb481dadb25c98e889cc4edc97f5581ef114c3bce2aaed1075b74",
            "platform": {
                "architecture": "386",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 758,
            "digest": "sha256:304a6bdfcaad365a31abbb7aba205b37611cd19934901cdcf9f812bbc70b3e04",
            "platform": {
                "architecture": "ppc64le",
                "os": "linux"
            }
        }
    ]
}
```
```shell
$ podman manifest inspect jp.icr.io/multi-arch/buildah-test:1.0 | jq -rc '.manifests[].platform'
{"architecture":"amd64","os":"linux"}
{"architecture":"s390x","os":"linux"}
{"architecture":"arm","os":"linux","variant":"v6"}
{"architecture":"arm","os":"linux","variant":"v7"}
{"architecture":"arm64","os":"linux"}
{"architecture":"386","os":"linux"}
{"architecture":"ppc64le","os":"linux"}
```
```shell
$ skopeo inspect --override-arch=arm --override-os=linux --override-variant=v7 docker://jp.icr.io/multi-arch/buildah-test:1.0
{
    "Name": "jp.icr.io/multi-arch/buildah-test",
    "Digest": "sha256:92251cd314782a2ef92a73b8ac8c9f2462337f6004a87ef9bd92ff9adad7f067",
    "RepoTags": [
        "1.0"
    ],
    "Created": "2022-07-05T06:04:36.885248758Z",
    "DockerVersion": "",
    "Labels": {
        "io.buildah.version": "1.24.2"
    },
    "Architecture": "arm",
    "Os": "linux",
    "Layers": [
        "sha256:6d34a08ca0d03c74fe8515811e6a1806c5c8799297c032a418c660b76a13836c",
        "sha256:bf4892210806256b8358e6e1d86a46ac17f1caa702530a1032309dc6213de4d4"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}
```
```shell
$  skopeo inspect --config --override-arch=arm --override-os=linux --override-variant=v7 docker://jp.icr.io/multi-arch/buildah-test:1.0
{
    "created": "2022-07-05T06:04:36.885248758Z",
    "architecture": "arm",
    "os": "linux",
    "config": {
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Entrypoint": [
            "/usr/bin/start.sh"
        ],
        "Labels": {
            "io.buildah.version": "1.24.2"
        }
    },
    "rootfs": {
        "type": "layers",
        "diff_ids": [
            "sha256:4fc242d58285699eca05db3cc7c7122a2b8e014d9481f323bd9277baacfa0628",
            "sha256:116ebe7e0830a7e8fb83f3d70e998940a8e4c21ca80c6ee288845ed856dad686"
        ]
    },
    "history": [
        {
            "created": "2022-04-05T00:19:59.790636867Z",
            "created_by": "/bin/sh -c #(nop) ADD file:5d673d25da3a14ce1f6cf66e4c7fd4f4b85a3759a9d93efb3fd9ff852b5b56e4 in / "
        },
        {
            "created": "2022-04-05T00:19:59.912662499Z",
            "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/sh\"]",
            "empty_layer": true
        },
        {
            "created": "2022-07-05T06:04:36.442780866Z",
            "created_by": "/bin/sh -c #(nop) ADD file:29224fab6275fcbcfeb9e59d018598d1ced47483889d97bc4ed12e8a41c8184f in /usr/bin ",
            "empty_layer": true
        },
        {
            "created": "2022-07-05T06:04:36.884863127Z",
            "created_by": "/bin/sh -c chmod a+x /usr/bin/start.sh",
            "empty_layer": true
        },
        {
            "created": "2022-07-05T06:04:37.000344163Z",
            "created_by": "/bin/sh -c #(nop) ENTRYPOINT [\"/usr/bin/start.sh\"]",
            "comment": "FROM docker.io/library/alpine:3.15"
        }
    ]
}
```
ICR から Container Image を pull してみます。
```shell
$ podman pull jp.icr.io/multi-arch/buildah-test:1.0
Trying to pull jp.icr.io/multi-arch/buildah-test:1.0...
Getting image source signatures
Copying blob 0ecd89e35dd4 done
Copying blob 6d34a08ca0d0 done
Copying config 0bc3e2b24a done
Writing manifest to image destination
Storing signatures
0bc3e2b24ac7e76f1fa58f4ad60d41ec596f4188dbe0b0b1f9faa4df16205dec

$ podman image ls
REPOSITORY                         TAG         IMAGE ID      CREATED       SIZE
jp.icr.io/multi-arch/buildah-test  1.0         0bc3e2b24ac7  27 hours ago  5.86 MB

$ podman image inspect 0bc3e2b24ac7
[
     {
          "Id": "0bc3e2b24ac7e76f1fa58f4ad60d41ec596f4188dbe0b0b1f9faa4df16205dec",
          "Digest": "sha256:92251cd314782a2ef92a73b8ac8c9f2462337f6004a87ef9bd92ff9adad7f067",
          "RepoTags": [
               "jp.icr.io/multi-arch/buildah-test:1.0"
          ],
          "RepoDigests": [
               "jp.icr.io/multi-arch/buildah-test@sha256:387922dcf0e66c7c611e913db52eb5e1545b80105824b8d0d2894837d6312001",
               "jp.icr.io/multi-arch/buildah-test@sha256:92251cd314782a2ef92a73b8ac8c9f2462337f6004a87ef9bd92ff9adad7f067"
          ],
          "Parent": "",
          "Comment": "",
          "Created": "2022-07-05T06:04:30.8304086Z",
          "Config": {
               "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
               ],
               "Entrypoint": [
                    "/usr/bin/start.sh"
               ],
               "Labels": {
                    "io.buildah.version": "1.24.2"
               }
          },
          "Version": "",
          "Author": "",
          "Architecture": "amd64",
          "Os": "linux",
          "Size": 5863548,
          "VirtualSize": 5863548,
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/lib/containers/storage/overlay/4fc242d58285699eca05db3cc7c7122a2b8e014d9481f323bd9277baacfa0628/diff",
                    "UpperDir": "/var/lib/containers/storage/overlay/03a764c09dcd6d27dcd7e4df2f836c49654df70a82873923a63d717d0a1afc87/diff",
                    "WorkDir": "/var/lib/containers/storage/overlay/03a764c09dcd6d27dcd7e4df2f836c49654df70a82873923a63d717d0a1afc87/work"
               }
          },
          "RootFS": {
               "Type": "layers",
               "Layers": [
                    "sha256:4fc242d58285699eca05db3cc7c7122a2b8e014d9481f323bd9277baacfa0628",
                    "sha256:1856f948c4bf972825d5ebc9ddbb3d6ccbd2c1d2a890bbf1cda21cfd13803116"
               ]
          },
          "Labels": {
               "io.buildah.version": "1.24.2"
          },
          "Annotations": {
               "org.opencontainers.image.base.digest": "sha256:a777c9c66ba177ccfea23f2a216ff6721e78a662cd17019488c417135299cd89",
               "org.opencontainers.image.base.name": "docker.io/library/alpine:3.15"
          },
          "ManifestType": "application/vnd.oci.image.manifest.v1+json",
          "User": "",
          "History": [
               {
                    "created": "2022-04-05T00:19:59.790636867Z",
                    "created_by": "/bin/sh -c #(nop) ADD file:5d673d25da3a14ce1f6cf66e4c7fd4f4b85a3759a9d93efb3fd9ff852b5b56e4 in / "
               },
               {
                    "created": "2022-04-05T00:19:59.912662499Z",
                    "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/sh\"]",
                    "empty_layer": true
               },
               {
                    "created": "2022-07-05T06:04:30.400728016Z",
                    "created_by": "/bin/sh -c #(nop) ADD file:29224fab6275fcbcfeb9e59d018598d1ced47483889d97bc4ed12e8a41c8184f in /usr/bin ",
                    "empty_layer": true
               },
               {
                    "created": "2022-07-05T06:04:30.829747547Z",
                    "created_by": "/bin/sh -c chmod a+x /usr/bin/start.sh",
                    "empty_layer": true
               },
               {
                    "created": "2022-07-05T06:04:31.000808315Z",
                    "created_by": "/bin/sh -c #(nop) ENTRYPOINT [\"/usr/bin/start.sh\"]",
                    "comment": "FROM docker.io/library/alpine:3.15"
               }
          ],
          "NamesHistory": [
               "jp.icr.io/multi-arch/buildah-test:1.0"
          ]
     }
]
```
```shell
$ podman run --rm -ti jp.icr.io/multi-arch/buildah-test:1.0
[Wed Jul  6 08:42:48 UTC 2022] : start.sh is running ...
[Wed Jul  6 08:42:58 UTC 2022] : start.sh is running ...
^C$
```
このように、`buildah manifest push --all` コマンドで IBM Cloud Container Registry (ICR) に `Multi-Architecture Image` を push することができます。