Repository: openshift/os
Pull request #1761: NO-JIRA: Containerfile: add metadata in last layer of node image
Author: login: jlebon
Labels: name: lgtm
description: Indicates that a PR is ready to be merged.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: jira/valid-reference
description: Indicates that this PR references a valid Jira ticket of any type.
Created: 2025-03-03T21:46:35Z | Merged: 2025-03-06T14:54:46Z

Prior to the layered RHCOS enhancement[[1]], there was tooling in ART and in the RHCOS release browser (notably linked from the release controller) that relied on the commit metadata JSON file published separately to be able to inspect the RPM contents of a build.

With the move to a layered node image, we lose this. The more container-native analogue would be SBOMs, which we should get for free once we move to Konflux. However, until then, let's provide an easy-to- generate alternative that still allows avoiding to download the whole image.

We add a new semantic where the last layer of the node image is actually a metadata-only layer. Then, tooling like `oc image extract` capable of extracting only specific layers from an image can be used to pull it down. E.g.

```
$ oc image extract quay.io/jlebon/rhcos:latest[-1] --file /usr/share/openshift/base/meta.json
$ jq . meta.json
```

Add a label to signal that this image implements this semantic.

Compared to SBOMs, this has the _major_ advantage that it's in-band and implicitly gets preserved as the image is slung around (whereas an attached SBOM may not until the tooling is careful to do so).

But like SBOMs, it has the pitfall that the metadata is only valid for the original node image, and obviously skews once more layering is done. The source of truth for the package list is always the rpmdb.

That said, a client implicitly knows if the metadata no longer matches the node content simply by the fact that it wouldn't be in the last layer.

[1]: https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/split-rhcos-into-layers.md


Diff:
diff --git a/Containerfile b/Containerfile
index cf5004be1..0563a1898 100644
--- a/Containerfile
+++ b/Containerfile
@@ -27,7 +27,7 @@
 #   --security-opt label=disable -t localhost/openshift-node-c9s \
 #   src/config
 
-FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev:c9s-coreos
+FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev:c9s-coreos as build
 ARG OPENSHIFT_CI=0
 # Avoid shipping modified .pyc files. Due to https://github.com/ostreedev/ostree/issues/1469,
 # any Python apps that run (e.g. dnf) will cause pyc creation.
@@ -37,3 +37,10 @@ RUN --mount=type=bind,target=/run/src --mount=type=secret,id=yumrepos,target=/et
   /run/src/scripts/apply-manifest /run/src/packages-openshift.yaml && \
   find /usr -name '*.pyc.bak' -exec sh -c 'mv $1 ${1%.bak}' _ {} \; && \
   ostree container commit
+
+FROM build as metadata
+RUN --mount=type=bind,target=/run/src /run/src/scripts/generate-metadata
+
+FROM build
+COPY --from=metadata /usr/share/openshift /usr/share/openshift
+LABEL io.openshift.metalayer=true
diff --git a/scripts/generate-metadata b/scripts/generate-metadata
new file mode 100755
index 000000000..f0b9c0ebf
--- /dev/null
+++ b/scripts/generate-metadata
@@ -0,0 +1,56 @@
+#!/usr/bin/python3 -u
+
+"""
+This script generates metadata intended to be stored as a separate layer.
+For now, only the RPM listing is generated. It follows the same schema as the
+`rpmostree.rpmdb.pkglist` commit metadata key in a base rpm-ostree compose.
+This is because it's intended to be consumed by code previously adapted for
+that purpose.
+
+Also, for now only a single `meta.json` file is generated. Obviously more
+metadata of interest can be generated in the future.
+"""
+
+import json
+import os
+import subprocess
+import sys
+
+# The "base" here denotes the fact that this describes the base node image and
+# will inherently not be valid if another layering operation is done on top of
+# this image to install packages. The source of truth for package lists will
+# always be the rpmdb.
+METADATA_FILE = "/usr/share/openshift/base/meta.json"
+
+
+def main():
+    metadata = {
+        # this follows the same schema as rpmostree.rpmdb.pkglist but let's
+        # not actually name it the same because it's not actually rpm-ostree
+        # generating this
+        'rpmdb.pkglist': get_rpmdb_pkglist()
+    }
+    os.makedirs(os.path.dirname(METADATA_FILE), exist_ok=True)
+    with open(METADATA_FILE, encoding='utf-8', mode='w') as f:
+        json.dump(metadata, f)
+
+
+def get_rpmdb_pkglist():
+    QUERYFORMAT = '%{NAME}\t%{EPOCH}\t%{VERSION}\t%{RELEASE}\t%{ARCH}\n'
+    out = subprocess.check_output(['rpm', '-qa', '--queryformat', QUERYFORMAT],
+                                  encoding='utf-8')
+    rpmdb = []
+    for line in out.splitlines():
+        n, e, v, r, a = line.split()
+        # canonicalize none to 0 to match rpm-ostree semantics
+        if e == '(none)':
+            e = 0
+        rpmdb.append([n, e, v, r, a])
+
+    # sort it by package name to be nice to humans who like JSON
+    rpmdb = sorted(rpmdb, key=lambda p: p[0])
+    return rpmdb
+
+
+if __name__ == '__main__':
+    sys.exit(main())


Reviews:
user: login: HuijingHei
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T12:07:12Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T13:01:22Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T13:12:47Z
user: login: jlebon
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T21:04:30Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T20:04:46Z
user: login: jbtrystram
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T15:47:04Z
user: login: jlebon
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T21:08:56Z

Review comments:
diff_hunk: @@ -0,0 +1,56 @@
+#!/usr/bin/python3 -u
+
+"""
+This script generates metadata intended to be stored as a separate layer.
+For now, only the RPM listing is generated. It follows the same schema as the
+`rpmostree.rpmdb.pkglist` commit metadata key in a base rpm-ostree compose.
+This is because it's intended to be consumed by code previously adapted for
+that purpose.
+
+Also, for now only a single `meta.json` file is generated. Obviously more
+metadata of interest can be generated in the future.
+"""
+
+import json
+import os
+import subprocess
+import sys
+
+# The "base" here denotes the fact that this describes the base node image and
+# will inherently not be valid if another layering operation is done on top of
+# this image to install packages. The source of truth for package lists will
+# always be the rpmdb.
+METADATA_FILE = "/usr/share/openshift/base/meta.json"
+
+
+def main():
+    metadata = {
+        # this follows the same schema as rpmostree.rpmdb.pkglist but let's
+        # not actually name it the same because it's not actually rpm-ostree
+        # generating this
+        'rpmdb.pkglist': get_rpmdb_pkglist()
+    }
+    os.makedirs(os.path.dirname(METADATA_FILE), exist_ok=True)
+    with open(METADATA_FILE, encoding='utf-8', mode='w') as f:
+        json.dump(metadata, f)
+
+
+def get_rpmdb_pkglist():
+    QUERYFORMAT = '%{NAME}\t%{EPOCH}\t%{VERSION}\t%{RELEASE}\t%{ARCH}\n'
+    out = subprocess.check_output(['rpm', '-qa', '--queryformat', QUERYFORMAT],
+                                  encoding='utf-8')
+    rpmdb = []
+    for line in out.splitlines():
+        n, e, v, r, a = line.split()
+        # canonicalize none to 0 to match rpm-ostree semantics
+        if e == '(none)':
+            e = 0
+        rpmdb.append([n, e, v, r, a])
+
+    # sort it by package name to be nice to humans who like JSON
path: scripts/generate-metadata
user: login: dustymabe
body: humans who like JSON :) 
created_at: 2025-03-04T13:01:21Z
author_association: MEMBER
diff_hunk: @@ -37,3 +37,10 @@ RUN --mount=type=bind,target=/run/src --mount=type=secret,id=yumrepos,target=/et
   /run/src/scripts/apply-manifest /run/src/packages-openshift.yaml && \
   find /usr -name '*.pyc.bak' -exec sh -c 'mv $1 ${1%.bak}' _ {} \; && \
   ostree container commit
+
+FROM build as metadata
+RUN --mount=type=bind,target=/run/src /run/src/scripts/generate-metadata
+
+FROM build
+COPY --from=metadata /usr/share/openshift /usr/share/openshift
+LABEL io.openshift.metalayer=true
path: Containerfile
user: login: dustymabe
body: I'm slightly confused.. I thought each operation in a Containerfile made a new layer anyway.. so how is what you have here different than not doing a multi-stage build and just doing:

```suggestion
RUN --mount=type=bind,target=/run/src /run/src/scripts/generate-metadata
LABEL io.openshift.metalayer=true
```


created_at: 2025-03-04T13:12:46Z
author_association: MEMBER
diff_hunk: @@ -37,3 +37,10 @@ RUN --mount=type=bind,target=/run/src --mount=type=secret,id=yumrepos,target=/et
   /run/src/scripts/apply-manifest /run/src/packages-openshift.yaml && \
   find /usr -name '*.pyc.bak' -exec sh -c 'mv $1 ${1%.bak}' _ {} \; && \
   ostree container commit
+
+FROM build as metadata
+RUN --mount=type=bind,target=/run/src /run/src/scripts/generate-metadata
+
+FROM build
+COPY --from=metadata /usr/share/openshift /usr/share/openshift
+LABEL io.openshift.metalayer=true
path: Containerfile
user: login: jlebon
body: Running the script itself adds more than just the metadata. It regenerates Python cache, it touches the rpmdb (just querying will change the timestamp), etc... Doing it this way ensures there's only _just_ the metadata in there.
created_at: 2025-03-04T21:04:30Z
author_association: MEMBER
diff_hunk: @@ -37,3 +37,10 @@ RUN --mount=type=bind,target=/run/src --mount=type=secret,id=yumrepos,target=/et
   /run/src/scripts/apply-manifest /run/src/packages-openshift.yaml && \
   find /usr -name '*.pyc.bak' -exec sh -c 'mv $1 ${1%.bak}' _ {} \; && \
   ostree container commit
+
+FROM build as metadata
+RUN --mount=type=bind,target=/run/src /run/src/scripts/generate-metadata
+
+FROM build
+COPY --from=metadata /usr/share/openshift /usr/share/openshift
+LABEL io.openshift.metalayer=true
path: Containerfile
user: login: dustymabe
body: 👍
created_at: 2025-03-05T20:04:46Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+#!/usr/bin/python3 -u
+
+"""
+This script generates metadata intended to be stored as a separate layer.
+For now, only the RPM listing is generated. It follows the same schema as the
+`rpmostree.rpmdb.pkglist` commit metadata key in a base rpm-ostree compose.
+This is because it's intended to be consumed by code previously adapted for
+that purpose.
+
+Also, for now only a single `meta.json` file is generated. Obviously more
+metadata of interest can be generated in the future.
+"""
+
+import json
+import os
+import subprocess
+import sys
+
+# The "base" here denotes the fact that this describes the base node image and
+# will inherently not be valid if another layering operation is done on top of
+# this image to install packages. The source of truth for package lists will
+# always be the rpmdb.
+METADATA_FILE = "/usr/share/openshift/base/meta.json"
+
+
+def main():
+    metadata = {
+        # this follows the same schema as rpmostree.rpmdb.pkglist but let's
+        # not actually name it the same because it's not actually rpm-ostree
+        # generating this
+        'rpmdb.pkglist': get_rpmdb_pkglist()
+    }
+    os.makedirs(os.path.dirname(METADATA_FILE), exist_ok=True)
+    with open(METADATA_FILE, encoding='utf-8', mode='w') as f:
+        json.dump(metadata, f)
+
+
+def get_rpmdb_pkglist():
+    QUERYFORMAT = '%{NAME}\t%{EPOCH}\t%{VERSION}\t%{RELEASE}\t%{ARCH}\n'
+    out = subprocess.check_output(['rpm', '-qa', '--queryformat', QUERYFORMAT],
+                                  encoding='utf-8')
+    rpmdb = []
+    for line in out.splitlines():
+        n, e, v, r, a = line.split()
+        # canonicalize none to 0 to match rpm-ostree semantics
+        if e == '(none)':
+            e = 0
path: scripts/generate-metadata
user: login: jbtrystram
body: shouldn't this be `e='0'` ? 
created_at: 2025-03-06T15:47:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,56 @@
+#!/usr/bin/python3 -u
+
+"""
+This script generates metadata intended to be stored as a separate layer.
+For now, only the RPM listing is generated. It follows the same schema as the
+`rpmostree.rpmdb.pkglist` commit metadata key in a base rpm-ostree compose.
+This is because it's intended to be consumed by code previously adapted for
+that purpose.
+
+Also, for now only a single `meta.json` file is generated. Obviously more
+metadata of interest can be generated in the future.
+"""
+
+import json
+import os
+import subprocess
+import sys
+
+# The "base" here denotes the fact that this describes the base node image and
+# will inherently not be valid if another layering operation is done on top of
+# this image to install packages. The source of truth for package lists will
+# always be the rpmdb.
+METADATA_FILE = "/usr/share/openshift/base/meta.json"
+
+
+def main():
+    metadata = {
+        # this follows the same schema as rpmostree.rpmdb.pkglist but let's
+        # not actually name it the same because it's not actually rpm-ostree
+        # generating this
+        'rpmdb.pkglist': get_rpmdb_pkglist()
+    }
+    os.makedirs(os.path.dirname(METADATA_FILE), exist_ok=True)
+    with open(METADATA_FILE, encoding='utf-8', mode='w') as f:
+        json.dump(metadata, f)
+
+
+def get_rpmdb_pkglist():
+    QUERYFORMAT = '%{NAME}\t%{EPOCH}\t%{VERSION}\t%{RELEASE}\t%{ARCH}\n'
+    out = subprocess.check_output(['rpm', '-qa', '--queryformat', QUERYFORMAT],
+                                  encoding='utf-8')
+    rpmdb = []
+    for line in out.splitlines():
+        n, e, v, r, a = line.split()
+        # canonicalize none to 0 to match rpm-ostree semantics
+        if e == '(none)':
+            e = 0
path: scripts/generate-metadata
user: login: jlebon
body: Good catch, thanks!
https://github.com/openshift/os/pull/1764
created_at: 2025-03-06T21:08:56Z
author_association: MEMBER

PR comments:
user: login: HuijingHei
created_at: 2025-03-04T12:50:39Z
body: CI error:
```
Running: dnf install --setopt=tsflags=nodocs --noplugins -y cri-o cri-tools conmon-rs openshift-clients openshift-kubelet openvswitch3.4 NetworkManager-ovs ose-aws-ecr-image-credential-provider ose-azure-acr-image-credential-provider ose-gcp-gcr-image-credential-provider
CentOS Stream 9 - BaseOS                         50 MB/s | 8.7 MB     00:00    
CentOS Stream 9 - AppStream                      95 MB/s |  23 MB     00:00    
CentOS Stream 9 - Extras packages               123 kB/s |  20 kB     00:00    
rhel-9-server-ose                                36 MB/s | 9.1 MB     00:00    
rhel-9-server-ose-ppc64le                       4.5 MB/s | 3.0 MB     00:00    
rhel-9-server-ose-s390x                         5.2 MB/s | 2.4 MB     00:00    
rhel-9-server-ose-aarch64                       9.5 MB/s | 7.2 MB     00:00    
No match for argument: openvswitch3.4
Error: Unable to find a match: openvswitch3.4
```
author_association: CONTRIBUTOR
user: login: jlebon
created_at: 2025-03-04T21:05:03Z
body: /retest
author_association: MEMBER
user: login: jlebon
created_at: 2025-03-05T01:15:30Z
body: OK, this now requires https://github.com/openshift/release/pull/62376.
author_association: MEMBER
user: login: jlebon
created_at: 2025-03-05T19:48:28Z
body: /retest
author_association: MEMBER
user: login: dustymabe
created_at: 2025-03-05T20:07:21Z
body: /approve
author_association: MEMBER
user: login: jlebon
created_at: 2025-03-05T23:20:17Z
body: e2e-aws failure looks unrelated. Cluster came up fine.
/override ci/prow/e2e-aws
author_association: MEMBER
user: login: openshift-ci[bot]
created_at: 2025-03-05T23:20:32Z
body: @jlebon: Overrode contexts on behalf of jlebon: ci/prow/e2e-aws

<details>

In response to [this](https://github.com/openshift/os/pull/1761#issuecomment-2702299880):

>e2e-aws failure looks unrelated. Cluster came up fine.
>/override ci/prow/e2e-aws


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: CONTRIBUTOR
user: login: openshift-ci-robot
created_at: 2025-03-06T02:09:38Z
body: @jlebon: This pull request explicitly references no jira issue.

<details>

In response to [this](https://github.com/openshift/os/pull/1761):

>Prior to the layered RHCOS enhancement[[1]], there was tooling in ART and in the RHCOS release browser (notably linked from the release controller) that relied on the commit metadata JSON file published separately to be able to inspect the RPM contents of a build.
>
>With the move to a layered node image, we lose this. The more container-native analogue would be SBOMs, which we should get for free once we move to Konflux. However, until then, let's provide an easy-to- generate alternative that still allows avoiding to download the whole image.
>
>We add a new semantic where the last layer of the node image is actually a metadata-only layer. Then, tooling like `oc image extract` capable of extracting only specific layers from an image can be used to pull it down. E.g.
>
>```
>$ oc image extract quay.io/jlebon/rhcos:latest[-1] --file /usr/share/openshift/base/meta.json
>$ jq . meta.json
>```
>
>Add a label to signal that this image implements this semantic.
>
>Compared to SBOMs, this has the _major_ advantage that it's in-band and implicitly gets preserved as the image is slung around (whereas an attached SBOM may not until the tooling is careful to do so).
>
>But like SBOMs, it has the pitfall that the metadata is only valid for the original node image, and obviously skews once more layering is done. The source of truth for the package list is always the rpmdb.
>
>That said, a client implicitly knows if the metadata no longer matches the node content simply by the fact that it wouldn't be in the last layer.
>
>[1]: https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/split-rhcos-into-layers.md


Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fos).  If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
</details>
author_association: NONE
user: login: HuijingHei
created_at: 2025-03-06T02:14:17Z
body: /lgtm
author_association: CONTRIBUTOR
user: login: openshift-ci[bot]
created_at: 2025-03-06T02:14:38Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/openshift/os/pull/1761#issuecomment-2701954197" title="Approved">dustymabe</a>*, *<a href="https://github.com/openshift/os/pull/1761#issuecomment-2702568132" title="LGTM">HuijingHei</a>*, *<a href="https://github.com/openshift/os/pull/1761#" title="Author self-approved">jlebon</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=openshift%2Fos).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/openshift/os/blob/master/OWNERS)~~ [HuijingHei,dustymabe,jlebon]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: openshift-ci-robot
created_at: 2025-03-06T05:02:28Z
body: /retest-required

Remaining retests: 0 against base HEAD 6655529da55ab0bdaeccc812dc66ea4cf03ea999 and 2 for PR HEAD 6241505f5835704397f9169b8b3abb4580de477c in total

author_association: NONE
user: login: jlebon
created_at: 2025-03-06T14:51:33Z
body: CI is hitting some regression from https://github.com/coreos/coreos-assembler/pull/4031.
/cc @nikita-dubrovskii 

But anyway, the image did build successfully.
/override ci/prow/rhcos-9-build-test-metal
/override ci/prow/scos-9-build-test-metal
author_association: MEMBER
user: login: openshift-ci[bot]
created_at: 2025-03-06T14:52:08Z
body: @jlebon: Overrode contexts on behalf of jlebon: ci/prow/rhcos-9-build-test-metal, ci/prow/scos-9-build-test-metal

<details>

In response to [this](https://github.com/openshift/os/pull/1761#issuecomment-2704074075):

>CI is hitting some regression from https://github.com/coreos/coreos-assembler/pull/4031.
>/cc @nikita-dubrovskii 
>
>But anyway, the image did build successfully.
>/override ci/prow/rhcos-9-build-test-metal
>/override ci/prow/scos-9-build-test-metal


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: CONTRIBUTOR
user: login: openshift-ci[bot]
created_at: 2025-03-06T14:53:03Z
body: @jlebon: all tests **passed!**


[Full PR test history](https://prow.ci.openshift.org/pr-history?org=openshift&repo=os&pr=1761). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:jlebon).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
author_association: CONTRIBUTOR
user: login: nikita-dubrovskii
created_at: 2025-03-06T14:54:39Z
body: @jlebon 

Yep, already disabled in https://github.com/coreos/coreos-assembler/pull/4035
author_association: CONTRIBUTOR
user: login: jlebon
created_at: 2025-03-06T15:01:32Z
body: > @jlebon
> 
> Yep, already disabled in [coreos/coreos-assembler#4035](https://github.com/coreos/coreos-assembler/pull/4035)

These tests are failing on x86_64, not s390x.
author_association: MEMBER