Repository: coreos/coreos-assembler
Pull request #4038: schema: support multiple images destinations
Author: login: jbtrystram
Created: 2025-03-11T10:00:56Z | Merged: 2025-03-27T13:40:50Z

In https://github.com/coreos/fedora-coreos-pipeline/pull/1091 we added the possibility of publishing container images to multiples repositories.

This needs to be reflected in the schema otherwhise we only record one image (currenlty the last pushed).

Add a `primary-image` key that extends image by adding an `additional-images` array.

Use that for relevant container images objects.

See https://github.com/coreos/fedora-coreos-pipeline/issues/1099


Diff:
diff --git a/pkg/builds/cosa_v1.go b/pkg/builds/cosa_v1.go
index d1a135d7b3..cb4b00509f 100644
--- a/pkg/builds/cosa_v1.go
+++ b/pkg/builds/cosa_v1.go
@@ -1,7 +1,7 @@
 package builds
 
 // generated by 'make schema'
-// source hash: 58da31f35c1405d5a752a297d0ee012f577720f810113d38cdabca102ec7edd8
+// source hash: 1711aa3ce997c644130139e1318d2683e2a5f11538e2b0528771a8658d62ee64
 
 type AdvisoryDiff []AdvisoryDiffItems
 
@@ -34,7 +34,7 @@ type Build struct {
 	Amis                      []Amis                `json:"amis,omitempty"`
 	Architecture              string                `json:"coreos-assembler.basearch,omitempty"`
 	Azure                     *Cloudartifact        `json:"azure,omitempty"`
-	BaseOsContainer           *Image                `json:"base-oscontainer,omitempty"`
+	BaseOsContainer           *PrimaryImage         `json:"base-oscontainer,omitempty"`
 	BuildArtifacts            *BuildArtifacts       `json:"images,omitempty"`
 	BuildID                   string                `json:"buildid"`
 	BuildRef                  string                `json:"ref,omitempty"`
@@ -50,7 +50,7 @@ type Build struct {
 	CosaImageChecksum         string                `json:"coreos-assembler.image-config-checksum,omitempty"`
 	CosaImageVersion          int                   `json:"coreos-assembler.image-genver,omitempty"`
 	Extensions                *Extensions           `json:"extensions,omitempty"`
-	ExtensionsContainer       *Image                `json:"extensions-container,omitempty"`
+	ExtensionsContainer       *PrimaryImage         `json:"extensions-container,omitempty"`
 	FedoraCoreOsParentCommit  string                `json:"fedora-coreos.parent-commit,omitempty"`
 	FedoraCoreOsParentVersion string                `json:"fedora-coreos.parent-version,omitempty"`
 	Gcp                       *Gcp                  `json:"gcp,omitempty"`
@@ -59,10 +59,10 @@ type Build struct {
 	ImageInputChecksum        string                `json:"coreos-assembler.image-input-checksum,omitempty"`
 	InputHashOfTheRpmOstree   string                `json:"rpm-ostree-inputhash"`
 	Koji                      *Koji                 `json:"koji,omitempty"`
-	KubevirtContainer         *Image                `json:"kubevirt,omitempty"`
+	KubevirtContainer         *PrimaryImage         `json:"kubevirt,omitempty"`
 	MetaStamp                 float64               `json:"coreos-assembler.meta-stamp,omitempty"`
 	Name                      string                `json:"name"`
-	Oscontainer               *Image                `json:"oscontainer,omitempty"`
+	Oscontainer               *PrimaryImage         `json:"oscontainer,omitempty"`
 	OstreeCommit              string                `json:"ostree-commit"`
 	OstreeContentBytesWritten int                   `json:"ostree-content-bytes-written,omitempty"`
 	OstreeContentChecksum     string                `json:"ostree-content-checksum"`
@@ -150,12 +150,14 @@ type Git struct {
 }
 
 type Image struct {
-	Comment string `json:"comment,omitempty"`
-	Digest  string `json:"digest,omitempty"`
-	Image   string `json:"image"`
-	Tags    []Tag  `json:"tags,omitempty"`
+	Comment string     `json:"comment,omitempty"`
+	Digest  string     `json:"digest,omitempty"`
+	Image   string     `json:"image"`
+	Tags    []ImageTag `json:"tags,omitempty"`
 }
 
+type ImageTag string
+
 type Koji struct {
 	BuildRelease string  `json:"release,omitempty"`
 	KojiBuildID  float64 `json:"build_id,omitempty"`
@@ -166,10 +168,18 @@ type PackageSetDifferences []PackageSetDifferencesItems
 
 type PackageSetDifferencesItems interface{}
 
+type PrimaryImage struct {
+	AdditionalImages []interface{}     `json:"additional-images,omitempty"`
+	Comment          string            `json:"comment,omitempty"`
+	Digest           string            `json:"digest,omitempty"`
+	Image            string            `json:"image"`
+	Tags             []PrimaryImageTag `json:"tags,omitempty"`
+}
+
+type PrimaryImageTag string
+
 type S3 struct {
 	Bucket    string `json:"bucket,omitempty"`
 	Key       string `json:"key,omitempty"`
 	PublicURL string `json:"public-url,omitempty"`
 }
-
-type Tag string
diff --git a/pkg/builds/schema_doc.go b/pkg/builds/schema_doc.go
index 761b05efa0..ebed72fec3 100644
--- a/pkg/builds/schema_doc.go
+++ b/pkg/builds/schema_doc.go
@@ -1,5 +1,5 @@
 // Generated by ./generate-schema.sh
-// Source hash: 58da31f35c1405d5a752a297d0ee012f577720f810113d38cdabca102ec7edd8
+// Source hash: 1711aa3ce997c644130139e1318d2683e2a5f11538e2b0528771a8658d62ee64
 // DO NOT EDIT
 
 package builds
@@ -91,6 +91,50 @@ var generatedSchemaJSON = `{
         }
       }
     },
+    "primary-image": {
+      "type": "object",
+      "required": [
+        "image"
+      ],
+      "optional": [
+        "digest",
+        "tags",
+        "comment",
+        "additional-images"
+      ],
+      "properties": {
+        "digest": {
+          "$id": "#/image/digest",
+          "type": "string",
+          "title": "Digest"
+        },
+        "comment": {
+          "$id": "#/image/comment",
+          "type": "string",
+          "title": "Comment"
+        },
+        "image": {
+          "$id": "#/image/image",
+          "type": "string",
+          "title": "Image"
+        },
+        "tags": {
+          "$id": "#/image/tags",
+          "type": "array",
+          "title": "Tags",
+          "items": {
+            "$id": "#/image/tags/item",
+            "title": "Tag",
+            "type": "string"
+          }
+       },
+        "additional-images": {
+          "$id": "#/image",
+          "type": "array",
+          "title": "Additional images"
+        }
+      }
+    },
     "cloudartifact": {
       "type": "object",
       "required": [
@@ -708,7 +752,7 @@ var generatedSchemaJSON = `{
       "$id": "#/properties/oscontainer",
       "type": "object",
       "title": "Oscontainer",
-      "$ref": "#/definitions/image"
+      "$ref": "#/definitions/primary-image"
     },
     "extensions": {
       "$id": "#/properties/extensions",
@@ -927,13 +971,13 @@ var generatedSchemaJSON = `{
       "$id": "#/properties/base-oscontainer",
       "type": "object",
       "title": "Base OS container",
-      "$ref": "#/definitions/image"
+      "$ref": "#/definitions/primary-image"
     },
     "extensions-container": {
       "$id": "#/properties/extensions-container",
       "type": "object",
       "title": "Extensions container",
-      "$ref": "#/definitions/image"
+      "$ref": "#/definitions/primary-image"
     },
     "gcp": {
       "$id": "#/properties/gcp",
@@ -974,7 +1018,7 @@ var generatedSchemaJSON = `{
       "$id": "#/properties/kubevirt",
       "type": "object",
       "title": "Kubevirt container",
-      "$ref": "#/definitions/image"
+      "$ref": "#/definitions/primary-image"
     },
     "ibmcloud": {
       "$id": "#/properties/ibmcloud",
diff --git a/src/cmd-coreos-prune b/src/cmd-coreos-prune
index 9f4bc36618..7916b4eed2 100755
--- a/src/cmd-coreos-prune
+++ b/src/cmd-coreos-prune
@@ -55,7 +55,7 @@ Build = collections.namedtuple("Build", ["id", "images", "arch", "meta_json"])
 # set metadata caching to 5m
 CACHE_MAX_AGE_METADATA = 60 * 5
 # These lists are up to date as of schema hash
-# 58da31f35c1405d5a752a297d0ee012f577720f810113d38cdabca102ec7edd8. If changing
+# 1711aa3ce997c644130139e1318d2683e2a5f11538e2b0528771a8658d62ee64. If changing
 # this hash, ensure that the list of SUPPORTED and UNSUPPORTED artifacts below
 # is up to date.
 SUPPORTED = ["amis", "gcp"]
@@ -185,12 +185,13 @@ def main():
                                     print(f"\t\t\tRelease {build_id} is a barrier release. Skipping container prune.")
                                     continue
                                 # Retrieve container tags excluding the stream name since it updates with each release.
-                                container_tags, container_repo = get_container_tags(meta_json, exclude=[stream])
-                                if container_tags:
-                                    for tag in container_tags:
-                                        prune_container(tag, args.dry_run, container_repo, args.registry_auth_file)
-                                else:
-                                    print(f"\t\t\tNo container tags to prune for build {build_id}.")
+                                containers = get_container_tags(meta_json, exclude=[stream])
+                                for (container_repo, container_tags) in containers:
+                                    if container_tags:
+                                        for tag in container_tags:
+                                            prune_container(tag, args.dry_run, container_repo, args.registry_auth_file)
+                                    else:
+                                        print(f"\t\t\tNo container tags to prune for build {build_id}.")
                     actions_completed.append(action)  # Append action to completed list
         # Only add policy-cleanup for the build in builds.json if any
         # of the cleanup actions were completed.
@@ -450,9 +451,14 @@ def get_container_tags(meta_json, exclude):
     if base_oscontainer:
         tags = base_oscontainer.get("tags", [])
         filtered_tags = [tag for tag in tags if tag not in exclude]
-        container_repo = base_oscontainer.get("image", "")
-        return filtered_tags, container_repo
-    return [], ""
+        container_repos = [(base_oscontainer.get("image", ""), filtered_tags)]
+        additional_images = base_oscontainer.get("additional-images", [])
+        for image in additional_images:
+            tags = image.get("tags", [])
+            filtered_tags = [tag for tag in tags if tag not in exclude]
+            container_repos.append((image.get("image", ""), filtered_tags))
+        return container_repos
+    return []
 
 
 def prune_container(tag, dry_run, container_repo, registry_auth_file):
diff --git a/src/cmd-push-container-manifest b/src/cmd-push-container-manifest
index 02078d50cc..eea16c52e0 100755
--- a/src/cmd-push-container-manifest
+++ b/src/cmd-push-container-manifest
@@ -122,11 +122,17 @@ def main():
             arch = manifest['platform']['architecture']
             if arch in map_arch:
                 arch = map_arch[arch]
-            buildmetas[arch][args.metajsonname] = {
+            image = {
                 'image': args.repo,
                 'digest': manifest['digest'],
                 'tags': args.tags
             }
+            if buildmetas[arch].get(args.metajsonname):
+                additional_images = buildmetas[arch][args.metajsonname].get('additional-images', [])
+                additional_images.append(image)
+                buildmetas[arch][args.metajsonname]['additional-images'] = additional_images
+            else:
+                buildmetas[arch][args.metajsonname] = image
             buildmetas[arch].write(artifact_name=args.metajsonname)
 
 
diff --git a/src/v1.json b/src/v1.json
index f7d75843ba..15f289cb3c 100644
--- a/src/v1.json
+++ b/src/v1.json
@@ -85,6 +85,50 @@
         }
       }
     },
+    "primary-image": {
+      "type": "object",
+      "required": [
+        "image"
+      ],
+      "optional": [
+        "digest",
+        "tags",
+        "comment",
+        "additional-images"
+      ],
+      "properties": {
+        "digest": {
+          "$id": "#/image/digest",
+          "type": "string",
+          "title": "Digest"
+        },
+        "comment": {
+          "$id": "#/image/comment",
+          "type": "string",
+          "title": "Comment"
+        },
+        "image": {
+          "$id": "#/image/image",
+          "type": "string",
+          "title": "Image"
+        },
+        "tags": {
+          "$id": "#/image/tags",
+          "type": "array",
+          "title": "Tags",
+          "items": {
+            "$id": "#/image/tags/item",
+            "title": "Tag",
+            "type": "string"
+          }
+       },
+        "additional-images": {
+          "$id": "#/image",
+          "type": "array",
+          "title": "Additional images"
+        }
+      }
+    },
     "cloudartifact": {
       "type": "object",
       "required": [
@@ -702,7 +746,7 @@
       "$id": "#/properties/oscontainer",
       "type": "object",
       "title": "Oscontainer",
-      "$ref": "#/definitions/image"
+      "$ref": "#/definitions/primary-image"
     },
     "extensions": {
       "$id": "#/properties/extensions",
@@ -921,13 +965,13 @@
       "$id": "#/properties/base-oscontainer",
       "type": "object",
       "title": "Base OS container",
-      "$ref": "#/definitions/image"
+      "$ref": "#/definitions/primary-image"
     },
     "extensions-container": {
       "$id": "#/properties/extensions-container",
       "type": "object",
       "title": "Extensions container",
-      "$ref": "#/definitions/image"
+      "$ref": "#/definitions/primary-image"
     },
     "gcp": {
       "$id": "#/properties/gcp",
@@ -968,7 +1012,7 @@
       "$id": "#/properties/kubevirt",
       "type": "object",
       "title": "Kubevirt container",
-      "$ref": "#/definitions/image"
+      "$ref": "#/definitions/primary-image"
     },
     "ibmcloud": {
       "$id": "#/properties/ibmcloud",


Reviews:
user: login: cverna
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-14T17:14:57Z
user: login: jlebon
body: Needs a `make schema-update`. LGTM otherwise!
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-16T14:07:48Z
user: login: ravanelli
body: LGTM
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-17T12:30:49Z
user: login: ravanelli
body: I would change the commit message:
```
Note that I had to force the update for entries in `go.sum` as the checksums had
changed. Fixes https://github.com/coreos/coreos-assembler/issues/4043.
```
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-17T14:26:41Z
user: login: jlebon
body: You should be able to test the GC parts using `--dry-run`.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T16:39:20Z
user: login: jbtrystram
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T14:53:48Z
user: login: jlebon
body: Nice work!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-27T13:40:16Z

Review comments:
diff_hunk: @@ -122,11 +122,17 @@ def main():
             arch = manifest['platform']['architecture']
             if arch in map_arch:
                 arch = map_arch[arch]
-            buildmetas[arch][args.metajsonname] = {
-                'image': args.repo,
-                'digest': manifest['digest'],
-                'tags': args.tags
-            }
+            image = {
+                    'image': args.repo,
+                    'digest': manifest['digest'],
+                    'tags': args.tags
+                }
+            if buildmetas[arch].get(args.metajsonname, 'null') != 'null':
path: src/cmd-push-container-manifest
user: login: cverna
body: by default the value of .get() for non existing key will be `None` so the following should work:
```suggestion
            if buildmetas[arch].get(args.metajsonname):
```
created_at: 2025-03-14T17:14:56Z
author_association: MEMBER
diff_hunk: @@ -122,11 +122,17 @@ def main():
             arch = manifest['platform']['architecture']
             if arch in map_arch:
                 arch = map_arch[arch]
-            buildmetas[arch][args.metajsonname] = {
-                'image': args.repo,
-                'digest': manifest['digest'],
-                'tags': args.tags
-            }
+            image = {
+                    'image': args.repo,
+                    'digest': manifest['digest'],
+                    'tags': args.tags
+                }
path: src/cmd-push-container-manifest
user: login: jlebon
body: Minor/optional: indentation looks weird here.
created_at: 2025-03-16T13:50:49Z
author_association: MEMBER
diff_hunk: @@ -450,9 +451,14 @@ def get_container_tags(meta_json, exclude):
     if base_oscontainer:
         tags = base_oscontainer.get("tags", [])
         filtered_tags = [tag for tag in tags if tag not in exclude]
-        container_repo = base_oscontainer.get("image", "")
-        return filtered_tags, container_repo
-    return [], ""
+        container_repos = [(base_oscontainer.get("image", ""), filtered_tags)]
+        additional_images = base_oscontainer.get("additional-images", [])
+        for image in additional_images:
+            tags = image.get("tags", [])
+            filtered_tags = [tag for tag in tags if tag not in exclude]
+            container_repos.append((image, filtered_tags))
path: src/cmd-coreos-prune
user: login: jlebon
body: Hmm, shouldn't this be

```suggestion
            container_repos.append((image["image"], filtered_tags))
```

?
created_at: 2025-03-24T16:38:57Z
author_association: MEMBER
diff_hunk: @@ -450,9 +451,14 @@ def get_container_tags(meta_json, exclude):
     if base_oscontainer:
         tags = base_oscontainer.get("tags", [])
         filtered_tags = [tag for tag in tags if tag not in exclude]
-        container_repo = base_oscontainer.get("image", "")
-        return filtered_tags, container_repo
-    return [], ""
+        container_repos = [(base_oscontainer.get("image", ""), filtered_tags)]
+        additional_images = base_oscontainer.get("additional-images", [])
+        for image in additional_images:
+            tags = image.get("tags", [])
+            filtered_tags = [tag for tag in tags if tag not in exclude]
+            container_repos.append((image, filtered_tags))
path: src/cmd-coreos-prune
user: login: jbtrystram
body: yes, good catch
created_at: 2025-03-26T14:53:48Z
author_association: MEMBER

PR comments:
user: login: jbtrystram
created_at: 2025-03-14T16:52:03Z
body: Ok just tested that, pushing a couple of manifests on a testing repo : 
```
jq '."base-oscontainer"' < builds/latest/x86_64/meta.json                                                                                
{
  "image": "quay.io/jbtrystramtestimages/testrepo1",
  "digest": "sha256:4d20a109cba763ca01d2ffa8c72b3e5f4038a619d50f54946836489499eeaf89",
  "tags": [
    "stable"
  ],
  "additional-images": [
    {
      "image": "quay.io/jbtrystramtestimages/testrepo2",
      "digest": "sha256:4d20a109cba763ca01d2ffa8c72b3e5f4038a619d50f54946836489499eeaf89",
      "tags": [
        "stable"
      ]
    },
    {
      "image": "quay.io/jbtrystramtestimages/fcos",
      "digest": "sha256:4d20a109cba763ca01d2ffa8c72b3e5f4038a619d50f54946836489499eeaf89",
      "tags": [
        "stable"
      ]
    }
  ]
}

```
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-03-17T10:28:22Z
body: > Needs a `make schema-update`. LGTM otherwise!

i am hitting https://github.com/coreos/coreos-assembler/issues/4043
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-03-24T09:26:41Z
body: The test failure have nothing to do with this change

/override ci/prow/rhcos
author_association: MEMBER
user: login: openshift-ci[bot]
created_at: 2025-03-24T09:26:55Z
body: @jbtrystram: Overrode contexts on behalf of jbtrystram: ci/prow/rhcos

<details>

In response to [this](https://github.com/coreos/coreos-assembler/pull/4038#issuecomment-2747443903):

>The test failure have nothing to do with this change
>
>/override ci/prow/rhcos


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: jbtrystram
created_at: 2025-03-26T15:54:16Z
body: I just duplicated the quay image in the latest rawhide `meta.json` and ran a dry-run , and the prune script was happy :
```
 "base-oscontainer": {
    ┊ ┊ "image": "quay.io/fedora/fedora-coreos",
    ┊ ┊ "digest": "sha256:b12c0ffaae46fb2e4ee5873a2ef3b67eba4a3007e947845bdcf4fc2e70fc71d8",
    ┊ ┊ "tags": ["testing", "41.20250315.2.0"],
    ┊ ┊ "additional_images": [
    ┊ ┊ ┊ {
    ┊ ┊ ┊ ┊ "image": "quay.io/fedora/fedora-coreos",
    ┊ ┊ ┊ ┊ "digest": "sha256:b12c0ffaae46fb2e4ee5873a2ef3b67eba4a3007e947845bdcf4fc2e70fc71d8",
    ┊ ┊ ┊ ┊ "tags": ["testing", "41.20250315.2.0"]
    ┊ ┊ ┊ }
    ┊ ┊ ]
    ┊ }
  ```
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-03-27T12:30:14Z
body: /retest
author_association: MEMBER
user: login: jlebon
created_at: 2025-03-27T13:40:44Z
body: Prow CI failure here is unrelated.

/override ci/prow/rhcos
author_association: MEMBER
user: login: openshift-ci[bot]
created_at: 2025-03-27T13:40:49Z
body: @jlebon: Overrode contexts on behalf of jlebon: ci/prow/rhcos

<details>

In response to [this](https://github.com/coreos/coreos-assembler/pull/4038#issuecomment-2758085297):

>Prow CI failure here is unrelated.
>
>/override ci/prow/rhcos


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE