Repository: openshift/os
Pull request #1498: COS-2902: Add C10s variant
Author: login: travier
Labels: name: lgtm
description: Indicates that a PR is ready to be merged.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: jira/valid-reference
description: Indicates that this PR references a valid Jira ticket of any type.
Created: 2024-04-30T16:05:10Z | Merged: 2025-04-05T07:45:25Z

c9s: Point image-c9s to image-rhel-9.4

---

c9s: Use RPMs from 9.4 RHAOS repo

---

Revert "c9s.repo: temporarily use mirrored repos"

This reverts commit 88e41a0fef0dbe8e0b2d103cadd37217c8cd0933.

---

DoNotMerge: CI changes to test C10S builds

---

manifests: Add initial c10s based variant

See: https://github.com/openshift/os/issues/1466

---

kola-denylist.yaml: Update for c10s variant

---

c10s.repo: Skip GPG checks for now

---

Corresponding Fedora CoreOS changes: https://github.com/coreos/fedora-coreos-config/pull/3015


Diff:
diff --git a/README.md b/README.md
index 6f5008c7b..5584ae8c5 100644
--- a/README.md
+++ b/README.md
@@ -15,7 +15,7 @@ supported:
 
 - `rhel-9.6`: RHEL 9.6-based CoreOS; without OpenShift components.
 - `ocp-rhel-9.6`: RHEL 9.6-based CoreOS; including OpenShift components.
-- `c9s`: CentOS Stream-based CoreOS, without OKD components.
+- `c9s`/`c10s`: CentOS Stream-based CoreOS, without OKD components.
 
 In the future, the `ocp-*` variants will be removed. Instead, OpenShift
 components will be layered by deriving from the `rhel-9.X`/`c9s` images.
diff --git a/build-node-image.sh b/build-node-image.sh
index 81c2f5788..658c97e33 100755
--- a/build-node-image.sh
+++ b/build-node-image.sh
@@ -14,6 +14,9 @@ if [ "${OPENSHIFT_CI}" != 0 ]; then
     /run/src/ci/get-ocp-repo.sh /etc/yum.repos.d/ocp.repo
 fi
 
+# add all the repos from the src repo into `/etc/yum.repos.d` so dnf sees them
+cat /run/src/*.repo >> /etc/yum.repos.d/git.repo
+
 source /etc/os-release
 
 # XXX: For SCOS, only allow certain packages to come from ART; everything else
@@ -21,7 +24,7 @@ source /etc/os-release
 if [ $ID = centos ]; then
     # this says: "if the line starts with [.*], turn off printing. if the line starts with [our-repo], turn it on."
     awk "/\[.*\]/{p=0} /\[rhel-9.6-server-ose-4.19\]/{p=1} p" /etc/yum.repos.d/*.repo > /etc/yum.repos.d/okd.repo.tmp
-    sed -i -e 's,rhel-9.6-server-ose-4.19,rhel-9.6-server-ose-4.19-okd,' /etc/yum.repos.d/okd.repo.tmp
+    sed -i -e 's,\[rhel-9.6-server-ose-4.19\],\[rhel-9.6-server-ose-4.19-okd\],' /etc/yum.repos.d/okd.repo.tmp
     echo 'includepkgs=openshift-*,ose-aws-ecr-*,ose-azure-acr-*,ose-gcp-gcr-*' >> /etc/yum.repos.d/okd.repo.tmp
     mv /etc/yum.repos.d/okd.repo{.tmp,}
 fi
@@ -34,10 +37,8 @@ mkdir -p /var/opt
 rpm-ostree experimental compose treefile-apply \
     --var id=$ID /run/src/packages-openshift.yaml
 
-# cleanup the repo file we injected
-if [ "${OPENSHIFT_CI}" != 0 ]; then
-    rm /etc/yum.repos.d/ocp.repo
-fi
+# cleanup any repo files we injected
+rm -f /etc/yum.repos.d/{ocp,git,okd}.repo
 
 find /usr -name '*.pyc.bak' -exec sh -c 'mv $1 ${1%.bak}' _ {} \;
 ostree container commit
diff --git a/c10s-mirror.repo b/c10s-mirror.repo
new file mode 100644
index 000000000..403d7c902
--- /dev/null
+++ b/c10s-mirror.repo
@@ -0,0 +1,36 @@
+# These are the official c10s repos. They are slower to update, but contain older
+# versions of packages, which is useful when pinning for lack of a "coreos-pool"
+# equivalent. When no pinning is needed you may find the compose repo URLs
+# defined in c10s.repo are quicker to get new content.
+
+[c10s-baseos-mirror]
+name=CentOS Stream 10 - BaseOS
+baseurl=https://mirror.stream.centos.org/10-stream/BaseOS/$basearch/os
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
+
+[c10s-appstream-mirror]
+name=CentOS Stream 10 - AppStream
+baseurl=https://mirror.stream.centos.org/10-stream/AppStream/$basearch/os
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
+
+[c10s-nfv-mirror]
+name=CentOS Stream 10 - NFV
+baseurl=https://mirror.stream.centos.org/10-stream/NFV/$basearch/os
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
+
+[c10s-rt-mirror]
+name=CentOS Stream 10 - RT
+baseurl=https://mirror.stream.centos.org/10-stream/RT/$basearch/os
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
diff --git a/c10s.repo b/c10s.repo
new file mode 100644
index 000000000..3504eb061
--- /dev/null
+++ b/c10s.repo
@@ -0,0 +1,77 @@
+# These are compose repo URLs that represent the latest composes in
+# CentOS Stream 10. Sometimes these repos get content a little faster
+# than the mirror repos defined in c10s-mirror.repo, but they won't
+# have multiple versions of packages, which make them not ideal when
+# needing to pin on older package versions.
+
+[c10s-baseos]
+name=CentOS Stream 10 - BaseOS
+baseurl=https://composes.stream.centos.org/stream-10/production/latest-CentOS-Stream/compose/BaseOS/$basearch/os
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
+
+[c10s-appstream]
+name=CentOS Stream 10 - AppStream
+baseurl=https://composes.stream.centos.org/stream-10/production/latest-CentOS-Stream/compose/AppStream/$basearch/os
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
+
+[c10s-nfv]
+name=CentOS Stream 10 - NFV
+baseurl=https://composes.stream.centos.org/stream-10/production/latest-CentOS-Stream/compose/NFV/$basearch/os
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
+
+[c10s-rt]
+name=CentOS Stream 10 - RT
+baseurl=https://composes.stream.centos.org/stream-10/production/latest-CentOS-Stream/compose/RT/$basearch/os
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
+
+# Note: We can't find a composes.stream.centos.org URL for this repo
+# so we use the mirror.stream.centos.org URL here.
+[c10s-extras-common]
+name=CentOS Stream 10 - Extras packages
+baseurl=https://mirror.stream.centos.org/SIGs/10-stream/extras/$basearch/extras-common
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
+
+# Note: We can't find a composes.stream.centos.org URL for this repo
+# so we use the mirror.stream.centos.org URL here.
+[c10s-sig-nfv]
+name=CentOS Stream 10 - SIG NFV
+baseurl=https://mirror.stream.centos.org/SIGs/10-stream/nfv/$basearch/openvswitch-2/
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-NFV
+
+# Note: We can't find a composes.stream.centos.org URL for this repo
+# so we use the mirror.stream.centos.org URL here.
+[c10s-sig-virtualization]
+name=CentOS Stream 10 - SIG Virtualization
+baseurl=https://mirror.stream.centos.org/SIGs/10-stream/virt/$basearch/kata-containers/
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Virtualization
+
+# Note: We can't find a composes.stream.centos.org URL for this repo
+# so we use the mirror.stream.centos.org URL here.
+[c10s-sig-cloud-okd]
+name=CentOS Stream 10 - SIG Cloud OKD 4.19
+baseurl=https://mirror.stream.centos.org/SIGs/10-stream/cloud/$basearch/okd-4.19/
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud
diff --git a/c9s.repo b/c9s.repo
index 0297db80a..61ec9c886 100644
--- a/c9s.repo
+++ b/c9s.repo
@@ -20,17 +20,6 @@ repo_gpgcheck=0
 enabled=1
 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
 
-[c9s-extras-common]
-name=CentOS Stream 9 - Extras packages
-# Note: the hardcoded x86_64 is not a mistake. Extras just has noarch RPMs that
-# contain GPG keys and repo files but for some reason only the x86_64 repo has
-# them...
-baseurl=https://mirror.stream.centos.org/SIGs/9-stream/extras/x86_64/extras-common
-gpgcheck=1
-repo_gpgcheck=0
-enabled=1
-gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
-
 [c9s-nfv]
 name=CentOS Stream 9 - NFV
 baseurl=https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/NFV/$basearch/os
@@ -47,6 +36,21 @@ repo_gpgcheck=0
 enabled=1
 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
 
+# Note: the hardcoded x86_64 is not a mistake. Extras just has noarch RPMs that
+# contain GPG keys and repo files but for some reason only the x86_64 repo has
+# them... Drop hardcoded arch once https://pagure.io/centos-infra/issue/1635 is fixed
+# Note: We can't find a composes.stream.centos.org URL for this repo
+# so we use the mirror.stream.centos.org URL here.
+[c9s-extras-common]
+name=CentOS Stream 9 - Extras packages
+baseurl=https://mirror.stream.centos.org/SIGs/9-stream/extras/x86_64/extras-common
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
+
+# Note: We can't find a composes.stream.centos.org URL for this repo
+# so we use the mirror.stream.centos.org URL here.
 [c9s-sig-nfv]
 name=CentOS Stream 9 - SIG NFV
 baseurl=https://mirror.stream.centos.org/SIGs/9-stream/nfv/$basearch/openvswitch-2/
@@ -55,6 +59,8 @@ repo_gpgcheck=0
 enabled=1
 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-NFV
 
+# Note: We can't find a composes.stream.centos.org URL for this repo
+# so we use the mirror.stream.centos.org URL here.
 [c9s-sig-virtualization]
 name=CentOS Stream 9 - SIG Virtualization
 baseurl=https://mirror.stream.centos.org/SIGs/9-stream/virt/$basearch/kata-containers/
@@ -63,6 +69,8 @@ repo_gpgcheck=0
 enabled=1
 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Virtualization
 
+# Note: We can't find a composes.stream.centos.org URL for this repo
+# so we use the mirror.stream.centos.org URL here.
 [c9s-sig-cloud-okd]
 name=CentOS Stream 9 - SIG Cloud OKD 4.19
 baseurl=https://mirror.stream.centos.org/SIGs/9-stream/cloud/$basearch/okd-4.19/
diff --git a/ci/prow-entrypoint.sh b/ci/prow-entrypoint.sh
index d49a60e03..65e07b0a7 100755
--- a/ci/prow-entrypoint.sh
+++ b/ci/prow-entrypoint.sh
@@ -82,16 +82,14 @@ cosa_build_extensions() {
 
 # Build QEMU image and run all kola tests
 kola_test_qemu() {
-    cosa buildextend-qemu
+    cosa osbuild qemu
     cosa kola run --parallel 2 --output-dir ${ARTIFACT_DIR:-/tmp}/kola --rerun --allow-rerun-success tags=needs-internet "$@"
 }
 
 # Build metal, metal4k & live images and run kola tests
 kola_test_metal() {
     # Build metal + installer now so we can test them
-    cosa buildextend-metal
-    cosa buildextend-metal4k
-    cosa buildextend-live
+    cosa osbuild metal metal4k live
 
     # Compress the metal and metal4k images now so we're testing
     # installs with the image format we ship
@@ -101,34 +99,6 @@ kola_test_metal() {
     kola testiso -S --output-dir ${ARTIFACT_DIR:-/tmp}/kola-testiso  --denylist-test iso-offline-install-iscsi* --denylist-test pxe-offline-install.rootfs-appended.bios
 }
 
-# Ensure that we can create all platform images for COSA CI
-cosa_buildextend_all() {
-    cosa buildextend-aliyun
-    cosa buildextend-aws
-    cosa buildextend-azure
-    cosa buildextend-azurestack
-    cosa buildextend-dasd
-    cosa buildextend-gcp
-    cosa buildextend-ibmcloud
-    cosa buildextend-kubevirt
-    cosa buildextend-live
-    cosa buildextend-metal
-    cosa buildextend-metal4k
-    cosa buildextend-nutanix
-    cosa buildextend-openstack
-    cosa buildextend-powervs
-    cosa buildextend-vmware
-
-    # Will be done in another step
-    # cosa buildextend-qemu
-
-    # Currently not available for RHCOS
-    # cosa buildextend-digitalocean
-    # cosa buildextend-exoscale
-    # cosa buildextend-virtualbox
-    # cosa buildextend-vultr
-}
-
 # Basic syntaxt validation for manifests
 validate() {
     # Create a temporary copy
@@ -312,10 +282,16 @@ main() {
             kola_test_metal
             ;;
         "scos-10-build-test-qemu")
-            exit 0
+            setup_user
+            cosa_init "c10s"
+            cosa_build
+            kola_test_qemu --tag '!openshift'
             ;;
         "scos-10-build-test-metal")
-            exit 0
+            setup_user
+            cosa_init "c10s"
+            cosa_build
+            kola_test_metal
             ;;
         "rhcos-10-build-test-qemu")
             exit 0
diff --git a/common.yaml b/common.yaml
index e142c8cd9..641723cb9 100644
--- a/common.yaml
+++ b/common.yaml
@@ -31,15 +31,18 @@ conditional-include:
   - if: basearch != "s390x"
     # And remove some cruft from grub2
     include: fedora-coreos-config/manifests/grub2-removals.yaml
-  - if: osversion == "c9s"
-    include: manifest-el9-shared.yaml
-  - if: osversion == "rhel-9.6"
-    include: manifest-el9-shared.yaml
   # Packages specific to el9
   - if: osversion == "c9s"
-    include: fedora-coreos-config/manifests/shared-el9.yaml
+    include:
+      - fedora-coreos-config/manifests/shared-el9.yaml
+      - manifest-el9-shared.yaml
   - if: osversion == "rhel-9.6"
-    include: fedora-coreos-config/manifests/shared-el9.yaml
+    include:
+      - fedora-coreos-config/manifests/shared-el9.yaml
+      - manifest-el9-shared.yaml
+  - if: osversion == "c10s"
+    include:
+      - fedora-coreos-config/manifests/shared-el10.yaml
   - if: inherit_tier_x == true
     include: fedora-coreos-config/manifests/tier-x.yaml
   - if: inherit_tier_x == false
@@ -222,7 +225,6 @@ packages:
  # Bootloader updater
  - bootupd
  # Containers
- - containernetworking-plugins
  - container-selinux
  # Needed for newer rpm-ostree
  - "'skopeo >= 2:1.7.0'"
@@ -253,9 +255,6 @@ packages:
  - qemu-guest-agent
  # passwd was obsoleted by shadow-utils in F40+, but still needed here
  - passwd
- # Gluster - Used for Openshift e2e gluster testcases
- # Reverts https://gitlab.cee.redhat.com/coreos/redhat-coreos/merge_requests/367 and add it for all arches
- - glusterfs-fuse
  # Needed for kernel-devel extension: https://bugzilla.redhat.com/show_bug.cgi?id=1885408
  # x86_64 and s390x have these packages installed as dependencies of other packages, ppc64le does not
  # FIXME: once the below BZs have been resolved to remove perl dependencies, this can be done in the extensions script
diff --git a/extensions-okd-c10s.yaml b/extensions-okd-c10s.yaml
new file mode 100644
index 000000000..47860d278
--- /dev/null
+++ b/extensions-okd-c10s.yaml
@@ -0,0 +1,103 @@
+# RPMs as operating system extensions, distinct from the base ostree commit/image
+# https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md
+# and https://github.com/coreos/fedora-coreos-tracker/issues/401
+
+# RULE: Do not add repos to specific extensions below if the extension is not
+# multi-arch, but the repos are. Instead, put them in the global repos list at
+# the top. Otherwise, we can have consistency issues across arches. See e.g.
+# https://issues.redhat.com/browse/OCPBUGS-52293.
+
+repos:
+  - c10s-sig-nfv
+  # Some of the extensions here have version bindings to host packages. Add the
+  # mirrors since those retain multiple versions of packages in case the latest
+  # compose has already moved since the last base image build.
+  - c10s-baseos-mirror
+  - c10s-appstream-mirror
+  - c10s-nfv-mirror
+  - c10s-rt-mirror
+
+extensions:
+  # https://issues.redhat.com/browse/RFE-4177
+  # wasm:
+  #   architectures:
+  #     - x86_64
+  #     - aarch64
+  #   packages:
+  #     - crun-wasm
+  # https://github.com/coreos/fedora-coreos-tracker/issues/1504
+  ipsec:
+    packages:
+      - libreswan
+      - NetworkManager-libreswan
+      - openvswitch3.5-ipsec
+  # https://github.com/coreos/fedora-coreos-tracker/issues/326
+  usbguard:
+    packages:
+      - usbguard
+  kerberos:
+    packages:
+      - krb5-workstation
+      - libkadm5
+  # https://github.com/kmods-via-containers/kmods-via-containers/issues/3
+  # https://gitlab.cee.redhat.com/coreos/redhat-coreos/merge_requests/866
+  # These are currently overlaid onto the host so that they can be bind-mounted
+  # into build containers... in the future they should be a `development`
+  # extension: https://github.com/openshift/machine-config-operator/pull/2143.
+  kernel-devel:
+    packages:
+      - kernel-devel
+      - kernel-headers
+    match-base-evr: kernel
+  # These are already in the base, so they're not OS extensions, but they're
+  # useful to have in RPM form to install in kmod build containers.
+  kernel:
+    kind: development
+    packages:
+      - kernel
+      - kernel-core
+      - kernel-modules
+      - kernel-modules-extra
+    match-base-evr: kernel
+  # GRPA-2822
+  # https://github.com/openshift/machine-config-operator/pull/1330
+  # https://github.com/openshift/enhancements/blob/master/enhancements/support-for-realtime-kernel.md
+  kernel-rt:
+    architectures:
+      - x86_64
+    repos:
+      # this is not available on all arches, so keep here and not in the global repo list
+      - c10s-nfv
+    packages:
+      - kernel-rt-core
+      - kernel-rt-kvm
+      - kernel-rt-modules
+      - kernel-rt-modules-extra
+      - kernel-rt-devel
+    match-base-evr: kernel
+  # https://github.com/openshift/machine-config-operator/pull/2456
+  # https://github.com/openshift/enhancements/blob/master/enhancements/sandboxed-containers/sandboxed-containers-tech-preview.md
+  # GRPA-3123
+  # kata-containers is not yet shipped in CS10, see https://mirror.stream.centos.org/SIGs/10-stream/virt/x86_64/
+  # Please uncomment the section once available
+  # sandboxed-containers:
+  #   architectures:
+  #     - x86_64
+  #   repos:
+  #     # this is not available on all arches, so keep here and not in the global repo list
+  #     - c10s-sig-virtualization
+  #   packages:
+  #     - kata-containers
+  # https://issues.redhat.com/browse/COS-2402
+  kernel-64k:
+    architectures:
+      - aarch64
+    packages:
+      - kernel-64k-core
+      - kernel-64k-modules
+      - kernel-64k-modules-core
+      - kernel-64k-modules-extra
+  # https://issues.redhat.com/browse/COS-2940
+  sysstat:
+    packages:
+      - sysstat
diff --git a/extensions-okd-c9s.yaml b/extensions-okd-c9s.yaml
index 8b522c466..e3cb4ac28 100644
--- a/extensions-okd-c9s.yaml
+++ b/extensions-okd-c9s.yaml
@@ -8,7 +8,7 @@
 # https://issues.redhat.com/browse/OCPBUGS-52293.
 
 repos:
-  - c9s-sig-virtualization
+  - c9s-sig-nfv
   # Some of the extensions here have version bindings to host packages. Add the
   # mirrors since those retain multiple versions of packages in case the latest
   # compose has already moved since the last base image build.
@@ -23,8 +23,6 @@ extensions:
   #   architectures:
   #     - x86_64
   #     - aarch64
-  #   repos:
-  #     - appstream
   #   packages:
   #     - crun-wasm
   # https://github.com/coreos/fedora-coreos-tracker/issues/1504
@@ -68,6 +66,7 @@ extensions:
     architectures:
       - x86_64
     repos:
+      # this is not available on all arches, so keep here and not in the global repo list
       - c9s-nfv
     packages:
       - kernel-rt-core
@@ -79,11 +78,11 @@ extensions:
   # https://github.com/openshift/machine-config-operator/pull/2456
   # https://github.com/openshift/enhancements/blob/master/enhancements/sandboxed-containers/sandboxed-containers-tech-preview.md
   # GRPA-3123
-  # - kata-containers
   sandboxed-containers:
     architectures:
       - x86_64
     repos:
+      # this is not available on all arches, so keep here and not in the global repo list
       - c9s-sig-virtualization
     packages:
       - kata-containers
diff --git a/image-c10s.yaml b/image-c10s.yaml
new file mode 120000
index 000000000..14763a255
--- /dev/null
+++ b/image-c10s.yaml
@@ -0,0 +1 @@
+image-rhel-9.6.yaml
\ No newline at end of file
diff --git a/kola-denylist.yaml b/kola-denylist.yaml
index 148b961a0..3a9b9f7aa 100644
--- a/kola-denylist.yaml
+++ b/kola-denylist.yaml
@@ -7,16 +7,19 @@
   tracker: https://github.com/openshift/os/issues/1237
   osversion:
     - c9s
+    - c10s
 
 - pattern: iso-live-login.uefi-secure
   tracker: https://github.com/openshift/os/issues/1237
   osversion:
     - c9s
+    - c10s
 
 - pattern: iso-as-disk.uefi-secure
   tracker: https://github.com/openshift/os/issues/1237
   osversion:
     - c9s
+    - c10s
 
 # The 4.17 and 4.18 build of Ignition encounters a FIPS panic so
 # we are using the 4.16 build for now while that is under investigation.
@@ -42,3 +45,44 @@
   tracker: https://github.com/openshift/os/issues/1751
   osversion:
     - c10s
+
+# Will fail until we have proper RHEL 10 builds of OCP packages
+- pattern: ext.config.version.rhel-major-version
+  tracker: TODO
+  osversion:
+    - c10s
+
+- pattern: multipath.day1
+  tracker: https://github.com/coreos/fedora-coreos-tracker/issues/1894
+  osversion:
+    - c10s
+
+- pattern: multipath.day2
+  tracker: https://github.com/coreos/fedora-coreos-tracker/issues/1894
+  osversion:
+    - c10s
+
+- pattern: multipath.partition
+  tracker: https://github.com/coreos/fedora-coreos-tracker/issues/1894
+  osversion:
+    - c10s
+
+- pattern: iso-offline-install*mpath.bios
+  tracker: https://github.com/coreos/fedora-coreos-tracker/issues/1894
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.root-reprovision.luks.multipath
+  tracker: https://github.com/coreos/fedora-coreos-tracker/issues/1894
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.networking.nm-ifcfg-rh-plugin
+  tracker: https://github.com/openshift/os/pull/1759#issuecomment-2779700581
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.networking.team-dhcp-via-ignition
+  tracker: https://github.com/openshift/os/pull/1759#issuecomment-2779700581
+  osversion:
+    - c10s
diff --git a/manifest-c10s.yaml b/manifest-c10s.yaml
new file mode 100644
index 000000000..fedc5321b
--- /dev/null
+++ b/manifest-c10s.yaml
@@ -0,0 +1,33 @@
+# Manifest for CentOS Stream CoreOS 10
+
+metadata:
+  license: MIT
+  name: scos
+  summary: CentOS Stream CoreOS 10
+
+variables:
+  id: "centos"
+  osversion: "c10s"
+  inherit_tier_x: true
+
+# Include manifests common to all RHEL and CentOS Stream versions
+include:
+  - common.yaml
+  - overrides-c10s.yaml
+
+repos:
+  - c10s-baseos
+  - c10s-appstream
+  - c10s-extras-common
+
+automatic-version-prefix: "10.0.<date:%Y%m%d>"
+# This ensures we're semver-compatible which OpenShift wants
+automatic-version-suffix: "-"
+
+mutate-os-release: "10"
+
+packages:
+ - centos-stream-release
+ - centos-release-cloud-common
+ - centos-release-nfv-common
+ - centos-release-virt-common
diff --git a/manifest-el9-shared.yaml b/manifest-el9-shared.yaml
index 7ae800c9f..4f592852e 100644
--- a/manifest-el9-shared.yaml
+++ b/manifest-el9-shared.yaml
@@ -1,5 +1,11 @@
 # Place here configuration that should happen on all el9* builds
 
+packages:
+ - containernetworking-plugins
+ # Gluster - Used for Openshift e2e gluster testcases
+ # Reverts https://gitlab.cee.redhat.com/coreos/redhat-coreos/merge_requests/367 and add it for all arches
+ - glusterfs-fuse
+
 #zram default config is in a subpackage in c10s
 # Meanwhile, remove the default config from the package
 remove-from-packages:
diff --git a/overlay.d/05rhcos/usr/lib/dracut/modules.d/40rhcos-fips/rhcos-fips.sh b/overlay.d/05rhcos/usr/lib/dracut/modules.d/40rhcos-fips/rhcos-fips.sh
index ac9129253..c28556bf7 100755
--- a/overlay.d/05rhcos/usr/lib/dracut/modules.d/40rhcos-fips/rhcos-fips.sh
+++ b/overlay.d/05rhcos/usr/lib/dracut/modules.d/40rhcos-fips/rhcos-fips.sh
@@ -69,6 +69,13 @@ finish() {
         fatal "FIPS mode is not enabled."
     fi
 
+    # on EL10 fips-mode-setup was removed and these steps are
+    # no longer needed. Please delete this function and sysroot_bwrap
+    # when EL9 is no longer supported.
+    if [ ! -e /sysroot/usr/bin/fips-mode-setup ]; then
+        return 0
+    fi
+
     # If we're running from a live system, then set things up so that the dracut fips
     # module will find the kernel binary.  TODO change dracut to look in /usr/lib/modules/$(uname -r)
     # directly.
diff --git a/overrides-c10s.yaml b/overrides-c10s.yaml
new file mode 100644
index 000000000..c32622859
--- /dev/null
+++ b/overrides-c10s.yaml
@@ -0,0 +1,10 @@
+# This is a poor man's version of an override lockfile for c10s. When needed, we
+# can enable the mirror repos here (which hold older RPMs) and list the NEVRs
+# we need in the `packages` section. When not needed. Empty or comment out this
+# file (except this comment).
+
+repos:
+ - c10s-baseos-mirror
+ - c10s-appstream-mirror
+
+#packages:
diff --git a/tests/kola/rpm-ostree/replace-rt-kernel/data/c10s.repo b/tests/kola/rpm-ostree/replace-rt-kernel/data/c10s.repo
new file mode 100644
index 000000000..6622a6bb6
--- /dev/null
+++ b/tests/kola/rpm-ostree/replace-rt-kernel/data/c10s.repo
@@ -0,0 +1,43 @@
+# This repo is based on the c10s.repo file at the top-level, but *always* refers
+# to the mirrors and never to the latest compose output. For more information, see:
+# https://github.com/openshift/os/commit/15c7bf96b70b014f6dd55f25195cf1a341b6c6e0
+
+[baseos]
+name=CentOS Stream 10 - BaseOS
+baseurl=https://mirror.stream.centos.org/10-stream/BaseOS/$basearch/os
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+
+[appstream]
+name=CentOS Stream 10 - AppStream
+baseurl=https://mirror.stream.centos.org/10-stream/AppStream/$basearch/os
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+
+[extras-common]
+name=CentOS Stream 10 - Extras packages
+baseurl=https://mirror.stream.centos.org/SIGs/10-stream/extras/$basearch/extras-common
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+
+[nfv]
+name=CentOS Stream 10 - NFV
+baseurl=https://mirror.stream.centos.org/10-stream/NFV/$basearch/os
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
+
+[rt]
+name=CentOS Stream 10 - RT
+baseurl=https://mirror.stream.centos.org/10-stream/RT/$basearch/os
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
+gpgcheck=1
+repo_gpgcheck=0
+enabled=1
diff --git a/tests/kola/rpm-ostree/replace-rt-kernel/test.sh b/tests/kola/rpm-ostree/replace-rt-kernel/test.sh
index 83713f92c..ac45f4914 100755
--- a/tests/kola/rpm-ostree/replace-rt-kernel/test.sh
+++ b/tests/kola/rpm-ostree/replace-rt-kernel/test.sh
@@ -25,9 +25,15 @@ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
     case "${major}" in
         9)
             repo_name=c9s.repo
+            if [ ! -e /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official ]; then
+                runv curl -sSLf https://centos.org/keys/RPM-GPG-KEY-CentOS-Official-SHA256 -o /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official
+            fi
             ;;
         10)
             repo_name=c10s.repo
+            if [ ! -e /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256 ]; then
+                runv curl -sSLf https://centos.org/keys/RPM-GPG-KEY-CentOS-Official-SHA256 -o /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
+            fi
             ;;
         *)  fatal "Unhandled major RHEL/SCOS VERSION=${major}"
             ;;
@@ -36,7 +42,6 @@ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
     # setup repos
     runv rm -rf /etc/yum.repos.d/*
     runv cp "$KOLA_EXT_DATA/$repo_name" /etc/yum.repos.d/cs.repo
-    runv curl -sSLf https://centos.org/keys/RPM-GPG-KEY-CentOS-Official -o /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official
     # Disable all repos except baseos and appstream as not all of them have support for all RHCOS/SCOS supported architectures
     runv sed -i 's/enabled=1/enabled=0/g' /etc/yum.repos.d/cs.repo
     runv sed -i '/\[baseos\]/,/^ *\[/ s/enabled=0/enabled=1/' /etc/yum.repos.d/cs.repo


Reviews:
user: login: LorbusChris
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-06-06T21:56:12Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-17T20:45:25Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-18T09:55:24Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-18T09:57:16Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-18T09:58:22Z
user: login: jbtrystram
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-18T13:05:45Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-18T19:54:23Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-18T19:58:09Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-18T19:58:50Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-18T20:00:15Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-19T10:22:12Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-19T10:23:19Z
user: login: jbtrystram
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-19T10:25:04Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-19T10:26:14Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-19T13:25:35Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-19T14:04:09Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-19T14:33:29Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-19T15:38:13Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-19T16:39:12Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-20T13:55:36Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-20T13:57:28Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-20T13:59:16Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-20T13:59:52Z
user: login: travier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-20T14:00:29Z
user: login: jbtrystram
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-21T09:46:33Z
user: login: jbtrystram
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-21T11:30:32Z
user: login: jlebon
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T15:35:23Z
user: login: jbtrystram
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T16:38:07Z
user: login: jlebon
body: Thanks all for working on this! A lot going on here. A bunch of commits here should be squashed together, others can be dropped. Can we add commit messages to the commits that have none?

I think to start, let's merge the smaller prep commits which don't directly introduce c10s-related manifests in a separate PR?
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T16:45:32Z
user: login: jbtrystram
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T11:31:32Z
user: login: jlebon
body: Thank you for adding more context in the commits!

First commit says "aglusterfs-fusend" which I think is a typo.

Otherwise, this looks sane to me!

/approve

I'd add a `/lgtm` but this will need a rebase anyway for https://github.com/openshift/os/pull/1780.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-04T16:06:55Z
user: login: jbtrystram
body: A few denylist entries can probably be removed
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T17:58:30Z
user: login: joelcapitao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T18:36:30Z
user: login: jbtrystram
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T18:40:46Z
user: login: joelcapitao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T18:42:51Z
user: login: joelcapitao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T18:44:12Z
user: login: joelcapitao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T18:44:47Z
user: login: joelcapitao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T18:44:53Z
user: login: joelcapitao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T18:45:04Z
user: login: jbtrystram
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T18:55:20Z
user: login: Prashanth684
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T20:59:28Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-05T00:03:18Z

Review comments:
diff_hunk: @@ -0,0 +1,63 @@
+[c10s-baseos]
+name=CentOS Stream 10 - BaseOS
+baseurl=https://composes.stream.centos.org/stream-10/production/latest-CentOS-Stream/compose/BaseOS/$basearch/os
+gpgcheck=0
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
+
+[c10s-appstream]
+name=CentOS Stream 10 - AppStream
+baseurl=https://composes.stream.centos.org/stream-10/production/latest-CentOS-Stream/compose/AppStream/$basearch/os
+gpgcheck=0
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
+
+# [c10s-extras-common]
+# name=CentOS Stream 10 - Extras packages
+# baseurl=https://mirror.stream.centos.org/SIGs/10-stream/extras/$basearch/extras-common
+# gpgcheck=1
+# repo_gpgcheck=0
+# enabled=1
+# gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
+
+[c10s-nfv]
+name=CentOS Stream 10 - NFV
+baseurl=https://composes.stream.centos.org/stream-10/production/latest-CentOS-Stream/compose/NFV/$basearch/os
+gpgcheck=0
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
+
+[c10s-rt]
+name=CentOS Stream 10 - RT
+baseurl=https://composes.stream.centos.org/stream-10/production/latest-CentOS-Stream/compose/RT/$basearch/os
+gpgcheck=0
+repo_gpgcheck=0
+enabled=1
+gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
+
+# [c10s-sig-nfv]
+# name=CentOS Stream 10 - SIG NFV
+# baseurl=https://mirror.stream.centos.org/SIGs/10-stream/nfv/$basearch/openvswitch-2/
+# gpgcheck=1
+# repo_gpgcheck=0
+# enabled=1
+# gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-SIG-NFV
+
+# [c10s-sig-virtualization]
+# name=CentOS Stream 10 - SIG Virtualization
+# baseurl=https://mirror.stream.centos.org/SIGs/10-stream/virt/$basearch/kata-containers/
+# gpgcheck=1
+# repo_gpgcheck=0
+# enabled=1
+# gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-SIG-Virtualization
+
+# [c10s-sig-cloud-okd]
+# name=CentOS Stream 10 - SIG Cloud OKD 4.15
+# baseurl=https://mirror.stream.centos.org/SIGs/10-stream/cloud/$basearch/okd-4.15/
path: c10s.repo
user: login: LorbusChris
body: ```suggestion
# baseurl=https://mirror.stream.centos.org/SIGs/10-stream/cloud/$basearch/okd-4.17/
```
This repo doesn't exist yet, but let's point it at 4.17 already. I'll set up the repo in the coming days.
created_at: 2024-06-06T21:56:12Z
author_association: CONTRIBUTOR
path: common-el9.yaml
user: login: dustymabe
body: I'm not opposed to the `common-el9.yaml` name here, but I think we should make an initial commit first that renames `manifest-el9-shared.yaml -> common-el9.yaml` and then add new changes to it. 
created_at: 2025-02-17T20:28:22Z
author_association: MEMBER
diff_hunk: @@ -37,9 +37,13 @@ conditional-include:
     include: manifest-el9-shared.yaml
   # Packages specific to el9
   - if: osversion == "c9s"
-    include: fedora-coreos-config/manifests/shared-el9.yaml
+    include:
+      - fedora-coreos-config/manifests/shared-el9.yaml
+      - common-el9.yaml
   - if: osversion == "rhel-9.6"
-    include: fedora-coreos-config/manifests/shared-el9.yaml
+    include:
+      - fedora-coreos-config/manifests/shared-el9.yaml
+      - common-el9.yaml
path: common.yaml
user: login: dustymabe
body: neat trick. didn't know you could specify multiple to include..


created_at: 2025-02-17T20:30:03Z
author_association: MEMBER
diff_hunk: @@ -10,14 +10,27 @@ ARG COSA
 ARG VARIANT
 RUN if [[ -z "$COSA" ]] ; then ci/get-ocp-repo.sh --ocp-layer packages-openshift.yaml; fi
 # on SCOS, we need to add the GPG keys of the various SIGs we need
-RUN if rpm -q centos-stream-release && ! rpm -q centos-release-cloud; then dnf install -y centos-release-{cloud,nfv,virt}-common; fi
-RUN mkdir -p /usr/share/distribution-gpg-keys/centos
-RUN ln -s /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial /usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Cloud
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-NFV
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Virtualization
-RUN if [[ -n "${VARIANT}" ]]; then MANIFEST="manifest-${VARIANT}.yaml"; EXTENSIONS="extensions-${VARIANT}.yaml"; else MANIFEST="manifest.yaml"; EXTENSIONS="extensions.yaml"; fi && rpm-ostree compose extensions --rootfs=/ --output-dir=/usr/share/rpm-ostree/extensions/ ./"${MANIFEST}" ./"${EXTENSIONS}"
+RUN if rpm -q centos-stream-release; then \
path: extensions/Dockerfile
user: login: dustymabe
body: Can we use the `RUN << EOF` pattern here instead of having the line continuation character `\` on each line?

Might make things easier to read.
created_at: 2025-02-17T20:31:41Z
author_association: MEMBER
diff_hunk: @@ -10,14 +10,27 @@ ARG COSA
 ARG VARIANT
 RUN if [[ -z "$COSA" ]] ; then ci/get-ocp-repo.sh --ocp-layer packages-openshift.yaml; fi
 # on SCOS, we need to add the GPG keys of the various SIGs we need
-RUN if rpm -q centos-stream-release && ! rpm -q centos-release-cloud; then dnf install -y centos-release-{cloud,nfv,virt}-common; fi
-RUN mkdir -p /usr/share/distribution-gpg-keys/centos
-RUN ln -s /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial /usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Cloud
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-NFV
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Virtualization
-RUN if [[ -n "${VARIANT}" ]]; then MANIFEST="manifest-${VARIANT}.yaml"; EXTENSIONS="extensions-${VARIANT}.yaml"; else MANIFEST="manifest.yaml"; EXTENSIONS="extensions.yaml"; fi && rpm-ostree compose extensions --rootfs=/ --output-dir=/usr/share/rpm-ostree/extensions/ ./"${MANIFEST}" ./"${EXTENSIONS}"
path: extensions/Dockerfile
user: login: dustymabe
body: +1 for breaking up this long line, but maybe we could also use the `RUN <<EOF` model here too? 
created_at: 2025-02-17T20:32:02Z
author_association: MEMBER
path: extensions-c10s.yaml
user: login: dustymabe
body: worth pointing out there is no `extensions-c9s.yaml` - maybe we want to match was is being done for c9s with the naming here?

OR this is a better scheme and we want to change c9s to match. 
created_at: 2025-02-17T20:38:51Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,85 @@
+# RPMs as operating system extensions, distinct from the base ostree commit/image
+# https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md
+# and https://github.com/coreos/fedora-coreos-tracker/issues/401
+
+# repos:
+#   - sig-virtualization
+
+extensions:
+  # https://issues.redhat.com/browse/RFE-4177
+  # wasm:
+  #   architectures:
+  #     - x86_64
+  #     - aarch64
+  #   repos:
+  #     - appstream
+  #   packages:
+  #     - crun-wasm
+  # https://github.com/coreos/fedora-coreos-tracker/issues/1504
+  ipsec:
+    packages:
+      - libreswan
+      - NetworkManager-libreswan
+  # https://github.com/coreos/fedora-coreos-tracker/issues/326
+  usbguard:
+    packages:
+      - usbguard
+  kerberos:
+    packages:
+      - krb5-workstation
+      - libkadm5
+  # https://github.com/kmods-via-containers/kmods-via-containers/issues/3
+  # https://gitlab.cee.redhat.com/coreos/redhat-coreos/merge_requests/866
+  # These are currently overlaid onto the host so that they can be bind-mounted
+  # into build containers... in the future they should be a `development`
+  # extension: https://github.com/openshift/machine-config-operator/pull/2143.
+  kernel-devel:
+    packages:
+      - kernel-devel
+      - kernel-headers
+    match-base-evr: kernel
+  # These are already in the base, so they're not OS extensions, but they're
+  # useful to have in RPM form to install in kmod build containers.
+  kernel:
+    kind: development
+    packages:
+      - kernel
+      - kernel-core
+      - kernel-modules
+      - kernel-modules-extra
+    match-base-evr: kernel
+  # GRPA-2822
+  # https://github.com/openshift/machine-config-operator/pull/1330
+  # https://github.com/openshift/enhancements/blob/master/enhancements/support-for-realtime-kernel.md
+  kernel-rt:
+    architectures:
+      - x86_64
+    repos:
+      - c10s-nfv
+    packages:
+      - kernel-rt-core
+      - kernel-rt-kvm
+      - kernel-rt-modules
+      - kernel-rt-modules-extra
+      - kernel-rt-devel
+    match-base-evr: kernel
+  # https://github.com/openshift/machine-config-operator/pull/2456
+  # https://github.com/openshift/enhancements/blob/master/enhancements/sandboxed-containers/sandboxed-containers-tech-preview.md
+  # GRPA-3123
+  # - kata-containers
+  # sandboxed-containers:
+  #   architectures:
+  #     - x86_64
+  #   repos:
+  #     - sig-virtualization10
+  #   packages:
+  #     - kata-containers
+  # https://issues.redhat.com/browse/COS-2402
+  kernel-64k:
+    architectures:
+      - aarch64
+    packages:
+      - kernel-64k-core
+      - kernel-64k-modules
+      - kernel-64k-modules-core
+      - kernel-64k-modules-extra
path: extensions-c10s.yaml
user: login: dustymabe
body: should sysstat be down here? it's in the c9s one. 
created_at: 2025-02-17T20:39:31Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,30 @@
+# Manifest for CentOS Stream CoreOS 10
+
+rojig:
path: manifest-c10s.yaml
user: login: dustymabe
body: ```suggestion
metadata:
```
created_at: 2025-02-17T20:40:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,30 @@
+# Manifest for CentOS Stream CoreOS 10
+
+rojig:
+  license: MIT
+  name: scos
+  summary: CentOS Stream CoreOS 10
+
+variables:
+  osversion: "c10s"
+  inherit_tier_x: true
+
+# Include manifests common to all RHEL and CentOS Stream versions
+include:
+  - common.yaml
+  - common-el10.yaml
+
+repos:
+  - c10s-baseos
+  - c10s-appstream
+
+automatic-version-prefix: "10.<date:%Y%m%d%H%M>"
path: manifest-c10s.yaml
user: login: dustymabe
body: ```suggestion
automatic-version-prefix: "10.0.<date:%Y%m%d>"
```
created_at: 2025-02-17T20:41:40Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,30 @@
+# Manifest for CentOS Stream CoreOS 10
+
+rojig:
+  license: MIT
+  name: scos
+  summary: CentOS Stream CoreOS 10
+
+variables:
+  osversion: "c10s"
+  inherit_tier_x: true
+
+# Include manifests common to all RHEL and CentOS Stream versions
+include:
+  - common.yaml
+  - common-el10.yaml
+
+repos:
+  - c10s-baseos
+  - c10s-appstream
+
+automatic-version-prefix: "10.<date:%Y%m%d%H%M>"
+# Eventually we should try to build these images as part of the c10s composes.
+# In that case, the versioning should instead be exactly the same as the pungi
+# compose ID.
path: manifest-c10s.yaml
user: login: dustymabe
body: Is this still true? otherwise let's delete the comment
created_at: 2025-02-17T20:41:52Z
author_association: MEMBER
path: manifest-ocp-rhel-10.0.yaml
user: login: dustymabe
body: In the new world where we are building node images as layered image builds on top of base image builds is this manifest needed? Or rather we'd just need a Dockerfile to feed into a container build? 
created_at: 2025-02-17T20:43:56Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,115 @@
+# Manifest for OKD node based on CentOS Stream CoreOS 1à
+# Note: this manifest is temporary; in the future, OKD components will be layered instead.
+
+rojig:
path: manifest-okd-c10s.yaml
user: login: dustymabe
body: metadata
created_at: 2025-02-17T20:44:04Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,115 @@
+# Manifest for OKD node based on CentOS Stream CoreOS 1à
path: manifest-okd-c10s.yaml
user: login: dustymabe
body: ```suggestion
# Manifest for OKD node based on CentOS Stream CoreOS 10
```
created_at: 2025-02-17T20:44:24Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,115 @@
+# Manifest for OKD node based on CentOS Stream CoreOS 1à
+# Note: this manifest is temporary; in the future, OKD components will be layered instead.
+
+rojig:
+  license: MIT
+  name: scos
+  summary: OKD 4.19
+
+variables:
+  osversion: "c10s"
+
+# Include manifests common to all RHEL and CentOS Stream versions and manifest
+# common to RHEL 10 & C10S variants
+include:
+  - manifest-c10s.yaml
+  - packages-openshift.yaml
+
+# Additional repos we need for OKD components
+repos:
+  # For containernetworking-plugins for cri-o
+  - rhel-9.6-appstream
path: manifest-okd-c10s.yaml
user: login: dustymabe
body: temporary?
created_at: 2025-02-17T20:44:49Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,30 @@
+# Manifest for CentOS Stream CoreOS 10
+
+rojig:
+  license: MIT
+  name: scos
+  summary: CentOS Stream CoreOS 10
+
+variables:
+  osversion: "c10s"
+  inherit_tier_x: true
+
+# Include manifests common to all RHEL and CentOS Stream versions
+include:
+  - common.yaml
+  - common-el10.yaml
+
+repos:
+  - c10s-baseos
+  - c10s-appstream
+
+automatic-version-prefix: "10.<date:%Y%m%d%H%M>"
path: manifest-c10s.yaml
user: login: travier
body: There are no point releases for C10S so the version is just 10.
created_at: 2025-02-18T09:55:24Z
author_association: MEMBER
diff_hunk: @@ -10,14 +10,27 @@ ARG COSA
 ARG VARIANT
 RUN if [[ -z "$COSA" ]] ; then ci/get-ocp-repo.sh --ocp-layer packages-openshift.yaml; fi
 # on SCOS, we need to add the GPG keys of the various SIGs we need
-RUN if rpm -q centos-stream-release && ! rpm -q centos-release-cloud; then dnf install -y centos-release-{cloud,nfv,virt}-common; fi
-RUN mkdir -p /usr/share/distribution-gpg-keys/centos
-RUN ln -s /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial /usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Cloud
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-NFV
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Virtualization
-RUN if [[ -n "${VARIANT}" ]]; then MANIFEST="manifest-${VARIANT}.yaml"; EXTENSIONS="extensions-${VARIANT}.yaml"; else MANIFEST="manifest.yaml"; EXTENSIONS="extensions.yaml"; fi && rpm-ostree compose extensions --rootfs=/ --output-dir=/usr/share/rpm-ostree/extensions/ ./"${MANIFEST}" ./"${EXTENSIONS}"
+RUN if rpm -q centos-stream-release; then \
path: extensions/Dockerfile
user: login: travier
body: I asked myself that as well but I don't know which version of podman/buildah added support thus I don't know if we have that in Prow.
created_at: 2025-02-18T09:57:16Z
author_association: MEMBER
path: common-el9.yaml
user: login: travier
body: Hum, not sure I follow, the common manifest is from openshift/os and the shared one is from f-c-c so we can not merge those.
created_at: 2025-02-18T09:58:22Z
author_association: MEMBER
path: manifest-ocp-rhel-10.0.yaml
user: login: jbtrystram
body: I think we want a manifest and use https://github.com/coreos/rpm-ostree/pull/5274
created_at: 2025-02-18T13:05:45Z
author_association: CONTRIBUTOR
path: manifest-ocp-rhel-10.0.yaml
user: login: dustymabe
body: but that would happen inside a container build, correct?
created_at: 2025-02-18T19:54:22Z
author_association: MEMBER
diff_hunk: @@ -10,14 +10,27 @@ ARG COSA
 ARG VARIANT
 RUN if [[ -z "$COSA" ]] ; then ci/get-ocp-repo.sh --ocp-layer packages-openshift.yaml; fi
 # on SCOS, we need to add the GPG keys of the various SIGs we need
-RUN if rpm -q centos-stream-release && ! rpm -q centos-release-cloud; then dnf install -y centos-release-{cloud,nfv,virt}-common; fi
-RUN mkdir -p /usr/share/distribution-gpg-keys/centos
-RUN ln -s /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial /usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Cloud
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-NFV
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Virtualization
-RUN if [[ -n "${VARIANT}" ]]; then MANIFEST="manifest-${VARIANT}.yaml"; EXTENSIONS="extensions-${VARIANT}.yaml"; else MANIFEST="manifest.yaml"; EXTENSIONS="extensions.yaml"; fi && rpm-ostree compose extensions --rootfs=/ --output-dir=/usr/share/rpm-ostree/extensions/ ./"${MANIFEST}" ./"${EXTENSIONS}"
+RUN if rpm -q centos-stream-release; then \
path: extensions/Dockerfile
user: login: dustymabe
body: looks like https://github.com/containers/buildah/pull/5092 landed at the end of 2023
created_at: 2025-02-18T19:58:09Z
author_association: MEMBER
diff_hunk: @@ -10,14 +10,27 @@ ARG COSA
 ARG VARIANT
 RUN if [[ -z "$COSA" ]] ; then ci/get-ocp-repo.sh --ocp-layer packages-openshift.yaml; fi
 # on SCOS, we need to add the GPG keys of the various SIGs we need
-RUN if rpm -q centos-stream-release && ! rpm -q centos-release-cloud; then dnf install -y centos-release-{cloud,nfv,virt}-common; fi
-RUN mkdir -p /usr/share/distribution-gpg-keys/centos
-RUN ln -s /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial /usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Cloud
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-NFV
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Virtualization
-RUN if [[ -n "${VARIANT}" ]]; then MANIFEST="manifest-${VARIANT}.yaml"; EXTENSIONS="extensions-${VARIANT}.yaml"; else MANIFEST="manifest.yaml"; EXTENSIONS="extensions.yaml"; fi && rpm-ostree compose extensions --rootfs=/ --output-dir=/usr/share/rpm-ostree/extensions/ ./"${MANIFEST}" ./"${EXTENSIONS}"
+RUN if rpm -q centos-stream-release; then \
path: extensions/Dockerfile
user: login: dustymabe
body: The CI run on this PR would tell us if it had it or not wouldn't it? 
created_at: 2025-02-18T19:58:50Z
author_association: MEMBER
path: common-el9.yaml
user: login: dustymabe
body: No. `fedora-coreos-config/manifests/shared-el9.yaml` and `manifest-el9-shared.yaml` are different. See https://github.com/openshift/os/commit/2210eccd87141d98ec7cef88cb07319a1e198f51
created_at: 2025-02-18T20:00:15Z
author_association: MEMBER
path: extensions-c10s.yaml
user: login: travier
body: I fixed the name
created_at: 2025-02-19T10:22:12Z
author_association: MEMBER
diff_hunk: @@ -10,14 +10,27 @@ ARG COSA
 ARG VARIANT
 RUN if [[ -z "$COSA" ]] ; then ci/get-ocp-repo.sh --ocp-layer packages-openshift.yaml; fi
 # on SCOS, we need to add the GPG keys of the various SIGs we need
-RUN if rpm -q centos-stream-release && ! rpm -q centos-release-cloud; then dnf install -y centos-release-{cloud,nfv,virt}-common; fi
-RUN mkdir -p /usr/share/distribution-gpg-keys/centos
-RUN ln -s /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial /usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Cloud
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-NFV
-RUN ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Virtualization
-RUN if [[ -n "${VARIANT}" ]]; then MANIFEST="manifest-${VARIANT}.yaml"; EXTENSIONS="extensions-${VARIANT}.yaml"; else MANIFEST="manifest.yaml"; EXTENSIONS="extensions.yaml"; fi && rpm-ostree compose extensions --rootfs=/ --output-dir=/usr/share/rpm-ostree/extensions/ ./"${MANIFEST}" ./"${EXTENSIONS}"
+RUN if rpm -q centos-stream-release; then \
path: extensions/Dockerfile
user: login: travier
body: true
created_at: 2025-02-19T10:23:19Z
author_association: MEMBER
path: manifest-ocp-rhel-10.0.yaml
user: login: jbtrystram
body: at some point yes, but we need https://issues.redhat.com/browse/COS-3024 first 
created_at: 2025-02-19T10:25:03Z
author_association: CONTRIBUTOR
path: common-el9.yaml
user: login: travier
body: Ah, my bad this got introduced recently so I had missed those during the rebase
created_at: 2025-02-19T10:26:14Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,30 @@
+# Manifest for CentOS Stream CoreOS 10
+
+rojig:
+  license: MIT
+  name: scos
+  summary: CentOS Stream CoreOS 10
+
+variables:
+  osversion: "c10s"
+  inherit_tier_x: true
+
+# Include manifests common to all RHEL and CentOS Stream versions
+include:
+  - common.yaml
+  - common-el10.yaml
+
+repos:
+  - c10s-baseos
+  - c10s-appstream
+
+automatic-version-prefix: "10.<date:%Y%m%d%H%M>"
path: manifest-c10s.yaml
user: login: dustymabe
body: Compare this with what we are doing for c9s. @jlebon chose 9.0.<date> intentionally. I think it was to preserve semver. 
created_at: 2025-02-19T13:25:34Z
author_association: MEMBER
diff_hunk: @@ -1,15 +1,15 @@
 [c10s-baseos]
 name=CentOS Stream 10 - BaseOS
 baseurl=https://composes.stream.centos.org/stream-10/production/latest-CentOS-Stream/compose/BaseOS/$basearch/os
-gpgcheck=1
+gpgcheck=0
path: c10s.repo
user: login: dustymabe
body: since it's not immediately obvious to me I think this commit could use a comment in the body of the commit that explains why this is necessary. If the comment is short we could add it here as a comment in the repo file as well and also add instructions for when to change it back to `gpgcheck=1` 
created_at: 2025-02-19T13:38:26Z
author_association: MEMBER
path: extensions/Dockerfile
user: login: dustymabe
body: This commit could use a message in the body.

Why can't we setup extensions repos on SCOS 10? 
created_at: 2025-02-19T13:41:46Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,15 @@
+#!/bin/bash
path: overlay.d/05rhcos/usr/lib/dracut/modules.d/50ignition-sgdisk/module-setup.sh
user: login: dustymabe
body: This overlay is specific, doesn't apply to el9 (IIUC) and also something we hope to drop one day. I think it should go in its own overlay dir (`overlay.d/XXignition-gdisk` ?) with an updated [README](https://github.com/openshift/os/blob/master/overlay.d/README.md) entry for rationale and then conditionally included on el10. 
created_at: 2025-02-19T13:45:45Z
author_association: MEMBER
diff_hunk: @@ -127,12 +127,20 @@ if [ "${rhel_version}" = 96 ]; then
     curl --fail -L http://base-4-19-rhel94.ocp.svc.cluster.local >> "$repo_path"
 fi
 
-# If we're building the SCOS OKD variant, then strip away all the RHEL repos and just keep the plashet.
+# If we are building the SCOS OKD 9 variant, then strip away all the RHEL repos and just keep the plashet.
 # Temporary workaround until we have all packages for SCOS in CentOS Stream.
+# If we are building the SCOS OKD 10 variant, then we need some RHEL packages for now.
 if [ "$osname" = scos ]; then
     info "Neutering RHEL repos for SCOS"
     awk '/server-ose/,/^$/' "$repo_path" > "$repo_path.tmp"
     mv "$repo_path.tmp" "$repo_path"
 fi
 
+centos_version=$(source /usr/lib/os-release; echo ${VERSION_ID//./})
+# Get RHEL 9 repos for C10S builds for now
path: ci/get-ocp-repo.sh
user: login: dustymabe
body: when can we drop this hack? 
created_at: 2025-02-19T13:47:09Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,115 @@
+# Manifest for OKD node based on CentOS Stream CoreOS 1à
+# Note: this manifest is temporary; in the future, OKD components will be layered instead.
+
+rojig:
+  license: MIT
+  name: scos
+  summary: OKD 4.19
+
+variables:
+  osversion: "c10s"
+
+# Include manifests common to all RHEL and CentOS Stream versions and manifest
+# common to RHEL 10 & C10S variants
+include:
+  - manifest-c10s.yaml
+  - packages-openshift.yaml
+
+# Additional repos we need for OKD components
+repos:
+  # For containernetworking-plugins for cri-o
+  - rhel-9.6-appstream
path: manifest-okd-c10s.yaml
user: login: dustymabe
body: IOW - why do we need this in our c10s manifest? Maybe a comment here would help.
created_at: 2025-02-19T13:52:32Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,89 @@
+# RPMs as operating system extensions, distinct from the base ostree commit/image
+# https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md
+# and https://github.com/coreos/fedora-coreos-tracker/issues/401
+
+# repos:
+#   - sig-virtualization
+
+extensions:
+  # https://issues.redhat.com/browse/RFE-4177
+  # wasm:
+  #   architectures:
+  #     - x86_64
+  #     - aarch64
+  #   repos:
+  #     - appstream
+  #   packages:
+  #     - crun-wasm
+  # https://github.com/coreos/fedora-coreos-tracker/issues/1504
+  ipsec:
+    packages:
+      - libreswan
+      - NetworkManager-libreswan
+  # https://github.com/coreos/fedora-coreos-tracker/issues/326
+  usbguard:
+    packages:
+      - usbguard
+  kerberos:
+    packages:
+      - krb5-workstation
+      - libkadm5
+  # https://github.com/kmods-via-containers/kmods-via-containers/issues/3
+  # https://gitlab.cee.redhat.com/coreos/redhat-coreos/merge_requests/866
+  # These are currently overlaid onto the host so that they can be bind-mounted
+  # into build containers... in the future they should be a `development`
+  # extension: https://github.com/openshift/machine-config-operator/pull/2143.
+  kernel-devel:
+    packages:
+      - kernel-devel
+      - kernel-headers
+    match-base-evr: kernel
+  # These are already in the base, so they're not OS extensions, but they're
+  # useful to have in RPM form to install in kmod build containers.
+  kernel:
+    kind: development
+    packages:
+      - kernel
+      - kernel-core
+      - kernel-modules
+      - kernel-modules-extra
+    match-base-evr: kernel
+  # GRPA-2822
+  # https://github.com/openshift/machine-config-operator/pull/1330
+  # https://github.com/openshift/enhancements/blob/master/enhancements/support-for-realtime-kernel.md
+  kernel-rt:
+    architectures:
+      - x86_64
+    repos:
+      - c10s-nfv
+    packages:
+      - kernel-rt-core
+      - kernel-rt-kvm
+      - kernel-rt-modules
+      - kernel-rt-modules-extra
+      - kernel-rt-devel
+    match-base-evr: kernel
+  # https://github.com/openshift/machine-config-operator/pull/2456
+  # https://github.com/openshift/enhancements/blob/master/enhancements/sandboxed-containers/sandboxed-containers-tech-preview.md
+  # GRPA-3123
+  # - kata-containers
+  # sandboxed-containers:
+  #   architectures:
+  #     - x86_64
+  #   repos:
+  #     - sig-virtualization10
+  #   packages:
+  #     - kata-containers
path: extensions-okd-c10s.yaml
user: login: dustymabe
body: why are these commented out in c10s and not commented out in c9s? A comment here would help. 
created_at: 2025-02-19T13:53:44Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,89 @@
+# RPMs as operating system extensions, distinct from the base ostree commit/image
+# https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md
+# and https://github.com/coreos/fedora-coreos-tracker/issues/401
+
+# repos:
+#   - sig-virtualization
+
+extensions:
+  # https://issues.redhat.com/browse/RFE-4177
+  # wasm:
+  #   architectures:
+  #     - x86_64
+  #     - aarch64
+  #   repos:
+  #     - appstream
+  #   packages:
+  #     - crun-wasm
+  # https://github.com/coreos/fedora-coreos-tracker/issues/1504
+  ipsec:
+    packages:
+      - libreswan
path: extensions-okd-c10s.yaml
user: login: dustymabe
body: in extensions-ocp-rhel-10.0.yaml we are pinning on `libreswan-4.6-3.el9_0.3` - do we need to do the same thing here? 
created_at: 2025-02-19T13:55:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,89 @@
+# RPMs as operating system extensions, distinct from the base ostree commit/image
+# https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md
+# and https://github.com/coreos/fedora-coreos-tracker/issues/401
+
+# repos:
+#   - sig-virtualization
+
+extensions:
+  # https://issues.redhat.com/browse/RFE-4177
+  # wasm:
+  #   architectures:
+  #     - x86_64
+  #     - aarch64
+  #   repos:
+  #     - appstream
+  #   packages:
+  #     - crun-wasm
+  # https://github.com/coreos/fedora-coreos-tracker/issues/1504
+  ipsec:
+    packages:
+      - libreswan
+      - NetworkManager-libreswan
+  # https://github.com/coreos/fedora-coreos-tracker/issues/326
+  usbguard:
+    packages:
+      - usbguard
+  kerberos:
+    packages:
+      - krb5-workstation
+      - libkadm5
+  # https://github.com/kmods-via-containers/kmods-via-containers/issues/3
+  # https://gitlab.cee.redhat.com/coreos/redhat-coreos/merge_requests/866
+  # These are currently overlaid onto the host so that they can be bind-mounted
+  # into build containers... in the future they should be a `development`
+  # extension: https://github.com/openshift/machine-config-operator/pull/2143.
+  kernel-devel:
+    packages:
+      - kernel-devel
+      - kernel-headers
+    match-base-evr: kernel
+  # These are already in the base, so they're not OS extensions, but they're
+  # useful to have in RPM form to install in kmod build containers.
+  kernel:
+    kind: development
+    packages:
+      - kernel
+      - kernel-core
+      - kernel-modules
+      - kernel-modules-extra
+    match-base-evr: kernel
+  # GRPA-2822
+  # https://github.com/openshift/machine-config-operator/pull/1330
+  # https://github.com/openshift/enhancements/blob/master/enhancements/support-for-realtime-kernel.md
+  kernel-rt:
+    architectures:
+      - x86_64
+    repos:
+      - c10s-nfv
+    packages:
+      - kernel-rt-core
+      - kernel-rt-kvm
+      - kernel-rt-modules
+      - kernel-rt-modules-extra
+      - kernel-rt-devel
+    match-base-evr: kernel
+  # https://github.com/openshift/machine-config-operator/pull/2456
+  # https://github.com/openshift/enhancements/blob/master/enhancements/sandboxed-containers/sandboxed-containers-tech-preview.md
+  # GRPA-3123
+  # - kata-containers
+  # sandboxed-containers:
+  #   architectures:
+  #     - x86_64
+  #   repos:
+  #     - sig-virtualization10
+  #   packages:
+  #     - kata-containers
+  # https://issues.redhat.com/browse/COS-2402
+  kernel-64k:
+    architectures:
+      - aarch64
+    packages:
+      - kernel-64k-core
+      - kernel-64k-modules
+      - kernel-64k-modules-core
+      - kernel-64k-modules-extra
+  # https://issues.redhat.com/browse/COS-2940
+  sysstat:
+    packages:
+      - sysstat
path: extensions-okd-c10s.yaml
user: login: dustymabe
body: `extensions-ocp-rhel-10.0.yaml` has `two-node-ha` here. Do we want that for c10s? 
created_at: 2025-02-19T13:56:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,115 @@
+# Manifest for OKD node based on CentOS Stream CoreOS 10
+# Note: this manifest is temporary; in the future, OKD components will be layered instead.
+
+metadata:
+  license: MIT
+  name: scos
+  summary: OKD 4.19
+
+variables:
+  osversion: "c10s"
+
+# Include manifests common to all RHEL and CentOS Stream versions and manifest
+# common to RHEL 10 & C10S variants
+include:
+  - manifest-c10s.yaml
+  - packages-openshift.yaml
+
+# Additional repos we need for OKD components
+repos:
+  # For containernetworking-plugins for cri-o
+  - rhel-9.6-appstream
+  # CentOS Extras Common repo for SIG RPM GPG keys
+  # - c10s-extras-common
+  # CentOS NFV SIG repo for openvswitch
+  # - c10s-sig-nfv
+  # For openvswitch
+  - rhel-9.6-fast-datapath
+  # CentOS Cloud SIG repo for cri-o, cri-tools and conmon-rs
+  # - c10s-sig-cloud-okd
+  # Include RHCOS 9 repo for oc, hyperkube
+  - rhel-9.6-server-ose-4.19
+
+# We include hours/minutes to avoid version number reuse
+automatic-version-prefix: "419.10.<date:%Y%m%d%H%M>"
+# This ensures we're semver-compatible which OpenShift wants
+automatic-version-suffix: "-"
+# Keep this is sync with the version in postprocess
+mutate-os-release: "4.19"
+
+postprocess:
+  - |
+     #!/usr/bin/env bash
+     set -xeo pipefail
+
+     # Tweak /usr/lib/os-release
+     grep -v -e "OSTREE_VERSION" -e "OPENSHIFT_VERSION" /etc/os-release > /usr/lib/os-release.stream
+     (
+     . /etc/os-release
+     cat > /usr/lib/os-release <<EOF
+     NAME="${NAME}"
+     ID="scos"
+     ID_LIKE="rhel fedora"
+     VERSION="${OSTREE_VERSION}"
+     VERSION_ID="${OPENSHIFT_VERSION}"
+     VARIANT="${VARIANT}"
+     VARIANT_ID=${VARIANT_ID}
+     PLATFORM_ID="${PLATFORM_ID}"
+     PRETTY_NAME="${NAME} ${OSTREE_VERSION}"
+     ANSI_COLOR="${ANSI_COLOR}"
+     CPE_NAME="${CPE_NAME}::coreos"
+     HOME_URL="${HOME_URL}"
+     DOCUMENTATION_URL="https://docs.okd.io/latest/welcome/index.html"
+     BUG_REPORT_URL="https://access.redhat.com/labs/rhir/"
+     REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
+     REDHAT_BUGZILLA_PRODUCT_VERSION="${OPENSHIFT_VERSION}"
+     REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
+     REDHAT_SUPPORT_PRODUCT_VERSION="${OPENSHIFT_VERSION}"
+     OPENSHIFT_VERSION="${OPENSHIFT_VERSION}"
+     OSTREE_VERSION="${OSTREE_VERSION}"
+     EOF
+     )
+     rm -f /etc/os-release
+     ln -s ../usr/lib/os-release /etc/os-release
+
+     # Tweak /etc/system-release, /etc/system-release-cpe & /etc/redhat-release
+     (
+     . /etc/os-release
+     cat > /usr/lib/system-release-cpe <<EOF
+     ${CPE_NAME}
+     EOF
+     cat > /usr/lib/system-release <<EOF
+     ${NAME} release ${VERSION_ID}
+     EOF
+     rm -f /etc/system-release-cpe /etc/system-release /etc/redhat-release
+     ln -s /usr/lib/system-release-cpe /etc/system-release-cpe
+     ln -s /usr/lib/system-release /etc/system-release
+     ln -s /usr/lib/system-release /etc/redhat-release
+     )
+
+     # Tweak /usr/lib/issue
+     cat > /usr/lib/issue <<EOF
+     \S \S{VERSION_ID}
+     EOF
+     rm -f /etc/issue /etc/issue.net
+     ln -s /usr/lib/issue /etc/issue
+     ln -s /usr/lib/issue /etc/issue.net
+
+packages:
+ # RPM GPG keys for CentOS SIG repos
+ # - centos-release-cloud-common
+ # - centos-release-nfv-common
+ # - centos-release-virt-common
path: manifest-okd-c10s.yaml
user: login: dustymabe
body: Add comment on why these are commented out? 
created_at: 2025-02-19T14:00:27Z
author_association: MEMBER
diff_hunk: @@ -1,15 +1,15 @@
 [c10s-baseos]
 name=CentOS Stream 10 - BaseOS
 baseurl=https://composes.stream.centos.org/stream-10/production/latest-CentOS-Stream/compose/BaseOS/$basearch/os
-gpgcheck=1
+gpgcheck=0
path: c10s.repo
user: login: travier
body: It's a leftover from when the c10s repos were not yet signed. Will remove
created_at: 2025-02-19T14:33:28Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,115 @@
+# Manifest for OKD node based on CentOS Stream CoreOS 10
+# Note: this manifest is temporary; in the future, OKD components will be layered instead.
+
+metadata:
+  license: MIT
+  name: scos
+  summary: OKD 4.19
+
+variables:
+  osversion: "c10s"
+
+# Include manifests common to all RHEL and CentOS Stream versions and manifest
+# common to RHEL 10 & C10S variants
+include:
+  - manifest-c10s.yaml
+  - packages-openshift.yaml
+
+# Additional repos we need for OKD components
+repos:
+  # For containernetworking-plugins for cri-o
+  - rhel-9.6-appstream
+  # CentOS Extras Common repo for SIG RPM GPG keys
+  # - c10s-extras-common
+  # CentOS NFV SIG repo for openvswitch
+  # - c10s-sig-nfv
+  # For openvswitch
+  - rhel-9.6-fast-datapath
+  # CentOS Cloud SIG repo for cri-o, cri-tools and conmon-rs
+  # - c10s-sig-cloud-okd
+  # Include RHCOS 9 repo for oc, hyperkube
+  - rhel-9.6-server-ose-4.19
+
+# We include hours/minutes to avoid version number reuse
+automatic-version-prefix: "419.10.<date:%Y%m%d%H%M>"
+# This ensures we're semver-compatible which OpenShift wants
+automatic-version-suffix: "-"
+# Keep this is sync with the version in postprocess
+mutate-os-release: "4.19"
+
+postprocess:
+  - |
+     #!/usr/bin/env bash
+     set -xeo pipefail
+
+     # Tweak /usr/lib/os-release
+     grep -v -e "OSTREE_VERSION" -e "OPENSHIFT_VERSION" /etc/os-release > /usr/lib/os-release.stream
+     (
+     . /etc/os-release
+     cat > /usr/lib/os-release <<EOF
+     NAME="${NAME}"
+     ID="scos"
+     ID_LIKE="rhel fedora"
+     VERSION="${OSTREE_VERSION}"
+     VERSION_ID="${OPENSHIFT_VERSION}"
+     VARIANT="${VARIANT}"
+     VARIANT_ID=${VARIANT_ID}
+     PLATFORM_ID="${PLATFORM_ID}"
+     PRETTY_NAME="${NAME} ${OSTREE_VERSION}"
+     ANSI_COLOR="${ANSI_COLOR}"
+     CPE_NAME="${CPE_NAME}::coreos"
+     HOME_URL="${HOME_URL}"
+     DOCUMENTATION_URL="https://docs.okd.io/latest/welcome/index.html"
+     BUG_REPORT_URL="https://access.redhat.com/labs/rhir/"
+     REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
+     REDHAT_BUGZILLA_PRODUCT_VERSION="${OPENSHIFT_VERSION}"
+     REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
+     REDHAT_SUPPORT_PRODUCT_VERSION="${OPENSHIFT_VERSION}"
+     OPENSHIFT_VERSION="${OPENSHIFT_VERSION}"
+     OSTREE_VERSION="${OSTREE_VERSION}"
+     EOF
+     )
+     rm -f /etc/os-release
+     ln -s ../usr/lib/os-release /etc/os-release
+
+     # Tweak /etc/system-release, /etc/system-release-cpe & /etc/redhat-release
+     (
+     . /etc/os-release
+     cat > /usr/lib/system-release-cpe <<EOF
+     ${CPE_NAME}
+     EOF
+     cat > /usr/lib/system-release <<EOF
+     ${NAME} release ${VERSION_ID}
+     EOF
+     rm -f /etc/system-release-cpe /etc/system-release /etc/redhat-release
+     ln -s /usr/lib/system-release-cpe /etc/system-release-cpe
+     ln -s /usr/lib/system-release /etc/system-release
+     ln -s /usr/lib/system-release /etc/redhat-release
+     )
+
+     # Tweak /usr/lib/issue
+     cat > /usr/lib/issue <<EOF
+     \S \S{VERSION_ID}
+     EOF
+     rm -f /etc/issue /etc/issue.net
+     ln -s /usr/lib/issue /etc/issue
+     ln -s /usr/lib/issue /etc/issue.net
+
+packages:
+ # RPM GPG keys for CentOS SIG repos
+ # - centos-release-cloud-common
+ # - centos-release-nfv-common
+ # - centos-release-virt-common
path: manifest-okd-c10s.yaml
user: login: travier
body: This was all leftover from when C10S was not released. Updating.
created_at: 2025-02-19T15:38:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,15 @@
+#!/bin/bash
path: overlay.d/05rhcos/usr/lib/dracut/modules.d/50ignition-sgdisk/module-setup.sh
user: login: travier
body: Should go away with https://github.com/coreos/fedora-coreos-config/pull/3368
created_at: 2025-02-19T16:39:12Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,115 @@
+# Manifest for OKD node based on CentOS Stream CoreOS 1à
+# Note: this manifest is temporary; in the future, OKD components will be layered instead.
+
+rojig:
+  license: MIT
+  name: scos
+  summary: OKD 4.19
+
+variables:
+  osversion: "c10s"
+
+# Include manifests common to all RHEL and CentOS Stream versions and manifest
+# common to RHEL 10 & C10S variants
+include:
+  - manifest-c10s.yaml
+  - packages-openshift.yaml
+
+# Additional repos we need for OKD components
+repos:
+  # For containernetworking-plugins for cri-o
+  - rhel-9.6-appstream
path: manifest-okd-c10s.yaml
user: login: travier
body: yes, it's temporary until we have this published in the RHAOS repo or cri-o stops depending on it in the RHEL 10 if that's possible.
created_at: 2025-02-20T13:55:36Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,30 @@
+# Manifest for CentOS Stream CoreOS 10
+
+rojig:
+  license: MIT
+  name: scos
+  summary: CentOS Stream CoreOS 10
+
+variables:
+  osversion: "c10s"
+  inherit_tier_x: true
+
+# Include manifests common to all RHEL and CentOS Stream versions
+include:
+  - common.yaml
+  - common-el10.yaml
+
+repos:
+  - c10s-baseos
+  - c10s-appstream
+
+automatic-version-prefix: "10.<date:%Y%m%d%H%M>"
path: manifest-c10s.yaml
user: login: travier
body: Hum, indeed., will do
created_at: 2025-02-20T13:57:28Z
author_association: MEMBER
path: extensions/Dockerfile
user: login: travier
body: I think it was because all the repos for them were missing. Now we should be able to build some of them.
created_at: 2025-02-20T13:59:16Z
author_association: MEMBER
diff_hunk: @@ -127,12 +127,20 @@ if [ "${rhel_version}" = 96 ]; then
     curl --fail -L http://base-4-19-rhel94.ocp.svc.cluster.local >> "$repo_path"
 fi
 
-# If we're building the SCOS OKD variant, then strip away all the RHEL repos and just keep the plashet.
+# If we are building the SCOS OKD 9 variant, then strip away all the RHEL repos and just keep the plashet.
 # Temporary workaround until we have all packages for SCOS in CentOS Stream.
+# If we are building the SCOS OKD 10 variant, then we need some RHEL packages for now.
 if [ "$osname" = scos ]; then
     info "Neutering RHEL repos for SCOS"
     awk '/server-ose/,/^$/' "$repo_path" > "$repo_path.tmp"
     mv "$repo_path.tmp" "$repo_path"
 fi
 
+centos_version=$(source /usr/lib/os-release; echo ${VERSION_ID//./})
+# Get RHEL 9 repos for C10S builds for now
path: ci/get-ocp-repo.sh
user: login: travier
body: Once we have RHEL 10 / RHAOS 10 repos will all the needed packages.
created_at: 2025-02-20T13:59:52Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,89 @@
+# RPMs as operating system extensions, distinct from the base ostree commit/image
+# https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md
+# and https://github.com/coreos/fedora-coreos-tracker/issues/401
+
+# repos:
+#   - sig-virtualization
+
+extensions:
+  # https://issues.redhat.com/browse/RFE-4177
+  # wasm:
+  #   architectures:
+  #     - x86_64
+  #     - aarch64
+  #   repos:
+  #     - appstream
+  #   packages:
+  #     - crun-wasm
+  # https://github.com/coreos/fedora-coreos-tracker/issues/1504
+  ipsec:
+    packages:
+      - libreswan
path: extensions-okd-c10s.yaml
user: login: travier
body: Good question, we will have to ask the ipsec/telco folks.
created_at: 2025-02-20T14:00:29Z
author_association: MEMBER
diff_hunk: @@ -115,24 +120,28 @@ fi
 mkdir -p "$output_dir"
 repo_path="$output_dir/ocp.repo"
 
-set -x
+info "Fetching ${ocp_version} ${rhel_version} repos"
+# set -x
 curl --fail -L "http://base-${ocp_version}-rhel${rhel_version}.ocp.svc.cluster.local" -o "$repo_path"
-set +x
+# set +x
 
 if [ "${rhel_version}" = 96 ]; then
     # XXX: also currently also add 9.4 repos for crun-wasm when building extensions
     # https://github.com/openshift/os/issues/1680
     # https://github.com/openshift/os/pull/1682
     # https://issues.redhat.com/browse/COS-3075
+    info "Fetching 4.19 94 repos"
     curl --fail -L http://base-4-19-rhel94.ocp.svc.cluster.local >> "$repo_path"
 fi
 
-# If we're building the SCOS OKD variant, then strip away all the RHEL repos and just keep the plashet.
+# If we are building the SCOS OKD 9 variant, then strip away all the RHEL repos and just keep the plashet.
 # Temporary workaround until we have all packages for SCOS in CentOS Stream.
-if [ "$osname" = scos ]; then
+# If we are building the SCOS OKD 10 variant, then we need some RHEL packages for now.
+if [ "$osname" = scos ] && [ "${centos_version}" != "10" ]; then
path: ci/get-ocp-repo.sh
user: login: jbtrystram
body: for the okd-c10s variant `version` here evaluates to `419.96....` because in this specific case it's pulling the version from the default variant (see line 96 above). I pushed a fix. 

This script is hard to maintain :D 
created_at: 2025-02-21T09:46:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,98 @@
+# RPMs as operating system extensions, distinct from the base ostree commit/image
+# https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md
+# and https://github.com/coreos/fedora-coreos-tracker/issues/401
+
+# repos:
+#   - sig-virtualization
+
+extensions:
+  # https://issues.redhat.com/browse/RFE-4177
+  # wasm:
+  #   architectures:
+  #     - x86_64
+  #     - aarch64
+  #   repos:
+  #     - appstream
+  #   packages:
+  #     - crun-wasm
+  # https://github.com/coreos/fedora-coreos-tracker/issues/1504
+  ipsec:
+    packages:
+      - libreswan
+      - NetworkManager-libreswan
+  # https://github.com/coreos/fedora-coreos-tracker/issues/326
+  usbguard:
+    packages:
+      - usbguard
+  kerberos:
+    packages:
+      - krb5-workstation
+      - libkadm5
+  # https://github.com/kmods-via-containers/kmods-via-containers/issues/3
+  # https://gitlab.cee.redhat.com/coreos/redhat-coreos/merge_requests/866
+  # These are currently overlaid onto the host so that they can be bind-mounted
+  # into build containers... in the future they should be a `development`
+  # extension: https://github.com/openshift/machine-config-operator/pull/2143.
+  kernel-devel:
+    packages:
+      - kernel-devel
+      - kernel-headers
+    match-base-evr: kernel
+  # These are already in the base, so they're not OS extensions, but they're
+  # useful to have in RPM form to install in kmod build containers.
+  kernel:
+    kind: development
+    packages:
+      - kernel
+      - kernel-core
+      - kernel-modules
+      - kernel-modules-extra
+    match-base-evr: kernel
+  # GRPA-2822
+  # https://github.com/openshift/machine-config-operator/pull/1330
+  # https://github.com/openshift/enhancements/blob/master/enhancements/support-for-realtime-kernel.md
+  kernel-rt:
+    architectures:
+      - x86_64
+    repos:
+      - c10s-nfv
+    packages:
+      - kernel-rt-core
+      - kernel-rt-kvm
+      - kernel-rt-modules
+      - kernel-rt-modules-extra
+      - kernel-rt-devel
+    match-base-evr: kernel
+  # https://github.com/openshift/machine-config-operator/pull/2456
+  # https://github.com/openshift/enhancements/blob/master/enhancements/sandboxed-containers/sandboxed-containers-tech-preview.md
+  # GRPA-3123
+  # - kata-containers
+  # sandboxed-containers:
+  #   architectures:
+  #     - x86_64
+  #   repos:
+  #     - sig-virtualization10
+  #   packages:
+  #     - kata-containers
+  # https://issues.redhat.com/browse/COS-2402
+  kernel-64k:
+    architectures:
+      - aarch64
+    packages:
+      - kernel-64k-core
+      - kernel-64k-modules
+      - kernel-64k-modules-core
+      - kernel-64k-modules-extra
+  # https://issues.redhat.com/browse/COS-2940
+  sysstat:
+    packages:
+      - sysstat
+  # https://issues.redhat.com/browse/OCPEDGE-1493
+  two-node-ha:
+    packages:
+      - pacemaker
+      - pcs
+      - fence-agents-all
+    repos:
+      - rhel-10.0-appstream
path: extensions-okd-c10s.yaml
user: login: jbtrystram
body: we need the rhel10 repos to be mirrored in openshift for this to go through : https://github.com/openshift/release/pull/61812
created_at: 2025-02-21T11:30:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,10 +15,10 @@ supported:
 
 - `rhel-9.6`: RHEL 9.6-based CoreOS; without OpenShift components.
 - `ocp-rhel-9.6`: RHEL 9.6-based CoreOS; including OpenShift components.
-- `c9s`: CentOS Stream-based CoreOS, without OKD components.
-- `okd-c9s`: CentOS Stream-based CoreOS, including OpenShift components. This
-  currently includes some packages from RHEL because not all packages required
-  by OpenShift are provided in CentOS Stream.
+- `c9s`/`c10s`: CentOS Stream-based CoreOS, without OKD components.
+- `okd-c9s`/`okd-c10s`: CentOS Stream-based CoreOS, including OpenShift
path: README.md
user: login: jlebon
body: No need to add an `okd-c10s` variant. In fact, let's delete the `okd-c9s` variant.
created_at: 2025-03-21T15:35:23Z
author_association: MEMBER
diff_hunk: @@ -15,10 +15,10 @@ supported:
 
 - `rhel-9.6`: RHEL 9.6-based CoreOS; without OpenShift components.
 - `ocp-rhel-9.6`: RHEL 9.6-based CoreOS; including OpenShift components.
-- `c9s`: CentOS Stream-based CoreOS, without OKD components.
-- `okd-c9s`: CentOS Stream-based CoreOS, including OpenShift components. This
-  currently includes some packages from RHEL because not all packages required
-  by OpenShift are provided in CentOS Stream.
+- `c9s`/`c10s`: CentOS Stream-based CoreOS, without OKD components.
+- `okd-c9s`/`okd-c10s`: CentOS Stream-based CoreOS, including OpenShift
path: README.md
user: login: jbtrystram
body: let's delete the okd-c9s variant in another PR, this one is already dragging enough as is

created_at: 2025-03-21T16:38:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,10 +15,10 @@ supported:
 
 - `rhel-9.6`: RHEL 9.6-based CoreOS; without OpenShift components.
 - `ocp-rhel-9.6`: RHEL 9.6-based CoreOS; including OpenShift components.
-- `c9s`: CentOS Stream-based CoreOS, without OKD components.
-- `okd-c9s`: CentOS Stream-based CoreOS, including OpenShift components. This
-  currently includes some packages from RHEL because not all packages required
-  by OpenShift are provided in CentOS Stream.
+- `c9s`/`c10s`: CentOS Stream-based CoreOS, without OKD components.
+- `okd-c9s`/`okd-c10s`: CentOS Stream-based CoreOS, including OpenShift
path: README.md
user: login: jlebon
body: https://github.com/openshift/os/pull/1780
created_at: 2025-03-28T16:06:35Z
author_association: MEMBER
diff_hunk: @@ -15,10 +15,10 @@ supported:
 
 - `rhel-9.6`: RHEL 9.6-based CoreOS; without OpenShift components.
 - `ocp-rhel-9.6`: RHEL 9.6-based CoreOS; including OpenShift components.
-- `c9s`: CentOS Stream-based CoreOS, without OKD components.
-- `okd-c9s`: CentOS Stream-based CoreOS, including OpenShift components. This
-  currently includes some packages from RHEL because not all packages required
-  by OpenShift are provided in CentOS Stream.
+- `c9s`/`c10s`: CentOS Stream-based CoreOS, without OKD components.
+- `okd-c9s`/`okd-c10s`: CentOS Stream-based CoreOS, including OpenShift
path: README.md
user: login: jlebon
body: But still, let's not add `okd-c10s` in this PR.
created_at: 2025-03-28T16:07:05Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,89 @@
+# RPMs as operating system extensions, distinct from the base ostree commit/image
+# https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md
+# and https://github.com/coreos/fedora-coreos-tracker/issues/401
+
+# repos:
+#   - sig-virtualization
+
+extensions:
+  # https://issues.redhat.com/browse/RFE-4177
+  # wasm:
+  #   architectures:
+  #     - x86_64
+  #     - aarch64
+  #   repos:
+  #     - appstream
+  #   packages:
+  #     - crun-wasm
+  # https://github.com/coreos/fedora-coreos-tracker/issues/1504
+  ipsec:
+    packages:
+      - libreswan
path: extensions-okd-c10s.yaml
user: login: jlebon
body: https://github.com/openshift/os/pull/1771 will unpin it soon. Probably fine to not worry about this here for now.
created_at: 2025-03-28T16:11:12Z
author_association: MEMBER
path: image-okd-c10s.yaml
user: login: jlebon
body: Nuke
created_at: 2025-03-28T16:12:11Z
author_association: MEMBER
path: manifest-okd-c10s.yaml
user: login: jlebon
body: Nuke
created_at: 2025-03-28T16:12:40Z
author_association: MEMBER
path: oscontainer.yaml
user: login: jlebon
body: Nuke this file. This is no longer needed since https://github.com/coreos/coreos-assembler/pull/3765.
created_at: 2025-03-28T16:14:56Z
author_association: MEMBER
diff_hunk: @@ -9,7 +9,14 @@ ADD . .
 ARG OPENSHIFT_CI=0
 ARG VARIANT=""
 RUN if [ "${OPENSHIFT_CI}" != 0 ]; then ci/get-ocp-repo.sh --ocp-layer packages-openshift.yaml; fi
-RUN --mount=type=secret,id=yumrepos,target=/os/secret.repo if [[ -n "${VARIANT}" ]]; then MANIFEST="manifest-${VARIANT}.yaml"; EXTENSIONS="extensions-${VARIANT}.yaml"; else MANIFEST="manifest.yaml"; EXTENSIONS="extensions.yaml"; fi && rpm-ostree compose extensions --rootfs=/ --output-dir=/usr/share/rpm-ostree/extensions/ ./"${MANIFEST}" ./"${EXTENSIONS}"
+RUN --mount=type=secret,id=yumrepos,target=/os/secret.repo if [[ -n "${VARIANT}" ]]; then \
path: extensions/Dockerfile
user: login: jlebon
body: Or let's just switch it to [a heredoc](https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/)? Not sure if the buildah in CI is new enough for it though, but could try.
created_at: 2025-03-28T16:17:07Z
author_association: MEMBER
diff_hunk: @@ -312,10 +312,16 @@ main() {
             kola_test_metal
             ;;
         "scos-10-build-test-qemu")
-            exit 0
+            setup_user
+            cosa_init "okd-c10s"
+            cosa_build
+            kola_test_qemu
             ;;
         "scos-10-build-test-metal")
-            exit 0
+            setup_user
+            cosa_init "okd-c10s"
+            cosa_build
+            kola_test_metal
path: ci/prow-entrypoint.sh
user: login: jlebon
body: Let's test the `c10s` variant instead.
created_at: 2025-03-28T16:17:37Z
author_association: MEMBER
diff_hunk: @@ -169,21 +172,24 @@ fi
 mkdir -p "$output_dir"
 repo_path="$output_dir/ocp.repo"
 
-set -x
+info "Fetching ${ocp_version} ${rhel_version} repos"
+# set -x
 curl --fail -L "http://base-${ocp_version}-rhel${rhel_version}.ocp.svc.cluster.local" -o "$repo_path"
-set +x
+# set +x
 
 if [ "${rhel_version}" = 96 ]; then
     # XXX: also currently also add 9.4 repos for crun-wasm when building extensions
     # https://github.com/openshift/os/issues/1680
     # https://github.com/openshift/os/pull/1682
     # https://issues.redhat.com/browse/COS-3075
+    info "Fetching 4.19 94 repos"
     curl --fail -L http://base-4-19-rhel94.ocp.svc.cluster.local >> "$repo_path"
 fi
 
-# If we're building the SCOS OKD variant, then strip away all the RHEL repos and just keep the plashet.
+# If we are building the SCOS OKD 9 variant, then strip away all the RHEL repos and just keep the plashet.
 # Temporary workaround until we have all packages for SCOS in CentOS Stream.
-if [ "$osname" = scos ]; then
+# If we are building the SCOS OKD 10 variant, then we need some RHEL packages for now.
path: ci/get-ocp-repo.sh
user: login: jlebon
body: The commit title says "Keep RHEL 9 repos", but we're keeping RHEL 10 repos, no?
created_at: 2025-03-28T16:22:23Z
author_association: MEMBER
path: overlay.d/22rhcos-mpathd/etc/multipath.conf
user: login: jlebon
body: Let's just drop this commit. We shouldn't need it soon now that https://github.com/dracut-ng/dracut-ng/pull/1259 has merged. We can leave the multipath tests denylisted until it's backported.
created_at: 2025-03-28T16:24:11Z
author_association: MEMBER
diff_hunk: @@ -313,13 +313,13 @@ main() {
             ;;
         "scos-10-build-test-qemu")
             setup_user
-            cosa_init "okd-c10s"
+            cosa_init "c10s"
path: ci/prow-entrypoint.sh
user: login: jlebon
body: Nice. Had an earlier comment about that in the patch series. Should squash this commit into that earlier one.
created_at: 2025-03-28T16:24:35Z
author_association: MEMBER
diff_hunk: @@ -131,7 +133,7 @@ else
     # first, make sure we're looking at the right manifest
     manifest="$cosa_workdir/src/config/manifest.yaml"
     if [ -f "$cosa_workdir/src/config.json" ]; then
-        variant="$(jq --raw-output '."coreos-assembler.config-variant"' "$cosa_workdir/src/config.json")"
+        variant="$(jq --raw-output '."coreos-assembler.config-variant"' < "$cosa_workdir"/src/config.json)"
path: ci/get-ocp-repo.sh
user: login: jlebon
body: Hmm, why?
created_at: 2025-03-28T16:28:35Z
author_association: MEMBER
diff_hunk: @@ -127,12 +127,20 @@ if [ "${rhel_version}" = 96 ]; then
     curl --fail -L http://base-4-19-rhel94.ocp.svc.cluster.local >> "$repo_path"
 fi
 
-# If we're building the SCOS OKD variant, then strip away all the RHEL repos and just keep the plashet.
+# If we are building the SCOS OKD 9 variant, then strip away all the RHEL repos and just keep the plashet.
 # Temporary workaround until we have all packages for SCOS in CentOS Stream.
+# If we are building the SCOS OKD 10 variant, then we need some RHEL packages for now.
 if [ "$osname" = scos ]; then
     info "Neutering RHEL repos for SCOS"
     awk '/server-ose/,/^$/' "$repo_path" > "$repo_path.tmp"
     mv "$repo_path.tmp" "$repo_path"
 fi
 
+centos_version=$(source /usr/lib/os-release; echo ${VERSION_ID//./})
+# Get RHEL 9 repos for C10S builds for now
path: ci/get-ocp-repo.sh
user: login: jlebon
body: Let's mention the packages we still need here, or ideally file an issue with the details and link to that here?
created_at: 2025-03-28T16:29:55Z
author_association: MEMBER
diff_hunk: @@ -72,7 +72,7 @@ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
             for x in ${kernel_pkgs}; do
                 args+=(--install "${x}")
             done
-            runv rpm-ostree override remove kernel{,-core,-modules,-modules-extra,-modules-core} "${args[@]}"
+            runv rpm-ostree override remove kernel{,-core} "${args[@]}"
path: tests/kola/rpm-ostree/replace-rt-kernel/test.sh
user: login: jlebon
body: Hmm, we shouldn't mix and match between kernel and kernel-rt packages though.

Anyway, this is moot since it looks like iptables was fixed to do this properly recently: https://gitlab.com/redhat/centos-stream/rpms/iptables/-/commit/46e0fa580ca4dfa77d5a652b75d2ae286abadf77

So we should be able to drop this commit.

(But nice commit message regardless BTW!)
created_at: 2025-03-28T16:41:56Z
author_association: MEMBER
path: overlay.d/22rhcos-mpathd/etc/multipath.conf
user: login: jbtrystram
body: I'm afraid it's going to be a while. I have a PR in cosa that was waiting on a dracut release to be packaged in fedora for 6 months... downstream packaging seems to be faster however.

hmm although I suppose we're not shipping those builds any time soon so it's probably ok to keep the tests denylisted
created_at: 2025-04-02T11:29:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,3 +45,64 @@
   tracker: https://github.com/openshift/os/issues/1751
   osversion:
     - c10s
+
+- pattern: ext.config.shared.kdump.crash
+  tracker: TODO
+  osversion:
+    - c10s
path: kola-denylist.yaml
user: login: jbtrystram
body: This can be removed
created_at: 2025-04-04T16:39:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,89 @@
+# RPMs as operating system extensions, distinct from the base ostree commit/image
+# https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md
+# and https://github.com/coreos/fedora-coreos-tracker/issues/401
+
+# repos:
+#   - sig-virtualization
+
+extensions:
+  # https://issues.redhat.com/browse/RFE-4177
+  # wasm:
+  #   architectures:
+  #     - x86_64
+  #     - aarch64
+  #   repos:
+  #     - appstream
+  #   packages:
+  #     - crun-wasm
+  # https://github.com/coreos/fedora-coreos-tracker/issues/1504
+  ipsec:
+    packages:
+      - libreswan
+      - NetworkManager-libreswan
+  # https://github.com/coreos/fedora-coreos-tracker/issues/326
+  usbguard:
+    packages:
+      - usbguard
+  kerberos:
+    packages:
+      - krb5-workstation
+      - libkadm5
+  # https://github.com/kmods-via-containers/kmods-via-containers/issues/3
+  # https://gitlab.cee.redhat.com/coreos/redhat-coreos/merge_requests/866
+  # These are currently overlaid onto the host so that they can be bind-mounted
+  # into build containers... in the future they should be a `development`
+  # extension: https://github.com/openshift/machine-config-operator/pull/2143.
+  kernel-devel:
+    packages:
+      - kernel-devel
+      - kernel-headers
+    match-base-evr: kernel
+  # These are already in the base, so they're not OS extensions, but they're
+  # useful to have in RPM form to install in kmod build containers.
+  kernel:
+    kind: development
+    packages:
+      - kernel
+      - kernel-core
+      - kernel-modules
+      - kernel-modules-extra
+    match-base-evr: kernel
+  # GRPA-2822
+  # https://github.com/openshift/machine-config-operator/pull/1330
+  # https://github.com/openshift/enhancements/blob/master/enhancements/support-for-realtime-kernel.md
+  kernel-rt:
+    architectures:
+      - x86_64
+    repos:
+      - c10s-nfv
+    packages:
+      - kernel-rt-core
+      - kernel-rt-kvm
+      - kernel-rt-modules
+      - kernel-rt-modules-extra
+      - kernel-rt-devel
+    match-base-evr: kernel
+  # https://github.com/openshift/machine-config-operator/pull/2456
+  # https://github.com/openshift/enhancements/blob/master/enhancements/sandboxed-containers/sandboxed-containers-tech-preview.md
+  # GRPA-3123
+  # - kata-containers
+  # sandboxed-containers:
+  #   architectures:
+  #     - x86_64
+  #   repos:
+  #     - sig-virtualization10
+  #   packages:
+  #     - kata-containers
path: extensions-okd-c10s.yaml
user: login: jbtrystram
body: @dustymabe  asked about why these are commented in an erlier review, let's add a comment saying it s because the kata container package is not available in c10s yet
created_at: 2025-04-04T17:46:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,89 @@
+# RPMs as operating system extensions, distinct from the base ostree commit/image
+# https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md
+# and https://github.com/coreos/fedora-coreos-tracker/issues/401
+
+# repos:
+#   - sig-virtualization
+
+extensions:
+  # https://issues.redhat.com/browse/RFE-4177
+  # wasm:
+  #   architectures:
+  #     - x86_64
+  #     - aarch64
+  #   repos:
+  #     - appstream
+  #   packages:
+  #     - crun-wasm
+  # https://github.com/coreos/fedora-coreos-tracker/issues/1504
+  ipsec:
+    packages:
+      - libreswan
+      - NetworkManager-libreswan
+  # https://github.com/coreos/fedora-coreos-tracker/issues/326
+  usbguard:
+    packages:
+      - usbguard
+  kerberos:
+    packages:
+      - krb5-workstation
+      - libkadm5
+  # https://github.com/kmods-via-containers/kmods-via-containers/issues/3
+  # https://gitlab.cee.redhat.com/coreos/redhat-coreos/merge_requests/866
+  # These are currently overlaid onto the host so that they can be bind-mounted
+  # into build containers... in the future they should be a `development`
+  # extension: https://github.com/openshift/machine-config-operator/pull/2143.
+  kernel-devel:
+    packages:
+      - kernel-devel
+      - kernel-headers
+    match-base-evr: kernel
+  # These are already in the base, so they're not OS extensions, but they're
+  # useful to have in RPM form to install in kmod build containers.
+  kernel:
+    kind: development
+    packages:
+      - kernel
+      - kernel-core
+      - kernel-modules
+      - kernel-modules-extra
+    match-base-evr: kernel
+  # GRPA-2822
+  # https://github.com/openshift/machine-config-operator/pull/1330
+  # https://github.com/openshift/enhancements/blob/master/enhancements/support-for-realtime-kernel.md
+  kernel-rt:
+    architectures:
+      - x86_64
+    repos:
+      - c10s-nfv
+    packages:
+      - kernel-rt-core
+      - kernel-rt-kvm
+      - kernel-rt-modules
+      - kernel-rt-modules-extra
+      - kernel-rt-devel
+    match-base-evr: kernel
+  # https://github.com/openshift/machine-config-operator/pull/2456
+  # https://github.com/openshift/enhancements/blob/master/enhancements/sandboxed-containers/sandboxed-containers-tech-preview.md
+  # GRPA-3123
+  # - kata-containers
+  # sandboxed-containers:
+  #   architectures:
+  #     - x86_64
+  #   repos:
+  #     - sig-virtualization10
+  #   packages:
+  #     - kata-containers
+  # https://issues.redhat.com/browse/COS-2402
+  kernel-64k:
+    architectures:
+      - aarch64
+    packages:
+      - kernel-64k-core
+      - kernel-64k-modules
+      - kernel-64k-modules-core
+      - kernel-64k-modules-extra
+  # https://issues.redhat.com/browse/COS-2940
+  sysstat:
+    packages:
+      - sysstat
path: extensions-okd-c10s.yaml
user: login: jbtrystram
body: @jcapiitao ^
created_at: 2025-04-04T17:46:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,3 +45,64 @@
   tracker: https://github.com/openshift/os/issues/1751
   osversion:
     - c10s
+
+- pattern: ext.config.shared.kdump.crash
+  tracker: TODO
+  osversion:
+    - c10s
+
+# TODO: Make this test EL9 only
+- pattern: ext.config.shared.networking.nm-ifcfg-rh-plugin
+  tracker: TODO
+  osversion:
+    - c10s
+
+# Will fail until we have proper RHEL 10 builds of OCP packages
+- pattern: ext.config.version.rhel-major-version
+  tracker: TODO
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.podman.dns
+  tracker: https://github.com/openshift/os/pull/1498
+  osversion:
+    - c10s
path: kola-denylist.yaml
user: login: jbtrystram
body: This one should pass (tested on rhel10)
created_at: 2025-04-04T17:56:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,3 +45,64 @@
   tracker: https://github.com/openshift/os/issues/1751
   osversion:
     - c10s
+
+- pattern: ext.config.shared.kdump.crash
+  tracker: TODO
+  osversion:
+    - c10s
+
+# TODO: Make this test EL9 only
+- pattern: ext.config.shared.networking.nm-ifcfg-rh-plugin
+  tracker: TODO
+  osversion:
+    - c10s
+
+# Will fail until we have proper RHEL 10 builds of OCP packages
+- pattern: ext.config.version.rhel-major-version
+  tracker: TODO
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.podman.dns
+  tracker: https://github.com/openshift/os/pull/1498
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.podman.rootless-pasta-networking
+  tracker: https://github.com/openshift/os/pull/1498
+  osversion:
+    - c10s
path: kola-denylist.yaml
user: login: jbtrystram
body: This one should pass (tested on rhel10)
created_at: 2025-04-04T17:56:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,3 +45,64 @@
   tracker: https://github.com/openshift/os/issues/1751
   osversion:
     - c10s
+
+- pattern: ext.config.shared.kdump.crash
+  tracker: TODO
+  osversion:
+    - c10s
+
+# TODO: Make this test EL9 only
+- pattern: ext.config.shared.networking.nm-ifcfg-rh-plugin
+  tracker: TODO
+  osversion:
+    - c10s
+
+# Will fail until we have proper RHEL 10 builds of OCP packages
+- pattern: ext.config.version.rhel-major-version
+  tracker: TODO
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.podman.dns
+  tracker: https://github.com/openshift/os/pull/1498
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.podman.rootless-pasta-networking
+  tracker: https://github.com/openshift/os/pull/1498
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.podman.rootless-systemd
+  tracker: https://github.com/openshift/os/pull/1498
+  osversion:
+    - c10s
path: kola-denylist.yaml
user: login: jbtrystram
body: This one should pass (tested on rhel10)
created_at: 2025-04-04T17:58:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,89 @@
+# RPMs as operating system extensions, distinct from the base ostree commit/image
+# https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md
+# and https://github.com/coreos/fedora-coreos-tracker/issues/401
+
+# repos:
+#   - sig-virtualization
+
+extensions:
+  # https://issues.redhat.com/browse/RFE-4177
+  # wasm:
+  #   architectures:
+  #     - x86_64
+  #     - aarch64
+  #   repos:
+  #     - appstream
+  #   packages:
+  #     - crun-wasm
+  # https://github.com/coreos/fedora-coreos-tracker/issues/1504
+  ipsec:
+    packages:
+      - libreswan
+      - NetworkManager-libreswan
+  # https://github.com/coreos/fedora-coreos-tracker/issues/326
+  usbguard:
+    packages:
+      - usbguard
+  kerberos:
+    packages:
+      - krb5-workstation
+      - libkadm5
+  # https://github.com/kmods-via-containers/kmods-via-containers/issues/3
+  # https://gitlab.cee.redhat.com/coreos/redhat-coreos/merge_requests/866
+  # These are currently overlaid onto the host so that they can be bind-mounted
+  # into build containers... in the future they should be a `development`
+  # extension: https://github.com/openshift/machine-config-operator/pull/2143.
+  kernel-devel:
+    packages:
+      - kernel-devel
+      - kernel-headers
+    match-base-evr: kernel
+  # These are already in the base, so they're not OS extensions, but they're
+  # useful to have in RPM form to install in kmod build containers.
+  kernel:
+    kind: development
+    packages:
+      - kernel
+      - kernel-core
+      - kernel-modules
+      - kernel-modules-extra
+    match-base-evr: kernel
+  # GRPA-2822
+  # https://github.com/openshift/machine-config-operator/pull/1330
+  # https://github.com/openshift/enhancements/blob/master/enhancements/support-for-realtime-kernel.md
+  kernel-rt:
+    architectures:
+      - x86_64
+    repos:
+      - c10s-nfv
+    packages:
+      - kernel-rt-core
+      - kernel-rt-kvm
+      - kernel-rt-modules
+      - kernel-rt-modules-extra
+      - kernel-rt-devel
+    match-base-evr: kernel
+  # https://github.com/openshift/machine-config-operator/pull/2456
+  # https://github.com/openshift/enhancements/blob/master/enhancements/sandboxed-containers/sandboxed-containers-tech-preview.md
+  # GRPA-3123
+  # - kata-containers
+  # sandboxed-containers:
+  #   architectures:
+  #     - x86_64
+  #   repos:
+  #     - sig-virtualization10
+  #   packages:
+  #     - kata-containers
+  # https://issues.redhat.com/browse/COS-2402
+  kernel-64k:
+    architectures:
+      - aarch64
+    packages:
+      - kernel-64k-core
+      - kernel-64k-modules
+      - kernel-64k-modules-core
+      - kernel-64k-modules-extra
+  # https://issues.redhat.com/browse/COS-2940
+  sysstat:
+    packages:
+      - sysstat
path: extensions-okd-c10s.yaml
user: login: joelcapitao
body: @jbtrystram Timothée has already added it in current PR
created_at: 2025-04-04T18:36:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,3 +45,37 @@
   tracker: https://github.com/openshift/os/issues/1751
   osversion:
     - c10s
+
+# TODO: Make this test EL9 only
+- pattern: ext.config.shared.networking.nm-ifcfg-rh-plugin
+  tracker: TODO
+  osversion:
+    - c10s
path: kola-denylist.yaml
user: login: jbtrystram
body: Also fixed in https://github.com/coreos/fedora-coreos-config/pull/3434
created_at: 2025-04-04T18:40:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -72,7 +72,7 @@ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
             for x in ${kernel_pkgs}; do
                 args+=(--install "${x}")
             done
-            runv rpm-ostree override remove kernel{,-core,-modules,-modules-extra,-modules-core} "${args[@]}"
+            runv rpm-ostree override remove kernel{,-core} "${args[@]}"
path: tests/kola/rpm-ostree/replace-rt-kernel/test.sh
user: login: joelcapitao
body: Indeed with iptables-1.8.11-8.el10 the test is ok :+1: 
Thank you for having dug down more on that 
created_at: 2025-04-04T18:42:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,3 +45,37 @@
   tracker: https://github.com/openshift/os/issues/1751
   osversion:
     - c10s
+
+# TODO: Make this test EL9 only
+- pattern: ext.config.shared.networking.nm-ifcfg-rh-plugin
+  tracker: TODO
+  osversion:
+    - c10s
path: kola-denylist.yaml
user: login: joelcapitao
body: Removed in [f71d5f9](https://github.com/openshift/os/pull/1498/commits/f71d5f9e4c5abdb9dfaf6946a95772e7bcedc1cc)
created_at: 2025-04-04T18:44:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,3 +45,64 @@
   tracker: https://github.com/openshift/os/issues/1751
   osversion:
     - c10s
+
+- pattern: ext.config.shared.kdump.crash
+  tracker: TODO
+  osversion:
+    - c10s
+
+# TODO: Make this test EL9 only
+- pattern: ext.config.shared.networking.nm-ifcfg-rh-plugin
+  tracker: TODO
+  osversion:
+    - c10s
+
+# Will fail until we have proper RHEL 10 builds of OCP packages
+- pattern: ext.config.version.rhel-major-version
+  tracker: TODO
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.podman.dns
+  tracker: https://github.com/openshift/os/pull/1498
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.podman.rootless-pasta-networking
+  tracker: https://github.com/openshift/os/pull/1498
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.podman.rootless-systemd
+  tracker: https://github.com/openshift/os/pull/1498
+  osversion:
+    - c10s
path: kola-denylist.yaml
user: login: joelcapitao
body: Removed in https://github.com/openshift/os/pull/1498/commits/fc2ab86dee92ab13a5af6656e3f29e2d6186d495
created_at: 2025-04-04T18:44:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,3 +45,64 @@
   tracker: https://github.com/openshift/os/issues/1751
   osversion:
     - c10s
+
+- pattern: ext.config.shared.kdump.crash
+  tracker: TODO
+  osversion:
+    - c10s
+
+# TODO: Make this test EL9 only
+- pattern: ext.config.shared.networking.nm-ifcfg-rh-plugin
+  tracker: TODO
+  osversion:
+    - c10s
+
+# Will fail until we have proper RHEL 10 builds of OCP packages
+- pattern: ext.config.version.rhel-major-version
+  tracker: TODO
+  osversion:
+    - c10s
+
+- pattern: ext.config.shared.podman.dns
+  tracker: https://github.com/openshift/os/pull/1498
+  osversion:
+    - c10s
path: kola-denylist.yaml
user: login: joelcapitao
body: Removed in https://github.com/openshift/os/pull/1498/commits/fc2ab86dee92ab13a5af6656e3f29e2d6186d495
created_at: 2025-04-04T18:44:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,3 +45,64 @@
   tracker: https://github.com/openshift/os/issues/1751
   osversion:
     - c10s
+
+- pattern: ext.config.shared.kdump.crash
+  tracker: TODO
+  osversion:
+    - c10s
path: kola-denylist.yaml
user: login: joelcapitao
body: Removed in https://github.com/openshift/os/pull/1498/commits/fc2ab86dee92ab13a5af6656e3f29e2d6186d495
created_at: 2025-04-04T18:45:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,3 +45,37 @@
   tracker: https://github.com/openshift/os/issues/1751
   osversion:
     - c10s
+
+# TODO: Make this test EL9 only
+- pattern: ext.config.shared.networking.nm-ifcfg-rh-plugin
+  tracker: TODO
+  osversion:
+    - c10s
path: kola-denylist.yaml
user: login: jbtrystram
body: it still needs to be denylisted until the next f-c-c submodule bmp
created_at: 2025-04-04T18:55:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3,7 +3,6 @@
 # and https://github.com/coreos/fedora-coreos-tracker/issues/401
 
 repos:
-  - c9s-sig-virtualization
path: extensions-okd-c9s.yaml
user: login: Prashanth684
body: ```
error: Packages not found: openvswitch3.5-ipsec
```
Looks like we need to add `c9s-sig-nfv` here
created_at: 2025-04-04T20:59:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3,7 +3,6 @@
 # and https://github.com/coreos/fedora-coreos-tracker/issues/401
 
 repos:
-  - c9s-sig-virtualization
path: extensions-okd-c9s.yaml
user: login: dustymabe
body: Should be fixed now.
created_at: 2025-04-05T00:03:18Z
author_association: MEMBER

PR comments:
user: login: travier
created_at: 2024-04-30T16:06:06Z
body: Do not merge as it includes changes that we do not want but that are here to test in CI (see commits).
We'll have to update the Prow CI config once ready.
/hold
author_association: MEMBER
user: login: travier
created_at: 2024-05-02T15:57:10Z
body: /retest
author_association: MEMBER
user: login: travier
created_at: 2024-05-03T12:32:40Z
body: /retest
author_association: MEMBER
user: login: travier
created_at: 2024-05-06T09:52:08Z
body: /retest
author_association: MEMBER
user: login: jlebon
created_at: 2024-05-06T17:52:56Z
body: Hmm, almost seems like the logs are truncated... This is probably fallout from https://github.com/coreos/coreos-assembler/pull/3785.
author_association: MEMBER
user: login: jlebon
created_at: 2024-05-06T18:21:14Z
body: Let's see if https://github.com/coreos/coreos-assembler/pull/3792 helps (then here, we'd `export TAIL_SLEEP=3` or something).
author_association: MEMBER
user: login: jlebon
created_at: 2024-05-07T03:21:14Z
body: /retest
author_association: MEMBER
user: login: jlebon
created_at: 2024-05-07T03:35:55Z
body: Ahh and indeed now we clearly see the error for all of them which means we were missing output. Yuck OK, so we need to fix the `tail` handling. Though I'll see if I can bang on the virtio-serial bug again.
author_association: MEMBER
user: login: travier
created_at: 2024-05-07T08:50:46Z
body: > Ahh and indeed now we clearly see the error for all of them which means we were missing output. Yuck OK, so we need to fix the `tail` handling. Though I'll see if I can bang on the virtio-serial bug again.

Thanks! I indeed had a missing change in those commits.
author_association: MEMBER
user: login: travier
created_at: 2024-05-07T08:53:18Z
body: Hum, the workaround is not that ugly and only impacts CI here so maybe we should merge it for now until we've fixed this COSA.
author_association: MEMBER
user: login: travier
created_at: 2024-05-07T09:48:04Z
body: CI fixes in https://github.com/openshift/release/pull/51750
author_association: MEMBER
user: login: jlebon
created_at: 2024-05-07T18:24:00Z
body: > and only impacts CI here

So far. :) It's racy so I don't see why it couldn't happen in the prod pipeline. I'd hate for someone to be debugging a failure there and working with incomplete output thinking the error is happening somewhere other than it really is.

I guess though a major hack is to just add a `sleep 1` at least on the cosa side.
author_association: MEMBER
user: login: travier
created_at: 2024-05-13T16:06:09Z
body: Failing on missing the teamd package.
author_association: MEMBER
user: login: travier
created_at: 2024-05-13T16:08:55Z
body: https://gitlab.com/redhat/centos-stream/rpms/NetworkManager/-/commit/2d2f70f74f4950b2dc6eef7e9541f32e6da0544f
author_association: MEMBER
user: login: travier
created_at: 2024-05-13T16:12:46Z
body: https://github.com/coreos/fedora-coreos-tracker/issues/1727
author_association: MEMBER
user: login: travier
created_at: 2024-05-14T10:33:51Z
body: /retest
author_association: MEMBER
user: login: travier
created_at: 2024-05-14T12:53:20Z
body: kdump not working:
```
systemctl status kola-runext.service:
× kola-runext.service
     Loaded: loaded (/etc/systemd/system/kola-runext.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2024-05-14 12:01:37 UTC; 207ms ago
   Duration: 79ms
    Process: 2200 ExecStart=/usr/local/bin/kola-runext-test.sh (code=exited, status=1/FAILURE)
   Main PID: 2200 (code=exited, status=1/FAILURE)
        CPU: 27ms

May 14 12:01:36 qemu0 systemd[1]: Started kola-runext.service.
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: + . /var/opt/kola/extdata/commonlib.sh
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: ++ IFS=' '
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: ++ read -r -a cmdline
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: + case "${AUTOPKGTEST_REBOOT_MARK:-}" in
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: + is_service_active kdump.service
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: + local service=kdump.service
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: + for _x in {0..60}
May 14 12:01:36 qemu0 kola-runext-test.sh[2201]: ++ systemctl is-active kdump.service
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: + '[' inactive '!=' activating ']'
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: + break
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: + systemctl is-active kdump.service
May 14 12:01:37 qemu0 kola-runext-test.sh[2202]: inactive
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: + fatal 'kdump.service failed to start'
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: + echo 'kdump.service failed to start'
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: kdump.service failed to start
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: + exit 1
May 14 12:01:37 qemu0 systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
May 14 12:01:37 qemu0 systemd[1]: kola-runext.service: Failed with result 'exit-code'.
--- [31mFAIL[0m: ext.config.shared.kdump.crash (101.06s)
        cluster.go:162: Error: Unit kola-runext.service exited with code 1
        cluster.go:162: 2024-05-14T12:01:37Z cli: Unit kola-runext.service exited with code 1
        harness.go:1263: kolet failed: : kolet run-test-unit failed: Process exited with status 1
```

teamd test not working (expected as we removed team support):
```
systemctl status kola-runext.service:
× kola-runext.service
     Loaded: loaded (/etc/systemd/system/kola-runext.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2024-05-14 12:02:35 UTC; 195ms ago
   Duration: 66ms
    Process: 2164 ExecStart=/usr/local/bin/kola-runext-test.sh (code=exited, status=1/FAILURE)
   Main PID: 2164 (code=exited, status=1/FAILURE)
        CPU: 20ms

May 14 12:02:35 qemu0 systemd[1]: Started kola-runext.service.
May 14 12:02:35 qemu0 kola-runext-test.sh[2164]: + . /var/opt/kola/extdata/commonlib.sh
May 14 12:02:35 qemu0 kola-runext-test.sh[2164]: ++ IFS=' '
May 14 12:02:35 qemu0 kola-runext-test.sh[2164]: ++ read -r -a cmdline
May 14 12:02:35 qemu0 kola-runext-test.sh[2164]: + team=team0
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ get_ipv4_for_nic team0
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ local nic_name=team0
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ local ip
May 14 12:02:35 qemu0 kola-runext-test.sh[2167]: +++ ip -j addr show team0
May 14 12:02:35 qemu0 kola-runext-test.sh[2168]: +++ jq -r '.[0].addr_info | map(select(.family == "inet")) | .[0].local'
May 14 12:02:35 qemu0 kola-runext-test.sh[2167]: Device "team0" does not exist.
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ ip=
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ '[' -z '' ']'
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ echo 'Error: can not get ip for team0'
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ exit 1
May 14 12:02:35 qemu0 kola-runext-test.sh[2164]: + nic_ip='Error: can not get ip for team0'
May 14 12:02:35 qemu0 systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
May 14 12:02:35 qemu0 systemd[1]: kola-runext.service: Failed with result 'exit-code'.
--- [31mFAIL[0m: ext.config.shared.networking.team-dhcp-via-ignition (59.40s)
        cluster.go:162: Error: Unit kola-runext.service exited with code 1
        cluster.go:162: 2024-05-14T12:02:35Z cli: Unit kola-runext.service exited with code 1
        harness.go:1263: kolet failed: : kolet run-test-unit failed: Process exited with status 1
```

No more ifcfg files:
```
=== RUN   non-exclusive-test-bucket-0/ext.config.shared.networking.nm-ifcfg-rh-plugin
systemctl status kola-runext-63.service:
× kola-runext-63.service
     Loaded: loaded (/etc/systemd/system/kola-runext-63.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2024-05-14 12:24:42 UTC; 248ms ago
   Duration: 59ms
    Process: 5657 ExecStart=/usr/local/bin/kola-runext-nm-ifcfg-rh-plugin (code=exited, status=1/FAILURE)
   Main PID: 5657 (code=exited, status=1/FAILURE)
        CPU: 21ms

May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ DOCUMENTATION_URL=https://docs.okd.io/latest/welcome/index.html
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ BUG_REPORT_URL=https://access.redhat.com/labs/rhir/
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_BUGZILLA_PRODUCT='OpenShift Container Platform'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_BUGZILLA_PRODUCT_VERSION=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_SUPPORT_PRODUCT='OpenShift Container Platform'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_SUPPORT_PRODUCT_VERSION=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ OPENSHIFT_VERSION=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ OSTREE_VERSION=416.10.202405141044-0
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + '[' scos == rhcos ']'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + is_scos
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + source /etc/os-release
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ NAME='CentOS Stream CoreOS'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ ID=scos
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ ID_LIKE='rhel fedora'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ VERSION=416.10.202405141044-0
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ VERSION_ID=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ VARIANT=CoreOS
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ VARIANT_ID=coreos
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ PLATFORM_ID=platform:el10
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ PRETTY_NAME='CentOS Stream CoreOS 416.10.202405141044-0'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ ANSI_COLOR='0;31'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ CPE_NAME=cpe:/o:centos:centos:10::coreos
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ HOME_URL=https://centos.org/
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ DOCUMENTATION_URL=https://docs.okd.io/latest/welcome/index.html
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ BUG_REPORT_URL=https://access.redhat.com/labs/rhir/
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_BUGZILLA_PRODUCT='OpenShift Container Platform'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_BUGZILLA_PRODUCT_VERSION=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_SUPPORT_PRODUCT='OpenShift Container Platform'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_SUPPORT_PRODUCT_VERSION=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ OPENSHIFT_VERSION=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ OSTREE_VERSION=416.10.202405141044-0
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + '[' scos == scos ']'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + '[' coreos == coreos ']'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + '[' 0 == 1 ']'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + fatal 'ifcfg-rh plugin not detected on RHCOS/SCOS'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + echo 'ifcfg-rh plugin not detected on RHCOS/SCOS'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ifcfg-rh plugin not detected on RHCOS/SCOS
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + exit 1
May 14 12:24:42 qemu0 systemd[1]: kola-runext-63.service: Main process exited, code=exited, status=1/FAILURE
May 14 12:24:42 qemu0 systemd[1]: kola-runext-63.service: Failed with result 'exit-code'.
```

Missing AMD ucode:
```
=== RUN   non-exclusive-test-bucket-0/ext.config.shared.files.amd-ucode-firmware
systemctl status kola-runext-28.service:
× kola-runext-28.service
     Loaded: loaded (/etc/systemd/system/kola-runext-28.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2024-05-14 12:28:59 UTC; 208ms ago
   Duration: 29ms
    Process: 13697 ExecStart=/usr/local/bin/kola-runext-amd-ucode-firmware (code=exited, status=1/FAILURE)
   Main PID: 13697 (code=exited, status=1/FAILURE)
        CPU: 10ms

May 14 12:28:59 qemu0 systemd[1]: Started kola-runext-28.service.
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: + . /var/opt/kola/extdata-28/commonlib.sh
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: ++ IFS=' '
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: ++ read -r -a cmdline
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: + ls '/usr/lib/firmware/amd-ucode/microcode*'
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13698]: ls: cannot access '/usr/lib/firmware/amd-ucode/microcode*': No such file or directory
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: + fatal 'no AMD microcode found on host'
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: + echo 'no AMD microcode found on host'
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: no AMD microcode found on host
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: + exit 1
May 14 12:28:59 qemu0 systemd[1]: kola-runext-28.service: Main process exited, code=exited, status=1/FAILURE
May 14 12:28:59 qemu0 systemd[1]: kola-runext-28.service: Failed with result 'exit-code'.
```

Change in systemd timeout:
```
=== RUN   non-exclusive-test-bucket-0/ext.config.shared.systemd.default-unit-timeouts
systemctl status kola-runext-79.service:
× kola-runext-79.service
     Loaded: loaded (/etc/systemd/system/kola-runext-79.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2024-05-14 12:29:10 UTC; 1s ago
   Duration: 114ms
    Process: 14540 ExecStart=/usr/local/bin/kola-runext-default-unit-timeouts (code=exited, status=1/FAILURE)
   Main PID: 14540 (code=exited, status=1/FAILURE)
        CPU: 55ms

May 14 12:29:10 qemu0 systemd[1]: Started kola-runext-79.service.
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + . /var/opt/kola/extdata-79/commonlib.sh
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: ++ IFS=' '
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: ++ read -r -a cmdline
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14541]: ++ systemctl show --value --property=DefaultTimeoutStartUSec
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + DefaultTimeoutStartUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14542]: ++ systemctl show --value --property=DefaultTimeoutStopUSec
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + DefaultTimeoutStopUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14543]: ++ systemctl show --value --property=DefaultTimeoutAbortUSec
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + DefaultTimeoutAbortUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14544]: ++ systemctl show --value --property=DefaultDeviceTimeoutUSec
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + DefaultDeviceTimeoutUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + error='DefaultTimeoutStartUsec=45s '
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + error+='DefaultTimeoutStopUSec=45s '
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + error+='DefaultTimeoutAbortUSec=45s '
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + error+=DefaultDeviceTimeoutUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + timeout='1min 30s'
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14545]: ++ rpm -q --queryformat '%{VERSION}' systemd
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + systemd_version=255.3
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + required_version=252.4
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + [[ 45s == \1\m\i\n\ \3\0\s ]]
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + fatal DefaultTimeoutStartUsec=45s DefaultTimeoutStopUSec=45s DefaultTimeoutAbortUSec=45s DefaultDeviceTimeoutUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + echo DefaultTimeoutStartUsec=45s DefaultTimeoutStopUSec=45s DefaultTimeoutAbortUSec=45s DefaultDeviceTimeoutUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: DefaultTimeoutStartUsec=45s DefaultTimeoutStopUSec=45s DefaultTimeoutAbortUSec=45s DefaultDeviceTimeoutUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + exit 1
May 14 12:29:10 qemu0 systemd[1]: kola-runext-79.service: Main process exited, code=exited, status=1/FAILURE
May 14 12:29:10 qemu0 systemd[1]: kola-runext-79.service: Failed with result 'exit-code'.
```

RHEL major version test:
```
=== RUN   non-exclusive-test-bucket-0/ext.config.version.rhel-major-version
systemctl status kola-runext-83.service:
× kola-runext-83.service
     Loaded: loaded (/etc/systemd/system/kola-runext-83.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2024-05-14 12:32:29 UTC; 350ms ago
   Duration: 4.889s
    Process: 15664 ExecStart=/usr/local/bin/kola-runext-rhel-major-version (code=exited, status=1/FAILURE)
   Main PID: 15664 (code=exited, status=1/FAILURE)
        CPU: 4.202s

May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16372]: ++ head -1
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + release=10
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ 10 != \1\0 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + for pkg in $(rpm -qa)
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16373]: ++ rpm -q '--queryformat=%{NAME}' libjose-12-1.el10.x86_64
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + name=libjose
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ libjose == \s\h\i\m\-\x\6\4 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ libjose == \s\h\i\m\-\a\a\6\4 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16375]: ++ rpm -q '--queryformat=%{RELEASE}' libjose-12-1.el10.x86_64
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16376]: ++ grep -oP 'el\s*\K\d+'
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16377]: ++ head -1
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + release=10
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ 10 != \1\0 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + for pkg in $(rpm -qa)
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16378]: ++ rpm -q '--queryformat=%{NAME}' shared-mime-info-2.3-4.el10.x86_64
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + name=shared-mime-info
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ shared-mime-info == \s\h\i\m\-\x\6\4 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ shared-mime-info == \s\h\i\m\-\a\a\6\4 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16381]: ++ grep -oP 'el\s*\K\d+'
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16380]: ++ rpm -q '--queryformat=%{RELEASE}' shared-mime-info-2.3-4.el10.x86_64
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16382]: ++ head -1
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + release=10
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ 10 != \1\0 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + for pkg in $(rpm -qa)
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16383]: ++ rpm -q '--queryformat=%{NAME}' conmon-2.1.10-2.1.rhaos4.16.el9.x86_64
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + name=conmon
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ conmon == \s\h\i\m\-\x\6\4 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ conmon == \s\h\i\m\-\a\a\6\4 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16386]: ++ grep -oP 'el\s*\K\d+'
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16385]: ++ rpm -q '--queryformat=%{RELEASE}' conmon-2.1.10-2.1.rhaos4.16.el9.x86_64
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16387]: ++ head -1
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + release=9
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ 9 != \1\0 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + fatal 'Found packages '\''conmon'\'' with release version '\''9'\'' that does not match the OS current version '\''10'\'''
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + echo 'Found packages '\''conmon'\'' with release version '\''9'\'' that does not match the OS current version '\''10'\'''
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: Found packages 'conmon' with release version '9' that does not match the OS current version '10'
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + exit 1
May 14 12:32:29 qemu0 systemd[1]: kola-runext-83.service: Main process exited, code=exited, status=1/FAILURE
May 14 12:32:29 qemu0 systemd[1]: kola-runext-83.service: Failed with result 'exit-code'.
May 14 12:32:29 qemu0 systemd[1]: kola-runext-83.service: Consumed 4.202s CPU time, 1.9M memory peak, 0B memory swap peak.
```
author_association: MEMBER
user: login: travier
created_at: 2024-05-14T16:39:38Z
body: ```
Installing packages: Packages not found: amd-gpu-firmware, intel-gpu-firmware, nvidia-gpu-firmware
```
author_association: MEMBER
user: login: travier
created_at: 2024-05-14T16:47:36Z
body: https://issues.redhat.com/browse/RHEL-36319
author_association: MEMBER
user: login: travier
created_at: 2024-05-23T13:26:57Z
body: ```
    --- [31mFAIL[0m: non-exclusive-test-bucket-0/ext.config.shared.podman.dns (180.51s)
            harness.go:106: TIMEOUT[3m0s]: ssh: sudo ./kolet run-test-unit --deny-reboots kola-runext-69.service
            harness.go:106: TIMEOUT[3m0s]: ssh: journalctl -t kola-runext-test.sh
```

```
    --- [31mFAIL[0m: non-exclusive-test-bucket-0/ext.config.shared.podman.rootless-pasta-networking (180.48s)
            harness.go:106: TIMEOUT[3m0s]: ssh: sudo ./kolet run-test-unit --deny-reboots kola-runext-70.service
            harness.go:106: TIMEOUT[3m0s]: ssh: journalctl -t kola-runext-rootless-pasta-networking
```

```
2024-05-23T09:56:53Z kola: Test timed out. Adding as candidate for rerun success: ext.config.shared.podman.rootless-systemd
--- [31mFAIL[0m: ext.config.shared.podman.rootless-systemd (904.79s)
        harness.go:106: TIMEOUT[15m0s]: ssh: sudo ./kolet run-test-unit kola-runext.service
        harness.go:106: TIMEOUT[15m0s]: ssh: journalctl -t kola-runext-rootless-systemd
```
author_association: MEMBER
user: login: travier
created_at: 2024-06-06T06:15:22Z
body: ```
error: Installing packages: Could not depsolve transaction; 1 problem detected:
 Problem: conflicting requests
  - nothing provides containernetworking-plugins >= 0.8.2-3 needed by cri-o-1.28.6-7.rhaos4.15.git70c2e96.el9.x86_64 from rhel-9.4-server-ose-4.17
  - nothing provides containernetworking-plugins >= 0.8.2-3 needed by cri-o-1.29.4-6.rhaos4.16.git0e93ae2.el9.x86_64 from rhel-9.4-server-ose-4.17
failed to execute cmd-fetch: exit status 1
```

We'll need new cri-o builds to move forward.
author_association: MEMBER
user: login: travier
created_at: 2024-08-28T13:02:36Z
body: We'll need the prep work for https://github.com/coreos/fedora-coreos-tracker/issues/1787 as well
author_association: MEMBER
user: login: jlebon
created_at: 2024-12-05T15:11:37Z
body: Note we should also add a `rhel-10.0` variant as part of this.
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-02-18T09:47:30Z
body: /retest
author_association: CONTRIBUTOR
user: login: HuijingHei
created_at: 2025-02-19T01:44:13Z
body: /retest
author_association: CONTRIBUTOR
user: login: travier
created_at: 2025-02-19T13:39:35Z
body: Needs https://github.com/coreos/fedora-coreos-config/pull/3367 and a fix for the gdisk requirement from the ignition dracut module.
author_association: MEMBER
user: login: dustymabe
created_at: 2025-02-19T14:05:19Z
body: Thank you for working on this @travier. I'm not going to lie it's getting extremely hard to follow everything and you've done a good job of getting it right so far.

Longer term I think we're going to need to simplify all of this. 
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-02-19T15:27:20Z
body: > Needs ... and a fix for the gdisk requirement from the ignition dracut module.

https://github.com/coreos/fedora-coreos-config/pull/3368

author_association: CONTRIBUTOR
user: login: dustymabe
created_at: 2025-02-19T21:55:25Z
body: CI failed with:

```
error: Installing packages: Unknown rpm-md repository: rhel-9.6-fast-datapath
```

author_association: MEMBER
user: login: jbtrystram
created_at: 2025-02-19T23:08:40Z
body: > CI failed with:
> 
> ```
> error: Installing packages: Unknown rpm-md repository: rhel-9.6-fast-datapath
> ```
> 

Looking at the logs, [this]( https://github.com/openshift/os/commit/fa906886466489a19ae3d8aeb6d9e238ca5dffbb) doesn't work as expected maybe


author_association: CONTRIBUTOR
user: login: travier
created_at: 2025-02-20T13:46:13Z
body: Will rebase again now that we have https://github.com/openshift/os/pull/1752.
author_association: MEMBER
user: login: dustymabe
created_at: 2025-02-20T16:36:17Z
body: I think building the ISO here is going to fail until `coreos-installer` is updated. It currently is at `coreos-installer-0.22.1-2.el10.x86_64` in c10s so building the Live ISO will fail until we get `0.23+`. 
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-02-27T13:31:35Z
body: /retest
author_association: CONTRIBUTOR
user: login: travier
created_at: 2025-03-10T14:49:45Z
body: I had pushed [`00a07bb` (#1498)](https://github.com/openshift/os/pull/1498/commits/00a07bbece76c8c57be0a46383db595f98a3c266) to try to fix this.
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-03-21T14:37:53Z
body: @travier regarding the last commit here to fix multipath, do you think we should make it into a el10-only manifest or an overlay is fine ?
author_association: CONTRIBUTOR
user: login: jbtrystram
created_at: 2025-03-24T09:31:15Z
body: I messed up the last rebase here
author_association: CONTRIBUTOR
user: login: jlebon
created_at: 2025-03-31T14:46:09Z
body: I've opened https://github.com/openshift/os/pull/1780 which cleans a lot of things up and should make landing this easier!
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-04-03T10:13:21Z
body: /retitle [COS-2902]:  Add C10S variant

author_association: CONTRIBUTOR
user: login: joelcapitao
created_at: 2025-04-03T10:23:48Z
body: The PR is currently rebased on top of https://github.com/openshift/os/pull/1780 which is not yet merged, so the tests will fail.
We cleaned up the history (some drop and squash commit) to include the latest comment.
author_association: CONTRIBUTOR
user: login: jbtrystram
created_at: 2025-04-03T14:22:22Z
body: [COS-2902](https://issues.redhat.com//browse/COS-2902): Add C10S variant
author_association: CONTRIBUTOR
user: login: joelcapitao
created_at: 2025-04-04T14:55:00Z
body: Requires https://github.com/coreos/coreos-assembler/pull/4064 
author_association: CONTRIBUTOR
user: login: joelcapitao
created_at: 2025-04-04T18:39:14Z
body: > Thank you for adding more context in the commits!
> 
> First commit says "aglusterfs-fusend" which I think is a typo.
> 
> Otherwise, this looks sane to me!
> 
> /approve
> 
> I'd add a `/lgtm` but this will need a rebase anyway for #1780.

alright, I fixed the typo
author_association: CONTRIBUTOR
user: login: jbtrystram
created_at: 2025-04-04T19:18:05Z
body: /unhold
author_association: CONTRIBUTOR
user: login: Prashanth684
created_at: 2025-04-04T21:06:43Z
body: will there be a way to switch between building c9s and c10s, or will we have a follow up moving to c10s (in packages-openshift.yaml  and other places) after this PR?
author_association: CONTRIBUTOR
user: login: dustymabe
created_at: 2025-04-04T23:50:12Z
body: I'm looking into the failures here.
author_association: MEMBER
user: login: dustymabe
created_at: 2025-04-05T00:04:01Z
body: Pushed up a few fixes.. still working on some other things.
author_association: MEMBER
user: login: dustymabe
created_at: 2025-04-05T02:25:02Z
body: ok a few more fixes now. I think (hope) the only thing left failing is the okd-scos-images
author_association: MEMBER
user: login: dustymabe
created_at: 2025-04-05T02:35:37Z
body: Please everyone refrain from pushing to this PR unless I ask you to do so. 
author_association: MEMBER
user: login: jlebon
created_at: 2025-04-05T02:42:08Z
body: > will there be a way to switch between building c9s and c10s, or will we have a follow up moving to c10s (in packages-openshift.yaml and other places) after this PR?

We'll maintain both c9s and c10s variants. The repos in `packages-openshift.yaml` should be adapted so it's conditional on the version as well as `centos` vs `rhel`.
author_association: MEMBER
user: login: openshift-ci[bot]
created_at: 2025-04-05T05:04:55Z
body: @travier: all tests **passed!**


[Full PR test history](https://prow.ci.openshift.org/pr-history?org=openshift&repo=os&pr=1498). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:travier).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
author_association: CONTRIBUTOR
user: login: jbtrystram
created_at: 2025-04-05T07:38:32Z
body: /retitle [COS-2902](https://issues.redhat.com//browse/COS-2902): Add C10s variant
author_association: CONTRIBUTOR
user: login: openshift-ci-robot
created_at: 2025-04-05T07:38:38Z
body: @travier: This pull request references [COS-2902](https://issues.redhat.com//browse/COS-2902) which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

<details>

In response to [this](https://github.com/openshift/os/pull/1498):

>c9s: Point image-c9s to image-rhel-9.4
>
>---
>
>c9s: Use RPMs from 9.4 RHAOS repo
>
>---
>
>Revert "c9s.repo: temporarily use mirrored repos"
>
>This reverts commit 88e41a0fef0dbe8e0b2d103cadd37217c8cd0933.
>
>---
>
>DoNotMerge: CI changes to test C10S builds
>
>---
>
>manifests: Add initial c10s based variant
>
>See: https://github.com/openshift/os/issues/1466
>
>---
>
>kola-denylist.yaml: Update for c10s variant
>
>---
>
>c10s.repo: Skip GPG checks for now
>
>---
>
>Corresponding Fedora CoreOS changes: https://github.com/coreos/fedora-coreos-config/pull/3015


Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fos).  If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
</details>
author_association: NONE
user: login: jbtrystram
created_at: 2025-04-05T07:41:07Z
body: /lgtm
author_association: CONTRIBUTOR
user: login: openshift-ci[bot]
created_at: 2025-04-05T07:41:27Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/openshift/os/pull/1498#issuecomment-2780400771" title="LGTM">jbtrystram</a>*, *<a href="https://github.com/openshift/os/pull/1498#pullrequestreview-2743530104" title="Approved">jlebon</a>*, *<a href="https://github.com/openshift/os/pull/1498#" title="Author self-approved">travier</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=openshift%2Fos).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/openshift/os/blob/master/OWNERS)~~ [jbtrystram,jlebon,travier]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: joelcapitao
created_at: 2025-04-08T14:37:47Z
body: > will there be a way to switch between building c9s and c10s, or will we have a follow up moving to c10s (in packages-openshift.yaml and other places) after this PR?

@Prashanth684 there is an on-going follow-up patch https://github.com/openshift/os/pull/1786 which implements what you are asking for (specially the commit https://github.com/openshift/os/pull/1786/commits/e6213566189af6602a5e6c3aa79390457e49b2f7)
author_association: CONTRIBUTOR