Repository: coreos/fedora-coreos-config
Pull request #3458: overlay: migrate existing systems to OCI
Author: login: jbtrystram
Created: 2025-04-15T15:08:06Z | Merged: 2025-05-22T03:43:58Z

In [1] we moved newly provisioned systems to be
deployed via container and thus retrieving updates via zincati from the OCI registry (quay.io/fedora/fedora-coreos).

This starts the migration script shipped in [2]
before Zincati to set up the migration at the next update.

A way to override this is to add another drop-in
with `ExecStartPre=` with a higher ranking name
under the systemd drop in path, e.g.
`/etc/systemd/system/zincati.service.d/90-no-oci.conf`.

[1] https://github.com/coreos/fedora-coreos-tracker/issues/1823
[2] https://github.com/coreos/fedora-coreos-config/pull/3355

Closes https://github.com/coreos/fedora-coreos-tracker/issues/1890


Diff:
diff --git a/manifests/fedora-coreos.yaml b/manifests/fedora-coreos.yaml
index 2e9e608a39..b7c1241ae4 100644
--- a/manifests/fedora-coreos.yaml
+++ b/manifests/fedora-coreos.yaml
@@ -33,7 +33,18 @@ conditional-include:
       # All Fedora CoreOS streams share the same pool for locked files.
       lockfile-repos:
         - fedora-coreos-pool
-
+  - if: stream == "next"
+    include:
+      ostree-layers:
+        - overlay/35oci-migration
+  - if: stream == "next-devel"
+    include:
+      ostree-layers:
+        - overlay/35oci-migration
+  - if: stream == "rawhide"
+    include:
+      ostree-layers:
+        - overlay/35oci-migration
 
 ostree-layers:
   - overlay/15fcos
diff --git a/overlay.d/35oci-migration/usr/lib/systemd/system-preset/35-oci-migration.preset b/overlay.d/35oci-migration/usr/lib/systemd/system-preset/35-oci-migration.preset
new file mode 100644
index 0000000000..27dca04f83
--- /dev/null
+++ b/overlay.d/35oci-migration/usr/lib/systemd/system-preset/35-oci-migration.preset
@@ -0,0 +1,2 @@
+# show a motd if the OCI migration cannot be done automatically
+enable coreos-oci-migration-motd.service
diff --git a/overlay.d/35oci-migration/usr/lib/systemd/system/coreos-oci-migration-motd.service b/overlay.d/35oci-migration/usr/lib/systemd/system/coreos-oci-migration-motd.service
new file mode 100644
index 0000000000..1c5cd06261
--- /dev/null
+++ b/overlay.d/35oci-migration/usr/lib/systemd/system/coreos-oci-migration-motd.service
@@ -0,0 +1,12 @@
+# This service is used for printing a message if
+# the oci migration failed or was opted out.
+[Unit]
+Description=Warn based on oci migration status
+Before=systemd-user-sessions.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/libexec/coreos-oci-migration-motd
+RemainAfterExit=yes
+[Install]
+WantedBy=multi-user.target
diff --git a/overlay.d/35oci-migration/usr/lib/systemd/system/zincati.service.d/010-oci-migration.conf b/overlay.d/35oci-migration/usr/lib/systemd/system/zincati.service.d/010-oci-migration.conf
new file mode 100644
index 0000000000..685c2eb310
--- /dev/null
+++ b/overlay.d/35oci-migration/usr/lib/systemd/system/zincati.service.d/010-oci-migration.conf
@@ -0,0 +1,5 @@
+[Service]
+StateDirectory=coreos-oci-migration
+ExecStartPre=-/usr/libexec/coreos-oci-rebase
+# wait for 3 min
+TimeoutStartSec=300
diff --git a/overlay.d/35oci-migration/usr/libexec/coreos-oci-migration-motd b/overlay.d/35oci-migration/usr/libexec/coreos-oci-migration-motd
new file mode 100755
index 0000000000..5b282d29ba
--- /dev/null
+++ b/overlay.d/35oci-migration/usr/libexec/coreos-oci-migration-motd
@@ -0,0 +1,78 @@
+#!/usr/bin/bash
+# This script checks if oci migration failed
+# or was opted-out and prints the appropriate message
+# to the console.
+#
+# See https://github.com/coreos/fedora-coreos-tracker/issues/1890
+# See migration script discussion in https://github.com/coreos/fedora-coreos-config/pull/3355
+
+OCI_MIGRATION_OPT_OUT=/var/lib/coreos-oci-migration/opt-out.stamp
+OCI_MIGRATION_FAILED=/var/lib/coreos-oci-migration/failed.stamp
+
+ZINCATI_ENABLED=$(systemctl is-enabled zincati.service)
+
+MOTD_PATH=/run/motd.d/30_oci_migration.motd
+# Start by deleting any previous motd we created,
+# e.g. if the migration failed then the user opted out, or fixed it.
+rm -f $MOTD_PATH
+
+opt_out_message="
+This system has been opted out of the migration to OCI images for
+updates as the opt-out stamp file exists.
+
+This system will keep updating using the legacy OSTree repository,
+but later this year new Fedora CoreOS updates will cease to be pushed
+to the OSTree repository.
+
+When ready, the migration can be resumed by deleting
+${OCI_MIGRATION_OPT_OUT}, then restarting zincati.service"
+
+failed_message="
+The migration to OCI images for updates failed. Check the logs of
+zincati.service for more details.
+
+This system will keep updating using the legacy OSTree repository,
+but later this year new Fedora CoreOS updates will cease to be pushed
+to the OSTree repository.
+
+When ready, the migration can be retried by deleting
+${OCI_MIGRATION_FAILED}, then restarting zincati.service"
+
+zincati_disabled_message="
+The zincati service is disabled on this system.
+
+If not done so already please migrate the system to the OCI backend
+for updates by running:
+
+stream=stable # or testing or next
+target=ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:\$stream
+sudo rpm-ostree rebase \$target --reboot
+
+Where \$stream matches the Fedora CoreOS stream the system is following."
+
+write_motd () {
+
+# Change the output color to yellow
+warn=$(echo -e '\033[0;33m')
+# No color
+nc=$(echo -e '\033[0m')
+
+cat << EOF > "${MOTD_PATH}"
+${warn}
+##########################################################################
+${1}
+
+To disable this warning, use:
+sudo systemctl disable coreos-oci-migration-motd.service
+##########################################################################
+${nc}
+EOF
+}
+
+if [ -e $OCI_MIGRATION_OPT_OUT ]; then
+  write_motd "$opt_out_message"
+elif [ -e $OCI_MIGRATION_FAILED ]; then
+  write_motd "$failed_message"
+elif [ "$ZINCATI_ENABLED" == "disabled" ]; then
+  write_motd "$zincati_disabled_message"
+fi
diff --git a/overlay.d/15fcos/usr/libexec/coreos-oci-rebase b/overlay.d/35oci-migration/usr/libexec/coreos-oci-rebase
similarity index 78%
rename from overlay.d/15fcos/usr/libexec/coreos-oci-rebase
rename to overlay.d/35oci-migration/usr/libexec/coreos-oci-rebase
index 61ab5be987..254674cbe4 100755
--- a/overlay.d/15fcos/usr/libexec/coreos-oci-rebase
+++ b/overlay.d/35oci-migration/usr/libexec/coreos-oci-rebase
@@ -8,6 +8,20 @@
 # see https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates
 # and https://github.com/coreos/fedora-coreos-tracker/issues/1823
 
+
+OCI_MIGRATION_OPT_OUT=/var/lib/coreos-oci-migration/opt-out.stamp
+OCI_MIGRATION_FAILED=/var/lib/coreos-oci-migration/failed.stamp
+
+if [ -e $OCI_MIGRATION_OPT_OUT ]; then
+    echo "${OCI_MIGRATION_OPT_OUT} to opt out of migration exists, doing nothing."
+    exit 0
+fi
+
+# create the failed stamp file as a precaution
+touch $OCI_MIGRATION_FAILED
+# if something goes wrong, exit the script so the failed stamp will remain
+set -euxo pipefail
+
 CINCINNATI_URL="${CINCINNATI_URL:-https://raw-updates.coreos.fedoraproject.org/v1/graph}"
 
 # Save the rpm-ostree status.
@@ -19,6 +33,7 @@ booted_imgref=$(jq -r '.deployments[0]."container-image-reference"' <<< "$status
 
 if [ "$booted_imgref" != "null" ]; then
     echo "The booted deployment is already an OCI container."
+    rm $OCI_MIGRATION_FAILED
     exit 0
 fi
 
@@ -27,7 +42,7 @@ origin=$(jq -r '.deployments[0].origin' <<< "$status" | cut -d ':' -f 1)
 origin_url=$(ostree remote show-url "$origin")
 if [ "$origin_url" != "https://ostree.fedoraproject.org" ]; then
     echo "ERROR: The OSTree origin is not matching the default Fedora CoreOS."
-    exit 0
+    exit 1
 fi
 
 # fetch the SHA checksum of the matching OCI image for the booted deployment
@@ -37,15 +52,20 @@ checksum=$(jq -r '.deployments[0].checksum' <<< "$status")
 arch=$(arch)
 cin_url="$CINCINNATI_URL?basearch=$arch&stream=$stream&oci=true"
 
+# Wait for the network to be online
+echo "Waiting for the network to be online."
+nm-online --timeout 120
+
 # Grab the OCI update graph that matches our stream and arch.
 echo "Fetching cincinnati update graph for stream $stream on $arch"
 cincinnati_graph=$(curl "$cin_url" -s)
 imgref=$(echo "$cincinnati_graph" | jq --arg VERSION "$version" -r '.nodes[]  | select(.version==$VERSION) | .payload')
 if [ "$imgref" == "" ]; then
     echo "ERROR: Could not find the current deployment in the update graph from cincinnati."
-    echo "   If it is proxied, you can override the cincinnati url with an environment varable."
+    echo "   If it is proxied, you can override the cincinnati url with an environment variable."
     echo "   e.g. CINCINNATI_URL=https://custom-cincinnati-address.com/v1/graph /usr/libexec/coreos-oci-rebase"
-    exit 0
+    echo "   Or add a drop-in to zincati with 'Environment=...'"
+    exit 1
 fi
 
 echo "Found OCI image $imgref in the update graph that matches the local deployment version."
@@ -69,3 +89,6 @@ cat > /run/zincati/booted-status-override.json << EOF
 EOF
 
 echo "Zincati will rebase to an OCI image for the next update."
+
+# All went well, delete the failed stamp
+rm $OCI_MIGRATION_FAILED
diff --git a/overlay.d/README.md b/overlay.d/README.md
index df67a9ac92..6fbbac6e5c 100644
--- a/overlay.d/README.md
+++ b/overlay.d/README.md
@@ -70,3 +70,16 @@ pre-existing LVM devices attached. See the tracker issue [1] for more
 information.
 
 [1] https://github.com/coreos/fedora-coreos-tracker/issues/1517
+
+35oci-migration
+-------------------
+
+Trigger a migration script to instruct Zincati to rebase the system
+to using OCI images from quay.io/fedora/fedora-coreos for future
+updates. [1]
+This is composed of a migration script [2] and a motd generator to
+instruct the user if something go wrong or show a reminder if the
+user opted out.
+
+[1] https://github.com/coreos/fedora-coreos-tracker/issues/1890
+[2] https://github.com/coreos/fedora-coreos-config/pull/3355


Reviews:
user: login: jbtrystram
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T17:28:19Z
user: login: dustymabe
body: Can we move this motd and the migration script into its own overlay  (i.e. not `15fcos`) that only gets included for F42? 

I just went through removing some old cruft from our packages lists and pulling it out of a generic overlay is much harder than just deleting a directory. 
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T18:36:17Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T18:46:13Z
user: login: jbtrystram
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T07:50:55Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T19:10:13Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-13T14:49:06Z
user: login: jbtrystram
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-13T15:51:13Z
user: login: jbtrystram
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-14T12:40:00Z
user: login: dustymabe
body: I think the code review LGTM. I'll try to run some tests.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-14T15:10:43Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T20:59:47Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T13:26:46Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T13:27:40Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T13:28:07Z
user: login: dustymabe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T16:27:36Z
user: login: dustymabe
body: LGTM - a few final comments.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T16:28:02Z
user: login: jbtrystram
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T20:44:17Z
user: login: dustymabe
body: LGTM
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-21T22:11:57Z
user: login: jlebon
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-26T14:48:23Z

Review comments:
diff_hunk: @@ -0,0 +1,14 @@
+# This service is used for printing a message if
+# the oci migration failed or was opted out.
+[Unit]
+Description=Check for oci migration stamps
+Before=systemd-user-sessions.service
+# we want to find out if the migration failed.
+# After=zincati.service
+# Requires=zincati.service
path: overlay.d/15fcos/usr/lib/systemd/system/coreos-oci-migration-motd.service
user: login: jbtrystram
body: So Zincati is ordered after `multi-user.target` which is dependant on `systemd-user-sessions.service`. 
Thus, this creates an ordering loop. Not sure how to deal with this !
created_at: 2025-04-16T17:28:19Z
author_association: MEMBER
diff_hunk: @@ -8,6 +8,15 @@
 # see https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates
 # and https://github.com/coreos/fedora-coreos-tracker/issues/1823
 
+
+OCI_MIGRATION_OPT_OUT=/var/lib/coreos-oci-migration-opt-out.stamp
+OCI_MIGRATION_FAILED=/run/zincati/coreos-oci-migration-failed.stamp
+
+if [ -e $OCI_MIGRATION_OPT_OUT ]; then
+    echo "The stamp file to opt out of migration exists, doing nothing."
path: overlay.d/15fcos/usr/libexec/coreos-oci-rebase
user: login: dustymabe
body: I would add an echo with the filepath here too. 
created_at: 2025-04-16T18:17:06Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,2 @@
+[Service]
+ExecStartPre=/usr/libexec/coreos-oci-rebase
path: overlay.d/15fcos/usr/lib/systemd/system/zincati.service.d/010-oci-migration.conf
user: login: dustymabe
body: ```suggestion
ExecStartPre=-/usr/libexec/coreos-oci-rebase
```

I wonder if we should not care about the exit code of the rebase script and then we can change the `exit 0` in the script where we are actually having a failure into `exit 1` now and `systemctl status zincati` will indicate some problem as well, although it won't be fatal. 
created_at: 2025-04-16T18:24:48Z
author_association: MEMBER
diff_hunk: @@ -27,6 +36,7 @@ origin=$(jq -r '.deployments[0].origin' <<< "$status" | cut -d ':' -f 1)
 origin_url=$(ostree remote show-url "$origin")
 if [ "$origin_url" != "https://ostree.fedoraproject.org" ]; then
     echo "ERROR: The OSTree origin is not matching the default Fedora CoreOS."
+    touch $OCI_MIGRATION_FAILED
path: overlay.d/15fcos/usr/libexec/coreos-oci-rebase
user: login: dustymabe
body: I'd almost rather set `-euo pipefail` this script and create the file on line 21 of this script and remove it on actual success (at the bottom).
created_at: 2025-04-16T18:34:41Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,14 @@
+# This service is used for printing a message if
+# the oci migration failed or was opted out.
+[Unit]
+Description=Check for oci migration stamps
+Before=systemd-user-sessions.service
+# we want to find out if the migration failed.
+# After=zincati.service
+# Requires=zincati.service
path: overlay.d/15fcos/usr/lib/systemd/system/coreos-oci-migration-motd.service
user: login: dustymabe
body: one way would be to make this not dependent on zincati at all :)

i.e. it inspects rpm-ostree and tells if we are booted from container or not.

this would mean we'd have to change the strategy a bit (and the motd message). i.e we could configure this service to ONLY run one boot AFTER we know the migration script ran somehow. So at that point the system should be booted from container and it's fair to MOTD the user. 
created_at: 2025-04-16T18:46:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,14 @@
+# This service is used for printing a message if
+# the oci migration failed or was opted out.
+[Unit]
+Description=Check for oci migration stamps
+Before=systemd-user-sessions.service
+# we want to find out if the migration failed.
+# After=zincati.service
+# Requires=zincati.service
path: overlay.d/15fcos/usr/lib/systemd/system/coreos-oci-migration-motd.service
user: login: jbtrystram
body:  > we could configure this service to ONLY run one boot AFTER we know the migration script ran somehow

this is complex to do as the migration script run with the `zincati` user, unprivileged, so we can't write a stamp that will survive a reboot.
 
 I am going to investigate this more today
 
 EDIT: obviously [systemd have a thing](https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#RuntimeDirectory=) :)
created_at: 2025-04-17T07:50:54Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,79 @@
+#!/usr/bin/bash
+# This script checks if oci migration failed
+# or was opted-out and prints the appropriate message
+# to the console.
+#
+# See https://github.com/coreos/fedora-coreos-tracker/issues/1890
+# See migration script discussion in https://github.com/coreos/fedora-coreos-config/pull/3355
+
+OCI_MIGRATION_OPT_OUT=/var/lib/coreos-oci-migration/opt-out.stamp
+OCI_MIGRATION_FAILED=/var/lib/coreos-oci-migration/failed.stamp
+
+ZINCATI_ENABLED=$(systemctl is-enabled zincati.service)
+
+MOTD_PATH=/run/motd.d/30_oci_migration.motd
+# Start by deleting any previous motd we created,
+# e.g. if the migration failed then the user opted out, or fixed it.
+rm -f $MOTD_PATH
+
+opt_out_message="This system will not migrate to OCI images
+for updates as the opt-out stamp file exists.
+
+This system will keep updating using the legacy ostree repository,
+until it is decommissioned. Currently this is planned for the release
+of Fedora 43.
+
+Once ready, the migration can be resumed by deleting this file:
+${OCI_MIGRATION_OPT_OUT}
+Then restarting zincati.service"
+
+failed_message="The migration to OCI images for updates failed.
+Check the logs of zincati.service for more details.
+
+This system will keep updating using the legacy ostree repository,
+until it is decommissioned. Currently this is planned for the release
+of Fedora 43.
+
+Since ready, the migration can be retried by deleting this file:
+${OCI_MIGRATION_FAILED}
+Then restarting zincati.service"
+
+zincati_disabled_message="The zincati service is disabled on this
+system.
+
+The system can still be updated using 'rpm-ostree update' but this
+will no longer work after the legacy OSTree repository is decommissioned.
+Currently this is planned for the release
+of Fedora 43.
+
+You can migrate to OCI images for upates by running the following:
+rpm-ostree rebase ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:\$steam
+Where \$stream matches the Fedora CoreOS stream you want to run.
+"
path: overlay.d/35oci-migration/usr/libexec/coreos-oci-migration-motd
user: login: dustymabe
body: I think this complexity is just another example of why I'd prefer to have this script just inspect the booted deployment and print a generic message based on that.

```
status=$(rpm-ostree status --json --booted)
booted_imgref=$(jq -r '.deployments[0]."container-image-reference"' <<< "$status")
if [ "$booted_imgref" != "null" ]; then...                                        
```
created_at: 2025-04-17T19:10:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,3 @@
+[Service]
+StateDirectory=coreos-oci-migration
path: overlay.d/35oci-migration/usr/lib/systemd/system/zincati.service.d/010-oci-migration.conf
user: login: dustymabe
body: I worry just a little bit about this somehow affecting zincati daemon when it runs.

This will set some env vars:

```
 StateDirectory=         │ /var/lib/                   │ $XDG_STATE_HOME           │ $STATE_DIRECTORY  
```

Are we sure this won't change any behavior of Zincati when it runs?

If we are worried about it we could just move to a model where we don't write the files under a directory but just directly in `/var/lib/`.
created_at: 2025-05-13T13:31:18Z
author_association: MEMBER
diff_hunk: @@ -33,6 +33,10 @@ conditional-include:
       # All Fedora CoreOS streams share the same pool for locked files.
       lockfile-repos:
         - fedora-coreos-pool
+  - if: ${stream} == "next"
path: manifests/fedora-coreos.yaml
user: login: dustymabe
body: ```suggestion
  - if: stream == "next"
```

I don't think the `${}` is needed, but please confirm
created_at: 2025-05-13T13:55:21Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,12 @@
+# This service is used for printing a message if
+# the oci migration failed or was opted out.
+[Unit]
+Description=Check for oci migration stamps
path: overlay.d/35oci-migration/usr/lib/systemd/system/coreos-oci-migration-motd.service
user: login: dustymabe
body: ```suggestion
Description=Warn based on oci migration status
```
created_at: 2025-05-13T14:33:12Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,79 @@
+#!/usr/bin/bash
+# This script checks if oci migration failed
+# or was opted-out and prints the appropriate message
+# to the console.
+#
+# See https://github.com/coreos/fedora-coreos-tracker/issues/1890
+# See migration script discussion in https://github.com/coreos/fedora-coreos-config/pull/3355
+
+OCI_MIGRATION_OPT_OUT=/var/lib/coreos-oci-migration/opt-out.stamp
+OCI_MIGRATION_FAILED=/var/lib/coreos-oci-migration/failed.stamp
+
+ZINCATI_ENABLED=$(systemctl is-enabled zincati.service)
+
+MOTD_PATH=/run/motd.d/30_oci_migration.motd
+# Start by deleting any previous motd we created,
+# e.g. if the migration failed then the user opted out, or fixed it.
+rm -f $MOTD_PATH
+
+opt_out_message="This system will not migrate to OCI images
+for updates as the opt-out stamp file exists.
+
+This system will keep updating using the legacy ostree repository,
+until it is decommissioned. Currently this is planned for the release
+of Fedora 43.
+
+Once ready, the migration can be resumed by deleting this file:
+${OCI_MIGRATION_OPT_OUT}
+Then restarting zincati.service"
+
+failed_message="The migration to OCI images for updates failed.
+Check the logs of zincati.service for more details.
+
+This system will keep updating using the legacy ostree repository,
+until it is decommissioned. Currently this is planned for the release
+of Fedora 43.
+
+Since ready, the migration can be retried by deleting this file:
+${OCI_MIGRATION_FAILED}
+Then restarting zincati.service"
+
+zincati_disabled_message="The zincati service is disabled on this
+system.
+
+The system can still be updated using 'rpm-ostree update' but this
+will no longer work after the legacy OSTree repository is decommissioned.
+Currently this is planned for the release
+of Fedora 43.
path: overlay.d/35oci-migration/usr/libexec/coreos-oci-migration-motd
user: login: dustymabe
body: We changed the strategy slightly such that `rpm-ostree upgrade` will continue to work even after the OCI migration. So we can drop this part.

See new wording suggestion in other comment. 
created_at: 2025-05-13T14:45:10Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,3 @@
+[Service]
+StateDirectory=coreos-oci-migration
path: overlay.d/35oci-migration/usr/lib/systemd/system/zincati.service.d/010-oci-migration.conf
user: login: jbtrystram
body: I haven't had any issues in my testing, Zincati updated fine
created_at: 2025-05-13T15:51:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,79 @@
+#!/usr/bin/bash
+# This script checks if oci migration failed
+# or was opted-out and prints the appropriate message
+# to the console.
+#
+# See https://github.com/coreos/fedora-coreos-tracker/issues/1890
+# See migration script discussion in https://github.com/coreos/fedora-coreos-config/pull/3355
+
+OCI_MIGRATION_OPT_OUT=/var/lib/coreos-oci-migration/opt-out.stamp
+OCI_MIGRATION_FAILED=/var/lib/coreos-oci-migration/failed.stamp
+
+ZINCATI_ENABLED=$(systemctl is-enabled zincati.service)
+
+MOTD_PATH=/run/motd.d/30_oci_migration.motd
+# Start by deleting any previous motd we created,
+# e.g. if the migration failed then the user opted out, or fixed it.
+rm -f $MOTD_PATH
+
+opt_out_message="This system will not migrate to OCI images
+for updates as the opt-out stamp file exists.
+
+This system will keep updating using the legacy ostree repository,
+until it is decommissioned. Currently this is planned for the release
+of Fedora 43.
+
+Once ready, the migration can be resumed by deleting this file:
+${OCI_MIGRATION_OPT_OUT}
+Then restarting zincati.service"
+
+failed_message="The migration to OCI images for updates failed.
+Check the logs of zincati.service for more details.
+
+This system will keep updating using the legacy ostree repository,
+until it is decommissioned. Currently this is planned for the release
+of Fedora 43.
+
+Since ready, the migration can be retried by deleting this file:
+${OCI_MIGRATION_FAILED}
+Then restarting zincati.service"
+
+zincati_disabled_message="The zincati service is disabled on this
+system.
+
+The system can still be updated using 'rpm-ostree update' but this
+will no longer work after the legacy OSTree repository is decommissioned.
+Currently this is planned for the release
+of Fedora 43.
path: overlay.d/35oci-migration/usr/libexec/coreos-oci-migration-motd
user: login: jbtrystram
body: I think you are misunderstanding this : this is shown when Zincati is disabled, so there is no oci migration happening here.
This says that if you want to disable zincati you can still manually update, as always, but as the ostree repo won't be populated after f43, then no more updates after this point.
created_at: 2025-05-14T12:40:00Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,77 @@
+#!/usr/bin/bash
+# This script checks if oci migration failed
+# or was opted-out and prints the appropriate message
+# to the console.
+#
+# See https://github.com/coreos/fedora-coreos-tracker/issues/1890
+# See migration script discussion in https://github.com/coreos/fedora-coreos-config/pull/3355
+
+OCI_MIGRATION_OPT_OUT=/var/lib/coreos-oci-migration/opt-out.stamp
+OCI_MIGRATION_FAILED=/var/lib/coreos-oci-migration/failed.stamp
+
+ZINCATI_ENABLED=$(systemctl is-enabled zincati.service)
+
+MOTD_PATH=/run/motd.d/30_oci_migration.motd
+# Start by deleting any previous motd we created,
+# e.g. if the migration failed then the user opted out, or fixed it.
+rm -f $MOTD_PATH
+
+opt_out_message="
+This system has been opted out of the migration to OCI images for
+updates as the opt-out stamp file exists.
+
+This system will keep updating using the legacy OSTree repository,
+but later this year new Fedora CoreOS updates will cease to be pushed
+to the OSTree repository.
+
+When ready, the migration can be resumed by deleting 
+${OCI_MIGRATION_OPT_OUT}, then restarting zincati.service"
+
+failed_message="
+The migration to OCI images for updates failed. Check the logs of
+zincati.service for more details.
+
+This system will keep updating using the legacy OSTree repository,
+but later this year new Fedora CoreOS updates will cease to be pushed
+to the OSTree repository.
+
+When ready, the migration can be retried by deleting 
+${OCI_MIGRATION_FAILED}, then restarting zincati.service"
+
+zincati_disabled_message="
+The zincati service is disabled on this system.
+
+If not done so already please migrate the system to the OCI backend
+for updates by running:
+
+stream=stable # or testing or next
+target=ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:\$stream
+sudo rpm-ostree rebase \$target
+
+Where \$stream matches the Fedora CoreOS stream the system is following."
+write_motd () {
path: overlay.d/35oci-migration/usr/libexec/coreos-oci-migration-motd
user: login: dustymabe
body: add blank line before this function. 
created_at: 2025-05-14T15:10:15Z
author_association: MEMBER
diff_hunk: @@ -8,6 +8,20 @@
 # see https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates
 # and https://github.com/coreos/fedora-coreos-tracker/issues/1823
 
+
+OCI_MIGRATION_OPT_OUT=/var/lib/coreos-oci-migration/opt-out.stamp
+OCI_MIGRATION_FAILED=/var/lib/coreos-oci-migration/failed.stamp
+
+if [ -e $OCI_MIGRATION_OPT_OUT ]; then
+    echo "${OCI_MIGRATION_OPT_OUT} to opt out of migration exists, doing nothing."
+    exit 0
+fi
+
+# create the failed stamp file as a precaution
+touch $OCI_MIGRATION_FAILED
path: overlay.d/35oci-migration/usr/libexec/coreos-oci-rebase
user: login: dustymabe
body: need to `rm $OCI_MIGRATION_FAILED` as part of the code below:

```
echo "The booted deployment is already an OCI container."
exit 0
```
created_at: 2025-05-20T20:35:50Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,78 @@
+#!/usr/bin/bash
+# This script checks if oci migration failed
+# or was opted-out and prints the appropriate message
+# to the console.
+#
+# See https://github.com/coreos/fedora-coreos-tracker/issues/1890
+# See migration script discussion in https://github.com/coreos/fedora-coreos-config/pull/3355
+
+OCI_MIGRATION_OPT_OUT=/var/lib/coreos-oci-migration/opt-out.stamp
+OCI_MIGRATION_FAILED=/var/lib/coreos-oci-migration/failed.stamp
+
+ZINCATI_ENABLED=$(systemctl is-enabled zincati.service)
+
+MOTD_PATH=/run/motd.d/30_oci_migration.motd
+# Start by deleting any previous motd we created,
+# e.g. if the migration failed then the user opted out, or fixed it.
+rm -f $MOTD_PATH
+
+opt_out_message="
+This system has been opted out of the migration to OCI images for
+updates as the opt-out stamp file exists.
+
+This system will keep updating using the legacy OSTree repository,
+but later this year new Fedora CoreOS updates will cease to be pushed
+to the OSTree repository.
+
+When ready, the migration can be resumed by deleting 
+${OCI_MIGRATION_OPT_OUT}, then restarting zincati.service"
+
+failed_message="
+The migration to OCI images for updates failed. Check the logs of
+zincati.service for more details.
+
+This system will keep updating using the legacy OSTree repository,
+but later this year new Fedora CoreOS updates will cease to be pushed
+to the OSTree repository.
+
+When ready, the migration can be retried by deleting 
+${OCI_MIGRATION_FAILED}, then restarting zincati.service"
+
+zincati_disabled_message="
+The zincati service is disabled on this system.
+
+If not done so already please migrate the system to the OCI backend
+for updates by running:
+
+stream=stable # or testing or next
+target=ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:\$stream
+sudo rpm-ostree rebase \$target
path: overlay.d/35oci-migration/usr/libexec/coreos-oci-migration-motd
user: login: dustymabe
body: just for completeness so we don't separately tell them they need to reboot:

```suggestion
sudo rpm-ostree rebase --reboot \$target
```
created_at: 2025-05-20T20:41:42Z
author_association: MEMBER
diff_hunk: @@ -37,15 +51,19 @@ checksum=$(jq -r '.deployments[0].checksum' <<< "$status")
 arch=$(arch)
 cin_url="$CINCINNATI_URL?basearch=$arch&stream=$stream&oci=true"
 
+# Wait for the network to be online
+nm-online -q
path: overlay.d/35oci-migration/usr/libexec/coreos-oci-rebase
user: login: dustymabe
body: I would specify a timeout here `--timeout 120` or something. Some machines can take some time to come online.
created_at: 2025-05-21T13:26:45Z
author_association: MEMBER
diff_hunk: @@ -8,6 +8,20 @@
 # see https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates
 # and https://github.com/coreos/fedora-coreos-tracker/issues/1823
 
+
+OCI_MIGRATION_OPT_OUT=/var/lib/coreos-oci-migration/opt-out.stamp
+OCI_MIGRATION_FAILED=/var/lib/coreos-oci-migration/failed.stamp
+
+if [ -e $OCI_MIGRATION_OPT_OUT ]; then
+    echo "${OCI_MIGRATION_OPT_OUT} to opt out of migration exists, doing nothing."
+    exit 0
+fi
+
+# create the failed stamp file as a precaution
+touch $OCI_MIGRATION_FAILED
path: overlay.d/35oci-migration/usr/libexec/coreos-oci-rebase
user: login: dustymabe
body: still need to address this
created_at: 2025-05-21T13:27:40Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,78 @@
+#!/usr/bin/bash
+# This script checks if oci migration failed
+# or was opted-out and prints the appropriate message
+# to the console.
+#
+# See https://github.com/coreos/fedora-coreos-tracker/issues/1890
+# See migration script discussion in https://github.com/coreos/fedora-coreos-config/pull/3355
+
+OCI_MIGRATION_OPT_OUT=/var/lib/coreos-oci-migration/opt-out.stamp
+OCI_MIGRATION_FAILED=/var/lib/coreos-oci-migration/failed.stamp
+
+ZINCATI_ENABLED=$(systemctl is-enabled zincati.service)
+
+MOTD_PATH=/run/motd.d/30_oci_migration.motd
+# Start by deleting any previous motd we created,
+# e.g. if the migration failed then the user opted out, or fixed it.
+rm -f $MOTD_PATH
+
+opt_out_message="
+This system has been opted out of the migration to OCI images for
+updates as the opt-out stamp file exists.
+
+This system will keep updating using the legacy OSTree repository,
+but later this year new Fedora CoreOS updates will cease to be pushed
+to the OSTree repository.
+
+When ready, the migration can be resumed by deleting 
+${OCI_MIGRATION_OPT_OUT}, then restarting zincati.service"
+
+failed_message="
+The migration to OCI images for updates failed. Check the logs of
+zincati.service for more details.
+
+This system will keep updating using the legacy OSTree repository,
+but later this year new Fedora CoreOS updates will cease to be pushed
+to the OSTree repository.
+
+When ready, the migration can be retried by deleting 
+${OCI_MIGRATION_FAILED}, then restarting zincati.service"
+
+zincati_disabled_message="
+The zincati service is disabled on this system.
+
+If not done so already please migrate the system to the OCI backend
+for updates by running:
+
+stream=stable # or testing or next
+target=ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:\$stream
+sudo rpm-ostree rebase \$target
path: overlay.d/35oci-migration/usr/libexec/coreos-oci-migration-motd
user: login: dustymabe
body: still need to address this (if you agree with the change).
created_at: 2025-05-21T13:28:06Z
author_association: MEMBER
diff_hunk: @@ -33,6 +33,10 @@ conditional-include:
       # All Fedora CoreOS streams share the same pool for locked files.
       lockfile-repos:
         - fedora-coreos-pool
+  - if: stream == "next"
+    include:
+      ostree-layers:
+        - overlay/35oci-migration
path: manifests/fedora-coreos.yaml
user: login: dustymabe
body: I don't know how we'll actually end up implementing this, but just in case we re-enable the `devel` stream, let's account for that and also we should probably just enable it on `rawhide` too just to see if any issues during build pop up:

```suggestion
  - if: stream == "next"
    include:
      ostree-layers:
        - overlay/35oci-migration
  - if: stream == "next-devel"
    include:
      ostree-layers:
        - overlay/35oci-migration
  - if: stream == "rawhide"
    include:
      ostree-layers:
        - overlay/35oci-migration
```
created_at: 2025-05-21T16:27:36Z
author_association: MEMBER
diff_hunk: @@ -33,6 +33,10 @@ conditional-include:
       # All Fedora CoreOS streams share the same pool for locked files.
       lockfile-repos:
         - fedora-coreos-pool
+  - if: stream == "next"
+    include:
+      ostree-layers:
+        - overlay/35oci-migration
path: manifests/fedora-coreos.yaml
user: login: jbtrystram
body: we don't have cincinnati updates for devel streams neither rawhide, and Zincati is disabled there, but I see how that would make the changes go through the pipeline yeah
created_at: 2025-05-21T20:44:17Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,12 @@
+# This service is used for printing a message if
+# the oci migration failed or was opted out.
+[Unit]
+Description=Warn based on oci migration status
path: overlay.d/35oci-migration/usr/lib/systemd/system/coreos-oci-migration-motd.service
user: login: jlebon
body: Minor: the convention is Title Case. Also s/oci/OCI/.
created_at: 2025-05-26T14:41:03Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,78 @@
+#!/usr/bin/bash
+# This script checks if oci migration failed
+# or was opted-out and prints the appropriate message
+# to the console.
+#
+# See https://github.com/coreos/fedora-coreos-tracker/issues/1890
+# See migration script discussion in https://github.com/coreos/fedora-coreos-config/pull/3355
+
+OCI_MIGRATION_OPT_OUT=/var/lib/coreos-oci-migration/opt-out.stamp
+OCI_MIGRATION_FAILED=/var/lib/coreos-oci-migration/failed.stamp
+
+ZINCATI_ENABLED=$(systemctl is-enabled zincati.service)
+
+MOTD_PATH=/run/motd.d/30_oci_migration.motd
+# Start by deleting any previous motd we created,
+# e.g. if the migration failed then the user opted out, or fixed it.
+rm -f $MOTD_PATH
+
+opt_out_message="
+This system has been opted out of the migration to OCI images for
+updates as the opt-out stamp file exists.
+
+This system will keep updating using the legacy OSTree repository,
+but later this year new Fedora CoreOS updates will cease to be pushed
+to the OSTree repository.
+
+When ready, the migration can be resumed by deleting
+${OCI_MIGRATION_OPT_OUT}, then restarting zincati.service"
+
+failed_message="
+The migration to OCI images for updates failed. Check the logs of
+zincati.service for more details.
+
+This system will keep updating using the legacy OSTree repository,
+but later this year new Fedora CoreOS updates will cease to be pushed
+to the OSTree repository.
+
+When ready, the migration can be retried by deleting
+${OCI_MIGRATION_FAILED}, then restarting zincati.service"
+
+zincati_disabled_message="
+The zincati service is disabled on this system.
+
+If not done so already please migrate the system to the OCI backend
+for updates by running:
path: overlay.d/35oci-migration/usr/libexec/coreos-oci-migration-motd
user: login: jlebon
body: Probably should mention that they should be fully up to date here so that there's no barrier concerns.
created_at: 2025-05-26T14:44:52Z
author_association: MEMBER
diff_hunk: @@ -37,15 +52,20 @@ checksum=$(jq -r '.deployments[0].checksum' <<< "$status")
 arch=$(arch)
 cin_url="$CINCINNATI_URL?basearch=$arch&stream=$stream&oci=true"
 
+# Wait for the network to be online
+echo "Waiting for the network to be online."
+nm-online --timeout 120
path: overlay.d/35oci-migration/usr/libexec/coreos-oci-rebase
user: login: jlebon
body: Hmm, wouldn't it be cleaner to use curl's `--retry` infrastructure instead?
created_at: 2025-05-26T14:46:37Z
author_association: MEMBER

PR comments:
user: login: dustymabe
created_at: 2025-04-15T16:24:23Z
body: ok. a few things here.

- we're going to need to roll this out to `next` first so maybe we just add this file in a conditional include (we can nest them now and it doesn't need to be a second file, see [this](https://github.com/openshift/os/blob/41f6a028d37b750db0bf4257447d809bd9cbe4bf/packages-openshift.yaml#L7-L14) for an example) based on the stream name.
- Maybe we should make it even easier for people to opt out by just touching a file. That means instead of telling them to drop down an override they touch a file instead. This means we'd modify the migration script to check and no-op based on the existence of a file.
- Should we consider also adding a MOTD helper to let people know if there system hasn't been migrated (for whatever reason). i.e. maybe the migration script failed OR they decided to opt out. If they decided to opt out I still think it would be useful to warn them in an MOTD and they can silence that too if they'd like. Perhaps the MOTD thing can be a separate thing that we roll out later?

author_association: MEMBER
user: login: dustymabe
created_at: 2025-04-17T19:12:22Z
body: IMO things here become way simpler if we do https://github.com/coreos/fedora-coreos-config/pull/3458#discussion_r2049507203 and just roll out the motd in the next release after we roll out the automated migration (i.e. separate PR)
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-05-07T13:19:48Z
body: > IMO things here become way simpler if we do [#3458 (comment)](https://github.com/coreos/fedora-coreos-config/pull/3458#discussion_r2049507203) and just roll out the motd in the next release after we roll out the automated migration (i.e. separate PR)

the issue I see going that route is : if a node get the ExecStartPre but zincati can't pull the OCI at the next update for some reason, it won't be able to get the next update, so won't ever get the motd to display
so except if the user go and look at the rpm-ostree status output it will silently get stuck

I would prefer to ship the motd in the same rollout. We can make it simpler if you think the complexity is not worth it, and we could move the `.path` systemd files to a simpler timer to show the motd without a reboot
author_association: MEMBER
user: login: dustymabe
created_at: 2025-05-07T14:59:36Z
body: > the issue I see going that route is : if a node get the ExecStartPre but zincati can't pull the OCI at the next update for some reason, it won't be able to get the next update, so won't ever get the motd to display so except if the user go and look at the rpm-ostree status output it will silently get stuck

TBH I've been thinking for some time that we need a motd that warns of the most recent deployment being stale (i.e. older than 30 days or something). I think if we had, it wouldn't give you what you want here, but it would be some sort of backstop that would let the user know something was going wrong with the update mechanism.

> 
> I would prefer to ship the motd in the same rollout. We can make it simpler if you think the complexity is not worth it, and we could move the `.path` systemd files to a simpler timer to show the motd without a reboot

Interested in thoughts from @jlebon @travier here. 

author_association: MEMBER
user: login: jlebon
created_at: 2025-05-08T16:33:36Z
body: I agree with @jbtrystram that the likelihood of failure is highest during migration and we want to cover the case where the system can't upgrade. But I also agree with @dustymabe that this seems quite complex.

Hmm, I'm confused why we need the motd stuff to live completely separately. Couldn't it be another `ExecStartPre=` that just checks if the migration was successful (e.g. check for `/run` stamp file written by the migration script) and writes out the appropriate motd? No additional path/timer/systemd units and presets.

Ahh, is it to handle the "disabled Zincati" case? People doing `rpm-ostree upgrade` directly are already diverging by not following the graph. But yeah, we could have a separate MOTD at some point before we stop OSTree releases that checks for this and emits a message.
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-05-09T07:12:32Z
body: >Couldn't it be another `ExecStartPre=` that just checks if the migration was successful (e.g. check for `/run` stamp file written by the migration script) and writes out the appropriate motd? No additional path/timer/systemd units and presets.

Then you'd only get the MOTD at next login I think.
As currently you can't have Zincati run before the thing writing the motd : https://github.com/coreos/fedora-coreos-config/pull/3458#discussion_r2047404019

That's why I ended up with the path unit , but I think it's a byproduct of `cosa run` getting a log in very early, in real life you'd get logged in after Zincati started i think.

author_association: MEMBER
user: login: jlebon
created_at: 2025-05-09T15:56:54Z
body: > That's why I ended up with the path unit , but I think it's a byproduct of `cosa run` getting a log in very early, in real life you'd get logged in after Zincati started i think.

Yeah, agree with that. Noticing it at the next login seems OK too.
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-05-13T09:56:45Z
body: Ok I Just tested this some more and after all the whole `.path` thing is really uneeded as the motd is refreshed at each login anyway.

> Couldn't it be another ExecStartPre= that just checks if the migration was successful (e.g. check for /run stamp file written by the migration script) and writes out the appropriate motd? No additional path/timer/systemd units and presets.

Then you can't silence it without having to do another zincati dropin 
author_association: MEMBER
user: login: dustymabe
created_at: 2025-05-13T14:50:29Z
body: Some wording suggestions for the motd messages:

```
opt_out_message="
This system has been opted out of the migration to OCI images for
updates as the opt-out stamp file exists.

This system will keep updating using the legacy OSTree repository,
but later this year new Fedora CoreOS updates will cease to be pushed
to the OSTree repository.

When ready, the migration can be resumed by deleting 
${OCI_MIGRATION_OPT_OUT}, then restarting zincati.service"

failed_message="
The migration to OCI images for updates failed. Check the logs of
zincati.service for more details.

This system will keep updating using the legacy OSTree repository,
but later this year new Fedora CoreOS updates will cease to be pushed
to the OSTree repository.

When ready, the migration can be retried by deleting 
${OCI_MIGRATION_FAILED}, then restarting zincati.service"

zincati_disabled_message="
The zincati service is disabled on this system.

If not done so already please migrate the system to the OCI backend
for updates by running:

stream=stable # or testing or next
target=ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:\$stream
sudo rpm-ostree rebase \$target

Where \$stream matches the Fedora CoreOS stream the system is following."
```
author_association: MEMBER
user: login: dustymabe
created_at: 2025-05-13T14:51:17Z
body: I can try to test this later this week if you like.
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-05-14T12:43:55Z
body: @dustymabe I applied the suggestions . Thanks 
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-05-14T17:07:32Z
body: I made a small ign config and instructions to make testing this easier : https://gist.github.com/jbtrystram/5f0f94e2ea47f46e16f0e7cafc4cc788

~it's just testing the MOTD though. I tested the zincati migration with the status override well enough in the other PR I t
think~
Updated this to also test the migration
author_association: MEMBER
user: login: dustymabe
created_at: 2025-05-20T21:05:07Z
body: Note that often `zincati.service` comes up before networking is fully up, which is fine for zincati because it will just try again in 5-10m, but it's pretty fatal for our migration script:

```
[core@localhost ~]$ sudo journalctl -u zincati -b0 | cat
May 20 20:53:22 localhost.localdomain systemd[1]: Starting zincati.service - Zincati Update Agent...
May 20 20:53:22 localhost.localdomain coreos-oci-rebase[2225]: Saving rpm-ostree status.
May 20 20:53:22 localhost.localdomain coreos-oci-rebase[2225]: Fetching cincinnati update graph for stream stable on x86_64
May 20 20:53:22 localhost.localdomain zincati[2248]: [INFO  zincati::cli::agent] starting update agent (zincati 0.0.30)
May 20 20:53:22 localhost.localdomain zincati[2248]: [INFO  zincati::cincinnati] Cincinnati service: https://updates.coreos.fedoraproject.org
May 20 20:53:22 localhost.localdomain zincati[2248]: [INFO  zincati::cli::agent] agent running on node 'a87dfdee0e7046ac89ad3f063d501dcf', in update group 'default'
May 20 20:53:22 localhost.localdomain zincati[2248]: [INFO  zincati::update_agent::actor] registering as the update driver for rpm-ostree
May 20 20:53:22 localhost.localdomain zincati[2248]: [INFO  zincati::update_agent::actor] initialization complete, auto-updates logic enabled
May 20 20:53:22 localhost.localdomain zincati[2248]: [INFO  zincati::strategy] update strategy: immediate
May 20 20:53:22 localhost.localdomain systemd[1]: Started zincati.service - Zincati Update Agent.
May 20 20:53:22 localhost.localdomain zincati[2248]: [INFO  zincati::update_agent::actor] reached steady state, periodically polling for updates
May 20 20:53:22 localhost.localdomain zincati[2248]: [ERROR zincati::cincinnati] failed to check Cincinnati for updates: client-side error: error sending request for url (https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&platform=qemu&node_uuid=a87dfdee0e7046ac89ad3f063d501dcf&os_version=41.20250331.3.0&group=default&stream=stable&os_checksum=1f21e2c4c522f97eea5d4383e831f8dc8e5a512e9de00d0fd679b689a1d507d3): error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
May 20 20:53:49 localhost.localdomain systemd[1]: Stopping zincati.service - Zincati Update Agent...
May 20 20:53:49 localhost.localdomain systemd[1]: zincati.service: Deactivated successfully.
May 20 20:53:49 localhost.localdomain systemd[1]: Stopped zincati.service - Zincati Update Agent.
```

Notice that because the script is `set -euo pipefail` we don't really get much indicator of what happened but there is enough clue here because zincati itself failed later with `Temporary failure in name resolution`.

We probably need to consider this more. Either remove `set -e` and do more manual error checking OR just enable `set -x` too so we get more in the logs.

We also need to perhaps do some sort of waiting in the script until networking comes up (maybe by executing `nm-online` or something.
author_association: MEMBER
user: login: jbtrystram
created_at: 2025-05-21T09:04:54Z
body: > Note that often `zincati.service` comes up before networking is fully up, which is fine for zincati because it will just try again in 5-10m, but it's pretty fatal for our migration script:

nice catch! 

> We also need to perhaps do some sort of waiting in the script until networking comes up (maybe by executing `nm-online` or something.

Why not simply order zincati after `network-online.target` in the drop in ?
(and also add `-x` )

author_association: MEMBER
user: login: dustymabe
created_at: 2025-05-21T12:40:58Z
body: > Why not simply order zincati after `network-online.target` in the drop in?

Right now we have a `ext.config.networking.network-online-service` test that makes sure network-online.target doesn't get pulled in. I'd be OK with temporarily disabling that test while we roll this out. We'd have to get it approved by others too.

Let's just modify the test to detect if the zincati override exists and if it does short circuit the test. 

Note also that we probably need `Wants=` and `After=` to actually pull in the target. 

> (and also add `-x` )

I'm ok with that. It will be ugly, but if you're looking at the logs it's probably because something went wrong and you probably want more info.


author_association: MEMBER
user: login: jbtrystram
created_at: 2025-05-21T13:39:26Z
body: Ok I updated the migration script with a call to `nm-online` to wait for network, after reading [this PR](https://github.com/coreos/fedora-coreos-config/pull/1088) i don't feel confident pulling `network-online.target` as it seems to have undesirable consequences in some edge-case. 
Here `nm-online` will timeout after 120s and fail the migration if no network is available, which is fine I think
 
author_association: MEMBER
user: login: dustymabe
created_at: 2025-05-21T16:24:04Z
body: CI failed with:

```
[+] Found files with whitespace at the end of line

./overlay.d/35oci-migration/usr/libexec/coreos-oci-migration-motd
```
author_association: MEMBER
user: login: dustymabe
created_at: 2025-05-21T16:34:50Z
body: This is how I tested this PR:

```
# 1. make overlay compatible as a systemd sysext

mkdir -p overlay.d/35oci-migration/usr/lib/extension-release.d/
echo "ID='_any'" > overlay.d/35oci-migration/usr/lib/extension-release.d/extension-release.oci-migration


# 2. Define a butane config to pick up those files from the overlay and write them out as sysext. Note had to disable selinux because running things on sysext was getting blocked

$ cat config.bu
variant: fcos
version: 1.6.0
passwd:
  users:
    - name: core
      password_hash: xyz
systemd:
  units:
    - name: systemd-sysext.service
      enabled: true
    - name: zincati.service
      enabled: false
storage:
  trees:
    - local: .
      path: /var/lib/extensions/oci-migration
  files:
    # SELinux->permissive because there are some denials running
    # things from a sysext.
    - path: /etc/selinux/config
      mode: 0644
      overwrite: true
      contents:
        inline: |
          SELINUXTYPE=targeted
          SELINUX=permissive

#3. Convert to Ignition

cat config.bu | butane --pretty --strict --files-dir overlay.d/35oci-migration/ > config.ign


#4. Launch VM using the Ignition config using the last `stable` stream release of F41

cosa buildfetch --force --build=41.20250331.3.0 --artifact=qemu
cosa decompress --build=41.20250331.3.0
cosa run --build=41.20250331.3.0 --ignition=config.ign --ignition-direct --devshell=false --usernet


#5. Once the machine is up log in as `core` user and daemon-reload to pick up sysext changes, etc.

sudo systemctl daemon-reload
sudo systemctl enable coreos-oci-migration-motd.service --now
sudo systemctl enable zincati --now
```


author_association: MEMBER
user: login: travier
created_at: 2025-05-22T08:36:08Z
body: To get the SELinux labels in order, we would need a service unit that runs the following before `systemd-sysext` gets started:

```
cd /var/lib/extensions/oci-migration
setfiles -r . /etc/selinux/targeted/contexts/files/file_contexts .
chcon --user=system_u --recursive .
```
author_association: MEMBER