Chapter 16. Creating and managing OSTree image updates You can easily create and manage OStree image updates for your RHEL for Edge systems and make them immediately available to RHEL for Edge devices. With OSTree, you can use image builder to create RHEL for Edge Commit or RHEL for Edge Container images as .tar files that contain OSTree commits. The OSTree update versioning system works as a "Git repository" that stores and versions the OSTree commits. The rpm-ostree image and package system then assembles the commits on the client device. When you create a new image with RHEL image builder to perform an update, RHEL image builder pulls updates from these repositories. 16.1. Basic concepts for OSTree Basic terms that OSTree and rpm-ostree use during image updates. rpm-ostree The technology on the edge device that handles how the OSTree commits are assembled on the device. It works as a hybrid between an image and a package system. With the rpm-ostree technology, you can make atomic upgrades and rollbacks to your system. OSTree OSTree is a technology that enables you to create commits and download bootable file system trees. You can also use it to deploy the trees and manage the boot loader configuration. Commit An OSTree commit contains a full operating system that is not directly bootable. To boot the system, you must deploy it, for example, with a RHEL Installable image. Reference It is also known as ref . An OSTree ref is similar to a Git branch and it is a name. The following reference names examples are valid: rhel/9/x86_64/edge ref-name app/org.gnome.Calculator/x86_64/stable ref-name-2 By default, image builder specifies rhel/9/USDARCH/edge as a path. The "USDARCH" value is determined by the host machine. Parent The parent argument is an OSTree commit that you can provide to build a new commit with image builder. You can use the parent argument to specify an existing ref that retrieves a parent commit for the new commit that you are building. You must specify the parent commit as a ref value to be resolved and pulled, for example rhel/9/x86_64/edge . You can use the --parent commit for the RHEL for Edge Commit ( .tar ) and RHEL for Edge Container ( .tar ) image types. Remote The http or https endpoint that hosts the OSTree content. This is analogous to the baseurl for a yum repository. Static delta Static deltas are a collection of updates generated between two OSTree commits. This enables the system client to fetch a smaller amount of files, which are larger in size. The static deltas updates are more network efficient because, when updating an ostree-based host, the system client will only fetch the objects from the new OSTree commit which do not exist on the system. Typically, the new OSTree commit contains many small files, which requires multiple TCP connections. Summary The summary file is a concise way of enumerating refs, checksums, and available static deltas in an OSTree repo. You can check the state of all the refs and static deltas available in an Ostree repo. However, you must generate the summary file every time a new ref, commit, or static-delta is added to the OSTree repo. 16.2. Creating OSTree repositories You can create OSTree repos with RHEL image builder by using either RHEL for Edge Commit (.tar) or RHEL for Edge Container (.tar) image types. These image types contain an OSTree repo that contains a single OSTree commit. You can extract the RHEL for Edge Commit (.tar) on a web server and it is ready to be served. You must import the RHEL for Edge Container (.tar) to a local container image storage or push the image to a container registry. After you start the container, it serves the commit over an integrated nginx web server. Use the RHEL for Edge Container (.tar) on a RHEL server with Podman to create an OSTree repo: Prerequisite You created a RHEL for Edge Container (.tar) image. Procedure Download the container image from image builder: Import the container into Podman: Start the container and make it available by using the port 8080 : Verification Check that the container is running: 16.3. Managing a centralized OSTree mirror For production environments, having a central OSTree mirror that serves all the commits has several advantages, including: Deduplicating and minimizing disk storage Optimizing the updates to clients by using static delta updates Pointing to a single OSTree mirror for their deployment life. To manage a centralized OSTree mirror, you must pull each commit from image builder into the centralized repository where it will be available to your users. Note You can also automate managing an OSTree mirror by using the infra.osbuild Ansible collection. See osbuild.infra Ansible . To create a centralized repository you can run the following commands directly on a web server: Procedure Create an empty blueprint, customizing it to use "rhel-92" as the distro: Push the blueprint to the server: Build a RHEL for Edge Commit ( .tar ) image from the blueprint you created: Retrieve the .tar file and decompress it to the disk: The /usr/share/nginx/html/repo location on disk will become the single OSTree repo for all refs and commits. Create another empty blueprint, customizing it to use "rhel-87" as the distro: Push the blueprint and create another RHEL for Edge Commit ( .tar ) image: Retrieve the .tar file and decompress it to the disk: Pull the commit to the local repo. By using ostree pull-local , you can copy the commit data from one local repo to another local repo. Optional: Inspect the status of the OSTree repo. The following is an output example: Update the RHEL 9.2 blueprint to include a new package and build a new commit, for example: Push the updated blueprint and create a new RHEL for Edge Commit ( .tar ) image, pointing the compose to the existing OSTree repo: Retrieve the .tar file and decompress it to the disk: Pull the commit to repo: Optional: Inspect the OSTree repo status again: 16.4. Performing updates by using static deltas You can use static deltas to improve the speed of your client repository updates. If static deltas for an update exist, they are used. If not, the update still happens, but with a lower speed. After building an OSTree system, if you want to retrieve the content from your client systems, you need to create a repository management. By default, the repository you use for production requires one HTTP fetch transaction per client request. If you perform releases only once a week, use "static deltas" to get client repository updates faster. Prerequisites You have a RHEL repo repository running with Podman. You have created a centralized OSTree mirror to use as the production repository. Procedure Pull the content update from the repo repository into the repo-prod repository: Note If you do not specify the refs branch, the ostree-pull-local command retrieves all the branches. As an alternative to the ostree-pull-local command, you can also use the ostree-pull command. The difference is that ostree-pull-local is optimized for copies only between repositories on the same system, while ostree-pull can download data from a remote repository. Apply a delta update against the commit. The static generation happens on the local repository, in this case, prod . After the delta update is generated, the update is done offline. Note You must have at least two commits in the repository where you want to generate static deltas. Optional: If you want to upgrade from the two commits, run the following command: Important Generating a full permutation of deltas across all versions can be a heavy workload for your systems. The OSTree core has some support for static deltas which refer to a parent. Update the summary file: You need to update the summary file, even if you choose to generate static deltas. The summary command cannot run concurrently. It must be triggered serially by other jobs. That is, you must generate a summary file after generating the static-deltas, because the command added a new static delta. Use the generated static delta to apply to the existing Operating System and perform updates: Apply the OSTree delta update: Reboot into the new image to apply the updates:

Commands:
composer-cli compose image _<UUID>
skopeo copy oci-archive:_<UUID>_-container.tar containers-storage:localhost/ostree
podman run -rm -p 8080:8080 ostree
podman ps -a
name = "minimal-rhel92" description = "minimal blueprint for ostree commit" version = "1.0.0" modules = [] groups = [] distro = "rhel-92"
composer-cli blueprints push minimal-rhel92.toml
composer-cli compose start-ostree minimal-rhel92 edge-commit
composer-cli compose image _<rhel-92-uuid> tar -xf <rhel-92-uuid>.tar -C /usr/share/nginx/html/
name = "minimal-rhel87" description = "minimal blueprint for ostree commit" version = "1.0.0" modules = [] groups = [] distro = "rhel-87"
composer-cli blueprints push minimal-rhel87.toml composer-cli compose start-ostree minimal-rhel87 edge-commit
composer-cli compose image <rhel-87-uuid> tar -xf <rhel-87-uuid>.tar
ostree --repo=/usr/share/nginx/html/repo pull-local repo
ostree --repo=/usr/share/nginx/html/repo refs rhel/8/x86_64/edge rhel/9/x86_64/edge ostree --repo=/usr/share/nginx/html/repo show rhel/8/x86_64/edge commit f7d4d95465fbd875f6358141f39d0c573df6a321627bafde68c73850667e5443 ContentChecksum: 41bf2f8b442a770e9bf03e096a46a286f5836e0a0702b7c3516ef4e0acec2dea Date: 2023-09-15 16:17:04 +0000 Version: 8.7 (no subject) ostree --repo=/usr/share/nginx/html/repo show rhel/9/x86_64/edge commit 89290dbfd6f749700c77cbc434c121432defb0c1c367532368eee170d9e53ea9 ContentChecksum: 70235bfb9cae82c53f856183750e809becf0b9b076122b19c40fec92fc6d74c1 Date: 2023-09-15 15:30:24 +0000 Version: 9.2 (no subject)
name = "minimal-rhel92" description = "minimal blueprint for ostree commit" version = "1.1.0" modules = [] groups = [] distro = "rhel-92" [[packages]] name = "strace" version = "*"
composer-cli blueprints push minimal-rhel92.toml composer-cli compose start-ostree minimal-rhel92 edge-commit --url http://localhost/repo --ref rhel/9/x86_64/edge
rm -rf repo composer-cli compose image <rhel-92-uuid> tar -xf <rhel-92-uuid>.tar
ostree --repo=/usr/share/nginx/html/repo pull-local repo
ostree --repo=/usr/share/nginx/html/repo refs rhel/8/x86_64/edge rhel/9/x86_64/edge ostree --repo=/usr/share/nginx/html/repo show rhel/8/x86_64/edge commit f7d4d95465fbd875f6358141f39d0c573df6a321627bafde68c73850667e5443 ContentChecksum: 41bf2f8b442a770e9bf03e096a46a286f5836e0a0702b7c3516ef4e0acec2dea Date: 2023-09-15 16:17:04 +0000 Version: 8.7 (no subject) ostree --repo=/usr/share/nginx/html/repo show rhel/9/x86_64/edge commit a35c3b1a9e731622f32396bb1aa84c73b16bd9b9b423e09d72efaca11b0411c9 Parent: 89290dbfd6f749700c77cbc434c121432defb0c1c367532368eee170d9e53ea9 ContentChecksum: 2335930df6551bf7808e49f8b35c45e3aa2a11a6c84d988623fd3f36df42a1f1 Date: 2023-09-15 18:21:31 +0000 Version: 9.2 (no subject) ostree --repo=/usr/share/nginx/html/repo log rhel/9/x86_64/edge commit a35c3b1a9e731622f32396bb1aa84c73b16bd9b9b423e09d72efaca11b0411c9 Parent: 89290dbfd6f749700c77cbc434c121432defb0c1c367532368eee170d9e53ea9 ContentChecksum: 2335930df6551bf7808e49f8b35c45e3aa2a11a6c84d988623fd3f36df42a1f1 Date: 2023-09-15 18:21:31 +0000 Version: 9.2 (no subject) commit 89290dbfd6f749700c77cbc434c121432defb0c1c367532368eee170d9e53ea9 ContentChecksum: 70235bfb9cae82c53f856183750e809becf0b9b076122b19c40fec92fc6d74c1 Date: 2023-09-15 15:30:24 +0000 Version: 9.2 (no subject) rpm-ostree db diff --repo=/usr/share/nginx/html/repo 89290dbfd6f749700c77cbc434c121432defb0c1c367532368eee170d9e53ea9 a35c3b1a9e731622f32396bb1aa84c73b16bd9b9b423e09d72efaca11b0411c9 ostree diff commit from: 89290dbfd6f749700c77cbc434c121432defb0c1c367532368eee170d9e53ea9 ostree diff commit to: a35c3b1a9e731622f32396bb1aa84c73b16bd9b9b423e09d72efaca11b0411c9 Added: elfutils-default-yama-scope-0.188-3.el9.noarch elfutils-libs-0.188-3.el9.x86_64 strace-5.18-2.el9.x86_64
ostree --repo=/usr/share/nginx/html/repo pull-local repo-prod
ostree --repo=/usr/share/nginx/html/client-server static-delta generate exampleos/x86_64/standard
ostree --repo=repo-prod static-delta generate --from=exampleos/x86_64/standard^^ --to=exampleos/x86_64/standard
ostree --repo=repo-prod summary -u
ostree --repo=/ostree/repo-prod static-delta <delta-update-file-path>
sudo rpm-ostree upgrade
systemctl reboot