Chapter 9. Managing RHEL bootc images After installing and deploying RHEL bootc images, you can perform management operations on your container images, such as changing or updating the systems. The system supports in-place transactional updates with rollback after deployment. This kind of management, also known as Day 2 management baseline, consists of transactionally fetching new operating system updates from a container registry and booting the system into them, while supporting manual, or automated rollbacks in case of failures. You can also rely on automatic updates, that are turned on by default. The systemd service unit and the systemd timer unit files check the container registry for updates and apply them to the system. You can trigger an update process with different events, such as updating an application. There are automation tools watching these updates and then triggering the CI/CD pipelines. A reboot is required, because the updates are transactional. For environments that require more sophisticated or scheduled rollouts, you must disable auto updates and use the bootc utility to update your operating system. See Day 2 operations support for more details. Figure 9.1. Manually updating an installed operating system, changing the container image reference or rolling back changes if needed 9.1. Switching the container image reference You can change the container image reference used for upgrades by using the bootc switch command. For example, you can switch from the stage to the production tag. The bootc switch command performs the same operations as the bootc upgrade command and additionally changes the container image reference. To manually switch an existing ostree-based container image reference, use the bootc switch command. Warning The use of rpm-ostree to make changes, or install content, is not supported. Prerequisites A booted system using bootc . Procedure Run the following command: Optionally, you can use the --apply option when you want to automatically take actions, such as rebooting if the system has changed. Note The bootc switch command has the same effect as bootc upgrade . The only difference is the container image reference is changed. This allows preserving the existing states in /etc and /var , for example, host SSH keys and home directories. Additional resources The bootc-switch man page 9.2. Adding modules to the bootc image initramfs The rhel9/rhel-bootc image uses the dracut infrastructure to build an initial RAM disk, the initrd during the image build time. The initrd is built and included in the /usr/lib/modules/USDkver/initramfs.img location inside the container. You can use a drop-in configuration file to override the dracut configuration, and place it in /usr/lib/dracut/dracut.conf.d/ <50-custom-added-modules.conf> And thus re-create initrd with the modules you want to add. Prerequisites A booted system using bootc. Procedure Re-create the initrd as part of a container build: Note By default the command attempts to pull the running kernel version, which causes an error. Explicitly pass to dracut the kernel version of the target to avoid errors. 9.3. Modifying and regenerating initrd The default container image includes a pre-generated initial RAM disk (initrd) in /usr/lib/modules/USDkver/initramfs.img . To regenerate the initrd , for example, to add a dracut module, follow the steps: Procedure Write your drop-in configuration file. For example: Place your drop-in configuration file in the location that dracut normally uses: /usr . For example: Regenerate the initrd as part of the container build. You must explicitly pass the kernel version to target to dracut , because it tries to pull the running kernel version, which can cause an error. The following is an example: 9.4. Performing manual updates from an installed operating system Installing image mode for RHEL is a one time task. You can perform any other management task, such as changing or updating the system, by pushing the changes to the container registry. When using image mode for RHEL, you can choose to perform manual updates for your systems. Manual updates are also useful if you have an automated way to perform updates, for example, by using Ansible. Because the automatic updates are enabled by default, to perform manual updates you must turn the automatic updates off. You can do this by choosing one of the following options: Running the bootc upgrade command Modifying the systemd timer file 9.5. Turning off automatic updates To perform manual updates you must turn off automatic updates. You can do this by choosing one of the following options in the procedure below. Procedure Disable the timer of a container build. By running the systemctl mask command: By modifying the systemd timer file. Use systemd "drop-ins" to override the timer. In the following example, updates are scheduled for once a week. Create an updates.conf file with the following content: Add you file to the directory you created: 9.6. Manually updating an installed operating system To manually fetch updates from a registry and boot the system into the new updates, use bootc upgrade . This command fetches the transactional in-place updates from the installed operating system to the container image registry. The command queries the registry and queues an updated container image for the boot. It stages the changes to the base image, while not changing the running system by default. Procedure Run the following command: The apply argument is optional and you can use it when you want to automatically take actions, such as rebooting if the system has changed. Note The bootc upgrade and bootc update commands are aliases. Additional resources The bootc-upgrade man page 9.7. Performing rollbacks from a updated operating system You can roll back to a boot entry to revert changes by using the bootc rollback command. This command changes the boot loader entry ordering by making the deployment under rollback queued for the boot. The current deployment then becomes the rollback. Any staged changes, such as a queued upgrade that was not applied, are discarded. After a rollback completes, the system reboots and the update timer runs within 1 to 3 hours which automatically updates and reboots your system to the image you just rolled back from. Warning If you perform a rollback, the system will automatically update again unless you turn off auto-updates. See Turning off automatic updates . Prerequisites You performed an update to the system. Procedure Run the following command: Note The bootc rollback command has the same effect as bootc upgrade . The only difference is the container image being tracked. This enables preserving the existing states in /etc and /var , for example, host SSH keys and home directories. Verification Use systemd journal to check the logged message for the detected rollback invocation. You can see a log similar to: Additional resources The bootc-rollback man page 9.8. Deploying updates to system groups You can change the configuration of your operating system by modifying the Containerfile. Then you can build and push your container image to the registry. When you boot your operating system, an update will be applied. You can also change the container image source by using the bootc switch command. The container registry is the source of truth. See Switching the container image reference . Usually, when deploying updates to system groups, you can use a central management service to provide a client to be installed on each system which connects to the central service. Often, the management service requires the client to perform a one time registration. The following is an example on how to deploy updates to system groups. You can modify it to create a persistent systemd service, if required. Note For clarity reasons, the Containerfile in the example is not optimized. For example, a better optimization to avoid creating multiple layers in the image is by invoking RUN a single time. You can install a client into an image mode for RHEL image and run it at startup to register the system. Prerequisites The management-client handles future connections to the server, by using a cron job or a separate systemd service. Procedure Create a management service with the following characteristics. It determines when to upgrade the system. Disable bootc-fetch-apply-updates.timer if it is included in the base image. Install the client by using dnf , or some other method that applies for your client. Inject the credentials for the management service into the image. 9.9. Checking inventory health Health checks are one of the Day 2 Operations. You can manually check the system health of the container images and events that are running inside the container. You can set health checks by creating the container on the command line. You can display the health check status of a container by using the podman inspect or podman ps commands. You can monitor and print events that occur in Podman by using the podman events command. Each event includes a timestamp, a type, a status, a name, if applicable, and an image, if applicable. For more information about health checks and events, see chapter Monitoring containers . 9.10. Automation and GitOps You can automate the building process by using CI/CD pipelines so that an update process can be triggered by events, such as updating an application. You can use automation tools that track these updates and trigger the CI/CD pipelines. The pipeline keeps the systems up to date by using the transactional background operating system updates. 9.11. Using Toolbx to inspect bootc containers Installing software on a system presents certain risks: it can change a system's behavior, and can leave unwanted files and directories behind after they are no longer needed. You can prevent these risks by installing your favorite development and debugging tools, editors, and software development kits (SDKs) into the Toolbx utility included in the RHEL bootc, an image fully mutable container without affecting the base operating system. You can perform changes on the host system with commands such as less , lsof , rsync , ssh , sudo , and unzip . The Toolbx utility performs the following actions: Pulling the registry.access.redhat.com/ubi9/toolbox:latest image to your local system Starting up a container from the image Running a shell inside the container from which you can access the host system Note Toolbx can run a root container or a rootless container, depending on the rights of the user who creates the Toolbx container. Utilities that would require root rights on the host system also should be run in root containers. The default container name is rhel-toolbox . To inspect bootc containers, follow the steps: Procedure Start a Toolbx container by using the toolbox create command and enter the container with the toolbox enter command. As a rootless user: As a root user: Verify that you pulled the correct image: Enter the Toolbx container: Optional: Check if you pulled the correct image Enter a command inside the <mytoolbox> container and display the name of the container and the image: Use the Toolbx to install the development tools: Install the tools of your choice, for example, the Emacs text editor, GCC compiler and GNU Debugger (GDB): Optional: Verify that the tools are installed: After installation, you can continue using those tools as a rootless user. Use Toolbx to troubleshoot the host system without installing them on the host system. Install the systemd suite to be able to run the journalctl command: Display log messages for all processes running on the host: Display log messages for the kernel: Install the nmap network scanning tool: Scan IP addresses and ports in a network: The -sS option performs a TCP SYN scan. Most of Nmap's scan types are only available to privileged users, because they send and receive raw packets, which requires root access on UNIX systems. Stop the Toolbx bootc container. Leave the container and return to the host: Stop the toolbox container: Optional: Remove the toolbox container: Alternatively, you can also use the podman rm command to remove the bootc container. Additional resources Debugging image mode hosts article

Commands:
sudo bootc switch [--apply] quay.io/ <namespace> / <image> : <tag>
FROM <baseimage> COPY <50-custom-added-modules> .conf /usr/lib/dracut/dracut.conf.d RUN set -x; kver=USD(cd /usr/lib/modules && echo *); dracut -vf /usr/lib/modules/USDkver/initramfs.img USDkver
dracutmodules = " module "
/usr/lib/dracut/dracut.conf.d/50-custom-added-modules.conf
FROM <baseimage> COPY 50-custom-added-modules.conf /usr/lib/dracut/dracut.conf.d RUN set -x; kver=USD(cd /usr/lib/modules && echo *); dracut -vf /usr/lib/modules/USDkver/initramfs.img USDkver
systemctl mask bootc-fetch-apply-updates.timer
Clear previous timers OnBootSec= OnBootSec=1w OnUnitInactiveSec=1w
mkdir -p /usr/lib/systemd/system/bootc-fetch-apply-updates.timer.d cp updates.conf /usr/lib/systemd/system/bootc-fetch-apply-updates.timer.d
bootc upgrade [--apply]
bootc rollback [-h|--help] [-V|--version]
journalctl -b
MESSAGE_ID=26f3b1eb24464d12aa5e7b544a6b5468
toolbox create <mytoolbox>
sudo toolbox create <mytoolbox> Created container: <mytoolbox> Enter with: toolbox enter
[user@toolbox ~]USD toolbox list IMAGE ID IMAGE NAME CREATED fe0ae375f149 registry.access.redhat.com/ubi{ProductVersion}/toolbox 5 weeks ago CONTAINER ID CONTAINER NAME CREATED STATUS IMAGE NAME 5245b924c2cb <mytoolbox> 7 minutes ago created registry.access.redhat.com/ubi{ProductVersion}/toolbox:8.9-6
[user@toolbox ~]USD toolbox enter <mytoolbox>
⬢ [user@toolbox ~]USD cat /run/.containerenv engine="podman-4.8.2" name=" <mytoolbox> " id="5245b924c2cb..." image="registry.access.redhat.com/ubi{ProductVersion}/toolbox" imageid="fe0ae375f14919cbc0596142e3aff22a70973a36e5a165c75a86ea7ec5d8d65c"
⬢[user@toolbox ~]USD sudo dnf install emacs gcc gdb
⬢[user@toolbox ~]USD dnf repoquery --info --installed <package_name>
⬢[root@toolbox ~]# dnf install systemd
⬢[root@toolbox ~]# j journalctl --boot -0 Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: microcode: updated ear> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: Linux version 6.6.8-10> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: Command line: BOOT_IMA> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: x86/split lock detecti> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: BIOS-provided physical>
⬢[root@toolbox ~]# journalctl --boot -0 --dmesg Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: microcode: updated ear> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: Linux version 6.6.8-10> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: Command line: BOOT_IMA> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: x86/split lock detecti> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: BIOS-provided physical> Jan 02 09:06:48 user-thinkpadp1gen4i.brq.csb kernel: BIOS-e820: [mem 0x0000>
⬢[root@toolbox ~]# dnf install nmap
⬢[root@toolbox ~]# nmap -sS scanme.nmap.org Starting Nmap 7.93 ( https://nmap.org ) at 2024-01-02 10:39 CET Stats: 0:01:01 elapsed; 0 hosts completed (0 up), 256 undergoing Ping Scan Ping Scan Timing: About 29.79% done; ETC: 10:43 (0:02:24 remaining) Nmap done: 256 IP addresses (0 hosts up) scanned in 206.45 seconds
⬢ [user@toolbox ~]USD exit
⬢ [user@toolbox ~]USD podman stop <mytoolbox>
⬢ [user@toolbox ~]USD toolbox rm <mytoolbox>