Chapter 7. RHCOS image layering Red Hat Enterprise Linux CoreOS (RHCOS) image layering allows you to easily extend the functionality of your base RHCOS image by layering additional images onto the base image. This layering does not modify the base RHCOS image. Instead, it creates a custom layered image that includes all RHCOS functionality and adds additional functionality to specific nodes in the cluster. 7.1. About RHCOS image layering Image layering allows you to customize the underlying node operating system on any of your cluster nodes. This helps keep everything up-to-date, including the node operating system and any added customizations such as specialized software. You create a custom layered image by using a Containerfile and applying it to nodes by using a custom object. At any time, you can remove the custom layered image by deleting that custom object. With RHCOS image layering, you can install RPMs into your base image, and your custom content will be booted alongside RHCOS. The Machine Config Operator (MCO) can roll out these custom layered images and monitor these custom containers in the same way it does for the default RHCOS image. RHCOS image layering gives you greater flexibility in how you manage your RHCOS nodes. Important Installing realtime kernel and extensions RPMs as custom layered content is not recommended. This is because these RPMs can conflict with RPMs installed by using a machine config. If there is a conflict, the MCO enters a degraded state when it tries to install the machine config RPM. You need to remove the conflicting extension from your machine config before proceeding. As soon as you apply the custom layered image to your cluster, you effectively take ownership of your custom layered images and those nodes. While Red Hat remains responsible for maintaining and updating the base RHCOS image on standard nodes, you are responsible for maintaining and updating images on nodes that use a custom layered image. You assume the responsibility for the package you applied with the custom layered image and any issues that might arise with the package. There are two methods for deploying a custom layered image onto your nodes: On-cluster layering With on-cluster layering , you create a MachineOSConfig object where you include the Containerfile and other parameters. The build is performed on your cluster and the resulting custom layered image is automatically pushed to your repository and applied to the machine config pool that you specified in the MachineOSConfig object. The entire process is performed completely within your cluster. Important On-cluster image layering is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Out-of-cluster layering With out-of-cluster layering , you create a Containerfile that references an OpenShift Container Platform image and the RPM that you want to apply, build the layered image in your own environment, and push the image to your repository. Then, in your cluster, create a MachineConfig object for the targeted node pool that points to the new image. The Machine Config Operator overrides the base RHCOS image, as specified by the osImageURL value in the associated machine config, and boots the new image. Important For both methods, use the same base RHCOS image installed on the rest of your cluster. Use the oc adm release info --image-for rhel-coreos command to obtain the base image used in your cluster. 7.2. Example Containerfiles RHCOS image layering allows you to use the following types of images to create custom layered images: OpenShift Container Platform Hotfixes . You can work with Customer Experience and Engagement (CEE) to obtain and apply Hotfix packages on top of your RHCOS image. In some instances, you might want a bug fix or enhancement before it is included in an official OpenShift Container Platform release. RHCOS image layering allows you to easily add the Hotfix before it is officially released and remove the Hotfix when the underlying RHCOS image incorporates the fix. Important Some Hotfixes require a Red Hat Support Exception and are outside of the normal scope of OpenShift Container Platform support coverage or life cycle policies. Hotfixes are provided to you based on Red Hat Hotfix policy . Apply it on top of the base image and test that new custom layered image in a non-production environment. When you are satisfied that the custom layered image is safe to use in production, you can roll it out on your own schedule to specific node pools. For any reason, you can easily roll back the custom layered image and return to using the default RHCOS. Example on-cluster Containerfile to apply a Hotfix # Using a 4.17.0 image containerfileArch: noarch content: |- FROM configs AS final #Install hotfix rpm RUN dnf install -y https://example.com/myrepo/haproxy-1.0.16-5.el8.src.rpm && \ dnf clean all && \ ostree container commit Example out-of-cluster Containerfile to apply a Hotfix # Using a 4.17.0 image FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256... #Install hotfix rpm RUN dnf install -y https://example.com/myrepo/haproxy-1.0.16-5.el8.src.rpm && \ dnf clean all && \ ostree container commit RHEL packages . You can download Red Hat Enterprise Linux (RHEL) packages from the Red Hat Customer Portal , such as chrony, firewalld, and iputils. Example out-of-cluster Containerfile to apply the libreswan utility # Get RHCOS base image of target cluster `oc adm release info --image-for rhel-coreos` # hadolint ignore=DL3006 FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256... # Install our config file COPY my-host-to-host.conf /etc/ipsec.d/ # RHEL entitled host is needed here to access RHEL packages # Install libreswan as extra RHEL package RUN dnf install -y libreswan && \ dnf clean all && \ systemctl enable ipsec && \ ostree container commit Because libreswan requires additional RHEL packages, the image must be built on an entitled RHEL host. For RHEL entitlements to work, you must copy the etc-pki-entitlement secret into the openshift-machine-config-operator namespace. Third-party packages . You can download and install RPMs from third-party organizations, such as the following types of packages: Bleeding edge drivers and kernel enhancements to improve performance or add capabilities. Forensic client tools to investigate possible and actual break-ins. Security agents. Inventory agents that provide a coherent view of the entire cluster. SSH Key management packages. Example on-cluster Containerfile to apply a third-party package from EPEL FROM configs AS final #Enable EPEL (more info at https://docs.fedoraproject.org/en-US/epel/ ) and install htop RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && \ dnf install -y htop && \ dnf clean all && \ ostree container commit Example out-of-cluster Containerfile to apply a third-party package from EPEL # Get RHCOS base image of target cluster `oc adm release info --image-for rhel-coreos` # hadolint ignore=DL3006 FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256... #Enable EPEL (more info at https://docs.fedoraproject.org/en-US/epel/ ) and install htop RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && \ dnf install -y htop && \ dnf clean all && \ ostree container commit This Containerfile installs the RHEL fish program. Because fish requires additional RHEL packages, the image must be built on an entitled RHEL host. For RHEL entitlements to work, you must copy the etc-pki-entitlement secret into the openshift-machine-api namespace. Example on-cluster Containerfile to apply a third-party package that has RHEL dependencies FROM configs AS final # RHEL entitled host is needed here to access RHEL packages # Install fish as third party package from EPEL RUN dnf install -y https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/f/fish-3.3.1-3.el9.x86_64.rpm && \ dnf clean all && \ ostree container commit Example out-of-cluster Containerfile to apply a third-party package that has RHEL dependencies # Get RHCOS base image of target cluster `oc adm release info --image-for rhel-coreos` # hadolint ignore=DL3006 FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256... # RHEL entitled host is needed here to access RHEL packages # Install fish as third party package from EPEL RUN dnf install -y https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/f/fish-3.3.1-3.el9.x86_64.rpm && \ dnf clean all && \ ostree container commit After you create the machine config, the Machine Config Operator (MCO) performs the following steps: Renders a new machine config for the specified pool or pools. Performs cordon and drain operations on the nodes in the pool or pools. Writes the rest of the machine config parameters onto the nodes. Applies the custom layered image to the node. Reboots the node using the new image. Important It is strongly recommended that you test your images outside of your production environment before rolling out to your cluster. 7.3. Using on-cluster layering to apply a custom layered image To apply a custom layered image to your cluster by using the on-cluster build process, make a MachineOSConfig custom resource (CR) that specifies the following parameters: the Containerfile to build the machine config pool to associate the build where the final image should be pushed and pulled from the push and pull secrets to use When you create the object, the Machine Config Operator (MCO) creates a MachineOSBuild object and a builder pod. The build process also creates transient objects, such as config maps, which are cleaned up after the build is complete. The MachineOSBuild object and the associated builder-* pod use the same naming scheme, <MachineOSConfig_CR_name>-<hash> , for example: Example MachineOSBuild object NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED layered-c8765e26ebc87e1e17a7d6e0a78e8bae False False True False False Example builder pod NAME READY STATUS RESTARTS AGE build-layered-c8765e26ebc87e1e17a7d6e0a78e8bae 2/2 Running 0 11m When the build is complete, the MCO pushes the new custom layered image to your repository and rolled out to the nodes in the associated machine config pool. You can see the digested image pull spec for the new custom layered image in the MachineOSBuild object and machine-os-builder pod. Tip You can test a MachineOSBuild object to make sure it builds correctly without rolling out the custom layered image to active nodes by using a custom machine config pool that contains non-production nodes. Alternatively, you can use a custom machine config pool that has no nodes. The MachineOSBuild object builds even if there are no nodes for the MCO to deploy the custom layered image onto. You should not need to interact with these new objects or the machine-os-builder pod. However, you can use all of these resources for troubleshooting, if necessary. You need a separate MachineOSConfig CR for each machine config pool where you want to use a custom layered image. Important On-cluster image layering is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Making certain changes to a MachineOSConfig object triggers an automatic rebuild of the associated custom layered image. You can mitigate the effects of the rebuild by pausing the machine config pool where the custom layered image is applied as described in "Pausing the machine config pools." While the pools are paused, the MCO does not roll out the newly built image to the nodes after the build is complete. However, the build will still run regardless of whether the pool is paused or not. For example, if you want to remove and replace a MachineOSCOnfig object, pausing the machine config pools before making the change prevents the MCO from reverting the associated nodes to the base image, reducing the number of reboots needed. When a machine config pool is paused, the oc get machineconfigpools reports the following status: Example output NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-a0b404d061a6183cc36d302363422aba True False False 3 3 3 0 4h14m worker rendered-worker-221507009cbcdec0eec8ab3ccd789d18 False False False 2 2 2 0 4h14m 1 1 The worker machine config pool is paused, as indicated by the three False statuses and the READYMACHINECOUNT at 0 . After the changes have been rolled out, you can unpause the machine config pool. In the case of a build failure, for example due to network issues or an invalid secret, the MCO retries the build three additional times before the job fails. The MCO creates a different build pod for each build attempt. You can use the build pod logs to troubleshoot any build failures. However, the MCO automatically removes these build pods after a short period of time. Example failed MachineOSBuild object NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED layered-c8765e26ebc87e1e17a7d6e0a78e8bae False False False False True On-cluster layering Technology Preview known limitations Note the following limitations when working with the on-cluster layering feature: On-cluster layering is supported only for OpenShift Container Platform clusters on the AMD64 architecture. On-cluster layering is not supported on multi-architecture compute machines, single-node OpenShift, or disconnected clusters. If you scale up a machine set that uses a custom layered image, the nodes reboot two times. The first, when the node is initially created with the base image and a second time when the custom layered image is applied. Node disruption policies are not supported on nodes with a custom layered image. As a result the following configuration changes cause a node reboot: Modifying the configuration files in the /var or /etc directory Adding or modifying a systemd service Changing SSH keys Removing mirroring rules from ICSP , ITMS , and IDMS objects Changing the trusted CA, by updating the user-ca-bundle configmap in the openshift-config namespace The images used in creating custom layered images take up space in your push registry. Always be aware of the free space in your registry and prune the images as needed. Prerequisites You have enabled the TechPreviewNoUpgrade feature set by using the feature gates. For more information, see "Enabling features using feature gates". You have a copy of the pull secret in the openshift-machine-config-operator namespace that the MCO needs to pull the base operating system image. You have the push secret of the registry that the MCO needs to push the new custom layered image to. You have a pull secret that your nodes need to pull the new custom layered image from your registry. This should be a different secret than the one used to push the image to the repository. You are familiar with how to configure a Containerfile. Instructions on how to create a Containerfile are beyond the scope of this documentation. Optional: You have a separate machine config pool for the nodes where you want to apply the custom layered image. Procedure Create a MachineOSconfig object: Create a YAML file similar to the following: apiVersion: machineconfiguration.openshift.io/v1aplha1 1 kind: MachineOSConfig metadata: name: layered 2 spec: machineConfigPool: name: worker 3 buildInputs: containerFile: 4 - containerfileArch: noarch content: |- FROM configs AS final RUN rpm-ostree install tree && \ ostree container commit imageBuilder: 5 imageBuilderType: PodImageBuilder baseImagePullSecret: 6 name: global-pull-secret-copy renderedImagePushspec: image-registry.openshift-image-registry.svc:5000/openshift/os-image:latest 7 renderedImagePushSecret: 8 name: builder-dockercfg-7lzwl buildOutputs: 9 currentImagePullSecret: name: builder-dockercfg-7lzwl 1 Specifies the machineconfiguration.openshift.io/v1 API that is required for MachineConfig CRs. 2 Specifies a name for the MachineOSConfig object. This name is used with other on-cluster layering resources. The examples in this documentation use the name layered . 3 Specifies the name of the machine config pool associated with the nodes where you want to deploy the custom layered image. The examples in this documentation use the worker machine config pool. 4 Specifies the Containerfile to configure the custom layered image. 5 Specifies the name of the image builder to use. This must be PodImageBuilder . 6 Specifies the name of the pull secret that the MCO needs in order to pull the base operating system image from the registry. 7 Specifies the image registry to push the newly built custom layered image to. This can be any registry that your cluster has access to. This example uses the internal OpenShift Container Platform registry. 8 Specifies the name of the push secret that the MCO needs in order to push the newly built custom layered image to that registry. 9 Specifies the secret required by the image registry that the nodes need in order to pull the newly built custom layered image. This should be a different secret than the one used to push the image to your repository. Create the MachineOSConfig object: USD oc create -f <file_name>.yaml If necessary, when the MachineOSBuild object has been created and is in the READY state, modify the node spec for the nodes where you want to use the new custom layered image: Check that the MachineOSBuild object is READY . When the SUCCEEDED value is True , the build is complete. USD oc get machineosbuild Example output showing that the MachineOSBuild object is ready NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED layered-ad5a3cad36303c363cf458ab0524e7c0-builder False False True False False Edit the nodes where you want to deploy the custom layered image by adding a label for the machine config pool you specified in the MachineOSConfig object: USD oc label node <node_name> 'node-role.kubernetes.io/<mcp_name>=' where: node-role.kubernetes.io/<mcp_name>= Specifies a node selector that identifies the nodes to deploy the custom layered image. When you save the changes, the MCO drains, cordons, and reboots the nodes. After the reboot, the node will be using the new custom layered image. Verification Verify that the new pods are ready by running the following command: USD oc get pods -n openshift-machine-config-operator Example output NAME READY STATUS RESTARTS AGE build-layered-ad5a3cad36303c363cf458ab0524e7c0-hxrws 2/2 Running 0 2m40s 1 # ... machine-os-builder-6fb66cfb99-zcpvq 1/1 Running 0 2m42s 2 1 This is the build pod where the custom layered image is building, named in the build-<MachineOSConfig_CR_name>-<hash> format. 2 This pod can be used for troubleshooting. Verify that the MachineOSConfig object contains a reference to the new custom layered image by running the following command: USD oc get machineosbuilds Example output NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED layered-ad5a3cad36303c363cf458ab0524e7c0 False True False False False 1 1 The MachineOSBuild is named in the <MachineOSConfig_CR_name>-<hash> format. Verify that the MachineOSBuild object contains a reference to the new custom layered image by running the following command: USD oc describe machineosbuild <object_name> Example output Name: layered-ad5a3cad36303c363cf458ab0524e7c0 # ... API Version: machineconfiguration.openshift.io/v1alpha1 Kind: MachineOSBuild # ... Spec: Config Generation: 1 Desired Config: Name: rendered-layered-ad5a3cad36303c363cf458ab0524e7c0 Machine OS Config: Name: layered Rendered Image Pushspec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images:layered-ad5a3cad36303c363cf458ab0524e7c0 # ... Last Transition Time: 2025-02-12T19:21:28Z Message: Build Ready Reason: Ready Status: True Type: Succeeded Final Image Pullspec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images@sha256:312e48825e074b01a913deedd6de68abd44894ede50b2d14f99d722f13cda04b 1 1 Digested image pull spec for the new custom layered image. Verify that the appropriate nodes are using the new custom layered image: Start a debug session as root for a control plane node: USD oc debug node/<node_name> Set /host as the root directory within the debug shell: sh-4.4# chroot /host Run the rpm-ostree status command to view that the custom layered image is in use: sh-5.1# rpm-ostree status Example output # ... Deployments: * ostree-unverified-registry:image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images@sha256:312e48825e074b01a913deedd6de68abd44894ede50b2d14f99d722f13cda04b Digest: sha256:312e48825e074b01a913deedd6de68abd44894ede50b2d14f99d722f13cda04b 1 Version: 418.94.202502100215-0 (2025-02-12T19:20:44Z) 1 Digested image pull spec for the new custom layered image. Additional resources Enabling features using feature gates Pausing the machine config pools 7.3.1. Modifying a custom layered image You can modify an on-cluster custom layered image, as needed. This allows you to install additional packages, remove existing packages, change the pull or push repositories, update secrets, or other similar changes. You can edit the MachineOSConfig object, apply changes to the YAML file that created the MachineOSConfig object, or create a new YAML file for that purpose. If you modify and apply the MachineOSConfig object YAML or create a new YAML file, the YAML overwrites any changes you made directly to the MachineOSConfig object itself. Making certain changes to a MachineOSConfig object triggers an automatic rebuild of the associated custom layered image. You can mitigate the effects of the rebuild by pausing the machine config pool where the custom layered image is applied as described in "Pausing the machine config pools." While the pools are paused, the MCO does not roll out the newly built image to the nodes after the build is complete. However, the build will still run regardless of whether the pool is paused or not. For example, if you want to remove and replace a MachineOSCOnfig object, pausing the machine config pools before making the change prevents the MCO from reverting the associated nodes to the base image, reducing the number of reboots needed. When a machine config pool is paused, the oc get machineconfigpools reports the following status: Example output NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-a0b404d061a6183cc36d302363422aba True False False 3 3 3 0 4h14m worker rendered-worker-221507009cbcdec0eec8ab3ccd789d18 False False False 2 2 2 0 4h14m 1 1 The worker machine config pool is paused, as indicated by the three False statuses and the READYMACHINECOUNT at 0 . After the changes have been rolled out, you can unpause the machine config pool. Prerequisites You have opted in to on-cluster layering by creating a MachineOSConfig object. Procedure Modify an object to update the associated custom layered image: Edit the MachineOSConfig object to modify the custom layered image. The following example adds the rngd daemon to nodes that already have the tree package that was installed using a custom layered image. apiVersion: machineconfiguration.openshift.io/v1alpha1 kind: MachineOSConfig metadata: name: layered spec: machineConfigPool: name: worker buildInputs: containerFile: - containerfileArch: noarch content: |- 1 FROM configs AS final RUN rpm-ostree install rng-tools && \ systemctl enable rngd && \ rpm-ostree cleanup -m && \ ostree container commit RUN rpm-ostree install tree && \ ostree container commit imageBuilder: imageBuilderType: PodImageBuilder baseImagePullSecret: name: global-pull-secret-copy 2 renderedImagePushspec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images:latest 3 renderedImagePushSecret: 4 name: new-secret-name buildOutputs: currentImagePullSecret: name: new-secret-name 5 1 Optional: Modify the Containerfile, for example to add or remove packages. 2 Optional: Update the secret needed to pull the base operating system image from the registry. 3 Optional: Modify the image registry to push the newly built custom layered image to. 4 Optional: Update the secret needed to push the newly built custom layered image to the registry. 5 Optional: Update the secret needed to pull the newly built custom layered image from the registry. When you save the changes, the MCO drains, cordons, and reboots the nodes. After the reboot, the node uses the cluster base Red Hat Enterprise Linux CoreOS (RHCOS) image. If your changes modify a secret only, no new build is triggered and no reboot is performed. Verification Verify that the new MachineOSBuild object was created by using the following command: USD oc get machineosbuild Example output NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED layered-a5457b883f5239cdcb71b57e1a30b6ef False False True False False layered-f91f0f5593dd337d89bf4d38c877590b False True False False False 1 1 The value True in the BUILDING column indicates that the MachineOSBuild object is building. When the SUCCEEDED column reports True , the build is complete. You can watch as the new machine config is rolled out to the nodes by using the following command: USD oc get machineconfigpools Example output NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-a0b404d061a6183cc36d302363422aba True False False 3 3 3 0 3h38m worker rendered-worker-221507009cbcdec0eec8ab3ccd789d18 False True False 2 2 2 0 3h38m 1 1 The value FALSE in the UPDATED column indicates that the MachineOSBuild object is building. When the UPDATED column reports FALSE , the new custom layered image has rolled out to the nodes. When the node is back in the Ready state, check that the changes were applied: Open an oc debug session to the node by running the following command: USD oc debug node/<node_name> Set /host as the root directory within the debug shell by running the following command: sh-5.1# chroot /host Use an appropriate command to verify that change was applied. The following examples shows that the rngd daemon was installed: sh-5.1# rpm -qa |grep rng-tools Example output rng-tools-6.17-3.fc41.x86_64 sh-5.1# rngd -v Example output rngd 6.16 Additional resources Pausing the machine config pools 7.3.2. Reverting an on-cluster custom layered image You can revert an on-cluster custom layered image from nodes by removing the label for the machine config pool (MCP) that you specified in the MachineOSConfig object. After you remove the label, the Machine Config Operator (MCO) reboots the nodes in that MCP with the cluster base Red Hat Enterprise Linux CoreOS (RHCOS) image, along with any previously-made machine config changes, overriding the custom layered image. Important If the node where the custom layered image is deployed uses a custom machine config pool, before you remove the label, make sure the node is associated with a second MCP. You can reapply the custom layered image to the node by using the oc label node/<node_name> 'node-role.kubernetes.io/<mcp_name>=' label. Prerequisites You have opted in to on-cluster layering by creating a MachineOSConfig object. Procedure Remove the label from the node by using the following command: USD oc label node/<node_name> node-role.kubernetes.io/<mcp_name>- When you save the changes, the MCO drains, cordons, and reboots the nodes. After the reboot, the node uses the cluster base Red Hat Enterprise Linux CoreOS (RHCOS) image. Verification You can verify that the custom layered image is removed by performing the following checks: Check that the worker machine config pool is updating with the machine config: USD oc get mcp Sample output NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-8332482204e0b76002f15ecad15b6c2d True False False 3 3 3 0 5h26m worker rendered-worker-bde4e4206442c0a48b1a1fb35ba56e85 False True False 3 2 2 0 5h26m 1 1 The value FALSE in the UPDATED column indicates that the MachineOSBuild object is building. When the UPDATED column reports FALSE , the base image has rolled out to the nodes. Check the nodes to see that scheduling on the nodes is disabled. This indicates that the change is being applied: USD oc get nodes Example output NAME STATUS ROLES AGE VERSION ip-10-0-148-79.us-west-1.compute.internal Ready worker 32m v1.31.3 ip-10-0-155-125.us-west-1.compute.internal Ready,SchedulingDisabled worker 35m v1.31.3 ip-10-0-170-47.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-174-77.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-211-49.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-218-151.us-west-1.compute.internal Ready worker 31m v1.31.3 When the node is back in the Ready state, check that the node is using the base image: Open an oc debug session to the node. For example: USD oc debug node/<node_name> Set /host as the root directory within the debug shell: sh-5.1# chroot /host Run an rpm-ostree status command to view that the base image is in use: sh-5.1# rpm-ostree status Example output State: idle Deployments: * ostree-unverified-image:containers-storage:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76721c875a2b79688be46b1dca654c2c6619a6be28b29a2822cd86c3f9d8e3c1 Digest: sha256:76721c875a2b79688be46b1dca654c2c6619a6be28b29a2822cd86c3f9d8e3c1 Version: 418.94.202501300706-0 (2025-01-30T07:10:58Z) 7.4. Using out-of-cluster layering to apply a custom layered image You can easily configure Red Hat Enterprise Linux CoreOS (RHCOS) image layering on the nodes in specific machine config pools. The Machine Config Operator (MCO) reboots those nodes with the new custom layered image, overriding the base Red Hat Enterprise Linux CoreOS (RHCOS) image. To apply a custom layered image to your cluster, you must have the custom layered image in a repository that your cluster can access. Then, create a MachineConfig object that points to the custom layered image. You need a separate MachineConfig object for each machine config pool that you want to configure. Important When you configure a custom layered image, OpenShift Container Platform no longer automatically updates any node that uses the custom layered image. You become responsible for manually updating your nodes as appropriate. If you roll back the custom layer, OpenShift Container Platform will again automatically update the node. See the Additional resources section that follows for important information about updating nodes that use a custom layered image. Prerequisites You must create a custom layered image that is based on an OpenShift Container Platform image digest, not a tag. Note You should use the same base RHCOS image that is installed on the rest of your cluster. Use the oc adm release info --image-for rhel-coreos command to obtain the base image being used in your cluster. For example, the following Containerfile creates a custom layered image from an OpenShift Container Platform 4.18 image and overrides the kernel package with one from CentOS 9 Stream: Example Containerfile for a custom layer image # Using a 4.18.0 image FROM quay.io/openshift-release/ocp-release@sha256... 1 #Install hotfix rpm RUN rpm-ostree override replace http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/kernel-{,core-,modules-,modules-core-,modules-extra-}5.14.0-295.el9.x86_64.rpm && \ 2 rpm-ostree cleanup -m && \ ostree container commit 1 Specifies the RHCOS base image of your cluster. 2 Replaces the kernel packages. Note Instructions on how to create a Containerfile are beyond the scope of this documentation. Because the process for building a custom layered image is performed outside of the cluster, you must use the --authfile /path/to/pull-secret option with Podman or Buildah. Alternatively, to have the pull secret read by these tools automatically, you can add it to one of the default file locations: ~/.docker/config.json , USDXDG_RUNTIME_DIR/containers/auth.json , ~/.docker/config.json , or ~/.dockercfg . Refer to the containers-auth.json man page for more information. You must push the custom layered image to a repository that your cluster can access. Procedure Create a machine config file. Create a YAML file similar to the following: apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker 1 name: os-layer-custom spec: osImageURL: quay.io/my-registry/custom-image@sha256... 2 1 Specifies the machine config pool to deploy the custom layered image. 2 Specifies the path to the custom layered image in the repository. Create the MachineConfig object: USD oc create -f <file_name>.yaml Important It is strongly recommended that you test your images outside of your production environment before rolling out to your cluster. Verification You can verify that the custom layered image is applied by performing any of the following checks: Check that the worker machine config pool has rolled out with the new machine config: Check that the new machine config is created: USD oc get mc Sample output NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 00-worker 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 01-master-container-runtime 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 01-master-kubelet 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 01-worker-container-runtime 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 01-worker-kubelet 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 99-master-generated-registries 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 99-master-ssh 3.2.0 98m 99-worker-generated-registries 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 99-worker-ssh 3.2.0 98m os-layer-custom 10s 1 rendered-master-15961f1da260f7be141006404d17d39b 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m rendered-worker-5aff604cb1381a4fe07feaf1595a797e 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m rendered-worker-5de4837625b1cbc237de6b22bc0bc873 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 4s 2 1 New machine config 2 New rendered machine config Check that the osImageURL value in the new machine config points to the expected image: USD oc describe mc rendered-worker-5de4837625b1cbc237de6b22bc0bc873 Example output Name: rendered-worker-5de4837625b1cbc237de6b22bc0bc873 Namespace: Labels: <none> Annotations: machineconfiguration.openshift.io/generated-by-controller-version: 5bdb57489b720096ef912f738b46330a8f577803 machineconfiguration.openshift.io/release-image-version: 4.18.0-ec.3 API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig ... Os Image URL: quay.io/my-registry/custom-image@sha256... Check that the associated machine config pool is updated with the new machine config: USD oc get mcp Sample output NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-15961f1da260f7be141006404d17d39b True False False 3 3 3 0 39m worker rendered-worker-5de4837625b1cbc237de6b22bc0bc873 True False False 3 0 0 0 39m 1 1 When the UPDATING field is True , the machine config pool is updating with the new machine config. In this case, you will not see the new machine config listed in the output. When the field becomes False , the worker machine config pool has rolled out to the new machine config. Check the nodes to see that scheduling on the nodes is disabled. This indicates that the change is being applied: USD oc get nodes Example output NAME STATUS ROLES AGE VERSION ip-10-0-148-79.us-west-1.compute.internal Ready worker 32m v1.31.3 ip-10-0-155-125.us-west-1.compute.internal Ready,SchedulingDisabled worker 35m v1.31.3 ip-10-0-170-47.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-174-77.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-211-49.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-218-151.us-west-1.compute.internal Ready worker 31m v1.31.3 When the node is back in the Ready state, check that the node is using the custom layered image: Open an oc debug session to the node. For example: USD oc debug node/ip-10-0-155-125.us-west-1.compute.internal Set /host as the root directory within the debug shell: sh-4.4# chroot /host Run the rpm-ostree status command to view that the custom layered image is in use: sh-4.4# sudo rpm-ostree status Example output Additional resources Updating with a RHCOS custom layered image 7.4.1. Reverting an out-of-cluster node You can revert an out-of-cluster custom layered image from the nodes in specific machine config pools. The Machine Config Operator (MCO) reboots those nodes with the cluster base Red Hat Enterprise Linux CoreOS (RHCOS) image, overriding the custom layered image. To remove a Red Hat Enterprise Linux CoreOS (RHCOS) custom layered image from your cluster, you need to delete the machine config that applied the image. Procedure Delete the machine config that applied the custom layered image. USD oc delete mc os-layer-custom After deleting the machine config, the nodes reboot. Verification You can verify that the custom layered image is removed by performing any of the following checks: Check that the worker machine config pool is updating with the machine config: USD oc get mcp Sample output NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-6faecdfa1b25c114a58cf178fbaa45e2 True False False 3 3 3 0 39m worker rendered-worker-6b000dbc31aaee63c6a2d56d04cd4c1b False True False 3 0 0 0 39m 1 1 When the UPDATING field is True , the machine config pool is updating with the machine config. When the field becomes False , the worker machine config pool has rolled out to the machine config. Check the nodes to see that scheduling on the nodes is disabled. This indicates that the change is being applied: USD oc get nodes Example output NAME STATUS ROLES AGE VERSION ip-10-0-148-79.us-west-1.compute.internal Ready worker 32m v1.31.3 ip-10-0-155-125.us-west-1.compute.internal Ready,SchedulingDisabled worker 35m v1.31.3 ip-10-0-170-47.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-174-77.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-211-49.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-218-151.us-west-1.compute.internal Ready worker 31m v1.31.3 When the node is back in the Ready state, check that the node is using the base image: Open an oc debug session to the node by running the following command: USD oc debug node/<node_name> Set /host as the root directory within the debug shell by running the following command: sh-5.1# chroot /host Run the rpm-ostree status command to view that the custom layered image is in use: sh-5.1# sudo rpm-ostree status Example output 7.5. Updating with a RHCOS custom layered image When you configure Red Hat Enterprise Linux CoreOS (RHCOS) image layering, OpenShift Container Platform no longer automatically updates the node pool that uses the custom layered image. You become responsible to manually update your nodes as appropriate. To update a node that uses a custom layered image, follow these general steps: The cluster automatically upgrades to version x.y.z+1, except for the nodes that use the custom layered image. You could then create a new Containerfile that references the updated OpenShift Container Platform image and the RPM that you had previously applied. Create a new machine config that points to the updated custom layered image. Updating a node with a custom layered image is not required. However, if that node gets too far behind the current OpenShift Container Platform version, you could experience unexpected results.

Commands:
Using a 4.17.0 image containerfileArch: noarch content: |- FROM configs AS final #Install hotfix rpm RUN dnf install -y https://example.com/myrepo/haproxy-1.0.16-5.el8.src.rpm && dnf clean all && ostree container commit
Using a 4.17.0 image FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 #Install hotfix rpm RUN dnf install -y https://example.com/myrepo/haproxy-1.0.16-5.el8.src.rpm && dnf clean all && ostree container commit
Get RHCOS base image of target cluster `oc adm release info --image-for rhel-coreos` hadolint ignore=DL3006 FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 Install our config file COPY my-host-to-host.conf /etc/ipsec.d/ RHEL entitled host is needed here to access RHEL packages Install libreswan as extra RHEL package RUN dnf install -y libreswan && dnf clean all && systemctl enable ipsec && ostree container commit
FROM configs AS final #Enable EPEL (more info at https://docs.fedoraproject.org/en-US/epel/ ) and install htop RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && dnf install -y htop && dnf clean all && ostree container commit
Get RHCOS base image of target cluster `oc adm release info --image-for rhel-coreos` hadolint ignore=DL3006 FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 #Enable EPEL (more info at https://docs.fedoraproject.org/en-US/epel/ ) and install htop RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && dnf install -y htop && dnf clean all && ostree container commit
FROM configs AS final RHEL entitled host is needed here to access RHEL packages Install fish as third party package from EPEL RUN dnf install -y https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/f/fish-3.3.1-3.el9.x86_64.rpm && dnf clean all && ostree container commit
Get RHCOS base image of target cluster `oc adm release info --image-for rhel-coreos` hadolint ignore=DL3006 FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 RHEL entitled host is needed here to access RHEL packages Install fish as third party package from EPEL RUN dnf install -y https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/f/fish-3.3.1-3.el9.x86_64.rpm && dnf clean all && ostree container commit
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED layered-c8765e26ebc87e1e17a7d6e0a78e8bae False False True False False
NAME READY STATUS RESTARTS AGE build-layered-c8765e26ebc87e1e17a7d6e0a78e8bae 2/2 Running 0 11m
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-a0b404d061a6183cc36d302363422aba True False False 3 3 3 0 4h14m worker rendered-worker-221507009cbcdec0eec8ab3ccd789d18 False False False 2 2 2 0 4h14m 1
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED layered-c8765e26ebc87e1e17a7d6e0a78e8bae False False False False True
apiVersion: machineconfiguration.openshift.io/v1aplha1 1 kind: MachineOSConfig metadata: name: layered 2 spec: machineConfigPool: name: worker 3 buildInputs: containerFile: 4 - containerfileArch: noarch content: |- FROM configs AS final RUN rpm-ostree install tree && ostree container commit imageBuilder: 5 imageBuilderType: PodImageBuilder baseImagePullSecret: 6 name: global-pull-secret-copy renderedImagePushspec: image-registry.openshift-image-registry.svc:5000/openshift/os-image:latest 7 renderedImagePushSecret: 8 name: builder-dockercfg-7lzwl buildOutputs: 9 currentImagePullSecret: name: builder-dockercfg-7lzwl
oc create -f <file_name>.yaml
oc get machineosbuild
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED layered-ad5a3cad36303c363cf458ab0524e7c0-builder False False True False False
oc label node <node_name> 'node-role.kubernetes.io/<mcp_name>='
oc get pods -n openshift-machine-config-operator
NAME READY STATUS RESTARTS AGE build-layered-ad5a3cad36303c363cf458ab0524e7c0-hxrws 2/2 Running 0 2m40s 1 machine-os-builder-6fb66cfb99-zcpvq 1/1 Running 0 2m42s 2
oc get machineosbuilds
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED layered-ad5a3cad36303c363cf458ab0524e7c0 False True False False False 1
oc describe machineosbuild <object_name>
Name: layered-ad5a3cad36303c363cf458ab0524e7c0 API Version: machineconfiguration.openshift.io/v1alpha1 Kind: MachineOSBuild Spec: Config Generation: 1 Desired Config: Name: rendered-layered-ad5a3cad36303c363cf458ab0524e7c0 Machine OS Config: Name: layered Rendered Image Pushspec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images:layered-ad5a3cad36303c363cf458ab0524e7c0 Last Transition Time: 2025-02-12T19:21:28Z Message: Build Ready Reason: Ready Status: True Type: Succeeded Final Image Pullspec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images@sha256:312e48825e074b01a913deedd6de68abd44894ede50b2d14f99d722f13cda04b 1
oc debug node/<node_name>
sh-4.4# chroot /host
sh-5.1# rpm-ostree status
Deployments: * ostree-unverified-registry:image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images@sha256:312e48825e074b01a913deedd6de68abd44894ede50b2d14f99d722f13cda04b Digest: sha256:312e48825e074b01a913deedd6de68abd44894ede50b2d14f99d722f13cda04b 1 Version: 418.94.202502100215-0 (2025-02-12T19:20:44Z)
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-a0b404d061a6183cc36d302363422aba True False False 3 3 3 0 4h14m worker rendered-worker-221507009cbcdec0eec8ab3ccd789d18 False False False 2 2 2 0 4h14m 1
apiVersion: machineconfiguration.openshift.io/v1alpha1 kind: MachineOSConfig metadata: name: layered spec: machineConfigPool: name: worker buildInputs: containerFile: - containerfileArch: noarch content: |- 1 FROM configs AS final RUN rpm-ostree install rng-tools && systemctl enable rngd && rpm-ostree cleanup -m && ostree container commit RUN rpm-ostree install tree && ostree container commit imageBuilder: imageBuilderType: PodImageBuilder baseImagePullSecret: name: global-pull-secret-copy 2 renderedImagePushspec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images:latest 3 renderedImagePushSecret: 4 name: new-secret-name buildOutputs: currentImagePullSecret: name: new-secret-name 5
oc get machineosbuild
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED layered-a5457b883f5239cdcb71b57e1a30b6ef False False True False False layered-f91f0f5593dd337d89bf4d38c877590b False True False False False 1
oc get machineconfigpools
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-a0b404d061a6183cc36d302363422aba True False False 3 3 3 0 3h38m worker rendered-worker-221507009cbcdec0eec8ab3ccd789d18 False True False 2 2 2 0 3h38m 1
oc debug node/<node_name>
sh-5.1# chroot /host
sh-5.1# rpm -qa |grep rng-tools
rng-tools-6.17-3.fc41.x86_64
sh-5.1# rngd -v
rngd 6.16
oc label node/<node_name> node-role.kubernetes.io/<mcp_name>-
oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-8332482204e0b76002f15ecad15b6c2d True False False 3 3 3 0 5h26m worker rendered-worker-bde4e4206442c0a48b1a1fb35ba56e85 False True False 3 2 2 0 5h26m 1
oc get nodes
NAME STATUS ROLES AGE VERSION ip-10-0-148-79.us-west-1.compute.internal Ready worker 32m v1.31.3 ip-10-0-155-125.us-west-1.compute.internal Ready,SchedulingDisabled worker 35m v1.31.3 ip-10-0-170-47.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-174-77.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-211-49.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-218-151.us-west-1.compute.internal Ready worker 31m v1.31.3
oc debug node/<node_name>
sh-5.1# chroot /host
sh-5.1# rpm-ostree status
State: idle Deployments: * ostree-unverified-image:containers-storage:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76721c875a2b79688be46b1dca654c2c6619a6be28b29a2822cd86c3f9d8e3c1 Digest: sha256:76721c875a2b79688be46b1dca654c2c6619a6be28b29a2822cd86c3f9d8e3c1 Version: 418.94.202501300706-0 (2025-01-30T07:10:58Z)
Using a 4.18.0 image FROM quay.io/openshift-release/ocp-release@sha256... 1 #Install hotfix rpm RUN rpm-ostree override replace http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/kernel-{,core-,modules-,modules-core-,modules-extra-}5.14.0-295.el9.x86_64.rpm && \ 2 rpm-ostree cleanup -m && ostree container commit
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker 1 name: os-layer-custom spec: osImageURL: quay.io/my-registry/custom-image@sha256... 2
oc create -f <file_name>.yaml
oc get mc
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 00-worker 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 01-master-container-runtime 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 01-master-kubelet 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 01-worker-container-runtime 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 01-worker-kubelet 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 99-master-generated-registries 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 99-master-ssh 3.2.0 98m 99-worker-generated-registries 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m 99-worker-ssh 3.2.0 98m os-layer-custom 10s 1 rendered-master-15961f1da260f7be141006404d17d39b 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m rendered-worker-5aff604cb1381a4fe07feaf1595a797e 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 95m rendered-worker-5de4837625b1cbc237de6b22bc0bc873 5bdb57489b720096ef912f738b46330a8f577803 3.2.0 4s 2
oc describe mc rendered-worker-5de4837625b1cbc237de6b22bc0bc873
Name: rendered-worker-5de4837625b1cbc237de6b22bc0bc873 Namespace: Labels: <none> Annotations: machineconfiguration.openshift.io/generated-by-controller-version: 5bdb57489b720096ef912f738b46330a8f577803 machineconfiguration.openshift.io/release-image-version: 4.18.0-ec.3 API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Os Image URL: quay.io/my-registry/custom-image@sha256
oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-15961f1da260f7be141006404d17d39b True False False 3 3 3 0 39m worker rendered-worker-5de4837625b1cbc237de6b22bc0bc873 True False False 3 0 0 0 39m 1
oc get nodes
NAME STATUS ROLES AGE VERSION ip-10-0-148-79.us-west-1.compute.internal Ready worker 32m v1.31.3 ip-10-0-155-125.us-west-1.compute.internal Ready,SchedulingDisabled worker 35m v1.31.3 ip-10-0-170-47.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-174-77.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-211-49.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-218-151.us-west-1.compute.internal Ready worker 31m v1.31.3
oc debug node/ip-10-0-155-125.us-west-1.compute.internal
sh-4.4# chroot /host
sh-4.4# sudo rpm-ostree status
State: idle Deployments: * ostree-unverified-registry:quay.io/my-registry/ Digest: sha256:
oc delete mc os-layer-custom
oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-6faecdfa1b25c114a58cf178fbaa45e2 True False False 3 3 3 0 39m worker rendered-worker-6b000dbc31aaee63c6a2d56d04cd4c1b False True False 3 0 0 0 39m 1
oc get nodes
NAME STATUS ROLES AGE VERSION ip-10-0-148-79.us-west-1.compute.internal Ready worker 32m v1.31.3 ip-10-0-155-125.us-west-1.compute.internal Ready,SchedulingDisabled worker 35m v1.31.3 ip-10-0-170-47.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-174-77.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-211-49.us-west-1.compute.internal Ready control-plane,master 42m v1.31.3 ip-10-0-218-151.us-west-1.compute.internal Ready worker 31m v1.31.3
oc debug node/<node_name>
sh-5.1# chroot /host
sh-5.1# sudo rpm-ostree status
State: idle Deployments: * ostree-unverified-registry:podman pull quay.io/openshift-release-dev/ocp-release@sha256:e2044c3cfebe0ff3a99fc207ac5efe6e07878ad59fd4ad5e41f88cb016dacd73 Digest: sha256:e2044c3cfebe0ff3a99fc207ac5efe6e07878ad59fd4ad5e41f88cb016dacd73