Chapter 1. About the Red Hat OpenStack Platform framework for upgrades The Red Hat OpenStack Platform (RHOSP) framework for upgrades is a workflow to upgrade your RHOSP environment from one long life version to the long life version. This workflow is an in-place solution and the upgrade occurs within your existing environment. 1.1. High-level changes in Red Hat OpenStack Platform 17.1 The following high-level changes occur during the upgrade to Red Hat OpenStack Platform (RHOSP) 17.1: The RHOSP upgrade and the operating system upgrade are separated into two distinct phases. You upgrade RHOSP first, then you upgrade the operating system. You can upgrade a portion of your Compute nodes to RHEL 9.2 while the rest of your Compute nodes remain on RHEL 8.4. This is called a Multi-RHEL environment. With an upgrade to Red Hat Ceph Storage 5, cephadm now manages Red Hat Ceph Storage. versions of Red Hat Ceph Storage were managed by ceph-ansible . You can upgrade your Red Hat Ceph Storage nodes from version 5 to version 6 after the upgrade to RHOSP 17.1 is complete. Otherwise, Red Hat Ceph Storage nodes can remain on version 5 with RHOSP 17.1 until the end of the Red Hat Ceph Storage 5 lifecycle. To upgrade from Red Hat Ceph Storage version 5 to version 6, begin with one of the following procedures for your environment: Director-deployed Red Hat Ceph Storage environments: Updating the cephadm client External Red Hat Ceph Storage cluster environments: Updating the Red Hat Ceph Storage container image By default, the RHOSP overcloud uses Open Virtual Network (OVN) as the default ML2 mechanism driver in versions 16.2 and 17.1. If your RHOSP 16.2 deployment uses the OVS mechanism driver, you must upgrade to 17.1 with the OVS mechanism driver. Do not attempt to change the mechanism driver during the upgrade. After the upgrade, you can migrate from the OVS to the OVN mechanism driver. See Migrating to the OVN mechanism driver . In ML2/OVN deployments, you can enable egress minimum and maximum bandwidth policies for hardware offloaded ports. For more information, see Configuring the Networking service for QoS policies in Configuring Red Hat OpenStack Platform networking . The undercloud and overcloud both run on Red Hat Enterprise Linux 9. 1.2. Changes in Red Hat Enterprise Linux 9 The Red Hat OpenStack Platform (RHOSP) 17.1 uses Red Hat Enterprise Linux (RHEL) 9.2 as the base operating system. As a part of the upgrade process, you will upgrade the base operating system of nodes to RHEL 9.2. Before beginning the upgrade, review the following information to familiarize yourself with RHEL 9: If your system contains packages with RSA/SHA-1 signatures, you must remove them or contact the vendor to get packages with RSA/SHA-256 signatures before you upgrade to RHOSP 17.1. For more information, see SHA-1 deprecation in Red Hat Enterprise Linux 9 . For more information about the latest changes in RHEL 9, see the Red Hat Enterprise Linux 9.2 Release Notes . For more information about the key differences between Red Hat Enterprise Linux 8 and 9, see Considerations in adopting RHEL 9 . For general information about Red Hat Enterprise Linux 9, see Product Documentation for Red Hat Enterprise Linux 9 . For more information about upgrading from RHEL 8 to RHEL 9, see Upgrading from RHEL 8 to RHEL 9 . 1.3. Upgrade framework for long life versions You can use the Red Hat OpenStack Platform (RHOSP) upgrade framework to perform an in-place upgrade path through multiple versions of the overcloud. The goal is to provide you with an opportunity to remain on certain OpenStack versions that are considered long life versions and upgrade when the long life version is available. The Red Hat OpenStack Platform upgrade process also upgrades the version of Red Hat Enterprise Linux (RHEL) on your nodes. This guide provides an upgrade framework through the following versions: Current Version Target Version Red Hat OpenStack Platform 16.2.4 and later Red Hat OpenStack Platform 17.1 latest For detailed support dates and information on the lifecycle support for Red Hat OpenStack Platform, see Red Hat OpenStack Platform Life Cycle . Upgrade paths for long life releases Familiarize yourself with the possible update and upgrade paths before you begin an upgrade. If you are using an environment that is earlier than RHOSP 16.2.4, before you upgrade from major version to major version, you must first update your existing environment to the latest minor release. For example, if your current deployment is Red Hat OpenStack Platform (RHOSP) 16.2.1 on Red Hat Enterprise Linux (RHEL) 8.4, you must perform a minor update to the latest RHOSP 16.2 version before you upgrade to RHOSP 17.1. Note You can view your current RHOSP and RHEL versions in the /etc/rhosp-release and /etc/redhat-release files. Table 1.1. Updates version path Current version Target version RHOSP 16.2.x on RHEL 8.4 RHOSP 16.2 latest on RHEL 8.4 latest RHOSP 17.0.x on RHEL 9.0 RHOSP 17.0 latest on RHEL 9.0 latest RHOSP 17.0.x on RHEL 9.0 RHOSP 17.1 latest on RHEL 9.2 latest RHOSP 17.1.x on RHEL 9.2 RHOSP 17.1 latest on RHEL 9.2 latest For more information, see Performing a minor update of Red Hat OpenStack Platform . Table 1.2. Upgrades version path Current version Target version RHOSP 16.2 on RHEL 8.4 RHOSP 17.1 latest on RHEL 9.2 latest Red Hat provides two options for upgrading your environment to the long life release: In-place upgrade Perform an upgrade of the services in your existing environment. This guide primarily focuses on this option. Parallel migration Create a new RHOSP 17.1 environment and migrate your workloads from your current environment to the new environment. For more information about RHOSP parallel migration, contact Red Hat Global Professional Services. 1.4. Upgrade duration and impact The durations in the following table were recorded in a test environment that consisted of an overcloud with 200 nodes, and 9 Ceph Storage hosts with 17 object storage daemons (OSDs) each. The durations in the table might not apply to all production environments. For example, if your hardware has low specifications or an extended boot period, allow for more time with these durations. Durations also depend on network I/O to container and package content, and disk I/O on the host. To accurately estimate the upgrade duration for each task, perform these procedures in a test environment with hardware that is similar to your production environment. Table 1.3. Duration and impact of In-place upgrade Duration Notes Undercloud upgrade 30 minutes No disruption to the overcloud. Overcloud adoption and preparation 10 minutes for bare metal adoption 30 minutes for upgrade prepare Duration scales based on the size of the overcloud. No disruption to the overcloud. Red Hat Ceph Storage upgrade Ceph upgrade ansible run: 90 minutes total, 10 minutes per node Ceph ansible run for cephadm adoption: 30 minutes total, 3 minutes per node Post ceph upgrade and adoption overcloud upgrade prepare: 20 minutes Duration scales based on the number of Storage hosts and OSDs. Storage performance is degraded. Overcloud OpenStack upgrade 120 minutes Duration scales based on the size of the overcloud. During this process, agents are restarted and API transactions might be lost. Disable client access to the OpenStack API during this stage. Undercloud system upgrade 40 minutes Includes multiple reboots. Reboot times are hardware dependent. Includes SELinux relabeling. Hosts with large numbers of files take significantly longer. No disruption to the overcloud. Overcloud non-Compute host system upgrade 30 minutes for upgrade prepare 40 minutes per host system upgrade Includes multiple reboots. Reboot times are hardware dependent. Includes SELinux relabeling. Hosts with large numbers of files take significantly longer. Performance is degraded. Overcloud Compute host upgrade 40 minutes per batch of hosts 30 minutes for upgrade prepare You run upgrade prepare on select batches of Compute nodes. The duration depends on the number of Compute nodes in each batch. There is no outage. Includes multiple reboots. Reboot times are hardware dependent. Includes SELinux relabeling. Hosts with large numbers of files take significantly longer. To prevent workload outages during the reboot, you can migrate the workloads to another host beforehand. 1.5. Planning and preparation for an in-place upgrade Before you conduct an in-place upgrade of your OpenStack Platform environment, create a plan for the upgrade and accommodate any potential obstacles that might block a successful upgrade. 1.5.1. Familiarize yourself with Red Hat OpenStack Platform 17.1 Before you perform an upgrade, familiarize yourself with Red Hat OpenStack Platform 17.1 to help you understand the resulting environment and any potential version-to-version changes that might affect your upgrade. To familiarize yourself with Red Hat OpenStack Platform 17.1, follow these suggestions: Read the release notes for all versions across the upgrade path and identify any potential aspects that require planning: Components that contain new features Known issues Open the release notes for each version using these links: Red Hat OpenStack Platform 16.2 , which is your source version Red Hat OpenStack Platform 17.1 which is your target version Read the Installing and managing Red Hat OpenStack Platform with director guide for version 17.1 and familiarize yourself with any new requirements and processes in this guide. Install a proof-of-concept Red Hat OpenStack Platform 17.1 undercloud and overcloud. Develop hands-on experience of the target OpenStack Platform version and investigate potential differences between the target version and your current version. 1.5.2. Minor version update requirement To upgrade from Red Hat OpenStack Platform (RHOSP) 16.2 to 17.1, your environment must be running RHOSP version 16.2.4 or later. If you are using a version of RHOSP that is earlier than 16.2.4, update the environment to the latest minor version of your current release. For example, update your Red Hat OpenStack Platform 16.2.3 environment to the latest 16.2 version before upgrading to Red Hat OpenStack Platform 17.1. For instructions on performing a minor version update for Red Hat OpenStack Platform 16.2, see Keeping Red Hat OpenStack Platform Updated . 1.5.3. Leapp upgrade usage in Red Hat OpenStack Platform The long-life Red Hat OpenStack Platform upgrade requires a base operating system upgrade from Red Hat Enterprise Linux (RHEL) 8.4 to RHEL 9.2. The upgrade process uses the Leapp utility to perform the upgrade to RHEL 9.2. However, some aspects of the Leapp upgrade are customized to ensure that you are upgrading specifically from RHEL 8.4 to RHEL 9.2. To upgrade your operating system to RHEL 9.2, see Performing the undercloud system upgrade . Limitations For information on potential limitations that might affect your upgrade, see the following sections from the Upgrading from RHEL 8 to RHEL 9 guide: Planning an upgrade Known issues If any known limitations affect your environment, seek advice from the Red Hat Technical Support Team . Troubleshooting For information about troubleshooting potential Leapp issues, see Troubleshooting in Upgrading from RHEL 8 to RHEL 9 . 1.5.4. Storage driver certification Before you upgrade, confirm deployed storage drivers are certified for use with Red Hat OpenStack Platform 17.1. For information on software certified for use with Red Hat OpenStack Platform 17.1, see Software certified for Red Hat OpenStack Platform 17 . 1.5.5. Supported upgrade scenarios Before proceeding with the upgrade, check that your overcloud is supported. Note If you are uncertain whether a particular scenario not mentioned in these lists is supported, seek advice from the Red Hat Technical Support Team . Supported scenarios The following in-place upgrade scenarios are tested and supported: Standard environments with default role types: Controller, Compute, and Ceph Storage OSD Split-Controller composable roles Ceph Storage composable roles Hyper-Converged Infrastructure: Compute and Ceph Storage OSD services on the same node Environments with Network Functions Virtualization (NFV) technologies: Single-root input/output virtualization (SR-IOV) and Data Plane Development Kit (DPDK) Environments with Instance HA enabled Edge and Distributed Compute Node (DCN) scenarios Note During an upgrade procedure, nova live migrations are supported. However, evacuations initiated by Instance HA are not supported. When you upgrade a Compute node, the node is shut down cleanly and any workload running on the node is not evacuated by Instance HA automatically. Instead, you must perform live migration manually. Unsupported scenarios The following in-place upgrade scenarios are not supported: Upgrades with a single Controller node 1.5.6. Red Hat Virtualization upgrade process If you are running your control plane on Red Hat Virtualization, there is no effect on the Red Hat OpenStack Platform (RHOSP) upgrade process. The RHOSP upgrade is the same regardless of whether or not an environment is running on Red Hat Virtualization. 1.5.7. Known issues that might block an upgrade Review the following known issues that might affect a successful upgrade. BZ#2224085 - Leapp is stuck in Interim System when - -debug is specified If you upgrade your operating system from RHEL 7.x to RHEL 8.x, or from RHEL 8.x to RHEL 9.x, do not run a Leapp upgrade with the --debug option. The system remains in the early console in setup code state and does not reboot automatically. To avoid this issue, the UpgradeLeappDebug parameter is set to false by default. Do not change this value in your templates. BZ#2203785 - Collectd sensubility stops working after overcloud node was rebooted. After rebooting an overcloud node, a permission issue causes collectd-sensubility to stop working. As a result, collectd-sensubility stops reporting container health. During an upgrade from Red Hat OpenStack Platform (RHOSP) 16.2 to RHOSP 17.1, overcloud nodes are rebooted as part of the Leapp upgrade. To ensure that collectd-sensubility continues to work, run the following command: BZ#2180542 - After upgrade, manila ceph-nfs fails to start with error: ceph-nfs start on leaf1-controller-0 returned 'error' because 'failed' The Pacemaker-controlled ceph-nfs resource requires a runtime directory to store some process data. The directory is created when you install or upgrade RHOSP. Currently, a reboot of the Controller nodes removes the directory, and the ceph-nfs service does not recover when the Controller nodes are rebooted. If all Controller nodes are rebooted, the ceph-nfs service fails permanently. You can apply the following workaround: If you reboot a Controller node, log in to the Controller node and create a /var/run/ceph directory: USD mkdir -p /var/run/ceph Repeat this step on all Controller nodes that have been rebooted. If the ceph-nfs-pacemaker service has been marked as failed, after creating the directory, run the following command from any of the Controller nodes: USD pcs resource cleanup BZ#2210873 - assimilate_{{ tripleo_cephadm_cluster }}.conf required if --crush-hierarchy is used If the CephPools parameter is defined with a set of pool overrides, you must add rule_name: replicated_rule to the definition to avoid pool creation failures during an upgrade from RHOSP 16.2 to 17.1. BZ#2245602 - Upgrade (OSP16.2 ->OSP17.1) controller-0 does not perform leapp upgrade due to packages missing ovn2.15 openvswitch2.15 If you upgrade from Red Hat OpenStack Platform (RHOSP) 13 to 16.1 or 16.2, or from RHOSP 16.2 to 17.1, do not include the system_upgrade.yaml file in the --answers-file answer-upgrade.yaml file. If the system_upgrade.yaml file is included in that file, the environments/lifecycle/upgrade-prepare.yaml file overwrites the parameters in the system_upgrade.yaml file. To avoid this issue, append the system_upgrade.yaml file to the openstack overcloud upgrade prepare command. For example: With this workaround, the parameters that are configured in the system_upgrade.yaml file overwrite the default parameters in the environments/lifecycle/upgrade-prepare.yaml file. BZ#2246409 - (OSP16.2->17.1) Cinder volume NFS mounts on compute nodes are preventing leapp upgrade During an upgrade from RHOSP 16.2 to 17.1, the operating system upgrade from RHEL 8.4 to RHEL 9.2 fails if Cinder volume NFS mounts are present on Compute nodes. Contact your Red Hat support representative for a workaround. BZ#2277756 - rolling update fails unless mon_mds_skip_sanity=true is set During an upgrade from Red Hat Ceph Storage 4 to 5, a known issue prevents Ceph Monitor nodes from being upgraded. After the first Ceph Monitor node is upgraded to version 5, the other Ceph Monitor nodes stop running and report the following message: Before you upgrade your Red Hat Ceph Storage nodes, apply the workaround in the Red Hat Knowledgebase solution RHCS during upgrade RHCS 4 RHCS 5 ceph-mon is failing with "FAILED ceph_assert(fs->mds_map.compat.compare(compat) == 0)" . After the upgrade is complete, the Red Hat Ceph Storage cluster is adopted by cephadm , which does not require this workaround. BZ#2259891 - During upgrade 16.2-17.1 with not internet on UC overcloud_upgrade_prepare.sh fails pulling registry.access.redhat.com/ubi8/pause In environments where the undercloud is not connected to the internet, an upgrade from Red Hat OpenStack Platform 16.2 to 17.1 fails because the infra_image value is not defined. The overcloud_upgrade_prepare.sh script tries to pull registry.access.redhat.com/ubi8/pause , which causes an error. To avoid this issue, manually add a pause container to your Satellite server: Import a pause container to your Satellite server, for example, k8s.gcr.io/pause:3.5 or registry.access.redhat.com/ubi8/pause . In the /usr/share/containers/containers.conf file, specify the pause container in your local Satellite URL. For example: Replace <LOCAL_SATELLITE_URL/pause:3.5> with your local Satellite URL and the pause container that you imported. Confirm that you can start a pod: BZ#2264543 - (FFU 16.2->17) leapp upgrade of ceph nodes is failing encrypted partition detected When you upgrade from Red Hat OpenStack Platform (RHOSP) 16.2 to RHOSP 17.1, the Leapp upgrade of the Red Hat Ceph Storage nodes fails because of an encrypted ceph-osd . Before you run the Leapp upgrade on your Red Hat Ceph Storage nodes, apply the workaround in the Red Hat Knowledgebase solution (FFU 16.2->17) leapp upgrade of ceph nodes is failing encrypted partition detected . BZ#2275097 - bridge_name is not translated to br-ex in osp 17.1 (ffu from 16.2) The bridge_name variable is no longer valid for nic-config templates in RHOSP 17.1. After an upgrade from RHOSP 16.2 to 17.1, if you run a stack update and the nic-config templates still include the bridge_name variable, an outage occurs. Before you upgrade to RHOSP 17.1, you need to rename the bridge_name variable. For more information, see the Red Hat Knowledgebase solution bridge_name is still present in templates during and post FFU causing further updates failure . BZ#2269009 - After cephadm adoption, haproxy fails to start when alertmanager is deployed If you deployed Alertmanager in a director-deployed Red Hat Ceph Storage environment, the upgrade from Red Hat Ceph Storage version 4 to version 5 fails. The failure occurs because HAProxy does not restart after you run the following command to configure cephadm on the Red Hat Ceph Storage nodes: After you run the command, the Red Hat Ceph Storage cluster status is HEALTH_WARN . For a workaround for this issue, see the Red Hat Knowledgebase solution HAProxy does not restart during RHOSP upgrade when RHCS is director-deployed and Alertmanager is enabled . BZ#2278835 - RHCS 6 - BLUESTORE_NO_PER_POOL_OMAP OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats You might see a health warning message similar to the following after upgrading from Red Hat Ceph Storage 5 to 6: You can clear this health warning message by following the instructions in the Red Hat Knowledgebase solution link: RHCS 6 - BLUESTORE_NO_PER_POOL_OMAP OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats . BZ#2225011 - (OSP 16.2 17.1) Undercloud upgrade fails on 'migrate existing introspection data' with lost connection to mysql. If the undercloud upgrade fails, you must restart the mySQL service before you run the undercloud upgrade again. For more information about restarting the mySQL service, see the Red Hat Knowledgebase solution Update from 16.2 to 17.1 failed on migrate existing introspection data in the undercloud . BZ#2269564 - (FFU) Upgrades takes longer time when cloud consist of 350+ nodes The time you will need to upgrade from Red Hat OpenStack Platform 16.2 to 17.1 increases with the number of nodes in a single role. To reduce the amount of time it takes to complete the upgrade, you can split your nodes into multiple roles. For more information, see the Red Hat Knowledgebase article How to split roles during upgrade from RHOSP 16.2 to RHOSP 17.1 . BZ#1947415 - (RHOSP16.1): Unable to delete DEFAULT volume type Starting with RHOSP 16.1.7, deleting the DEFAULT volume type is allowed. However, the DEFAULT volume type is hard coded in the cinder.conf file, and therefore it must exist during a fast forward upgrade. If you deleted the DEFAULT volume type, do not perform an upgrade from RHOSP 16.2 to RHOSP 17.1 until after you perform the workaround described in the Red Hat Knowledgebase solution Performing online database updates failed . BZ#2305981 - OSP16.2 to OSP17.1 upgrade breaks GRUB and makes it try to boot RHEL7 When you upgrade from RHOSP 16.2 to 17.1, during the system upgrade, a known issue causes GRUB to contain RHEL 7 entries instead of RHEL 8 entries. As a result, the hosts cannot reboot. This issue affects environments that previously ran RHOSP 13.0 or earlier. Workaround: See the Red Hat Knowledgebase solution Openstack 16 to 17 FFU - During LEAPP upgrade UEFI systems do not boot due to invalid /boot/grub2/grub.cfg . BZ#2278831 - No disk space check causing unbootable node during leapp upgrade The Leapp version that upgrades Red Hat Enterprise Linux 8.4 to 9.2 does not verify whether all partitions have enough disk space. Before you perform the Red Hat OpenStack Platform system upgrade, you must manually check that all partitions have at least 3 GB of disk space. Failure to do so can cause the node to reboot and enter into an emergency shell. BZ#2259795 - Incorrect validation of Podman version If you perform an upgrade of your RHOSP environment to 17.1.x, the pre-upgrade package_version validation fails because the validation cannot find a matching podman version. Workaround: To skip the package_version validation, use the --skiplist package-version option when you run the pre-upgrade validation: BZ#2295414 -Horizon dashboard internal server error Static file compression does not run automatically after an upgrade from Red Hat OpenStack Platform (RHOSP) 16.2 to 17.1. As a result, the missing static files cause the Red Hat OpenStack Platform (RHOSP) dashboard (horizon) to fail. To run the compression manually after the upgrade, see Compressing Red Hat OpenStack Platform dashboard files . BZ#2295407 - Ceph gets the eus repos enabled If you are using director-deployed Red Hat Ceph Storage 5 nodes, during an upgrade from RHOSP 16.2 to 17.1, the EUS repositories that are specified in the UpgradeInitCommand parameter override the repositories in the Red Hat Ceph Storage role. Workaround : To use the repositories listed in your Red Hat Ceph Storage nodes, add the following parameters: In the upgrades-environment.yaml file, add the CephStorageUpgradeInitCommand : In the system_upgrade.yaml file, add the CephStorageUpgradeLeappCommandOptions and CephStorageLeappInitCommand parameters: BZ#2264174 (OSP17.1) Undercloud upgrade hangs during upgrade 16.2 to 17.1 (DB sync) In RHOSP 17.1, you can use the net_config_override variable in the undercloud.conf file to identify an alternate network configuration file. In that alternate file, you must include the IP addresses that are used for the Virtual IPs (VIPs). If the IP addresses are not present in that file, when you run openstack undercloud install , the DB sync hangs. For example: BZ#2320138 (FFU) 16.2->17.1 upgrade fails if extra openvswitch packages are installed An upgrade from Red Hat OpenStack Platform (RHOSP) 16.2 to 17.1 fails if additional, non-core openvswitch packages are installed. Workaround: See the Red Hat Knowledgebase solution FFU is failing on Ansible task special treatment for OpenvSwitch . BZ#2224967 RHCS 6.1 Ceph Dashboard Graphs show "No Data" for the Object Gateway After completing an upgrade from 16.2 to 17.1 and Red Hat Ceph Storage 7, no data (N/A) is displayed in the Overall performance section of the OSD dashboard. Workaround: See the Red Hat Knowledgebase solution Ceph dashboard is not showing performance metrics graphs on RHCS 6 . BZ#2346107 grafana.update.checker trying to reach Internet in a disconnected environment After an upgrade from Red Hat Ceph Storage 6 to 7, if you have a disconnected Red Hat OpenStack environment, Grafana attempts to access the internet to download updates. As a result, Grafana times out. Workaround: See BZ#2346107 . FFU upgrade from 16.2.6 to 17.1.4 - openstack leapp upgrade of compute with NVIDIA GPU card failed If you attempt to perform a Leapp OS upgrade with NVIDIA drivers, the system upgrade fails with the following error in /var/log/leapp/leapp-report.txt : Workaround: Remove the NVIDIA driver. For example: Remove the loaded module kernels: Upgrade the Compute node: After the server reboot, re-install the NVIDIA drivers for the appropriate operating system (RHEL 9.2). If necessary, re-create the mdev devices. 1.5.8. Backup and restore Before you upgrade your Red Hat OpenStack Platform (RHOSP) 16.2 environment, back up the undercloud and overcloud control plane by using one of the following options: Back up your nodes before you perform an upgrade. For more information about backing up nodes before you upgrade, see Red Hat OpenStack Platform 16.2 Backing up and restoring the undercloud and control plane nodes . Back up the undercloud node after you perform the undercloud upgrade and before you perform the overcloud upgrade. For more information about backing up the undercloud, see Creating a backup of the undercloud node in the Red Hat OpenStack Platform 17.1 Backing up and restoring the undercloud and control plane nodes . Use a third-party backup and recovery tool that suits your environment. For more information about certified backup and recovery tools, see the Red Hat Ecosystem catalog . 1.5.9. Proxy configuration If you use a proxy with your Red Hat OpenStack Platform 16.2 environment, the proxy configuration in the /etc/environment file will persist past the operating system upgrade and the Red Hat OpenStack Platform 17.1 upgrade. For more information about proxy configuration for Red Hat OpenStack Platform 16.2, see Considerations when running the undercloud with a proxy in Installing and managing Red Hat OpenStack Platform with director . For more information about proxy configuration for Red Hat OpenStack Platform 17.1, see Considerations when running the undercloud with a proxy in Installing and managing Red Hat OpenStack Platform with director . 1.5.10. Planning for a Compute node upgrade After you upgrade your Compute nodes from Red Hat OpenStack Platform (RHOSP) 16.2 to RHOSP 17.1, you can choose one of the following options to upgrade the Compute host operating system: Keep a portion of your Compute nodes on Red Hat Enterprise Linux (RHEL) 8.4, and upgrade the rest to RHEL 9.2. This is referred to as a Multi-RHEL environment. Upgrade all Compute nodes to RHEL 9.2, and complete the upgrade of the environment. Keep all Compute nodes on RHEL 8.4. The lifecycle of RHEL 8.4 applies. Benefits of a Multi-RHEL environment You must upgrade all of your Compute nodes to RHEL 9.2 to take advantage of any hardware-related features that are only supported in RHOSP 17.1, such as vTPM and Secure Boot. However, you might require that some or all of your Compute nodes remain on RHEL 8.4. For example, if you certified an application for RHEL 8, you can keep your Compute nodes running on RHEL 8.4 to support the application without blocking the entire upgrade. The option to upgrade part of your Compute nodes to RHEL 9.2 gives you more control over your upgrade process. You can prioritize upgrading the RHOSP environment within a planned maintenance window and defer the operating system upgrade to another time. Less downtime is required, which minimizes the impact to end users. Note If you plan to upgrade from RHOSP 17.1 to RHOSP 18.0, you must upgrade all hosts to RHEL 9.2. If you continue to run RHEL 8.4 on your hosts beyond the Extended Life Cycle Support phase, you must obtain a TUS subscription. Limitations of a Multi-RHEL environment The following limitations apply in a Multi-RHEL environment: Compute nodes running RHEL 8 cannot consume NVMe-over-TCP Cinder volumes. You cannot use different paths for socket files on RHOSP 16.2 and 17.1 for collectd monitoring. You cannot mix ML2/OVN and ML2/OVS mechanism drivers. For example, if your RHOSP 16.2 deployment included ML2/OVN, your Multi-RHEL environment must use ML2/OVN. FIPS is not supported in a Multi-RHEL environment. FIPs deployment is a Day 1 operation. FIPS is not supported in RHOSP 16.2. As a result, when you upgrade from RHOSP 16.2 to 17.1, the 17.1 environment does not include FIPS. Instance HA is not supported in a Multi-RHEL environment. Edge topologies are currently not supported. Important All HCI nodes in supported Hyperconverged Infrastructure environments must use the same version of Red Hat Enterprise Linux as the version used by the Red Hat OpenStack Platform controllers. If you wish to use multiple Red Hat Enterprise versions in a hybrid state on HCI nodes in the same Hyperconverged Infrastructure environment, contact the Red Hat Customer Experience and Engagement team to discuss a support exception. Upgrading Compute nodes Use one of the following options to upgrade your Compute nodes: To perform a Multi-RHEL upgrade of your Compute nodes, see Upgrading Compute nodes to a Multi-RHEL environment . To upgrade all Compute nodes to RHEL 9.2, see Upgrading Compute nodes to RHEL 9.2 . If you are keeping all of your Compute nodes on RHEL 8.4, no additional configuration is required. 1.6. Repositories This section contains the repositories for the undercloud and overcloud. Refer to this section when you need to enable repositories in certain situations: Enabling repositories when registering to the Red Hat Customer Portal. Enabling and synchronizing repositories to your Red Hat Satellite Server. Enabling repositories when registering to your Red Hat Satellite Server. 1.6.1. Undercloud repositories You run Red Hat OpenStack Platform (RHOSP) 17.1 on Red Hat Enterprise Linux (RHEL) 9.2. RHEL 8.4 Compute nodes are also supported in a Multi-RHEL environment when upgrading from RHOSP 16.2. Note If you synchronize repositories with Red Hat Satellite, you can enable specific versions of the Red Hat Enterprise Linux repositories. However, the repository remains the same despite the version you choose. For example, you can enable the 9.2 version of the BaseOS repository, but the repository name is still rhel-9-for-x86_64-baseos-eus-rpms despite the specific version you choose. Warning Any repositories except the ones specified here are not supported. Unless recommended, do not enable any other products or repositories except the ones listed in the following tables or else you might encounter package dependency issues. Do not enable Extra Packages for Enterprise Linux (EPEL). Core repositories The following table lists core repositories for installing the undercloud. Name Repository Description of requirement Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) Extended Update Support (EUS) rhel-9-for-x86_64-baseos-eus-rpms Base operating system repository for x86_64 systems. Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) rhel-9-for-x86_64-appstream-eus-rpms Contains Red Hat OpenStack Platform dependencies. Red Hat Enterprise Linux 9 for x86_64 - High Availability (RPMs) Extended Update Support (EUS) rhel-9-for-x86_64-highavailability-eus-rpms High availability tools for Red Hat Enterprise Linux. Used for Controller node high availability. Red Hat OpenStack Platform for RHEL 9 (RPMs) openstack-17.1-for-rhel-9-x86_64-rpms Core Red Hat OpenStack Platform repository, which contains packages for Red Hat OpenStack Platform director. Red Hat Fast Datapath for RHEL 9 (RPMS) fast-datapath-for-rhel-9-x86_64-rpms Provides Open vSwitch (OVS) packages for OpenStack Platform. Red Hat Enterprise Linux 8.4 for x86_64 - BaseOS (RPMs) Telecommunications Update Service (TUS) rhel-8-for-x86_64-baseos-tus-rpms Base operating system repository for x86_64 systems. Red Hat Enterprise Linux 8.4 for x86_64 - AppStream (RPMs) rhel-8-for-x86_64-appstream-tus-rpms Contains Red Hat OpenStack Platform dependencies. Red Hat OpenStack Platform for RHEL 8 x86_64 (RPMs) openstack-17.1-for-rhel-8-x86_64-rpms Core Red Hat OpenStack Platform repository. 1.6.2. Overcloud repositories You run Red Hat OpenStack Platform (RHOSP) 17.1 on Red Hat Enterprise Linux (RHEL) 9.2. RHEL 8.4 Compute nodes are also supported in a Multi-RHEL environment when upgrading from RHOSP 16.2. Note If you synchronize repositories with Red Hat Satellite, you can enable specific versions of the Red Hat Enterprise Linux repositories. However, the repository remains the same despite the version you choose. For example, you can enable the 9.2 version of the BaseOS repository, but the repository name is still rhel-9-for-x86_64-baseos-eus-rpms despite the specific version you choose. Warning Any repositories except the ones specified here are not supported. Unless recommended, do not enable any other products or repositories except the ones listed in the following tables or else you might encounter package dependency issues. Do not enable Extra Packages for Enterprise Linux (EPEL). Controller node repositories The following table lists core repositories for Controller nodes in the overcloud. Name Repository Description of requirement Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) Extended Update Support (EUS) rhel-9-for-x86_64-baseos-eus-rpms Base operating system repository for x86_64 systems. Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) rhel-9-for-x86_64-appstream-eus-rpms Contains Red Hat OpenStack Platform dependencies. Red Hat Enterprise Linux 9 for x86_64 - High Availability (RPMs) Extended Update Support (EUS) rhel-9-for-x86_64-highavailability-eus-rpms High availability tools for Red Hat Enterprise Linux. Red Hat OpenStack Platform for RHEL 9 x86_64 (RPMs) openstack-17.1-for-rhel-9-x86_64-rpms Core Red Hat OpenStack Platform repository. Red Hat Fast Datapath for RHEL 9 (RPMS) fast-datapath-for-rhel-9-x86_64-rpms Provides Open vSwitch (OVS) packages for OpenStack Platform. Red Hat Ceph Storage Tools 6 for RHEL 9 x86_64 (RPMs) rhceph-6-tools-for-rhel-9-x86_64-rpms Tools for Red Hat Ceph Storage 6 for Red Hat Enterprise Linux 9. Red Hat Enterprise Linux 8.4 for x86_64 - BaseOS (RPMs) Telecommunications Update Service (TUS) rhel-8-for-x86_64-baseos-tus-rpms Base operating system repository for x86_64 systems. Red Hat Enterprise Linux 8.4 for x86_64 - AppStream (RPMs) rhel-8-for-x86_64-appstream-tus-rpms Contains Red Hat OpenStack Platform dependencies. Red Hat Enterprise Linux 8 for x86_64 - High Availability (RPMs) Telecommunications Update Service (TUS) rhel-8-for-x86_64-highavailability-tus-rpms High availability tools for Red Hat Enterprise Linux. Red Hat OpenStack Platform for RHEL 8 x86_64 (RPMs) openstack-17.1-for-rhel-8-x86_64-rpms Core Red Hat OpenStack Platform repository. Compute and ComputeHCI node repositories The following table lists core repositories for Compute and ComputeHCI nodes in the overcloud. Name Repository Description of requirement Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) Extended Update Support (EUS) rhel-9-for-x86_64-baseos-eus-rpms Base operating system repository for x86_64 systems. Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) rhel-9-for-x86_64-appstream-eus-rpms Contains Red Hat OpenStack Platform dependencies. Red Hat Enterprise Linux 9 for x86_64 - High Availability (RPMs) Extended Update Support (EUS) rhel-9-for-x86_64-highavailability-eus-rpms High availability tools for Red Hat Enterprise Linux. Red Hat OpenStack Platform for RHEL 9 x86_64 (RPMs) openstack-17.1-for-rhel-9-x86_64-rpms Core Red Hat OpenStack Platform repository. Red Hat Fast Datapath for RHEL 9 (RPMS) fast-datapath-for-rhel-9-x86_64-rpms Provides Open vSwitch (OVS) packages for OpenStack Platform. Red Hat Ceph Storage Tools 6 for RHEL 9 x86_64 (RPMs) rhceph-6-tools-for-rhel-9-x86_64-rpms Tools for Red Hat Ceph Storage 6 for Red Hat Enterprise Linux 9. Red Hat Enterprise Linux 8.4 for x86_64 - BaseOS (RPMs) Telecommunications Update Service (TUS) rhel-8-for-x86_64-baseos-tus-rpms Base operating system repository for x86_64 systems. Red Hat Enterprise Linux 8.4 for x86_64 - AppStream (RPMs) rhel-8-for-x86_64-appstream-tus-rpms Contains Red Hat OpenStack Platform dependencies. Red Hat OpenStack Platform for RHEL 8 x86_64 (RPMs) openstack-17.1-for-rhel-8-x86_64-rpms Core Red Hat OpenStack Platform repository. Ceph Storage node repositories The following table lists Ceph Storage related repositories for the overcloud. Name Repository Description of requirement Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) rhel-9-for-x86_64-baseos-rpms Base operating system repository for x86_64 systems. Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) rhel-9-for-x86_64-appstream-rpms Contains Red Hat OpenStack Platform dependencies. Red Hat OpenStack Platform Deployment Tools for RHEL 9 x86_64 (RPMs) openstack-17.1-deployment-tools-for-rhel-9-x86_64-rpms Packages to help director configure Ceph Storage nodes. This repository is included with standalone Ceph Storage subscriptions. If you use a combined OpenStack Platform and Ceph Storage subscription, use the openstack-17.1-for-rhel-9-x86_64-rpms repository. Red Hat OpenStack Platform for RHEL 9 x86_64 (RPMs) openstack-17.1-for-rhel-9-x86_64-rpms Packages to help director configure Ceph Storage nodes. This repository is included with combined Red Hat OpenStack Platform and Red Hat Ceph Storage subscriptions. If you use a standalone Red Hat Ceph Storage subscription, use the openstack-17.1-deployment-tools-for-rhel-9-x86_64-rpms repository. Red Hat Ceph Storage Tools 6 for RHEL 9 x86_64 (RPMs) rhceph-6-tools-for-rhel-9-x86_64-rpms Provides tools for nodes to communicate with the Ceph Storage cluster. Red Hat Fast Datapath for RHEL 9 (RPMS) fast-datapath-for-rhel-9-x86_64-rpms Provides Open vSwitch (OVS) packages for OpenStack Platform. If you are using OVS on Ceph Storage nodes, add this repository to the network interface configuration (NIC) templates. 1.6.3. Red Hat Satellite Server 6 considerations If you use Red Hat Satellite Server 6 to host RPMs and container images for your Red Hat OpenStack Platform (RHOSP) environment and you plan to use Satellite 6 to deliver content during the RHOSP 17.1 upgrade, the following must be true: Your Satellite Server hosts RHOSP 16.2 RPMs and container images. Note If the Red Hat Ceph Storage container image is hosted on a Satellite Server, then you must download a copy of the image to the undercloud before starting the Red Hat Ceph Storage upgrade. To copy this image, see Downloading Red Hat Ceph Storage containers to the undercloud from Satellite . You have registered all nodes in your RHOSP 16.2 environment to your Satellite Server. For example, you used an activation key linked to a RHOSP 16.2 content view to register nodes to RHOSP 16.2 content. Note If you are using an isolated environment where the undercloud does not have access to the internet, a known issue causes an upgrade from Red Hat OpenStack Platform 16.2 to 17.1 to fail. For a workaround, see the known issue for BZ2259891 in Known issues that might block an upgrade . Recommendations for RHOSP upgrades Enable and synchronize the necessary RPM repositories for both the RHOSP 16.2 undercloud and overcloud. This includes the necessary Red Hat Enterprise Linux (RHEL) 9.2 repositories. Create custom products on your Satellite Server to host container images for RHOSP 17.1. Create and promote a content view for RHOSP 17.1 upgrade and include the following content in the content view: RHEL 8 repositories: Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) Red Hat Enterprise Linux 8 for x86_64 - High Availability (RPMs) Red Hat Fast Datapath for RHEL 8 (RPMs) RHEL 9 repositories: Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs) All undercloud and overcloud RPM repositories, including RHEL 9.2 repositories. To avoid issues enabling the RHEL repositories, ensure that you include the correct version of the RHEL repositories, which is 9.2. RHOSP 17.1 container images. Associate an activation key with the RHOSP 17.1 content view that you have created for the RHOSP 17.1 upgrade. Check that no node has the katello-host-tools-fact-plugin package installed. The Leapp upgrade does not upgrade this package. Leaving this package on a RHEL 9.2 system causes subscription-manager to report errors. You can configure Satellite Server to host RHOSP 17.1 container images. To upgrade from RHOSP 16.2 to 17.1, you need the following container images: Container images that are hosted on the rhosp-rhel8 namespace: rhosp-rhel8/openstack-collectd rhosp-rhel8/openstack-nova-libvirt Container images that are hosted on the rhosp-rhel9 namespace. For information about configuring the rhosp-rhel9 namespace container images, see Preparing a Satellite server for container images in Installing and managing Red Hat OpenStack Platform with director . If you use a Red Hat Ceph Storage subscription and have configured director to use the overcloud-minimal image for Red Hat Ceph Storage nodes, on your Satellite Server you must create a content view and add the following RHEL 9.2 repositories to it: Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) For more information, see Importing Content and Managing Content Views in the Red Hat Satellite Managing Content guide.

Commands:
sudo podman exec -it collectd setfacl -R -m u:collectd:rwx /run/podman
openstack overcloud upgrade prepare --answers-file answer-upgrade.yaml / -r roles-data.yaml / -n networking-data.yaml / -e system_upgrade.yaml / -e upgrade_environment.yaml /
"FAILED ceph_assert(fs->mds_map.compat.compare(compat) == 0)"
infra_image="<LOCAL_SATELLITE_URL/pause:3.5>"
podman pod create
openstack overcloud external-upgrade run --skip-tags ceph_ansible_remote_tmp --stack <stack> --tags cephadm_adopt 2>&1
[WRN] BLUESTORE_NO_PER_POOL_OMAP
validation run -i inventory.yaml --group pre-upgrade --skiplist package-version
parameter_defaults: UpgradeInitCommand: | sudo subscription-manager repos --disable=* CephStorageUpgradeInitCommand: | sudo subscription-manager repos --disable=* if USD( grep -q 9.2 /etc/os-release ) then sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms --enable=rhel-9-for-x86_64-appstream-rpms --enable=openstack-17.1-deployment-tools-for-rhel-9-x86_64-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms sudo podman ps | grep -q ceph && subscription-manager repos --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms sudo subscription-manager release --set=9.2 else sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-tus-rpms --enable=rhel-8-for-x86_64-appstream-tus-rpms --enable=rhel-8-for-x86_64-highavailability-tus-rpms --enable=openstack-17.1-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms sudo podman ps | grep -q ceph && subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms sudo subscription-manager release --set=8.4 fi if USD(sudo podman ps | grep -q ceph ) then sudo dnf -y install cephadm fi
LeappRepoInitCommand: | subscription-manager repos --disable=* CephStorageUpgradeLeappCommandOptions: "--enablerepo=rhel-9-for-x86_64-baseos-rpms --enablerepo=rhel-9-for-x86_64-appstream-rpms --enablerepo=openstack-17.1-for-rhel-9-x86_64-rpms --enablerepo=fast-datapath-for-rhel-9-x86_64-rpms CephStorageLeappInitCommand: | subscription-manager repos --disable=* subscription-manager release --unset subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms --enable=rhel-9-for-x86_64-appstream-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms leapp answer --add --section check_vdo.confirm=True leapp answer --add --section check_vdo.no_vdo_devices=True
network_config: name: br-ctlplane type: ovs_bridge use_dhcp: false dns_servers: 172.16.0.1 10.0.0.1 domain: lab.example.com ovs_extra: "br-set-external-id br-ctlplane bridge-id br-ctlplane" addresses: ip_netmask: 192.168.24.1/24 ip_netmask: 192.168.24.2/32 <---------------------------| ip_netmask: 192.168.24.3/32 <---------------------------| members: type: interface name: eth0
Summary: Leapp has detected that the NVIDIA proprietary driver has been loaded, which also means the nouveau driver is blacklisted. If you upgrade now, you will end up without a graphical session, as the newer kernel won't be able to load the NVIDIA driver module and nouveau will still be blacklisted. Please uninstall the NVIDIA graphics driver before upgrading to make sure you have a graphical session after upgrading.
sudo dnf remove -y NVIDIA-vGPU-rhel-8.4-525.105.14.x86_64
rmmod nvidia_vgpu_vfio rmmod nvidia
openstack overcloud upgrade run --tag system_upgrade --limit <compute-0>
rhel-8-for-x86_64-appstream-tus-rpms
rhel-8-for-x86_64-baseos-tus-rpms
rhel-8-for-x86_64-highavailability-tus-rpms
fast-datapath-for-rhel-8-x86_64-rpms
rhel-9-for-x86_64-appstream-eus-rpms
rhel-9-for-x86_64-baseos-eus-rpms
satellite-client-6-for-rhel-9-x86_64-rpms
rhel-9-for-x86_64-appstream-eus-rpms
rhel-9-for-x86_64-baseos-eus-rpms