Chapter 4. Preparing for an overcloud upgrade You must complete some initial steps to prepare for the overcloud upgrade. 4.1. Preparing for overcloud service downtime The overcloud upgrade process disables the main control plane services at key points. You cannot use any overcloud services to create new resources when these key points are reached. Workloads that are running in the overcloud remain active during the upgrade process, which means instances continue to run during the upgrade of the control plane. During an upgrade of Compute nodes, these workloads can be live migrated to Compute nodes that are already upgraded. It is important to plan a maintenance window to ensure that no users can access the overcloud services during the upgrade. Affected by overcloud upgrade OpenStack Platform services Unaffected by overcloud upgrade Instances running during the upgrade Ceph Storage OSDs (backend storage for instances) Linux networking Open vSwitch networking Undercloud 4.2. Disabling fencing in the overcloud Before you upgrade the overcloud, ensure that fencing is disabled. When you upgrade the overcloud, you upgrade each Controller node individually to retain high availability functionality. If fencing is deployed in your environment, the overcloud might detect certain nodes as disabled and attempt fencing operations, which can cause unintended results. If you have enabled fencing in the overcloud, you must temporarily disable fencing for the duration of the upgrade to avoid any unintended results. Note When you complete the upgrade of your Red Hat OpenStack Platform environment, you must re-enable fencing in the overcloud. For more information about re-enabling fencing, see Re-enabling fencing in the overcloud . Procedure Log in to the undercloud host as the stack user. Source the stackrc undercloud credentials file: For each Controller node, log in to the Controller node and run the Pacemaker command to disable fencing: USD ssh tripleo-admin@<controller_ip> "sudo pcs property set stonith-enabled=false" Replace <controller_ip> with the IP address of a Controller node. You can find the IP addresses of your Controller nodes at /etc/hosts or /var/lib/mistral . If you use SBD fencing, disable SBD fencing on the pacemaker_remote nodes: Note Ensure that you take note of the original value of the watchdog timer device interval. You must reset the watchdog timer device interval to its original value after you upgrade the control plane nodes. For more information, see Re-enabling fencing in the overcloud . In the fencing.yaml environment file, set the EnableFencing parameter to false to ensure that fencing stays disabled during the upgrade process. Additional Resources Fencing Controller nodes with STONITH 4.3. Undercloud node database backup You can use the openstack undercloud backup --db-only command to create a standalone database backup that runs on the undercloud node. You can also use that backup to recover the state of the database in the event that it becomes corrupted. For more information about backing up the undercloud database, see Creating a standalone database backup of the undercloud nodes in Red Hat OpenStack Platform 17.1 Backing up and restoring the undercloud and control plane nodes . 4.4. Updating composable services in custom roles_data files This section contains information about new and deprecated composable services. All nodes The following services have been deprecated for all nodes. Remove them from all roles. Service Reason OS::TripleO::Services::CinderBackendDellEMCXTREMIOISCSI OS::TripleO::Services::CinderBackendDellPs OS::TripleO::Services::CinderBackendVRTSHyperScale Deprecated services. OS::TripleO::Services::Ec2Api Deprecated service. OS::TripleO::Services::Fluentd Deprecated in favour of OS::TripleO::Services::Rsyslog . OS::TripleO::Services::FluentdAlt Deprecated service. OS::TripleO::Services::Keepalived Deprecated service. OS::TripleO::Services::MistralApi OS::TripleO::Services::MistralEngine OS::TripleO::Services::MistralEventEngine OS::TripleO::Services::MistralExecutor Deprecated services. OS::TripleO::Services::NeutronLbaasv2Agent OS::TripleO::Services::NeutronLbaasv2Api OpenStack Networking (neutron) Load Balancing as a Service deprecated in favour of Octavia. OS::TripleO::Services::NeutronML2FujitsuCfab OS::TripleO::Services::NeutronML2FujitsuFossw Deprecated services. OS::TripleO::Services::NeutronSriovHostConfig Deprecated service. OS::TripleO::Services::NovaConsoleauth This service is removed. OS::TripleO::Services::Ntp Deprecated in favor of OS::TripleO::Services::Timesync . OS::TripleO::Services::OpenDaylightApi OS::TripleO::Services::OpenDaylightOvs OpenDaylight is no longer supported. OS::TripleO::Services::OpenShift::GlusterFS OS::TripleO::Services::OpenShift::Infra OS::TripleO::Services::OpenShift::Master OS::TripleO::Services::OpenShift::Worker Deprecated services. OS::TripleO::Services::PankoApi The OpenStack Telemetry services are deprecated in favor of Service Telemetry Framework (STF) for metrics and monitoring. The legacy telemetry services are only available in RHOSP 17.1 to help facilitate the transition to STF and will be removed in a future version of RHOSP. OS::TripleO::Services::Rear Deprecated service. OS::TripleO::Services::SaharaApi OS::TripleO::Services::SaharaEngine Deprecated services. OS::TripleO::Services::SensuClient OS::TripleO::Services::SensuClientAlt Deprecated services. OS::TripleO::Services::SkydiveAgent OS::TripleO::Services::SkydiveAnalyzer Skydive is no longer supported. OS::TripleO::Services::Tacker Tacker is no longer supported. OS::TripleO::Services::TripleoUI Deprecated service. OS::TripleO::Services::UndercloudMinionMessaging OS::TripleO::Services::UndercloudUpgradeEphemeralHeat Deprecated services. OS::TripleO::Services::Zaqar Deprecated service. Controller nodes The following services are new for Controller nodes. Add them to your Controller role. Service Reason OS::TripleO::Services::GlanceApiInternal Service for the internal instance of the Image service (glance) API to provide location data to administrators and services that require it, such as the Block Storage service (cinder) and the Compute service (nova). Compute nodes By default, 17.1 Compute nodes run the OS::TripleO::Services::NovaLibvirt service. However, if you perform the RHOSP upgrade with the Compute nodes running the OS::TripleO::Services::NovaLibvirt service, the virtual machine instances appear as shut off. To prevent this issue, all Compute nodes that are on RHEL 8.4 must run the OS::TripleO::Services::NovaLibvirtLegacy service, and the container image must be based on UBI-8. After the RHOSP upgrade, if you want to upgrade your Compute nodes to RHEL 9.2, your Compute nodes must run the OS::TripleO::Services::NovaLibvirt service and the container image must be based on UBI-9, or your virtual machine instances appear as shut off. For more information about upgrading the operating system on Compute nodes, see Upgrading all Compute nodes to RHEL 9.2 and Upgrading Compute nodes to a Multi-RHEL environment . 4.5. Checking custom Puppet parameters If you use the ExtraConfig interfaces for customizations of Puppet parameters, Puppet might report duplicate declaration errors during the upgrade. This is due to changes in the interfaces provided by the puppet modules themselves. This procedure shows how to check for any custom ExtraConfig hieradata parameters in your environment files. Note If your environment uses LDAP backends, remove the following deprecated parameters from the keystone_domain_specific_ldap_backend.yaml environment file to prevent overcloud upgrade failure: user_allow_create user_allow_update user_allow_delete group_allow_create group_allow_update group_allow_delete For more information about removing these parameters, see the Red Hat Knowledgebase solution Overcloud upgrade to RHOSP 17.1 failed due to Keystone error when deprecated ldap related options are present in templates . Procedure Select an environment file and the check if it has an ExtraConfig parameter: If the results show an ExtraConfig parameter for any role (e.g. ControllerExtraConfig ) in the chosen file, check the full parameter structure in that file. If the parameter contains any puppet Hierdata with a SECTION/parameter syntax followed by a value , it might have been been replaced with a parameter with an actual Puppet class. For example: Check the director's Puppet modules to see if the parameter now exists within a Puppet class. For example: If so, change to the new interface. The following are examples to demonstrate the change in syntax: Example 1: Changes to: Example 2: Changes to: 4.6. Firewall rule change In Red Hat OpenStack Platform (RHOSP) 17.1.4, iptables rules were replaced with nftables rules. If your RHOSP templates include firewall rules, for example, tripleo::tripleo_firewall::firewall_rules , you must redefine them by using the ExtraFirewallRules parameter. For more information about using the ExtraFirewallRules parameter, see Adding services to the overcloud firewall in Hardening Red Hat OpenStack Platform . 4.7. Final review before upgrade Complete a final check of all preparation steps before you begin the upgrade. 4.7.1. Upgrade command overview The upgrade process involves different commands that you run at certain stages of the process. Important This section only contains information about each command. You must run these commands in a specific order and provide options specific to your overcloud. Wait until you receive instructions to run these commands at the appropriate step. 4.7.1.1. openstack overcloud upgrade prepare This command performs the initial preparation steps for the overcloud upgrade, which includes replacing the current overcloud plan on the undercloud with the new OpenStack Platform 17.1 overcloud plan and your updated environment files. This command functions similar to the openstack overcloud deploy command and uses many of the same options. Before you run the openstack overcloud upgrade prepare command, you must perform the overcloud adoption. For more information about overcloud adoption, see Performing the overcloud adoption and preparation . 4.7.1.2. openstack overcloud upgrade run This command performs the upgrade process. Director creates a set of Ansible playbooks based on the new OpenStack Platform 17.1 overcloud plan and runs the fast forward tasks on the entire overcloud. This includes running the upgrade process through each OpenStack Platform version from 16.2 to 17.1. In addition to the standard upgrade process, this command can perform a Leapp upgrade of the operating system on overcloud nodes. Run these tasks using the --tags option. Upgrade task tags for Leapp system_upgrade Task that combines tasks from system_upgrade_prepare , system_upgrade_run , and system_upgrade_reboot . system_upgrade_prepare Tasks to prepare for the operating system upgrade with Leapp. system_upgrade_run Tasks to run Leapp and upgrade the operating system. system_upgrade_reboot Tasks to reboot a system and complete the operating system upgrade. 4.7.1.3. openstack overcloud external-upgrade run This command performs upgrade tasks outside the standard upgrade process. Director creates a set of Ansible playbooks based on the new OpenStack Platform 17.1 overcloud plan and you run specific tasks using the --tags option. External task tags for container management container_image_prepare Tasks for pulling container images to the undercloud registry and preparing the images for the overcloud to use. 4.7.2. Upgrade Parameters You can modify the behavior of the upgrade process with upgrade parameters. Parameter Description UpgradeInitCommand Command or script snippet to run on all overcloud nodes to initialize the upgrade process. For example, a repository switch. UpgradeInitCommonCommand Common commands required by the upgrades process. This should not normally be modified by the operator and is set and unset in the major-upgrade-composable-steps.yaml and major-upgrade-converge.yaml environment files. UpgradeLeappCommandOptions Additional command line options to append to the Leapp command. UpgradeLeappDebug Print debugging output when running Leapp. The default value is false . UpgradeLeappDevelSkip Skip Leapp checks by setting env variables when running Leapp in development/testing. For example, LEAPP_DEVEL_SKIP_RHSM=1. UpgradeLeappEnabled Use Leapp for operating system upgrade. The default value is false . UpgradeLeappPostRebootDelay Maximum (seconds) to wait for machine to reboot and respond to a test command. The default value is 120 . UpgradeLeappRebootTimeout Timeout (seconds) for the OS upgrade phase via Leapp. The default value is 3600 . UpgradeLeappToInstall List of packages to install after Leapp upgrade. UpgradeLeappToRemove List of packages to remove during Leapp upgrade. 4.7.3. Custom files to include in your deployment If any overcloud nodes in your deployment are dedicated Object Storage (swift) nodes, you must copy the default roles_data.yaml file and edit ObjectStorage to remove deprecated_server_resource_name: 'SwiftStorage' . Then use the --roles-file option to pass the file to the openstack overcloud upgrade prepare command. 4.7.4. New environment files to include with your deployment In addition to your regular overcloud environment files, you must include new environment files to facilitate the upgrade to Red Hat OpenStack Platform (RHOSP) 17.1. File Notes /home/stack/templates/upgrades-environment.yaml This file contains the parameters specific to the upgrade. This file is necessary only for the duration of the upgrade. /home/stack/containers-prepare-parameter.yaml The file that contains the source and preparation steps. This is the same file that you use with the undercloud upgrade. /home/stack/templates/ceph.yaml This file contains the parameters that Ceph Storage is required to override. Add these files to the end of your environment file listing when you run the following commands: openstack overcloud upgrade prepare openstack overcloud deploy 4.7.5. Environment files to remove from your deployment Remove any environment files specific to your OpenStack Platform Red Hat OpenStack Platform 16.2: Red Hat OpenStack Platform 16.2 container image list Red Hat OpenStack Platform 16.2 Customer Portal or Satellite rhel-registration scripts Remove these files from the list of environment files you include when you run the following commands: openstack overcloud upgrade prepare openstack overcloud deploy 4.7.6. Upgrading IPA services If TLS everywhere is enabled in your environment, add an additional permission to the Nova Host Manager role to allow the creation of DNS zone entries. Prerequisites Check whether the Nova Host Management permission is included in your environment: If you already have this permission, skip the following procedure. Procedure Log in to the undercloud host as the stack user. Source the stackrc undercloud credentials file: Add the Nova Host Management permission: Create an environment file called ipa_environment.yaml and include the following configuration: Save the environment file. 4.7.7. Upgrade checklist Use the following checklist to determine your readiness to upgrade the overcloud: Item Complete Validated a working overcloud. Y / N Performed a Relax-and-Recover (ReaR) backup of the overcloud control plane. For more information, see Red Hat OpenStack Platform 16.2 Backing up and restoring the undercloud and control plane nodes . Y / N Created a backup of the database that runs on the undercloud node. For more information, see Creating a backup of the undercloud node in Red Hat OpenStack Platform 17.1 Backing up and restoring the undercloud and control plane nodes . Y / N Updated your registration details to Red Hat OpenStack Platform 17.1 repositories and converted your environment file to use the Ansible-based method. Y / N Updated your network configuration templates. Y / N Updated your environment file list with new environment files for Red Hat OpenStack Platform 17.1. Y / N Optional: If your deployment includes dedicated Object Storage (swift) nodes: Copied the roles_data.yaml file, removed deprecated_server_resource_name: 'SwiftStorage' , and passed the file to the openstack overcloud upgrade prepare command. Y / N Removed old environment files only relevant to Red Hat OpenStack Platform 16.2, such as old Red Hat registration and container image location files. Y / N

Commands:
source ~/stackrc
ssh tripleo-admin@<controller_ip> "sudo pcs property set stonith-enabled=false"
pcs property set stonith-watchdog-timeout=0
grep ExtraConfig ~/templates/custom-config.yaml
parameter_defaults: ExtraConfig: neutron::config::dhcp_agent_config: 'DEFAULT/dnsmasq_local_resolv': value: 'true'
grep dnsmasq_local_resolv
parameter_defaults: ExtraConfig: neutron::config::dhcp_agent_config: 'DEFAULT/dnsmasq_local_resolv': value: 'true'
parameter_defaults: ExtraConfig: neutron::agents::dhcp::dnsmasq_local_resolv: true
parameter_defaults: ExtraConfig: ceilometer::config::ceilometer_config: 'oslo_messaging_rabbit/rabbit_qos_prefetch_count': value: '32'
parameter_defaults: ExtraConfig: oslo::messaging::rabbit::rabbit_qos_prefetch_count: '32'
ipa privilege-show "Nova Host Management"
source ~/stackrc
kinit admin ipa privilege-add-permission 'Nova Host Management' --permission 'System: Modify Realm Domains'
resource_registry: OS::TripleO::Services::IpaClient: /usr/share/openstack-tripleo-heat-templates/deployment/ipa/ipaservices-baremetal-ansible.yaml parameter_defaults: IdMServer: USDIPA_FQDN IdMDomain: USDIPA_DOMAIN IdMInstallClientPackages: False