Chapter 11. Updating the overcloud for director Operator After you update the openstackclient pod, update the overcloud by running the overcloud and container image preparation deployments, updating your nodes, and running the overcloud update converge deployment. During a minor update, the control plane API is available. 11.1. Preparing director Operator for a minor update The Red Hat OpenStack Platform (RHOSP) minor update process workflow: Prepare your environment for the RHOSP minor update. Update the openstackclient pod image to the latest OpenStack 16.2.z version. Update the overcloud to the latest OpenStack 16.2.z version. Update all Red Hat Ceph Storage services. Run the convergence deployment to refresh your overcloud stack. 11.1.1. Locking the environment to a Red Hat Enterprise Linux release Red Hat OpenStack Platform (RHOSP) 16.2 is supported on Red Hat Enterprise Linux (RHEL) 8.4. Before you perform the update, lock the overcloud repositories to the RHEL 8.4 release to avoid upgrading the operating system to a newer minor release. Procedure Copy the rhsm.yaml file to openstackclient : Open a remote shell on the openstackclient pod: Open the rhsm.yaml file and check if your subscription management configuration includes the rhsm_release parameter. If the rhsm_release parameter is not present, add it and set it to 8.4 : Save the overcloud subscription management environment file. Create a playbook that contains a task to lock the operating system version to RHEL 8.4 on all nodes: Run the ansible playbook on the openstackclient pod: Use the --limit option to apply the content to all RHOSP nodes. Do not run this playbook against Red Hat Ceph Storage nodes because you are probably using a different subscription for these nodes. Note To manually lock a node to a version, log in to the node and run the subscription-manager release command: 11.1.2. Changing to Extended Update Support (EUS) repositories Your Red Hat OpenStack Platform (RHOSP) subscription includes repositories for Red Hat Enterprise Linux (RHEL) 8.4 Extended Update Support (EUS). The EUS repositories include the latest security patches and bug fixes for RHEL 8.4. Switch to the following repositories before you perform an update. Table 11.1. EUS repositories for RHEL 8.4 Standard repository EUS repository rhel-8-for-x86_64-baseos-rpms rhel-8-for-x86_64-baseos-eus-rpms rhel-8-for-x86_64-appstream-rpms rhel-8-for-x86_64-appstream-eus-rpms rhel-8-for-x86_64-highavailability-rpms rhel-8-for-x86_64-highavailability-eus-rpms Important You must use EUS repositories to retain compatibility with a specific version of Podman. Later versions of Podman are untested with RHOSP 16.2 and can cause unexpected results. Prerequisites Copy the rhsm.yaml file for the openstackclient pod to the /home/cloud-admin directory. Procedure Open a remote shell on the openstackclient pod: Open the rhsm.yaml file and check the rhsm_repos parameter in your subscription management configuration. If this parameter does not include the EUS repositories, change the relevant repositories to the EUS versions: parameter_defaults: RhsmVars: rhsm_repos: - rhel-8-for-x86_64-baseos-eus-rpms - rhel-8-for-x86_64-appstream-eus-rpms - rhel-8-for-x86_64-highavailability-eus-rpms - ansible-2.9-for-rhel-8-x86_64-rpms - openstack-16.2-for-rhel-8-x86_64-rpms - rhceph-4-tools-for-rhel-8-x86_64-rpms - fast-datapath-for-rhel-8-x86_64-rpms Save the overcloud subscription management environment file. Create a playbook that contains a task to set the repositories to RHEL 8.4 EUS on all nodes: Run the change_eus.yaml playbook: USD ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory /home/cloud-admin/change_eus.yaml --limit Controller,Compute Use the --limit option to apply the content to all RHOSP nodes. Do not run this playbook against Red Hat Ceph Storage nodes because they use a different subscription. 11.1.3. Updating Red Hat Openstack Platform and Ansible repositories Update your repositories to use Red Hat OpenStack Platform (RHOSP) 16.2 and Ansible 2.9 packages. For more information, see Overcloud repositories . Prerequisites You have copied the rhsm.yaml file for the openstackclient pod to the /home/cloud-admin directory. Procedure Open a remote shell on the openstackclient pod: Open the rhsm.yaml file and check the rhsm_repos parameter in your subscription management configuration. If the rhsm_repos parameter is using the RHOSP 16.1 and Ansible 2.8 repositories, change the repository to the correct versions: parameter_defaults: RhsmVars: rhsm_repos: - rhel-8-for-x86_64-baseos-eus-rpms - rhel-8-for-x86_64-appstream-eus-rpms - rhel-8-for-x86_64-highavailability-eus-rpms - ansible-2.9-for-rhel-8-x86_64-rpms - openstack-16.2-for-rhel-8-x86_64-rpms - fast-datapath-for-rhel-8-x86_64-rpms Save the overcloud subscription management environment file. Create a playbook that contains a task to set the repositories to RHOSP {osp_curr_ver} on all RHOSP nodes: USD cat > ~/update_rhosp_repos.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: change osp repos command: subscription-manager repos --disable=openstack-16.1-for-rhel-8-x86_64-rpms --enable=openstack-16.2-for-rhel-8-x86_64-rpms --disable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms become: true EOF Run the update_rhosp_repos.yaml playbook: USD ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory /home/cloud-admin/update_rhosp_repos.yaml --limit Controller,Compute Use the --limit option to apply the content to all RHOSP nodes. Do not run this playbook against Red Hat Ceph Storage nodes because they use a different subscription. Create a playbook that contains a task to set the repositories to RHOSP {osp_curr_ver} on all Red Hat Ceph Storage nodes: USD cat > ~/update_ceph_repos.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: change ceph repos command: subscription-manager repos --disable=openstack-16-deployment-tools-for-rhel-8-x86_64-rpms --enable=openstack-16.2-deployment-tools-for-rhel-8-x86_64-rpms --disable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms become: true EOF Run the update_ceph_repos.yaml playbook: USD ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory /home/cloud-admin/update_ceph_repos.yaml --limit CephStorage Use the --limit option to apply the content to Red Hat Ceph Storage nodes. 11.1.4. Setting the container-tools version Set the container-tools module to version 2.0 to ensure you use the correct package versions on all nodes. Procedure Open a remote shell on the openstackclient pod: Create a playbook that contains a task to set the container-tools module to version 3.0 on all nodes: USD cat > ~/container-tools.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: disable default dnf module for container-tools command: dnf module reset container-tools become: true - name: set dnf module for container-tools:3.0 command: dnf module enable -y container-tools:3.0 become: true - name: disable dnf module for virt:8.2 command: dnf module disable -y virt:8.2 become: true - name: set dnf module for virt:rhel command: dnf module enable -y virt:rhel become: true EOF Run the container-tools.yaml playbook against all nodes: USD ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory ~/container-tools.yaml 11.1.5. Updating the container image preparation file The container preparation file is the file that contains the ContainerImagePrepare parameter. You use this file to define the rules for obtaining container images for the overcloud. Before you update your environment, check the file to ensure that you obtain the correct image versions. Procedure Edit the container preparation file. The default name for this file is usually containers-prepare-parameter.yaml . Check the tag parameter is set to 16.2 for each rule set: parameter_defaults: ContainerImagePrepare: - push_destination: true set: ... tag: '16.2' tag_from_label: '{version}-{release}' Note If you do not want to use a specific tag for the update, such as 16.2 or 16.2.2 , remove the tag key-value pair and specify tag_from_label only. This uses the installed Red Hat OpenStack Platform version to determine the value for the tag to use as part of the update process. Save this file. 11.1.6. Disabling fencing in the overcloud Before you update the overcloud, ensure that fencing is disabled. If fencing is deployed in your environment during the Controller nodes update process, the overcloud might detect certain nodes as disabled and attempt fencing operations, which can cause unintended results. If you have enabled fencing in the overcloud, you must temporarily disable fencing for the duration of the update to avoid any unintended results. Procedure Open a remote shell on the openstackclient pod: Log in to a Controller node and run the Pacemaker command to disable fencing: Replace <controller-0.ctlplane> with the name of your Controller node. In the fencing.yaml environment file, set the EnableFencing parameter to false to ensure that fencing stays disabled during the update process. Additional Resources Fencing Controller nodes with STONITH 11.2. Running the overcloud update preparation for director Operator To prepare the overcloud for the update process, generate an update prepare configuration, which creates updated ansible playbooks and prepares the nodes for the update. Procedure Modify the heat parameter ConfigMap called fencing.yaml , to disable fencing for the duration of the update: Create an OpenStackConfigGenerator resource called osconfiggenerator-update-prepare.yaml : Apply the configuration: Wait until the update preparation process completes. 11.3. Running the container image preparation for director Operator Before you can update the overcloud, you must prepare all container image configurations that are required for your environment. To complete the container image preparation, you must run the overcloud deployment against tasks that have the container_image_prepare tag. Procedure Create an osdeploy job called osdeploy-container-image-prepare.yaml : Apply the configuration: 11.4. Optional: Updating the ovn-controller container on all overcloud servers If you deployed your overcloud with the Modular Layer 2 Open Virtual Network mechanism driver (ML2/OVN), update the ovn-controller container to the latest RHOSP 16.2 version. The update occurs on every overcloud server that runs the ovn-controller container. Important The following procedure updates the ovn-controller containers on servers that are assigned the Compute role before it updates the ovn-northd service on servers that are assigned the Controller role. If you accidentally updated the ovn-northd service before following this procedure, you might not be able to reach your virtual machines or create new virtual machines or virtual networks. The following procedure restores connectivity. Procedure Create an osdeploy job called osdeploy-ovn-update.yaml : Apply the configuration: Wait until the ovn-controller container update completes. 11.5. Updating all Controller nodes on director Operator Update all the Controller nodes to the latest Red Hat OpenStack Platform (RHOSP) 16.2 version. Important Until BZ#1872404 is resolved, for nodes based on composable roles, you must update the Database role first, before you can update Controller , Messaging , Compute , Ceph , and other roles. Procedure Create an osdeploy job called osdeploy-controller-update.yaml : Apply the configuration: Wait until the Controller node update completes. 11.6. Updating all Compute nodes on director Operator Update all Compute nodes to the latest Red Hat OpenStack Platform (RHOSP) 16.2 version. To update Compute nodes, run a deployment with the limit: Compute option to restrict operations to the Compute nodes only. Procedure Create an osdeploy job called osdeploy-compute-update.yaml : Apply the configuration: Wait until the Compute node update completes. 11.7. Updating all HCI Compute nodes on director Operator Update the Hyperconverged Infrastructure (HCI) Compute nodes to the latest Red Hat OpenStack Platform (RHOSP) 16.2 version. To update the HCI Compute nodes, run a deployment with the limit: ComputeHCI option to restrict operations to only the HCI nodes. You must also run a deployment with the mode: external-update and tags: ["ceph"] options to perform an update to a containerized Red Hat Ceph Storage 4 cluster. Procedure Create an osdeploy job called osdeploy-computehci-update.yaml : Apply the configuration: Wait until the ComputeHCI node update completes. Create an osdeploy job called osdeploy-ceph-update.yaml : Apply the configuration: Wait until the Red Hat Ceph Storage node update completes. 11.8. Updating all Red Hat Ceph Storage nodes on director Operator Update the Red Hat Ceph Storage nodes to the latest Red Hat OpenStack Platform (RHOSP) 16.2 version. Important RHOSP 16.2 is supported on RHEL 8.4. However, hosts that are mapped to the CephStorage role update to the latest major RHEL release. For more information, see Red Hat Ceph Storage: Supported configurations . Procedure Create an osdeploy job called osdeploy-cephstorage-update.yaml : Apply the configuration: Wait until the Red Hat Ceph Storage node update completes. Create an osdeploy job called osdeploy-ceph-update.yaml : Apply the configuration: Wait until the Red Hat Ceph Storage node update completes. 11.9. Performing online database updates on director Operator Some overcloud components require an online update or migration of their databases tables. Online database updates apply to the following components: OpenStack Block Storage (cinder) OpenStack Compute (nova) Procedure Create an osdeploy job called osdeploy-online-migration.yaml : Apply the configuration: 11.10. Finalizing the update To finalize the update to the latest Red Hat OpenStack Platform 16.2 version, you must update the overcloud generated configuration. This ensures that the stack resource structure aligns with a regular deployment of OSP 16.2 and you can perform standard overcloud deployments in the future. Procedure Re-enable fencing in the fencing.yaml environment file: Regenerate the default configuration, ensuring that lifecycle/update-prepare.yaml is not included in the heatEnvs. For more information, see Configuring overcloud software with the director Operator . Delete OpenStackConfigGenerator, ConfigVersion, and configuration deployment resources. Replace <type> with the type of resource to delete. Replace <name> with the name of the resource to delete. Wait until the update finalization completes.

Commands:
oc cp rhsm.yaml openstackclient:/home/cloud-admin/rhsm.yaml
oc rsh openstackclient
parameter_defaults: RhsmVars: ... rhsm_username: "myusername" rhsm_password: "p@55w0rd!" rhsm_org_id: "1234567" rhsm_pool_ids: "1a85f9223e3d5e43013e3d6e8ff506fd" rhsm_method: "portal" rhsm_release: "8.4"
cat > ~/set_release.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: set release to 8.4 command: subscription-manager release --set=8.4 become: true EOF
ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory /home/cloud-admin/set_release.yaml --limit Controller,Compute
sudo subscription-manager release --set=8.4
oc rsh openstackclient
parameter_defaults: RhsmVars: rhsm_repos: - rhel-8-for-x86_64-baseos-eus-rpms - rhel-8-for-x86_64-appstream-eus-rpms - rhel-8-for-x86_64-highavailability-eus-rpms - ansible-2.9-for-rhel-8-x86_64-rpms - openstack-16.2-for-rhel-8-x86_64-rpms - rhceph-4-tools-for-rhel-8-x86_64-rpms - fast-datapath-for-rhel-8-x86_64-rpms
cat > ~/change_eus.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: change to eus repos command: subscription-manager repos --disable=rhel-8-for-x86_64-baseos-rpms --disable=rhel-8-for-x86_64-appstream-rpms --disable=rhel-8-for-x86_64-highavailability-rpms --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms become: true EOF
ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory /home/cloud-admin/change_eus.yaml --limit Controller,Compute
oc rsh openstackclient
parameter_defaults: RhsmVars: rhsm_repos: - rhel-8-for-x86_64-baseos-eus-rpms - rhel-8-for-x86_64-appstream-eus-rpms - rhel-8-for-x86_64-highavailability-eus-rpms - ansible-2.9-for-rhel-8-x86_64-rpms - openstack-16.2-for-rhel-8-x86_64-rpms - fast-datapath-for-rhel-8-x86_64-rpms
cat > ~/update_rhosp_repos.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: change osp repos command: subscription-manager repos --disable=openstack-16.1-for-rhel-8-x86_64-rpms --enable=openstack-16.2-for-rhel-8-x86_64-rpms --disable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms become: true EOF
ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory /home/cloud-admin/update_rhosp_repos.yaml --limit Controller,Compute
cat > ~/update_ceph_repos.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: change ceph repos command: subscription-manager repos --disable=openstack-16-deployment-tools-for-rhel-8-x86_64-rpms --enable=openstack-16.2-deployment-tools-for-rhel-8-x86_64-rpms --disable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms become: true EOF
ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory /home/cloud-admin/update_ceph_repos.yaml --limit CephStorage
oc rsh openstackclient
cat > ~/container-tools.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: disable default dnf module for container-tools command: dnf module reset container-tools become: true - name: set dnf module for container-tools:3.0 command: dnf module enable -y container-tools:3.0 become: true - name: disable dnf module for virt:8.2 command: dnf module disable -y virt:8.2 become: true - name: set dnf module for virt:rhel command: dnf module enable -y virt:rhel become: true EOF
ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory ~/container-tools.yaml
parameter_defaults: ContainerImagePrepare: - push_destination: true set: tag: '16.2' tag_from_label: '{version}-{release}'
oc rsh openstackclient
ssh <controller-0.ctlplane> "sudo pcs property set stonith-enabled=false"
parameter_defaults: EnableFencing: false
cat <<EOF > osconfiggenerator-update-prepare.yaml apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackConfigGenerator metadata: name: "update" namespace: openstack spec: gitSecret: git-secret heatEnvs: - lifecycle/update-prepare.yaml heatEnvConfigMap: heat-env-config-update tarballConfigMap: tripleo-tarball-config-update EOF
oc apply -f osconfiggenerator-update-prepare.yaml
cat <<EOF > osdeploy-container-image-prepare.yaml apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackDeploy metadata: name: container_image_prepare spec: configVersion: <config_version> configGenerator: update mode: external-update advancedSettings: tags: - container_image_prepare EOF
oc apply -f osdeploy-container-image-prepare.yaml
cat <<EOF > osdeploy-ovn-update.yaml apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackDeploy metadata: name: ovn-update spec: configVersion: <config_version> configGenerator: update mode: update advancedSettings: tags: - ovn EOF
oc apply -f osdeploy-ovn-update.yaml
cat <<EOF > osdeploy-controller-update.yaml apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackDeploy metadata: name: controller-update spec: configVersion: <config_version> configGenerator: update mode: update advancedSettings: limit: Controller EOF
oc apply -f osdeploy-controller-update.yaml
cat <<EOF > osdeploy-compute-update.yaml apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackDeploy metadata: name: compute-update spec: configVersion: <config_version> configGenerator: update mode: update advancedSettings: limit: Compute EOF
oc apply -f osdeploy-compute-update.yaml
cat <<EOF > osdeploy-computehci-update.yaml apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackDeploy metadata: name: computehci-update spec: configVersion: <config_version> configGenerator: update mode: update advancedSettings: limit: ComputeHCI EOF
oc apply -f osdeploy-computehci-update.yaml
cat <<EOF > osdeploy-ceph-update.yaml apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackDeploy metadata: name: ceph-update spec: configVersion: <config_version> configGenerator: update mode: external-update advancedSettings: tags: - ceph EOF
oc apply -f osdeploy-ceph-update.yaml
cat <<EOF > osdeploy-cephstorage-update.yaml apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackDeploy metadata: name: cephstorage-update spec: configVersion: <config_version> configGenerator: update mode: update advancedSettings: limit: CephStorage EOF
oc apply -f osdeploy-cephstorage-update.yaml
cat <<EOF > osdeploy-ceph-update.yaml apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackDeploy metadata: name: ceph-update spec: configVersion: <config_version> configGenerator: update mode: external-update advancedSettings: tags: - ceph EOF
oc apply -f osdeploy-ceph-update.yaml
cat <<EOF > osdeploy-online-migration.yaml apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackDeploy metadata: name: online-migration spec: configVersion: <config_version> configGenerator: update mode: external-update advancedSettings: tags: - online_upgrade EOF
oc apply -f osdeploy-online-migration.yaml
parameter_defaults: EnableFencing: true
oc delete <type> <name>