Chapter 1. Red Hat OpenStack Services on OpenShift 18.0 adoption overview Adoption is the process of migrating a Red Hat OpenStack Platform (RHOSP) 17.1 overcloud to a Red Hat OpenStack Services on OpenShift 18.0 data plane. To ensure that you understand the entire adoption process and how to sufficiently prepare your RHOSP environment, review the prerequisites, adoption process, and post-adoption tasks. Important It is important to read the whole adoption guide before you start the adoption. You should form an understanding of the procedure, prepare the necessary configuration snippets for each service ahead of time, and test the procedure in a representative test environment before you adopt your main environment. 1.1. Adoption limitations Before you proceed with the adoption, check which features are considered a Technology Preview or are unsupported. Technology Preview The following features are considered a Technology Preview and have not been tested within the context of the Red Hat OpenStack Services on OpenShift adoption: Bare Metal Provisioning service (ironic) NFS Ganesha back end for Shared File Systems service (manila) iSCSI, NFS, and FC-based drivers for Block Storage service (cinder) The following Compute service (nova) features: Compute hosts with /var/lib/nova/instances on NFS NUMA aware vswitches PCI passthrough by flavor SR-IOV trusted virtual functions RX and TX queue sizes vGPU Virtio multiqueue Emulated virtual Trusted Platform Module (vTPM) UEFI AMD SEV Direct download from Rados Block Device (RBD) File-backed memory Provider.yaml Unsupported features The adoption process does not support the following features: Red Hat OpenStack Platform (RHOSP) 17.1 multi-cell deployments instanceHA DCN Designate Loadbalancer service (octavia) BGP IPv6 NFS back end for ephemeral Compute service virtual machine instances storage Adopting a FIPS environment The Key Manager service only supports the simple crypto plug-in The Block Storage service only supports RBD back-end adoption 1.2. Known issues Review the following known issues that might affect a successful adoption: Adoption of combined Controller/Networker nodes not verified Red Hat has not verified a process for adoption of a Red Hat OpenStack Platform (RHOSP) 17.1 environment where Controller and Networker roles are composed together on Controller nodes. If your RHOSP 17.1 environment does use combined Controller/Networker roles on the Controller nodes, the documented adoption process will not produce the expected results. Adoption of RHOSP 17.1 environments that use dedicated Networker nodes has been verified to work as documented. 1.3. Adoption prerequisites Before you begin the adoption procedure, complete the following prerequisites: Planning information Review the Adoption limitations . Review the Red Hat OpenShift Container Platform (RHOCP) requirements, data plane node requirements, Compute node requirements, and so on. For more information, see Planning your deployment . Review the adoption-specific networking requirements. For more information, see Configuring the network for the RHOSO deployment . Review the adoption-specific storage requirements. For more information, see Storage requirements . Review how to customize your deployed control plane with the services that are required for your environment. For more information, see Customizing the Red Hat OpenStack Services on OpenShift deployment . Familiarize yourself with the following RHOCP concepts that are used during adoption: Overview of nodes About node selectors Machine configuration overview Back-up information Back up your Red Hat OpenStack Platform (RHOSP) 17.1 environment by using one of the following options: The Relax-and-Recover tool. For more information, see Backing up the undercloud and the control plane nodes by using the Relax-and-Recover tool in Backing up and restoring the undercloud and control plane nodes . The Snapshot and Revert tool. For more information, see Backing up your Red Hat OpenStack Platform cluster by using the Snapshot and Revert tool in Backing up and restoring the undercloud and control plane nodes . A third-party backup and recovery tool. For more information about certified backup and recovery tools, see the Red Hat Ecosystem Catalog . Back up the configuration files from the RHOSP services and director on your file system. For more information, see Pulling the configuration from a director deployment . Compute Upgrade your Compute nodes to Red Hat Enterprise Linux 9.2. For more information, see Upgrading all Compute nodes to RHEL 9.2 in Framework for upgrades (16.2 to 17.1) . Perform a minor update to the latest RHOSP version. For more information, see Performing a minor update of Red Hat OpenStack Platform . If the systemd-container package is not installed on your Compute hosts, install it by using the following command: USD sudo dnf -y install systemd-container Reboot all hypervisors one by one to activate the systemd-container package. To avoid interrupting your workloads during the reboot, live migrate virtual machine instances before rebooting a node. For more information, see Rebooting Compute nodes in Performing a minor update of Red Hat OpenStack Platform . ML2/OVS If you use the Modular Layer 2 plug-in with Open vSwitch mechanism driver (ML2/OVS), migrate it to the Modular Layer 2 plug-in with Open Virtual Networking (ML2/OVN) mechanism driver. For more information, see Migrating to the OVN mechanism driver . Tools Install the oc command line tool on your workstation. Install the podman command line tool on your workstation. RHOSP 17.1 release The RHOSP 17.1 cloud is updated to the latest minor version of the 17.1 release. RHOSP 17.1 hosts All control plane and data plane hosts of the RHOSP 17.1 cloud are up and running, and continue to run throughout the adoption procedure. 1.4. Guidelines for planning the adoption When planning to adopt a Red Hat OpenStack Services on OpenShift (RHOSO) 18.0 environment, consider the scope of the change. An adoption is similar in scope to a data center upgrade. Different firmware levels, hardware vendors, hardware profiles, networking interfaces, storage interfaces, and so on affect the adoption process and can cause changes in behavior during the adoption. Review the following guidelines to adequately plan for the adoption and increase the chance that you complete the adoption successfully: Important All commands in the adoption documentation are examples. Do not copy and paste the commands without understanding what the commands do. To minimize the risk of an adoption failure, reduce the number of environmental differences between the staging environment and the production sites. If the staging environment is not representative of the production sites, or a staging environment is not available, then you must plan to include contingency time in case the adoption fails. Review your custom Red Hat OpenStack Platform (RHOSP) service configuration at every major release. Every major release upgrades through multiple OpenStack releases. Each major release might deprecate configuration options or change the format of the configuration. Prepare a Method of Procedure (MOP) that is specific to your environment to reduce the risk of variance or omitted steps when running the adoption process. You can use representative hardware in a staging environment to prepare a MOP and validate any content changes. Include a cross-section of firmware versions, additional interface or device hardware, and any additional software in the representative staging environment to ensure that it is broadly representative of the variety that is present in the production environments. Ensure that you validate any Red Hat Enterprise Linux update or upgrade in the representative staging environment. Use Satellite for localized and version-pinned RPM content where your data plane nodes are located. In the production environment, use the content that you tested in the staging environment. 1.5. Adoption process overview Familiarize yourself with the steps of the adoption process and the optional post-adoption tasks. Main adoption process Migrate TLS everywhere (TLS-e) to the Red Hat OpenStack Services on OpenShift (RHOSO) deployment . Migrate your existing databases to the new control plane . Adopt your Red Hat OpenStack Platform 17.1 control plane services to the new RHOSO 18.0 deployment . Adopt the RHOSO 18.0 data plane . Migrate the Object Storage service (swift) to the RHOSO nodes . Migrate the Red Hat Ceph Storage cluster . Migrate the monitoring stack component to new nodes within an existing Red Hat Ceph Storage cluster . Migrate Red Hat Ceph Storage MDS to new nodes within the existing cluster . Migrate Red Hat Ceph Storage RGW to external RHEL nodes . Migrate Red Hat Ceph Storage RBD to external RHEL nodes . Post-adoption tasks Optional: Run tempest to verify that the entire adoption process is working properly. For more information, see Validating and troubleshooting the deployed cloud . Optional: Perform a minor update from RHEL 9.2 to 9.4. You can perform a minor update any time after you complete the adoption procedure. For more information, see Updating your environment to the latest maintenance release . Optional: Verify that you migrated all services from the Controller nodes, and then power off the nodes. If any services are still running in the Controller nodes, such as Open Virtual Networking (ML2/OVN), Object Storage service (swift), or Red Hat Ceph Storage, do not power off the nodes. 1.6. Identity service authentication If you have custom policies enabled, contact Red Hat Support before adopting a director OpenStack deployment. You must complete the following steps for adoption: Remove custom policies. Run the adoption. Re-add custom policies by using the new SRBAC syntax. After you adopt a director-based OpenStack deployment to a Red Hat OpenStack Services on OpenShift deployment, the Identity service performs user authentication and authorization by using Secure RBAC (SRBAC). If SRBAC is already enabled, then there is no change to how you perform operations. If SRBAC is disabled, then adopting a director-based OpenStack deployment might change how you perform operations due to changes in API access policies. For more information on SRBAC, see Secure role based access control in Red Hat OpenStack Services on OpenShift in Performing security operations . 1.7. Configuring the network for the Red Hat OpenStack Services on OpenShift deployment When you adopt a new Red Hat OpenStack Services on OpenShift (RHOSO) deployment, you must align the network configuration with the adopted cluster to maintain connectivity for existing workloads. Perform the following tasks to incorporate the existing network configuration: Configure Red Hat OpenShift Container Platform (RHOCP) worker nodes to align VLAN tags and IP Address Management (IPAM) configuration with the existing deployment. Configure control plane services to use compatible IP ranges for service and load-balancing IP addresses. Configure data plane nodes to use corresponding compatible configuration for VLAN tags and IPAM. When configuring nodes and services, the general approach is as follows: For IPAM, you can either reuse subnet ranges from the existing deployment or, if there is a shortage of free IP addresses in existing subnets, define new ranges for the new control plane services. If you define new ranges, you configure IP routing between the old and new ranges. For more information, see Planning your IPAM configuration . For VLAN tags, always reuse the configuration from the existing deployment. Note For more information about the network architecture and configuration, see Preparing networks for Red Hat OpenStack Services on OpenShift in Deploying Red Hat OpenStack Services on OpenShift and About networking in Networking . 1.7.1. Retrieving the network configuration from your existing deployment You must determine which isolated networks are defined in your existing deployment. After you retrieve your network configuration, you have the following information: A list of isolated networks that are used in the existing deployment. For each of the isolated networks, the VLAN tag and IP ranges used for dynamic address allocation. A list of existing IP address allocations that are used in the environment. When reusing the existing subnet ranges to host the new control plane services, these addresses are excluded from the corresponding allocation pools. Procedure Find the network configuration in the network_data.yaml file. For example: Retrieve the VLAN tag that is used in the vlan key and the IP range in the ip_subnet key for each isolated network from the network_data.yaml file. When reusing subnet ranges from the existing deployment for the new control plane services, the ranges are split into separate pools for control plane services and load-balancer IP addresses. Use the tripleo-ansible-inventory.yaml file to determine the list of IP addresses that are already consumed in the adopted environment. For each listed host in the file, make a note of the IP and VIP addresses that are consumed by the node. For example: Note In this example, the 172.17.0.2 and 172.17.0.100 values are consumed and are not available for the new control plane services until the adoption is complete. Repeat this procedure for each isolated network and each host in the configuration. 1.7.2. Planning your IPAM configuration In a Red Hat OpenStack Services on OpenShift (RHOSO) deployment, each service that is deployed on the Red Hat OpenShift Container Platform (RHOCP) worker nodes requires an IP address from the IP Address Management (IPAM) pool. In a Red Hat OpenStack Platform (RHOSP) deployment, all services that are hosted on a Controller node share the same IP address. The RHOSO control plane has different requirements for the number of IP addresses that are made available for services. Depending on the size of the IP ranges that are used in the existing RHOSO deployment, you might reuse these ranges for the RHOSO control plane. The total number of IP addresses that are required for the new control plane services in each isolated network is calculated as the sum of the following: The number of RHOCP worker nodes. Each worker node requires 1 IP address in the NodeNetworkConfigurationPolicy custom resource (CR). The number of IP addresses required for the data plane nodes. Each node requires an IP address from the NetConfig CRs. The number of IP addresses required for control plane services. Each service requires an IP address from the NetworkAttachmentDefinition CRs. This number depends on the number of replicas for each service. The number of IP addresses required for load balancer IP addresses. Each service requires a Virtual IP address from the IPAddressPool CRs. For example, a simple single worker node RHOCP deployment with Red Hat OpenShift Local has the following IP ranges defined for the internalapi network: 1 IP address for the single worker node 1 IP address for the data plane node NetworkAttachmentDefinition CRs for control plane services: X.X.X.30-X.X.X.70 (41 addresses) IPAllocationPool CRs for load balancer IPs: X.X.X.80-X.X.X.90 (11 addresses) This example shows a total of 54 IP addresses allocated to the internalapi allocation pools. The requirements might differ depending on the list of RHOSP services to be deployed, their replica numbers, and the number of RHOCP worker nodes and data plane nodes. Additional IP addresses might be required in future RHOSP releases, so you must plan for some extra capacity for each of the allocation pools that are used in the new environment. After you determine the required IP pool size for the new deployment, you can choose to define new IP address ranges or reuse your existing IP address ranges. Regardless of the scenario, the VLAN tags in the existing deployment are reused in the new deployment. Ensure that the VLAN tags are properly retained in the new configuration. For more information, see Configuring isolated networks . 1.7.2.1. Configuring new subnet ranges You can define new IP ranges for control plane services that belong to a different subnet that is not used in the existing cluster. Then you configure link local IP routing between the existing and new subnets to enable existing and new service deployments to communicate. This involves using the director mechanism on a pre-adopted cluster to configure additional link local routes. This enables the data plane deployment to reach out to Red Hat OpenStack Platform (RHOSP) nodes by using the existing subnet addresses. You can use new subnet ranges with any existing subnet configuration, and when the existing cluster subnet ranges do not have enough free IP addresses for the new control plane services. You must size the new subnet appropriately to accommodate the new control plane services. There are no specific requirements for the existing deployment allocation pools that are already consumed by the RHOSP environment. Important Defining a new subnet for Storage and Storage management is not supported because Compute service (nova) and Red Hat Ceph Storage do not allow modifying those networks during adoption. In the following procedure, you configure NetworkAttachmentDefinition custom resources (CRs) to use a different subnet from what is configured in the network_config section of the OpenStackDataPlaneNodeSet CR for the same networks. The new range in the NetworkAttachmentDefinition CR is used for control plane services, while the existing range in the OpenStackDataPlaneNodeSet CR is used to manage IP Address Management (IPAM) for data plane nodes. The values that are used in the following procedure are examples. Use values that are specific to your configuration. Procedure Configure link local routes on the existing deployment nodes for the control plane subnets. This is done through director configuration: 1 The new control plane subnet. 2 The control plane IP address of the existing data plane node. Repeat this configuration for other networks that need to use different subnets for the new and existing parts of the deployment. Apply the new configuration to every RHOSP node: Optional: Include the --templates option to use your own templates instead of the default templates located in /usr/share/openstack-tripleo-heat-templates . Replace <templates_directory> with the path to the directory that contains your templates. Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud . Include the --network-config optional argument to provide the network definitions to the cli-overcloud-node-network-config.yaml Ansible playbook. The cli-overcloud-node-network-config.yaml playbook uses the os-net-config tool to apply the network configuration on the deployed nodes. If you do not use --network-config to provide the network definitions, then you must configure the {{role.name}}NetworkConfigTemplate parameters in your network-environment.yaml file, otherwise the default network definitions are used. Replace <deployment_file> with the name of the heat environment file to generate for inclusion in the deployment command, for example /home/stack/templates/overcloud-baremetal-deployed.yaml . Replace <node_definition_file> with the name of your node definition file, for example, overcloud-baremetal-deploy.yaml . Ensure that the network_config_update variable is set to true in the node definition file. Note Network configuration changes are not applied by default to avoid the risk of network disruption. You must enforce the changes by setting the StandaloneNetworkConfigUpdate: true in the director configuration files. Confirm that there are new link local routes to the new subnet on each node. For example: You also must configure link local routes to existing deployment on Red Hat OpenStack Services on OpenShift (RHOSO) worker nodes. This is achieved by adding routes entries to the NodeNetworkConfigurationPolicy CRs for each network. For example: 1 The original subnet of the isolated network on the data plane. 2 The Red Hat OpenShift Container Platform (RHOCP) worker network interface that corresponds to the isolated network on the data plane. As a result, the following route is added to your RHOCP nodes: Later, during the data plane adoption, in the network_config section of the OpenStackDataPlaneNodeSet CR, add the same link local routes for the new control plane subnet ranges. For example: List the IP addresses that are used for the data plane nodes in the existing deployment as ansibleHost and fixedIP . For example: Important Do not change RHOSP node IP addresses during the adoption process. List previously used IP addresses in the fixedIP fields for each node entry in the nodes section of the OpenStackDataPlaneNodeSet CR. Expand the SSH range for the firewall configuration to include both subnets to allow SSH access to data plane nodes from both subnets: This provides SSH access from the new subnet to the RHOSP nodes as well as the RHOSP subnets. Set edpm_network_config_update: true to enforce the changes that you are applying to the nodes. 1.7.2.2. Reusing existing subnet ranges You can reuse existing subnet ranges if they have enough IP addresses to allocate to the new control plane services. You configure the new control plane services to use the same subnet as you used in the Red Hat OpenStack Platform (RHOSP) environment, and configure the allocation pools that are used by the new services to exclude IP addresses that are already allocated to existing cluster nodes. By reusing existing subnets, you avoid additional link local route configuration between the existing and new subnets. If your existing subnets do not have enough IP addresses in the existing subnet ranges for the new control plane services, you must create new subnet ranges. For more information, see Using new subnet ranges . No special routing configuration is required to reuse subnet ranges. However, you must ensure that the IP addresses that are consumed by RHOSP services do not overlap with the new allocation pools configured for Red Hat OpenStack Services on OpenShift control plane services. If you are especially constrained by the size of the existing subnet, you may have to apply elaborate exclusion rules when defining allocation pools for the new control plane services. For more information, see Configuring isolated networks . 1.7.3. Configuring isolated networks Before you begin replicating your existing VLAN and IPAM configuration in the Red Hat OpenStack Services on OpenShift (RHOSO) environment, you must have the following IP address allocations for the new control plane services: 1 IP address for each isolated network on each Red Hat OpenShift Container Platform (RHOCP) worker node. You configure these IP addresses in the NodeNetworkConfigurationPolicy custom resources (CRs) for the RHOCP worker nodes. For more information, see Configuring RHOCP worker nodes . 1 IP range for each isolated network for the data plane nodes. You configure these ranges in the NetConfig CRs for the data plane nodes. For more information, see Configuring data plane nodes . 1 IP range for each isolated network for control plane services. These ranges enable pod connectivity for isolated networks in the NetworkAttachmentDefinition CRs. For more information, see Configuring the networking for control plane services . 1 IP range for each isolated network for load balancer IP addresses. These IP ranges define load balancer IP addresses for MetalLB in the IPAddressPool CRs. For more information, see Configuring the networking for control plane services . Note The exact list and configuration of isolated networks in the following procedures should reflect the actual Red Hat OpenStack Platform environment. The number of isolated networks might differ from the examples used in the procedures. The IPAM scheme might also differ. Only the parts of the configuration that are relevant to configuring networks are shown. The values that are used in the following procedures are examples. Use values that are specific to your configuration. 1.7.3.1. Configuring isolated networks on RHOCP worker nodes To connect service pods to isolated networks on Red Hat OpenShift Container Platform (RHOCP) worker nodes that run Red Hat OpenStack Platform services, physical network configuration on the hypervisor is required. This configuration is managed by the NMState operator, which uses NodeNetworkConfigurationPolicy custom resources (CRs) to define the desired network configuration for the nodes. Procedure For each RHOCP worker node, define a NodeNetworkConfigurationPolicy CR that describes the desired network configuration. For example: 1.7.3.2. Configuring isolated networks on control plane services After the NMState operator creates the desired hypervisor network configuration for isolated networks, you must configure the Red Hat OpenStack Platform (RHOSP) services to use the configured interfaces. You define a NetworkAttachmentDefinition custom resource (CR) for each isolated network. In some clusters, these CRs are managed by the Cluster Network Operator, in which case you use Network CRs instead. For more information, see Cluster Network Operator in Networking . Procedure Define a NetworkAttachmentDefinition CR for each isolated network. For example: Important Ensure that the interface name and IPAM range match the configuration that you used in the NodeNetworkConfigurationPolicy CRs. Optional: When reusing existing IP ranges, you can exclude part of the range that is used in the existing deployment by using the exclude parameter in the NetworkAttachmentDefinition pool. For example: 1 Defines the start of the IP range. 2 Defines the end of the IP range. 3 Excludes part of the IP range. This example excludes IP addresses 172.17.0.24/32 and 172.17.0.44/31 from the allocation pool. If your RHOSP services require load balancer IP addresses, define the pools for these services in an IPAddressPool CR. For example: Note The load balancer IP addresses belong to the same IP range as the control plane services, and are managed by MetalLB. This pool should also be aligned with the RHOSP configuration. Define IPAddressPool CRs for each isolated network that requires load balancer IP addresses. Optional: When reusing existing IP ranges, you can exclude part of the range by listing multiple entries in the addresses section of the IPAddressPool . For example: The example above would exclude the 172.17.0.65 address from the allocation pool. 1.7.3.3. Configuring isolated networks on data plane nodes Data plane nodes are configured by the OpenStack Operator and your OpenStackDataPlaneNodeSet custom resources (CRs). The OpenStackDataPlaneNodeSet CRs define your desired network configuration for the nodes. Your Red Hat OpenStack Services on OpenShift (RHOSO) network configuration should reflect the existing Red Hat OpenStack Platform (RHOSP) network setup. You must pull the network_data.yaml files from each RHOSP node and reuse them when you define the OpenStackDataPlaneNodeSet CRs. The format of the configuration does not change, so you can put network templates under edpm_network_config_template variables, either for all nodes or for each node. To ensure that the latest network configuration is used during the data plane adoption, you should also set edpm_network_config_update: true in the nodeTemplate field of the OpenStackDataPlaneNodeSet CR. Procedure Configure a NetConfig CR with your desired VLAN tags and IPAM configuration. For example: Optional: In the NetConfig CR, list multiple ranges for the allocationRanges field to exclude some of the IP addresses, for example, to accommodate IP addresses that are already consumed by the adopted environment: This example excludes the 172.17.0.200 address from the pool. 1.8. Storage requirements Storage in a Red Hat OpenStack Platform (RHOSP) deployment refers to the following types: The storage that is needed for the service to run The storage that the service manages Before you can deploy the services in Red Hat OpenStack Services on OpenShift (RHOSO), you must review the storage requirements, plan your Red Hat OpenShift Container Platform (RHOCP) node selection, prepare your RHOCP nodes, and so on. 1.8.1. Storage driver certification Before you adopt your Red Hat OpenStack Platform 17.1 deployment to a Red Hat OpenStack Services on OpenShift (RHOSO) 18.0 deployment, confirm that your deployed storage drivers are certified for use with RHOSO 18.0. For information on software certified for use with RHOSO 18.0, see the Red Hat Ecosystem Catalog . 1.8.2. Block Storage service guidelines Prepare to adopt your Block Storage service (cinder): Take note of the Block Storage service back ends that you use. Determine all the transport protocols that the Block Storage service back ends use, such as RBD, iSCSI, FC, NFS, NVMe-TCP, and so on. You must consider them when you place the Block Storage services and when the right storage transport-related binaries are running on the Red Hat OpenShift Container Platform (RHOCP) nodes. For more information about each storage transport protocol, see RHOCP preparation for Block Storage service adoption . Use a Block Storage service volume service to deploy each Block Storage service volume back end. For example, you have an LVM back end, a Ceph back end, and two entries in cinderVolumes , and you cannot set global defaults for all volume services. You must define a service for each of them: apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: enabled: true template: cinderVolumes: lvm: customServiceConfig: | [DEFAULT] debug = True [lvm] < . . . > ceph: customServiceConfig: | [DEFAULT] debug = True [ceph] < . . . > Warning Check that all configuration options are still valid for RHOSO 18.0 version. Configuration options might be deprecated, removed, or added. This applies to both back-end driver-specific configuration options and other generic options. 1.8.3. Limitations for adopting the Block Storage service Before you begin the Block Storage service (cinder) adoption, review the following limitations: There is no global nodeSelector option for all Block Storage service volumes. You must specify the nodeSelector for each back end. There are no global customServiceConfig or customServiceConfigSecrets options for all Block Storage service volumes. You must specify these options for each back end. Support for Block Storage service back ends that require kernel modules that are not included in Red Hat Enterprise Linux is not tested in Red Hat OpenStack Services on OpenShift (RHOSO). 1.8.4. RHOCP preparation for Block Storage service adoption Before you deploy Red Hat OpenStack Platform (RHOSP) in Red Hat OpenShift Container Platform (RHOCP) nodes, ensure that the networks are ready, that you decide which RHOCP nodes to restrict, and that you make any necessary changes to the RHOCP nodes. Node selection You might need to restrict the RHOCP nodes where the Block Storage service volume and backup services run. An example of when you need to restrict nodes for a specific Block Storage service is when you deploy the Block Storage service with the LVM driver. In that scenario, the LVM data where the volumes are stored only exists in a specific host, so you need to pin the Block Storage-volume service to that specific RHOCP node. Running the service on any other RHOCP node does not work. You cannot use the RHOCP host node name to restrict the LVM back end. You need to identify the LVM back end by using a unique label, an existing label, or a new label: apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: secret: osp-secret storageClass: local-storage cinder: enabled: true template: cinderVolumes: lvm-iscsi: nodeSelector: lvm: cinder-volumes < . . . > For more information about node selection, see About node selectors . Note If your nodes do not have enough local disk space for temporary images, you can use a remote NFS location by setting the extra volumes feature, extraMounts . Transport protocols Some changes to the storage transport protocols might be required for RHOCP: If you use a MachineConfig to make changes to RHOCP nodes, the nodes reboot. Check the back-end sections that are listed in the enabled_backends configuration option in your cinder.conf file to determine the enabled storage back-end sections. Depending on the back end, you can find the transport protocol by viewing the volume_driver or target_protocol configuration options. The icssid service, multipathd service, and NVMe-TCP kernel modules start automatically on data plane nodes. NFS RHOCP connects to NFS back ends without additional changes. Rados Block Device and Red Hat Ceph Storage RHOCP connects to Red Hat Ceph Storage back ends without additional changes. You must provide credentials and configuration files to the services. iSCSI To connect to iSCSI volumes, the iSCSI initiator must run on the RHOCP hosts where the volume and backup services run. The Linux Open iSCSI initiator does not support network namespaces, so you must only run one instance of the service for the normal RHOCP usage, as well as the RHOCP CSI plugins and the RHOSP services. If you are not already running iscsid on the RHOCP nodes, then you must apply a MachineConfig . For example: apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker service: cinder name: 99-master-cinder-enable-iscsid spec: config: ignition: version: 3.2.0 systemd: units: - enabled: true name: iscsid.service If you use labels to restrict the nodes where the Block Storage services run, you must use a MachineConfigPool to limit the effects of the MachineConfig to the nodes where your services might run. For more information, see About node selectors . If you are using a single node deployment to test the process, replace worker with master in the MachineConfig . For production deployments that use iSCSI volumes, configure multipathing for better I/O. FC The Block Storage service volume and Block Storage service backup services must run in an RHOCP host that has host bus adapters (HBAs). If some nodes do not have HBAs, then use labels to restrict where these services run. For more information, see About node selectors . If you have virtualized RHOCP clusters that use FC you need to expose the host HBAs inside the virtual machine. For production deployments that use FC volumes, configure multipathing for better I/O. NVMe-TCP To connect to NVMe-TCP volumes, load NVMe-TCP kernel modules on the RHOCP hosts. If you do not already load the nvme-fabrics module on the RHOCP nodes where the volume and backup services are going to run, then you must apply a MachineConfig . For example: If you use labels to restrict the nodes where Block Storage services run, use a MachineConfigPool to limit the effects of the MachineConfig to the nodes where your services run. For more information, see About node selectors . If you use a single node deployment to test the process, replace worker with master in the MachineConfig . Only load the nvme-fabrics module because it loads the transport-specific modules, such as TCP, RDMA, or FC, as needed. For production deployments that use NVMe-TCP volumes, use multipathing for better I/O. For NVMe-TCP volumes, RHOCP uses native multipathing, called ANA. After the RHOCP nodes reboot and load the nvme-fabrics module, you can confirm that the operating system is configured and that it supports ANA by checking the host: Important ANA does not use the Linux Multipathing Device Mapper, but RHOCP requires multipathd to run on Compute nodes for the Compute service (nova) to be able to use multipathing. Multipathing is automatically configured on data plane nodes when they are provisioned. Multipathing Use multipathing for iSCSI and FC protocols. To configure multipathing on these protocols, you perform the following tasks: Prepare the RHOCP hosts Configure the Block Storage services Prepare the Compute service nodes Configure the Compute service To prepare the RHOCP hosts, ensure that the Linux Multipath Device Mapper is configured and running on the RHOCP hosts by using MachineConfig . For example: # Includes the /etc/multipathd.conf contents and the systemd unit changes apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker service: cinder name: 99-master-cinder-enable-multipathd spec: config: ignition: version: 3.2.0 storage: files: - path: /etc/multipath.conf overwrite: false # Mode must be decimal, this is 0600 mode: 384 user: name: root group: name: root contents: # Source can be a http, https, tftp, s3, gs, or data as defined in rfc2397. # This is the rfc2397 text/plain string format source: data:,defaults%20%7B%0A%20%20user_friendly_names%20no%0A%20%20recheck_wwid%20yes%0A%20%20skip_kpartx%20yes%0A%20%20find_multipaths%20yes%0A%7D%0A%0Ablacklist%20%7B%0A%7D systemd: units: - enabled: true name: multipathd.service If you use labels to restrict the nodes where Block Storage services run, you need to use a MachineConfigPool to limit the effects of the MachineConfig to only the nodes where your services run. For more information, see About node selectors . If you are using a single node deployment to test the process, replace worker with master in the MachineConfig . Cinder volume and backup are configured by default to use multipathing. 1.8.5. Converting the Block Storage service configuration In your deployment, you use the same cinder.conf file for all the services. To prepare your Block Storage service (cinder) configuration for adoption, split this single-file configuration into individual configurations for each Block Storage service service. Review the following information to guide you in coverting your configuration: Determine what part of the configuration is generic for all the Block Storage services and remove anything that would change when deployed in Red Hat OpenShift Container Platform (RHOCP), such as the connection in the [database] section, the transport_url and log_dir in the [DEFAULT] sections, the whole [coordination] and [barbican] sections. The remaining generic configuration goes into the customServiceConfig option, or a Secret custom resource (CR) and is then used in the customServiceConfigSecrets section, at the cinder: template: level. Determine if there is a scheduler-specific configuration and add it to the customServiceConfig option in cinder: template: cinderScheduler . Determine if there is an API-specific configuration and add it to the customServiceConfig option in cinder: template: cinderAPI . If the Block Storage service backup is deployed, add the Block Storage service backup configuration options to customServiceConfig option, or to a Secret CR that you can add to customServiceConfigSecrets section at the cinder: template: cinderBackup: level. Remove the host configuration in the [DEFAULT] section to support multiple replicas later. Determine the individual volume back-end configuration for each of the drivers. The configuration is in the specific driver section, and it includes the [backend_defaults] section and FC zoning sections if you use them. The Block Storage service operator does not support a global customServiceConfig option for all volume services. Each back end has its own section under cinder: template: cinderVolumes , and the configuration goes in the customServiceConfig option or in a Secret CR and is then used in the customServiceConfigSecrets section. If any of the Block Storage service volume drivers require a custom vendor image, find the location of the image in the Red Hat Ecosystem Catalog , and create or modify an OpenStackVersion CR to specify the custom image by using the key from the cinderVolumes section. For example, if you have the following configuration: spec: cinder: enabled: true template: cinderVolume: pure: customServiceConfigSecrets: - openstack-cinder-pure-cfg < . . . > Then the OpenStackVersion CR that describes the container image for that back end looks like the following example: apiVersion: core.openstack.org/v1beta1 kind: OpenStackVersion metadata: name: openstack spec: customContainerImages: cinderVolumeImages: pure: registry.connect.redhat.com/purestorage/openstack-cinder-volume-pure-rhosp-18-0' Note The name of the OpenStackVersion must match the name of your OpenStackControlPlane CR. If your Block Storage services use external files, for example, for a custom policy, or to store credentials or SSL certificate authority bundles to connect to a storage array, make those files available to the right containers. Use Secrets or ConfigMap to store the information in RHOCP and then in the extraMounts key. For example, for Red Hat Ceph Storage credentials that are stored in a Secret called ceph-conf-files , you patch the top-level extraMounts key in the OpenstackControlPlane CR: spec: extraMounts: - extraVol: - extraVolType: Ceph mounts: - mountPath: /etc/ceph name: ceph readOnly: true propagation: - CinderVolume - CinderBackup - Glance volumes: - name: ceph projected: sources: - secret: name: ceph-conf-files For a service-specific file, such as the API policy, you add the configuration on the service itself. In the following example, you include the CinderAPI configuration that references the policy you are adding from a ConfigMap called my-cinder-conf that has a policy key with the contents of the policy: spec: cinder: enabled: true template: cinderAPI: customServiceConfig: | [oslo_policy] policy_file=/etc/cinder/api/policy.yaml extraMounts: - extraVol: - extraVolType: Ceph mounts: - mountPath: /etc/cinder/api name: policy readOnly: true propagation: - CinderAPI volumes: - name: policy projected: sources: - configMap: name: my-cinder-conf items: - key: policy path: policy.yaml 1.8.6. Changes to CephFS through NFS Important This content in this section is available in this release as a Technology Preview , and therefore is not fully supported by Red Hat. It should only be used for testing, and should not be deployed in a production environment. For more information, see Technology Preview . Before you begin the adoption, review the following information to understand the changes to CephFS through NFS between Red Hat OpenStack Platform (RHOSP) 17.1 and Red Hat OpenStack Services on OpenShift (RHOSO) 18.0: If the RHOSP 17.1 deployment uses CephFS through NFS as a back end for Shared File Systems service (manila), you cannot directly import the ceph-nfs service on the RHOSP Controller nodes into RHOSO 18.0. In RHOSO 18.0, the Shared File Systems service only supports using a clustered NFS service that is directly managed on the Red Hat Ceph Storage cluster. Adoption with the ceph-nfs service involves a data path disruption to existing NFS clients. On RHOSP 17.1, Pacemaker controls the high availability of the ceph-nfs service. This service is assigned a Virtual IP (VIP) address that is also managed by Pacemaker. The VIP is typically created on an isolated StorageNFS network. The Controller nodes have ordering and collocation constraints established between this VIP, ceph-nfs , and the Shared File Systems service (manila) share manager service. Prior to adopting Shared File Systems service, you must adjust the Pacemaker ordering and collocation constraints to separate the share manager service. This establishes ceph-nfs with its VIP as an isolated, standalone NFS service that you can decommission after completing the RHOSO adoption. In Red Hat Ceph Storage 7, a native clustered Ceph NFS service has to be deployed on the Red Hat Ceph Storage cluster by using the Ceph Orchestrator prior to adopting the Shared File Systems service. This NFS service eventually replaces the standalone NFS service from RHOSP 17.1 in your deployment. When the Shared File Systems service is adopted into the RHOSO 18.0 environment, it establishes all the existing exports and client restrictions on the new clustered Ceph NFS service. Clients can continue to read and write data on existing NFS shares, and are not affected until the old standalone NFS service is decommissioned. After the service is decommissioned, you can re-mount the same share from the new clustered Ceph NFS service during a scheduled downtime. To ensure that NFS users are not required to make any networking changes to their existing workloads, assign an IP address from the same isolated StorageNFS network to the clustered Ceph NFS service. NFS users only need to discover and re-mount their shares by using new export paths. When the adoption is complete, RHOSO users can query the Shared File Systems service API to list the export locations on existing shares to identify the preferred paths to mount these shares. These preferred paths correspond to the new clustered Ceph NFS service in contrast to other non-preferred export paths that continue to be displayed until the old isolated, standalone NFS service is decommissioned. For more information on setting up a clustered NFS service, see Creating an NFS Ganesha cluster . 1.9. Red Hat Ceph Storage prerequisites Before you migrate your Red Hat Ceph Storage cluster daemons from your Controller nodes, complete the following tasks in your Red Hat OpenStack Platform 17.1 environment: Upgrade your Red Hat Ceph Storage cluster to release 7. For more information, see Upgrading Red Hat Ceph Storage 6 to 7 in Framework for upgrades (16.2 to 17.1) . Your Red Hat Ceph Storage 7 deployment is managed by cephadm . The undercloud is still available, and the nodes and networks are managed by director. If you use an externally deployed Red Hat Ceph Storage cluster, you must recreate a ceph-nfs cluster in the target nodes as well as propogate the StorageNFS network. Complete the prerequisites for your specific Red Hat Ceph Storage environment: Red Hat Ceph Storage with monitoring stack components Red Hat Ceph Storage RGW Red Hat Ceph Storage RBD NFS Ganesha 1.9.1. Completing prerequisites for a Red Hat Ceph Storage cluster with monitoring stack components Complete the following prerequisites before you migrate a Red Hat Ceph Storage cluster with monitoring stack components. Note In addition to updating the container images related to the monitoring stack, you must update the configuration entry related to the container_image_base . This has an impact on all the Red Hat Ceph Storage daemons that rely on the undercloud images. New daemons are deployed by using the new image registry location that is configured in the Red Hat Ceph Storage cluster. Procedure Gather the current status of the monitoring stack. Verify that the hosts have no monitoring label, or grafana , prometheus , or alertmanager , in cases of a per daemons placement evaluation: Note The entire relocation process is driven by cephadm and relies on labels to be assigned to the target nodes, where the daemons are scheduled. For more information about assigning labels to nodes, review the Red Hat Knowledgebase article Red Hat Ceph Storage: Supported configurations . Confirm that the cluster is healthy and that both ceph orch ls and ceph orch ps return the expected number of deployed daemons. Review and update the container image registry: Note If you run the Red Hat Ceph Storage externalization procedure after you migrate the Red Hat OpenStack Platform control plane, update the container images in the Red Hat Ceph Storage cluster configuration. The current container images point to the undercloud registry, which might not be available anymore. Because the undercloud is not available after adoption is complete, replace the undercloud-provided images with an alternative registry. Remove the undercloud container images: 1.9.2. Completing prerequisites for Red Hat Ceph Storage RGW migration Complete the following prerequisites before you begin the Ceph Object Gateway (RGW) migration. Procedure Check the current status of the Red Hat Ceph Storage nodes: Log in to controller-0 and check the Pacemaker status to identify important information for the RGW migration: Identify the ranges of the storage networks. The following is an example and the values might differ in your environment: 1 br-ex represents the External Network, where in the current environment, HAProxy has the front-end Virtual IP (VIP) assigned. 2 vlan30 represents the Storage Network, where the new RGW instances should be started on the Red Hat Ceph Storage nodes. Identify the network that you previously had in HAProxy and propagate it through director to the Red Hat Ceph Storage nodes. Use this network to reserve a new VIP that is owned by Red Hat Ceph Storage as the entry point for the RGW service. Log in to controller-0 and find the ceph_rgw section in the current HAProxy configuration: Confirm that the network is used as an HAProxy front end. The following example shows that controller-0 exposes the services by using the external network, which is absent from the Red Hat Ceph Storage nodes. You must propagate the external network through director: Note If the target nodes are not managed by director, you cannot use this procedure to configure the network. An administrator must manually configure all the required networks. Propagate the HAProxy front-end network to Red Hat Ceph Storage nodes. In the NIC template that you use to define the ceph-storage network interfaces, add the new config section in the Red Hat Ceph Storage network configuration template file, for example, /home/stack/composable_roles/network/nic-configs/ceph-storage.j2 : --- network_config: - type: interface name: nic1 use_dhcp: false dns_servers: {{ ctlplane_dns_nameservers }} addresses: - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }} routes: {{ ctlplane_host_routes }} - type: vlan vlan_id: {{ storage_mgmt_vlan_id }} device: nic1 addresses: - ip_netmask: {{ storage_mgmt_ip }}/{{ storage_mgmt_cidr }} routes: {{ storage_mgmt_host_routes }} - type: interface name: nic2 use_dhcp: false defroute: false - type: vlan vlan_id: {{ storage_vlan_id }} device: nic2 addresses: - ip_netmask: {{ storage_ip }}/{{ storage_cidr }} routes: {{ storage_host_routes }} - type: ovs_bridge name: {{ neutron_physical_bridge_name }} dns_servers: {{ ctlplane_dns_nameservers }} domain: {{ dns_search_domains }} use_dhcp: false addresses: - ip_netmask: {{ external_ip }}/{{ external_cidr }} routes: {{ external_host_routes }} members: [] - type: interface name: nic3 primary: true Add the External Network to the bare metal file, for example, /home/stack/composable_roles/network/baremetal_deployment.yaml that is used by metalsmith : Note Ensure that network_config_update is enabled for network propagation to the target nodes when os-net-config is triggered. - name: CephStorage count: 3 hostname_format: cephstorage-%index% instances: - hostname: cephstorage-0 name: ceph-0 - hostname: cephstorage-1 name: ceph-1 - hostname: cephstorage-2 name: ceph-2 defaults: profile: ceph-storage network_config: template: /home/stack/composable_roles/network/nic-configs/ceph-storage.j2 network_config_update: true networks: - network: ctlplane vif: true - network: storage - network: storage_mgmt - network: external Configure the new network on the bare metal nodes: Verify that the new network is configured on the Red Hat Ceph Storage nodes: 1.9.3. Completing prerequisites for a Red Hat Ceph Storage RBD migration Complete the following prerequisites before you begin the Red Hat Ceph Storage Rados Block Device (RBD) migration. The target CephStorage or ComputeHCI nodes are configured to have both storage and storage_mgmt networks. This ensures that you can use both Red Hat Ceph Storage public and cluster networks from the same node. From Red Hat OpenStack Platform 17.1 and later you do not have to run a stack update. NFS Ganesha is migrated from a director deployment to cephadm . For more information, see Creating an NFS Ganesha cluster . Ceph Metadata Server, monitoring stack, Ceph Object Gateway, and any other daemon that is deployed on Controller nodes. The daemons distribution follows the cardinality constraints that are described in Red Hat Ceph Storage: Supported configurations . The Red Hat Ceph Storage cluster is healthy, and the ceph -s command returns HEALTH_OK . Run os-net-config on the bare metal node and configure additional networks: If target nodes are CephStorage , ensure that the network is defined in the bare metal file for the CephStorage nodes, for example, /home/stack/composable_roles/network/baremetal_deployment.yaml : - name: CephStorage count: 2 instances: - hostname: oc0-ceph-0 name: oc0-ceph-0 - hostname: oc0-ceph-1 name: oc0-ceph-1 defaults: networks: - network: ctlplane vif: true - network: storage_cloud_0 subnet: storage_cloud_0_subnet - network: storage_mgmt_cloud_0 subnet: storage_mgmt_cloud_0_subnet network_config: template: templates/single_nic_vlans/single_nic_vlans_storage.j2 Add the missing network: Verify that the storage network is configured on the target nodes: 1.9.4. Creating an NFS Ganesha cluster Important This content in this section is available in this release as a Technology Preview , and therefore is not fully supported by Red Hat. It should only be used for testing, and should not be deployed in a production environment. For more information, see Technology Preview . If you use CephFS through NFS with the Shared File Systems service (manila), you must create a new clustered NFS service on the Red Hat Ceph Storage cluster. This service replaces the standalone, Pacemaker-controlled ceph-nfs service that you use in Red Hat OpenStack Platform (RHOSP) 17.1. Procedure Identify the Red Hat Ceph Storage nodes to deploy the new clustered NFS service, for example, cephstorage-0 , cephstorage-1 , cephstorage-2 . Note You must deploy this service on the StorageNFS isolated network so that you can mount your existing shares through the new NFS export locations. You can deploy the new clustered NFS service on your existing CephStorage nodes or HCI nodes, or on new hardware that you enrolled in the Red Hat Ceph Storage cluster. If you deployed your Red Hat Ceph Storage nodes with director, propagate the StorageNFS network to the target nodes where the ceph-nfs service is deployed. Note If the target nodes are not managed by director, you cannot use this procedure to configure the network. An administrator must manually configure all the required networks. Identify the node definition file, overcloud-baremetal-deploy.yaml , that is used in the RHOSP environment. For more information about identifying the overcloud-baremetal-deploy.yaml file, see Customizing overcloud networks in Customizing the Red Hat OpenStack Services on OpenShift deployment . Edit the networks that are associated with the Red Hat Ceph Storage nodes to include the StorageNFS network: - name: CephStorage count: 3 hostname_format: cephstorage-%index% instances: - hostname: cephstorage-0 name: ceph-0 - hostname: cephstorage-1 name: ceph-1 - hostname: cephstorage-2 name: ceph-2 defaults: profile: ceph-storage network_config: template: /home/stack/network/nic-configs/ceph-storage.j2 network_config_update: true networks: - network: ctlplane vif: true - network: storage - network: storage_mgmt - network: storage_nfs Edit the network configuration template file, for example, /home/stack/network/nic-configs/ceph-storage.j2 , for the Red Hat Ceph Storage nodes to include an interface that connects to the StorageNFS network: - type: vlan device: nic2 vlan_id: {{ storage_nfs_vlan_id }} addresses: - ip_netmask: {{ storage_nfs_ip }}/{{ storage_nfs_cidr }} routes: {{ storage_nfs_host_routes }} Update the Red Hat Ceph Storage nodes: When the update is complete, ensure that a new interface is created in theRed Hat Ceph Storage nodes and that they are tagged with the VLAN that is associated with StorageNFS . Identify the IP address from the StorageNFS network to use as the Virtual IP address (VIP) for the Ceph NFS service: In a running cephadm shell, identify the hosts for the NFS service: Label each host that you identified. Repeat this command for each host that you want to label: Replace <hostname> with the name of the host that you identified. Create the NFS cluster: Replace <VIP> with the VIP for the Ceph NFS service. Note You must set the ingress-mode argument to haproxy-protocol . No other ingress-mode is supported. This ingress mode allows you to enforce client restrictions through the Shared File Systems service. For more information on deploying the clustered Ceph NFS service, see the Management of NFS-Ganesha gateway using the Ceph Orchestrator (Limited Availability) in Red Hat Ceph Storage 7 Operations Guide . Check the status of the NFS cluster: 1.10. Comparing configuration files between deployments To help you manage the configuration for your director and Red Hat OpenStack Platform (RHOSP) services, you can compare the configuration files between your director deployment and the Red Hat OpenStack Services on OpenShift (RHOSO) cloud by using the os-diff tool. Prerequisites Golang is installed and configured on your environment: Procedure Configure the /etc/os-diff/os-diff.cfg file and the /etc/os-diff/ssh.config file according to your environment. To allow os-diff to connect to your clouds and pull files from the services that you describe in the config.yaml file, you must set the following options in the os-diff.cfg file: [Default] local_config_dir=/tmp/ service_config_file=config.yaml [Tripleo] ssh_cmd=ssh -F ssh.config 1 director_host=standalone 2 container_engine=podman connection=ssh remote_config_path=/tmp/tripleo local_config_path=/tmp/ [Openshift] ocp_local_config_path=/tmp/ocp connection=local ssh_cmd="" 1 Instructs os-diff to access your director host through SSH. The default value is ssh -F ssh.config . However, you can set the value without an ssh.config file, for example, ssh -i /home/user/.ssh/id_rsa stack@my.undercloud.local . 2 The host to use to access your cloud, and the podman/docker binary is installed and allowed to interact with the running containers. You can leave this key blank. If you use a host file to connect to your cloud, configure the ssh.config file to allow os-diff to access your RHOSP environment, for example: Host * IdentitiesOnly yes Host virthost Hostname virthost IdentityFile ~/.ssh/id_rsa User root StrictHostKeyChecking no UserKnownHostsFile=/dev/null Host standalone Hostname standalone IdentityFile <path to SSH key> User root StrictHostKeyChecking no UserKnownHostsFile=/dev/null Host crc Hostname crc IdentityFile ~/.ssh/id_rsa User stack StrictHostKeyChecking no UserKnownHostsFile=/dev/null Replace <path to SSH key> with the path to your SSH key. You must provide a value for IdentityFile to get full working access to your RHOSP environment. If you use an inventory file to connect to your cloud, generate the ssh.config file from your Ansible inventory, for example, tripleo-ansible-inventory.yaml file: Verification Test your connection:

Commands:
sudo dnf -y install systemd-container
- name: InternalApi mtu: 1500 vip: true vlan: 20 name_lower: internal_api dns_domain: internal.mydomain.tld. service_net_map_replace: internal subnets: internal_api_subnet: ip_subnet: '172.17.0.0/24' allocation_pools: [{'start': '172.17.0.4', 'end': '172.17.0.250'}]
Standalone: hosts: standalone: internal_api_ip: 172.17.0.100 standalone: children: Standalone: {} vars: internal_api_vip: 172.17.0.2
network_config: - type: ovs_bridge name: br-ctlplane routes: - ip_netmask: 0.0.0.0/0 next_hop: 192.168.1.1 - ip_netmask: 172.31.0.0/24 1 next_hop: 192.168.1.100 2
(undercloud)USD openstack overcloud network provision [--templates <templates_directory> \] --output <deployment_file> /home/stack/templates/<networks_definition_file>
(undercloud)USD openstack overcloud node provision [--templates <templates_directory> \] --stack <stack> --network-config --output <deployment_file> /home/stack/templates/<node_definition_file>
ip route | grep 172 172.31.0.0/24 via 192.168.122.100 dev br-ctlplane
- destination: 192.168.122.0/24 1 next-hop-interface: ospbr 2
ip route | grep 192 192.168.122.0/24 dev ospbr proto static scope link
nodeTemplate: ansible: ansibleUser: root ansibleVars: additional_ctlplane_host_routes: - ip_netmask: 172.31.0.0/24 next_hop: '{{ ctlplane_ip }}' edpm_network_config_template: | network_config: - type: ovs_bridge routes: {{ ctlplane_host_routes + additional_ctlplane_host_routes }}
nodes: standalone: ansible: ansibleHost: 192.168.122.100 ansibleUser: "" hostName: standalone networks: - defaultRoute: true fixedIP: 192.168.122.100 name: ctlplane subnetName: subnet1
edpm_sshd_allowed_ranges: - 192.168.122.0/24 - 172.31.0.0/24
apiVersion: v1 items: - apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy spec: desiredState: interfaces: - description: internalapi vlan interface ipv4: address: - ip: 172.17.0.10 prefix-length: 24 dhcp: false enabled: true ipv6: enabled: false name: enp6s0.20 state: up type: vlan vlan: base-iface: enp6s0 id: 20 reorder-headers: true - description: storage vlan interface ipv4: address: - ip: 172.18.0.10 prefix-length: 24 dhcp: false enabled: true ipv6: enabled: false name: enp6s0.21 state: up type: vlan vlan: base-iface: enp6s0 id: 21 reorder-headers: true - description: tenant vlan interface ipv4: address: - ip: 172.19.0.10 prefix-length: 24 dhcp: false enabled: true ipv6: enabled: false name: enp6s0.22 state: up type: vlan vlan: base-iface: enp6s0 id: 22 reorder-headers: true nodeSelector: kubernetes.io/hostname: ocp-worker-0 node-role.kubernetes.io/worker: ""
apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: internalapi namespace: openstack spec: config: | { "cniVersion": "0.3.1", "name": "internalapi", "type": "macvlan", "master": "enp6s0.20", "ipam": { "type": "whereabouts", "range": "172.17.0.0/24", "range_start": "172.17.0.20", "range_end": "172.17.0.50" } }
apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: internalapi namespace: openstack spec: config: | { "cniVersion": "0.3.1", "name": "internalapi", "type": "macvlan", "master": "enp6s0.20", "ipam": { "type": "whereabouts", "range": "172.17.0.0/24", "range_start": "172.17.0.20", 1 "range_end": "172.17.0.50", 2 "exclude": [ 3 "172.17.0.24/32", "172.17.0.44/31" ] } }
- apiVersion: metallb.io/v1beta1 kind: IPAddressPool spec: addresses: - 172.17.0.60-172.17.0.70
- apiVersion: metallb.io/v1beta1 kind: IPAddressPool spec: addresses: - 172.17.0.60-172.17.0.64 - 172.17.0.66-172.17.0.70
apiVersion: network.openstack.org/v1beta1 kind: NetConfig metadata: name: netconfig spec: networks: - name: internalapi dnsDomain: internalapi.example.com subnets: - name: subnet1 allocationRanges: - end: 172.17.0.250 start: 172.17.0.100 cidr: 172.17.0.0/24 vlan: 20 - name: storage dnsDomain: storage.example.com subnets: - name: subnet1 allocationRanges: - end: 172.18.0.250 start: 172.18.0.100 cidr: 172.18.0.0/24 vlan: 21 - name: tenant dnsDomain: tenant.example.com subnets: - name: subnet1 allocationRanges: - end: 172.19.0.250 start: 172.19.0.100 cidr: 172.19.0.0/24 vlan: 22
apiVersion: network.openstack.org/v1beta1 kind: NetConfig metadata: name: netconfig spec: networks: - name: internalapi dnsDomain: internalapi.example.com subnets: - name: subnet1 allocationRanges: - end: 172.17.0.199 start: 172.17.0.100 - end: 172.17.0.250 start: 172.17.0.201 cidr: 172.17.0.0/24 vlan: 20
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: cinder: enabled: true template: cinderVolumes: lvm: customServiceConfig: | [DEFAULT] debug = True [lvm] < . . . > ceph: customServiceConfig: | [DEFAULT] debug = True [ceph] < . . . >
oc label nodes worker0 lvm=cinder-volumes
apiVersion: core.openstack.org/v1beta1 kind: OpenStackControlPlane metadata: name: openstack spec: secret: osp-secret storageClass: local-storage cinder: enabled: true template: cinderVolumes: lvm-iscsi: nodeSelector: lvm: cinder-volumes < . . . >
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker service: cinder name: 99-master-cinder-enable-iscsid spec: config: ignition: version: 3.2.0 systemd: units: - enabled: true name: iscsid.service
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker service: cinder name: 99-master-cinder-load-nvme-fabrics spec: config: ignition: version: 3.2.0 storage: files: - path: /etc/modules-load.d/nvme_fabrics.conf overwrite: false # Mode must be decimal, this is 0644 mode: 420 user: name: root group: name: root contents: # Source can be a http, https, tftp, s3, gs, or data as defined in rfc2397. # This is the rfc2397 text/plain string format source: data:,nvme-fabrics
cat /sys/module/nvme_core/parameters/multipath
Includes the /etc/multipathd.conf contents and the systemd unit changes apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker service: cinder name: 99-master-cinder-enable-multipathd spec: config: ignition: version: 3.2.0 storage: files: - path: /etc/multipath.conf overwrite: false # Mode must be decimal, this is 0600 mode: 384 user: name: root group: name: root contents: # Source can be a http, https, tftp, s3, gs, or data as defined in rfc2397. # This is the rfc2397 text/plain string format source: data:,defaults%20%7B%0A%20%20user_friendly_names%20no%0A%20%20recheck_wwid%20yes%0A%20%20skip_kpartx%20yes%0A%20%20find_multipaths%20yes%0A%7D%0A%0Ablacklist%20%7B%0A%7D systemd: units: - enabled: true name: multipathd.service
spec: cinder: enabled: true template: cinderVolume: pure: customServiceConfigSecrets: - openstack-cinder-pure-cfg < . . . >
apiVersion: core.openstack.org/v1beta1 kind: OpenStackVersion metadata: name: openstack spec: customContainerImages: cinderVolumeImages: pure: registry.connect.redhat.com/purestorage/openstack-cinder-volume-pure-rhosp-18-0'
spec: extraMounts: - extraVol: - extraVolType: Ceph mounts: - mountPath: /etc/ceph name: ceph readOnly: true propagation: - CinderVolume - CinderBackup - Glance volumes: - name: ceph projected: sources: - secret: name: ceph-conf-files
spec: cinder: enabled: true template: cinderAPI: customServiceConfig: | [oslo_policy] policy_file=/etc/cinder/api/policy.yaml extraMounts: - extraVol: - extraVolType: Ceph mounts: - mountPath: /etc/cinder/api name: policy readOnly: true propagation: - CinderAPI volumes: - name: policy projected: sources: - configMap: name: my-cinder-conf items: - key: policy path: policy.yaml
[tripleo-admin@controller-0 ~]USD sudo cephadm shell -- ceph orch host ls HOST ADDR LABELS STATUS cephstorage-0.redhat.local 192.168.24.11 osd mds cephstorage-1.redhat.local 192.168.24.12 osd mds cephstorage-2.redhat.local 192.168.24.47 osd mds controller-0.redhat.local 192.168.24.35 _admin mon mgr controller-1.redhat.local 192.168.24.53 mon _admin mgr controller-2.redhat.local 192.168.24.10 mon _admin mgr 6 hosts in cluster
ceph config dump mgr advanced mgr/cephadm/container_image_alertmanager undercloud-0.ctlplane.redhat.local:8787/rh-osbs/openshift-ose-prometheus-alertmanager:v4.10 mgr advanced mgr/cephadm/container_image_base undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph mgr advanced mgr/cephadm/container_image_grafana undercloud-0.ctlplane.redhat.local:8787/rh-osbs/grafana:latest mgr advanced mgr/cephadm/container_image_node_exporter undercloud-0.ctlplane.redhat.local:8787/rh-osbs/openshift-ose-prometheus-node-exporter:v4.10 mgr advanced mgr/cephadm/container_image_prometheus undercloud-0.ctlplane.redhat.local:8787/rh-osbs/openshift-ose-prometheus:v4.10
cephadm shell -- ceph config rm mgr mgr/cephadm/container_image_base for i in prometheus grafana alertmanager node_exporter; do cephadm shell -- ceph config rm mgr mgr/cephadm/container_image_USDi done
(undercloud) [stack@undercloud-0 ~]USD metalsmith list +------------------------+ +----------------+ | IP Addresses | | Hostname | +------------------------+ +----------------+ | ctlplane=192.168.24.25 | | cephstorage-0 | | ctlplane=192.168.24.10 | | cephstorage-1 | | ctlplane=192.168.24.32 | | cephstorage-2 | | ctlplane=192.168.24.28 | | compute-0 | | ctlplane=192.168.24.26 | | compute-1 | | ctlplane=192.168.24.43 | | controller-0 | | ctlplane=192.168.24.7 | | controller-1 | | ctlplane=192.168.24.41 | | controller-2 | +------------------------+ +----------------+
Full List of Resources: * ip-192.168.24.46 (ocf:heartbeat:IPaddr2): Started controller-0 * ip-10.0.0.103 (ocf:heartbeat:IPaddr2): Started controller-1 * ip-172.17.1.129 (ocf:heartbeat:IPaddr2): Started controller-2 * ip-172.17.3.68 (ocf:heartbeat:IPaddr2): Started controller-0 * ip-172.17.4.37 (ocf:heartbeat:IPaddr2): Started controller-1 * Container bundle set: haproxy-bundle [undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-haproxy:pcmklatest]: * haproxy-bundle-podman-0 (ocf:heartbeat:podman): Started controller-2 * haproxy-bundle-podman-1 (ocf:heartbeat:podman): Started controller-0 * haproxy-bundle-podman-2 (ocf:heartbeat:podman): Started controller-1
[heat-admin@controller-0 ~]USD ip -o -4 a 1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever 2: enp1s0 inet 192.168.24.45/24 brd 192.168.24.255 scope global enp1s0\ valid_lft forever preferred_lft forever 2: enp1s0 inet 192.168.24.46/32 brd 192.168.24.255 scope global enp1s0\ valid_lft forever preferred_lft forever 7: br-ex inet 10.0.0.122/24 brd 10.0.0.255 scope global br-ex\ valid_lft forever preferred_lft forever 1 8: vlan70 inet 172.17.5.22/24 brd 172.17.5.255 scope global vlan70\ valid_lft forever preferred_lft forever 8: vlan70 inet 172.17.5.94/32 brd 172.17.5.255 scope global vlan70\ valid_lft forever preferred_lft forever 9: vlan50 inet 172.17.2.140/24 brd 172.17.2.255 scope global vlan50\ valid_lft forever preferred_lft forever 10: vlan30 inet 172.17.3.73/24 brd 172.17.3.255 scope global vlan30\ valid_lft forever preferred_lft forever 2 10: vlan30 inet 172.17.3.68/32 brd 172.17.3.255 scope global vlan30\ valid_lft forever preferred_lft forever 11: vlan20 inet 172.17.1.88/24 brd 172.17.1.255 scope global vlan20\ valid_lft forever preferred_lft forever 12: vlan40 inet 172.17.4.24/24 brd 172.17.4.255 scope global vlan40\ valid_lft forever preferred_lft forever
less /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg listen ceph_rgw bind 10.0.0.103:8080 transparent bind 172.17.3.68:8080 transparent mode http balance leastconn http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto http if !{ ssl_fc } http-request set-header X-Forwarded-Port %[dst_port] option httpchk GET /swift/healthcheck option httplog option forwardfor server controller-0.storage.redhat.local 172.17.3.73:8080 check fall 5 inter 2000 rise 2 server controller-1.storage.redhat.local 172.17.3.146:8080 check fall 5 inter 2000 rise 2 server controller-2.storage.redhat.local 172.17.3.156:8080 check fall 5 inter 2000 rise 2
[controller-0]USD ip -o -4 a 7: br-ex inet 10.0.0.106/24 brd 10.0.0.255 scope global br-ex\ valid_lft forever preferred_lft forever
--- network_config: - type: interface name: nic1 use_dhcp: false dns_servers: {{ ctlplane_dns_nameservers }} addresses: - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }} routes: {{ ctlplane_host_routes }} - type: vlan vlan_id: {{ storage_mgmt_vlan_id }} device: nic1 addresses: - ip_netmask: {{ storage_mgmt_ip }}/{{ storage_mgmt_cidr }} routes: {{ storage_mgmt_host_routes }} - type: interface name: nic2 use_dhcp: false defroute: false - type: vlan vlan_id: {{ storage_vlan_id }} device: nic2 addresses: - ip_netmask: {{ storage_ip }}/{{ storage_cidr }} routes: {{ storage_host_routes }} - type: ovs_bridge name: {{ neutron_physical_bridge_name }} dns_servers: {{ ctlplane_dns_nameservers }} domain: {{ dns_search_domains }} use_dhcp: false addresses: - ip_netmask: {{ external_ip }}/{{ external_cidr }} routes: {{ external_host_routes }} members: [] - type: interface name: nic3 primary: true
- name: CephStorage count: 3 hostname_format: cephstorage-%index% instances: - hostname: cephstorage-0 name: ceph-0 - hostname: cephstorage-1 name: ceph-1 - hostname: cephstorage-2 name: ceph-2 defaults: profile: ceph-storage network_config: template: /home/stack/composable_roles/network/nic-configs/ceph-storage.j2 network_config_update: true networks: - network: ctlplane vif: true - network: storage - network: storage_mgmt - network: external
(undercloud) [stack@undercloud-0]USD openstack overcloud node provision -o overcloud-baremetal-deployed-0.yaml --stack overcloud --network-config -y USDPWD/composable_roles/network/baremetal_deployment.yaml
ip -o -4 a 1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever 2: enp1s0 inet 192.168.24.54/24 brd 192.168.24.255 scope global enp1s0\ valid_lft forever preferred_lft forever 11: vlan40 inet 172.17.4.43/24 brd 172.17.4.255 scope global vlan40\ valid_lft forever preferred_lft forever 12: vlan30 inet 172.17.3.23/24 brd 172.17.3.255 scope global vlan30\ valid_lft forever preferred_lft forever 14: br-ex inet 10.0.0.133/24 brd 10.0.0.255 scope global br-ex\ valid_lft forever preferred_lft forever
- name: CephStorage count: 2 instances: - hostname: oc0-ceph-0 name: oc0-ceph-0 - hostname: oc0-ceph-1 name: oc0-ceph-1 defaults: networks: - network: ctlplane vif: true - network: storage_cloud_0 subnet: storage_cloud_0_subnet - network: storage_mgmt_cloud_0 subnet: storage_mgmt_cloud_0_subnet network_config: template: templates/single_nic_vlans/single_nic_vlans_storage.j2
openstack overcloud node provision -o overcloud-baremetal-deployed-0.yaml --stack overcloud-0 /--network-config -y --concurrency 2 /home/stack/metalsmith-0.yaml
(undercloud) [stack@undercloud ~]USD ssh heat-admin@192.168.24.14 ip -o -4 a 1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever 5: br-storage inet 192.168.24.14/24 brd 192.168.24.255 scope global br-storage\ valid_lft forever preferred_lft forever 6: vlan1 inet 192.168.24.14/24 brd 192.168.24.255 scope global vlan1\ valid_lft forever preferred_lft forever 7: vlan11 inet 172.16.11.172/24 brd 172.16.11.255 scope global vlan11\ valid_lft forever preferred_lft forever 8: vlan12 inet 172.16.12.46/24 brd 172.16.12.255 scope global vlan12\ valid_lft forever preferred_lft forever
- name: CephStorage count: 3 hostname_format: cephstorage-%index% instances: - hostname: cephstorage-0 name: ceph-0 - hostname: cephstorage-1 name: ceph-1 - hostname: cephstorage-2 name: ceph-2 defaults: profile: ceph-storage network_config: template: /home/stack/network/nic-configs/ceph-storage.j2 network_config_update: true networks: - network: ctlplane vif: true - network: storage - network: storage_mgmt - network: storage_nfs
- type: vlan device: nic2 vlan_id: {{ storage_nfs_vlan_id }} addresses: - ip_netmask: {{ storage_nfs_ip }}/{{ storage_nfs_cidr }} routes: {{ storage_nfs_host_routes }}
openstack overcloud node provision --stack overcloud --network-config -y -o overcloud-baremetal-deployed-storage_nfs.yaml --concurrency 2 /home/stack/network/baremetal_deployment.yaml
openstack port list -c "Fixed IP Addresses" --network storage_nfs
ceph orch host ls
ceph orch host label add <hostname> nfs
ceph nfs cluster create cephfs "label:nfs" --ingress --virtual-ip=<VIP> --ingress-mode=haproxy-protocol
ceph nfs cluster ls ceph nfs cluster info cephfs
dnf install -y golang-github-openstack-k8s-operators-os-diff
[Default] local_config_dir=/tmp/ service_config_file=config.yaml [Tripleo] ssh_cmd=ssh -F ssh.config 1 director_host=standalone 2 container_engine=podman connection=ssh remote_config_path=/tmp/tripleo local_config_path=/tmp/ [Openshift] ocp_local_config_path=/tmp/ocp connection=local ssh_cmd=""
Host * IdentitiesOnly yes Host virthost Hostname virthost IdentityFile ~/.ssh/id_rsa User root StrictHostKeyChecking no UserKnownHostsFile=/dev/null Host standalone Hostname standalone IdentityFile <path to SSH key> User root StrictHostKeyChecking no UserKnownHostsFile=/dev/null Host crc Hostname crc IdentityFile ~/.ssh/id_rsa User stack StrictHostKeyChecking no UserKnownHostsFile=/dev/null
os-diff configure -i tripleo-ansible-inventory.yaml -o ssh.config --yaml
ssh -F ssh.config standalone