Chapter 5. Deploying a DCN node set Node sets are deployed using the same procedures, whether you deploy them at the central location, or at a remote location. Deploy each edge location in a separate availability zone. For example, deploy the central location node set at az0 , deploy the first edge site at az1 , and so on. 5.1. Configuring the data plane node networks You must configure the data plane node networks to accommodate the Red Hat Ceph Storage networking requirements. Prerequisites Control plane deployment is complete but has not yet been modified to use Ceph Storage. The data plane nodes have been pre-provisioned with an operating system. The data plane nodes are accessible through an SSH key that Ansible can use. If you are using HCI, then the data plane nodes have disks available to be used as Ceph OSDs. There are a minimum of three available data plane nodes. Ceph Storage clusters must have a minimum of three nodes to ensure redundancy. Procedure Create a file on your workstation named dcn-data-plane-networks.yaml to defin the OpenStackDataPlaneNodeSet CR that configures the dataplane node networks: Specify the services to apply to the nodes: Set the edpm_enable_chassis_gw and edpm_ovn_availability_zones fields on the data plane: Optional: The ceph-hci-pre service prepares data plane nodes to host Red Hat Ceph Storage services after network configuration using the edpm_ceph_hci_pre edpm-ansible role. By default, the edpm_ceph_hci_pre_enabled_services parameter of this role only contains RBD , RGW , and NFS services. DCN only supports RBD services at DCN sites. If you are deploying HCI, disable the RGW and NFS services by adding the edpm_ceph_hci_pre_enabled_services parameter, and adding only ceph RBD services. Note If other services, such as the Dashboard, are deployed with HCI nodes, they must be added to the edpm_ceph_hci_pre_enabled_services parameter list. For more information about this role, see edpm_ceph_hci_pre role. Configure the Red Hat Ceph Storage cluster network for storage management. The following example has 3 nodes. It assumes the storage management is on VLAN23 : Apply the CR: Replace <dataplane_cr_file> with the name of your file. Note Ansible does not configure or validate the networks until the OpenStackDataPlaneDeployment CRD is created. Create an OpenStackDataPlaneDeployment CRD, as described in Creating the data plane in the Deploying Red Hat OpenStack Services on OpenShift guide, which has the OpenStackDataPlaneNodeSet CRD file defined above to have Ansible configure the services on the data plane nodes. To confirm the network is configured, complete the following steps: SSH into a data plane node. Use the ip a command to display configured networks. Confirm the storage networks are in the list of configured networks. 5.2. Configuring and deploying hyperconverged Red Hat Ceph Storage Note The following steps are specifically for a hyperconverged configuration of Red hat Ceph Storage (RHCS), and are not required if you have deployed an external RHCS cluster. Configure and deploy Red Hat Ceph Storage by editing the configuration file and using the cephadm utility. Procedure Edit the Red Hat Ceph Storage configuration file. Add the Storage and Storage Management network ranges. Red Hat Ceph Storage uses the Storage network as the Red Hat Ceph Storage public_network and the Storage Management network as the cluster_network . The following example is for a configuration file entry where the Storage network range is 172.18.0.0/24 and the Storage Management network range is 172.20.0.0/24 : Add collocation boundaries between the Compute service and Ceph OSD services. Boundaries should be set between collocated Compute service and Ceph OSD services to reduce CPU and memory contention. The following is an example for a Ceph configuration file entry with these boundaries set: In this example, the osd_memory_target_autotune parameter is set to true so that the OSD daemons adjust memory consumption based on the osd_memory_target option. The autotune_memory_target_ratio defaults to 0.7. This means 70 percent of the total RAM in the system is the starting point from which any memory consumed by non-autotuned Ceph daemons is subtracted. The remaining memory is divided between the OSDs; assuming all OSDs have osd_memory_target_autotune set to true. For HCI deployments, you can set mgr/cephadm/autotune_memory_target_ratio to 0.2 so that more memory is available for the Compute service. For additional information about service collocation, see Collocating services in a HCI environment for NUMA nodes . Note If these values need to be adjusted after the deployment, use the ceph config set osd <key> <value> command. Deploy Ceph Storage with the edited configuration file on a data plane node: USD cephadm bootstrap --config <config_file> --mon-ip <data_plane_node_ip> --skip-monitoring-stack Replace <config_file> with the name of your Ceph configuration file. Replace <data_plane_node_ip> with the Storage network IP address of the data plane node on which Red Hat Ceph Storage will be installed. Note The --skip-monitoring-stack option is used in the cephadm bootstrap command to skip the deployment of monitoring services. This ensures the Red Hat Ceph Storage deployment completes successfully if monitoring services have been previously deployed as part of any other preceding process. If monitoring services have not been deployed, see the Red Hat Ceph Storage documentation for information and procedures on enabling monitoring services. After the Red Hat Ceph Storage cluster is bootstrapped on the first EDPM node, see Red Hat Ceph Storage installation in the Red Hat Ceph Storage Installation Guide to add the other EDPM nodes to the Ceph cluster. 5.3. Configuring the DCN data plane Red Hat Ceph Storage must be configured as the storage solution before the data plane nodes can use it. Prerequisites Complete the procedures in Integrating Red Hat Ceph Storage . Procedure Edit the OpenStackDataPlaneNodeSet CR. To make the cephx key and configuration file available for the Compute service (nova), use the extraMounts parameter. The following is an example of using the extraMounts parameter for this purpose: Create a ConfigMap to add required configuration details to the the Compute service (nova). Create a file called ceph-nova-az0.yaml and add contents similar to the following. You must add the the Image service (glance) endpoint for the local availability zone, as well as set the cross_az_attach parameter to false: Create the ConfigMap : Create a custom Compute (nova) service to use the ConfigMap. Create a file called nova-custom-az0.yaml and add contents similar to the following. You must add the name of the ConfigMap that you just created under the dataSources field: Create the custom service: Note You must create a unique ConfigMap and custom Compute service for each availability zone. Append the availability zone to the end of these file names as shown in the steps. Locate the services list in the CR. Edit the services list to restore all of the services removed in Configuring the data plane node networks . Restoring the full services list allows the remaining jobs to be run that complete the configuration of the HCI environment. The following is an example of a full services list with the additional services in bold: Note In addition to restoring the default service list, the ceph-client service is added after the run-os service. The ceph-client service configures EDPM nodes as clients of a Red Hat Ceph Storage server. This service distributes the files necessary for the clients to connect to the Red Hat Ceph Storage server. The ceph-hci-pre service is only needed when you deploy HCI. Optional: You can assign compute nodes to Compute service (nova) cells the same as you can in any other environment. Replace the nova service in your OpenStackDataPlaneNodeSet CR with your custom nova service: For more information, see Connecting an OpenStackDataPlaneNodeSetSR to a Compute cell . Note If you are using cells, then the neutron-metadata service is unique per cell and defined separately. For example neutron-metadata-cell1 : The nova-custom-ceph service is unique for each availability zone and defined separately. For example, nova-custom-ceph-az0 : Optional: If you are deploying Red Hat Ceph Storage (RHCS) as a hyperconverged solution, complete the following steps: Create a ConfigMap to set the reserved_host_memory_mb parameter to a value appropriate for your configuration: The following is an example of a ConfigMap used for this purpose: Note The value for the reserved_host_memory_mb parameter may be set so that the Compute service scheduler does not give memory to a virtual machine that a Ceph OSD on the same server needs. The example reserves 5 GB per OSD for 10 OSDs per host in addition to the default reserved memory for the hypervisor. In an IOPS-optimized cluster, you can improve performance by reserving more memory for each OSD. The 5 GB number is provided as a starting point which can be further tuned if necessary. Edit the OpenStackDataPlaneService/nova-custom-ceph-az file. Add reserved-memory-nova to the configMaps list in the OpenStackDataPlaneService CR called ceph-nova-az0 that you created earlier: Apply the CR changes. Replace <dataplane_cr_file> with the name of your file. Note Ansible does not configure or validate the networks until the OpenStackDataPlaneDeployment CRD is created. Create an OpenStackDataPlaneDeployment CRD, as described in Creating the data plane in the Deploying Red Hat OpenStack Services on OpenShift guide, which has the OpenStackDataPlaneNodeSet CRD file defined above to have Ansible configure the services on the data plane nodes. example-node-set-resource 5.4. Example node set resource The following example has three nodes. It assumes the storage management network range is 172.20.0.0/24 and that it is on vlan23 . Note It is not necessary to add the storage management network to the networkAttachments key. 5.5. Updating the control plane After you have deployed the data plane at the central location, you must update the control plane. Prerequisites You have deployed a node set at the central location using Red Hat OpenStack services on OpenShift. You have deployed Red Hat Ceph Storage (RHCS). Procedure Optional: Configure the Block Storage backup service in your openstack_control_plane.yaml file: For more information about configuring the Block Storage backup service, see Configuring the Block Storage backup service . Update the Block Storage cinder volume service in your openstack_control_plane.yaml file: For more information about configuring the Block Storage volume service, see Configuring the volume service . Add the extraMounts field to your openstack_control_plane.yaml file to define the services that require access to the Red Hat Ceph Storage secret: Update the Image service (glance) in your openstack_control_plane.yaml file to configure the Block Storage service as the backend: Apply the changes made to the OpenStackControlPlane CR: Add the AZ to a host aggregate. This allows OpenStack administrators to schedule workloads to a geographical location based on image metadata. Open a terminal to the openstackclient pod: Create a new OpenStack aggregate: Label the OpenStack aggregate with the name of the AZ: Add each host in the AZ to the aggregate: 5.6. Updating the control plane after deploying DCN edge locations You can create additional node sets using the OpenStackDataPlaneNodeSet custom resource (CR). Use a unique availability zone, and the VLANs, NIC mappings and IP addresses specific to the site you are deploying. For more information on deploying an OpenStackDataPlaneNodeSet , see Creating the data plane . When you deploy a DCN node set with storage, you must update two fields of the OpenStackControlPlane CR at the central location: cinderVolumes glanceAPIs Neutron OVN Note If you are using cells, you must also configure cells for the new DCN site. Prerequisites You have deployed the central location. You have deployed an additional OpenStackDataPlane node set. Procedure In the neutron service configuration, update the customServiceConfig field to add the new availability zone and network leaf: In the OVN service configuration, update the availability zones: Update the cinderVolumes field in the OpenStackControlPlane CR to add the availability zones definitions of the remote location. Each cinder volume service in each availability zone uses the Glance API server for its availability zone. For example, glance_api_servers = https://glance-az1-internal.openstack.svc:9292 : Register an Image Service (glance) pod to the Identity Service (keystone) catalog: In DCN, an Image Service pod is deployed for each node set. A single Image Service pod is registered to the Identity service (keystone) catalog at any one time. For this reason, in the top-level Glance CR, a keystoneEndpoint parameter is defined and exposed. Unless a single instance is deployed, the human operator can choose, before the main OpenStackControlPlane CR is applied, which instance should be registered in keystone. Because our default endpoint is the az0 glance API the keystoneEndpoint is set to az0: Update the glanceAPIs field: For the node sets at az0 , the glanceAPIs field configures the Image service pods for the central location. When you add an additional node set in AZ1, the OpenStackControlPlane CR is updated such that the glanceAPIs field contains the Image service (glance) pod definition for AZ0 and AZ1. Additionally, the Image service pod for AZ1 defines the ceph the backend for the central location, and the AZ0 Image service pod for the central location is updated so that it has the ceph backend definition for AZ1. Note Availability zone az0 is of type split , and all other availability zones are of type edge . The split type is for cloud users to use when uploading images. The edge type is created so that when Cinder or Nova interact with Glance they can be configured to whichever glance is local to them. Use at least 3 replicas for the default split glance pods and 2 replicas for the edge glance pods and increase replicas proportionally to the workload. Optional: Update the Cell configuration. By default, compute nodes across all availability zones (AZs) are placed in a common cell called cell1 . You can increase the performance of large deployments by partitioning compute nodes into separate cells. For DCN deployments, place each availability zone into its own cell. For more information, see Adding Compute cells to the control plane . Apply the changes made to the OpenStackControlPlane CR: Continue to update the control plane for each additional edge site that is added. Add Red Hat Storage Service (RHCS) to your OpenShift secrets as needed. In the neutron service configuration, update the customServiceConfig field to add the new availability zone and network leaf: In the OVN service configuration, update the availability zones Add an additional availability zone the cinderVolumes service field: Add an additional availability zone to the glanceAPIs field: As you add additional AZs, you must ensure that each Image service pod definition contains the storage configuration of the central location (AZ0), and its own local ceph configuration. You must also ensure that the central location has the storage definition of all other sites. This creates a hub and spoke relationship between the central location Image service pod, and the Image service pods for geographically dispersed node sets: Apply the changes made to the OpenStackControlPlane CR: Add the AZ to a host aggregate. This allows OpenStack administrators to schedule workloads to a geographical location by passing the --availabliity-zone argument: Open a terminal to the openstackclient pod: Create a new OpenStack aggregate: Label the OpenStack aggregate with the name of the AZ: Add each host in the AZ to the aggregate:

Commands:
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet metadata: name: dcn-data-plane-networks namespace: openstack spec: env: - name: ANSIBLE_FORCE_COLOR value: "True"
spec: services: - bootstrap - configure-network - validate-network - install-os - ceph-hci-pre - configure-os - ssh-known-hosts - run-os - reboot-os
spec: env: - name: ANSIBLE_FORCE_COLOR value: "True" networkAttachments: - ctlplane nodeTemplate: ansible: ansiblePort: 22 ansibleUser: cloud-admin ansibleVars: edpm_enable_chassis_gw: true edpm_ovn_availability_zones: - az0
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet metadata: name: openstack-edpm namespace: openstack spec: env: - name: ANSIBLE_FORCE_COLOR value: "True" networkAttachments: - ctlplane nodeTemplate: ansible: ansiblePort: 22 ansibleUser: cloud-admin ansibleVars: edpm_ceph_hci_pre_enabled_services: - ceph_mon - ceph_mgr - ceph_osd
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet metadata: name: openstack-edpm namespace: openstack spec: env: - name: ANSIBLE_FORCE_COLOR value: "True" networkAttachments: - ctlplane nodeTemplate: ansible: ansiblePort: 22 ansibleUser: cloud-admin ansibleVars: edpm_ceph_hci_pre_enabled_services: - ceph_mon - ceph_mgr - ceph_osd edpm_fips_mode: check edpm_iscsid_image: {{ registry_url }}/openstack-iscsid:{{ image_tag }} edpm_logrotate_crond_image: {{ registry_url }}/openstack-cron:{{ image_tag }} edpm_network_config_hide_sensitive_logs: false edpm_network_config_os_net_config_mappings: edpm-compute-0: nic1: 52:54:00:1e:af:6b nic2: 52:54:00:d9:cb:f4 edpm-compute-1: nic1: 52:54:00:f2:bc:af nic2: 52:54:00:f1:c7:dd edpm-compute-2: nic1: 52:54:00:dd:33:14 nic2: 52:54:00:50:fb:c3 edpm_network_config_template: | --- {% set mtu_list = [ctlplane_mtu] %} {% for network in nodeset_networks %} {{ mtu_list.append(lookup(vars, networks_lower[network] ~ _mtu)) }} {%- endfor %} {% set min_viable_mtu = mtu_list | max %} network_config: - type: ovs_bridge name: {{ neutron_physical_bridge_name }} mtu: {{ min_viable_mtu }} use_dhcp: false dns_servers: {{ ctlplane_dns_nameservers }} domain: {{ dns_search_domains }} addresses: - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }} routes: {{ ctlplane_host_routes }} members: - type: interface name: nic2 mtu: {{ min_viable_mtu }} # force the MAC address of the bridge to this interface primary: true {% for network in nodeset_networks %} - type: vlan mtu: {{ lookup(vars, networks_lower[network] ~ _mtu) }} vlan_id: {{ lookup(vars, networks_lower[network] ~ _vlan_id) }} addresses: - ip_netmask: {{ lookup(vars, networks_lower[network] ~ _ip) }}/{{ lookup(vars, networks_lower[network] ~ _cidr) }} routes: {{ lookup(vars, networks_lower[network] ~ _host_routes) }} {% endfor %} edpm_neutron_metadata_agent_image: {{ registry_url }}/openstack-neutron-metadata-agent-ovn:{{ image_tag }} edpm_nodes_validation_validate_controllers_icmp: false edpm_nodes_validation_validate_gateway_icmp: false edpm_selinux_mode: enforcing edpm_sshd_allowed_ranges: - 192.168.111.0/24 - 192.168.122.0/24 - 192.168.133.0/24 - 192.168.144.0/24 edpm_sshd_configure_firewall: true enable_debug: false gather_facts: false image_tag: current-podified neutron_physical_bridge_name: br-ex neutron_public_interface_name: eth0 service_net_map: nova_api_network: internalapi nova_libvirt_network: internalapi storage_mgmt_cidr: "24" storage_mgmt_host_routes: [] storage_mgmt_mtu: 9000 storage_mgmt_vlan_id: 23 storage_mtu: 9000 timesync_ntp_servers: - hostname: pool.ntp.org ansibleSSHPrivateKeySecret: dataplane-ansible-ssh-private-key-secret managementNetwork: ctlplane networks: - defaultRoute: true name: ctlplane subnetName: subnet1 - name: internalapi subnetName: subnet1 - name: storage subnetName: subnet1 - name: tenant subnetName: subnet1 nodes: edpm-compute-0: ansible: host: 192.168.122.100 hostName: compute-0 networks: - defaultRoute: true fixedIP: 192.168.122.100 name: ctlplane subnetName: subnet1 - name: internalapi subnetName: subnet1 - name: storage subnetName: subnet1 - name: storagemgmt subnetName: subnet1 - name: tenant subnetName: subnet1 edpm-compute-1: ansible: host: 192.168.122.101 hostName: compute-1 networks: - defaultRoute: true fixedIP: 192.168.122.101 name: ctlplane subnetName: subnet1 - name: internalapi subnetName: subnet1 - name: storage subnetName: subnet1 - name: storagemgmt subnetName: subnet1 - name: tenant subnetName: subnet1 edpm-compute-2: ansible: host: 192.168.122.102 hostName: compute-2 networks: - defaultRoute: true fixedIP: 192.168.122.102 name: ctlplane subnetName: subnet1 - name: internalapi subnetName: subnet1 - name: storage subnetName: subnet1 - name: storagemgmt subnetName: subnet1 - name: tenant subnetName: subnet1 preProvisioned: true services: - bootstrap - configure-network - validate-network - install-os - ceph-hci-pre - configure-os - ssh-known-hosts - run-os - reboot-os
oc apply -f <dataplane_cr_file>
[global] public_network = 172.18.0.0/24 cluster_network = 172.20.0.0/24
[osd] osd_memory_target_autotune = true osd_numa_auto_affinity = true [mgr] mgr/cephadm/autotune_memory_target_ratio = 0.2
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet spec: nodeTemplate: extraMounts: - extraVolType: Ceph volumes: - name: ceph secret: secretName: ceph-conf-files mounts: - name: ceph mountPath: "/etc/ceph" readOnly: true
apiVersion: v1 kind: ConfigMap metadata: name: ceph-nova-az0 namespace: openstack data: 03-ceph-nova.conf: [libvirt] images_type = rbd images_rbd_pool = vms images_rbd_ceph_conf = /etc/ceph/az0.conf images_rbd_glance_store_name = az0 images_rbd_glance_copy_poll_interval = 15 images_rbd_glance_copy_timeout = 600 rbd_user = openstack rbd_secret_uuid = 9cfb3a03-3f91-516a-881e-a675f67c30ea hw_disk_discard = unmap volume_use_multipath = False [glance] endpoint_override = http://glance-az0-internal.openstack.svc:9292 valid_interfaces = internal [cinder] cross_az_attach = False catalog_info = volumev3:cinderv3:internalURL
create -f ceph-nova-az0.yaml
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneService metadata: name: nova-custom-ceph-az0 spec: addCertMounts: false caCerts: combined-ca-bundle dataSources: - configMapRef: name: ceph-nova-az0 - secretRef: name: nova-cell1-compute-config - secretRef: name: nova-migration-ssh-key edpmServiceType: nova playbook: osp.edpm.nova tlsCerts: default: contents: - dnsnames - ips edpmRoleServiceName: nova issuer: osp-rootca-issuer-internal networks: - ctlplane
create -f nova-custom-ceph-az0.yaml
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet spec: services: - bootstrap - configure-network - validate-network - install-os - ceph-hci-pre - configure-os - ssh-known-hosts - run-os - reboot-os - install-certs - ceph-client - ovn - neutron-metadata - libvirt - nova-custom-ceph-az0
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet metadata: name: openstack-cell2 spec: services: - download-cache - bootstrap - configure-network - validate-network - install-os - configure-os - ssh-known-hosts - run-os - ovn - libvirt - *nova-cell-custom*
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneService metadata: labels: app.kubernetes.io/instance: neutron-metadata-cell1 app.kubernetes.io/name: openstackdataplaneservice app.kubernetes.io/part-of: openstack-operator name: neutron-metadata-cell1
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneService metadata: labels: app.kubernetes.io/instance: nova-custom-ceph-az0 app.kubernetes.io/name: openstackdataplaneservice app.kubernetes.io/part-of: openstack-operator name: nova-custom-ceph-az0 namespace: openstack
apiVersion: v1 kind: ConfigMap metadata: name: reserved-memory-nova data: 04-reserved-memory-nova.conf: | [DEFAULT] reserved_host_memory_mb=75000
kind: OpenStackDataPlaneService <...> spec: configMaps: - ceph-nova - reserved-memory-nova
oc apply -f <dataplane_cr_file>
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet metadata: name: openstack-edpm namespace: openstack spec: env: - name: ANSIBLE_FORCE_COLOR value: "True" networkAttachments: - ctlplane nodeTemplate: ansible: ansiblePort: 22 ansibleUser: cloud-admin ansibleVars: edpm_ceph_hci_pre_enabled_services: - ceph_mon - ceph_mgr - ceph_osd edpm_fips_mode: check edpm_iscsid_image: {{ registry_url }}/openstack-iscsid:{{ image_tag }} edpm_logrotate_crond_image: {{ registry_url }}/openstack-cron:{{ image_tag }} edpm_network_config_hide_sensitive_logs: false edpm_network_config_os_net_config_mappings: edpm-compute-0: nic1: 52:54:00:1e:af:6b nic2: 52:54:00:d9:cb:f4 edpm-compute-1: nic1: 52:54:00:f2:bc:af nic2: 52:54:00:f1:c7:dd edpm-compute-2: nic1: 52:54:00:dd:33:14 nic2: 52:54:00:50:fb:c3 edpm_network_config_template: | --- {% set mtu_list = [ctlplane_mtu] %} {% for network in nodeset_networks %} {{ mtu_list.append(lookup( vars , networks_lower[network] ~ _mtu )) }} {%- endfor %} {% set min_viable_mtu = mtu_list | max %} network_config: - type: ovs_bridge name: {{ neutron_physical_bridge_name }} mtu: {{ min_viable_mtu }} use_dhcp: false dns_servers: {{ ctlplane_dns_nameservers }} domain: {{ dns_search_domains }} addresses: - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }} routes: {{ ctlplane_host_routes }} members: - type: interface name: nic2 mtu: {{ min_viable_mtu }} # force the MAC address of the bridge to this interface primary: true {% for network in nodeset_networks %} - type: vlan mtu: {{ lookup( vars , networks_lower[network] ~ _mtu ) }} vlan_id: {{ lookup( vars , networks_lower[network] ~ _vlan_id ) }} addresses: - ip_netmask: {{ lookup( vars , networks_lower[network] ~ _ip ) }}/{{ lookup( vars , networks_lower[network] ~ _cidr ) }} routes: {{ lookup( vars , networks_lower[network] ~ _host_routes ) }} {% endfor %} edpm_neutron_metadata_agent_image: {{ registry_url }}/openstack-neutron-metadata-agent-ovn:{{ image_tag }} edpm_nodes_validation_validate_controllers_icmp: false edpm_nodes_validation_validate_gateway_icmp: false edpm_selinux_mode: enforcing edpm_sshd_allowed_ranges: - 192.168.111.0/24 - 192.168.122.0/24 - 192.168.133.0/24 - 192.168.144.0/24 edpm_sshd_configure_firewall: true enable_debug: false gather_facts: false image_tag: current-podified neutron_physical_bridge_name: br-ex neutron_public_interface_name: eth0 service_net_map: nova_api_network: internalapi nova_libvirt_network: internalapi storage_mgmt_cidr: "24" storage_mgmt_host_routes: [] storage_mgmt_mtu: 9000 storage_mgmt_vlan_id: 23 storage_mtu: 9000 timesync_ntp_servers: - hostname: pool.ntp.org ansibleSSHPrivateKeySecret: dataplane-ansible-ssh-private-key-secret managementNetwork: ctlplane networks: - defaultRoute: true name: ctlplane subnetName: subnet1 - name: internalapi subnetName: subnet1 - name: storage subnetName: subnet1 - name: tenant subnetName: subnet1 nodes: edpm-compute-0: ansible: host: 192.168.122.100 hostName: compute-0 networks: - defaultRoute: true fixedIP: 192.168.122.100 name: ctlplane subnetName: subnet1 - name: internalapi subnetName: subnet1 - name: storage subnetName: subnet1 - name: storagemgmt subnetName: subnet1 - name: tenant subnetName: subnet1 edpm-compute-1: ansible: host: 192.168.122.101 hostName: compute-1 networks: - defaultRoute: true fixedIP: 192.168.122.101 name: ctlplane subnetName: subnet1 - name: internalapi subnetName: subnet1 - name: storage subnetName: subnet1 - name: storagemgmt subnetName: subnet1 - name: tenant subnetName: subnet1 edpm-compute-2: ansible: host: 192.168.122.102 hostName: compute-2 networks: - defaultRoute: true fixedIP: 192.168.122.102 name: ctlplane subnetName: subnet1 - name: internalapi subnetName: subnet1 - name: storage subnetName: subnet1 - name: storagemgmt subnetName: subnet1 - name: tenant subnetName: subnet1 preProvisioned: true services: - bootstrap - configure-network - validate-network - install-os - ceph-hci-pre - configure-os - ssh-known-hosts - run-os - reboot-os
cinderBackup: customServiceConfig: | [DEFAULT] backup_driver = cinder.backup.drivers.ceph.CephBackupDriver backup_ceph_pool = backups backup_ceph_user = openstack
cinderVolumes: az0: customServiceConfig: | [DEFAULT] enabled_backends = ceph glance_api_servers = https://glance-az0-internal.openstack.svc:9292 [ceph] volume_backend_name = ceph volume_driver = cinder.volume.drivers.rbd.RBDDriver rbd_ceph_conf = /etc/ceph/az0.conf rbd_user = openstack rbd_pool = volumes rbd_flatten_volume_from_snapshot = False rbd_secret_uuid = 795dcbca-e715-5ac3-9b7e-a3f5c64eb89f rbd_cluster_name = az0 backend_availability_zone = az0
extraMounts: - extraVol: - extraVolType: Ceph mounts: - mountPath: /etc/ceph name: ceph readOnly: true propagation: - az0 - CinderBackup volumes: - name: ceph projected: sources: - secret: name: ceph-conf-az-0
glanceAPIs: az0: customServiceConfig: | [DEFAULT] enabled_import_methods = [web-download,copy-image,glance-direct] enabled_backends = az0:rbd [glance_store] default_backend = az0 [az0] rbd_store_ceph_conf = /etc/ceph/az0.conf store_description = "az0 RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True
apply -f openstack_control_plane.yaml
oc rsh openstackclient
openstack aggregate create <aggregate_name>
openstack aggregate set --zone <availability_zone> <aggregate_name>
openstack aggregate add host <aggregate_name> <compute_node_1> openstack aggregate add host <aggregate_name> <compute_node_2>
customServiceConfig: | [DEFAULT] router_scheduler_driver = neutron.scheduler.l3_agent_scheduler.AZLeastRoutersScheduler network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.AZAwareWeightScheduler default_availability_zones = az0,az1 [ml2_type_vlan] network_vlan_ranges = datacentre:1:1000,leaf1:1:1000 [neutron] physnets = datacentre,leaf
ovnController: external-ids: availability-zones: - az0 - az1 enable-chassis-as-gateway ovn-bridge: br-int onv-enap-type: geneve system-id: random networkAttachment: tenant nicMappings: datacentre: ospbr
cinderVolumes: az0: customServiceConfig: | [DEFAULT] . az1: customServiceConfig: | [DEFAULT] enabled_backends = ceph glance_api_servers = https://glance-az1-internal.openstack.svc:9292 [ceph] volume_backend_name = az1 volume_driver = cinder.volume.drivers.rbd.RBDDriver rbd_ceph_conf = /etc/ceph/az1.conf rbd_user = openstack rbd_pool = volumes rbd_flatten_volume_from_snapshot = False rbd_secret_uuid = 19ccdd60-79a0-5f0f-aece-ece700e514f8 rbd_cluster_name = az1 backend_availability_zone = az1
spec: <...> glance: enabled: true keystoneEndpoint: az0 glanceAPIs: az0: apiTimeout: 60
glanceAPIs: az1: customServiceConfig: | [DEFAULT] enabled_import_methods = [web-download,copy-image,glance-direct] enabled_backends = az0:rbd,az1:rbd [glance_store] default_backend = az1 [az1] rbd_store_ceph_conf = /etc/ceph/az1.conf store_description = "az1 RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True [az0] rbd_store_ceph_conf = /etc/ceph/az0.conf store_description = "az0 RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True networkAttachments: - storage override: service: internal: metadata: annotations: metallb.universe.tf/address-pool: internalapi metallb.universe.tf/allow-shared-ip: internalapi metallb.universe.tf/loadBalancerIPs: 172.17.0.81 spec: type: LoadBalancer replicas: 2 type: edge az0: customServiceConfig: | [DEFAULT] enabled_import_methods = [web-download,copy-image,glance-direct] enabled_backends = az0:rbd,az1:rbd [glance_store] default_backend = az0 [az0] rbd_store_ceph_conf = /etc/ceph/az0.conf store_description = "az0 RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True [az1] rbd_store_ceph_conf = /etc/ceph/az1.conf store_description = "az1 RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True networkAttachments: - storage override: service: internal: metadata: annotations: metallb.universe.tf/address-pool: internalapi metallb.universe.tf/allow-shared-ip: internalapi metallb.universe.tf/loadBalancerIPs: 172.17.0.80 spec: type: LoadBalancer replicas: 3 type: split
apply -f openstack_control_plane.yaml
customServiceConfig: | [DEFAULT] router_scheduler_driver = neutron.scheduler.l3_agent_scheduler.AZLeastRoutersScheduler network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.AZAwareWeightScheduler default_availability_zones = az0,az1,az2 [ml2_type_vlan] network_vlan_ranges = datacentre:1:1000,leaf1:1:1000,leaf2:1:1000 [neutron] physnets = datacentre,leaf1,leaf2
ovnController: external-ids: availability-zones: - az0 - az1 - az2 enable-chassis-as-gateway ovn-bridge: br-int onv-enap-type: geneve system-id: random networkAttachment: tenant nicMappings: datacentre: ospbr
cinderVolumes: az0: customServiceConfig: | [DEFAULT] az1: customServiceConfig: | [DEFAULT] az2: customServiceConfig: | [DEFAULT] enabled_backends = ceph glance_api_servers = https://glance-az2-internal.openstack.svc:9292 [ceph] volume_backend_name = ceph volume_driver = cinder.volume.drivers.rbd.RBDDriver rbd_ceph_conf = /etc/ceph/az2.conf rbd_user = openstack rbd_pool = volumes rbd_flatten_volume_from_snapshot = False rbd_secret_uuid = 5c0c7a8e-55b1-5fa8-bc5c-9756b7862d2f rbd_cluster_name = az2 backend_availability_zone = az2
glanceAPIs: az0: customServiceConfig: | [DEFAULT] enabled_import_methods = [web-download,copy-image,glance-direct] enabled_backends = az0:rbd,az1:rbd,az2:rbd [glance_store] default_backend = az0 [az0] rbd_store_ceph_conf = /etc/ceph/az0.conf store_description = "az0 RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True [az1] rbd_store_ceph_conf = /etc/ceph/az1.conf store_description = "az1 RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True [az2] rbd_store_ceph_conf = /etc/ceph/az2.conf store_description = "az2 RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True networkAttachments: - storage override: service: internal: metadata: annotations: metallb.universe.tf/address-pool: internalapi metallb.universe.tf/allow-shared-ip: internalapi metallb.universe.tf/loadBalancerIPs: 172.17.0.80 spec: type: LoadBalancer replicas: 3 type: split az1: customServiceConfig: | [DEFAULT] enabled_import_methods = [web-download,copy-image,glance-direct] enabled_backends = az0:rbd,az1:rbd [glance_store] default_backend = az1 [az1] rbd_store_ceph_conf = /etc/ceph/az1.conf store_description = "az1 RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True [az0] rbd_store_ceph_conf = /etc/ceph/az0.conf store_description = "az0 RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True networkAttachments: - storage override: service: internal: metadata: annotations: metallb.universe.tf/address-pool: internalapi metallb.universe.tf/allow-shared-ip: internalapi metallb.universe.tf/loadBalancerIPs: 172.17.0.81 spec: type: LoadBalancer replicas: 2 type: edge az2: customServiceConfig: | [DEFAULT] enabled_import_methods = [web-download,copy-image,glance-direct] enabled_backends = az0:rbd,az2:rbd [glance_store] default_backend = az2 [az2] rbd_store_ceph_conf = /etc/ceph/az2.conf store_description = "az2 RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True [az0] rbd_store_ceph_conf = /etc/ceph/az0.conf store_description = "az0 RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True networkAttachments: - storage override: service: internal: metadata: annotations: metallb.universe.tf/address-pool: internalapi metallb.universe.tf/allow-shared-ip: internalapi metallb.universe.tf/loadBalancerIPs: 172.17.0.82 spec: type: LoadBalancer replicas: 2 type: edge
apply -f openstack_control_plane.yaml
oc rsh openstackclient
openstack aggregate create <aggregate_name>
openstack aggregate set --zone <availability_zone> <aggregate_name>
openstack aggregate add host <aggregate_name> <compute_node_1> openstack aggregate add host <aggregate_name> <compute_node_2>