Chapter 2. Planning a Distributed Compute Node (DCN) deployment When you plan your DCN architecture, check that the technologies that you need are available and supported. 2.1. Considerations for storage on DCN architecture The following features are not currently supported for DCN architectures: Copying a volume between edge sites. You can work around this by creating an image from the volume and using the Image service (glance) to copy the image. After the image is copied, you can create a volume from it. Ceph Rados Gateway (RGW) at the edge sites. CephFS at the edge sites. Instance high availability (HA) at the edge sites. RBD mirroring between edge sites. Instance migration, live or cold, either between edge sites, or from the central location to edge sites. You can only migrate instances within a site boundary. To move an image between sites, you must snapshot the image, and use glance image-import . Additionally, you must consider the following: You must upload images to the central location before copying them to edge sites. A copy of each image must exist in the Image service at the central location. You must use the RBD storage driver for the Image, Compute and Block Storage services. For each site, including the central location, assign a unique availability zone. You can migrate an offline volume from an edge site to the central location, or vice versa. You cannot migrate volumes directly between edge sites. 2.2. Considerations for networking on DCN architecture The following features are not currently supported for DCN architectures: DHCP on DPDK nodes TC Flower Hardware Offload The following ML2/OVN networking technologies are fully supported: Routed provider networks OVN GW (networker node) with Neutron AZs supported Additionally, you must consider the following: Network latency: Balance the latency as measured in round-trip time (RTT), with the expected number of concurrent API operations to maintain acceptable performance. Maximum TCP/IP throughput is inversely proportional to RTT. You can mitigate some issues with high-latency connections with high bandwidth by tuning kernel TCP parameters. Contact Red Hat Support if a cross-site communication exceeds 100 ms. Network drop outs: If the edge site temporarily loses connection to the central site, then no control plane API or CLI operations can be executed at the impacted edge site for the duration of the outage. For example, Compute nodes at the edge site are consequently unable to create a snapshot of an instance, issue an auth token, or delete an image. General control plane API and CLI operations remain functional during this outage, and can continue to serve any other edge sites that have a working connection. Image type: You must use raw images when deploying a DCN architecture with Ceph storage. Image sizing: Compute images: Compute images are downloaded from the central location. These images are potentially large files that are transferred across all necessary networks from the central site to the edge site during provisioning. Instance images: If there is no block storage at the edge, then the Image service images traverse the WAN during first use. The images are copied or cached locally to the target edge nodes for all subsequent use. There is no size limit for images. Transfer times vary with available bandwidth and network latency. Provider networks are the most common approach for DCN deployments. Note that the Networking service (neutron) does not validate where you can attach available networks. For example, if you use a provider network called "site-a" only in edge site A, the Networking service does not validate and prevent you from attaching "site-a" to an instance at site B, which does not work. Site-specific networks: A limitation in DCN networking arises if you use networks that are specific to a certain site: When you deploy centralized neutron controllers with Compute nodes, there are no triggers in the Networking service to identify a certain Compute node as a remote node. Consequently, the Compute nodes receive a list of other Compute nodes and automatically form tunnels between each other. The tunnels are formed from edge to edge through the central site. If you use VXLAN or GENEVE, every Compute node at every site forms a tunnel with every other Compute node, whether or not they are local or remote. This is not an issue if you are using the same networks everywhere. When you use VLANs, the Networking service expects that all Compute nodes have the same bridge mappings, and that all VLANs are available at every site. If edge servers are not pre-provisioned, you must configure DHCP relay for introspection and provisioning on routed segments. Routing must be configured either on the cloud or within the networking infrastructure that connects each edge site to the hub. You should implement a networking design that allocates an L3 subnet for each RHOSO cluster network (external, internal API, and so on), unique to each site. 2.3. IP Address pool sizing for the internalapi network The Image service operator creates an endpoint for each Image service pod with its own DNS name, such as glance-az0-internal.openstack.svc:9292 . Each Compute service and Block storage service in each availability zone uses the Image service (glance) API server in the same availability zone. For example, when you update the cinderVolumes field in the OpenStackControlPlane custom resource (CR), add a field called glance_api_servers under customServiceConfig : The Image service endpoint DNS name maps to a load balancer IP address in the internalapi address pool as indicated by the internal metadata annotations: The range of addresses in this address pool should be sized according to the number of DCN sites. For example, the following shows only 10 available addresses in the internalapi network. Use commands like the following after updating the glance section of the OpenStackControlPlane CR in order to confirm that the Glance Operator has created the service endpoint and route.

Commands:
cinderVolumes: az0: customServiceConfig: | [DEFAULT] enabled_backends = az0 glance_api_servers = https://glance-az0-internal.openstack.svc:9292
[glance_store] default_backend = ceph [ceph] rbd_store_ceph_conf = /etc/ceph/ceph.conf store_description = "ceph RBD backend" rbd_store_pool = images rbd_store_user = openstack rbd_thin_provisioning = True networkAttachments: - storage override: service: internal: metadata: annotations: metallb.universe.tf/address-pool: internalapi metallb.universe.tf/allow-shared-ip: internalapi metallb.universe.tf/loadBalancerIPs: 172.17.0.80
oc get ipaddresspool -n metallb-system NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES ctlplane true false ["192.168.122.80-192.168.122.90"] internalapi true false ["172.17.0.80-172.17.0.90"] storage true false ["172.18.0.80-172.18.0.90"] tenant true false ["172.19.0.80-172.19.0.90"]
oc get svc | grep glance glance-az0-internal LoadBalancer 172.30.217.178 172.17.0.80 9292:32134/TCP 24h glance-az0-public ClusterIP 172.30.78.47 <none> 9292/TCP 24h glance-az1-internal LoadBalancer 172.30.52.123 172.17.0.81 9292:31679/TCP 23h glance-c1ca8-az0-external-api ClusterIP None <none> 9292/TCP 24h glance-c1ca8-az0-internal-api ClusterIP None <none> 9292/TCP 24h glance-c1ca8-az1-edge-api ClusterIP None <none> 9292/TCP 23h oc get route | grep glance glance-az0-public glance-az0-public-openstack.apps.ocp.openstack.lab glance-az0-public glance-az0-public reencrypt/Redirect None glance-default-public glance-default-public-openstack.apps.ocp.openstack.lab glance-default-public glance-default-public reencrypt/Redirect None