Chapter 5. Preparing networks for Red Hat OpenStack Services on OpenShift To prepare for configuring and deploying your Red Hat OpenStack Services on OpenShift (RHOSO) environment, you must configure the Red Hat OpenShift Container Platform (RHOCP) networks on your RHOCP cluster. 5.1. Default Red Hat OpenStack Services on OpenShift networks The following physical data center networks are typically implemented for a Red Hat OpenStack Services on OpenShift (RHOSO) deployment: Control plane network: This network is used by the OpenStack Operator for Ansible SSH access to deploy and connect to the data plane nodes from the Red Hat OpenShift Container Platform (RHOCP) environment. This network is also used by data plane nodes for live migration of instances. External network: (Optional) You can configure an external network if one is required for your environment. For example, you might create an external network for any of the following purposes: To provide virtual machine instances with Internet access. To create flat provider networks that are separate from the control plane. To configure VLAN provider networks on a separate bridge from the control plane. To provide access to virtual machine instances with floating IPs on a network other than the control plane network. Internal API network: This network is used for internal communication between RHOSO components. Storage network: This network is used for block storage, RBD, NFS, FC, and iSCSI. Tenant (project) network: This network is used for data communication between virtual machine instances within the cloud deployment. Storage Management network: (Optional) This network is used by storage components. For example, Red Hat Ceph Storage uses the Storage Management network in a hyperconverged infrastructure (HCI) environment as the cluster_network to replicate data. Note For more information on Red Hat Ceph Storage network configuration, see Ceph network configuration in the Red Hat Ceph Storage Configuration Guide . The following table details the default networks used in a RHOSO deployment. If required, you can update the networks for your environment. Note By default, the control plane and external networks do not use VLANs. Networks that do not use VLANs must be placed on separate NICs. You can use a VLAN for the control plane network on new RHOSO deployments. You can also use the Native VLAN on a trunked interface as the non-VLAN network. For example, you can have the control plane and the internal API on one NIC, and the external network with no VLAN on a separate NIC. Table 5.1. Default RHOSO networks Network name VLAN CIDR NetConfig allocationRange MetalLB IPAddressPool range net-attach-def ipam range OCP worker nncp range ctlplane n/a 192.168.122.0/24 192.168.122.100 - 192.168.122.250 192.168.122.80 - 192.168.122.90 192.168.122.30 - 192.168.122.70 192.168.122.10 - 192.168.122.20 external n/a 10.0.0.0/24 10.0.0.100 - 10.0.0.250 n/a n/a internalapi 20 172.17.0.0/24 172.17.0.100 - 172.17.0.250 172.17.0.80 - 172.17.0.90 172.17.0.30 - 172.17.0.70 172.17.0.10 - 172.17.0.20 storage 21 172.18.0.0/24 172.18.0.100 - 172.18.0.250 n/a 172.18.0.30 - 172.18.0.70 172.18.0.10 - 172.18.0.20 tenant 22 172.19.0.0/24 172.19.0.100 - 172.19.0.250 n/a 172.19.0.30 - 172.19.0.70 172.19.0.10 - 172.19.0.20 storageMgmt 23 172.20.0.0/24 172.20.0.100 - 172.20.0.250 n/a 172.20.0.30 - 172.20.0.70 172.20.0.10 - 172.20.0.20 5.2. Preparing RHOCP for RHOSO networks The Red Hat OpenStack Services on OpenShift (RHOSO) services run as a Red Hat OpenShift Container Platform (RHOCP) workload. You use the NMState Operator to connect the worker nodes to the required isolated networks. You use the MetalLB Operator to expose internal service endpoints on the isolated networks. By default, the public service endpoints are exposed as RHOCP routes. Note The examples in the following procedures use IPv4 addresses. You can use IPv6 addresses instead of IPv4 addresses. Dual stack IPv4/6 is not available. For information about how to configure IPv6 addresses, see the following resources in the RHOCP Networking guide: Installing the Kubernetes NMState Operator Configuring MetalLB address pools 5.2.1. Preparing RHOCP with isolated network interfaces Create a NodeNetworkConfigurationPolicy ( nncp ) CR to configure the interfaces for each isolated network on each worker node in RHOCP cluster. Procedure Create a NodeNetworkConfigurationPolicy ( nncp ) CR file on your workstation, for example, openstack-nncp.yaml . Retrieve the names of the worker nodes in the RHOCP cluster: Discover the network configuration: Replace <worker_node> with the name of a worker node retrieved in step 2, for example, worker-1 . Repeat this step for each worker node. In the nncp CR file, configure the interfaces for each isolated network on each worker node in the RHOCP cluster. For information about the default physical data center networks that must be configured with network isolation, see Default Red Hat OpenStack Services on OpenShift networks . In the following example, the nncp CR configures the enp6s0 interface for worker node 1, osp-enp6s0-worker-1 , to use VLAN interfaces with IPv4 addresses for network isolation: Create the nncp CR in the cluster: Verify that the nncp CR is created: 5.2.2. Attaching service pods to the isolated networks Create a NetworkAttachmentDefinition ( net-attach-def ) custom resource (CR) for each isolated network to attach the service pods to the networks. Procedure Create a NetworkAttachmentDefinition ( net-attach-def ) CR file on your workstation, for example, openstack-net-attach-def.yaml . In the NetworkAttachmentDefinition CR file, configure a NetworkAttachmentDefinition resource for each isolated network to attach a service deployment pod to the network. The following examples create a NetworkAttachmentDefinition resource for the internalapi , storage , ctlplane , and tenant networks of type macvlan : 1 The namespace where the services are deployed. 2 The node interface name associated with the network, as defined in the nncp CR. 3 The whereabouts CNI IPAM plugin to assign IPs to the created pods from the range .30 - .70 . 4 The IP address pool range must not overlap with the MetalLB IPAddressPool range and the NetConfig allocationRange . Create the NetworkAttachmentDefinition CR in the cluster: Verify that the NetworkAttachmentDefinition CR is created: 5.2.3. Preparing RHOCP for RHOSO network VIPS The Red Hat OpenStack Services on OpenShift (RHOSO) services run as a Red Hat OpenShift Container Platform (RHOCP) workload. You must create an L2Advertisement resource to define how the Virtual IPs (VIPs) are announced, and an IPAddressPool resource to configure which IPs can be used as VIPs. In layer 2 mode, one node assumes the responsibility of advertising a service to the local network. Procedure Create an IPAddressPool CR file on your workstation, for example, openstack-ipaddresspools.yaml . In the IPAddressPool CR file, configure an IPAddressPool resource on the isolated network to specify the IP address ranges over which MetalLB has authority: 1 The IPAddressPool range must not overlap with the whereabouts IPAM range and the NetConfig allocationRange . For information about how to configure the other IPAddressPool resource parameters, see Configuring MetalLB address pools in the RHOCP Networking guide. Create the IPAddressPool CR in the cluster: Verify that the IPAddressPool CR is created: Create a L2Advertisement CR file on your workstation, for example, openstack-l2advertisement.yaml . In the L2Advertisement CR file, configure L2Advertisement CRs to define which node advertises a service to the local network. Create one L2Advertisement resource for each network. In the following example, each L2Advertisement CR specifies that the VIPs requested from the network address pools are announced on the interface that is attached to the VLAN: 1 The interface where the VIPs requested from the VLAN address pool are announced. For information about how to configure the other L2Advertisement resource parameters, see Configuring MetalLB with a L2 advertisement and label in the RHOCP Networking guide. Create the L2Advertisement CRs in the cluster: Verify that the L2Advertisement CRs are created: If your cluster has OVNKubernetes as the network back end, then you must enable global forwarding so that MetalLB can work on a secondary network interface. Check the network back end used by your cluster: If the back end is OVNKubernetes, then run the following command to enable global IP forwarding: 5.3. Creating the data plane network To create the data plane network, you define a NetConfig custom resource (CR) and specify all the subnets for the data plane networks. You must define at least one control plane network for your data plane. You can also define VLAN networks to create network isolation for composable networks, such as InternalAPI , Storage , and External . Each network definition must include the IP address assignment. Tip Use the following commands to view the NetConfig CRD definition and specification schema: Procedure Create a file named openstack_netconfig.yaml on your workstation. Add the following configuration to openstack_netconfig.yaml to create the NetConfig CR: In the openstack_netconfig.yaml file, define the topology for each data plane network. To use the default Red Hat OpenStack Services on OpenShift (RHOSO) networks, you must define a specification for each network. For information about the default RHOSO networks, see Default Red Hat OpenStack Services on OpenShift networks . The following example creates isolated networks for the data plane: 1 The name of the network, for example, CtlPlane . 2 The IPv4 subnet specification. 3 The name of the subnet, for example, subnet1 . 4 The NetConfig allocationRange . The allocationRange must not overlap with the MetalLB IPAddressPool range and the IP address pool range. 5 Optional: List of IP addresses from the allocation range that must not be used by data plane nodes. 6 The network VLAN. For information about the default RHOSO networks, see Default Red Hat OpenStack Services on OpenShift networks . 7 bgpnet0 is the network used by the data plane node to communicate with its BGP peer. 8 bgpnet1 is the network used by the data plane node to communicate with its BGP peer. Save the openstack_netconfig.yaml definition file. Create the data plane network: To verify that the data plane network is created, view the openstacknetconfig resource: If you see errors, check the underlying network-attach-definition and node network configuration policies:

Commands:
oc get nodes -l node-role.kubernetes.io/worker -o jsonpath="{.items[*].metadata.name}"
oc get nns/<worker_node> -o yaml | more
apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: osp-enp6s0-worker-1 spec: desiredState: interfaces: - description: internalapi vlan interface ipv4: address: - ip: 172.17.0.10 prefix-length: 24 enabled: true dhcp: false ipv6: enabled: false name: internalapi state: up type: vlan vlan: base-iface: enp6s0 id: 20 reorder-headers: true - description: storage vlan interface ipv4: address: - ip: 172.18.0.10 prefix-length: 24 enabled: true dhcp: false ipv6: enabled: false name: storage state: up type: vlan vlan: base-iface: enp6s0 id: 21 reorder-headers: true - description: tenant vlan interface ipv4: address: - ip: 172.19.0.10 prefix-length: 24 enabled: true dhcp: false ipv6: enabled: false name: tenant state: up type: vlan vlan: base-iface: enp6s0 id: 22 reorder-headers: true - description: Configuring enp6s0 ipv4: address: - ip: 192.168.122.10 prefix-length: 24 enabled: true dhcp: false ipv6: enabled: false mtu: 1500 name: enp6s0 state: up type: ethernet nodeSelector: kubernetes.io/hostname: worker-1 node-role.kubernetes.io/worker: ""
oc apply -f openstack-nncp.yaml
oc get nncp -w NAME STATUS REASON osp-enp6s0-worker-1 Progressing ConfigurationProgressing osp-enp6s0-worker-1 Progressing ConfigurationProgressing osp-enp6s0-worker-1 Available SuccessfullyConfigured
apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: internalapi namespace: openstack 1 spec: config: | { "cniVersion": "0.3.1", "name": "internalapi", "type": "macvlan", "master": "internalapi", 2 "ipam": { 3 "type": "whereabouts", "range": "172.17.0.0/24", "range_start": "172.17.0.30", 4 "range_end": "172.17.0.70" } } --- apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: ctlplane namespace: openstack spec: config: | { "cniVersion": "0.3.1", "name": "ctlplane", "type": "macvlan", "master": "enp6s0", "ipam": { "type": "whereabouts", "range": "192.168.122.0/24", "range_start": "192.168.122.30", "range_end": "192.168.122.70" } } --- apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: storage namespace: openstack spec: config: | { "cniVersion": "0.3.1", "name": "storage", "type": "macvlan", "master": "storage", "ipam": { "type": "whereabouts", "range": "172.18.0.0/24", "range_start": "172.18.0.30", "range_end": "172.18.0.70" } } --- apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: tenant namespace: openstack spec: config: | { "cniVersion": "0.3.1", "name": "tenant", "type": "macvlan", "master": "tenant", "ipam": { "type": "whereabouts", "range": "172.19.0.0/24", "range_start": "172.19.0.30", "range_end": "172.19.0.70" } }
oc apply -f openstack-net-attach-def.yaml
oc get net-attach-def -n openstack
apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: internalapi namespace: metallb-system spec: addresses: - 172.17.0.80-172.17.0.90 1 autoAssign: true avoidBuggyIPs: false --- apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: namespace: metallb-system name: ctlplane spec: addresses: - 192.168.122.80-192.168.122.90 autoAssign: true avoidBuggyIPs: false --- apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: namespace: metallb-system name: storage spec: addresses: - 172.18.0.80-172.18.0.90 autoAssign: true avoidBuggyIPs: false --- apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: namespace: metallb-system name: tenant spec: addresses: - 172.19.0.80-172.19.0.90 autoAssign: true avoidBuggyIPs: false
oc apply -f openstack-ipaddresspools.yaml
oc describe -n metallb-system IPAddressPool
apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: internalapi namespace: metallb-system spec: ipAddressPools: - internalapi interfaces: - internalapi 1 --- apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: ctlplane namespace: metallb-system spec: ipAddressPools: - ctlplane interfaces: - enp6s0 --- apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: storage namespace: metallb-system spec: ipAddressPools: - storage interfaces: - storage --- apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: tenant namespace: metallb-system spec: ipAddressPools: - tenant interfaces: - tenant
oc apply -f openstack-l2advertisement.yaml
oc get -n metallb-system L2Advertisement NAME IPADDRESSPOOLS IPADDRESSPOOL SELECTORS INTERFACES ctlplane ["ctlplane"] ["enp6s0"] internalapi ["internalapi"] ["internalapi"] storage ["storage"] ["storage"] tenant ["tenant"] ["tenant"]
oc get network.operator cluster --output=jsonpath='{.spec.defaultNetwork.type}'
oc patch network.operator cluster -p '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"ipForwarding": "Global"}}}}}' --type=merge
oc describe crd netconfig oc explain netconfig.spec
apiVersion: network.openstack.org/v1beta1 kind: NetConfig metadata: name: openstacknetconfig namespace: openstack
spec: networks: - name: CtlPlane 1 dnsDomain: ctlplane.example.com subnets: 2 - name: subnet1 3 allocationRanges: 4 - end: 192.168.122.120 start: 192.168.122.100 - end: 192.168.122.200 start: 192.168.122.150 cidr: 192.168.122.0/24 gateway: 192.168.122.1 - name: InternalApi dnsDomain: internalapi.example.com subnets: - name: subnet1 allocationRanges: - end: 172.17.0.250 start: 172.17.0.100 excludeAddresses: 5 - 172.17.0.10 - 172.17.0.12 cidr: 172.17.0.0/24 vlan: 20 6 - name: External dnsDomain: external.example.com subnets: - name: subnet1 allocationRanges: - end: 10.0.0.250 start: 10.0.0.100 cidr: 10.0.0.0/24 gateway: 10.0.0.1 - name: Storage dnsDomain: storage.example.com subnets: - name: subnet1 allocationRanges: - end: 172.18.0.250 start: 172.18.0.100 cidr: 172.18.0.0/24 vlan: 21 - name: Tenant dnsDomain: tenant.example.com subnets: - name: subnet1 allocationRanges: - end: 172.19.0.250 start: 172.19.0.100 cidr: 172.19.0.0/24 vlan: 22 - name: bgpnet0 7 dnsDomain: bgpnet0.example.com mtu: 1500 subnets: - allocationRanges: - end: 100.64.0.6 start: 100.64.0.1 cidr: 100.64.0.0/29 gateway: 100.64.0.1 name: subnet0 routes: - destination: 0.0.0.0/0 nexthop: 100.64.0.1 - allocationRanges: - end: 100.64.1.6 start: 100.64.1.1 cidr: 100.64.1.0/29 gateway: 100.64.1.1 name: subnet1 routes: - destination: 0.0.0.0/0 nexthop: 100.64.1.1 - allocationRanges: - end: 100.64.2.6 start: 100.64.2.1 cidr: 100.64.2.0/29 gateway: 100.64.2.1 name: subnet2 routes: - destination: 0.0.0.0/0 nexthop: 100.64.2.1 - name: bgpnet1 8 dnsDomain: bgpnet1.example.com mtu: 1500 subnets: - allocationRanges: - end: 100.65.0.6 start: 100.65.0.1 cidr: 100.65.0.0/29 gateway: 100.65.0.1 name: subnet0 routes: - destination: 0.0.0.0/0 nexthop: 100.65.0.1 - allocationRanges: - end: 100.65.1.6 start: 100.65.1.1 cidr: 100.65.1.0/29 gateway: 100.65.1.1 name: subnet1 routes: - destination: 0.0.0.0/0 nexthop: 100.65.1.1 - allocationRanges: - end: 100.65.2.6 start: 100.65.2.1 cidr: 100.65.2.0/29 gateway: 100.65.2.1 name: subnet2 routes: - destination: 0.0.0.0/0 nexthop: 100.65.2.1
oc create -f openstack_netconfig.yaml -n openstack
oc get netconfig/openstacknetconfig -n openstack
oc get network-attachment-definitions -n openstack oc get nncp