Chapter 3. Creating networks with the director Operator Use the OpenStackNetConfig resource to create networks and bridges on OpenShift Virtualization worker nodes to connect your virtual machines to these networks. You must create one control plane network for your overcloud and additional networks to implement network isolation for your composable networks. 3.1. Understanding virtual machine bridging with OpenStackNet When you create virtual machines with the OpenStackVMSet resource, you must connect these virtual machines to the relevant Red Hat OpenStack Platform (RHOSP) networks. The OpenStackNetConfig resource includes an attachConfigurations option which is a hash of nodeNetworkConfigurationPolicy . Each specified attachConfiguration in the OpenStackNetConfig creates an OpenStackNet Attachment, which passes network interface data to the NodeNetworkConfigurationPolicy resource in OpenShift. The NodeNetworkConfigurationPolicy resource uses the nmstate API to configure the end state of the network configuration on each OCP worker node. Each network, configured in the OpenStackNetConfig, references one of the attachConfigurations . Inside the virtual machines, there is one interface per network. Through this method, you can create required bridges on OCP worker nodes and connect your Controller virtual machines to RHOSP networks. For example, if you create a br-osp attachConfiguration and set the nodeNetworkConfigurationPolicy option to create a Linux bridge and connect the bridge to a NIC on each worker, the NodeNetworkConfigurationPolicy resource configures each OCP worker node to match this desired end state: After you apply this configuration, each worker contains a new bridge named br-osp , which is connected to the enp6s0 NIC on each host. Dedicated NICs are required to deploy RHOSP. All RHOSP Controller virtual machines can connect to the br-osp bridge for control plane network traffic. If you specify an Internal API network through VLAN 20, you can set the attachConfiguration option to modify the networking configuration on each OCP worker node and connect the VLAN to the existing br-osp bridge: The br-osp already exists and is connected to the enp6s0 NIC on each host, so no change occurs to the bridge itself. However, the InternalAPI OpenStackNet associates VLAN 20 to this network, which means RHOSP Controller virtual machines can connect to the VLAN 20 on the br-osp bridge for Internal API network traffic. When you create virtual machines with the OpenStackVMSet resource, the virtual machines use multiple Virtio devices connected to each network. OpenShift Virtualization sorts the network names in alphabetical order except for the default network, which is always the first interface. For example, if you create the default RHOSP networks with OpenStackNetConfig, the interface configuration for Controller virtual machines resembles the following example: This configuration results in the following network-to-interface mapping for Controller nodes: Table 3.1. Default network-to-interface mapping Network Interface default nic1 ctlplane nic2 external nic3 internalapi nic4 storage nic5 storagemgmt nic6 tenant nic7 Note The role NIC template used by OpenStackVMSet is auto generated. It can be overwritten by adding a nic-template.role.j2 file to your tarball file. Include the binary contents of the tarball file in an OpenShift ConfigMap names tripleo-tarball-config . Additional resources "Updating node network configuration" 3.2. Creating an overcloud control plane network with OpenStackNetConfig You must define at least one control plane network for your overcloud in OpenStackNetConfig. In addition to IP address assignment, the network definition includes the mapping information for OpenStackNetAttachment. OpenShift Virtualization uses this information to attach any virtual machines to the network. Prerequisites Ensure your OpenShift Container Platform cluster is operational and you have installed the director Operator correctly. Ensure that you have installed the oc command line tool on your workstation. Procedure Create a file named osnetconfig.yaml on your workstation. Include the resource specification for the control plane network, which is named ctlplane . For example, the specification for a control plane that uses a Linux bridge connected to the enp6s0 Ethernet device on each worker node is as follows: Set the following values in the networks specification: name Set to the name of the control plane network, which is Control. nameLower Set to the lower name of the control plane network, which is ctlplane. subnets Set the subnet specifications. subnets.name Set the name of the control plane subnet, which is ctlplane. subnets.attachConfiguration Set the reference to which of the attach configuration should be used. subnets.ipv4 Details of the ipv4 subnet with allocationStart, allocationEnd, cidr, gateway and optional list of routes (with destination and nexthop) For descriptions of the values you can use in this section, view the specification schema in the custom resource definition for the openstacknetconfig CRD: Save the file when you have finished configuring the network specification. Create the control plane network: Verification View the resource for the control plane network: 3.3. Creating VLAN networks for network isolation with OpenStackNetConfig You must create additional networks to implement network isolation for your composable networks. To accomplish this network isolation, you can place your composable networks on individual VLAN networks. In addition to IP address assignment, the OpenStackNetConfig resource includes information to define the network configuration policy that OpenShift Virtualization uses to attach any virtual machines to VLAN networks. To use the default Red Hat OpenStack Platform networks, you must create an OpenStackNetConfig resource which defines each network. Table 3.2. Default Red Hat OpenStack Platform networks Network VLAN CIDR Allocation External 10 10.0.0.0/24 10.0.0.10 - 10.0.0.250 InternalApi 20 172.17.0.0/24 172.17.0.10 - 172.17.0.250 Storage 30 172.18.0.0/24 172.18.0.10 - 172.18.0.250 StorageMgmt 40 172.19.0.0/24 172.19.0.10 - 172.19..250 Tenant 50 172.20.0.0/24 172.20.0.10 - 172.20.0.250 Important To use different networking details for each network, you must create a custom network_data.yaml file. Prerequisites Ensure your OpenShift Container Platform cluster is operational and you have installed the director Operator correctly. Ensure that you have installed the oc command line tool on your workstation. Procedure Create a file for your network configuration. Include the resource specification for the VLAN network. For example, the specification for internal API, storage, storage mgmt, tenant, and external network that manages VLAN-tagged traffic over Linux bridges br-ex and br-osp connected to the enp6s0 and enp7s0 Ethernet device on each worker node is as follows: When you use VLAN for network isolation with linux-bridge the following happens: The director Operator creates a Node Network Configuration Policy for the bridge interface specified in the resource, which uses nmstate to configure the bridge on worker nodes. The director Operator creates a Network Attach Definition for each network, which defines the Multus CNI plugin configuration. When you specify the VLAN ID on the Network Attach Definition, the Multus CNI plugin enables vlan-filtering on the bridge. The director Operator attaches a dedicated interface for each network on a virtual machine. This means that the network template for the OpenStackVMSet is a multi-NIC network template. Set the following values in the resource specification: metadata.name Set to the name of the OpenStackNetConfig. spec Set the network configuration for attaching the networks and the network specifics. For descriptions of the values you can use in this section, view the specification schema in the custom resource definition for the openstacknetconfig CRD: Save the file when you have finished configuring the network specification. Create the network configuration: Verification View the OpenStackNetConfig API and created child resources: If you see errors, check the underlying network-attach-definition and node network configuration policies: 3.4. Configuring jumbo frames with OpenStackNetConfig To use Jumbo Frames for a bridge, you can create a configuration for the device to configure the correct MTU: 3.5. Static IP reservation with OpenStackNetConfig You can use the OpenStackNetConfig specification reservations parameter to reserve a static IP address per host and network. The reservations provided there are populated down to the` OpenStackNet` specifications reservations and have precedence over any auto generated IPs. The following example shows an overcloud with 3 Controllers and 2 Compute nodes, all nodes have static reservations except controller-2 and compute-1 :

Commands:
apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackNetConfig metadata: name: openstacknetconfig spec: attachConfigurations: br-osp: nodeNetworkConfigurationPolicy: nodeSelector: node-role.kubernetes.io/worker: "" desiredState: interfaces: - bridge: options: stp: enabled: false port: - name: enp6s0 description: Linux bridge with enp6s0 as a port name: br-osp state: up type: linux-bridge mtu: 1500 ... networks: - name: Control nameLower: ctlplane subnets: - name: ctlplane ipv4: allocationEnd: 192.168.25.250 allocationStart: 192.168.25.100 cidr: 192.168.25.0/24 gateway: 192.168.25.1 attachConfiguration: br-osp
apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackNetConfig metadata: name: openstacknetconfig spec: attachConfigurations: br-osp: ... networks: ... - isControlPlane: false mtu: 1500 name: InternalApi nameLower: internal_api subnets: - attachConfiguration: br-osp ipv4: allocationEnd: 172.17.0.250 allocationStart: 172.17.0.10 cidr: 172.17.0.0/24 gateway: 172.17.0.1 routes: - destination: 172.17.1.0/24 nexthop: 172.17.0.1 - destination: 172.17.2.0/24 nexthop: 172.17.0.1 name: internal_api vlan: 20
interfaces: - masquerade: {} model: virtio name: default - bridge: {} model: virtio name: ctlplane - bridge: {} model: virtio name: external - bridge: {} model: virtio name: internalapi - bridge: {} model: virtio name: storage - bridge: {} model: virtio name: storagemgmt - bridge: {} model: virtio name: tenant
apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackNetConfig metadata: name: openstacknetconfig spec: attachConfigurations: br-osp: nodeNetworkConfigurationPolicy: nodeSelector: node-role.kubernetes.io/worker: "" desiredState: interfaces: - bridge: options: stp: enabled: false port: - name: enp6s0 description: Linux bridge with enp6s0 as a port name: br-osp state: up type: linux-bridge mtu: 1500 # optional DnsServers list dnsServers: - 192.168.25.1 # optional DnsSearchDomains list dnsSearchDomains: - osptest.test.metalkube.org - some.other.domain # DomainName of the OSP environment domainName: osptest.test.metalkube.org networks: - name: Control nameLower: ctlplane subnets: - name: ctlplane ipv4: allocationEnd: 172.22.0.250 allocationStart: 172.22.0.100 cidr: 172.22.0.0/24 gateway: 172.22.0.1 attachConfiguration: br-osp # optional: configure static mapping for the networks per nodes. If there is none, a random gets created reservations: controller-0: ipReservations: ctlplane: 172.22.0.120 compute-0: ipReservations: ctlplane: 172.22.0.140
oc describe crd openstacknetconfig
oc create -f osnetconfig.yaml -n openstack
oc get openstacknetconfig/openstacknetconfig
kind: OpenStackNetConfig metadata: name: openstacknetconfig spec: attachConfigurations: br-osp: nodeNetworkConfigurationPolicy: nodeSelector: node-role.kubernetes.io/worker: "" desiredState: interfaces: - bridge: options: stp: enabled: false port: - name: enp7s0 description: Linux bridge with enp7s0 as a port name: br-osp state: up type: linux-bridge mtu: 1500 br-ex: nodeNetworkConfigurationPolicy: nodeSelector: node-role.kubernetes.io/worker: "" desiredState: interfaces: - bridge: options: stp: enabled: false port: - name: enp6s0 description: Linux bridge with enp6s0 as a port name: br-ex state: up type: linux-bridge mtu: 1500 # optional DnsServers list dnsServers: - 172.22.0.1 # optional DnsSearchDomains list dnsSearchDomains: - osptest.test.metalkube.org - some.other.domain # DomainName of the OSP environment domainName: osptest.test.metalkube.org networks: - name: Control nameLower: ctlplane subnets: - name: ctlplane ipv4: allocationEnd: 172.22.0.250 allocationStart: 172.22.0.10 cidr: 172.22.0.0/24 gateway: 172.22.0.1 attachConfiguration: br-osp - name: InternalApi nameLower: internal_api mtu: 1350 subnets: - name: internal_api attachConfiguration: br-osp vlan: 20 ipv4: allocationEnd: 172.17.0.250 allocationStart: 172.17.0.10 cidr: 172.17.0.0/24 - name: External nameLower: external subnets: - name: external ipv4: allocationEnd: 10.0.0.250 allocationStart: 10.0.0.10 cidr: 10.0.0.0/24 gateway: 10.0.0.1 attachConfiguration: br-ex - name: Storage nameLower: storage mtu: 1500 subnets: - name: storage ipv4: allocationEnd: 172.18.0.250 allocationStart: 172.18.0.10 cidr: 172.18.0.0/24 vlan: 30 attachConfiguration: br-osp - name: StorageMgmt nameLower: storage_mgmt mtu: 1500 subnets: - name: storage_mgmt ipv4: allocationEnd: 172.19.0.250 allocationStart: 172.19.0.10 cidr: 172.19.0.0/24 vlan: 40 attachConfiguration: br-osp - name: Tenant nameLower: tenant vip: False mtu: 1500 subnets: - name: tenant ipv4: allocationEnd: 172.20.0.250 allocationStart: 172.20.0.10 cidr: 172.20.0.0/24 vlan: 50 attachConfiguration: br-osp
oc describe crd openstacknetconfig
oc apply -f openstacknetconfig.yaml -n openstack
oc get openstacknetconfig/openstacknetconfig -n openstack oc get openstacknetattachment -n openstack oc get openstacknet -n openstack
oc get network-attachment-definitions -n openstack oc get nncp
apiVersion: osp-director.openstack.org/v1beta1 kind: OpenStackNetConfig metadata: name: openstacknetconfig spec: attachConfigurations: br-osp: nodeNetworkConfigurationPolicy: nodeSelector: node-role.kubernetes.io/worker: "" desiredState: interfaces: - bridge: options: stp: enabled: false port: - name: enp7s0 description: Linux bridge with enp7s0 as a port name: br-osp state: up type: linux-bridge mtu: 9000 - name: enp7s0 description: Configuring enp7s0 on workers type: ethernet state: up mtu: 9000
spec: ... reservations: compute-0: ipReservations: ctlplane: 172.22.0.140 internal_api: 172.17.0.40 storage: 172.18.0.40 tenant: 172.20.0.40 macReservations: {} controller-0: ipReservations: ctlplane: 172.22.0.120 external: 10.0.0.20 internal_api: 172.17.0.20 storage: 172.18.0.20 storage_mgmt: 172.19.0.20 tenant: 172.20.0.20 macReservations: {} controller-1: ipReservations: ctlplane: 172.22.0.130 external: 10.0.0.30 internal_api: 172.17.0.30 storage: 172.18.0.30 storage_mgmt: 172.19.0.30 tenant: 172.20.0.30 macReservations: {} controlplane: ipReservations: ctlplane: 172.22.0.110 external: 10.0.0.10 internal_api: 172.17.0.10 storage: 172.18.0.10 storage_mgmt: 172.19.0.10 macReservations: {} openstackclient-0: ipReservations: ctlplane: 172.22.0.251 external: 10.0.0.251 internal_api: 172.17.0.251 macReservations: {}