Chapter 5. Planning an OVS-DPDK deployment To optimize your Open vSwitch with Data Plane Development Kit (OVS-DPDK) deployment for NFV in Red Hat OpenStack Services on OpenShift (RHOSO) environments, you should understand how OVS-DPDK uses the Compute node hardware (CPU, NUMA nodes, memory, NICs). This understanding will help you to determine the values required for the parameters used in your OVS-DPDK configuration. Important When using OVS-DPDK and the OVS native firewall (a stateful firewall based on conntrack), you can track only packets that use ICMPv4, ICMPv6, TCP, and UDP protocols. OVS marks all other types of network traffic as invalid. Important Red Hat does not support the use of OVS-DPDK for non-NFV workloads. If you need OVS-DPDK functionality for non-NFV workloads, contact your Technical Account Manager (TAM) or open a customer service request case to discuss a Support Exception and other options. To open a customer service request case, go to Create a case and choose Account > Customer Service Request . 5.1. OVS-DPDK with CPU partitioning and NUMA topology OVS-DPDK partitions the hardware resources for host, guests, and itself. The OVS-DPDK Poll Mode Drivers (PMDs) run DPDK active loops, which require dedicated CPU cores. Therefore you must allocate some CPUs, and huge pages, to OVS-DPDK. A sample partitioning includes 16 cores per NUMA node on dual-socket Compute nodes. The traffic requires additional NICs because you cannot share NICs between the host and OVS-DPDK. Figure 5.1. NUMA topology: OVS-DPDK with CPU partitioning Note You must reserve DPDK PMD threads on both NUMA nodes, even if a NUMA node does not have an associated DPDK NIC. For optimum OVS-DPDK performance, reserve a block of memory local to the NUMA node. Choose NICs associated with the same NUMA node that you use for memory and CPU pinning. Ensure that both bonded interfaces are from NICs on the same NUMA node. 5.2. OVS-DPDK parameters This section describes how OVS-DPDK uses parameters to configure the CPU and memory for optimum performance. Use this information to evaluate the hardware support on your Compute nodes and how to partition the hardware to optimize your OVS-DPDK deployment. This section describes the data plane parameters used in custom resources (CRs) to configure an OVS-DPDK deployment. Note Always pair CPU sibling threads, or logical CPUs, together in the physical core when allocating CPU cores. For details on how to determine the CPU and NUMA nodes on your Compute nodes, see Discovering your NUMA node topology . Use this information to map CPU and other parameters to support the host, guest instance, and OVS-DPDK process needs. 5.2.1. Data plane (EDPM) Ansible variables The following variables are part of data plane (EDPM) Ansible roles: edpm_ovs_dpdk Enables you to add, modify, and delete OVS-DPDK configurations, by using values defined in the OVS-DPDK edpm Ansible variables. edpm_ovs_dpdk_pmd_core_list Provides the CPU cores that are used for the DPDK poll mode drivers (PMD). It is recommended that you choose CPU cores that are associated with the local NUMA nodes of the DPDK interfaces. edpm_ovs_dpdk_lcore_list List of CPU cores to be used for DPDK lcore threads. edpm_tuned_profile Name of the custom TuneD profile. The default value is throughput-performance . edpm_tuned_isolated_cores A set of CPU cores isolated from the host processes. edpm_ovs_dpdk_socket_memory Specifies the amount of memory in MB to pre-allocate from the hugepage pool, per NUMA node. dpm_ovs_dpdk_socket_memory is the other_config:dpdk-socket-mem value in OVS. Observe the following recommendations: Provide as a comma-separated list. For a NUMA node without a DPDK NIC, use the static recommendation of 1024MB (1GB). Calculate the edpm_ovs_dpdk_socket_memory value from the MTU value of each NIC on the NUMA node. The following equation approximates the value: 800 is the overhead value. 4096 * 64 is the number of packets in the mempool. Add the MEMORY_REQD_PER_MTU for each of the MTU values set on the NUMA node and add another 512MB as buffer. Round the value up to a multiple of 1024 . edpm_ovs_dpdk_memory_channels Maps memory channels in the CPU per NUMA node. edpm_ovs_dpdk_memory_channels is the other_config:dpdk-extra="-n <value>" value in OVS. Observe the following recommendations: Use dmidecode -t memory or your hardware manual to determine the number of memory channels available. Use ls /sys/devices/system/node/node* -d to determine the number of NUMA nodes. Divide the number of memory channels available by the number of NUMA nodes. edpm_ovs_dpdk_vhost_postcopy_support Enable or disable OVS-DPDK vhost post-copy support. Setting this to true enables post-copy support for all vhost user client ports. edpm_nova_libvirt_qemu_group Set edpm_nova_libvirt_qemu_group to hugetlbfs `so that the `ovs-vswitchd and qemu processes can access the shared huge pages and UNIX socket that configures the virtio-net device . This value is role-specific and should be applied to any role leveraging OVS-DPDK. edpm_ovn_bridge_mappings List of bridge and dpdk ports mappings. edpm_kernel_args Provides multiple kernel arguments to /etc/default/grub for the compute nodes at boot time. 5.2.2. Configuration map parameters The following list describes parameters that you can use in ConfigMap sections: cpu_shared_set List or range of host CPU cores used to determine the host CPUs that instance emulator threads should be offloaded to for instances configured with the share emulator thread policy ( hw::emulator_threads_policy=share ). cpu_dedicated_set A comma-separated list or range of physical host CPU numbers to which processes for pinned instance CPUs can be scheduled. Exclude all cores from the edpm_ovs_dpdk_pmd_core_list . Include all remaining cores. Pair the sibling threads together. reserved_host_memory_mb Reserves memory in MB for tasks on the host. Use the static recommended value of 4096MB . 5.3. Two NUMA node example OVS-DPDK deployment The Red Hat OpenStack Services on OpenShift (RHOSO) Compute node in the following example includes two NUMA nodes: NUMA 0 has logical cores 0-7 (four physical cores). The sibling thread pairs are (0,1), (2,3), (4,5), and (6,7) NUMA 1 has cores 8-15. The sibling thread pairs are (8,9), (10,11), (12,13), and (14,15). Each NUMA node connects to a physical NIC, namely NIC1 on NUMA 0, and NIC2 on NUMA 1. Figure 5.2. OVS-DPDK: two NUMA nodes example Note Reserve the first physical cores or both thread pairs on each NUMA node (0,1 and 8,9) for non-datapath DPDK processes. This example also assumes a 1500 MTU configuration, so the OvsDpdkSocketMemory is the same for all use cases: OvsDpdkSocketMemory: "1024,1024" NIC 1 for DPDK, with one physical core for PMD In this use case, you allocate one physical core on NUMA 0 for PMD. You must also allocate one physical core on NUMA 1, even though DPDK is not enabled on the NIC for that NUMA node. The remaining cores are allocated for guest instances. The resulting parameter settings are: edpm_ovs_dpdk_pmd_core_list: "2,3,10,11" cpu_dedicated_set: "4,5,6,7,12,13,14,15" NIC 1 for DPDK, with two physical cores for PMD In this use case, you allocate two physical cores on NUMA 0 for PMD. You must also allocate one physical core on NUMA 1, even though DPDK is not enabled on the NIC for that NUMA node. The remaining cores are allocated for guest instances. The resulting parameter settings are: edpm_ovs_dpdk_pmd_core_list: "2,3,4,5,10,11" cpu_dedicated_set: "6,7,12,13,14,15" NIC 2 for DPDK, with one physical core for PMD In this use case, you allocate one physical core on NUMA 1 for PMD. You must also allocate one physical core on NUMA 0, even though DPDK is not enabled on the NIC for that NUMA node. The remaining cores are allocated for guest instances. The resulting parameter settings are: edpm_ovs_dpdk_pmd_core_list: "2,3,10,11" cpu_dedicated_set: "4,5,6,7,12,13,14,15" NIC 2 for DPDK, with two physical cores for PMD In this use case, you allocate two physical cores on NUMA 1 for PMD. You must also allocate one physical core on NUMA 0, even though DPDK is not enabled on the NIC for that NUMA node. The remaining cores are allocated for guest instances. The resulting parameter settings are: edpm_ovs_dpdk_pmd_core_list: "2,3,10,11,12,13" cpu_dedicated_set: "4,5,6,7,14,15" NIC 1 and NIC2 for DPDK, with two physical cores for PMD In this use case, you allocate two physical cores on each NUMA node for PMD. The remaining cores are allocated for guest instances. The resulting parameter settings are: edpm_ovs_dpdk_pmd_core_list: "2,3,4,5,10,11,12,13" cpu_dedicated_set: "6,7,14,15" 5.4. Topology of an NFV OVS-DPDK deployment This example deployment shows an OVS-DPDK configuration and consists of two virtual network functions (VNFs) with two interfaces each: The management interface, represented by mgt . The data plane interface. In the OVS-DPDK deployment, the VNFs operate with inbuilt DPDK that supports the physical interface. OVS-DPDK enables bonding at the vSwitch level. For improved performance in your OVS-DPDK deployment, it is recommended that you separate kernel and OVS-DPDK NICs. To separate the management ( mgt ) network, connected to the Base provider network for the virtual machine, ensure you have additional NICs. The Compute node consists of two regular NICs for the Red Hat OpenStack Platform API management that can be reused by the Ceph API but cannot be shared with any OpenStack project. Figure 5.3. Compute node: NFV OVS-DPDK OVS-DPDK Topology for NFV The following image shows the topology for OVS-DPDK on an NFV environment. Figure 5.4. OVS-DPDK Topology for NFV

Commands:
MEMORY_REQD_PER_MTU = (ROUNDUP_PER_MTU + 800) * (4096 * 64) Bytes
OvsDpdkSocketMemory: "1024,1024"
edpm_ovs_dpdk_pmd_core_list: "2,3,10,11" cpu_dedicated_set: "4,5,6,7,12,13,14,15"
edpm_ovs_dpdk_pmd_core_list: "2,3,4,5,10,11" cpu_dedicated_set: "6,7,12,13,14,15"
edpm_ovs_dpdk_pmd_core_list: "2,3,10,11" cpu_dedicated_set: "4,5,6,7,12,13,14,15"
edpm_ovs_dpdk_pmd_core_list: "2,3,10,11,12,13" cpu_dedicated_set: "4,5,6,7,14,15"
edpm_ovs_dpdk_pmd_core_list: "2,3,4,5,10,11,12,13" cpu_dedicated_set: "6,7,14,15"