Chapter 9. Planning your OVS-DPDK deployment To optimize your Open vSwitch with Data Plane Development Kit (OVS-DPDK) deployment for NFV, you should understand how OVS-DPDK uses the Compute node hardware (CPU, NUMA nodes, memory, NICs) and the considerations for determining the individual OVS-DPDK parameters based on your Compute node. Important When using OVS-DPDK and the OVS native firewall (a stateful firewall based on conntrack), you can track only packets that use ICMPv4, ICMPv6, TCP, and UDP protocols. OVS marks all other types of network traffic as invalid. Important Red Hat does not support the use of OVS-DPDK for non-NFV workloads. If you need OVS-DPDK functionality for non-NFV workloads, contact your Technical Account Manager (TAM) or open a customer service request case to discuss a Support Exception and other options. To open a customer service request case, go to Create a case and choose Account > Customer Service Request . 9.1. OVS-DPDK with CPU partitioning and NUMA topology OVS-DPDK partitions the hardware resources for host, guests, and itself. The OVS-DPDK Poll Mode Drivers (PMDs) run DPDK active loops, which require dedicated CPU cores. Therefore you must allocate some CPUs, and huge pages, to OVS-DPDK. A sample partitioning includes 16 cores per NUMA node on dual-socket Compute nodes. The traffic requires additional NICs because you cannot share NICs between the host and OVS-DPDK. Figure 9.1. NUMA topology: OVS-DPDK with CPU partitioning Note You must reserve DPDK PMD threads on both NUMA nodes, even if a NUMA node does not have an associated DPDK NIC. For optimum OVS-DPDK performance, reserve a block of memory local to the NUMA node. Choose NICs associated with the same NUMA node that you use for memory and CPU pinning. Ensure that both bonded interfaces are from NICs on the same NUMA node. 9.2. OVS-DPDK parameters This section describes how OVS-DPDK uses parameters within the director network_environment.yaml heat templates to configure the CPU and memory for optimum performance. Use this information to evaluate the hardware support on your Compute nodes and how to partition the hardware to optimize your OVS-DPDK deployment. Note Always pair CPU sibling threads, or logical CPUs, together in the physical core when allocating CPU cores. For details on how to determine the CPU and NUMA nodes on your Compute nodes, see Discovering your NUMA node topology . Use this information to map CPU and other parameters to support the host, guest instance, and OVS-DPDK process needs. 9.2.1. CPU parameters OVS-DPDK uses the following parameters for CPU partitioning: OvsPmdCoreList Provides the CPU cores that are used for the DPDK poll mode drivers (PMD). Choose CPU cores that are associated with the local NUMA nodes of the DPDK interfaces. Use OvsPmdCoreList for the pmd-cpu-mask value in OVS. Use the following recommendations for OvsPmdCoreList : Pair the sibling threads together. Performance depends on the number of physical cores allocated for this PMD Core list. On the NUMA node which is associated with DPDK NIC, allocate the required cores. For NUMA nodes with a DPDK NIC, determine the number of physical cores required based on the performance requirement, and include all the sibling threads or logical CPUs for each physical core. For NUMA nodes without DPDK NICs, allocate the sibling threads or logical CPUs of any physical core except the first physical core of the NUMA node. Note You must reserve DPDK PMD threads on both NUMA nodes, even if a NUMA node does not have an associated DPDK NIC. NovaComputeCpuDedicatedSet A comma-separated list or range of physical host CPU numbers to which processes for pinned instance CPUs can be scheduled. For example, NovaComputeCpuDedicatedSet: [4-12,^8,15] reserves cores from 4-12 and 15, excluding 8. Exclude all cores from the OvsPmdCoreList . Include all remaining cores. Pair the sibling threads together. NovaComputeCpuSharedSet A comma-separated list or range of physical host CPU numbers used to determine the host CPUs for instance emulator threads. IsolCpusList A set of CPU cores isolated from the host processes. IsolCpusList is the isolated_cores value in the cpu-partitioning-variable.conf file for the tuned-profiles-cpu-partitioning component. Use the following recommendations for IsolCpusList : Match the list of cores in OvsPmdCoreList and NovaComputeCpuDedicatedSet . Pair the sibling threads together. DerivePciWhitelistEnabled To reserve virtual functions (VF) for VMs, use the NovaPCIPassthrough parameter to create a list of VFs passed through to Nova. VFs excluded from the list remain available for the host. For each VF in the list, populate the address parameter with a regular expression that resolves to the address value. The following is an example of the manual list creation process. If NIC partitioning is enabled in a device named eno2 , list the PCI addresses of the VFs with the following command: In this case, the VFs 0, 4, and 6 are used by eno2 for NIC Partitioning. Manually configure NovaPCIPassthrough to include VFs 1-3, 5, and 7, and consequently exclude VFs 0,4, and 6, as in the following example: 9.2.2. Memory parameters OVS-DPDK uses the following memory parameters: OvsDpdkMemoryChannels Maps memory channels in the CPU per NUMA node. OvsDpdkMemoryChannels is the other_config:dpdk-extra="-n <value>" value in OVS. Observe the following recommendations for OvsDpdkMemoryChannels : Use dmidecode -t memory or your hardware manual to determine the number of memory channels available. Use ls /sys/devices/system/node/node* -d to determine the number of NUMA nodes. Divide the number of memory channels available by the number of NUMA nodes. NovaReservedHostMemory Reserves memory in MB for tasks on the host. NovaReservedHostMemory is the reserved_host_memory_mb value for the Compute node in nova.conf . Observe the following recommendation for NovaReservedHostMemory : Use the static recommended value of 4096 MB. OvsDpdkSocketMemory Specifies the amount of memory in MB to pre-allocate from the hugepage pool, per NUMA node. OvsDpdkSocketMemory is the other_config:dpdk-socket-mem value in OVS. Observe the following recommendations for OvsDpdkSocketMemory : Provide as a comma-separated list. For a NUMA node without a DPDK NIC, use the static recommendation of 1024 MB (1GB) Calculate the OvsDpdkSocketMemory value from the MTU value of each NIC on the NUMA node. The following equation approximates the value for OvsDpdkSocketMemory : MEMORY_REQD_PER_MTU = (ROUNDUP_PER_MTU + 800) * (4096 * 64) Bytes 800 is the overhead value. 4096 * 64 is the number of packets in the mempool. Add the MEMORY_REQD_PER_MTU for each of the MTU values set on the NUMA node and add another 512 MB as buffer. Round the value up to a multiple of 1024. Sample Calculation - MTU 2000 and MTU 9000 DPDK NICs dpdk0 and dpdk1 are on the same NUMA node 0, and configured with MTUs 9000, and 2000 respectively. The sample calculation to derive the memory required is as follows: Round off the MTU values to the nearest multiple of 1024 bytes. Calculate the required memory for each MTU value based on these rounded byte values. Calculate the combined total memory required, in bytes. This calculation represents (Memory required for MTU of 9000) + (Memory required for MTU of 2000) + (512 MB buffer). Convert the total memory required into MB. Round this value up to the nearest 1024. Use this value to set OvsDpdkSocketMemory . OvsDpdkSocketMemory: "4096,1024" Sample Calculation - MTU 2000 DPDK NICs dpdk0 and dpdk1 are on the same NUMA node 0, and each are configured with MTUs of 2000. The sample calculation to derive the memory required is as follows: Round off the MTU values to the nearest multiple of 1024 bytes. Calculate the required memory for each MTU value based on these rounded byte values. Calculate the combined total memory required, in bytes. This calculation represents (Memory required for MTU of 2000) + (512 MB buffer). Convert the total memory required into MB. Round this value up to the nearest multiple of 1024. Use this value to set OvsDpdkSocketMemory . OvsDpdkSocketMemory: "2048,1024" 9.2.3. Networking parameters OvsDpdkDriverType Sets the driver type used by DPDK. Use the default value of vfio-pci . NeutronDatapathType Datapath type for OVS bridges. DPDK uses the default value of netdev . NeutronVhostuserSocketDir Sets the vhost-user socket directory for OVS. Use /var/lib/vhost_sockets for vhost client mode. 9.2.4. Other parameters NovaSchedulerEnabledFilters Provides an ordered list of filters that the Compute node uses to find a matching Compute node for a requested guest instance. VhostuserSocketGroup Sets the vhost-user socket directory group. The default value is qemu . Set VhostuserSocketGroup to hugetlbfs so that the ovs-vswitchd and qemu processes can access the shared huge pages and unix socket that configures the virtio-net device. This value is role-specific and should be applied to any role leveraging OVS-DPDK. Important To use the parameter VhostuserSocketGroup you must also set NeutronVhostuserSocketDir . For more information, see Section 9.2.3, "Networking parameters" . KernelArgs Provides multiple kernel arguments to /etc/default/grub for the Compute node at boot time. Add the following values based on your configuration: hugepagesz : Sets the size of the huge pages on a CPU. This value can vary depending on the CPU hardware. Set to 1G for OVS-DPDK deployments ( default_hugepagesz=1GB hugepagesz=1G ). Use this command to check for the pdpe1gb CPU flag that confirms your CPU supports 1G. hugepages count : Sets the number of huge pages available based on available host memory. Use most of your available memory, except NovaReservedHostMemory . You must also configure the huge pages count value within the flavor of your Compute nodes. iommu : For Intel CPUs, add "intel_iommu=on iommu=pt" isolcpus : Sets the CPU cores for tuning. This value matches IsolCpusList . For more information about CPU isolation, see the Red Hat Knowledgebase solution OpenStack CPU isolation guidance for RHEL 8 and RHEL 9 . DdpPackage Configures Dynamic Device Personalization (DDP), to apply a profile package to a device at deployment to change the packet processing pipeline of the device. Add the following lines to your network_environment.yaml template to include the DDP package: parameter_defaults: ComputeOvsDpdkSriovParameters: DdpPackage: "ddp-comms" OvsDpdkExtra Enables you to pass additional configuration parameters with the other_config:dpdk-extra parameter. It is used for environments that use NIC partitioning with NVIDIA Mellanox cards to avoid connectivity issues. For these use cases, set OvsDpdkExtra to -a 0000:00:00.0 which causes the allow list of PCI addresses to allow no addresses. Example 9.2.5. VM instance flavor specifications Before deploying VM instances in an NFV environment, create a flavor that utilizes CPU pinning, huge pages, and emulator thread pinning. hw:cpu_policy When this parameter is set to dedicated , the guest uses pinned CPUs. Instances created from a flavor with this parameter set have an effective overcommit ratio of 1:1. The default value is shared . hw:mem_page_size Set this parameter to a valid string of a specific value with standard suffix (For example, 4KB , 8MB , or 1GB ). Use 1GB to match the hugepagesz boot parameter. Calculate the number of huge pages available for the virtual machines by subtracting OvsDpdkSocketMemory from the boot parameter. The following values are also valid: small (default) - The smallest page size is used large - Only use large page sizes. (2MB or 1GB on x86 architectures) any - The compute driver can attempt to use large pages, but defaults to small if none available. hw:emulator_threads_policy Set the value of this parameter to share so that emulator threads are locked to CPUs that you've identified in the heat parameter, NovaComputeCpuSharedSet . If an emulator thread is running on a vCPU with the poll mode driver (PMD) or real-time processing, you can experience negative effects, such as packet loss. 9.3. Saving power in OVS-DPDK deployments A power save profile, cpu-partitioning-powersave , has been introduced in Red Hat Enterprise Linux 9 (RHEL 9), and is now available in Red Hat OpenStack Platform (RHOSP) 17.1.3. This TuneD profile is the base building block to save power in RHOSP 17.1 NFV environments. Prerequisites Access to the undercloud host and credentials for the stack user. The CPUs on which you want to achieve power savings are enabled to allow higher C-states. For more information, see the max_power_state option in the man page for tuned-profiles-cpu-partitioning(7) . Procedure Log in to the undercloud as the stack user. Source the stackrc file: Create an Ansible playbook YAML file, for example, /home/stack/cli-overcloud-tuned-maxpower-conf.yaml . Add the following configuration to your cli-overcloud-tuned-maxpower-conf.yaml file: cat <<EOF > /home/stack/cli-overcloud-tuned-maxpower-conf.yaml {% raw %} --- #/home/stack/cli-overcloud-tuned-maxpower-conf.yaml - name: Overcloud Node set tuned power state hosts: compute-0 compute-1 any_errors_fatal: true gather_facts: false pre_tasks: - name: Wait for provisioned nodes to boot wait_for_connection: timeout: 600 delay: 10 connection: local tasks: - name: Check the max power state for this system become: true block: - name: Get power states shell: "for s in /sys/devices/system/cpu/cpu2/cpuidle/*; do grep . USDs/{name,latency}; done" register: _list_of_power_states - name: Print available power states debug: msg: "{{ _list_of_power_states.stdout.split('\n') }}" - name: Check for active tuned power-save profile stat: path: "/etc/tuned/active_profile" register: _active_profile - name: Check the profile slurp: path: "/etc/tuned/active_profile" when: _active_profile.stat.exists register: _active_profile_name - name: Print states debug: var: (_active_profile_name.content|b64decode|string) - name: Check the max power state for this system block: - name: Check if the cstate config is present in the conf file lineinfile: dest: /etc/tuned/cpu-partitioning-powersave-variables.conf regexp: '^max_power_state' line: 'max_power_state=cstate.name:C6' register: _cstate_entry_check {% endraw %} EOF Add the power save profile to your roles data file. For more information, see 10.2. Generating roles and image files . Add the cli-overcloud-tuned-maxpower-conf.yaml playbook to your bare metal nodes definition file. For more information, see 10.5. Creating a bare metal nodes definition file . Ensure that you have set queue size in your NIC configuration template. For more information, see 10.6. Creating a NIC configuration template . Additional resources force_latency 9.4. Two NUMA node example OVS-DPDK deployment The Compute node in the following example includes two NUMA nodes: NUMA 0 has cores 0-7. The sibling thread pairs are (0,1), (2,3), (4,5), and (6,7) NUMA 1 has cores 8-15. The sibling thread pairs are (8,9), (10,11), (12,13), and (14,15). Each NUMA node connects to a physical NIC, namely NIC1 on NUMA 0, and NIC2 on NUMA 1. Figure 9.2. OVS-DPDK: two NUMA nodes example Note Reserve the first physical cores or both thread pairs on each NUMA node (0,1 and 8,9) for non-datapath DPDK processes. This example also assumes a 1500 MTU configuration, so the OvsDpdkSocketMemory is the same for all use cases: OvsDpdkSocketMemory: "1024,1024" NIC 1 for DPDK, with one physical core for PMD In this use case, you allocate one physical core on NUMA 0 for PMD. You must also allocate one physical core on NUMA 1, even though DPDK is not enabled on the NIC for that NUMA node. The remaining cores are allocated for guest instances. The resulting parameter settings are: OvsPmdCoreList: "2,3,10,11" NovaComputeCpuDedicatedSet: "4,5,6,7,12,13,14,15" NIC 1 for DPDK, with two physical cores for PMD In this use case, you allocate two physical cores on NUMA 0 for PMD. You must also allocate one physical core on NUMA 1, even though DPDK is not enabled on the NIC for that NUMA node. The remaining cores are allocated for guest instances. The resulting parameter settings are: OvsPmdCoreList: "2,3,4,5,10,11" NovaComputeCpuDedicatedSet: "6,7,12,13,14,15" NIC 2 for DPDK, with one physical core for PMD In this use case, you allocate one physical core on NUMA 1 for PMD. You must also allocate one physical core on NUMA 0, even though DPDK is not enabled on the NIC for that NUMA node. The remaining cores are allocated for guest instances. The resulting parameter settings are: OvsPmdCoreList: "2,3,10,11" NovaComputeCpuDedicatedSet: "4,5,6,7,12,13,14,15" NIC 2 for DPDK, with two physical cores for PMD In this use case, you allocate two physical cores on NUMA 1 for PMD. You must also allocate one physical core on NUMA 0, even though DPDK is not enabled on the NIC for that NUMA node. The remaining cores are allocated for guest instances. The resulting parameter settings are: OvsPmdCoreList: "2,3,10,11,12,13" NovaComputeCpuDedicatedSet: "4,5,6,7,14,15" NIC 1 and NIC2 for DPDK, with two physical cores for PMD In this use case, you allocate two physical cores on each NUMA node for PMD. The remaining cores are allocated for guest instances. The resulting parameter settings are: OvsPmdCoreList: "2,3,4,5,10,11,12,13" NovaComputeCpuDedicatedSet: "6,7,14,15" 9.5. Topology of an NFV OVS-DPDK deployment This example deployment shows an OVS-DPDK configuration and consists of two virtual network functions (VNFs) with two interfaces each: The management interface, represented by mgt . The data plane interface. In the OVS-DPDK deployment, the VNFs operate with inbuilt DPDK that supports the physical interface. OVS-DPDK enables bonding at the vSwitch level. For improved performance in your OVS-DPDK deployment, it is recommended that you separate kernel and OVS-DPDK NICs. To separate the management ( mgt ) network, connected to the Base provider network for the virtual machine, ensure you have additional NICs. The Compute node consists of two regular NICs for the Red Hat OpenStack Platform API management that can be reused by the Ceph API but cannot be shared with any OpenStack project. Figure 9.3. Compute node: NFV OVS-DPDK NFV OVS-DPDK topology The following image shows the topology for OVS-DPDK for NFV. It consists of Compute and Controller nodes with 1 or 10 Gbps NICs, and the director node. Figure 9.4. NFV topology: OVS-DPDK

Commands:
[tripleo-admin@compute-0 ~]USD ls -lh /sys/class/net/eno2/device/ | grep virtfn lrwxrwxrwx. 1 root root 0 Apr 16 09:58 virtfn0 -> ../0000:18:06.0 lrwxrwxrwx. 1 root root 0 Apr 16 09:58 virtfn1 -> ../0000:18:06.1 lrwxrwxrwx. 1 root root 0 Apr 16 09:58 virtfn2 -> ../0000:18:06.2 lrwxrwxrwx. 1 root root 0 Apr 16 09:58 virtfn3 -> ../0000:18:06.3 lrwxrwxrwx. 1 root root 0 Apr 16 09:58 virtfn4 -> ../0000:18:06.4 lrwxrwxrwx. 1 root root 0 Apr 16 09:58 virtfn5 -> ../0000:18:06.5 lrwxrwxrwx. 1 root root 0 Apr 16 09:58 virtfn6 -> ../0000:18:06.6 lrwxrwxrwx. 1 root root 0 Apr 16 09:58 virtfn7 -> ../0000:18:06.7
NovaPCIPassthrough: - physical_network: "sriovnet2" address: {"domain": ".*", "bus": "18", "slot": "06", "function": "[1-3]"} - physical_network: "sriovnet2" address: {"domain": ".*", "bus": "18", "slot": "06", "function": "[5]"} - physical_network: "sriovnet2" address: {"domain": ".*", "bus": "18", "slot": "06", "function": "[7]"}
The MTU value of 9000 becomes 9216 bytes. The MTU value of 2000 becomes 2048 bytes.
Memory required for 9000 MTU = (9216 + 800) * (4096*64) = 2625634304 Memory required for 2000 MTU = (2048 + 800) * (4096*64) = 746586112
2625634304 + 746586112 + 536870912 = 3909091328 bytes.
3909091328 / (1024*1024) = 3728 MB.
3724 MB rounds up to 4096 MB.
OvsDpdkSocketMemory: "4096,1024"
The MTU value of 2000 becomes 2048 bytes.
Memory required for 2000 MTU = (2048 + 800) * (4096*64) = 746586112
746586112 + 536870912 = 1283457024 bytes.
1283457024 / (1024*1024) = 1224 MB.
1224 MB rounds up to 2048 MB.
OvsDpdkSocketMemory: "2048,1024"
lshw -class processor | grep pdpe1gb
parameter_defaults: ComputeOvsDpdkSriovParameters: DdpPackage: "ddp-comms"
parameter_defaults: ComputeOvsDpdkSriovParameters: KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=48 intel_iommu=on iommu=pt isolcpus=1-11,13-23" IsolCpusList: "1-11,13-23" OvsDpdkSocketMemory: "4096" OvsDpdkMemoryChannels: "4" OvsDpdkExtra: "-a 0000:00:00.0" NovaReservedHostMemory: 4096 OvsPmdCoreList: "1,13,2,14,3,15" OvsDpdkCoreList: "0,12" NovaComputeCpuDedicatedSet: [ 4-11 , 16-23 ] NovaComputeCpuSharedSet: [ 0 , 12 ]
source ~/stackrc
cat <<EOF > /home/stack/cli-overcloud-tuned-maxpower-conf.yaml {% raw %} --- #/home/stack/cli-overcloud-tuned-maxpower-conf.yaml - name: Overcloud Node set tuned power state hosts: compute-0 compute-1 any_errors_fatal: true gather_facts: false pre_tasks: - name: Wait for provisioned nodes to boot wait_for_connection: timeout: 600 delay: 10 connection: local tasks: - name: Check the max power state for this system become: true block: - name: Get power states shell: "for s in /sys/devices/system/cpu/cpu2/cpuidle/*; do grep . USDs/{name,latency}; done" register: _list_of_power_states - name: Print available power states debug: msg: "{{ _list_of_power_states.stdout.split('\n') }}" - name: Check for active tuned power-save profile stat: path: "/etc/tuned/active_profile" register: _active_profile - name: Check the profile slurp: path: "/etc/tuned/active_profile" when: _active_profile.stat.exists register: _active_profile_name - name: Print states debug: var: (_active_profile_name.content|b64decode|string) - name: Check the max power state for this system block: - name: Check if the cstate config is present in the conf file lineinfile: dest: /etc/tuned/cpu-partitioning-powersave-variables.conf regexp: '^max_power_state' line: 'max_power_state=cstate.name:C6' register: _cstate_entry_check {% endraw %} EOF
OvsDpdkSocketMemory: "1024,1024"
OvsPmdCoreList: "2,3,10,11" NovaComputeCpuDedicatedSet: "4,5,6,7,12,13,14,15"
OvsPmdCoreList: "2,3,4,5,10,11" NovaComputeCpuDedicatedSet: "6,7,12,13,14,15"
OvsPmdCoreList: "2,3,10,11" NovaComputeCpuDedicatedSet: "4,5,6,7,12,13,14,15"
OvsPmdCoreList: "2,3,10,11,12,13" NovaComputeCpuDedicatedSet: "4,5,6,7,14,15"
OvsPmdCoreList: "2,3,4,5,10,11,12,13" NovaComputeCpuDedicatedSet: "6,7,14,15"