Chapter 4. Configuring CPUs on Compute nodes As a cloud administrator, you can configure the scheduling and placement of instances for optimal performance by creating customized flavors to target specialized workloads, including NFV and High Performance Computing (HPC). Use the following features to tune your instances for optimal CPU performance: CPU pinning : Pin virtual CPUs to physical CPUs. Emulator threads : Pin emulator threads associated with the instance to physical CPUs. CPU feature flags : Configure the standard set of CPU feature flags that are applied to instances to improve live migration compatibility across Compute nodes. 4.1. Configuring CPU pinning on Compute nodes You can configure each instance CPU process to run on a dedicated host CPU by enabling CPU pinning on the Compute nodes. When an instance uses CPU pinning, each instance vCPU process is allocated its own host pCPU that no other instance vCPU process can use. Instances that run on Compute nodes with CPU pinning enabled have a NUMA topology. Each NUMA node of the instance NUMA topology maps to a NUMA node on the host Compute node. You can configure the Compute scheduler to schedule instances with dedicated (pinned) CPUs and instances with shared (floating) CPUs on the same Compute node. To configure CPU pinning on Compute nodes that have a NUMA topology, you must complete the following: Designate Compute nodes for CPU pinning. Configure the Compute nodes to reserve host cores for pinned instance vCPU processes, floating instance vCPU processes, and host processes. Deploy the overcloud. Create a flavor for launching instances that require CPU pinning. Create a flavor for launching instances that use shared, or floating, CPUs. Note Configuring CPU pinning creates an implicit NUMA topology on the instance even if a NUMA topology is not requested. Do not run NUMA and non-NUMA virtual machines (VMs) on the same hosts. For more information, see Constraints when using NUMA . 4.1.1. Prerequisites You know the NUMA topology of your Compute node. You have configured NovaReservedHugePages on the Compute nodes. For more information, see Configuring huge pages on Compute nodes . 4.1.2. Designating Compute nodes for CPU pinning To designate Compute nodes for instances with pinned CPUs, you must create a new role file to configure the CPU pinning role, and configure the bare metal nodes with a CPU pinning resource class to use to tag the Compute nodes for CPU pinning. Note The following procedure applies to new overcloud nodes that have not yet been provisioned. To assign a resource class to an existing overcloud node that has already been provisioned, you must use the scale down procedure to unprovision the node, then use the scale up procedure to reprovision the node with the new resource class assignment. For more information, see Scaling overcloud nodes . Procedure Log in to the undercloud as the stack user. Source the stackrc file: Generate a new roles data file named roles_data_cpu_pinning.yaml that includes the Controller , Compute , and ComputeCPUPinning roles, along with any other roles that you need for the overcloud: Open roles_data_cpu_pinning.yaml and edit or add the following parameters and sections: Section/Parameter Current value New value Role comment Role: Compute Role: ComputeCPUPinning Role name name: Compute name: ComputeCPUPinning description Basic Compute Node role CPU Pinning Compute Node role HostnameFormatDefault %stackname%-novacompute-%index% %stackname%-novacomputepinning-%index% deprecated_nic_config_name compute.yaml compute-cpu-pinning.yaml Register the CPU pinning Compute nodes for the overcloud by adding them to your node definition template, node.json or node.yaml . For more information, see Registering nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide. Inspect the node hardware: For more information, see Creating an inventory of the bare-metal node hardware in the Installing and managing Red Hat OpenStack Platform with director guide. Tag each bare metal node that you want to designate for CPU pinning with a custom CPU pinning resource class: Replace <node> with the ID of the bare metal node. Add the ComputeCPUPinning role to your node definition file, overcloud-baremetal-deploy.yaml , and define any predictive node placements, resource classes, network topologies, or other attributes that you want to assign to your nodes: 1 You can reuse an existing network topology or create a new custom network interface template for the role. For more information, see Custom network interface templates in the Installing and managing Red Hat OpenStack Platform with director guide. If you do not define the network definitions by using the network_config property, then the default network definitions are used. For more information about the properties you can use to configure node attributes in your node definition file, see Bare metal node provisioning attributes . For an example node definition file, see Example node definition file . Run the provisioning command to provision the new nodes for your role: Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud . Include the --network-config optional argument to provide the network definitions to the cli-overcloud-node-network-config.yaml Ansible playbook. If you do not define the network definitions by using the network_config property, then the default network definitions are used. Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active : If you did not run the provisioning command with the --network-config option, then configure the <Role>NetworkConfigTemplate parameters in your network-environment.yaml file to point to your NIC template files: Replace <cpu_pinning_net_top> with the name of the file that contains the network topology of the ComputeCPUPinning role, for example, compute.yaml to use the default network topology. 4.1.3. Configuring Compute nodes for CPU pinning Configure CPU pinning on your Compute nodes based on the NUMA topology of the nodes. Reserve some CPU cores across all the NUMA nodes for the host processes for efficiency. Assign the remaining CPU cores to managing your instances. This procedure uses the following NUMA topology, with eight CPU cores spread across two NUMA nodes, to illustrate how to configure CPU pinning: Table 4.1. Example of NUMA Topology NUMA Node 0 NUMA Node 1 Core 0 Core 1 Core 2 Core 3 Core 4 Core 5 Core 6 Core 7 The procedure reserves cores 0 and 4 for host processes, cores 1, 3, 5 and 7 for instances that require CPU pinning, and cores 2 and 6 for floating instances that do not require CPU pinning. Procedure Create an environment file to configure Compute nodes to reserve cores for pinned instances, floating instances, and host processes, for example, cpu_pinning.yaml . To schedule instances with a NUMA topology on NUMA-capable Compute nodes, add NUMATopologyFilter to the NovaSchedulerEnabledFilters parameter in your Compute environment file, if not already present: For more information on NUMATopologyFilter , see Compute scheduler filters . To reserve physical CPU cores for the dedicated instances, add the following configuration to cpu_pinning.yaml : To reserve physical CPU cores for the shared instances, add the following configuration to cpu_pinning.yaml : If you are not using file-backed memory, specify the amount of RAM to reserve for host processes: Replace <ram> with the amount of RAM to reserve in MB. To ensure that host processes do not run on the CPU cores reserved for instances, set the parameter IsolCpusList to the CPU cores you have reserved for instances: Specify the value of the IsolCpusList parameter using a list, or ranges, of CPU indices separated by a comma. Add your new files to the stack with your other environment files and deploy the overcloud: 4.1.4. Creating a dedicated CPU flavor for instances To enable your cloud users to create instances that have dedicated CPUs, you can create a flavor with a dedicated CPU policy for launching instances. Prerequisites Simultaneous multithreading (SMT) is enabled on the host. The Compute node is configured to allow CPU pinning. For more information, see Configuring CPU pinning on the Compute nodes . Procedure Source the overcloudrc file: Create a flavor for instances that require CPU pinning: To request pinned CPUs, set the hw:cpu_policy property of the flavor to dedicated : If you are not using file-backed memory, set the hw:mem_page_size property of the flavor to enable NUMA-aware memory allocation: Replace <page_size> with one of the following valid values: large : Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems. small : (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages). any : Selects the page size by using the hw_mem_page_size set on the image. If the page size is not specified by the image, selects the largest available page size, as determined by the libvirt driver. <pagesize> : Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB. Note To set hw:mem_page_size to small or any , you must have configured the amount of memory pages to reserve on each NUMA node for processes that are not instances. For more information, see Configuring huge pages on Compute nodes . To place each vCPU on thread siblings, set the hw:cpu_thread_policy property of the flavor to require : Note If the host does not have an SMT architecture or enough CPU cores with available thread siblings, scheduling fails. To prevent this, set hw:cpu_thread_policy to prefer instead of require . The prefer policy is the default policy that ensures that thread siblings are used when available. If you use hw:cpu_thread_policy=isolate , you must have SMT disabled or use a platform that does not support SMT. Verification To verify the flavor creates an instance with dedicated CPUs, use your new flavor to launch an instance: 4.1.5. Creating a shared CPU flavor for instances To enable your cloud users to create instances that use shared, or floating, CPUs, you can create a flavor with a shared CPU policy for launching instances. Prerequisites The Compute node is configured to reserve physical CPU cores for the shared CPUs. For more information, see Configuring CPU pinning on the Compute nodes . Procedure Source the overcloudrc file: Create a flavor for instances that do not require CPU pinning: To request floating CPUs, set the hw:cpu_policy property of the flavor to shared : If you are not using file-backed memory, set the hw:mem_page_size property of the flavor to enable NUMA-aware memory allocation: Replace <page_size> with one of the following valid values: large : Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems. small : (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages). any : Selects the page size by using the hw_mem_page_size set on the image. If the page size is not specified by the image, selects the largest available page size, as determined by the libvirt driver. <pagesize> : Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB. Note To set hw:mem_page_size to small or any , you must have configured the amount of memory pages to reserve on each NUMA node for processes that are not instances. For more information, see Configuring huge pages on Compute nodes . 4.1.6. Creating a mixed CPU flavor for instances To enable your cloud users to create instances that have a mix of dedicated and shared CPUs, you can create a flavor with a mixed CPU policy for launching instances. Procedure Source the overcloudrc file: Create a flavor for instances that require a mixed of dedicated and shared CPUs: Specify which CPUs must be dedicated or shared: Replace <CPU_number> with the CPUs that must be either dedicated or shared: To specify dedicated CPUs, specify the CPU number or CPU range. For example, set the property to 2-3 to specify that CPUs 2 and 3 are dedicated and all the remaining CPUs are shared. To specify shared CPUs, prepend the CPU number or CPU range with a caret (^). For example, set the property to ^0-1 to specify that CPUs 0 and 1 are shared and all the remaining CPUs are dedicated. If you are not using file-backed memory, set the hw:mem_page_size property of the flavor to enable NUMA-aware memory allocation: Replace <page_size> with one of the following valid values: large : Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems. small : (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages). any : Selects the page size by using the hw_mem_page_size set on the image. If the page size is not specified by the image, selects the largest available page size, as determined by the libvirt driver. <pagesize> : Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB. Note To set hw:mem_page_size to small or any , you must have configured the amount of memory pages to reserve on each NUMA node for processes that are not instances. For more information, see Configuring huge pages on Compute nodes . 4.1.7. Configuring CPU pinning on Compute nodes with simultaneous multithreading (SMT) If a Compute node supports simultaneous multithreading (SMT), group thread siblings together in either the dedicated or the shared set. Thread siblings share some common hardware which means it is possible for a process running on one thread sibling to impact the performance of the other thread sibling. For example, the host identifies four logical CPU cores in a dual core CPU with SMT: 0, 1, 2, and 3. Of these four, there are two pairs of thread siblings: Thread sibling 1: logical CPU cores 0 and 2 Thread sibling 2: logical CPU cores 1 and 3 In this scenario, do not assign logical CPU cores 0 and 1 as dedicated and 2 and 3 as shared. Instead, assign 0 and 2 as dedicated and 1 and 3 as shared. The files /sys/devices/system/cpu/cpuN/topology/thread_siblings_list , where N is the logical CPU number, contain the thread pairs. You can use the following command to identify which logical CPU cores are thread siblings: The following output indicates that logical CPU core 0 and logical CPU core 2 are threads on the same core: 4.1.8. Additional resources Discovering your NUMA node topology 4.2. Configuring emulator threads Compute nodes have overhead tasks associated with the hypervisor for each instance, known as emulator threads. By default, emulator threads run on the same CPUs as the instance, which impacts the performance of the instance. You can configure the emulator thread policy to run emulator threads on separate CPUs to those the instance uses. Note To avoid packet loss, you must never preempt the vCPUs in an NFV deployment. Prerequisites CPU pinning must be enabled. Procedure Log in to the undercloud as the stack user. Open your Compute environment file. To reserve physical CPU cores for instances that require CPU pinning, configure the NovaComputeCpuDedicatedSet parameter in the Compute environment file. For example, the following configuration sets the dedicated CPUs on a Compute node with a 32-core CPU: For more information, see Configuring CPU pinning on the Compute nodes . To reserve physical CPU cores for the emulator threads, configure the NovaComputeCpuSharedSet parameter in the Compute environment file. For example, the following configuration sets the shared CPUs on a Compute node with a 32-core CPU: Note The Compute scheduler also uses the CPUs in the shared set for instances that run on shared, or floating, CPUs. For more information, see Configuring CPU pinning on Compute nodes Add the Compute scheduler filter NUMATopologyFilter to the NovaSchedulerEnabledFilters parameter, if not already present. Add your Compute environment file to the stack with your other environment files and deploy the overcloud: Configure a flavor that runs emulator threads for the instance on a dedicated CPU, which is selected from the shared CPUs configured using NovaComputeCpuSharedSet : For more information about configuration options for hw:emulator_threads_policy , see Emulator threads policy in Flavor metadata . 4.3. Configuring CPU feature flags for instances You can enable or disable CPU feature flags for an instance without changing the settings on the host Compute node and rebooting the Compute node. By configuring the standard set of CPU feature flags that are applied to instances, you are helping to achieve live migration compatibility across Compute nodes. You are also helping to manage the performance and security of the instances, by disabling flags that have a negative impact on the security or performance of the instances with a particular CPU model, or enabling flags that provide mitigation from a security problem or alleviates performance problems. 4.3.1. Configuring CPU feature flags for instances Configure the Compute service to apply CPU feature flags to instances with specific vCPU models. Procedure Log in to the undercloud as the stack user. Source the stackrc file: Open your Compute environment file. Configure the instance CPU mode: Replace <cpu_mode> with the CPU mode of each instance on the Compute node. Set to one of the following valid values: host-model : (Default) Use the CPU model of the host Compute node. Use this CPU mode to automatically add critical CPU flags to the instance to provide mitigation from security flaws. custom : Use to configure the specific CPU models each instance should use. Note You can also set the CPU mode to host-passthrough to use the same CPU model and feature flags as the Compute node for the instances hosted on that Compute node. Optional: If you set NovaLibvirtCPUMode to custom , configure the instance CPU models that you want to customise: Replace <cpu_model> with a list of the CPU models that the host supports. List the CPU models in order, placing the more common and less advanced CPU models first in the list, and the more feature-rich CPU models last, for example: For a list of model names, see the /usr/share/libvirt/cpu_map.xml file, or use one of the following commands on the host Compute node: For a RHEL version 8.4 Compute node: For a RHEL version 9.2 Compute node: Replace <arch> with the name of the architecture of the Compute node, for example, x86_64 . Configure the CPU feature flags for instances with the specified CPU models: Replace <cpu_feature_flags> with a comma-separated list of feature flags to enable or disable. Prefix each flag with "+" to enable the flag, or "-" to disable it. If a prefix is not specified, the flag is enabled. For a list of the available feature flags for a given CPU model, see /usr/share/libvirt/cpu_map/*.xml . The following example enables the CPU feature flags pcid and ssbd for the IvyBridge and Cascadelake-Server models, and disables the feature flag mtrr . Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

Commands:
[stack@director ~]USD source ~/stackrc
(undercloud)USD openstack overcloud roles generate -o /home/stack/templates/roles_data_cpu_pinning.yaml Compute:ComputeCPUPinning Compute Controller
(undercloud)USD openstack overcloud node introspect --all-manageable --provide
(undercloud)USD openstack baremetal node set --resource-class baremetal.CPU-PINNING <node>
- name: Controller count: 3 - name: Compute count: 3 - name: ComputeCPUPinning count: 1 defaults: resource_class: baremetal.CPU-PINNING network_config: template: /home/stack/templates/nic-config/myRoleTopology.j2 1
(undercloud)USD openstack overcloud node provision --stack <stack> [--network-config \] --output /home/stack/templates/overcloud-baremetal-deployed.yaml /home/stack/templates/overcloud-baremetal-deploy.yaml
(undercloud)USD watch openstack baremetal node list
parameter_defaults: ComputeNetworkConfigTemplate: /home/stack/templates/nic-configs/compute.j2 ComputeCPUPinningNetworkConfigTemplate: /home/stack/templates/nic-configs/<cpu_pinning_net_top>.j2 ControllerNetworkConfigTemplate: /home/stack/templates/nic-configs/controller.j2
parameter_defaults: NovaSchedulerEnabledFilters: - AvailabilityZoneFilter - ComputeFilter - ComputeCapabilitiesFilter - ImagePropertiesFilter - ServerGroupAntiAffinityFilter - ServerGroupAffinityFilter - PciPassthroughFilter - NUMATopologyFilter
parameter_defaults: ComputeCPUPinningParameters: NovaComputeCpuDedicatedSet: 1,3,5,7
parameter_defaults: ComputeCPUPinningParameters: NovaComputeCpuSharedSet: 2,6
parameter_defaults: ComputeCPUPinningParameters: NovaReservedHugePages: <ram>
parameter_defaults: ComputeCPUPinningParameters: IsolCpusList: 1-3,5-7
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -r /home/stack/templates/roles_data_cpu_pinning.yaml -e /home/stack/templates/network-environment.yaml -e /home/stack/templates/cpu_pinning.yaml -e /home/stack/templates/overcloud-baremetal-deployed.yaml -e /home/stack/templates/node-info.yaml
(undercloud)USD source ~/overcloudrc
(overcloud)USD openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <no_reserved_vcpus> pinned_cpus
(overcloud)USD openstack flavor set --property hw:cpu_policy=dedicated pinned_cpus
(overcloud)USD openstack flavor set --property hw:mem_page_size=<page_size> pinned_cpus
(overcloud)USD openstack flavor set --property hw:cpu_thread_policy=require pinned_cpus
(overcloud)USD openstack server create --flavor pinned_cpus --image <image> pinned_cpu_instance
(undercloud)USD source ~/overcloudrc
(overcloud)USD openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <no_reserved_vcpus> floating_cpus
(overcloud)USD openstack flavor set --property hw:cpu_policy=shared floating_cpus
(overcloud)USD openstack flavor set --property hw:mem_page_size=<page_size> pinned_cpus
(undercloud)USD source ~/overcloudrc
(overcloud)USD openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <number_of_reserved_vcpus> --property hw:cpu_policy=mixed mixed_CPUs_flavor
(overcloud)USD openstack flavor set --property hw:cpu_dedicated_mask=<CPU_number> mixed_CPUs_flavor
(overcloud)USD openstack flavor set --property hw:mem_page_size=<page_size> pinned_cpus
grep -H . /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | sort -n -t ':' -k 2 -u
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0,2 /sys/devices/system/cpu/cpu2/topology/thread_siblings_list:1,3
parameter_defaults: NovaComputeCpuDedicatedSet: 2-15,18-31
parameter_defaults: NovaComputeCpuSharedSet: 0,1,16,17
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml
(overcloud)USD openstack flavor set --property hw:cpu_policy=dedicated --property hw:emulator_threads_policy=share dedicated_emulator_threads
[stack@director ~]USD source ~/stackrc
parameter_defaults: ComputeParameters: NovaLibvirtCPUMode: <cpu_mode>
parameter_defaults: ComputeParameters: NovaLibvirtCPUMode: 'custom' NovaLibvirtCPUModels: <cpu_model>
NovaLibvirtCPUModels: - SandyBridge - IvyBridge - Haswell-noTSX-IBRS
sudo podman exec -it nova_libvirt virsh cpu-models <arch>
sudo podman exec -it nova_virtqemud virsh cpu-models <arch>
parameter_defaults: ComputeParameters: NovaLibvirtCPUModelExtraFlags: <cpu_feature_flags>
parameter_defaults: ComputeParameters: NovaLibvirtCPUMode: 'custom' NovaLibvirtCPUModels: - IvyBridge - Cascadelake-Server NovaLibvirtCPUModelExtraFlags: 'pcid,+ssbd,-mtrr'
(undercloud)USD openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/<compute_environment_file>.yaml