Chapter 4. Configuring CPUs on Compute nodes Warning The content for this feature is available in this release as a Documentation Preview , and therefore is not fully verified by Red Hat. Use it only for testing, and do not use in a production environment. As a cloud administrator, you can configure the scheduling and placement of instances for optimal performance by creating customized flavors to target specialized workloads, including NFV and High Performance Computing (HPC). Use the following features to tune your instances for optimal CPU performance: CPU pinning : Pin virtual CPUs to physical CPUs. Emulator threads : Pin emulator threads associated with the instance to physical CPUs. CPU feature flags : Configure the standard set of CPU feature flags that are applied to instances to improve live migration compatibility across Compute nodes. 4.1. Configuring CPU pinning on Compute nodes You can configure each instance CPU process to run on a dedicated host CPU by enabling CPU pinning on the Compute nodes. When an instance uses CPU pinning, each instance vCPU process is allocated its own host pCPU that no other instance vCPU process can use. Instances that run on Compute nodes with CPU pinning enabled have a NUMA topology. Each NUMA node of the instance NUMA topology maps to a NUMA node on the host Compute node. You can configure the Compute scheduler to schedule instances with dedicated (pinned) CPUs and instances with shared (floating) CPUs on the same Compute node. To configure CPU pinning on Compute nodes that have a NUMA topology, you must complete the following: Designate Compute nodes for CPU pinning. Configure the Compute nodes to reserve host cores for pinned instance vCPU processes, floating instance vCPU processes, and host processes. Deploy the data plane. Create a flavor for launching instances that require CPU pinning. Create a flavor for launching instances that use shared, or floating, CPUs. Note Configuring CPU pinning creates an implicit NUMA topology on the instance even if a NUMA topology is not requested. Do not run NUMA and non-NUMA virtual machines (VMs) on the same hosts. 4.1.1. Prerequisites You know the NUMA topology of your Compute node. The oc command line tool is installed on your workstation. You are logged in to Red Hat OpenStack Services on OpenShift (RHOSO) as a user with cluster-admin privileges. 4.1.2. Designating and configuring Compute nodes for CPU pinning To designate Compute nodes for instances with pinned CPUs, you must create and configure a new OpenStackDataPlaneNodeSet custom resource (CR) to configure the nodes that are designated for CPU pinning. Configure CPU pinning on your Compute nodes based on the NUMA topology of the nodes. Reserve some CPU cores across all the NUMA nodes for the host processes for efficiency. Assign the remaining CPU cores to managing your instances. This procedure uses the following NUMA topology, with eight CPU cores spread across two NUMA nodes, to illustrate how to configure CPU pinning: Table 4.1. Example of NUMA Topology NUMA Node 0 NUMA Node 1 Core 0 Core 1 Core 4 Core 5 Core 2 Core 3 Core 6 Core 7 The procedure reserves cores 0 and 4 for host processes, cores 1, 3, 5 and 7 for instances that require CPU pinning, and cores 2 and 6 for floating instances that do not require CPU pinning. Note The following procedure applies to new OpenStackDataPlaneNodeSet CRs that have not yet been provisioned. To reconfigure an existing OpenStackDataPlaneNodeSet that has already been provisioned, you must first drain the guest instances from all the nodes in the OpenStackDataPlaneNodeSet . Note Configuring CPU pinning creates an implicit NUMA topology on the instance even if a NUMA topology is not requested. Do not run NUMA and non-NUMA virtual machines (VMs) on the same hosts. Prerequisites You have selected the OpenStackDataPlaneNodeSet CR that defines the nodes for which you want to designate and configure CPU pinning. For more information about creating an OpenStackDataPlaneNodeSet CR, see Creating the data plane in the Deploying Red Hat OpenStack Services on OpenShift guide. Procedure Create or update the ConfigMap CR named nova-extra-config.yaml and set the values of the parameters under [compute] and [default]: 1 The name of the new Compute configuration file. The nova-operator generates the default configuration file with the name 01-nova.conf . Do not use the default name, because it would override the infrastructure configuration, such as the transport_url . The nova-compute service applies every file under /etc/nova/nova.conf.d/ in lexicographical order, therefore configurations defined in later files override the same configurations defined in an earlier file. 2 Reserves physical CPU cores for the shared instances. 3 Reserves physical CPU cores for the dedicated instances. 4 Specifies the amount memory to reserve per NUMA node. For more information about creating ConfigMap objects, see Creating and using config maps . Create a new OpenStackDataPlaneDeployment CR to configure the services on the data plane nodes and deploy the data plane, and save it to a file named compute_cpu_pinning_deploy.yaml on your workstation: For more information about creating an OpenStackDataPlaneDeployment CR, see Deploying the data plane in the Deploying Red Hat OpenStack Services on OpenShift guide. In the compute_cpu_pinning_deploy.yaml , specify nodeSets to include all the OpenStackDataPlaneNodeSet CRs you want to deploy. Ensure that you include the OpenStackDataPlaneNodeSet CR that you selected as a prerequisite. That OpenStackDataPlaneNodeSet CR defines the nodes you want to designate for CPU pinning. Warning You can configure only whole node sets. Reconfiguring a subset of the nodes within a node set is not supported. If you need to reconfigure a subset of nodes within a node set, you must scale the node set down, and create a new node set from the previously removed nodes. Warning If your deployment has more than one node set, changes to the nova-extra-config.yaml ConfigMap might directly affect more than one node set, depending on how the node sets and the DataPlaneServices are configured. To check if a node set uses the nova-extra-config ConfigMap and therefore will be affected by the reconfiguration, complete the following steps: Check the services list of the node set and find the name of the DataPlaneService that points to nova. Ensure that the value of the edpmServiceType field of the DataPlaneService is set to nova . If the dataSources list of the DataPlaneService contains a configMapRef named nova-extra-config , then this node set uses this ConfigMap and therefore will be affected by the configuration changes in this ConfigMap . If some of the node sets that are affected should not be reconfigured, you must create a new DataPlaneService pointing to a separate ConfigMap for these node sets. Replace <nodeSet_name> with the names of the OpenStackDataPlaneNodeSet CRs that you want to include in your data plane deployment. Save the compute_cpu_pinning_deploy.yaml deployment file. Deploy the data plane: Verify that the data plane is deployed: Access the remote shell for openstackclient and verify that the deployed Compute nodes are visible on the control plane: 4.1.3. Creating a dedicated CPU flavor for instances To enable your cloud users to create instances that have dedicated CPUs, you can create a flavor with a dedicated CPU policy for launching instances. Prerequisites Simultaneous multithreading (SMT) is configured on the host if you intend to use the required cpu_thread_policy . You can have a mix of SMT and non-SMT Compute hosts. Flavors with the require cpu_thread_policy will land on SMT hosts, and flavors with isolate will land on non-SMT. The Compute node is configured to allow CPU pinning. For more information, see Configuring CPU pinning on the Compute nodes . Procedure Create a flavor for instances that require CPU pinning: If you are not using file-backed memory, set the hw:mem_page_size property of the flavor to enable NUMA-aware memory allocation: Replace <page_size> with one of the following valid values: large : Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems. small : (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages). any : Selects the page size by using the hw_mem_page_size set on the image. If the page size is not specified by the image, selects the largest available page size, as determined by the libvirt driver. <pagesize> : Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB. Note To set hw:mem_page_size to small or any , you must have configured the amount of memory pages to reserve on each NUMA node for processes that are not instances. To request pinned CPUs, set the hw:cpu_policy property of the flavor to dedicated : Optional: To place each vCPU on thread siblings, set the hw:cpu_thread_policy property of the flavor to require : Note If the host does not have an SMT architecture or enough CPU cores with available thread siblings, scheduling fails. To prevent this, set hw:cpu_thread_policy to prefer instead of require . The prefer policy is the default policy that ensures that thread siblings are used when available. If you use hw:cpu_thread_policy=isolate , you must have SMT disabled or use a platform that does not support SMT. To verify the flavor creates an instance with dedicated CPUs, use your new flavor to launch an instance: 4.1.4. Creating a shared CPU flavor for instances To enable your cloud users to create instances that use shared, or floating, CPUs, you can create a flavor with a shared CPU policy for launching instances. Prerequisites The Compute node is configured to reserve physical CPU cores for the shared CPUs. For more information, see Configuring CPU pinning on the Compute nodes . Procedure Create a flavor for instances that do not require CPU pinning: To request floating CPUs, set the hw:cpu_policy property of the flavor to shared : 4.1.5. Creating a mixed CPU flavor for instances To enable your cloud users to create instances that have a mix of dedicated and shared CPUs, you can create a flavor with a mixed CPU policy for launching instances. Procedure Create a flavor for instances that require a mix of dedicated and shared CPUs: Specify which CPUs must be dedicated or shared: Replace <CPU_MASK> with the CPUs that must be either dedicated or shared: To specify dedicated CPUs, specify the CPU number or CPU range. For example, set the property to 2-3 to specify that CPUs 2 and 3 are dedicated and all the remaining CPUs are shared. To specify shared CPUs, prepend the CPU number or CPU range with a caret (^). For example, set the property to ^0-1 to specify that CPUs 0 and 1 are shared and all the remaining CPUs are dedicated. If you are not using file-backed memory, set the hw:mem_page_size property of the flavor to enable NUMA-aware memory allocation: Replace <page_size> with one of the following valid values: large : Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems. small : (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages). any : Selects the page size by using the hw_mem_page_size set on the image. If the page size is not specified by the image, selects the largest available page size, as determined by the libvirt driver. <pagesize> : Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB. Note To set hw:mem_page_size to small or any , you must have configured the amount of memory pages to reserve on each NUMA node for processes that are not instances. 4.1.6. Configuring CPU pinning on Compute nodes with simultaneous multithreading (SMT) If a Compute node supports simultaneous multithreading (SMT), group thread siblings together in either the dedicated or the shared set. Thread siblings share some common hardware which means it is possible for a process running on one thread sibling to impact the performance of the other thread sibling. For example, the host identifies four logical CPU cores in a dual core CPU with SMT: 0, 1, 2, and 3. Of these four, there are two pairs of thread siblings: Thread sibling 1: logical CPU cores 0 and 2 Thread sibling 2: logical CPU cores 1 and 3 In this scenario, do not assign logical CPU cores 0 and 1 as dedicated and 2 and 3 as shared. Instead, assign 0 and 2 as dedicated and 1 and 3 as shared. The files /sys/devices/system/cpu/cpuN/topology/thread_siblings_list , where N is the logical CPU number, contain the thread pairs. You can use the following command to identify which logical CPU cores are thread siblings: The following output indicates that logical CPU core 0 and logical CPU core 2 are threads on the same core: 4.1.7. Additional resources Discovering your NUMA node topology

Commands:
apiVersion: v1 kind: ConfigMap metadata: name: nova-extra-config namespace: openstack data: 25-nova-cpu-pinning.conf: | 1 [compute] cpu_shared_set = 2,6 2 cpu_dedicated_set = 1,3,5,7 3 [DEFAULT] reserved_huge_pages = node:0,size:4,count:131072 4 reserved_huge_pages = node:1,size:4,count:131072
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: openstack-edpm-cpu-pinning
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: openstack-edpm-cpu-pinning spec: nodeSets: - openstack-edpm - compute-cpu-pinning - - <nodeSet_name>
oc create -f compute_cpu_pinning_deploy.yaml
oc get openstackdataplanenodeset NAME STATUS MESSAGE compute-cpu-pinning True Deployed
oc rsh -n openstack openstackclient openstack hypervisor list
openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <num_guest_vcpus> pinned_cpus
openstack --os-compute-api=2.86 flavor set --property hw:mem_page_size=<page_size> pinned_cpus
openstack --os-compute-api=2.86 flavor set --property hw:cpu_policy=dedicated pinned_cpus
openstack --os-compute-api=2.86 flavor set --property hw:cpu_thread_policy=require pinned_cpus
openstack server create --flavor pinned_cpus --image <image> pinned_cpu_instance
openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <no_reserved_vcpus> floating_cpus
openstack --os-compute-api=2.86 flavor set --property hw:cpu_policy=shared floating_cpus
openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <number_of_reserved_vcpus> --property hw:cpu_policy=mixed mixed_CPUs_flavor
openstack --os-compute-api=2.86 flavor set --property hw:cpu_dedicated_mask=<CPU_MASK> mixed_CPUs_flavor
openstack --os-compute-api=2.86 flavor set --property hw:mem_page_size=<page_size> mixed_CPUs_flavor
grep -H . /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | sort -n -t ':' -k 2 -u
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0,2 /sys/devices/system/cpu/cpu2/topology/thread_siblings_list:1,3