Chapter 11. Enabling RT-KVM for NFV Workloads To facilitate installing and configuring Red Hat Enterprise Linux Real Time KVM (RT-KVM), Red Hat OpenStack Platform provides the following features: A real-time Compute node role that provisions Red Hat Enterprise Linux for real-time. The additional RT-KVM kernel module. Automatic configuration of the Compute node. 11.1. Planning for your RT-KVM Compute nodes When planning for RT-KVM Compute nodes, ensure that the following tasks are completed: You must use Red Hat certified servers for your RT-KVM Compute nodes. For more information, see Red Hat Enterprise Linux for Real Time certified servers . Register your undercloud and attach a valid Red Hat OpenStack Platform subscription. For more information, see Registering the undercloud and attaching subscriptions in the Director Installation and Usage guide. Enable the repositories that are required for the undercloud, such as the rhel-9-server-nfv-rpms repository for RT-KVM, and update the system packages to the latest versions. Note You need a separate subscription to a Red Hat OpenStack Platform for Real Time SKU before you can access this repository. For more information, see Enabling repositories for the undercloud in the Director Installation and Usage guide. Building the real-time image Install the libguestfs-tools package on the undercloud to get the virt-customize tool: Important If you install the libguestfs-tools package on the undercloud, disable iscsid.socket to avoid port conflicts with the tripleo_iscsid service on the undercloud: Extract the images: Copy the default image: Register your image to enable Red Hat repositories relevant to your customizations. Replace [username] and [password] with valid credentials in the following example. Note For security, you can remove credentials from the history file if they are used on the command prompt. You can delete individual lines in history using the history -d command followed by the line number. Find a list of pool IDs from your account's subscriptions, and attach the appropriate pool ID to your image. Add the repositories necessary for Red Hat OpenStack Platform with NFV. Create a script to configure real-time capabilities on the image. Run the script to configure the real-time image: Note If you see the following line in the rt.sh script output, "grubby fatal error: unable to find a suitable template" , you can ignore this error. Examine the virt-customize.log file that resulted from the command, to check that the packages installed correctly using the rt.sh script . Relabel SELinux: Extract vmlinuz and initrd: Note The software version in the vmlinuz and initramfs filenames vary with the kernel version. Upload the image: You now have a real-time image you can use with the ComputeOvsDpdkRT composable role on your selected Compute nodes. Modifying BIOS settings on RT-KVM Compute nodes To reduce latency on your RT-KVM Compute nodes, disable all options for the following parameters in your Compute node BIOS settings: Power Management Hyper-Threading CPU sleep states Logical processors 11.2. Configuring OVS-DPDK with RT-KVM Note You must determine the best values for the OVS-DPDK parameters that you set in the network-environment.yaml file to optimize your OpenStack network for OVS-DPDK. For more details, see Deriving DPDK parameters with workflows . 11.2.1. Generating the ComputeOvsDpdk composable role Use the ComputeOvsDpdkRT role to specify Compute nodes for the real-time compute image. Generate roles_data.yaml for the ComputeOvsDpdkRT role. 11.2.2. Configuring the OVS-DPDK parameters Important Determine the best values for the OVS-DPDK parameters in the network-environment.yaml file to optimize your deployment. For more information, see Section 9.1, "Deriving DPDK parameters with workflows" . Add the NIC configuration for the OVS-DPDK role you use under resource_registry : resource_registry: # Specify the relative/absolute path to the config files you want to use for override the default. OS::TripleO::ComputeOvsDpdkRT::Net::SoftwareConfig: nic-configs/compute-ovs-dpdk.yaml OS::TripleO::Controller::Net::SoftwareConfig: nic-configs/controller.yaml Under parameter_defaults , set the OVS-DPDK, and RT-KVM parameters: # DPDK compute node. ComputeOvsDpdkRTParameters: KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1-7,17-23,9-15,25-31" TunedProfileName: "realtime-virtual-host" IsolCpusList: "1,2,3,4,5,6,7,9,10,17,18,19,20,21,22,23,11,12,13,14,15,25,26,27,28,29,30,31" NovaComputeCpuDedicatedSet: ['2,3,4,5,6,7,18,19,20,21,22,23,10,11,12,13,14,15,26,27,28,29,30,31'] NovaReservedHostMemory: 4096 OvsDpdkSocketMemory: "1024,1024" OvsDpdkMemoryChannels: "4" OvsPmdCoreList: "1,17,9,25" VhostuserSocketGroup: "hugetlbfs" ComputeOvsDpdkRTImage: "overcloud-realtime-compute" 11.2.3. Deploying the overcloud Deploy the overcloud for ML2-OVS: (undercloud) [stack@undercloud-0 ~]USD openstack overcloud deploy \ --templates \ -r /home/stack/ospd-16-vlan-dpdk-ctlplane-bonding-rt/roles_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs-dpdk.yaml \ -e /home/stack/ospd-16-vxlan-dpdk-data-bonding-rt-hybrid/containers-prepare-parameter.yaml \ -e /home/stack/ospd-16-vxlan-dpdk-data-bonding-rt-hybrid/network-environment.yaml 11.3. Launching an RT-KVM instance Perform the following steps to launch an RT-KVM instance on a real-time enabled Compute node: Create an RT-KVM flavor on the overcloud: Launch an RT-KVM instance: To verify that the instance uses the assigned emulator threads, run the following command:

Commands:
(undercloud) [stack@undercloud-0 ~]USD sudo dnf install libguestfs-tools
sudo systemctl disable --now iscsid.socket
(undercloud) [stack@undercloud-0 ~]USD tar -xf /usr/share/rhosp-director-images/overcloud-full.tar (undercloud) [stack@undercloud-0 ~]USD tar -xf /usr/share/rhosp-director-images/ironic-python-agent.tar
(undercloud) [stack@undercloud-0 ~]USD cp overcloud-hardened-uefi-full.qcow2 overcloud-realtime-compute.qcow2
virt-customize -a overcloud-realtime-compute.qcow2 --run-command 'subscription-manager register --username=[username] --password=[password]' subscription-manager release --set 8.4
sudo subscription-manager list --all --available | less virt-customize -a overcloud-realtime-compute.qcow2 --run-command 'subscription-manager attach --pool [pool-ID]'
virt-customize -a overcloud-realtime-compute.qcow2 --run-command 'sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms --enable=openstack-16.2-for-rhel-8-x86_64-rpms --enable=rhel-8-for-x86_64-nfv-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms'
(undercloud) [stack@undercloud-0 ~]USD cat <<'EOF' > rt.sh #!/bin/bash set -eux dnf -v -y --setopt=protected_packages= erase kernel.USD(uname -m) dnf -v -y install kernel-rt kernel-rt-kvm tuned-profiles-nfv-host grubby --set-default /boot/vmlinuz*rt* EOF
(undercloud) [stack@undercloud-0 ~]USD virt-customize -a overcloud-realtime-compute.qcow2 -v --run rt.sh 2>&1 | tee virt-customize.log
(undercloud) [stack@undercloud-0 ~]USD cat virt-customize.log | grep Verifying Verifying : kernel-3.10.0-957.el7.x86_64 1/1 Verifying : 10:qemu-kvm-tools-rhev-2.12.0-18.el7_6.1.x86_64 1/8 Verifying : tuned-profiles-realtime-2.10.0-6.el7_6.3.noarch 2/8 Verifying : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/8 Verifying : tuned-profiles-nfv-host-2.10.0-6.el7_6.3.noarch 4/8 Verifying : kernel-rt-kvm-3.10.0-957.10.1.rt56.921.el7.x86_64 5/8 Verifying : tuna-0.13-6.el7.noarch 6/8 Verifying : kernel-rt-3.10.0-957.10.1.rt56.921.el7.x86_64 7/8 Verifying : rt-setup-2.0-6.el7.x86_64 8/8
(undercloud) [stack@undercloud-0 ~]USD virt-customize -a overcloud-realtime-compute.qcow2 --selinux-relabel
(undercloud) [stack@undercloud-0 ~]USD mkdir image (undercloud) [stack@undercloud-0 ~]USD guestmount -a overcloud-realtime-compute.qcow2 -i --ro image (undercloud) [stack@undercloud-0 ~]USD cp image/boot/vmlinuz-3.10.0-862.rt56.804.el7.x86_64 ./overcloud-realtime-compute.vmlinuz (undercloud) [stack@undercloud-0 ~]USD cp image/boot/initramfs-3.10.0-862.rt56.804.el7.x86_64.img ./overcloud-realtime-compute.initrd (undercloud) [stack@undercloud-0 ~]USD guestunmount image
(undercloud) [stack@undercloud-0 ~]USD openstack overcloud image upload --update-existing --os-image-name overcloud-realtime-compute.qcow2
(undercloud) [stack@undercloud-0 ~]USD openstack overcloud roles generate -o roles_data.yaml Controller ComputeOvsDpdkRT
resource_registry: # Specify the relative/absolute path to the config files you want to use for override the default. OS::TripleO::ComputeOvsDpdkRT::Net::SoftwareConfig: nic-configs/compute-ovs-dpdk.yaml OS::TripleO::Controller::Net::SoftwareConfig: nic-configs/controller.yaml
DPDK compute node. ComputeOvsDpdkRTParameters: KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1-7,17-23,9-15,25-31" TunedProfileName: "realtime-virtual-host" IsolCpusList: "1,2,3,4,5,6,7,9,10,17,18,19,20,21,22,23,11,12,13,14,15,25,26,27,28,29,30,31" NovaComputeCpuDedicatedSet: ['2,3,4,5,6,7,18,19,20,21,22,23,10,11,12,13,14,15,26,27,28,29,30,31'] NovaReservedHostMemory: 4096 OvsDpdkSocketMemory: "1024,1024" OvsDpdkMemoryChannels: "4" OvsPmdCoreList: "1,17,9,25" VhostuserSocketGroup: "hugetlbfs" ComputeOvsDpdkRTImage: "overcloud-realtime-compute"
(undercloud) [stack@undercloud-0 ~]USD openstack overcloud deploy --templates -r /home/stack/ospd-16-vlan-dpdk-ctlplane-bonding-rt/roles_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs-dpdk.yaml -e /home/stack/ospd-16-vxlan-dpdk-data-bonding-rt-hybrid/containers-prepare-parameter.yaml -e /home/stack/ospd-16-vxlan-dpdk-data-bonding-rt-hybrid/network-environment.yaml
openstack flavor create r1.small 99 4096 20 4 openstack flavor set --property hw:cpu_policy=dedicated 99 openstack flavor set --property hw:cpu_realtime=yes 99 openstack flavor set --property hw:mem_page_size=1GB 99 openstack flavor set --property hw:cpu_realtime_mask="^0-1" 99 openstack flavor set --property hw:cpu_emulator_threads=isolate 99
openstack server create --image <rhel> --flavor r1.small --nic net-id=<dpdk-net> test-rt
virsh dumpxml <instance-id> | grep vcpu -A1 <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='3'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='7'/> <emulatorpin cpuset='0-1'/> <vcpusched vcpus='2-3' scheduler='fifo' priority='1'/> </cputune>