Chapter 19. Creating nested virtual machines You can use nested virtual machines (VMs) if you require a different host operating system than what your local host is running. This eliminates the need for additional physical hardware. Warning Red Hat currently provides nested virtualization only as a Technology Preview , and it is therefore unsupported. Additionally, nested virtualization has only been tested on a limited set of architectures and operating system versions. Before you use this feature in your environment, see Restrictions and limitations for nested virtualization . 19.1. What is nested virtualization? With nested virtualization, you can run virtual machines (VMs) within other VMs. A standard VM that runs on a physical host can also act as a second hypervisor and create its own VMs. Nested virtualization terminology Level 0 ( L0 ) A physical host, a bare-metal machine. Level 1 ( L1 ) A standard VM, running on an L0 physical host, that can act as an additional virtual host. Level 2 ( L2 ) A nested VM running on an L1 virtual host. Important: The second level of virtualization severely limits the performance of an L2 VM. For this reason, nested virtualization is primarily intended for development and testing scenarios, such as: Debugging hypervisors in a constrained environment Testing larger virtual deployments on a limited amount of physical resources Warning Red Hat currently provides nested virtualization only as a Technology Preview , and it is therefore unsupported. Additionally, nested virtualization has only been tested on a limited set of architectures and operating system versions. Before you use this feature in your environment, see Restrictions and limitations for nested virtualization . Additional resources Restrictions and limitations for nested virtualization 19.2. Restrictions and limitations for nested virtualization Keep the following restrictions in mind when using nested virtualization. To learn more about the relevant terminology for nested virtualization, see What is nested virtualization? Warning Red Hat currently does not support nested virtualization, and only provides nesting as a Technology Preview . Tested architectures The L0 host must be an Intel, AMD, IBM POWER9, or IBM Z system. Nested virtualization currently does not work on other architectures, such as ARM. Tested environments To create nested virtual machines (VMs), you must use the following versions of operating systems: On the L0 host : On the L1 VMs : On the L2 VMs : RHEL 8.2 and later RHEL 7.8 and later RHEL 7.8 and later RHEL 8.2 and later RHEL 8.2 and later Windows Server 2016 Windows Server 2019 Note Creating RHEL L1 VMs is not tested when used in other Red Hat virtualization offerings. These include: Red Hat Virtualization Red Hat OpenStack Platform OpenShift Virtualization In addition, on IBM POWER9, nested virtualization currently only works under the following circumstances: Both the L0 host and the L1 VM use RHEL 8 The L2 VM uses RHEL 8, or RHEL 7 with a rhel-alt kernel. The L1 VM and L2 VM are not running in POWER8 compatibility mode. Hypervisor limitations Currently, Red Hat tests nesting only on RHEL-KVM. When RHEL is used as the L0 hypervisor, you can use RHEL or Windows as the L1 hypervisor. When using an L1 RHEL 8 VM on a non-KVM L0 hypervisor, such as VMware ESXi or Amazon Web Services (AWS), creating L2 VMs in the RHEL 8 guest operating system might work, but is not tested. Feature limitations Use of L2 VMs as hypervisors and creating L3 guests has not been properly tested and is not expected to work. Migrating VMs currently does not work on AMD systems if nested virtualization has been enabled on the L0 host. On an IBM Z system, huge-page backing storage and nested virtualization cannot be used at the same time. subs="+quotes,attributes"] Some features available on the L0 host might be unavailable for the L1 hypervisor. For example, on IBM POWER 9 hardware, the External Interrupt Virtualization Engine (XIVE) does not work. However, L1 VMs can use the emulated XIVE interrupt controller to start L2 VMs. Additional resources Creating a nested virtual machine on Intel Creating a nested virtual machine on AMD Creating a nested virtual machine on IBM Z Creating a nested virtual machine on IBM POWER9 19.3. Creating a nested virtual machine on Intel Follow the steps below to enable and configure nested virtualization on an Intel host. Warning Red Hat currently provides nested virtualization only as a Technology Preview , and it is therefore unsupported. Additionally, nested virtualization has only been tested on a limited set of architectures and operating system versions. Before you use this feature in your environment, see Restrictions and limitations for nested virtualization . Prerequisites An L0 RHEL 8 host running an L1 virtual machine (VM). The hypervisor CPU must support nested virtualization. To verify, use the cat /proc/cpuinfo command on the L0 hypervisor. If the output of the command includes the vmx and ept flags, creating L2 VMs is possible. This is generally the case on Intel Xeon v3 cores and later. Ensure that nested virtualization is enabled on the L0 host: If the command returns 1 or Y , the feature is enabled. Skip the remaining prerequisite steps, and continue with the Procedure section. If the command returns 0 or N but your system supports nested virtualization, use the following steps to enable the feature. Unload the kvm_intel module: Activate the nesting feature: The nesting feature is now enabled, but only until the reboot of the L0 host. To enable it permanently, add the following line to the /etc/modprobe.d/kvm.conf file: Procedure Configure your L1 VM for nested virtualization. Open the XML configuration of the VM. The following example opens the configuration of the Intel-L1 VM: Configure the VM to use host-passthrough CPU mode by editing the <cpu> element: If you require the VM to use a specific CPU model, configure the VM to use custom CPU mode. Inside the <cpu> element, add a <feature policy='require' name='vmx'/> element and a <model> element with the CPU model specified inside. For example: Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM . 19.4. Creating a nested virtual machine on AMD Follow the steps below to enable and configure nested virtualization on an AMD host. Warning Red Hat currently provides nested virtualization only as a Technology Preview , and it is therefore unsupported. Additionally, nested virtualization has only been tested on a limited set of architectures and operating system versions. Before you use this feature in your environment, see Restrictions and limitations for nested virtualization . Prerequisites An L0 RHEL 8 host running an L1 virtual machine (VM). The hypervisor CPU must support nested virtualization. To verify, use the cat /proc/cpuinfo command on the L0 hypervisor. If the output of the command includes the svm and npt flags, creating L2 VMs is possible. This is generally the case on AMD EPYC cores and later. Ensure that nested virtualization is enabled on the L0 host: If the command returns 1 or Y , the feature is enabled. Skip the remaining prerequisite steps, and continue with the Procedure section. If the command returns 0 or N , use the following steps to enable the feature. Stop all running VMs on the L0 host. Unload the kvm_amd module: Activate the nesting feature: The nesting feature is now enabled, but only until the reboot of the L0 host. To enable it permanently, add the following to the /etc/modprobe.d/kvm.conf file: Procedure Configure your L1 VM for nested virtualization. Open the XML configuration of the VM. The following example opens the configuration of the AMD-L1 VM: Configure the VM to use host-passthrough CPU mode by editing the <cpu> element: If you require the VM to use a specific CPU model, configure the VM to use custom CPU mode. Inside the <cpu> element, add a <feature policy='require' name='svm'/> element and a <model> element with the CPU model specified inside. For example: Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM . 19.5. Creating a nested virtual machine on IBM Z Follow the steps below to enable and configure nested virtualization on an IBM Z host. Note IBM Z does not really provide a bare-metal L0 host. Instead, user systems are set up on a logical partition (LPAR), which is already a virtualized system, so it is often referred to as L1 . However, for better alignment with other architectures in this guide, the following steps refer to IBM Z as if it provides an L0 host. To learn more about nested virtualization, see: What is nested virtualization? Warning Red Hat currently provides nested virtualization only as a Technology Preview , and it is therefore unsupported. Additionally, nested virtualization has only been tested on a limited set of architectures and operating system versions. Before you use this feature in your environment, see Restrictions and limitations for nested virtualization . Prerequisites An L0 RHEL 8 host running an L1 virtual machine (VM). The hypervisor CPU must support nested virtualization. To verify this is the case, use the cat /proc/cpuinfo command on the L0 hypervisor. If the output of the command includes the sie flag, creating L2 VMs is possible. Ensure that nested virtualization is enabled on the L0 host: If the command returns 1 or Y , the feature is enabled. Skip the remaining prerequisite steps, and continue with the Procedure section. If the command returns 0 or N , use the following steps to enable the feature. Stop all running VMs on the L0 host. Unload the kvm module: Activate the nesting feature: The nesting feature is now enabled, but only until the reboot of the L0 host. To enable it permanently, add the following line to the /etc/modprobe.d/kvm.conf file: Procedure Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM . 19.6. Creating a nested virtual machine on IBM POWER9 Follow the steps below to enable and configure nested virtualization on an IBM POWER9 host. Note IBM POWER9 does not really provide a bare-metal L0 host. Instead, user systems are set up on a logical partition (LPAR), which is already a virtualized system, so it is often referred to as L1 . However, for better alignment with other architectures in this guide, the following steps refer to IBM POWER9 as if it provides an L0 host. To learn more about nested virtualization, see: What is nested virtualization? Warning Nested virtualization is currently provided only as a Technology Preview on the IBM POWER9 architecture, and is therefore unsupported. In addition, creating nested virtual machines (VMs) is not possible on versions of IBM POWER systems, such as IBM POWER8. Prerequisites An L0 RHEL 8 host is running an L1 VM. The L1 VM is using RHEL 8 as the guest operating system. Nested virtualization is enabled on the L0 host: If the command returns 1 or Y , the feature is enabled. Skip the remaining prerequisite steps, and continue with the Procedure section. If the command returns 0 or N , use the following steps to enable the feature: Stop all running VMs on the L0 host. Unload the kvm module: Activate the nesting feature: The nesting feature is now enabled, but only until the reboot of the L0 host. To enable it permanently, add the following line to the /etc/modprobe.d/kvm.conf file: Procedure To ensure that the L1 VM can create L2 VMs, add the cap-nested-hv parameter to the machine type of the L1 VM. To do so, use the virsh edit command to modify the L1 VM's XML configuration, and the following line to the <features> section: Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM . To significantly improve the performance of L2 VMs, Red Hat recommends adding the`cap-nested-hv` parameter to the XML configurations of L2 VMs as well. For instructions, see the step. Additional information Note that IBM POWER8 as the architecture for the L2 VM currently does not supported.

Commands:
*modprobe kvm hpage=1 nested=1* modprobe: ERROR: could not insert 'kvm': Invalid argument *dmesg |tail -1* [90226.508366] kvm-s390: A KVM host that supports nesting cannot back its KVM guests with huge pages
cat /sys/module/kvm_intel/parameters/nested
modprobe -r kvm_intel
modprobe kvm_intel nested=1
options kvm_intel nested=1
virsh edit Intel-L1
<cpu mode='host-passthrough' />
<cpu mode ='custom' match ='exact' check='partial'> <model fallback='allow'> Haswell-noTSX </model> <feature policy='require' name='vmx'/> </cpu>
cat /sys/module/kvm_amd/parameters/nested
modprobe -r kvm_amd
modprobe kvm_amd nested=1
options kvm_amd nested=1
virsh edit AMD-L1
<cpu mode='host-passthrough' />
<cpu mode="custom" match="exact" check="none"> <model fallback="allow"> EPYC-IBPB </model> <feature policy="require" name="svm"/> </cpu>
cat /sys/module/kvm/parameters/nested
modprobe -r kvm
modprobe kvm nested=1
options kvm nested=1
cat /sys/module/kvm_hv/parameters/nested
modprobe -r kvm_hv
modprobe kvm_hv nested=1
options kvm_hv nested=1
<nested-hv state='on'/>