Chapter 7. Virtual machines 7.1. Creating virtual machines Use one of these procedures to create a virtual machine: Quick Start guided tour Running the wizard Pasting a pre-configured YAML file with the virtual machine wizard Using the CLI Importing a VMware virtual machine or template with the virtual machine wizard Warning Do not create virtual machines in openshift-* namespaces. Instead, create a new namespace or use an existing namespace without the openshift prefix. When you create virtual machines from the web console, select a virtual machine template that is configured with a boot source. Virtual machine templates with a boot source are labeled as Available boot source or they display a customized label text. Using templates with an available boot source expedites the process of creating virtual machines. Templates without a boot source are labeled as Boot source required . You can use these templates if you complete the steps for adding a boot source to the virtual machine . 7.1.1. Using a Quick Start to create a virtual machine The web console provides Quick Starts with instructional guided tours for creating virtual machines. You can access the Quick Starts catalog by selecting the Help menu in the Administrator perspective to view the Quick Starts catalog. When you click on a Quick Start tile and begin the tour, the system guides you through the process. Tasks in a Quick Start begin with selecting a Red Hat template. Then, you can add a boot source and import the operating system image. Finally, you can save the custom template and use it to create a virtual machine. Prerequisites Access to the website where you can download the URL link for the operating system image. Procedure In the web console, select Quick Starts from the Help menu. Click on a tile in the Quick Starts catalog. For example: Creating a Red Hat Linux Enterprise Linux virtual machine . Follow the instructions in the guided tour and complete the tasks for importing an operating system image and creating a virtual machine. The Virtual Machines tab displays the virtual machine. 7.1.2. Running the virtual machine wizard to create a virtual machine The web console features a wizard that guides you through the process of selecting a virtual machine template and creating a virtual machine. Red Hat virtual machine templates are preconfigured with an operating system image, default settings for the operating system, flavor (CPU and memory), and workload type (server). When templates are configured with a boot source, they are labeled with a customized label text or the default label text Available boot source . These templates are then ready to be used for creating virtual machines. You can select a template from the list of preconfigured templates, review the settings, and create a virtual machine in the Create virtual machine from template wizard. If you choose to customize your virtual machine, the wizard guides you through the General , Networking , Storage , Advanced , and Review steps. All required fields displayed by the wizard are marked by a *. Create network interface controllers (NICs) and storage disks later and attach them to virtual machines. Procedure Click Workloads Virtualization from the side menu. From the Virtual Machines tab or the Templates tab, click Create and select Virtual Machine with Wizard . Select a template that is configured with a boot source. Click to go to the Review and create step. Clear the Start this virtual machine after creation checkbox if you do not want to start the virtual machine now. Click Create virtual machine and exit the wizard or continue with the wizard to customize the virtual machine. Click Customize virtual machine to go to the General step. Optional: Edit the Name field to specify a custom name for the virtual machine. Optional: In the Description field, add a description. Click to go to the Networking step. A nic0 NIC is attached by default. Optional: Click Add Network Interface to create additional NICs. Optional: You can remove any or all NICs by clicking the Options menu and selecting Delete . A virtual machine does not need a NIC attached to be created. You can create NICs after the virtual machine has been created. Click to go to the Storage step. Optional: Click Add Disk to create additional disks. These disks can be removed by clicking the Options menu and selecting Delete . Optional: Click the Options menu to edit the disk and save your changes. Click to go to the Advanced step and review the details for Cloud-init . Click to go to the Review step and review the settings for the virtual machine. Click Create Virtual Machine . Click See virtual machine details to view the Overview for this virtual machine. The virtual machine is listed in the Virtual Machines tab. Refer to the virtual machine wizard fields section when running the web console wizard. 7.1.2.1. Virtual machine wizard fields Name Parameter Description Name The name can contain lowercase letters ( a-z ), numbers ( 0-9 ), and hyphens ( - ), up to a maximum of 253 characters. The first and last characters must be alphanumeric. The name must not contain uppercase letters, spaces, periods ( . ), or special characters. Description Optional description field. Operating System The primary operating system that is selected for the virtual machine in the template. You cannot edit this field when creating a virtual machine from a template. Boot Source Import via URL (creates PVC) Import content from an image available from an HTTP or HTTPS endpoint. Example: Obtaining a URL link from the web page with the operating system image. Clone existing PVC (creates PVC) Select an existent persistent volume claim available on the cluster and clone it. Import via Registry (creates PVC) Provision virtual machine from a bootable operating system container located in a registry accessible from the cluster. Example: kubevirt/cirros-registry-disk-demo . PXE (network boot - adds network interface) Boot an operating system from a server on the network. Requires a PXE bootable network attachment definition. Persistent Volume Claim project Project name that you want to use for cloning the PVC. Persistent Volume Claim name PVC name that should apply to this virtual machine template if you are cloning an existing PVC. Mount this as a CD-ROM boot source A CD-ROM requires an additional disk for installing the operating system. Select the checkbox to add a disk and customize it later. Flavor Tiny, Small, Medium, Large, Custom Presets that determine the amount of CPU and memory allocated to the virtual machine. The presets displayed for Flavor are determined by the operating system. Workload Type Desktop A virtual machine configuration for use on a desktop. Ideal for consumption on a small scale. Recommended for use with the web console. Server Balances performance and it is compatible with a wide range of server workloads. High-Performance A virtual machine configuration that is optimized for high-performance workloads. Start this virtual machine after creation. This checkbox is selected by default and the virtual machine starts running after creation. Clear the checkbox if you do not want the virtual machine to start when it is created. 7.1.2.2. Networking fields Name Description Name Name for the network interface controller. Model Indicates the model of the network interface controller. Supported values are e1000e and virtio . Network List of available network attachment definitions. Type List of available binding methods. For the default pod network, masquerade is the only recommended binding method. For secondary networks, use the bridge binding method. The masquerade method is not supported for non-default networks. Select SR-IOV if you configured an SR-IOV network device and defined that network in the namespace. MAC Address MAC address for the network interface controller. If a MAC address is not specified, one is assigned automatically. 7.1.2.3. Storage fields Name Selection Description Source Blank (creates PVC) Create an empty disk. Import via URL (creates PVC) Import content via URL (HTTP or HTTPS endpoint). Use an existing PVC Use a PVC that is already available in the cluster. Clone existing PVC (creates PVC) Select an existing PVC available in the cluster and clone it. Import via Registry (creates PVC) Import content via container registry. Container (ephemeral) Upload content from a container located in a registry accessible from the cluster. The container disk should be used only for read-only filesystems such as CD-ROMs or temporary virtual machines. Name Name of the disk. The name can contain lowercase letters ( a-z ), numbers ( 0-9 ), hyphens ( - ), and periods ( . ), up to a maximum of 253 characters. The first and last characters must be alphanumeric. The name must not contain uppercase letters, spaces, or special characters. Size Size of the disk in GiB. Type Type of disk. Example: Disk or CD-ROM Interface Type of disk device. Supported interfaces are virtIO , SATA , and SCSI . Storage Class The storage class that is used to create the disk. Advanced Volume Mode Defines whether the persistent volume uses a formatted file system or raw block state. Default is Filesystem . Advanced Access Mode Access mode of the persistent volume. Supported access modes are Single User (RWO) , Shared Access (RWX) , and Read Only (ROX) . Advanced storage settings The following advanced storage settings are available for Blank , Import via URL , and Clone existing PVC disks. These parameters are optional. If you do not specify these parameters, the system uses the default values from the kubevirt-storage-class-defaults config map. Name Parameter Description Volume Mode Filesystem Stores the virtual disk on a file system-based volume. Block Stores the virtual disk directly on the block volume. Only use Block if the underlying storage supports it. Access Mode Single User (RWO) The disk can be mounted as read/write by a single node. Shared Access (RWX) The disk can be mounted as read/write by many nodes. Note This is required for some features, such as live migration of virtual machines between nodes. Read Only (ROX) The disk can be mounted as read-only by many nodes. 7.1.2.4. Cloud-init fields Name Description Hostname Sets a specific hostname for the virtual machine. Authorized SSH Keys The user's public key that is copied to ~/.ssh/authorized_keys on the virtual machine. Custom script Replaces other options with a field in which you paste a custom cloud-init script. For more information on the kubevirt-storage-class-defaults config map, see Storage defaults for data volumes . 7.1.2.5. Pasting in a pre-configured YAML file to create a virtual machine Create a virtual machine by writing or pasting a YAML configuration file. A valid example virtual machine configuration is provided by default whenever you open the YAML edit screen. If your YAML configuration is invalid when you click Create , an error message indicates the parameter in which the error occurs. Only one error is shown at a time. Note Navigating away from the YAML screen while editing cancels any changes to the configuration you have made. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Click Create and select Virtual Machine With YAML . Write or paste your virtual machine configuration in the editable window. Alternatively, use the example virtual machine provided by default in the YAML screen. Optional: Click Download to download the YAML configuration file in its present state. Click Create to create the virtual machine. The virtual machine is listed in the Virtual Machines tab. 7.1.3. Using the CLI to create a virtual machine You can create a virtual machine from a virtualMachine manifest. Procedure Edit the VirtualMachine manifest for your VM. For example, the following manifest configures a Red Hat Enterprise Linux (RHEL) VM: Example 7.1. Example manifest for a RHEL VM 1 Specify the name of the virtual machine. 2 Specify the password for cloud-user. Create a virtual machine by using the manifest file: USD oc create -f <vm_manifest_file>.yaml Optional: Start the virtual machine: USD virtctl start <vm_name> 7.1.4. Virtual machine storage volume types Storage volume type Description ephemeral A local copy-on-write (COW) image that uses a network volume as a read-only backing store. The backing volume must be a PersistentVolumeClaim . The ephemeral image is created when the virtual machine starts and stores all writes locally. The ephemeral image is discarded when the virtual machine is stopped, restarted, or deleted. The backing volume (PVC) is not mutated in any way. persistentVolumeClaim Attaches an available PV to a virtual machine. Attaching a PV allows for the virtual machine data to persist between sessions. Importing an existing virtual machine disk into a PVC by using CDI and attaching the PVC to a virtual machine instance is the recommended method for importing existing virtual machines into OpenShift Container Platform. There are some requirements for the disk to be used within a PVC. dataVolume Data volumes build on the persistentVolumeClaim disk type by managing the process of preparing the virtual machine disk via an import, clone, or upload operation. VMs that use this volume type are guaranteed not to start until the volume is ready. Specify type: dataVolume or type: "" . If you specify any other value for type , such as persistentVolumeClaim , a warning is displayed, and the virtual machine does not start. cloudInitNoCloud Attaches a disk that contains the referenced cloud-init NoCloud data source, providing user data and metadata to the virtual machine. A cloud-init installation is required inside the virtual machine disk. containerDisk References an image, such as a virtual machine disk, that is stored in the container image registry. The image is pulled from the registry and attached to the virtual machine as a disk when the virtual machine is launched. A containerDisk volume is not limited to a single virtual machine and is useful for creating large numbers of virtual machine clones that do not require persistent storage. Only RAW and QCOW2 formats are supported disk types for the container image registry. QCOW2 is recommended for reduced image size. Note A containerDisk volume is ephemeral. It is discarded when the virtual machine is stopped, restarted, or deleted. A containerDisk volume is useful for read-only file systems such as CD-ROMs or for disposable virtual machines. emptyDisk Creates an additional sparse QCOW2 disk that is tied to the life-cycle of the virtual machine interface. The data survives guest-initiated reboots in the virtual machine but is discarded when the virtual machine stops or is restarted from the web console. The empty disk is used to store application dependencies and data that otherwise exceeds the limited temporary file system of an ephemeral disk. The disk capacity size must also be provided. 7.1.5. About RunStrategies for virtual machines A RunStrategy for virtual machines determines a virtual machine instance's (VMI) behavior, depending on a series of conditions. The spec.runStrategy setting exists in the virtual machine configuration process as an alternative to the spec.running setting. The spec.runStrategy setting allows greater flexibility for how VMIs are created and managed, in contrast to the spec.running setting with only true or false responses. However, the two settings are mutually exclusive. Only either spec.running or spec.runStrategy can be used. An error occurs if both are used. There are four defined RunStrategies. Always A VMI is always present when a virtual machine is created. A new VMI is created if the original stops for any reason, which is the same behavior as spec.running: true . RerunOnFailure A VMI is re-created if the instance fails due to an error. The instance is not re-created if the virtual machine stops successfully, such as when it shuts down. Manual The start , stop , and restart virtctl client commands can be used to control the VMI's state and existence. Halted No VMI is present when a virtual machine is created, which is the same behavior as spec.running: false . Different combinations of the start , stop and restart virtctl commands affect which RunStrategy is used. The following table follows a VM's transition from different states. The first column shows the VM's initial RunStrategy . Each additional column shows a virtctl command and the new RunStrategy after that command is run. Initial RunStrategy start stop restart Always - Halted Always RerunOnFailure - Halted RerunOnFailure Manual Manual Manual Manual Halted Always - - Note In OpenShift Virtualization clusters installed using installer-provisioned infrastructure, when a node fails the MachineHealthCheck and becomes unavailable to the cluster, VMs with a RunStrategy of Always or RerunOnFailure are rescheduled on a new node. apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine spec: RunStrategy: Always 1 template: ... 1 The VMI's current RunStrategy setting. 7.1.6. Additional resources The VirtualMachineSpec definition in the KubeVirt v0.36.5 API Reference provides broader context for the parameters and hierarchy of the virtual machine specification. Note The KubeVirt API Reference is the upstream project reference and might contain parameters that are not supported in OpenShift Virtualization. Prepare a container disk before adding it to a virtual machine as a containerDisk volume. See Deploying machine health checks for further details on deploying and enabling machine health checks. See Installer-provisioned infrastructure overview for further details on installer-provisioned infrastructure. See Configuring the SR-IOV Network Operator for further details on the SR-IOV Network Operator. 7.2. Editing virtual machines You can update a virtual machine configuration using either the YAML editor in the web console or the OpenShift client on the command line. You can also update a subset of the parameters in the Virtual Machine Details screen. 7.2.1. Editing a virtual machine in the web console Edit select values of a virtual machine in the web console by clicking the pencil icon to the relevant field. Other values can be edited using the CLI. Labels and annotations are editable for both preconfigured Red Hat templates and your custom virtual machine templates. All other values are editable only for custom virtual machine templates that users have created using the Red Hat templates or the Create Virtual Machine Template wizard. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine. Click the Details tab. Click the pencil icon to make a field editable. Make the relevant changes and click Save . Note If the virtual machine is running, changes to Boot Order or Flavor will not take effect until you restart the virtual machine. You can view pending changes by clicking View Pending Changes on the right side of the relevant field. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts. 7.2.2. Editing a virtual machine YAML configuration using the web console You can edit the YAML configuration of a virtual machine in the web console. Some parameters cannot be modified. If you click Save with an invalid configuration, an error message indicates the parameter that cannot be changed. If you edit the YAML configuration while the virtual machine is running, changes will not take effect until you restart the virtual machine. Note Navigating away from the YAML screen while editing cancels any changes to the configuration you have made. Procedure Click Workloads Virtualization from the side menu. Select a virtual machine. Click the YAML tab to display the editable configuration. Optional: You can click Download to download the YAML file locally in its current state. Edit the file and click Save . A confirmation message shows that the modification has been successful and includes the updated version number for the object. 7.2.3. Editing a virtual machine YAML configuration using the CLI Use this procedure to edit a virtual machine YAML configuration using the CLI. Prerequisites You configured a virtual machine with a YAML object configuration file. You installed the oc CLI. Procedure Run the following command to update the virtual machine configuration: USD oc edit <object_type> <object_ID> Open the object configuration. Edit the YAML. If you edit a running virtual machine, you need to do one of the following: Restart the virtual machine. Run the following command for the new configuration to take effect: USD oc apply <object_type> <object_ID> 7.2.4. Adding a virtual disk to a virtual machine Use this procedure to add a virtual disk to a virtual machine. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview screen. Click the Disks tab. In the Add Disk window, specify the Source , Name , Size , Type , Interface , and Storage Class . Optional: In the Advanced list, specify the Volume Mode and Access Mode for the virtual disk. If you do not specify these parameters, the system uses the default values from the kubevirt-storage-class-defaults config map. Click Add . Note If the virtual machine is running, the new disk is in the pending restart state and will not be attached until you restart the virtual machine. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts. For more information on the kubevirt-storage-class-defaults config map, see Storage defaults for data volumes . 7.2.4.1. Storage fields Name Selection Description Source Blank (creates PVC) Create an empty disk. Import via URL (creates PVC) Import content via URL (HTTP or HTTPS endpoint). Use an existing PVC Use a PVC that is already available in the cluster. Clone existing PVC (creates PVC) Select an existing PVC available in the cluster and clone it. Import via Registry (creates PVC) Import content via container registry. Container (ephemeral) Upload content from a container located in a registry accessible from the cluster. The container disk should be used only for read-only filesystems such as CD-ROMs or temporary virtual machines. Name Name of the disk. The name can contain lowercase letters ( a-z ), numbers ( 0-9 ), hyphens ( - ), and periods ( . ), up to a maximum of 253 characters. The first and last characters must be alphanumeric. The name must not contain uppercase letters, spaces, or special characters. Size Size of the disk in GiB. Type Type of disk. Example: Disk or CD-ROM Interface Type of disk device. Supported interfaces are virtIO , SATA , and SCSI . Storage Class The storage class that is used to create the disk. Advanced Volume Mode Defines whether the persistent volume uses a formatted file system or raw block state. Default is Filesystem . Advanced Access Mode Access mode of the persistent volume. Supported access modes are Single User (RWO) , Shared Access (RWX) , and Read Only (ROX) . Advanced storage settings The following advanced storage settings are available for Blank , Import via URL , and Clone existing PVC disks. These parameters are optional. If you do not specify these parameters, the system uses the default values from the kubevirt-storage-class-defaults config map. Name Parameter Description Volume Mode Filesystem Stores the virtual disk on a file system-based volume. Block Stores the virtual disk directly on the block volume. Only use Block if the underlying storage supports it. Access Mode Single User (RWO) The disk can be mounted as read/write by a single node. Shared Access (RWX) The disk can be mounted as read/write by many nodes. Note This is required for some features, such as live migration of virtual machines between nodes. Read Only (ROX) The disk can be mounted as read-only by many nodes. 7.2.5. Adding a network interface to a virtual machine Use this procedure to add a network interface to a virtual machine. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview screen. Click the Network Interfaces tab. Click Add Network Interface . In the Add Network Interface window, specify the Name , Model , Network , Type , and MAC Address of the network interface. Click Add . Note If the virtual machine is running, the new network interface is in the pending restart state and changes will not take effect until you restart the virtual machine. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts. 7.2.5.1. Networking fields Name Description Name Name for the network interface controller. Model Indicates the model of the network interface controller. Supported values are e1000e and virtio . Network List of available network attachment definitions. Type List of available binding methods. For the default pod network, masquerade is the only recommended binding method. For secondary networks, use the bridge binding method. The masquerade method is not supported for non-default networks. Select SR-IOV if you configured an SR-IOV network device and defined that network in the namespace. MAC Address MAC address for the network interface controller. If a MAC address is not specified, one is assigned automatically. 7.2.6. Editing CD-ROMs for Virtual Machines Use the following procedure to edit CD-ROMs for virtual machines. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview screen. Click the Disks tab. Click the Options menu for the CD-ROM that you want to edit and select Edit . In the Edit CD-ROM window, edit the fields: Source , Persistent Volume Claim , Name , Type , and Interface . Click Save . 7.3. Editing boot order You can update the values for a boot order list by using the web console or the CLI. With Boot Order in the Virtual Machine Overview page, you can: Select a disk or network interface controller (NIC) and add it to the boot order list. Edit the order of the disks or NICs in the boot order list. Remove a disk or NIC from the boot order list, and return it back to the inventory of bootable sources. 7.3.1. Adding items to a boot order list in the web console Add items to a boot order list by using the web console. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview screen. Click the Details tab. Click the pencil icon that is located on the right side of Boot Order . If a YAML configuration does not exist, or if this is the first time that you are creating a boot order list, the following message displays: No resource selected. VM will attempt to boot from disks by order of appearance in YAML file. Click Add Source and select a bootable disk or network interface controller (NIC) for the virtual machine. Add any additional disks or NICs to the boot order list. Click Save . Note If the virtual machine is running, changes to Boot Order will not take effect until you restart the virtual machine. You can view pending changes by clicking View Pending Changes on the right side of the Boot Order field. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts. 7.3.2. Editing a boot order list in the web console Edit the boot order list in the web console. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview screen. Click the Details tab. Click the pencil icon that is located on the right side of Boot Order . Choose the appropriate method to move the item in the boot order list: If you do not use a screen reader, hover over the arrow icon to the item that you want to move, drag the item up or down, and drop it in a location of your choice. If you use a screen reader, press the Up Arrow key or Down Arrow key to move the item in the boot order list. Then, press the Tab key to drop the item in a location of your choice. Click Save . Note If the virtual machine is running, changes to the boot order list will not take effect until you restart the virtual machine. You can view pending changes by clicking View Pending Changes on the right side of the Boot Order field. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts. 7.3.3. Editing a boot order list in the YAML configuration file Edit the boot order list in a YAML configuration file by using the CLI. Procedure Open the YAML configuration file for the virtual machine by running the following command: USD oc edit vm example Edit the YAML file and modify the values for the boot order associated with a disk or network interface controller (NIC). For example: disks: - bootOrder: 1 1 disk: bus: virtio name: containerdisk - disk: bus: virtio name: cloudinitdisk - cdrom: bus: virtio name: cd-drive-1 interfaces: - boot Order: 2 2 macAddress: '02:96:c4:00:00' masquerade: {} name: default 1 The boot order value specified for the disk. 2 The boot order value specified for the network interface controller. Save the YAML file. Click reload the content to apply the updated boot order values from the YAML file to the boot order list in the web console. 7.3.4. Removing items from a boot order list in the web console Remove items from a boot order list by using the web console. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview screen. Click the Details tab. Click the pencil icon that is located on the right side of Boot Order . Click the Remove icon to the item. The item is removed from the boot order list and saved in the list of available boot sources. If you remove all items from the boot order list, the following message displays: No resource selected. VM will attempt to boot from disks by order of appearance in YAML file. Note If the virtual machine is running, changes to Boot Order will not take effect until you restart the virtual machine. You can view pending changes by clicking View Pending Changes on the right side of the Boot Order field. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts. 7.4. Deleting virtual machines You can delete a virtual machine from the web console or by using the oc command line interface. 7.4.1. Deleting a virtual machine using the web console Deleting a virtual machine permanently removes it from the cluster. Note When you delete a virtual machine, the data volume it uses is automatically deleted. Procedure In the OpenShift Virtualization console, click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Click the Options menu of the virtual machine that you want to delete and select Delete Virtual Machine . Alternatively, click the virtual machine name to open the Virtual Machine Overview screen and click Actions Delete Virtual Machine . In the confirmation pop-up window, click Delete to permanently delete the virtual machine. 7.4.2. Deleting a virtual machine by using the CLI You can delete a virtual machine by using the oc command line interface (CLI). The oc client enables you to perform actions on multiple virtual machines. Note When you delete a virtual machine, the data volume it uses is automatically deleted. Prerequisites Identify the name of the virtual machine that you want to delete. Procedure Delete the virtual machine by running the following command: USD oc delete vm <vm_name> Note This command only deletes objects that exist in the current project. Specify the -n <project_name> option if the object you want to delete is in a different project or namespace. 7.5. Managing virtual machine instances If you have standalone virtual machine instances (VMIs) that were created independently outside of the OpenShift Virtualization environment, you can manage them by using the web console or the command-line interface (CLI). 7.5.1. About virtual machine instances A virtual machine instance (VMI) is a representation of a running virtual machine (VM). When a VMI is owned by a VM or by another object, you manage it through its owner in the web console or by using the oc command-line interface (CLI). A standalone VMI is created and started independently with a script, through automation, or by using other methods in the CLI. In your environment, you might have standalone VMIs that were developed and started outside of the OpenShift Virtualization environment. You can continue to manage those standalone VMIs by using the CLI. You can also use the web console for specific tasks associated with standalone VMIs: List standalone VMIs and their details. Edit labels and annotations for a standalone VMI. Delete a standalone VMI. When you delete a VM, the associated VMI is automatically deleted. You delete a standalone VMI directly because it is not owned by VMs or other objects. Note Before you uninstall OpenShift Virtualization, list and view the standalone VMIs by using the CLI or the web console. Then, delete any outstanding VMIs. 7.5.2. Listing all virtual machine instances using the CLI You can list all virtual machine instances (VMIs) in your cluster, including standalone VMIs and those owned by virtual machines, by using the oc command-line interface (CLI). Procedure List all VMIs by running the following command: USD oc get vmis 7.5.3. Listing standalone virtual machine instances using the web console Using the web console, you can list and view standalone virtual machine instances (VMIs) in your cluster that are not owned by virtual machines (VMs). Note VMIs that are owned by VMs or other objects are not displayed in the web console. The web console displays only standalone VMIs. If you want to list all VMIs in your cluster, you must use the CLI. Procedure Click Workloads Virtualization from the side menu. A list of VMs and standalone VMIs displays. You can identify standalone VMIs by the dark colored badges that display to the virtual machine instance names. 7.5.4. Editing a standalone virtual machine instance using the web console You can edit annotations and labels for a standalone virtual machine instance (VMI) using the web console. Other items displayed in the Details page for a standalone VMI are not editable. Procedure Click Workloads Virtualization from the side menu. A list of virtual machines (VMs) and standalone VMIs displays. Click the name of a standalone VMI to open the Virtual Machine Instance Overview screen. Click the Details tab. Click the pencil icon that is located on the right side of Annotations . Make the relevant changes and click Save . Note To edit labels for a standalone VMI, click Actions and select Edit Labels. Make the relevant changes and click Save . 7.5.5. Deleting a standalone virtual machine instance using the CLI You can delete a standalone virtual machine instance (VMI) by using the oc command-line interface (CLI). Prerequisites Identify the name of the VMI that you want to delete. Procedure Delete the VMI by running the following command: USD oc delete vmi <vmi_name> 7.5.6. Deleting a standalone virtual machine instance using the web console Delete a standalone virtual machine instance (VMI) from the web console. Procedure In the OpenShift Container Platform web console, click Workloads Virtualization from the side menu. Click the ... button of the standalone virtual machine instance (VMI) that you want to delete and select Delete Virtual Machine Instance . Alternatively, click the name of the standalone VMI. The Virtual Machine Instance Overview page displays. Select Actions Delete Virtual Machine Instance . In the confirmation pop-up window, click Delete to permanently delete the standalone VMI. 7.6. Controlling virtual machine states You can stop, start, restart, and unpause virtual machines from the web console. Note To control virtual machines from the command-line interface (CLI), use the virtctl client . 7.6.1. Starting a virtual machine You can start a virtual machine from the web console. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Find the row that contains the virtual machine that you want to start. Navigate to the appropriate menu for your use case: To stay on this page, where you can perform actions on multiple virtual machines: Click the Options menu located at the far right end of the row. To view comprehensive information about the selected virtual machine before you start it: Access the Virtual Machine Overview screen by clicking the name of the virtual machine. Click Actions . Select Start Virtual Machine . In the confirmation window, click Start to start the virtual machine. Note When you start virtual machine that is provisioned from a URL source for the first time, the virtual machine has a status of Importing while OpenShift Virtualization imports the container from the URL endpoint. Depending on the size of the image, this process might take several minutes. 7.6.2. Restarting a virtual machine You can restart a running virtual machine from the web console. Important To avoid errors, do not restart a virtual machine while it has a status of Importing . Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Find the row that contains the virtual machine that you want to restart. Navigate to the appropriate menu for your use case: To stay on this page, where you can perform actions on multiple virtual machines: Click the Options menu located at the far right end of the row. To view comprehensive information about the selected virtual machine before you restart it: Access the Virtual Machine Overview screen by clicking the name of the virtual machine. Click Actions . Select Restart Virtual Machine . In the confirmation window, click Restart to restart the virtual machine. 7.6.3. Stopping a virtual machine You can stop a virtual machine from the web console. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Find the row that contains the virtual machine that you want to stop. Navigate to the appropriate menu for your use case: To stay on this page, where you can perform actions on multiple virtual machines: Click the Options menu located at the far right end of the row. To view comprehensive information about the selected virtual machine before you stop it: Access the Virtual Machine Overview screen by clicking the name of the virtual machine. Click Actions . Select Stop Virtual Machine . In the confirmation window, click Stop to stop the virtual machine. 7.6.4. Unpausing a virtual machine You can unpause a paused virtual machine from the web console. Prerequisites At least one of your virtual machines must have a status of Paused . Note You can pause virtual machines by using the virtctl client. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Find the row that contains the virtual machine that you want to unpause. Navigate to the appropriate menu for your use case: To stay on this page, where you can perform actions on multiple virtual machines: In the Status column, click Paused . To view comprehensive information about the selected virtual machine before you unpause it: Access the Virtual Machine Overview screen by clicking the name of the virtual machine. Click the pencil icon that is located on the right side of Status . In the confirmation window, click Unpause to unpause the virtual machine. 7.7. Accessing virtual machine consoles OpenShift Virtualization provides different virtual machine consoles that you can use to accomplish different product tasks. You can access these consoles through the OpenShift Container Platform web console and by using CLI commands. 7.7.1. Accessing virtual machine consoles in the OpenShift Container Platform web console You can connect to virtual machines by using the serial console or the VNC console in the OpenShift Container Platform web console. You can connect to Windows virtual machines by using the desktop viewer console, which uses RDP (remote desktop protocol), in the OpenShift Container Platform web console. 7.7.1.1. Connecting to the serial console Connect to the serial console of a running virtual machine from the Console tab in the Virtual Machine Overview screen of the web console. Procedure In the OpenShift Virtualization console, click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview page. Click Console . The VNC console opens by default. Select Disconnect before switching to ensure that only one console session is open at a time. Otherwise, the VNC console session remains active in the background. Click the VNC Console drop-down list and select Serial Console . Click Disconnect to end the console session. Optional: Open the serial console in a separate window by clicking Open Console in New Window . 7.7.1.2. Connecting to the VNC console Connect to the VNC console of a running virtual machine from the Console tab in the Virtual Machine Overview screen of the web console. Procedure In the OpenShift Virtualization console, click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview page. Click the Console tab. The VNC console opens by default. Optional: Open the VNC console in a separate window by clicking Open Console in New Window . Optional: Send key combinations to the virtual machine by clicking Send Key . 7.7.1.3. Connecting to a Windows virtual machine with RDP The desktop viewer console, which utilizes the Remote Desktop Protocol (RDP), provides a better console experience for connecting to Windows virtual machines. To connect to a Windows virtual machine with RDP, download the console.rdp file for the virtual machine from the Consoles tab in the Virtual Machine Details screen of the web console and supply it to your preferred RDP client. Prerequisites A running Windows virtual machine with the QEMU guest agent installed. The qemu-guest-agent is included in the VirtIO drivers. A layer-2 NIC attached to the virtual machine. An RDP client installed on a machine on the same network as the Windows virtual machine. Procedure In the OpenShift Virtualization console, click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a Windows virtual machine to open the Virtual Machine Overview screen. Click the Console tab. In the Console list, select Desktop Viewer . In the Network Interface list, select the layer-2 NIC. Click Launch Remote Desktop to download the console.rdp file. Open an RDP client and reference the console.rdp file. For example, using remmina : USD remmina --connect /path/to/console.rdp Enter the Administrator user name and password to connect to the Windows virtual machine. 7.7.1.4. Copying the SSH command from the web console Copy the command to access a running virtual machine (VM) via SSH from the Actions list in the web console. Procedure In the OpenShift Container Platform console, click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview page. From the Actions list, select Copy SSH Command . You can now paste this command onto the OpenShift CLI ( oc ). 7.7.2. Accessing virtual machine consoles by using CLI commands 7.7.2.1. Accessing a virtual machine instance via SSH You can use SSH to access a virtual machine (VM) after you expose port 22 on it. The virtctl expose command forwards a virtual machine instance (VMI) port to a node port and creates a service for enabled access. The following example creates the fedora-vm-ssh service that forwards traffic from a specific port of cluster nodes to port 22 of the <fedora-vm> virtual machine. Prerequisites You must be in the same project as the VMI. The VMI you want to access must be connected to the default pod network by using the masquerade binding method. The VMI you want to access must be running. Install the OpenShift CLI ( oc ). Procedure Run the following command to create the fedora-vm-ssh service: 1 <fedora-vm> is the name of the VM that you run the fedora-vm-ssh service on. Check the service to find out which port the service acquired: USD oc get svc Example output NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE fedora-vm-ssh NodePort 127.0.0.1 <none> 22:32551/TCP 6s + In this example, the service acquired the 32551 port. Log in to the VMI via SSH. Use the ipAddress of any of the cluster nodes and the port that you found in the step: USD ssh username@<node_IP_address> -p 32551 7.7.2.2. Accessing the serial console of a virtual machine instance The virtctl console command opens a serial console to the specified virtual machine instance. Prerequisites The virt-viewer package must be installed. The virtual machine instance you want to access must be running. Procedure Connect to the serial console with virtctl : USD virtctl console <VMI> 7.7.2.3. Accessing the graphical console of a virtual machine instances with VNC The virtctl client utility can use the remote-viewer function to open a graphical console to a running virtual machine instance. This capability is included in the virt-viewer package. Prerequisites The virt-viewer package must be installed. The virtual machine instance you want to access must be running. Note If you use virtctl via SSH on a remote machine, you must forward the X session to your machine. Procedure Connect to the graphical interface with the virtctl utility: USD virtctl vnc <VMI> If the command failed, try using the -v flag to collect troubleshooting information: USD virtctl vnc <VMI> -v 4 7.7.2.4. Connecting to a Windows virtual machine with an RDP console The Remote Desktop Protocol (RDP) provides a better console experience for connecting to Windows virtual machines. To connect to a Windows virtual machine with RDP, specify the IP address of the attached L2 NIC to your RDP client. Prerequisites A running Windows virtual machine with the QEMU guest agent installed. The qemu-guest-agent is included in the VirtIO drivers. A layer 2 NIC attached to the virtual machine. An RDP client installed on a machine on the same network as the Windows virtual machine. Procedure Log in to the OpenShift Virtualization cluster through the oc CLI tool as a user with an access token. USD oc login -u <user> https://<cluster.example.com>:8443 Use oc describe vmi to display the configuration of the running Windows virtual machine. USD oc describe vmi <windows-vmi-name> Example output ... spec: networks: - name: default pod: {} - multus: networkName: cnv-bridge name: bridge-net ... status: interfaces: - interfaceName: eth0 ipAddress: 198.51.100.0/24 ipAddresses: 198.51.100.0/24 mac: a0:36:9f:0f:b1:70 name: default - interfaceName: eth1 ipAddress: 192.0.2.0/24 ipAddresses: 192.0.2.0/24 2001:db8::/32 mac: 00:17:a4:77:77:25 name: bridge-net ... Identify and copy the IP address of the layer 2 network interface. This is 192.0.2.0 in the above example, or 2001:db8:: if you prefer IPv6. Open an RDP client and use the IP address copied in the step for the connection. Enter the Administrator user name and password to connect to the Windows virtual machine. 7.8. Triggering virtual machine failover by resolving a failed node If a node fails and machine health checks are not deployed on your cluster, virtual machines (VMs) with RunStrategy: Always configured are not automatically relocated to healthy nodes. To trigger VM failover, you must manually delete the Node object. Note If you installed your cluster by using installer-provisioned infrastructure and you properly configured machine health checks: Failed nodes are automatically recycled. Virtual machines with RunStrategy set to Always or RerunOnFailure are automatically scheduled on healthy nodes. 7.8.1. Prerequisites A node where a virtual machine was running has the NotReady condition . The virtual machine that was running on the failed node has RunStrategy set to Always . You have installed the OpenShift CLI ( oc ). 7.8.2. Deleting nodes from a bare metal cluster When you delete a node using the CLI, the node object is deleted in Kubernetes, but the pods that exist on the node are not deleted. Any bare pods not backed by a replication controller become inaccessible to OpenShift Container Platform. Pods backed by replication controllers are rescheduled to other available nodes. You must delete local manifest pods. Procedure Delete a node from an OpenShift Container Platform cluster running on bare metal by completing the following steps: Mark the node as unschedulable: USD oc adm cordon <node_name> Drain all pods on the node: USD oc adm drain <node_name> --force=true This step might fail if the node is offline or unresponsive. Even if the node does not respond, it might still be running a workload that writes to shared storage. To avoid data corruption, power down the physical hardware before you proceed. Delete the node from the cluster: USD oc delete node <node_name> Although the node object is now deleted from the cluster, it can still rejoin the cluster after reboot or if the kubelet service is restarted. To permanently delete the node and all its data, you must decommission the node . If you powered down the physical hardware, turn it back on so that the node can rejoin the cluster. 7.8.3. Verifying virtual machine failover After all resources are terminated on the unhealthy node, a new virtual machine instance (VMI) is automatically created on a healthy node for each relocated VM. To confirm that the VMI was created, view all VMIs by using the oc CLI. 7.8.3.1. Listing all virtual machine instances using the CLI You can list all virtual machine instances (VMIs) in your cluster, including standalone VMIs and those owned by virtual machines, by using the oc command-line interface (CLI). Procedure List all VMIs by running the following command: USD oc get vmis 7.9. Installing the QEMU guest agent on virtual machines The QEMU guest agent is a daemon that runs on the virtual machine and passes information to the host about the virtual machine, users, file systems, and secondary networks. 7.9.1. Installing QEMU guest agent on a Linux virtual machine The qemu-guest-agent is widely available and available by default in Red Hat virtual machines. Install the agent and start the service Procedure Access the virtual machine command line through one of the consoles or by SSH. Install the QEMU guest agent on the virtual machine: USD yum install -y qemu-guest-agent Ensure the service is persistent and start it: USD systemctl enable --now qemu-guest-agent You can also install and start the QEMU guest agent by using the custom script field in the cloud-init section of the wizard when creating either virtual machines or virtual machines templates in the web console. 7.9.2. Installing QEMU guest agent on a Windows virtual machine For Windows virtual machines, the QEMU guest agent is included in the VirtIO drivers, which can be installed using one of the following procedures: 7.9.2.1. Installing VirtIO drivers on an existing Windows virtual machine Install the VirtIO drivers from the attached SATA CD drive to an existing Windows virtual machine. Note This procedure uses a generic approach to adding drivers to Windows. The process might differ slightly between versions of Windows. Refer to the installation documentation for your version of Windows for specific installation steps. Procedure Start the virtual machine and connect to a graphical console. Log in to a Windows user session. Open Device Manager and expand Other devices to list any Unknown device . Open the Device Properties to identify the unknown device. Right-click the device and select Properties . Click the Details tab and select Hardware Ids in the Property list. Compare the Value for the Hardware Ids with the supported VirtIO drivers. Right-click the device and select Update Driver Software . Click Browse my computer for driver software and browse to the attached SATA CD drive, where the VirtIO drivers are located. The drivers are arranged hierarchically according to their driver type, operating system, and CPU architecture. Click to install the driver. Repeat this process for all the necessary VirtIO drivers. After the driver installs, click Close to close the window. Reboot the virtual machine to complete the driver installation. 7.9.2.2. Installing VirtIO drivers during Windows installation Install the VirtIO drivers from the attached SATA CD driver during Windows installation. Note This procedure uses a generic approach to the Windows installation and the installation method might differ between versions of Windows. Refer to the documentation for the version of Windows that you are installing. Procedure Start the virtual machine and connect to a graphical console. Begin the Windows installation process. Select the Advanced installation. The storage destination will not be recognized until the driver is loaded. Click Load driver . The drivers are attached as a SATA CD drive. Click OK and browse the CD drive for the storage driver to load. The drivers are arranged hierarchically according to their driver type, operating system, and CPU architecture. Repeat the two steps for all required drivers. Complete the Windows installation. 7.10. Viewing the QEMU guest agent information for virtual machines When the QEMU guest agent runs on the virtual machine, you can use the web console to view information about the virtual machine, users, file systems, and secondary networks. 7.10.1. Prerequisites Install the QEMU guest agent on the virtual machine. 7.10.2. About the QEMU guest agent information in the web console When the QEMU guest agent is installed, the Details pane within the Virtual Machine Overview tab and the Details tab display information about the hostname, operating system, time zone, and logged in users. The Virtual Machine Overview shows information about the guest operating system installed on the virtual machine. The Details tab displays a table with information for logged in users. The Disks tab displays a table with information for file systems. Note If the QEMU guest agent is not installed, the Virtual Machine Overview tab and the Details tab display information about the operating system that was specified when the virtual machine was created. 7.10.3. Viewing the QEMU guest agent information in the web console You can use the web console to view information for virtual machines that is passed by the QEMU guest agent to the host. Procedure Click Workloads Virtual Machines from the side menu. Click the Virtual Machines tab. Select a virtual machine name to open the Virtual Machine Overview screen and view the Details pane. Click Logged in users to view the Details tab that shows information for users. Click the Disks tab to view information about the file systems. 7.11. Managing config maps, secrets, and service accounts in virtual machines You can use secrets, config maps, and service accounts to pass configuration data to virtual machines. For example, you can: Give a virtual machine access to a service that requires credentials by adding a secret to the virtual machine. Store non-confidential configuration data in a config map so that a pod or another object can consume the data. Allow a component to access the API server by associating a service account with that component. Note OpenShift Virtualization exposes secrets, config maps, and service accounts as virtual machine disks so that you can use them across platforms without additional overhead. 7.11.1. Adding a secret, config map, or service account to a virtual machine Add a secret, config map, or service account to a virtual machine by using the OpenShift Container Platform web console. Prerequisites The secret, config map, or service account that you want to add must exist in the same namespace as the target virtual machine. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview screen. Click the Environment tab. Click Select a resource and select a secret, config map, or service account from the list. A six character serial number is automatically generated for the selected resource. Click Save . Optional. Add another object by clicking Add Config Map, Secret or Service Account . Note You can reset the form to the last saved state by clicking Reload . The Environment resources are added to the virtual machine as disks. You can mount the secret, config map, or service account as you would mount any other disk. If the virtual machine is running, changes will not take effect until you restart the virtual machine. The newly added resources are marked as pending changes for both the Environment and Disks tab in the Pending Changes banner at the top of the page. Verification From the Virtual Machine Overview page, click the Disks tab. Check to ensure that the secret, config map, or service account is included in the list of disks. Optional. Choose the appropriate method to apply your changes: If the virtual machine is running, restart the virtual machine by clicking Actions Restart Virtual Machine . If the virtual machine is stopped, start the virtual machine by clicking Actions Start Virtual Machine . You can now mount the secret, config map, or service account as you would mount any other disk. 7.11.2. Removing a secret, config map, or service account from a virtual machine Remove a secret, config map, or service account from a virtual machine by using the OpenShift Container Platform web console. Prerequisites You must have at least one secret, config map, or service account that is attached to a virtual machine. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview screen. Click the Environment tab. Find the item that you want to delete in the list, and click Remove on the right side of the item. Click Save . Note You can reset the form to the last saved state by clicking Reload . Verification From the Virtual Machine Overview page, click the Disks tab. Check to ensure that the secret, config map, or service account that you removed is no longer included in the list of disks. 7.11.3. Additional resources Providing sensitive data to pods Understanding and creating service accounts Understanding config maps 7.12. Installing VirtIO driver on an existing Windows virtual machine 7.12.1. Understanding VirtIO drivers VirtIO drivers are paravirtualized device drivers required for Microsoft Windows virtual machines to run in OpenShift Virtualization. The supported drivers are available in the container-native-virtualization/virtio-win container disk of the Red Hat Ecosystem Catalog . The container-native-virtualization/virtio-win container disk must be attached to the virtual machine as a SATA CD drive to enable driver installation. You can install VirtIO drivers during Windows installation on the virtual machine or added to an existing Windows installation. After the drivers are installed, the container-native-virtualization/virtio-win container disk can be removed from the virtual machine. See also: Installing Virtio drivers on a new Windows virtual machine . 7.12.2. Supported VirtIO drivers for Microsoft Windows virtual machines Table 7.1. Supported drivers Driver name Hardware ID Description viostor VEN_1AF4&DEV_1001 VEN_1AF4&DEV_1042 The block driver. Sometimes displays as an SCSI Controller in the Other devices group. viorng VEN_1AF4&DEV_1005 VEN_1AF4&DEV_1044 The entropy source driver. Sometimes displays as a PCI Device in the Other devices group. NetKVM VEN_1AF4&DEV_1000 VEN_1AF4&DEV_1041 The network driver. Sometimes displays as an Ethernet Controller in the Other devices group. Available only if a VirtIO NIC is configured. 7.12.3. Adding VirtIO drivers container disk to a virtual machine OpenShift Virtualization distributes VirtIO drivers for Microsoft Windows as a container disk, which is available from the Red Hat Ecosystem Catalog . To install these drivers to a Windows virtual machine, attach the container-native-virtualization/virtio-win container disk to the virtual machine as a SATA CD drive in the virtual machine configuration file. Prerequisites Download the container-native-virtualization/virtio-win container disk from the Red Hat Ecosystem Catalog . This is not mandatory, because the container disk will be downloaded from the Red Hat registry if it not already present in the cluster, but it can reduce installation time. Procedure Add the container-native-virtualization/virtio-win container disk as a cdrom disk in the Windows virtual machine configuration file. The container disk will be downloaded from the registry if it is not already present in the cluster. spec: domain: devices: disks: - name: virtiocontainerdisk bootOrder: 2 1 cdrom: bus: sata volumes: - containerDisk: image: container-native-virtualization/virtio-win name: virtiocontainerdisk 1 OpenShift Virtualization boots virtual machine disks in the order defined in the VirtualMachine configuration file. You can either define other disks for the virtual machine before the container-native-virtualization/virtio-win container disk or use the optional bootOrder parameter to ensure the virtual machine boots from the correct disk. If you specify the bootOrder for a disk, it must be specified for all disks in the configuration. The disk is available once the virtual machine has started: If you add the container disk to a running virtual machine, use oc apply -f <vm.yaml> in the CLI or reboot the virtual machine for the changes to take effect. If the virtual machine is not running, use virtctl start <vm> . After the virtual machine has started, the VirtIO drivers can be installed from the attached SATA CD drive. 7.12.4. Installing VirtIO drivers on an existing Windows virtual machine Install the VirtIO drivers from the attached SATA CD drive to an existing Windows virtual machine. Note This procedure uses a generic approach to adding drivers to Windows. The process might differ slightly between versions of Windows. Refer to the installation documentation for your version of Windows for specific installation steps. Procedure Start the virtual machine and connect to a graphical console. Log in to a Windows user session. Open Device Manager and expand Other devices to list any Unknown device . Open the Device Properties to identify the unknown device. Right-click the device and select Properties . Click the Details tab and select Hardware Ids in the Property list. Compare the Value for the Hardware Ids with the supported VirtIO drivers. Right-click the device and select Update Driver Software . Click Browse my computer for driver software and browse to the attached SATA CD drive, where the VirtIO drivers are located. The drivers are arranged hierarchically according to their driver type, operating system, and CPU architecture. Click to install the driver. Repeat this process for all the necessary VirtIO drivers. After the driver installs, click Close to close the window. Reboot the virtual machine to complete the driver installation. 7.12.5. Removing the VirtIO container disk from a virtual machine After installing all required VirtIO drivers to the virtual machine, the container-native-virtualization/virtio-win container disk no longer needs to be attached to the virtual machine. Remove the container-native-virtualization/virtio-win container disk from the virtual machine configuration file. Procedure Edit the configuration file and remove the disk and the volume . USD oc edit vm <vm-name> spec: domain: devices: disks: - name: virtiocontainerdisk bootOrder: 2 cdrom: bus: sata volumes: - containerDisk: image: container-native-virtualization/virtio-win name: virtiocontainerdisk Reboot the virtual machine for the changes to take effect. 7.13. Installing VirtIO driver on a new Windows virtual machine 7.13.1. Prerequisites Windows installation media accessible by the virtual machine, such as importing an ISO into a data volume and attaching it to the virtual machine. 7.13.2. Understanding VirtIO drivers VirtIO drivers are paravirtualized device drivers required for Microsoft Windows virtual machines to run in OpenShift Virtualization. The supported drivers are available in the container-native-virtualization/virtio-win container disk of the Red Hat Ecosystem Catalog . The container-native-virtualization/virtio-win container disk must be attached to the virtual machine as a SATA CD drive to enable driver installation. You can install VirtIO drivers during Windows installation on the virtual machine or added to an existing Windows installation. After the drivers are installed, the container-native-virtualization/virtio-win container disk can be removed from the virtual machine. See also: Installing VirtIO driver on an existing Windows virtual machine . 7.13.3. Supported VirtIO drivers for Microsoft Windows virtual machines Table 7.2. Supported drivers Driver name Hardware ID Description viostor VEN_1AF4&DEV_1001 VEN_1AF4&DEV_1042 The block driver. Sometimes displays as an SCSI Controller in the Other devices group. viorng VEN_1AF4&DEV_1005 VEN_1AF4&DEV_1044 The entropy source driver. Sometimes displays as a PCI Device in the Other devices group. NetKVM VEN_1AF4&DEV_1000 VEN_1AF4&DEV_1041 The network driver. Sometimes displays as an Ethernet Controller in the Other devices group. Available only if a VirtIO NIC is configured. 7.13.4. Adding VirtIO drivers container disk to a virtual machine OpenShift Virtualization distributes VirtIO drivers for Microsoft Windows as a container disk, which is available from the Red Hat Ecosystem Catalog . To install these drivers to a Windows virtual machine, attach the container-native-virtualization/virtio-win container disk to the virtual machine as a SATA CD drive in the virtual machine configuration file. Prerequisites Download the container-native-virtualization/virtio-win container disk from the Red Hat Ecosystem Catalog . This is not mandatory, because the container disk will be downloaded from the Red Hat registry if it not already present in the cluster, but it can reduce installation time. Procedure Add the container-native-virtualization/virtio-win container disk as a cdrom disk in the Windows virtual machine configuration file. The container disk will be downloaded from the registry if it is not already present in the cluster. spec: domain: devices: disks: - name: virtiocontainerdisk bootOrder: 2 1 cdrom: bus: sata volumes: - containerDisk: image: container-native-virtualization/virtio-win name: virtiocontainerdisk 1 OpenShift Virtualization boots virtual machine disks in the order defined in the VirtualMachine configuration file. You can either define other disks for the virtual machine before the container-native-virtualization/virtio-win container disk or use the optional bootOrder parameter to ensure the virtual machine boots from the correct disk. If you specify the bootOrder for a disk, it must be specified for all disks in the configuration. The disk is available once the virtual machine has started: If you add the container disk to a running virtual machine, use oc apply -f <vm.yaml> in the CLI or reboot the virtual machine for the changes to take effect. If the virtual machine is not running, use virtctl start <vm> . After the virtual machine has started, the VirtIO drivers can be installed from the attached SATA CD drive. 7.13.5. Installing VirtIO drivers during Windows installation Install the VirtIO drivers from the attached SATA CD driver during Windows installation. Note This procedure uses a generic approach to the Windows installation and the installation method might differ between versions of Windows. Refer to the documentation for the version of Windows that you are installing. Procedure Start the virtual machine and connect to a graphical console. Begin the Windows installation process. Select the Advanced installation. The storage destination will not be recognized until the driver is loaded. Click Load driver . The drivers are attached as a SATA CD drive. Click OK and browse the CD drive for the storage driver to load. The drivers are arranged hierarchically according to their driver type, operating system, and CPU architecture. Repeat the two steps for all required drivers. Complete the Windows installation. 7.13.6. Removing the VirtIO container disk from a virtual machine After installing all required VirtIO drivers to the virtual machine, the container-native-virtualization/virtio-win container disk no longer needs to be attached to the virtual machine. Remove the container-native-virtualization/virtio-win container disk from the virtual machine configuration file. Procedure Edit the configuration file and remove the disk and the volume . USD oc edit vm <vm-name> spec: domain: devices: disks: - name: virtiocontainerdisk bootOrder: 2 cdrom: bus: sata volumes: - containerDisk: image: container-native-virtualization/virtio-win name: virtiocontainerdisk Reboot the virtual machine for the changes to take effect. 7.14. Advanced virtual machine management 7.14.1. Automating management tasks You can automate OpenShift Virtualization management tasks by using Red Hat Ansible Automation Platform. Learn the basics by using an Ansible Playbook to create a new virtual machine. 7.14.1.1. About Red Hat Ansible Automation Ansible is an automation tool used to configure systems, deploy software, and perform rolling updates. Ansible includes support for OpenShift Virtualization, and Ansible modules enable you to automate cluster management tasks such as template, persistent volume claim, and virtual machine operations. Ansible provides a way to automate OpenShift Virtualization management, which you can also accomplish by using the oc CLI tool or APIs. Ansible is unique because it allows you to integrate KubeVirt modules with other Ansible modules. 7.14.1.2. Automating virtual machine creation You can use the kubevirt_vm Ansible Playbook to create virtual machines in your OpenShift Container Platform cluster using Red Hat Ansible Automation Platform. Prerequisites Red Hat Ansible Engine version 2.8 or newer Procedure Edit an Ansible Playbook YAML file so that it includes the kubevirt_vm task: kubevirt_vm: namespace: name: cpu_cores: memory: disks: - name: volume: containerDisk: image: disk: bus: Note This snippet only includes the kubevirt_vm portion of the playbook. Edit the values to reflect the virtual machine you want to create, including the namespace , the number of cpu_cores , the memory , and the disks . For example: kubevirt_vm: namespace: default name: vm1 cpu_cores: 1 memory: 64Mi disks: - name: containerdisk volume: containerDisk: image: kubevirt/cirros-container-disk-demo:latest disk: bus: virtio If you want the virtual machine to boot immediately after creation, add state: running to the YAML file. For example: kubevirt_vm: namespace: default name: vm1 state: running 1 cpu_cores: 1 1 Changing this value to state: absent deletes the virtual machine, if it already exists. Run the ansible-playbook command, using your playbook's file name as the only argument: USD ansible-playbook create-vm.yaml Review the output to determine if the play was successful: Example output (...) TASK [Create my first VM] ************************************************************************ changed: [localhost] PLAY RECAP ******************************************************************************************************** localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 If you did not include state: running in your playbook file and you want to boot the VM now, edit the file so that it includes state: running and run the playbook again: USD ansible-playbook create-vm.yaml To verify that the virtual machine was created, try to access the VM console . 7.14.1.3. Example: Ansible Playbook for creating virtual machines You can use the kubevirt_vm Ansible Playbook to automate virtual machine creation. The following YAML file is an example of the kubevirt_vm playbook. It includes sample values that you must replace with your own information if you run the playbook. --- - name: Ansible Playbook 1 hosts: localhost connection: local tasks: - name: Create my first VM kubevirt_vm: namespace: default name: vm1 cpu_cores: 1 memory: 64Mi disks: - name: containerdisk volume: containerDisk: image: kubevirt/cirros-container-disk-demo:latest disk: bus: virtio Additional information Intro to Playbooks Tools for Validating Playbooks 7.14.2. Using EFI mode for virtual machines You can boot a virtual machine (VM) in Extensible Firmware Interface (EFI) mode. 7.14.2.1. About EFI mode for virtual machines Extensible Firmware Interface (EFI), like legacy BIOS, initializes hardware components and operating system image files when a computer starts. EFI supports more modern features and customization options than BIOS, enabling faster boot times. It stores all the information about initialization and startup in a file with a .efi extension, which is stored on a special partition called EFI System Partition (ESP). The ESP also contains the boot loader programs for the operating system that is installed on the computer. Note OpenShift Virtualization only supports a virtual machine (VM) with Secure Boot when using EFI mode. If Secure Boot is not enabled, the VM crashes repeatedly. However, the VM might not support Secure Boot. Before you boot a VM, verify that it supports Secure Boot by checking the VM settings. 7.14.2.2. Booting virtual machines in EFI mode You can configure a virtual machine to boot in EFI mode by editing the VM or VMI manifest. Prerequisites Install the OpenShift CLI ( oc ). Procedure Create a YAML file that defines a VM object or a Virtual Machine Instance (VMI) object. Use the firmware stanza of the example YAML file: Booting in EFI mode with secure boot active apiversion: kubevirt.io/v1 kind: VirtualMachineInstance metadata: labels: special: vmi-secureboot name: vmi-secureboot spec: domain: devices: disks: - disk: bus: virtio name: containerdisk features: acpi: {} smm: enabled: true 1 firmware: bootloader: efi: secureBoot: true 2 ... 1 OpenShift Virtualization requires System Management Mode ( SMM ) to be enabled for Secure Boot in EFI mode to occur. 2 OpenShift Virtualization only supports a virtual machine (VM) with Secure Boot when using EFI mode. If Secure Boot is not enabled, the VM crashes repeatedly. However, the VM might not support Secure Boot. Before you boot a VM, verify that it supports Secure Boot by checking the VM settings. Apply the manifest to your cluster by running the following command: USD oc create -f <file_name>.yaml 7.14.3. Configuring PXE booting for virtual machines PXE booting, or network booting, is available in OpenShift Virtualization. Network booting allows a computer to boot and load an operating system or other program without requiring a locally attached storage device. For example, you can use it to choose your desired OS image from a PXE server when deploying a new host. 7.14.3.1. Prerequisites A Linux bridge must be connected . The PXE server must be connected to the same VLAN as the bridge. 7.14.3.2. PXE booting with a specified MAC address As an administrator, you can boot a client over the network by first creating a NetworkAttachmentDefinition object for your PXE network. Then, reference the network attachment definition in your virtual machine instance configuration file before you start the virtual machine instance. You can also specify a MAC address in the virtual machine instance configuration file, if required by the PXE server. Prerequisites A Linux bridge must be connected. The PXE server must be connected to the same VLAN as the bridge. Procedure Configure a PXE network on the cluster: Create the network attachment definition file for PXE network pxe-net-conf : apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: pxe-net-conf spec: config: '{ "cniVersion": "0.3.1", "name": "pxe-net-conf", "plugins": [ { "type": "cnv-bridge", "bridge": "br1", "vlan": 1 1 }, { "type": "cnv-tuning" 2 } ] }' 1 Optional: The VLAN tag. 2 The cnv-tuning plug-in provides support for custom MAC addresses. Note The virtual machine instance will be attached to the bridge br1 through an access port with the requested VLAN. Create the network attachment definition by using the file you created in the step: USD oc create -f pxe-net-conf.yaml Edit the virtual machine instance configuration file to include the details of the interface and network. Specify the network and MAC address, if required by the PXE server. If the MAC address is not specified, a value is assigned automatically. However, note that at this time, MAC addresses assigned automatically are not persistent. Ensure that bootOrder is set to 1 so that the interface boots first. In this example, the interface is connected to a network called <pxe-net> : interfaces: - masquerade: {} name: default - bridge: {} name: pxe-net macAddress: de:00:00:00:00:de bootOrder: 1 Note Boot order is global for interfaces and disks. Assign a boot device number to the disk to ensure proper booting after operating system provisioning. Set the disk bootOrder value to 2 : devices: disks: - disk: bus: virtio name: containerdisk bootOrder: 2 Specify that the network is connected to the previously created network attachment definition. In this scenario, <pxe-net> is connected to the network attachment definition called <pxe-net-conf> : networks: - name: default pod: {} - name: pxe-net multus: networkName: pxe-net-conf Create the virtual machine instance: USD oc create -f vmi-pxe-boot.yaml Example output virtualmachineinstance.kubevirt.io "vmi-pxe-boot" created Wait for the virtual machine instance to run: USD oc get vmi vmi-pxe-boot -o yaml | grep -i phase phase: Running View the virtual machine instance using VNC: USD virtctl vnc vmi-pxe-boot Watch the boot screen to verify that the PXE boot is successful. Log in to the virtual machine instance: USD virtctl console vmi-pxe-boot Verify the interfaces and MAC address on the virtual machine and that the interface connected to the bridge has the specified MAC address. In this case, we used eth1 for the PXE boot, without an IP address. The other interface, eth0 , got an IP address from OpenShift Container Platform. USD ip addr Example output ... 3. eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether de:00:00:00:00:de brd ff:ff:ff:ff:ff:ff 7.14.3.3. Template: Virtual machine instance configuration file for PXE booting apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachineInstance metadata: creationTimestamp: null labels: special: vmi-pxe-boot name: vmi-pxe-boot spec: domain: devices: disks: - disk: bus: virtio name: containerdisk bootOrder: 2 - disk: bus: virtio name: cloudinitdisk interfaces: - masquerade: {} name: default - bridge: {} name: pxe-net macAddress: de:00:00:00:00:de bootOrder: 1 machine: type: "" resources: requests: memory: 1024M networks: - name: default pod: {} - multus: networkName: pxe-net-conf name: pxe-net terminationGracePeriodSeconds: 0 volumes: - name: containerdisk containerDisk: image: kubevirt/fedora-cloud-container-disk-demo - cloudInitNoCloud: userData: | #!/bin/bash echo "fedora" | passwd fedora --stdin name: cloudinitdisk status: {} 7.14.3.4. OpenShift Virtualization networking glossary OpenShift Virtualization provides advanced networking functionality by using custom resources and plug-ins. The following terms are used throughout OpenShift Virtualization documentation: Container Network Interface (CNI) a Cloud Native Computing Foundation project, focused on container network connectivity. OpenShift Virtualization uses CNI plug-ins to build upon the basic Kubernetes networking functionality. Multus a "meta" CNI plug-in that allows multiple CNIs to exist so that a pod or virtual machine can use the interfaces it needs. Custom resource definition (CRD) a Kubernetes API resource that allows you to define custom resources, or an object defined by using the CRD API resource. Network attachment definition a CRD introduced by the Multus project that allows you to attach pods, virtual machines, and virtual machine instances to one or more networks. Preboot eXecution Environment (PXE) an interface that enables an administrator to boot a client machine from a server over the network. Network booting allows you to remotely load operating systems and other software onto the client. 7.14.4. Managing guest memory If you want to adjust guest memory settings to suit a specific use case, you can do so by editing the guest's YAML configuration file. OpenShift Virtualization allows you to configure guest memory overcommitment and disable guest memory overhead accounting. Warning The following procedures increase the chance that virtual machine processes will be killed due to memory pressure. Proceed only if you understand the risks. 7.14.4.1. Configuring guest memory overcommitment If your virtual workload requires more memory than available, you can use memory overcommitment to allocate all or most of the host's memory to your virtual machine instances (VMIs). Enabling memory overcommitment means that you can maximize resources that are normally reserved for the host. For example, if the host has 32 GB RAM, you can use memory overcommitment to fit 8 virtual machines (VMs) with 4 GB RAM each. This allocation works under the assumption that the virtual machines will not use all of their memory at the same time. Important Memory overcommitment increases the potential for virtual machine processes to be killed due to memory pressure (OOM killed). The potential for a VM to be OOM killed varies based on your specific configuration, node memory, available swap space, virtual machine memory consumption, the use of kernel same-page merging (KSM), and other factors. Procedure To explicitly tell the virtual machine instance that it has more memory available than was requested from the cluster, edit the virtual machine configuration file and set spec.domain.memory.guest to a higher value than spec.domain.resources.requests.memory . This process is called memory overcommitment. In this example, 1024M is requested from the cluster, but the virtual machine instance is told that it has 2048M available. As long as there is enough free memory available on the node, the virtual machine instance will consume up to 2048M. kind: VirtualMachine spec: template: domain: resources: requests: memory: 1024M memory: guest: 2048M Note The same eviction rules as those for pods apply to the virtual machine instance if the node is under memory pressure. Create the virtual machine: USD oc create -f <file_name>.yaml 7.14.4.2. Disabling guest memory overhead accounting A small amount of memory is requested by each virtual machine instance in addition to the amount that you request. This additional memory is used for the infrastructure that wraps each VirtualMachineInstance process. Though it is not usually advisable, it is possible to increase the virtual machine instance density on the node by disabling guest memory overhead accounting. Important Disabling guest memory overhead accounting increases the potential for virtual machine processes to be killed due to memory pressure (OOM killed). The potential for a VM to be OOM killed varies based on your specific configuration, node memory, available swap space, virtual machine memory consumption, the use of kernel same-page merging (KSM), and other factors. Procedure To disable guest memory overhead accounting, edit the YAML configuration file and set the overcommitGuestOverhead value to true . This parameter is disabled by default. kind: VirtualMachine spec: template: domain: resources: overcommitGuestOverhead: true requests: memory: 1024M Note If overcommitGuestOverhead is enabled, it adds the guest overhead to memory limits, if present. Create the virtual machine: USD oc create -f <file_name>.yaml 7.14.5. Using huge pages with virtual machines You can use huge pages as backing memory for virtual machines in your cluster. 7.14.5.1. Prerequisites Nodes must have pre-allocated huge pages configured . 7.14.5.2. What huge pages do Memory is managed in blocks known as pages. On most systems, a page is 4Ki. 1Mi of memory is equal to 256 pages; 1Gi of memory is 256,000 pages, and so on. CPUs have a built-in memory management unit that manages a list of these pages in hardware. The Translation Lookaside Buffer (TLB) is a small hardware cache of virtual-to-physical page mappings. If the virtual address passed in a hardware instruction can be found in the TLB, the mapping can be determined quickly. If not, a TLB miss occurs, and the system falls back to slower, software-based address translation, resulting in performance issues. Since the size of the TLB is fixed, the only way to reduce the chance of a TLB miss is to increase the page size. A huge page is a memory page that is larger than 4Ki. On x86_64 architectures, there are two common huge page sizes: 2Mi and 1Gi. Sizes vary on other architectures. To use huge pages, code must be written so that applications are aware of them. Transparent Huge Pages (THP) attempt to automate the management of huge pages without application knowledge, but they have limitations. In particular, they are limited to 2Mi page sizes. THP can lead to performance degradation on nodes with high memory utilization or fragmentation due to defragmenting efforts of THP, which can lock memory pages. For this reason, some applications may be designed to (or recommend) usage of pre-allocated huge pages instead of THP. In OpenShift Virtualization, virtual machines can be configured to consume pre-allocated huge pages. 7.14.5.3. Configuring huge pages for virtual machines You can configure virtual machines to use pre-allocated huge pages by including the memory.hugepages.pageSize and resources.requests.memory parameters in your virtual machine configuration. The memory request must be divisible by the page size. For example, you cannot request 500Mi memory with a page size of 1Gi . Note The memory layouts of the host and the guest OS are unrelated. Huge pages requested in the virtual machine manifest apply to QEMU. Huge pages inside the guest can only be configured based on the amount of available memory of the virtual machine instance. If you edit a running virtual machine, the virtual machine must be rebooted for the changes to take effect. Prerequisites Nodes must have pre-allocated huge pages configured. Procedure In your virtual machine configuration, add the resources.requests.memory and memory.hugepages.pageSize parameters to the spec.domain . The following configuration snippet is for a virtual machine that requests a total of 4Gi memory with a page size of 1Gi : kind: VirtualMachine ... spec: domain: resources: requests: memory: "4Gi" 1 memory: hugepages: pageSize: "1Gi" 2 ... 1 The total amount of memory requested for the virtual machine. This value must be divisible by the page size. 2 The size of each huge page. Valid values for x86_64 architecture are 1Gi and 2Mi . The page size must be smaller than the requested memory. Apply the virtual machine configuration: USD oc apply -f <virtual_machine>.yaml 7.14.6. Enabling dedicated resources for virtual machines To improve performance, you can dedicate node resources, such as CPU, to a virtual machine. 7.14.6.1. About dedicated resources When you enable dedicated resources for your virtual machine, your virtual machine's workload is scheduled on CPUs that will not be used by other processes. By using dedicated resources, you can improve the performance of the virtual machine and the accuracy of latency predictions. 7.14.6.2. Prerequisites The CPU Manager must be configured on the node. Verify that the node has the cpumanager = true label before scheduling virtual machine workloads. The virtual machine must be powered off. 7.14.6.3. Enabling dedicated resources for a virtual machine You can enable dedicated resources for a virtual machine in the Details tab. Virtual machines that were created by using a Red Hat template or the wizard can be enabled with dedicated resources. Procedure Click Workloads Virtual Machines from the side menu. Select a virtual machine to open the Virtual Machine tab. Click the Details tab. Click the pencil icon to the right of the Dedicated Resources field to open the Dedicated Resources window. Select Schedule this workload with dedicated resources (guaranteed policy) . Click Save . 7.14.7. Scheduling virtual machines You can schedule a virtual machine (VM) on a node by ensuring that the VM's CPU model and policy attribute are matched for compatibility with the CPU models and policy attributes supported by the node. 7.14.7.1. Understanding policy attributes You can schedule a virtual machine (VM) by specifying a policy attribute and a CPU feature that is matched for compatibility when the VM is scheduled on a node. A policy attribute specified for a VM determines how that VM is scheduled on a node. Policy attribute Description force The VM is forced to be scheduled on a node. This is true even if the host CPU does not support the VM's CPU. require Default policy that applies to a VM if the VM is not configured with a specific CPU model and feature specification. If a node is not configured to support CPU node discovery with this default policy attribute or any one of the other policy attributes, VMs are not scheduled on that node. Either the host CPU must support the VM's CPU or the hypervisor must be able to emulate the supported CPU model. optional The VM is added to a node if that VM is supported by the host's physical machine CPU. disable The VM cannot be scheduled with CPU node discovery. forbid The VM is not scheduled even if the feature is supported by the host CPU and CPU node discovery is enabled. 7.14.7.2. Setting a policy attribute and CPU feature You can set a policy attribute and CPU feature for each virtual machine (VM) to ensure that it is scheduled on a node according to policy and feature. The CPU feature that you set is verified to ensure that it is supported by the host CPU or emulated by the hypervisor. Procedure Edit the domain spec of your VM configuration file. The following example sets the CPU feature and the require policy for a virtual machine instance (VMI): apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: myvmi spec: domain: cpu: features: - name: apic 1 policy: require 2 1 Name of the CPU feature for the VM or VMI. 2 Policy attribute for the VM or VMI. 7.14.7.3. Scheduling virtual machines with the supported CPU model You can configure a CPU model for a virtual machine (VM) or a virtual machine instance (VMI) to schedule it on a node where its CPU model is supported. Procedure Edit the domain spec of your virtual machine configuration file. The following example shows a specific CPU model defined for a VMI: apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachineInstance metadata: name: myvmi spec: domain: cpu: model: Conroe 1 1 CPU model for the VMI. 7.14.7.4. Scheduling virtual machines with the host model When the CPU model for a virtual machine (VM) is set to host-model , the VM inherits the CPU model of the node where it is scheduled. Procedure Edit the domain spec of your VM configuration file. The following example shows host-model being specified for the virtual machine instance (VMI): apiVersion: kubevirt/v1alpha3 kind: VirtualMachineInstance metadata: name: myvmi spec: domain: cpu: model: host-model 1 1 The VM or VMI that inherits the CPU model of the node where it is scheduled. 7.15. Importing virtual machines 7.15.1. TLS certificates for data volume imports 7.15.1.1. Adding TLS certificates for authenticating data volume imports TLS certificates for registry or HTTPS endpoints must be added to a config map to import data from these sources. This config map must be present in the namespace of the destination data volume. Create the config map by referencing the relative file path for the TLS certificate. Procedure Ensure you are in the correct namespace. The config map can only be referenced by data volumes if it is in the same namespace. USD oc get ns Create the config map: USD oc create configmap <configmap-name> --from-file=</path/to/file/ca.pem> 7.15.1.2. Example: Config map created from a TLS certificate The following example is of a config map created from ca.pem TLS certificate. apiVersion: v1 kind: ConfigMap metadata: name: tls-certs data: ca.pem: | -----BEGIN CERTIFICATE----- ... <base64 encoded cert> ... -----END CERTIFICATE----- 7.15.2. Importing virtual machine images with data volumes Use the Containerized Data Importer (CDI) to import a virtual machine image into a persistent volume claim (PVC) by using a data volume. You can attach a data volume to a virtual machine for persistent storage. The virtual machine image can be hosted at an HTTP or HTTPS endpoint, or built into a container disk and stored in a container registry. Important When you import a disk image into a PVC, the disk image is expanded to use the full storage capacity that is requested in the PVC. To use this space, the disk partitions and file system(s) in the virtual machine might need to be expanded. The resizing procedure varies based on the operating system installed on the virtual machine. Refer to the operating system documentation for details. 7.15.2.1. Prerequisites If the endpoint requires a TLS certificate, the certificate must be included in a config map in the same namespace as the data volume and referenced in the data volume configuration. To import a container disk: You might need to prepare a container disk from a virtual machine image and store it in your container registry before importing it. If the container registry does not have TLS, you must add the registry to the cdi-insecure-registries config map before you can import a container disk from it. You might need to define a storage class or prepare CDI scratch space for this operation to complete successfully. 7.15.2.2. CDI supported operations matrix This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space. Content types HTTP HTTPS HTTP basic auth Registry Upload KubeVirt(QCOW2) [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2** [✓] GZ* [✓] XZ* [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2* □ GZ □ XZ [✓] QCOW2* [✓] GZ* [✓] XZ* KubeVirt (RAW) [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW* □ GZ □ XZ [✓] RAW* [✓] GZ* [✓] XZ* [✓] Supported operation □ Unsupported operation * Requires scratch space ** Requires scratch space if a custom certificate authority is required 7.15.2.3. About data volumes DataVolume objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared. 7.15.2.4. Importing a virtual machine image into storage by using a data volume You can import a virtual machine image into storage by using a data volume. The virtual machine image can be hosted at an HTTP or HTTPS endpoint or the image can be built into a container disk and stored in a container registry. You specify the data source for the image in a VirtualMachine configuration file. When the virtual machine is created, the data volume with the virtual machine image is imported into storage. Prerequisites To import a virtual machine image you must have the following: A virtual machine disk image in RAW, ISO, or QCOW2 format, optionally compressed by using xz or gz . An HTTP or HTTPS endpoint where the image is hosted, along with any authentication credentials needed to access the data source. To import a container disk, you must have a virtual machine image built into a container disk and stored in a container registry, along with any authentication credentials needed to access the data source. If the virtual machine must communicate with servers that use self-signed certificates or certificates not signed by the system CA bundle, you must create a config map in the same namespace as the data volume. Procedure If your data source requires authentication, create a Secret manifest, specifying the data source credentials, and save it as endpoint-secret.yaml : apiVersion: v1 kind: Secret metadata: name: endpoint-secret 1 labels: app: containerized-data-importer type: Opaque data: accessKeyId: "" 2 secretKey: "" 3 1 Specify the name of the Secret . 2 Specify the Base64-encoded key ID or user name. 3 Specify the Base64-encoded secret key or password. Apply the Secret manifest: USD oc apply -f endpoint-secret.yaml Edit the VirtualMachine manifest, specifying the data source for the virtual machine image you want to import, and save it as vm-fedora-datavolume.yaml : 1 Specify the name of the virtual machine. 2 Specify the name of the data volume. 3 Specify http for an HTTP or HTTPS endpoint. Specify registry for a container disk image imported from a registry. 4 The source of the virtual machine image you want to import. This example references a virtual machine image at an HTTPS endpoint. An example of a container registry endpoint is url: "docker://kubevirt/fedora-cloud-container-disk-demo:latest" . 5 Required if you created a Secret for the data source. 6 Optional: Specify a CA certificate config map. Create the virtual machine: USD oc create -f vm-fedora-datavolume.yaml Note The oc create command creates the data volume and the virtual machine. The CDI controller creates an underlying PVC with the correct annotation and the import process begins. When the import is complete, the data volume status changes to Succeeded . You can start the virtual machine. Data volume provisioning happens in the background, so there is no need to monitor the process. Verification The importer pod downloads the virtual machine image or container disk from the specified URL and stores it on the provisioned PV. View the status of the importer pod by running the following command: USD oc get pods Monitor the data volume until its status is Succeeded by running the following command: USD oc describe dv fedora-dv 1 1 Specify the data volume name that you defined in the VirtualMachine manifest. Verify that provisioning is complete and that the virtual machine has started by accessing its serial console: USD virtctl console vm-fedora-datavolume 7.15.2.5. Additional resources Configure preallocation mode to improve write performance for data volume operations. 7.15.3. Importing virtual machine images into block storage with data volumes You can import an existing virtual machine image into your OpenShift Container Platform cluster. OpenShift Virtualization uses data volumes to automate the import of data and the creation of an underlying persistent volume claim (PVC). Important When you import a disk image into a PVC, the disk image is expanded to use the full storage capacity that is requested in the PVC. To use this space, the disk partitions and file system(s) in the virtual machine might need to be expanded. The resizing procedure varies based on the operating system that is installed on the virtual machine. Refer to the operating system documentation for details. 7.15.3.1. Prerequisites If you require scratch space according to the CDI supported operations matrix , you must first define a storage class or prepare CDI scratch space for this operation to complete successfully. 7.15.3.2. About data volumes DataVolume objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared. 7.15.3.3. About block persistent volumes A block persistent volume (PV) is a PV that is backed by a raw block device. These volumes do not have a file system and can provide performance benefits for virtual machines by reducing overhead. Raw block volumes are provisioned by specifying volumeMode: Block in the PV and persistent volume claim (PVC) specification. 7.15.3.4. Creating a local block persistent volume Create a local block persistent volume (PV) on a node by populating a file and mounting it as a loop device. You can then reference this loop device in a PV manifest as a Block volume and use it as a block device for a virtual machine image. Procedure Log in as root to the node on which to create the local PV. This procedure uses node01 for its examples. Create a file and populate it with null characters so that it can be used as a block device. The following example creates a file loop10 with a size of 2Gb (20 100Mb blocks): USD dd if=/dev/zero of=<loop10> bs=100M count=20 Mount the loop10 file as a loop device. USD losetup </dev/loop10>d3 <loop10> 1 2 1 File path where the loop device is mounted. 2 The file created in the step to be mounted as the loop device. Create a PersistentVolume manifest that references the mounted loop device. kind: PersistentVolume apiVersion: v1 metadata: name: <local-block-pv10> annotations: spec: local: path: </dev/loop10> 1 capacity: storage: <2Gi> volumeMode: Block 2 storageClassName: local 3 accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - <node01> 4 1 The path of the loop device on the node. 2 Specifies it is a block PV. 3 Optional: Set a storage class for the PV. If you omit it, the cluster default is used. 4 The node on which the block device was mounted. Create the block PV. # oc create -f <local-block-pv10.yaml> 1 1 The file name of the persistent volume created in the step. 7.15.3.5. Importing a virtual machine image into block storage by using a data volume You can import a virtual machine image into block storage by using a data volume. You reference the data volume in a VirtualMachine manifest before you create a virtual machine. Prerequisites A virtual machine disk image in RAW, ISO, or QCOW2 format, optionally compressed by using xz or gz . An HTTP or HTTPS endpoint where the image is hosted, along with any authentication credentials needed to access the data source. Procedure If your data source requires authentication, create a Secret manifest, specifying the data source credentials, and save it as endpoint-secret.yaml : apiVersion: v1 kind: Secret metadata: name: endpoint-secret 1 labels: app: containerized-data-importer type: Opaque data: accessKeyId: "" 2 secretKey: "" 3 1 Specify the name of the Secret . 2 Specify the Base64-encoded key ID or user name. 3 Specify the Base64-encoded secret key or password. Apply the Secret manifest: USD oc apply -f endpoint-secret.yaml Create a DataVolume manifest, specifying the data source for the virtual machine image and Block for storage.volumeMode . apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: import-pv-datavolume 1 spec: storageClassName: local 2 source: http: url: "https://mirror.arizona.edu/fedora/linux/releases/35/Cloud/x86_64/images/Fedora-Cloud-Base-35-1.2.x86_64.qcow2" 3 secretRef: endpoint-secret 4 storage: volumeMode: Block 5 resources: requests: storage: 10Gi 1 Specify the name of the data volume. 2 Optional: Set the storage class or omit it to accept the cluster default. 3 Specify the HTTP or HTTPS URL of the image to import. 4 Required if you created a Secret for the data source. 5 The volume mode and access mode are detected automatically for known storage provisioners. Otherwise, specify Block . Create the data volume to import the virtual machine image: USD oc create -f import-pv-datavolume.yaml You can reference this data volume in a VirtualMachine manifest before you create a virtual machine. 7.15.3.6. CDI supported operations matrix This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space. Content types HTTP HTTPS HTTP basic auth Registry Upload KubeVirt(QCOW2) [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2** [✓] GZ* [✓] XZ* [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2* □ GZ □ XZ [✓] QCOW2* [✓] GZ* [✓] XZ* KubeVirt (RAW) [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW* □ GZ □ XZ [✓] RAW* [✓] GZ* [✓] XZ* [✓] Supported operation □ Unsupported operation * Requires scratch space ** Requires scratch space if a custom certificate authority is required 7.15.3.7. Additional resources Configure preallocation mode to improve write performance for data volume operations. 7.15.4. Importing a single Red Hat Virtualization virtual machine You can import a single Red Hat Virtualization (RHV) virtual machine into OpenShift Virtualization by using the VM Import wizard or the CLI. 7.15.4.1. OpenShift Virtualization storage feature matrix The following table describes the OpenShift Virtualization storage types that support VM import. Table 7.3. OpenShift Virtualization storage feature matrix RHV VM import OpenShift Container Storage: RBD block-mode volumes Yes OpenShift Virtualization hostpath provisioner No Other multi-node writable storage Yes [1] Other single-node writable storage Yes [2] PVCs must request a ReadWriteMany access mode. PVCs must request a ReadWriteOnce access mode. 7.15.4.2. Prerequisites for importing a virtual machine Importing a virtual machine from Red Hat Virtualization (RHV) into OpenShift Virtualization has the following prerequisites: You must have admin user privileges. Storage: The OpenShift Virtualization local and shared persistent storage classes must support VM import. If you are using Ceph RBD block-mode volumes and the available storage space is too small for the virtual disk, the import process bar stops at 75% for more than 20 minutes and the migration does not succeed. No error message is displayed in the web console. BZ#1910019 Networks: The RHV and OpenShift Virtualization networks must either have the same name or be mapped to each other. The RHV VM network interface must be e1000 , rtl8139 , or virtio . VM disks: The disk interface must be sata , virtio_scsi , or virtio . The disk must not be configured as a direct LUN. The disk status must not be illegal or locked . The storage type must be image . SCSI reservation must be disabled. ScsiGenericIO must be disabled. VM configuration: If the VM uses GPU resources, the nodes providing the GPUs must be configured. The VM must not be configured for vGPU resources. The VM must not have snapshots with disks in an illegal state. The VM must not have been created with OpenShift Container Platform and subsequently added to RHV. The VM must not be configured for USB devices. The watchdog model must not be diag288 . 7.15.4.3. Importing a virtual machine with the VM Import wizard You can import a single virtual machine with the VM Import wizard. Procedure In the web console, click Workloads Virtual Machines . Click Create Virtual Machine and select Import with Wizard . Select Red Hat Virtualization (RHV) from the Provider list. Select Connect to New Instance or a saved RHV instance. If you select Connect to New Instance , fill in the following fields: API URL : For example, https://<RHV_Manager_FQDN>/ovirt-engine/api CA certificate : Click Browse to upload the RHV Manager CA certificate or paste the CA certificate into the field. View the CA certificate by running the following command: USD openssl s_client -connect <RHV_Manager_FQDN>:443 -showcerts < /dev/null The CA certificate is the second certificate in the output. Username : RHV Manager user name, for example, ocpadmin@internal Password : RHV Manager password If you select a saved RHV instance, the wizard connects to the RHV instance using the saved credentials. Click Check and Save and wait for the connection to complete. Note The connection details are stored in a secret. If you add a provider with an incorrect URL, user name, or password, click Workloads Secrets and delete the provider secret. Select a cluster and a virtual machine. Click . In the Review screen, review your settings. Optional: You can select Start virtual machine on creation . Click Edit to update the following settings: General Name : The VM name is limited to 63 characters. General Description : Optional description of the VM. Storage Class : Select NFS or ocs-storagecluster-ceph-rbd . If you select ocs-storagecluster-ceph-rbd , you must set the Volume Mode of the disk to Block . Advanced Volume Mode : Select Block . Advanced Volume Mode : Select Block . Networking Network : You can select a network from a list of available network attachment definition objects. Click Import or Review and Import , if you have edited the import settings. A Successfully created virtual machine message and a list of resources created for the virtual machine are displayed. The virtual machine appears in Workloads Virtual Machines . Virtual machine wizard fields Name Parameter Description Name The name can contain lowercase letters ( a-z ), numbers ( 0-9 ), and hyphens ( - ), up to a maximum of 253 characters. The first and last characters must be alphanumeric. The name must not contain uppercase letters, spaces, periods ( . ), or special characters. Description Optional description field. Operating System The primary operating system that is selected for the virtual machine in the template. You cannot edit this field when creating a virtual machine from a template. Boot Source Import via URL (creates PVC) Import content from an image available from an HTTP or HTTPS endpoint. Example: Obtaining a URL link from the web page with the operating system image. Clone existing PVC (creates PVC) Select an existent persistent volume claim available on the cluster and clone it. Import via Registry (creates PVC) Provision virtual machine from a bootable operating system container located in a registry accessible from the cluster. Example: kubevirt/cirros-registry-disk-demo . PXE (network boot - adds network interface) Boot an operating system from a server on the network. Requires a PXE bootable network attachment definition. Persistent Volume Claim project Project name that you want to use for cloning the PVC. Persistent Volume Claim name PVC name that should apply to this virtual machine template if you are cloning an existing PVC. Mount this as a CD-ROM boot source A CD-ROM requires an additional disk for installing the operating system. Select the checkbox to add a disk and customize it later. Flavor Tiny, Small, Medium, Large, Custom Presets that determine the amount of CPU and memory allocated to the virtual machine. The presets displayed for Flavor are determined by the operating system. Workload Type Desktop A virtual machine configuration for use on a desktop. Ideal for consumption on a small scale. Recommended for use with the web console. Server Balances performance and it is compatible with a wide range of server workloads. High-Performance A virtual machine configuration that is optimized for high-performance workloads. Start this virtual machine after creation. This checkbox is selected by default and the virtual machine starts running after creation. Clear the checkbox if you do not want the virtual machine to start when it is created. Networking fields Name Description Name Name for the network interface controller. Model Indicates the model of the network interface controller. Supported values are e1000e and virtio . Network List of available network attachment definitions. Type List of available binding methods. For the default pod network, masquerade is the only recommended binding method. For secondary networks, use the bridge binding method. The masquerade method is not supported for non-default networks. Select SR-IOV if you configured an SR-IOV network device and defined that network in the namespace. MAC Address MAC address for the network interface controller. If a MAC address is not specified, one is assigned automatically. Storage fields Name Selection Description Source Blank (creates PVC) Create an empty disk. Import via URL (creates PVC) Import content via URL (HTTP or HTTPS endpoint). Use an existing PVC Use a PVC that is already available in the cluster. Clone existing PVC (creates PVC) Select an existing PVC available in the cluster and clone it. Import via Registry (creates PVC) Import content via container registry. Container (ephemeral) Upload content from a container located in a registry accessible from the cluster. The container disk should be used only for read-only filesystems such as CD-ROMs or temporary virtual machines. Name Name of the disk. The name can contain lowercase letters ( a-z ), numbers ( 0-9 ), hyphens ( - ), and periods ( . ), up to a maximum of 253 characters. The first and last characters must be alphanumeric. The name must not contain uppercase letters, spaces, or special characters. Size Size of the disk in GiB. Type Type of disk. Example: Disk or CD-ROM Interface Type of disk device. Supported interfaces are virtIO , SATA , and SCSI . Storage Class The storage class that is used to create the disk. Advanced Volume Mode Defines whether the persistent volume uses a formatted file system or raw block state. Default is Filesystem . Advanced storage settings Name Parameter Description Volume Mode Filesystem Stores the virtual disk on a file system-based volume. Block Stores the virtual disk directly on the block volume. Only use Block if the underlying storage supports it. Access Mode [1] Single User (RWO) The disk can be mounted as read/write by a single node. Shared Access (RWX) The disk can be mounted as read/write by many nodes. Read Only (ROX) The disk can be mounted as read-only by many nodes. You can change the access mode by using the command line interface. 7.15.4.4. Importing a virtual machine with the CLI You can import a virtual machine with the CLI by creating the Secret and VirtualMachineImport custom resources (CRs). The Secret CR stores the RHV Manager credentials and CA certificate. The VirtualMachineImport CR defines the parameters of the VM import process. Optional: You can create a ResourceMapping CR that is separate from the VirtualMachineImport CR. A ResourceMapping CR provides greater flexibility, for example, if you import additional RHV VMs. Important The default target storage class must be NFS. Cinder does not support RHV VM import. Procedure Create the Secret CR by running the following command: USD cat <<EOF | oc create -f - apiVersion: v1 kind: Secret metadata: name: rhv-credentials namespace: default 1 type: Opaque stringData: ovirt: | apiUrl: <api_endpoint> 2 username: ocpadmin@internal password: 3 caCert: | -----BEGIN CERTIFICATE----- 4 -----END CERTIFICATE----- EOF 1 Optional. You can specify a different namespace in all the CRs. 2 Specify the API endpoint of the RHV Manager, for example, \"https://www.example.com:8443/ovirt-engine/api" 3 Specify the password for ocpadmin@internal . 4 Specify the RHV Manager CA certificate. You can obtain the CA certificate by running the following command: USD openssl s_client -connect :443 -showcerts < /dev/null Optional: Create a ResourceMapping CR if you want to separate the resource mapping from the VirtualMachineImport CR by running the following command: USD cat <<EOF | kubectl create -f - apiVersion: v2v.kubevirt.io/v1alpha1 kind: ResourceMapping metadata: name: resourcemapping_example namespace: default spec: ovirt: networkMappings: - source: name: <rhv_logical_network>/<vnic_profile> 1 target: name: <target_network> 2 type: pod storageMappings: 3 - source: name: <rhv_storage_domain> 4 target: name: <target_storage_class> 5 volumeMode: <volume_mode> 6 EOF 1 Specify the RHV logical network and vNIC profile. 2 Specify the OpenShift Virtualization network. 3 If storage mappings are specified in both the ResourceMapping and the VirtualMachineImport CRs, the VirtualMachineImport CR takes precedence. 4 Specify the RHV storage domain. 5 Specify nfs or ocs-storagecluster-ceph-rbd . 6 If you specified the ocs-storagecluster-ceph-rbd storage class, you must specify Block as the volume mode. Create the VirtualMachineImport CR by running the following command: USD cat <<EOF | oc create -f - apiVersion: v2v.kubevirt.io/v1beta1 kind: VirtualMachineImport metadata: name: vm-import namespace: default spec: providerCredentialsSecret: name: rhv-credentials namespace: default # resourceMapping: 1 # name: resourcemapping-example # namespace: default targetVmName: vm_example 2 startVm: true source: ovirt: vm: id: <source_vm_id> 3 name: <source_vm_name> 4 cluster: name: <source_cluster_name> 5 mappings: 6 networkMappings: - source: name: <source_logical_network>/<vnic_profile> 7 target: name: <target_network> 8 type: pod storageMappings: 9 - source: name: <source_storage_domain> 10 target: name: <target_storage_class> 11 accessMode: <volume_access_mode> 12 diskMappings: - source: id: <source_vm_disk_id> 13 target: name: <target_storage_class> 14 EOF 1 If you create a ResourceMapping CR, uncomment the resourceMapping section. 2 Specify the target VM name. 3 Specify the source VM ID, for example, 80554327-0569-496b-bdeb-fcbbf52b827b . You can obtain the VM ID by entering https://www.example.com/ovirt-engine/api/vms/ in a web browser on the Manager machine to list all VMs. Locate the VM you want to import and its corresponding VM ID. You do not need to specify a VM name or cluster name. 4 If you specify the source VM name, you must also specify the source cluster. Do not specify the source VM ID. 5 If you specify the source cluster, you must also specify the source VM name. Do not specify the source VM ID. 6 If you create a ResourceMapping CR, comment out the mappings section. 7 Specify the logical network and vNIC profile of the source VM. 8 Specify the OpenShift Virtualization network. 9 If storage mappings are specified in both the ResourceMapping and the VirtualMachineImport CRs, the VirtualMachineImport CR takes precedence. 10 Specify the source storage domain. 11 Specify the target storage class. 12 Specify ReadWriteOnce , ReadWriteMany , or ReadOnlyMany . If no access mode is specified, {virt} determines the correct volume access mode based on the Host Migration mode setting of the RHV VM or on the virtual disk access mode: If the RHV VM migration mode is Allow manual and automatic migration , the default access mode is ReadWriteMany . If the RHV virtual disk access mode is ReadOnly , the default access mode is ReadOnlyMany . For all other settings, the default access mode is ReadWriteOnce . 13 Specify the source VM disk ID, for example, 8181ecc1-5db8-4193-9c92-3ddab3be7b05 . You can obtain the disk ID by entering https://www.example.com/ovirt-engine/api/vms/vm23 in a web browser on the Manager machine and reviewing the VM details. 14 Specify the target storage class. Follow the progress of the virtual machine import to verify that the import was successful: USD oc get vmimports vm-import -n default The output indicating a successful import resembles the following example: Example output ... status: conditions: - lastHeartbeatTime: "2020-07-22T08:58:52Z" lastTransitionTime: "2020-07-22T08:58:52Z" message: Validation completed successfully reason: ValidationCompleted status: "True" type: Valid - lastHeartbeatTime: "2020-07-22T08:58:52Z" lastTransitionTime: "2020-07-22T08:58:52Z" message: 'VM specifies IO Threads: 1, VM has NUMA tune mode specified: interleave' reason: MappingRulesVerificationReportedWarnings status: "True" type: MappingRulesVerified - lastHeartbeatTime: "2020-07-22T08:58:56Z" lastTransitionTime: "2020-07-22T08:58:52Z" message: Copying virtual machine disks reason: CopyingDisks status: "True" type: Processing dataVolumes: - name: fedora32-b870c429-11e0-4630-b3df-21da551a48c0 targetVmName: fedora32 7.15.4.4.1. Creating a config map for importing a VM You can create a config map to map the Red Hat Virtualization (RHV) virtual machine operating system to an OpenShift Virtualization template if you want to override the default vm-import-controller mapping or to add additional mappings. The default vm-import-controller config map contains the following RHV operating systems and their corresponding common OpenShift Virtualization templates. Table 7.4. Operating system and template mapping RHV VM operating system OpenShift Virtualization template rhel_6_9_plus_ppc64 rhel6.9 rhel_6_ppc64 rhel6.9 rhel_6 rhel6.9 rhel_6x64 rhel6.9 rhel_7_ppc64 rhel7.7 rhel_7_s390x rhel7.7 rhel_7x64 rhel7.7 rhel_8x64 rhel8.1 sles_11_ppc64 opensuse15.0 sles_11 opensuse15.0 sles_12_s390x opensuse15.0 ubuntu_12_04 ubuntu18.04 ubuntu_12_10 ubuntu18.04 ubuntu_13_04 ubuntu18.04 ubuntu_13_10 ubuntu18.04 ubuntu_14_04_ppc64 ubuntu18.04 ubuntu_14_04 ubuntu18.04 ubuntu_16_04_s390x ubuntu18.04 windows_10 win10 windows_10x64 win10 windows_2003 win10 windows_2003x64 win10 windows_2008R2x64 win2k8 windows_2008 win2k8 windows_2008x64 win2k8 windows_2012R2x64 win2k12r2 windows_2012x64 win2k12r2 windows_2016x64 win2k16 windows_2019x64 win2k19 windows_7 win10 windows_7x64 win10 windows_8 win10 windows_8x64 win10 windows_xp win10 Procedure In a web browser, identify the REST API name of the RHV VM operating system by navigating to http://<RHV_Manager_FQDN>/ovirt-engine/api/vms/<VM_ID> . The operating system name appears in the <os> section of the XML output, as shown in the following example: ... <os> ... <type>rhel_8x64</type> </os> View a list of the available OpenShift Virtualization templates: USD oc get templates -n openshift --show-labels | tr ',' '\n' | grep os.template.kubevirt.io | sed -r 's#os.template.kubevirt.io/(.*)=.*#\1#g' | sort -u Example output fedora31 fedora32 ... rhel8.1 rhel8.2 ... If an OpenShift Virtualization template that matches the RHV VM operating system does not appear in the list of available templates, create a template with the OpenShift Virtualization web console. Create a config map to map the RHV VM operating system to the OpenShift Virtualization template: USD cat <<EOF | oc create -f - apiVersion: v1 kind: ConfigMap metadata: name: os-configmap namespace: default 1 data: guestos2common: | "Red Hat Enterprise Linux Server": "rhel" "CentOS Linux": "centos" "Fedora": "fedora" "Ubuntu": "ubuntu" "openSUSE": "opensuse" osinfo2common: | "<rhv-operating-system>": "<vm-template>" 2 EOF 1 Optional: You can change the value of the namespace parameter. 2 Specify the REST API name of the RHV operating system and its corresponding VM template as shown in the following example. Config map example USD cat <<EOF | oc apply -f - apiVersion: v1 kind: ConfigMap metadata: name: os-configmap namespace: default data: osinfo2common: | "other_linux": "fedora31" EOF Verify that the custom config map was created: USD oc get cm -n default os-configmap -o yaml Patch the vm-import-controller-config config map to apply the new config map: USD oc patch configmap vm-import-controller-config -n openshift-cnv --patch '{ "data": { "osConfigMap.name": "os-configmap", "osConfigMap.namespace": "default" 1 } }' 1 Update the namespace if you changed it in the config map. Verify that the template appears in the OpenShift Virtualization web console: Click Workloads Virtualization from the side menu. Click the Virtual Machine Templates tab and find the template in the list. 7.15.4.5. Troubleshooting a virtual machine import 7.15.4.5.1. Logs You can check the VM Import Controller pod log for errors. Procedure View the VM Import Controller pod name by running the following command: USD oc get pods -n <namespace> | grep import 1 1 Specify the namespace of your imported virtual machine. Example output vm-import-controller-f66f7d-zqkz7 1/1 Running 0 4h49m View the VM Import Controller pod log by running the following command: USD oc logs <vm-import-controller-f66f7d-zqkz7> -f -n <namespace> 1 1 Specify the VM Import Controller pod name and the namespace. 7.15.4.5.2. Error messages The following error message might appear: The following error message is displayed in the VM Import Controller pod log and the progress bar stops at 10% if the OpenShift Virtualization storage PV is not suitable: Failed to bind volumes: provisioning failed for PVC You must use a compatible storage class. The Cinder storage class is not supported. 7.15.4.5.3. Known issues If you are using Ceph RBD block-mode volumes and the available storage space is too small for the virtual disk, the import process bar stops at 75% for more than 20 minutes and the migration does not succeed. No error message is displayed in the web console. BZ#1910019 7.15.5. Importing a single VMware virtual machine or template You can import a VMware vSphere 6.5, 6.7, or 7.0 VM or VM template into OpenShift Virtualization by using the VM Import wizard. If you import a VM template, OpenShift Virtualization creates a virtual machine based on the template. 7.15.5.1. OpenShift Virtualization storage feature matrix The following table describes the OpenShift Virtualization storage types that support VM import. Table 7.5. OpenShift Virtualization storage feature matrix VMware VM import OpenShift Container Storage: RBD block-mode volumes Yes OpenShift Virtualization hostpath provisioner Yes Other multi-node writable storage Yes [1] Other single-node writable storage Yes [2] PVCs must request a ReadWriteMany access mode. PVCs must request a ReadWriteOnce access mode. 7.15.5.2. Preparing a VDDK image The import process uses the VMware Virtual Disk Development Kit (VDDK) to copy the VMware virtual disk. You can download the VDDK SDK, create a VDDK image, upload the image to an image registry, and add it to the v2v-vmware config map. You can configure either an internal OpenShift Container Platform image registry or a secure external image registry for the VDDK image. The registry must be accessible to your OpenShift Virtualization environment. Note Storing the VDDK image in a public registry might violate the terms of the VMware license. 7.15.5.2.1. Configuring an internal image registry You can configure the internal OpenShift Container Platform image registry on bare metal by updating the Image Registry Operator configuration. You can access the registry directly, from within the OpenShift Container Platform cluster, or externally, by exposing the registry with a route. Changing the image registry's management state To start the image registry, you must change the Image Registry Operator configuration's managementState from Removed to Managed . Procedure Change managementState Image Registry Operator configuration from Removed to Managed . For example: USD oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}' Configuring registry storage for bare metal and other manual installations As a cluster administrator, following installation you must configure your registry to use storage. Prerequisites Cluster administrator permissions. A cluster that uses manually-provisioned Red Hat Enterprise Linux CoreOS (RHCOS) nodes, such as bare metal. Persistent storage provisioned for your cluster, such as Red Hat OpenShift Container Storage. Important OpenShift Container Platform supports ReadWriteOnce access for image registry storage when you have only one replica. To deploy an image registry that supports high availability with two or more replicas, ReadWriteMany access is required. Must have 100Gi capacity. Procedure To configure your registry to use storage, change the spec.storage.pvc in the configs.imageregistry/cluster resource. Note When using shared storage, review your security settings to prevent outside access. Verify that you do not have a registry pod: USD oc get pod -n openshift-image-registry -l docker-registry=default Example output No resourses found in openshift-image-registry namespace Note If you do have a registry pod in your output, you do not need to continue with this procedure. Check the registry configuration: USD oc edit configs.imageregistry.operator.openshift.io Example output storage: pvc: claim: Leave the claim field blank to allow the automatic creation of an image-registry-storage PVC. Check the clusteroperator status: USD oc get clusteroperator image-registry Example output NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE image-registry 4.7 True False False 6h50m Ensure that your registry is set to managed to enable building and pushing of images. Run: Then, change the line to Accessing registry directly from the cluster You can access the registry from inside the cluster. Procedure Access the registry from the cluster by using internal routes: Access the node by getting the node's name: USD oc get nodes USD oc debug nodes/<node_name> To enable access to tools such as oc and podman on the node, run the following command: sh-4.2# chroot /host Log in to the container image registry by using your access token: sh-4.2# oc login -u kubeadmin -p <password_from_install_log> https://api-int.<cluster_name>.<base_domain>:6443 sh-4.2# podman login -u kubeadmin -p USD(oc whoami -t) image-registry.openshift-image-registry.svc:5000 You should see a message confirming login, such as: Login Succeeded! Note You can pass any value for the user name; the token contains all necessary information. Passing a user name that contains colons will result in a login failure. Since the Image Registry Operator creates the route, it will likely be similar to default-route-openshift-image-registry.<cluster_name> . Perform podman pull and podman push operations against your registry: Important You can pull arbitrary images, but if you have the system:registry role added, you can only push images to the registry in your project. In the following examples, use: Component Value <registry_ip> 172.30.124.220 <port> 5000 <project> openshift <image> image <tag> omitted (defaults to latest ) Pull an arbitrary image: sh-4.2# podman pull name.io/image Tag the new image with the form <registry_ip>:<port>/<project>/<image> . The project name must appear in this pull specification for OpenShift Container Platform to correctly place and later access the image in the registry: sh-4.2# podman tag name.io/image image-registry.openshift-image-registry.svc:5000/openshift/image Note You must have the system:image-builder role for the specified project, which allows the user to write or push an image. Otherwise, the podman push in the step will fail. To test, you can create a new project to push the image. Push the newly tagged image to your registry: sh-4.2# podman push image-registry.openshift-image-registry.svc:5000/openshift/image Exposing a secure registry manually Instead of logging in to the OpenShift Container Platform registry from within the cluster, you can gain external access to it by exposing it with a route. This allows you to log in to the registry from outside the cluster using the route address, and to tag and push images to an existing project by using the route host. Prerequisites: The following prerequisites are automatically performed: Deploy the Registry Operator. Deploy the Ingress Operator. Procedure You can expose the route by using DefaultRoute parameter in the configs.imageregistry.operator.openshift.io resource or by using custom routes. To expose the registry using DefaultRoute : Set DefaultRoute to True : USD oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge Log in with podman : USD HOST=USD(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}') USD podman login -u kubeadmin -p USD(oc whoami -t) --tls-verify=false USDHOST 1 1 --tls-verify=false is needed if the cluster's default certificate for routes is untrusted. You can set a custom, trusted certificate as the default certificate with the Ingress Operator. To expose the registry using custom routes: Create a secret with your route's TLS keys: USD oc create secret tls public-route-tls \ -n openshift-image-registry \ --cert=</path/to/tls.crt> \ --key=</path/to/tls.key> This step is optional. If you do not create a secret, the route uses the default TLS configuration from the Ingress Operator. On the Registry Operator: spec: routes: - name: public-routes hostname: myregistry.mycorp.organization secretName: public-route-tls ... Note Only set secretName if you are providing a custom TLS configuration for the registry's route. 7.15.5.2.2. Configuring an external image registry If you use an external image registry for the VDDK image, you can add the external image registry's certificate authorities to the OpenShift Container Platform cluster. Optionally, you can create a pull secret from your Docker credentials and add it to your service account. Adding certificate authorities to the cluster You can add certificate authorities (CA) to the cluster for use when pushing and pulling images with the following procedure. Prerequisites You must have cluster administrator privileges. You must have access to the public certificates of the registry, usually a hostname/ca.crt file located in the /etc/docker/certs.d/ directory. Procedure Create a ConfigMap in the openshift-config namespace containing the trusted certificates for the registries that use self-signed certificates. For each CA file, ensure the key in the ConfigMap is the hostname of the registry in the hostname[..port] format: USD oc create configmap registry-cas -n openshift-config \ --from-file=myregistry.corp.com..5000=/etc/docker/certs.d/myregistry.corp.com:5000/ca.crt \ --from-file=otherregistry.com=/etc/docker/certs.d/otherregistry.com/ca.crt Update the cluster image configuration: USD oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge Allowing pods to reference images from other secured registries The .dockercfg USDHOME/.docker/config.json file for Docker clients is a Docker credentials file that stores your authentication information if you have previously logged into a secured or insecure registry. To pull a secured container image that is not from OpenShift Container Platform's internal registry, you must create a pull secret from your Docker credentials and add it to your service account. Procedure If you already have a .dockercfg file for the secured registry, you can create a secret from that file by running: USD oc create secret generic <pull_secret_name> \ --from-file=.dockercfg=<path/to/.dockercfg> \ --type=kubernetes.io/dockercfg Or if you have a USDHOME/.docker/config.json file: USD oc create secret generic <pull_secret_name> \ --from-file=.dockerconfigjson=<path/to/.docker/config.json> \ --type=kubernetes.io/dockerconfigjson If you do not already have a Docker credentials file for the secured registry, you can create a secret by running: USD oc create secret docker-registry <pull_secret_name> \ --docker-server=<registry_server> \ --docker-username=<user_name> \ --docker-password=<password> \ --docker-email=<email> To use a secret for pulling images for pods, you must add the secret to your service account. The name of the service account in this example should match the name of the service account the pod uses. The default service account is default : USD oc secrets link default <pull_secret_name> --for=pull 7.15.5.2.3. Creating and using a VDDK image You can download the VMware Virtual Disk Development Kit (VDDK), build a VDDK image, and push the VDDK image to your image registry. You then add the VDDK image to the v2v-vmware config map. Prerequisites You must have access to an OpenShift Container Platform internal image registry or a secure external registry. Procedure Create and navigate to a temporary directory: USD mkdir /tmp/<dir_name> && cd /tmp/<dir_name> In a browser, navigate to VMware code and click SDKs . Under Compute Virtualization , click Virtual Disk Development Kit (VDDK) . Select the VDDK version that corresponds to your VMware vSphere version, for example, VDDK 7.0 for vSphere 7.0, click Download , and then save the VDDK archive in the temporary directory. Extract the VDDK archive: USD tar -xzf VMware-vix-disklib-<version>.x86_64.tar.gz Create a Dockerfile : USD cat > Dockerfile <<EOF FROM busybox:latest COPY vmware-vix-disklib-distrib /vmware-vix-disklib-distrib RUN mkdir -p /opt ENTRYPOINT ["cp", "-r", "/vmware-vix-disklib-distrib", "/opt"] EOF Build the image: USD podman build . -t <registry_route_or_server_path>/vddk:<tag> 1 1 Specify your image registry: For an internal OpenShift Container Platform registry, use the internal registry route, for example, image-registry.openshift-image-registry.svc:5000/openshift/vddk:<tag> . For an external registry, specify the server name, path, and tag, for example, server.example.com:5000/vddk:<tag> . Push the image to the registry: USD podman push <registry_route_or_server_path>/vddk:<tag> Ensure that the image is accessible to your OpenShift Virtualization environment. Edit the v2v-vmware config map in the openshift-cnv project: USD oc edit configmap v2v-vmware -n openshift-cnv Add the vddk-init-image parameter to the data stanza: ... data: vddk-init-image: <registry_route_or_server_path>/vddk:<tag> 7.15.5.3. Importing a virtual machine with the VM Import wizard You can import a single virtual machine with the VM Import wizard. You can also import a VM template. If you import a VM template, OpenShift Virtualization creates a virtual machine based on the template. Prerequisites You must have admin user privileges. The VMware Virtual Disk Development Kit (VDDK) image must be in an image registry that is accessible to your OpenShift Virtualization environment. The VDDK image must be added to the v2v-vmware config map. The VM must be powered off. Virtual disks must be connected to IDE or SCSI controllers. If virtual disks are connected to a SATA controller, you can change them to IDE controllers and then migrate the VM. The OpenShift Virtualization local and shared persistent storage classes must support VM import. The OpenShift Virtualization storage must be large enough to accommodate the virtual disk. Warning If you are using Ceph RBD block-mode volumes, the storage must be large enough to accommodate the virtual disk. If the disk is too large for the available storage, the import process fails and the PV that is used to copy the virtual disk is not released. You will not be able to import another virtual machine or to clean up the storage because there are insufficient resources to support object deletion. To resolve this situation, you must add more object storage devices to the storage back end. The OpenShift Virtualization egress network policy must allow the following traffic: Destination Protocol Port VMware ESXi hosts TCP 443 VMware ESXi hosts TCP 902 VMware vCenter TCP 5840 Procedure In the web console, click Workloads Virtual Machines . Click Create Virtual Machine and select Import with Wizard . Select VMware from the Provider list. Select Connect to New Instance or a saved vCenter instance. If you select Connect to New Instance , enter the vCenter hostname , Username , and Password . If you select a saved vCenter instance, the wizard connects to the vCenter instance using the saved credentials. Click Check and Save and wait for the connection to complete. Note The connection details are stored in a secret. If you add a provider with an incorrect hostname, user name, or password, click Workloads Secrets and delete the provider secret. Select a virtual machine or a template. Click . In the Review screen, review your settings. Click Edit to update the following settings: General : Description Operating System Flavor Memory CPUs Workload Profile Networking : Name Model Network Type MAC Address Storage : Click the Options menu of the VM disk and select Edit to update the following fields: Name Source : For example, Import Disk . Size Interface Storage Class : Select NFS or ocs-storagecluster-ceph-rbd (ceph-rbd) . If you select ocs-storagecluster-ceph-rbd , you must set the Volume Mode of the disk to Block . Other storage classes might work, but they are not officially supported. Advanced Volume Mode : Select Block . Advanced Access Mode Advanced Cloud-init : Form : Enter the Hostname and Authenticated SSH Keys . Custom script : Enter the cloud-init script in the text field. Advanced Virtual Hardware : You can attach a virtual CD-ROM to the imported virtual machine. Click Import or Review and Import , if you have edited the import settings. A Successfully created virtual machine message and a list of resources created for the virtual machine are displayed. The virtual machine appears in Workloads Virtual Machines . Virtual machine wizard fields Name Parameter Description Name The name can contain lowercase letters ( a-z ), numbers ( 0-9 ), and hyphens ( - ), up to a maximum of 253 characters. The first and last characters must be alphanumeric. The name must not contain uppercase letters, spaces, periods ( . ), or special characters. Description Optional description field. Operating System The primary operating system that is selected for the virtual machine in the template. You cannot edit this field when creating a virtual machine from a template. Boot Source Import via URL (creates PVC) Import content from an image available from an HTTP or HTTPS endpoint. Example: Obtaining a URL link from the web page with the operating system image. Clone existing PVC (creates PVC) Select an existent persistent volume claim available on the cluster and clone it. Import via Registry (creates PVC) Provision virtual machine from a bootable operating system container located in a registry accessible from the cluster. Example: kubevirt/cirros-registry-disk-demo . PXE (network boot - adds network interface) Boot an operating system from a server on the network. Requires a PXE bootable network attachment definition. Persistent Volume Claim project Project name that you want to use for cloning the PVC. Persistent Volume Claim name PVC name that should apply to this virtual machine template if you are cloning an existing PVC. Mount this as a CD-ROM boot source A CD-ROM requires an additional disk for installing the operating system. Select the checkbox to add a disk and customize it later. Flavor Tiny, Small, Medium, Large, Custom Presets that determine the amount of CPU and memory allocated to the virtual machine. The presets displayed for Flavor are determined by the operating system. Workload Type Desktop A virtual machine configuration for use on a desktop. Ideal for consumption on a small scale. Recommended for use with the web console. Server Balances performance and it is compatible with a wide range of server workloads. High-Performance A virtual machine configuration that is optimized for high-performance workloads. Start this virtual machine after creation. This checkbox is selected by default and the virtual machine starts running after creation. Clear the checkbox if you do not want the virtual machine to start when it is created. Cloud-init fields Name Description Hostname Sets a specific hostname for the virtual machine. Authorized SSH Keys The user's public key that is copied to ~/.ssh/authorized_keys on the virtual machine. Custom script Replaces other options with a field in which you paste a custom cloud-init script. Networking fields Name Description Name Name for the network interface controller. Model Indicates the model of the network interface controller. Supported values are e1000e and virtio . Network List of available network attachment definitions. Type List of available binding methods. For the default pod network, masquerade is the only recommended binding method. For secondary networks, use the bridge binding method. The masquerade method is not supported for non-default networks. Select SR-IOV if you configured an SR-IOV network device and defined that network in the namespace. MAC Address MAC address for the network interface controller. If a MAC address is not specified, one is assigned automatically. Storage fields Name Selection Description Source Blank (creates PVC) Create an empty disk. Import via URL (creates PVC) Import content via URL (HTTP or HTTPS endpoint). Use an existing PVC Use a PVC that is already available in the cluster. Clone existing PVC (creates PVC) Select an existing PVC available in the cluster and clone it. Import via Registry (creates PVC) Import content via container registry. Container (ephemeral) Upload content from a container located in a registry accessible from the cluster. The container disk should be used only for read-only filesystems such as CD-ROMs or temporary virtual machines. Name Name of the disk. The name can contain lowercase letters ( a-z ), numbers ( 0-9 ), hyphens ( - ), and periods ( . ), up to a maximum of 253 characters. The first and last characters must be alphanumeric. The name must not contain uppercase letters, spaces, or special characters. Size Size of the disk in GiB. Type Type of disk. Example: Disk or CD-ROM Interface Type of disk device. Supported interfaces are virtIO , SATA , and SCSI . Storage Class The storage class that is used to create the disk. Advanced Volume Mode Defines whether the persistent volume uses a formatted file system or raw block state. Default is Filesystem . Advanced Access Mode Access mode of the persistent volume. Supported access modes are Single User (RWO) , Shared Access (RWX) , and Read Only (ROX) . Advanced storage settings The following advanced storage settings are available for Blank , Import via URL , and Clone existing PVC disks. These parameters are optional. If you do not specify these parameters, the system uses the default values from the kubevirt-storage-class-defaults config map. Name Parameter Description Volume Mode Filesystem Stores the virtual disk on a file system-based volume. Block Stores the virtual disk directly on the block volume. Only use Block if the underlying storage supports it. Access Mode Single User (RWO) The disk can be mounted as read/write by a single node. Shared Access (RWX) The disk can be mounted as read/write by many nodes. Note This is required for some features, such as live migration of virtual machines between nodes. Read Only (ROX) The disk can be mounted as read-only by many nodes. 7.15.5.3.1. Updating the NIC name of an imported virtual machine You must update the NIC name of a virtual machine that you imported from VMware to conform to OpenShift Virtualization naming conventions. Procedure Log in to the virtual machine. Navigate to the /etc/sysconfig/network-scripts directory. Rename the network configuration file: USD mv vmnic0 ifcfg-eth0 1 1 The first network configuration file is named ifcfg-eth0 . Additional network configuration files are numbered sequentially, for example, ifcfg-eth1 , ifcfg-eth2 . Update the NAME and DEVICE parameters in the network configuration file: NAME=eth0 DEVICE=eth0 Restart the network: USD systemctl restart network 7.15.5.4. Troubleshooting a virtual machine import 7.15.5.4.1. Logs You can check the V2V Conversion pod log for errors. Procedure View the V2V Conversion pod name by running the following command: USD oc get pods -n <namespace> | grep v2v 1 1 Specify the namespace of your imported virtual machine. Example output kubevirt-v2v-conversion-f66f7d-zqkz7 1/1 Running 0 4h49m View the V2V Conversion pod log by running the following command: USD oc logs <kubevirt-v2v-conversion-f66f7d-zqkz7> -f -n <namespace> 1 1 Specify the VM Conversion pod name and the namespace. 7.15.5.4.2. Error messages The following error messages might appear: If the VMware VM is not shut down before import, the imported virtual machine displays the error message, Readiness probe failed in the OpenShift Container Platform console and the V2V Conversion pod log displays the following error message: INFO - have error: ('virt-v2v error: internal error: invalid argument: libvirt domain 'v2v_migration_vm_1' is running or paused. It must be shut down in order to perform virt-v2v conversion',)" The following error message is displayed in the OpenShift Container Platform console if a non-admin user tries to import a VM: Could not load config map vmware-to-kubevirt-os in kube-public namespace Restricted Access: configmaps "vmware-to-kubevirt-os" is forbidden: User cannot get resource "configmaps" in API group "" in the namespace "kube-public" Only an admin user can import a VM. 7.16. Cloning virtual machines 7.16.1. Enabling user permissions to clone data volumes across namespaces The isolating nature of namespaces means that users cannot by default clone resources between namespaces. To enable a user to clone a virtual machine to another namespace, a user with the cluster-admin role must create a new cluster role. Bind this cluster role to a user to enable them to clone virtual machines to the destination namespace. 7.16.1.1. Prerequisites Only a user with the cluster-admin role can create cluster roles. 7.16.1.2. About data volumes DataVolume objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared. 7.16.1.3. Creating RBAC resources for cloning data volumes Create a new cluster role that enables permissions for all actions for the datavolumes resource. Procedure Create a ClusterRole manifest: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: <datavolume-cloner> 1 rules: - apiGroups: ["cdi.kubevirt.io"] resources: ["datavolumes/source"] verbs: ["*"] 1 Unique name for the cluster role. Create the cluster role in the cluster: USD oc create -f <datavolume-cloner.yaml> 1 1 The file name of the ClusterRole manifest created in the step. Create a RoleBinding manifest that applies to both the source and destination namespaces and references the cluster role created in the step. apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: <allow-clone-to-user> 1 namespace: <Source namespace> 2 subjects: - kind: ServiceAccount name: default namespace: <Destination namespace> 3 roleRef: kind: ClusterRole name: datavolume-cloner 4 apiGroup: rbac.authorization.k8s.io 1 Unique name for the role binding. 2 The namespace for the source data volume. 3 The namespace to which the data volume is cloned. 4 The name of the cluster role created in the step. Create the role binding in the cluster: USD oc create -f <datavolume-cloner.yaml> 1 1 The file name of the RoleBinding manifest created in the step. 7.16.2. Cloning a virtual machine disk into a new data volume You can clone the persistent volume claim (PVC) of a virtual machine disk into a new data volume by referencing the source PVC in your data volume configuration file. Warning Cloning operations between different volume modes are not supported. The volumeMode values must match in both the source and target specifications. For example, if you attempt to clone from a persistent volume (PV) with volumeMode: Block to a PV with volumeMode: Filesystem , the operation fails with an error message. 7.16.2.1. Prerequisites Users need additional permissions to clone the PVC of a virtual machine disk into another namespace. 7.16.2.2. About data volumes DataVolume objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared. 7.16.2.3. Cloning the persistent volume claim of a virtual machine disk into a new data volume You can clone a persistent volume claim (PVC) of an existing virtual machine disk into a new data volume. The new data volume can then be used for a new virtual machine. Note When a data volume is created independently of a virtual machine, the lifecycle of the data volume is independent of the virtual machine. If the virtual machine is deleted, neither the data volume nor its associated PVC is deleted. Prerequisites Determine the PVC of an existing virtual machine disk to use. You must power down the virtual machine that is associated with the PVC before you can clone it. Install the OpenShift CLI ( oc ). Procedure Examine the virtual machine disk you want to clone to identify the name and namespace of the associated PVC. Create a YAML file for a data volume that specifies the name of the new data volume, the name and namespace of the source PVC, and the size of the new data volume. For example: apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <cloner-datavolume> 1 spec: source: pvc: namespace: "<source-namespace>" 2 name: "<my-favorite-vm-disk>" 3 pvc: accessModes: - ReadWriteOnce resources: requests: storage: <2Gi> 4 1 The name of the new data volume. 2 The namespace where the source PVC exists. 3 The name of the source PVC. 4 The size of the new data volume. You must allocate enough space, or the cloning operation fails. The size must be the same as or larger than the source PVC. Start cloning the PVC by creating the data volume: USD oc create -f <cloner-datavolume>.yaml Note Data volumes prevent a virtual machine from starting before the PVC is prepared, so you can create a virtual machine that references the new data volume while the PVC clones. 7.16.2.4. Template: Data volume clone configuration file example-clone-dv.yaml apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: "example-clone-dv" spec: source: pvc: name: source-pvc namespace: example-ns pvc: accessModes: - ReadWriteOnce resources: requests: storage: "1G" 7.16.2.5. CDI supported operations matrix This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space. Content types HTTP HTTPS HTTP basic auth Registry Upload KubeVirt(QCOW2) [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2** [✓] GZ* [✓] XZ* [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2* □ GZ □ XZ [✓] QCOW2* [✓] GZ* [✓] XZ* KubeVirt (RAW) [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW* □ GZ □ XZ [✓] RAW* [✓] GZ* [✓] XZ* [✓] Supported operation □ Unsupported operation * Requires scratch space ** Requires scratch space if a custom certificate authority is required 7.16.3. Cloning a virtual machine by using a data volume template You can create a new virtual machine by cloning the persistent volume claim (PVC) of an existing VM. By including a dataVolumeTemplate in your virtual machine configuration file, you create a new data volume from the original PVC. Warning Cloning operations between different volume modes are not supported. The volumeMode values must match in both the source and target specifications. For example, if you attempt to clone from a persistent volume (PV) with volumeMode: Block to a PV with volumeMode: Filesystem , the operation fails with an error message. 7.16.3.1. Prerequisites Users need additional permissions to clone the PVC of a virtual machine disk into another namespace. 7.16.3.2. About data volumes DataVolume objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared. 7.16.3.3. Creating a new virtual machine from a cloned persistent volume claim by using a data volume template You can create a virtual machine that clones the persistent volume claim (PVC) of an existing virtual machine into a data volume. Reference a dataVolumeTemplate in the virtual machine manifest and the source PVC is cloned to a data volume, which is then automatically used for the creation of the virtual machine. Note When a data volume is created as part of the data volume template of a virtual machine, the lifecycle of the data volume is then dependent on the virtual machine. If the virtual machine is deleted, the data volume and associated PVC are also deleted. Prerequisites Determine the PVC of an existing virtual machine disk to use. You must power down the virtual machine that is associated with the PVC before you can clone it. Install the OpenShift CLI ( oc ). Procedure Examine the virtual machine you want to clone to identify the name and namespace of the associated PVC. Create a YAML file for a VirtualMachine object. The following virtual machine example clones my-favorite-vm-disk , which is located in the source-namespace namespace. The 2Gi data volume called favorite-clone is created from my-favorite-vm-disk . For example: apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: labels: kubevirt.io/vm: vm-dv-clone name: vm-dv-clone 1 spec: running: false template: metadata: labels: kubevirt.io/vm: vm-dv-clone spec: domain: devices: disks: - disk: bus: virtio name: root-disk resources: requests: memory: 64M volumes: - dataVolume: name: favorite-clone name: root-disk dataVolumeTemplates: - metadata: name: favorite-clone spec: pvc: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi source: pvc: namespace: "source-namespace" name: "my-favorite-vm-disk" 1 The virtual machine to create. Create the virtual machine with the PVC-cloned data volume: USD oc create -f <vm-clone-datavolumetemplate>.yaml 7.16.3.4. Template: Data volume virtual machine configuration file example-dv-vm.yaml apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: labels: kubevirt.io/vm: example-vm name: example-vm spec: dataVolumeTemplates: - metadata: name: example-dv spec: pvc: accessModes: - ReadWriteOnce resources: requests: storage: 1G source: http: url: "" 1 running: false template: metadata: labels: kubevirt.io/vm: example-vm spec: domain: cpu: cores: 1 devices: disks: - disk: bus: virtio name: example-dv-disk machine: type: q35 resources: requests: memory: 1G terminationGracePeriodSeconds: 0 volumes: - dataVolume: name: example-dv name: example-dv-disk 1 The HTTP source of the image you want to import, if applicable. 7.16.3.5. CDI supported operations matrix This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space. Content types HTTP HTTPS HTTP basic auth Registry Upload KubeVirt(QCOW2) [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2** [✓] GZ* [✓] XZ* [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2* □ GZ □ XZ [✓] QCOW2* [✓] GZ* [✓] XZ* KubeVirt (RAW) [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW* □ GZ □ XZ [✓] RAW* [✓] GZ* [✓] XZ* [✓] Supported operation □ Unsupported operation * Requires scratch space ** Requires scratch space if a custom certificate authority is required 7.16.4. Cloning a virtual machine disk into a new block storage data volume You can clone the persistent volume claim (PVC) of a virtual machine disk into a new block data volume by referencing the source PVC in your data volume configuration file. Warning Cloning operations between different volume modes are not supported. The volumeMode values must match in both the source and target specifications. For example, if you attempt to clone from a persistent volume (PV) with volumeMode: Block to a PV with volumeMode: Filesystem , the operation fails with an error message. 7.16.4.1. Prerequisites Users need additional permissions to clone the PVC of a virtual machine disk into another namespace. 7.16.4.2. About data volumes DataVolume objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared. 7.16.4.3. About block persistent volumes A block persistent volume (PV) is a PV that is backed by a raw block device. These volumes do not have a file system and can provide performance benefits for virtual machines by reducing overhead. Raw block volumes are provisioned by specifying volumeMode: Block in the PV and persistent volume claim (PVC) specification. 7.16.4.4. Creating a local block persistent volume Create a local block persistent volume (PV) on a node by populating a file and mounting it as a loop device. You can then reference this loop device in a PV manifest as a Block volume and use it as a block device for a virtual machine image. Procedure Log in as root to the node on which to create the local PV. This procedure uses node01 for its examples. Create a file and populate it with null characters so that it can be used as a block device. The following example creates a file loop10 with a size of 2Gb (20 100Mb blocks): USD dd if=/dev/zero of=<loop10> bs=100M count=20 Mount the loop10 file as a loop device. USD losetup </dev/loop10>d3 <loop10> 1 2 1 File path where the loop device is mounted. 2 The file created in the step to be mounted as the loop device. Create a PersistentVolume manifest that references the mounted loop device. kind: PersistentVolume apiVersion: v1 metadata: name: <local-block-pv10> annotations: spec: local: path: </dev/loop10> 1 capacity: storage: <2Gi> volumeMode: Block 2 storageClassName: local 3 accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - <node01> 4 1 The path of the loop device on the node. 2 Specifies it is a block PV. 3 Optional: Set a storage class for the PV. If you omit it, the cluster default is used. 4 The node on which the block device was mounted. Create the block PV. # oc create -f <local-block-pv10.yaml> 1 1 The file name of the persistent volume created in the step. 7.16.4.5. Cloning the persistent volume claim of a virtual machine disk into a new data volume You can clone a persistent volume claim (PVC) of an existing virtual machine disk into a new data volume. The new data volume can then be used for a new virtual machine. Note When a data volume is created independently of a virtual machine, the lifecycle of the data volume is independent of the virtual machine. If the virtual machine is deleted, neither the data volume nor its associated PVC is deleted. Prerequisites Determine the PVC of an existing virtual machine disk to use. You must power down the virtual machine that is associated with the PVC before you can clone it. Install the OpenShift CLI ( oc ). At least one available block persistent volume (PV) that is the same size as or larger than the source PVC. Procedure Examine the virtual machine disk you want to clone to identify the name and namespace of the associated PVC. Create a YAML file for a data volume that specifies the name of the new data volume, the name and namespace of the source PVC, volumeMode: Block so that an available block PV is used, and the size of the new data volume. For example: apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <cloner-datavolume> 1 spec: source: pvc: namespace: "<source-namespace>" 2 name: "<my-favorite-vm-disk>" 3 pvc: accessModes: - ReadWriteOnce resources: requests: storage: <2Gi> 4 volumeMode: Block 5 1 The name of the new data volume. 2 The namespace where the source PVC exists. 3 The name of the source PVC. 4 The size of the new data volume. You must allocate enough space, or the cloning operation fails. The size must be the same as or larger than the source PVC. 5 Specifies that the destination is a block PV Start cloning the PVC by creating the data volume: USD oc create -f <cloner-datavolume>.yaml Note Data volumes prevent a virtual machine from starting before the PVC is prepared, so you can create a virtual machine that references the new data volume while the PVC clones. 7.16.4.6. CDI supported operations matrix This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space. Content types HTTP HTTPS HTTP basic auth Registry Upload KubeVirt(QCOW2) [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2** [✓] GZ* [✓] XZ* [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2* □ GZ □ XZ [✓] QCOW2* [✓] GZ* [✓] XZ* KubeVirt (RAW) [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW* □ GZ □ XZ [✓] RAW* [✓] GZ* [✓] XZ* [✓] Supported operation □ Unsupported operation * Requires scratch space ** Requires scratch space if a custom certificate authority is required 7.17. Virtual machine networking 7.17.1. Configuring the virtual machine for the default pod network You can connect a virtual machine to the default internal pod network by configuring its network interface to use the masquerade binding mode. Note The KubeMacPool component provides a MAC address pool service for virtual machine NICs in designated namespaces. It is not enabled by default. Enable a MAC address pool in a namespace by applying the KubeMacPool label to that namespace. 7.17.1.1. Configuring masquerade mode from the command line You can use masquerade mode to hide a virtual machine's outgoing traffic behind the pod IP address. Masquerade mode uses Network Address Translation (NAT) to connect virtual machines to the pod network backend through a Linux bridge. Enable masquerade mode and allow traffic to enter the virtual machine by editing your virtual machine configuration file. Prerequisites The virtual machine must be configured to use DHCP to acquire IPv4 addresses. The examples below are configured to use DHCP. Procedure Edit the interfaces spec of your virtual machine configuration file: kind: VirtualMachine spec: domain: devices: interfaces: - name: default masquerade: {} 1 ports: - port: 80 2 networks: - name: default pod: {} 1 Connect using masquerade mode. 2 Optional: List the ports that you want to expose from the virtual machine, each specified by the port field. The port value must be a number between 0 and 65536. When the ports array is not used, all ports in the valid range are open to incoming traffic. In this example, incoming traffic is allowed on port 80 . Note Ports 49152 and 49153 are reserved for use by the libvirt platform and all other incoming traffic to these ports is dropped. Create the virtual machine: USD oc create -f <vm-name>.yaml 7.17.1.2. Creating a service from a virtual machine Create a service from a running virtual machine by first creating a Service object to expose the virtual machine. The ClusterIP service type exposes the virtual machine internally, within the cluster. The NodePort or LoadBalancer service types expose the virtual machine externally, outside of the cluster. This procedure presents an example of how to create, connect to, and expose a Service object of type: ClusterIP as a virtual machine-backed service. Note ClusterIP is the default service type , if the service type is not specified. Procedure Edit the virtual machine YAML as follows: apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: vm-ephemeral namespace: example-namespace spec: running: false template: metadata: labels: special: key 1 spec: domain: devices: disks: - name: containerdisk disk: bus: virtio - name: cloudinitdisk disk: bus: virtio interfaces: - masquerade: {} name: default resources: requests: memory: 1024M networks: - name: default pod: {} volumes: - name: containerdisk containerDisk: image: kubevirt/fedora-cloud-container-disk-demo - name: cloudinitdisk cloudInitNoCloud: userData: | #!/bin/bash echo "fedora" | passwd fedora --stdin 1 Add the label special: key in the spec.template.metadata.labels section. Note Labels on a virtual machine are passed through to the pod. The labels on the VirtualMachine configuration, for example special: key , must match the labels in the Service YAML selector attribute, which you create later in this procedure. Save the virtual machine YAML to apply your changes. Edit the Service YAML to configure the settings necessary to create and expose the Service object: apiVersion: v1 kind: Service metadata: name: vmservice 1 namespace: example-namespace 2 spec: ports: - port: 27017 protocol: TCP targetPort: 22 3 selector: special: key 4 type: ClusterIP 5 1 Specify the name of the service you are creating and exposing. 2 Specify namespace in the metadata section of the Service YAML that corresponds to the namespace you specify in the virtual machine YAML. 3 Add targetPort: 22 , exposing the service on SSH port 22 . 4 In the spec section of the Service YAML, add special: key to the selector attribute, which corresponds to the labels you added in the virtual machine YAML configuration file. 5 In the spec section of the Service YAML, add type: ClusterIP for a ClusterIP service. To create and expose other types of services externally, outside of the cluster, such as NodePort and LoadBalancer , replace type: ClusterIP with type: NodePort or type: LoadBalancer , as appropriate. Save the Service YAML to store the service configuration. Create the ClusterIP service: USD oc create -f <service_name>.yaml Start the virtual machine. If the virtual machine is already running, restart it. Query the Service object to verify it is available and is configured with type ClusterIP . Verification Run the oc get service command, specifying the namespace that you reference in the virtual machine and Service YAML files. USD oc get service -n example-namespace Example output NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE vmservice ClusterIP 172.30.3.149 <none> 27017/TCP 2m As shown from the output, vmservice is running. The TYPE displays as ClusterIP , as you specified in the Service YAML. Establish a connection to the virtual machine that you want to use to back your service. Connect from an object inside the cluster, such as another virtual machine. Edit the virtual machine YAML as follows: apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: vm-connect namespace: example-namespace spec: running: false template: spec: domain: devices: disks: - name: containerdisk disk: bus: virtio - name: cloudinitdisk disk: bus: virtio interfaces: - masquerade: {} name: default resources: requests: memory: 1024M networks: - name: default pod: {} volumes: - name: containerdisk containerDisk: image: kubevirt/fedora-cloud-container-disk-demo - name: cloudinitdisk cloudInitNoCloud: userData: | #!/bin/bash echo "fedora" | passwd fedora --stdin Run the oc create command to create a second virtual machine, where file.yaml is the name of the virtual machine YAML: USD oc create -f <file.yaml> Start the virtual machine. Connect to the virtual machine by running the following virtctl command: USD virtctl -n example-namespace console <new-vm-name> Note For service type LoadBalancer , use the vinagre client to connect your virtual machine by using the public IP and port. External ports are dynamically allocated when using service type LoadBalancer . Run the ssh command to authenticate the connection, where 172.30.3.149 is the ClusterIP of the service and fedora is the user name of the virtual machine: USD ssh fedora@172.30.3.149 -p 27017 Verification You receive the command prompt of the virtual machine backing the service you want to expose. You now have a service backed by a running virtual machine. 7.17.2. Attaching a virtual machine to a Linux bridge network By default, OpenShift Virtualization is installed with a single, internal pod network. You must create a Linux bridge network attachment definition (NAD) in order to connect to additional networks. To attach a virtual machine to an additional network: Create a Linux bridge node network configuration policy. Create a Linux bridge network attachment definition. Configure the virtual machine, enabling the virtual machine to recognize the network attachment definition. For more information about scheduling, interface types, and other node networking activities, see the node networking section. 7.17.2.1. Connecting to the network through the network attachment definition 7.17.2.1.1. Creating a Linux bridge node network configuration policy Use a NodeNetworkConfigurationPolicy manifest YAML file to create the Linux bridge. Procedure Create the NodeNetworkConfigurationPolicy manifest. This example includes sample values that you must replace with your own information. apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: br1-eth1-policy 1 spec: desiredState: interfaces: - name: br1 2 description: Linux bridge with eth1 as a port 3 type: linux-bridge 4 state: up 5 ipv4: enabled: false 6 bridge: options: stp: enabled: false 7 port: - name: eth1 8 1 Name of the policy. 2 Name of the interface. 3 Optional: Human-readable description of the interface. 4 The type of interface. This example creates a bridge. 5 The requested state for the interface after creation. 6 Disables IPv4 in this example. 7 Disables STP in this example. 8 The node NIC to which the bridge is attached. 7.17.2.2. Creating a Linux bridge network attachment definition 7.17.2.2.1. Prerequisites A Linux bridge must be configured and attached on every node. See the node networking section for more information. Warning Configuring IP address management (IPAM) in a network attachment definition for virtual machines is not supported. 7.17.2.2.2. Creating a Linux bridge network attachment definition in the web console The network attachment definition is a custom resource that exposes layer-2 devices to a specific namespace in your OpenShift Virtualization cluster. Network administrators can create network attachment definitions to provide existing layer-2 networking to pods and virtual machines. Procedure In the web console, click Networking Network Attachment Definitions . Click Create Network Attachment Definition . Note The network attachment definition must be in the same namespace as the pod or virtual machine. Enter a unique Name and optional Description . Click the Network Type list and select CNV Linux bridge . Enter the name of the bridge in the Bridge Name field. Optional: If the resource has VLAN IDs configured, enter the ID numbers in the VLAN Tag Number field. Optional: Select MAC Spoof Check to enable MAC spoof filtering. This feature provides security against a MAC spoofing attack by allowing only a single MAC address to exit the pod. Click Create . Note A Linux bridge network attachment definition is the most efficient method for connecting a virtual machine to a VLAN. 7.17.2.2.3. Creating a Linux bridge network attachment definition in the CLI As a network administrator, you can configure a network attachment definition of type cnv-bridge to provide layer-2 networking to pods and virtual machines. Prerequisites The node must support nftables and the nft binary must be deployed to enable MAC spoof check. Procedure Create a network attachment definition in the same namespace as the virtual machine. Add the virtual machine to the network attachment definition, as in the following example: apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: <bridge-network> 1 annotations: k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/<bridge-interface> 2 spec: config: '{ "cniVersion": "0.3.1", "name": "<bridge-network>", 3 "type": "cnv-bridge", 4 "bridge": "<bridge-interface>", 5 "macspoofchk": true, 6 "vlan": 1 7 }' 1 The name for the NetworkAttachmentDefinition object. 2 Optional: Annotation key-value pair for node selection, where bridge-interface must match the name of a bridge configured on some nodes. If you add this annotation to your network attachment definition, your virtual machine instances will only run on the nodes that have the bridge-interface bridge connected. 3 The name for the configuration. It is recommended to match the configuration name to the name value of the network attachment definition. 4 The actual name of the Container Network Interface (CNI) plug-in that provides the network for this network attachment definition. Do not change this field unless you want to use a different CNI. 5 The name of the Linux bridge configured on the node. 6 Optional: Flag to enable MAC spoof check. When set to true , you cannot change the MAC address of the pod or guest interface. This attribute provides security against a MAC spoofing attack by allowing only a single MAC address to exit the pod. 7 Optional: The VLAN tag. No additional VLAN configuration is required on the node network configuration policy. Note A Linux bridge network attachment definition is the most efficient method for connecting a virtual machine to a VLAN. Create the network attachment definition: USD oc create -f <network-attachment-definition.yaml> 1 1 Where <network-attachment-definition.yaml> is the file name of the network attachment definition manifest. Verification Verify that the network attachment definition was created by running the following command: USD oc get network-attachment-definition <bridge-network> 7.17.2.3. Configuring the virtual machine for a Linux bridge network 7.17.2.3.1. Creating a NIC for a virtual machine in the web console Create and attach additional NICs to a virtual machine from the web console. Procedure In the correct project in the OpenShift Virtualization console, click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview screen. Click Network Interfaces to display the NICs already attached to the virtual machine. Click Add Network Interface to create a new slot in the list. Use the Network drop-down list to select the network attachment definition for the additional network. Fill in the Name , Model , Type , and MAC Address for the new NIC. Click Add to save and attach the NIC to the virtual machine. 7.17.2.3.2. Networking fields Name Description Name Name for the network interface controller. Model Indicates the model of the network interface controller. Supported values are e1000e and virtio . Network List of available network attachment definitions. Type List of available binding methods. For the default pod network, masquerade is the only recommended binding method. For secondary networks, use the bridge binding method. The masquerade method is not supported for non-default networks. Select SR-IOV if you configured an SR-IOV network device and defined that network in the namespace. MAC Address MAC address for the network interface controller. If a MAC address is not specified, one is assigned automatically. 7.17.2.3.3. Attaching a virtual machine to an additional network in the CLI Attach a virtual machine to an additional network by adding a bridge interface and specifying a network attachment definition in the virtual machine configuration. This procedure uses a YAML file to demonstrate editing the configuration and applying the updated file to the cluster. You can alternatively use the oc edit <object> <name> command to edit an existing virtual machine. Prerequisites Shut down the virtual machine before editing the configuration. If you edit a running virtual machine, you must restart the virtual machine for the changes to take effect. Procedure Create or edit a configuration of a virtual machine that you want to connect to the bridge network. Add the bridge interface to the spec.template.spec.domain.devices.interfaces list and the network attachment definition to the spec.template.spec.networks list. This example adds a bridge interface called bridge-net that connects to the a-bridge-network network attachment definition: apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: <example-vm> spec: template: spec: domain: devices: interfaces: - masquerade: {} name: <default> - bridge: {} name: <bridge-net> 1 ... networks: - name: <default> pod: {} - name: <bridge-net> 2 multus: networkName: <a-bridge-network> 3 ... 1 The name of the bridge interface. 2 The name of the network. This value must match the name value of the corresponding spec.template.spec.domain.devices.interfaces entry. 3 The name of the network attachment definition, prefixed by the namespace where it exists. The namespace must be either the default namespace or the same namespace where the VM is to be created. In this case, multus is used. Multus is a cloud network interface (CNI) plug-in that allows multiple CNIs to exist so that a pod or virtual machine can use the interfaces it needs. Apply the configuration: USD oc apply -f <example-vm.yaml> Optional: If you edited a running virtual machine, you must restart it for the changes to take effect. 7.17.3. Configuring IP addresses for virtual machines You can configure either dynamically or statically provisioned IP addresses for virtual machines. Prerequisites The virtual machine must connect to an external network . You must have a DHCP server available on the additional network to configure a dynamic IP for the virtual machine. 7.17.3.1. Configuring an IP address for a new virtual machine using cloud-init You can use cloud-init to configure an IP address when you create a virtual machine. The IP address can be dynamically or statically provisioned. Procedure Create a virtual machine configuration and include the cloud-init network details in the spec.volumes.cloudInitNoCloud.networkData field of the virtual machine configuration: To configure a dynamic IP, specify the interface name and the dhcp4 boolean: kind: VirtualMachine spec: ... volumes: - cloudInitNoCloud: networkData: | version: 2 ethernets: eth1: 1 dhcp4: true 2 1 The interface name. 2 Uses DHCP to provision an IPv4 address. To configure a static IP, specify the interface name and the IP address: kind: VirtualMachine spec: ... volumes: - cloudInitNoCloud: networkData: | version: 2 ethernets: eth1: 1 addresses: - 10.10.10.14/24 2 1 The interface name. 2 The static IP address for the virtual machine. 7.17.4. Configuring an SR-IOV network device for virtual machines You can configure a Single Root I/O Virtualization (SR-IOV) device for virtual machines in your cluster. This process is similar but not identical to configuring an SR-IOV device for OpenShift Container Platform. Note Live migration is not supported for virtual machines that are attached to an SR-IOV network interface. 7.17.4.1. Prerequisites You must have installed the SR-IOV Operator. You must have configured the SR-IOV Operator. 7.17.4.2. Automated discovery of SR-IOV network devices The SR-IOV Network Operator searches your cluster for SR-IOV capable network devices on worker nodes. The Operator creates and updates a SriovNetworkNodeState custom resource (CR) for each worker node that provides a compatible SR-IOV network device. The CR is assigned the same name as the worker node. The status.interfaces list provides information about the network devices on a node. Important Do not modify a SriovNetworkNodeState object. The Operator creates and manages these resources automatically. 7.17.4.2.1. Example SriovNetworkNodeState object The following YAML is an example of a SriovNetworkNodeState object created by the SR-IOV Network Operator: An SriovNetworkNodeState object apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodeState metadata: name: node-25 1 namespace: openshift-sriov-network-operator ownerReferences: - apiVersion: sriovnetwork.openshift.io/v1 blockOwnerDeletion: true controller: true kind: SriovNetworkNodePolicy name: default spec: dpConfigVersion: "39824" status: interfaces: 2 - deviceID: "1017" driver: mlx5_core mtu: 1500 name: ens785f0 pciAddress: "0000:18:00.0" totalvfs: 8 vendor: 15b3 - deviceID: "1017" driver: mlx5_core mtu: 1500 name: ens785f1 pciAddress: "0000:18:00.1" totalvfs: 8 vendor: 15b3 - deviceID: 158b driver: i40e mtu: 1500 name: ens817f0 pciAddress: 0000:81:00.0 totalvfs: 64 vendor: "8086" - deviceID: 158b driver: i40e mtu: 1500 name: ens817f1 pciAddress: 0000:81:00.1 totalvfs: 64 vendor: "8086" - deviceID: 158b driver: i40e mtu: 1500 name: ens803f0 pciAddress: 0000:86:00.0 totalvfs: 64 vendor: "8086" syncStatus: Succeeded 1 The value of the name field is the same as the name of the worker node. 2 The interfaces stanza includes a list of all of the SR-IOV devices discovered by the Operator on the worker node. 7.17.4.3. Configuring SR-IOV network devices The SR-IOV Network Operator adds the SriovNetworkNodePolicy.sriovnetwork.openshift.io CustomResourceDefinition to OpenShift Container Platform. You can configure an SR-IOV network device by creating a SriovNetworkNodePolicy custom resource (CR). Note When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator might drain the nodes, and in some cases, reboot nodes. It might take several minutes for a configuration change to apply. Prerequisites You installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-admin role. You have installed the SR-IOV Network Operator. You have enough available nodes in your cluster to handle the evicted workload from drained nodes. You have not selected any control plane nodes for SR-IOV network device configuration. Procedure Create an SriovNetworkNodePolicy object, and then save the YAML in the <name>-sriov-node-network.yaml file. Replace <name> with the name for this configuration. apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: name: <name> 1 namespace: openshift-sriov-network-operator 2 spec: resourceName: <sriov_resource_name> 3 nodeSelector: feature.node.kubernetes.io/network-sriov.capable: "true" 4 priority: <priority> 5 mtu: <mtu> 6 numVfs: <num> 7 nicSelector: 8 vendor: "<vendor_code>" 9 deviceID: "<device_id>" 10 pfNames: ["<pf_name>", ...] 11 rootDevices: ["<pci_bus_id>", "..."] 12 deviceType: vfio-pci 13 isRdma: false 14 1 Specify a name for the CR object. 2 Specify the namespace where the SR-IOV Operator is installed. 3 Specify the resource name of the SR-IOV device plug-in. You can create multiple SriovNetworkNodePolicy objects for a resource name. 4 Specify the node selector to select which nodes are configured. Only SR-IOV network devices on selected nodes are configured. The SR-IOV Container Network Interface (CNI) plug-in and device plug-in are deployed only on selected nodes. 5 Optional: Specify an integer value between 0 and 99 . A smaller number gets higher priority, so a priority of 10 is higher than a priority of 99 . The default value is 99 . 6 Optional: Specify a value for the maximum transmission unit (MTU) of the virtual function. The maximum MTU value can vary for different NIC models. 7 Specify the number of the virtual functions (VF) to create for the SR-IOV physical network device. For an Intel network interface controller (NIC), the number of VFs cannot be larger than the total VFs supported by the device. For a Mellanox NIC, the number of VFs cannot be larger than 128 . 8 The nicSelector mapping selects the Ethernet device for the Operator to configure. You do not need to specify values for all the parameters. It is recommended to identify the Ethernet adapter with enough precision to minimize the possibility of selecting an Ethernet device unintentionally. If you specify rootDevices , you must also specify a value for vendor , deviceID , or pfNames . If you specify both pfNames and rootDevices at the same time, ensure that they point to an identical device. 9 Optional: Specify the vendor hex code of the SR-IOV network device. The only allowed values are either 8086 or 15b3 . 10 Optional: Specify the device hex code of SR-IOV network device. The only allowed values are 158b , 1015 , 1017 . 11 Optional: The parameter accepts an array of one or more physical function (PF) names for the Ethernet device. 12 The parameter accepts an array of one or more PCI bus addresses for the physical function of the Ethernet device. Provide the address in the following format: 0000:02:00.1 . 13 The vfio-pci driver type is required for virtual functions in OpenShift Virtualization. 14 Optional: Specify whether to enable remote direct memory access (RDMA) mode. For a Mellanox card, set isRdma to false . The default value is false . Note If isRDMA flag is set to true , you can continue to use the RDMA enabled VF as a normal network device. A device can be used in either mode. Optional: Label the SR-IOV capable cluster nodes with SriovNetworkNodePolicy.Spec.NodeSelector if they are not already labeled. For more information about labeling nodes, see "Understanding how to update labels on nodes". Create the SriovNetworkNodePolicy object: USD oc create -f <name>-sriov-node-network.yaml where <name> specifies the name for this configuration. After applying the configuration update, all the pods in sriov-network-operator namespace transition to the Running status. To verify that the SR-IOV network device is configured, enter the following command. Replace <node_name> with the name of a node with the SR-IOV network device that you just configured. USD oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}' 7.17.4.4. steps Configuring an SR-IOV network attachment for virtual machines 7.17.5. Defining an SR-IOV network You can create a network attachment for a Single Root I/O Virtualization (SR-IOV) device for virtual machines. After the network is defined, you can attach virtual machines to the SR-IOV network. 7.17.5.1. Prerequisites You must have configured an SR-IOV device for virtual machines . 7.17.5.2. Configuring SR-IOV additional network You can configure an additional network that uses SR-IOV hardware by creating a SriovNetwork object. When you create a SriovNetwork object, the SR-IOV Operator automatically creates a NetworkAttachmentDefinition object. Users can then attach virtual machines to the SR-IOV network by specifying the network in the virtual machine configurations. Note Do not modify or delete a SriovNetwork object if it is attached to any pods or virtual machines in the running state. Prerequisites Install the OpenShift CLI ( oc ). Log in as a user with cluster-admin privileges. Procedure Create the following SriovNetwork object, and then save the YAML in the <name>-sriov-network.yaml file. Replace <name> with a name for this additional network. apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetwork metadata: name: <name> 1 namespace: openshift-sriov-network-operator 2 spec: resourceName: <sriov_resource_name> 3 networkNamespace: <target_namespace> 4 vlan: <vlan> 5 spoofChk: "<spoof_check>" 6 linkState: <link_state> 7 maxTxRate: <max_tx_rate> 8 minTxRate: <min_rx_rate> 9 vlanQoS: <vlan_qos> 10 trust: "<trust_vf>" 11 capabilities: <capabilities> 12 1 Replace <name> with a name for the object. The SR-IOV Network Operator creates a NetworkAttachmentDefinition object with same name. 2 Specify the namespace where the SR-IOV Network Operator is installed. 3 Replace <sriov_resource_name> with the value for the .spec.resourceName parameter from the SriovNetworkNodePolicy object that defines the SR-IOV hardware for this additional network. 4 Replace <target_namespace> with the target namespace for the SriovNetwork. Only pods or virtual machines in the target namespace can attach to the SriovNetwork. 5 Optional: Replace <vlan> with a Virtual LAN (VLAN) ID for the additional network. The integer value must be from 0 to 4095 . The default value is 0 . 6 Optional: Replace <spoof_check> with the spoof check mode of the VF. The allowed values are the strings "on" and "off" . Important You must enclose the value you specify in quotes or the CR is rejected by the SR-IOV Network Operator. 7 Optional: Replace <link_state> with the link state of virtual function (VF). Allowed value are enable , disable and auto . 8 Optional: Replace <max_tx_rate> with a maximum transmission rate, in Mbps, for the VF. 9 Optional: Replace <min_tx_rate> with a minimum transmission rate, in Mbps, for the VF. This value should always be less than or equal to Maximum transmission rate. Note Intel NICs do not support the minTxRate parameter. For more information, see BZ#1772847 . 10 Optional: Replace <vlan_qos> with an IEEE 802.1p priority level for the VF. The default value is 0 . 11 Optional: Replace <trust_vf> with the trust mode of the VF. The allowed values are the strings "on" and "off" . Important You must enclose the value you specify in quotes or the CR is rejected by the SR-IOV Network Operator. 12 Optional: Replace <capabilities> with the capabilities to configure for this network. To create the object, enter the following command. Replace <name> with a name for this additional network. USD oc create -f <name>-sriov-network.yaml Optional: To confirm that the NetworkAttachmentDefinition object associated with the SriovNetwork object that you created in the step exists, enter the following command. Replace <namespace> with the namespace you specified in the SriovNetwork object. USD oc get net-attach-def -n <namespace> 7.17.5.3. steps Attaching a virtual machine to an SR-IOV network. 7.17.6. Attaching a virtual machine to an SR-IOV network You can attach a virtual machine to use a Single Root I/O Virtualization (SR-IOV) network as a secondary network. 7.17.6.1. Prerequisites You must have configured an SR-IOV device for virtual machines . You must have defined an SR-IOV network . 7.17.6.2. Attaching a virtual machine to an SR-IOV network You can attach the virtual machine to the SR-IOV network by including the network details in the virtual machine configuration. Procedure Include the SR-IOV network details in the spec.domain.devices.interfaces and spec.networks of the virtual machine configuration: kind: VirtualMachine ... spec: domain: devices: interfaces: - name: <default> 1 masquerade: {} 2 - name: <nic1> 3 sriov: {} networks: - name: <default> 4 pod: {} - name: <nic1> 5 multus: networkName: <sriov-network> 6 ... 1 A unique name for the interface that is connected to the pod network. 2 The masquerade binding to the default pod network. 3 A unique name for the SR-IOV interface. 4 The name of the pod network interface. This must be the same as the interfaces.name that you defined earlier. 5 The name of the SR-IOV interface. This must be the same as the interfaces.name that you defined earlier. 6 The name of the SR-IOV network attachment definition. Apply the virtual machine configuration: USD oc apply -f <vm-sriov.yaml> 1 1 The name of the virtual machine YAML file. 7.17.7. Viewing the IP address of NICs on a virtual machine You can view the IP address for a network interface controller (NIC) by using the web console or the oc client. The QEMU guest agent displays additional information about the virtual machine's secondary networks. 7.17.7.1. Viewing the IP address of a virtual machine interface in the CLI The network interface configuration is included in the oc describe vmi <vmi_name> command. You can also view the IP address information by running ip addr on the virtual machine, or by running oc get vmi <vmi_name> -o yaml . Procedure Use the oc describe command to display the virtual machine interface configuration: USD oc describe vmi <vmi_name> Example output ... Interfaces: Interface Name: eth0 Ip Address: 10.244.0.37/24 Ip Addresses: 10.244.0.37/24 fe80::858:aff:fef4:25/64 Mac: 0a:58:0a:f4:00:25 Name: default Interface Name: v2 Ip Address: 1.1.1.7/24 Ip Addresses: 1.1.1.7/24 fe80::f4d9:70ff:fe13:9089/64 Mac: f6:d9:70:13:90:89 Interface Name: v1 Ip Address: 1.1.1.1/24 Ip Addresses: 1.1.1.1/24 1.1.1.2/24 1.1.1.4/24 2001:de7:0:f101::1/64 2001:db8:0:f101::1/64 fe80::1420:84ff:fe10:17aa/64 Mac: 16:20:84:10:17:aa 7.17.7.2. Viewing the IP address of a virtual machine interface in the web console The IP information displays in the Virtual Machine Overview screen for the virtual machine. Procedure In the OpenShift Virtualization console, click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine name to open the Virtual Machine Overview screen. The information for each attached NIC is displayed under IP Address . 7.17.8. Using a MAC address pool for virtual machines The KubeMacPool component provides a MAC address pool service for virtual machine NICs in designated namespaces. Enable a MAC address pool in a namespace by applying the KubeMacPool label to that namespace. 7.17.8.1. About KubeMacPool If you enable the KubeMacPool component for a namespace, virtual machine NICs in that namespace are allocated MAC addresses from a MAC address pool. This ensures that the NIC is assigned a unique MAC address that does not conflict with the MAC address of another virtual machine. Virtual machine instances created from that virtual machine retain the assigned MAC address across reboots. Note KubeMacPool does not handle virtual machine instances created independently from a virtual machine. KubeMacPool is disabled by default. Enable a MAC address pool for a namespace by applying the KubeMacPool label to the namespace. 7.17.8.2. Enabling a MAC address pool for a namespace in the CLI Enable a MAC address pool for virtual machines in a namespace by applying the mutatevirtualmachines.kubemacpool.io=allocate label to the namespace. Procedure Add the KubeMacPool label to the namespace. The following example adds the KubeMacPool label to two namespaces, <namespace1> and <namespace2> : USD oc label namespace <namespace1> <namespace2> mutatevirtualmachines.kubemacpool.io=allocate 7.17.8.3. Disabling a MAC address pool for a namespace in the CLI Disable a MAC address pool for virtual machines in a namespace by removing the mutatevirtualmachines.kubemacpool.io label. Procedure Remove the KubeMacPool label from the namespace. The following example removes the KubeMacPool label from two namespaces, <namespace1> and <namespace2> : USD oc label namespace <namespace1> <namespace2> mutatevirtualmachines.kubemacpool.io- 7.18. Virtual machine disks 7.18.1. Storage features Use the following table to determine feature availability for local and shared persistent storage in OpenShift Virtualization. 7.18.1.1. OpenShift Virtualization storage feature matrix Table 7.6. OpenShift Virtualization storage feature matrix Virtual machine live migration Host-assisted virtual machine disk cloning Storage-assisted virtual machine disk cloning Virtual machine snapshots OpenShift Container Storage: RBD block-mode volumes Yes Yes Yes Yes OpenShift Virtualization hostpath provisioner No Yes No No Other multi-node writable storage Yes [1] Yes Yes [2] Yes [2] Other single-node writable storage No Yes Yes [2] Yes [2] PVCs must request a ReadWriteMany access mode. Storage provider must support both Kubernetes and CSI snapshot APIs Note You cannot live migrate virtual machines that use: A storage class with ReadWriteOnce (RWO) access mode Passthrough features such as SR-IOV and GPU Do not set the evictionStrategy field to LiveMigrate for these virtual machines. 7.18.2. Configuring local storage for virtual machines You can configure local storage for your virtual machines by using the hostpath provisioner feature. 7.18.2.1. About the hostpath provisioner The hostpath provisioner is a local storage provisioner designed for OpenShift Virtualization. If you want to configure local storage for virtual machines, you must enable the hostpath provisioner first. When you install the OpenShift Virtualization Operator, the hostpath provisioner Operator is automatically installed. To use it, you must: Configure SELinux: If you use Red Hat Enterprise Linux CoreOS (RHCOS) 8 workers, you must create a MachineConfig object on each node. Otherwise, apply the SELinux label container_file_t to the persistent volume (PV) backing directory on each node. Create a HostPathProvisioner custom resource. Create a StorageClass object for the hostpath provisioner. The hostpath provisioner Operator deploys the provisioner as a DaemonSet on each node when you create its custom resource. In the custom resource file, you specify the backing directory for the persistent volumes that the hostpath provisioner creates. 7.18.2.2. Configuring SELinux for the hostpath provisioner on Red Hat Enterprise Linux CoreOS (RHCOS) 8 You must configure SELinux before you create the HostPathProvisioner custom resource. To configure SELinux on Red Hat Enterprise Linux CoreOS (RHCOS) 8 workers, you must create a MachineConfig object on each node. Prerequisites Create a backing directory on each node for the persistent volumes (PVs) that the hostpath provisioner creates. Important The backing directory must not be located in the filesystem's root directory because the / partition is read-only on RHCOS. For example, you can use /var/<directory_name> but not /<directory_name> . Procedure Create the MachineConfig file. For example: USD touch machineconfig.yaml Edit the file, ensuring that you include the directory where you want the hostpath provisioner to create PVs. For example: apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: name: 50-set-selinux-for-hostpath-provisioner labels: machineconfiguration.openshift.io/role: worker spec: config: ignition: version: 3.2.0 systemd: units: - contents: | [Unit] Description=Set SELinux chcon for hostpath provisioner Before=kubelet.service [Service] ExecStart=/usr/bin/chcon -Rt container_file_t <backing_directory_path> 1 [Install] WantedBy=multi-user.target enabled: true name: hostpath-provisioner.service 1 Specify the backing directory where you want the provisioner to create PVs. This directory must not be located in the filesystem's root directory ( / ). Create the MachineConfig object: USD oc create -f machineconfig.yaml -n <namespace> 7.18.2.3. Using the hostpath provisioner to enable local storage To deploy the hostpath provisioner and enable your virtual machines to use local storage, first create a HostPathProvisioner custom resource. Prerequisites Create a backing directory on each node for the persistent volumes (PVs) that the hostpath provisioner creates. Important The backing directory must not be located in the filesystem's root directory because the / partition is read-only on Red Hat Enterprise Linux CoreOS (RHCOS). For example, you can use /var/<directory_name> but not /<directory_name> . Apply the SELinux context container_file_t to the PV backing directory on each node. For example: USD sudo chcon -t container_file_t -R <backing_directory_path> Note If you use Red Hat Enterprise Linux CoreOS (RHCOS) 8 workers, you must configure SELinux by using a MachineConfig manifest instead. Procedure Create the HostPathProvisioner custom resource file. For example: USD touch hostpathprovisioner_cr.yaml Edit the file, ensuring that the spec.pathConfig.path value is the directory where you want the hostpath provisioner to create PVs. For example: apiVersion: hostpathprovisioner.kubevirt.io/v1beta1 kind: HostPathProvisioner metadata: name: hostpath-provisioner spec: imagePullPolicy: IfNotPresent pathConfig: path: "<backing_directory_path>" 1 useNamingPrefix: false 2 workload: 3 1 Specify the backing directory where you want the provisioner to create PVs. This directory must not be located in the filesystem's root directory ( / ). 2 Change this value to true if you want to use the name of the persistent volume claim (PVC) that is bound to the created PV as the prefix of the directory name. 3 Optional: You can use the spec.workload field to configure node placement rules for the hostpath provisioner. Note If you did not create the backing directory, the provisioner attempts to create it for you. If you did not apply the container_file_t SELinux context, this can cause Permission denied errors. Create the custom resource in the openshift-cnv namespace: USD oc create -f hostpathprovisioner_cr.yaml -n openshift-cnv Additional resources Specifying nodes for virtualization components 7.18.2.4. Creating a storage class When you create a storage class, you set parameters that affect the dynamic provisioning of persistent volumes (PVs) that belong to that storage class. You cannot update a StorageClass object's parameters after you create it. Important When using OpenShift Virtualization with OpenShift Container Platform Container Storage, specify RBD block mode persistent volume claims (PVCs) when creating virtual machine disks. With virtual machine disks, RBD block mode volumes are more efficient and provide better performance than Ceph FS or RBD filesystem-mode PVCs. To specify RBD block mode PVCs, use the 'ocs-storagecluster-ceph-rbd' storage class and VolumeMode: Block . Procedure Create a YAML file for defining the storage class. For example: USD touch storageclass.yaml Edit the file. For example: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: hostpath-provisioner 1 provisioner: kubevirt.io/hostpath-provisioner reclaimPolicy: Delete 2 volumeBindingMode: WaitForFirstConsumer 3 1 You can optionally rename the storage class by changing this value. 2 The two possible reclaimPolicy values are Delete and Retain . If you do not specify a value, the storage class defaults to Delete . 3 The volumeBindingMode value determines when dynamic provisioning and volume binding occur. Specify WaitForFirstConsumer to delay the binding and provisioning of a PV until after a pod that uses the persistent volume claim (PVC) is created. This ensures that the PV meets the pod's scheduling requirements. Note Virtual machines use data volumes that are based on local PVs. Local PVs are bound to specific nodes. While the disk image is prepared for consumption by the virtual machine, it is possible that the virtual machine cannot be scheduled to the node where the local storage PV was previously pinned. To solve this problem, use the Kubernetes pod scheduler to bind the PVC to a PV on the correct node. By using StorageClass with volumeBindingMode set to WaitForFirstConsumer , the binding and provisioning of the PV is delayed until a Pod is created using the PVC. Create the StorageClass object: USD oc create -f storageclass.yaml Additional resources Storage classes 7.18.3. Reserving PVC space for file system overhead By default, the Containerized Data Importer (CDI) reserves space for file system overhead data in persistent volume claims (PVCs) that use the Filesystem volume mode. You can set the percentage that CDI reserves for this purpose globally and for specific storage classes. 7.18.3.1. How file system overhead affects space for virtual machine disks When you add a virtual machine disk to a persistent volume claim (PVC) that uses the Filesystem volume mode, you must ensure that there is enough space on the PVC for: The virtual machine disk. The space that the Containerized Data Importer (CDI) reserves for file system overhead, such as metadata. By default, CDI reserves 5.5% of the PVC space for overhead, reducing the space available for virtual machine disks by that amount. If a different value works better for your use case, you can configure the overhead value by editing the CDI object. You can change the value globally and you can specify values for specific storage classes. 7.18.3.2. Overriding the default file system overhead value Change the amount of persistent volume claim (PVC) space that the Containerized Data Importer (CDI) reserves for file system overhead by editing the spec.config.filesystemOverhead attribute of the CDI object. Prerequisites Install the OpenShift CLI ( oc ). Procedure Open the CDI object for editing by running the following command: USD oc edit cdi Edit the spec.config.filesystemOverhead fields, populating them with your chosen values: ... spec: config: filesystemOverhead: global: "<new_global_value>" 1 storageClass: <storage_class_name>: "<new_value_for_this_storage_class>" 2 1 The file system overhead percentage that CDI uses across the cluster. For example, global: "0.07" reserves 7% of the PVC for file system overhead. 2 The file system overhead percentage for the specified storage class. For example, mystorageclass: "0.04" changes the default overhead value for PVCs in the mystorageclass storage class to 4%. Save and exit the editor to update the CDI object. Verification View the CDI status and verify your changes by running the following command: USD oc get cdi -o yaml 7.18.4. Configuring CDI to work with namespaces that have a compute resource quota You can use the Containerized Data Importer (CDI) to import, upload, and clone virtual machine disks into namespaces that are subject to CPU and memory resource restrictions. 7.18.4.1. About CPU and memory quotas in a namespace A resource quota , defined by the ResourceQuota object, imposes restrictions on a namespace that limit the total amount of compute resources that can be consumed by resources within that namespace. The CDI object defines the user configuration for the Containerized Data Importer (CDI). The CPU and memory request and limit values for the CDI object are set to a default value of 0. This ensures that pods created by CDI that do not specify compute resource requirements are given the default values and are allowed to run in a namespace that is restricted with a quota. 7.18.4.2. Editing the CDI object to override CPU and memory defaults Modify the default settings for CPU and memory requests and limits for your use case by editing the spec.config.podResourceRequirements fields of the CDI object. Prerequisites Install the OpenShift CLI ( oc ). Procedure Edit the CDI object by running the following command: USD oc edit cdi Change the default CPU and memory requests and limits by editing the spec.config.podResourceRequirements fields of the CDI object: apiVersion: cdi.kubevirt.io/v1beta1 kind: CDI ... spec: config: podResourceRequirements: limits: cpu: "4" memory: "1Gi" requests: cpu: "1" memory: "250Mi" ... Save and exit the editor to update the CDI object. Verification View the status field of the CDI object to verify your changes: USD oc get cdi -o yaml 7.18.4.3. Additional resources Resource quotas per project 7.18.5. Using preallocation for data volumes The Containerized Data Importer can preallocate disk space to improve write performance when creating data volumes. You can enable preallocation globally for the cluster or for specific data volumes. 7.18.5.1. About preallocation The Containerized Data Importer (CDI) can use the QEMU preallocate mode for data volumes to improve write performance. You can use preallocation mode for importing and uploading operations and when creating blank data volumes. If preallocation is enabled, CDI uses the better preallocation method depending on the underlying file system and device type: fallocate If the file system supports it, CDI uses the operating system's fallocate call to preallocate space by using the posix_fallocate function, which allocates blocks and marks them as uninitialized. full If fallocate mode cannot be used, full mode allocates space for the image by writing data to the underlying storage. Depending on the storage location, all the empty allocated space might be zeroed. 7.18.5.2. Enabling preallocation globally You can enable cluster-wide preallocation mode for the Containerized Data Importer (CDI) by adding the preallocation field to the CDI object. Prerequisites Install the OpenShift client ( oc ). Procedure Use the oc client to edit the CDI object: USD oc edit cdi Set the spec.config.preallocation field with a value of true : apiVersion: cdi.kubevirt.io/v1beta1 kind: CDI metadata: ... spec: config: preallocation: true 1 1 The preallocation field is a boolean that defaults to false. Save and exit your editor to update the CDI object and enable preallocation mode. 7.18.5.3. Enabling preallocation for a data volume If preallocation is not globally enabled for the cluster, you can enable it for specific data volumes by including the spec.preallocation field in the data volume manifest. You can enable preallocation mode in either the web console or by using the OpenShift client ( oc ). Preallocation mode is supported for all CDI source types. Procedure Specify the spec.preallocation field in the data volume manifest: apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: preallocated-datavolume spec: source: 1 ... pvc: ... preallocation: true 2 1 All CDI source types support preallocation, however preallocation is ignored for cloning operations. 2 The preallocation field is a boolean that defaults to false. 7.18.6. Uploading local disk images by using the web console You can upload a locally stored disk image file by using the web console. 7.18.6.1. Prerequisites You must have a virtual machine image file in IMG, ISO, or QCOW2 format. If you require scratch space according to the CDI supported operations matrix , you must first define a storage class or prepare CDI scratch space for this operation to complete successfully. 7.18.6.2. CDI supported operations matrix This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space. Content types HTTP HTTPS HTTP basic auth Registry Upload KubeVirt(QCOW2) [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2** [✓] GZ* [✓] XZ* [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2* □ GZ □ XZ [✓] QCOW2* [✓] GZ* [✓] XZ* KubeVirt (RAW) [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW* □ GZ □ XZ [✓] RAW* [✓] GZ* [✓] XZ* [✓] Supported operation □ Unsupported operation * Requires scratch space ** Requires scratch space if a custom certificate authority is required 7.18.6.3. Uploading an image file using the web console Use the web console to upload an image file to a new persistent volume claim (PVC). You can later use this PVC to attach the image to new virtual machines. Prerequisites You must have one of the following: A raw virtual machine image file in either ISO or IMG format. A virtual machine image file in QCOW2 format. For best results, compress your image file according to the following guidelines before you upload it: Compress a raw image file by using xz or gzip . Note Using a compressed raw image file results in the most efficient upload. Compress a QCOW2 image file by using the method that is recommended for your client: If you use a Linux client, sparsify the QCOW2 file by using the virt-sparsify tool. If you use a Windows client, compress the QCOW2 file by using xz or gzip . Procedure From the side menu of the web console, click Storage Persistent Volume Claims . Click the Create Persistent Volume Claim drop-down list to expand it. Click With Data Upload Form to open the Upload Data to Persistent Volume Claim page. Click Browse to open the file manager and select the image that you want to upload, or drag the file into the Drag a file here or browse to upload field. Optional: Set this image as the default image for a specific operating system. Select the Attach this data to a virtual machine operating system check box. Select an operating system from the list. The Persistent Volume Claim Name field is automatically filled with a unique name and cannot be edited. Take note of the name assigned to the PVC so that you can identify it later, if necessary. Select a storage class from the Storage Class list. In the Size field, enter the size value for the PVC. Select the corresponding unit of measurement from the drop-down list. Warning The PVC size must be larger than the size of the uncompressed virtual disk. Select an Access Mode that matches the storage class that you selected. Click Upload . 7.18.6.4. Additional resources Configure preallocation mode to improve write performance for data volume operations. 7.18.7. Uploading local disk images by using the virtctl tool You can upload a locally stored disk image to a new or existing data volume by using the virtctl command-line utility. 7.18.7.1. Prerequisites Install the kubevirt-virtctl package. If you require scratch space according to the CDI supported operations matrix , you must first define a storage class or prepare CDI scratch space for this operation to complete successfully. 7.18.7.2. About data volumes DataVolume objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared. 7.18.7.3. Creating an upload data volume You can manually create a data volume with an upload data source to use for uploading local disk images. Procedure Create a data volume configuration that specifies spec: source: upload{} : apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <upload-datavolume> 1 spec: source: upload: {} pvc: accessModes: - ReadWriteOnce resources: requests: storage: <2Gi> 2 1 The name of the data volume. 2 The size of the data volume. Ensure that this value is greater than or equal to the size of the disk that you upload. Create the data volume by running the following command: USD oc create -f <upload-datavolume>.yaml 7.18.7.4. Uploading a local disk image to a data volume You can use the virtctl CLI utility to upload a local disk image from a client machine to a data volume (DV) in your cluster. You can use a DV that already exists in your cluster or create a new DV during this procedure. Note After you upload a local disk image, you can add it to a virtual machine. Prerequisites You must have one of the following: A raw virtual machine image file in either ISO or IMG format. A virtual machine image file in QCOW2 format. For best results, compress your image file according to the following guidelines before you upload it: Compress a raw image file by using xz or gzip . Note Using a compressed raw image file results in the most efficient upload. Compress a QCOW2 image file by using the method that is recommended for your client: If you use a Linux client, sparsify the QCOW2 file by using the virt-sparsify tool. If you use a Windows client, compress the QCOW2 file by using xz or gzip . The kubevirt-virtctl package must be installed on the client machine. The client machine must be configured to trust the OpenShift Container Platform router's certificate. Procedure Identify the following items: The name of the upload data volume that you want to use. If this data volume does not exist, it is created automatically. The size of the data volume, if you want it to be created during the upload procedure. The size must be greater than or equal to the size of the disk image. The file location of the virtual machine disk image that you want to upload. Upload the disk image by running the virtctl image-upload command. Specify the parameters that you identified in the step. For example: USD virtctl image-upload dv <datavolume_name> \ 1 --size=<datavolume_size> \ 2 --image-path=</path/to/image> \ 3 1 The name of the data volume. 2 The size of the data volume. For example: --size=500Mi , --size=1G 3 The file path of the virtual machine disk image. Note If you do not want to create a new data volume, omit the --size parameter and include the --no-create flag. When uploading a disk image to a PVC, the PVC size must be larger than the size of the uncompressed virtual disk. To allow insecure server connections when using HTTPS, use the --insecure parameter. Be aware that when you use the --insecure flag, the authenticity of the upload endpoint is not verified. Optional. To verify that a data volume was created, view all data volumes by running the following command: USD oc get dvs 7.18.7.5. CDI supported operations matrix This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space. Content types HTTP HTTPS HTTP basic auth Registry Upload KubeVirt(QCOW2) [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2** [✓] GZ* [✓] XZ* [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2* □ GZ □ XZ [✓] QCOW2* [✓] GZ* [✓] XZ* KubeVirt (RAW) [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW* □ GZ □ XZ [✓] RAW* [✓] GZ* [✓] XZ* [✓] Supported operation □ Unsupported operation * Requires scratch space ** Requires scratch space if a custom certificate authority is required 7.18.7.6. Additional resources Configure preallocation mode to improve write performance for data volume operations. 7.18.8. Uploading a local disk image to a block storage data volume You can upload a local disk image into a block data volume by using the virtctl command-line utility. In this workflow, you create a local block device to use as a persistent volume, associate this block volume with an upload data volume, and use virtctl to upload the local disk image into the data volume. 7.18.8.1. Prerequisites Install the kubevirt-virtctl package. If you require scratch space according to the CDI supported operations matrix , you must first define a storage class or prepare CDI scratch space for this operation to complete successfully. 7.18.8.2. About data volumes DataVolume objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared. 7.18.8.3. About block persistent volumes A block persistent volume (PV) is a PV that is backed by a raw block device. These volumes do not have a file system and can provide performance benefits for virtual machines by reducing overhead. Raw block volumes are provisioned by specifying volumeMode: Block in the PV and persistent volume claim (PVC) specification. 7.18.8.4. Creating a local block persistent volume Create a local block persistent volume (PV) on a node by populating a file and mounting it as a loop device. You can then reference this loop device in a PV manifest as a Block volume and use it as a block device for a virtual machine image. Procedure Log in as root to the node on which to create the local PV. This procedure uses node01 for its examples. Create a file and populate it with null characters so that it can be used as a block device. The following example creates a file loop10 with a size of 2Gb (20 100Mb blocks): USD dd if=/dev/zero of=<loop10> bs=100M count=20 Mount the loop10 file as a loop device. USD losetup </dev/loop10>d3 <loop10> 1 2 1 File path where the loop device is mounted. 2 The file created in the step to be mounted as the loop device. Create a PersistentVolume manifest that references the mounted loop device. kind: PersistentVolume apiVersion: v1 metadata: name: <local-block-pv10> annotations: spec: local: path: </dev/loop10> 1 capacity: storage: <2Gi> volumeMode: Block 2 storageClassName: local 3 accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - <node01> 4 1 The path of the loop device on the node. 2 Specifies it is a block PV. 3 Optional: Set a storage class for the PV. If you omit it, the cluster default is used. 4 The node on which the block device was mounted. Create the block PV. # oc create -f <local-block-pv10.yaml> 1 1 The file name of the persistent volume created in the step. 7.18.8.5. Creating an upload data volume You can manually create a data volume with an upload data source to use for uploading local disk images. Procedure Create a data volume configuration that specifies spec: source: upload{} : apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <upload-datavolume> 1 spec: source: upload: {} pvc: accessModes: - ReadWriteOnce resources: requests: storage: <2Gi> 2 1 The name of the data volume. 2 The size of the data volume. Ensure that this value is greater than or equal to the size of the disk that you upload. Create the data volume by running the following command: USD oc create -f <upload-datavolume>.yaml 7.18.8.6. Uploading a local disk image to a data volume You can use the virtctl CLI utility to upload a local disk image from a client machine to a data volume (DV) in your cluster. You can use a DV that already exists in your cluster or create a new DV during this procedure. Note After you upload a local disk image, you can add it to a virtual machine. Prerequisites You must have one of the following: A raw virtual machine image file in either ISO or IMG format. A virtual machine image file in QCOW2 format. For best results, compress your image file according to the following guidelines before you upload it: Compress a raw image file by using xz or gzip . Note Using a compressed raw image file results in the most efficient upload. Compress a QCOW2 image file by using the method that is recommended for your client: If you use a Linux client, sparsify the QCOW2 file by using the virt-sparsify tool. If you use a Windows client, compress the QCOW2 file by using xz or gzip . The kubevirt-virtctl package must be installed on the client machine. The client machine must be configured to trust the OpenShift Container Platform router's certificate. Procedure Identify the following items: The name of the upload data volume that you want to use. If this data volume does not exist, it is created automatically. The size of the data volume, if you want it to be created during the upload procedure. The size must be greater than or equal to the size of the disk image. The file location of the virtual machine disk image that you want to upload. Upload the disk image by running the virtctl image-upload command. Specify the parameters that you identified in the step. For example: USD virtctl image-upload dv <datavolume_name> \ 1 --size=<datavolume_size> \ 2 --image-path=</path/to/image> \ 3 1 The name of the data volume. 2 The size of the data volume. For example: --size=500Mi , --size=1G 3 The file path of the virtual machine disk image. Note If you do not want to create a new data volume, omit the --size parameter and include the --no-create flag. When uploading a disk image to a PVC, the PVC size must be larger than the size of the uncompressed virtual disk. To allow insecure server connections when using HTTPS, use the --insecure parameter. Be aware that when you use the --insecure flag, the authenticity of the upload endpoint is not verified. Optional. To verify that a data volume was created, view all data volumes by running the following command: USD oc get dvs 7.18.8.7. CDI supported operations matrix This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space. Content types HTTP HTTPS HTTP basic auth Registry Upload KubeVirt(QCOW2) [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2** [✓] GZ* [✓] XZ* [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2* □ GZ □ XZ [✓] QCOW2* [✓] GZ* [✓] XZ* KubeVirt (RAW) [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW* □ GZ □ XZ [✓] RAW* [✓] GZ* [✓] XZ* [✓] Supported operation □ Unsupported operation * Requires scratch space ** Requires scratch space if a custom certificate authority is required 7.18.8.8. Additional resources Configure preallocation mode to improve write performance for data volume operations. 7.18.9. Managing offline virtual machine snapshots You can create, restore, and delete virtual machine (VM) snapshots for VMs that are powered off (offline). OpenShift Virtualization supports offline VM snapshots on: Red Hat OpenShift Container Storage Any other storage provider with the Container Storage Interface (CSI) driver that supports the Kubernetes Volume Snapshot API 7.18.9.1. About virtual machine snapshots A snapshot represents the state and data of a virtual machine (VM) at a specific point in time. You can use a snapshot to restore an existing VM to a state (represented by the snapshot) for backup and disaster recovery or to rapidly roll back to a development version. An offline VM snapshot is created from a VM that is powered off (Stopped state). The snapshot stores a copy of each Container Storage Interface (CSI) volume attached to the VM and a copy of the VM specification and metadata. Snapshots cannot be changed after creation. With the offline VM snapshots feature, cluster administrators and application developers can: Create a new snapshot List all snapshots attached to a specific VM Restore a VM from a snapshot Delete an existing VM snapshot 7.18.9.1.1. Virtual machine snapshot controller and custom resource definitions (CRDs) The VM snapshot feature introduces three new API objects defined as CRDs for managing snapshots: VirtualMachineSnapshot : Represents a user request to create a snapshot. It contains information about the current state of the VM. VirtualMachineSnapshotContent : Represents a provisioned resource on the cluster (a snapshot). It is created by the VM snapshot controller and contains references to all resources required to restore the VM. VirtualMachineRestore : Represents a user request to restore a VM from a snapshot. The VM snapshot controller binds a VirtualMachineSnapshotContent object with the VirtualMachineSnapshot object for which it was created, with a one-to-one mapping. 7.18.9.2. Creating an offline virtual machine snapshot in the web console You can create a virtual machine (VM) snapshot by using the web console. Note The VM snapshot only includes disks that meet the following requirements: Must be either a data volume or persistent volume claim Belong to a storage class that supports Container Storage Interface (CSI) volume snapshots If your VM storage includes disks that do not support snapshots, you can either edit them or contact your cluster administrator. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview screen. If the virtual machine is running, click Actions Stop Virtual Machine to power it down. Click the Snapshots tab and then click Take Snapshot . Fill in the Snapshot Name and optional Description fields. Expand Disks included in this Snapshot to see the storage volumes to be included in the snapshot. If your VM has disks that cannot be included in the snapshot and you still wish to proceed, select the I am aware of this warning and wish to proceed checkbox. Click Save . 7.18.9.3. Creating an offline virtual machine snapshot in the CLI You can create a virtual machine (VM) snapshot for an offline VM by creating a VirtualMachineSnapshot object. Prerequisites Ensure that the persistent volume claims (PVCs) are in a storage class that supports Container Storage Interface (CSI) volume snapshots. Install the OpenShift CLI ( oc ). Power down the VM for which you want to create a snapshot. Procedure Create a YAML file to define a VirtualMachineSnapshot object that specifies the name of the new VirtualMachineSnapshot and the name of the source VM. For example: apiVersion: snapshot.kubevirt.io/v1alpha1 kind: VirtualMachineSnapshot metadata: name: my-vmsnapshot 1 spec: source: apiGroup: kubevirt.io kind: VirtualMachine name: my-vm 2 1 The name of the new VirtualMachineSnapshot object. 2 The name of the source VM. Create the VirtualMachineSnapshot resource. The snapshot controller creates a VirtualMachineSnapshotContent object, binds it to the VirtualMachineSnapshot and updates the status and readyToUse fields of the VirtualMachineSnapshot object. USD oc create -f <my-vmsnapshot>.yaml Verification Verify that the VirtualMachineSnapshot object is created and bound with VirtualMachineSnapshotContent . The readyToUse flag must be set to true . USD oc describe vmsnapshot <my-vmsnapshot> Example output apiVersion: snapshot.kubevirt.io/v1alpha1 kind: VirtualMachineSnapshot metadata: creationTimestamp: "2020-09-30T14:41:51Z" finalizers: - snapshot.kubevirt.io/vmsnapshot-protection generation: 5 name: mysnap namespace: default resourceVersion: "3897" selfLink: /apis/snapshot.kubevirt.io/v1alpha1/namespaces/default/virtualmachinesnapshots/my-vmsnapshot uid: 28eedf08-5d6a-42c1-969c-2eda58e2a78d spec: source: apiGroup: kubevirt.io kind: VirtualMachine name: my-vm status: conditions: - lastProbeTime: null lastTransitionTime: "2020-09-30T14:42:03Z" reason: Operation complete status: "False" 1 type: Progressing - lastProbeTime: null lastTransitionTime: "2020-09-30T14:42:03Z" reason: Operation complete status: "True" 2 type: Ready creationTime: "2020-09-30T14:42:03Z" readyToUse: true 3 sourceUID: 355897f3-73a0-4ec4-83d3-3c2df9486f4f virtualMachineSnapshotContentName: vmsnapshot-content-28eedf08-5d6a-42c1-969c-2eda58e2a78d 4 1 The status field of the Progressing condition specifies if the snapshot is still being created. 2 The status field of the Ready condition specifies if the snapshot creation process is complete. 3 Specifies if the snapshot is ready to be used. 4 Specifies that the snapshot is bound to a VirtualMachineSnapshotContent object created by the snapshot controller. Check the spec:volumeBackups property of the VirtualMachineSnapshotContent resource to verify that the expected PVCs are included in the snapshot. 7.18.9.4. Restoring a virtual machine from a snapshot in the web console You can restore a virtual machine (VM) to a configuration represented by a snapshot in the web console. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview screen. If the virtual machine is running, click Actions Stop Virtual Machine to power it down. Click the Snapshots tab. The page displays a list of snapshots associated with the virtual machine. Choose one of the following methods to restore a VM snapshot: For the snapshot that you want to use as the source to restore the VM, click Restore . Select a snapshot to open the Snapshot Details screen and click Actions Restore Virtual Machine Snapshot . In the confirmation pop-up window, click Restore to restore the VM to its configuration represented by the snapshot. 7.18.9.5. Restoring a virtual machine from a snapshot in the CLI You can restore an existing virtual machine (VM) to a configuration by using a VM snapshot . Prerequisites Install the OpenShift CLI ( oc ). Power down the VM you want to restore to a state. Procedure Create a YAML file to define a VirtualMachineRestore object that specifies the name of the VM you want to restore and the name of the snapshot to be used as the source. For example: apiVersion: snapshot.kubevirt.io/v1alpha1 kind: VirtualMachineRestore metadata: name: my-vmrestore 1 spec: target: apiGroup: kubevirt.io kind: VirtualMachine name: my-vm 2 virtualMachineSnapshotName: my-vmsnapshot 3 1 The name of the new VirtualMachineRestore object. 2 The name of the target VM you want to restore. 3 The name of the VirtualMachineSnapshot object to be used as the source. Create the VirtualMachineRestore resource. The snapshot controller updates the status fields of the VirtualMachineRestore object and replaces the existing VM configuration with the snapshot content. USD oc create -f <my-vmrestore>.yaml Verification Verify that the VM is restored to the state represented by the snapshot. The complete flag must be set to true . USD oc get vmrestore <my-vmrestore> Example output apiVersion: snapshot.kubevirt.io/v1alpha1 kind: VirtualMachineRestore metadata: creationTimestamp: "2020-09-30T14:46:27Z" generation: 5 name: my-vmrestore namespace: default ownerReferences: - apiVersion: kubevirt.io/v1alpha3 blockOwnerDeletion: true controller: true kind: VirtualMachine name: my-vm uid: 355897f3-73a0-4ec4-83d3-3c2df9486f4f resourceVersion: "5512" selfLink: /apis/snapshot.kubevirt.io/v1alpha1/namespaces/default/virtualmachinerestores/my-vmrestore uid: 71c679a8-136e-46b0-b9b5-f57175a6a041 spec: target: apiGroup: kubevirt.io kind: VirtualMachine name: my-vm virtualMachineSnapshotName: my-vmsnapshot status: complete: true 1 conditions: - lastProbeTime: null lastTransitionTime: "2020-09-30T14:46:28Z" reason: Operation complete status: "False" 2 type: Progressing - lastProbeTime: null lastTransitionTime: "2020-09-30T14:46:28Z" reason: Operation complete status: "True" 3 type: Ready deletedDataVolumes: - test-dv1 restoreTime: "2020-09-30T14:46:28Z" restores: - dataVolumeName: restore-71c679a8-136e-46b0-b9b5-f57175a6a041-datavolumedisk1 persistentVolumeClaim: restore-71c679a8-136e-46b0-b9b5-f57175a6a041-datavolumedisk1 volumeName: datavolumedisk1 volumeSnapshotName: vmsnapshot-28eedf08-5d6a-42c1-969c-2eda58e2a78d-volume-datavolumedisk1 1 Specifies if the process of restoring the VM to the state represented by the snapshot is complete. 2 The status field of the Progressing condition specifies if the VM is still being restored. 3 The status field of the Ready condition specifies if the VM restoration process is complete. 7.18.9.6. Deleting a virtual machine snapshot in the web console You can delete an existing virtual machine snapshot by using the web console. Procedure Click Workloads Virtualization from the side menu. Click the Virtual Machines tab. Select a virtual machine to open the Virtual Machine Overview screen. Click the Snapshots tab. The page displays a list of snapshots associated with the virtual machine. Choose one of the following methods to delete a virtual machine snapshot: Click the Options menu of the virtual machine snapshot that you want to delete and select Delete Virtual Machine Snapshot . Select a snapshot to open the Snapshot Details screen and click Actions Delete Virtual Machine Snapshot . In the confirmation pop-up window, click Delete to delete the snapshot. 7.18.9.7. Deleting a virtual machine snapshot in the CLI You can delete an existing virtual machine (VM) snapshot by deleting the appropriate VirtualMachineSnapshot object. Prerequisites Install the OpenShift CLI ( oc ). Procedure Delete the VirtualMachineSnapshot object. The snapshot controller deletes the VirtualMachineSnapshot along with the associated VirtualMachineSnapshotContent object. USD oc delete vmsnapshot <my-vmsnapshot> Verification Verify that the snapshot is deleted and no longer attached to this VM: USD oc get vmsnapshot 7.18.9.8. Additional resources CSI Volume Snapshots 7.18.10. Moving a local virtual machine disk to a different node Virtual machines that use local volume storage can be moved so that they run on a specific node. You might want to move the virtual machine to a specific node for the following reasons: The current node has limitations to the local storage configuration. The new node is better optimized for the workload of that virtual machine. To move a virtual machine that uses local storage, you must clone the underlying volume by using a data volume. After the cloning operation is complete, you can edit the virtual machine configuration so that it uses the new data volume, or add the new data volume to another virtual machine . Note Users without the cluster-admin role require additional user permissions to clone volumes across namespaces. 7.18.10.1. Cloning a local volume to another node You can move a virtual machine disk so that it runs on a specific node by cloning the underlying persistent volume claim (PVC). To ensure the virtual machine disk is cloned to the correct node, you must either create a new persistent volume (PV) or identify one on the correct node. Apply a unique label to the PV so that it can be referenced by the data volume. Note The destination PV must be the same size or larger than the source PVC. If the destination PV is smaller than the source PVC, the cloning operation fails. Prerequisites The virtual machine must not be running. Power down the virtual machine before cloning the virtual machine disk. Procedure Either create a new local PV on the node, or identify a local PV already on the node: Create a local PV that includes the nodeAffinity.nodeSelectorTerms parameters. The following manifest creates a 10Gi local PV on node01 . kind: PersistentVolume apiVersion: v1 metadata: name: <destination-pv> 1 annotations: spec: accessModes: - ReadWriteOnce capacity: storage: 10Gi 2 local: path: /mnt/local-storage/local/disk1 3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node01 4 persistentVolumeReclaimPolicy: Delete storageClassName: local volumeMode: Filesystem 1 The name of the PV. 2 The size of the PV. You must allocate enough space, or the cloning operation fails. The size must be the same as or larger than the source PVC. 3 The mount path on the node. 4 The name of the node where you want to create the PV. Identify a PV that already exists on the target node. You can identify the node where a PV is provisioned by viewing the nodeAffinity field in its configuration: USD oc get pv <destination-pv> -o yaml The following snippet shows that the PV is on node01 : Example output ... spec: nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname 1 operator: In values: - node01 2 ... 1 The kubernetes.io/hostname key uses the node hostname to select a node. 2 The hostname of the node. Add a unique label to the PV: USD oc label pv <destination-pv> node=node01 Create a data volume manifest that references the following: The PVC name and namespace of the virtual machine. The label you applied to the PV in the step. The size of the destination PV. apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <clone-datavolume> 1 spec: source: pvc: name: "<source-vm-disk>" 2 namespace: "<source-namespace>" 3 pvc: accessModes: - ReadWriteOnce selector: matchLabels: node: node01 4 resources: requests: storage: <10Gi> 5 1 The name of the new data volume. 2 The name of the source PVC. If you do not know the PVC name, you can find it in the virtual machine configuration: spec.volumes.persistentVolumeClaim.claimName . 3 The namespace where the source PVC exists. 4 The label that you applied to the PV in the step. 5 The size of the destination PV. Start the cloning operation by applying the data volume manifest to your cluster: USD oc apply -f <clone-datavolume.yaml> The data volume clones the PVC of the virtual machine into the PV on the specific node. 7.18.11. Expanding virtual storage by adding blank disk images You can increase your storage capacity or create new data partitions by adding blank disk images to OpenShift Virtualization. 7.18.11.1. About data volumes DataVolume objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared. 7.18.11.2. Creating a blank disk image with data volumes You can create a new blank disk image in a persistent volume claim by customizing and deploying a data volume configuration file. Prerequisites At least one available persistent volume. Install the OpenShift CLI ( oc ). Procedure Edit the data volume configuration file: apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: blank-image-datavolume spec: source: blank: {} pvc: # Optional: Set the storage class or omit to accept the default # storageClassName: "hostpath" accessModes: - ReadWriteOnce resources: requests: storage: 500Mi Create the blank disk image by running the following command: USD oc create -f <blank-image-datavolume>.yaml 7.18.11.3. Template: Data volume configuration file for blank disk images blank-image-datavolume.yaml apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: blank-image-datavolume spec: source: blank: {} pvc: # Optional: Set the storage class or omit to accept the default # storageClassName: "hostpath" accessModes: - ReadWriteOnce resources: requests: storage: 500Mi 7.18.11.4. Additional resources Configure preallocation mode to improve write performance for data volume operations. 7.18.12. Cloning a data volume using smart-cloning Smart-cloning is a built-in feature of OpenShift Container Platform Storage (OCS), designed to enhance performance of the cloning process. Clones created with smart-cloning are faster and more efficient than host-assisted cloning. You do not need to perform any action to enable smart-cloning, but you need to ensure your storage environment is compatible with smart-cloning to use this feature. When you create a data volume with a persistent volume claim (PVC) source, you automatically initiate the cloning process. You always receive a clone of the data volume, if your environment supports smart-cloning or not. However, you will only receive the performance benefits of smart cloning if you storage provider supports smart-cloning. 7.18.12.1. Understanding smart-cloning When a data volume is smart-cloned, the following occurs: A snapshot of the source persistent volume claim (PVC) is created. A PVC is created from the snapshot. The snapshot is deleted. 7.18.12.2. Cloning a data volume Prerequisites For smart-cloning to occur, the following conditions are required. Your storage provider must support snapshots. The source and target PVCs must be defined to the same namespace. The source and target PVCs must be defined to the same storage class. The VolumeSnapshotClass object must reference the storage class defined to both the source and target PVCs. If any of these prerequisites are not met, host-assisted cloning automatically occurs when you create a data volume with a PVC source. Procedure To initiate cloning of a data volume: Create a YAML file for a DataVolume object that specifies the name of the new data volume, the name and namespace of the source PVC, and the size of the new data volume. This example clones a source PVC in block mode, so volumeMode: Block is used: apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <cloner-datavolume> 1 spec: source: pvc: namespace: "<source-namespace>" 2 name: "<my-favorite-vm-disk>" 3 pvc: accessModes: - ReadWriteMany resources: requests: storage: <2Gi> 4 volumeMode: Block 5 1 The name of the new data volume. 2 The namespace where the source PVC exists. 3 The name of the source PVC. 4 The size of the new data volume. You must allocate enough space, or the cloning operation fails. The size must be the same as or larger than the source PVC. 5 Specifies that the destination is a block PV. Start cloning the PVC by creating the data volume: USD oc create -f <cloner-datavolume>.yaml Note Data volumes prevent a virtual machine from starting before the PVC is prepared, so you can create a virtual machine that references the new data volume while the PVC clones. 7.18.12.3. Additional resources Cloning the persistent volume claim of a virtual machine disk into a new data volume Configure preallocation mode to improve write performance for data volume operations. 7.18.13. Storage defaults for data volumes The kubevirt-storage-class-defaults config map provides access mode and volume mode defaults for data volumes. You can edit or add storage class defaults to the config map to create data volumes in the web console that better match the underlying storage. 7.18.13.1. About storage settings for data volumes Data volumes require a defined access mode and volume mode to be created in the web console. These storage settings are configured by default with a ReadWriteOnce access mode and Filesystem volume mode. You can modify these settings by editing the kubevirt-storage-class-defaults config map in the openshift-cnv namespace. You can also add settings for other storage classes to create data volumes in the web console for different storage types. Note You must configure storage settings that are supported by the underlying storage. All data volumes that you create in the web console use the default storage settings unless you specify a storage class that is also defined in the config map. 7.18.13.1.1. Access modes Data volumes support the following access modes: ReadWriteOnce : The volume can be mounted as read-write by a single node. ReadWriteOnce has greater versatility and is the default setting. ReadWriteMany : The volume can be mounted as read-write by many nodes. ReadWriteMany is required for some features, such as live migration of virtual machines between nodes. ReadWriteMany is recommended if the underlying storage supports it. 7.18.13.1.2. Volume modes The volume mode defines if a volume is intended to be used with a formatted file system or to remain in raw block state. Data volumes support the following volume modes: Filesystem : Creates a file system on the data volume. This is the default setting. Block : Creates a block data volume. Only use Block if the underlying storage supports it. 7.18.13.2. Editing the kubevirt-storage-class-defaults config map in the web console Modify the storage settings for data volumes by editing the kubevirt-storage-class-defaults config map in the openshift-cnv namespace. You can also add settings for other storage classes to create data volumes in the web console for different storage types. Note You must configure storage settings that are supported by the underlying storage. Procedure Click Workloads Config Maps from the side menu. In the Project list, select openshift-cnv . Click kubevirt-storage-class-defaults to open the Config Map Overview . Click the YAML tab to display the editable configuration. Update the data values with the storage configuration that is appropriate for your underlying storage: ... data: accessMode: ReadWriteOnce 1 volumeMode: Filesystem 2 <new>.accessMode: ReadWriteMany 3 <new>.volumeMode: Block 4 1 The default accessMode is ReadWriteOnce . 2 The default volumeMode is Filesystem . 3 If you add an access mode for a storage class, replace the <new> part of the parameter with the storage class name. 4 If you add a volume mode for a storage class, replace the <new> part of the parameter with the storage class name. Click Save to update the config map. 7.18.13.3. Editing the kubevirt-storage-class-defaults config map in the CLI Modify the storage settings for data volumes by editing the kubevirt-storage-class-defaults config map in the openshift-cnv namespace. You can also add settings for other storage classes to create data volumes in the web console for different storage types. Note You must configure storage settings that are supported by the underlying storage. Procedure Edit the config map by running the following command: USD oc edit configmap kubevirt-storage-class-defaults -n openshift-cnv Update the data values of the config map: ... data: accessMode: ReadWriteOnce 1 volumeMode: Filesystem 2 <new>.accessMode: ReadWriteMany 3 <new>.volumeMode: Block 4 1 The default accessMode is ReadWriteOnce . 2 The default volumeMode is Filesystem . 3 If you add an access mode for a storage class, replace the <new> part of the parameter with the storage class name. 4 If you add a volume mode for a storage class, replace the <new> part of the parameter with the storage class name. Save and exit the editor to update the config map. 7.18.13.4. Example of multiple storage class defaults The following YAML file is an example of a kubevirt-storage-class-defaults config map that has storage settings configured for two storage classes, migration and block . Ensure that all settings are supported by your underlying storage before you update the config map. kind: ConfigMap apiVersion: v1 metadata: name: kubevirt-storage-class-defaults namespace: openshift-cnv ... data: accessMode: ReadWriteOnce volumeMode: Filesystem nfs-sc.accessMode: ReadWriteMany nfs-sc.volumeMode: Filesystem block-sc.accessMode: ReadWriteMany block-sc.volumeMode: Block 7.18.14. Creating and using boot sources A boot source contains a bootable operating system (OS) and all of the configuration settings for the OS, such as drivers. You use a boot source to create virtual machine templates with specific configurations. These templates can be used to create any number of available virtual machines. Quick Start tours are available in the OpenShift Container Platform web console to assist you in creating a custom boot source, uploading a boot source, and other tasks. Select Quick Starts from the Help menu to view the Quick Start tours. Prerequisites To add a boot source, you must be logged in as a user with the os-images.kubevirt.io:edit RBAC role or as an administrator. You do not need special privileges to create a virtual machine from a template with a boot source attached. Note For information about requirements for Microsoft Windows, refer to Creating installation media for Windows boot sources . 7.18.14.1. Adding a boot source for a virtual machine template A boot source can be configured for any virtual machine template that you want to use for creating virtual machines or custom templates. When virtual machine templates are configured with a boot source, they are labeled Available in the Templates tab. After you add a boot source to a template, you can create a new virtual machine from the template. There are four methods for selecting and adding a boot source in the web console: Upload local file (creates PVC) Import via URL (creates PVC) Clone existing PVC (creates PVC) Import via Registry (creates PVC) Prerequisites To add a boot source, you must be logged in as a user with the os-images.kubevirt.io:edit RBAC role or as an administrator. You do not need special privileges to create a virtual machine from a template with a boot source added. To upload a local file, the operating system image file must exist on your local machine. To import via URL, access to the web server with the operating system image is required. For example: the Red Hat Enterprise Linux web page with images. To clone an existing PVC, access to the project with a PVC is required. To import via registry, access to the container registry is required. Procedure In the OpenShift Virtualization console, click Workloads Virtualization from the side menu. Click the Templates tab. Identify the virtual machine template for which you want to configure a boot source and click Add source . In the Add boot source to template window, click Select boot source , select a method for creating a persistent volume claim (PVC): Upload local file , Import via URL , Clone existing PVC , or Import via Registry . Optional: Click This is a CD-ROM boot source to mount a CD-ROM and use it to install the operating system on to an empty disk. The additional empty disk is automatically created and mounted by OpenShift Virtualization. If the additional disk is not needed, you can remove it when you create the virtual machine. Enter a value for Persistent Volume Claim size to specify the PVC size that is adequate for the uncompressed image and any additional space that is required. Optional: Enter a name for Source provider to associate the name with this template. Advanced: Click Storage class and select the storage class that is used to create the disk. Advanced: Click Access mode and select an access mode for the persistent volume. Supported access modes are: Single User (RWO) , Shared Access (RWX) , and Read Only (ROX) . Advanced: Click Volume mode if you want to select Block instead of the default value Filesystem . Select the appropriate method to save your boot source: Click Save and upload if you uploaded a local file. Click Save and import if you imported content from a URL or the registry. Click Save and clone if you cloned an existing PVC. Your custom virtual machine template with a boot source is listed in the Templates tab, and you can create virtual machines by using this template. 7.18.14.2. Creating a virtual machine from a template with an attached boot source After you add a boot source to a template, you can create a new virtual machine from the template. Procedure In the OpenShift Container Platform web console, click Workloads > Virtualization in the side menu. From the Virtual Machines tab or the Templates tab, click Create and select Virtual Machine with Wizard . In the General step, select an OS from the Operating System list that has the (Source available) label to the OS and version name. The (Source available) label indicates that a boot source is available for this OS. In the Select a template step, select an OS from the Operating System list that has the (Source available) label to the OS and version name. The (Source available) label indicates that a boot source is available for this OS. Click Review and Confirm . Review your virtual machine settings and edit them, if required. Click Create Virtual Machine to create your virtual machine. The Successfully created virtual machine page is displayed. 7.18.14.3. Additional resources Uploading local disk images 7.18.15. Using container disks with virtual machines You can build a virtual machine image into a container disk and store it in your container registry. You can then import the container disk into persistent storage for a virtual machine or attach it directly to the virtual machine for ephemeral storage. Important If you use large container disks, I/O traffic might increase, impacting worker nodes. This can lead to unavailable nodes. You can resolve this by: Pruning DeploymentConfig objects Configuring garbage collection 7.18.15.1. About container disks A container disk is a virtual machine image that is stored as a container image in a container image registry. You can use container disks to deliver the same disk images to multiple virtual machines and to create large numbers of virtual machine clones. A container disk can either be imported into a persistent volume claim (PVC) by using a data volume that is attached to a virtual machine, or attached directly to a virtual machine as an ephemeral containerDisk volume. 7.18.15.1.1. Importing a container disk into a PVC by using a data volume Use the Containerized Data Importer (CDI) to import the container disk into a PVC by using a data volume. You can then attach the data volume to a virtual machine for persistent storage. 7.18.15.1.2. Attaching a container disk to a virtual machine as a containerDisk volume A containerDisk volume is ephemeral. It is discarded when the virtual machine is stopped, restarted, or deleted. When a virtual machine with a containerDisk volume starts, the container image is pulled from the registry and hosted on the node that is hosting the virtual machine. Use containerDisk volumes for read-only file systems such as CD-ROMs or for disposable virtual machines. Important Using containerDisk volumes for read-write file systems is not recommended because the data is temporarily written to local storage on the hosting node. This slows live migration of the virtual machine, such as in the case of node maintenance, because the data must be migrated to the destination node. Additionally, all data is lost if the node loses power or otherwise shuts down unexpectedly. 7.18.15.2. Preparing a container disk for virtual machines You must build a container disk with a virtual machine image and push it to a container registry before it can used with a virtual machine. You can then either import the container disk into a PVC using a data volume and attach it to a virtual machine, or you can attach the container disk directly to a virtual machine as an ephemeral containerDisk volume. The size of a disk image inside a container disk is limited by the maximum layer size of the registry where the container disk is hosted. Note For Red Hat Quay , you can change the maximum layer size by editing the YAML configuration file that is created when Red Hat Quay is first deployed. Prerequisites Install podman if it is not already installed. The virtual machine image must be either QCOW2 or RAW format. Procedure Create a Dockerfile to build the virtual machine image into a container image. The virtual machine image must be owned by QEMU, which has a UID of 107 , and placed in the /disk/ directory inside the container. Permissions for the /disk/ directory must then be set to 0440 . The following example uses the Red Hat Universal Base Image (UBI) to handle these configuration changes in the first stage, and uses the minimal scratch image in the second stage to store the result: USD cat > Dockerfile << EOF FROM registry.access.redhat.com/ubi8/ubi:latest AS builder ADD --chown=107:107 <vm_image>.qcow2 /disk/ 1 RUN chmod 0440 /disk/* FROM scratch COPY --from=builder /disk/* /disk/ EOF 1 Where <vm_image> is the virtual machine image in either QCOW2 or RAW format. To use a remote virtual machine image, replace <vm_image>.qcow2 with the complete url for the remote image. Build and tag the container: USD podman build -t <registry>/<container_disk_name>:latest . Push the container image to the registry: USD podman push <registry>/<container_disk_name>:latest If your container registry does not have TLS you must add it as an insecure registry before you can import container disks into persistent storage. 7.18.15.3. Disabling TLS for a container registry to use as insecure registry You can disable TLS (transport layer security) for a container registry by adding the registry to the cdi-insecure-registries config map. Prerequisites Log in to the cluster as a user with the cluster-admin role. Procedure Add the registry to the cdi-insecure-registries config map in the openshift-cnv namespace. USD oc patch configmap cdi-insecure-registries -n openshift-cnv \ --type merge -p '{"data":{"mykey": "<insecure-registry-host>:5000"}}' 1 1 Replace <insecure-registry-host> with the registry hostname. 7.18.15.4. steps Import the container disk into persistent storage for a virtual machine . Create a virtual machine that uses a containerDisk volume for ephemeral storage. 7.18.16. Preparing CDI scratch space 7.18.16.1. About data volumes DataVolume objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared. 7.18.16.2. Understanding scratch space The Containerized Data Importer (CDI) requires scratch space (temporary storage) to complete some operations, such as importing and uploading virtual machine images. During this process, CDI provisions a scratch space PVC equal to the size of the PVC backing the destination data volume (DV). The scratch space PVC is deleted after the operation completes or aborts. You can define the storage class that is used to bind the scratch space PVC in the spec.config.scratchSpaceStorageClass field of the CDI object. If the defined storage class does not match a storage class in the cluster, then the default storage class defined for the cluster is used. If there is no default storage class defined in the cluster, the storage class used to provision the original DV or PVC is used. Note CDI requires requesting scratch space with a file volume mode, regardless of the PVC backing the origin data volume. If the origin PVC is backed by block volume mode, you must define a storage class capable of provisioning file volume mode PVCs. Manual provisioning If there are no storage classes, CDI uses any PVCs in the project that match the size requirements for the image. If there are no PVCs that match these requirements, the CDI import pod remains in a Pending state until an appropriate PVC is made available or until a timeout function kills the pod. 7.18.16.3. CDI operations that require scratch space Type Reason Registry imports CDI must download the image to a scratch space and extract the layers to find the image file. The image file is then passed to QEMU-IMG for conversion to a raw disk. Upload image QEMU-IMG does not accept input from STDIN. Instead, the image to upload is saved in scratch space before it can be passed to QEMU-IMG for conversion. HTTP imports of archived images QEMU-IMG does not know how to handle the archive formats CDI supports. Instead, the image is unarchived and saved into scratch space before it is passed to QEMU-IMG. HTTP imports of authenticated images QEMU-IMG inadequately handles authentication. Instead, the image is saved to scratch space and authenticated before it is passed to QEMU-IMG. HTTP imports of custom certificates QEMU-IMG inadequately handles custom certificates of HTTPS endpoints. Instead, CDI downloads the image to scratch space before passing the file to QEMU-IMG. 7.18.16.4. Defining a storage class in the CDI object Dynamically provision scratch space for CDI operations by defining a storage class in the CDI object. Prerequisites Install the OpenShift client ( oc ). Procedure Use the oc client to edit the CDI object: USD oc edit cdi Edit the spec.config.scratchSpaceStorageClass field to match a storage class in the cluster. apiVersion: cdi.kubevirt.io/v1beta1 kind: CDI ... spec: config: scratchSpaceStorageClass: "<storage_class>" 1 ... 1 Replace <storage_class> with the storage class that you want to use to provision scratch space for CDI operations in your cluster. Save and exit your default editor to update the CDI object. 7.18.16.5. CDI supported operations matrix This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space. Content types HTTP HTTPS HTTP basic auth Registry Upload KubeVirt(QCOW2) [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2** [✓] GZ* [✓] XZ* [✓] QCOW2 [✓] GZ* [✓] XZ* [✓] QCOW2* □ GZ □ XZ [✓] QCOW2* [✓] GZ* [✓] XZ* KubeVirt (RAW) [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW [✓] GZ [✓] XZ [✓] RAW* □ GZ □ XZ [✓] RAW* [✓] GZ* [✓] XZ* [✓] Supported operation □ Unsupported operation * Requires scratch space ** Requires scratch space if a custom certificate authority is required Additional resources See the Dynamic provisioning section for more information on storage classes and how these are defined in the cluster. 7.18.17. Re-using persistent volumes To re-use a statically provisioned persistent volume (PV), you must first reclaim the volume. This involves deleting the PV so that the storage configuration can be re-used. 7.18.17.1. About reclaiming statically provisioned persistent volumes When you reclaim a persistent volume (PV), you unbind the PV from a persistent volume claim (PVC) and delete the PV. Depending on the underlying storage, you might need to manually delete the shared storage. You can then re-use the PV configuration to create a PV with a different name. Statically provisioned PVs must have a reclaim policy of Retain to be reclaimed. If they do not, the PV enters a failed state when the PVC is unbound from the PV. Important The Recycle reclaim policy is deprecated in OpenShift Container Platform 4. 7.18.17.2. Reclaiming statically provisioned persistent volumes Reclaim a statically provisioned persistent volume (PV) by unbinding the persistent volume claim (PVC) and deleting the PV. You might also need to manually delete the shared storage. Reclaiming a statically provisioned PV is dependent on the underlying storage. This procedure provides a general approach that might need to be customized depending on your storage. Procedure Ensure that the reclaim policy of the PV is set to Retain : Check the reclaim policy of the PV: USD oc get pv <pv_name> -o yaml | grep 'persistentVolumeReclaimPolicy' If the persistentVolumeReclaimPolicy is not set to Retain , edit the reclaim policy with the following command: USD oc patch pv <pv_name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' Ensure that no resources are using the PV: USD oc describe pvc <pvc_name> | grep 'Mounted By:' Remove any resources that use the PVC before continuing. Delete the PVC to release the PV: USD oc delete pvc <pvc_name> Optional: Export the PV configuration to a YAML file. If you manually remove the shared storage later in this procedure, you can refer to this configuration. You can also use spec parameters in this file as the basis to create a new PV with the same storage configuration after you reclaim the PV: USD oc get pv <pv_name> -o yaml > <file_name>.yaml Delete the PV: USD oc delete pv <pv_name> Optional: Depending on the storage type, you might need to remove the contents of the shared storage folder: USD rm -rf <path_to_share_storage> Optional: Create a PV that uses the same storage configuration as the deleted PV. If you exported the reclaimed PV configuration earlier, you can use the spec parameters of that file as the basis for a new PV manifest: Note To avoid possible conflict, it is good practice to give the new PV object a different name than the one that you deleted. USD oc create -f <new_pv_name>.yaml Additional resources Configuring local storage for virtual machines The OpenShift Container Platform Storage documentation has more information on Persistent Storage . 7.18.18. Deleting data volumes You can manually delete a data volume by using the oc command-line interface. Note When you delete a virtual machine, the data volume it uses is automatically deleted. 7.18.18.1. About data volumes DataVolume objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared. 7.18.18.2. Listing all data volumes You can list the data volumes in your cluster by using the oc command-line interface. Procedure List all data volumes by running the following command: USD oc get dvs 7.18.18.3. Deleting a data volume You can delete a data volume by using the oc command-line interface (CLI). Prerequisites Identify the name of the data volume that you want to delete. Procedure Delete the data volume by running the following command: USD oc delete dv <datavolume_name> Note This command only deletes objects that exist in the current project. Specify the -n <project_name> option if the object you want to delete is in a different project or namespace.

Commands:
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: labels: app: <vm_name> 1 name: <vm_name> spec: dataVolumeTemplates: - apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <vm_name> spec: sourceRef: kind: DataSource name: rhel9 namespace: openshift-virtualization-os-images storage: resources: requests: storage: 30Gi running: false template: metadata: labels: kubevirt.io/domain: <vm_name> spec: domain: cpu: cores: 1 sockets: 2 threads: 1 devices: disks: - disk: bus: virtio name: rootdisk - disk: bus: virtio name: cloudinitdisk interfaces: - masquerade: {} name: default rng: {} features: smm: enabled: true firmware: bootloader: efi: {} resources: requests: memory: 8Gi evictionStrategy: LiveMigrate networks: - name: default pod: {} volumes: - dataVolume: name: <vm_name> name: rootdisk - cloudInitNoCloud: userData: |- #cloud-config user: cloud-user password: '<password>' 2 chpasswd: { expire: False } name: cloudinitdisk
oc create -f <vm_manifest_file>.yaml
virtctl start <vm_name>
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine spec: RunStrategy: Always 1 template:
oc edit <object_type> <object_ID>
oc apply <object_type> <object_ID>
oc edit vm example
disks: - bootOrder: 1 1 disk: bus: virtio name: containerdisk - disk: bus: virtio name: cloudinitdisk - cdrom: bus: virtio name: cd-drive-1 interfaces: - boot Order: 2 2 macAddress: '02:96:c4:00:00' masquerade: {} name: default
oc delete vm <vm_name>
oc get vmis
oc delete vmi <vmi_name>
remmina --connect /path/to/console.rdp
virtctl expose vm <fedora-vm> --port=22 --name=fedora-vm-ssh --type=NodePort 1
oc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE fedora-vm-ssh NodePort 127.0.0.1 <none> 22:32551/TCP 6s
ssh username@<node_IP_address> -p 32551
virtctl console <VMI>
virtctl vnc <VMI>
virtctl vnc <VMI> -v 4
oc login -u <user> https://<cluster.example.com>:8443
oc describe vmi <windows-vmi-name>
spec: networks: - name: default pod: {} - multus: networkName: cnv-bridge name: bridge-net status: interfaces: - interfaceName: eth0 ipAddress: 198.51.100.0/24 ipAddresses: 198.51.100.0/24 mac: a0:36:9f:0f:b1:70 name: default - interfaceName: eth1 ipAddress: 192.0.2.0/24 ipAddresses: 192.0.2.0/24 2001:db8::/32 mac: 00:17:a4:77:77:25 name: bridge-net
oc adm cordon <node_name>
oc adm drain <node_name> --force=true
oc delete node <node_name>
oc get vmis
yum install -y qemu-guest-agent
systemctl enable --now qemu-guest-agent
spec: domain: devices: disks: - name: virtiocontainerdisk bootOrder: 2 1 cdrom: bus: sata volumes: - containerDisk: image: container-native-virtualization/virtio-win name: virtiocontainerdisk
oc edit vm <vm-name>
spec: domain: devices: disks: - name: virtiocontainerdisk bootOrder: 2 cdrom: bus: sata volumes: - containerDisk: image: container-native-virtualization/virtio-win name: virtiocontainerdisk
spec: domain: devices: disks: - name: virtiocontainerdisk bootOrder: 2 1 cdrom: bus: sata volumes: - containerDisk: image: container-native-virtualization/virtio-win name: virtiocontainerdisk
oc edit vm <vm-name>
spec: domain: devices: disks: - name: virtiocontainerdisk bootOrder: 2 cdrom: bus: sata volumes: - containerDisk: image: container-native-virtualization/virtio-win name: virtiocontainerdisk
kubevirt_vm: namespace: name: cpu_cores: memory: disks: - name: volume: containerDisk: image: disk: bus:
kubevirt_vm: namespace: default name: vm1 cpu_cores: 1 memory: 64Mi disks: - name: containerdisk volume: containerDisk: image: kubevirt/cirros-container-disk-demo:latest disk: bus: virtio
kubevirt_vm: namespace: default name: vm1 state: running 1 cpu_cores: 1
ansible-playbook create-vm.yaml
(...) TASK [Create my first VM] ************************************************************************ changed: [localhost] PLAY RECAP ******************************************************************************************************** localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ansible-playbook create-vm.yaml
--- - name: Ansible Playbook 1 hosts: localhost connection: local tasks: - name: Create my first VM kubevirt_vm: namespace: default name: vm1 cpu_cores: 1 memory: 64Mi disks: - name: containerdisk volume: containerDisk: image: kubevirt/cirros-container-disk-demo:latest disk: bus: virtio
apiversion: kubevirt.io/v1 kind: VirtualMachineInstance metadata: labels: special: vmi-secureboot name: vmi-secureboot spec: domain: devices: disks: - disk: bus: virtio name: containerdisk features: acpi: {} smm: enabled: true 1 firmware: bootloader: efi: secureBoot: true 2
oc create -f <file_name>.yaml
apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: pxe-net-conf spec: config: '{ "cniVersion": "0.3.1", "name": "pxe-net-conf", "plugins": [ { "type": "cnv-bridge", "bridge": "br1", "vlan": 1 1 }, { "type": "cnv-tuning" 2 } ] }'
oc create -f pxe-net-conf.yaml
interfaces: - masquerade: {} name: default - bridge: {} name: pxe-net macAddress: de:00:00:00:00:de bootOrder: 1
devices: disks: - disk: bus: virtio name: containerdisk bootOrder: 2
networks: - name: default pod: {} - name: pxe-net multus: networkName: pxe-net-conf
oc create -f vmi-pxe-boot.yaml
virtualmachineinstance.kubevirt.io "vmi-pxe-boot" created
oc get vmi vmi-pxe-boot -o yaml | grep -i phase phase: Running
virtctl vnc vmi-pxe-boot
virtctl console vmi-pxe-boot
ip addr
3. eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether de:00:00:00:00:de brd ff:ff:ff:ff:ff:ff
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachineInstance metadata: creationTimestamp: null labels: special: vmi-pxe-boot name: vmi-pxe-boot spec: domain: devices: disks: - disk: bus: virtio name: containerdisk bootOrder: 2 - disk: bus: virtio name: cloudinitdisk interfaces: - masquerade: {} name: default - bridge: {} name: pxe-net macAddress: de:00:00:00:00:de bootOrder: 1 machine: type: "" resources: requests: memory: 1024M networks: - name: default pod: {} - multus: networkName: pxe-net-conf name: pxe-net terminationGracePeriodSeconds: 0 volumes: - name: containerdisk containerDisk: image: kubevirt/fedora-cloud-container-disk-demo - cloudInitNoCloud: userData: | #!/bin/bash echo "fedora" | passwd fedora --stdin name: cloudinitdisk status: {}
kind: VirtualMachine spec: template: domain: resources: requests: memory: 1024M memory: guest: 2048M
oc create -f <file_name>.yaml
kind: VirtualMachine spec: template: domain: resources: overcommitGuestOverhead: true requests: memory: 1024M
oc create -f <file_name>.yaml
kind: VirtualMachine spec: domain: resources: requests: memory: "4Gi" 1 memory: hugepages: pageSize: "1Gi" 2
oc apply -f <virtual_machine>.yaml
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: myvmi spec: domain: cpu: features: - name: apic 1 policy: require 2
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachineInstance metadata: name: myvmi spec: domain: cpu: model: Conroe 1
apiVersion: kubevirt/v1alpha3 kind: VirtualMachineInstance metadata: name: myvmi spec: domain: cpu: model: host-model 1
oc get ns
oc create configmap <configmap-name> --from-file=</path/to/file/ca.pem>
apiVersion: v1 kind: ConfigMap metadata: name: tls-certs data: ca.pem: | -----BEGIN CERTIFICATE----- ... <base64 encoded cert> -----END CERTIFICATE-----
apiVersion: v1 kind: Secret metadata: name: endpoint-secret 1 labels: app: containerized-data-importer type: Opaque data: accessKeyId: "" 2 secretKey: "" 3
oc apply -f endpoint-secret.yaml
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: creationTimestamp: null labels: kubevirt.io/vm: vm-fedora-datavolume name: vm-fedora-datavolume 1 spec: dataVolumeTemplates: - metadata: creationTimestamp: null name: fedora-dv 2 spec: storage: resources: requests: storage: 10Gi storageClassName: local source: http: 3 url: "https://mirror.arizona.edu/fedora/linux/releases/35/Cloud/x86_64/images/Fedora-Cloud-Base-35-1.2.x86_64.qcow2" 4 secretRef: endpoint-secret 5 certConfigMap: "" 6 status: {} running: true template: metadata: creationTimestamp: null labels: kubevirt.io/vm: vm-fedora-datavolume spec: domain: devices: disks: - disk: bus: virtio name: datavolumedisk1 machine: type: "" resources: requests: memory: 1.5Gi terminationGracePeriodSeconds: 60 volumes: - dataVolume: name: fedora-dv name: datavolumedisk1 status: {}
oc create -f vm-fedora-datavolume.yaml
oc get pods
oc describe dv fedora-dv 1
virtctl console vm-fedora-datavolume
dd if=/dev/zero of=<loop10> bs=100M count=20
losetup </dev/loop10>d3 <loop10> 1 2
kind: PersistentVolume apiVersion: v1 metadata: name: <local-block-pv10> annotations: spec: local: path: </dev/loop10> 1 capacity: storage: <2Gi> volumeMode: Block 2 storageClassName: local 3 accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - <node01> 4
oc create -f <local-block-pv10.yaml> 1
apiVersion: v1 kind: Secret metadata: name: endpoint-secret 1 labels: app: containerized-data-importer type: Opaque data: accessKeyId: "" 2 secretKey: "" 3
oc apply -f endpoint-secret.yaml
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: import-pv-datavolume 1 spec: storageClassName: local 2 source: http: url: "https://mirror.arizona.edu/fedora/linux/releases/35/Cloud/x86_64/images/Fedora-Cloud-Base-35-1.2.x86_64.qcow2" 3 secretRef: endpoint-secret 4 storage: volumeMode: Block 5 resources: requests: storage: 10Gi
oc create -f import-pv-datavolume.yaml
openssl s_client -connect <RHV_Manager_FQDN>:443 -showcerts < /dev/null
cat <<EOF | oc create -f - apiVersion: v1 kind: Secret metadata: name: rhv-credentials namespace: default 1 type: Opaque stringData: ovirt: | apiUrl: <api_endpoint> 2 username: ocpadmin@internal password: 3 caCert: | -----BEGIN CERTIFICATE----- 4 -----END CERTIFICATE----- EOF
openssl s_client -connect :443 -showcerts < /dev/null
cat <<EOF | kubectl create -f - apiVersion: v2v.kubevirt.io/v1alpha1 kind: ResourceMapping metadata: name: resourcemapping_example namespace: default spec: ovirt: networkMappings: - source: name: <rhv_logical_network>/<vnic_profile> 1 target: name: <target_network> 2 type: pod storageMappings: 3 - source: name: <rhv_storage_domain> 4 target: name: <target_storage_class> 5 volumeMode: <volume_mode> 6 EOF
cat <<EOF | oc create -f - apiVersion: v2v.kubevirt.io/v1beta1 kind: VirtualMachineImport metadata: name: vm-import namespace: default spec: providerCredentialsSecret: name: rhv-credentials namespace: default resourceMapping: 1 name: resourcemapping-example namespace: default targetVmName: vm_example 2 startVm: true source: ovirt: vm: id: <source_vm_id> 3 name: <source_vm_name> 4 cluster: name: <source_cluster_name> 5 mappings: 6 networkMappings: - source: name: <source_logical_network>/<vnic_profile> 7 target: name: <target_network> 8 type: pod storageMappings: 9 - source: name: <source_storage_domain> 10 target: name: <target_storage_class> 11 accessMode: <volume_access_mode> 12 diskMappings: - source: id: <source_vm_disk_id> 13 target: name: <target_storage_class> 14 EOF
oc get vmimports vm-import -n default
status: conditions: - lastHeartbeatTime: "2020-07-22T08:58:52Z" lastTransitionTime: "2020-07-22T08:58:52Z" message: Validation completed successfully reason: ValidationCompleted status: "True" type: Valid - lastHeartbeatTime: "2020-07-22T08:58:52Z" lastTransitionTime: "2020-07-22T08:58:52Z" message: 'VM specifies IO Threads: 1, VM has NUMA tune mode specified: interleave' reason: MappingRulesVerificationReportedWarnings status: "True" type: MappingRulesVerified - lastHeartbeatTime: "2020-07-22T08:58:56Z" lastTransitionTime: "2020-07-22T08:58:52Z" message: Copying virtual machine disks reason: CopyingDisks status: "True" type: Processing dataVolumes: - name: fedora32-b870c429-11e0-4630-b3df-21da551a48c0 targetVmName: fedora32
<os> <type>rhel_8x64</type> </os>
oc get templates -n openshift --show-labels | tr ',' '\n' | grep os.template.kubevirt.io | sed -r 's#os.template.kubevirt.io/(.*)=.*#\1#g' | sort -u
fedora31 fedora32 rhel8.1 rhel8.2
cat <<EOF | oc create -f - apiVersion: v1 kind: ConfigMap metadata: name: os-configmap namespace: default 1 data: guestos2common: | "Red Hat Enterprise Linux Server": "rhel" "CentOS Linux": "centos" "Fedora": "fedora" "Ubuntu": "ubuntu" "openSUSE": "opensuse" osinfo2common: | "<rhv-operating-system>": "<vm-template>" 2 EOF
cat <<EOF | oc apply -f - apiVersion: v1 kind: ConfigMap metadata: name: os-configmap namespace: default data: osinfo2common: | "other_linux": "fedora31" EOF
oc get cm -n default os-configmap -o yaml
oc patch configmap vm-import-controller-config -n openshift-cnv --patch '{ "data": { "osConfigMap.name": "os-configmap", "osConfigMap.namespace": "default" 1 } }'
oc get pods -n <namespace> | grep import 1
vm-import-controller-f66f7d-zqkz7 1/1 Running 0 4h49m
oc logs <vm-import-controller-f66f7d-zqkz7> -f -n <namespace> 1
Failed to bind volumes: provisioning failed for PVC
oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'
oc get pod -n openshift-image-registry -l docker-registry=default
No resourses found in openshift-image-registry namespace
oc edit configs.imageregistry.operator.openshift.io
storage: pvc: claim:
oc get clusteroperator image-registry
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE image-registry 4.7 True False False 6h50m
oc edit configs.imageregistry/cluster
managementState: Removed
managementState: Managed
oc get nodes
oc debug nodes/<node_name>
sh-4.2# chroot /host
sh-4.2# oc login -u kubeadmin -p <password_from_install_log> https://api-int.<cluster_name>.<base_domain>:6443
sh-4.2# podman login -u kubeadmin -p USD(oc whoami -t) image-registry.openshift-image-registry.svc:5000
Login Succeeded!
sh-4.2# podman pull name.io/image
sh-4.2# podman tag name.io/image image-registry.openshift-image-registry.svc:5000/openshift/image
sh-4.2# podman push image-registry.openshift-image-registry.svc:5000/openshift/image
oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge
HOST=USD(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
podman login -u kubeadmin -p USD(oc whoami -t) --tls-verify=false USDHOST 1
oc create secret tls public-route-tls -n openshift-image-registry --cert=</path/to/tls.crt> --key=</path/to/tls.key>
spec: routes: - name: public-routes hostname: myregistry.mycorp.organization secretName: public-route-tls
oc create configmap registry-cas -n openshift-config --from-file=myregistry.corp.com..5000=/etc/docker/certs.d/myregistry.corp.com:5000/ca.crt --from-file=otherregistry.com=/etc/docker/certs.d/otherregistry.com/ca.crt
oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge
oc create secret generic <pull_secret_name> --from-file=.dockercfg=<path/to/.dockercfg> --type=kubernetes.io/dockercfg
oc create secret generic <pull_secret_name> --from-file=.dockerconfigjson=<path/to/.docker/config.json> --type=kubernetes.io/dockerconfigjson
oc create secret docker-registry <pull_secret_name> --docker-server=<registry_server> --docker-username=<user_name> --docker-password=<password> --docker-email=<email>
oc secrets link default <pull_secret_name> --for=pull
mkdir /tmp/<dir_name> && cd /tmp/<dir_name>
tar -xzf VMware-vix-disklib-<version>.x86_64.tar.gz
cat > Dockerfile <<EOF FROM busybox:latest COPY vmware-vix-disklib-distrib /vmware-vix-disklib-distrib RUN mkdir -p /opt ENTRYPOINT ["cp", "-r", "/vmware-vix-disklib-distrib", "/opt"] EOF
podman build . -t <registry_route_or_server_path>/vddk:<tag> 1
podman push <registry_route_or_server_path>/vddk:<tag>
oc edit configmap v2v-vmware -n openshift-cnv
data: vddk-init-image: <registry_route_or_server_path>/vddk:<tag>
mv vmnic0 ifcfg-eth0 1
NAME=eth0 DEVICE=eth0
systemctl restart network
oc get pods -n <namespace> | grep v2v 1
kubevirt-v2v-conversion-f66f7d-zqkz7 1/1 Running 0 4h49m
oc logs <kubevirt-v2v-conversion-f66f7d-zqkz7> -f -n <namespace> 1
INFO - have error: ('virt-v2v error: internal error: invalid argument: libvirt domain 'v2v_migration_vm_1' is running or paused. It must be shut down in order to perform virt-v2v conversion',)"
Could not load config map vmware-to-kubevirt-os in kube-public namespace Restricted Access: configmaps "vmware-to-kubevirt-os" is forbidden: User cannot get resource "configmaps" in API group "" in the namespace "kube-public"
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: <datavolume-cloner> 1 rules: - apiGroups: ["cdi.kubevirt.io"] resources: ["datavolumes/source"] verbs: ["*"]
oc create -f <datavolume-cloner.yaml> 1
apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: <allow-clone-to-user> 1 namespace: <Source namespace> 2 subjects: - kind: ServiceAccount name: default namespace: <Destination namespace> 3 roleRef: kind: ClusterRole name: datavolume-cloner 4 apiGroup: rbac.authorization.k8s.io
oc create -f <datavolume-cloner.yaml> 1
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <cloner-datavolume> 1 spec: source: pvc: namespace: "<source-namespace>" 2 name: "<my-favorite-vm-disk>" 3 pvc: accessModes: - ReadWriteOnce resources: requests: storage: <2Gi> 4
oc create -f <cloner-datavolume>.yaml
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: "example-clone-dv" spec: source: pvc: name: source-pvc namespace: example-ns pvc: accessModes: - ReadWriteOnce resources: requests: storage: "1G"
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: labels: kubevirt.io/vm: vm-dv-clone name: vm-dv-clone 1 spec: running: false template: metadata: labels: kubevirt.io/vm: vm-dv-clone spec: domain: devices: disks: - disk: bus: virtio name: root-disk resources: requests: memory: 64M volumes: - dataVolume: name: favorite-clone name: root-disk dataVolumeTemplates: - metadata: name: favorite-clone spec: pvc: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi source: pvc: namespace: "source-namespace" name: "my-favorite-vm-disk"
oc create -f <vm-clone-datavolumetemplate>.yaml
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: labels: kubevirt.io/vm: example-vm name: example-vm spec: dataVolumeTemplates: - metadata: name: example-dv spec: pvc: accessModes: - ReadWriteOnce resources: requests: storage: 1G source: http: url: "" 1 running: false template: metadata: labels: kubevirt.io/vm: example-vm spec: domain: cpu: cores: 1 devices: disks: - disk: bus: virtio name: example-dv-disk machine: type: q35 resources: requests: memory: 1G terminationGracePeriodSeconds: 0 volumes: - dataVolume: name: example-dv name: example-dv-disk
dd if=/dev/zero of=<loop10> bs=100M count=20
losetup </dev/loop10>d3 <loop10> 1 2
kind: PersistentVolume apiVersion: v1 metadata: name: <local-block-pv10> annotations: spec: local: path: </dev/loop10> 1 capacity: storage: <2Gi> volumeMode: Block 2 storageClassName: local 3 accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - <node01> 4
oc create -f <local-block-pv10.yaml> 1
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <cloner-datavolume> 1 spec: source: pvc: namespace: "<source-namespace>" 2 name: "<my-favorite-vm-disk>" 3 pvc: accessModes: - ReadWriteOnce resources: requests: storage: <2Gi> 4 volumeMode: Block 5
oc create -f <cloner-datavolume>.yaml
kind: VirtualMachine spec: domain: devices: interfaces: - name: default masquerade: {} 1 ports: - port: 80 2 networks: - name: default pod: {}
oc create -f <vm-name>.yaml
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: vm-ephemeral namespace: example-namespace spec: running: false template: metadata: labels: special: key 1 spec: domain: devices: disks: - name: containerdisk disk: bus: virtio - name: cloudinitdisk disk: bus: virtio interfaces: - masquerade: {} name: default resources: requests: memory: 1024M networks: - name: default pod: {} volumes: - name: containerdisk containerDisk: image: kubevirt/fedora-cloud-container-disk-demo - name: cloudinitdisk cloudInitNoCloud: userData: | #!/bin/bash echo "fedora" | passwd fedora --stdin
apiVersion: v1 kind: Service metadata: name: vmservice 1 namespace: example-namespace 2 spec: ports: - port: 27017 protocol: TCP targetPort: 22 3 selector: special: key 4 type: ClusterIP 5
oc create -f <service_name>.yaml
oc get service -n example-namespace
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE vmservice ClusterIP 172.30.3.149 <none> 27017/TCP 2m
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: vm-connect namespace: example-namespace spec: running: false template: spec: domain: devices: disks: - name: containerdisk disk: bus: virtio - name: cloudinitdisk disk: bus: virtio interfaces: - masquerade: {} name: default resources: requests: memory: 1024M networks: - name: default pod: {} volumes: - name: containerdisk containerDisk: image: kubevirt/fedora-cloud-container-disk-demo - name: cloudinitdisk cloudInitNoCloud: userData: | #!/bin/bash echo "fedora" | passwd fedora --stdin
oc create -f <file.yaml>
virtctl -n example-namespace console <new-vm-name>
ssh fedora@172.30.3.149 -p 27017
apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: br1-eth1-policy 1 spec: desiredState: interfaces: - name: br1 2 description: Linux bridge with eth1 as a port 3 type: linux-bridge 4 state: up 5 ipv4: enabled: false 6 bridge: options: stp: enabled: false 7 port: - name: eth1 8
apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: <bridge-network> 1 annotations: k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/<bridge-interface> 2 spec: config: '{ "cniVersion": "0.3.1", "name": "<bridge-network>", 3 "type": "cnv-bridge", 4 "bridge": "<bridge-interface>", 5 "macspoofchk": true, 6 "vlan": 1 7 }'
oc create -f <network-attachment-definition.yaml> 1
oc get network-attachment-definition <bridge-network>
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: <example-vm> spec: template: spec: domain: devices: interfaces: - masquerade: {} name: <default> - bridge: {} name: <bridge-net> 1 networks: - name: <default> pod: {} - name: <bridge-net> 2 multus: networkName: <a-bridge-network> 3
oc apply -f <example-vm.yaml>
kind: VirtualMachine spec: volumes: - cloudInitNoCloud: networkData: | version: 2 ethernets: eth1: 1 dhcp4: true 2
kind: VirtualMachine spec: volumes: - cloudInitNoCloud: networkData: | version: 2 ethernets: eth1: 1 addresses: - 10.10.10.14/24 2
apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodeState metadata: name: node-25 1 namespace: openshift-sriov-network-operator ownerReferences: - apiVersion: sriovnetwork.openshift.io/v1 blockOwnerDeletion: true controller: true kind: SriovNetworkNodePolicy name: default spec: dpConfigVersion: "39824" status: interfaces: 2 - deviceID: "1017" driver: mlx5_core mtu: 1500 name: ens785f0 pciAddress: "0000:18:00.0" totalvfs: 8 vendor: 15b3 - deviceID: "1017" driver: mlx5_core mtu: 1500 name: ens785f1 pciAddress: "0000:18:00.1" totalvfs: 8 vendor: 15b3 - deviceID: 158b driver: i40e mtu: 1500 name: ens817f0 pciAddress: 0000:81:00.0 totalvfs: 64 vendor: "8086" - deviceID: 158b driver: i40e mtu: 1500 name: ens817f1 pciAddress: 0000:81:00.1 totalvfs: 64 vendor: "8086" - deviceID: 158b driver: i40e mtu: 1500 name: ens803f0 pciAddress: 0000:86:00.0 totalvfs: 64 vendor: "8086" syncStatus: Succeeded
apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: name: <name> 1 namespace: openshift-sriov-network-operator 2 spec: resourceName: <sriov_resource_name> 3 nodeSelector: feature.node.kubernetes.io/network-sriov.capable: "true" 4 priority: <priority> 5 mtu: <mtu> 6 numVfs: <num> 7 nicSelector: 8 vendor: "<vendor_code>" 9 deviceID: "<device_id>" 10 pfNames: ["<pf_name>", ...] 11 rootDevices: ["<pci_bus_id>", "..."] 12 deviceType: vfio-pci 13 isRdma: false 14
oc create -f <name>-sriov-node-network.yaml
oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'
apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetwork metadata: name: <name> 1 namespace: openshift-sriov-network-operator 2 spec: resourceName: <sriov_resource_name> 3 networkNamespace: <target_namespace> 4 vlan: <vlan> 5 spoofChk: "<spoof_check>" 6 linkState: <link_state> 7 maxTxRate: <max_tx_rate> 8 minTxRate: <min_rx_rate> 9 vlanQoS: <vlan_qos> 10 trust: "<trust_vf>" 11 capabilities: <capabilities> 12
oc create -f <name>-sriov-network.yaml
oc get net-attach-def -n <namespace>
kind: VirtualMachine spec: domain: devices: interfaces: - name: <default> 1 masquerade: {} 2 - name: <nic1> 3 sriov: {} networks: - name: <default> 4 pod: {} - name: <nic1> 5 multus: networkName: <sriov-network> 6
oc apply -f <vm-sriov.yaml> 1
oc describe vmi <vmi_name>
Interfaces: Interface Name: eth0 Ip Address: 10.244.0.37/24 Ip Addresses: 10.244.0.37/24 fe80::858:aff:fef4:25/64 Mac: 0a:58:0a:f4:00:25 Name: default Interface Name: v2 Ip Address: 1.1.1.7/24 Ip Addresses: 1.1.1.7/24 fe80::f4d9:70ff:fe13:9089/64 Mac: f6:d9:70:13:90:89 Interface Name: v1 Ip Address: 1.1.1.1/24 Ip Addresses: 1.1.1.1/24 1.1.1.2/24 1.1.1.4/24 2001:de7:0:f101::1/64 2001:db8:0:f101::1/64 fe80::1420:84ff:fe10:17aa/64 Mac: 16:20:84:10:17:aa
oc label namespace <namespace1> <namespace2> mutatevirtualmachines.kubemacpool.io=allocate
oc label namespace <namespace1> <namespace2> mutatevirtualmachines.kubemacpool.io-
touch machineconfig.yaml
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: name: 50-set-selinux-for-hostpath-provisioner labels: machineconfiguration.openshift.io/role: worker spec: config: ignition: version: 3.2.0 systemd: units: - contents: | [Unit] Description=Set SELinux chcon for hostpath provisioner Before=kubelet.service [Service] ExecStart=/usr/bin/chcon -Rt container_file_t <backing_directory_path> 1 [Install] WantedBy=multi-user.target enabled: true name: hostpath-provisioner.service
oc create -f machineconfig.yaml -n <namespace>
sudo chcon -t container_file_t -R <backing_directory_path>
touch hostpathprovisioner_cr.yaml
apiVersion: hostpathprovisioner.kubevirt.io/v1beta1 kind: HostPathProvisioner metadata: name: hostpath-provisioner spec: imagePullPolicy: IfNotPresent pathConfig: path: "<backing_directory_path>" 1 useNamingPrefix: false 2 workload: 3
oc create -f hostpathprovisioner_cr.yaml -n openshift-cnv
touch storageclass.yaml
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: hostpath-provisioner 1 provisioner: kubevirt.io/hostpath-provisioner reclaimPolicy: Delete 2 volumeBindingMode: WaitForFirstConsumer 3
oc create -f storageclass.yaml
oc edit cdi
spec: config: filesystemOverhead: global: "<new_global_value>" 1 storageClass: <storage_class_name>: "<new_value_for_this_storage_class>" 2
oc get cdi -o yaml
oc edit cdi
apiVersion: cdi.kubevirt.io/v1beta1 kind: CDI spec: config: podResourceRequirements: limits: cpu: "4" memory: "1Gi" requests: cpu: "1" memory: "250Mi"
oc get cdi -o yaml
oc edit cdi
apiVersion: cdi.kubevirt.io/v1beta1 kind: CDI metadata: spec: config: preallocation: true 1
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: preallocated-datavolume spec: source: 1 pvc: preallocation: true 2
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <upload-datavolume> 1 spec: source: upload: {} pvc: accessModes: - ReadWriteOnce resources: requests: storage: <2Gi> 2
oc create -f <upload-datavolume>.yaml
virtctl image-upload dv <datavolume_name> \ 1 --size=<datavolume_size> \ 2 --image-path=</path/to/image> \ 3
oc get dvs
dd if=/dev/zero of=<loop10> bs=100M count=20
losetup </dev/loop10>d3 <loop10> 1 2
kind: PersistentVolume apiVersion: v1 metadata: name: <local-block-pv10> annotations: spec: local: path: </dev/loop10> 1 capacity: storage: <2Gi> volumeMode: Block 2 storageClassName: local 3 accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - <node01> 4
oc create -f <local-block-pv10.yaml> 1
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <upload-datavolume> 1 spec: source: upload: {} pvc: accessModes: - ReadWriteOnce resources: requests: storage: <2Gi> 2
oc create -f <upload-datavolume>.yaml
virtctl image-upload dv <datavolume_name> \ 1 --size=<datavolume_size> \ 2 --image-path=</path/to/image> \ 3
oc get dvs
apiVersion: snapshot.kubevirt.io/v1alpha1 kind: VirtualMachineSnapshot metadata: name: my-vmsnapshot 1 spec: source: apiGroup: kubevirt.io kind: VirtualMachine name: my-vm 2
oc create -f <my-vmsnapshot>.yaml
oc describe vmsnapshot <my-vmsnapshot>
apiVersion: snapshot.kubevirt.io/v1alpha1 kind: VirtualMachineSnapshot metadata: creationTimestamp: "2020-09-30T14:41:51Z" finalizers: - snapshot.kubevirt.io/vmsnapshot-protection generation: 5 name: mysnap namespace: default resourceVersion: "3897" selfLink: /apis/snapshot.kubevirt.io/v1alpha1/namespaces/default/virtualmachinesnapshots/my-vmsnapshot uid: 28eedf08-5d6a-42c1-969c-2eda58e2a78d spec: source: apiGroup: kubevirt.io kind: VirtualMachine name: my-vm status: conditions: - lastProbeTime: null lastTransitionTime: "2020-09-30T14:42:03Z" reason: Operation complete status: "False" 1 type: Progressing - lastProbeTime: null lastTransitionTime: "2020-09-30T14:42:03Z" reason: Operation complete status: "True" 2 type: Ready creationTime: "2020-09-30T14:42:03Z" readyToUse: true 3 sourceUID: 355897f3-73a0-4ec4-83d3-3c2df9486f4f virtualMachineSnapshotContentName: vmsnapshot-content-28eedf08-5d6a-42c1-969c-2eda58e2a78d 4
apiVersion: snapshot.kubevirt.io/v1alpha1 kind: VirtualMachineRestore metadata: name: my-vmrestore 1 spec: target: apiGroup: kubevirt.io kind: VirtualMachine name: my-vm 2 virtualMachineSnapshotName: my-vmsnapshot 3
oc create -f <my-vmrestore>.yaml
oc get vmrestore <my-vmrestore>
apiVersion: snapshot.kubevirt.io/v1alpha1 kind: VirtualMachineRestore metadata: creationTimestamp: "2020-09-30T14:46:27Z" generation: 5 name: my-vmrestore namespace: default ownerReferences: - apiVersion: kubevirt.io/v1alpha3 blockOwnerDeletion: true controller: true kind: VirtualMachine name: my-vm uid: 355897f3-73a0-4ec4-83d3-3c2df9486f4f resourceVersion: "5512" selfLink: /apis/snapshot.kubevirt.io/v1alpha1/namespaces/default/virtualmachinerestores/my-vmrestore uid: 71c679a8-136e-46b0-b9b5-f57175a6a041 spec: target: apiGroup: kubevirt.io kind: VirtualMachine name: my-vm virtualMachineSnapshotName: my-vmsnapshot status: complete: true 1 conditions: - lastProbeTime: null lastTransitionTime: "2020-09-30T14:46:28Z" reason: Operation complete status: "False" 2 type: Progressing - lastProbeTime: null lastTransitionTime: "2020-09-30T14:46:28Z" reason: Operation complete status: "True" 3 type: Ready deletedDataVolumes: - test-dv1 restoreTime: "2020-09-30T14:46:28Z" restores: - dataVolumeName: restore-71c679a8-136e-46b0-b9b5-f57175a6a041-datavolumedisk1 persistentVolumeClaim: restore-71c679a8-136e-46b0-b9b5-f57175a6a041-datavolumedisk1 volumeName: datavolumedisk1 volumeSnapshotName: vmsnapshot-28eedf08-5d6a-42c1-969c-2eda58e2a78d-volume-datavolumedisk1
oc delete vmsnapshot <my-vmsnapshot>
oc get vmsnapshot
kind: PersistentVolume apiVersion: v1 metadata: name: <destination-pv> 1 annotations: spec: accessModes: - ReadWriteOnce capacity: storage: 10Gi 2 local: path: /mnt/local-storage/local/disk1 3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node01 4 persistentVolumeReclaimPolicy: Delete storageClassName: local volumeMode: Filesystem
oc get pv <destination-pv> -o yaml
spec: nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname 1 operator: In values: - node01 2
oc label pv <destination-pv> node=node01
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <clone-datavolume> 1 spec: source: pvc: name: "<source-vm-disk>" 2 namespace: "<source-namespace>" 3 pvc: accessModes: - ReadWriteOnce selector: matchLabels: node: node01 4 resources: requests: storage: <10Gi> 5
oc apply -f <clone-datavolume.yaml>
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: blank-image-datavolume spec: source: blank: {} pvc: # Optional: Set the storage class or omit to accept the default # storageClassName: "hostpath" accessModes: - ReadWriteOnce resources: requests: storage: 500Mi
oc create -f <blank-image-datavolume>.yaml
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: blank-image-datavolume spec: source: blank: {} pvc: # Optional: Set the storage class or omit to accept the default # storageClassName: "hostpath" accessModes: - ReadWriteOnce resources: requests: storage: 500Mi
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <cloner-datavolume> 1 spec: source: pvc: namespace: "<source-namespace>" 2 name: "<my-favorite-vm-disk>" 3 pvc: accessModes: - ReadWriteMany resources: requests: storage: <2Gi> 4 volumeMode: Block 5
oc create -f <cloner-datavolume>.yaml
data: accessMode: ReadWriteOnce 1 volumeMode: Filesystem 2 <new>.accessMode: ReadWriteMany 3 <new>.volumeMode: Block 4
oc edit configmap kubevirt-storage-class-defaults -n openshift-cnv
data: accessMode: ReadWriteOnce 1 volumeMode: Filesystem 2 <new>.accessMode: ReadWriteMany 3 <new>.volumeMode: Block 4
kind: ConfigMap apiVersion: v1 metadata: name: kubevirt-storage-class-defaults namespace: openshift-cnv data: accessMode: ReadWriteOnce volumeMode: Filesystem nfs-sc.accessMode: ReadWriteMany nfs-sc.volumeMode: Filesystem block-sc.accessMode: ReadWriteMany block-sc.volumeMode: Block
cat > Dockerfile << EOF FROM registry.access.redhat.com/ubi8/ubi:latest AS builder ADD --chown=107:107 <vm_image>.qcow2 /disk/ 1 RUN chmod 0440 /disk/* FROM scratch COPY --from=builder /disk/* /disk/ EOF
podman build -t <registry>/<container_disk_name>:latest .
podman push <registry>/<container_disk_name>:latest
oc patch configmap cdi-insecure-registries -n openshift-cnv --type merge -p '{"data":{"mykey": "<insecure-registry-host>:5000"}}' 1
oc edit cdi
apiVersion: cdi.kubevirt.io/v1beta1 kind: CDI spec: config: scratchSpaceStorageClass: "<storage_class>" 1
oc get pv <pv_name> -o yaml | grep 'persistentVolumeReclaimPolicy'
oc patch pv <pv_name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
oc describe pvc <pvc_name> | grep 'Mounted By:'
oc delete pvc <pvc_name>
oc get pv <pv_name> -o yaml > <file_name>.yaml
oc delete pv <pv_name>
rm -rf <path_to_share_storage>
oc create -f <new_pv_name>.yaml
oc get dvs
oc delete dv <datavolume_name>