Chapter 14. Support 14.1. Support overview You can collect data about your environment, monitor the health of your cluster and virtual machines (VMs), and troubleshoot OpenShift Virtualization resources with the following tools. 14.1.1. Web console The OpenShift Container Platform web console displays resource usage, alerts, events, and trends for your cluster and for OpenShift Virtualization components and resources. Table 14.1. Web console pages for monitoring and troubleshooting Page Description Overview page Cluster details, status, alerts, inventory, and resource usage Virtualization Overview tab OpenShift Virtualization resources, usage, alerts, and status Virtualization Top consumers tab Top consumers of CPU, memory, and storage Virtualization Migrations tab Progress of live migrations VirtualMachines VirtualMachine VirtualMachine details Metrics tab VM resource usage, storage, network, and migration VirtualMachines VirtualMachine VirtualMachine details Events tab List of VM events VirtualMachines VirtualMachine VirtualMachine details Diagnostics tab VM status conditions and volume snapshot status 14.1.2. Collecting data for Red Hat Support When you submit a support case to Red Hat Support, it is helpful to provide debugging information. You can gather debugging information by performing the following steps: Collecting data about your environment Configure Prometheus and Alertmanager and collect must-gather data for OpenShift Container Platform and OpenShift Virtualization. Collecting data about VMs Collect must-gather data and memory dumps from VMs. must-gather tool for OpenShift Virtualization Configure and use the must-gather tool. 14.1.3. Monitoring You can monitor the health of your cluster and VMs. For details about monitoring tools, see the Monitoring overview . 14.1.4. Troubleshooting Troubleshoot OpenShift Virtualization components and VMs and resolve issues that trigger alerts in the web console. Events View important life-cycle information for VMs, namespaces, and resources. Logs View and configure logs for OpenShift Virtualization components and VMs. Runbooks Diagnose and resolve issues that trigger OpenShift Virtualization alerts in the web console. Troubleshooting data volumes Troubleshoot data volumes by analyzing conditions and events. 14.2. Collecting data for Red Hat Support When you submit a support case to Red Hat Support, it is helpful to provide debugging information for OpenShift Container Platform and OpenShift Virtualization by using the following tools: must-gather tool The must-gather tool collects diagnostic information, including resource definitions and service logs. Prometheus Prometheus is a time-series database and a rule evaluation engine for metrics. Prometheus sends alerts to Alertmanager for processing. Alertmanager The Alertmanager service handles alerts received from Prometheus. The Alertmanager is also responsible for sending the alerts to external notification systems. For information about the OpenShift Container Platform monitoring stack, see About OpenShift Container Platform monitoring . 14.2.1. Collecting data about your environment Collecting data about your environment minimizes the time required to analyze and determine the root cause. Prerequisites Set the retention time for Prometheus metrics data to a minimum of seven days. Configure the Alertmanager to capture relevant alerts and to send alert notifications to a dedicated mailbox so that they can be viewed and persisted outside the cluster. Record the exact number of affected nodes and virtual machines. Procedure Collect must-gather data for the cluster . Collect must-gather data for Red Hat OpenShift Data Foundation , if necessary. Collect must-gather data for OpenShift Virtualization . Collect Prometheus metrics for the cluster . 14.2.2. Collecting data about virtual machines Collecting data about malfunctioning virtual machines (VMs) minimizes the time required to analyze and determine the root cause. Prerequisites Linux VMs: Install the latest QEMU guest agent . Windows VMs: Record the Windows patch update details. Install the latest VirtIO drivers . Install the latest QEMU guest agent . If Remote Desktop Protocol (RDP) is enabled, try to connect to the VMs with RDP by using the web console or the command line to determine whether there is a problem with the connection software. Procedure Collect must-gather data for the VMs using the /usr/bin/gather script. Collect screenshots of VMs that have crashed before you restart them. Collect memory dumps from VMs before remediation attempts. Record factors that the malfunctioning VMs have in common. For example, the VMs have the same host or network. 14.2.3. Using the must-gather tool for OpenShift Virtualization You can collect data about OpenShift Virtualization resources by running the must-gather command with the OpenShift Virtualization image. The default data collection includes information about the following resources: OpenShift Virtualization Operator namespaces, including child objects OpenShift Virtualization custom resource definitions Namespaces that contain virtual machines Basic virtual machine definitions Procedure Run the following command to collect data about OpenShift Virtualization: USD oc adm must-gather \ --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.13.11 \ -- /usr/bin/gather 14.2.3.1. must-gather tool options You can specify a combination of scripts and environment variables for the following options: Collecting detailed virtual machine (VM) information from a namespace Collecting detailed information about specified VMs Collecting image, image-stream, and image-stream-tags information Limiting the maximum number of parallel processes used by the must-gather tool 14.2.3.1.1. Parameters Environment variables You can specify environment variables for a compatible script. NS=<namespace_name> Collect virtual machine information, including virt-launcher pod details, from the namespace that you specify. The VirtualMachine and VirtualMachineInstance CR data is collected for all namespaces. VM=<vm_name> Collect details about a particular virtual machine. To use this option, you must also specify a namespace by using the NS environment variable. PROS=<number_of_processes> Modify the maximum number of parallel processes that the must-gather tool uses. The default value is 5 . Important Using too many parallel processes can cause performance issues. Increasing the maximum number of parallel processes is not recommended. Scripts Each script is compatible only with certain environment variable combinations. /usr/bin/gather Use the default must-gather script, which collects cluster data from all namespaces and includes only basic VM information. This script is compatible only with the PROS variable. /usr/bin/gather --vms_details Collect VM log files, VM definitions, control-plane logs, and namespaces that belong to OpenShift Virtualization resources. Specifying namespaces includes their child objects. If you use this parameter without specifying a namespace or VM, the must-gather tool collects this data for all VMs in the cluster. This script is compatible with all environment variables, but you must specify a namespace if you use the VM variable. /usr/bin/gather --images Collect image, image-stream, and image-stream-tags custom resource information. This script is compatible only with the PROS variable. 14.2.3.1.2. Usage and examples Environment variables are optional. You can run a script by itself or with one or more compatible environment variables. Table 14.2. Compatible parameters Script Compatible environment variable /usr/bin/gather PROS=<number_of_processes> /usr/bin/gather --vms_details For a namespace: NS=<namespace_name> For a VM: VM=<vm_name> NS=<namespace_name> PROS=<number_of_processes> /usr/bin/gather --images PROS=<number_of_processes> Syntax USD oc adm must-gather \ --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.13.11 \ -- <environment_variable_1> <environment_variable_2> <script_name> Default data collection parallel processes By default, five processes run in parallel. USD oc adm must-gather \ --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.13.11 \ -- PROS=5 /usr/bin/gather 1 1 You can modify the number of parallel processes by changing the default. Detailed VM information The following command collects detailed VM information for the my-vm VM in the mynamespace namespace: USD oc adm must-gather \ --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.13.11 \ -- NS=mynamespace VM=my-vm /usr/bin/gather --vms_details 1 1 The NS environment variable is mandatory if you use the VM environment variable. Image, image-stream, and image-stream-tags information The following command collects image, image-stream, and image-stream-tags information from the cluster: USD oc adm must-gather \ --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.13.11 \ /usr/bin/gather --images 14.3. Monitoring 14.3.1. Monitoring overview You can monitor the health of your cluster and virtual machines (VMs) with the following tools: OpenShift Container Platform cluster checkup framework Run automated tests on your cluster with the OpenShift Container Platform cluster checkup framework to check the following conditions: Network connectivity and latency between two VMs attached to a secondary network interface VM running a Data Plane Development Kit (DPDK) workload with zero packet loss Important The OpenShift Container Platform cluster checkup framework is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Prometheus queries for virtual resources Query vCPU, network, storage, and guest memory swapping usage and live migration progress. VM custom metrics Configure the node-exporter service to expose internal VM metrics and processes. xref:[VM health checks] Configure readiness, liveness, and guest agent ping probes and a watchdog for VMs. Important The guest agent ping probe is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . 14.3.2. OpenShift Container Platform cluster checkup framework OpenShift Virtualization includes predefined checkups that can be used for cluster maintenance and troubleshooting. Important The OpenShift Container Platform cluster checkup framework is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . 14.3.2.1. About the OpenShift Container Platform cluster checkup framework A checkup is an automated test workload that allows you to verify if a specific cluster functionality works as expected. The cluster checkup framework uses native Kubernetes resources to configure and execute the checkup. By using predefined checkups, cluster administrators and developers can improve cluster maintainability, troubleshoot unexpected behavior, minimize errors, and save time. They can also review the results of the checkup and share them with experts for further analysis. Vendors can write and publish checkups for features or services that they provide and verify that their customer environments are configured correctly. Running a predefined checkup in an existing namespace involves setting up a service account for the checkup, creating the Role and RoleBinding objects for the service account, enabling permissions for the checkup, and creating the input config map and the checkup job. You can run a checkup multiple times. Important You must always: Verify that the checkup image is from a trustworthy source before applying it. Review the checkup permissions before creating the Role and RoleBinding objects. 14.3.2.2. Virtual machine latency checkup You use a predefined checkup to verify network connectivity and measure latency between two virtual machines (VMs) that are attached to a secondary network interface. The latency checkup uses the ping utility. You run a latency checkup by performing the following steps: Create a service account, roles, and rolebindings to provide cluster access permissions to the latency checkup. Create a config map to provide the input to run the checkup and to store the results. Create a job to run the checkup. Review the results in the config map. Optional: To rerun the checkup, delete the existing config map and job and then create a new config map and job. When you are finished, delete the latency checkup resources. Prerequisites You installed the OpenShift CLI ( oc ). The cluster has at least two worker nodes. The Multus Container Network Interface (CNI) plugin is installed on the cluster. You configured a network attachment definition for a namespace. Procedure Create a ServiceAccount , Role , and RoleBinding manifest for the latency checkup: Example 14.1. Example role manifest file --- apiVersion: v1 kind: ServiceAccount metadata: name: vm-latency-checkup-sa --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kubevirt-vm-latency-checker rules: - apiGroups: ["kubevirt.io"] resources: ["virtualmachineinstances"] verbs: ["get", "create", "delete"] - apiGroups: ["subresources.kubevirt.io"] resources: ["virtualmachineinstances/console"] verbs: ["get"] - apiGroups: ["k8s.cni.cncf.io"] resources: ["network-attachment-definitions"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kubevirt-vm-latency-checker subjects: - kind: ServiceAccount name: vm-latency-checkup-sa roleRef: kind: Role name: kubevirt-vm-latency-checker apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kiagnose-configmap-access rules: - apiGroups: [ "" ] resources: [ "configmaps" ] verbs: ["get", "update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kiagnose-configmap-access subjects: - kind: ServiceAccount name: vm-latency-checkup-sa roleRef: kind: Role name: kiagnose-configmap-access apiGroup: rbac.authorization.k8s.io Apply the ServiceAccount , Role , and RoleBinding manifest: USD oc apply -n <target_namespace> -f <latency_sa_roles_rolebinding>.yaml 1 1 <target_namespace> is the namespace where the checkup is to be run. This must be an existing namespace where the NetworkAttachmentDefinition object resides. Create a ConfigMap manifest that contains the input parameters for the checkup: Example input config map apiVersion: v1 kind: ConfigMap metadata: name: kubevirt-vm-latency-checkup-config data: spec.timeout: 5m spec.param.networkAttachmentDefinitionNamespace: <target_namespace> spec.param.networkAttachmentDefinitionName: "blue-network" 1 spec.param.maxDesiredLatencyMilliseconds: "10" 2 spec.param.sampleDurationSeconds: "5" 3 spec.param.sourceNode: "worker1" 4 spec.param.targetNode: "worker2" 5 1 The name of the NetworkAttachmentDefinition object. 2 Optional: The maximum desired latency, in milliseconds, between the virtual machines. If the measured latency exceeds this value, the checkup fails. 3 Optional: The duration of the latency check, in seconds. 4 Optional: When specified, latency is measured from this node to the target node. If the source node is specified, the spec.param.targetNode field cannot be empty. 5 Optional: When specified, latency is measured from the source node to this node. Apply the config map manifest in the target namespace: USD oc apply -n <target_namespace> -f <latency_config_map>.yaml Create a Job manifest to run the checkup: Example job manifest apiVersion: batch/v1 kind: Job metadata: name: kubevirt-vm-latency-checkup spec: backoffLimit: 0 template: spec: serviceAccountName: vm-latency-checkup-sa restartPolicy: Never containers: - name: vm-latency-checkup image: registry.redhat.io/container-native-virtualization/vm-network-latency-checkup-rhel9:v4.13.0 securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] runAsNonRoot: true seccompProfile: type: "RuntimeDefault" env: - name: CONFIGMAP_NAMESPACE value: <target_namespace> - name: CONFIGMAP_NAME value: kubevirt-vm-latency-checkup-config - name: POD_UID valueFrom: fieldRef: fieldPath: metadata.uid Apply the Job manifest: USD oc apply -n <target_namespace> -f <latency_job>.yaml Wait for the job to complete: USD oc wait job kubevirt-vm-latency-checkup -n <target_namespace> --for condition=complete --timeout 6m Review the results of the latency checkup by running the following command. If the maximum measured latency is greater than the value of the spec.param.maxDesiredLatencyMilliseconds attribute, the checkup fails and returns an error. USD oc get configmap kubevirt-vm-latency-checkup-config -n <target_namespace> -o yaml Example output config map (success) apiVersion: v1 kind: ConfigMap metadata: name: kubevirt-vm-latency-checkup-config namespace: <target_namespace> data: spec.timeout: 5m spec.param.networkAttachmentDefinitionNamespace: <target_namespace> spec.param.networkAttachmentDefinitionName: "blue-network" spec.param.maxDesiredLatencyMilliseconds: "10" spec.param.sampleDurationSeconds: "5" spec.param.sourceNode: "worker1" spec.param.targetNode: "worker2" status.succeeded: "true" status.failureReason: "" status.completionTimestamp: "2022-01-01T09:00:00Z" status.startTimestamp: "2022-01-01T09:00:07Z" status.result.avgLatencyNanoSec: "177000" status.result.maxLatencyNanoSec: "244000" 1 status.result.measurementDurationSec: "5" status.result.minLatencyNanoSec: "135000" status.result.sourceNode: "worker1" status.result.targetNode: "worker2" 1 The maximum measured latency in nanoseconds. Optional: To view the detailed job log in case of checkup failure, use the following command: USD oc logs job.batch/kubevirt-vm-latency-checkup -n <target_namespace> Delete the job and config map that you previously created by running the following commands: USD oc delete job -n <target_namespace> kubevirt-vm-latency-checkup USD oc delete config-map -n <target_namespace> kubevirt-vm-latency-checkup-config Optional: If you do not plan to run another checkup, delete the roles manifest: USD oc delete -f <latency_sa_roles_rolebinding>.yaml 14.3.2.3. DPDK checkup Use a predefined checkup to verify that your OpenShift Container Platform cluster node can run a virtual machine (VM) with a Data Plane Development Kit (DPDK) workload with zero packet loss. The DPDK checkup runs traffic between a traffic generator pod and a VM running a test DPDK application. You run a DPDK checkup by performing the following steps: Create a service account, role, and role bindings for the DPDK checkup and a service account for the traffic generator pod. Create a security context constraints resource for the traffic generator pod. Create a config map to provide the input to run the checkup and to store the results. Create a job to run the checkup. Review the results in the config map. Optional: To rerun the checkup, delete the existing config map and job and then create a new config map and job. When you are finished, delete the DPDK checkup resources. Prerequisites You have access to the cluster as a user with cluster-admin permissions. You have installed the OpenShift CLI ( oc ). You have configured the compute nodes to run DPDK applications on VMs with zero packet loss. Important The traffic generator pod created by the checkup has elevated privileges: It runs as root. It has a bind mount to the node's file system. The container image of the traffic generator is pulled from the upstream Project Quay container registry. Procedure Create a ServiceAccount , Role , and RoleBinding manifest for the DPDK checkup and the traffic generator pod: Example 14.2. Example service account, role, and rolebinding manifest file --- apiVersion: v1 kind: ServiceAccount metadata: name: dpdk-checkup-sa --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kiagnose-configmap-access rules: - apiGroups: [ "" ] resources: [ "configmaps" ] verbs: [ "get", "update" ] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kiagnose-configmap-access subjects: - kind: ServiceAccount name: dpdk-checkup-sa roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kiagnose-configmap-access --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kubevirt-dpdk-checker rules: - apiGroups: [ "kubevirt.io" ] resources: [ "virtualmachineinstances" ] verbs: [ "create", "get", "delete" ] - apiGroups: [ "subresources.kubevirt.io" ] resources: [ "virtualmachineinstances/console" ] verbs: [ "get" ] - apiGroups: [ "" ] resources: [ "pods" ] verbs: [ "create", "get", "delete" ] - apiGroups: [ "" ] resources: [ "pods/exec" ] verbs: [ "create" ] - apiGroups: [ "k8s.cni.cncf.io" ] resources: [ "network-attachment-definitions" ] verbs: [ "get" ] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kubevirt-dpdk-checker subjects: - kind: ServiceAccount name: dpdk-checkup-sa roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubevirt-dpdk-checker --- apiVersion: v1 kind: ServiceAccount metadata: name: dpdk-checkup-traffic-gen-sa Apply the ServiceAccount , Role , and RoleBinding manifest: USD oc apply -n <target_namespace> -f <dpdk_sa_roles_rolebinding>.yaml Create a SecurityContextConstraints manifest for the traffic generator pod: Example security context constraints manifest apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: name: dpdk-checkup-traffic-gen allowHostDirVolumePlugin: true allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: false allowPrivilegedContainer: false allowedCapabilities: - IPC_LOCK - NET_ADMIN - NET_RAW - SYS_RESOURCE defaultAddCapabilities: null fsGroup: type: RunAsAny groups: [] readOnlyRootFilesystem: false requiredDropCapabilities: null runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny seccompProfiles: - runtime/default - unconfined supplementalGroups: type: RunAsAny users: - system:serviceaccount:dpdk-checkup-ns:dpdk-checkup-traffic-gen-sa Apply the SecurityContextConstraints manifest: USD oc apply -f <dpdk_scc>.yaml Create a ConfigMap manifest that contains the input parameters for the checkup: Example input config map apiVersion: v1 kind: ConfigMap metadata: name: dpdk-checkup-config data: spec.timeout: 10m spec.param.networkAttachmentDefinitionName: <network_name> 1 spec.param.trafficGeneratorRuntimeClassName: <runtimeclass_name> 2 spec.param.trafficGeneratorImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-traffic-gen:v0.1.1" 3 spec.param.vmContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-vm:v0.1.1" 4 1 The name of the NetworkAttachmentDefinition object. 2 The RuntimeClass resource that the traffic generator pod uses. 3 The container image for the traffic generator. In this example, the image is pulled from the upstream Project Quay Container Registry. 4 The container disk image for the VM. In this example, the image is pulled from the upstream Project Quay Container Registry. Apply the ConfigMap manifest in the target namespace: USD oc apply -n <target_namespace> -f <dpdk_config_map>.yaml Create a Job manifest to run the checkup: Example job manifest apiVersion: batch/v1 kind: Job metadata: name: dpdk-checkup spec: backoffLimit: 0 template: spec: serviceAccountName: dpdk-checkup-sa restartPolicy: Never containers: - name: dpdk-checkup image: registry.redhat.io/container-native-virtualization/kubevirt-dpdk-checkup-rhel9:v4.13.0 imagePullPolicy: Always securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] runAsNonRoot: true seccompProfile: type: "RuntimeDefault" env: - name: CONFIGMAP_NAMESPACE value: <target-namespace> - name: CONFIGMAP_NAME value: dpdk-checkup-config - name: POD_UID valueFrom: fieldRef: fieldPath: metadata.uid Apply the Job manifest: USD oc apply -n <target_namespace> -f <dpdk_job>.yaml Wait for the job to complete: USD oc wait job dpdk-checkup -n <target_namespace> --for condition=complete --timeout 10m Review the results of the checkup by running the following command: USD oc get configmap dpdk-checkup-config -n <target_namespace> -o yaml Example output config map (success) apiVersion: v1 kind: ConfigMap metadata: name: dpdk-checkup-config data: spec.timeout: 1h2m spec.param.NetworkAttachmentDefinitionName: "mlx-dpdk-network-1" spec.param.trafficGeneratorRuntimeClassName: performance-performance-zeus10 spec.param.trafficGeneratorImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-traffic-gen:v0.1.1" spec.param.vmContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-vm:v0.1.1" status.succeeded: true status.failureReason: " " status.startTimestamp: 2022-12-21T09:33:06+00:00 status.completionTimestamp: 2022-12-21T11:33:06+00:00 status.result.actualTrafficGeneratorTargetNode: worker-dpdk1 status.result.actualDPDKVMTargetNode: worker-dpdk2 status.result.dropRate: 0 Delete the job and config map that you previously created by running the following commands: USD oc delete job -n <target_namespace> dpdk-checkup USD oc delete config-map -n <target_namespace> dpdk-checkup-config Optional: If you do not plan to run another checkup, delete the ServiceAccount , Role , and RoleBinding manifest: USD oc delete -f <dpdk_sa_roles_rolebinding>.yaml 14.3.2.3.1. DPDK checkup config map parameters The following table shows the mandatory and optional parameters that you can set in the data stanza of the input ConfigMap manifest when you run a cluster DPDK readiness checkup: Table 14.3. DPDK checkup config map parameters Parameter Description Is Mandatory spec.timeout The time, in minutes, before the checkup fails. True spec.param.networkAttachmentDefinitionName The name of the NetworkAttachmentDefinition object of the SR-IOV NICs connected. True spec.param.trafficGeneratorRuntimeClassName The RuntimeClass resource that the traffic generator pod uses. True spec.param.trafficGeneratorImage The container image for the traffic generator. The default value is quay.io/kiagnose/kubevirt-dpdk-checkup-traffic-gen:main . False spec.param.trafficGeneratorNodeSelector The node on which the traffic generator pod is to be scheduled. The node should be configured to allow DPDK traffic. False spec.param.trafficGeneratorPacketsPerSecond The number of packets per second, in kilo (k) or million(m). The default value is 14m. False spec.param.trafficGeneratorEastMacAddress The MAC address of the NIC connected to the traffic generator pod or VM. The default value is a random MAC address in the format 50:xx:xx:xx:xx:01 . False spec.param.trafficGeneratorWestMacAddress The MAC address of the NIC connected to the traffic generator pod or VM. The default value is a random MAC address in the format 50:xx:xx:xx:xx:02 . False spec.param.vmContainerDiskImage The container disk image for the VM. The default value is quay.io/kiagnose/kubevirt-dpdk-checkup-vm:main . False spec.param.DPDKLabelSelector The label of the node on which the VM runs. The node should be configured to allow DPDK traffic. False spec.param.DPDKEastMacAddress The MAC address of the NIC that is connected to the VM. The default value is a random MAC address in the format 60:xx:xx:xx:xx:01 . False spec.param.DPDKWestMacAddress The MAC address of the NIC that is connected to the VM. The default value is a random MAC address in the format 60:xx:xx:xx:xx:02 . False spec.param.testDuration The duration, in minutes, for which the traffic generator runs. The default value is 5 minutes. False spec.param.portBandwidthGB The maximum bandwidth of the SR-IOV NIC. The default value is 10GB. False spec.param.verbose When set to true , it increases the verbosity of the checkup log. The default value is false . False 14.3.2.3.2. Building a container disk image for RHEL virtual machines You can build a custom Red Hat Enterprise Linux (RHEL) 8 OS image in qcow2 format and use it to create a container disk image. You can store the container disk image in a registry that is accessible from your cluster and specify the image location in the spec.param.vmContainerDiskImage attribute of the DPDK checkup config map. To build a container disk image, you must create an image builder virtual machine (VM). The image builder VM is a RHEL 8 VM that can be used to build custom RHEL images. Prerequisites The image builder VM must run RHEL 8.7 and must have a minimum of 2 CPU cores, 4 GiB RAM, and 20 GB of free space in the /var directory. You have installed the image builder tool and its CLI ( composer-cli ) on the VM. You have installed the virt-customize tool: # dnf install libguestfs-tools You have installed the Podman CLI tool ( podman ). Procedure Verify that you can build a RHEL 8.7 image: # composer-cli distros list Note To run the composer-cli commands as non-root, add your user to the weldr or root groups: # usermod -a -G weldr user USD newgrp weldr Enter the following command to create an image blueprint file in TOML format that contains the packages to be installed, kernel customizations, and the services to be disabled during boot time: USD cat << EOF > dpdk-vm.toml name = "dpdk_image" description = "Image to use with the DPDK checkup" version = "0.0.1" distro = "rhel-87" [[packages]] name = "dpdk" [[packages]] name = "dpdk-tools" [[packages]] name = "driverctl" [[packages]] name = "tuned-profiles-cpu-partitioning" [customizations.kernel] append = "default_hugepagesz=1GB hugepagesz=1G hugepages=8 isolcpus=2-7" [customizations.services] disabled = ["NetworkManager-wait-online", "sshd"] EOF Push the blueprint file to the image builder tool by running the following command: # composer-cli blueprints push dpdk-vm.toml Generate the system image by specifying the blueprint name and output file format. The Universally Unique Identifier (UUID) of the image is displayed when you start the compose process. # composer-cli compose start dpdk_image qcow2 Wait for the compose process to complete. The compose status must show FINISHED before you can continue to the step. # composer-cli compose status Enter the following command to download the qcow2 image file by specifying its UUID: # composer-cli compose image <UUID> Create the customization scripts by running the following commands: USD cat <<EOF >customize-vm echo isolated_cores=2-7 > /etc/tuned/cpu-partitioning-variables.conf tuned-adm profile cpu-partitioning echo "options vfio enable_unsafe_noiommu_mode=1" > /etc/modprobe.d/vfio-noiommu.conf EOF USD cat <<EOF >first-boot driverctl set-override 0000:06:00.0 vfio-pci driverctl set-override 0000:07:00.0 vfio-pci mkdir /mnt/huge mount /mnt/huge --source nodev -t hugetlbfs -o pagesize=1GB EOF Use the virt-customize tool to customize the image generated by the image builder tool: USD virt-customize -a <UUID>.qcow2 --run=customize-vm --firstboot=first-boot --selinux-relabel To create a Dockerfile that contains all the commands to build the container disk image, enter the following command: USD cat << EOF > Dockerfile FROM scratch COPY <uuid>-disk.qcow2 /disk/ EOF where: <uuid>-disk.qcow2 Specifies the name of the custom image in qcow2 format. Build and tag the container by running the following command: USD podman build . -t dpdk-rhel:latest Push the container disk image to a registry that is accessible from your cluster by running the following command: USD podman push dpdk-rhel:latest Provide a link to the container disk image in the spec.param.vmContainerDiskImage attribute in the DPDK checkup config map. 14.3.2.4. Additional resources Attaching a virtual machine to multiple networks Using a virtual function in DPDK mode with an Intel NIC Using SR-IOV and the Node Tuning Operator to achieve a DPDK line rate Installing image builder How to register and subscribe a RHEL system to the Red Hat Customer Portal using Red Hat Subscription Manager 14.3.3. Prometheus queries for virtual resources OpenShift Virtualization provides metrics that you can use to monitor the consumption of cluster infrastructure resources, including vCPU, network, storage, and guest memory swapping. You can also use metrics to query live migration status. Use the OpenShift Container Platform monitoring dashboard to query virtualization metrics. 14.3.3.1. Prerequisites To use the vCPU metric, the schedstats=enable kernel argument must be applied to the MachineConfig object. This kernel argument enables scheduler statistics used for debugging and performance tuning and adds a minor additional load to the scheduler. For more information, see Adding kernel arguments to nodes . For guest memory swapping queries to return data, memory swapping must be enabled on the virtual guests. 14.3.3.2. Querying metrics The OpenShift Container Platform monitoring dashboard enables you to run Prometheus Query Language (PromQL) queries to examine metrics visualized on a plot. This functionality provides information about the state of a cluster and any user-defined workloads that you are monitoring. As a cluster administrator, you can query metrics for all core OpenShift Container Platform and user-defined projects. As a developer, you must specify a project name when querying metrics. You must have the required privileges to view metrics for the selected project. 14.3.3.2.1. Querying metrics for all projects as a cluster administrator As a cluster administrator or as a user with view permissions for all projects, you can access metrics for all default OpenShift Container Platform and user-defined projects in the Metrics UI. Prerequisites You have access to the cluster as a user with the cluster-admin cluster role or with view permissions for all projects. You have installed the OpenShift CLI ( oc ). Procedure From the Administrator perspective in the OpenShift Container Platform web console, select Observe Metrics . To add one or more queries, do any of the following: Option Description Create a custom query. Add your Prometheus Query Language (PromQL) query to the Expression field. As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. You can use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. You can also move your mouse pointer over a suggested item to view a brief description of that item. Add multiple queries. Select Add query . Duplicate an existing query. Select the Options menu to the query, then choose Duplicate query . Disable a query from being run. Select the Options menu to the query and choose Disable query . To run queries that you created, select Run queries . The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message. Note Queries that operate on large amounts of data might time out or overload the browser when drawing time series graphs. To avoid this, select Hide graph and calibrate your query using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs. Note By default, the query table shows an expanded view that lists every metric and its current value. You can select ˅ to minimize the expanded view for a query. Optional: The page URL now contains the queries you ran. To use this set of queries again in the future, save this URL. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. You can select which metrics are shown by doing any of the following: Option Description Hide all metrics from a query. Click the Options menu for the query and click Hide all series . Hide a specific metric. Go to the query table and click the colored square near the metric name. Zoom into the plot and change the time range. Either: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu in the left upper corner to select the time range. Reset the time range. Select Reset zoom . Display outputs for all queries at a specific point in time. Hold the mouse cursor on the plot at that point. The query outputs will appear in a pop-up box. Hide the plot. Select Hide graph . 14.3.3.2.2. Querying metrics for user-defined projects as a developer You can access metrics for a user-defined project as a developer or as a user with view permissions for the project. In the Developer perspective, the Metrics UI includes some predefined CPU, memory, bandwidth, and network packet queries for the selected project. You can also run custom Prometheus Query Language (PromQL) queries for CPU, memory, bandwidth, network packet and application metrics for the project. Note Developers can only use the Developer perspective and not the Administrator perspective. As a developer, you can only query metrics for one project at a time. Prerequisites You have access to the cluster as a developer or as a user with view permissions for the project that you are viewing metrics for. You have enabled monitoring for user-defined projects. You have deployed a service in a user-defined project. You have created a ServiceMonitor custom resource definition (CRD) for the service to define how the service is monitored. Procedure From the Developer perspective in the OpenShift Container Platform web console, select Observe Metrics . Select the project that you want to view metrics for in the Project: list. Select a query from the Select query list, or create a custom PromQL query based on the selected query by selecting Show PromQL . The metrics from the queries are visualized on the plot. Note In the Developer perspective, you can only run one query at a time. Explore the visualized metrics by doing any of the following: Option Description Zoom into the plot and change the time range. Either: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu in the left upper corner to select the time range. Reset the time range. Select Reset zoom . Display outputs for all queries at a specific point in time. Hold the mouse cursor on the plot at that point. The query outputs appear in a pop-up box. 14.3.3.3. Virtualization metrics The following metric descriptions include example Prometheus Query Language (PromQL) queries. These metrics are not an API and might change between versions. Note The following examples use topk queries that specify a time period. If virtual machines are deleted during that time period, they can still appear in the query output. 14.3.3.3.1. vCPU metrics The following query can identify virtual machines that are waiting for Input/Output (I/O): kubevirt_vmi_vcpu_wait_seconds Returns the wait time (in seconds) for a virtual machine's vCPU. Type: Counter. A value above '0' means that the vCPU wants to run, but the host scheduler cannot run it yet. This inability to run indicates that there is an issue with I/O. Note To query the vCPU metric, the schedstats=enable kernel argument must first be applied to the MachineConfig object. This kernel argument enables scheduler statistics used for debugging and performance tuning and adds a minor additional load to the scheduler. Example vCPU wait time query topk(3, sum by (name, namespace) (rate(kubevirt_vmi_vcpu_wait_seconds[6m]))) > 0 1 1 This query returns the top 3 VMs waiting for I/O at every given moment over a six-minute time period. 14.3.3.3.2. Network metrics The following queries can identify virtual machines that are saturating the network: kubevirt_vmi_network_receive_bytes_total Returns the total amount of traffic received (in bytes) on the virtual machine's network. Type: Counter. kubevirt_vmi_network_transmit_bytes_total Returns the total amount of traffic transmitted (in bytes) on the virtual machine's network. Type: Counter. Example network traffic query topk(3, sum by (name, namespace) (rate(kubevirt_vmi_network_receive_bytes_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_network_transmit_bytes_total[6m]))) > 0 1 1 This query returns the top 3 VMs transmitting the most network traffic at every given moment over a six-minute time period. 14.3.3.3.3. Storage metrics 14.3.3.3.3.1. Storage-related traffic The following queries can identify VMs that are writing large amounts of data: kubevirt_vmi_storage_read_traffic_bytes_total Returns the total amount (in bytes) of the virtual machine's storage-related traffic. Type: Counter. kubevirt_vmi_storage_write_traffic_bytes_total Returns the total amount of storage writes (in bytes) of the virtual machine's storage-related traffic. Type: Counter. Example storage-related traffic query topk(3, sum by (name, namespace) (rate(kubevirt_vmi_storage_read_traffic_bytes_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_storage_write_traffic_bytes_total[6m]))) > 0 1 1 This query returns the top 3 VMs performing the most storage traffic at every given moment over a six-minute time period. 14.3.3.3.3.2. Storage snapshot data kubevirt_vmsnapshot_disks_restored_from_source_total Returns the total number of virtual machine disks restored from the source virtual machine. Type: Gauge. kubevirt_vmsnapshot_disks_restored_from_source_bytes Returns the amount of space in bytes restored from the source virtual machine. Type: Gauge. Examples of storage snapshot data queries kubevirt_vmsnapshot_disks_restored_from_source_total{vm_name="simple-vm", vm_namespace="default"} 1 1 This query returns the total number of virtual machine disks restored from the source virtual machine. kubevirt_vmsnapshot_disks_restored_from_source_bytes{vm_name="simple-vm", vm_namespace="default"} 1 1 This query returns the amount of space in bytes restored from the source virtual machine. 14.3.3.3.3.3. I/O performance The following queries can determine the I/O performance of storage devices: kubevirt_vmi_storage_iops_read_total Returns the amount of write I/O operations the virtual machine is performing per second. Type: Counter. kubevirt_vmi_storage_iops_write_total Returns the amount of read I/O operations the virtual machine is performing per second. Type: Counter. Example I/O performance query topk(3, sum by (name, namespace) (rate(kubevirt_vmi_storage_iops_read_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_storage_iops_write_total[6m]))) > 0 1 1 This query returns the top 3 VMs performing the most I/O operations per second at every given moment over a six-minute time period. 14.3.3.3.4. Guest memory swapping metrics The following queries can identify which swap-enabled guests are performing the most memory swapping: kubevirt_vmi_memory_swap_in_traffic_bytes_total Returns the total amount (in bytes) of memory the virtual guest is swapping in. Type: Gauge. kubevirt_vmi_memory_swap_out_traffic_bytes_total Returns the total amount (in bytes) of memory the virtual guest is swapping out. Type: Gauge. Example memory swapping query topk(3, sum by (name, namespace) (rate(kubevirt_vmi_memory_swap_in_traffic_bytes_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_memory_swap_out_traffic_bytes_total[6m]))) > 0 1 1 This query returns the top 3 VMs where the guest is performing the most memory swapping at every given moment over a six-minute time period. Note Memory swapping indicates that the virtual machine is under memory pressure. Increasing the memory allocation of the virtual machine can mitigate this issue. 14.3.3.3.5. Live migration metrics The following metrics can be queried to show live migration status: kubevirt_migrate_vmi_data_processed_bytes The amount of guest operating system data that has migrated to the new virtual machine (VM). Type: Gauge. kubevirt_migrate_vmi_data_remaining_bytes The amount of guest operating system data that remains to be migrated. Type: Gauge. kubevirt_migrate_vmi_dirty_memory_rate_bytes The rate at which memory is becoming dirty in the guest operating system. Dirty memory is data that has been changed but not yet written to disk. Type: Gauge. kubevirt_migrate_vmi_pending_count The number of pending migrations. Type: Gauge. kubevirt_migrate_vmi_scheduling_count The number of scheduling migrations. Type: Gauge. kubevirt_migrate_vmi_running_count The number of running migrations. Type: Gauge. kubevirt_migrate_vmi_succeeded The number of successfully completed migrations. Type: Gauge. kubevirt_migrate_vmi_failed The number of failed migrations. Type: Gauge. 14.3.3.4. Additional resources Monitoring overview Querying Prometheus Prometheus query examples 14.3.4. Exposing custom metrics for virtual machines OpenShift Container Platform includes a preconfigured, preinstalled, and self-updating monitoring stack that provides monitoring for core platform components. This monitoring stack is based on the Prometheus monitoring system. Prometheus is a time-series database and a rule evaluation engine for metrics. In addition to using the OpenShift Container Platform monitoring stack, you can enable monitoring for user-defined projects by using the CLI and query custom metrics that are exposed for virtual machines through the node-exporter service. 14.3.4.1. Configuring the node exporter service The node-exporter agent is deployed on every virtual machine in the cluster from which you want to collect metrics. Configure the node-exporter agent as a service to expose internal metrics and processes that are associated with virtual machines. Prerequisites Install the OpenShift Container Platform CLI oc . Log in to the cluster as a user with cluster-admin privileges. Create the cluster-monitoring-config ConfigMap object in the openshift-monitoring project. Configure the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project by setting enableUserWorkload to true . Procedure Create the Service YAML file. In the following example, the file is called node-exporter-service.yaml . kind: Service apiVersion: v1 metadata: name: node-exporter-service 1 namespace: dynamation 2 labels: servicetype: metrics 3 spec: ports: - name: exmet 4 protocol: TCP port: 9100 5 targetPort: 9100 6 type: ClusterIP selector: monitor: metrics 7 1 The node-exporter service that exposes the metrics from the virtual machines. 2 The namespace where the service is created. 3 The label for the service. The ServiceMonitor uses this label to match this service. 4 The name given to the port that exposes metrics on port 9100 for the ClusterIP service. 5 The target port used by node-exporter-service to listen for requests. 6 The TCP port number of the virtual machine that is configured with the monitor label. 7 The label used to match the virtual machine's pods. In this example, any virtual machine's pod with the label monitor and a value of metrics will be matched. Create the node-exporter service: USD oc create -f node-exporter-service.yaml 14.3.4.2. Configuring a virtual machine with the node exporter service Download the node-exporter file on to the virtual machine. Then, create a systemd service that runs the node-exporter service when the virtual machine boots. Prerequisites The pods for the component are running in the openshift-user-workload-monitoring project. Grant the monitoring-edit role to users who need to monitor this user-defined project. Procedure Log on to the virtual machine. Download the node-exporter file on to the virtual machine by using the directory path that applies to the version of node-exporter file. USD wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz Extract the executable and place it in the /usr/bin directory. USD sudo tar xvf node_exporter-1.3.1.linux-amd64.tar.gz \ --directory /usr/bin --strip 1 "*/node_exporter" Create a node_exporter.service file in this directory path: /etc/systemd/system . This systemd service file runs the node-exporter service when the virtual machine reboots. [Unit] Description=Prometheus Metrics Exporter After=network.target StartLimitIntervalSec=0 [Service] Type=simple Restart=always RestartSec=1 User=root ExecStart=/usr/bin/node_exporter [Install] WantedBy=multi-user.target Enable and start the systemd service. USD sudo systemctl enable node_exporter.service USD sudo systemctl start node_exporter.service Verification Verify that the node-exporter agent is reporting metrics from the virtual machine. USD curl http://localhost:9100/metrics Example output go_gc_duration_seconds{quantile="0"} 1.5244e-05 go_gc_duration_seconds{quantile="0.25"} 3.0449e-05 go_gc_duration_seconds{quantile="0.5"} 3.7913e-05 14.3.4.3. Creating a custom monitoring label for virtual machines To enable queries to multiple virtual machines from a single service, add a custom label in the virtual machine's YAML file. Prerequisites Install the OpenShift Container Platform CLI oc . Log in as a user with cluster-admin privileges. Access to the web console for stop and restart a virtual machine. Procedure Edit the template spec of your virtual machine configuration file. In this example, the label monitor has the value metrics . spec: template: metadata: labels: monitor: metrics Stop and restart the virtual machine to create a new pod with the label name given to the monitor label. 14.3.4.3.1. Querying the node-exporter service for metrics Metrics are exposed for virtual machines through an HTTP service endpoint under the /metrics canonical name. When you query for metrics, Prometheus directly scrapes the metrics from the metrics endpoint exposed by the virtual machines and presents these metrics for viewing. Prerequisites You have access to the cluster as a user with cluster-admin privileges or the monitoring-edit role. You have enabled monitoring for the user-defined project by configuring the node-exporter service. Procedure Obtain the HTTP service endpoint by specifying the namespace for the service: USD oc get service -n <namespace> <node-exporter-service> To list all available metrics for the node-exporter service, query the metrics resource. USD curl http://<172.30.226.162:9100>/metrics | grep -vE "^#|^USD" Example output node_arp_entries{device="eth0"} 1 node_boot_time_seconds 1.643153218e+09 node_context_switches_total 4.4938158e+07 node_cooling_device_cur_state{name="0",type="Processor"} 0 node_cooling_device_max_state{name="0",type="Processor"} 0 node_cpu_guest_seconds_total{cpu="0",mode="nice"} 0 node_cpu_guest_seconds_total{cpu="0",mode="user"} 0 node_cpu_seconds_total{cpu="0",mode="idle"} 1.10586485e+06 node_cpu_seconds_total{cpu="0",mode="iowait"} 37.61 node_cpu_seconds_total{cpu="0",mode="irq"} 233.91 node_cpu_seconds_total{cpu="0",mode="nice"} 551.47 node_cpu_seconds_total{cpu="0",mode="softirq"} 87.3 node_cpu_seconds_total{cpu="0",mode="steal"} 86.12 node_cpu_seconds_total{cpu="0",mode="system"} 464.15 node_cpu_seconds_total{cpu="0",mode="user"} 1075.2 node_disk_discard_time_seconds_total{device="vda"} 0 node_disk_discard_time_seconds_total{device="vdb"} 0 node_disk_discarded_sectors_total{device="vda"} 0 node_disk_discarded_sectors_total{device="vdb"} 0 node_disk_discards_completed_total{device="vda"} 0 node_disk_discards_completed_total{device="vdb"} 0 node_disk_discards_merged_total{device="vda"} 0 node_disk_discards_merged_total{device="vdb"} 0 node_disk_info{device="vda",major="252",minor="0"} 1 node_disk_info{device="vdb",major="252",minor="16"} 1 node_disk_io_now{device="vda"} 0 node_disk_io_now{device="vdb"} 0 node_disk_io_time_seconds_total{device="vda"} 174 node_disk_io_time_seconds_total{device="vdb"} 0.054 node_disk_io_time_weighted_seconds_total{device="vda"} 259.79200000000003 node_disk_io_time_weighted_seconds_total{device="vdb"} 0.039 node_disk_read_bytes_total{device="vda"} 3.71867136e+08 node_disk_read_bytes_total{device="vdb"} 366592 node_disk_read_time_seconds_total{device="vda"} 19.128 node_disk_read_time_seconds_total{device="vdb"} 0.039 node_disk_reads_completed_total{device="vda"} 5619 node_disk_reads_completed_total{device="vdb"} 96 node_disk_reads_merged_total{device="vda"} 5 node_disk_reads_merged_total{device="vdb"} 0 node_disk_write_time_seconds_total{device="vda"} 240.66400000000002 node_disk_write_time_seconds_total{device="vdb"} 0 node_disk_writes_completed_total{device="vda"} 71584 node_disk_writes_completed_total{device="vdb"} 0 node_disk_writes_merged_total{device="vda"} 19761 node_disk_writes_merged_total{device="vdb"} 0 node_disk_written_bytes_total{device="vda"} 2.007924224e+09 node_disk_written_bytes_total{device="vdb"} 0 14.3.4.4. Creating a ServiceMonitor resource for the node exporter service You can use a Prometheus client library and scrape metrics from the /metrics endpoint to access and view the metrics exposed by the node-exporter service. Use a ServiceMonitor custom resource definition (CRD) to monitor the node exporter service. Prerequisites You have access to the cluster as a user with cluster-admin privileges or the monitoring-edit role. You have enabled monitoring for the user-defined project by configuring the node-exporter service. Procedure Create a YAML file for the ServiceMonitor resource configuration. In this example, the service monitor matches any service with the label metrics and queries the exmet port every 30 seconds. apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: k8s-app: node-exporter-metrics-monitor name: node-exporter-metrics-monitor 1 namespace: dynamation 2 spec: endpoints: - interval: 30s 3 port: exmet 4 scheme: http selector: matchLabels: servicetype: metrics 1 The name of the ServiceMonitor . 2 The namespace where the ServiceMonitor is created. 3 The interval at which the port will be queried. 4 The name of the port that is queried every 30 seconds Create the ServiceMonitor configuration for the node-exporter service. USD oc create -f node-exporter-metrics-monitor.yaml 14.3.4.4.1. Accessing the node exporter service outside the cluster You can access the node-exporter service outside the cluster and view the exposed metrics. Prerequisites You have access to the cluster as a user with cluster-admin privileges or the monitoring-edit role. You have enabled monitoring for the user-defined project by configuring the node-exporter service. Procedure Expose the node-exporter service. USD oc expose service -n <namespace> <node_exporter_service_name> Obtain the FQDN (Fully Qualified Domain Name) for the route. USD oc get route -o=custom-columns=NAME:.metadata.name,DNS:.spec.host Example output NAME DNS node-exporter-service node-exporter-service-dynamation.apps.cluster.example.org Use the curl command to display metrics for the node-exporter service. USD curl -s http://node-exporter-service-dynamation.apps.cluster.example.org/metrics Example output go_gc_duration_seconds{quantile="0"} 1.5382e-05 go_gc_duration_seconds{quantile="0.25"} 3.1163e-05 go_gc_duration_seconds{quantile="0.5"} 3.8546e-05 go_gc_duration_seconds{quantile="0.75"} 4.9139e-05 go_gc_duration_seconds{quantile="1"} 0.000189423 14.3.4.5. Additional resources Configuring the monitoring stack Enabling monitoring for user-defined projects Managing metrics Reviewing monitoring dashboards Monitoring application health by using health checks Creating and using config maps Controlling virtual machine states 14.3.5. Virtual machine health checks You can configure virtual machine (VM) health checks by defining readiness and liveness probes in the VirtualMachine resource. 14.3.5.1. About readiness and liveness probes Use readiness and liveness probes to detect and handle unhealthy virtual machines (VMs). You can include one or more probes in the specification of the VM to ensure that traffic does not reach a VM that is not ready for it and that a new VM is created when a VM becomes unresponsive. A readiness probe determines whether a VM is ready to accept service requests. If the probe fails, the VM is removed from the list of available endpoints until the VM is ready. A liveness probe determines whether a VM is responsive. If the probe fails, the VM is deleted and a new VM is created to restore responsiveness. You can configure readiness and liveness probes by setting the spec.readinessProbe and the spec.livenessProbe fields of the VirtualMachine object. These fields support the following tests: HTTP GET The probe determines the health of the VM by using a web hook. The test is successful if the HTTP response code is between 200 and 399. You can use an HTTP GET test with applications that return HTTP status codes when they are completely initialized. TCP socket The probe attempts to open a socket to the VM. The VM is only considered healthy if the probe can establish a connection. You can use a TCP socket test with applications that do not start listening until initialization is complete. Guest agent ping The probe uses the guest-ping command to determine if the QEMU guest agent is running on the virtual machine. 14.3.5.1.1. Defining an HTTP readiness probe Define an HTTP readiness probe by setting the spec.readinessProbe.httpGet field of the virtual machine (VM) configuration. Procedure Include details of the readiness probe in the VM configuration file. Sample readiness probe with an HTTP GET test # ... spec: readinessProbe: httpGet: 1 port: 1500 2 path: /healthz 3 httpHeaders: - name: Custom-Header value: Awesome initialDelaySeconds: 120 4 periodSeconds: 20 5 timeoutSeconds: 10 6 failureThreshold: 3 7 successThreshold: 3 8 # ... 1 The HTTP GET request to perform to connect to the VM. 2 The port of the VM that the probe queries. In the above example, the probe queries port 1500. 3 The path to access on the HTTP server. In the above example, if the handler for the server's /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is removed from the list of available endpoints. 4 The time, in seconds, after the VM starts before the readiness probe is initiated. 5 The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds . 6 The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds . 7 The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready . 8 The number of times that the probe must report success, after a failure, to be considered successful. The default is 1. Create the VM by running the following command: USD oc create -f <file_name>.yaml 14.3.5.1.2. Defining a TCP readiness probe Define a TCP readiness probe by setting the spec.readinessProbe.tcpSocket field of the virtual machine (VM) configuration. Procedure Include details of the TCP readiness probe in the VM configuration file. Sample readiness probe with a TCP socket test # ... spec: readinessProbe: initialDelaySeconds: 120 1 periodSeconds: 20 2 tcpSocket: 3 port: 1500 4 timeoutSeconds: 10 5 # ... 1 The time, in seconds, after the VM starts before the readiness probe is initiated. 2 The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds . 3 The TCP action to perform. 4 The port of the VM that the probe queries. 5 The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds . Create the VM by running the following command: USD oc create -f <file_name>.yaml 14.3.5.1.3. Defining an HTTP liveness probe Define an HTTP liveness probe by setting the spec.livenessProbe.httpGet field of the virtual machine (VM) configuration. You can define both HTTP and TCP tests for liveness probes in the same way as readiness probes. This procedure configures a sample liveness probe with an HTTP GET test. Procedure Include details of the HTTP liveness probe in the VM configuration file. Sample liveness probe with an HTTP GET test # ... spec: livenessProbe: initialDelaySeconds: 120 1 periodSeconds: 20 2 httpGet: 3 port: 1500 4 path: /healthz 5 httpHeaders: - name: Custom-Header value: Awesome timeoutSeconds: 10 6 # ... 1 The time, in seconds, after the VM starts before the liveness probe is initiated. 2 The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds . 3 The HTTP GET request to perform to connect to the VM. 4 The port of the VM that the probe queries. In the above example, the probe queries port 1500. The VM installs and runs a minimal HTTP server on port 1500 via cloud-init. 5 The path to access on the HTTP server. In the above example, if the handler for the server's /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is deleted and a new VM is created. 6 The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds . Create the VM by running the following command: USD oc create -f <file_name>.yaml 14.3.5.2. Defining a watchdog You can define a watchdog to monitor the health of the guest operating system by performing the following steps: Configure a watchdog device for the virtual machine (VM). Install the watchdog agent on the guest. The watchdog device monitors the agent and performs one of the following actions if the guest operating system is unresponsive: poweroff : The VM powers down immediately. If spec.running is set to true or spec.runStrategy is not set to manual , then the VM reboots. reset : The VM reboots in place and the guest operating system cannot react. Note The reboot time might cause liveness probes to time out. If cluster-level protections detect a failed liveness probe, the VM might be forcibly rescheduled, increasing the reboot time. shutdown : The VM gracefully powers down by stopping all services. Note Watchdog is not available for Windows VMs. 14.3.5.2.1. Configuring a watchdog device for the virtual machine You configure a watchdog device for the virtual machine (VM). Prerequisites The VM must have kernel support for an i6300esb watchdog device. Red Hat Enterprise Linux (RHEL) images support i6300esb . Procedure Create a YAML file with the following contents: apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: labels: kubevirt.io/vm: vm2-rhel84-watchdog name: <vm-name> spec: running: false template: metadata: labels: kubevirt.io/vm: vm2-rhel84-watchdog spec: domain: devices: watchdog: name: <watchdog> i6300esb: action: "poweroff" 1 # ... 1 Specify poweroff , reset , or shutdown . The example above configures the i6300esb watchdog device on a RHEL8 VM with the poweroff action and exposes the device as /dev/watchdog . This device can now be used by the watchdog binary. Apply the YAML file to your cluster by running the following command: USD oc apply -f <file_name>.yaml Important This procedure is provided for testing watchdog functionality only and must not be run on production machines. Run the following command to verify that the VM is connected to the watchdog device: USD lspci | grep watchdog -i Run one of the following commands to confirm the watchdog is active: Trigger a kernel panic: # echo c > /proc/sysrq-trigger Stop the watchdog service: # pkill -9 watchdog 14.3.5.2.2. Installing the watchdog agent on the guest You install the watchdog agent on the guest and start the watchdog service. Procedure Log in to the virtual machine as root user. Install the watchdog package and its dependencies: # yum install watchdog Uncomment the following line in the /etc/watchdog.conf file and save the changes: #watchdog-device = /dev/watchdog Enable the watchdog service to start on boot: # systemctl enable --now watchdog.service 14.3.5.3. Defining a guest agent ping probe Define a guest agent ping probe by setting the spec.readinessProbe.guestAgentPing field of the virtual machine (VM) configuration. Important The guest agent ping probe is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Prerequisites The QEMU guest agent must be installed and enabled on the virtual machine. Procedure Include details of the guest agent ping probe in the VM configuration file. For example: Sample guest agent ping probe # ... spec: readinessProbe: guestAgentPing: {} 1 initialDelaySeconds: 120 2 periodSeconds: 20 3 timeoutSeconds: 10 4 failureThreshold: 3 5 successThreshold: 3 6 # ... 1 The guest agent ping probe to connect to the VM. 2 Optional: The time, in seconds, after the VM starts before the guest agent probe is initiated. 3 Optional: The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds . 4 Optional: The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds . 5 Optional: The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready . 6 Optional: The number of times that the probe must report success, after a failure, to be considered successful. The default is 1. Create the VM by running the following command: USD oc create -f <file_name>.yaml 14.3.5.4. Additional resources Monitoring application health by using health checks 14.4. Troubleshooting OpenShift Virtualization provides tools and logs for troubleshooting virtual machines and virtualization components. You can troubleshoot OpenShift Virtualization components by using the tools provided in the web console or by using the oc CLI tool. 14.4.1. Events OpenShift Container Platform events are records of important life-cycle information and are useful for monitoring and troubleshooting virtual machine, namespace, and resource issues. VM events: Navigate to the Events tab of the VirtualMachine details page in the web console. Namespace events You can view namespace events by running the following command: USD oc get events -n <namespace> See the list of events for details about specific events. Resource events You can view resource events by running the following command: USD oc describe <resource> <resource_name> 14.4.2. Logs You can review the following logs for troubleshooting: Virtual machine OpenShift Virtualization pod Aggregated OpenShift Virtualization logs 14.4.2.1. Viewing virtual machine logs with the web console You can view virtual machine logs with the OpenShift Container Platform web console. Procedure Navigate to Virtualization VirtualMachines . Select a virtual machine to open the VirtualMachine details page. On the Details tab, click the pod name to open the Pod details page. Click the Logs tab to view the logs. 14.4.2.2. Viewing OpenShift Virtualization pod logs You can view logs for OpenShift Virtualization pods by using the oc CLI tool. You can configure the verbosity level of the logs by editing the HyperConverged custom resource (CR). 14.4.2.2.1. Viewing OpenShift Virtualization pod logs with the CLI You can view logs for the OpenShift Virtualization pods by using the oc CLI tool. Procedure View a list of pods in the OpenShift Virtualization namespace by running the following command: USD oc get pods -n openshift-cnv Example 14.3. Example output NAME READY STATUS RESTARTS AGE disks-images-provider-7gqbc 1/1 Running 0 32m disks-images-provider-vg4kx 1/1 Running 0 32m virt-api-57fcc4497b-7qfmc 1/1 Running 0 31m virt-api-57fcc4497b-tx9nc 1/1 Running 0 31m virt-controller-76c784655f-7fp6m 1/1 Running 0 30m virt-controller-76c784655f-f4pbd 1/1 Running 0 30m virt-handler-2m86x 1/1 Running 0 30m virt-handler-9qs6z 1/1 Running 0 30m virt-operator-7ccfdbf65f-q5snk 1/1 Running 0 32m virt-operator-7ccfdbf65f-vllz8 1/1 Running 0 32m View the pod log by running the following command: USD oc logs -n openshift-cnv <pod_name> Note If a pod fails to start, you can use the -- option to view logs from the last attempt. To monitor log output in real time, use the -f option. Example 14.4. Example output {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373695Z"} {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373726Z"} {"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:462","timestamp":"2022-04-17T08:58:37.373782Z"} {"component":"virt-handler","level":"info","msg":"CPU features of a minimum baseline CPU model: map[apic:true clflush:true cmov:true cx16:true cx8:true de:true fpu:true fxsr:true lahf_lm:true lm:true mca:true mce:true mmx:true msr:true mtrr:true nx:true pae:true pat:true pge:true pni:true pse:true pse36:true sep:true sse:true sse2:true sse4.1:true ssse3:true syscall:true tsc:true]","pos":"cpu_plugin.go:96","timestamp":"2022-04-17T08:58:37.390221Z"} {"component":"virt-handler","level":"warning","msg":"host model mode is expected to contain only one model","pos":"cpu_plugin.go:103","timestamp":"2022-04-17T08:58:37.390263Z"} {"component":"virt-handler","level":"info","msg":"node-labeller is running","pos":"node_labeller.go:94","timestamp":"2022-04-17T08:58:37.391011Z"} 14.4.2.2.2. Configuring OpenShift Virtualization pod log verbosity You can configure the verbosity level of OpenShift Virtualization pod logs by editing the HyperConverged custom resource (CR). Procedure To set log verbosity for specific components, open the HyperConverged CR in your default text editor by running the following command: USD oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv Set the log level for one or more components by editing the spec.logVerbosityConfig stanza. For example: apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged spec: logVerbosityConfig: kubevirt: virtAPI: 5 1 virtController: 4 virtHandler: 3 virtLauncher: 2 virtOperator: 6 1 The log verbosity value must be an integer in the range 1-9 , where a higher number indicates a more detailed log. In this example, the virtAPI component logs are exposed if their priority level is 5 or higher. Apply your changes by saving and exiting the editor. 14.4.2.2.3. Common error messages The following error messages might appear in OpenShift Virtualization logs: ErrImagePull or ImagePullBackOff Indicates an incorrect deployment configuration or problems with the images that are referenced. 14.4.2.3. Viewing aggregated OpenShift Virtualization logs with the LokiStack You can view aggregated logs for OpenShift Virtualization pods and containers by using the LokiStack in the web console. Prerequisites You deployed the LokiStack. Procedure Navigate to Observe Logs in the web console. Select application , for virt-launcher pod logs, or infrastructure , for OpenShift Virtualization control plane pods and containers, from the log type list. Click Show Query to display the query field. Enter the LogQL query in the query field and click Run Query to display the filtered logs. 14.4.2.3.1. OpenShift Virtualization LogQL queries You can view and filter aggregated logs for OpenShift Virtualization components by running Loki Query Language (LogQL) queries on the Observe Logs page in the web console. The default log type is infrastructure . The virt-launcher log type is application . Optional: You can include or exclude strings or regular expressions by using line filter expressions. Note If the query matches a large number of logs, the query might time out. Table 14.4. OpenShift Virtualization LogQL example queries Component LogQL query All {log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" cdi-apiserver cdi-deployment cdi-operator {log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" |kubernetes_labels_app_kubernetes_io_component="storage" hco-operator {log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" |kubernetes_labels_app_kubernetes_io_component="deployment" kubemacpool {log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" |kubernetes_labels_app_kubernetes_io_component="network" virt-api virt-controller virt-handler virt-operator {log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" |kubernetes_labels_app_kubernetes_io_component="compute" ssp-operator {log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" |kubernetes_labels_app_kubernetes_io_component="schedule" Container {log_type=~".+",kubernetes_container_name=~"<container>|<container>"} 1 |json|kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" 1 Specify one or more containers separated by a pipe ( | ). virt-launcher You must select application from the log type list before running this query. {log_type=~".+", kubernetes_container_name="compute"}|json |!= "custom-ga-command" 1 1 |!= "custom-ga-command" excludes libvirt logs that contain the string custom-ga-command . ( BZ#2177684 ) You can filter log lines to include or exclude strings or regular expressions by using line filter expressions. Table 14.5. Line filter expressions Line filter expression Description |= "<string>" Log line contains string != "<string>" Log line does not contain string |~ "<regex>" Log line contains regular expression !~ "<regex>" Log line does not contain regular expression Example line filter expression {log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" |= "error" != "timeout" 14.4.2.3.2. Additional resources for LokiStack and LogQL About log storage Deploying the LokiStack LogQL log queries in the Grafana documentation 14.4.3. Troubleshooting data volumes You can check the Conditions and Events sections of the DataVolume object to analyze and resolve issues. 14.4.3.1. About data volume conditions and events You can diagnose data volume issues by examining the output of the Conditions and Events sections generated by the command: USD oc describe dv <DataVolume> The Conditions section displays the following Types : Bound Running Ready The Events section provides the following additional information: Type of event Reason for logging Source of the event Message containing additional diagnostic information. The output from oc describe does not always contains Events . An event is generated when the Status , Reason , or Message changes. Both conditions and events react to changes in the state of the data volume. For example, if you misspell the URL during an import operation, the import generates a 404 message. That message change generates an event with a reason. The output in the Conditions section is updated as well. 14.4.3.2. Analyzing data volume conditions and events By inspecting the Conditions and Events sections generated by the describe command, you determine the state of the data volume in relation to persistent volume claims (PVCs), and whether or not an operation is actively running or completed. You might also receive messages that offer specific details about the status of the data volume, and how it came to be in its current state. There are many different combinations of conditions. Each must be evaluated in its unique context. Examples of various combinations follow. Bound - A successfully bound PVC displays in this example. Note that the Type is Bound , so the Status is True . If the PVC is not bound, the Status is False . When the PVC is bound, an event is generated stating that the PVC is bound. In this case, the Reason is Bound and Status is True . The Message indicates which PVC owns the data volume. Message , in the Events section, provides further details including how long the PVC has been bound ( Age ) and by what resource ( From ), in this case datavolume-controller : Example output Status: Conditions: Last Heart Beat Time: 2020-07-15T03:58:24Z Last Transition Time: 2020-07-15T03:58:24Z Message: PVC win10-rootdisk Bound Reason: Bound Status: True Type: Bound ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Bound 24s datavolume-controller PVC example-dv Bound Running - In this case, note that Type is Running and Status is False , indicating that an event has occurred that caused an attempted operation to fail, changing the Status from True to False . However, note that Reason is Completed and the Message field indicates Import Complete . In the Events section, the Reason and Message contain additional troubleshooting information about the failed operation. In this example, the Message displays an inability to connect due to a 404 , listed in the Events section's first Warning . From this information, you conclude that an import operation was running, creating contention for other operations that are attempting to access the data volume: Example output Status: Conditions: Last Heart Beat Time: 2020-07-15T04:31:39Z Last Transition Time: 2020-07-15T04:31:39Z Message: Import Complete Reason: Completed Status: False Type: Running ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Error 12s (x2 over 14s) datavolume-controller Unable to connect to http data source: expected status code 200, got 404. Status: 404 Not Found Ready - If Type is Ready and Status is True , then the data volume is ready to be used, as in the following example. If the data volume is not ready to be used, the Status is False : Example output Status: Conditions: Last Heart Beat Time: 2020-07-15T04:31:39Z Last Transition Time: 2020-07-15T04:31:39Z Status: True Type: Ready 14.5. OpenShift Virtualization runbooks Runbooks for the OpenShift Virtualization Operator are maintained in the openshift/runbooks Git repository, and you can view them on GitHub. To diagnose and resolve issues that trigger OpenShift Virtualization alerts , follow the procedures in the runbooks. OpenShift Virtualization alerts are displayed in the Virtualization Overview tab in the web console. 14.5.1. CDIDataImportCronOutdated View the runbook for the CDIDataImportCronOutdated alert. 14.5.2. CDIDataVolumeUnusualRestartCount View the runbook for the CDIDataVolumeUnusualRestartCount alert. 14.5.3. CDIDefaultStorageClassDegraded View the runbook for the CDIDefaultStorageClassDegraded alert. 14.5.4. CDIMultipleDefaultVirtStorageClasses View the runbook for the CDIMultipleDefaultVirtStorageClasses alert. 14.5.5. CDINoDefaultStorageClass View the runbook for the CDINoDefaultStorageClass alert. 14.5.6. CDINotReady View the runbook for the CDINotReady alert. 14.5.7. CDIOperatorDown View the runbook for the CDIOperatorDown alert. 14.5.8. CDIStorageProfilesIncomplete View the runbook for the CDIStorageProfilesIncomplete alert. 14.5.9. CnaoDown View the runbook for the CnaoDown alert. 14.5.10. CnaoNMstateMigration View the runbook for the CnaoNMstateMigration alert. 14.5.11. HCOInstallationIncomplete View the runbook for the HCOInstallationIncomplete alert. 14.5.12. HPPNotReady View the runbook for the HPPNotReady alert. 14.5.13. HPPOperatorDown View the runbook for the HPPOperatorDown alert. 14.5.14. HPPSharingPoolPathWithOS View the runbook for the HPPSharingPoolPathWithOS alert. 14.5.15. KubemacpoolDown View the runbook for the KubemacpoolDown alert. 14.5.16. KubeMacPoolDuplicateMacsFound View the runbook for the KubeMacPoolDuplicateMacsFound alert. 14.5.17. KubeVirtComponentExceedsRequestedCPU The KubeVirtComponentExceedsRequestedCPU alert is deprecated . 14.5.18. KubeVirtComponentExceedsRequestedMemory The KubeVirtComponentExceedsRequestedMemory alert is deprecated . 14.5.19. KubeVirtCRModified View the runbook for the KubeVirtCRModified alert. 14.5.20. KubeVirtDeprecatedAPIRequested View the runbook for the KubeVirtDeprecatedAPIRequested alert. 14.5.21. KubeVirtNoAvailableNodesToRunVMs View the runbook for the KubeVirtNoAvailableNodesToRunVMs alert. 14.5.22. KubevirtVmHighMemoryUsage View the runbook for the KubevirtVmHighMemoryUsage alert. 14.5.23. KubeVirtVMIExcessiveMigrations View the runbook for the KubeVirtVMIExcessiveMigrations alert. 14.5.24. LowKVMNodesCount View the runbook for the LowKVMNodesCount alert. 14.5.25. LowReadyVirtControllersCount View the runbook for the LowReadyVirtControllersCount alert. 14.5.26. LowReadyVirtOperatorsCount View the runbook for the LowReadyVirtOperatorsCount alert. 14.5.27. LowVirtAPICount View the runbook for the LowVirtAPICount alert. 14.5.28. LowVirtControllersCount View the runbook for the LowVirtControllersCount alert. 14.5.29. LowVirtOperatorCount View the runbook for the LowVirtOperatorCount alert. 14.5.30. NetworkAddonsConfigNotReady View the runbook for the NetworkAddonsConfigNotReady alert. 14.5.31. NoLeadingVirtOperator View the runbook for the NoLeadingVirtOperator alert. 14.5.32. NoReadyVirtController View the runbook for the NoReadyVirtController alert. 14.5.33. NoReadyVirtOperator View the runbook for the NoReadyVirtOperator alert. 14.5.34. OrphanedVirtualMachineInstances View the runbook for the OrphanedVirtualMachineInstances alert. 14.5.35. OutdatedVirtualMachineInstanceWorkloads View the runbook for the OutdatedVirtualMachineInstanceWorkloads alert. 14.5.36. SingleStackIPv6Unsupported View the runbook for the SingleStackIPv6Unsupported alert. 14.5.37. SSPCommonTemplatesModificationReverted View the runbook for the SSPCommonTemplatesModificationReverted alert. 14.5.38. SSPDown View the runbook for the SSPDown alert. 14.5.39. SSPFailingToReconcile View the runbook for the SSPFailingToReconcile alert. 14.5.40. SSPHighRateRejectedVms View the runbook for the SSPHighRateRejectedVms alert. 14.5.41. SSPTemplateValidatorDown View the runbook for the SSPTemplateValidatorDown alert. 14.5.42. UnsupportedHCOModification View the runbook for the UnsupportedHCOModification alert. 14.5.43. VirtAPIDown View the runbook for the VirtAPIDown alert. 14.5.44. VirtApiRESTErrorsBurst View the runbook for the VirtApiRESTErrorsBurst alert. 14.5.45. VirtApiRESTErrorsHigh View the runbook for the VirtApiRESTErrorsHigh alert. 14.5.46. VirtControllerDown View the runbook for the VirtControllerDown alert. 14.5.47. VirtControllerRESTErrorsBurst View the runbook for the VirtControllerRESTErrorsBurst alert. 14.5.48. VirtControllerRESTErrorsHigh View the runbook for the VirtControllerRESTErrorsHigh alert. 14.5.49. VirtHandlerDaemonSetRolloutFailing View the runbook for the VirtHandlerDaemonSetRolloutFailing alert. 14.5.50. VirtHandlerRESTErrorsBurst View the runbook for the VirtHandlerRESTErrorsBurst alert. 14.5.51. VirtHandlerRESTErrorsHigh View the runbook for the VirtHandlerRESTErrorsHigh alert. 14.5.52. VirtOperatorDown View the runbook for the VirtOperatorDown alert. 14.5.53. VirtOperatorRESTErrorsBurst View the runbook for the VirtOperatorRESTErrorsBurst alert. 14.5.54. VirtOperatorRESTErrorsHigh View the runbook for the VirtOperatorRESTErrorsHigh alert. 14.5.55. VirtualMachineCRCErrors The runbook for the VirtualMachineCRCErrors alert is deprecated because the alert was renamed to VMStorageClassWarning . View the runbook for the VMStorageClassWarning alert. 14.5.56. VMCannotBeEvicted View the runbook for the VMCannotBeEvicted alert. 14.5.57. VMStorageClassWarning View the runbook for the VMStorageClassWarning alert.

Commands:
oc adm must-gather --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.13.11 -- /usr/bin/gather
oc adm must-gather --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.13.11 -- <environment_variable_1> <environment_variable_2> <script_name>
oc adm must-gather --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.13.11 -- PROS=5 /usr/bin/gather 1
oc adm must-gather --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.13.11 -- NS=mynamespace VM=my-vm /usr/bin/gather --vms_details 1
oc adm must-gather --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.13.11 /usr/bin/gather --images
--- apiVersion: v1 kind: ServiceAccount metadata: name: vm-latency-checkup-sa --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kubevirt-vm-latency-checker rules: - apiGroups: ["kubevirt.io"] resources: ["virtualmachineinstances"] verbs: ["get", "create", "delete"] - apiGroups: ["subresources.kubevirt.io"] resources: ["virtualmachineinstances/console"] verbs: ["get"] - apiGroups: ["k8s.cni.cncf.io"] resources: ["network-attachment-definitions"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kubevirt-vm-latency-checker subjects: - kind: ServiceAccount name: vm-latency-checkup-sa roleRef: kind: Role name: kubevirt-vm-latency-checker apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kiagnose-configmap-access rules: - apiGroups: [ "" ] resources: [ "configmaps" ] verbs: ["get", "update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kiagnose-configmap-access subjects: - kind: ServiceAccount name: vm-latency-checkup-sa roleRef: kind: Role name: kiagnose-configmap-access apiGroup: rbac.authorization.k8s.io
oc apply -n <target_namespace> -f <latency_sa_roles_rolebinding>.yaml 1
apiVersion: v1 kind: ConfigMap metadata: name: kubevirt-vm-latency-checkup-config data: spec.timeout: 5m spec.param.networkAttachmentDefinitionNamespace: <target_namespace> spec.param.networkAttachmentDefinitionName: "blue-network" 1 spec.param.maxDesiredLatencyMilliseconds: "10" 2 spec.param.sampleDurationSeconds: "5" 3 spec.param.sourceNode: "worker1" 4 spec.param.targetNode: "worker2" 5
oc apply -n <target_namespace> -f <latency_config_map>.yaml
apiVersion: batch/v1 kind: Job metadata: name: kubevirt-vm-latency-checkup spec: backoffLimit: 0 template: spec: serviceAccountName: vm-latency-checkup-sa restartPolicy: Never containers: - name: vm-latency-checkup image: registry.redhat.io/container-native-virtualization/vm-network-latency-checkup-rhel9:v4.13.0 securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] runAsNonRoot: true seccompProfile: type: "RuntimeDefault" env: - name: CONFIGMAP_NAMESPACE value: <target_namespace> - name: CONFIGMAP_NAME value: kubevirt-vm-latency-checkup-config - name: POD_UID valueFrom: fieldRef: fieldPath: metadata.uid
oc apply -n <target_namespace> -f <latency_job>.yaml
oc wait job kubevirt-vm-latency-checkup -n <target_namespace> --for condition=complete --timeout 6m
oc get configmap kubevirt-vm-latency-checkup-config -n <target_namespace> -o yaml
apiVersion: v1 kind: ConfigMap metadata: name: kubevirt-vm-latency-checkup-config namespace: <target_namespace> data: spec.timeout: 5m spec.param.networkAttachmentDefinitionNamespace: <target_namespace> spec.param.networkAttachmentDefinitionName: "blue-network" spec.param.maxDesiredLatencyMilliseconds: "10" spec.param.sampleDurationSeconds: "5" spec.param.sourceNode: "worker1" spec.param.targetNode: "worker2" status.succeeded: "true" status.failureReason: "" status.completionTimestamp: "2022-01-01T09:00:00Z" status.startTimestamp: "2022-01-01T09:00:07Z" status.result.avgLatencyNanoSec: "177000" status.result.maxLatencyNanoSec: "244000" 1 status.result.measurementDurationSec: "5" status.result.minLatencyNanoSec: "135000" status.result.sourceNode: "worker1" status.result.targetNode: "worker2"
oc logs job.batch/kubevirt-vm-latency-checkup -n <target_namespace>
oc delete job -n <target_namespace> kubevirt-vm-latency-checkup
oc delete config-map -n <target_namespace> kubevirt-vm-latency-checkup-config
oc delete -f <latency_sa_roles_rolebinding>.yaml
--- apiVersion: v1 kind: ServiceAccount metadata: name: dpdk-checkup-sa --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kiagnose-configmap-access rules: - apiGroups: [ "" ] resources: [ "configmaps" ] verbs: [ "get", "update" ] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kiagnose-configmap-access subjects: - kind: ServiceAccount name: dpdk-checkup-sa roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kiagnose-configmap-access --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kubevirt-dpdk-checker rules: - apiGroups: [ "kubevirt.io" ] resources: [ "virtualmachineinstances" ] verbs: [ "create", "get", "delete" ] - apiGroups: [ "subresources.kubevirt.io" ] resources: [ "virtualmachineinstances/console" ] verbs: [ "get" ] - apiGroups: [ "" ] resources: [ "pods" ] verbs: [ "create", "get", "delete" ] - apiGroups: [ "" ] resources: [ "pods/exec" ] verbs: [ "create" ] - apiGroups: [ "k8s.cni.cncf.io" ] resources: [ "network-attachment-definitions" ] verbs: [ "get" ] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kubevirt-dpdk-checker subjects: - kind: ServiceAccount name: dpdk-checkup-sa roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubevirt-dpdk-checker --- apiVersion: v1 kind: ServiceAccount metadata: name: dpdk-checkup-traffic-gen-sa
oc apply -n <target_namespace> -f <dpdk_sa_roles_rolebinding>.yaml
apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: name: dpdk-checkup-traffic-gen allowHostDirVolumePlugin: true allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: false allowPrivilegedContainer: false allowedCapabilities: - IPC_LOCK - NET_ADMIN - NET_RAW - SYS_RESOURCE defaultAddCapabilities: null fsGroup: type: RunAsAny groups: [] readOnlyRootFilesystem: false requiredDropCapabilities: null runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny seccompProfiles: - runtime/default - unconfined supplementalGroups: type: RunAsAny users: - system:serviceaccount:dpdk-checkup-ns:dpdk-checkup-traffic-gen-sa
oc apply -f <dpdk_scc>.yaml
apiVersion: v1 kind: ConfigMap metadata: name: dpdk-checkup-config data: spec.timeout: 10m spec.param.networkAttachmentDefinitionName: <network_name> 1 spec.param.trafficGeneratorRuntimeClassName: <runtimeclass_name> 2 spec.param.trafficGeneratorImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-traffic-gen:v0.1.1" 3 spec.param.vmContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-vm:v0.1.1" 4
oc apply -n <target_namespace> -f <dpdk_config_map>.yaml
apiVersion: batch/v1 kind: Job metadata: name: dpdk-checkup spec: backoffLimit: 0 template: spec: serviceAccountName: dpdk-checkup-sa restartPolicy: Never containers: - name: dpdk-checkup image: registry.redhat.io/container-native-virtualization/kubevirt-dpdk-checkup-rhel9:v4.13.0 imagePullPolicy: Always securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] runAsNonRoot: true seccompProfile: type: "RuntimeDefault" env: - name: CONFIGMAP_NAMESPACE value: <target-namespace> - name: CONFIGMAP_NAME value: dpdk-checkup-config - name: POD_UID valueFrom: fieldRef: fieldPath: metadata.uid
oc apply -n <target_namespace> -f <dpdk_job>.yaml
oc wait job dpdk-checkup -n <target_namespace> --for condition=complete --timeout 10m
oc get configmap dpdk-checkup-config -n <target_namespace> -o yaml
apiVersion: v1 kind: ConfigMap metadata: name: dpdk-checkup-config data: spec.timeout: 1h2m spec.param.NetworkAttachmentDefinitionName: "mlx-dpdk-network-1" spec.param.trafficGeneratorRuntimeClassName: performance-performance-zeus10 spec.param.trafficGeneratorImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-traffic-gen:v0.1.1" spec.param.vmContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-vm:v0.1.1" status.succeeded: true status.failureReason: " " status.startTimestamp: 2022-12-21T09:33:06+00:00 status.completionTimestamp: 2022-12-21T11:33:06+00:00 status.result.actualTrafficGeneratorTargetNode: worker-dpdk1 status.result.actualDPDKVMTargetNode: worker-dpdk2 status.result.dropRate: 0
oc delete job -n <target_namespace> dpdk-checkup
oc delete config-map -n <target_namespace> dpdk-checkup-config
oc delete -f <dpdk_sa_roles_rolebinding>.yaml
dnf install libguestfs-tools
composer-cli distros list
usermod -a -G weldr user
newgrp weldr
cat << EOF > dpdk-vm.toml name = "dpdk_image" description = "Image to use with the DPDK checkup" version = "0.0.1" distro = "rhel-87" [[packages]] name = "dpdk" [[packages]] name = "dpdk-tools" [[packages]] name = "driverctl" [[packages]] name = "tuned-profiles-cpu-partitioning" [customizations.kernel] append = "default_hugepagesz=1GB hugepagesz=1G hugepages=8 isolcpus=2-7" [customizations.services] disabled = ["NetworkManager-wait-online", "sshd"] EOF
composer-cli blueprints push dpdk-vm.toml
composer-cli compose start dpdk_image qcow2
composer-cli compose status
composer-cli compose image <UUID>
cat <<EOF >customize-vm echo isolated_cores=2-7 > /etc/tuned/cpu-partitioning-variables.conf tuned-adm profile cpu-partitioning echo "options vfio enable_unsafe_noiommu_mode=1" > /etc/modprobe.d/vfio-noiommu.conf EOF
cat <<EOF >first-boot driverctl set-override 0000:06:00.0 vfio-pci driverctl set-override 0000:07:00.0 vfio-pci mkdir /mnt/huge mount /mnt/huge --source nodev -t hugetlbfs -o pagesize=1GB EOF
virt-customize -a <UUID>.qcow2 --run=customize-vm --firstboot=first-boot --selinux-relabel
cat << EOF > Dockerfile FROM scratch COPY <uuid>-disk.qcow2 /disk/ EOF
podman build . -t dpdk-rhel:latest
podman push dpdk-rhel:latest
topk(3, sum by (name, namespace) (rate(kubevirt_vmi_vcpu_wait_seconds[6m]))) > 0 1
topk(3, sum by (name, namespace) (rate(kubevirt_vmi_network_receive_bytes_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_network_transmit_bytes_total[6m]))) > 0 1
topk(3, sum by (name, namespace) (rate(kubevirt_vmi_storage_read_traffic_bytes_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_storage_write_traffic_bytes_total[6m]))) > 0 1
kubevirt_vmsnapshot_disks_restored_from_source_total{vm_name="simple-vm", vm_namespace="default"} 1
kubevirt_vmsnapshot_disks_restored_from_source_bytes{vm_name="simple-vm", vm_namespace="default"} 1
topk(3, sum by (name, namespace) (rate(kubevirt_vmi_storage_iops_read_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_storage_iops_write_total[6m]))) > 0 1
topk(3, sum by (name, namespace) (rate(kubevirt_vmi_memory_swap_in_traffic_bytes_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_memory_swap_out_traffic_bytes_total[6m]))) > 0 1
kind: Service apiVersion: v1 metadata: name: node-exporter-service 1 namespace: dynamation 2 labels: servicetype: metrics 3 spec: ports: - name: exmet 4 protocol: TCP port: 9100 5 targetPort: 9100 6 type: ClusterIP selector: monitor: metrics 7
oc create -f node-exporter-service.yaml
wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
sudo tar xvf node_exporter-1.3.1.linux-amd64.tar.gz --directory /usr/bin --strip 1 "*/node_exporter"
[Unit] Description=Prometheus Metrics Exporter After=network.target StartLimitIntervalSec=0 [Service] Type=simple Restart=always RestartSec=1 User=root ExecStart=/usr/bin/node_exporter [Install] WantedBy=multi-user.target
sudo systemctl enable node_exporter.service sudo systemctl start node_exporter.service
curl http://localhost:9100/metrics
go_gc_duration_seconds{quantile="0"} 1.5244e-05 go_gc_duration_seconds{quantile="0.25"} 3.0449e-05 go_gc_duration_seconds{quantile="0.5"} 3.7913e-05
spec: template: metadata: labels: monitor: metrics
oc get service -n <namespace> <node-exporter-service>
curl http://<172.30.226.162:9100>/metrics | grep -vE "^#|^USD"
node_arp_entries{device="eth0"} 1 node_boot_time_seconds 1.643153218e+09 node_context_switches_total 4.4938158e+07 node_cooling_device_cur_state{name="0",type="Processor"} 0 node_cooling_device_max_state{name="0",type="Processor"} 0 node_cpu_guest_seconds_total{cpu="0",mode="nice"} 0 node_cpu_guest_seconds_total{cpu="0",mode="user"} 0 node_cpu_seconds_total{cpu="0",mode="idle"} 1.10586485e+06 node_cpu_seconds_total{cpu="0",mode="iowait"} 37.61 node_cpu_seconds_total{cpu="0",mode="irq"} 233.91 node_cpu_seconds_total{cpu="0",mode="nice"} 551.47 node_cpu_seconds_total{cpu="0",mode="softirq"} 87.3 node_cpu_seconds_total{cpu="0",mode="steal"} 86.12 node_cpu_seconds_total{cpu="0",mode="system"} 464.15 node_cpu_seconds_total{cpu="0",mode="user"} 1075.2 node_disk_discard_time_seconds_total{device="vda"} 0 node_disk_discard_time_seconds_total{device="vdb"} 0 node_disk_discarded_sectors_total{device="vda"} 0 node_disk_discarded_sectors_total{device="vdb"} 0 node_disk_discards_completed_total{device="vda"} 0 node_disk_discards_completed_total{device="vdb"} 0 node_disk_discards_merged_total{device="vda"} 0 node_disk_discards_merged_total{device="vdb"} 0 node_disk_info{device="vda",major="252",minor="0"} 1 node_disk_info{device="vdb",major="252",minor="16"} 1 node_disk_io_now{device="vda"} 0 node_disk_io_now{device="vdb"} 0 node_disk_io_time_seconds_total{device="vda"} 174 node_disk_io_time_seconds_total{device="vdb"} 0.054 node_disk_io_time_weighted_seconds_total{device="vda"} 259.79200000000003 node_disk_io_time_weighted_seconds_total{device="vdb"} 0.039 node_disk_read_bytes_total{device="vda"} 3.71867136e+08 node_disk_read_bytes_total{device="vdb"} 366592 node_disk_read_time_seconds_total{device="vda"} 19.128 node_disk_read_time_seconds_total{device="vdb"} 0.039 node_disk_reads_completed_total{device="vda"} 5619 node_disk_reads_completed_total{device="vdb"} 96 node_disk_reads_merged_total{device="vda"} 5 node_disk_reads_merged_total{device="vdb"} 0 node_disk_write_time_seconds_total{device="vda"} 240.66400000000002 node_disk_write_time_seconds_total{device="vdb"} 0 node_disk_writes_completed_total{device="vda"} 71584 node_disk_writes_completed_total{device="vdb"} 0 node_disk_writes_merged_total{device="vda"} 19761 node_disk_writes_merged_total{device="vdb"} 0 node_disk_written_bytes_total{device="vda"} 2.007924224e+09 node_disk_written_bytes_total{device="vdb"} 0
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: k8s-app: node-exporter-metrics-monitor name: node-exporter-metrics-monitor 1 namespace: dynamation 2 spec: endpoints: - interval: 30s 3 port: exmet 4 scheme: http selector: matchLabels: servicetype: metrics
oc create -f node-exporter-metrics-monitor.yaml
oc expose service -n <namespace> <node_exporter_service_name>
oc get route -o=custom-columns=NAME:.metadata.name,DNS:.spec.host
NAME DNS node-exporter-service node-exporter-service-dynamation.apps.cluster.example.org
curl -s http://node-exporter-service-dynamation.apps.cluster.example.org/metrics
go_gc_duration_seconds{quantile="0"} 1.5382e-05 go_gc_duration_seconds{quantile="0.25"} 3.1163e-05 go_gc_duration_seconds{quantile="0.5"} 3.8546e-05 go_gc_duration_seconds{quantile="0.75"} 4.9139e-05 go_gc_duration_seconds{quantile="1"} 0.000189423
spec: readinessProbe: httpGet: 1 port: 1500 2 path: /healthz 3 httpHeaders: - name: Custom-Header value: Awesome initialDelaySeconds: 120 4 periodSeconds: 20 5 timeoutSeconds: 10 6 failureThreshold: 3 7 successThreshold: 3 8
oc create -f <file_name>.yaml
spec: readinessProbe: initialDelaySeconds: 120 1 periodSeconds: 20 2 tcpSocket: 3 port: 1500 4 timeoutSeconds: 10 5
oc create -f <file_name>.yaml
spec: livenessProbe: initialDelaySeconds: 120 1 periodSeconds: 20 2 httpGet: 3 port: 1500 4 path: /healthz 5 httpHeaders: - name: Custom-Header value: Awesome timeoutSeconds: 10 6
oc create -f <file_name>.yaml
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: labels: kubevirt.io/vm: vm2-rhel84-watchdog name: <vm-name> spec: running: false template: metadata: labels: kubevirt.io/vm: vm2-rhel84-watchdog spec: domain: devices: watchdog: name: <watchdog> i6300esb: action: "poweroff" 1
oc apply -f <file_name>.yaml
lspci | grep watchdog -i
echo c > /proc/sysrq-trigger
pkill -9 watchdog
yum install watchdog
#watchdog-device = /dev/watchdog
systemctl enable --now watchdog.service
spec: readinessProbe: guestAgentPing: {} 1 initialDelaySeconds: 120 2 periodSeconds: 20 3 timeoutSeconds: 10 4 failureThreshold: 3 5 successThreshold: 3 6
oc create -f <file_name>.yaml
oc get events -n <namespace>
oc describe <resource> <resource_name>
oc get pods -n openshift-cnv
NAME READY STATUS RESTARTS AGE disks-images-provider-7gqbc 1/1 Running 0 32m disks-images-provider-vg4kx 1/1 Running 0 32m virt-api-57fcc4497b-7qfmc 1/1 Running 0 31m virt-api-57fcc4497b-tx9nc 1/1 Running 0 31m virt-controller-76c784655f-7fp6m 1/1 Running 0 30m virt-controller-76c784655f-f4pbd 1/1 Running 0 30m virt-handler-2m86x 1/1 Running 0 30m virt-handler-9qs6z 1/1 Running 0 30m virt-operator-7ccfdbf65f-q5snk 1/1 Running 0 32m virt-operator-7ccfdbf65f-vllz8 1/1 Running 0 32m
oc logs -n openshift-cnv <pod_name>
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373695Z"} {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373726Z"} {"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:462","timestamp":"2022-04-17T08:58:37.373782Z"} {"component":"virt-handler","level":"info","msg":"CPU features of a minimum baseline CPU model: map[apic:true clflush:true cmov:true cx16:true cx8:true de:true fpu:true fxsr:true lahf_lm:true lm:true mca:true mce:true mmx:true msr:true mtrr:true nx:true pae:true pat:true pge:true pni:true pse:true pse36:true sep:true sse:true sse2:true sse4.1:true ssse3:true syscall:true tsc:true]","pos":"cpu_plugin.go:96","timestamp":"2022-04-17T08:58:37.390221Z"} {"component":"virt-handler","level":"warning","msg":"host model mode is expected to contain only one model","pos":"cpu_plugin.go:103","timestamp":"2022-04-17T08:58:37.390263Z"} {"component":"virt-handler","level":"info","msg":"node-labeller is running","pos":"node_labeller.go:94","timestamp":"2022-04-17T08:58:37.391011Z"}
oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged spec: logVerbosityConfig: kubevirt: virtAPI: 5 1 virtController: 4 virtHandler: 3 virtLauncher: 2 virtOperator: 6
{log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster"
{log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" |kubernetes_labels_app_kubernetes_io_component="storage"
{log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" |kubernetes_labels_app_kubernetes_io_component="deployment"
{log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" |kubernetes_labels_app_kubernetes_io_component="network"
{log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" |kubernetes_labels_app_kubernetes_io_component="compute"
{log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" |kubernetes_labels_app_kubernetes_io_component="schedule"
{log_type=~".+",kubernetes_container_name=~"<container>|<container>"} 1 |json|kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster"
{log_type=~".+", kubernetes_container_name="compute"}|json |!= "custom-ga-command" 1
{log_type=~".+"}|json |kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster" |= "error" != "timeout"
oc describe dv <DataVolume>
Status: Conditions: Last Heart Beat Time: 2020-07-15T03:58:24Z Last Transition Time: 2020-07-15T03:58:24Z Message: PVC win10-rootdisk Bound Reason: Bound Status: True Type: Bound Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Bound 24s datavolume-controller PVC example-dv Bound
Status: Conditions: Last Heart Beat Time: 2020-07-15T04:31:39Z Last Transition Time: 2020-07-15T04:31:39Z Message: Import Complete Reason: Completed Status: False Type: Running Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Error 12s (x2 over 14s) datavolume-controller Unable to connect to http data source: expected status code 200, got 404. Status: 404 Not Found
Status: Conditions: Last Heart Beat Time: 2020-07-15T04:31:39Z Last Transition Time: 2020-07-15T04:31:39Z Status: True Type: Ready