Chapter 13. Nodes 13.1. Node maintenance Nodes can be placed into maintenance mode by using the oc adm utility or NodeMaintenance custom resources (CRs). Note The node-maintenance-operator (NMO) is no longer shipped with OpenShift Virtualization. It is deployed as a standalone Operator from the OperatorHub in the OpenShift Container Platform web console or by using the OpenShift CLI ( oc ). For more information on remediation, fencing, and maintaining nodes, see the Workload Availability for Red Hat OpenShift documentation. Important Virtual machines (VMs) must have a persistent volume claim (PVC) with a shared ReadWriteMany (RWX) access mode to be live migrated. The Node Maintenance Operator watches for new or deleted NodeMaintenance CRs. When a new NodeMaintenance CR is detected, no new workloads are scheduled and the node is cordoned off from the rest of the cluster. All pods that can be evicted are evicted from the node. When a NodeMaintenance CR is deleted, the node that is referenced in the CR is made available for new workloads. Note Using a NodeMaintenance CR for node maintenance tasks achieves the same results as the oc adm cordon and oc adm drain commands using standard OpenShift Container Platform custom resource processing. 13.1.1. Eviction strategies Placing a node into maintenance marks the node as unschedulable and drains all the VMs and pods from it. You can configure eviction strategies for virtual machines (VMs) or for the cluster. VM eviction strategy The VM LiveMigrate eviction strategy ensures that a virtual machine instance (VMI) is not interrupted if the node is placed into maintenance or drained. VMIs with this eviction strategy will be live migrated to another node. You can configure eviction strategies for virtual machines (VMs) by using the OpenShift Container Platform web console or the command line . Important The default eviction strategy is LiveMigrate . A non-migratable VM with a LiveMigrate eviction strategy might prevent nodes from draining or block an infrastructure upgrade because the VM is not evicted from the node. This situation causes a migration to remain in a Pending or Scheduling state unless you shut down the VM manually. You must set the eviction strategy of non-migratable VMs to LiveMigrateIfPossible , which does not block an upgrade, or to None , for VMs that should not be migrated. Cluster eviction strategy You can configure an eviction strategy for the cluster to prioritize workload continuity or infrastructure upgrade. Table 13.1. Cluster eviction strategies Eviction strategy Description Interrupts workflow Blocks upgrades LiveMigrate 1 Prioritizes workload continuity over upgrades. No Yes 2 LiveMigrateIfPossible Prioritizes upgrades over workload continuity to ensure that the environment is updated. Yes No None 3 Shuts down VMs with no eviction strategy. Yes No Default eviction strategy for multi-node clusters. If a VM blocks an upgrade, you must shut down the VM manually. Default eviction strategy for single-node OpenShift. 13.1.1.1. Configuring a VM eviction strategy using the command line You can configure an eviction strategy for a virtual machine (VM) by using the command line. Important The default eviction strategy is LiveMigrate . A non-migratable VM with a LiveMigrate eviction strategy might prevent nodes from draining or block an infrastructure upgrade because the VM is not evicted from the node. This situation causes a migration to remain in a Pending or Scheduling state unless you shut down the VM manually. You must set the eviction strategy of non-migratable VMs to LiveMigrateIfPossible , which does not block an upgrade, or to None , for VMs that should not be migrated. Procedure Edit the VirtualMachine resource by running the following command: USD oc edit vm <vm_name> -n <namespace> Example eviction strategy apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: <vm_name> spec: template: spec: evictionStrategy: LiveMigrateIfPossible 1 # ... 1 Specify the eviction strategy. The default value is LiveMigrate . Restart the VM to apply the changes: USD virtctl restart <vm_name> -n <namespace> 13.1.1.2. Configuring a cluster eviction strategy by using the command line You can configure an eviction strategy for a cluster by using the command line. Procedure Edit the hyperconverged resource by running the following command: USD oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv Set the cluster eviction strategy as shown in the following example: Example cluster eviction strategy apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged spec: evictionStrategy: LiveMigrate # ... 13.1.2. Run strategies The spec.runStrategy key determines how a VM behaves under certain conditions. 13.1.2.1. Run strategies The spec.runStrategy key has four possible values: Always The virtual machine instance (VMI) is always present when a virtual machine (VM) is created on another node. A new VMI is created if the original stops for any reason. RerunOnFailure The VMI is re-created on another node if the instance fails. The instance is not re-created if the VM stops successfully, such as when it is shut down. Manual You control the VMI state manually with the start , stop , and restart virtctl client commands. The VM is not automatically restarted. Halted No VMI is present when a VM is created. Different combinations of the virtctl start , stop and restart commands affect the run strategy. The following table describes a VM's transition between states. The first column shows the VM's initial run strategy. The remaining columns show a virtctl command and the new run strategy after that command is run. Table 13.2. Run strategy before and after virtctl commands Initial run strategy Start Stop Restart Always - Halted Always RerunOnFailure - Halted RerunOnFailure Manual Manual Manual Manual Halted Always - - Note If a node in a cluster installed by using installer-provisioned infrastructure fails the machine health check and is unavailable, VMs with runStrategy: Always or runStrategy: RerunOnFailure are rescheduled on a new node. 13.1.2.2. Configuring a VM run strategy by using the command line You can configure a run strategy for a virtual machine (VM) by using the command line. Procedure Edit the VirtualMachine resource by running the following command: USD oc edit vm <vm_name> -n <namespace> Example run strategy apiVersion: kubevirt.io/v1 kind: VirtualMachine spec: runStrategy: Always # ... 13.1.3. Maintaining bare metal nodes When you deploy OpenShift Container Platform on bare metal infrastructure, there are additional considerations that must be taken into account compared to deploying on cloud infrastructure. Unlike in cloud environments where the cluster nodes are considered ephemeral, re-provisioning a bare metal node requires significantly more time and effort for maintenance tasks. When a bare metal node fails, for example, if a fatal kernel error happens or a NIC card hardware failure occurs, workloads on the failed node need to be restarted elsewhere else on the cluster while the problem node is repaired or replaced. Node maintenance mode allows cluster administrators to gracefully power down nodes, moving workloads to other parts of the cluster and ensuring workloads do not get interrupted. Detailed progress and node status details are provided during maintenance. 13.1.4. Additional resources About live migration 13.2. Managing node labeling for obsolete CPU models You can schedule a virtual machine (VM) on a node as long as the VM CPU model and policy are supported by the node. 13.2.1. About node labeling for obsolete CPU models The OpenShift Virtualization Operator uses a predefined list of obsolete CPU models to ensure that a node supports only valid CPU models for scheduled VMs. By default, the following CPU models are eliminated from the list of labels generated for the node: Example 13.1. Obsolete CPU models This predefined list is not visible in the HyperConverged CR. You cannot remove CPU models from this list, but you can add to the list by editing the spec.obsoleteCPUs.cpuModels field of the HyperConverged CR. 13.2.2. About node labeling for CPU features Through the process of iteration, the base CPU features in the minimum CPU model are eliminated from the list of labels generated for the node. For example: An environment might have two supported CPU models: Penryn and Haswell . If Penryn is specified as the CPU model for minCPU , each base CPU feature for Penryn is compared to the list of CPU features supported by Haswell . Example 13.2. CPU features supported by Penryn Example 13.3. CPU features supported by Haswell If both Penryn and Haswell support a specific CPU feature, a label is not created for that feature. Labels are generated for CPU features that are supported only by Haswell and not by Penryn . Example 13.4. Node labels created for CPU features after iteration 13.2.3. Configuring obsolete CPU models You can configure a list of obsolete CPU models by editing the HyperConverged custom resource (CR). Procedure Edit the HyperConverged custom resource, specifying the obsolete CPU models in the obsoleteCPUs array. For example: apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: obsoleteCPUs: cpuModels: 1 - "<obsolete_cpu_1>" - "<obsolete_cpu_2>" minCPUModel: "<minimum_cpu_model>" 2 1 Replace the example values in the cpuModels array with obsolete CPU models. Any value that you specify is added to a predefined list of obsolete CPU models. The predefined list is not visible in the CR. 2 Replace this value with the minimum CPU model that you want to use for basic CPU features. If you do not specify a value, Penryn is used by default. 13.3. Preventing node reconciliation Use skip-node annotation to prevent the node-labeller from reconciling a node. 13.3.1. Using skip-node annotation If you want the node-labeller to skip a node, annotate that node by using the oc CLI. Prerequisites You have installed the OpenShift CLI ( oc ). Procedure Annotate the node that you want to skip by running the following command: USD oc annotate node <node_name> node-labeller.kubevirt.io/skip-node=true 1 1 Replace <node_name> with the name of the relevant node to skip. Reconciliation resumes on the cycle after the node annotation is removed or set to false. 13.3.2. Additional resources Managing node labeling for obsolete CPU models 13.4. Deleting a failed node to trigger virtual machine failover If a node fails and node health checks are not deployed on your cluster, virtual machines (VMs) with runStrategy: Always configured are not automatically relocated to healthy nodes. 13.4.1. Prerequisites A node where a virtual machine was running has the NotReady condition . The virtual machine that was running on the failed node has runStrategy set to Always . You have installed the OpenShift CLI ( oc ). 13.4.2. Deleting nodes from a bare metal cluster When you delete a node using the CLI, the node object is deleted in Kubernetes, but the pods that exist on the node are not deleted. Any bare pods not backed by a replication controller become inaccessible to OpenShift Container Platform. Pods backed by replication controllers are rescheduled to other available nodes. You must delete local manifest pods. Procedure Delete a node from an OpenShift Container Platform cluster running on bare metal by completing the following steps: Mark the node as unschedulable: USD oc adm cordon <node_name> Drain all pods on the node: USD oc adm drain <node_name> --force=true This step might fail if the node is offline or unresponsive. Even if the node does not respond, it might still be running a workload that writes to shared storage. To avoid data corruption, power down the physical hardware before you proceed. Delete the node from the cluster: USD oc delete node <node_name> Although the node object is now deleted from the cluster, it can still rejoin the cluster after reboot or if the kubelet service is restarted. To permanently delete the node and all its data, you must decommission the node . If you powered down the physical hardware, turn it back on so that the node can rejoin the cluster. 13.4.3. Verifying virtual machine failover After all resources are terminated on the unhealthy node, a new virtual machine instance (VMI) is automatically created on a healthy node for each relocated VM. To confirm that the VMI was created, view all VMIs by using the oc CLI. 13.4.3.1. Listing all virtual machine instances using the CLI You can list all virtual machine instances (VMIs) in your cluster, including standalone VMIs and those owned by virtual machines, by using the oc command-line interface (CLI). Procedure List all VMIs by running the following command: USD oc get vmis -A 13.5. Activating kernel samepage merging (KSM) OpenShift Virtualization can activate kernel samepage merging (KSM) when nodes are overloaded. KSM deduplicates identical data found in the memory pages of virtual machines (VMs). If you have very similar VMs, KSM can make it possible to schedule more VMs on a single node. Important You must only use KSM with trusted workloads. 13.5.1. Prerequisites Ensure that an administrator has configured KSM support on any nodes where you want OpenShift Virtualization to activate KSM. 13.5.2. About using OpenShift Virtualization to activate KSM You can configure OpenShift Virtualization to activate kernel samepage merging (KSM) when nodes experience memory overload. 13.5.2.1. Configuration methods You can enable or disable the KSM activation feature for all nodes by using the OpenShift Container Platform web console or by editing the HyperConverged custom resource (CR). The HyperConverged CR supports more granular configuration. CR configuration You can configure the KSM activation feature by editing the spec.configuration.ksmConfiguration stanza of the HyperConverged CR. You enable the feature and configure settings by editing the ksmConfiguration stanza. You disable the feature by deleting the ksmConfiguration stanza. You can allow OpenShift Virtualization to enable KSM on only a subset of nodes by adding node selection syntax to the ksmConfiguration.nodeLabelSelector field. Note Even if the KSM activation feature is disabled in OpenShift Virtualization, an administrator can still enable KSM on nodes that support it. 13.5.2.2. KSM node labels OpenShift Virtualization identifies nodes that are configured to support KSM and applies the following node labels: kubevirt.io/ksm-handler-managed: "false" This label is set to "true" when OpenShift Virtualization activates KSM on a node that is experiencing memory overload. This label is not set to "true" if an administrator activates KSM. kubevirt.io/ksm-enabled: "false" This label is set to "true" when KSM is activated on a node, even if OpenShift Virtualization did not activate KSM. These labels are not applied to nodes that do not support KSM. 13.5.3. Configuring KSM activation by using the web console You can allow OpenShift Virtualization to activate kernel samepage merging (KSM) on all nodes in your cluster by using the OpenShift Container Platform web console. Procedure From the side menu, click Virtualization Overview . Select the Settings tab. Select the Cluster tab. Expand Resource management . Enable or disable the feature for all nodes: Set Kernel Samepage Merging (KSM) to on. Set Kernel Samepage Merging (KSM) to off. 13.5.4. Configuring KSM activation by using the CLI You can enable or disable OpenShift Virtualization's kernel samepage merging (KSM) activation feature by editing the HyperConverged custom resource (CR). Use this method if you want OpenShift Virtualization to activate KSM on only a subset of nodes. Procedure Open the HyperConverged CR in your default editor by running the following command: USD oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv Edit the ksmConfiguration stanza: To enable the KSM activation feature for all nodes, set the nodeLabelSelector value to {} . For example: apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: configuration: ksmConfiguration: nodeLabelSelector: {} # ... To enable the KSM activation feature on a subset of nodes, edit the nodeLabelSelector field. Add syntax that matches the nodes where you want OpenShift Virtualization to enable KSM. For example, the following configuration allows OpenShift Virtualization to enable KSM on nodes where both <first_example_key> and <second_example_key> are set to "true" : apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: configuration: ksmConfiguration: nodeLabelSelector: matchLabels: <first_example_key>: "true" <second_example_key>: "true" # ... To disable the KSM activation feature, delete the ksmConfiguration stanza. For example: apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: configuration: # ... Save the file. 13.5.5. Additional resources Specifying nodes for virtual machines Placing pods on specific nodes using node selectors Managing kernel samepage merging in the Red Hat Enterprise Linux (RHEL) documentation

Commands:
oc edit vm <vm_name> -n <namespace>
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: <vm_name> spec: template: spec: evictionStrategy: LiveMigrateIfPossible 1
virtctl restart <vm_name> -n <namespace>
oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged spec: evictionStrategy: LiveMigrate
oc edit vm <vm_name> -n <namespace>
apiVersion: kubevirt.io/v1 kind: VirtualMachine spec: runStrategy: Always
"486" Conroe athlon core2duo coreduo kvm32 kvm64 n270 pentium pentium2 pentium3 pentiumpro phenom qemu32 qemu64
apic clflush cmov cx16 cx8 de fpu fxsr lahf_lm lm mca mce mmx msr mtrr nx pae pat pge pni pse pse36 sep sse sse2 sse4.1 ssse3 syscall tsc
aes apic avx avx2 bmi1 bmi2 clflush cmov cx16 cx8 de erms fma fpu fsgsbase fxsr hle invpcid lahf_lm lm mca mce mmx movbe msr mtrr nx pae pat pcid pclmuldq pge pni popcnt pse pse36 rdtscp rtm sep smep sse sse2 sse4.1 sse4.2 ssse3 syscall tsc tsc-deadline x2apic xsave
aes avx avx2 bmi1 bmi2 erms fma fsgsbase hle invpcid movbe pcid pclmuldq popcnt rdtscp rtm sse4.2 tsc-deadline x2apic xsave
apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: obsoleteCPUs: cpuModels: 1 - "<obsolete_cpu_1>" - "<obsolete_cpu_2>" minCPUModel: "<minimum_cpu_model>" 2
oc annotate node <node_name> node-labeller.kubevirt.io/skip-node=true 1
oc adm cordon <node_name>
oc adm drain <node_name> --force=true
oc delete node <node_name>
oc get vmis -A
oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: configuration: ksmConfiguration: nodeLabelSelector: {}
apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: configuration: ksmConfiguration: nodeLabelSelector: matchLabels: <first_example_key>: "true" <second_example_key>: "true"
apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: configuration: