OpenShift Virtualization (OCP-V) - Live Migration - Requirements and Configuration

### OpenShift Virtualization

Red Hat® OpenShift® Virtualization は Red Hat OpenShift に含まれる機能であり、組織が新規および既存の仮想マシン (VM) ワークロードを実行およびデプロイするための先進的なプラットフォームを提供します。このソリューションにより、従来の仮想マシンを、信頼できる、一貫した包括的なハイブリッドクラウド・アプリケーション・プラットフォームに簡単に移行できます。

OpenShift Virtualization は、VM の移行を単純化するとともに、クラウドネイティブ・アプリケーション・プラットフォームのシンプルさと速度を利用してインフラストラクチャのモダナイゼーションの道筋を提供します。また、先進的な管理の原則を取り入れつつ既存の仮想化への投資を維持することを目指しており、Red Hat の包括的な仮想化ソリューションの基盤となります。

https://www.redhat.com/ja/technologies/cloud-computing/openshift/virtualization

### Live Migration

OCP 4.18 Bare Metal 環境 OpenShift Virtualization (OCP-V) で Live Migration を行ってみます。

https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/virtualization/live-migration

#### Live migration requirements
VM の Live Migration 前提条件として、以下が挙げられます。

> - The cluster must have shared storage with ReadWriteMany (RWX) access mode.
> - The cluster must have sufficient RAM and network bandwidth. 
> - If a VM uses a host model CPU, the nodes must support the CPU.
> - Configuring a dedicated Multus network for live migration is highly recommended. A dedicated network minimizes the effects of network saturation on tenant workloads during migration. 

##### The cluster must have shared storage with ReadWriteMany (RWX) access mode.
Live Migration 実行時に、Migration 元 VM Pod と Migration 先 VM Pod 間で PVC を共有する必要があります。なお、VM が使用する PVC の Access Mode を ReadWriteOnce (RWO) に設定すると、以下のように Live Migration 不可能な状態 `(Not migratable / LiveMigratable = False)` で VM が稼働します。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/c8e19aeb-47db-4044-8a81-7713d36863d2.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/097b4f62-23d9-4950-b6e9-1f5566730d09.png)

##### The cluster must have sufficient RAM and network bandwidth
Live Migration 実行時に、十分な量の RAM と Network Bandwidth が必要です。

##### If a VM uses a host model CPU, the nodes must support the CPU

CPU Model に `Host Model` を使用している場合、VM Pod は Node の 物理CPU に最適な 仮想CPU (`QEMU Predefined named CPU`) を自動選択して稼働します。Live Migration 先 Node が同じ CPU をサポートしている必要があります。

https://kubevirt.io/user-guide/compute/virtual_hardware/#enabling-default-cluster-cpu-model

Model
https://kubevirt.io/user-guide/compute/virtual_hardware/#model

CPU model and topology
https://libvirt.org/formatdomain.html#cpu-model-and-topology

なお、`Host Model` は、OCP-V (Libvirt) から QEMU の CPU Model 設定をするためのオプションの1つです。

https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html

> Libvirt supports a third way to configure CPU models known as “Host model”. This uses the QEMU “Named model” feature, automatically picking a CPU model that is similar the host CPU, and then adding extra features to approximate the host model as closely as possible. This does not guarantee the CPU family, stepping, etc will precisely match the host CPU, as they would with “Host passthrough”, but gives much of the benefit of passthrough, while making live migration safe.

VM が使用している CPU Model は、以下のように VirtualMachineInstance から確認することができます。
```shell
$ oc -o json get vmi rhel9-custom01-upload | jq -r '.spec.domain.cpu'
{
  "cores": 1,
  "maxSockets": 4,
  "model": "host-model",
  "sockets": 1,
  "threads": 1
}
```

VM 内部からは、以下の要領で確認することができます。
```shell
$ virtctl console rhel9-custom01-upload
：
～中略～
：
[cloud-user@rhel9-custom01-upload ~]$ lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          45 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   1
  On-line CPU(s) list:    0
Vendor ID:                GenuineIntel
  Model name:             Intel Xeon Processor (SapphireRapids)
    CPU family:           6
    Model:                143
    Thread(s) per core:   1
    Core(s) per socket:   1
    Socket(s):            1
    Stepping:             4
    BogoMIPS:             4199.99
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge m
                          ca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall
                           nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopo
                          logy cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma
                          cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadli
                          ne_timer aes xsave avx f16c rdrand hypervisor lahf_lm
                          abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibr
                          s_enhanced tpr_shadow ept vpid ept_ad fsgsbase tsc_adj
                          ust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq
                          rdseed adx smap avx512ifma clflushopt clwb avx512cd sh
                          a_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves
                          avx_vnni avx512_bf16 wbnoinvd arat vnmi avx512vbmi umi
                          p pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_v
                          nni avx512_bitalg avx512_vpopcntdq rdpid cldemote movd
                          iri movdir64b fsrm md_clear serialize amx_bf16 avx512_
                          fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization features:
  Virtualization:         VT-x
  Hypervisor vendor:      KVM
  Virtualization type:    full
Caches (sum of all):
  L1d:                    32 KiB (1 instance)
  L1i:                    32 KiB (1 instance)
  L2:                     4 MiB (1 instance)
  L3:                     16 MiB (1 instance)
NUMA:
  NUMA node(s):           1
  NUMA node0 CPU(s):      0
Vulnerabilities:
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Unknown: No mitigations
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prct
                          l
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointe
                          r sanitization
  Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditiona
                          l; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop,
                          KVM SW loop
  Srbds:                  Not affected
  Tsx async abort:        Not affected
  
[cloud-user@rhel9-custom01-upload ~]$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 143
model name      : Intel Xeon Processor (SapphireRapids)
stepping        : 4
microcode       : 0x1
cpu MHz         : 2099.998
cache size      : 16384 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 32
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
vmx flags       : vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb tsc_offset vtpr mtf ept vpid unrestricted_guest shadow_vmcs pml
bugs            : spectre_v1 spectre_v2 spec_store_bypass swapgs mmio_unknown eibrs_pbrsb bhi
bogomips        : 4199.99
clflush size    : 64
cache_alignment : 64
address sizes   : 45 bits physical, 48 bits virtual
power management:
```
VM Pod (Virt Launcher) の QEMU (qemu-kvm) からは、以下の要領で確認することができます。qemu-kvm プロセスで `-cpu SapphireRapids` が指定されていること、QEMU XML で `cpu mode='host-model'` が指定されていることが分かります。


```shell
$ oc get pod
NAME                                        READY   STATUS    RESTARTS   AGE
virt-launcher-rhel9-custom01-upload-mq2sr   1/1     Running   0          16m

$ oc exec -ti virt-launcher-rhel9-custom01-upload-mq2sr -- bash
bash-5.1$

bash-5.1$ pgrep -a qemu-kvm | tr ',' '\n' | grep -- "-cpu"
acpi=on -accel kvm -cpu SapphireRapids

bash-5.1$ ls -l /var/run/libvirt/qemu/run
total 32
-rw-------. 1 qemu qemu     0 Jun  4 00:22 autostarted
drwxr-sr-x. 3 qemu qemu    36 Jun  4 00:22 channel
drwxrwx---. 2 qemu qemu     6 Jun  4 00:22 dbus
-rw-r--r--. 1 qemu qemu     2 Jun  4 00:22 driver.pid
drwxr-sr-x. 2 qemu qemu     6 Jun  4 00:22 passt
drwxr-sr-x. 2 qemu qemu     6 Jun  4 00:22 slirp
-rw-r--r--. 1 qemu qemu     2 Jun  4 00:22 z-ocpv-vmtest-custom_rhel9-custom01-upload.pid
-rw-------. 1 qemu qemu 23505 Jun  4 01:04 z-ocpv-vmtest-custom_rhel9-custom01-upload.xml

bash-5.1$ grep -n -A 10 "<cpu mode" /var/run/libvirt/qemu/run/z-ocpv-vmtest-custom_rhel9-custom01-upload.xml
206:  <cpu mode='host-model' check='partial'>
207-    <topology sockets='4' dies='1' clusters='1' cores='1' threads='1'/>
208-  </cpu>
209-  <chardevStdioLogd/>
210-  <nodename index='2'/>
211-  <fdset index='0'/>
212-  <blockjobs active='no'/>
213-  <agentTimeout>-2</agentTimeout>
214-  <domain type='kvm' id='1'>
215-    <name>z-ocpv-vmtest-custom_rhel9-custom01-upload</name>
216-    <uuid>bad11d07-82d7-5175-b29a-c85462c781ed</uuid>
--
254:    <cpu mode='custom' match='exact' check='full'>
255-      <model fallback='forbid'>SapphireRapids</model>
256-      <vendor>Intel</vendor>
257-      <topology sockets='4' dies='1' clusters='1' cores='1' threads='1'/>
258-      <feature policy='require' name='ss'/>
259-      <feature policy='require' name='vmx'/>
260-      <feature policy='require' name='hypervisor'/>
261-      <feature policy='require' name='tsc_adjust'/>
262-      <feature policy='require' name='cldemote'/>
263-      <feature policy='require' name='movdiri'/>
264-      <feature policy='require' name='movdir64b'/>
```

##### Configuring a dedicated Multus network for live migration is highly recommended
以下の要領で、Live Migration 専用 Network を設定することができます　`NetworkAttachmentDefinition (NAD)` で Bridge Network を設定した後、HyperConverged CR で Network を Live Migration 専用として指定します。

Configuring a dedicated network for live migration
https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/virtualization/networking#virt-dedicated-network-live-migration

#### Configuring live migration

HyperConverged CR と OCP Console から Live Migration の設定を行うことができます。

https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/virtualization/live-migration#virt-configuring-live-migration


Configuring live migration limits and timeouts
https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/virtualization/live-migration#virt-configuring-live-migration-limits_virt-configuring-live-migration

```shell
$ oc explain HyperConverged.spec.liveMigrationConfig
GROUP:      hco.kubevirt.io
KIND:       HyperConverged
VERSION:    v1beta1

FIELD: liveMigrationConfig <Object>


DESCRIPTION:
    Live migration limits and timeouts are applied so that migration processes
    do not overwhelm the cluster.

FIELDS:
  allowAutoConverge     <boolean>
    AllowAutoConverge allows the platform to compromise performance/availability
    of VMIs to
    guarantee successful VMI live migrations. Defaults to false

  allowPostCopy <boolean>
    When enabled, KubeVirt attempts to use post-copy live-migration in case it
    reaches its completion timeout while attempting pre-copy live-migration.
    Post-copy migrations allow even the busiest VMs to successfully
    live-migrate.
    However, events like a network failure or a failure in any of the source or
    destination nodes can cause the migrated VM to crash or reach inconsistency.
    Enable this option when evicting nodes is more important than keeping VMs
    alive.
    Defaults to false.

  bandwidthPerMigration <string>
    Bandwidth limit of each migration, the value is quantity of bytes per second
    (e.g. 2048Mi = 2048MiB/sec)

  completionTimeoutPerGiB       <integer>
    If a migrating VM is big and busy, while the connection to the destination
    node
    is slow, migration may never converge. The completion timeout is calculated
    based on completionTimeoutPerGiB times the size of the guest (both RAM and
    migrated disks, if any). For example, with completionTimeoutPerGiB set to
    800,
    a virtual machine instance with 6GiB memory will timeout if it has not
    completed migration in 1h20m. Use a lower completionTimeoutPerGiB to induce
    quicker failure, so that another destination or post-copy is attempted. Use
    a
    higher completionTimeoutPerGiB to let workload with spikes in its memory
    dirty
    rate to converge.
    The format is a number.

  network       <string>
    The migrations will be performed over a dedicated multus network to minimize
    disruption to tenant workloads due to network saturation when VM live
    migrations are triggered.

  parallelMigrationsPerCluster  <integer>
    Number of migrations running in parallel in the cluster.

  parallelOutboundMigrationsPerNode     <integer>
    Maximum number of outbound migrations per node.

  progressTimeout       <integer>
    The migration will be canceled if memory copy fails to make progress in this
    time, in seconds.
```

OCP Console (Virtualization/Overview/Settings) から、一部の設定を行うことができます。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/4f1894ee-cfcb-469c-a955-2eeeaf430ef8.png)


##### Live migration policies

MigrationPolicy CR で Live Migration Policy を設定することができます。これによって HyperConverged CR で指定した Default を、適宜変更することができます。MigrationPolicy を適用する範囲を、`MigrationPolicy.spec.selectors` で指定します。

Live migration policies
https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/virtualization/live-migration#live-migration-policies


![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/cfba4559-8741-4dd9-8bbb-37b1eb72b233.png)


![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/0a6f8de7-96c6-40f3-9e1a-80468feb31e8.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/59f35b00-e9a0-428b-a515-71304f10c03a.png)


```shell
$ oc explain MigrationPolicy
GROUP:      migrations.kubevirt.io
KIND:       MigrationPolicy
VERSION:    v1alpha1

DESCRIPTION:
    MigrationPolicy holds migration policy (i.e. configurations) to apply to a
    VM or group of VMs

FIELDS:
  apiVersion    <string>
    APIVersion defines the versioned schema of this representation of an object.
    Servers should convert recognized schemas to the latest internal value, and
    may reject unrecognized values. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

  kind  <string>
    Kind is a string value representing the REST resource this object
    represents. Servers may infer this from the endpoint the client submits
    requests to. Cannot be updated. In CamelCase. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

  metadata      <ObjectMeta>
    Standard object's metadata. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

  spec  <Object> -required-
    <no description>

  status        <Object>
    <no description>
```
```shell
$ oc explain MigrationPolicy.spec
GROUP:      migrations.kubevirt.io
KIND:       MigrationPolicy
VERSION:    v1alpha1

FIELD: spec <Object>


DESCRIPTION:
    <empty>
FIELDS:
  allowAutoConverge     <boolean>
    <no description>

  allowPostCopy <boolean>
    <no description>

  bandwidthPerMigration <Object>
    <no description>

  completionTimeoutPerGiB       <integer>
    <no description>

  selectors     <Object> -required-
    <no description>
```
```shell
$ oc explain MigrationPolicy.spec.selectors
GROUP:      migrations.kubevirt.io
KIND:       MigrationPolicy
VERSION:    v1alpha1

FIELD: selectors <Object>


DESCRIPTION:
    <empty>
FIELDS:
  namespaceSelector     <map[string]string>
    <no description>

  virtualMachineInstanceSelector        <map[string]string>
    <no description>
```