RPi5のAlmaLinux9でMicroShiftを起動したときにflannelがうまく動かない

## 問題
- PaspberryPi5上のAlmaLinux9.4でMicroShift（4.8）を立ち上げるとflannelのPodが作られない
    ```bash
    $ oc get pods -n kube-system | grep flannel
    kube-flannel-ds-24dll   0/1     CreateContainerError   0          5m43s
    ```

## 解決策
- cgroupv2のメモリコントローラを有効化する
    ```bash
    $ sudo sed -i -e "1 s/$/ cgroup_enable=memory/" /boot/cmdline.txt
    $ sudo reboot
    ```

## 解決までの経緯
- flannelの状態を確認
    ```bash
    $ oc get pods -n kube-system | grep flannel
    kube-flannel-ds-24dll   0/1     CreateContainerError   0          5m43s
    ```
- `kube-flannel-ds-24dll`の詳細を確認
    ```bash
    $ oc describe pods -n kube-system  kube-flannel-ds-24dll
    Name:                 kube-flannel-ds-24dll
    Namespace:            kube-system
    ...（中略）...
      Normal   Pulled     5m13s  kubelet            Successfully pulled image "quay.io/microshift/flannel:4.8.0-0.okd-2021-10-10-030117" in 48.989413278s
      Normal   Created    5m13s  kubelet            Created container install-cni
      Normal   Started    5m12s  kubelet            Started container install-cni
      Warning  Failed     5m11s  kubelet            Error: container create failed: time="2024-09-03T07:48:21Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-20e7bcad2a1c0b50e78a7dd5e3c9ba6b49a0b6f944fe70679856282e296cda2b.scope/memory.events: no such file or directory"
    time="2024-09-03T07:48:21Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-20e7bcad2a1c0b50e78a7dd5e3c9ba6b49a0b6f944fe70679856282e296cda2b.scope/memory.max: no such file or directory"
      Warning  Failed  5m10s  kubelet  Error: container create failed: time="2024-09-03T07:48:22Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-07e79f778e1a0368459d234e3a201be6565c77aaa696778120de03ae1e592658.scope/memory.events: no such file or directory"
    time="2024-09-03T07:48:22Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-07e79f778e1a0368459d234e3a201be6565c77aaa696778120de03ae1e592658.scope/memory.max: no such file or directory"
      Warning  Failed  4m55s  kubelet  Error: container create failed: time="2024-09-03T07:48:37Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-2c2894680182139a1967ee1fd01e8e2a764f09dea7aa346f90cad5abc99ac691.scope/memory.events: no such file or directory"
    time="2024-09-03T07:48:37Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-2c2894680182139a1967ee1fd01e8e2a764f09dea7aa346f90cad5abc99ac691.scope/memory.max: no such file or directory"
      Warning  Failed  4m33s  kubelet  Error: container create failed: time="2024-09-03T07:48:59Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-60877050a3d72ca863da01d0adbca82c36960c2424abb2fede98463527773847.scope/memory.events: no such file or directory"
    time="2024-09-03T07:48:59Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-60877050a3d72ca863da01d0adbca82c36960c2424abb2fede98463527773847.scope/memory.max: no such file or directory"
      Warning  Failed  4m19s  kubelet  Error: container create failed: time="2024-09-03T07:49:13Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-036db4110351bb9e944850917b4d08354ba2cf09e85ace3ab6de1cc56bbd0a89.scope/memory.events: no such file or directory"
    time="2024-09-03T07:49:13Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-036db4110351bb9e944850917b4d08354ba2cf09e85ace3ab6de1cc56bbd0a89.scope/memory.max: no such file or directory"
      Warning  Failed  4m3s  kubelet  Error: container create failed: time="2024-09-03T07:49:29Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-6716142d9b83c6891318fa461b26bc46e5535b85cab3f86ffffa8fbb4fedca99.scope/memory.events: no such file or directory"
    time="2024-09-03T07:49:29Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-6716142d9b83c6891318fa461b26bc46e5535b85cab3f86ffffa8fbb4fedca99.scope/memory.max: no such file or directory"
      Warning  Failed  3m52s  kubelet  Error: container create failed: time="2024-09-03T07:49:40Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-77663b47bcb91fac2b2e28c054974f31cd808d004ac43906551472c7ba87fba2.scope/memory.events: no such file or directory"
    time="2024-09-03T07:49:40Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-77663b47bcb91fac2b2e28c054974f31cd808d004ac43906551472c7ba87fba2.scope/memory.max: no such file or directory"
      Warning  Failed  3m40s  kubelet  Error: container create failed: time="2024-09-03T07:49:52Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/crio-8078e233d83a6185a68c9c325b43adb090f1fe9dfb4c3ab886eaf4b4195733d9.scope/memory.events: no such file or directory"
    time="2024-09-03T07:49:52Z" level=error msg="runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/system.slice/crio-8078e233d83a6185a68c9c325b43adb090f1fe9dfb4c3ab886eaf4b4195733d9.scope/memory.max: no such file or directory"
      Normal  Pulled  78s (x19 over 5m12s)  kubelet  Container image "quay.io/microshift/flannel:4.8.0-0.okd-2021-10-10-030117" already present on machine
    ```
- cgroup関連（memory?）の問題っぽいので、cgroupv2が有効になっているか確認
    ```bash
    $ mount | grep cgroup
    cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)
    ```
- 利用可能なcgroupのコントローラを確認
    ```bash
    $ cat /sys/fs/cgroup/cgroup.controllers
    cpuset cpu io pids
    ```
- `memory`の記述がないので、カーネルリングバッファのcgroup関連のメッセージを検索
    ```bash
    $ dmesg | grep -i cgroup
    [    0.000811] cgroup: Disabling memory control group subsystem
    [    3.568089] SELinux:  policy capability cgroup_seclabel=1
    [    3.868919] systemd[1]: Relabelled /dev, /dev/shm, /run, /sys/fs/cgroup in 19.206ms.
    ```
- `Disabling memory control group subsystem`とあるので、メモリコントローラを有効化したい
- `/boot/cmdline.txt`というbootプロセス中にLinuxカーネルに渡されるカーネルパラメーターを含む、Raspberry Piシステム上のコンフィグファイルがあるらしいのでこれを編集
    ```bash
    $ cat /boot/cmdline.txt
    console=serial0,115200 console=tty1 root=PARTUUID=1f4a55c6-02 rootfstype=ext4 rootwait
    
    $ sudo sed -i -e "1 s/$/ cgroup_enable=memory/" /boot/cmdline.txt
    
    $ cat /boot/cmdline.txt
    console=serial0,115200 console=tty1 root=PARTUUID=1f4a55c6-02 rootfstype=ext4 rootwait cgroup_enable=memory
    ```
- 再起動
    ```bash
    $ sudo reboot
    ```
- 再起動後、カーネルリングバッファを確認
    ```bash
    $ dmesg | grep -i "cgroup\|memory"
    [    0.000000] Reserved memory: created CMA memory pool at 0x0000000001c00000, size 320 MiB
    [    0.000000] Early memory node ranges
    [    0.000000] Kernel command line: reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe  smsc95xx.macaddr=2C:CF:67:44:93:06 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  console=ttyAMA10,115200 console=tty1 root=PARTUUID=1f4a55c6-02 rootfstype=ext4 rootwait cgroup_enable=memory
    [    0.000000] cgroup: Enabling memory control group subsystem
    [    0.000000] Memory: 7799420K/8380416K available (13120K kernel code, 2206K rwdata, 4284K rodata, 4928K init, 1102K bss, 253316K reserved, 327680K cma-reserved)
    [    2.654119] Freeing unused kernel memory: 4928K
    [    3.594043] SELinux:  policy capability cgroup_seclabel=1
    [    3.888098] systemd[1]: Relabelled /dev, /dev/shm, /run, /sys/fs/cgroup in 19.263ms.
    ```
- cgroupコントローラの確認
    ```bash
    $ cat /sys/fs/cgroup/cgroup.controllers
    cpuset cpu io memory pids
    ```
- flannelのPodが正常起動していることを確認
    ```bash
    $ oc get pods -n kube-system | grep flannel
    kube-flannel-ds-vm2q4   1/1     Running   0          13m
    ```