LXD上でのpacemakerのクラスタのインストール

LXD(システムコンテナ)上でPacemakerをセットアップしてみました。

# 環境 - まずは単ノードで
KVM上のCentOS 8
CentOS-Stream-8-x86_64-2021102

# LXDのインストール
以下のページを参考にしました。
* https://www.cyberciti.biz/faq/set-up-use-lxd-on-centos-rhel-8-x/

## 手順
以下の操作は途中まではrootユーザで実行しています

### KVMホスト上にCentOSのインストールを実行
インストール後にselinuxをdisabledに設定。firewallも停止。
EPELの有効化とアップデート

```
# yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
(snip)
# yum -y update
(snip)
```

### snapdのインストール前の設定と再起動

```
[root@cent8-01 ~]# yum search snapd
メタデータの期限切れの最終確認: 0:10:26 時間前の 2021年11月29日 16時29分31秒 に実施しました。
=============================================================================== 名前 完全一致: snapd ================================================================================
snapd.x86_64 : A transactional software package manager
============================================================================== 名前 & 概要 一致: snapd ==============================================================================
snapd-devel.noarch : Development files for snapd
snapd-glib.x86_64 : Library providing a GLib interface to snapd
snapd-glib-devel.x86_64 : Development files for snapd-glib
snapd-glib-tests.x86_64 : Installed tests for snapd-glib
snapd-qt.x86_64 : Library providing a Qt5 interface to snapd
snapd-qt-devel.x86_64 : Development files for snapd-qt
snapd-qt-qml.x86_64 : Library providing a Qt5 QML interface to snapd
snapd-qt-tests.x86_64 : Installed tests for snapd-qt
snapd-selinux.noarch : SELinux module for snapd

[root@cent8-01 ~]# grubby --args="namespace.unpriv_enable=1" --update-kernel="$(grubby --default-kernel)"

[root@cent8-01 ~]# sh -c 'echo "user.max_user_namespaces=2147483647" >> /etc/sysctl.d/99-userns.conf'

[root@cent8-01 ~]# shutdown -r now 
```

### 設定の確認とsnapdのインストールと再起動

```
[root@cent8-01 ~]# cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-348.2.1.el8_5.x86_64 root=/dev/mapper/cs-root ro resume=/dev/mapper/cs-swap rd.lvm.lv=cs/root rd.lvm.lv=cs/swap rhgb quiet namespace.unpriv_enable=1

[root@cent8-01 ~]# yum -y install snapd

[root@cent8-01 ~]# ln -s /var/lib/snapd/snap /snap
[root@cent8-01 ~]# 
[root@cent8-01 ~]# systemctl enable --now snapd.socket
Created symlink /etc/systemd/system/sockets.target.wants/snapd.socket → /usr/lib/systemd/system/snapd.socket.
[root@cent8-01 ~]# systemctl status snapd.socket
● snapd.socket - Socket activation for snappy daemon
   Loaded: loaded (/usr/lib/systemd/system/snapd.socket; enabled; vendor preset: disabled)
   Active: active (listening) since Mon 2021-11-29 16:46:49 JST; 6s ago
   Listen: /run/snapd.socket (Stream)
           /run/snapd-snap.socket (Stream)
    Tasks: 0 (limit: 49678)
   Memory: 4.0K
   CGroup: /system.slice/snapd.socket

11月 29 16:46:49 cent8-01 systemd[1]: Starting Socket activation for snappy daemon.
11月 29 16:46:49 cent8-01 systemd[1]: Listening on Socket activation for snappy daemon.
[root@cent8-01 ~]# systemctl status snapd.service
● snapd.service - Snap Daemon
   Loaded: loaded (/usr/lib/systemd/system/snapd.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

[root@cent8-01 ~]# systemctl enable snapd.service
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.service → /usr/lib/systemd/system/snapd.service.

[root@cent8-01 ~]# systemctl start snapd.service

[root@cent8-01 ~]# shutdown -r now
```

### lxdのインストール

```
[root@cent8-01 ~]# snap install lxd
2021-11-29T16:50:24+09:00 INFO Waiting for automatic snapd restart...
lxd 4.20 from Canonical✓ installed

[root@cent8-01 ~]# useradd yamauchi
[root@cent8-01 ~]# usermod -a -G lxd yamauchi
[root@cent8-01 ~]# su - yamauchi
[yamauchi@cent8-01 ~]$ newgrp lxd

[yamauchi@cent8-01 ~]$ id
uid=1001(yamauchi) gid=971(lxd) groups=971(lxd),1001(yamauchi)

[yamauchi@cent8-01 ~]$ lxc list
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:20.04
Or for a virtual machine: lxc launch ubuntu:20.04 --vm

+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+

```

### lxdのconfigurationの実行
途中storage backendだけ、OSインストール時のファイルシステムのlvmで指定しています。

```
[yamauchi@cent8-01 ~]$ lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (ceph, btrfs, dir, lvm) [default=btrfs]: lvm
Create a new LVM pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GB of the new loop device (1GB minimum) [default=5GB]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the LXD server to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes] 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
```

### イメージの確認
多くの公式イメージが出力される。

```
[yamauchi@cent8-01 ~] lxc image list images:
```

### とりあえず、centos8のイメージを取得

```
[yamauchi@cent8-01 ~]$ lxc launch images:centos/8/amd64 cent8-01
Creating cent8-01
Starting cent8-01  

[yamauchi@cent8-01 ~]$ lxc list
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
|   NAME   |  STATE  |         IPV4          |                     IPV6                      |   TYPE    | SNAPSHOTS |
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| cent8-01 | RUNNING | 10.144.107.190 (eth0) | fd42:26e0:cb93:870a:216:3eff:fe63:367d (eth0) | CONTAINER | 0         |
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
```

### 起動しているコンテナへ接続

```
[yamauchi@cent8-01 ~]$ lxc exec cent8-01 bash
```

### High Availabilityを有効にしてコンテナにpcs,pacemaker,fence-agentsをインストール

```
[root@cent8-01 ~]# dnf --enablerepo=ha -y install pacemaker pcs fence-agents-all
CentOS Linux 8 - AppStream                                                                                                                           3.7 MB/s | 8.1 MB     00:02    
CentOS Linux 8 - BaseOS                                                                                                                              2.9 MB/s | 3.5 MB     00:01    
CentOS Linux 8 - Extras                                                                                                                               17 kB/s |  10 kB     00:00    
CentOS Linux 8 - HighAvailability                                                                                                                    528 kB/s | 521 kB     00:00    
Dependencies resolved.
=====================================================================================================================================================================================
 Package                                             Architecture                Version                                                        Repository                      Size
=====================================================================================================================================================================================
Installing:
 fence-agents-all                                    x86_64                      4.2.1-75.el8                                                   appstream                       22 k
 pacemaker                                           x86_64                      2.1.0-8.el8                                                    ha                             456 k
 pcs                                                 x86_64                      0.10.8-1.el8                                                   ha                              13 M
(snip)
```

### 一旦、コンテナから抜けて、コンテナを停止

```
[root@cent8-01 ~]# exit
exit

[yamauchi@cent8-01 ~]$ lxc stop  cent8-01
[yamauchi@cent8-01 ~]$ lxc list
+----------+---------+------+------+-----------+-----------+
|   NAME   |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+----------+---------+------+------+-----------+-----------+
| cent8-01 | STOPPED |      |      | CONTAINER | 0         |
+----------+---------+------+------+-----------+-----------+
```

### cont8-01をpublishしてから、cent8-02のイメージを作成

```
[yamauchi@cent8-01 ~]$ lxc publish cent8-01 --alias pacemaker
Instance published with fingerprint: f6716cdfd9643b59c6f96786b8d2306726dae386259cd9cfd3e77dfce78e8900
[yamauchi@cent8-01 ~]$ lxc list
+----------+---------+------+------+-----------+-----------+
|   NAME   |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+----------+---------+------+------+-----------+-----------+
| cent8-01 | STOPPED |      |      | CONTAINER | 0         |
+----------+---------+------+------+-----------+-----------+

[yamauchi@cent8-01 ~]$  lxc image list
+-----------+--------------+--------+----------------------------------+--------------+-----------+----------+------------------------------+
|   ALIAS   | FINGERPRINT  | PUBLIC |           DESCRIPTION            | ARCHITECTURE |   TYPE    |   SIZE   |         UPLOAD DATE          |
+-----------+--------------+--------+----------------------------------+--------------+-----------+----------+------------------------------+
| pacemaker | f6716cdfd964 | no     | Centos 8 x86_64 (20211127_07:08) | x86_64       | CONTAINER | 308.77MB | Nov 29, 2021 at 8:04am (UTC) |
+-----------+--------------+--------+----------------------------------+--------------+-----------+----------+------------------------------+
|           | 6b5db6ae52c3 | no     | Centos 8 amd64 (20211127_07:08)  | x86_64       | CONTAINER | 127.49MB | Nov 29, 2021 at 7:56am (UTC) |
+-----------+--------------+--------+----------------------------------+--------------+-----------+----------+------------------------------+

[yamauchi@cent8-01 ~]$ lxc launch pacemaker cent8-02
Creating cent8-02
Starting cent8-02                          
[yamauchi@cent8-01 ~]$ lxc list
+----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
|   NAME   |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| cent8-01 | STOPPED |                      |                                               | CONTAINER | 0         |
+----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| cent8-02 | RUNNING | 10.144.107.86 (eth0) | fd42:26e0:cb93:870a:216:3eff:fe88:8ae1 (eth0) | CONTAINER | 0         |
+----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
```

### cont8-01も再起動

```
[yamauchi@cent8-01 ~]$ lxc start cent8-01
[yamauchi@cent8-01 ~]$ lxc list
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
|   NAME   |  STATE  |         IPV4          |                     IPV6                      |   TYPE    | SNAPSHOTS |
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| cent8-01 | RUNNING | 10.144.107.190 (eth0) | fd42:26e0:cb93:870a:216:3eff:fe63:367d (eth0) | CONTAINER | 0         |
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| cent8-02 | RUNNING | 10.144.107.86 (eth0)  | fd42:26e0:cb93:870a:216:3eff:fe88:8ae1 (eth0) | CONTAINER | 0         |
+----------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
```

### クラスタの構築
2つのコンソールからそれぞれのコンテナへ接続

```
[yamauchi@cent8-01 ~]$ lxc exec cent8-01 bash
[root@cent8-01 ~]#

[yamauchi@cent8-01 ~]$ lxc exec cent8-02 bash
[root@cent8-02 ~]# 
```

それぞれのコンテナでhaclusterユーザのパスワードを設定し、pcsdサービスを起動

```
[root@cent8-01 ~]# passwd hacluster
Changing password for user hacluster.
New password: 
Retype new password: 
passwd: all authentication tokens updated successfully.

[root@cent8-01 ~]# systemctl enable pcsd --now
Created symlink /etc/systemd/system/multi-user.target.wants/pcsd.service → /usr/lib/systemd/system/pcsd.service.
[root@cent8-01 ~]# systemctl status  pcsd
● pcsd.service - PCS GUI and remote configuration interface
   Loaded: loaded (/usr/lib/systemd/system/pcsd.service; enabled; vendor preset: disabled)
  Drop-In: /run/systemd/system/pcsd.service.d
           └─zzz-lxc-service.conf
   Active: active (running) since Mon 2021-11-29 08:10:11 UTC; 5s ago
     Docs: man:pcsd(8)
           man:pcs(8)
 Main PID: 1367 (pcsd)
    Tasks: 1 (limit: 49678)
   Memory: 38.8M
   CGroup: /system.slice/pcsd.service
           └─1367 /usr/libexec/platform-python -Es /usr/sbin/pcsd

Nov 29 08:10:10 cent8-01 systemd[1]: Starting PCS GUI and remote configuration interface...
Nov 29 08:10:11 cent8-01 systemd[1]: Started PCS GUI and remote configuration interface.
```

片方のコンテナで、pcsコマンドで各コンテナを認証

```
[root@cent8-01 ~]# pcs host auth cent8-01 cent8-02
Username: hacluster
Password: 
cent8-01: Authorized
cent8-02: Authorized
```

一旦、pcsコマンドでクラスタをセットアップ
 インターフェースは、最初は既存のインターフェースを指定

```
[root@cent8-01 ~]# pcs cluster setup cluster_name cent8-01 addr=10.144.107.190 cent8-02 addr=10.144.107.86
Destroying cluster on hosts: 'cent8-01', 'cent8-02'...
cent8-01: Successfully destroyed cluster
cent8-02: Successfully destroyed cluster
Requesting remove 'pcsd settings' from 'cent8-01', 'cent8-02'
cent8-01: successful removal of the file 'pcsd settings'
cent8-02: successful removal of the file 'pcsd settings'
Sending 'corosync authkey', 'pacemaker authkey' to 'cent8-01', 'cent8-02'
cent8-01: successful distribution of the file 'corosync authkey'
cent8-01: successful distribution of the file 'pacemaker authkey'
cent8-02: successful distribution of the file 'corosync authkey'
cent8-02: successful distribution of the file 'pacemaker authkey'
Sending 'corosync.conf' to 'cent8-01', 'cent8-02'
cent8-01: successful distribution of the file 'corosync.conf'
cent8-02: successful distribution of the file 'corosync.conf'
Cluster has been successfully set up.
```

pcsでクラスタを起動しようとするが失敗

```
[root@cent8-01 ~]# pcs cluster start --all
cent8-01: Error connecting to cent8-01 - (HTTP error: 400)
cent8-02: Error connecting to cent8-02 - (HTTP error: 400)
Error: unable to start all nodes
cent8-01: Error connecting to cent8-01 - (HTTP error: 400)
cent8-02: Error connecting to cent8-02 - (HTTP error: 400)
```

どうやら、問題は、knet指定がうまく動作していないようなので、両コンテナのcorosync.confのtotemブロックを以下のように変更
transportをudpに、crypto_cipher/crypto_hashをnoneに、interfaceブロックを追加

```
totem {
    version: 2
    cluster_name: cluster_name
    transport: udp
    crypto_cipher: none
    crypto_hash: none
    interface {
        ringnumber: 0
    }
}
(snip)
```

再度、クラスタを起動

```
[root@cent8-01 ~]# pcs cluster start --all
cent8-01: Starting Cluster...
cent8-02: Starting Cluster...
```

無事に起動

```
[root@cent8-01 ~]# crm_mon -rfA1
Cluster Summary:
  * Stack: corosync
  * Current DC: cent8-02 (version 2.1.0-8.el8-7c3f660707) - partition with quorum
  * Last updated: Mon Nov 29 08:18:54 2021
  * Last change:  Mon Nov 29 08:18:49 2021 by hacluster via crmd on cent8-02
  * 2 nodes configured
  * 0 resource instances configured

Node List:
  * Online: [ cent8-01 cent8-02 ]

Full List of Resources:
  * No resources

Migration Summary:
```
# 関連リンク
https://linuxcontainers.org/ja/lxd/introduction/
https://computingforgeeks.com/run-linux-containers-with-lxc-lxd-on-rocky-almalinux/

以上、