IBM Cloud: Classic/PowerVS環境で同一VLAN内でのマルチキャスト(multicast)通信をテストしてみた


# 1. はじめに

IBM Cloudでは一般的にはルーターを超えて（＝VLANを跨って）マルチキャスト通信は利用できないが、Classic Infrastructure環境では同一VLAN内であればマルチキャスト（multicast)通信が利用可能であることは昔から知られていた。本稿では改めてその確認を行なった。
また、Power Systems Virtual Server(PowerVS)でも一般的にはルーター超えてマルチキャスト通信はできないが、同一VLAN内ではどのような挙動になるかを確認してみた。

*P.S. 結果としてはPower Systems Virtual Serverでも同一VLAN内であればmulticast通信は機能しましたが、今回のテスト結果が未来永劫保証されるものではないので注意してください　。*

# 2. multicastのテストツール

[これ](https://github.com/nin2yasu/multicast-test)を利用した。python3で稼働するように、オリジナルから少しコードを編集している。（オリジナル版は、`TypeError: a bytes-like object is required, not 'str'`というエラーが発生するため）。Pythonが導入されていれば以下でセットアップは完了である。

```
# python3 --version
Python 3.6.8
# git clone https://github.com/nin2yasu/multicast-test
# cd multicast-test/
```



- 2023/02/16 追記
ちなみに、一方の環境で`ping 239.192.0.29`を実行し、他方の環境で`tcpdump -i any ip multicast -nn`とかでも確認できたので、わざわざこんなアプリを動かす必要もなかったかもしれない。




# 3. Classic

## 3-1. Classic(Public VLAN)

- 同一VLAN/同一Subnet内のVSI同士のマルチキャスト通信
- デフォルトではマルチキャストアドレス（224.0.0.0/4）はpublic側を通ってしまうが、それでも通信できている。

```txt:1台目でmulticastを実行
[root@syasuda-muticast1 multicast-test]# python3 mctest.py -send 'test in Classic'
Sending to 232.8.8.8 (TTL 10): test in Classic: 2023-02-13 19:23:12
Sending to 232.8.8.8 (TTL 10): test in Classic: 2023-02-13 19:23:13
Sending to 232.8.8.8 (TTL 10): test in Classic: 2023-02-13 19:23:14
Sending to 232.8.8.8 (TTL 10): test in Classic: 2023-02-13 19:23:15
Sending to 232.8.8.8 (TTL 10): test in Classic: 2023-02-13 19:23:16
```
```txt:2台目でmulticastを受信(public IPから受信）
[root@syasuda-muticast2 multicast-test]# python3 mctest.py -rec
Listing on 232.8.8.8 port 1900
Received on 232.8.8.8 from 165.xxx.xx.xx from port 36425: b'test in Classic: 2023-02-13 19:23:12'
Received on 232.8.8.8 from 165.xxx.xx.xx from port 36425: b'test in Classic: 2023-02-13 19:23:13'
Received on 232.8.8.8 from 165.xxx.xx.xx from port 36425: b'test in Classic: 2023-02-13 19:23:14'
Received on 232.8.8.8 from 165.xxx.xx.xx from port 36425: b'test in Classic: 2023-02-13 19:23:15'
Received on 232.8.8.8 from 165.xxx.xx.xx from port 36425: b'test in Classic: 2023-02-13 19:23:16'
```
```txt:2台目でtcpdumpを確認(public IPから受信）
[root@syasuda-muticast2 ~]# tcpdump -i any -nn ip multicast
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
19:23:12.153875 IP 165.xxx.xx.xx.36425 > 232.8.8.8.1900: UDP, length 36
19:23:13.154804 IP 165.xxx.xx.xx.36425 > 232.8.8.8.1900: UDP, length 36
19:23:14.156082 IP 165.xxx.xx.xx.36425 > 232.8.8.8.1900: UDP, length 36
19:23:15.157509 IP 165.xxx.xx.xx.36425 > 232.8.8.8.1900: UDP, length 36
19:23:16.158804 IP 165.xxx.xx.xx.36425 > 232.8.8.8.1900: UDP, length 36
```

## 3-2. Classic(Private VLAN)

Classicでprivate側を使わせるために、以下を両ノードで設定。

```
# ip route add 224.0.0.0/4 dev eth0
```

```txt:1台目でmulticastを実行
[root@syasuda-muticast1 multicast-test]# python3 mctest.py -send 'test in Classic'
Sending to 232.8.8.8 (TTL 10): test in Classic: 2023-02-13 19:58:06
Sending to 232.8.8.8 (TTL 10): test in Classic: 2023-02-13 19:58:07
Sending to 232.8.8.8 (TTL 10): test in Classic: 2023-02-13 19:58:08
Sending to 232.8.8.8 (TTL 10): test in Classic: 2023-02-13 19:58:09
Sending to 232.8.8.8 (TTL 10): test in Classic: 2023-02-13 19:58:10
```
```txt:2台目でmulticastを受信(private IPから受信）
[root@syasuda-muticast2 multicast-test]# python3 mctest.py -rec
Listing on 232.8.8.8 port 1900
Received on 232.8.8.8 from 10.193.37.176 from port 58444: b'test in Classic: 2023-02-13 19:58:06'
Received on 232.8.8.8 from 10.193.37.176 from port 58444: b'test in Classic: 2023-02-13 19:58:07'
Received on 232.8.8.8 from 10.193.37.176 from port 58444: b'test in Classic: 2023-02-13 19:58:08'
Received on 232.8.8.8 from 10.193.37.176 from port 58444: b'test in Classic: 2023-02-13 19:58:09'
Received on 232.8.8.8 from 10.193.37.176 from port 58444: b'test in Classic: 2023-02-13 19:58:10'
```
```txt:2台目でtcpdumpを確認(private IPから受信）
[root@syasuda-muticast2 ~]# tcpdump -i any -nn ip multicast
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
19:58:06.532406 IP 10.193.37.176.58444 > 232.8.8.8.1900: UDP, length 36
19:58:07.533355 IP 10.193.37.176.58444 > 232.8.8.8.1900: UDP, length 36
19:58:08.534710 IP 10.193.37.176.58444 > 232.8.8.8.1900: UDP, length 36
19:58:09.536060 IP 10.193.37.176.58444 > 232.8.8.8.1900: UDP, length 36
19:58:10.537338 IP 10.193.37.176.58444 > 232.8.8.8.1900: UDP, length 36
```

# 4. PowerVS

PowerVSには、インターネット側に接続するために利用するインターフェースが配置されているpublic network(env2)と、Direct Link接続などで利用するPrivate network(env3)が存在し、それぞれが異なるVLANに属している。何も設定していないと、マルチキャスト通信はdefault gatewayが存在するPublic VLANを利用して通信する。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/62127/da181296-2673-aee7-ac1e-aa1cdc062eb4.png)


```txt:Serial情報

[root@power-rhel8 ~]# lshw|head -8
power-rhel8
    description: pSeries LPAR
    product: IBM,9009-22A
    vendor: IBM
    serial: IBM,0678A7850
    width: 64 bits
    capabilities: smp
    configuration: uuid=145bd61c-d217-4948-bc0d-3d9be9b6d5e9


[root@power-rhel8-2 ~]# lshw|head -8
power-rhel8-2
    description: pSeries LPAR
    product: IBM,9009-22A
    vendor: IBM
    serial: IBM,0678A79D0
    width: 64 bits
    capabilities: smp
    configuration: uuid=6f50d8bb-a22c-44ed-9816-62a2e6fbe6a9
```

なお、上記のようにシリアル番号が異なることから、異なるHW筐体上にそれぞれのサーバーが存在していることがわかる。

## 4-1. PowerVS(Public VLAN)

```
[root@power-rhel8 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.6 (Ootpa)

[root@power-rhel8 ~]# uname -a
Linux power-rhel8 4.18.0-372.32.1.el8_6.ppc64le #1 SMP Fri Oct 7 11:37:39 EDT 2022 ppc64le ppc64le ppc64le GNU/Linux
```

```txt:1台目でmulticastを実行
[root@power-rhel8 multicast-test]# python3 ./mctest.py -send 'test in PowerVS'
Sending to 232.8.8.8 (TTL 10): test in PowerVS: 2023-02-14 12:36:08
Sending to 232.8.8.8 (TTL 10): test in PowerVS: 2023-02-14 12:36:09
Sending to 232.8.8.8 (TTL 10): test in PowerVS: 2023-02-14 12:36:10
Sending to 232.8.8.8 (TTL 10): test in PowerVS: 2023-02-14 12:36:11
Sending to 232.8.8.8 (TTL 10): test in PowerVS: 2023-02-14 12:36:12
Sending to 232.8.8.8 (TTL 10): test in PowerVS: 2023-02-14 12:36:13
```
```txt:1台目でtcpdumpを確認
[root@power-rhel8 ~]# tcpdump -i any -nn ip multicast
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
12:36:08.374732 IP 192.168.186.58.52652 > 232.8.8.8.1900: UDP, length 36
12:36:09.375816 IP 192.168.186.58.52652 > 232.8.8.8.1900: UDP, length 36
12:36:10.376888 IP 192.168.186.58.52652 > 232.8.8.8.1900: UDP, length 36
12:36:11.377949 IP 192.168.186.58.52652 > 232.8.8.8.1900: UDP, length 36
12:36:12.379058 IP 192.168.186.58.52652 > 232.8.8.8.1900: UDP, length 36
12:36:13.380127 IP 192.168.186.58.52652 > 232.8.8.8.1900: UDP, length 36
```
```txt:2台目でmulticastを受信
[root@power-rhel8-2 multicast-test]# python mctest.py -rec
Listing on 232.8.8.8 port 1900
Received on 232.8.8.8 from 192.168.186.58 from port 52652: b'test in PowerVS: 2023-02-14 12:36:08'
Received on 232.8.8.8 from 192.168.186.58 from port 52652: b'test in PowerVS: 2023-02-14 12:36:09'
Received on 232.8.8.8 from 192.168.186.58 from port 52652: b'test in PowerVS: 2023-02-14 12:36:10'
Received on 232.8.8.8 from 192.168.186.58 from port 52652: b'test in PowerVS: 2023-02-14 12:36:11'
Received on 232.8.8.8 from 192.168.186.58 from port 52652: b'test in PowerVS: 2023-02-14 12:36:12'
Received on 232.8.8.8 from 192.168.186.58 from port 52652: b'test in PowerVS: 2023-02-14 12:36:13'
```
```txt:2台目でtcpdumpを確認（なぜか -i anyだとちゃんとtcpdumpに出てこなかった）
[root@power-rhel8-2 multicast-test]# tcpdump -i env2 ip multicast -nn
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
12:36:08.375822 IP 192.168.186.58.52652 > 232.8.8.8.1900: UDP, length 36
12:36:09.376911 IP 192.168.186.58.52652 > 232.8.8.8.1900: UDP, length 36
12:36:10.377975 IP 192.168.186.58.52652 > 232.8.8.8.1900: UDP, length 36
12:36:11.379046 IP 192.168.186.58.52652 > 232.8.8.8.1900: UDP, length 36
12:36:12.380171 IP 192.168.186.58.52652 > 232.8.8.8.1900: UDP, length 36
12:36:13.381223 IP 192.168.186.58.52652 > 232.8.8.8.1900: UDP, length 36
```

## 4-2. PowerVS(Private VLAN)

private VLANを利用するように、両ノードで以下を実行。

```
# ip route add 224.0.0.0/4 dev env3
```

```txt:1台目でmulticastを実行
[root@power-rhel8 multicast-test]# python3 ./mctest.py -send 'test in PowerVS-private'
Sending to 232.8.8.8 (TTL 10): test in PowerVS-private: 2023-02-14 12:39:02
Sending to 232.8.8.8 (TTL 10): test in PowerVS-private: 2023-02-14 12:39:03
Sending to 232.8.8.8 (TTL 10): test in PowerVS-private: 2023-02-14 12:39:04
Sending to 232.8.8.8 (TTL 10): test in PowerVS-private: 2023-02-14 12:39:05
Sending to 232.8.8.8 (TTL 10): test in PowerVS-private: 2023-02-14 12:39:06
```
```txt:1台目でtcpdumpを確認
[root@power-rhel8 ~]# tcpdump -i any -nn ip multicast
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
12:39:02.893416 IP 192.168.50.224.39516 > 232.8.8.8.1900: UDP, length 44
12:39:03.894550 IP 192.168.50.224.39516 > 232.8.8.8.1900: UDP, length 44
12:39:04.895639 IP 192.168.50.224.39516 > 232.8.8.8.1900: UDP, length 44
12:39:05.896713 IP 192.168.50.224.39516 > 232.8.8.8.1900: UDP, length 44
12:39:06.897775 IP 192.168.50.224.39516 > 232.8.8.8.1900: UDP, length 44
```
```txt:2台目でmulticastを受信
[root@power-rhel8-2 multicast-test]# python mctest.py -rec
Listing on 232.8.8.8 port 1900
Received on 232.8.8.8 from 192.168.50.224 from port 39516: b'test in PowerVS-private: 2023-02-14 12:39:02'
Received on 232.8.8.8 from 192.168.50.224 from port 39516: b'test in PowerVS-private: 2023-02-14 12:39:03'
Received on 232.8.8.8 from 192.168.50.224 from port 39516: b'test in PowerVS-private: 2023-02-14 12:39:04'
Received on 232.8.8.8 from 192.168.50.224 from port 39516: b'test in PowerVS-private: 2023-02-14 12:39:05'
Received on 232.8.8.8 from 192.168.50.224 from port 39516: b'test in PowerVS-private: 2023-02-14 12:39:06'
```
```txt:2台目でtcpdumpを確認（なぜか -i anyだとちゃんとtcpdumpに出てこなかった）
[root@power-rhel8-2 multicast-test]# tcpdump -i env3 ip multicast -nn
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
12:39:02.894697 IP 192.168.50.224.39516 > 232.8.8.8.1900: UDP, length 44
12:39:03.895841 IP 192.168.50.224.39516 > 232.8.8.8.1900: UDP, length 44
12:39:04.896912 IP 192.168.50.224.39516 > 232.8.8.8.1900: UDP, length 44
12:39:05.897962 IP 192.168.50.224.39516 > 232.8.8.8.1900: UDP, length 44
12:39:06.899033 IP 192.168.50.224.39516 > 232.8.8.8.1900: UDP, length 44
```

## 4-3. PowerVS(Private VLAN) - pingを使ったmulticastの検証

その他の確認方法として、こういうのも考えられる。

### 4-3-1. RHEL8-RHEL8の通信
```txt:1台目(RHEL8)でping
[root@power-rhel8 ~]# ping 239.192.0.29
PING 239.192.0.29 (239.192.0.29) 56(84) bytes of data.
```
```txt:2台目(RHEL8)でtcpdumpを確認
[root@power-rhel8-2 ~]# tcpdump -i env3 ip multicast -nn
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on env3, link-type EN10MB (Ethernet), capture size 262144 bytes
17:08:08.882327 IP 192.168.50.224 > 239.192.0.29: ICMP echo request, id 3, seq 1, length 64
17:08:09.906234 IP 192.168.50.224 > 239.192.0.29: ICMP echo request, id 3, seq 2, length 64
17:08:10.946240 IP 192.168.50.224 > 239.192.0.29: ICMP echo request, id 3, seq 3, length 64
17:08:11.986224 IP 192.168.50.224 > 239.192.0.29: ICMP echo request, id 3, seq 4, length 64
17:08:13.026225 IP 192.168.50.224 > 239.192.0.29: ICMP echo request, id 3, seq 5, length 64
```

### 4-3-2. RHEL8-AIXの通信

```
# prtconf 2>&1 |head -18
System Model: IBM,9009-22A
Machine Serial Number: 78A7B20
Processor Type: PowerPC_POWER9
Processor Implementation Mode: POWER 9
Processor Version: PV_9_Compat
Number Of Processors: 1
Processor Clock Speed: 2500 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 4 power2-d9c9cdb3-00000665
Memory Size: 2048 MB
Good Memory Size: 2048 MB
Platform Firmware level: VL950_099
Firmware Version: IBM,FW950.11 (VL950_075)
Console Login: enable
Auto Restart: true
Full Core: false
NX Crypto Acceleration: Capable and Enabled
```

上記のようにこのAIXが利用している物理環境のシリアル番号がも先ほどのRHEL環境とは異なるため、異なる物理サーバー上に存在することがわかる。


```txt:1台目(RHEL8)でping
[root@power-rhel8 ~]# ping 239.192.0.29
PING 239.192.0.29 (239.192.0.29) 56(84) bytes of data.
```
```txt:2台目(AIX)でtcpdumpを確認

# oslevel -s
7200-04-01-1939

# tcpdump -i en1 ip multicast -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en1, link-type EN10MB (Ethernet), capture size 262144 bytes
09:38:46.674332 IP 192.168.50.224 > 239.192.0.29: ICMP echo request, id 5, seq 1, length 64
09:38:47.720673 IP 192.168.50.224 > 239.192.0.29: ICMP echo request, id 5, seq 2, length 64
09:38:48.760705 IP 192.168.50.224 > 239.192.0.29: ICMP echo request, id 5, seq 3, length 64
09:38:49.800747 IP 192.168.50.224 > 239.192.0.29: ICMP echo request, id 5, seq 4, length 64
09:38:50.840810 IP 192.168.50.224 > 239.192.0.29: ICMP echo request, id 5, seq 5, length 64
09:38:51.880826 IP 192.168.50.224 > 239.192.0.29: ICMP echo request, id 5, seq 6, length 64
```