IBM Cloud: VPCでAZまたぎでGRE+マルチキャスト通信を試してみた

# 1. はじめに
IBM Cloud VPCは、2023年2月現在は、TCP/UDP/ICMPしか通信は許可されていない。つまり、GREやマルチキャスト通信は許可されていない。現時点では、`Allow IP Spoofing`をenabledにすれば、GREは通信可能であり、またこのGREを使ってマルチキャスト通信が可能であることを確認した（これは明示的に記載されていない仕様のため、将来に渡って変更される可能性があることに注意）。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/62127/4791311a-6d8c-48f4-c24b-16b9763d14d5.png)

# 2. GREの構成

以下のサーバーの間に、192.168.10.0/30のGREを構成する。
- Tokyo regionのAZ1: 10.0.0.12
- Tokyo regionのAZ2: 10.5.0.6


```
[root@syasuda-multicast1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:00:08:02:70:58 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.12/24 brd 10.0.0.255 scope global dynamic noprefixroute eth0
       valid_lft 350sec preferred_lft 350sec
    inet6 fe80::8ff:fe02:7058/64 scope link
       valid_lft forever preferred_lft forever

[root@syasuda-multicast2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:00:03:0d:bf:b2 brd ff:ff:ff:ff:ff:ff
    inet 10.5.0.6/24 brd 10.5.0.255 scope global dynamic noprefixroute eth0
       valid_lft 322sec preferred_lft 322sec
    inet6 fe80::3ff:fe0d:bfb2/64 scope link
       valid_lft forever preferred_lft forever
```


1. `Allow IP Spoofing`をEnabledにする。
2. [GREの設定方法](https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/configuring-a-gre-tunnel-using-nmcli-to-encapsulate-layer-3-traffic-in-ipv4-packets_configuring-ip-tunnels)を参照する。`gre0`というインターフェース名は予約されているので利用してはいけない。
3. （デフォルトゲートウェイに紐づいている）`eth0`ではなく、今回作成した`gre1`インターフェースを利用してマルチキャスト通信が行われるように、このインターフェースに`224.0.0.0/4`の経路を追加している。

```txt:1台目
[root@syasuda-multicast1 ~]# nmcli connection add type ip-tunnel ip-tunnel.mode gre con-name gre1 ifname gre1 remote 10.5.0.6 local 10.0.0.12
[root@syasuda-multicast1 ~]# nmcli connection modify gre1 ipv4.method manual
[root@syasuda-multicast1 ~]# nmcli connection modify gre1 ipv4.addresses '192.168.10.1/30'
[root@syasuda-multicast1 ~]# nmcli connection modify gre1 +ipv4.routes '224.0.0.0/4'
[root@syasuda-multicast1 ~]# nmcli connection up gre1
[root@syasuda-multicast1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:00:08:02:70:58 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.12/24 brd 10.0.0.255 scope global dynamic noprefixroute eth0
       valid_lft 265sec preferred_lft 265sec
    inet6 fe80::8ff:fe02:7058/64 scope link
       valid_lft forever preferred_lft forever
3: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
4: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
5: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
19: gre1@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1476 qdisc noqueue state UNKNOWN group default qlen 1000
    link/gre 10.0.0.12 peer 10.5.0.6
    inet 192.168.10.1/30 brd 192.168.10.3 scope global noprefixroute gre1
       valid_lft forever preferred_lft forever
    inet6 fe80::36f6:977d:7c17:cce3/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
[root@syasuda-multicast1 ~]# ip r
default via 10.0.0.1 dev eth0
default via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.12 metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.12 metric 100
192.168.10.0/30 dev gre1 proto kernel scope link src 192.168.10.1 metric 675
224.0.0.0/4 dev gre1 proto static scope link metric 675
```

```txt:2台目
[root@syasuda-multicast2 ~]# nmcli connection add type ip-tunnel ip-tunnel.mode gre con-name gre1 ifname gre1 remote 10.0.0.12 local 10.5.0.6
[root@syasuda-multicast2 ~]# nmcli connection modify gre1 ipv4.method manual
[root@syasuda-multicast2 ~]# nmcli connection modify gre1 ipv4.addresses '192.168.10.2/30'
[root@syasuda-multicast2 ~]# nmcli connection modify gre1 +ipv4.routes '224.0.0.0/4'
[root@syasuda-multicast2 ~]# nmcli connection up gre1
[root@syasuda-multicast2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:00:03:0d:bf:b2 brd ff:ff:ff:ff:ff:ff
    inet 10.5.0.6/24 brd 10.5.0.255 scope global dynamic noprefixroute eth0
       valid_lft 279sec preferred_lft 279sec
    inet6 fe80::3ff:fe0d:bfb2/64 scope link
       valid_lft forever preferred_lft forever
3: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
4: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
5: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
19: gre1@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1476 qdisc noqueue state UNKNOWN group default qlen 1000
    link/gre 10.5.0.6 peer 10.0.0.12
    inet 192.168.10.2/30 brd 192.168.10.3 scope global noprefixroute gre1
       valid_lft forever preferred_lft forever
    inet6 fe80::f06a:4490:be38:d218/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
[root@syasuda-multicast2 ~]# ip r
default via 10.5.0.1 dev eth0
default via 10.5.0.1 dev eth0 proto dhcp src 10.5.0.6 metric 100
10.5.0.0/24 dev eth0 proto kernel scope link src 10.5.0.6 metric 100
192.168.10.0/30 dev gre1 proto kernel scope link src 192.168.10.2 metric 675
224.0.0.0/4 dev gre1 proto static scope link metric 675
```

# 3. GREのテスト

```
[root@syasuda-multicast1 ~]# ping 192.168.10.2
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.
64 bytes from 192.168.10.2: icmp_seq=1 ttl=64 time=1.76 ms
64 bytes from 192.168.10.2: icmp_seq=2 ttl=64 time=2.02 ms
64 bytes from 192.168.10.2: icmp_seq=3 ttl=64 time=9.81 ms
64 bytes from 192.168.10.2: icmp_seq=4 ttl=64 time=1.88 ms
64 bytes from 192.168.10.2: icmp_seq=5 ttl=64 time=1.89 ms
```

# 4. GREを利用したマルチキャスト通信のテスト

[これ](https://github.com/nin2yasu/multicast-test)を利用した。python3で稼働するように、オリジナルから少しコードを編集している。（オリジナル版は、`TypeError: a bytes-like object is required, not 'str'`というエラーが発生するため）。Pythonが導入されていれば以下でセットアップは完了である。

```
[root@syasuda-multicast1 ~]# python --version
Python 3.6.8
[root@syasuda-multicast1 ~]# git clone https://github.com/nin2yasu/multicast-test
[root@syasuda-multicast1 ~]# cd multicast-test/
```


```txt:1台目でmulticastを実行
[root@syasuda-multicast1 multicast-test]# python mctest.py -send 'test in VPC'
Sending to 232.8.8.8 (TTL 10): test in VPC: 2023-02-14 01:13:25
Sending to 232.8.8.8 (TTL 10): test in VPC: 2023-02-14 01:13:26
Sending to 232.8.8.8 (TTL 10): test in VPC: 2023-02-14 01:13:27
Sending to 232.8.8.8 (TTL 10): test in VPC: 2023-02-14 01:13:28
Sending to 232.8.8.8 (TTL 10): test in VPC: 2023-02-14 01:13:29
```
```txt:1台目でtcpdumpを確認
[root@syasuda-multicast1 ~]# tcpdump -i any ip multicast -nn
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
01:13:25.229956 IP 192.168.10.1.41425 > 232.8.8.8.1900: UDP, length 32
01:13:26.231314 IP 192.168.10.1.41425 > 232.8.8.8.1900: UDP, length 32
01:13:27.232626 IP 192.168.10.1.41425 > 232.8.8.8.1900: UDP, length 32
01:13:28.233964 IP 192.168.10.1.41425 > 232.8.8.8.1900: UDP, length 32
01:13:29.235405 IP 192.168.10.1.41425 > 232.8.8.8.1900: UDP, length 32
```
```txt:2台目でmulticastを受信
[root@syasuda-multicast2 multicast-test]# python mctest.py -rec
Listing on 232.8.8.8 port 1900
Received on 232.8.8.8 from 192.168.10.1 from port 41425: b'test in VPC: 2023-02-14 01:13:25'
Received on 232.8.8.8 from 192.168.10.1 from port 41425: b'test in VPC: 2023-02-14 01:13:26'
Received on 232.8.8.8 from 192.168.10.1 from port 41425: b'test in VPC: 2023-02-14 01:13:27'
Received on 232.8.8.8 from 192.168.10.1 from port 41425: b'test in VPC: 2023-02-14 01:13:28'
Received on 232.8.8.8 from 192.168.10.1 from port 41425: b'test in VPC: 2023-02-14 01:13:29'
```
```txt:2台目でtcpdumpを確認
[root@syasuda-multicast2 multicast-test]# tcpdump -i any ip multicast -nn
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
01:13:24.639570 IP 192.168.10.1.41425 > 232.8.8.8.1900: UDP, length 32
01:13:25.640978 IP 192.168.10.1.41425 > 232.8.8.8.1900: UDP, length 32
01:13:26.642276 IP 192.168.10.1.41425 > 232.8.8.8.1900: UDP, length 32
01:13:27.643590 IP 192.168.10.1.41425 > 232.8.8.8.1900: UDP, length 32
01:13:28.644982 IP 192.168.10.1.41425 > 232.8.8.8.1900: UDP, length 32
```


# 5. その他のマルチキャスト確認方法

```txt:1台目でmulticast宛にping
[root@syasuda-multicast1 ~]# ping 239.192.0.29
PING 239.192.0.29 (239.192.0.29) 56(84) bytes of data.
```

```txt:2台目でtcpdump
[root@syasuda-multicast2 ~]# tcpdump -i any ip multicast
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
03:04:12.898194 IP 192.168.10.1 > 239.192.0.29: ICMP echo request, id 12, seq 1, length 64
03:04:13.942707 IP 192.168.10.1 > 239.192.0.29: ICMP echo request, id 12, seq 2, length 64
03:04:14.966638 IP 192.168.10.1 > 239.192.0.29: ICMP echo request, id 12, seq 3, length 64
03:04:15.990598 IP 192.168.10.1 > 239.192.0.29: ICMP echo request, id 12, seq 4, length 64
03:04:17.014608 IP 192.168.10.1 > 239.192.0.29: ICMP echo request, id 12, seq 5, length 64

```