Azure x IBMCloud Active/Active 冗長化接続(IPsecVPN+BGP)

# 目的
Microsoft AzureとIBMCloudをインターネットを介したIPsecVPNを**Active/Active**で接続する。
次に、BGPを用いた経路交換を実施し、AzureとIBMCloud間で仮想マシン間の通信が可能か確認する。
最後に、インタフェース障害や機器障害を想定したテストを実施し、問題なく通信が可能か確認する。

**細かすぎる説明は省略する場合があります。**
基本的には、[Azure x IBMCloud IPsecVPN/BGP接続](https://qiita.com/khoshina/items/a549f491411980e9b344)の記事の発展形となる。
そのため、細かい説明は上記の記事を参照してください。

# 構成
* ローカル端末：macOS Big Sur 11.6 1台
* プラットフォーム:Microsoft Azure
    * 仮想ネットワーク:PublicIPアドレス/仮想NWゲートウェイ/ローカルNWゲートウェイ 2サービス
    * Virtual Machines:CentOS 7.9 1台
* プラットフォーム:IBMCloud
    * ルータ:Vyatta AT&T vRouter 5600 1912g 2台
    * 仮想サーバ:CentOS 7.7(Virtual Server for Classic) 1台

## 構成図
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/ca892627-0a3d-19f6-08fa-313cea34e7ae.png)



# Azureサービスのデプロイ
Active/ActiveでIPsecVPNをデプロイする際の変更点のみ記載する。

## Azure VPN Gateway(仮想ネットワークゲートウェイ)の作成

アクティブ/アクティブの有効化と、２つのアドレスに対してBGP接続するための設定

![貼り付けた画像_2022_04_01_12_43.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/4cd337e5-3ece-89ca-67fa-eb68d1e8bc66.png)

## Azure ローカルネットワークゲートウェイの作成
Azure側の２台に対して、IBMCloud側も2台のルータで接続するため、ローカルネットワークゲートウェイも2台分作成する。

1台目:IBMCloud#1の定義
![貼り付けた画像_2022_04_01_12_53.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/9564fd6d-1479-6722-b32d-ed7286abc3f2.png)

2台目:IBMCloud#2の定義
![貼り付けた画像_2022_04_01_12_54.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/f77254cc-b366-16ae-beea-7f838039dee8.png)

## Azure接続用IPsecVPNの設定
２台分のIPSecVPNの設定が必要

１台目：Azure#1 ~ IBMCloud#1間
![貼り付けた画像_2022_04_01_12_56.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/3f18beb9-7095-bff8-5cca-a4b4d974ebaf.png)

2台目：Azure#2 ~ IBMCloud#2間
![貼り付けた画像_2022_04_01_12_57.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/2dedc2df-3721-f939-9fc8-1a6720d2b4fd.png)

# IBMCloud側の設定(Vyatta/VRA)

## IPsecVPN

1号機側の設定
**20.222.88.aaa**はAzure側のGW#1
**128.168.104.aaa**はIBMCloud側のVRA#1

```
set interfaces vti vti0 address 169.254.21.1/30
set interfaces vti vti0 description Azure
set interfaces vti vti0 mtu 1436

set security vpn ipsec esp-group ESP-Azure compression disable
set security vpn ipsec esp-group ESP-Azure lifetime 3600
set security vpn ipsec esp-group ESP-Azure mode tunnel
set security vpn ipsec esp-group ESP-Azure pfs dh-group20
set security vpn ipsec esp-group ESP-Azure proposal 1 encryption aes256
set security vpn ipsec esp-group ESP-Azure proposal 1 hash sha2_256
set security vpn ipsec ike-group IKE-Azure dead-peer-detection action restart
set security vpn ipsec ike-group IKE-Azure dead-peer-detection interval 10
set security vpn ipsec ike-group IKE-Azure dead-peer-detection timeout 30
set security vpn ipsec ike-group IKE-Azure ike-version 2
set security vpn ipsec ike-group IKE-Azure lifetime 3600
set security vpn ipsec ike-group IKE-Azure proposal 1 dh-group 20
set security vpn ipsec ike-group IKE-Azure proposal 1 encryption aes256
set security vpn ipsec ike-group IKE-Azure proposal 1 hash sha2_256
set security vpn ipsec site-to-site peer 20.222.88.aaa authentication mode pre-shared-secret
set security vpn ipsec site-to-site peer 20.222.88.aaa authentication pre-shared-secret ibmcloudazure
set security vpn ipsec site-to-site peer 20.222.88.aaa connection-type initiate
set security vpn ipsec site-to-site peer 20.222.88.aaa default-esp-group ESP-Azure
set security vpn ipsec site-to-site peer 20.222.88.aaa ike-group IKE-Azure
set security vpn ipsec site-to-site peer 20.222.88.aaa local-address 128.168.104.aaa
set security vpn ipsec site-to-site peer 20.222.88.aaa vti bind vti0
set security vpn ipsec site-to-site peer 20.222.88.aaa vti esp-group ESP-Azure
```

2号機側の設定
**20.222.89.bbb**はAzure側のGW#2
**128.168.104.bbb**はIBMCloud側のVRA#2

```
set interfaces vti vti0 address 169.254.22.1/30
set interfaces vti vti0 description Azure
set interfaces vti vti0 mtu 1436

set security vpn ipsec esp-group ESP-Azure compression disable
set security vpn ipsec esp-group ESP-Azure lifetime 3600
set security vpn ipsec esp-group ESP-Azure mode tunnel
set security vpn ipsec esp-group ESP-Azure pfs dh-group20
set security vpn ipsec esp-group ESP-Azure proposal 1 encryption aes256
set security vpn ipsec esp-group ESP-Azure proposal 1 hash sha2_256
set security vpn ipsec ike-group IKE-Azure dead-peer-detection action restart
set security vpn ipsec ike-group IKE-Azure dead-peer-detection interval 10
set security vpn ipsec ike-group IKE-Azure dead-peer-detection timeout 30
set security vpn ipsec ike-group IKE-Azure ike-version 2
set security vpn ipsec ike-group IKE-Azure lifetime 3600
set security vpn ipsec ike-group IKE-Azure proposal 1 dh-group 20
set security vpn ipsec ike-group IKE-Azure proposal 1 encryption aes256
set security vpn ipsec ike-group IKE-Azure proposal 1 hash sha2_256
set security vpn ipsec site-to-site peer 20.222.89.bbb authentication mode pre-shared-secret
set security vpn ipsec site-to-site peer 20.222.89.bbb authentication pre-shared-secret ibmcloudazure
set security vpn ipsec site-to-site peer 20.222.89.bbb connection-type initiate
set security vpn ipsec site-to-site peer 20.222.89.bbb default-esp-group ESP-Azure
set security vpn ipsec site-to-site peer 20.222.89.bbb ike-group IKE-Azure
set security vpn ipsec site-to-site peer 20.222.89.bbb local-address 128.168.104.bbb
set security vpn ipsec site-to-site peer 20.222.89.bbb vti bind vti0
set security vpn ipsec site-to-site peer 20.222.89.bbb vti esp-group ESP-Azure
```

## BGPの設定(IBMCloud ~ Azure間)

1号機側の設定
```
set protocols bgp 65001 address-family ipv4-unicast network 10.192.145.192/26
set protocols bgp 65001 neighbor 169.254.21.2 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp 65001 neighbor 169.254.21.2 remote-as 65515
set protocols bgp 65001 neighbor 169.254.21.2 timers holdtime 30
set protocols bgp 65001 neighbor 169.254.21.2 timers keepalive 10
set protocols bgp 65001 parameters router-id 169.254.21.1
```

2号機側の設定
```
set protocols bgp 65001 address-family ipv4-unicast network 10.192.145.192/26
set protocols bgp 65001 neighbor 169.254.22.2 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp 65001 neighbor 169.254.22.2 remote-as 65515
set protocols bgp 65001 neighbor 169.254.22.2 timers holdtime 30
set protocols bgp 65001 neighbor 169.254.22.2 timers keepalive 10
set protocols bgp 65001 parameters router-id 169.254.22.1
```

## OSPFの設定(VRA１号機〜VRA２号機間)

1号機側の設定
10.192.94.146はVRA#1のdp0bond0
10.192.94.175はVRA#2のdp0bond0

```
set interfaces tunnel tun0 address 172.31.255.1/30
set interfaces tunnel tun0 encapsulation gre
set interfaces tunnel tun0 local-ip 10.192.94.146
set interfaces tunnel tun0 multicast enable
set interfaces tunnel tun0 remote-ip 10.192.94.175

set protocols ospf area 0.0.0.0 network 172.31.255.0/30
set protocols ospf parameters router-id 172.31.255.1
set protocols ospf passive-interface default
set protocols ospf passive-interface-exclude tun0
set protocols ospf redistribute bgp
```

2号機側の設定
10.192.94.175はVRA#2のdp0bond0
10.192.94.146はVRA#1のdp0bond0

```
set interfaces tunnel tun0 address 172.31.255.2/30
set interfaces tunnel tun0 encapsulation gre
set interfaces tunnel tun0 local-ip 10.192.94.175
set interfaces tunnel tun0 multicast enable
set interfaces tunnel tun0 remote-ip 10.192.94.146

set protocols ospf area 0.0.0.0 network 172.31.255.0/30
set protocols ospf parameters router-id 172.31.255.2
set protocols ospf passive-interface default
set protocols ospf passive-interface-exclude tun0
set protocols ospf redistribute bgp
```

# Azure側状態確認

## IPsecVPNの状態
![貼り付けた画像_2022_04_01_14_19.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/d8fac061-5580-a841-2a96-e172a3761ba0.png)


## BGPの状態
![貼り付けた画像_2022_04_01_14_18.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/3aa3046a-f12d-33d5-475b-c0b86b5b45f0.png)


# IBMCloud側状態確認

## VPNの状態

1号機側

```
vyatta@kou-vra001:~$    show vpn ike sa
Peer ID / IP                            Local ID / IP
------------                            -------------
20.222.88.aaa                           128.168.104.aaa

    State    Encrypt       Hash    D-H Grp  A-Time  L-Time IKEv
    -----  ------------  --------  -------  ------  ------ ----
    up     aes256        sha2_256  20       1740    3600    2

vyatta@kou-vra001:~$ show vpn ipsec sa
Peer ID / IP                            Local ID / IP
------------                            -------------
20.222.88.aaa                           128.168.104.aaa

    Tunnel  Id          State  Bytes Out/In   Encrypt       Hash      DH A-Time  L-Time
    ------  ----------  -----  -------------  ------------  --------  -- ------  ------
    vti     7           up     22.2K/24.0K    aes256        sha2_256  20 19757   3600
```

2号機側:VPN

```
vyatta@kou-vra002:~$ show vpn ike sa
Peer ID / IP                            Local ID / IP
------------                            -------------
20.222.89.bbb                            128.168.104.bbb

    State    Encrypt       Hash    D-H Grp  A-Time  L-Time IKEv
    -----  ------------  --------  -------  ------  ------ ----
    up     aes256        sha2_256  20       660     3600    2

vyatta@kou-vra002:~$ show vpn ipsec sa
Peer ID / IP                            Local ID / IP
------------                            -------------
20.222.89.bbb                            128.168.104.bbb

    Tunnel  Id          State  Bytes Out/In   Encrypt       Hash      DH A-Time  L-Time
    ------  ----------  -----  -------------  ------------  --------  -- ------  ------
    vti     7           up     6.4K/13.2K     aes256        sha2_256  20 18676   3600
```

## BGP/OSPF/ルーティングの状態

1号機側

```
vyatta@kou-vra001:~$ show ip bgp neighbors
BGP neighbor is 169.254.21.2, remote AS 65515, local AS 65001, external link
  BGP version 4, remote router ID 192.168.0.5
  BGP state = Established, up for 00:28:23
  Last read 00:28:23, hold time is 30, keepalive interval is 10 seconds
  Configured hold time is 30, keepalive interval is 10 seconds
  Neighbor capabilities:
    Route refresh: advertised and received (new)
    Address family IPv4 Unicast: advertised and received
    Address family IPv6 Unicast: received
  Received 200 messages, 0 notifications, 0 in queue
  Sent 174 messages, 0 notifications, 0 in queue
  Route refresh request: received 0, sent 0
  Minimum time between advertisement runs is 30 seconds
 For address family: IPv4 Unicast
  BGP table version 8, neighbor version 8
  Index 1, Offset 0, Mask 0x2
    Graceful restart: received
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor (both)
  1 accepted prefixes
  1 announced prefixes

 Connections established 7; dropped 6
Local host: 169.254.21.1, Local port: 57254
Foreign host: 169.254.21.2, Foreign port: 179
Nexthop: 169.254.21.1
Nexthop global: ::
Nexthop local: fe80::200:5efe:80a8:68e9
BGP connection: non shared network
Last Reset: 00:29:57, due to BGP Notification sent
Notification Error Message: (Cease/Other Configuration Change.)

vyatta@kou-vra001:~$ show ip ospf neighbor

Total number of full neighbors: 1
OSPF process 0 routing-instance(default):
Neighbor ID     Pri   State            Dead Time   Address         Interface
172.31.255.2      1   Full/ -          00:00:32    172.31.255.2    tun0
vyatta@kou-vra001:~$ show ip route bgp
IP Route Table for Routing-instance "default"
B    *> 192.168.0.0/16 [20/0] via 169.254.21.2, vti0, 00:28:31
vyatta@kou-vra001:~$ show ip route ospf
IP Route Table for Routing-instance "default"
O E2    192.168.0.0/16 [110/1] via 172.31.255.2, tun0, 00:08:17
```

2号機側

```
vyatta@kou-vra002:~$ show ip bgp neighbors
BGP neighbor is 169.254.22.2, remote AS 65515, local AS 65001, external link
  BGP version 4, remote router ID 192.168.0.4
  BGP state = Established, up for 00:09:37
  Last read 00:09:37, hold time is 30, keepalive interval is 10 seconds
  Configured hold time is 30, keepalive interval is 10 seconds
  Neighbor capabilities:
    Route refresh: advertised and received (new)
    Address family IPv4 Unicast: advertised and received
    Address family IPv6 Unicast: received
  Received 72 messages, 0 notifications, 0 in queue
  Sent 61 messages, 0 notifications, 0 in queue
  Route refresh request: received 0, sent 0
  Minimum time between advertisement runs is 30 seconds
 For address family: IPv4 Unicast
  BGP table version 8, neighbor version 8
  Index 1, Offset 0, Mask 0x2
    Graceful restart: received
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor (both)
  1 accepted prefixes
  1 announced prefixes

 Connections established 5; dropped 4
Local host: 169.254.22.1, Local port: 57766
Foreign host: 169.254.22.2, Foreign port: 179
Nexthop: 169.254.22.1
Nexthop global: ::
Nexthop local: fe80::200:5efe:80a8:68fd
BGP connection: non shared network
Last Reset: 00:11:29, due to BGP Notification sent
Notification Error Message: (Cease/Other Configuration Change.)

vyatta@kou-vra002:~$ show ip ospf neighbor

Total number of full neighbors: 1
OSPF process 0 routing-instance(default):
Neighbor ID     Pri   State            Dead Time   Address         Interface
172.31.255.1      1   Full/ -          00:00:36    172.31.255.1    tun0
vyatta@kou-vra002:~$ show ip route bgp
IP Route Table for Routing-instance "default"
B    *> 192.168.0.0/16 [20/0] via 169.254.22.2, vti0, 00:09:43
vyatta@kou-vra002:~$ show ip route ospf
IP Route Table for Routing-instance "default"
O E2    192.168.0.0/16 [110/1] via 172.31.255.1, tun0, 00:29:58
```

## エンドツーエンド疎通確認
### ICMP

AzureサーバからIBMCloudのサーバに対してICMP
```
[azureuser@SV1 ~]$ ping 10.192.145.230
PING 10.192.145.230 (10.192.145.230) 56(84) bytes of data.
64 bytes from 10.192.145.230: icmp_seq=1 ttl=62 time=4.87 ms
64 bytes from 10.192.145.230: icmp_seq=2 ttl=62 time=5.08 ms
64 bytes from 10.192.145.230: icmp_seq=3 ttl=62 time=4.87 ms
64 bytes from 10.192.145.230: icmp_seq=4 ttl=62 time=4.82 ms
64 bytes from 10.192.145.230: icmp_seq=5 ttl=62 time=4.85 ms
```

IBMCloudサーバからAzureのサーバに対してポート80での疎通確認
```
[root@azure-sv ~]# hping3 -S -p 80 192.168.1.4
HPING 192.168.1.4 (eth0 192.168.1.4): S set, 40 headers + 0 data bytes
len=46 ip=192.168.1.4 ttl=62 DF id=0 sport=80 flags=SA seq=0 win=29200 rtt=4.9 ms
len=46 ip=192.168.1.4 ttl=62 DF id=0 sport=80 flags=SA seq=1 win=29200 rtt=5.8 ms
len=46 ip=192.168.1.4 ttl=62 DF id=0 sport=80 flags=SA seq=2 win=29200 rtt=6.7 ms
len=46 ip=192.168.1.4 ttl=62 DF id=0 sport=80 flags=SA seq=3 win=29200 rtt=5.6 ms
len=46 ip=192.168.1.4 ttl=62 DF id=0 sport=80 flags=SA seq=4 win=29200 rtt=5.5 ms
```

### traceroute/tracepath

Azureサーバからの経路確認
Active/Activeで接続されているため、疎通確認の度に経路が変わっていることがわかる

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/32c4b783-1ecc-66e4-10ce-8b4099086882.png)

```
[azureuser@SV1 ~]$ sudo traceroute -I 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
 1  169.254.22.1 (169.254.22.1)  5.965 ms  5.972 ms  5.974 ms
 2  * * *
 3  10.192.145.230 (10.192.145.230)  7.746 ms  7.751 ms  7.750 ms
[azureuser@SV1 ~]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
 1  169.254.21.1 (169.254.21.1)  5.753 ms  5.705 ms  5.674 ms
 2  * * *
 3  * * 10.192.145.230 (10.192.145.230)  5.526 ms
[azureuser@SV1 ~]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
 1  169.254.22.1 (169.254.22.1)  4.512 ms 169.254.21.1 (169.254.21.1)  4.086 ms 169.254.22.1 (169.254.22.1)  4.435 ms
 2  * * *
 3  10.192.145.230 (10.192.145.230)  5.076 ms  5.064 ms  5.031 ms
[azureuser@SV1 ~]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
 1  169.254.21.1 (169.254.21.1)  3.886 ms 169.254.22.1 (169.254.22.1)  3.789 ms  3.754 ms
 2  * * *
 3  10.192.145.230 (10.192.145.230)  5.207 ms  5.185 ms *
```

IBMCloudサーバからの経路確認

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/b3310e3f-a178-b97f-efb2-35a1fd2d1ea9.png)

```
[root@ibmcloud-sv ~]# tracepath 192.168.1.4
 1?: [LOCALHOST]                                         pmtu 1500
 1:  10.192.145.193                                        0.644ms
 1:  10.192.145.193                                        0.311ms
 2:  10.192.94.146                                         0.727ms
 3:  10.192.94.146                                         0.860ms pmtu 1422
 3:  192.168.1.4                                           4.790ms reached
     Resume: pmtu 1422 hops 3 back 3
[root@ibmcloud-sv ~]# tracepath 192.168.1.4
 1?: [LOCALHOST]                                         pmtu 1500
 1:  10.192.145.193                                        0.656ms
 1:  10.192.145.193                                        0.395ms
 2:  10.192.94.175                                         0.872ms
 3:  10.192.94.175                                         0.778ms pmtu 1422
 3:  192.168.1.4                                           5.004ms reached
     Resume: pmtu 1422 hops 3 back 3
```

### HTTP

AzureからIBMCloudのHTTP(80)サーバへの疎通確認
```
[azureuser@SV1 html]$ while true; do curl -w "gettime:%{time_total} " http://10.192.145.230; sleep 1; done
gettime:0.010 ibmcloud sv1
gettime:0.008 ibmcloud sv1
gettime:0.008 ibmcloud sv1
gettime:0.007 ibmcloud sv1
gettime:0.008 ibmcloud sv1
```

IBMCloudからAzureのHTTP(80)サーバへの疎通確認
```
[root@ibmcloud-sv ~]# while true; do curl -w "gettime:%{time_total} " http://192.168.1.4; sleep 1; done
gettime:0.010 azure sv1
gettime:0.008 azure sv1
gettime:0.009 azure sv1
gettime:0.008 azure sv1
gettime:0.009 azure sv1
```

# 障害時の通信確認
## VRA#1側のPublicインタフェース障害

![貼り付けた画像_2022_04_01_16_44.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/3408066f-a353-dc76-15aa-eaa5e8fb9a99.png)

ポータルのページからVRA#1のPublicインタフェースをdisableにする
![貼り付けた画像_2022_04_01_13_59.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/296b0753-6ff4-fdec-1e2c-1aa75a65cc5d.png)

### Azure側の状態確認

![貼り付けた画像_2022_04_01_14_06.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/bc873e1a-12c8-3624-95f0-90aa92b9fbcd.png)

![貼り付けた画像_2022_04_01_14_12.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/a0ee8978-ea87-62e1-ec8a-360214255af3.png)

### VRA#1側の状態確認

dp0bon1(publicインタフェース)とvti0がダウンしていることがわかる
```
vyatta@kou-vra001:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface       IP Address                        S/L  Speed/Duplex  Description
---------       ----------                        ---  ------------  -----------
dp0bond0        10.192.94.146/26                  u/u  2g/full
dp0bond1        128.168.104.aaa/27                u/D  -/-
dp0s0           -                                 u/u  a-1g/a-full
dp0s1           -                                 u/D  auto/auto
dp0s2           -                                 u/u  a-1g/a-full
dp0s3           -                                 u/D  auto/auto
erspan0         -                                 A/D  -/-
tun0            172.31.255.1/30                   u/u  -/-
vti0            169.254.21.1/30                   A/D  -/-           Azure
```

ipsecも停止している
```
vyatta@kou-vra001:~$ show vpn ike sa
Peer ID / IP                            Local ID / IP
------------                            -------------
20.222.88.aaa                           128.168.104.aaa

    State    Encrypt       Hash    D-H Grp  A-Time  L-Time IKEv
    -----  ------------  --------  -------  ------  ------ ----
    init   n/a           n/a       n/a      n/a     3600    2

vyatta@kou-vra001:~$ show vpn ipsec sa
vyatta@kou-vra001:~$
```

bgpも停止している
```
vyatta@kou-vra001:~$ show ip bgp neighbors
BGP neighbor is 169.254.21.2, remote AS 65515, local AS 65001, external link
  BGP version 4, remote router ID 0.0.0.0
  BGP state = Active
  Last read 00:01:06, hold time is 30, keepalive interval is 10 seconds
  Configured hold time is 30, keepalive interval is 10 seconds
  Received 0 messages, 0 notifications, 0 in queue
  Sent 0 messages, 0 notifications, 0 in queue
  Route refresh request: received 0, sent 0
  Minimum time between advertisement runs is 30 seconds
 For address family: IPv4 Unicast
  BGP table version 11, neighbor version 0
  Index 1, Offset 0, Mask 0x2
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor (both)
  0 accepted prefixes
  0 announced prefixes

 Connections established 10; dropped 10
Next connect timer due in 34 seconds
Last Reset: 00:05:19, due to BGP Notification sent
Notification Error Message: (Hold Timer Expired/Unspecified Error Subcode)
```

Azure(192.168.0.0/16)へのルートはOSPFでVRA#2からルーティングをもらっている
```
vyatta@kou-vra001:~$ show ip route ospf
IP Route Table for Routing-instance "default"
O E2 *> 192.168.0.0/16 [110/1] via 172.31.255.2, tun0, 00:23:10
```

### traceroute/tracepath

Azure側からはVRA#2側のみを通って通信していることがわかる

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/2c8240a6-0c4e-c263-dee4-b79de55c67cb.png)

```
[azureuser@SV1 html]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
 1  169.254.22.1 (169.254.22.1)  4.346 ms  4.285 ms  4.281 ms
 2  * * *
 3  10.192.145.230 (10.192.145.230)  5.905 ms * *
[azureuser@SV1 html]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
 1  169.254.22.1 (169.254.22.1)  3.819 ms  3.776 ms  3.732 ms
 2  * * *
 3  10.192.145.230 (10.192.145.230)  6.349 ms *  6.303 ms
```

IBMCloud側では、VRA#1~VRA#2間のOSPF経路を経由して、VRA#2側から通信している事がわかる

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/76d3d788-8022-3fce-92de-9692e56257d2.png)

```
[root@ibmcloud-sv ~]# tracepath 192.168.1.4
 1?: [LOCALHOST]                                         pmtu 1500
 1:  10.192.145.193                                        0.947ms
 1:  10.192.145.193                                        0.635ms
 2:  10.192.94.175                                         0.820ms
 3:  10.192.94.175                                         0.463ms pmtu 1422
 3:  192.168.1.4                                           4.576ms reached
     Resume: pmtu 1422 hops 3 back 3
[root@ibmcloud-sv ~]# tracepath 192.168.1.4
 1?: [LOCALHOST]                                         pmtu 1500
 1:  10.192.145.193                                        0.538ms
 1:  10.192.145.193                                        0.360ms
 2:  10.192.94.146                                         0.768ms
 3:  10.192.94.175                                         0.703ms pmtu 1422
 3:  192.168.1.4                                           4.034ms reached
     Resume: pmtu 1422 hops 3 back 3
```

### HTTP
azure側からibmcloud側のサーバに対して、http getに30秒
```
[azureuser@SV1 html]$ while true; do curl -w "gettime:%{time_total} " http://10.192.145.230; sleep 1; done
gettime:0.009 ibmcloud sv1
gettime:0.010 ibmcloud sv1
gettime:31.080 ibmcloud sv1
gettime:0.009 ibmcloud sv1
gettime:0.009 ibmcloud sv1
```

ibmcloud側からazure側のサーバに対して、http getに23秒
```
[root@ibmcloud-sv ~]# while true; do curl -w "gettime:%{time_total} " http://192.168.1.4; sleep 1; done
gettime:0.009 azure sv1
gettime:0.009 azure sv1
gettime:23.617 azure sv1
gettime:0.010 azure sv1
gettime:0.010 azure sv1
```


## VRA#2側のPublicインタフェース障害
ポータルのページからVRA#2のPublicインタフェースをdisableにする

![貼り付けた画像_2022_04_01_16_27.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/a16e0844-f9bc-dbdf-08f6-f77e7714e846.png)


### Azure側の状態確認
![貼り付けた画像_2022_04_01_14_26.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/35e5962c-419e-e4c4-a4fe-5bfe60e93389.png)

![貼り付けた画像_2022_04_01_14_27.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/ae536b30-17f8-31f4-03e9-60661ad06d9b.png)

### VRA#2側の状態確認

dp0bon1(publicインタフェース)とvti0がダウンしていることがわかる
```
vyatta@kou-vra002:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface       IP Address                        S/L  Speed/Duplex  Description
---------       ----------                        ---  ------------  -----------
dp0bond0        10.192.94.175/26                  u/u  2g/full
dp0bond1        128.168.104.bbb/27                u/D  -/-
dp0s0           -                                 u/u  a-1g/a-full
dp0s1           -                                 u/D  auto/auto
dp0s2           -                                 u/u  a-1g/a-full
dp0s3           -                                 u/D  auto/auto
erspan0         -                                 A/D  -/-
tun0            172.31.255.2/30                   u/u  -/-
vti0            169.254.22.1/30                   A/D  -/-           Azure
```

ipsecも停止している
```
vyatta@kou-vra002:~$ show vpn ike sa
Peer ID / IP                            Local ID / IP
------------                            -------------
20.222.89.bbb                            128.168.104.bbb

    State    Encrypt       Hash    D-H Grp  A-Time  L-Time IKEv
    -----  ------------  --------  -------  ------  ------ ----
    init   n/a           n/a       n/a      n/a     3600    2

vyatta@kou-vra002:~$ show vpn ipse sa
vyatta@kou-vra002:~$
```

bgpも停止している
```
vyatta@kou-vra002:~$ show ip bgp neighbors
BGP neighbor is 169.254.22.2, remote AS 65515, local AS 65001, external link
  BGP version 4, remote router ID 0.0.0.0
  BGP state = Active
  Last read 00:00:10, hold time is 30, keepalive interval is 10 seconds
  Configured hold time is 30, keepalive interval is 10 seconds
  Received 0 messages, 0 notifications, 0 in queue
  Sent 0 messages, 0 notifications, 0 in queue
  Route refresh request: received 0, sent 0
  Minimum time between advertisement runs is 30 seconds
 For address family: IPv4 Unicast
  BGP table version 9, neighbor version 0
  Index 1, Offset 0, Mask 0x2
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor (both)
  0 accepted prefixes
  0 announced prefixes

 Connections established 6; dropped 6
Next connect timer due in 87 seconds
Last Reset: 00:04:07, due to BGP Notification sent
Notification Error Message: (Hold Timer Expired/Unspecified Error Subcode)
```

Azure(192.168.0.0/16)へのルートはOSPFでVRA#2からルーティングをもらっている
```
vyatta@kou-vra002:~$ show ip route ospf
IP Route Table for Routing-instance "default"
O E2 *> 192.168.0.0/16 [110/1] via 172.31.255.1, tun0, 00:09:22
```


### traceroute/tracepath

Azure側からはVRA#1側のみを通って通信していることがわかる

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/fdb066a5-3b7b-f637-b8b2-22809d85fd13.png)

```
[azureuser@SV1 html]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
 1  169.254.21.1 (169.254.21.1)  6.663 ms  6.610 ms  6.570 ms
 2  * * *
 3  10.192.145.230 (10.192.145.230)  7.672 ms  7.663 ms *
[azureuser@SV1 html]$ traceroute 10.192.145.230
traceroute to 10.192.145.230 (10.192.145.230), 30 hops max, 60 byte packets
 1  169.254.21.1 (169.254.21.1)  4.283 ms  4.223 ms  4.191 ms
 2  * * *
 3  10.192.145.230 (10.192.145.230)  4.533 ms  4.881 ms *
```

IBMCloud側では、VRA#1~VRA#2間のOSPF経路を経由して、VRA#1側から通信している事がわかる

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/644bf385-a75c-ee64-7650-6d4c5c38f7fc.png)

```
[root@ibmcloud-sv ~]# tracepath 192.168.1.4
 1?: [LOCALHOST]                                         pmtu 1500
 1:  10.192.145.193                                        0.587ms
 1:  10.192.145.193                                        0.376ms
 2:  10.192.94.175                                         0.689ms
 3:  10.192.94.146                                         0.876ms pmtu 1422
 3:  192.168.1.4                                           4.822ms reached
     Resume: pmtu 1422 hops 3 back 3
[root@ibmcloud-sv ~]# tracepath 192.168.1.4
 1?: [LOCALHOST]                                         pmtu 1500
 1:  10.192.145.193                                        0.938ms
 1:  10.192.145.193                                        0.611ms
 2:  10.192.94.146                                         1.172ms
 3:  10.192.94.146                                         0.552ms pmtu 1422
 3:  192.168.1.4                                           5.104ms reached
     Resume: pmtu 1422 hops 3 back 3
```


### HTTP
azure側からibmcloud側のサーバに対して、http getに30秒
```
[azureuser@SV1 html]$ while true; do curl -w "gettime:%{time_total} " http://10.192.145.230; sleep 1; done
gettime:0.010 ibmcloud sv1
gettime:0.009 ibmcloud sv1
gettime:31.062 ibmcloud sv1
gettime:0.010 ibmcloud sv1
gettime:0.010 ibmcloud sv1
```

ibmcloud側からazure側のサーバに対して、http getに30秒
```
[root@ibmcloud-sv ~]# while true; do curl -w "gettime:%{time_total} " http://192.168.1.4; sleep 1; done
gettime:0.010 azure sv1
gettime:0.010 azure sv1
gettime:31.072 azure sv1
gettime:0.010 azure sv1
gettime:0.011 azure sv1
```

## VRA#1/VRA#2機器障害
確認結果はほぼ同じなので割愛する

VRA#1障害
Azure to IBMCloud
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/a5d9503e-fa6c-3788-884e-0d32667cdac6.png)

IBMCloud to Azure
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/a2fad8c2-8ec7-0b3f-14c7-476739fdc8b7.png)

VRA#2障害
Azure to IBMCloud
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/ad4cf2e0-112e-3ca3-31ff-9497de361a16.png)

IBMCloud to Azure
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/703043/b65c8096-623d-c1ea-31a1-c9d2dc2234e0.png)