Oracle Cloud: NEC IXルーターを冗長構成でIKEv2  IPSec VPN接続してみてみた

NECさんがIXルーターとOracle CloudとのIKEv2で冗長構成のIPSec VPN接続コンフィグを作成してくれました。
　・[Oracle CloudとのVPN接続:冗長構成](https://jpn.nec.com/univerge/ix/Support/OCI/index.html)
NEC IXルーターはVPN構築に適した、企業向けの高速アクセスルータです。
企業向けということで機器冗長構成も作成してしてくれたので、障害テストもやってみます。
ということで、早速やってみてみます。

# ■ 構成図
冗長はIPsec接続とオンプレミス側の顧客構内機器(CPE)を別々に設定することで、以下の構成のような冗長構成を組むことが出来ます。
<img width="1225" alt="構成図.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/647a996f-df41-e7e5-32e4-329c200ce57d.png">
OCIのDRGは物理的に機器冗長されているので、１台のCPE用にIPSec Connectionを作成すると２つのそれぞれ異なる機器のTunnel用Public IPが払い出されます
今回は、下記図のMultiple CPE構成でCPE側も２台構成で冗長させるので、合計４つのTunnelを張ることになります
<img width="983" alt="構成冗長.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/50525c9b-9cc5-f3f2-1c12-2840971b6c4c.png">


# ■ Oracle Cloud 側作業
Oracle Cloud(OCI)側作業は簡単です。CPEとIPSecリソースを作成するだけです。
構成図を参考に値を埋めて作成します。

## ● CPE(顧客構内機器)作成

### ① CPE作成
OCIのWebコンソールから、[ネットワーク] > [顧客構内機器] 画面から、[顧客構内機器の作成]をクリックし
以下内容を設定し、[CPEの作成]をクリック

    名前： 適当な名前を設定
    パブリックIPアドレス: 100.100.100.101
    ベンダー: NEC
    プラットフォームバージョン: 対象となるバージョンがあれば選択

<img width="1016" alt="01.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/02a521ae-fe10-8216-733c-acb48ba6d6a9.png">



### ② CPE作成確認
<img width="1014" alt="02.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/54163d45-f9aa-e0bf-aab4-aa8dfd799745.png">


## ● IPSec Connection作成
### ① IPSec Connection作成
OCIのWebコンソールから、[ネットワーク] > [IPSec接続] 画面から、[IPSec接続]をクリックし
以下内容を設定し、[IPSec接続の作成]をクリック

    名前： 適当な名前を設定
    顧客構内機器: 作成したCPEを選択
    動的ルーティンティング・ゲートウェイ: DRGを選択

<img width="733" alt="01.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/04a0c2c6-cf2b-eb40-a815-84ec38dfa39c.png">

拡張オプションの表示をクリックし、
・[CPE IKE識別子]タブ 
そのままデフォルト
<img width="484" alt="02.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/2ebd61da-01ff-73ec-4e56-f0d0b42bef08.png">


・[トンネル1]タブ

    名前:適当な名前を設定
    共有シークレット: 共有シークレットを設定
    IKEバージョン: IKEv2を設定
    ルーティング・タイプ: BGP動的ルーティングを設定
    BGP ASN: 65100を設定
    トンネル内インターフェース-CPE: 192.168.254.13/30 
    トンネル内インターフェース-Oracle 192.168.254.14/30 

<img width="479" alt="03.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/de0072ee-6ac9-221e-514d-201035d5d96b.png">

・[トンネル2]タブ
<img width="485" alt="04.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/e576dbee-08a1-820c-5f48-4517bec6e48b.png">


### ②IPSec Connection作成確認
<img width="1009" alt="05.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/a46dca91-685b-ae36-1864-d0b49737cfd4.png">


# ■ NEC IX Router設定


## ● 初期化
### ① コンフィグレーションモードで[erase startup-config]コマンドを実行し、スタートアップコンフィグを削除

```shell_session
Router01# enable-config
    Enter configuration commands, one per line. End with CNTL/Z.
Router01(config)# erase startup-config
    Are you sure you want to erase the startup-configuration? (Yes or [No]): Yes
```


### ② オペレーションモードに戻り、[default-console command-line]コマンドを実行

```shell_session
Router01(config)# exit
Router01# default-console command-line
    % You must RELOAD the router for this configuration to take effect.
```

### ③ 再起動

```shell_session
Router01# reload
    % Warning: current running-configuration is not saved yet.
    Notice: The router will be RELOADED. This is to ensure that
            the peripheral devices are properly initialized.
    Are you sure you want to reload the router? (Yes or [No]): Yes

    NEC Bootstrap Software
    Copyright (c) NEC Corporation 2001-2017. All rights reserved.

    %BOOT-INFO: Trying flash load, exec-image [ix2105-ms-9.6.12.a.ldc].
    Loading: #################################################################################### [OK]


    Starting at 0x20000

    Configuring router subsystems (before IDB proc): done.
    Constructing IDB(Interface Database): done.
    Configuring router subsystems (after IDB proc): done.
    Initializing router subsystems: done.
    Starting router subsystems: done.

    All router subsystems coming up.



    NEC Portable Internetwork Core Operating System Software
    Copyright Notices:
    Copyright (c) NEC Corporation 2001-2017. All rights reserved.
    Copyright (c) 1985-1998 OpenROUTE Networks, Inc.
    Copyright (c) 1984-1987, 1989 J. Noel Chiappa.

Router#
```

### ④ 初期化確認

```shell_session
Router# enable-config
    Enter configuration commands, one per line. End with CNTL/Z.
    Router(config)# show running-config
    ! NEC Portable Internetwork Core Operating System Software
    ! IX Series IX2105 (magellan-sec) Software, Version 9.6.12A, MAINTENANCE RELEASE SOFTWARE
    ! Compiled Oct 05-Thu-2017 19:27:09 JST #2
    ! Current time Mar 18-Wed-2020 13:46:56 JST
    !
    timezone +09 00
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    device GigaEthernet0
    !
    device GigaEthernet1
    !
    interface GigaEthernet0.0
    no ip address
    shutdown
    !
    interface GigaEthernet1.0
    no ip address
    shutdown
    !
    interface Loopback0.0
    no ip address
    !
    interface Null0.0
    no ip address

Router(config)#
```


# ■ NEC IX Router初期設定
ホスト名、Internet(ppp)接続、VRRP, LANとWANアドレス設定などなど設定します

## ● Master Router01
### ① ユーザー作成

```shell_session
Router(config)# username oracle password plain <パスワード> administrator
    % User 'oracle' has been added.
```

### ② ホスト名、ssh接続、Loggingなど初期設定
・ Host名設定

```shell_session
hostname Router01
```

・ ssh有効化

```shell_session
ssh-server ip enable
```


・ イベントログ取得設定

```shell_session
logging buffered 131072
logging subsystem all warn
logging timestamp datetime
```



### ③ Ineternet(ppp)接続設定

```shell-session
ppp profile web-ppp-gigaethernet0.1
  authentication myname [プロバイダ接続用ユーザID]
  authentication password [プロバイダ接続用ユーザID] [プロバイダ接続用パスワード]
!
interface GigaEthernet0.1
  description WAN1
  encapsulation pppoe
  auto-connect
  ppp binding web-ppp-gigaethernet0.1
  ip address ipcp
  ip tcp adjust-mss auto
  ip napt enable
  ip napt static GigaEthernet0.1 udp 500
  ip napt static GigaEthernet0.1 udp 4500
  ip napt static GigaEthernet0.1 50
  no shutdown
!
proxy-dns ip enable
proxy-dns interface GigaEthernet0.1 priority 254
```


### ④ LAN側インターフェースとVRRP,DHCP設定

```shell_session
ip route default GigaEthernet0.1
ip dhcp enable
vrrp enable
!
interface GigaEthernet1.0
  description LAN1
  ip address 192.168.100.211/24
  ip dhcp binding web-dhcp-gigaethernet1.0
  vrrp 10 ip 192.168.100.254
  vrrp 10 priority 100
  vrrp 10 ip virtual-host
  no shutdown
!
ip dhcp profile web-dhcp-gigaethernet1.0
　assignable-range 192.168.100.10 192.168.100.50
　dns-server 192.168.100.211
```


### ⑤ 設定確認

```shell_session
Router01(config)# show running-config
    ! NEC Portable Internetwork Core Operating System Software
    ! IX Series IX2105 (magellan-sec) Software, Version 9.6.12A, MAINTENANCE RELEASE SOFTWARE
    ! Compiled Oct 05-Thu-2017 19:27:09 JST #2
    ! Current time Mar 18-Wed-2020 15:57:48 JST
    !
    hostname Router01
    timezone +09 00
    !
    username oracle password hash AD8SFBB9DSVDSBN7 administrator
    !
    logging buffered 131072
    logging subsystem all warn
    logging timestamp datetime
    !
    !
    ip ufs-cache enable
    ip multipath per-flow
    ip route default GigaEthernet0.1
    ip dhcp enable
    !
    !
    !
    !
    !
    !
    !
    !
    !
    proxy-dns ip enable
    proxy-dns interface GigaEthernet0.1 priority 254
    !
    !
    ssh-server ip enable
    !
    !
    !
    !
    ppp profile web-ppp-gigaethernet0.1
    authentication myname [プロバイダ接続用ユーザID]
    authentication password [プロバイダ接続用ユーザID] [プロバイダ接続用パスワード]  
    !
    ip dhcp profile web-dhcp-gigaethernet1.0
    assignable-range 192.168.100.10 192.168.100.50
    dns-server 192.168.100.211
    !
    device GigaEthernet0
    !
    device GigaEthernet1
    !
    interface GigaEthernet0.0
    no ip address
    shutdown
    !
    interface GigaEthernet1.0
    description LAN1
    ip address 192.168.100.211/24
    ip dhcp binding web-dhcp-gigaethernet1.0
    no shutdown
    !
    interface GigaEthernet0.1
    description WAN1
    encapsulation pppoe
    auto-connect
    ppp binding web-ppp-gigaethernet0.1
    ip address ipcp
    ip tcp adjust-mss auto
    ip napt enable
    no shutdown
    !
    interface Loopback0.0
    no ip address
    !
    interface Null0.0
    no ip address
```


### ⑥ IP Address確認

```shell_session
Router01(config)# show ip address
    Interface GigaEthernet0.1 is up, line protocol is up
    Internet address is 100.100.100.101/32
    Broadcast address is 255.255.255.255
    Peer address is 200.100.100.246
    Address determined by IPCP
    Primary DNS server is 200.200.200.2
    Secondary DNS server is 200.200.200.3
    Interface GigaEthernet1.0 is up, line protocol is up
    Internet address is 192.168.100.211/24
    Broadcast address is 255.255.255.255
    Address determined by config
    Interface Null0.0 is up, line protocol is up
    Interface is unnumbered.
```

### ⑦ Internet接続確認

```shell-session
Router01(config)# ping google.com
    Looking up ipv4 address for "google.com" ...Success
    PING 100.100.100.101 > 172.217.175.78 56 data bytes
    64 bytes from 172.217.175.78: icmp_seq=0 ttl=56 time=15.248 ms
    64 bytes from 172.217.175.78: icmp_seq=1 ttl=56 time=2.767 ms
    64 bytes from 172.217.175.78: icmp_seq=2 ttl=56 time=2.882 ms
    64 bytes from 172.217.175.78: icmp_seq=3 ttl=56 time=5.910 ms

    --- 172.217.175.78 ping statistics ---
    4 packets transmitted, 4 packets received, 0% packet loss
    round-trip (ms)  min/avg/max = 2.767/6.701/15.248
```

### ⑧ VRRP設定確認

・ Master Router01

```shell_session
Router01(config)# show vrrp
Interface            VRID Pri Pre State      Master addr
GigaEthernet1.0        10 100 P   master     192.168.100.211
```

・ Master Router02

```shell_session
Router02(config)# show vrrp
  Interface            VRID Pri Pre State      Master addr
  GigaEthernet1.0        10  95 P   backup     192.168.100.211
```

## ● Standby Router02設定
上記設定の様にRouter02用のIPを使用して設定

### ・設定確認

```shell_session
Router01(config)# show running-config
    ! NEC Portable Internetwork Core Operating System Software
    ! IX Series IX2105 (magellan-sec) Software, Version 9.6.12A, MAINTENANCE RELEASE SOFTWARE
    ! Compiled Oct 05-Thu-2017 19:27:09 JST #2
    ! Current time Mar 18-Wed-2020 16:10:15 JST
    !
    hostname Router01
    timezone +09 00
    !
    username oracle password hash AD8SFBB9DSVDSBN7 administrator
    !
    logging buffered 131072
    logging subsystem all warn
    logging timestamp datetime
    !
    !
    ip ufs-cache enable
    ip multipath per-flow
    ip route default GigaEthernet0.1
    ip dhcp enable
    !
    !
    !
    !
    !
    !
    !
    !
    !
    proxy-dns ip enable
    proxy-dns interface GigaEthernet0.1 priority 254
    !
    !
    ssh-server ip enable
    !
    !
    !
    !
    ppp profile web-ppp-gigaethernet0.1
    authentication myname [プロバイダ接続用ユーザID]
    authentication password [プロバイダ接続用ユーザID] [プロバイダ接続用パスワード]
    !
    ip dhcp profile web-dhcp-gigaethernet1.0
    assignable-range 192.168.100.60 192.168.100.90
    dns-server 192.168.100.212
    !
    device GigaEthernet0
    !
    device GigaEthernet1
    !
    interface GigaEthernet0.0
    no ip address
    shutdown
    !
    interface GigaEthernet1.0
    description LAN1
    ip address 192.168.100.212/24
    ip dhcp binding web-dhcp-gigaethernet1.0
    no shutdown
    !
    interface GigaEthernet0.1
    description WAN1
    encapsulation pppoe
    auto-connect
    ppp binding web-ppp-gigaethernet0.1
    ip address ipcp
    ip tcp adjust-mss auto
    ip napt enable
    no shutdown
    !
    interface Loopback0.0
    no ip address
    !
    interface Null0.0
    no ip address
```



# ■ NEC IX Router IPSec VPNとBGP設定
LAN interface, IPSec VPNとBGPを設定
## ● Master Router01設定

```shell_session
ikev2 authentication psk id ipv4 140.204.100.101 key char SharedSecret01
ikev2 authentication psk id ipv4 140.204.100.102 key char SharedSecret02
!
ikev2 default-profile
  dpd interval 10
  source-address GigaEthernet0.1
!
interface Tunnel0.0
  tunnel mode ipsec-ikev2
  ip address 192.168.254.13/30
  ip tcp adjust-mss auto
  ikev2 child-pfs 1536-bit
  ikev2 child-proposal enc aes-cbc-256
  ikev2 child-proposal integrity sha1
  ikev2 connect-type auto
  ikev2 ipsec pre-fragment
  ikev2 outgoing-interface GigaEthernet0.1
  ikev2 sa-proposal enc aes-cbc-256
  ikev2 sa-proposal integrity sha2-384
  ikev2 sa-proposal dh 1536-bit
  ikev2 peer 140.204.100.101 authentication psk id ipv4 140.204.100.101
  no shutdown
!
interface Tunnel1.0
  tunnel mode ipsec-ikev2
  ip address 192.168.254.17/30
  ip tcp adjust-mss auto
  ikev2 child-pfs 1536-bit
  ikev2 child-proposal enc aes-cbc-256
  ikev2 child-proposal integrity sha1
  ikev2 connect-type auto
  ikev2 ipsec pre-fragment
  ikev2 outgoing-interface GigaEthernet0.1
  ikev2 sa-proposal enc aes-cbc-256
  ikev2 sa-proposal integrity sha2-384
  ikev2 sa-proposal dh 1536-bit
  ikev2 peer 140.204.100.102 authentication psk id ipv4 140.204.100.102
  no shutdown
!
route-map pri1 permit 10
  set metric 5
  set local-preference 200
!
route-map pri2 permit 10
  set metric 10
  set local-preference 150
!  
router bgp 65100
  neighbor 192.168.254.14 remote-as 31898
  neighbor 192.168.254.14 timers 10 30
  neighbor 192.168.254.18 remote-as 31898
  neighbor 192.168.254.18 timers 10 30
  neighbor 192.168.100.212 remote-as 65100
  neighbor 192.168.100.212 timers 10 30
  address-family ipv4 unicast
    neighbor 192.168.254.14 route-map pri1 in
    neighbor 192.168.254.14 route-map pri1 out
    neighbor 192.168.254.18 route-map pri2 in
    neighbor 192.168.254.18 route-map pri2 out
    neighbor 192.168.100.212 next-hop-self
    network 192.168.100.0/24
```

## ● Standby Router02設定

```shell_session
ikev2 authentication psk id ipv4 129.146.200.201 key char SharedSecret01
ikev2 authentication psk id ipv4 129.146.200.202 key char SharedSecret02
!
ikev2 default-profile
  child-pfs 1536-bit
  child-proposal enc aes-cbc-256
  child-proposal integrity sha1
  dpd interval 10
  sa-proposal enc aes-cbc-256
  sa-proposal integrity sha2-384
  sa-proposal dh 1536-bit
  source-address GigaEthernet0.1
!
interface Tunnel0.0
  tunnel mode ipsec-ikev2
  ip address 192.168.254.21/30
  ip tcp adjust-mss auto
  ikev2 connect-type auto
  ikev2 ipsec pre-fragment
  ikev2 outgoing-interface GigaEthernet0.1
  ikev2 peer 129.146.200.201 authentication psk id ipv4 129.146.200.201
  no shutdown
!
interface Tunnel1.0
  tunnel mode ipsec-ikev2
  ip address 192.168.254.25/30
  ip tcp adjust-mss auto
  ikev2 connect-type auto
  ikev2 ipsec pre-fragment
  ikev2 outgoing-interface GigaEthernet0.1
  ikev2 peer 129.146.200.202 authentication psk id ipv4 129.146.200.202
  no shutdown
!
router bgp 65100
  neighbor 192.168.254.22 remote-as 31898
  neighbor 192.168.254.22 timers 10 30
  neighbor 192.168.254.26 remote-as 31898
  neighbor 192.168.254.26 timers 10 30
  neighbor 192.168.100.211 remote-as 65100
  neighbor 192.168.100.211 timers 10 30
  address-family ipv4 unicast
    neighbor 192.168.254.22 route-map pri1 in
    neighbor 192.168.254.22 route-map pri1 out
    neighbor 192.168.254.26 route-map pri2 in
    neighbor 192.168.254.26 route-map pri2 out
    neighbor 192.168.100.211 next-hop-self
    network 192.168.100.0/24
```


# ■ BGP確認

### ① show ip bgp summary
すべてのBGP接続のステータスを表示

```shell_session
Router01(config)# show ip bgp summary
    BGP router ID 192.168.254.17, local AS number 65100
    2 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    192.168.100.212 4    65100 779     779     2:09:09       ESTABLISHED
    192.168.254.14  4    31898 842     786     2:10:35       ESTABLISHED
    192.168.254.18  4    31898 841     786     2:10:33       ESTABLISHED

    Total number of neighbors 3
```

### ② show ip bgp
BGPルーティング テーブルのエントリを表示

```shell_session
Router01(config)# show ip bgp
    BGP table version is 9, local router ID is 192.168.254.17
    Local AS number 65100
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *  10.0.0.0/24        192.168.254.18          10        150  31898 i
    *>                    192.168.254.14           5        200  31898 i
    * i                   192.168.100.212          5        200  31898 i
    *  134.70.80.0/23     192.168.254.18          10        150  31898 i
    *>                    192.168.254.14           5        200  31898 i
    * i                   192.168.100.212          5        200  31898 i
    *  134.70.82.0/23     192.168.254.18          10        150  31898 i
    *>                    192.168.254.14           5        200  31898 i
    * i                   192.168.100.212          5        200  31898 i
    *  140.91.32.0/23     192.168.254.18          10        150  31898 i
    *>                    192.168.254.14           5        200  31898 i
    * i                   192.168.100.212          5        200  31898 i
    *  140.204.8.128/25   192.168.254.18          10        150  31898 i
    *>                    192.168.254.14           5        200  31898 i
    * i                   192.168.100.212          5        200  31898 i
    *  192.29.36.0/22     192.168.254.18          10        150  31898 i
    *>                    192.168.254.14           5        200  31898 i
    * i                   192.168.100.212          5        200  31898 i
    *  192.29.40.0/22     192.168.254.18          10        150  31898 i
    *>                    192.168.254.14           5        200  31898 i
    * i                   192.168.100.212          5        200  31898 i
    *  192.29.44.0/25     192.168.254.18          10        150  31898 i
    *>                    192.168.254.14           5        200  31898 i
    * i                   192.168.100.212          5        200  31898 i
    *> 192.168.100.0/24   0.0.0.0                  1             i
    * i                   192.168.100.212          1        100  i

    Total number of prefixes 26
```

### ③ show ip bgp neighbors <neighbor-ip-address> advertised-routes
neighborにアドバタイズされたすべてのルートを表示

```shell_session
Router01(config)# show ip bgp neighbors 192.168.254.14 advertised-routes
    BGP table version is 9, local router ID is 192.168.254.17
    Local AS number 65100
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 192.168.100.0/24   192.168.254.13           5             i

    Total number of prefixes 1
```

### ④ show ip bgp neighbors <neighbor-ip-address> received-routes
neighborから受信されたすべてのルートを表示

```shell_session
Router01(config)# show ip bgp neighbors 192.168.254.14 received-routes
    BGP table version is 9, local router ID is 192.168.254.17
    Local AS number 65100
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 10.0.0.0/24        192.168.254.14           5        200  31898 i
    *> 134.70.80.0/23     192.168.254.14           5        200  31898 i
    *> 134.70.82.0/23     192.168.254.14           5        200  31898 i
    *> 140.91.32.0/23     192.168.254.14           5        200  31898 i
    *> 140.204.8.128/25   192.168.254.14           5        200  31898 i
    *> 192.29.36.0/22     192.168.254.14           5        200  31898 i
    *> 192.29.40.0/22     192.168.254.14           5        200  31898 i
    *> 192.29.44.0/25     192.168.254.14           5        200  31898 i

    Total number of prefixes 8
```


### ⑤ show ip route
ルーティング テーブルの内容を表示

```shell_session
Router01(config)# show ip route
    IP Routing Table - 14 entries, 7 hidden, 2027 frees
    Entries: 5 Connected, 1 Static, 0 RIP, 0 OSPF, 8 BGP
    Codes: C - Connected, S - Static, R - RIP, O - OSPF, IA - OSPF inter area
        N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
        E1 - OSPF external type 1, E2 - OSPF external type 2, B - BGP
        * - Candidate default, s - Summary
    Timers: Age
    S*   0.0.0.0/0 [1/1] is directly connected, GigaEthernet0.1, 1d9h48m10s
        10.0.0.0/8 is subnetted, 1 subnets
    B      10.0.0.0/24 [20/5] via 192.168.254.14, Tunnel0.0, 4:32:50
        134.70.0.0/16 is subnetted, 2 subnets
    B      134.70.80.0/23 [20/5] via 192.168.254.14, Tunnel0.0, 4:29:32
    B      134.70.82.0/23 [20/5] via 192.168.254.14, Tunnel0.0, 4:29:32
        140.91.0.0/16 is subnetted, 1 subnets
    B      140.91.32.0/23 [20/5] via 192.168.254.14, Tunnel0.0, 4:29:32
        140.204.0.0/16 is subnetted, 1 subnets
    B      140.204.8.128/25 [20/5] via 192.168.254.14, Tunnel0.0, 4:29:32
    B    192.29.36.0/22 [20/5] via 192.168.254.14, Tunnel0.0, 4:29:32
    B    192.29.40.0/22 [20/5] via 192.168.254.14, Tunnel0.0, 4:29:32
        192.29.44.0/24 is subnetted, 1 subnets
    B      192.29.44.0/25 [20/5] via 192.168.254.14, Tunnel0.0, 4:29:32
        192.168.100.0/24 is subnetted, 2 subnets
    C      192.168.100.0/24 [0/1] is directly connected, GigaEthernet1.0, 1d12m59s
    C      192.168.100.254/32 [0/1] is directly connected, Virtual10, 1d12m4s
        192.168.254.0/24 is subnetted, 2 subnets
    C      192.168.254.12/30 [0/1] is directly connected, Tunnel0.0, 5:49:32
    C      192.168.254.16/30 [0/1] is directly connected, Tunnel1.0, 5:49:31
        202.223.119.0/24 is subnetted, 1 subnets
    C      202.223.119.246/32 [0/1] is directly connected, GigaEthernet0.1, 1d9h49m59s
```


# ■ Oracle Cloud IPSec接続確認
IPSecステータスとBGPステータスが「稼働中」でグリーン色になっていればOK

① Router01
<img width="1009" alt="06.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/72786903-8976-3761-d35c-0094a17e8d59.png">


② Router02
<img width="1255" alt="06.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/5a750cf3-d255-4750-009c-126c401a2488.png">


# ■ 接続確認
## ● ping確認
① onp-inst01 -> OCI tokyo-inst01

```shell_session
[root@onp-inst01:~]$ ping 10.0.0.2 -c 3
    PING 10.0.0.2 (10.0.0.2): 56 data bytes
    64 bytes from 10.0.0.2: icmp_seq=0 ttl=62 time=46.431 ms
    64 bytes from 10.0.0.2: icmp_seq=1 ttl=62 time=48.592 ms
    64 bytes from 10.0.0.2: icmp_seq=2 ttl=62 time=61.959 ms

    --- 10.0.0.2 ping statistics ---
    3 packets transmitted, 3 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 46.431/52.327/61.959/6.868 ms
```

② onp-inst01 -> OCI Objectstorage

```shell-session
[root@onp-inst01:~]$ ping objectstorage.ap-tokyo-1.oraclecloud.com -c 3
    PING objectstorage.ap-tokyo-1.oraclecloud.com (134.70.80.1): 56 data bytes
    64 bytes from 134.70.80.1: icmp_seq=0 ttl=62 time=15.937 ms
    64 bytes from 134.70.80.1: icmp_seq=1 ttl=62 time=296.453 ms
    64 bytes from 134.70.80.1: icmp_seq=2 ttl=62 time=72.517 ms

    --- objectstorage.ap-tokyo-1.oraclecloud.com ping statistics ---
    3 packets transmitted, 3 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 15.937/128.302/296.453/121.123 ms
```

## ● traceroute確認
① onp-inst01 -> OCI tokyo-inst01

```shell_session
[root@onp-inst01:~]$ traceroute -I 10.0.0.2
    traceroute to 10.0.0.2 (10.0.0.2), 64 hops max, 72 byte packets
    1  * * *
    2  * * *
    3  10.0.0.2 (10.0.0.2)  32.602 ms  33.115 ms  35.621 ms
```


② onp-inst01 -> OCI Objectstorage

```shell-session
[root@onp-inst01:~]$ traceroute -I objectstorage.ap-tokyo-1.oraclecloud.com
    traceroute to objectstorage.ap-tokyo-1.oraclecloud.com (134.70.80.1), 64 hops max, 72 byte packets
    1  * * *
    2  * * *
    3  134.70.80.1 (134.70.80.1)  18.641 ms  37.581 ms  32.537 ms
```


# ■ LAN側抜線障害テスト
Master Router01のLAN側を抜線するとはVRRPでIPフェールオーバーしますが、WAN側のIPSec トンネルは生きているので、どうなるのでしょうか。
動作は以下の様な感じになるはずです
    抜線したMaster Router01のVRRPがStandby Router02へフェールオーバー
    抜線したMaster Router01側では、On-PremiseのRouteが対抗のOCI DRGへ伝搬されなくなり、
    DRGからのパケットはVRRPがフェールオーバーした Standby Router02のみへ送信される
ということで抜線してみてみます

## ● Master Router01のLAN抜線
### ① 状態確認

・VRRP確認
Master Router01(192.168.100.211)にVRRPのIPがあるMasterノードであることを確認

```shell_session
Router01(config)# show vrrp
    Interface            VRID Pri Pre State      Master addr
    GigaEthernet1.0        10 100 P   master     192.168.100.211
```


・BGP全確立確認
全てのBGP確立がされてESTABLISHED場外であることを確認

    eBGP: ２つのTunnel Peerアドレス(92.168.254.14,92.168.254.18)
    iBGP: Standby Router02アドレス(192.168.100.212)

```shell_session
Router01(config)# show ip bgp summary
    BGP router ID 192.168.254.17, local AS number 65001
    2 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    192.168.100.212 4    65001 13673   13680   1d4h20m39s    ESTABLISHED
    192.168.254.14  4    31898 14654   13690   1d7h24m23s    ESTABLISHED
    192.168.254.18  4    31898 14658   13689   6:48:24       ESTABLISHED

    Total number of neighbors 3
```


・BGP Neighbor側CPE Route伝搬確認
BGP Neighbor側となるOCI DRGが CPE(IX Router01)のROuter(CIDR:192.168.100.0/24)を伝搬し疎通状態であることを確認

```shell_session
Router01(config)# show ip bgp neighbors 192.168.254.14 advertised-routes
    BGP table version is 25, local router ID is 192.168.254.17
    Local AS number 65001
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 192.168.100.0/24   192.168.254.13           5             i

    Total number of prefixes 1
```


・BGP CPE側Neighbor Route伝搬確認
BGP CPE(IX Router01)側が、BGP Neighbor側となるOCI DRGののRouteを伝搬し疎通状態であることを確認
OCI DRGののRouteはここでは、以下、VCN Subnet(10.0.0.0/24)と Tokyo RegioのOSN CIDR群

```shell_session
Router01(config)# show ip bgp neighbors 192.168.254.14 received-routes
    BGP table version is 25, local router ID is 192.168.254.17
    Local AS number 65001
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 10.0.0.0/24        192.168.254.14           5        200  31898 i
    *> 134.70.80.0/23     192.168.254.14           5        200  31898 i
    *> 134.70.82.0/23     192.168.254.14           5        200  31898 i
    *> 140.91.32.0/23     192.168.254.14           5        200  31898 i
    *> 140.204.8.128/25   192.168.254.14           5        200  31898 i
    *> 192.29.36.0/22     192.168.254.14           5        200  31898 i
    *> 192.29.40.0/22     192.168.254.14           5        200  31898 i
    *> 192.29.44.0/25     192.168.254.14           5        200  31898 i

    Total number of prefixes 8
```


### ② 抜線前ping実行
抜線前にonp-inst01からping実行し抜線したらどうなるか、確認できるようにします

```shell_session
[root@onp-inst01:~]$ ping 10.0.0.2
    PING 10.0.0.2 (10.0.0.2): 56 data bytes
    64 bytes from 10.0.0.2: icmp_seq=0 ttl=62 time=5.662 ms
    64 bytes from 10.0.0.2: icmp_seq=1 ttl=62 time=7.754 ms
    64 bytes from 10.0.0.2: icmp_seq=2 ttl=62 time=6.971 ms
    ・・・
```


### ③ 抜線
Master Router01のLANを抜線
![抜線01.jpg](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/98a17526-e725-25f6-ce8b-9bd98b9902d8.jpeg)





### ④ 抜線後ping確認
②で実行中のping状態を確認し疎通できていることを確認
ここでは抜線後、３回Request timeout し復旧しています


```shell_session
[root@onp-inst01:~]$ ping 10.0.0.2
    PING 10.0.0.2 (10.0.0.2): 56 data bytes
    64 bytes from 10.0.0.2: icmp_seq=0 ttl=62 time=5.662 ms
    64 bytes from 10.0.0.2: icmp_seq=1 ttl=62 time=7.754 ms
    64 bytes from 10.0.0.2: icmp_seq=2 ttl=62 time=6.971 ms
    ・・・
    64 bytes from 10.0.0.2: icmp_seq=21 ttl=62 time=7.318 ms
    64 bytes from 10.0.0.2: icmp_seq=22 ttl=62 time=8.815 ms
    64 bytes from 10.0.0.2: icmp_seq=23 ttl=62 time=7.748 ms
    Request timeout for icmp_seq 24
    Request timeout for icmp_seq 25
    Request timeout for icmp_seq 26
    64 bytes from 10.0.0.2: icmp_seq=27 ttl=62 time=6.116 ms
    64 bytes from 10.0.0.2: icmp_seq=28 ttl=62 time=7.787 ms
    64 bytes from 10.0.0.2: icmp_seq=29 ttl=62 time=7.315 ms
    64 bytes from 10.0.0.2: icmp_seq=30 ttl=62 time=6.390 ms
```


### ⑤ Router01状態確認

・VRRP確認

```shell_session
Router01(config)# show vrrp
    Interface            VRID Pri Pre State      Master addr
    GigaEthernet1.0        10 100 P   initialize 0.0.0.0
```


・BGP状態確認
LAN側が抜線されているのでStandby Router02とのBGPが確立されていないことを確認

```shell_session
Router01(config)# show ip bgp summary
    BGP router ID 192.168.254.17, local AS number 65001
    2 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    192.168.100.212 4    65001 13762   13773   0:00:08       IDLE
    192.168.254.14  4    31898 14753   13784   1d7h39m41s    ESTABLISHED
    192.168.254.18  4    31898 14756   13783   7:03:42       ESTABLISHED
```


・BGP Neighbor側CPE Route伝搬確認
BGP Neighbor側となるOCI DRGが CPE(IX Router01)のRoute(CIDR:192.168.100.0/24)を伝搬せずCPE-DRG間疎通されないことを確認

```shell_session
Router01(config)# show ip bgp neighbors 192.168.254.14 advertised-routes
    BGP table version is 27, local router ID is 192.168.254.17
    Local AS number 65001
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path

    Total number of prefixes 0
```


### ⑥ Router02確認
・VRRP確認
VRRPがRouter02へフェールオーバーされていることを確認

```shell_session
Router02(config)# show vrrp
    Interface            VRID Pri Pre State      Master addr
    GigaEthernet1.0        10  95 P   master     192.168.100.212
```


・BGP状態確認
LAN側が抜線されているのでMaster Router01とのBGPが確立されていないことを確認


```shell_session
Router02(config)# show ip bgp summary
    BGP router ID 192.168.254.25, local AS number 65001
    2 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    192.168.100.211 4    65001 13772   13779   0:12:02       CONNECT
    192.168.254.22  4    31898 14885   13907   1d14h37m1s    ESTABLISHED
    192.168.254.26  4    31898 14837   13858   10:24:27      ESTABLISHED
```


・BGP Neighbor側CPE Route伝搬確認
抜線前とかわらず、BGP Neighbor側となるOCI DRGが CPE(IX Router01)のROuter(CIDR:192.168.100.0/24)を伝搬し疎通状態であることを確認

```shell_session
Router02(config)# show ip bgp neighbors 192.168.254.22 advertised-routes
    BGP table version is 9, local router ID is 192.168.254.25
    Local AS number 65001
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 192.168.100.0/24   192.168.254.21           5             i

    Total number of prefixes 1
```

・BGP CPE側Neighbor Route伝搬確認
抜線前とかわらず、BGP CPE(IX Router01)側が、BGP Neighbor側となるOCI DRGののRouteを伝搬し疎通状態であることを確認

```shell_session
Router02(config)# show ip bgp neighbors 192.168.254.22 received-routes
    BGP table version is 9, local router ID is 192.168.254.25
    Local AS number 65001
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 10.0.0.0/24        192.168.254.22           5        200  31898 i
    *> 134.70.80.0/23     192.168.254.22           5        200  31898 i
    *> 134.70.82.0/23     192.168.254.22           5        200  31898 i
    *> 140.91.32.0/23     192.168.254.22           5        200  31898 i
    *> 140.204.8.128/25   192.168.254.22           5        200  31898 i
    *> 192.29.36.0/22     192.168.254.22           5        200  31898 i
    *> 192.29.40.0/22     192.168.254.22           5        200  31898 i
    *> 192.29.44.0/25     192.168.254.22           5        200  31898 i

    Total number of prefixes 8
```



# ■ 抜線切り戻し
### ① 抜線切り戻し
![切り戻しのコピー.jpg](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109087/1cfc8e00-bef8-0c80-066b-57c6e4724d40.jpeg)




### ② 抜線切り戻しping確認
ここでは抜線後、1回Request timeoutし復旧しています

```shell_session
[root@onp-inst01:~]$ ping 10.0.0.2
    PING 10.0.0.2 (10.0.0.2): 56 data bytes
    64 bytes from 10.0.0.2: icmp_seq=0 ttl=62 time=5.662 ms
    64 bytes from 10.0.0.2: icmp_seq=1 ttl=62 time=7.754 ms
    64 bytes from 10.0.0.2: icmp_seq=2 ttl=62 time=6.971 ms
    ・・・
    64 bytes from 10.0.0.2: icmp_seq=1067 ttl=62 time=7.508 ms
    64 bytes from 10.0.0.2: icmp_seq=1068 ttl=62 time=7.073 ms
    64 bytes from 10.0.0.2: icmp_seq=1069 ttl=62 time=8.185 ms
    Request timeout for icmp_seq 1070
    64 bytes from 10.0.0.2: icmp_seq=1071 ttl=62 time=5.640 ms
    64 bytes from 10.0.0.2: icmp_seq=1072 ttl=62 time=7.220 ms
    64 bytes from 10.0.0.2: icmp_seq=1073 ttl=62 time=8.407 ms
```


### ③ Router02確認

・VRRP状態確認
VRRPがRouter01へフェールオーバー(フェールバック)されていることを確認

```shell_session
Router02(config)# show vrrp
    Interface            VRID Pri Pre State      Master addr
    GigaEthernet1.0        10  95 P   backup     192.168.100.211
```



### ④ Router01確認

・VRRP状態確認
VRRPがRouter01へフェールオーバー(フェールバック)されていることを確認

```shell_session
Router01(config)# show vrrp
    Interface            VRID Pri Pre State      Master addr
    GigaEthernet1.0        10 100 P   master     192.168.100.211
```


・BGP全確立確認
全てのBGP確立がされてESTABLISHED状態であることを確認

```shell_session
Router01(config)# show ip bgp summary
    BGP router ID 192.168.254.17, local AS number 65001
    2 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    192.168.100.212 4    65001 13811   13822   0:07:38       ESTABLISHED
    192.168.254.14  4    31898 14911   13933   1d8h4m13s     ESTABLISHED
    192.168.254.18  4    31898 14913   13932   7:28:14       ESTABLISHED

    Total number of neighbors 3
```

・BGP Neighbor側CPE Route伝搬確認
BGP Neighbor側となるOCI DRGが CPE(IX Router01)のROuter(CIDR:192.168.100.0/24)を伝搬し疎通状態であることを確認

```shell_session
Router01(config)# show ip bgp neighbors 192.168.254.14 advertised-routes
    BGP table version is 29, local router ID is 192.168.254.17
    Local AS number 65001
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 192.168.100.0/24   192.168.254.13           5             i

    Total number of prefixes 1
```


・BGP CPE側Neighbor Route伝搬確認
BGP CPE(IX Router01)側が、BGP Neighbor側となるOCI DRGののRouteを伝搬し疎通状態であることを確認

```shell_session
Router01(config)#  show ip bgp neighbors 192.168.254.14 received-routes
    BGP table version is 29, local router ID is 192.168.254.17
    Local AS number 65001
    Status codes: s - suppressed, * - valid, h - history
                > - best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop            Metric     LocPrf  Path
    *> 10.0.0.0/24        192.168.254.14           5        200  31898 i
    *> 134.70.80.0/23     192.168.254.14           5        200  31898 i
    *> 134.70.82.0/23     192.168.254.14           5        200  31898 i
    *> 140.91.32.0/23     192.168.254.14           5        200  31898 i
    *> 140.204.8.128/25   192.168.254.14           5        200  31898 i
    *> 192.29.36.0/22     192.168.254.14           5        200  31898 i
    *> 192.29.40.0/22     192.168.254.14           5        200  31898 i
    *> 192.29.44.0/25     192.168.254.14           5        200  31898 i

    Total number of prefixes 8
```


# ■ WAN側Tunnelダウン障害テスト
WAN側Internet IPは固定IPでないため、WAN抜線を行うと、IPが変わりIPSecの設定をし直さなくてはいけないため、
Tunnelダウンで障害を発生させます。
Master Route01は、WAN側を抜線、IP/TunnelダウンさせてもLAN側はアップでVRRPもMaster Route01のままです。
そのため、On-PremisのインスタンスがOCIへパケットを流すと、WAN側がダウンしている VRRP Master Router01へ送信されます。
が、iBGP動作でRouter01へ送信されたパケットは、Router02へ流れOCI DRGへ送信される動作になります。
ということで、確認してみてみます。


## ● Master Router01のTunnel0.0,Tunnel1.0 ダウン障害検証

### ① Tunnnelダウン前状態確認


・VRRP確認
 Master Router01(192.168.100.211)にVRRPのIPがあるMasterノードであることを確認

```shell_session
Router01(config)# show vrrp
Interface            VRID Pri Pre State      Master addr
GigaEthernet1.0        10 100 P   master     192.168.100.211
Router01(config)#
```

・BGP全確立確認
全てのBGP確立がされてESTABLISHED場外であることを確認

```shell_session
Router01(config)# show ip bgp summ
    BGP router ID 192.168.254.17, local AS number 65001
    2 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    192.168.100.212 4    65001 18691   18702   13:40:56      ESTABLISHED
    192.168.254.14  4    31898 20132   18813   1d21h37m31s   ESTABLISHED
    192.168.254.18  4    31898 20125   18802   2:48:54       ESTABLISHED
```


・ Tunnel0.0アップ確認

```shell_session
Router01(config)# show interfaces Tunnel0.0
    Interface Tunnel0.0 is up
    ・・・
```


・ Tunnel1.0アップ確認

```shell_session
Router01(config)# show interfaces Tunnel1.0
Interface Tunnel1.0 is up
    ・・・
```


・Route確認
OCI側のCIDRが、Tunnel0.0, Tunnel1.0を経由することを確認

```shell_session
Router01(config)# show ip route
    IP Routing Table - 14 entries, 7 hidden, 2027 frees
    Entries: 5 Connected, 1 Static, 0 RIP, 0 OSPF, 8 BGP
    Codes: C - Connected, S - Static, R - RIP, O - OSPF, IA - OSPF inter area
        N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
        E1 - OSPF external type 1, E2 - OSPF external type 2, B - BGP
        * - Candidate default, s - Summary
    Timers: Age
    S*   0.0.0.0/0 [1/1] is directly connected, GigaEthernet0.1, 3d9h32m3s
        10.0.0.0/8 is subnetted, 1 subnets
    B      10.0.0.0/24 [20/5] via 192.168.254.14, Tunnel0.0, 1d21h38m17s
        134.70.0.0/16 is subnetted, 2 subnets
    B      134.70.80.0/23 [20/5] via 192.168.254.14, Tunnel0.0, 1d21h38m17s
    B      134.70.82.0/23 [20/5] via 192.168.254.14, Tunnel0.0, 1d21h38m17s
        140.91.0.0/16 is subnetted, 1 subnets
    B      140.91.32.0/23 [20/5] via 192.168.254.14, Tunnel0.0, 1d21h38m17s
        140.204.0.0/16 is subnetted, 1 subnets
    B      140.204.8.128/25 [20/5] via 192.168.254.14, Tunnel0.0, 1d21h38m17s
    B    192.29.36.0/22 [20/5] via 192.168.254.14, Tunnel0.0, 1d21h38m18s
    B    192.29.40.0/22 [20/5] via 192.168.254.14, Tunnel0.0, 1d21h38m18s
        192.29.44.0/24 is subnetted, 1 subnets
    B      192.29.44.0/25 [20/5] via 192.168.254.14, Tunnel0.0, 1d21h38m18s
        192.168.100.0/24 is subnetted, 2 subnets
    C      192.168.100.0/24 [0/1] is directly connected, GigaEthernet1.0, 13:41:43
    C      192.168.100.254/32 [0/1] is directly connected, Virtual10, 13:41:40
        192.168.254.0/24 is subnetted, 2 subnets
    C      192.168.254.12/30 [0/1] is directly connected, Tunnel0.0, 2d5h33m25s
    C      192.168.254.16/30 [0/1] is directly connected, Tunnel1.0, 2:51:41
```


### ② Tunnleダウン前ping実行

```shell_session
[root@onp-inst01:~]$ ping 10.0.0.2
    PING 10.0.0.2 (10.0.0.2): 56 data bytes
    64 bytes from 10.0.0.2: icmp_seq=0 ttl=62 time=45.499 ms
    64 bytes from 10.0.0.2: icmp_seq=1 ttl=62 time=18.790 ms
    64 bytes from 10.0.0.2: icmp_seq=2 ttl=62 time=90.103 ms
    ・・・
```



### ③ Tunnelダウン障害実行

・ Tunnel0.0ダウン実行
VRRP Master Router01のTunnel0.0ダウン実行

```shell_session
Router01(config)# interface Tunnel0.0
    Router01(config-Tunnel0.0)# shutdown
```


・Tunnel0.0ダウン確認

```shell_session
Router01(config)# show interfaces Tunnel0.0
    Interface Tunnel0.0 is administratively down
```

・Tunnel1.0ダウン

```shell_session
Router01(config-Tunnel0.0)# interface Tunnel1.0
    Router01(config-Tunnel1.0)# shutdown
```


・Tunnel1.0ダウン確認

```shell_session
Router01(config)# show interfaces Tunnel1.0
    Interface Tunnel1.0 is administratively down
```



### ④ 状態確認


・VRRP確認
Tunnelが全てダウンしてもVRRPは変わらずRouter01(192.168.100.211)がMasterであることを確認

```shell_session
Router01(config)# show vrrp
Interface            VRID Pri Pre State      Master addr
GigaEthernet1.0        10 100 P   master     192.168.100.211
```


・BGP状態確認
TunnelのBGP(192.168.254.14, 192.168.254.18)が全てダウン(IDLE)状態であることを確認

```shell_session
Router01(config)# show ip bgp summary
    BGP router ID 192.168.254.17, local AS number 65001
    2 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    192.168.100.212 4    65001 18721   18733   13:45:57      ESTABLISHED
    192.168.254.14  4    31898 20154   18837   0:01:11       IDLE
    192.168.254.18  4    31898 20149   18828   0:00:50       IDLE
```


・Rote確認
OCI側のCIDRが、Router02(192.168.100.212)を経由することを確認

```shell_session
Router01(config)# show ip route
    IP Routing Table - 12 entries, 6 hidden, 2030 frees
    Entries: 3 Connected, 1 Static, 0 RIP, 0 OSPF, 8 BGP
    Codes: C - Connected, S - Static, R - RIP, O - OSPF, IA - OSPF inter area
        N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
        E1 - OSPF external type 1, E2 - OSPF external type 2, B - BGP
        * - Candidate default, s - Summary
    Timers: Age
    S*   0.0.0.0/0 [1/1] is directly connected, GigaEthernet0.1, 3d9h36m29s
        10.0.0.0/8 is subnetted, 1 subnets
    B      10.0.0.0/24 [200/5] via 192.168.100.212, GigaEthernet1.0, 0:01:22
        134.70.0.0/16 is subnetted, 2 subnets
    B      134.70.80.0/23 [200/5] via 192.168.100.212, GigaEthernet1.0, 0:01:22
    B      134.70.82.0/23 [200/5] via 192.168.100.212, GigaEthernet1.0, 0:01:22
        140.91.0.0/16 is subnetted, 1 subnets
    B      140.91.32.0/23 [200/5] via 192.168.100.212, GigaEthernet1.0, 0:01:22
        140.204.0.0/16 is subnetted, 1 subnets
    B      140.204.8.128/25 [200/5] via 192.168.100.212, GigaEthernet1.0, 0:01:22
    B    192.29.36.0/22 [200/5] via 192.168.100.212, GigaEthernet1.0, 0:01:22
    B    192.29.40.0/22 [200/5] via 192.168.100.212, GigaEthernet1.0, 0:01:22
        192.29.44.0/24 is subnetted, 1 subnets
    B      192.29.44.0/25 [200/5] via 192.168.100.212, GigaEthernet1.0, 0:01:22
        192.168.100.0/24 is subnetted, 2 subnets
    C      192.168.100.0/24 [0/1] is directly connected, GigaEthernet1.0, 13:46:08
    C      192.168.100.254/32 [0/1] is directly connected, Virtual10, 13:46:08
```



### ⑤ ping 確認
全てのTunnelがダウンしたので、ここではtimeoutが25回でてるが、iBGPにより経路がRouter02へ切り替わり疎通再開できていることを確認

```shell_session
[root@onp-inst01:~]$ ping 10.0.0.2
    PING 10.0.0.2 (10.0.0.2): 56 data bytes
    64 bytes from 10.0.0.2: icmp_seq=0 ttl=62 time=45.499 ms
    64 bytes from 10.0.0.2: icmp_seq=1 ttl=62 time=18.790 ms
    64 bytes from 10.0.0.2: icmp_seq=2 ttl=62 time=90.103 ms
    64 bytes from 10.0.0.2: icmp_seq=3 ttl=62 time=20.731 ms
    64 bytes from 10.0.0.2: icmp_seq=4 ttl=62 time=48.073 ms
    64 bytes from 10.0.0.2: icmp_seq=5 ttl=62 time=9.557 ms
    64 bytes from 10.0.0.2: icmp_seq=6 ttl=62 time=13.235 ms
    Request timeout for icmp_seq 7
    Request timeout for icmp_seq 8
    Request timeout for icmp_seq 9
    Request timeout for icmp_seq 10
    Request timeout for icmp_seq 11
    Request timeout for icmp_seq 12
    Request timeout for icmp_seq 13
    Request timeout for icmp_seq 14
    Request timeout for icmp_seq 15
    Request timeout for icmp_seq 16
    Request timeout for icmp_seq 17
    Request timeout for icmp_seq 18
    Request timeout for icmp_seq 19
    Request timeout for icmp_seq 20
    Request timeout for icmp_seq 21
    Request timeout for icmp_seq 22
    Request timeout for icmp_seq 23
    Request timeout for icmp_seq 24
    Request timeout for icmp_seq 25
    Request timeout for icmp_seq 26
    Request timeout for icmp_seq 27
    Request timeout for icmp_seq 28
    Request timeout for icmp_seq 29
    Request timeout for icmp_seq 30
    Request timeout for icmp_seq 31
    64 bytes from 10.0.0.2: icmp_seq=32 ttl=62 time=33.984 ms
    64 bytes from 10.0.0.2: icmp_seq=33 ttl=62 time=37.560 ms
    64 bytes from 10.0.0.2: icmp_seq=34 ttl=62 time=54.988 ms
    ・・・
```


## ● 切り戻し(障害復旧)

### ① Tunnelアップ
Tunnelアップさせて切り戻し(障害復旧)

・Tunnel0.0 アップ

```shell_session
Router01(config)# interface Tunnel0.0
Router01(config-Tunnel0.0)# no shutdown
```


・Tunnel0.0 アップ確認

```shell_session
Router01(config)# show interface Tunnel0.0
    Interface Tunnel0.0 is up
```


・Tunnel1.0 アップ

```shell_session
Router01(config-Tunnel0.0)# interface Tunnel1.0
Router01(config-Tunnel1.0)# no shutdown
```


・Tunnel1.0 アップ確認

```shell_session
Router01(config)# show interface Tunnel1.0
    Interface Tunnel1.0 is up
```


### ② BGP復旧確認
TunnelのBGP(192.168.254.14, 192.168.254.18)が全て復旧したことを確認

```shell_session
Router01(config)# show ip bgp summary
    BGP router ID 192.168.254.17, local AS number 65001
    2 BGP AS-PATH entries

    Neighbor         V    AS    MsgRcvd MsgSent Up/DownTime   State
    192.168.100.212 4    65001 18738   18751   13:48:44      ESTABLISHED
    192.168.254.14  4    31898 20166   18848   0:01:25       ESTABLISHED
    192.168.254.18  4    31898 20158   18837   0:01:04       ESTABLISHED
```


### ③ Route復旧確認
OCI側のCIDRが、Tunnel0.0, Tunnel1.0を経由するよう復旧したことを確認

```shell_session
Router01(config)# show ip route
    IP Routing Table - 14 entries, 7 hidden, 2027 frees
    Entries: 5 Connected, 1 Static, 0 RIP, 0 OSPF, 8 BGP
    Codes: C - Connected, S - Static, R - RIP, O - OSPF, IA - OSPF inter area
        N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
        E1 - OSPF external type 1, E2 - OSPF external type 2, B - BGP
        * - Candidate default, s - Summary
    Timers: Age
    S*   0.0.0.0/0 [1/1] is directly connected, GigaEthernet0.1, 3d9h39m13s
        10.0.0.0/8 is subnetted, 1 subnets
    B      10.0.0.0/24 [20/5] via 192.168.254.14, Tunnel0.0, 0:01:33
        134.70.0.0/16 is subnetted, 2 subnets
    B      134.70.80.0/23 [20/5] via 192.168.254.14, Tunnel0.0, 0:01:33
    B      134.70.82.0/23 [20/5] via 192.168.254.14, Tunnel0.0, 0:01:33
        140.91.0.0/16 is subnetted, 1 subnets
    B      140.91.32.0/23 [20/5] via 192.168.254.14, Tunnel0.0, 0:01:33
        140.204.0.0/16 is subnetted, 1 subnets
    B      140.204.8.128/25 [20/5] via 192.168.254.14, Tunnel0.0, 0:01:34
    B    192.29.36.0/22 [20/5] via 192.168.254.14, Tunnel0.0, 0:01:34
    B    192.29.40.0/22 [20/5] via 192.168.254.14, Tunnel0.0, 0:01:34
        192.29.44.0/24 is subnetted, 1 subnets
    B      192.29.44.0/25 [20/5] via 192.168.254.14, Tunnel0.0, 0:01:34
        192.168.100.0/24 is subnetted, 2 subnets
    C      192.168.100.0/24 [0/1] is directly connected, GigaEthernet1.0, 13:48:53
    C      192.168.100.254/32 [0/1] is directly connected, Virtual10, 13:48:52
        192.168.254.0/24 is subnetted, 2 subnets
    C      192.168.254.12/30 [0/1] is directly connected, Tunnel0.0, 0:01:41
    C      192.168.254.16/30 [0/1] is directly connected, Tunnel1.0, 0:01:19
```



### ④ ping確認
特にtimeoutせず、通信できていることを確認

```shell_session
    ・・・
    64 bytes from 10.0.0.2: icmp_seq=239 ttl=62 time=29.308 ms
    64 bytes from 10.0.0.2: icmp_seq=240 ttl=62 time=46.684 ms
    64 bytes from 10.0.0.2: icmp_seq=241 ttl=62 time=30.180 ms
    64 bytes from 10.0.0.2: icmp_seq=242 ttl=62 time=44.906 ms
    64 bytes from 10.0.0.2: icmp_seq=243 ttl=62 time=27.194 ms
    64 bytes from 10.0.0.2: icmp_seq=244 ttl=62 time=17.733 ms
    ・・・
```


# ■ コンフィグ確認
## ● Master Router01


```shell_session
Router01(config)# show running-config
! NEC Portable Internetwork Core Operating System Software
! IX Series IX2105 (magellan-sec) Software, Version 9.6.12A, MAINTENANCE RELEASE SOFTWARE
! Compiled Oct 05-Thu-2017 19:27:09 JST #2
! Current time Mar 19-Thu-2020 23:59:33 JST
!
hostname Router01
timezone +09 00
!
username oracle password hash AD8SFBB9DSVDSBN7 administrator
!
logging buffered 131072
logging subsystem all warn
logging timestamp datetime
!
!
ip ufs-cache enable
ip multipath per-flow
ip route default GigaEthernet0.1
ip dhcp enable
!
!
!
!
!
!
!
!
!
proxy-dns ip enable
proxy-dns interface GigaEthernet0.1 priority 254
!
!
ssh-server ip enable
!
vrrp enable
!
ikev2 authentication psk id ipv4 140.204.100.101 key char SharedSecret01
ikev2 authentication psk id ipv4 140.204.100.102 key char SharedSecret02
!
!
route-map pri1 permit 10
  set metric 5
  set local-preference 200
!
route-map pri2 permit 10
  set metric 10
  set local-preference 150
!
ppp profile web-ppp-gigaethernet0.1
  authentication myname [プロバイダ接続用ユーザID]
  authentication password [プロバイダ接続用ユーザID] [プロバイダ接続用パスワード]
!
ip dhcp profile web-dhcp-gigaethernet1.0
  assignable-range 192.168.100.10 192.168.100.50
  dns-server 192.168.100.211
!
router bgp 65100
  neighbor 192.168.254.14 remote-as 31898
  neighbor 192.168.254.14 timers 10 30
  neighbor 192.168.254.18 remote-as 31898
  neighbor 192.168.254.18 timers 10 30
  neighbor 192.168.100.212 remote-as 65100
  neighbor 192.168.100.212 timers 10 30
  address-family ipv4 unicast
    neighbor 192.168.254.14 route-map pri1 in
    neighbor 192.168.254.14 route-map pri1 out
    neighbor 192.168.254.18 route-map pri2 in
    neighbor 192.168.254.18 route-map pri2 out
    neighbor 192.168.100.212 next-hop-self
    network 192.168.100.0/24
!
ikev2 default-profile
  child-pfs 1536-bit
  child-proposal enc aes-cbc-256
  child-proposal integrity sha1
  dpd interval 10
  sa-proposal enc aes-cbc-256
  sa-proposal integrity sha2-384
  sa-proposal dh 1536-bit
  source-address GigaEthernet0.1
!
device GigaEthernet0
!
device GigaEthernet1
!
interface GigaEthernet0.0
  no ip address
  shutdown
!
interface GigaEthernet1.0
  description LAN1
  ip address 192.168.100.211/24
  ip dhcp binding web-dhcp-gigaethernet1.0
  vrrp 10 ip 192.168.100.254
  vrrp 10 ip virtual-host
  no shutdown
!
interface GigaEthernet0.1
  description WAN1
  encapsulation pppoe
  auto-connect
  ppp binding web-ppp-gigaethernet0.1
  ip address ipcp
  ip tcp adjust-mss auto
  ip napt enable
  ip napt static GigaEthernet0.1 udp 500
  ip napt static GigaEthernet0.1 udp 4500
  ip napt static GigaEthernet0.1 50
  no shutdown
!
interface Loopback0.0
  no ip address
!
interface Null0.0
  no ip address
!
interface Tunnel0.0
  tunnel mode ipsec-ikev2
  ip address 192.168.254.13/30
  ip tcp adjust-mss auto
  ikev2 connect-type auto
  ikev2 ipsec pre-fragment
  ikev2 outgoing-interface GigaEthernet0.1
  ikev2 peer 140.204.100.101 authentication psk id ipv4 140.204.100.101
  no shutdown
!
interface Tunnel1.0
  tunnel mode ipsec-ikev2
  ip address 192.168.254.17/30
  ip tcp adjust-mss auto
  ikev2 connect-type auto
  ikev2 ipsec pre-fragment
  ikev2 outgoing-interface GigaEthernet0.1
  ikev2 peer 140.204.100.102 authentication psk id ipv4 140.204.100.102
  no shutdown
```

・保存

```shell_session
Router01(config)# write memory
    Building configuration...
    % Warning: do NOT enter CNTL/Z while saving to avoid config corruption.
```

## ● Standby Router02

```shell_session
Router02(config)# show running-config
Current configuration : 3498 bytes

! NEC Portable Internetwork Core Operating System Software
! IX Series IX2105 (magellan-sec) Software, Version 9.2.20, RELEASE SOFTWARE
! Compiled Aug 19-Wed-2015 16:25:46 JST #2
! Current time Mar 19-Thu-2020 21:20:39 JST
!
!
hostname Router02
timezone +09 00
!
!
!
username oracle password hash AD8SFBB9DSVDSBN7 administrator
!
!
!
!
!
!
logging buffered 131072
logging subsystem all warn
logging timestamp datetime
!
!
ip ufs-cache enable
ip multipath per-flow
ip route default GigaEthernet0.1
ip dhcp enable
!
!
!
!
!
!
!
!
!
!
!
proxy-dns ip enable
proxy-dns interface GigaEthernet0.1 priority 254
!
!
ssh-server ip enable
!
!
!
!
!
vrrp enable
!
ikev2 authentication psk id ipv4 129.146.200.201 key char SharedSecret01
ikev2 authentication psk id ipv4 129.146.200.202 key char SharedSecret02
!
!
!
!
route-map pri1 permit 10
  set metric 5
  set local-preference 200
!
route-map pri2 permit 10
  set metric 10
  set local-preference 150
!
ppp profile web-ppp-gigaethernet0.1
  authentication myname [プロバイダ接続用ユーザID]
  authentication password [プロバイダ接続用ユーザID] [プロバイダ接続用パスワード]
!
ip dhcp profile web-dhcp-gigaethernet1.0
  assignable-range 192.168.100.100 192.168.100.110
  dns-server 192.168.100.212
!
router bgp 65100
  neighbor 192.168.254.22 remote-as 31898
  neighbor 192.168.254.22 timers 10 30
  neighbor 192.168.254.26 remote-as 31898
  neighbor 192.168.254.26 timers 10 30
  neighbor 192.168.100.211 remote-as 65100
  neighbor 192.168.100.211 timers 10 30
  address-family ipv4 unicast
    neighbor 192.168.254.22 route-map pri1 in
    neighbor 192.168.254.22 route-map pri1 out
    neighbor 192.168.254.26 route-map pri2 in
    neighbor 192.168.254.26 route-map pri2 out
    neighbor 192.168.100.211 next-hop-self
    network 192.168.100.0/24
!
ikev2 default-profile
  child-pfs 1536-bit
  child-proposal enc aes-cbc-256
  child-proposal integrity sha1
  dpd interval 10
  sa-proposal enc aes-cbc-256
  sa-proposal integrity sha2-384
  sa-proposal dh 1536-bit
  source-address GigaEthernet0.1
!
device GigaEthernet0
!
device GigaEthernet1
!
interface GigaEthernet0.0
  no ip address
  shutdown
!
interface GigaEthernet1.0
  description LAN1
  ip address 192.168.100.212/24
  ip dhcp binding web-dhcp-gigaethernet1.0
  vrrp 10 ip 192.168.100.254
  vrrp 10 priority 95
  vrrp 10 ip virtual-host
  no shutdown
!
interface GigaEthernet0.1
  description WAN1
  encapsulation pppoe
  auto-connect
  ppp binding web-ppp-gigaethernet0.1
  ip address ipcp
  ip tcp adjust-mss auto
  ip napt enable
  ip napt static GigaEthernet0.1 udp 500
  ip napt static GigaEthernet0.1 udp 4500
  ip napt static GigaEthernet0.1 50
  no shutdown
!
interface Loopback0.0
  no ip address
!
interface Null0.0
  no ip address
!
interface Tunnel0.0
  tunnel mode ipsec-ikev2
  ip address 192.168.254.21/30
  ip tcp adjust-mss auto
  ikev2 connect-type auto
  ikev2 ipsec pre-fragment
  ikev2 outgoing-interface GigaEthernet0.1
  ikev2 peer 129.146.200.201 authentication psk id ipv4 129.146.200.201
  no shutdown
!
interface Tunnel1.0
  tunnel mode ipsec-ikev2
  ip address 192.168.254.25/30
  ip tcp adjust-mss auto
  ikev2 connect-type auto
  ikev2 ipsec pre-fragment
  ikev2 outgoing-interface GigaEthernet0.1
  ikev2 peer 129.146.200.202 authentication psk id ipv4 129.146.200.202
  no shutdown
```

・保存

```shell_session
Router02(config)# write memory
    Building configuration...
    % Warning: do NOT enter CNTL/Z while saving to avoid config corruption.
```


# ■ 参考
### ● NEC 技術情報
　・[UNIVERGE IXシリーズ 技術情報](https://jpn.nec.com/univerge/ix/Support/OCI/index.html)
　・[Oracle CloudとのVPN接続](https://jpn.nec.com/univerge/ix/Support/OCI/index.html)
　・[Oracle Cloudの設定(BGP)](https://jpn.nec.com/univerge/ix/Support/OCI/bgp/index.html?)
　・[UNIVERGE IXシリーズ　障害切り分けガイドライン](https://jpn.nec.com/univerge/ix/Support/Troubleshooting/eventlog.html)
### ● Oracle Cloud Infrastructureドキュメント 
　・[NEC IXシリーズ](https://docs.oracle.com/cd/E97706_01/Content/Network/Reference/necixCPE.htm)
### ● Blog
　・[Oracle Cloud:NEC UNIVERGE IXルーターでOCIへIPsec VPN接続してみてみた](https://qiita.com/shirok/items/6501abbc63175bc68ce3)
　・[UNIVERGE IXルーターとOracle CloudをIPSec接続してAS-Path Prependで経路制御してみてみた](https://qiita.com/shirok/items/be7d6b06d7345831d84c)
　・[Transit Routing + IPSec VPN / FastConnectで Object Storage, Autonomous Databaseへ接続してみてみた](https://qiita.com/shirok/items/ec0e5a09346422346fd4)