AWS DX Hands-on LAB

###AWS 専用線アクセス体験ラボトレーニング
- AWS Direct Connect （AWS DX）へオンプレミスから接続する方法をハンズオン形式で体験できる
- オンプレミス側（CGW）はVyOS,VSRX（Juniper）、CSR（Cisco）のマルチベンダー構成
    - マルチベンダー構成にしているのは、ハンズオン環境で色々な機器を体験してもらうためで、実際に冗長構成を構築するときは 同じベンダー・同じバージョン での構築をお勧めします。（講師：AWSJ SA NWSP さん談）

####ハンズオンの流れ
- 自分のAWS アカウントIDを ラボ専用WEBサイトから登録すると、AWS DX の仮想インターフェース（VIF）が承認待ちの状態で作成される。 
- 下記ネットワーク図 左側のAS：65000 の構成はCloud Formation（cfn） で自動作成します
- 承認待ちで作成されたVIFの承認作業を行い、VPCにアタッチします。
- 2つのVPCとサブネットをAWS側に作成し、それぞれに作業用EC2（bastionとdx-training）を起動
- cfnで作成されたオンプレ側（AS：65000）と AWS側（AS：64512）の間に確立されるBGPピアを使い、オンプレ側のVyOSにアクセス。VyOSを踏み台にしてVSRXやCSRの設定を自分行いネットワーク図右側のオンプレ環境（AS：65001）とAWS側（AS:64512）のBGPピアを確立させる。 

####ハンズオン環境
![dxhandson.png](https://qiita-image-store.s3.amazonaws.com/0/71534/4c275ccb-f89b-f569-9187-de665871c366.png)


#### 構築後はハンズオン環境を使って、自由に検証作業をすることができます。
##### 検証その１：AS65000-VLAN 1018 をUPさせる
初期の設定では Virtual Interfaces：selfdxlab-vyos-18 の BGP State が down 状態なので 下記設定をVyOSへ追加しavailable にしてみます。
![VyOS18_BGP_Down.PNG](https://qiita-image-store.s3.amazonaws.com/0/71534/2e42430a-c5c8-fe7e-6c00-57a0dbb142cc.png)

```
aws@vyos:~$ sh ip bgp
BGP table version is 0, local router ID is 192.168.10.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.7.0.0/16      169.254.0.1                            0 64512 i
*> 192.168.10.0     0.0.0.0                  1         32768 i
```

1.interface ethernet eth0 にVLAN（vif）3318 を追加

```ps1:add_vif_3318_to_interface_ethernet_eth0
interfaces {
    ethernet eth0 {
        duplex auto
        hw-id 00:50:56:8f:af:34
        mtu 1522
        smp_affinity auto
        speed auto
        vif 1018 {
            address 169.254.0.2/30
            mtu 1500
        }
        vif 3318 {
            address 169.254.100.10/30
            mtu 1500
        }
```
2.BGP 設定にselfdxlab-vyos-18 (neighbor  169.254.100.9) のPeer設定を追加

```ps1:add_BGP_setting_to_VyOS
protocols {
    bgp 65000 {
        neighbor 169.254.0.1 {
            password ****************
            remote-as 64512
        }
        neighbor 169.254.100.9 {
            local-as 65001 {
                no-prepend
            }
            password ****************
            remote-as 64512
        }
        network 192.168.10.0/24 {
        }
    }
```
3.BGP 設定追加した neighbor 169.254.100.9 の Peerが追加されている事を確認します

```ps1:cheacking_BGP_Prre_neighbor-169.254.100.9
aws@vyos:~$ sh ip bgp
BGP table version is 0, local router ID is 192.168.10.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*  10.7.0.0/16      169.254.100.9                          0 64512 i
*>                  169.254.0.1                            0 64512 i
*> 192.168.10.0     0.0.0.0                  1         32768 i
```
しばらくするとAMCでもStateが down から available に状態が変わります
![VyOS18_BGP_Up.PNG](https://qiita-image-store.s3.amazonaws.com/0/71534/bbb68704-bb31-7dbe-7190-575fa242b409.png)

4.現在ベストパスになっている neighbor：169.254.0.1 をshutdownしてみます。

```ps1:shutdown_BGP_neighbor-169.254.0.1
aws@vyos:~$ configure
[edit]
aws@vyos# set protocols bgp 65000 neighbor 169.254.0.1 shutdown
[edit]
aws@vyos# commit
[edit]
aws@vyos# exit
Warning: configuration changes have not been saved.
exit
aws@vyos:~$ sh ip bgp
BGP table version is 0, local router ID is 192.168.10.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.7.0.0/16      169.254.100.9                          0 64512 i
*> 192.168.10.0     0.0.0.0                  1         32768 i

Total number of prefixes 2
```
neighbor：169.254.0.1 をshutdown しても neighbor：169.254.100.9 がベストパスとなり、通信が継続されます

```ps1:icmp_packet_doesn't_lost_when_commit_neighbor_shutdown
64 bytes from 192.168.10.10: icmp_seq=177 ttl=63 time=4.33 ms
64 bytes from 192.168.10.10: icmp_seq=178 ttl=63 time=4.38 ms
64 bytes from 192.168.10.10: icmp_seq=179 ttl=63 time=4.45 ms
64 bytes from 192.168.10.10: icmp_seq=180 ttl=63 time=4.32 ms
64 bytes from 192.168.10.10: icmp_seq=181 ttl=63 time=4.60 ms
64 bytes from 192.168.10.10: icmp_seq=182 ttl=63 time=4.44 ms
64 bytes from 192.168.10.10: icmp_seq=183 ttl=63 time=4.35 ms
64 bytes from 192.168.10.10: icmp_seq=184 ttl=63 time=4.46 ms
64 bytes from 192.168.10.10: icmp_seq=185 ttl=63 time=4.33 ms
64 bytes from 192.168.10.10: icmp_seq=186 ttl=63 time=4.41 ms
64 bytes from 192.168.10.10: icmp_seq=187 ttl=63 time=4.39 ms → この辺でcommit
64 bytes from 192.168.10.10: icmp_seq=188 ttl=63 time=4.35 ms
64 bytes from 192.168.10.10: icmp_seq=189 ttl=63 time=4.30 ms
64 bytes from 192.168.10.10: icmp_seq=190 ttl=63 time=4.35 ms
64 bytes from 192.168.10.10: icmp_seq=191 ttl=63 time=4.48 ms
64 bytes from 192.168.10.10: icmp_seq=192 ttl=63 time=4.47 ms
64 bytes from 192.168.10.10: icmp_seq=193 ttl=63 time=4.35 ms
64 bytes from 192.168.10.10: icmp_seq=194 ttl=63 time=4.40 ms
64 bytes from 192.168.10.10: icmp_seq=195 ttl=63 time=4.46 ms
64 bytes from 192.168.10.10: icmp_seq=196 ttl=63 time=4.48 ms
```
##### 検証その2：トラフィックエンジニアリング
vsrx と csr で同じ宛先（AS64512：172.16.0.0./16）に複数経路を保有するので、vsrxの優先度をcsrより上げる設定を投入し、トラフィックの流れを 正常時はvsrx、異常時はcsr になるようにオンプレ側（AS65001）の内部経路を制御します。

1. vsrx と csr の間にiBGPピア用の Lookback と ospf を設定
     - vsrx と csr にiBGPの起点としてLoopbackを設定し、OSPFでお互いのLoopbackを伝搬

     ```ps1:loopback_and_osfp_setting_on_vsrx
        #Loopbackインターフェース設定
        aws@vsrx1> show configuration interfaces lo0 | display set
        set interfaces lo0 unit 0 family inet address 10.0.0.255/32

        #ospf 設定
        aws@vsrx1> show configuration protocols ospf | display set
        set protocols ospf area 0.0.0.0 interface lo0.0 passive
        set protocols ospf area 0.0.0.0 interface ge-0/0/1.0

        #trust security zone で インターフェース（Loopback と ge-0/0/1.0）で許可設定
        aws@vsrx1> show configuration security zones security-zone trust interfaces lo0 | display set
        set security zones security-zone trust interfaces lo0.0 host-inbound-traffic system-services all

        aws@vsrx1> show configuration security zones security-zone trust interfaces ge-0/0/1.0 | display set
        set security zones security-zone trust interfaces ge-0/0/1.0 host-inbound-traffic protocols ospf

       #ospf設定確認
       aws@vsrx1> show route protocol ospf brief
       
       inet.0: 12 destinations, 13 routes (12 active, 0 holddown, 0 hidden)
       + = Active Route, - = Last Active, * = Both
       
       10.0.0.254/32      *[OSPF/10] 2w1d 12:51:55, metric 2
                           > to 192.168.10.30 via ge-0/0/1.0
       224.0.0.5/32       *[OSPF/10] 2w1d 20:14:59, metric 1
                             MultiRecv

        #crsのLoopbackへ疎通確認
        aws@vsrx1> ping 10.0.0.255 count 3
        PING 10.0.0.255 (10.0.0.255): 56 data bytes
        64 bytes from 10.0.0.255: icmp_seq=0 ttl=64 time=0.218 ms
        64 bytes from 10.0.0.255: icmp_seq=1 ttl=64 time=0.330 ms
        64 bytes from 10.0.0.255: icmp_seq=2 ttl=64 time=0.385 ms

     ```

     ```ps1:loopback_and_osfp_setting_on_csr
        !
        interface Loopback0
         ip address 10.0.0.254 255.255.255.255
        !
        !
        router ospf 100
         passive-interface Loopback0
         network 10.0.0.254 0.0.0.0 area 0.0.0.0
         network 192.168.10.0 0.0.0.255 area 0.0.0.0
        !
 
       #ospf設定確認
        csr#show ip route ospf
        Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
               D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
               N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
               E1 - OSPF external type 1, E2 - OSPF external type 2
               i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
               ia - IS-IS inter area, * - candidate default, U - per-user static route
               o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
               a - application route
               + - replicated route, % - next hop override, p - overrides from PfR
        
        Gateway of last resort is not set
        
              10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
        O        10.0.0.255/32 [110/1] via 192.168.10.20, 2w1d, GigabitEthernet2       

        #vsrxのLoopbackへ疎通確認
        csr#ping 10.0.0.255
        Type escape sequence to abort.
        Sending 5, 100-byte ICMP Echos to 10.0.0.255, timeout is 2 seconds:
        !!!!!
        Success rate is 100 percent (5/5), round-trip min/avg/max = 2/8/19 ms

     ```

2. iBGP の設定

```ps1:loopback_and_iBGP_setting_on_vsrx
#iBGPの NextHop Self 用ポリシー設定
aws@vsrx1> show configuration policy-options policy-statement NEXT-HOP-SELF | display set
set policy-options policy-statement NEXT-HOP-SELF term DEFAULT then next-hop self

#iBGPの設定
aws@vsrx1> show configuration protocols bgp group IBGP | display set
set protocols bgp group IBGP type internal
set protocols bgp group IBGP local-address 10.0.0.255
set protocols bgp group IBGP export NEXT-HOP-SELF
set protocols bgp group IBGP local-as 65001
set protocols bgp group IBGP neighbor 10.0.0.254

#iBGPの状態確認（State: Established を確認します）
aws@vsrx1> show bgp neighbor 10.0.0.254
Peer: 10.0.0.254+179 AS 65001  Local: 10.0.0.255+63481 AS 65001
  Type: Internal    State: Established    Flags: <Sync>
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: None
  Export: [ NEXT-HOP-SELF ]
  Options: <Preference LocalAddress LocalAS Refresh>
  Local Address: 10.0.0.255 Holdtime: 90 Preference: 170 Local AS: 65001 Local System AS: 65001
  Number of flaps: 0
  Peer ID: 192.168.10.30   Local ID: 10.0.0.255        Active Holdtime: 90
  Keepalive Interval: 30         Peer index: 0
  BFD: disabled, down
  NLRI for restart configured on peer: inet-unicast
  NLRI advertised by peer: inet-unicast
  NLRI for this session: inet-unicast
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Peer does not support Restarter functionality
  Peer does not support Receiver functionality
  Peer supports 4 byte AS extension (peer-as 65001)
  Peer does not support Addpath
  Table inet.0 Bit: 10001
    RIB State: BGP restart is complete
    Send state: in sync
    Active prefixes:              0
    Received prefixes:            1
    Accepted prefixes:            1
    Suppressed due to damping:    0
    Advertised prefixes:          1
  Last traffic (seconds): Received 5    Sent 27   Checked 28
  Input messages:  Total 48982  Updates 6       Refreshes 0     Octets 930823
  Output messages: Total 49689  Updates 2       Refreshes 0     Octets 944199
  Output Queue[0]: 0
```

```ps1:loopback_and_iBGP_setting_on_csr
csr#sh run | begin router bgp 65001
router bgp 65001
 bgp log-neighbor-changes
 network 192.168.10.0
 neighbor 10.0.0.255 remote-as 65001
 neighbor 10.0.0.255 update-source Loopback0
 neighbor 10.0.0.255 next-hop-self

#iBGPの状態確認（state = Established を確認します）
csr#sh ip bgp nei 10.0.0.255
BGP neighbor is 10.0.0.255,  remote AS 65001, internal link
  BGP version 4, remote router ID 10.0.0.255
  BGP state = Established, up for 2w1d
  Last read 00:00:01, last write 00:00:22, hold time is 90, keepalive interval is 30 seconds
  Neighbor sessions:
    1 active, is not multisession capable (disabled)
  Neighbor capabilities:
    Route refresh: advertised and received(new)
    Four-octets ASN Capability: advertised and received
    Address family IPv4 Unicast: advertised and received
    Graceful Restart Capability: received
      Remote Restart timer is 120 seconds
      Address families advertised by peer:
        none
    Enhanced Refresh Capability: advertised
    Multisession Capability:
    Stateful switchover support enabled: NO for session 1
  Message statistics:
    InQ depth is 0
    OutQ depth is 0

                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:                6          2
    Keepalives:         49014      49725
    Route Refresh:          0          0
    Total:              49021      49728
  Do log neighbor state changes (via global configuration)
  Default minimum time between advertisement runs is 0 seconds
```
3.eBGPの優先度設定（Local Preferecce と AS Path Prepend）

```ps1:eBGP_LP_setting_on_vsrx
#受信ルートに LP:200 を設定
aws@vsrx1> show configuration policy-options policy-statement ROUTE-IN | display set
set policy-options policy-statement ROUTE-IN term DEFAULT then local-preference 200

#受信ルートフィルターにEBGP を設定
aws@vsrx1> show configuration protocols bgp group EBGP import | display set
set protocols bgp group EBGP import ROUTE-IN

```

```ps1:eBGP_LP_and_pass-prepend_setting_on_csr
#AWS側のAS番号を正規表現で access-list 1 へ定義
!
ip as-path access-list 1 permit ^64512$
!
#オンプレミス側から広報するネットワークプリフィックスを定義
!
ip prefix-list 10 seq 5 permit 192.168.10.0/24
!
#eBGP送信用のルートフィルタでAS Path Prepend を設定
!
route-map ROUTE-OUT permit 10
 match ip address prefix-list 10
 set as-path prepend 65001 65001
!
#eBGP受信ルートフィルターの設定で LP:100 を設定
!
route-map ROUTE-IN permit 10
 match as-path 1
 set local-preference 100
!
#AWS側のeBGP neighbor のIN・OUTの経路へフィルタを設定
!
 neighbor 169.254.100.5 route-map ROUTE-IN in
 neighbor 169.254.100.5 route-map ROUTE-OUT out
!

```
4.経路制御の設定確認（vsrx）

- vsrxで受信するAWS側 BGP Peer（169.254.100.1）からの経路にはLP:200が設定されています
- csrからの受信ルート（

```ps1:eBGP_inbound_route_LP_setting_check_on_vsrx
aws@vsrx1> show route protocol bgp

inet.0: 12 destinations, 13 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.16.0.0/16      *[BGP/170] 2w2d 01:34:29, localpref 200
                      AS path: 64512 I
                    > to 169.254.100.1 via ge-0/0/0.0
192.168.10.0/24     [BGP/170] 2w1d 17:21:46, MED 0, localpref 100, from 10.0.0.254
                      AS path: I
                    > to 192.168.10.30 via ge-0/0/1.0
aws@vsrx1> show route receive-protocol bgp 10.0.0.254

inet.0: 12 destinations, 13 routes (12 active, 0 holddown, 0 hidden)
  Prefix                  Nexthop              MED     Lclpref    AS path
  192.168.10.0/24         10.0.0.254           0       100        I

```

- vsrxからAWS側 BGP Peer（169.254.100.1）への送信経路にはAS Path Prependが設定されていません

```ps1:eBGP_outbound_route_LP_setting_check_on_vsrx
aws@vsrx1> show route advertising-protocol bgp 169.254.100.1

inet.0: 12 destinations, 13 routes (12 active, 0 holddown, 0 hidden)
  Prefix                  Nexthop              MED     Lclpref    AS path
* 192.168.10.0/24         Self                                    I

```

- vsrxで受信するAWS側 BGP Peer（169.254.100.1）からの経路にはAS Path Prependが設定されていません。
- オンプレ側からの見たときには、LPが優先されます。

```ps1:eBGP_inbound_route_prepend_check_on_vsrx
aws@vsrx1> show route receive-protocol bgp 169.254.100.1

inet.0: 12 destinations, 13 routes (12 active, 0 holddown, 0 hidden)
  Prefix                  Nexthop              MED     Lclpref    AS path
* 172.16.0.0/16           169.254.100.1                           64512 I

```
5.経路制御の設定確認（csr）

- csrで受信するAWS側VPC（172.16.0.0/16）の経路は AWS側VGW BGP Peer（169.254.100.5）からの経路に設定されたLP:100 より、vsrxとのiBPG Peer（10.0.0.255）からの経路に設定された LP:200 の方が優先度が高いため、vsrxからの経路がベストパスとなります。

```ps1:eBGP_inbound_route_LP_setting_check_on_csr
csr#sh ip bgp
BGP table version is 4, local router ID is 192.168.10.30
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i 172.16.0.0       10.0.0.255                    200      0 64512 i
 *                    169.254.100.5                 100      0 64512 i
 *>  192.168.10.0     0.0.0.0                  0         32768 i
```

- csrからAWS側VGW BGP Peer（169.254.100.5）へ広報するオンプレ側（192.168.10.0/24）の経路はAS Path Prepend が設定されているため、AWSからオンプレへ向かう経路のベストパスはvsrxになります。

6.経路制御の動作確認

- オンプレ側のubuntuから、AWS側のEC2（rx-traning）にICMPを送りvsrxとAWS VGW間のeGBP セッションを切断、csr側にトラフィックが迂回されるか確認します。

```ps1:eBGP_deactivate_neighber_on_vsrx
#現在の経路を確認
#AWS VPC（172.16.0.0/16）へのベストパスはAWS VGW（169.254.100.1）
aws@vsrx1> show route 172.16.0.0

inet.0: 12 destinations, 13 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.16.0.0/16      *[BGP/170] 00:00:05, localpref 200
                      AS path: 64512 I
                    > to 169.254.100.1 via ge-0/0/0.0

#AWS VGW（169.254.100.1）とのeBGPセッションを切断
aws@vsrx1> configure exclusive
warning: uncommitted changes will be discarded on exit
Entering configuration mode

[edit]
aws@vsrx1# deactivate protocols bgp group EBGP neighbor 169.254.100.1

[edit]
aws@vsrx1# commit check
configuration check succeeds

[edit]
aws@vsrx1# commit and-quit
commit complete
Exiting configuration mode

#AWS VGW（169.254.100.1）とのeBGPセッションを切断後の経路を確認
#AWS VPC（172.16.0.0/16）へのベストパスは csrとのiBGP（10.0.0.254）からの
#経路に変更されている
aws@vsrx1> show route 172.16.0.0

inet.0: 12 destinations, 13 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.16.0.0/16      *[BGP/170] 00:00:18, MED 0, localpref 100, from 10.0.0.254
                      AS path: 64512 I
                    > to 192.168.10.30 via ge-0/0/1.0
```

- vsrxとVPC VGWとのセッション切断時にオンプレ側のubuntuから、AWS側のEC2（rx-traning）にICMPを送り続けても接続が切れないことを確認

```ps1:ICMP_cheack_from_ubuntu_to_VPC_traning_server
aws@ubuntu:~$ ping 172.16.0.100
PING 172.16.0.100 (172.16.0.100) 56(84) bytes of data.
64 bytes from 172.16.0.100: icmp_seq=1 ttl=243 time=8.41 ms
64 bytes from 172.16.0.100: icmp_seq=2 ttl=243 time=7.82 ms
64 bytes from 172.16.0.100: icmp_seq=3 ttl=243 time=9.75 ms
64 bytes from 172.16.0.100: icmp_seq=4 ttl=243 time=8.52 ms
64 bytes from 172.16.0.100: icmp_seq=5 ttl=243 time=6.82 ms
64 bytes from 172.16.0.100: icmp_seq=6 ttl=243 time=10.6 ms
64 bytes from 172.16.0.100: icmp_seq=7 ttl=243 time=9.14 ms
64 bytes from 172.16.0.100: icmp_seq=8 ttl=243 time=6.63 ms　→　この辺でcommit
64 bytes from 172.16.0.100: icmp_seq=9 ttl=243 time=7.30 ms
64 bytes from 172.16.0.100: icmp_seq=10 ttl=243 time=6.23 ms
64 bytes from 172.16.0.100: icmp_seq=11 ttl=243 time=5.86 ms
64 bytes from 172.16.0.100: icmp_seq=12 ttl=243 time=6.33 ms
64 bytes from 172.16.0.100: icmp_seq=13 ttl=243 time=6.39 ms
64 bytes from 172.16.0.100: icmp_seq=14 ttl=243 time=6.13 ms
64 bytes from 172.16.0.100: icmp_seq=15 ttl=243 time=6.18 ms
64 bytes from 172.16.0.100: icmp_seq=16 ttl=243 time=6.07 ms

```
- vsrxとVPC VGWとのセッションを復旧させます

```ps1:eBGP_deactivate_neighber_on_vsrx
aws@vsrx1> configure exclusive
warning: uncommitted changes will be discarded on exit
Entering configuration mode

[edit]
aws@vsrx1# activate protocols bgp group EBGP neighbor 169.254.100.1

[edit]
aws@vsrx1# commit and-quit
commit complete
Exiting configuration mode

```
##### 検証その3：障害時のフェールオーバー（Keepalive と Hold Timer）
- vsrx側の Keepalive と Hold Timer を変更

```ps1:edit_Keepalive_and_Hold-Timer_on_vsrx
#現在の設定値（初期値）を確認
aws@vsrx1> show bgp neighbor 169.254.100.1 | match Holdtime
  Local Address: 169.254.100.2 Holdtime: 90 Preference: 170
  Peer ID: 103.246.151.102 Local ID: 10.0.0.255        Active Holdtime: 90

aws@vsrx1> show bgp neighbor 169.254.100.1 | match Keepalive
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Keepalive Interval: 30         Peer index: 0

#Keepaliveを10s Hold Timerを30sに設定
aws@vsrx1> configure exclusive
warning: uncommitted changes will be discarded on exit
Entering configuration mode
[edit]
aws@vsrx1# set protocols bgp group EBGP neighbor 169.254.100.1 hold-time 30

[edit]
aws@vsrx1# commit check
configuration check succeeds

[edit]
aws@vsrx1# commit and-quit
commit complete
Exiting configuration mode

#設定変更を確認（Holdtimeが90から30に変更されている）
aws@vsrx1> show bgp neighbor 169.254.100.1 | match Holdtime
  Options: <Preference LocalAddress HoldTime AuthKey PeerAS Refresh>
  Local Address: 169.254.100.2 Holdtime: 30 Preference: 170
  Peer ID: 103.246.151.102 Local ID: 10.0.0.255        Active Holdtime: 30

aws@vsrx1> show bgp neighbor 169.254.100.1 | match Keepalive
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Keepalive Interval: 10         Peer index: 0

#AS:64512のneighbor（169.254.100.1）とのbgpセッションをsoftリセット
aws@vsrx1> clear bgp neighbor 169.254.100.1 soft-inbound

```

- csr側の Keepalive と Hold Timer を変更

```ps1:edit_Keepalive_and_Hold-Timer_on_csr
#現在の設定値（初期値）を確認
csr#sh ip bgp neighbors 169.254.100.5 | include hold
  Last read 00:00:17, last write 00:00:17, hold time is 90, keepalive interval is 30 seconds
minRTT: 2 ms, maxRTT: 1000 ms, ACK hold: 200 ms

#Keepaliveを10s Hold Timerを30sに設定
csr#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
csr(config)#router bgp 65001
csr(config-router)#neighbor 169.254.100.5 timers 10 30
csr(config-router)#^Z

#設定変更を確認
csr#sh run | include 169.254.100.5
 neighbor 169.254.100.5 remote-as 64512
 neighbor 169.254.100.5 password aws123
 neighbor 169.254.100.5 timers 10 30
 neighbor 169.254.100.5 route-map ROUTE-IN in
 neighbor 169.254.100.5 route-map ROUTE-OUT out

#AS:64512のneighbor（169.254.100.5）とのbgpセッションをsoftリセット
csr#clear ip bgp 169.254.100.5 soft in
csr#
csr#sh ip bgp neighbors 169.254.100.5 | include hold
  Last read 00:00:14, last write 00:00:11, hold time is 90, keepalive interval is 30 seconds
  Configured hold time is 30, keepalive interval is 10 seconds
  Minimum holdtime from neighbor is 0 seconds
minRTT: 2 ms, maxRTT: 1000 ms, ACK hold: 200 m

```

##### 検証その4：障害時のフェールオーバー（Bidirectional Forwarding Direction）
- vsrx側の BFD を設定

```ps1:edit_BFD_setting_on_vsrx
#初期のBFDを確認（BFDの設定はない状態）
aws@vsrx1> show bfd session extensive

0 sessions, 0 clients
Cumulative transmit rate 0.0 pps, cumulative receive rate 0.0 pps

#BFDを設定する（300ミリ秒間隔でBFDパケットを送信し、3度受け取れない場合は障害と判断する）
aws@vsrx1> configure exclusive
warning: uncommitted changes will be discarded on exit
Entering configuration mode
[edit]
aws@vsrx1# set protocols bgp group EBGP bfd-liveness-detection minimum-interval 300 multiplier 3

[edit]
aws@vsrx1# commit check
configuration check succeeds

[edit]
aws@vsrx1# commit and-quit
commit complete
Exiting configuration mode

#設定変更後のBFDを確認
aws@vsrx1> show bfd session extensive
                                                  Detect   Transmit
Address                  State     Interface      Time     Interval  Multiplier
169.254.100.1            Up        ge-0/0/0.0     0.900     0.300        3
 Client BGP, TX interval 0.300, RX interval 0.300
 Session up time 00:02:15
 Local diagnostic None, remote diagnostic None
 Remote state Up, version 1
 Min async interval 0.300, min slow interval 1.000
 Adaptive async TX interval 0.300, RX interval 0.300
 Local min TX interval 0.300, minimum RX interval 0.300, multiplier 3
 Remote min TX interval 0.300, min RX interval 0.300, multiplier 3
 Local discriminator 1, remote discriminator 65923
 Echo mode disabled/inactive
 Remote is control-plane independent

1 sessions, 1 clients
Cumulative transmit rate 3.3 pps, cumulative receive rate 3.3 pps

```

- csr側の BFD を設定

```ps1:edit_BFD_setting_on_csr
#初期のBFDを確認（BFDの設定はない状態）
csr#show bfd neighbors details
csr#

#BFDを設定する（300ミリ秒間隔でBFDパケットを送信し、3度受け取れない場合は障害と判断する）
csr#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
csr(config)#bfd slow-timers 10000
csr(config)#interface GigabitEthernet1.3218
csr(config-subif)#bfd interval 300 min_rx 300 multiplier 3
csr(config-subif)#router bgp 65001
csr(config-router)#neighbor 169.254.100.5 fall-over bfd
csr(config-router)#^Z
csr#

#設定変更後のBFDを確認
csr#show bfd neighbors details

IPv4 Sessions
NeighAddr                              LD/RD         RH/RS     State     Int
169.254.100.5                        4097/1592       Up        Up        Gi1.3218
Session state is UP and not using echo function.
Session Host: Software
OurAddr: 169.254.100.6
Handle: 1
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 300000, MinRxInt: 300000, Multiplier: 3
Received MinRxInt: 300000, Received Multiplier: 3
Holddown (hits): 899(0), Hello (hits): 300(1152)
Rx Count: 1014, Rx Interval (ms) min/max/avg: 297/646/300 last: 1 ms ago
Tx Count: 1154, Tx Interval (ms) min/max/avg: 3/646/263 last: 34 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: BGP CEF
Uptime: 00:05:04
Last packet: Version: 1                  - Diagnostic: 0
             State bit: Up               - Demand bit: 0
             Poll bit: 0                 - Final bit: 0
             C bit: 1
             Multiplier: 3               - Length: 24
             My Discr.: 1592             - Your Discr.: 4097
             Min tx interval: 300000     - Min rx interval: 300000
             Min Echo interval: 0

```
##### 検証その5：Route 53 Resolver for Hybrid Clouds
Route 53 Resolver Endpoint（Inbound）を作成してオンプレ側からの名前解決をしてみます。

1. Resolver Endpoint（Inbound）の作成
   ![R53-Endpoint01.PNG](https://qiita-image-store.s3.amazonaws.com/0/71534/512b06c5-333a-fa8d-bc9d-942d0d7c6978.png)
・endpointを作成するVPNとsubnetを選択
![R53-Endpoint02.PNG](https://qiita-image-store.s3.amazonaws.com/0/71534/17325809-ae43-a236-d0fe-98a93044ab18.png)
![R53-Endpoint03.PNG](https://qiita-image-store.s3.amazonaws.com/0/71534/f4e66212-fa63-3837-762d-99576790a512.png)
・vpc-dxtraning に inbound endpoint が作成されました
![R53-Endpoint04.PNG](https://qiita-image-store.s3.amazonaws.com/0/71534/ccc06d23-366d-ce4b-9d92-3a2906359963.png)

2. 作成された inbound endpoint のIPアドレスを確認
![R53-Endpoint05.PNG](https://qiita-image-store.s3.amazonaws.com/0/71534/8f8e8757-dba0-d387-c13c-115b5895cc4e.png)

3. オンプレ側のubuntuから名前解決してみます

```ps1:name_resolve_from_ubuntu_to_r53_inbound-gw
root@ubuntu:~# dig @172.16.0.231 aws.amazon.com → フルリゾルバの動作

; <<>> DiG 9.9.5-3ubuntu0.7-Ubuntu <<>> @172.16.0.231 aws.amazon.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52397
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;aws.amazon.com.                        IN      A

;; ANSWER SECTION:
aws.amazon.com.         60      IN      CNAME   1.aws-lbr.amazonaws.com.
1.aws-lbr.amazonaws.com. 60     IN      CNAME   aws-us-west-2.amazon.com.
aws-us-west-2.amazon.com. 60    IN      A       52.119.161.208

;; Query time: 236 msec
;; SERVER: 172.16.0.231#53(172.16.0.231)
;; WHEN: Sat Dec 29 13:16:55 JST 2018
;; MSG SIZE  rcvd: 110

root@ubuntu:~# dig @172.16.0.231 ip-172-16-0-100.ap-northeast-1.compute.internal　→　VPC サブネット内 EC2の名前解決

; <<>> DiG 9.9.5-3ubuntu0.7-Ubuntu <<>> @172.16.0.231 ip-172-16-0-100.ap-northeast-1.compute.internal
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21414
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ip-172-16-0-100.ap-northeast-1.compute.internal. IN A

;; ANSWER SECTION:
ip-172-16-0-100.ap-northeast-1.compute.internal. 60 IN A 172.16.0.100

;; Query time: 10 msec
;; SERVER: 172.16.0.231#53(172.16.0.231)
;; WHEN: Sat Dec 29 13:17:37 JST 2018
;; MSG SIZE  rcvd: 8

```
実際の環境では、オンプレ側のリカーシブに条件付きフォワーダ設定をします。

##### 検証その6：VSRX と CSR でvrrp を設定（失敗）
設定してみたのですが、結果はうまくいきませんでした。
私の設定間違いか multicast が通らないのか原因究明に至らなかったのですが、設定方法だけ備忘録として記載しています。

1. VSRXでvrrpの設定
    - 本設定の投入を実施するSSH接続が使用している側のインターフェース設定を変更するので、念のため commit confirmed 1 で設定を反映させる

```ps1:vrrp_setting_on_vsrx
aws@vsrx1> configure exclusive
warning: uncommitted changes will be discarded on exit
Entering configuration mode

[edit]
aws@vsrx1# show interfaces | display set
set interfaces ge-0/0/0 flexible-vlan-tagging
set interfaces ge-0/0/0 mtu 1522
set interfaces ge-0/0/0 unit 0 vlan-id 3018
set interfaces ge-0/0/0 unit 0 family inet mtu 1500
set interfaces ge-0/0/0 unit 0 family inet address 169.254.100.2/30
set interfaces ge-0/0/1 unit 0 family inet address 192.168.10.20/24 vrrp-group 1 virtual-address 192.168.10.25
set interfaces ge-0/0/1 unit 0 family inet address 192.168.10.20/24 vrrp-group 1 priority 150
set interfaces ge-0/0/1 unit 0 family inet address 192.168.10.20/24 vrrp-group 1 no-preempt
set interfaces ge-0/0/1 unit 0 family inet address 192.168.10.20/24 vrrp-group 1 accept-data
set interfaces ge-0/0/2 unit 0 family inet address 10.251.3.18/21
set interfaces lo0 unit 0 family inet address 10.0.0.255/32

set security zones security-zone trust interfaces ge-0/0/1.0 host-inbound-traffic protocols vrrp

#confirmed を1分に設定してcommit
aws@vsrx1# commit confirmed ?
Possible completions:
  <[Enter]>            Execute this command
  <timeout>            Number of minutes until automatic rollback (1..65535)
  and-quit             Quit configuration mode if commit succeeds
  comment              Message to write to commit log
  |                    Pipe through a command
[edit]
aws@vsrx1# commit confirmed 1

```
  
2. CSRでvrrpの設定


```ps1:vrrp_setting_on_csr
csr#sh running-config interface gigabitEthernet 2
Building configuration...

Current configuration : 191 bytes
!
interface GigabitEthernet2
 ip address 192.168.10.30 255.255.255.0
 no ip redirects
 negotiation auto
 vrrp 1 description "vrrp to Juniper"
 vrrp 1 ip 192.168.10.25
 no vrrp 1 preempt
end
```
3. 動作確認（vsrx と csr 両方ともmasterになっています）

```ps1:vrrp_status_on_vsrx
aws@vsrx1# run show vrrp detail
Physical interface: ge-0/0/1, Unit: 0, Address: 192.168.10.20/24
  Index: 70, SNMP ifIndex: 518, VRRP-Traps: disabled
  Interface state: up, Group: 1, State: master, VRRP Mode: Active
  Priority: 150, Advertisement interval: 1, Authentication type: none
  Advertisement threshold: 3, Computed send rate: 0
  Preempt: no, Accept-data mode: yes, VIP count: 1, VIP: 192.168.10.25
  Advertisement Timer: 0.803s, Master router: 192.168.10.20
  Virtual router uptime: 4d 14:41, Master router uptime: 4d 13:53
  Virtual Mac: 00:00:5e:00:01:01
  Tracking: disabled

[edit]
aws@vsrx1# exit
Exiting configuration mode

aws@vsrx1> show vrrp
Interface     State       Group   VR state VR Mode   Timer    Type   Address
ge-0/0/1.0    up              1   master   Active      A  0.070 lcl    192.168.10.20
                                                                vip    192.168.10.25
```

```ps1:vrrp_status_on_csr
csr#show vrrp
GigabitEthernet2 - Group 1
"vrrp to Juniper"
  State is Master
  Virtual IP address is 192.168.10.25
  Virtual MAC address is 0000.5e00.0101
  Advertisement interval is 1.000 sec
  Preemption disabled
  Priority is 100
  Master Router is 192.168.10.30 (local), priority is 100
  Master Advertisement interval is 1.000 sec
  Master Down interval is 3.609 sec

csr#show vrrp bri
csr#show vrrp brief
Interface          Grp Pri Time  Own Pre State   Master addr     Group addr
Gi2                1   100 3609          Master  192.168.10.30   192.168.10.25
```

####まとめ
オンプレミスからAWS VPC までDX経由で接続する場合は、APNさん提供の接続サービスを使用することが多かったのですが、今回のラボ参加で Shared Virtual Interfaces を許可して CGWでBGP設定する検証作業ができ、とても勉強になりました。

また、ラボ参加した当日からしばらくのあいだ環境を使わせていただけたので、R53 Resolver Endpoint の検証など行う事ができました。

DX接続を検討している方にとって、とても有意義なハンズオンになると思います。