SONiC/Leaf-spine testing

(moved from SONiC)
=== Leaf/spine configuration ===
We are going to use the diagram below to setup the leaf/spine. Each lead switch has 1 connection to each spine switch. There is no connection between the spines. The first setup is to identify the interfaces on each switch.
==== Interface mapping ====
show interfaces description
on S5232F-ON (Spine)
 Interface Oper Admin Alias Description
 ----------- ------ ------- ------- -------------
 Ethernet0 down down Eth1/1 N/A
 Ethernet4 down down Eth1/2 N/A
 Ethernet8 down down Eth1/3 N/A
 Ethernet12 down down Eth1/4 N/A
 Ethernet16 down down Eth1/5 N/A
 Ethernet20 down down Eth1/6 N/A
 -
 -
 Ethernet120 down down Eth1/31 N/A
 Ethernet124 down down Eth1/32 N/A
 Ethernet128 down down Eth1/33 N/A
 Ethernet129 down down Eth1/34 N/A
on both spines switches, we will use interfaces Ethernet120 on Sonic side which is interface E1/31 on Dell switch and Ethernet124 which is E1/32
On S5248F-ON (leaf)
leaf1# show interface description
 Interface Oper Admin Alias Description
 ----------- ------ ------- ------- -------------
 Ethernet0 down down Eth1/1 N/A
 Ethernet1 down down Eth1/2 N/A
 Ethernet2 down down Eth1/3 N/A
 Ethernet3 down down Eth1/4 N/A
 -
 -
 Ethernet72 down down Eth1/55 N/A
 Ethernet76 down down Eth1/56 N/A
On the leaf's we will use interfaces Ethernet72 and Ethernet76 which are E1/55 and E1/56 on the Dell switch
Now that we have the interface mapping from Sonic to Dell it is time to setup IP address on each of those interfaces. Below is the diagram showing the IP's and network that we will use for the leaf/spine setup.
==== Interface configuration ====
spine1
 sonic-cli
 configure terminal
 interface Ethernet120
 description dell-leaf2
 no shutdown
 ip address 172.16.1.9/30
 interface Ethernet124
 description dell-leaf1
 no shutdown
 ip address 172.16.1.1/30
 end 
 write memory
spine2
 sonic-cli
 configure terminal
 interface Ethernet120
 description dell-leaf2
 no shutdown
 ip address 172.16.1.14/30
 interface Ethernet124
 description dell-leaf1
 no shutdown
 ip address 172.16.1.6/30
 end 
 write memory
leaf1
 sonic-cli
 configure terminal
 interface Ethernet72
 description dell-spine2
 no shutdown
 ip address 172.16.1.6/30
 interface Ethernet76
 description dell-spine1
 no shutdown
 ip address 172.16.1.2/30
 leaf2
 sonic-cli
 configure terminal
 interface Ethernet72
 description dell-spine2
 no shutdown
 ip address 172.16.1.14/30
 interface Ethernet76
 description dell-spine1
 no shutdown
 ip address 172.16.1.10/30
Check connectivity between spine1 and leaf1
 dell-spine1# ping 172.16.1.2
 PING 172.16.1.2 (172.16.1.2) 56(84) bytes of data.
 64 bytes from 172.16.1.2: icmp_seq=1 ttl=64 time=0.348 ms
 64 bytes from 172.16.1.2: icmp_seq=2 ttl=64 time=0.274 ms
 64 bytes from 172.16.1.2: icmp_seq=3 ttl=64 time=0.268 ms
 dell-leaf1# ping 172.16.1.1
 PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data.
 64 bytes from 172.16.1.1: icmp_seq=1 ttl=64 time=0.339 ms
 64 bytes from 172.16.1.1: icmp_seq=2 ttl=64 time=0.303 ms
 dell-spine2# ping 172.16.1.14
 PING 172.16.1.14 (172.16.1.14) 56(84) bytes of data.
 64 bytes from 172.16.1.14: icmp_seq=1 ttl=64 time=0.356 ms
 64 bytes from 172.16.1.14: icmp_seq=2 ttl=64 time=0.296 ms 
Now that we have connectivity, the next step is to setup BGP
==== BGP configuration ====
spine1
 sonic-cli
 configure terminal
 router bgp 65030
 router-id 10.0.1.13
 address-family ipv4 unicast
 redistribute connected
 neighbor 172.16.1.10
 remote-as 65033
 address-family ipv4 unicast
 activate
 neighbor 172.16.1.2
 remote-as 65032
 address-family ipv4 unicast
 activate
 end
 write memory
spine2
 sonic-cli
 configure terminal
 router bgp 65030
 router-id 10.0.1.14
 address-family ipv4 unicast
 redistribute connected
 neighbor 172.16.1.14
 remote-as 65033
 address-family ipv4 unicast
 activate
 neighbor 172.16.1.6
 remote-as 65032
 address-family ipv4 unicast
 activate 
 end
 write memory
leaf 1
 router bgp 65032
 router-id 10.0.1.24
 address-family ipv4 unicast
 redistribute connected
 neighbor 172.16.1.1
 remote-as 65030
 address-family ipv4 unicast
 activate
 neighbor 172.16.1.5
 remote-as 65030
 address-family ipv4 unicast
 activate
 end
 write memory
leaf 2
 router bgp 65033
 router-id 10.0.1.25
 address-family ipv4 unicast
 redistribute connected
 neighbor 172.16.1.9
 remote-as 65030
 address-family ipv4 unicast
 activate
 neighbor 172.16.1.13
 remote-as 65030
 address-family ipv4 unicast
 activate
 end
 write memory
verification
show bgp ipv4 unicast summary
spine1
 BGP router identifier 10.0.1.13, local AS number 65030
 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd
 172.16.1.10 4 65033 142 141 0 0 02:13:29 3
 172.16.1.2 4 65032 152 163 0 0 02:23:07 3
 Total number of neighbors 2
 Total number of neighbors established 2
spine2
 BGP router identifier 10.0.1.14, local AS number 65030
 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd
 172.16.1.14 4 65033 145 144 0 0 02:16:31 3
 172.16.1.6 4 65032 160 159 0 0 02:31:48 3
 Total number of neighbors 2
 Total number of neighbors established 2
leaf1
 BGP router identifier 10.0.1.24, local AS number 65032
 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd
 172.16.1.1 4 65030 156 159 0 0 02:27:13 5
 172.16.1.5 4 65030 161 164 0 0 02:32:49 5
 Total number of neighbors 2
 Total number of neighbors established 2
leaf2
 BGP router identifier 10.0.1.25, local AS number 65033
 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd
 172.16.1.13 4 65030 151 154 0 0 02:22:01 5
 172.16.1.9 4 65030 151 154 0 0 02:22:04 5
 Total number of neighbors 2
 Total number of neighbors established 2
=== EVPN/VXLAN Configuration ===
==== Spine ====
On both spine's the only thing needed to be configured is EVPN and the configuration is the same some both.
Spine1
 router bgp 65030
 !
 address-family l2vpn evpn
 !
 neighbor 172.16.1.10
 !
 address-family l2vpn evpn
 activate
 !
 neighbor 172.16.1.2
 !
 address-family l2vpn evpn
 activate
Verification
 dell-spine1# show bgp l2vpn evpn summary
 BGP router identifier 10.0.1.13, local AS number 65030
 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd
 172.16.1.10 4 65033 9612 9605 0 0 00:00:35 0
 172.16.1.2 4 65032 9627 9622 0 0 00:00:33 0
Spine2
 router bgp 65030
 !
 address-family l2vpn evpn
 !
 neighbor 172.16.1.14
 !
 address-family l2vpn evpn
 activate
 !
 neighbor 172.16.1.6
 !
 address-family l2vpn evpn
 activate
Verification
 dell-spine2# show bgp l2vpn evpn summary
 BGP router identifier 10.0.1.14, local AS number 65030
 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd
 172.16.1.14 4 65033 9612 9605 0 0 00:00:35 0
 172.16.1.6 4 65032 9627 9622 0 0 00:00:33 0
==== Leaf ====
leaf1
 interface Loopback 1
 description "VXLAN source IP"
 ip address 10.10.10.1/32
 interface vxlan vtep1
 source-ip 10.10.10.1
 router bgp 65032
 address-family l2vpn evpn
 advertise-all-vni
 !
 neighbor 172.16.1.1
 !
 address-family l2vpn evpn
 activate
 !
 neighbor 172.16.1.5
 !
 address-family l2vpn evpn
 activate
 ip vrf Vrf_codfw
 ip anycast-mac-address 00:00:00:10:10:10
 
 router bgp 65032 vrf Vrf_codfw
 router-id 10.0.1.24
 !
 address-family ipv4 unicast
 redistribute connected
 !
 address-family l2vpn evpn
 advertise ipv4 unicast
 !
 interface Vlan4000
 description "IRB VLAN"
 ip vrf forwarding Vrf_codfw
 ipv6 enable
 !
 interface Vlan2004
 description private1-e-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.64.254/22
 !
 interface vxlan vtep1
 map vni 102004 vlan 2004
 map vni 404000 vlan 4000
 map vni 404000 vrf Vrf_codfw
 !
 interface Ethernet0
 description test_srv1
 no shutdown
 switchport access Vlan 2004
Verification
 dell-leaf1# show vxlan tunnel
 Name SIP DIP source operstatus
 ======= ====== ====== ====== ========
The vxlan tunnel is not formed yet, because the switch hasn't learned any routes from the end devices (servers)
 dell-leaf1# show vxlan vlanvnimap
 VLAN VNI
 ====== =====
 Vlan2004 102004
 Vlan4000 404000
 Total count : 2
 
 dell-leaf1# show vxlan vrfvnimap
 VRF VNI
 ====== =====
 Vrf_codfw 404000
 Total count : 1
leaf 2
 interface Loopback 1
 description "VXLAN source IP"
 ip address 10.10.10.2/32
 interface vxlan vtep1
 source-ip 10.10.10.2
 router bgp 65033
 address-family l2vpn evpn
 advertise-all-vni
 !
 neighbor 172.16.1.9
 !
 address-family l2vpn evpn
 activate
 !
 neighbor 172.16.1.13
 !
 address-family l2vpn evpn
 activate
 ip vrf Vrf_codfw
 ip anycast-mac-address 00:00:00:10:10:10
 
 router bgp 65033 vrf Vrf_codfw
 router-id 10.0.1.25
 !
 address-family ipv4 unicast
 redistribute connected
 !
 address-family l2vpn evpn
 advertise ipv4 unicast
 !
 interface Vlan4000
 description "IRB VLAN"
 ip vrf forwarding Vrf_codfw
 ipv6 enable
 !
 interface Vlan2004
 description private1-e-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.64.254/22
 !
 interface vxlan vtep1
 map vni 102004 vlan 2004
 map vni 404000 vlan 4000
 map vni 404000 vrf Vrf_codfw
 !
 interface Ethernet0
 description test_srv2
 no shutdown
 switchport access Vlan 2004
Verification
 dell-leaf1# show vxlan tunnel
 Name SIP DIP source operstatus
 ======= ====== ====== ====== ========
The vxlan tunnel is not formed yet, because the switch hasn't learned any routes from the end devices (servers)
 dell-leaf2# show vxlan vlanvnimap
 VLAN VNI
 ====== =====
 Vlan2004 102004
 Vlan4000 404000
 Total count : 2
 
 dell-leaf1# show vxlan vrfvnimap
 VRF VNI
 ====== =====
 Vrf_codfw 404000
 Total count : 1
== End devices connectivity ==
I setup up 2 servers, one with IP address 10.192.64.10/22 connected to leaf 1 and the second with IP address 10.192.64.20/22 connected to leaf2. After running a ping test from server one to server 2. I checked the vxlan tunnel on both leaf's switches.
 dell-leaf1# show vxlan tunnel
 Name SIP DIP source operstatus
 ======= ====== ====== ====== ========
 EVPN_10.10.10.2 10.10.10.1 10.10.10.2 EVPN oper_up
 
 dell-leaf2# show vxlan tunnel
 Name SIP DIP source operstatus
 ======= ====== ====== ====== ========
 EVPN_10.10.10.1 10.10.10.2 10.10.10.1 EVPN oper_up
=== Some routes informations ===
type 5 routes 
 dell-leaf2# show bgp l2vpn evpn route type prefix
 BGP table version is 19, local router ID is 10.0.1.25
 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
 Origin codes: i - IGP, e - EGP, ? - incomplete
 EVPN type-1 prefix: [1]:[ESI]:[EthTag]
 EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
 EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
 EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
 EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
 Network Next Hop Metric LocPrf Weight Path
 Extended Community
 Route Distinguisher: 10.0.1.24:5096
 * [5]:[0]:[22]:[10.192.64.0]
 10.10.10.1 0 65030 65032 ?
 RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
 *> [5]:[0]:[22]:[10.192.64.0]
 10.10.10.1 0 65030 65032 ?
 RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
 Route Distinguisher: 10.0.1.25:5096
 *> [5]:[0]:[22]:[10.192.64.0]
 10.10.10.2 0 32768 ?
 ET:8 RT:65033:40400 Rmac:3c:2c:30:4c:81:83
 Displayed 2 prefixes (3 paths) (of requested type)