SONiC/VXLAN-EVPN Network Testing - Sonic on Dell switches

This page details the individual tests carried out to validate the Dell hardware / Sonic OS platform. For further information please see the Dell Enterprise Sonic Evaluation write up.
= Test Topology =
This test was done on 4 Dell switches. Two S5232F devices acting as 'Spines' as two S5248F's operating as Leaf/edge switches. All are based on the Broadcom Trident 3 ASIC and ran Dell SONiC-OS-3.4.1-Enterprise_Base. We have 1 server connected to leaf1 on port Ethernet 0 and the other server connected to leaf 2 on port Eth1/1.
= Physical Tests =
== Transceiver Compatibility ==
=== FS.com 40GBase-PSM4 QSFP+ ===
 dell-spine1# show interface transceiver Ethernet 4
 
 Ethernet4
 ---------------------------------------------------------------------
 Attribute : Value/State
 ---------------------------------------------------------------------
 connector-type : MPO
 date-code : 2020-08-13
 form-factor : QSFP+
 cable-length(m) : 0
 display-name : QSFP+ 40GBASE-PSM4
 max-module-power(Watts) : 3.5
 max-port-power(Watts) : 5
 vendor-oui : 44-7C-7F
 present : PRESENT
 serial-no : F1930364698
 vendor : FS
 vendor-part : QSFP-PLR4-40G
 vendor-rev : 1B
=== Finisar 10GBase-LR SFP+ ===
 dell-leaf2# show interface transceiver Ethernet 0
 
 Ethernet0
 ---------------------------------------------------------------------
 Attribute : Value/State
 ---------------------------------------------------------------------
 connector-type : LC
 date-code : 2014-07-19
 form-factor : SFP+
 cable-length(m) : 0
 display-name : SFP+ 10GBASE-LR
 max-module-power(Watts) : 2
 max-port-power(Watts) : 2.5
 vendor-oui : 00-90-65
 present : PRESENT
 serial-no : AS32MX4
 vendor : FINISAR CORP.
 vendor-part : FTLX1471D3BCL
 vendor-rev : A
=== FS.com 10GBase-LR SFP+ ===
 dell-leaf2# show interface transceiver Ethernet 1
 
 Ethernet1
 ---------------------------------------------------------------------
 Attribute : Value/State
 ---------------------------------------------------------------------
 connector-type : LC
 date-code : 2018-05-04
 form-factor : SFP+
 cable-length(m) : 0
 display-name : SFP+ 10GBASE-LR
 max-module-power(Watts) : 2
 max-port-power(Watts) : 2.5
 vendor-oui : 00-90-65
 present : PRESENT
 serial-no : G1804036447
 vendor : FiberStore
 vendor-part : SFP-10GLR-31
 vendor-rev : A
=== Juniper 10G 3m DAC 10GBASE-CR-DAC-3.0M ===
 dell-leaf2# show interface transceiver Ethernet 2
 
 Ethernet2
 ---------------------------------------------------------------------
 Attribute : Value/State
 ---------------------------------------------------------------------
 date-code : 2011-11-29
 form-factor : SFP+
 cable-length(m) : 3
 display-name : SFP+ 10GBASE-CR-DAC-3.0M
 max-module-power(Watts) : 2
 max-port-power(Watts) : 2.5
 vendor-oui : 41-50-48
 present : PRESENT
 serial-no : APF11460020W1Y
 vendor : Amphenol
 vendor-part : 584990002
 vendor-rev : A
=== FS.com 10G 3m DAC ===
 dell-leaf2# show interface transceiver Ethernet 4
 
 Ethernet4
 ---------------------------------------------------------------------
 Attribute : Value/State
 ---------------------------------------------------------------------
 date-code : 2019-06-12
 form-factor : SFP+
 cable-length(m) : 3
 display-name : SFP+ 10GBASE-CR-DAC-3.0M
 max-module-power(Watts) : 2
 max-port-power(Watts) : 2.5
 vendor-oui : 00-00-00
 present : PRESENT
 serial-no : G1906269417-1
 vendor : FS
 vendor-part : SFPP-PC03
 vendor-rev : A
=== Juniper 40GBase-SR4 QSFP+ ===
 dell-spine1# show interface transceiver Ethernet 12
 
 Ethernet12
 ---------------------------------------------------------------------
 Attribute : Value/State
 ---------------------------------------------------------------------
 connector-type : MPO
 date-code : 2013-12-04
 form-factor : QSFP+
 cable-length(m) : 0
 display-name : QSFP+ 40GBASE-SR4
 max-module-power(Watts) : 1.5
 max-port-power(Watts) : 5
 vendor-oui : 00-17-6A
 present : PRESENT
 serial-no : QD491487
 vendor : AVAGO
 vendor-part : AFBR-79EQDZ-JU1
 vendor-rev : 01
=== FS.com SFP 1000BASE-T ===
The switch is able to see the transceiver but no way to test since the switch doesn't allow to set the interface to 1G.
 Ethernet32 SFP 1000BASE-T 1.5 | 2.5 Fiberstore G1804219349 SFP-GB-GE-T N/A N/A Ready
To set the interface to 1G, you need to change the portgroup first from 25G to 10G, then change the port speed to 1G:
 sudo config portgroup speed 9 10000
 dell-leaf1(conf-if-Ethernet32)# speed 1000
 interface Ethernet32
 description test_srv1_1GTBase
 mtu 9100
 speed 1000
 fec none
 no shutdown
 dell-leaf1# show interface Ethernet 32
 
 Ethernet32 is up, line protocol is up
 Hardware is Eth
 Description: test_srv1_1GTBase
 Mode of IPV4 address assignment: not-set
 Mode of IPV6 address assignment: not-set
 Interface IPv6 oper status: Disabled
 IP MTU 9100 bytes
 LineSpeed 1GB, Auto-negotiation off
 FEC: DISABLED
 Last clearing of "show interface" counters: never
 10 seconds input rate 0 packets/sec, 0 bits/sec, 0 Bytes/sec
 10 seconds output rate 0 packets/sec, 0 bits/sec, 0 Bytes/sec
 Input statistics:
 28 packets, 2072 octets
 28 Multicasts, 0 Broadcasts, 0 Unicasts
 0 error, 28 discarded, 0 Oversize
 0 Packets (128 to 255 Octects)
 Output statistics:
 2379 packets, 620868 octets
 2379 Multicasts, 0 Broadcasts, 0 Unicasts
 0 error, 0 discarded, 0 Oversize
 dell-leaf1# show mac address-table interface Ethernet 32
 -----------------------------------------------------------
 VLAN MAC-ADDRESS TYPE INTERFACE 
 -----------------------------------------------------------
 2004 D0:94:66:86:4D:6C DYNAMIC Ethernet32
== 'Breakout' Transceivers ==
=== 4x10G QSFP+ in 'Breakout' Mode ===
The switch is able to see the break out cable on both ends. the version used is PSM4
 Ethernet0 QSFP+ 40GBASE-PSM4 3.5 | 5.0 FS F1930364699 QSFP-PLR4-40G N/A N/A Ready
 Ethernet16 SFP+ 10GBASE-LR 2.0 | 2.5 FINISAR CORP. AP40JAN FTLX1471D3BCL N/A N/A Ready
Command for break out 
 sudo config interface breakout Ethernet0 '4x10G'
 
 After running Logic to limit the impact
 Final list of ports to be deleted :
 {
 "Ethernet0": "100000"
 }
 Final list of ports to be added :
 {
 "Ethernet2": "10000",
 "Ethernet3": "10000",
 "Ethernet0": "10000",
 "Ethernet1": "10000"
 }
 
 dell-spine1# show interface breakout
 -----------------------------------------------
 Port Breakout Mode Status Interfaces
 -----------------------------------------------
 1/1 4x10G Completed Eth1/1/1
 Eth1/1/2
 Eth1/1/3
 Eth1/1/4
=== 4x25G QSFP28 in 'Brekaout' Mode ===
Not sure we really need this, be good to know if it's an option. I believe it can be done with 100GBase-PSM4 -> 4x25GBase-LR4:
https://community.fs.com/blog/brief-introduction-to-qsfp-100g-psm4-optical-transceiver.html
100GBase-SR4 should also work over MMF to 4x25GBase-SR
Or otherwise a DAC/AOC:
https://www.fs.com/de-en/products/116289.html
https://www.fs.com/products/70439.html
== Transceiver DOM Supprt ==
Digital Optical Monitoring is the standard that transceiver modules use to send statistics to the element they are plugged into (i.e. the switch).
We need to validate that we can see this data on the Dell switches, particularly the Rx light level which is important for longer optical links.
=== SFP+ / SFP28 ===
Works ok:
 dell-leaf1# show interface transceiver dom Ethernet 0
 
 -----------------------------------------------------------------------
 Ethernet0 
 -----------------------------------------------------------------------
 Identifier: SFP
 Vendor Name: FINISAR CORP.
 Vendor Part: FTLX1471D3BCL
 ChannelMonitorValues:
 Rx1Power: -2.0094 dBm
 Tx1Bias: 41.5540 mA
 Tx1Power: -1.5009 dBm
 ChannelThresholdValues:
 RxPowerHighAlarm : 2.5001 dBm
 RxPowerHighWarning: 2.0000 dBm
 RxPowerLowAlarm : -20.0000 dBm
 RxPowerLowWarning : -18.0134 dBm
 TxBiasHighAlarm : 85.0000 mA
 TxBiasHighWarning : 80.0000 mA
 TxBiasLowAlarm : 15.0000 mA
 TxBiasLowWarning : 20.0000 mA
 TxPowerHighAlarm : 2.0000 dBm
 TxPowerHighWarning: 0.9999 dBm
 TxPowerLowAlarm : -7.9997 dBm
 TxPowerLowWarning : -7.0006 dBm
 ModuleMonitorValues:
 Temperature: 36.6055 C
 Vcc: 3.3757 Volts
 ModuleThresholdValues:
 TempHighAlarm : 78.0000 C
 TempHighWarning: 73.0000 C
 TempLowAlarm : -13.0000 C
 TempLowWarning : -8.0000 C
 VccHighAlarm : 3.7000 Volts
 VccHighWarning : 3.6000 Volts
 VccLowAlarm : 2.9000 Volts
 VccLowWarning : 3.0000 Volts
=== QSFP+ / QSFP28 ===
 dell-spine1# show interface transceiver dom Eth1/1/1
 
 -----------------------------------------------------------------------
 Eth1/1/1 
 -----------------------------------------------------------------------
 Identifier: QSFP+
 Vendor Name: FS
 Vendor Part: QSFP-PLR4-40G
 ChannelMonitorValues:
 Rx1Power: -1.7289 dBm
 Rx2Power: -2.6898 dBm
 Rx3Power: -2.3980 dBm
 Rx4Power: -inf dBm
(note above is in breakout mode with only 3 ends connected, so makes sense Rx4Power is zero).
= Functional Tests =
== L2 Access port ==
 dell-leaf1# show running-configuration interface Ethernet 0
 interface Ethernet0
 description test_srv1
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport access Vlan 2004
 
 dell-leaf1# show running-configuration interface Eth1/1
 interface Eth1/1
 description test_Srv2
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport access Vlan 2005
 interface Vlan2004
 description private1-e-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.64.254/22
 interface Vlan2005
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.80.254/22
Test server 1 in vlan 2004
 ppaul@srv1:~$ ip -br link show enp59s0f0
 enp59s0f0 UP 40:a8:f0:2c:83:10 <BROADCAST,MULTICAST,UP,LOWER_UP>
Mac is learnt on leaf 1 where test srv1 is connected to port 0, and shown in local Vlan table:
 dell-leaf1# show mac address-table Vlan 2004
 -----------------------------------------------------------
 VLAN MAC-ADDRESS TYPE INTERFACE 
 -----------------------------------------------------------
 2004 40:A8:F0:2C:83:10 DYNAMIC Ethernet0 
As well as being associated with the attached EVPN VNI:
 dell-leaf1# show evpn mac vni 102004
 Number of MACs (local and remote) known for this VNI: 2
 MAC Type Intf/Remote VTEP VLAN Seq #'s
 00:00:00:10:10:10 local Vlan2004 2004 0/0
 40:a8:f0:2c:83:10 local Ethernet0 2004 0/0
== L2 Trunk port ==
Interface config:
 dell-leaf1# show running-configuration interface Ethernet 0
 !
 interface Ethernet0
 description test_srv1
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport trunk allowed Vlan 2004-2005
With the server configured with an 802.1q sub-interface for each Vlan MACs were learnt correctly on the switch:
 
 dell-leaf1# show mac address-table interface Ethernet 0
 -----------------------------------------------------------
 VLAN MAC-ADDRESS TYPE INTERFACE 
 -----------------------------------------------------------
 2004 40:A8:F0:2C:83:10 DYNAMIC Ethernet0 
 2005 40:A8:F0:2C:83:10 DYNAMIC Ethernet0
== Routing to end host from Vlan interface in VRF ==
=== IPv4 ===
Server connected via L2 access/trunk port. Switch config:
 interface Vlan2004
 description private1-e-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.64.254/22
 ipv6 anycast-address 2620:0000:0861:011c::254/64
 
10.192.64.10 is the ip address of test server 1 if we want to ping this test server from leaf 1 we get:
 dell-leaf1# ping vrf Vrf_codfw 10.192.64.10
 ping: Warning: source address might be selected on device other than Vrf_codfw.
 PING 10.192.64.10 (10.192.64.10) from 10.192.64.254 Vrf_codfw: 56(84) bytes of data.
 64 bytes from 10.192.64.10: icmp_seq=1 ttl=64 time=0.312 ms
 64 bytes from 10.192.64.10: icmp_seq=2 ttl=64 time=0.302 ms
 ^C
 --- 10.192.64.10 ping statistics ---
 2 packets transmitted, 2 received, 0% packet loss, time 1006ms
=== IPv6 ===
The server is configured with IPv6 global unicast address 2620:0000:0861:011c::10, which we also get a response from:
 dell-leaf1# ping vrf Vrf_codfw 2620:0000:0861:011c::10
 ping6: Warning: source address might be selected on device other than Vrf_codfw.
 PING 2620:0000:0861:011c::10(2620:0:861:11c::10) from 2620:0:861:11c::254 Vrf_codfw: 56 data bytes
 64 bytes from 2620:0:861:11c::10: icmp_seq=1 ttl=64 time=0.385 ms
 64 bytes from 2620:0:861:11c::10: icmp_seq=2 ttl=64 time=0.305 ms
== Routed port in VRF ==
Config added as follows to switch port:
 interface Ethernet2
 description srv1_nic2
 mtu 9100
 speed 10000
 fec none
 no shutdown
 ip vrf forwarding Vrf_codfw
 ip address 192.0.2.8/31
 ipv6 address 2620:0000:0861:011e::1/64
Server port IPs were set as follows:
 root@srv1:~# ip -br addr show dev enp59s0f1
 enp59s0f1 UP 192.0.2.9/31 2620:0:861:11e::2/64 fe80::42a8:f0ff:fe2c:8314/64 
===IPv4===
IPv4 pingable both sides:
 dell-leaf1# ping vrf Vrf_codfw -I 192.0.2.8 192.0.2.9
 PING 192.0.2.9 (192.0.2.9) from 192.0.2.8 Vrf_codfw: 56(84) bytes of data.
 64 bytes from 192.0.2.9: icmp_seq=1 ttl=64 time=0.265 ms
 64 bytes from 192.0.2.9: icmp_seq=2 ttl=64 time=0.250 ms
 root@srv1:~# ping 192.0.2.8
 PING 192.0.2.8 (192.0.2.8) 56(84) bytes of data.
 64 bytes from 192.0.2.8: icmp_seq=1 ttl=64 time=0.171 ms
 64 bytes from 192.0.2.8: icmp_seq=2 ttl=64 time=0.185 ms
 
===IPv6===
Same with IPv6:
 dell-leaf1# ping vrf Vrf_codfw -I 2620:0000:0861:011e::1 2620:0000:0861:011e::2
 PING 2620:0000:0861:011e::2(2620:0:861:11e::2) from 2620:0:861:11e::1 Vrf_codfw: 56 data bytes
 64 bytes from 2620:0:861:11e::2: icmp_seq=1 ttl=64 time=0.312 ms
 64 bytes from 2620:0:861:11e::2: icmp_seq=2 ttl=64 time=0.247 ms
 root@srv1:~# ping 2620:0000:0861:011e::1
 PING 2620:0000:0861:011e::1(2620:0:861:11e::1) 56 data bytes
 64 bytes from 2620:0:861:11e::1: icmp_seq=1 ttl=64 time=0.209 ms
 64 bytes from 2620:0:861:11e::1: icmp_seq=2 ttl=64 time=0.214 ms
== Jumbo frames across L2 Vlan ==
===Same Switch===
===Remote via VXLAN===
== Jumbo frames L3 routing ==
===Same Switch===
Switch port had MTU set to 9100, Vlan interface did not have a specific MTU configured but it defaults to 9100:
 interface Ethernet0
 description test_srv1
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport trunk allowed Vlan 2004-2005
 dell-leaf1# show running-configuration interface Vlan 2004
 !
 interface Vlan2004
 description private1-e-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.64.254/22
 ipv6 anycast-address 2620:0000:0861:011c::254/64
 dell-leaf1# show interface Vlan 2004
 
 Vlan2004 is up, line protocol is up
 Description: private1-e-codfw
 Mode of IPV4 address assignment: not-set
 Mode of IPV6 address assignment: not-set
 Interface IPv6 oper status: Disabled
 IP MTU 9100 bytes
Server also had MTU set to 9100:
 root@srv1:~# ip -4 -d addr show enp59s0f0.2004
 6: enp59s0f0.2004@enp59s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP group default qlen 1000
 link/ether 40:a8:f0:2c:83:10 brd ff:ff:ff:ff:ff:ff promiscuity 0 
 vlan protocol 802.1Q id 2004 <REORDER_HDR> numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
 inet 10.192.64.10/22 scope global enp59s0f0.2004
 valid_lft forever preferred_lft forever
With this config we can ping the Vlan interface on the switch with a 9000 byte packet and DF bit set:
 root@srv1:~# ping -s 9000 -M do 10.192.64.254
 PING 10.192.64.254 (10.192.64.254) 9000(9028) bytes of data.
 9008 bytes from 10.192.64.254: icmp_seq=1 ttl=64 time=0.244 ms
 9008 bytes from 10.192.64.254: icmp_seq=2 ttl=64 time=0.247 ms
 9008 bytes from 10.192.64.254: icmp_seq=3 ttl=64 time=0.260 ms
 9008 bytes from 10.192.64.254: icmp_seq=4 ttl=64 time=0.265 ms
===Remote via VXLAN===
== Remote MAC Learning on Vlan/VNI ==
MAC 40:A8:F0:2C:83:10 is being learnt from srv1 connected to leaf1 on Vlan2004.
Logging on to leaf2 we can see it is receiving an EVPN type 2 route corresponding to this MAC:
 dell-leaf2# show bgp l2vpn evpn route detail 
 BGP routing table entry for 10.0.1.24:2004:[2]:[0]:[48]:[40:a8:f0:2c:83:10]:[32]:[10.192.64.10]
 Paths: (2 available, best #1)
 Advertised to non peer-group peers:
 172.16.1.9 172.16.1.13
 Route [2]:[0]:[48]:[40:a8:f0:2c:83:10]:[32]:[10.192.64.10] VNI 102004/404000
 65030 65032
 10.10.10.1 from 172.16.1.9 (10.0.1.13)
 Origin IGP, valid, external, best (Router ID)
 Extended Community: RT:65032:102004 RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
 Last update: Fri Apr 1 11:27:40 2022
 Route [2]:[0]:[48]:[40:a8:f0:2c:83:10]:[32]:[10.192.64.10] VNI 102004/404000
 65030 65032
 10.10.10.1 from 172.16.1.13 (10.0.1.14)
 Origin IGP, valid, external
 Extended Community: RT:65032:102004 RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
 Last update: Fri Apr 1 11:27:40 2022
 
This is processed correctly and added to the local EVPN database:
 
 dell-leaf2# show evpn mac vni 102004 mac 40:A8:F0:2C:83:10
 MAC: 40:a8:f0:2c:83:10
 Remote VTEP: 10.10.10.1
 Local Seq: 0 Remote Seq: 0
 Kernel Add: Success, Add ReAttempt:0
 Neighbors:
 10.192.64.10 Active
 2620:0:861:11c::10 Active
 fe80::42a8:f0ff:fe2c:8310 Active
The MAC is also visible in the local L2 forwarding table for Vlan 2004, the "interface" it was "learnt" on shows as VxLAN with the IP of the remote VTEP as expected:
 dell-leaf2# show mac address-table Vlan 2004
 -----------------------------------------------------------
 VLAN MAC-ADDRESS TYPE INTERFACE 
 -----------------------------------------------------------
 2004 40:A8:F0:2C:83:10 DYNAMIC VxLAN DIP: 10.10.10.1
== Client to Client L2 Unicast forwarding ==
=== Same Switch (Pure L2) ===
Ethernet0 and Ethernet2 on Leaf1 were configured as trunks allowing the same Vlans:
 
 interface Ethernet0
 description test_srv1
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport trunk allowed Vlan 2004-2005
 interface Ethernet2
 description srv1_nic2
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport trunk allowed Vlan 2004-2005
Both of these ports were connected to the same server, so in order to make it work the second sub-interface was added to a different Linux network namespace:
 root@srv1:~# ip -br link show enp59s0f0.2004
 enp59s0f0.2004@enp59s0f0 UP 40:a8:f0:2c:83:10 <BROADCAST,MULTICAST,UP,LOWER_UP>
 root@srv1:~# ip -br addr show enp59s0f0.2004
 enp59s0f0.2004@enp59s0f0 UP 10.192.64.10/24 2620:0:861:11c::10/64 fe80::42a8:f0ff:fe2c:8310/64
 root@srv1:~# ip route get 10.192.64.30
 10.192.64.30 dev enp59s0f0.2004 src 10.192.64.10 uid 0
 root@srv1:~# ip netns exec TESTNS ip -br link show dev enp59s0f1.2004
 enp59s0f1.2004@if5 UP 40:a8:f0:2c:83:14 <BROADCAST,MULTICAST,UP,LOWER_UP>
 root@srv1:~# ip netns exec TESTNS ip -br addr show dev enp59s0f1.2004
 enp59s0f1.2004@if5 UP 10.192.64.30/24 fe80::42a8:f0ff:fe2c:8314/64
 root@srv1:~# ip netns exec TESTNS ip route get 10.192.64.20
 10.192.64.20 dev enp59s0f1.2004 src 10.192.64.30 uid 0
This use of separate namespaces on the system isolates the two ports, so the "default" namespace, with a port connected to Ethernet0, does not know about the interface in the other one, which is connected to Ethernet2. As such when we send frames from the default namespace for the IP configured on the TESTNS port it will send it out to switch port Ethernet 0, and the switch should forward it via Ethernet 2, looping back to the same server. We only had a single server so this simulates having two separate boxes and sending traffic between them. Results were as expected.
Both MACs are properly learnt on the switch as expected:
 dell-leaf1# show mac address-table Vlan 2004
 -----------------------------------------------------------
 VLAN MAC-ADDRESS TYPE INTERFACE 
 -----------------------------------------------------------
 2004 40:A8:F0:2C:83:10 DYNAMIC Ethernet0 
 2004 40:A8:F0:2C:83:14 DYNAMIC Ethernet2
ARP works and pings flow successfully via the switch:
 root@srv1:~# ip neigh show 10.192.64.30
 10.192.64.30 dev enp59s0f0.2004 lladdr 40:a8:f0:2c:83:14 STALE
 root@srv1:~# ip netns exec TESTNS ip neigh show 10.192.64.10
 10.192.64.10 dev enp59s0f1.2004 lladdr 40:a8:f0:2c:83:10 STALE
 root@srv1:~# ping 10.192.64.30
 PING 10.192.64.30 (10.192.64.30) 56(84) bytes of data.
 64 bytes from 10.192.64.30: icmp_seq=1 ttl=64 time=0.167 ms
 64 bytes from 10.192.64.30: icmp_seq=2 ttl=64 time=0.136 ms
 
 root@srv1:~# ip netns exec TESTNS ping -c 2 10.192.64.10
 PING 10.192.64.10 (10.192.64.10) 56(84) bytes of data.
 64 bytes from 10.192.64.10: icmp_seq=1 ttl=64 time=0.138 ms
 64 bytes from 10.192.64.10: icmp_seq=2 ttl=64 time=0.135 ms
TCPDUMP shows the traffic coming in externally with MACs as expected:
 root@srv1:~# tcpdump -e -i enp59s0f0.2004 -l -p -nn icmp
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on enp59s0f0.2004, link-type EN10MB (Ethernet), capture size 262144 bytes
 13:12:24.539625 40:a8:f0:2c:83:14 > 40:a8:f0:2c:83:10, ethertype IPv4 (0x0800), length 98: 10.192.64.30 > 10.192.64.10: ICMP echo request, id 6059, seq 1, length 64
 13:12:24.539655 40:a8:f0:2c:83:10 > 40:a8:f0:2c:83:14, ethertype IPv4 (0x0800), length 98: 10.192.64.10 > 10.192.64.30: ICMP echo reply, id 6059, seq 1, length 64
 13:12:25.545664 40:a8:f0:2c:83:14 > 40:a8:f0:2c:83:10, ethertype IPv4 (0x0800), length 98: 10.192.64.30 > 10.192.64.10: ICMP echo request, id 6059, seq 2, length 64
 13:12:25.545680 40:a8:f0:2c:83:10 > 40:a8:f0:2c:83:14, ethertype IPv4 (0x0800), length 98: 10.192.64.10 > 10.192.64.30: ICMP echo reply, id 6059, seq 2, length 64
=== Remote Switch (VXLAN tunneled) ===
Two servers were connected, SRV1 to Leaf1 Ethernet0, and SRV2 to Leaf2 Eth1/1. These ports were configured as follows:
SRV1 was connected to Vlan2005 on Leaf1 with 802.1q trunk encap:
 interface Ethernet0
 description test_srv1
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport trunk allowed Vlan 2004-2005
 root@srv1:~# ip -br link show enp59s0f0.2005
 enp59s0f0.2005@enp59s0f0 UP 40:a8:f0:2c:83:10 <BROADCAST,MULTICAST,UP,LOWER_UP>
The MAC is learnt locally on Leaf1 as expected:
 dell-leaf1# show mac address-table interface Ethernet 0 | grep 2005
 2005 40:A8:F0:2C:83:10 DYNAMIC Ethernet0
SRV2 was connected to Vlan2005 on Leaf2 as a regular access port:
 interface Eth1/1
 description test_Srv2
 mtu 9100
 speed 10000
 fec none
 no shutdown
 switchport access Vlan 2005
 root@srv2:~# ip -br link show dev enp59s0f0
 enp59s0f0 UP 40:a8:f0:2c:31:68 <BROADCAST,MULTICAST,UP,LOWER_UP>
Again the MAC is properly learnt on Leaf2:
 
 dell-leaf2# show mac address-table interface Eth1/1
 -----------------------------------------------------------
 VLAN MAC-ADDRESS TYPE INTERFACE
 -----------------------------------------------------------
 2005 40:A8:F0:2C:31:68 DYNAMIC Eth1/1
In terms of EVPN we can see that Leaf1 receives an both a plain MAC-only EVPN route for this address, as well as one with the IP configured on SRV2 included. Both have a next-hop of Leaf2's Loopback1 interface (VTEP IP):
 BGP routing table entry for 10.0.1.25:2005:[2]:[0]:[48]:[40:a8:f0:2c:31:68]
 Paths: (2 available, best #2)
 Advertised to non peer-group peers:
 172.16.1.1 172.16.1.5
 Route [2]:[0]:[48]:[40:a8:f0:2c:31:68] VNI 102005
 65030 65033
 10.10.10.2 from 172.16.1.1 (10.0.1.13)
 Origin IGP, valid, external
 Extended Community: RT:65033:102005 ET:8
 Last update: Fri Apr 8 15:05:41 2022
 Route [2]:[0]:[48]:[40:a8:f0:2c:31:68] VNI 102005
 65030 65033
 10.10.10.2 from 172.16.1.5 (10.0.1.14)
 Origin IGP, valid, external, best (Older Path)
 Extended Community: RT:65033:102005 ET:8
 Last update: Tue Apr 5 14:58:20 2022
 BGP routing table entry for 10.0.1.25:2005:[2]:[0]:[48]:[40:a8:f0:2c:31:68]:[32]:[10.192.80.10]
 Paths: (2 available, best #2)
 Advertised to non peer-group peers:
 172.16.1.1 172.16.1.5
 Route [2]:[0]:[48]:[40:a8:f0:2c:31:68]:[32]:[10.192.80.10] VNI 102005/404000
 65030 65033
 10.10.10.2 from 172.16.1.1 (10.0.1.13)
 Origin IGP, valid, external
 Extended Community: RT:65033:102005 RT:65033:404000 ET:8 Rmac:3c:2c:30:4c:81:83
 Last update: Fri Apr 8 15:05:41 2022
 Route [2]:[0]:[48]:[40:a8:f0:2c:31:68]:[32]:[10.192.80.10] VNI 102005/404000
 65030 65033
 10.10.10.2 from 172.16.1.5 (10.0.1.14)
 Origin IGP, valid, external, best (Older Path)
 Extended Community: RT:65033:102005 RT:65033:404000 ET:8 Rmac:3c:2c:30:4c:81:83
 Last update: Tue Apr 5 14:58:20 2022
This is properly processed and the EVPN database lists it correctly:
 dell-leaf1# show evpn mac vni 102005 mac 40:a8:f0:2c:31:68
 MAC: 40:a8:f0:2c:31:68
 Remote VTEP: 10.10.10.2
 Local Seq: 0 Remote Seq: 0
 Kernel Add: Success, Add ReAttempt:0
 Neighbors:
 10.192.80.10 Active
 2620:0:861:cabf::10 Active
 fe80::42a8:f0ff:fe2c:3168 Active
When we look at the local Vlan forwarding table on Leaf1 we can see that this MAC has been added, with the remote VTEP listed against it:
 dell-leaf1# show mac address-table dynamic Vlan 2005
 -----------------------------------------------------------
 VLAN MAC-ADDRESS TYPE INTERFACE
 -----------------------------------------------------------
 2005 40:A8:F0:2C:31:68 DYNAMIC VxLAN DIP: 10.10.10.2
 2005 40:A8:F0:2C:83:10 DYNAMIC Ethernet0
The reverse is true on Leaf2 for SRV1's MAC learnt on Leaf1, I omit the EVPN details as they work/look much the same in reverse:
 dell-leaf2# show mac address-table dynamic Vlan 2005
 -----------------------------------------------------------
 VLAN MAC-ADDRESS TYPE INTERFACE 
 -----------------------------------------------------------
 2005 40:A8:F0:2C:31:68 DYNAMIC Eth1/1 
 2005 40:A8:F0:2C:83:10 DYNAMIC VxLAN DIP: 10.10.10.1
With this in place we should be able to send unicast Ethernet frames between the two servers, which the switches will send over the IP underlay using VXLAN. We test this with a simple ping from SRV1 to SRV2:
 root@srv1:~# ping -c 2 10.192.80.10
 PING 10.192.80.10 (10.192.80.10) 56(84) bytes of data.
 64 bytes from 10.192.80.10: icmp_seq=1 ttl=64 time=0.206 ms
 64 bytes from 10.192.80.10: icmp_seq=2 ttl=64 time=0.179 ms
 --- 10.192.80.10 ping statistics ---
 2 packets transmitted, 2 received, 0% packet loss, time 1029ms
 rtt min/avg/max/mdev = 0.179/0.192/0.206/0.019 ms
Doing a TCPdump we can see the MAC addresses are expected:
 root@srv2:~# tcpdump -e -i enp59s0f0 -l -p -nn icmp
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on enp59s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
 09:08:46.807595 40:a8:f0:2c:83:10 > 40:a8:f0:2c:31:68, ethertype IPv4 (0x0800), length 98: 10.192.80.187 > 10.192.80.10: ICMP echo request, id 23100, seq 1, length 64
 09:08:46.807628 40:a8:f0:2c:31:68 > 40:a8:f0:2c:83:10, ethertype IPv4 (0x0800), length 98: 10.192.80.10 > 10.192.80.187: ICMP echo reply, id 23100, seq 1, length 64
 09:08:47.827376 40:a8:f0:2c:83:10 > 40:a8:f0:2c:31:68, ethertype IPv4 (0x0800), length 98: 10.192.80.187 > 10.192.80.10: ICMP echo request, id 23100, seq 2, length 64
 09:08:47.827403 40:a8:f0:2c:31:68 > 40:a8:f0:2c:83:10, ethertype IPv4 (0x0800), length 98: 10.192.80.10 > 10.192.80.187: ICMP echo reply, id 23100, seq 2, length 64
== Client to Client broadcast forwarding / ingress replication ==
=== Same Switch (Pure L2) ===
Config the same as in 3.8.1
ARP cache was deleted in the TESTNS namespace / on port enp59s0f1, before initiating a ping:
 root@srv1:~# ip netns exec TESTNS bash
 root@srv1:~# ip neigh del 10.192.64.10 dev enp59s0f1.2004
 root@srv1:~# ping 10.192.64.10
 PING 10.192.64.10 (10.192.64.10) 56(84) bytes of data.
 64 bytes from 10.192.64.10: icmp_seq=1 ttl=64 time=0.388 ms
 64 bytes from 10.192.64.10: icmp_seq=2 ttl=64 time=0.127 ms
TCPDUMP in the default netns show's the broadcast arriving on the port from switch:
 root@srv1:~# tcpdump -i enp59s0f0.2004 -e -l -p -nn arp
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on enp59s0f0.2004, link-type EN10MB (Ethernet), capture size 262144 bytes
 13:30:08.545177 40:a8:f0:2c:83:14 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.192.64.10 tell 10.192.64.30, length 46
=== Remote Switch (VXLAN tunneled) ===
Switch and server ports were configured the same as for the unicast test 3.8.2.
We do not have IP Multicast configured in the underlay network, so instead Ingress Replication is used. In this case the edge device receiving the BUM frame duplicates it and sends a copy to each remote switch that has originated an EVPN type 3 route to say it is participating in that VNI/Vlan.
In our lab Leaf1 and Leaf2 both have Vlan2005 configured, bound to VXLAN VNI 102005. We thus see EVPN type 3 routes on each that have originated on the remote side:
Leaf1 output showing route from Leaf2:
 dell-leaf1# show bgp l2vpn evpn route detail type multicast | find 10.0.1.25
 Route Distinguisher: 10.0.1.25:2005
 BGP routing table entry for 10.0.1.25:2005:[3]:[0]:[32]:[10.10.10.2]
 Paths: (2 available, best #2)
 Advertised to non peer-group peers:
 172.16.1.1 172.16.1.5
 65030 65033
 10.10.10.2 from 172.16.1.1 (10.0.1.13)
 Origin IGP, valid, external
 Extended Community: RT:65033:102005 ET:8
 Last update: Fri Apr 8 15:05:42 2022
 PMSI Tunnel Type: Ingress Replication, label: 102005
 65030 65033
 10.10.10.2 from 172.16.1.5 (10.0.1.14)
 Origin IGP, valid, external, best (Older Path)
 Extended Community: RT:65033:102005 ET:8
 Last update: Tue Apr 5 14:58:21 2022
 PMSI Tunnel Type: Ingress Replication, label: 102005
This is also visible in the below command, with the loopback1 IP of Leaf2 (10.10.10.2) shown as a remote VTEP with flooding enabled:
 dell-leaf1# show evpn vni 102005
 VNI: 102005
 Type: L2
 Tenant VRF: default
 Client State: Up
 VxLAN interface: vtep1-2005
 VxLAN ifIndex: 134
 Local VTEP IP: 10.10.10.1
 Mcast group: 0.0.0.0
 Remote VTEPs for this VNI:
 10.10.10.2 flood: HER
 Kernel Add: Success, Add ReAttempt:0
 Number of MACs (local and remote) known for this VNI: 2
 Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 6
 Advertise-gw-macip: No
Leaf2 output shows a similar route, but for Leaf1, and again the EVPN database shows the remote IP as being part of the VNI:
 dell-leaf2# show bgp l2vpn evpn route detail type multicast | find 2005
 Route Distinguisher: 10.0.1.24:2005
 BGP routing table entry for 10.0.1.24:2005:[3]:[0]:[32]:[10.10.10.1]
 Paths: (2 available, best #2)
 Advertised to non peer-group peers:
 172.16.1.9 172.16.1.13
 65030 65032
 10.10.10.1 from 172.16.1.9 (10.0.1.13)
 Origin IGP, valid, external
 Extended Community: RT:65032:102005 ET:8
 Last update: Fri Apr 8 15:05:42 2022
 PMSI Tunnel Type: Ingress Replication, label: 102005
 65030 65032
 10.10.10.1 from 172.16.1.13 (10.0.1.14)
 Origin IGP, valid, external, best (Older Path)
 Extended Community: RT:65032:102005 ET:8
 Last update: Tue Apr 5 14:58:21 2022
 PMSI Tunnel Type: Ingress Replication, label: 102005
 dell-leaf2# show evpn vni 102005
 VNI: 102005
 Type: L2
 Tenant VRF: Vrf_codfw
 Client State: Up
 VxLAN interface: vtep1-2005
 VxLAN ifIndex: 77
 Local VTEP IP: 10.10.10.2
 Mcast group: 0.0.0.0
 Remote VTEPs for this VNI:
 10.10.10.1 flood: HER
 Kernel Add: Success, Add ReAttempt:0
 Number of MACs (local and remote) known for this VNI: 3
 Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 10
 Advertise-gw-macip: No
This should ensure that all broadcast frames transmitted by SRV1 will be received by SRV2 (possibly with the exception of ARP due to ARP suppression).
To test we generate a random broadcast frame on SRV1, connected to Leaf1:
 root@srv1:~# echo "test broadcast in vlan2005" | nc -q1 -u -s 10.192.80.187 -b 255.255.255.255 12345
A TCPdump shows this is received as expected on SRV2, connected to Leaf2:
 root@srv2:~# tcpdump -X -e -i enp59s0f0 -l -p -nn src 10.192.80.187
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on enp59s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
 09:35:31.670833 40:a8:f0:2c:83:10 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 69: 10.192.80.187.34163 > 255.255.255.255.12345: UDP, length 27
 0x0000: 4500 0037 a5d8 4000 4011 3963 0ac0 50bb E..7..@.@.9c..P.
 0x0010: ffff ffff 8573 3039 0023 68d1 7465 7374 .....s09.#h.test
 0x0020: 2062 726f 6164 6361 7374 2069 6e20 766c .broadcast.in.vl
 0x0030: 616e 3230 3035 0a an2005.
We can conclude that ingress replication is thus working as it should for broadcast traffic.
== Client to Client L2 multicast forwarding / ingress replication ==
=== Same Switch (Pure L2) ===
Config the same as in 3.8.1.
Multicast generated in TESTNS with Netcat:
 root@srv1:~# echo "test multicast in vlan2004" | nc -q1 -u -b 224.0.0.26 12345
This can be observed leaving the server in a TCPdump:
 root@srv1:~# tcpdump -e -i enp59s0f1.2004 -l -p -nn
 listening on enp59s0f1.2004, link-type EN10MB (Ethernet), capture size 262144 bytes
 17:02:40.891494 40:a8:f0:2c:83:14 > 01:00:5e:00:00:1a, ethertype IPv4 (0x0800), length 69: 10.192.64.129.38067 > 224.0.0.26.12345: UDP, length 27
The frame is recieved on the other port having been flooded within the Vlan by the switch:
 root@srv1:~# tcpdump -X -e -i enp59s0f0 -l -nn net 224.0.0.0/8
 listening on enp59s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
 17:02:40.891570 40:a8:f0:2c:83:14 > 01:00:5e:00:00:1a, ethertype 802.1Q (0x8100), length 73: vlan 2004, p 0, ethertype IPv4, 10.192.64.129.38067 > 224.0.0.26.12345: UDP, length 27
 0x0000: 4500 0037 6f76 4000 0111 dee4 0ac0 4081 E..7ov@.......@.
 0x0010: e000 001a 94b3 3039 0023 73a4 7465 7374 ......09.#s.test
 0x0020: 206d 756c 7469 6361 7374 2069 6e20 766c .multicast.in.vl
 0x0030: 616e 3230 3034 0a an2004.
=== Remote Switch (VXLAN tunneled) ===
Two servers are connected, one to each Leaf, the same as in test 3.8.2 / 3.9.2.
We generate a multicast frame on SRV1:
 root@srv1:~# echo "test multicast in vlan2005" | nc -q1 -u -s 10.192.80.187 -b 224.0.0.26 12345 
This can be seen on the wire going out from SRV1 as expected:
 root@srv1:~# tcpdump -e -i enp59s0f0.2005 -l -p -nn net 224.0.0.0/8
 listening on enp59s0f0.2005, link-type EN10MB (Ethernet), capture size 262144 bytes
 18:08:46.017656 40:a8:f0:2c:83:10 > 01:00:5e:00:00:1a, ethertype IPv4 (0x0800), length 69: 10.192.80.187.35480 > 224.0.0.26.12345: UDP, length 27
The frame is received on SRV2, connected to the same Vlan but off Leaf2, showing the multicasts have been sent over the VXLAN fabric as required:
 root@srv2:~# tcpdump -X -i enp59s0f0 -l -nn -e net 224.0.0.0/8
 listening on enp59s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
 18:09:04.782380 40:a8:f0:2c:83:10 > 01:00:5e:00:00:1a, ethertype IPv4 (0x0800), length 69: 10.192.80.187.35480 > 224.0.0.26.12345: UDP, length 27
 0x0000: 4500 0037 9a63 4000 0111 a3bd 0ac0 50bb E..7.c@.......P.
 0x0010: e000 001a 8a98 3039 0023 6d84 7465 7374 ......09.#m.test
 0x0020: 206d 756c 7469 6361 7374 2069 6e20 766c .multicast.in.vl
 0x0030: 616e 3230 3035 0a an2005.
== Inter-Vlan/subnet routing via IRB interfaces on same switch ==
Leaf1 has Vlan interfaces for Vlan2004 and Vlan2005 configured:
 interface Vlan2004
 description private1-e-codfw
 ip vrf forwarding Vrf_codfw
 ipv6 enable
 ip anycast-address 10.192.64.254/24
 ipv6 anycast-address 2620:0:861:11c::254/64
 interface Vlan2005
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ipv6 enable
 ip anycast-address 10.192.80.254/22
 ipv6 anycast-address 2620:0:861:cabf::254/64
=== IPv4 ===
 root@srv1:~# mtr --address 10.192.64.129 -n -r 10.192.80.187
 Start: 2022-05-05T18:42:03+0000
 HOST: srv1 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 10.192.64.254 0.0% 10 0.2 0.3 0.2 0.6 0.1
 2.|-- 10.192.80.187 0.0% 10 0.2 0.2 0.2 0.2 0.0
=== IPv6 ===
 root@srv1:~# mtr --address 2620:0:861:11c::129 -n -r 2620:0:861:cabf::187
 Start: 2022-05-05T18:41:03+0000
 HOST: srv1 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 2620:0:861:11c::254 0.0% 10 0.4 0.8 0.3 5.0 1.4
 2.|-- 2620:0:861:cabf::187 0.0% 10 0.2 0.3 0.2 1.1 0.3
== Inter-Vlan/subnet routing via IRB interfaces on separate switches ==
=== IPv4 ===
From test server 1 in Vlan 2004 on leaf1 ping test server2 in vlan 2005 on leaf2
 root@srv1:~# mtr --address 10.192.64.10 -n -r -c 1 10.192.80.10
 Start: 2022-04-28T10:03:53+0000
 HOST: srv1 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 10.192.64.254 0.0% 1 0.4 0.4 0.4 0.4 0.0
 2.|-- 10.192.80.254 0.0% 1 0.4 0.4 0.4 0.4 0.0
 3.|-- 10.192.80.10 0.0% 1 0.3 0.3 0.3 0.3 0.0
=== IPv6 ===
Routing works as expected, we can ping between subnets across switches just fine:
 root@srv1:~# ping -I 2620:0:861:11c::10 2620:0:861:cabf::10
 PING 2620:0:861:cabf::10(2620:0:861:cabf::10) from 2620:0:861:11c::10 : 56 data bytes
 64 bytes from 2620:0:861:cabf::10: icmp_seq=1 ttl=62 time=0.227 ms
 64 bytes from 2620:0:861:cabf::10: icmp_seq=2 ttl=62 time=0.176 ms
One issue we note, however, is that when doing a traceroute the *remote* switch shows up with an IPv6 link-local address.
 root@srv1:~# mtr --address 2620:0:861:11c::10 -n -r 2620:0:861:cabf::10
 Start: 2022-04-28T10:14:45+0000
 HOST: srv1 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 2620:0:861:11c::254 0.0% 10 0.4 0.4 0.3 0.4 0.0
 2.|-- fe80::3e2c:30ff:fe4c:8183 0.0% 10 0.3 0.8 0.3 4.9 1.4
 3.|-- 2620:0:861:cabf::10 0.0% 10 0.3 0.3 0.3 0.3 0.0
The equivalent at hop 2 in the IPv4 test was the address of the Vlan2005 interface on Leaf 2 (10.192.80.254). That interface is configured with a global unicast IPv6 address, and v6 is enabled, but it doesn't use it:
 interface Vlan2005
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ipv6 enable
 ip anycast-address 10.192.80.254/22
 ipv6 anycast-address 2620:0:861:cabf::254/64
Looking a little closer it seems the address used to source the ICMPv6 PTB messages is the one assigned to Vlan4000 on the switch (this is the Vlan created to bind to our VRF for VXLAN encap):
 admin@dell-leaf2:~$ ip -br addr show | grep 8183
 Vlan4000@Bridge UP fe80::3e2c:30ff:fe4c:8183/64 
 dell-leaf2# show running-configuration interface Vlan 4000
 !
 interface Vlan4000
 description "IRB VLAN"
 ip vrf forwarding Vrf_codfw
 ipv6 enable
We see the same behaviour when doing a trace which routes out via an external device connected to the Spine layer within the VRF:
 root@srv1:~# mtr --address 2620:0:861:11c::10 -n -r 2001:67c:930:400::26 
 Start: 2022-04-28T10:39:04+0000
 HOST: srv1 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 2620:0:861:11c::254 0.0% 10 0.3 0.3 0.3 0.3 0.0
 2.|-- fe80::e29:efff:fee1:fb01 0.0% 10 0.5 0.4 0.4 0.5 0.0
 3.|-- 2001:67c:930:400::26 0.0% 10 5.5 6.1 2.0 11.1 2.4
In this case again the IP used by Spine1 (hop 2) is link local. Looking at the device this IP is seemingly used on both the external interface and the VLan4000 device:
 admin@dell-spine1:~$ ip -br addr show | grep "fee1:fb01"
 Ethernet104 UP 172.16.1.25/30 2001:67c:930:400::25/64 fe80::e29:efff:fee1:fb01/64 
 Vlan4000@Bridge UP fe80::e29:efff:fee1:fb01/64 
It certainly would be better if the system had of used global unicast address 2001:67c:930:400::25, or another unicast address on the device, to source the ICMP PTB messages. In contrast the Juniper QFX paltform does use global unicast addressing to source ICMPs, which makes troubleshooting easier as the switches can be identified in a trace:
 cmooney@elastic1089:~$ sudo traceroute -I -w 1 -s 2620:0:861:109:10:64:130:7 2620:0:861:10d:10:64:134:2
 traceroute to 2620:0:861:10d:10:64:134:2 (2620:0:861:10d:10:64:134:2), 30 hops max, 80 byte packets
 1 irb-1031.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:109::1) 0.610 ms 0.583 ms 0.576 ms
 2 irb-1035.lsw1-f1-eqiad.eqiad.wmnet (2620:0:861:10d::1) 0.515 ms 0.509 ms 0.504 ms
 3 dumpsdata1007.eqiad.wmnet (2620:0:861:10d:10:64:134:2) 0.170 ms * *
== IPv6 RA Generation from Vlan Interfaces ==
NOT WORKING
Vlan configuration as follows on Leaf2:
 interface Vlan2005
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ipv6 enable
 ip anycast-address 10.192.80.254/22
 ipv6 anycast-address 2620:0:861:cabf::254/6
No IPv6 RAs are recieved on SRV2 which is connected on an access port in this Vlan however:
 root@srv2:~# tcpdump -i enp59s0f0 -l -nn icmp6
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on enp59s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
 ^C
 0 packets captured
 0 packets received by filter
 0 packets dropped by kernel
== BGP Peering on Vlan segment to end device ==
=== IPv4 only ===
Srv1 is reachable in Vlan2004 on IP 10.192.64.10
 dell-leaf1# show ip arp vrf Vrf_codfw 
 Type: R - Remote Neighbor entries (EVPN)
 -------------------------------------------------------------------------------------------
 Address Hardware address Interface Egress Interface Type
 -------------------------------------------------------------------------------------------
 10.192.64.10 40:a8:f0:2c:83:10 Vlan2004 Ethernet0 Dynamic 
We configure a BGP session to it within the VRF as follows:
 router bgp 65032 vrf Vrf_codfw
 router-id 10.0.1.24
 log-neighbor-changes
 bestpath as-path multipath-relax
 !
 address-family ipv4 unicast
 redistribute connected 
 maximum-paths 16
 !
 neighbor 10.192.64.10
 remote-as 64600
 timers 5 20
 advertisement-interval 0
 bfd
 local-as 14907 no-prepend replace-as
 !
 address-family ipv4 unicast
 activate
 route-map NO_HOST_ROUTES out
 remove-private-as
 !
The "NO_HOST_ROUTES" filter is designed to prevent /32 host routes from being sent externally. These end up in the VRF BGP RIB due to the import of EVPN type 2 MAC/IP routes to the local table. Instead we only want to send the routes that have originated locally or as type 5 EVPN routes, i.e. the subnets allocated to our Vlans and not every host IP from the Vlan.
 ip prefix-list NO_HOST_ROUTES seq 5 permit 0.0.0.0/0 le 29
 route-map NO_HOST_ROUTES permit 100
 match ip address prefix-list NO_HOST_ROUTES
Srv1 is set up for BGP (running FRR in this case), configured to peer with the IP of Leaf1 on Vlan2004:
 router bgp 64600
 neighbor 10.192.64.254 remote-as 14907
 neighbor 10.192.64.254 bfd
 !
 address-family ipv4 unicast
 exit-address-family
With this in place the session establishes:
 srv1# show bgp ipv4 unicast summary 
 BGP router identifier 1.1.1.1, local AS number 64600 vrf-id 0
 BGP table version 15
 RIB entries 5, using 920 bytes of memory
 Peers 1, using 723 KiB of memory
 
 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
 10.192.64.254 4 14907 482 396 0 0 0 00:21:21 2 1 N/A
 
 Total number of neighbors 1
The prefix for Vlan2004 is originated locally by leaf1 (due to the 'redistributed connected'), the prefix for Vlan2005 is in the BGP RIB on Leaf1 already having been redistributed from the EVPN. Both are learnt on the server side:
 srv1# show bgp ipv4 unicast neighbors 10.192.64.254 routes 
 BGP table version is 24, local router ID is 1.1.1.1, vrf id 0
 Default local pref 100, local AS 64600
 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
 i internal, r RIB-failure, S Stale, R Removed
 Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
 Origin codes: i - IGP, e - EGP, ? - incomplete
 RPKI validation codes: V valid, I invalid, N Not found
 
 Network Next Hop Metric LocPrf Weight Path
 *> 10.192.64.0/22 10.192.64.254 0 0 14907 ?
 *> 10.192.80.0/22 10.192.64.254 0 14907 ?
The as-path just shows the "fake" AS configured in the 'local-as' command, as desired. The Spine and Leaf2 AS numbers would appear for 10.192.80.0/22 but we have configured the sessions to "remove-private-as" to validate that works as expected.
=== IPv4 carrying IPv4 & IPv6 address families ===
Configuration on the switch was the same as the last test, but the IPv6 unicast address family was also activated for the server peer:
 router bgp 65032 vrf Vrf_codfw
 neighbor 10.192.64.10
 remote-as 64600
 !
 address-family ipv6 unicast
 activate
 remove-private-as
 !
With the same done server-side the BGP session restarted, and came back up announcing IPv6 addresses also:
 dell-leaf1# show bgp ipv6 unicast vrf Vrf_codfw summary
 BGP router identifier 10.0.1.24, local AS number 65032 
 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd 
 10.192.64.10 4 64600 144556 173488 0 0 00:06:06 1 
 
 Total number of neighbors 1
 Total number of neighbors established 1
Address was received as expected:
 dell-leaf1# show bgp ipv6 unicast vrf Vrf_codfw neighbors 10.192.64.10 routes 
 BGP routing table information for VRF default
 Router identifier 10.0.1.24, local AS number 65032 
 Route status codes: * - valid, > - best
 Origin codes: i - IGP, e - EGP, ? - incomplete
 Network Next Hop Metric LocPref Weight Path
 *> 2620:0:861:babe::1/1282620:0:861:11c::10 0 64600 ?
 Total number of prefixes 1
And placed into the local VRF routing table as required:
 dell-leaf1# show ipv6 route vrf Vrf_codfw 2620:0:861:babe::1
 Codes: K - kernel route, C - connected, S - static, B - BGP, O - OSPF
 > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
 Destination Gateway Dist/Metric Uptime 
 --------------------------------------------------------------------------------------------------------------------
 B>* 2620:0:861:babe::1/128 via fe80::42a8:f0ff:fe2c:8310 Vlan2004 20/0 00:05:50
=== IPv4 & IPv6 each carrying their own address family ===
== BGP Peering on Vlan segment from Anycast GW IP ==
=== IPv4 only ===
=== IPv4 carrying IPv4 & IPv6 address families ===
=== IPv4 & IPv6 each carrying their own address family ===
== BGP Peering on Vlan segment with Anycast GW, from Unicast IP ==
NOT WORKING
It does not seem possible to configure an additional IP address on a Vlan interface configured with an Anycast GW.
For instance with this config on Vlan2005 on Leaf2:
 interface Vlan2005
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.80.254/22
 ipv6 anycast-address 2620:0:861:cabf::254/64
We cannot add an additional (unique to this switch) IP address in addition to the anycast IP (used on all switches):
 dell-leaf2# configure terminal
 dell-leaf2(config)# interface Vlan2005
 dell-leaf2(conf-if-Vlan2005)# ip address 10.192.80.253/22 secondary
 %Error: Primary IPv4 address is not configured for interface: Vlan2005
 dell-leaf2(conf-if-Vlan2005)# ip address 10.192.80.253/22
 %Error: IP overlap on same interface with IP or IP Anycast 10.192.80.254/22
This is not something that makes much of a difference to us generally. We can likely create unique loopback interfaces, in the overlay VRF, on the switch with unique IPs per device, if needed. The only scenario this is probably needed is if we have an MC-LAG configured, and need to peer to both switches over the bonded link. This is not something that is currently likely for us to require.
== BGP Re-establishment if device peered to Anycast GW moved to new switch (i.e. VM live motion) ==
== eBGP Peering in VRF to external device ==
The VRF was enabled on the spine switches, each of which was connected to a Juniper device to test eBGP routing to an external element from the VRF.
Interface config on Spine1, connected to the Juniper switch was as follows:
 dell-spine1# show running-configuration interface Eth1/27
 !
 interface Eth1/27
 description link_to_lsw3-et-0/0/50
 mtu 9100
 speed 40000
 fec none
 no shutdown
 ip vrf forwarding Vrf_codfw
 ip address 172.16.1.25/30
 ipv6 address 2001:67c:930:400::25/64
 ipv6 enable
BGP sessions were configured, one over IPv4 for IPv4 address-fam, one over IPv6 for IPv6 address-fam:
 router bgp 65030 vrf Vrf_codfw
 router-id 10.0.1.13
 !
 address-family ipv4 unicast
 maximum-paths 8
 maximum-paths ibgp 8
 !
 address-family ipv6 unicast
 maximum-paths 8
 maximum-paths ibgp 8
 !
 neighbor 172.16.1.26
 remote-as 65034
 !
 address-family ipv4 unicast
 activate
 !
 neighbor 2001:67c:930:400::26
 remote-as 65034
 !
 address-family ipv4 unicast
 !
 address-family ipv6 unicast
 activate
BGP established as expected for each address family:
 dell-spine1# show bgp ipv4 unicast vrf Vrf_codfw summary
 BGP router identifier 10.0.1.13, local AS number 65030
 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd
 172.16.1.26 4 65034 24829 22586 0 0 01w0d20h 1 
 dell-spine1# show bgp ipv6 unicast vrf Vrf_codfw summary
 BGP router identifier 10.0.1.13, local AS number 65030
 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd
 2001:67c:930:400::26 4 65034 167 150 0 0 01:04:40 1
Routes are accepted:
 dell-spine1# show bgp ipv4 unicast vrf Vrf_codfw neighbors 172.16.1.26 routes
 BGP routing table information for VRF default
 Router identifier 10.0.1.13, local AS number 65030
 Route status codes: * - valid, > - best
 Origin codes: i - IGP, e - EGP, ? - incomplete
 Network Next Hop Metric LocPref Weight Path
 *> 0.0.0.0/0 172.16.1.26 65034 i
 Total number of prefixes 1
 dell-spine1# show bgp ipv6 unicast vrf Vrf_codfw neighbors 2001:67c:930:400::26 routes
 BGP routing table information for VRF default
 Router identifier 10.0.1.13, local AS number 65030
 Route status codes: * - valid, > - best
 Origin codes: i - IGP, e - EGP, ? - incomplete
 Network Next Hop Metric LocPref Weight Path
 *> ::/0 2001:67c:930:400::26 65034 i
 Total number of prefixes 1
And make their way into the local FIB:
 dell-spine1# show ip route vrf Vrf_codfw 0.0.0.0/0
 Codes: K - kernel route, C - connected, S - static, B - BGP, O - OSPF
 > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
 Destination Gateway Dist/Metric Uptime
 --------------------------------------------------------------------------------------------------------------------
 B>* 0.0.0.0/0 via 172.16.1.26 Eth1/27 20/0 01w0d20h
 dell-spine1# show ipv6 route vrf Vrf_codfw ::/0
 Codes: K - kernel route, C - connected, S - static, B - BGP, O - OSPF
 > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
 Destination Gateway Dist/Metric Uptime
 --------------------------------------------------------------------------------------------------------------------
 B>* ::/0 via fe80::2e21:31ff:fefa:f1bb Eth1/27 20/0 01:04:58
== BFD Support on Peering to External Device (VRF) ==
BFD was enabled for the peering to srv1 in Vlan2004 as follows on the switch:
 router bgp 65032 vrf Vrf_codfw
 neighbor 10.192.64.10
 remote-as 64600
 bfd
Once it was configured on the server as well the session came up:
 dell-leaf1# show bfd peers vrf Vrf_codfw
 BFD Peers:
 
 peer 10.192.64.10 vrf Vrf_codfw interface Vlan2004
 ID: 624049837
 Remote ID: 3910201229
 Status: up
 Uptime: 0 day(s), 0 hour(s), 0 min(s), 46 sec(s)
 Diagnostics: ok
 Remote diagnostics: ok
 Peer Type: dynamic
 Local timers:
 Detect-multiplier: 3
 Receive interval: 300ms
 Transmission interval: 300ms
 Echo transmission interval: 0ms
 Remote timers:
 Detect-multiplier: 3
 Receive interval: 300ms
 Transmission interval: 300ms
 Echo transmission interval: 50ms
== BFD Support on BGP peering in global table ==
BFD was set up in the underlay network / global table on the BGP peerings between Spine1 and the two Leaf devices. Simply needs the keyword 'bfd' added under the BGP neighbor definition:
 router bgp 65030
 neighbor 172.16.1.10
 remote-as 65033
 bfd
Once configured both sides the sessions come up:
 dell-spine1# show bfd peers brief 
 Session Count: 2
 SessionId LocalAddress PeerAddress Status Vrf 
 ========= ============ =========== ====== === 
 2747556315 172.16.1.9 172.16.1.10 UP default 
 1863681777 172.16.1.1 172.16.1.2 UP default 
 dell-spine1# show bfd peers 
 BFD Peers:
 
 
 peer 172.16.1.10 vrf default interface Eth1/31
 ID: 2747556315
 Remote ID: 1404963392
 Status: up
 Uptime: 0 day(s), 0 hour(s), 1 min(s), 27 sec(s)
 Diagnostics: ok
 Remote diagnostics: ok
 Peer Type: dynamic
 Local timers:
 Detect-multiplier: 3
 Receive interval: 300ms
 Transmission interval: 300ms
 Echo transmission interval: 0ms
 Remote timers:
 Detect-multiplier: 3
 Receive interval: 300ms
 Transmission interval: 300ms
 Echo transmission interval: 50ms
 
 
 peer 172.16.1.2 vrf default interface Eth1/32
 ID: 1863681777
 Remote ID: 2448635902
 Status: up
 Uptime: 0 day(s), 0 hour(s), 5 min(s), 53 sec(s)
 Diagnostics: ok
 Remote diagnostics: ok
 Peer Type: dynamic
 Local timers:
 Detect-multiplier: 3
 Receive interval: 300ms
 Transmission interval: 300ms
 Echo transmission interval: 0ms
 Remote timers:
 Detect-multiplier: 3
 Receive interval: 300ms
 Transmission interval: 300ms
 Echo transmission interval: 50ms
== BGP Route propagaton from unicast peer into EVPN and into remote VRF table ==
=== IPv4 ===
We learn 1.1.1.1/32 from srv1 on Leaf1:
 dell-leaf1# show bgp ipv4 unicast vrf Vrf_codfw neighbors 10.192.64.10 routes 
 BGP routing table information for VRF default
 Router identifier 10.0.1.24, local AS number 65032 
 Route status codes: * - valid, > - best
 Origin codes: i - IGP, e - EGP, ? - incomplete
 Network Next Hop Metric LocPref Weight Path
 *> 1.1.1.1/32 10.192.64.10 0 64600 ?
 Total number of prefixes 1
We can see this being learnt on Leaf 2 as an EVPN type-5 route, next-hop the Leaf1 VTEP IP:
 dell-leaf2# show bgp l2vpn evpn route detail | find 1.1.1.1
 BGP routing table entry for 10.0.1.24:5096:[5]:[0]:[32]:[1.1.1.1]
 Paths: (2 available, best #1)
 Advertised to non peer-group peers:
 172.16.1.9 172.16.1.13
 Route [5]:[0]:[32]:[1.1.1.1] VNI 404000
 65030 65032 64600
 10.10.10.1 from 172.16.1.9 (10.0.1.13)
 Origin incomplete, valid, external, best (Router ID)
 Extended Community: RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
 Last update: Fri Apr 15 15:14:11 2022
 Route [5]:[0]:[32]:[1.1.1.1] VNI 404000
 65030 65032 64600
 10.10.10.1 from 172.16.1.13 (10.0.1.14)
 Origin incomplete, valid, external
 Extended Community: RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
 Last update: Fri Apr 15 15:14:11 2022
This is correctly added to the local VRF routing table on Leaf 2:
 dell-leaf2# show ip route vrf Vrf_codfw 1.1.1.1
 Codes: K - kernel route, C - connected, S - static, B - BGP, O - OSPF
 > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
 Destination Gateway Dist/Metric Uptime 
 --------------------------------------------------------------------------------------------------------------------
 B>* 1.1.1.1/32 via 10.10.10.1 Vlan4000 20/0 00:26:49 
We can send packets from srv2 (connected to Leaf 2 on Vlan2005) and get a response from 1.1.1.1 on srv1 (connected to Leaf1):
 ppaul@srv2:~$ mtr -n -r 1.1.1.1 -c 3
 Start: 2022-04-15T15:30:42+0000
 HOST: srv2 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 10.192.80.254 0.0% 3 0.2 0.3 0.2 0.3 0.0
 2.|-- 10.192.64.254 0.0% 3 0.3 0.3 0.3 0.3 0.0
 3.|-- 1.1.1.1 0.0% 3 0.3 0.3 0.3 0.3 0.0
=== IPv6 ===
We learn 2620:0:861:babe::1/128 via BGP from srv1 connected over Vlan2004 on Leaf1:
 dell-leaf1# show bgp ipv6 unicast vrf Vrf_codfw neighbors 10.192.64.10 routes
 BGP routing table information for VRF default
 Router identifier 10.0.1.24, local AS number 65032 
 Route status codes: * - valid, > - best
 Origin codes: i - IGP, e - EGP, ? - incomplete
 Network Next Hop Metric LocPref Weight Path
 *> 2620:0:861:babe::1/1282620:0:861:11c::10 0 64600 ?
 Total number of prefixes 1
This is received on Leaf 2 as an EVPN type-5 with next-hop of Leaf1's VTEP IP:
 dell-leaf2# show bgp l2vpn evpn route detail | find babe
 BGP routing table entry for 10.0.1.24:5096:[5]:[0]:[128]:[2620:0:861:babe::1]
 Paths: (2 available, best #1)
 Advertised to non peer-group peers:
 172.16.1.9 172.16.1.13
 Route [5]:[0]:[128]:[2620:0:861:babe::1] VNI 404000
 65030 65032 64600
 10.10.10.1 from 172.16.1.9 (10.0.1.13)
 Origin incomplete, valid, external, best (Router ID)
 Extended Community: RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
 Last update: Fri Apr 15 15:28:43 2022
 Route [5]:[0]:[128]:[2620:0:861:babe::1] VNI 404000
 65030 65032 64600
 10.10.10.1 from 172.16.1.13 (10.0.1.14)
 Origin incomplete, valid, external
 Extended Community: RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
 Last update: Fri Apr 15 15:28:43 2022
It's added to the local VRF routing table on Leaf 2:
 dell-leaf2# show ipv6 route vrf Vrf_codfw 2620:0:861:babe::1
 Codes: K - kernel route, C - connected, S - static, B - BGP, O - OSPF
 > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
 Destination Gateway Dist/Metric Uptime 
 --------------------------------------------------------------------------------------------------------------------
 B>* 2620:0:861:babe::1/128 via ::ffff:10.10.10.1 Vlan4000 20/0 00:11:10 
And we can send traffic to it from srv2 (connected to Leaf2), similar to the v4 test:
 root@srv2:~# mtr -n -r -c 3 2620:0:861:babe::1
 Start: 2022-04-15T15:37:51+0000
 HOST: srv2 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 2620:0:861:cabf::254 0.0% 3 0.4 0.4 0.4 0.4 0.0
 2.|-- fe80::3e2c:30ff:fe4b:903 0.0% 3 0.3 0.3 0.3 0.4 0.0
 3.|-- 2620:0:861:babe::1 0.0% 3 0.3 0.3 0.3 0.3 0.0
== BGP Route propagation to external hosts from VRF ==
=== IPv4 ===
BGP peering in vrf Vrf_codfw to srv1 was set up the same way as in test 3.15.
On the server we can see that we learn the routes for Vlan2004 (originated by leaf1) and Vlan2005 (originated by leaf2 and propagated as an EVPN type 5 to leaf 1):
 srv1# show bgp ipv4 unicast neighbors 10.192.64.254 routes 
 BGP table version is 15, local router ID is 1.1.1.1, vrf id 0
 Default local pref 100, local AS 64600
 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
 i internal, r RIB-failure, S Stale, R Removed
 Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
 Origin codes: i - IGP, e - EGP, ? - incomplete
 RPKI validation codes: V valid, I invalid, N Not found
 
 Network Next Hop Metric LocPrf Weight Path
 *> 10.192.64.0/22 10.192.64.254 0 0 14907 65032 ?
 *> 10.192.80.0/22 10.192.64.254 0 14907 65032 65030 65033 ?
=== IPv6 ===
With IPv6 peering to srv1 we see the same thing, we learn the /64 subnets attached to Vlan2004 (local on BGP peer Leaf 1) and also Vlan 2005 (on Leaf 2, and learnt by Leaf 1 over EVPN):
 srv1# show bgp ipv6 unicast neighbors 10.192.64.254 routes 
 BGP table version is 10, local router ID is 1.1.1.1, vrf id 0
 Default local pref 100, local AS 64600
 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
 i internal, r RIB-failure, S Stale, R Removed
 Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
 Origin codes: i - IGP, e - EGP, ? - incomplete
 RPKI validation codes: V valid, I invalid, N Not found
 
 Network Next Hop Metric LocPrf Weight Path
 *> 2620:0:861:11c::/64
 fe80::200:ff:fe10:1010
 0 0 14907 ?
 *> 2620:0:861:cabf::/64
 fe80::200:ff:fe10:1010
 0 14907 65030 65033 ?
== ECMP Routing in overlay network ==
In modern Leaf/Spine fabrics ECMP is a key requirement to make use of multiple redundant links. To test this is working correctly for routes learnt in EVPN type 5 announcements we augmented the configuration described in section 3.19 "eBGP Peering in VRF to external device".
In that section an external device was connected to Spine1 and BGP peering established within the VRF. For this test we connected that same device to Spine2 also, and configured BGP peering from Spine2 for it, announcing the same default routes. Interface and BGP configuration was similar, and BGP established as in that test:
 dell-spine2# show bgp ipv4 unicast vrf Vrf_codfw summary
 BGP router identifier 10.0.1.14, local AS number 65030 
 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd 
 172.16.1.30 4 65034 24762 22530 0 0 00:00:46 1 
 
 dell-spine2# show bgp ipv4 unicast vrf Vrf_codfw neighbors 172.16.1.30 routes 
 BGP routing table information for VRF default
 Router identifier 10.0.1.14, local AS number 65030 
 Route status codes: * - valid, > - best
 Origin codes: i - IGP, e - EGP, ? - incomplete
 Network Next Hop Metric LocPref Weight Path
 *> 0.0.0.0/0 172.16.1.30 65034 i
 
 
 dell-spine2# show bgp ipv6 unicast vrf Vrf_codfw summary
 BGP router identifier 10.0.1.14, local AS number 65030 
 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd 
 2001:67c:930:401::30 4 65034 24 37 0 0 00:00:23 1 
 
 Total number of neighbors 1
 Total number of neighbors established 1
 
 dell-spine2# show bgp ipv6 unicast vrf Vrf_codfw neighbors 2001:67c:930:401::30 routes 
 BGP routing table information for VRF default
 Router identifier 10.0.1.14, local AS number 65030 
 Route status codes: * - valid, > - best
 Origin codes: i - IGP, e - EGP, ? - incomplete
 Network Next Hop Metric LocPref Weight Path
 *> ::/0 2001:67c:930:401::30 65034 i
With that set up we now have a situation where Spine1 and Spine2 are learning a default route in each address family, from an external peer. These should then get announced to the Leaf devices as separate EVPN type 5 prefixes, with next-hop set to the VTEP IP of Spine1 and Spine2. Ultimately the LEAF devices should then ECMP traffic to both spines based on these two routes.
Looking on Leaf2 we can see this is the case, we have 2 v4 and 2 v6 addresses, 1 of each from each spine:
 dell-leaf2# show bgp l2vpn evpn route type prefix
 BGP table version is 2, local router ID is 10.0.1.25
 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
 Origin codes: i - IGP, e - EGP, ? - incomplete
 EVPN type-1 prefix: [1]:[ESI]:[EthTag]
 EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
 EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
 EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
 EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
 Network Next Hop Metric LocPrf Weight Path
 Extended Community
 Route Distinguisher: 10.0.1.13:5096
 *> [5]:[0]:[0]:[0.0.0.0]
 10.10.10.3 0 65030 65034 i
 RT:65030:404000 ET:8 Rmac:0c:29:ef:e1:fb:01
 *> [5]:[0]:[0]:[::] 10.10.10.3 0 65030 65034 i
 RT:65030:404000 ET:8 Rmac:0c:29:ef:e1:fb:01
 Route Distinguisher: 10.0.1.14:5096
 *> [5]:[0]:[0]:[0.0.0.0]
 10.10.10.4 0 65030 65034 i
 RT:65030:404000 ET:8 Rmac:0c:29:ef:e1:fc:81
 *> [5]:[0]:[0]:[::] 10.10.10.4 0 65030 65034 i
 RT:65030:404000 ET:8 Rmac:0c:29:ef:e1:fc:81
=== IPv4 ===
Sticking with Leaf2 if we look at the local VRF routing table it does indeed have both defaults, and both are in the FIB indicating it should ECMP across the two next-hops:
 dell-leaf2# show ip route vrf Vrf_codfw 0.0.0.0/0
 Codes: K - kernel route, C - connected, S - static, B - BGP, O - OSPF
 > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
 Destination Gateway Dist/Metric Uptime
 --------------------------------------------------------------------------------------------------------------------
 B>* 0.0.0.0/0 via 10.10.10.3 Vlan4000 20/0 00:09:44
 * via 10.10.10.4 Vlan4000
If we move to SRV2, which is connected on an access port to Leaf2, we can see the different link addresses of the connections from Spine1 or Spine2 in a traceroute if we change the source address (and thus the ECMP hash):
 root@srv2:~# mtr --address 10.192.80.10 -n -r -c 3 10.0.1.26
 Start: 2022-04-28T12:15:45+0000
 HOST: srv2 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 10.192.80.254 0.0% 3 0.3 0.3 0.3 0.3 0.0
 2.|-- 172.16.1.29 0.0% 3 0.3 0.4 0.3 0.4 0.0
 3.|-- 10.0.1.26 0.0% 3 21.2 18.8 13.7 21.5 4.4
^ Hop 2 is Spine2 in this case.
 root@srv2:~# mtr --address 10.192.80.21 -n -r -c 3 10.0.1.26
 Start: 2022-04-28T12:16:01+0000
 HOST: srv2 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 10.192.80.254 0.0% 3 0.3 0.3 0.3 0.3 0.0
 2.|-- 172.16.1.25 0.0% 3 0.3 0.3 0.3 0.4 0.0
 3.|-- 10.0.1.26 0.0% 3 71.6 32.4 11.4 71.6 33.9
^ Hop 2 is Spine1 in this case.
=== IPv6 ===
Again if we look on Leaf2 the two EVPN routes have been added to the local VRF table and it indicates both are being used:
 dell-leaf2# show ipv6 route vrf Vrf_codfw ::/0
 Codes: K - kernel route, C - connected, S - static, B - BGP, O - OSPF
 > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
 Destination Gateway Dist/Metric Uptime 
 --------------------------------------------------------------------------------------------------------------------
 B>* ::/0 via ::ffff:10.10.10.3 Vlan4000 20/0 00:15:08 
 * via ::ffff:10.10.10.4 Vlan4000 
Doing multiple traceroutes, changing the source IP address, we can see different IPs at hop 2, depending on if the packets are sent by Leaf2 to Spine1 or Spine2:
 Source: 2620:0:861:cabf::66
 Start: 2022-04-28T12:34:11+0000
 HOST: srv2 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 2620:0:861:cabf::254 0.0% 2 0.3 0.5 0.3 0.6 0.2
 2.|-- fe80::e29:efff:fee1:fb01 0.0% 2 0.4 0.4 0.4 0.5 0.1
 3.|-- 2606:4700:4700::1111 0.0% 2 4.2 5.9 4.2 7.7 2.4
 Source: 2620:0:861:cabf::67
 Start: 2022-04-28T12:34:17+0000
 HOST: srv2 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 2620:0:861:cabf::254 0.0% 2 0.3 0.5 0.3 0.6 0.2
 2.|-- fe80::e29:efff:fee1:fb01 0.0% 2 0.4 0.4 0.4 0.4 0.1
 3.|-- 2606:4700:4700::1111 0.0% 2 25.1 14.9 4.7 25.1 14.4
 Source: 2620:0:861:cabf::68
 Start: 2022-04-28T12:34:24+0000
 HOST: srv2 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 2620:0:861:cabf::254 0.0% 2 0.3 0.4 0.3 0.6 0.2
 2.|-- fe80::e29:efff:fee1:fc81 0.0% 2 0.4 0.4 0.4 0.4 0.0
 3.|-- 2606:4700:4700::1111 0.0% 2 4.3 4.6 4.3 4.9 0.4
 Source: 2620:0:861:cabf::69
 Start: 2022-04-28T12:34:30+0000
 HOST: srv2 Loss% Snt Last Avg Best Wrst StDev
 1.|-- 2620:0:861:cabf::254 0.0% 2 0.4 0.5 0.4 0.7 0.2
 2.|-- fe80::e29:efff:fee1:fc81 0.0% 2 0.4 0.4 0.4 0.5 0.1
 3.|-- 2606:4700:4700::1111 0.0% 2 9.4 8.7 8.0 9.4 1.0
== ARP Supression ==
== ND Suppression ==
== DHCP Relay & Option 82 insertion ==
=== DHCP Relay ===
DHCP relay can be configured on a Vlan interface, for instance like this on Vlan 2005 on Leaf2:
 interface Vlan2005
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ip dhcp-relay 10.192.64.10 vrf Vrf_codfw
 ip anycast-address 10.192.80.254/22
In the above case the 10.192.64.10 IP is configured on SRV1, which is connected to Vlan 2004 on Leaf1.
On SRV2, which is connected via an access port to Leaf2 on Vlan2005, we can then issue a DHCP request, which completes as desired:
 root@srv2:~# dhclient -v enp59s0f0
 Internet Systems Consortium DHCP Client 4.3.5
 Copyright 2004-2016 Internet Systems Consortium.
 All rights reserved.
 For info, please visit https://www.isc.org/software/dhcp/
 Listening on LPF/enp59s0f0/40:a8:f0:2c:31:68
 Sending on LPF/enp59s0f0/40:a8:f0:2c:31:68
 Sending on Socket/fallback
 DHCPDISCOVER on enp59s0f0 to 255.255.255.255 port 67 interval 3 (xid=0xe029ca10)
 DHCPREQUEST of 10.192.80.129 on enp59s0f0 to 255.255.255.255 port 67 (xid=0x10ca29e0)
 DHCPOFFER of 10.192.80.129 from 10.192.80.254
 DHCPACK of 10.192.80.129 from 10.192.80.254
 bound to 10.192.80.129 -- renewal in 147 seconds.
Looking on SRV1 we can see that this has been relayed from the IPv4 address on Vlan 2005 of Leaf2:
 root@srv1:/etc/dhcp# tcpdump -i enp59s0f0.2004 -l -nn udp port 67 or udp port 68
 listening on enp59s0f0.2004, link-type EN10MB (Ethernet), capture size 262144 bytes
 14:45:58.969588 IP 10.192.80.254.67 > 10.192.64.10.67: BOOTP/DHCP, Request from 40:a8:f0:2c:31:68, length 300
 14:45:59.971058 IP 10.192.64.10.67 > 10.192.80.254.67: BOOTP/DHCP, Reply, length 323
 14:46:01.890686 IP 10.192.80.254.67 > 10.192.64.10.67: BOOTP/DHCP, Request from 40:a8:f0:2c:31:68, length 300
 14:46:01.890952 IP 10.192.64.10.67 > 10.192.80.254.67: BOOTP/DHCP, Reply, length 323
 14:46:08.511262 IP 10.192.80.254.67 > 10.192.64.10.67: BOOTP/DHCP, Request from 40:a8:f0:2c:31:68, length 300
 14:46:08.511501 IP 10.192.64.10.67 > 10.192.80.254.67: BOOTP/DHCP, Reply, length 323
This worked fine in this case, when Vlan2005 was only configured on Leaf2. We found, however, that when Vlan 2005 was enabled on both switches, and both had the same IP address configured as Anycast GW, it failed. The reason for this is the source IP Leaf2 used to sent the DHCP packets was also configured on Leaf1. So when SRV1 replied to the request Leaf1 tried to process the packet itself, instead of sending it to Leaf2.
Obviously the relayed DHCP packet needs to come from an IP address only configured on the device that sends it, so that the replies go back to the correct device. An Anycast GW IP is thus not suitable, as it is shared on them all. Unfortunately SONiC does not allow a 'secondary', unique, IP to be added in addition to the Anycast GW IP on a Vlan interface. To get around this limitation we created a new loopback interface on Leaf2, assigned it an IP address, and placed it in the VRF:
 dell-leaf2# show running-configuration interface Loopback 2
 !
 interface Loopback 2
 ip vrf forwarding Vrf_codfw
 ip address 1.2.3.4/32
We then added this config to the Vlan interface to tell it to source the DHCP relays from that IP:
 interface Vlan2005 
 description private1-f-codfw
 ip vrf forwarding Vrf_codfw
 ip anycast-address 10.192.80.254/22
 ip dhcp-relay 10.192.64.10 vrf Vrf_codfw
 ip dhcp-relay source-interface Loopback2
With this in place we re-tried the DHCP request from SRV2. We did see the packet come in from source IP 1.2.3.4, but unfortunately the reply went to 10.192.80.254 still:
 root@srv1:/etc/dhcp# tcpdump -i enp59s0f0.2004 -l -nn udp port 67 or udp port 68
 listening on enp59s0f0.2004, link-type EN10MB (Ethernet), capture size 262144 bytes
 15:15:05.556947 IP 1.2.3.4.67 > 10.192.64.10.67: BOOTP/DHCP, Request from 40:a8:f0:2c:31:68, length 302
 15:15:06.558359 IP 10.192.64.10.67 > 10.192.80.254.67: BOOTP/DHCP, Reply, length 323
Further investigation revealed that we could also add this command to the Vlan interface configuration:
 interface Vlan2005 
 ip dhcp-relay 10.192.64.10 vrf Vrf_codfw
 ip dhcp-relay source-interface Loopback2
 ip dhcp-relay link-select
This command caused two additional DHCP Option 82 elements to be added to the packet:
 Option 82 Suboption: (5) Link selection
 Length: 4
 Link selection: 10.192.80.254
 Option 82 Suboption: (11) Server ID Override
 Length: 4
 Server ID Override: 10.192.80.254
Our vanilla ISC DHCPd test server replied to the source IP of the packet correctly when these attributes were present, and the replies went back to Leaf2 correctly and ultimately the end server:
 root@srv1:~# tcpdump -i enp59s0f0.2004 -l -nn udp port 67 or udp port 68
 listening on enp59s0f0.2004, link-type EN10MB (Ethernet), capture size 262144 bytes
 15:12:02.011778 IP 1.2.3.4.67 > 10.192.64.10.67: BOOTP/DHCP, Request from 40:a8:f0:2c:31:68, length 314
 15:12:02.011987 IP 10.192.64.10.67 > 1.2.3.4.67: BOOTP/DHCP, Reply, length 335
So if we use Anycast GW across multiple switches we need to create a switch-specific Loopback interface, each with a unique IP address, and add this as the "ip dhcp-relay source-interface" as well as enable "ip dhcp-relay link-select".
=== Option 82 ===
The SONiC devices do insert Option 82 into DHCP messages they relay, as can be seen in the following snippet from a packet capture:
 Option: (82) Agent Information Option
 Length: 29
 Option 82 Suboption: (1) Agent Circuit ID
 Length: 8
 Agent Circuit ID: 566c616e32303035
 Option 82 Suboption: (2) Agent Remote ID
 Length: 17
 Agent Remote ID: 30303a30303a30303a31303a31303a3130
Decoding these values we see the following information:
 Agent Circuit ID: Vlan2005
 Agent Remote ID: 00:00:00:10:10:10
The remote ID is the MAC address of the Vlan2005 interface on Leaf2 which sourced the packet:
 admin@dell-leaf2:~$ ip -br link show Vlan2005
 Vlan2005@Bridge UP 00:00:00:10:10:10 <BROADCAST,MULTICAST,UP,LOWER_UP> 
Unfortunately the SONiC config does not seem to provide any mechanism to customize what is included in these values. Specifically it does not seem to allow us to include the switch name, and access port ID the DHCP request was received on, which our install process currently uses to identify the server and assign the correct IP.
== IP Filters on Routed interface ==
=== IPv4 ===
=== IPv6 ===
== IP Filters on IRB interface ==
=== IPv4 ===
=== IPv6 ===
== Filter access to RE/CPU/Device Services ==
Evaluate if there is a mechanism like the loopback filter in JunOS, or specific daemon filters (SNMP, vty ACL, NTP acls etc) on Cisco. Basically to prevent remote users trying to SSH to switch or similar. If nothing specific it can be done in-band on the external data ports, but it's trickier to implement.
= Failover Tests =
== Spine Switch Failure ==
== Leaf to Spine Link Failure ==
= Management Tests =
== User Account Creation ==
== SSH Access to Management ==
== SSH Key Auth ==
== Management VRF ==
We should attempt to place the dedicated management port in a specific VRF ("mgmt" for instance).
Ideally all following tests would be configured with this, and all functions would work / have a way to configure them to work when access is in mgmt vrf.
 Interface IPv4 address/mask Master Admin/Oper BGP Neighbor Neighbor IP Flags
 ----------- ------------------- --------- ------------ -------------- ------------- -------
 Ethernet72 172.16.1.6/30 up/up N/A N/A
 Ethernet76 172.16.1.2/30 up/up N/A N/A
 Loopback0 10.0.1.24/32 up/up N/A N/A
 Loopback1 10.10.10.1/32 up/up N/A N/A
 Vlan2004 10.192.64.254/22 Vrf_codfw up/up N/A N/A A
 Vlan2005 10.192.80.254/22 Vrf_codfw down/down N/A N/A A
 docker0 240.127.1.1/24 up/down N/A N/A
 eth0 10.193.0.185/16 mgmt up/up N/A N/A
== SNMP RO Access ==
Works fine, devices added to LibreNMS system which is polling them via SNMP fine.
== NTP ==
Switch was configured to act as an NTP client in the mangement VRF towards our NTP servers:
 dell-leaf2# show running-configuration | grep ntp
 ntp server 208.80.153.77 minpoll 6 maxpoll 10
 ntp server 208.80.154.10 minpoll 6 maxpoll 10
 ntp server 208.80.155.108 minpoll 6 maxpoll 10
 ntp vrf mgmt
Following this NTP sync was ok:
 dell-leaf2# show ntp associations 
 remote refid st t when poll reach delay offset jitter 
 ------------------------------------------------------------------------------------------------------ 
 208.80.153.77 162.159.200.1 4 u 27 64 3 0.299 4.204 0.426 
 *208.80.154.10 170.187.158.81 3 u 40 64 17 31.728 0.097 2.565 
 208.80.155.108 104.171.113.34 3 u 31 64 3 33.109 4.581 0.419 
 ------------------------------------------------------------------------------------------------------
 * master (synced), # master (unsynced), + selected, - candidate, ~ configured
== LLDP == 
=== Server Side ===
Server can see the switch and gets the switch port ok:
 root@srv1:~# lldpcli show neighbors ports eno1
 -------------------------------------------------------------------------------
 LLDP neighbors:
 -------------------------------------------------------------------------------
 Interface: eno1, via: LLDP, RID: 1, Time: 0 day, 00:00:01
 Chassis: 
 ChassisID: mac 3c:2c:30:4b:09:00
 SysName: dell-leaf1
 SysDescr: SONiC Software Version: SONiC.3.4.1-Enterprise_Base - HwSku: DellEMC-S5248f-P-25G-DPB - Distribution: Debian 9.13 - Kernel: 4.9.0-11-2-amd64
 MgmtIP: 10.193.0.185
 Capability: Bridge, off
 Capability: Router, on
 Capability: Wlan, off
 Capability: Station, on
 Port: 
 PortID: local Ethernet32
 PortDescr: test_srv1_1GTBase
 TTL: 120
 -------------------------------------------------------------------------------
===Switch Side===
The switch can also see the server details:
 dell-leaf1# show lldp neighbor Ethernet 32
 -----------------------------------------------------------
 LLDP Neighbors 
 -----------------------------------------------------------
 Interface: Ethernet32,via: LLDP
 Chassis:
 ChassisID: d0:94:66:86:4d:6c
 SysName: srv1
 SysDescr: Ubuntu 18.04.6 LTS Linux 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64
 TTL: 120
 MgmtIP: 1.1.1.1
 MgmtIP: 2620:0:861:11c::10
 Port
 PortID: d0:94:66:86:4d:6c
 PortDescr: eno1
 -----------------------------------------------------------
===Per-interface LLDP control===
The command "no lldp enable" was added to the switch config for Ethernet32, after a short time the switch details were no longer visible from the server:
 root@srv1:~# lldpcli show neighbors ports eno1
 -------------------------------------------------------------------------------
 LLDP neighbors:
 -------------------------------------------------------------------------------
 root@srv1:~# 
Same goes for the switch-side:
 
 dell-leaf1# show lldp neighbor Ethernet 32
 -----------------------------------------------------------
 LLDP Neighbors 
 -----------------------------------------------------------
 dell-leaf1#
== sFlow Export ==
Set up sFlow export. At a minimum I guess we could just capture the packets with tcpdump and compare to what the Junipers send. We could also try to set up pmacct somewhere or something like that to validate the flow data is ok.
== Prometheus Export ==
Set up telegraf etc. as per their guide. We can test with curl don't need to actually set up our Prometheus to scrape it.
== Puppet Agent ==
It would be interesting to test the puppet agent compatibility. We may not go down that road but good to know.
= Automation Tests =
== RESTCONF ==
Basic tests to make sure we can talk to the interface and apply config.
== Partial config replace ==
i.e. validate we can do a replace on a specific section, i.e. "bgp", without touching entire config.