Довольно популярным запросом у заказчиков стало наращивание существующей фабрики Cisco коммутаторами Maipu. Недавно нам удалось наконец собрать необходимый пул оборудования и разобраться, что может и чего не может оборудование Maipu в таких кейсах. Ранее, кстати, рассказал, как мы расширяем существующую DMVPN-сеть Cisco с помощью Maipu, и про коммутаторы Maipu для сетей ЦОД. Ограничение, которое мы увидели сразу - невозможность реализовать схему Multisite, которую предлагает Cisco. В этой схеме существует специальная роль коммутаторов -- Border Gateway. Эти коммутаторы терминируют VXLAN-туннели внутри фабрики и создают новые туннели до других дата-центров вот так: Такой подход позволяет ограничить распространение информации, связанной с работой Control Plane, что позволяет лучше масштабировать сети датацентров и сузить домены отказа. При этом между датацентрами работает и L2, и L3. Функционал основан на нескольких драфтах RFC, и пока не очень распространён. Что-то подобное умеет делать ещё Arista, но на Maipu данная схема недоступна. Поэтому мы пробуем классический Multipod - полный обмен маршрутами между двумя наборами Leaf/Spine-коммутаторов через EBGP. Другой вариант - объединение фабрик через L3, но сейчас не об этом.  Схема, которую нам удалось собрать из имеющегося оборудования, выглядит так: Второе заметное ограничение - отсутствие на Maipu возможности распространять Broadcast/Unknown Unicast-трафик с помощью PIM и Multicast. Используем BGP Ingress Replication на обоих подах.  На этом грустная часть истории заканчивается и начинается интересная. Фабрику настраиваем так же, как и в предыдущей серии: в Underlay OSPF и IBGP. Такой вариант мы чаще всего встречаем у заказчиков, эксплуатирующих фабрику на Cisco. Хосты сразу подключаем к двум Leaf-коммутаторам, это VPC и MLAG соответственно. Между Spine-коммутаторами, где у нас, собственно, и происходит соединение двух фабрик, настраиваем EBGP. Для того, чтобы VXLAN-туннели корректно строились от одного Leaf-коммутатора до другого, сохраняем значение атрибута Next-Hop в передаваемых EVPN-маршрутах. Конфиги Leaf-коммутаторов Cisco Nexus: N9k-Leaf1 N9k-Leaf2 cfs eth   distribute nv overlay evpn feature ospf feature bgp feature pim feature fabric forwarding feature interface-vlan feature vn-segment-vlan-based feature hsrp feature lacp feature vpc feature lldp feature bfd feature nv overlay   fabric forwarding anycast-gateway-mac   0001.0001.0100   vlan 1,41-42,999 vlan 41     vn-segment 100 vlan 42     vn-segment 200 vlan 999     vn-segment 1   route-map DIRECT-BGP-RM permit 10   match   tag 12345 vrf context Keepalive   vrf context vxlan-100   vni 1   rd   100:42     address-family ipv4 unicast       route-target import 100:2       route-target import 100:2 evpn       route-target export 100:2       route-target export 100:2 evpn vpc domain 1     peer-keepalive destination 10.253.1.5 source 10.253.1.4 vrf Keepalive   interface Vlan41   no   shutdown   vrf   member vxlan-100   ip   address 172.16.129.1/24     fabric forwarding mode anycast-gateway   interface Vlan42   no   shutdown   vrf   member vxlan-100   ip   address 172.16.130.1/24     fabric forwarding mode anycast-gateway   interface Vlan999   no   shutdown   vrf   member vxlan-100   ip   forward   interface port-channel30     switchport     switchport mode trunk     switchport trunk allowed vlan 41-42   vpc 1   interface port-channel54     switchport     switchport mode trunk     spanning-tree port type network   vpc   peer-link   interface nve1   no   shutdown     host-reachability protocol bgp     source-interface loopback101     member vni 1 associate-vrf     member vni 100       ingress-replication protocol bgp     member vni 200       ingress-replication protocol bgp   interface Ethernet1/1     switchport     switchport mode trunk     switchport trunk allowed vlan 41-42     channel-group 30 mode active   no   shutdown   interface Ethernet1/48     description -= Peer Keepalive Link =-   vrf   member Keepalive   ip   address 10.253.1.4/31   no   shutdown   interface Ethernet1/53     description -= Spine =-   ip   address 10.253.1.1/31   ip   ospf network point-to-point   ip   router ospf 1 area 0.0.0.0   ip pim   sparse-mode   no   shutdown   interface Ethernet1/54     description -= Peer Link =-     switchport     switchport mode trunk     channel-group 54 mode active   no   shutdown   interface loopback0   ip   address 10.253.127.1/32   ip   router ospf 1 area 0.0.0.0   interface loopback101   ip   address 10.253.127.253/32   ip   address 10.253.127.40/32 secondary   ip   router ospf 1 area 0.0.0.0   router ospf 1     router-id 10.253.127.1 router bgp 200   address-family l2vpn evpn     neighbor 10.253.127.3       remote-as 200       update-source loopback0       address-family ipv4 unicast         advertisement-interval 1       address-family l2vpn evpn         send-community         send-community extended   vrf   vxlan-100       address-family ipv4 unicast         network 172.16.129.0/24         network 172.16.130.0/24         advertise l2vpn evpn         redistribute direct route-map DIRECT-BGP-RM evpn   vni   100 l2             rd   100:40       route-target import 100:1       route-target export 100:1   vni   200 l2             rd   200:40       route-target import 200:1     route-target   export 200:1 cfs eth   distribute nv overlay evpn feature ospf feature bgp feature pim feature fabric forwarding feature interface-vlan feature vn-segment-vlan-based feature hsrp feature lacp feature vpc feature lldp feature bfd feature nv overlay   fabric forwarding anycast-gateway-mac   0001.0001.0100   vlan 1,41-42,999 vlan 41     vn-segment 100 vlan 42     vn-segment 200 vlan 999     vn-segment 1   route-map DIRECT-BGP-RM permit 10   match   tag 12345 vrf context Keepalive   vrf context vxlan-100   vni 1   rd   100:42     address-family ipv4 unicast       route-target import 100:2       route-target import 100:2 evpn       route-target export 100:2       route-target export 100:2 evpn vpc domain 1     peer-keepalive destination 10.253.1.4 source 10.253.1.5 vrf Keepalive   interface Vlan41   no   shutdown   vrf   member vxlan-100   ip   address 172.16.129.1/24     fabric forwarding mode anycast-gateway   interface Vlan42   no   shutdown   vrf   member vxlan-100   ip   address 172.16.130.1/24     fabric forwarding mode anycast-gateway   interface Vlan999   no   shutdown   vrf   member vxlan-100   ip   forward   interface port-channel30     switchport     switchport mode trunk     switchport trunk allowed vlan 41-42   vpc 1   interface port-channel54     switchport     switchport mode trunk     spanning-tree port type network   vpc   peer-link   interface nve1   no   shutdown     host-reachability protocol bgp     source-interface loopback101     member vni 1 associate-vrf     member vni 100       ingress-replication protocol bgp     member vni 200       ingress-replication protocol bgp   interface Ethernet1/1     switchport     switchport mode trunk     switchport trunk allowed vlan 41-42     channel-group 30 mode active   no   shutdown   interface Ethernet1/48     description -= Peer Keepalive Link =-   vrf   member Keepalive   ip   address 10.253.1.5/31   no   shutdown   interface Ethernet1/53     description -= Spine =-   ip   address 10.253.1.3/31   ip   ospf network point-to-point   ip   router ospf 1 area 0.0.0.0   ip   pim sparse-mode   no   shutdown   interface Ethernet1/54     description -= Peer Link =-     switchport     switchport mode trunk     channel-group 54 mode active   no   shutdown   interface loopback0   ip   address 10.253.127.2/32   ip   router ospf 1 area 0.0.0.0   interface loopback101   ip   address 10.253.127.254/32   ip   address 10.253.127.40/32 secondary   ip   router ospf 1 area 0.0.0.0   router ospf 1     router-id 10.253.127.2 router bgp 200   address-family l2vpn evpn     neighbor 10.253.127.3       remote-as 200       update-source loopback0       address-family ipv4 unicast         advertisement-interval 1       address-family l2vpn evpn         send-community         send-community extended   vrf   vxlan-100       address-family ipv4 unicast         network 172.16.129.0/24         network 172.16.130.0/24       advertise l2vpn evpn         redistribute direct route-map DIRECT-BGP-RM evpn   vni   100 l2             rd   100:40       route-target import 100:1       route-target export 100:1   vni   200 l2             rd   200:40       route-target import 200:1       route-target export 200:1 Spine Nexus: N9k-Spine nv   overlay evpn feature ospf feature bgp feature pim feature fabric forwarding feature interface-vlan feature vn-segment-vlan-based feature hsrp feature lacp feature lldp feature bfd feature nv overlay   vlan 1   route-map BGP-OUT-RM permit 10   set   ip next-hop unchanged   vrf context vxlan-100   vni 1   rd   100:4     address-family ipv4 unicast       route-target import 100:2       route-target export 100:2   interface Ethernet1/48   ip   address 11.0.0.2/24   ip   router ospf 1 area 0.0.0.0   no   shutdown   interface Ethernet1/49     description -= External Port =-   ip address 10.254.100.3/24   ip   ospf mtu-ignore   ip   router ospf 1 area 0.0.0.0   no   shutdown   interface Ethernet1/53     description -= Leaf 1 =-   ip   address 10.253.1.0/31   ip   ospf network point-to-point   ip   router ospf 1 area 0.0.0.0   ip   pim sparse-mode   no   shutdown  interface Ethernet1/54     description -= Leaf 2 =-   ip   address 10.253.1.2/31   ip   ospf network point-to-point   ip   router ospf 1 area 0.0.0.0   ip   pim sparse-mode   no   shutdown   interface loopback0   ip   address 10.253.127.3/32   ip   router ospf 1 area 0.0.0.0   router ospf 1     router-id 10.253.127.3   router bgp 200     neighbor 10.253.127.1       remote-as 200       update-source loopback0       address-family ipv4 unicast         advertisement-interval 1       address-family l2vpn evpn         send-community         send-community extended     neighbor 10.253.127.2       remote-as 200       update-source loopback0       address-family ipv4 unicast         advertisement-interval 1       address-family l2vpn evpn         send-community         send-community extended     neighbor 10.254.127.1       remote-as 100       update-source loopback0       ebgp-multihop 5       address-family ipv4 unicast         advertisement-interval 1       address-family l2vpn evpn         send-community         send-community extended         route-map BGP-OUT-RM out     neighbor 10.254.127.2       remote-as 100       update-source loopback0       ebgp-multihop 5       address-family ipv4 unicast         advertisement-interval 1       address-family l2vpn evpn         send-community         send-community extended         route-map BGP-OUT-RM out   vrf   vxlan-100       address-family ipv4 unicast         advertise l2vpn evpn evpn   vni   100 l2             rd   100:3       route-target import 100:1       route-target export 100:1   vni   200 l2             rd   200:3       route-target import 200:1     route-target export 200:1 Leaf Maipu: Maipu-L11 Maipu-L12 ip vrf   100  rd   100:2  l3vnid   1  address-family evpn   route-target import 100:2 ipv4     route-target export 100:2 ipv4   exit  exit   link-aggregation 64 mode lacp link-aggregation 250 mode lacp   vlan 1  exit   vlan 11-13   mlag domain 1  node   id 1  node   role-priority 50  role   preempt  system-mac 0001.7a95.000b  keepalive ip destination 192.168.168.41   source 192.168.168.40  exit   vxlan 100  vxlan   vnid 100  address-family evpn   rd   100:10     route-target import 100:1     route-target export 100:1   exit  exit vxlan 200  vxlan   vnid 200  address-family evpn   rd   200:10     route-target import 200:1     route-target export 200:1   exit  exit     interface tengigabitethernet0/48  link-aggregation 64 active  exit interface 100ge0/49  no   switchport  ip   address 10.254.1.1 255.255.255.254  no ip   redirects  ip   ospf network point-to-point  ip   ospf bfd  ip mtu   exception forward  exit interface 100ge0/50  no   switchport  ip   address 10.254.2.1 255.255.255.254  no ip   redirects  ip   ospf network point-to-point  ip   ospf bfd  ip mtu   exception forward  exit interface 100ge0/55  fec   mode rs auto  link-aggregation 250 active  exit interface 100ge0/56  fec   mode rs auto  link-aggregation 250 active  exit !end   interface link-aggregation64  switchport mode trunk  switchport trunk allowed vlan add 1,11-12  switchport trunk pvid vlan 1  vxlan   100 encapsulation vlan 11  vxlan   200 encapsulation vlan 12  mlag   group 1  exit interface link-aggregation250  switchport mode trunk  switchport trunk allowed vlan add 11-13  switchport trunk pvid vlan 1  mlag   peer-link  exit   interface dc0  ip   address 192.168.168.40 255.255.0.0  exit   interface loopback1  description Router-ID  ip   address 10.254.127.11 255.255.255.255  exit   interface loopback2  description VTEP  ip   address 10.254.128.11 255.255.255.255  ip   address 10.254.128.10 255.255.255.255 secondary  exit     interface loopback100  ip vrf   forwarding 100  ip   address 172.16.250.11 255.255.255.255  exit   interface vxlan100  ip vrf   forwarding 100  vxlan   distribute-gateway  ip   address 172.16.129.1 255.255.255.0  mac-address 0001.0001.0100  exit   interface vxlan200  ip vrf   forwarding 100  vxlan   distribute-gateway  ip   address 172.16.130.1 255.255.255.0  mac-address 0001.0001.0100  exit   interface nve1  source   10.254.128.10  mac-address 0088.9999.1112  vxlan   100,200 ingress-replication protocol bgp  exit   router ospf 100  router-id 10.254.127.11  bfd all-interfaces  network 10.254.1.1 0.0.0.0 area 0  network 10.254.2.1 0.0.0.0 area 0  network 10.254.127.11 0.0.0.0 area 0  network 10.254.128.10 0.0.0.0 area 0  maximum-paths 8  exit   router bgp 100  no   auto-summary  no   synchronization  bgp   router-id 10.254.127.11  neighbor spine peer-group  neighbor spine remote-as 100  neighbor spine update-source loopback1  neighbor 10.254.127.1 peer-group spine  neighbor 10.254.127.2 peer-group spine  maximum-paths ibgp 8  address-family l2vpn evpn     neighbor spine activate     neighbor spine send-community both     neighbor 10.254.127.1 activate     neighbor 10.254.127.1 send-community both     neighbor 10.254.127.2 activate     neighbor 10.254.127.2 send-community both     exit-address-family  address-family ipv4 vrf 100     advertise-l2vpn-evpn     network 172.16.129.0 255.255.255.0     network 172.16.130.0 255.255.255.0     network 172.16.250.11 255.255.255.255     network 172.16.250.12 255.255.255.255     exit-address-family  exit ip vrf   100  rd   100:2  l3vnid   1  address-family evpn   route-target import 100:2 ipv4     route-target export 100:2 ipv4   exit  exit   link-aggregation 64 mode lacp link-aggregation 250 mode lacp   vlan 1  exit   vlan 11-13   mlag domain 1  node   id 2  system-mac 0001.7a95.000b  keepalive ip destination 192.168.168.40   source 192.168.168.41  exit       vxlan 100  vxlan   vnid 100  address-family evpn   rd   100:10     route-target import 100:1     route-target export 100:1   exit  exit vxlan 200  vxlan   vnid 200  address-family evpn   rd   200:10     route-target import 200:1     route-target export 200:1   exit  exit     interface tengigabitethernet0/48  link-aggregation 64 active  exit interface 100ge0/49  no   switchport  ip   address 10.254.1.3 255.255.255.254  no ip   redirects  ip   ospf network point-to-point  ip   ospf bfd  ip mtu   exception forward  exit interface 100ge0/50  no   switchport  ip   address 10.254.2.3 255.255.255.254  no ip   redirects  ip   ospf network point-to-point  ip   ospf bfd  ip mtu   exception forward  exit interface 100ge0/55  fec   mode rs auto  link-aggregation 250 active  exit interface 100ge0/56  fec   mode rs auto  link-aggregation 250 active  exit !end   interface link-aggregation64  switchport mode trunk  switchport trunk allowed vlan add 1,11-12  switchport trunk pvid vlan 1  vxlan   100 encapsulation vlan 11  vxlan   200 encapsulation vlan 12  mlag   group 1  exit interface link-aggregation250  switchport mode trunk  switchport trunk allowed vlan add 11-13  switchport trunk pvid vlan 1  mlag   peer-link  exit   interface dc0  ip   address 192.168.168.41 255.255.0.0  exit   interface loopback1  description Router-ID  ip   address 10.254.127.12 255.255.255.255  exit   interface loopback2  description VTEP  ip   address 10.254.128.12 255.255.255.255  ip   address 10.254.128.10 255.255.255.255 secondary  exit     interface loopback100  ip vrf   forwarding 100  ip   address 172.16.250.12 255.255.255.255  exit   interface vxlan100  ip vrf   forwarding 100  vxlan   distribute-gateway  ip   address 172.16.129.1 255.255.255.0  mac-address 0001.0001.0100  exit   interface vxlan200  ip vrf   forwarding 100  vxlan   distribute-gateway  ip   address 172.16.130.1 255.255.255.0  mac-address 0001.0001.0100  exit   interface nve1  source   10.254.128.10  mac-address 0088.9999.1112  vxlan   100,200 ingress-replication protocol bgp  exit   router ospf 100  router-id 10.254.127.12  bfd all-interfaces  network 10.254.1.3 0.0.0.0 area 0  network 10.254.2.3 0.0.0.0 area 0  network 10.254.127.12 0.0.0.0 area 0  network 10.254.128.10 0.0.0.0 area 0  maximum-paths 8  exit   router bgp 100  no   auto-summary  no   synchronization  bgp   router-id 10.254.127.12  neighbor spine peer-group  neighbor spine remote-as 100  neighbor spine update-source loopback1  neighbor 10.254.127.1 peer-group spine  neighbor 10.254.127.2 peer-group spine  maximum-paths ibgp 8  address-family l2vpn evpn     neighbor spine activate     neighbor spine send-community both     neighbor 10.254.127.1 activate     neighbor 10.254.127.1 send-community both     neighbor 10.254.127.2 activate     neighbor 10.254.127.2 send-community both     exit-address-family  address-family ipv4 vrf 100     advertise-l2vpn-evpn     network 172.16.129.0 255.255.255.0     network 172.16.130.0 255.255.255.0     network 172.16.250.11 255.255.255.255     network 172.16.250.12 255.255.255.255     exit-address-family  exit Spine Maipu: Maipu-S1 Maipu-S2 ip vrf   100  rd   100:3  l3vnid   1  address-family evpn     route-target import 100:2 ipv4     route-target export 100:2 ipv4   exit  exit   vxlan 100  vxlan   vnid 100  address-family evpn   rd   100:1     route-target import 100:1     route-target export 100:1   exit  exit vxlan 200  vxlan   vnid 200  address-family evpn   rd   200:1     route-target import 200:1     route-target export 200:1   exit  exit   interface 100ge0/1  no   switchport  ip   address 10.254.1.0 255.255.255.254  no ip   redirects  ip   ospf network point-to-point  ip   ospf bfd  ip mtu   exception forward  exit   interface 100ge0/2  no   switchport  ip   address 10.254.1.2 255.255.255.254  no ip   redirects  ip   ospf network point-to-point  ip   ospf bfd  ip mtu   exception forward  exit   interface 100ge0/6  no   switchport  ip   address 10.254.1.4 255.255.255.254  no ip   redirects  ip   ospf network point-to-point  ip   ospf bfd  ip mtu   exception forward  exit   interface 100ge0/10  no   switchport  ip   address 10.254.1.6 255.255.255.254  no ip   redirects  ip   ospf network point-to-point  ip   ospf bfd  ip mtu   exception forward  exit   interface 100ge0/14  no   switchport  ip   address 10.254.100.1 255.255.255.0  ip   ospf mtu-ignore  exit !end   interface loopback1  description Router-ID  ip   address 10.254.127.1 255.255.255.255  exit   router ospf 100  router-id 10.254.127.1  bfd all-interfaces  network 10.254.1.0 0.0.0.0 area 0  network 10.254.1.2 0.0.0.0 area 0  network 10.254.1.4 0.0.0.0 area 0  network 10.254.1.6 0.0.0.0 area 0  network 10.254.100.0 0.0.0.255 area 0  network 10.254.127.1 0.0.0.0 area 0  exit   router bgp 100  no   auto-summary  no   synchronization  neighbor leaf peer-group  neighbor leaf remote-as 100  neighbor leaf update-source loopback1  neighbor leaf advertisement-interval 0  neighbor 10.253.127.3 remote-as 200  neighbor 10.253.127.3 ebgp-multihop 10  neighbor 10.253.127.3 update-source loopback1  neighbor 10.253.127.3 advertisement-interval   0  neighbor 10.253.127.3 attribute-unchanged  neighbor 10.254.127.11 peer-group leaf  neighbor 10.254.127.12 peer-group leaf  neighbor 10.254.127.21 peer-group leaf  address-family l2vpn evpn     neighbor 10.253.127.3 activate     neighbor 10.253.127.3 send-community both     neighbor 10.253.127.3 attribute-unchanged     neighbor 10.254.127.11 activate     neighbor 10.254.127.11 route-reflector-client     neighbor 10.254.127.11 send-community both     neighbor 10.254.127.12 activate     neighbor 10.254.127.12 route-reflector-client     neighbor 10.254.127.12 send-community both     neighbor 10.254.127.21 activate     neighbor 10.254.127.21 route-reflector-client     neighbor 10.254.127.21 send-community both     exit-address-family  address-family ipv4 vrf 100     advertise-l2vpn-evpn   exit-address-family  exit ip vrf   100  rd   100:3  l3vnid   1  address-family evpn     route-target import 100:2 ipv4     route-target export 100:2 ipv4   exit  exit   vxlan 100  vxlan   vnid 100  address-family evpn   rd   100:2     route-target import 100:1     route-target export 100:1   exit  exit vxlan 200  vxlan   vnid 200  address-family evpn   rd   200:2     route-target import 200:1     route-target export 200:1   exit  exit   interface 100ge0/1  no   switchport  ip   address 10.254.2.0 255.255.255.254  no ip   redirects  ip   ospf network point-to-point  ip   ospf bfd  ip mtu   exception forward  exit   interface 100ge0/2  no   switchport  ip   address 10.254.2.2 255.255.255.254  no ip   redirects  ip   ospf network point-to-point  ip   ospf bfd  ip mtu   exception forward  exit   interface 100ge0/6  no   switchport  ip   address 10.254.2.4 255.255.255.254  no ip   redirects  ip   ospf network point-to-point  ip   ospf bfd  ip mtu   exception forward  exit   interface 100ge0/10  no   switchport  ip   address 10.254.2.6 255.255.255.254  no ip   redirects  ip   ospf network point-to-point  ip   ospf bfd  ip mtu   exception forward  exit   interface 100ge0/14  no   switchport  ip   address 10.254.100.2 255.255.255.0  ip   ospf mtu-ignore  exit !end   interface loopback1  description Router-ID  ip   address 10.254.127.2 255.255.255.255  exit   router ospf 100  router-id 10.254.127.2  bfd all-interfaces  network 2.2.2.2 0.0.0.0 area 0  network 10.254.2.0 0.0.0.0 area 0  network 10.254.2.2 0.0.0.0 area 0  network 10.254.2.4 0.0.0.0 area 0  network 10.254.100.0 0.0.0.255 area 0  network 10.254.127.2 0.0.0.0 area 0  exit   router bgp 100  no   auto-summary  no   synchronization  neighbor leaf peer-group  neighbor leaf remote-as 100  neighbor leaf update-source loopback1  neighbor leaf advertisement-interval 0  neighbor 10.253.127.3 remote-as 200  neighbor 10.253.127.3 ebgp-multihop 10  neighbor 10.253.127.3 update-source   loopback1  neighbor 10.253.127.3 advertisement-interval   0  neighbor 10.253.127.3 attribute-unchanged  neighbor 10.254.127.11 peer-group leaf  neighbor 10.254.127.12 peer-group leaf  neighbor 10.254.127.21 peer-group leaf  address-family l2vpn evpn     neighbor 10.253.127.3 activate     neighbor 10.253.127.3 send-community both     neighbor 10.253.127.3 attribute-unchanged     neighbor 10.254.127.11 activate     neighbor 10.254.127.11 route-reflector-client     neighbor 10.254.127.11 send-community both     neighbor 10.254.127.12 activate     neighbor 10.254.127.12 route-reflector-client     neighbor 10.254.127.12 send-community both     neighbor 10.254.127.21 activate     neighbor 10.254.127.21 route-reflector-client     neighbor 10.254.127.21 send-community both     exit-address-family  address-family ipv4 vrf 100     advertise-l2vpn-evpn   exit-address-family  exit Хосты друг друга пингуют, прочий трафик тоже ходит. Честно говоря, в этом мы особенно и не сомневались. Опасения вызывало то, как поймут друг друга фабрики, в которых присутствует с одной стороны VPC, а с другой Multi-Chassis LAG. Подёргав интерфейсы с каждой из сторон, убедились в том, что всё работает корректно. Впрочем, на случай проблем с MLAG, на коммутаторах Maipu серии ЦОД доступно стекирование.   Вот так выглядят маршруты на Nexus: N9k-Leaf1#   sh bgp l2vpn evpn BGP routing table   information for VRF default, address family L2VPN EVPN BGP table version   is 1286, Local Router ID is 10.253.127.1 Status:   s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best Path type:   i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected Origin codes: i -   IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2      Network         Next   Hop          Metric   LocPrf     Weight Path Route Distinguisher:   100:1 *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[0]:[0.0.0.0]/216                       10.254.128.20                0          100       0   100 i *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.129.1]/272                       10.254.128.20                0          100       0   100 i *>i[3]:[0]:[32]:[10.254.128.20]/88                       10.254.128.20                0          100       0   100 i   Route   Distinguisher: 100:10 *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[0]:[0.0.0.0]/216                       10.254.128.10                0          100       0   100 i *>i[2]:[0]:[0]:[48]:[dc4a.3e3b.cbf8]:[0]:[0.0.0.0]/216                       10.254.128.10                0                100          0 100 i *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.129.1]/272                       10.254.128.10                0          100       0   100 i *>i[2]:[0]:[0]:[48]:[dc4a.3e3b.cbf8]:[32]:[172.16.129.11]/272                       10.254.128.10                0          100       0   100 i *>i[3]:[0]:[32]:[10.254.128.10]/88                       10.254.128.10                0          100       0   100 i   Route Distinguisher:   100:40      (L2VNI 100) *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[0]:[0.0.0.0]/216                       10.254.128.10                0          100       0   100 i * i                   10.254.128.20                0          100       0   100 i *>i[2]:[0]:[0]:[48]:[dc4a.3e3b.cbf8]:[0]:[0.0.0.0]/216                       10.254.128.10                0          100       0   100 i *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.129.1]/272                       10.254.128.10                0          100          0 100 i * i                   10.254.128.20                0          100       0   100 i *>i[2]:[0]:[0]:[48]:[dc4a.3e3b.cbf8]:[32]:[172.16.129.11]/272                       10.254.128.10                0          100       0   100 i *>l[3]:[0]:[32]:[10.253.127.40]/88                       10.253.127.40                     100              32768   i *>i[3]:[0]:[32]:[10.254.128.10]/88                       10.254.128.10                0          100       0   100 i *>i[3]:[0]:[32]:[10.254.128.20]/88                       10.254.128.20                0          100       0   100 i   Route   Distinguisher: 200:10 *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[0]:[0.0.0.0]/216                       10.254.128.10                0          100       0   100 i *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.130.1]/272                       10.254.128.10                0          100       0   100 i *>i[3]:[0]:[32]:[10.254.128.10]/88                       10.254.128.10                0          100       0   100 i   Route   Distinguisher: 200:40      (L2VNI 200) *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[0]:[0.0.0.0]/216                       10.254.128.10                0          100       0   100 i *>l[2]:[0]:[0]:[48]:[dc4a.3e3b.cbe9]:[0]:[0.0.0.0]/216                       10.253.127.40                     100              32768   i *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.130.1]/272                       10.254.128.10                0          100       0   100 i *>l[2]:[0]:[0]:[48]:[dc4a.3e3b.cbe9]:[32]:[172.16.130.11]/272                       10.253.127.40                     100              32768   i *>l[3]:[0]:[32]:[10.253.127.40]/88                       10.253.127.40                     100              32768 i *>i[3]:[0]:[32]:[10.254.128.10]/88                       10.254.128.10                0          100       0   100 i   Route   Distinguisher: 100:42      (L3VNI 1) *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[0]:[0.0.0.0]/216                       10.254.128.10                0          100       0   100 i *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.129.1]/272                       10.254.128.10                0          100       0   100 i * i                10.254.128.20                    0          100       0   100 i *>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.130.1]/272                       10.254.128.10                0          100       0   100 i *>i[2]:[0]:[0]:[48]:[dc4a.3e3b.cbf8]:[32]:[172.16.129.11]/272                       10.254.128.10                  0          100       0   100 i *>l[5]:[0]:[0]:[24]:[172.16.129.0]/224                         10.253.127.40                       100              32768 i *>l[5]:[0]:[0]:[24]:[172.16.130.0]/224                         10.253.127.40                             100       32768   i А вот вывод с Maipu: L11#show bgp l2vpn evpn all   all-type BGP local router ID is   10.254.127.11   Status codes: s suppressed, d   damped, h history, * valid, > best, i - internal,             S   Stale Origin codes: i - IGP, e - EGP, ?   - incomplete EVPN Information for Route   Distinguisher:100:1  MAC/IP Advertisement Routes:               Network(ETID:MAC:IP)                                Next Hop                    Metric    LocPrf Weight Path [B]*>i0:48:0001.0001.0100:0:0.0.0.0/96                   10.254.128.20            0            100      0   i [B]* i                                                   10.254.128.20            0            100      0   i [B]*>i0:48:0001.0001.0100:32:172.16.129.1/128                 10.254.128.20            0          100              0   i [B]* i                                                     10.254.128.20            0          100      0   i  Inclusive Multicast Ethernet Tag Routes:               Network(Originating IP   Addr)                        Next Hop                    Metric    LocPrf Weight Path [B]*>i0:32:10.254.128.20/72                                     10.254.128.20            0          100      0   i [B]* i                                                   10.254.128.20            0          100      0   i EVPN Information for Route   Distinguisher:100:2  MAC/IP Advertisement Routes:               Network(ETID:MAC:IP)                                Next Hop                    Metric    LocPrf Weight Path [B]*> 0:48:0001.0001.0100:32:172.16.129.1/128                 0.0.0.0                        0                     32768 i [B]* i                                                     10.254.128.20            0          100      0   i [B]*>   0:48:dc4a.3e3b.cbf8:32:172.16.129.11/128                0.0.0.0                        0                     32768 i [B]*>   0:48:0001.0001.0100:32:172.16.130.1/128                 0.0.0.0                        0                     32768 i [B]*>i0:48:dc4a.3e3b.cbe9:32:172.16.130.11/128               10.253.127.40            0          100              0   200 i  IP Prefix Routes:               Network(ETID:IP)                                         Next Hop                    Metric    LocPrf Weight Path [B]*> 0:24:172.16.129.0/72                                       0.0.0.0                        0          32768 i [B]*> 0:24:172.16.130.0/72                                       0.0.0.0                        0          32768 i EVPN Information for Route   Distinguisher:100:10  MAC/IP Advertisement Routes:               Network(ETID:MAC:IP)                                Next Hop                    Metric    LocPrf Weight Path [B]*>   0:48:0001.0001.0100:0:0.0.0.0/96                   0.0.0.0                        0          32768   i [B]* i                                                     10.254.128.20            0          100      0   i [B]*>   0:48:dc4a.3e3b.cbf8:0:0.0.0.0/96                    0.0.0.0                        0          32768   i [B]*> 0:48:0001.0001.0100:32:172.16.129.1/128                 0.0.0.0                        0                     32768 i [B]* i                                                     10.254.128.20            0          100      0   i [B]*>   0:48:dc4a.3e3b.cbf8:32:172.16.129.11/128                0.0.0.0                        0                     32768 i  Inclusive Multicast Ethernet Tag Routes:               Network(Originating IP   Addr)                        Next Hop                    Metric    LocPrf Weight Path [B]*>i0:32:10.253.127.40/72                                     10.253.127.40            0          100      0   200 i [B]*> 0:32:10.254.128.10/72                                     0.0.0.0                        0          32768 i [B]*>i0:32:10.254.128.20/72                                     10.254.128.20            0          100      0   i EVPN Information for Route   Distinguisher:100:40  Inclusive Multicast Ethernet Tag Routes:               Network(Originating IP   Addr)                        Next Hop                    Metric    LocPrf Weight Path [B]* i0:32:10.253.127.40/72                                       10.253.127.40            0          100      0   200 i [B]*>i                                                              10.253.127.40            0          100      0   200 i EVPN Information for Route   Distinguisher:200:10  MAC/IP Advertisement Routes:               Network(ETID:MAC:IP)                                Next Hop                    Metric    LocPrf Weight Path [B]*>   0:48:0001.0001.0100:0:0.0.0.0/96                   0.0.0.0                        0          32768   i [B]*>i0:48:dc4a.3e3b.cbe9:0:0.0.0.0/96                   10.253.127.40            0            100      0   200 i [B]*                                                       0.0.0.0                        0          32768 i [B]*>i0:48:dc4a.3e3b.cbf8:0:0.0.0.0/96                    10.253.127.40            0            100      0   200 i [B]*>   0:48:0001.0001.0100:32:172.16.130.1/128                 0.0.0.0                        0                     32768 i [B]*>i0:48:dc4a.3e3b.cbe9:32:172.16.130.11/128               10.253.127.40            0          100              0   200 i  Inclusive Multicast Ethernet Tag Routes:               Network(Originating IP   Addr)                        Next Hop                    Metric    LocPrf Weight Path [B]*>i0:32:10.253.127.40/72                                     10.253.127.40            0            100      0   200 i [B]*>   0:32:10.254.128.10/72                                     0.0.0.0                        0          32768 i EVPN Information for Route   Distinguisher:200:40  MAC/IP Advertisement Routes:               Network(ETID:MAC:IP)                                Next Hop                    Metric    LocPrf Weight Path [B]*>i0:48:dc4a.3e3b.cbe9:0:0.0.0.0/96                   10.253.127.40            0            100      0   200 i [B]* i                                                     10.253.127.40            0          100      0   200 i [B]*   i0:48:dc4a.3e3b.cbf8:0:0.0.0.0/96                     10.253.127.40            0            100      0   200 i [B]*>i                                                                10.253.127.40            0          100      0   200 i [B]*>i0:48:dc4a.3e3b.cbe9:32:172.16.130.11/128               10.253.127.40            0          100              0   200 i [B]* i                                                     10.253.127.40            0          100      0   200 i  Inclusive Multicast Ethernet Tag Routes:               Network(Originating IP   Addr)                        Next Hop                    Metric    LocPrf Weight Path [B]* i0:32:10.253.127.40/72                                       10.253.127.40            0          100      0   200 i [B]*>i                                                                10.253.127.40            0          100      0   200 i  Проверили доступность хостов и по L2, при помещении их в один VXLAN-сегмент. Главный вывод - мы не напоролись ни на какой подводный камень. Получили работающий Asymmetric Routing & Bridging. Фабрику Cisco можно нарастить подами Maipu, пусть и при определённых ограничениях.  Хорошим продолжением было бы тестирование такой схемы на пилотном сегменте реальной сети так, чтобы подключить побольше хостов разных типов, получить много MAC-адресов, маршрутов и реальный трафик.  На момент написания поста мы уже получили в лабораторию набор оборудования Maipu, включающий коммутаторы линейки ЦОД и готовы тестировать более конкретные, жизненные кейсы для наших заказчиков. Кроме того, 18 октября у нас пройдёт онлайн-митап, на котором мы расскажем о доступных сегодня в России альтернативах и на примере Maipu продемонстрируем, как они справляются с типовыми кейсами заказчиков.    Эксперт