ソフトウェアから見た
Disaggregated Distributed Backbone Router (DDBR)
へのチャレンジ
Tetsuya Murakami
October 26 2023

Confidential | ©2023 Arrcus

1

DDBR Architecture – Flexibility & High Scalability
Control Plane Cluster• Commodity multi-core x86 server• Runs VDR Control & Management planes

Underlay Connector• Layer 3 fully managed switch• Out of band (OOB) Control & Management connectivity

113

Fabric Card
• Single-chip or Multi-chip Ramon platform• Future-proof to support any Next-Gen siliconLine Card• High Density ports: 10/25/100/400 Gigabit Ethernet• Jericho 2 today, Jericho 3 or any Next-Gen silicon

Breaking the scale barrier: up to 7,680 x 100GE ports
2Arrcus, Inc Confidential - August 2020

DDBR components – ODM whitebox lineupRamon – 19.2Tbps

Ramon3 – 51.2Tbps

Fabric CardLine Card
Jericho2cJericho2

Jericho3
Jericho2c+Jericho2

3

Available today

Available 2024

ODM hardware – Fabric Card
UfiSpace S9705-48D

f0f1
enp2s0f0
enp2s0f1-Broadcom Ramon ASIC
Intel Broadwell-DE processor
48x400GE (fabric)
DDBR’s Backplane Switch FabricArrcus, Inc 2022 Confidential Material under NDAODM hardware – Line Card or Standalone
UfiSpace S9700-53DX

High Density 100GE-Broadcom J2 ASIC
Intel Broadwell-DE processor
Access ports: 40x100GE-or 80x10/25GE w/ breakout

Fabric ports: 13x400GE
Switching capacity 4.8Tbps
Large routing table
8GB deep packet buffer
Redundant PS and fansArrcus, Inc 2022 Confidential Material under NDAODM hardware – Line Card
UfiSpace S9700-23D

400GE Line Card-Broadcom J2 ASIC
Intel Broadwell-DE processor
Access ports: 10x400GE-or 40x100GE w/ breakout

Fabric ports: 13x400GE
400GE ZR support
Switching capacity 4.8Tbps
Large routing table
8GB deep packet buffer
Redundant PS and fansArrcus, Inc 2022 Confidential Material under NDAODM hardware – Line Card
UfiSpace S9710-76D

High Density 400GE Line Card-Broadcom J2c+ ASIC
Intel Skylake-D processor
Access ports: 36x40/100/400GE
Fabric ports: 40x400GE
400GE ZR and OpenZR+ support
Switching capacity 14.4Tbps
Large routing table
16GB deep packet buffer
SyncE and IEEE1588v2
Redundant PS and fansArrcus, Inc 2022 Confidential Material under NDA

Compute Server & Underlay Connector• CS (Compute Server)• x86 Server 20-core CPU, 128GB RAM• 2x NVMe SSD RAID 1 storage• NIC:§ Intel X710 10GE NIC§ Intel E810 100GE NIC§ Mellanox ConnectX-6 DX• UC (Underlay Connector):• Broadcom Trident3 switch• Quanta IX8A

swp17 swp19

swp49

swp18 swp20

swp50

swp51

swp56

DDBR is ready?
• Hardware design is fully defined.
However,• Software design/architecture is very unclear.
••••

No available open source – Can’t try DDBR easily.
No suitable Operating System.
Need to design/develop software from the scratch.
There are several challenging….

Confidential | ©2023 Arrcus

9Slot identity and interface naming?
Problem:§ Unlike a real chassis, DDBR does not have physical slot identifiers that can beused in interface naming.
§ How do we identify and name interfaces on DDBR LCs.
Solution:§ User configures a slot number against the serial number for a LC/FC to make it apart of the cluster§ Until this happens LCs and FCs cannot join the cluster and participate in the datapath§ Examples:- eth5_20 ==> Port 20 on slot 5- eth5_20s3 ==> Break out #3 on port 20 on slot 5- eth5_20s3_1 ==> Sub-interface 1 on break out #3 on port 20 on slot 5

Confidential | ©2023 Arrcus

10ifindex allocation problem?
Problem:§ ifindex is allocated by Linux kernel when creating interfaces in kernel§ ifindex is used system-wide to uniquely identify an interface§ In DDBR, there are many Linux kernel instances and interfaces are
distributed across many LCs§ How do we ensure a unique system-wide ifindex
Solution:§ Move ifindex allocation responsibility to interface manager outside of Linux§ Change ASIC interface handling to accept app provided ifindex§ Entails major rework of interface information flow within the system
Confidential | ©2023 Arrcus

11Interface discovery and punt path?
Problem:§ How does NOS control plane get to know about interfaces located on LCs§ How does control plane send and receive packets through thoseinterfaces
Solution:§ As a LC joins the cluster its interfaces are discovered, virtual
representations of those interfaces are created on the CS§ A L3 tunnel (i.e., VxLAN) is provisioned between the CS and each LC totransmit and receive packets over these interfaces

Confidential | ©2023 Arrcus

12How distributing software components?
Problem:• What software components need to be distributed to LC/FC?
• Does the operating system (i.e, Linux) need to be synced up between LC/FC/CS? If
yes, how to sync up?
Solution:• In order to simplify the software architecture, only FIB should be distributed toLC/FC• FIB on CS should manage the entire information and distribute partial information toFIB on LC/FC• The operating system on CS/LC needs to be synced up because FIB information isneeded to handle interfaces, ICMP error, etc…
• The operating system running on CS has the entire FIB. The operating systemrunning on LC has the partial FIB only.
Confidential | ©2023 Arrcus

13Distributed software architecture

ComputeServer•••••ConfigControlplane

Datastore

FIB(Linux kernel,Global plugins)

BGP, ISIS, OSPF
L3 RIB, L2 RIB
IfMgr, AdjMgr
ACL, QoS, CoPP
SystemMessage Lib
Brokers

Punt Path
Manager (CP)Linux Kernel

Punt Path

Linux Kernel
FIB

Fabric Card

Linux KernelLinux Kernel

FIBFIB

Punt Path
Manager (LC)Punt Path
Manager (LC)Line CardLine Card
Confidential | ©2023 Arrcus

14

Global identifier problem?
Problem:§ Certain data path identifiers required by ASIC need to be globally scoped.
§ ASIC specific§ Traditional solution of piggy backing on top of control plane download
sucks§ How do we allocate and distribute such globally scoped, system-wide
ASIC specific identifiers
Solution:§ A global FIB that runs on the CS allocates the global identifier.
§ A global identifier is distributed to FIB instances running on LC.
§ ASIC can accept it?
Confidential | ©2023 Arrcus

15

Runtime orchestration?
Problem:§ How do we download and run the right set of NOS binaries on the appropriate nodes§ How do we run an active and a standby instance of the control plane on the same host to
get soft high availability
Solution:§ Break up NOS into several different container images§ Containers are lightweight and provide enough separation to simulate an entire chassis
on a single host
§ Choice of several open source orchestration systems to manage and interconnectcontainers§ Kubernetes (k8s) might be chosen as the orchestration system

Confidential | ©2023 Arrcus

16

DDBR Containers
Controlplane processes

ConfigCP

CP-DPATHFIB(LXK,Global plugins)

Message LibBrokers

Datastore

CP-PPM

K8S Master

Punt Path
Manager (CP)DockerRegistry
ComputeServer

Linux KernelLinux KernelLinux KernelLinux Kernel
FIB

KubeletFIB

Kubelet

LC-DPATH

Fabric Card

LC-DPATH

FIB

KubeletPunt Path
Manager (LC)Line Card

LC-DPATH
Punt Path
Manager (LC)Line Card
Confidential | ©2023 Arrcus

17

DDBR Underlay?
Problem:§§§§

ALL nodes need a host OS
Who does the installation of the host OS
Who configures the Control Plane Switch (CPS)
Who assigns node addresses in the control plane network

Solution:§ DDBR underlay tools – collection of tools and scripts that can bootstrap a DDBR cluster frombare metal§ Installs host OS on all nodes, configures the CPS and the node interfaces that connect to it§ Creates ssh keys and login on all nodes for internal access§ Runs NTP between nodes§ Runs an internal DHCP, DNS server, Docker registry service and finally starts off the k8sorchestration services

Confidential | ©2023 Arrcus

18Networking functions supported in the operating system
Problem:• How to leverage the networking functions in Linux like bridge, lacp, etc• How to manage the bridge interface among multiple LCs.
• How to manage lacp on LC and/or CS.

Solution:• Create the bridge interfaces on multiple LCs and connecting to thepath/interface toward to FC.
• Disable lacp on LC and running on lacp on CS only.
• The lacp packets is managed by the punt path manager.
• It causes some overhead to process lacp packets. Hence, it is difficult to achieve
lacp fast.

Confidential | ©2023 Arrcus

19Summary▪

Hardware design for DDBR is securely defined.

▪

Software design/architecture for DDBR has not been discussed sofar…
Need to consider a lot of stuff related to software architecture for
DDBR
Require TRY & ERROR approach to verify the architecture.
Take long time for the development▪

No open source tools, No operating system suitable for DDBR
Need to modify/tweak the existing open source, operating system.

Arrcus, Inc 2022 Confidential Material under NDA

Thank You!

Confidential | ©2023 Arrcus

21