MPLS Japan 2022
Evolution of Whitebox Solution
Tetsuya Murakami
tetsuya@arrcus.com
1Arrcus Confidential

Agenda▪ Use case
▪ Integration with Open Source Software (SONiC)▪ Challenge on scalability, performance, security▪ Summary▪ Q&A

2Arrcus Confidential

Enterprises Need Agility and Flexibility in A Hybrid World
RetailFinancial
Services
Web3.0 Apps
Cloud -> Multi-Cloud

4G -> 5G

Healthcare

3

Cloud -> Edge

Arrcus Confidential

Industrial
Automation

However … Legacy Networks are a Huge Bottleneck
Telco/5G Carriers
Problem with current Transport Networks§ Expensive§ Inflexible§ Not Software-defined§ Not Multi-cloud, 5G, Edge readyEnterprises(Retail, Financial Services)

4Clouds,CDNs

Arrcus Confidential

Commercial Software Structure for Whitebox▪ Microservices Architecture

Operations, Management, & Cloud Consumption Layer (Controller based Orchestration)

Open Adaptive
Cloud/Industry
Standard Interfaces

ArcAPICLISNMPRESTCONF

NETCONF

Real Time Streaming Telemetry(JSON, Google Protocol Buffer,
Kafka, & gRPC)▪ Components implemented in user space▪ Minimal required support of Linux Kernel▪ Container ready

ProgrammaticAPIs

IPv4/IPv6
EVPN/VxLAN
SR-MPLS/SRv6Fabrics

Configuration Store(YANG based model)

BGP
IS-IS
OSPF
LSVRLabelManager
LDP

System InfrastructureLinuxNetworking

System & PowerManagement

Any Router or Switch Platform

Data Store(JSON based model)

ACL
Interface
Adjacency
CoS
BFD
ManagerManager
QoS
Application Oriented Routing & ProtocolsRouting(RIB/IM/Tunnel Manager)sFlow▪ Modularity
LLDP
LACP
LAG

FIB/ACL/CoS/QoS
DPAL

HW Device Drivers + BSP
Linux Kernel

Router or Switch SDK

3rd Party Hardware

Arrcus, Inc 2022 Confidential Material under NDA▪ Replace/Add/Use any component withminimal impact▪ Every component implemented as a multithreaded separate process▪ Hardware Agnostic▪ Router, Switch and/or Server ready▪ Intel and/or ARM processor support▪ Dataplane Adaptation Layer (DPAL)

Integration with OSS (SONiC): Architecture

6

Arrcus Confidential

Integration with OSS (SONiC): Routing

7

Arrcus Confidential

Challenge on Whitebox software with OSS▪ Scalability/Performance Measurement▪ Security Issue

8

Arrcus Confidential

Scalability/Performance Measurement9

Arrcus Confidential

ArcOS/ArcOS+SONiC Convergence Measurement
Topology
IXIA -----Router#1 ---- (8 ebgp session)— 7326-56X(ArcOS w/ SONiC/ArcOS) ---------- Router#2

BGP Profile
30K BGP IPV6 host routes learnt
from 8 ebgp sessions (8 ways ecmp/multipath)

ArcOS
Convergence Time

ArcOS + SonicConvergence Time

Comments
BGP Convergence (Control plane)

1.2 secs

2.3 secs
Forwarding Convergence (Data plane)

4 secs1015 secs

Arrcus Confidential

ArcOS/FRR Convergence Measurement
Topology
IXIA --------- UUT (FRR/ArcOS) ---------- Router

BGP Profile
32 BGP Sessions/ 900K Prefixes/
28.8M Paths
Profile Used …
•32 BGP Sessions on IXIA
•Each BGP Session advertisingsame 900K prefixes

ArcOS
Convergence Time
50 secs

44 secs

FRR
Convergence Time
5 mins 10 secs

4 mins 30 secs

Comments
Test Methodology: Route convergencemeasured by bringing up interfaces
between IXIA and UUT.
This convergence number is an average of 5
iterations.
Test methodology: Keep BGP sessions up
and measure convergence by only
advertising routes.
This convergence number is an average of 5
iterations.

11Arrcus Confidential

Scale and Performance Analysis on XGS family§ArcOS on IX8Routes

With Acks

No Acks

50k v47.91 s~6.3 k/sec.678 s~73 k/sec§

Local Programming Rate with/without HW Route Ack

ArcOS on IX8
1 v4

15-25 ms

Others on TD3
1 v4Definition§ HW Route Ack = explicit Ack from HWwhen a route is installedGoals
1.

Measure local Route Programming Rates
with and without HW Route Ack enabled

2.

E2E BGP route advertisement latency with
HW Route Ack enabled

30-50 ms§

E2E Route Advertisement Latency With Route Ack

Others on TD3

ArcOS on IX8Routes

Acks

No Acks

RoutesAcks

No Acks

40K v4

5.8-6.2s

0.6-0.7s

40K v412.75-13s2.2-2.3s

40K v6

8.5-9s

0.8-0.9s

40K v6

16.5-17s

2.2-2.25s

250K v4

39-40s8.5-9.5s

250K v4

78-80s15-15.5s

Network Convergence Time
12Arrcus Confidential

3.

Network convergence N1-DUT-N2§§Measure time taken to advertise a SINGLE
BGP route received when it is being held tillroute is programmed in HW and an explicit
Ack has been received
For X prefixes, measure (last update
received by N2 – last update sent by N1)
once BGP neighbor from DUT to N1 isunshut

Measurements done on TD3 HW, ALPM
mode, subnet routes

Routing Performance Analysis for DNX family
Traffic Convergence

Hardware Programming Rate

257060151517105Routes/sec

1750

2558515144403020

Without RIB-FIB
Q2C

J2

Traffic Convergence§ Under 30 seconds traffic convergence§ Full Internet Route Table (IPv4 + IPv6)

With RIB-FIB

Without RIB-FIB
Q2C

J2

Hardware Programming§ Up to 58k routes/sec§ Full Internet Route Table (IPv4 + IPv6)

One Architecture. One Software
1321

15

11.8

12.0

1000
With RIB-FIB

19510020

Seconds

2020

Seconds

Control Plane ConvergenceArrcus Confidential

With RIB-FIB

Without RIB-FIB
Q2CJ2Control Plane Convergence§ Under 12 seconds convergence§ Full Internet Route Table (IPv4 + IPv6)

Challenge to use OSS for Whitebox▪ Lacks support for Holistic CLI interface▪ Includes lack of support for programmatic APIs

▪ Lacks Telemetry at scale▪ Less performance/scalability▪ Lacks support for Hardware resource usage▪ Lacks Routing support for DNX

14Arrcus Confidential

Scalability/Performance Analysis▪ SAI API is slower than the traditional SDK▪ SDK might not be optimized even though using
the same SDK (there are some spaces toimprove the performance).
▪ However, SDK is not disclosed at all… In addition,
SDK is not available for all available chipsets.

15

Arrcus Confidential

Security Issue

16

Arrcus Confidential

Security issue▪ Sometimes, someone running out of the device areusing the local service running inside of the device
like syslog, ntp, etc. As the result, a lot of resources
are consumed on the device and some services aregoing down…
▪ Someone might steal the information from the device
or might inject the invalid routing/operation to thedevice…

17Arrcus Confidential

Why…
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address
udp
00 0.0.0.0:55015
udp
00 0.0.0.0:56204
udp
00 0.0.0.0:123
udp
00 0.0.0.0:49591
udp
00 0.0.0.0:33435
udp
00 0.0.0.0:50172
udp
0
0 0.0.0.0:58719
udp
00 0.0.0.0:44045
udp
00 0.0.0.0:60505
udp
0
0 0.0.0.0:36466
udp
00 0.0.0.0:53112
udp
00 0.0.0.0:53122
udp
00 0.0.0.0:36741
udp
0
0 0.0.0.0:53640
udp
0
0 0.0.0.0:53894
udp
0
0 0.0.0.0:46166
udp
0
0 0.0.0.0:38519
udp6
00 :::123

Foreign Address
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
:::*

State

PID/Program name1879/rsyslogd
2946/rsyslogd
3194/ntpd
3200/rsyslogd
3217/rsyslogd
5613/rsyslogd
3290/rsyslogd
2642/rsyslogd
5634/rsyslogd
2016/rsyslogd
3339/rsyslogd
5607/rsyslogd
3227/rsyslogd
2872/rsyslogd
2665/rsyslogd
3140/rsyslogd
1105/rsyslogd
3194/ntpd

Issue: some applications open up the port without specifying the address
18

Arrcus Confidential

Security Analysis▪ Security Vulnerabilities – Port scan reveals lots ofprocesses using 0.0.0.0 address▪ Redist db a single point of failure▪ 32 bit system that manages configdb, state db, routingand SAI▪ Limited ACL Support▪ Limited support for Filtering rules in Hardware

19Arrcus Confidential