Merchant Silicon in Carrier Networks
November 1, 2022

Broadcom
Proprietary
IncandProprietaryConfidential.
and Confidential.
Copyright © 2021
© 2021
Broadcom.
BroadcomAllInc.
RightsAll rights
Reserved.
reserved.
The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
1| Broadcom

Agenda
•
•
•
•
•
•
•

2

Introduction
DNX Portfolio and Cadence
Partners & Ecosystem
DNX in Access Edge and CoreDNX Architecture Efficiency
DNX DDC for AI / ML
Summary|Broadcom Inc Proprietary and Confidential. © 2021 Broadcom Inc. All rights reserved.Broadcom Merchant Silicon Switch Portfolio
Enterprise

Hyperscale Data Center

TridentTomahawk

Domain-Specific Switching Silicon
3|Broadcom Inc Proprietary and Confidential. © 2021 Broadcom Inc. All rights reserved.

Service Provider

Jericho / QumranRamonDNX – Deployed In All Networks
Jericho & Ramon

QumranDistributed / Chassis

Centralized / Fixed• Non-blocking switching• Controlled latency• Deep packet buffering (lossless)• Large-scale Traffic Management• Programmable PP• Millions of IP routes• Redundancy and protection• Unified silicon architecture & SDK

5|Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

Merchant Silicon Jericho Innovation Cadence
Jericho4

60x INCREASE INPERFORMANCE IN 10YEARS!!

Jericho3

Jericho2c+Jericho2
Jericho+
Jericho

Arad

480G

1.2T

1.8T

9.6T

14.4T

28.8T

5nm

7nm

16nm

28nm

28nm

40nm

2013
7|

2015

Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

2017

2019

2021

2023

Merchant Silicon Ramon Innovation Cadence
40x INCREASE INPERFORMANCE IN 10YEARS!!

Ramon3

51.2T

5nm
Ramon

9.6T

16nm
FE3600FE1600

1.32T

3.6T

28nm

40nm

20138|

2015

Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

201720192021

2023

3 Generations of Qumran in Access, Metro and Edge
5nm Lineup refresh of all Centralized Devices

••
•
•
•

100G, 50G and 25G lanes
Line rate MACsec on all ports
Low power 4W-5W per 100G

14.4T
Q2c+7.2T

Q2

4.8T
Q3m

JR3-software compatible

Q2m

QMX

Q3c

Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

1.6-2.4T

600-800G
800G

160-300G

Q3u180-360G360G
Q3n32-120G
Q2n|

3.2T

Q3aQ2uQUX

1.6-2.4T

800G
Q2aQAX

6.4T

3.6T

No feature compromise in Qumran vs Jericho
Q2c9Q380G

Partners & Ecosystem

Broadcom Proprietary and Confidential. Copyright © 2022 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Merchant Silicon Enabled Ecosystem
Orchestration & Automation

Network Controller

Disaggregated NOS Options

NV Controller

OEM SW Partners

ManagementNOS
SDKODM HW Partners

11|Tomahawk

Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

Trident

Jericho

OEM HW Partners

Merchant Silicon-Based OEM Platforms for Every Network NodeVendor

12|AccessAggregation

Service Edge

CoreDCIDC Leaf/Spine

DC Top of Rack

7020R

7280R
7500R

7280R
7500R

7800R37500R

7800R37500R / R7280R

7300X
7500R/E

7260X
7280R/E

516X

517x

NCS540

ASR900/907
NCS5500/560

NCS5500/NCS5700

R6672/R6X71/
R6675

R6675/R6672/
R6273/R6274

SSR8000

HDS8000

HDS8000

CE6850/55
CE5800CE8800CE12800

CE12800

NE9000NE5000E/40E

CE8800CE12800

CE6850/55

S6300/RA5300

S12500R

S12500R

S12500R

S12500

S9800

S6800

DRX-30DRX-90DRX-90ACX7024
ACX710ACX5448
ACX7100
ACX7509

QFX5xxx

QFX5xxx

7210
7250-IXR-e

7250-IXR

7250-IXR-s
7220-IXR
7250 IXR-6e
7250 IXR-10e

7220-IXR
Nuage 7850

8180

7250-IXR-R4

Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

NCS5500/NCS5700
NCS6000

NCS5500/NCS5700
NCS6000

OEM NOS + White Box Ecosystem

Metro Aggregation

Hurricane3

13|

QumranMX

Jericho2

Broadcom Inc Proprietary and Confidential. © 2021 Broadcom Inc. All rights reserved.

DC Spine

Metro Core

Metro Edge

Ramon

Tomahawk

Tomahawk2

Trident3

Trident4

DC LeafDisaggregated Solution for Every SP Network NodeOLTvBNG

Metro Aggregation

Metro Core and Edge

Cell Site
Router

Fronthaul
Gateway

QumranAX

Jericho2cJericho2cJericho2

QumranAX

Monterey

Coming soon

14|

Broadcom Inc Proprietary and Confidential. © 2021 Broadcom Inc. All rights reserved.

Merchant Silicon-Based ODM Platforms for Every Network NodeVendor

15|

AccessAggregation

Service Edge

CoreDCIDC Leaf/Spine

DC Top of Rack

AS5915
AS7315
AS7316

AS7535
AS7525
AS7515AS5926
AS5916
AS7946

AS7946
AS7936
AS7926

AS7926
AS7826
AS5835

AS973X AS7812
AS972X AS7712
AS971X AS7710

AS7826 AS583X
AS7726 AS5812
AS7716 AS4630

STX-60D0

STX-60D0

STX-60x0

STC-60x0

STC-60x0

SNJ-60x0
SNH-60x0

SNJ-61x0
SNH-61x2

S3020
S3010

S3020

S3020

D8060 DX510
DX510 DX500

D8060 DX510
DX510 DX500

DX430 DX410
DX400 D7020
D8030 DX010

DX040 DX032
D3040 D3030
DX030 E1062

AGC7008S
DT-G26X2SL

DT-G26X2SL
AGCV208S
AGC7648

DT-G32X2SL
AGCX424S
AGC5648SDT-V36C4S
AGC023S
AGCC048

AGCXD40SAGCVA48S
AGCC048

DD-S32C8 AG9064
DD-S64C4S AGC032
DD-S32C4S AG5648

DD-T32C2S AGV848
AG6248C AG9032
AG7648 AGV424

D8216
D8212

D8216
D8212

D7332
ASL74xx
D6432 / D6456

NE1054
D6356
D6332

T4024

T4032

T4048

T9036 T7080
T9018 T7040
T9010

T9036 T7080
T9018 T7040
T9010

T8128 T7128
T9064 T7064
T9032 T7032

T4048 IX7
T8024 IX8

G3100
S9501

S9600
S9500
S9510

S9600
S9510

S9700
S9710
S9701

S9700
S9710S9705

S9300
S9100
S8900

S9310
S8810
S7810

Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

DNX in Access Edge and Core

Broadcom Proprietary and Confidential. Copyright © 2022 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Merchant Silicon In The Service Provider Space
4G/5G Cell-Site Routers (AT&T SIAD, TIP DCSG)

Virtual Broadband Network Gateway

Qumran-AX
Qumran-UX

TIP Distributed Disaggregated Backbone Router (DDBR)

Qumran2cCore & Edge Router via DDCJericho2

Ramon

17|

Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

Qumran-Based Router Integrating UPF•

Single-chip Session, Forwarding and Routing plane––––––––••

GTP-U processing
Carrier-Grade Segment Routing
Network Slicing
HQoS and packet buffering
Routing, ACL
Any tunneling technology (TE, SR, etc.)
Advanced telemetry, statistics and Instrumentation
Future-proofing programmability

N3
S1

CUPS and Disaggregation ready model
Significant cost and power saving

x86

Qumran18|Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

JR2/J2C+ Series GTP-U Service Scale in Centralized Mode
Q2c+JR2Q2c

J2xQ2aQ2u

Q2n

3.6Tbps

2.4Tbps

1.2Tbps

800Gbps

400Gbps

180Gbps

40GbpsNew ConnectionRate

100150K/s

100150K/s

100150K/s

100150K/s

50-75K/s

50-75K/s

25-40K/s

Fixed WirelessService Scale *

500K

120K

120K

120K

30K

30K

30K

Mobility Service
Scale *
Assumptions

1M

320K

300K

300K

100K

100K

100K

N3-Facing
Interfaces(no over-subscription)•– Fixed Wireless Access (FWA)– Min bandwidth requirement of 100Mb/s per UE (10K UE / 1Tbps)– Upstream metering and downstream shaping per UE– VOQ and counter per UE, with smart counter eviction– Mobility– Min bandwidth requirement of 10Mb/s per UE (100K UE / 1Tbps)– On demand fine grained Upstream metering and downstream shaping, or per UE group (or service type)– On demand fine grained VOQ or per UE group (or service type)– Counter per UE, with smart counter eviction* 2-hop processing where applicable
19|Broadcom Inc Proprietary and Confidential. © 2021 Broadcom Inc. All rights reserved.

DNX Architecture Efficiency vs. Ethernet

Broadcom Proprietary and Confidential. Copyright © 2022 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Power Efficient Architecture - Ramon• Ramon is the industry most power efficient switch• 25% power saving compared to best of class equivalent Ethernet switch• Ramon is used as a high-performance cell switching fabric device, Zero design overhead– Optimized design– Doesn’t carry multipurpose logic targeted to reduce R&D spending over multiple tape outs
– Light, elegant and proven design– Fine-tuned for multiple generations
Ramon Design Concept“Light” Fabric Cores

Ramon

Jericho21|Jericho

Ramon

Jericho

JerichoBroadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

Core 0Core 1

Core 4

Core 6

Core 2

Core 3

Core 5Core 7

Core 8Core 9Core 10

Core 11Core 12Core 13Core 14

Core 15

Power Efficient Architecture – Jericho• Same “zero design overhead” concept• Optimized for line-card / NCP functionaility, no combo functionaility overheads• Light and optimized fabic cores reduce total device power• No fabric logic overhead on NIF coresPower, Ethernet Cores X2 vs. Fabric

Qumran Design Concept
Ethernet Cores

Jericho Design Concept

Ethernet Cores

Ethernet Cores“Light” Fabric Cores

Core 0Core 0Core 1Core 0Core 1Core 0Core 1Core 0Core 1Core 2Core 3

Core 2Core 3~=Core 2Core 3Core 2Core 3
Core 022|Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

+Core 1

Combo Arch – “One arch fits all”
• Each core can act as fabric or Ethernet• Generic engines carry power overhead vs. optimized logic

Combo Design Concept
Jericho Design Concept
Ethernet Cores

Core 0Core 223|Core 1

Core 3

Combo Cores“Light” Fabric Cores

Core 0Core 2Core 1

Core 3Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

Combo Cores

Fabric
Logic

Fabric
Logic

Fabric
Logic

FabricLogic

Core 0Core 1

Core 0Core 1Fabric
Logic

Fabric
Logic

Fabric
Logic

FabricLogic

Core 2

Core 3

Core 2

Core 3

Network View – What Do You Optimize For?
• One device fits all• Zero design overhead• Optimized to reduce R&D resources• Optimized for low power

Ramon Design Concept

Ramon Design Concept“Light” Fabric Cores
Core 0Core 1Core 4“Light” Fabric Cores
Core 6Core 0Core 1Core 4Combo Cores

Jericho Design Concept
Ethernet Cores

Core 0Core 224Core 1Core 3|Core 0Core 2Core 2Core 3Core 5Core 7Core 2Core 3Core 5Core 7Core 8Core 9Core 10Core 11Core 8Core 9Core 10Core 11Core 12Core 13Core 14Core 15Core 12Core 13Core 14Core 15Jericho Design Concept“Light” Fabric Cores

Core 1Core 3

Ethernet Cores

Core 0Core 2Core 1Core 3Core 0Core 2Jericho Design Concept

Jericho Design Concept“Light” Fabric Cores

Core 1Core 3

Ethernet Cores

Core 0Core 2Core 1

Core 3“Light” Fabric Cores

Core 0Core 2

Combo Design Concept

Combo Design Concept
Core 6Core 1Core 3Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

Ethernet Cores

Core 0Core 2

Core 1

Core 3

Core 0Core 2

Combo Cores

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

FabricLogic

Core 0Core 1

Core 0Core 1

Core 0Core 1

Core 0Core 1Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

FabricLogic

Core 2Core 3Core 2Core 3Core 2Core 3Core 2

Core 3

Combo Design Concept

Combo Cores

Fabric
Logic

Fabric
Logic

Fabric
Logic

FabricLogic

Core 0Core 1

Core 0Core 1Core 1

Core 3Fabric
Logic

Fabric
Logic

Fabric
Logic

FabricLogic

Core 2

Core 3

Core 2

Core 3

Combo Cores

Fabric
Logic

Combo Design Concept“Light” Fabric Cores

Combo Cores

Combo Cores

Combo Cores

Combo Design Concept

Combo Cores

Combo Cores

Combo Design Concept

Combo Cores

Combo Cores

Combo Cores

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

FabricLogic

FabricLogic

Core 0Core 1Core 0Core 1Core 0Core 1Core 0Core 1Core 0Core 1Core 0Core 1

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

Fabric
Logic

FabricLogic

Core 2Core 3Core 2Core 3Core 2Core 3Core 2Core 3Core 2Core 3Core 2Core 3

DNX DDC for AI / MLBroadcom Proprietary and Confidential. Copyright © 2022 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

AI Network WorkloadRequirements• Objective– Reduce job completion time
– Increase GPU utilization• Characteristics– High network utilization– Small Number of flows at high bandwidth– Uneven bandwidth flows

26|Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

Network Pain Points• Standard Load balancing is flow-based• Leads to– Uneven link utilization– Sub-optimal network utilization– Frequent Congestion

With DNX

The Distributed Disaggregated Chassis (DDC)

Native for AI/ML network

Industry-largest router

Wide, open eco-system• Perfect load balancing• Flat network - single routingdomain• E2E scheduling• No congestion• Scalable and high radix• Disaggregated solution• Yielding the industry-largest
single routing domain• Logically-equivalent tovertically integrated solutions• AT&T first to deploy at thecore network• Adopted by leading carriersand cloud providers• OEM and 3rd party software
over Jericho and Ramon
fixed boxes

With DNX
27|Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

DDC White Box Ecosystem
In Deployment2023• DDC scale-out to 18,000 400GE ports• DDC scale-out to 32,000 800GE ports• Network Cloud Processor 3 (Jericho2c+)– 40x400G fabric ports– 36x400GE network ports• Jericho3 NCP– 18 x 800GE / 36 x 400GE network ports– 20 x 800G fabric ports• Network Cloud Fabric (NCF, Ramon)– 48x400G fabric ports• Ramon3 NCF– 64 x 800G fabric ports

Tomahawk4

NCP4 28.8T J3 5nm

NCP2 8T J2 16nm

600W Max power

500W Max power

NCP3 28.8T J2c+ 7nm
1000W Max power

10x400GE +13x400G

NCF2 51.2T R3 5nm

NCF 9.6T Ramon 16nm
48x400G
NCP in Production
28|36x400GE + 20x800G

2019

Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

64x800G

36x400GE + 40x400G

2021

2023

Reducing Job Completion Time with DNX DDC
Perfect load balancing••

Larger messages in certain flows can overload the links
DNX fabric uses all links for every flow – cell spraying

Message Size Impact on Job Completion Time
5000
4000

Jericho / Ramon cluster
Leaf-Spine Ethernet cluster

3000
2000
1000
064KB

A
29|BBroadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

128KB

256KB

512KB

1,024KB

2,048KB

4,096KB

Reducing Job Completion Time with DNX DDC
End-to-End Congestion Control•Ethernet PFC propagates to all links, creates HOLblocking•

DNX E2E scheduling eliminates HOL blocking– Flow held at the source

Congestion Level Impact on Job Completion Time54.5
4Jericho / Ramon cluster
Leaf-Spine Ethernet cluster

3.5
32.5
2
1.5
10.5
0Low @ GPU Rx

A30|

BBroadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

Ethernet PFC

Mid @ GPU Rx

High @ GPU RxSummary – DDC in the AI/ML Network
Perfect load balancing
Cell spraying over the cell-based DNX fabricNo hashing, no ECMP polarization
End-to-end scheduling
VOQ-based
Lossless RDMA operation
Deep buffering and Virtual output queuing
Low and controlled latency

Single-tier, large-scale routing domain for the entire network
Simplified management•31

Further reading– “Distributed Decoupling Chassis” - white paper (in Chinese) by China Telecom and Broadcom– “Reducing job completion time in AI/ML clusters” – Broadcom blog by Xiguang (Henry) Wu|Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.

Summary•••••

32

DNX is the leading switch/routing technology, deployed in the world’s largest carriers and MSDCs
Mature and field-proven SDK and architecture (10th generation)
Extensive and robust portfolio with committed investment and cadence
Vibrant hardware & Software ecosystem enabling multiple solutions to the end customer
Access, BNG, Edge, Peering, and Core rich feature-set, all deployed in T1 carrier networks|Broadcom Inc Proprietary and Confidential. © 2019 Broadcom Inc. All rights reserved.Thank You

Broadcom
Proprietary
IncandProprietary
Confidential.
and Confidential.
Copyright © 2021
© 2021
Broadcom.
BroadcomAllInc.
RightsAll rights
Reserved.
reserved.
The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
33 | Broadcom

backupNovember 1, 2022

Broadcom
Proprietary
IncandProprietary
Confidential.
and Confidential.
Copyright © 2021
© 2021
Broadcom.
BroadcomAllInc.
RightsAll rights
Reserved.
reserved.
The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
34 | Broadcom

UPF Node – Example••Qumran offloading the UPF datapath

––––

•SMF

UPF scales from 80Gbps to 7.2Tbps, from 10GE to 400GE–•Registration
ProtocolTermination/encapsulation the GTPUv1 PDU with the PDU
Container(0x85) extension header according to 3GPP
Same forwarding (Match-Action table) implemented in UPF datapath
MEC Servers aggregation RouterQoS
DPI redirect

NOS

NOS run exactly same UPF function as current X86 UPF does
Can act as slow path (network-defined)
Extend the NOS HAL to populate data-path flow tables in Qumran

GTP Control Redirect to Slow-path–Includes: GTPCv1, GTPCv2, GTP’, GTPUv1-signaling

HALBRCMSDKPCIeor Eth

Switch Host CPU–––

Host UPF
as slow path

Control

DNX

Unknown TEID/User (Miss)
Known TEID/User (Hit)

Flowtable

DNX based node (WhiteBox/OEM)

35|Broadcom Inc Proprietary and Confidential. © 2021 Broadcom Inc. All rights reserved.

InstallTEID
Entry

SMF

Cloudifying The Slow Path•••UPF Control Plane

Unknown TEID sent to a remote “slow path” service
New TEID configuration provided to the local x86 hostLocal host installs the TEID in Qumran

HAL
BRCM
SDK

InstallTEID
EntryNOS

PCIex86

Qumranx86

Qumranx86

Unknown TEID/User (Miss)
Known TEID/User (Hit)36|Broadcom Inc Proprietary and Confidential. © 2021 Broadcom Inc. All rights reserved.

QumranUPF Market Trends
DNX

x86 + SmartNICGTP Tunnels

x86

37|

Broadcom Inc Proprietary and Confidential. © 2021 Broadcom Inc. All rights reserved.

Jericho2 Series UPF: GTP-U Released!
In Trials with a Chinese Carrier••••••••••

UPF-phase 1Now

UPF-phase 2
Q4 2021

Control plane trapGTPUv1 Encapsulation and Termination
GTPUv1 \w Session Container Termination
Up to 200K* TEID/User-IP ScaleCountersUpstream Policing + Downstream Shaping
FWA Use Case
Mobility Use Case
Lawful intercept
N6 Tunneling: MPLS, IP, SR, etc.

New features• GTPUv1 \w Session Container Encapsulation• QoS Mapping (QFI)• Up to 0.5M-1M* TEID/User-IP Scale on-chip* Jericho2 scale
38|Broadcom Inc Proprietary and Confidential. © 2021 Broadcom Inc. All rights reserved.

Released Segment Routing Techniques“Classic” SRv6uSID

G-SID

MPLS-SRAvailability

DNX SDK in GA

DNX SDK in GA

DNX SDK in GA

DNX SDK in GA

Compression

Full 128b, nocompression

32b prefix, 16b uSID
(48b prefix in theplan)48b prefix, 32b GSID

N/A

Up to 11 SID

Up to 66 uSID

(Roadmap to increase inJ2c+)

VPN (mix)
example1 x VPN + 10 x SID

1 x “classic” (VPN) +60 x uSID

1 x “classic” (VPN) +12 x G-SID

DeploymentStatus
Standard
39

Up to 16 G-SIDs

JR2 max.
scale|Interop by CMCC

Up to 14 labels

N/A

Deployed

Deployed

JR2 with ZTE, Huawei,H3C

Deployedspring-srv6-networkprogramming-24

spring-net-pgm-extensionsrv6-usid-07

generalized-srv6-for-cmpr01

draft-ietf-spring-segmentrouting-mpls-22

Broadcom Inc Proprietary and Confidential. © 2021 Broadcom Inc. All rights reserved.