2SOLUTION BRIEF

WHITE PAPER

Executive Summary
The Artificial intelligence (AI) revolution is here. The digital
revolution over the last few decades has taken the next giant
leap in the form of Artificial intelligence and Machine
Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Learning (ML). AI and ML have immense potential to disrupt
AI Data Centers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
and transform entire industries. Supermicro is already seeing
GPU Communications in an AI Data Center . . . . . . . . . . . . . . . . . 2
several breakthrough innovations emerge in healthcare,
Network Fabric in AI Data Center . . . . . . . . . . . . . . . . . . . . . . . . . . 3
banking, automotive, retail, etc., to name a few sectors.
Supermicro SSE-T8032S Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Advancements in large language models (LLM) and training
Supermicro Enterprise SONiC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
and inferencing technologies are breaking barriers andSummary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
enabling transformative innovations. Supermicro is a leader
More Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
in the AI infrastructure space and has been constantly rolling
Appendix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
out advanced solutions that have enabled these innovations.
This paper will cover the infrastructure challenges posed by
AI workloads on the network fabric and how Supermicro's
400G 25.6Tbps high-speed Ethernet switch SSE-T8032S running Supermicro Enterprise SONiC is uniquely equipped to tackle
them. The results from the performance and benchmarking tests run on a cluster of AMD instinct MI300X nodes interconnectedby an SSE-T8032S switch fabric and Broadcom 400G Network interface cards (NIC) discussed in this paper further validate that
the SSE-T8032S switch running Supermicro Enterprise SONiC is a great choice for modern high-performance AI network fabric.
TABLE OF CONTENTS

AI Datacenters
AI workloads have specific, unique requirements. The unprecedented growth in LLMs in terms of the number of parameters andsize of the data sets demands a new type of infrastructure with specialized hardware like AI accelerators and GPUs. Largeclusters of GPUs, in many cases, tens of thousands of them, are interconnected to form a unified high-performance computing
environment to perform massively parallel computations needed to train these massive LLMs. The AI workloads running onsuch large, distributed computing environments must constantly communicate and synchronize to complete the job in realtime. One of the key metrics used to measure the executing efficiency of the AI infrastructure is the job completion time (JCT),
which effectively is the overall time taken between the job assignment and collective completion. The network fabricconnecting these distributed GPUs must be best designed to optimize the job completion time of the training jobs to improveoverall efficiency and resource utilization of the AI infrastructure.

GPU Communications in an AI Data Center
AI workloads are massive in scale, and techniques such as data parallelism and model parallelism enable them to be broken
into mini-batches and run across multiple GPU nodes. The GPU nodes constantly communicate using the MPI collective
communications library (CCL) primitives to synchronize and exchange data, which demands an efficient network fabric in theAI infrastructure. All-reduce and all-gather are some of the most commonly used CCL primitives in AI networks. All-reduce isused to aggregate and then redistribute data from cluster nodes as they coordinate and work on completing the job. For each
iteration, the nodes perform a reduce operation and update all the nodes with the computed result. These data-intensive
communication operations can cause all nodes to transmit simultaneously, causing network in-cast scenarios and leading toincreased congestion in the network.

Figure 1 – Network performance in large AI Clusters (Source: OCP2022 Keynote by Meta)© 2024 Copyright Super Micro Computer, Inc. All rights reserved

July 2024

2

Unlike traditional data centers, the workloads in the AI infrastructure are very different. The AI workloads constantly exchange
messages between the GPU nodes in the cluster to collectively complete the job. These message exchanges generate increased
bursts of significant inter-GPU traffic. These messages initiate and terminate within the cluster between the same set of GPUs,
causing the flow entropy to be very low. This low entropy bursty traffic is more prone to congestion and sub-optimal data
forwarding that can cause link saturation and packet loss. Congestion or loss of packets in the network fabric can slow computenodes, increasing job completion time and possibly blocking job completion altogether. For instance, Figure 1 above shows theoverall time taken by the network in training four different AI models (M1, M2, M3, M4) within Meta's AI data center. It shows thatin large clusters, as much as 40-50% of the total time is consumed by the network transport that connects the GPU nodes, ofwhich about 33% of the time is wasted waiting for the network. This clearly makes network fabric the most critical piece in theAI infrastructure puzzle.

Network Fabric in an AI Data Center
As the compute clusters scale, the AI data center's performance greatly depends on a highly robust network that can enable
seamless communication within the cluster. If the network fabric in the AI data center is not architected right, it can lead toperformance bottlenecks, causing sub-optimal functioning of the expensive GPU resources. As more GPUs are inter-connectedin larger clusters, the increased message exchange and synchronization between them need a higher aggregate cluster
bandwidth for all the GPUs to communicate effectively. This mandates the network fabric connecting the compute clusters beof high bandwidth and non-blocking. The distributed nature of the AI workload execution translates to multiple iterations ofcomputation, updating, and synchronization of the mini-batches to successfully complete the job. In such a scenario, even asingle slow node could end up causing the entire cluster to slow down, making it very sensitive to packet loss or delays in thenetwork. So, it is not just the individual packet latency but the tail latency of the network that also needs to be low for all thenodes to successfully exchange messages and proceed to the next iteration without having to wait unnecessarily on other GPU
nodes. All these requirements warrant a need for a high bandwidth, non-blocking network fabric with low tail latency andefficient congestion control mechanism to meet the stringent Quality of Service (QoS) demands posed by the distributed
computing challenges in the AI infrastructure. Ethernet already has all these capabilities needed in an AI network fabric built
into it. RDMA over Converged Ethernet (RoCE) technology enables RDMA high throughput and low latency message transfer
capability on Ethernet. A standard CLOS network fabric running high-speed ethernet with RoCEv2 is thus an ideal solution for
an AI network of any scale.

Supermicro SSE-T8032S Switch
Supermicro SSE-T8032S switch platform offers 64 400G (2*400G with 32 Physical OSFP) ports in a 1 RU form factor. The
forwarding engine of the switch is built with a Broadcom Tomahawk 4 ASIC that can support 25.6Tbs of switching capacity.
Broadcom Tomahawk 4 provides several advanced features like better in-cast absorption, lower end-to-end latency, advanced
load balancing capabilities, and efficient congestion management mechanisms. The Supermicro Enterprise SONiC has acomprehensive set of all the features that the AI networks need. The single chip architecture of SSE-T8032S powered by
Tomahawk 4 running Supermicro Enterprise SONiC makes it an excellent switch platform with the correct port density in a 1 RU
form factor to build a non-blocking low latency CLOS network fabric to interconnect the current generation GPUs in the AI datacenters.

Supermicro Enterprise SONiC
The SSE-T8032S switches come with the Supermicro Enterprise SONiC network operating system. Software for OpenNetworking in the Cloud (SONiC) is an open-source Debian Linux based network operating system that has seen increased© 2024 Copyright Super Micro Computer, Inc. All rights reserved

July 20243

adoption in the industry across major cloud providers and enterprises. Supermicro SONiC is hardened with more features andusability features to deploy and manage at scale. The Supermicro Enterprise SONiC has all the needed software support torealize all the advanced hardware capabilities of the Tomahawk 4 switch, such as RoCE and congestion control technologies.
Let's look at RoCE and the congestion control aspects that make SSE-T8032S running Supermicro Enterprise SONiC suitable formodern AI network fabrics.

RDMA over Converged Ethernet (RoCE):
Remote Direct Memory Access (RDMA) is a network data transfer technology employed in large, distributed computing thatallows applications to read from and write into the host adapters directly. RDMA bypasses the operating system kernel and CPU,
immensely bringing down data transfer latency. Distributed AI applications with a lot of synchronous messaging benefit greatly
from RDMA technology and, hence are a widely deployed protocol in AI infrastructure for the compute nodes to communicate
between them. With RoCE, all the performance benefits and optimization of the RDMA technology are now available in ethernet
transport, making Ethernet even more suitable for a scale-out AI network fabric. Large AI network fabrics are prone tocongestion, and unfortunately, RDMA does not have congestion control built into it natively and relies on the transport protocol.
Ethernet does have a suite of congestion management technologies that can be deployed alongside RoCE for congestion
control.

Priority Flow Control (PFC):
Priority flow control is a link level flow control mechanism used to pause the transmission of the network's flows, causing
congestion. Traffic is classified into queues based on priority and assigned a watermark. When the watermark is breached, aPFC message is generated towards the upstream device, signaling to pause the flow transmission until the receipt of asubsequent message to resume. The upstream device resumes transmission when the PFC is de-asserted. PFC thus mitigatescongestion by pausing the congesting queues instead of dropping packets.

Explicit Congestion Notification (ECN)ECN manages congestion by proactively signaling the transmitting host of the potential congestion in the network withoutpausing the traffic. Unlike PFC, in the case of ECN, the congestion signaling is propagated all the way to the end host and notjust to the upstream device. During congestion, as the egress queue starts building up in an ECN capable device, it starts toprobabilistically mark the Congestion Experienced (CE) bits in the DSCP packets in the IP header of the data packets to notify
the destination host of the potential congestion. As congestion continues to build up, more packets are marked until all thepackets towards the destination are marked to signal congestion. On seeing the CE bit set in the data frames, the destination
host would then send a Congestion Notification Packet (CNP) to notify the sender of the flows about the congestion. The senderwould react to CNP by reducing the data transmission rate to clear the congestion. Unlike PFC, ECN handles congestion byslowing down the frame transmission and shaping the flow instead of pausing transmission.

Data Center Quantized Congestion Notification (DCQCN):DCQCN is basically a means of managing congestion by deploying both ECN and PFC simultaneously to get the best of both. InDCQCN, as congestion in the network builds up, ECN is configured in a way to get triggered first and start marking the packet tosignal the host to slow down the transmission. If that does not happen on time and the congestion continues to build up, PFC
kicks in to pause the offending flows. The goal essentially is to use ECN as your first line of defense to tackle congestion without
having to stop data transmission as best as possible before resorting to PFC as a backup mechanism. The effectiveness of© 2024 Copyright Super Micro Computer, Inc. All rights reserved

July 2024

4

DCQCN depends on having the right PFC and ECN parameters such that PFC does not prematurely start pausing packets before
ECN attempts to throttle down the transmission rate to handle the congestion.

Figure 2 - Data Center Quantized Congestion Notification

SSE-T8032S Fabric Test
Supermicro SSE-T8032S interoperates with all the leading current generation GPU and NIC products to create high-performance
network fabric in the AI data centers. We will now focus on one of the recent benchmarking tests performed on an AI cluster
built using AMD GPUs and Broadcom GPUs NICs. A small-scale AI cluster was emulated using Supermicro SSE-T8032S switches
to connect Supermicro AS-8125GS-TNMR2 GPU Systems using Broadcom's latest 400G Network interface cards (NIC) to evaluate
GPU cluster performance. Supermicro AS-8125GS-TNMR2 is a high-performance GPU accelerator platform that integrates eight
fully connected AMD Instinct MI300X GPU modules using an AMD infinity fabric. The topology below shows how a simple twotier network fabric was built using SSE-T8032S to interconnect these 32 GPUs into a small cluster. RDMA was provisioned on theBroadcom 400G NICs to enable high throughput and low latency message transmission between the GPU nodes. DCQCN
congestion control feature was configured both on the SSE-T8032S nodes and Broadcom NICs, interconnecting the 32-node
AMD Instinct MI300X cluster to enable a loss-less fabric for the RDMA endpoints to communicate efficiently using RoCEv2.

© 2024 Copyright Super Micro Computer, Inc. All rights reserved

July 20245Figure 3 - Topology for RCCL test on 8032 fabric for AMD MI300X cluster
ROCm communication collective library (RCCL) benchmarking tests were run on this cluster to emulate various CCL
communication, such as all-reduce, all-to-all, scatter, etc., to measure the cluster's performance. The standard RCCL tests
emulate the CCL operations supported on the ROCm framework by scheduling jobs of different sizes and measuring the GPUperformance. The tests report three metrics – operation time, algorithm bandwidth, and bus bandwidth. While operation timeand algorithm bandwidth are an excellent criterion for point-to-point operations like send, receive, etc., but are not an accurate
measure of performance in case of collective operations like all-reduce, all-to-all, etc., which involve multiple GPU nodes talking
to the same GPU at the same time. Algorithm bandwidth decreases as the number of ranks in the cluster increases; hence, bus
bandwidth is used in operations like all-reduce. Bus bandwidth is more reflective of the actual bandwidth. It is computed byapplying an additional calculation to the algorithm bandwidth to factor in the number of ranks and the collective operationtype. The bus bandwidth equates to 2(n-1)/n * algorithm bandwidth for an all-reduce operation. Table 1 below captures theRCCL test result run for an all-reduce operation in the lab setting. For the 32-node AMD MI300X GPU cluster connected by a twotier network fabric with SSE-T8032S, the bus bandwidth was measured at 280GBps. Please see the appendix for details on thehardware and software components used to construct this test scenario.

© 2024 Copyright Super Micro Computer, Inc. All rights reserved

July 2024

6

Table 1 - RCCL test results for All-reduce operation on 8032 fabric for AMD MI300X cluster

Summary
AI is a transformative technology that is continuously evolving at a rapid pace. AI workloads need a new class of scalable, robust,and efficient data center infrastructure to meet the compute and power demands of these large-scale distributed systems. A
robust AI infrastructure calls for a very robust and efficient network fabric to interconnect the massively distributed GPUresources, reduce job completion time, and drive peak performance of the overall system. This makes network fabric the mostcritical component in the AI data center. The right kind of network can enable the utilization of expensive GPUs in theinfrastructure and significantly drive down the overall spending on AI infrastructure. Supermicro SSE-T8032S, Tomahawk 4
based 64 port 400G switch running Supermicro Enterprise SONiC, is an excellent switch with all the features needed to build alarge scale, high throughput, low latency network with efficient congestion management techniques to meet the needs of thecurrent generation AI data center. The SSE-T8032S switches also come as part of the Supermicro rack scale solutions - a fully
validated solution for the AI infrastructure with extremely low lead times. Using Supermicro switches in the infrastructure helps
in having a more tightly integrated overall offering. Pre-validated designs and the cluster orchestration tools from Supermicro
radically accelerate day 0 deployments, aiding in faster infrastructure build-out.

For More Information
Supermicro | Products | Networking | SSE-T8032S
http://benchmark.supermicro.com/report/HPC/amd-mi300x-cluster-rccl-tests
GitHub - ROCm/rccl-tests: RCCL Performance Benchmark Tests© 2024 Copyright Super Micro Computer, Inc. All rights reserved

July 2024

7

Appendix – Component Details
1. Cluster Hardware Components
Item Number
AS -8125GS-TNMR2
SSE-T8032S
CBL-NTWK-0976-15M-J
CBL-NTWK-1107-15M-J

DescriptionDO , x PU F U I
XOx xO FP Ports, U, Front airflow, C P Us ( leaf,spine)[ ]O FP to x
Q FP
Y cable, 6 W , . ,JPC
O FP Cable
W.

QuantityH

62. Server Components
Hardware
CPUGPUMemoryDrive
NetworkPSU
PSU
FansDescription
D PYC 96 96 Core Processor, @ .
Hz 96Cores/ 9 hreads
D Instinct I
X 9 B7 W
icron C F
WC BBHFF/s 96 B
amsung ZQL
HCL
7 . B
Broadcom 76 PCIe en . x 6 low profile standard form factor
single Q FP DD connector
PW K 6PW DF 9 F

Quantity6/

3. Software Components
ItemLinux OS
Kernel
ROCm Software
BMC
Redfish
CPLD
BIOS
BKC
OpenMPI
UCX
Broadcom Driver

Version Information
Ubuntu . . L. .
generic
6. .
. .
. .
F .6 . 7.
. 6.
Open PI v . . av . .
netxtreme bnxt_en .

.

9. .

.

4. Switch SSE-T8032S Specifications
ItemPorts
Switch Capacity
Packet Buffer
OSCPU

Detailed Information
6 x
in O FP + x
FP+.6 bps aggregated switching capacity
B memory packet buffer
O iC O . . nterprise_ dvanced
IL X 6 Xeon core CPU© 2024 Copyright Super Micro Computer, Inc. All rights reserved

July 2024

8SUPERMICROAs a global leader in high performance, high efficiency server
technology and innovation, we develop and provide end-to-end
green computing solutions to the data center, cloud computing,enterprise IT, big data, HPC, and embedded markets. Our Building
Block Solutions® approach allows us to provide a broad range ofSKUs, and enables us to build and deliver application-optimized
solutions based upon your requirements.

© 2024 Copyright Super Micro Computer, Inc. All rights reserved

July 20249