HPC/AI
NVIDIA IO
NVIDIA Networking Product Marketing
HPC/AI Technical Computing Director

AIExploding Data and Model Size

175
Zettabytes

393 TB
COVID-19 Graph Analytics

287 TB/day
ECMWF

58Zettabytes# Parameters (Log scale)

GPT-3
175 BnTuring-NLG
17 Bn
GPT-2 8B
8.3 Bn

BERT
340 M
Transformer
65 M

2010

2012

2014

2016

2018

2020

2022

2024

16 TB/sec
SKA

550 TB
NASA Mars Landing Sim.

2017

2018

2019

2020

Big data growth

Growth in scientific data

Exploding model size

90% of the World’s Data in last 2 Years

Fueled by Accurate Sensors & Simulations

Driving Superhuman Capabilities

AIIO
Source for Big Data Growth chart: IDC – The Digitization of the World (May 2020)
storagenewsletter.com: 64.2ZB of Data Created or Replicated in 2020

2021

NVIDIA Magnum IO:
AI

ENTERPRISE
AI

HPCIO

NVIDIAHPCSPARKRAPIDSSDKAI:
TRITON,TRT,MERLIN

CLARA

METRO

DRIVE

ISAAC

AERIAL

CUDA-XCUDADOCAMAGNUM IO

NETWORK IO

STORAGE IO

IN-NETWORK COMPUTEIO MANAGEMENT

Magnum IO
IO

Network IO:Storage IO

In-Network Compute

IO ManagementΣNCCL
NVSHMEMASAP2GPUDirect Storage

SHARP

NVMe SNAP

Hardware Tag
Matching

GPUDirect RDMA
HPC-X, UCX

DPDK
MOFED

NVIDIA イノベーション | ピークパフォーマンス | どこでもRDMA | 柔軟な抽象化

Ethernet NetQUFM

GPUDirect

RDMA

10Magnum IOアーキテクチャは、DMAエンジンを活用して不要なコピーを排除し、IOを最適化

No GPUDirect

GPUDirect

Network handled by CPU and CPUMemory

Network goes directly to GPU memory

2 Full copy operations 0232

PCIe transactions

1↓

GPU utilization↑↑

CPU usage↓↑

Latency↓

18x Latency

.

2.6

MPI

Magnum IO GPUDirect Async
HPCCPUSystemMemory

NVIDIA GPUDirect•InfiniBandor
Ethernet
AdapterCPU•Magnum IO

NVIDIA GPUDirect Async

GPUGPU•

GPUDirect Async•GPUDirect Async-Kernel Initiated GDA-KI
GPU NICCPU•GDA-KI.

GPU CPU.

•PCIe Switch•CPU.

•GPUGPUCPU

GPUDirect Peer2PeerGPUGPU•GPUDirect RDMAShared System Memory
GPUDirect Peer2PeerGPU••

CPU GPU.

PCIe.

GPUDirect RDMA
GPUDirect Async

SM – Streaming Multiprocessor
Improving Network Performance of HPC Systems Using NVIDIA Magnum IO NVSHMEM and GPUDirect Async

NVLinkGPUGPU
GDA-KI

GPUDirect StorageGPU

Without GPUDirect Storage

Without GPUDirect Storage

System Memory

System Memory

CPUCPU

PCIe Switch
PCIe SwitchNICsStorage
NICsGPU

Low Bandwidth | High Latency | Limited CapacityStorageNICsGPU
Storage fabric
NICs 9 & 10 usedfor GDS

Higher Bandwidth | Lower Latency
O(PB) capacity | CUDA programming model

GPUDirect Storage
GPU:GPUDirect Storage

CPUBlueField Data Processing Unit

Infrastructure Services

Data Center on a Chip
• 16 Arm 64-Bit Cores• 16 Core / 256 Threads Datapath Accelerator• ConnectX InfiniBand / Ethernet• DDR memory interface• PCIe switch

BlueField Infrastructure
Compute Platform

BlueField SNAP
Host• BlueField SNAP

NVMe

Standard
NVMe driver

NVMe SSD•• BlueField SNAP
NVMe
VirtIO-blk•

PCIe

PCIe
NVMe

Hypervisor/VMsStandard
VirtIO-blk
generic driver

VirtIO-blk

BlueField SNAP

• VM.

••

.

Bring your own
storage software
stack

BlueField DPU3

SHARP V3

Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)SHARP V364

▪

AI
SHARP

SHARP V2▪
•

Switch• In-Network••

Aggregated
Data

•
• MPI
•

SHMEM

SwitchHPC•AggregatedResultSwitchAggregatedResultData• Barrier, Reduce, All-Reduce, Broadcast• Sum, Min, Max, Min-loc, max-loc, OR, XOR, AND•8) 8/16/32/64• Bfloat16

ServerServerServerServerServer

MVAPICH

SHARP

NCCL All Reduce Performance with NVIDIA InfiniBand

InfiniBand SHARP maintains bandwidth on large scale reaching up to 100% advantage
InfiniBand SHARP100

Where you can get Magnum
IO TechnologiesMagnum
IO Resources

Storage IO
GPU Direct Storage

Network IO
GPU Direct RDMA, Peer-to-Peer in CUDA
Toolkit (v6-11.5.1)
NCCL, HPC-X UCX, MPI in HPC SDKIO Management
NetQ

NVIDIA Deep Learning Institute Courses
The Fundamentals of RDMA Programming
Working with Mellanox OFED in InfiniBand
Environments

NCCL
ASAP2In-Network Computing

UFM
Nsight Cluster Debug and Profiling

SHARP

Magnum IO ManagementNVIDIA Nsights Graphics•• Nsight➢ GPUDirect RDMA and GPUDirect Storage• Nsight Magnum IO

UFM Platforms

AI

UFM TelemetryReal-Time Monitoring

UFM EnterpriseManagement, Monitoring & Orchestration(UFM Enterprise includes UFM Telemetry)

UFM Cyber-AI

Cyber Intelligence and Analytics(UFM Cyber-AI includes UFM Enterprise)NVIDIA NetQ – Transforming Network Operations
SaaS

AI Driven Cyber
Security(Morpheus)

CLIWebhook

Orchestration andOperations Systems
Integration

GUI

Events

Notification System

API Gateway
Analytics Applications

Messaging System

On-Premise Telemetry Aggregator (OPTA)
Telemetry

NoSQL
DatabaseTelemetry
Aggregation andAnalytics

Telemetry

Agent

Agent

SONiC

CUMULUS

Agent

DebianDPU

Telemetry from
Switches and DPUs