AI

HPC

2024/6/27(@_ksasaki)

GPU

HPC

NVIDIA GPU
Fermi(2010)

Kepler(2012)

Maxwell(2014)

Pascal(2016)

Volta(2017)

Turing(2018)

Ampere(2020)

Hopper(2022)

Ada
Lovelace(2022)

Compute Capability

2.0 / 2.1

3.0/3.5/3.7

5.0 / 5.2

6.0 / 6.1

7.0

7.5

8.0 / 8.6

9.0

8.9

HPC

M2050

P100

V100

TSUBAME2DL

K20X

TSUBAME2.5

M40

A100
A30

TSUBAME3DL

A40

K2/VDI

GeForce

K1

T4

P4
M60

K6000

M6000

P5000

GTX
580

GTX
780

GTX
980

GTX
1080

L40S
L4A16

V100

6000

GB200
B200
B100

H100TSUBAME4Blackwell(2024)

GV100

RTX8000

RTX A6000

RTX 6000
Ada

TITAN V

RTX2080

RTX 3090

RTX 4090

TOP500:

https://top500.org/system/180236/

https://top500.org/system/180239/

NVIDIA H100 Tensor Core GPU
Unprecedented Performance, Scalability, andSecurity for Every Data Center
HIGHEST AI AND HPC PERFORMANCE
4PF FP8 (6X)| 2PF FP16 (3X)| 1PF TF32 (3X)| 60TF FP64 (3X)
3TB/s (1.5X), 80GB HBM3 memory

TRANSFORMER MODEL OPTIMIZATIONS
6X faster on largest transformer models

HIGHEST UTILIZATION EFFICIENCY AND
SECURITY
7 Fully isolated & secured instances, guaranteed QoS
2nd Gen MIG | Confidential Computing

FASTEST, SCALABLE INTERCONNECT
900 GB/s GPU-2-GPU connectivity (1.5X)
up to 256 GPUs with NVLink Switch | 128GB/s PCI Gen5

FP8, FP16, TF32 performance include sparsity. X-factor compared to A100DGX H100 Hardware Feature SummaryGPU
GPU memory
Performance
NVswitchesCPUs

8 x NVIDIA H100 Tensor Core GPUs
640 GB total
32 petaFLOPs FP84
Dual Intel Xeon 8480C, 56 cores (112 for 2 sockets),
2.0 GHz (base), 2.9 GHz (All Core Turbo), 3.8 GHz (Max Turbo)

System Memory
2 TBNetworking
4 x OSFP ports provide connectivity to 8 x NVIDIA
connectivity and speed ConnectX-7s
8 x 400Gb/s InfiniBand/Ethernet*
2 x dual-port NVIDIA ConnectX-7
1 x 400Gb/s InfiniBand/Ethernet and1 x 200Gb/s InfiniBand/Ethernet*
Cache Storage

8 x U.2 3.84TB NVMe Secure Erase Drives

Boot Storage

2 x 1.92TB M.2 NVMe (software encryptable)

Host Management

On-board 10Gb/s RJ-45 Ethernet
100Gb/s QSFP Ethernet NIC

Remote System
Management

Baseboard Management Controller (BMC)
1Gb/s RJ-45 network connectivity
Remote Keyboard, Video, Mouse (KVM)Remote Storage
RedFish, IPMI management

Operating System

DGX OS based on Ubuntu
Additional support for Ubuntu and Red Hat Enterprise LinuxNVIDIA DGX H100
The world’s first AI system with the NVIDIA H100 Tensor Core GPU* Subject to Change

DGX H100 Exploded View

Bezel
GPU trayMotherboard tray

Fan Modules

ChassisFront Console
Board

Front Cage
Secure Erase Drives

Power Supplies

DGX H100 Motherboard Tray
PCIe card riser forslots 1 and 3

32 x 64GB DIMMs for a
total of 2TB of systemmemoryNetworking modules. Each includes
4 x ConnectX-7 network devices

100Gb Ethernet NIC (top)ConnectX-7 storage
networking (bottom)

Midplane connectivity
Power, PCIe, sensors andsignaling communications

OSFP transceivers forexternal ConnectX-7
cluster networking
PCIe card riser
for slots 2 and 4
M.2 boot drive carrier (top)ConnectX-7 storage
networking (bottom)

Dual x86 Gen5 PCIecapable CPUs

4 x Densilink cables tocommunicate ConnectX-7 network
devices to external ports

DGX H100 Block Diagramhttps://www.nvidia.com/en-us/on-demand/session/gtc24-s62421/

https://www.nvidia.com/en-us/on-demand/session/gtc24-s62421/

ISC 2024

High Performance, EnergyEfficient Systems inTOP500/GREEN500
Grace Hopper is powering the next waveof new Exaflop AI systems around the globe

Powering Worlds Fastest, Greenest
Supercomputers
TOP500The platform of choice76%
use NVIDIA

1.5 EF
added to top10#1

5 of thepowered byGH200

top10 poweredby GH200

Green500

Energy Efficiency

23
NVIDIA of the
top 30NVIDIA GH200 Grace HopperSuperchip
Built for the New Era of AI Supercomputing

CPU to GPU Bandwidth

900GB/sGPU Memory Bandwidth

5TB/s

NVLink-C2C

HBM3e

Energy Efficiency

QFT Quantum Simulation

90X

200X

MILC Efficiency vs 2S x86 CPUs

Performance vs 2S x86 CPUs

Performance vs H100 80GB50X

LLM Inference

624GB High-Speed Memory | 4 PF AI Perf | 72 Arm Cores

Preliminary measured performance, subject to change
Energy Efficiency: GH200 144GB vs 2S Xeon 8480+ CPU for MILC running dataset: Apex Medium
QFT Quantum Simulation: QFT 2S Xeon 8480+ vs GH200 144GB
LLM Inference: Llama.cpp (2S 8480+) and TensorRT-LLM (GH200, H100) | Max Batch Llama2 70B | Throughput includes time to first token + token
generation time

JEDI, #1 on ISC 2024 Green 500Grace Hopper in 6 of the Top 10 Green 500• JEDI, at EuroHPC/ FZJ, Julich, Germany• 48 nodes, 192 GH200 at 73 GF/Watt• Jupiter Booster - ~6,000 nodes, ~24,000 GH200• 36x FP64 energy efficiency vs Titan, Oakridge in2012.
• 18,000 K20X was 2 GF/Watt.

• GH200 over 1,000x more energy efficient on raw
compute• Mixed precision and the AI performance,• AI can bring time-to-solution algorithmic
improvements to solving scientific compute problems
faster.

Isambard-AI , #2 on ISC 2024 Green500
University of Bristol• Isambard-AI , University of Bristol• 0.7 AI EF | 42 nodes | 168 GH200 Superchips at 69 GF/Watt• 22 TB LPDDR | 16 TB HBM | 38TB Total Memory• Phase 2: 12 cabinets | 660 blades• 21 AI EF | 1,320 nodes | 5,280 GH200 Superchips• 675 TB LPDDR | 506 TB HBM | 1.2 PB Total Memory• AI-based Supercomputer to Foster Scientific Innovation•

United Kingdom•

5,280 GH200 GPUs | Grace CPUs with over 55,000 Arm Neoverse V2 Cores

First European Grace HopperSupercomputer Online• Fastest AI Supercomputer in Europe• 20 Exaflops of AI• 10X more energy efficient than Piz Daint at 64 GF/Watt• Powered by 10,000 Grace Hopper Superchips• HPC and AI to Advance Weather, Climate (1km globalmodels), and Material Science

Green500 (2024/6)

Rank Name

1 JEDIComputer

BullSequana XH3000, Grace Hopper Superchip 72C 3GHz, NVIDIA GH200 Superchip, Quad-Rail NVIDIA
InfiniBand NDR200

EnergyEfficiency[GFlops/Watts] Processor

72.73Grace Hopper Superchip 72C 3GHz

2 Isambard-AI phase 1HPE Cray EX254n, NVIDIA Grace 72C 3.1GHz, NVIDIA GH200 Superchip, Slingshot-1168.83NVIDIA Grace 72C 3.1GHz

3 Helios GPUHPE Cray EX254n, NVIDIA Grace 72C 3.1GHz, NVIDIA GH200 Superchip, Slingshot-11
ThinkSystem SR670 V2, Intel Xeon Platinum 8362 32C 2.8GHz, NVIDIA H100 80GB PCIe, Infiniband
HDR

66.95NVIDIA Grace 72C 3.1GHz

HPE Cray EX254n, NVIDIA Grace 72C 3.1GHz, NVIDIA GH200 Superchip, Slingshot-11
ThinkSystem SD665-N V3, AMD EPYC 9354 32C 3.25GHz, Nvidia H100 94Gb SXM5, Infiniband
NDR200HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot11

64.38NVIDIA Grace 72C 3.1GHz

HPE Cray EX254n, NVIDIA Grace 72C 3.1GHz, NVIDIA GH200 Superchip, Slingshot-11
HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot11
HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot11

59.29NVIDIA Grace 72C 3.1GHz
AMD Optimized 3rd Generation EPYC 64C
58.02 2GHz
AMD Optimized 3rd Generation EPYC 64C
56.98 2GHz

4 Henri
5 preAlps

6 HoreKa-Teal
7 Frontier TDS
8 Venado
9 Adastra
10 Setonix – GPU

65.40Intel Xeon Platinum 8362 32C 2.8GHz

62.96AMD EPYC 9354 32C 3.25GHz
AMD Optimized 3rd Generation EPYC 64C
62.68 2GHzBefore Transformers = 8x / 2yrs
Transformers = 256x / 2yrs

GPT-MoE-1.8T

10,000,000,000

Training Compute (petaFLOPs)

MT NLG 530B
Chinchilla
BLOOM
PaLM

GPT3-175B
100,000,000

Microsoft T-NLG
GPT-2 1.5B
Megatron-NLG

1,000,000

Xception

InceptionV3

10,000

XLNet

VGG-19 Resnet
Seq2Seq

Wav2Vec 2.0
MoCo ResNet50

BERT Large
GPT-1
Transformer
ResNeXt

DenseNet201

ELMo

AlexNet
100
2012

2014

2016

2018

2020

2022

2024

NVIDIA Blackwell

AI
GPU 1

20 PetaFLOPS (FP4)AI

4X Training | 30X Inference | 25X Energy Efficiency & TCOAI10GPU

AI SUPERCHIP

2nd GEN TRANSFORMER ENGINE

5th GENERATION NVLINKRAS ENGINESECURE AI

DECOMPRESSION ENGINE

208B Transistors

FP4/FP6 Tensor Core

Scales to 576 GPUs

100% In-System
Self-Test

Full Performance
Encryption & TEE

800 GB/s

AI
21GPU

2NV-HBI 10TB/s#1#2192GB HBM3e10 PetaFLOPS FP8 | 20 PetaFLOPS FP4192GB HBM3e | 8 TB/sec HBM Bandwidth | 1.8TB/s NVLink

1GPU5NVLink

NVLink

7.2 TB/s

Sharp v4 plus FP8
3.6 TF In-Network Compute
NVLink

576 GPUNDR InfiniBand400Gbps

18GB200 NVL72GB200 NVL72

36 GRACE CPUs
72 BLACKWELL GPUs
Fully Connected NVLink
Switch Rack

Training
Inference
NVL Model Size
Multi-Node All-to-All
Multi-Node All-Reduce

720 PFLOPs
1,440 PFLOPs
27T params
130 TB/s
260 TB/s

Grace Blackwell Superchip
Architecture for Simulation + AI

Grace
CPUHardware
Coherency

PCIe5 x 16

≤ 192 GBBlackwellGPU≤ 480 GBHBM≤ 192 GBBlackwellGPU

CPU LPDDR5x

HBM

NVLINK

Assumed Network

GH200(1 Grace | 1 Hopper)
NDR200GB200(1 Grace | 2 Blackwell)
XDR400Super Chip TDP

700W

Up to 2.7kW

CPUHBM / LPDDR (GB)HBM BandwidthGrace96 GB |Up to 480GB
1xGraceUp to 384 GB |Up to 480 GB
4xFP64 FMA (Vector)
FP32 FMA (Vector)

1x1x

2.6x
2.6x

GB200 NVL72
GB200 NVL72 Rack

BLACKWELL

NVLINK C2C

GB200 SUPERCHIP
40 PETAFLOPS FP4 AI INFERENCE
20 PETAFLOPS FP8 AI TRAINING
864GB FAST MEMORY
GB200 SUPERCHIP COMPUTE TRAY
2x GB200
80 PETAFLOPS FP4 AI INFERENCE
40 PETAFLOPS FP8 AI TRAINING
1728 GB FAST MEMORY
1U Liquid Cooled
18 Per Rack

NVLINK SWITCH TRAY
2x NVLINK SWITCH CHIP
14.4 TB/s Total Bandwidth
SHARPv4 FP64/32/16/81U Liquid Cooled
9 Per Rack

Blackwell

GB200 NVL72

HGX B200

HGX B100

Compute for Trillion Parameter Scale AI
Maximum Performance and Lowest TCO

Best Performance and TCO for HGX Platform

Drop-in Upgrade for Existing Hopper InfrastructureBlackwell
GB200 NVL72

HGX B200

HGX B100

72888 (Hopper)

FP4 Tensor Core

1,440 petaFLOPS

144 petaFLOPS

112 petaFLOPS-

FP8/FP6/INT8

720 petaFLOPS

72 petaFLOPS

56 petaFLOPS

32 petaFLOPS

Fast Memory

Up to 30 TB

up to 1.5 TB

Up to 1.5TB

640GBAggregate Memory Bandwidth

Up to 600 TB/s

Up to 64 TB/s

Up to 64 TB/s

24 TB/s

Aggregate NVLink Bandwidth

130 TB/s

14.4 TB/s

14.4 TB/s

7.2 TB/s

2592 Arm Neoverse V2 cores---GPUs

CPU Cores

HGX H100

Per GPU Specifications
FP4 Tensor Core

20 petaFLOPS

18 petaFLOPS

14 petaFLOPS-

FP8/FP6 Tensor Core

10 petaFLOPS

9 petaFLOPS

7 petaFLOPS

4 petaFLOPS

INT8 Tensor Core

10 petaOPS

9 petaOPS

7 petaOPs

4 petaOPS

FP16/BF16 Tensor Core

5 petaFLOPS

4.5 petaFLOPS

3.5 petaFLOPS

2 petaFLOPS

TF32 Tensor Core

2.5 petaFLOPS

2.2 petaFLOPS

1.8 petaFLOPS

0.98 petaFLOPSFP64 Tensor Core

45 teraFLOPS

40 teraFLOPS

30 teraFLOPS

67 teraFLOPS

GPU memory | Bandwidth

Up to 192 GB HBM3e | Up to 8 TB/s

80GB HBM3 | 3TB/sMulti-Instance GPU (MIG)

77Decompression EngineYes-Decoders
Power

2x 7 NVDEC
2x 7 NVJPEG
Configurable up to 1,200W

Interconnect
Server options

Configurable up to 1,000W

Configurable up to 700W

5th Generation NVLink: 1.8TB/s
PCIe Gen6: 256GB/s
NVIDIA GB200 NVL72 partner and
NVIDIA-Certified Systems with 72GPUs

NVIDIA HGX B200 partner and
NVIDIA-Certified Systems with 8GPUs

1.Preliminary specifications subject to change. All Tensor Core numbers with sparsity.
2.GB200 Superchip configuration includes 2 high performance B200 GPUs and one Grace CPU700W
4th Generation NVLink: 0.9TB/s
PCIe Gen5: 128GB/s

NVIDIA HGX B100 partner and
NVIDIA-Certified Systems with 8GPUs

NVIDIA Blackwell Platform

Spectrum X800 Switch
Quantum X800 Switch

NVLINK Switch
HGX B100

GB200 Superchip
Compute Node

ConnectX-8 SuperNIC

BlueField-3 SuperNIC