NVIDIA コンピューティング &ネットワーキング最新情報
2022/12/5
佐々木 邦暢 (@_ksasaki)
エヌビディア合同会社

GPUNVIDIA GPUFermi(2010)

HPC

Kepler(2012)

Maxwell(2014)

Pascal(2016)

Volta(2017)

P100

V100Turing(2018)

Ampere(2020)

M2070

K80
DL

A40

T4

K16000GTX 580

K6000

GTX 780

L40

P4

K2/VDI

H100

A100

M40

DL

GeForceAda
Hopper Lovelace(2022)(2022)

A16

V100

M60

M6000

P5000

GTX 980

GTX
1080

GV100

RTX
8000

RTX A6000

RTX 6000

TITAN V

RTX
2080

RTX 3080 Ti

RTX 4090

AI
10,000,000,000

70%

MONTHS TO TRAIN MODELSEXPLODING COMPUTATIONAL REQUIREMENTS

TRANSFORMERS TRANSFORMING AI

Transformer AI Models = 275x / 2yrs
AI Models Excluding Transformers = 8x / 2yrs

DesiredMegatron-Turing NLG 530B

1,000,000,000

AI Papers
In last 2 years discuss Transformer Models

GPT-3

SEGFORMER
Sematic Segmentation

10,000,000

Training PetaFLOPSMEGAMOLBART
Drug Discovery with AI

DECISION TRANSFORMER
Reinforcement Learning

Microsoft T-NLG
GPT-2
Megatron
XLNet

1,000,000
Xception
InceptionV3

100,000

SUPERGLUE LEADERBOARD
Difficult NLU Tasks
Seq2Seq

10,000ResnetWav2Vec 2.0
MoCo ResNet50

SwitchTransformer
BERT Large
1.6T
GPT-1

Performance

100,000,000

Today

Transformer

ResNeXt

VGG-19

GPT-3 (175B parameters)
3.5 months to train on 128x A100

ELMo
DenseNet201

1,000AlexNet
100

Scale (# GPUs)
2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022

MegaMolBART: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/megamolbart | SegFormer: https://arxiv.org/abs/2105.15203 | Decision Transformer: https://arxiv.org/pdf/2106.01345.pdf | SuperGLUE: https://super.gluebenchmark.com/leaderboard
Exploding Computational Requirements, source: NVIDIA Analysis and https://github.com/amirgholami/ai_and_memory_wall

NVIDIA H100
AI

AI/HPC
4PF FP8 (6X)| 2PF FP16 (3X)| 1PF TF32 (3X)| 67TF FP64 (3X)
3.35TB/s (1.5X), 80GB HBM3 memory

TRANSFORMER
6X faster on largest transformer models

7 Fully isolated & secured instances, guaranteed QoS
2nd Gen MIG | Confidential Computing

900 GB/s GPU-2-GPU connectivity (1.5X) | 128GB/s PCI Gen5

Custom TSMC 4N Process | 4.9 TB/s Total External B/W

https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet

FP8, FP16, TF32 performance include sparsity. X-factor compared to A100

H100

TRAINING

Up to 9X More Throughput
20 hrs totrain

HPC

Up to 30X More Throughput

Up to 7X Higher Performance
7X

30X

A100Throughput per GPU

9X
Faster

4000

8000

6X

25X

5X

20X

20X
4X

15X

16X

3X

10X
2X

7 days totrain

5X
1X

0X

128

7X

30X

6X

H100

Throughput – Sequences / Sec

INFERENCE#GPUs

Mixture of Experts (395B) Training vs A100

2 sec

1.5 sec

1 sec

Latency

Megatron 530B Throughput vs A100

0X

3D FFT

Genome Sequencing

HPC App Performance vs A100

Projected performance subject to change
Training Mixture of Experts (MoE) Transformer Switch-XXL variant with 395B parameters on 1T token dataset | Inference on Megatron 530B parameter model based chatbot for input sequence length=128, output sequence length =20 | HPC 3D
FFT (4K^3) throughput | A100 cluster: HDR IB network | H100 cluster: NVLink Switch System, NDR IB
HPC Genome Sequencing (Smith-Waterman) | 1 A100 | 1 H100

MULTI-INSTANCE GPU (MIG)

GPC
1c.4g.40gb

1H100 GPU

GPC

GPC

7GPC

GPC

GPC

GPC

コンピュート
インスタンスコンピュートインスタンス

メモリ

メモリ

メモリ

GPU インスタンス
4g.40gb

GPU インスタンス
2g.20gb

GPU インスタンス

1c.4g.40gb

1c.4g.40gb

1c.4g.40gb

GPU

1g.10gb

DGX H100 Hardware Feature SummaryGPU

8 x NVIDIA H100 Tensor Core GPUs

GPU memory

640 GB total

Performance

32 petaFLOPs FP8NVswitches

4CPUs

2 x x86

System Memory

2 TB

Networking
4 x OSFP ports provide connectivity to 8 x NVIDIAconnectivity and speed ConnectX-7s
8 x 400Gb/s InfiniBand/Ethernet*
2 x dual-port NVIDIA ConnectX-7
1 x 400Gb/s InfiniBand/Ethernet and1 x 200Gb/s InfiniBand/Ethernet*
Cache Storage

8 x U.2 3.84TB NVMe Secure Erase Drives

Boot Storage

2 x 1.92TB M.2 NVMe (software encryptable)

Host Management

On-board 10Gb/s RJ-45 Ethernet
Optional 50Gb/s QSFP Ethernet NIC

Remote System
Management

Baseboard Management Controller (BMC)
1Gb/s RJ-45 network connectivity
Remote Keyboard, Video, Mouse (KVM)
Remote Storage
RedFish, IPMI management

Operating System

DGX OS based on Ubuntu
Additional support for Ubuntu and Red Hat EnterpriseLinuxNVIDIA DGX H100
The world’s first AI system with the NVIDIA H100 Tensor Core GPU* Subject to Change

DGX H100 Exploded ViewBezel
GPU trayMotherboard

Fan Modules

ChassisFront Console
Board

Front Cage
Secure Erase Drives

Power Supplies

DGX H100 Block Diagram

Understanding DGX BasePOD vs DGX SuperPOD

NVIDIA DGX BasePOD

Scalable, foundational architecture••••••Flexible reference architectures
Powered by Base Command
Sold through channel
Partner professional services
Validated against key benchmarks
Foundation for partner branded offerings

Customer needs:​• Choice of flexible vs performance optimized designs• Inclusion of non-SuperPOD certified storage

NVIDIA DGX SuperPOD

Digital twin of NVIDIA’s infrastructure• Turnkey data center product• Powered by Base Command• Sold through channel, but with mandatory…
…white-glove NVIDIA PS – order > install > commission• Certified performance for the most complex workloads• No customization, no partner re-branding
Customer needs:• a replica of NVIDIA’s infrastructure​• a turnkey deployment​• full-stack support of the entire deployment

NVIDIA H100 PCIE

AI/HPC
3.2PF FP8 (5X) | 1.6PF FP16 (2.5X) | 800TF TF32 (2.5X) | 48TF FP64 (2.5X)
6X faster Dynamic Programming with DPX Instructions
2TB/s , 80GB HBM2e memory

Configurable TDP - 200W to 350W
2 Slot FHFL mainstream form factor

7 Fully isolated & secured instances, guaranteed QoS
2nd Gen MIG | Confidential Computing

128GB/s PCI Gen5600 GB/s GPU-2-GPU connectivity (5X PCIe Gen5)
up to 2 GPUs with NVLink Bridge

FP8, FP16, TF32 performance include sparsity. X-factor compared to A100

NVIDIA H100 GPU + NVIDIA AI ENTERPRISE
NVIDIA AI ENTERPRISE
5
Compute Accelerator for AI, HPC, DataProcessing• PCIeH100
H100 GPU 1

NVIDIA Certified
NVIDIA AI Enterprise• Business Standard Support•: nvidia.com/activate-h100

Fastest Mainstream Compute
Ready for Enterprise AI

H100 PCIe350W | 80GB
2-Slot FHFL

NVIDIA AI Enterprise not included with HGX H100 (SXM form factor)

H100

GREEN500

https://blogs.nvidia.co.jp/2022/11/16/green/NVIDIA RTX 6000 Ada

NVIDIAL40
Ada for the Data Center

Ada for the Enterprise Desktop• Up to 2x Faster than RTX A6000• Up to 2x Faster than A40• 48GB GDDR6 w/ECC Memory• 48GB GDDR6 w/ECC Memory• 4x DP1.4 Display Outputs• 4x DP1.4 Display Outputs• PCIe Gen 4• PCIe Gen 4• vGPU Support

Ada is incredibly energy efficient – over twice the performance at the same power compared to Ampere.

• vGPU Support• Available from channel partners
starting in December 2022, OEMpartners early 2023

• Availability starting in December 2022

*Specifications, availability dates subject to change. vGPU support in Q1 2023

Ada Lovelace
RTX

NVIDIA GRACE CPU

Grace CPUAI

HPCCPUCPUSuperchip Design with 144 high-performance Neoverse V2 Cores
Estimated Specrate2017_int_base of over 740

World’s first LPDDR5x memory with ECC, 1TB/s Memory Bandwidth2X Perf/Watt, CPU Cores + Memory in 500W

22x density of DIMM based designs

NVIDIA
RTX, HPC, AI, Omniverse: 2023

Grace Hopper
AICPU+GPUGrace CPU plus Hopper GPU Acceleration

GPU600GBEnables Giant AI Models for Training & Inference

3.5TB/s
LPDDR5x and HBM3900GB/S
NVLink-C2C connecting Grace to Hopper

GPU

15
NVLink-C2C vs PCIe

NVIDIA
RTX, HPC, AI, Omniverse: 2023HPC

Grace CPUGrace Hopper
HPC

4

HGX A100 Cluster

X86 CPU

GRACE CPU Superchip

GRACE HOPPER Superchip

Grace Hopper Superchip and Grace CPU Superchip Pre-silicon ESTIMATES ONLY: | Performance comparisons: (Left) AMD Milan CPU baseline| BWA-MEM2 (Single Socket/Chip) | NEMO (Dual Socket/Chip) | (Right) Cluster
Comparison: 32 node HGX (4x A100 SXM4 80 GB & 4x HDR InfiniBand) baseline and 128x Single Grace-Hopper 66T / 80 GB nodes | LAMMPS datasets ReaxFF/c & SNAP (geomean) | OpenFOAM dataset Lid driven cavity |GROMACS dataset STMV 1066628 atoms

AI

HPC

LiU

NERSC

NVIDIALANLNVIDIA (EOS)

18 EFLOPS AI Perf
Quantum-2 400Gb/s InfiniBand

ANLCambridge-1
JülichBristolCSCSCINECABSC

TACC UoF

Naver

U ofTsukuba
JAMSTEC

C-DAC

LANL (Venado)
1 EFLOPS AI Perf

BSC (MareNostrum 5)

18 EFLOPS AI Perf
Quantum-2 400Gb/s InfiniBand

CSCS (ALPS)

20 EFLOPS AI Perf

Hopper + X86 Systems: University of Tsukuba, Bristol, and TACC | Grace Hopper / Grace CPU Superchips Systems: BSC, CSCS, and LANLNVIDIANVIDIA HPC

NVIDIA AI

GPUCPUInfrastructure Workload• Communications• Storage• Security and Isolation

NIC

NVIDIA Omniverse

AI

PLATFORMS

NVIDIA HPC

NVIDIA AIGPUNVIDIA Omniverse

GPUFREEDCPUCPU
Cloud NativeSupercomputing

FREEDBlueField DPUNIC

AI APPLICATIONFRAMEWORKPLATFORMS

NVIDIA HPC

NVIDIA AI

ACCELERATION LIBRARIESIn-Network Computing
GPUCPUDPUNVIDIA Omniverse

DPU

Infrastructure Services

BlueField-2

BlueField-3200Gb/s

400Gb/s

DOCA
Orchestration

RDMA215Mpps

Computeプラットフォーム
インフラストラク
チャ370Mpps

SPECINT2K1
7: 9.8SPECINT2K17: 42

17GB/s

80GB/s

BlueField Infrastructure
Compute Platform

Security

Networking

Management

Acceleration Libraries
Telemetry

StorageNVIDIA Quantum-2 InfiniBand platform In-Network Computing

QUANTUM-2 SWITCH
64-Ports of 400 Gbps or 128-Ports of 200Gbps

CONNECTX-7 INFINIBAND
16 Core / 256 Threads Datapath Accelerator

BLUEFIELD-3 INFINIBAND
16 Arm 64-Bit CoresSHARPv3 Small Message Data Reductions

Full Transport Offload and Telemetry

16 Core / 256 Threads DatapathAccelerator

SHARPv3 Large Message Data Reductions

Hardware-Based RDMA / GPUDirect

Full Transport Offload and Telemetry

32X More AI Acceleration Engines

MPI Tag Matching and All-to-All

Hardware-Based RDMA / GPUDirect

PCIe Gen 5

Computational Storage
Security Engines
MPI Tag Matching and All-to-All

CloudBare-Metal ServerManagement

DOCA Services

DOCA SDKBlueField DPU

Management

HPCServerDPUUFM

DOCA

HPC

HPC-X

UCC

MVAPICH

UCX

OPEN-MPI

FILE SYSTEMSCHEDULERS

Storage

Performance
Isolation

Orchestration

SDKWith BlueField DPU

Barcelona
Supercomputing Center

DurhamUniversity​

Georgia Instituteof Technology

Japan
Meteorological Agency

Quantum In-Network Computing

Los Alamos
National Laboratory

Ohio State
University

Sandia National
Laboratory

Technical University
MunichUniversity College
LondonNVIDIA

ビジネス
アプリケーション

カスタマー
エンゲージメント

医療診断

異常検知

NGC
アプリケーション
フレームワーク

Con

開発者
ツールキット

品質保証

自動化

マーケティングAI
Metropolis

Riva

ML & DATA ANALYTICS

++

++

Drive

MerlinAI TRAINING & INFERENCE

HIGH PERFORMANCECOMPUTING

TensorRT

NVIDIA HPC SDKCOMPUTE

高速化
ライブラリ

シミュレーション

Clara...
VIRTUAL GPU
SOFTWARE

RENDERING &VISUALIZATIONIndeX

DOCA

FLEET
COMMAND

NVIDIA GPUOperator

OptiX

NETWORKING, STORAGE & SECURITY

CUDA-XTRITON
INFERENCESERVER

MANAGEMENTMAGNUM IO

NVIDIA CERTIFIEDVALIDATEDSOLUTIONS

DGX
HGX

サーバー及びクラウド

EGX

DCGM

UFM

ハードウェア技術
GPU

NVSwitch

BlueField DPU

SMART NIC

Mellanox Switch