PROGRAMMING ENVIRONMENTFOR HPC+AI
Oct 2019

Introduction
CUDA Development Ecosystem

AGENDA

Mixed Precision Acceleration and Tensor CoresNGC

2

RISE OF GPU COMPUTINGAPPLICATIONSALGORITHMS
SYSTEMS
CUDA

1000X
by2025

GPU-Computing perf
1.5X per year

107
106

1.1X per year

105
104
103

1.5X per year

102
Single-threaded perf
ARCHITECTURE

Original data up to the year 2010 collected and plotted by M. Horowitz, F. Labonte, O. Shacham,K. Olukotun, L. Hammond, and C. Batten New plot and data collected for 2010-2015 by K. Rupp

1980

1990

2000

2010

2020

3

ACCELERATED COMPUTINGIS FULL-STACK OPTIMIZATION
2X More Performance with Software Optimizations Alone
HPC Applications Speedup
24x

CUDA 10CUBLAS 10
CUFFT 1020xVolta

16x2X

12x8xon same
hardware

CUDA 8CUBLAS 8
CUFFT 84x0x
Nov-16

Mar-172x Broadwell vs 4xP100

Nov-17Mar-18

Nov-182x Broadwell vs 4xV100

HPC Apps: AMBER, Chroma, GROMACS, GTC, LAMMPS, MILC, NAMD, QE, RTM, SPECFEM3D, VASP

4NVIDIA DATA CENTER PLATFORM
Single Platform Drives Utilization and Productivity
CUSTOMERUSE CASESSpeech

Translate

Recommender

Healthcare

Manufacturing

FinanceCONSUMER INTERNET & INDUSTRY APPLICATIONS

Molecular
Simulations

Amber

cuDF

cuML
cuDNN

cuGRAPH

DEEP LEARNING
cuDNN

CUTLASSOpenACC

Knowledge
Workers

VIRTUAL GRAPHICS

Applications

HPC

TensorRT

Creative &Technical+600

NAMD

MACHINE LEARNING

Seismic
Mapping

SCIENTIFIC APPLICATIONSAPPS &FRAMEWORKS

CUDA-X &NVIDIA SDKs

WeatherForecasting

VIRTUAL GPU

cuFFTvDWSvPCvAPPS

CUDA & CORE LIBRARIES - cuBLAS | NCCLTESLA GPUs& SYSTEMSTESLA GPUNVIDIA DGX FAMILYNVIDIA HGXEVERY OEMEVERY MAJOR CLOUD5BEYOND MOORE’S LAW
2013

Progress Of Stack In 6 Years
1000

cuBLAS: 5.0

2019
cuBLAS: 10.0
cuFFT: 10.0

cuRAND: 5.0

cuRAND: 10.0

cuSPARSE: 5.0
NPP: 5.0
Thrust: 1.5.3
CUDA: 5.0
Resource Mgr: r304
Base OS: CentOS 6.2

Relative
Performance
Relative
Performance

cuFFT: 5.0

cuSOLVER: 10.0

100
GPU-Accelerated
Computing

NPP: 10.0
Thrust: 1.9.0
CUDA: 10.0

10Moore’s Law
CPU1MarchMar-12

Accelerated ServerWith Fermi

cuSPARSE: 10.0

2013
Mar-132014
Mar-142015
Mar-152016
Mar-162017
Mar-172018
Mar-18Resource Mgr: r384
Base OS: Ubuntu 16.04

2019
Mar-19Accelerated Serverwith Volta
6

CUDA DEVELOPMENTECOSYSTEM
7CUDA DEVELOPMENT ECOSYSTEMGPU Users

DomainSpecialists

ProblemSpecialists

New Algorithm Developers andOptimization Experts
CUDA-C++
CUDA FortranApplications

Frameworks

Ease of use

Libraries

Directives andStandard Languages

Extended Standard
Languages

Specialized Performance

CUDA: Programming Model, GPU Architecture, System Architecture
8

CUDA TOOLKIT

Libraries, Languages and Development Tools for GPU Computing
Programming
Approaches
Development
Environment

Language Support

Libraries

Programming Languages“Drop-in” Acceleration

Maximum Flexibility

Nsight IDECNVIDIA
Visual Profiler

C++

CUDA Profiling
Tools Interface

CUDA-GDB
Debugger

CUDA
MEMCHECK

FortranCompile new
languages to CUDA
9RAPID ADOPTION OF OPENACC
Simple| Performant| Portable
200
ADS CFD
CFD

ANSYS FLUENT
CFDCASTRO / MAESTRO
Astrophysics

CGYRO
PlasmaCOSMO
Weather

E3SM

Climate

150

FINE / OPEN

FLASH

CFD

Astrophysics

HIPSTAR

IBM-CFD

CFD

NEKCEM

Electromagnetics

CFD

SANJEEVINIDrug Discovery

GAMERA
Seismic

LAVA

GAUSSIAN
Chemistry

LSDALTON

GTC

HIFUN
CFD

10050

Plasma Physics

CFD

Quantum Chemistry

Astrophysics

MAS

MPAS-ASOMASYNOPSYSVASP

VMDPhysics

Material Science

ACROSS SCIENTIFIC DOMAINS

Material Science

WeatherMolecular Dynamics

0Mar-15

Mar-16Mar-17Mar-18Mar-19~200 APPS BEING ACCELERATED
10DIRECTIVE-BASED HPC PROGRAMMING
Who’s Using OpenACC?

3 OF TOP 5 HPC APPS

5 OF 13 CAAR CODESACCELERATED APPS
19410011653

GTC16 GTC17 GTC18 GTC19725 TRAINED EXPERTSSLACK MEMBERS

160,000+ DOWNLOADS
846326102
GTC17GTC18GTC1911OPENMP OFFLOADING12NEW LANGUAGES
Developers want to build frontends for:
•CUDAC, C++, FortranJava, Python, R, DSLsLLVM Compiler
For CUDA

Target other processors like:•ARM, FPGA, GPUs, x86

CUDA Compiler Contributed toOpen Source LLVM

New Language
SupportNVIDIAGPUsx86CPUs

New ProcessorSupport

13MIXED PRECISIONACCELERATION ANDTENSOR CORES
14TENSOR CORES FOR SCIENCE
Mixed-Precision Computing

V100
TFLOPS
140

125

120
100
80
60

FP16 Solver
3.5x fasterFP16-FP21-FP32-FP64
25x fasterFP16/FP32/FP64
4x faster

PLASMA FUSIONAPPLICATION

EARTHQUAKE SIMULATIONMIXED PRECISION WEATHERPREDICTION

40207.8

15.7

0FP64+ MULTI-PRECISION

15AUTOMATIC MIXED PRECISION
Easy to Use, Greater Performance and Boost in Productivity

Insert ~ two lines of code to introduce Automatic Mixed-Precision and get upto 3X speedup
AMP uses a graph optimization technique to determine FP16 and FP32 operations
Support for TensorFlow, PyTorch and MXNet

Unleash Next-Generation AI Performance and Accelerate Time to ROI

16

ENABLING AUTOMATIC MIXED PRECISION
Add Just A Few Lines of Code, Get Upto 3X Speedup
os.environ['TF_ENABLE_AUTO_MIXED_PRECISION'] = '1'
OR

export TF_ENABLE_AUTO_MIXED_PRECISION=1TensorFlow

Explicit optimizer wrapper available inNVIDIA Container 19.07+, TF 1.14+, TF
2.0:

GA

opt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)

PyTorchmodel, optimizer = amp.initialize(model, optimizer,opt_level="O1")with amp.scale_loss(loss, optimizer) as scaled_loss:
scaled_loss.backward()

MXNet

amp.init()
amp.init_trainer(trainer)
with amp.scale_loss(loss, trainer) as scaled_loss:
autograd.backward(scaled_loss)

More details: https://developer.nvidia.com/automatic-mixed-precision

GAGA
ComingSoon

173

ENABLING AUTOMATIC MIXED PRECISION
Add Just A Few Lines of Code• TensorFlow•

NVIDIA container 19.03+:• export TF_ENABLE_AUTO_MIXED_PRECISION=1 [automatic casting and automatic loss scaling]•Available in NVIDIA container 19.07+, TF 1.14+, TF 2.0:•

We provide an explicit optimizer wrapper to perform loss scaling – which can also enable auto-casting for you:import tensorflow as tf
opt = tf.train.GradientDescentOptimizer(0.5)
opt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)184

ENABLING AUTOMATIC MIXED PRECISION
Add Just A Few Lines of Code•

PyTorch•Two steps: initialization and wrapping backpropagationfrom apex import amp
model = …
optimizer = SomeOptimizer(model.parameters(), …)# …
model, optimizer = amp.initialize(model, optimizer, opt_level=“O1”)# …
for train_loop():loss = loss_fn(model(x), y)with amp.scale_loss(loss, optimizer) as scaled_loss:
scaled_loss.backward()# Can manipulate the .grads if you’d like
optimizer.step()
195

ENABLING AUTOMATIC MIXED PRECISION
Add Just A Few Lines of Code•MXNET•NVIDIA container 19.03+ and MXNET 1.5:

from mxnet.contrib import amp
amp.init()
net = get_network()
trainer = mx.gluon.Trainer(...)
amp.init_trainer(trainer)
for data in dataloader:with autograd.record(True):out = net(data)l = loss(out)with amp.scale_loss(l, trainer) as scaled_loss:
autograd.backward(scaled_loss)
trainer.step()206

AUTOMATIC MIXED PRECISION IN TENSORFLOW
Upto 3X Speedup

TensorFlow Medium Post: Automatic Mixed Precision in TensorFlow for Faster AI Training on NVIDIA GPUs
All models can be found at:https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow, except for ssd-rn50-fpn-640, which is here: https://github.com/tensorflow/models/tree/master/research/object_detection
All performance collected on 1xV100-16GB, except bert-squadqa on 1xV100-32GB.
Speedup is the ratio of time to train for a fixed number of epochs in single-precision and Automatic Mixed Precision. Number of epochs for each model was matching the literature or common practice (it was also confirmed that both training sessions achieved the same model accuracy).
217Batch sizes:. rn50 (v1.5): 128 for FP32, 256 for AMP+XLA; ssd-rn50-fpn-640: 8 for FP32, 16 for AMP+XLA; NCF: 1M for FP32 and AMP+XLA; bert-squadqa: 4 for FP32, 10 for AMP+XLA; GNMT: 128 for FP32, 192 for AMP.

AUTOMATIC MIXED PRECISION IN PYTORCH
https://developer.nvidia.com/automatic-mixed-precision●

Plot shows ResNet-50 result with/without automatic mixed
precision(AMP)●More AMP enabled model scripts coming soon:

2X

Mask-R CNN, GNMT, NCF, etc.

AMP
EnabledFP32

Source: https://github.com/NVIDIA/apex/tree/master/examples/imagenet

MixedPrecision

228

AUTOMATIC MIXED PRECISION IN MXNET
AMP speedup ~1.5X to 2X in comparison with FP32

https://github.com/apache/incubator-mxnet/pull/14173

239

Matching Accuracy for FP32 and Mixed Precision
Model Script

Framework

Data Set

Automatic or
ManualMixed-Precision

FP32
Accuracy

Mixed-Precisi
on Accuracy

FP32
Throughput

Mixed-Precision
Throughput

Speedup

BERT Q&A

TensorFlow

SQuaD

AMP

90.83
Top 1

90.99
Top 1

66.65
sentences/sec

129.16sentences/sec

1.94

TensorFlow

COCO 2017

AMP

0.268
mAP0.269mAP569images/sec

752
images/sec

1.32

PyTorchWMT16
English toGerman

Manual

24.16
BLEU

24.22
BLEU

314,831
tokens/sec

738,521
tokens/sec

2.35

PyTorchMovieLens
20M

Manual

0.959
HR0.960HR

55,004,590
samples/sec

99,332,230
items/sec

1.81

TensorFlow

DAGM 2007

AMP

0.965-0.988

0.960-0.988

445
images/sec

491
images/sec

1.10

MXNet

ImageNet

Manual

76.67
Top 1%

76.49
Top 1%

2,957
images/sec

10,263
images/sec

3.47

PyTorchLJ Speech
DatasetAMP

0.3629/-6.1087

0.3645/-6.0258

10,843 tok/s
257,687 smp/s

12,742 tok/s
500,375 smp/s

1.18/
1.94

(2)

SSD w/RN50
(1)

GNMT
(3)

NeuralCollaborativeFilter
(1)

U-Net
Industrial(1)

ResNet-50 v1.5
(1)

Tacotron 2 /WaveGlow 1.0
(1)

24
25

Values are measured with model running on (1) DGX-1V 8GPU 16G, (2) DGX-1V 8GPU 32G or (3) DGX-2V 16GPU 32G

NGC25

NGC: GPU-OPTIMIZED SOFTWARE HUB
Ready-to-run GPU Optimized Software, Anywhere

50+ Containers
DL, ML, HPC

15+ Model Training Scripts

NLP, Image Classification, Object Detection &more

NGC

60 Pre-trained Models

NLP, Image Classification, Object Detection& moreIndustry Workflows

Medical Imaging, Intelligent VideoAnalytics

On-premCloud

Hybrid Cloud

Multi-cloud26

NGC MODEL SCRIPTS

Tensor Core Optimized Deep Learning Examples
18 Available●
Tensor Core optimized for greater performance●
Test drive automatic mixed precision●
Actively updated by NVIDIA●
State-of-the-art accuracy using Tensor Cores●
Serves as a reference implementation●Exposes hyperparameters and source code for further adjustment
Accessible via:
●NVIDIA NGC https://ngc.nvidia.com/catalog/model-scripts●GitHub https://www.github.com/NVIDIA/deeplearningexamples●NVIDIA NGC Framework containers https://ngc.nvidia.com/catalog/containers
27

NVIDIA NGC MODEL SCRIPTSTensor Core Examples Built for Multiple Use Cases and Frameworks

A dedicated hub to download Tensor Core Optimized Deep Learning Examples on NGC

https://ngc.nvidia.com/catalog/model-scripts?quickFilter=deep-learning

28
19

NGC PRE-TRAINED MODEL FILES AVAILABLE•Published for multiple frameworks: TensorRT, TensorFlow, PyTorch, and MXNet.

•Trained on multiple industry data sets: ImageNet, MSCOCO, LibreSpeech,
Wikipedia/BookCorpus, etc.

•Provided in the most popular precisions: FP32, FP16, and INT8•

Key customer benefits:○

Quick experimentation with our trained models to understand inference
performance/latency○

Building blocks for model ensembling pipelines, such as Speech AI: ASR->BERT Q&A->TTS○

Reproducibility of our inference benchmarks
29Introduction
CUDA Development Ecosystem

AGENDA

Mixed Precision Acceleration and Tensor CoresNGC30