Sony's deep learning development environment
for massively large scale training
Takuya Narihira & Hisahiro Suganuma
8th ADAC Workshop, Oct 30, 2019Sony Corporation R&D center
Copyright 2019 Sony Corporation(Supplementary information)

Agenda• Deep Learning at Sony• Large Scale Training•Large model training•

Distributed training

2

2019.10.30

Sony Corporation R&D CenterADAC

Deep Learning at Sony

Sony’s Product

Deep Learning is being utilized in many application domains.
4

2019.10.30

Sony Corporation R&D CenterADAC

Software 2.0

https://medium.com/@karpathy/software-2-0-a64152b37c35
https://media.neurips.cc/Conferences/NIPS2018/Slides/Olukotun-NIPS-2018.pdf

Software 1.0••••

Write in code (C++, Python …)
Requires domain expertise
Hard to maintain
Does not scale

Software 2.0••••

Written in the weight of NN by optimization
Requires data
Easier to maintain (Data tells)
Scale empowered by data

“Gradient Descent can write code better than you. I’m sorry.“
Andrej Karpathy

We need techniques, infrastructures and echo-systems for software 2.0 era
52019.10.30

Sony Corporation R&D CenterADAC

Sony's deep learning software
Neural Network Console

Neural Network Libraries

dl.sony.comnnabla.org

GUI based deep learning IDE

Deep learning framework with Python API

Windows Desktop (free) & Cloud (paid)

Open source (Apache 2.0 license)

Intuitive, fast and easy to deploy
62019.10.30

Sony Corporation R&D CenterADAC

NNC demo

7

2019.10.30

Sony Corporation
R&D Center
Easy
model
building, experiment, and deployment

ADACNeural Network Console x ABCI
8
Training
can be distributed over GPUs by just clicking a couple of times in GUI

AI Bridging Cloud Infrastructure (ABCI)

https://github.com/aistairc/abci-docs/

The world's first large-scale Open AI Computing Infrastructure,
constructed and operated by National Institute of Advanced
Industrial Science and Technology (AIST).

92019.10.30

Sony Corporation R&D Center

ADACNeural Network Libraries

nnabla Model Format Converter

FormatConverter
Model
FormatConsoleProgramming
APIs & Runtime
Computebackend
OS102019.10.30

nnabla(C++,Python, CLI)

TF

NNB

ONNX

nnablaC-runtime

3rd party
Framework/Runtime

NNP

TF, PyTorch, TensorRT, SNPE etc.

CPU

CUDA

OpenCLCPU

ARM CMSISLinux, Windows, macOS, Android, iOS, RasPI, Embeded etc.
Sony Corporation R&D Center

ADAC

Easy network definition

Both static & dynamic graph paradigms are supported
112019.10.30

Sony Corporation R&D CenterADAC

nnabla-examples: A bunch of SoTA training scripts

https://github.com/sony/nnabla-examples

Generative models for content creation
Pix2PixHD

StarGANInstaGAN

Mask  Texture

Black  Blonde

Skirt  Pants

MUNIT

CycleGAN

Self-Attention GAN“dog”

Drawing  Texture

Horse  Zebra

Label  Image

Please use research baselines & application prototyping etc.
122019.10.30

Sony Corporation R&D CenterADAC

Large Scale TrainingWhy large scale training is important? 1/2Bigger models are better in performance (accuracy)
Recognition Cite: GPipe

Generation Cite: BigGAN

Performance (better)

Class conditional image generator

Image generation models has also becomebigger (requires 512 cores TPUv3 pod to train)
Network size (bigger)

Biggest models win inrecognition tasks

14

2019.10.30

Sony Corporation R&D CenterBigger

BetterADAC

Why large scale training is important? 2/2
Large dataset gives better performance

Data size becomes bigger

GAN progress on facial generation

Pretraining on bigger dataset gives
better performancehttps://twitter.com/goodfellow_ian/status/1084973596236144640?s=20

https://research.fb.com/publications/exploring-the-limits-ofweakly-supervised-pretraining/Bigger model, dataset, and data size impose large memory andlonger training time15

2019.10.30

Sony Corporation R&D CenterADAC

Large scale training: Large model execution in nnabla

Presentation only
16

2019.10.30

Sony Corporation R&D CenterADAC

Distributed Training

Introduction of techniques for scaling up
distributed training- toward over 100K mini-batch training –Hisahiro SUGANUMASony Corporation R&D center
Copyright 2019 Sony Corporation(Supplementary information)

Agenda•Overview of data parallel distributed training•

Problems regarding scaling up disrtibuted training•Techniques for large scale distributed training••

Brief introduction of our new optimizer “STiLL”

Experimental Result : ImageNet/ResNet-50 training

19

2019.10.30

Copyright 2019 Sony Corporation

Brief introduction of our recent activities• We are studying distributed training to train DNN models in shorter time.
• And to benchmark our framework, we have published the training results ofImageNet/ResNet-50 that contain the time of training, the final accuracy andtechniques for large batch training.

630 sec/bs=34k

224 sec/bs=34,68k

112 sec/bs=54k

Publish date20

2019.10.30

Copyright 2019 Sony Corporation

68 sec/bs=96k

Brief overview of Distributed Training

Paradigms of distributed training : Data and Model Parallelism
Data parallelism

Model parallelism

split
Train data

Train data

split
modelmodel

Each GPU has the same model weight andtraining with sharded data.

Each GPU has a different part of model weight.

Data parallelism

Model parallelism

Pros- It is simple to scale out the number of workers.

- It can deal with very big model like “Transformer”.

ConsIncreasing batch-size make optimization difficult
significantly.

- Suffering from computational inefficiency. ( Recently it isgetting more better.)

Scope of my talk22

2019.10.30

Copyright 2019 Sony Corporation

DNN Training on single GPU.
• DNN Training basically employs “Mini-batch training”• Training consist of fetch data, forward, backward and update weights.
• Iterative training with each split dataset “mini-batch” to optimize DNN parameters(weight)
gradually.
One iteration in the training
InputDataset
DNN Model

Fetch dataForwardBackward

Training
mini-batch training with a single GPUCPUTraining

23

2019.10.30

UpdateCopyright 2019 Sony CorporationGPUGPUGPU

N/W

OutputTrained ModelData-parallel distributed DNN Training• Data parallel training needs to synchronize the gradients among the all workers before it
updates the weights since it must maintain the consistency of the model.
• Synchronization is the obviously overhead to the time of training.
Fetch data
Input

ForwardTraining

DatasetBackward

sync Update

Training

Training

sync

DNN Model

Trained Modelmini-batch training for each GPUtraining

CPUCPUCPUGPUGPUGPU

GPU

GPU
GPU

N/W

N/W

GPU

training

parameters
GPUGPUGPUCPU

CPUCPUGPUGPU

parameters

N/W

N/W

GPU
2019.10.30

gradient-sync inter-GPU/internodeCPUtraining GPU

24

UpdateOutput

Copyright 2019 Sony Corporation

GPU
GPU

training

N/WN/WGPU

SyncGPUGPU
GPU

parameters
GPUN/WN/WCPU
GPU
parameters
GPUGPU

Problems regarding large scale training,and how do we solve them ?

Problems of distributed large-batch training.
1. To achieve convergence with very small number of update steps•

The larger the mini-batch size is, the more number of GPUs we can use.

•However, larger mini-batch size also makes it more difficult for convergence.
Mini-batch size will be N-times size, if using N GPUs.
The point is developing the way to train even with large mini-batch size.

2. To reduce gradient synchronization time•

Ring topology becomes too slow, if using more than hundreds of GPUs .

Out of scope in this talk

The point is developing a topology (for collective communication) which allow for an
efficient synchronization even with the thousands of GPUs.

26

2019.10.30

Copyright 2019 Sony Corporation

Convergence degradation in large-scale distributed DNN Training▶ Mini-batch size is also increased by a multiple of the number of GPUs.

With SGD, it is hard to train DNN with large batch-size because …

1.The number of training updates decreases(∝1/#workers)
𝑤𝑤 ∶ Weights
𝐵𝐵1 ∶ Mini−batch size𝜂𝜂 ∶ Learning rate

𝑤𝑤 𝑡𝑡+1 = 𝑤𝑤 𝑡𝑡 −2 workers

1 worker𝜂𝜂� 𝛻𝛻𝑙𝑙(𝑥𝑥𝑖𝑖 , 𝑤𝑤 𝑡𝑡 )𝐵𝐵1𝑥𝑥𝑖𝑖 ∈𝐵𝐵1

2. Easy to converge into sharp minima

𝑤𝑤 𝑡𝑡+1 = 𝑤𝑤 𝑡𝑡 −𝜂𝜂� 𝛻𝛻𝑙𝑙 𝑥𝑥𝑖𝑖 , 𝑤𝑤 𝑡𝑡
2 𝐵𝐵1𝑥𝑥𝑖𝑖 ∈𝐵𝐵2

Optimal#steps is halved.
Example: optimization in 2-dimension.

27

2019.10.30

Copyright 2019 Sony Corporation※ N. S. Keskar et al. “On Large-Batch Training for Deep Learning: Generalization Gapand Sharp Minima” ICLR 2017

Real world approaches to scale batch-size.
1. Tuning hyperparameters on SGD•

Many techniques were proposed.
E.g. “Learning rate warmup”, “LARS”, “LAMB”, …etc.

•We employ this way for several reasons. Mainly we consider that SGD to train DNN is wellstudied and we can use more wisdom of the past researches

2. Using second or higher order optimization like “Newton’s method”•They could optimize with smaller steps without sharp minima. But most of them need enormous
computing resource due to using hessian.

•

Recently proposed approximation have made us optimize DNN in real.

28

2019.10.30

Copyright 2019 Sony Corporation

Overview of common techniques for large batch training
Learning rate warmup*- Linearly scaling up LR in several epochs.

Warming up LR.

Label Smoothing***- Smoothing i-th label of Softmax output q(0 < ε < 1, K is the number of classes.)*** C. Szegedy et al Rethinking the Inception Architecture for Computer Vision(arxiv.org/abs/1512.00567)

LARS optimizer**

Zero-γ-init- Layer-wise Adaptive LR Scaling with the ratio ofweights and grads.

- Initilizing γ of batchnorm in residual path to 0.

conv* T.He et al. Bag of Tricks for Image Classification with Convolutional Neural Networks (arxiv.org/abs/1812.01187)** Y. you et al. Large Batch Training of Convolutional Networks (arxiv.org/abs/1708.03888)

BNRelu↑ Init γ = 0 here.

Up to 64K mini-batch these are enough to train.
However we need another technique to increase batch-size more and more !
29

2019.10.30

Copyright 2019 Sony Corporation

STiLL : Smoothly Transition from LAMB* to LARS.
• We proposed the new optimizer “STiLL” that uses two optimizer, LAMB and LARS at
the same time.
• STiLL starts from LAMB:LARS=100:0 to update the weights, then it linearly changes
the ratio of them during warmup duration.
• After the warmup duration, it uses only LARS to update weights.

*Y.You et al, Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
https://arxiv.org/abs/1904.00962

30

2019.10.30

Copyright 2019 Sony Corporation

Why we mixed them?
• We expected that LAMB accelerates convergence in the beginning of training andLARS helps to get better generalization performance by combining them.
↓ batch training with LARS and LAMB (these cases did not converge.)

LAMB is initially faster like AdamLARS reaches better solution
like Momentum SGD

312019.10.30

Copyright 2019 Sony Corporation

Other Technique – Smoothout, that strengthens regularization.
• Smoothout* is one of the regularization technique proposed by W.Wen.

•1.

Add noise from Uniform dist. to the weights before forward computation.

2.

Do forward and backward computation.

3.

Sub. noise from the weights.

4.

Update weights.

“Smoothout” can avoid sharp-minima, that is more important in large-batchtraining. We use this over 64K batch training.

*W.Wen et al. SmoothOut: Smoothing Out Sharp Minima to Improve Generalizationin Deep Learning https://arxiv.org/abs/1805.07898
32

2019.10.30

Copyright 2019 Sony Corporation

Experiments

Experiments(1): Performance comparison with LARS, LAMB and STiLL• Firstly, we conducted ImageNet/ResNet-50 training while changing optimizers,“LAMB”, “LARS” and “STiLL”, and increasing batch-size from 72K to 96K.
• STiLL outperforms “LAMB” and “LARS” in the final validation accuracy.

Training curve of LAMB, LARS and STiLL#epoch 70

Top-1 accuracy of ImageNet/ResNet-50
model after training for 90 epochs
80

9075%

34

2019.10.30

Copyright 2019 Sony Corporation

Mini-Batch Size

LAMB

LARS

STiLL

72K

70.9%

75.5%

76.1%

80K

N/A

75.1%

75.9%

96K

N/A

N/A

75.5%

Experiments(2): Performance of NNL in ABCI.
• We evaluated wallclock time to train for 90 epochs and convergence accuracy withSTiLL in ABCI.
- Settings- Training curve
Batch size

64K, 80K, 96K

STiLL’s Initial/MAX LR

Init: 1.36(LARS)/0.0012(LAMB), Max: 26.1

LARS Momentum

0.955

Warmup/LR Decay

Warmup:25epoch, and then poly decay.

Label-smoothing α

0.1

Weight decay

8e-5- Results
BS#GPU

WallclockAccuracy

64K

epoch
90

2048

97 sec76.13 %

80K

90

2560

79.7 sec

75.93 %

96K

90

3072

67.65 sec

75.46 %

35

2019.10.30

Copyright 2019 Sony Corporation

Experiments (3): Training longer epochs could give better performance• Then we made extra experiments to see if the model get better generalization
performance by training longer epochs.
• In the result, the model could reach good solution even over 100K batch training.
- Training curve- Settings- Use the same settings as the previous exp.

- Results
BSepoch#GPUAcc

96K

1103072

76.19%

128K

150409676.12 %

36

2019.10.30

Copyright 2019 Sony Corporation

Summary• We proposed the new optimizer “STiLL” for large-scale training.
• Then empirically showed that “STiLL” outperforms “LARS” and “LAMB” in Imageclassification task.
• Also we succeeded to train with 128K batch, however even longer epoch we neednow.

37

2019.10.30

Copyright 2019 Sony Corporation

Question?

38

2019.10.30

Copyright 2019 Sony Corporation

SONY is a registered trademark of Sony Corporation.
Names of Sony products and services are the registered trademarks and/or trademarks of Sony Corporation or its Group companies.
Other company names and product names are registered trademarks and/or trademarks of the respective companies.