8th ADAC Workshop
Yet Another Accelerated SGD: ResNet-50 Training in 70.4 sec.
October 30, 2019
Fujitsu Laboratories. LTD.
senior researcher

Masafumi Yamazaki
0Copy right 2019 FUJITSU LABORATORIES LIMITEDBackground The scale of Deep
Neural Network (DNN)
is growing

Complexity of Training
Neural Networks

widergap
The large-scaleparallelization is one of
the efficient way toaccelerate training speed.

Source： Dario Amodei and Danny Hernandez, https://openai.com/blog/ai-and-compute/
1Copy right 2019 FUJITSU LABORATORIES LIMITEDContents
1. Distributed Deep Learning History of DNN training speed challenges

 Our contributions

3. Optimization Points
 Backward/Allreduce parallelization

2. Key points for distributedtraining Optimization of initialization andunnecessary processing DNN training using Data parallel
Synchronous-SGD

 Optimization for MLPerf v0.6 (TheRule was changed from v0.5)

 Optimal mini-batch size
 Allreduce algorithms

4. Evaluation and results
2Copy right 2019 FUJITSU LABORATORIES LIMITEDHistory of training speed of DNN
2012

2017

ILSVRC2012• Barklay U.

AlexNettraining in 5-6 days using 2 GPUs

2014

• PFN
Training in 15 Minutes using 1,024 GPUs

2018

ILSVRC2014• google
GoogleNet
training in a week using few GPUs
(estimation in their paper)

2015• FacebookResNet-50 Training in 1 hour using 256 GPUs

• TencentTraining in 6.6 minute using 2,048 GPUs
• Google
Training in 1.8 minute using 1,024 TPUs

2019
ILSVRC2015• MICROSOFT ResNet• SonyTraining in 2.0 minute using 3,456 GPUs
…

2016

3

Copy right 2019 FUJITSU LABORATORIES LIMITEDOur contribution ... speed up
 In 2015, our group in Fujitsu Laboratories began working on large-scale distributed
trainingYearHardware#GPUs

DNN / Dataset

TimeRemarks

Feb, 2016

Tatara (Kyushu Univ.)

64

AlexNet / ImageNet-June, 2016

TSUBAME 2.5

256

AlexNet / ImageNet-Aug., 2018

ABCI~4096

ResNet-50 / ImageNet(6.6 minute)

The Accuracy didn’t reach 75%

April, 2019

ABCI

2048

ResNet-50 / ImageNet

74.7 seconds

arXiv:1903.12650

June, 2019

ABCI

2048

ResNet-50 / ImageNet

70.4 seconds

MLPerf v0.6

*1*1 SWoPP2016 「MPIを用いたDL処理高速化の提案」- evaluated the Allreduce algorithm- proposed running computation and communication processes in parallel- reported how accuracy worsened with large mini batch sizes.
4

Copy right 2019 FUJITSU LABORATORIES LIMITEDHistory of training speed of DNN
2012

2017

ILSVRC2012• Barklay U.

AlexNettraining in 5-6 days using 2 GPUs

2014

• PFN
Training in 15 Minutes using 1,024 GPUs

2018

ILSVRC2014• google
GoogleNet
training in a week using few GPUs
(estimation in their paper)

2015• Facebook
ResNet-50 Training in 1 hour using 256 GPUs

• TencentTraining in 6.6 minute using 2,048 GPUs
• Google
Training in 1.8 minute using 1,024 TPUs

2019
ILSVRC2015• MICROSOFT ResNet• SonyTraining in 2.0 minute using 3,456 GPUs
• MLPerf v0.6 (Our work)
Training in 1.2 minute using 2,048 GPUs

2016

• Huawei
Training in 1.0 minute using 1,024 A910
processors
5

Copy right 2019 FUJITSU LABORATORIES LIMITEDOur contribution ... mini-batch size

ImageNet top-1 Validation accuracy

 Facebook was able to increase the mini-batch size up to 8k using ideas such as
warm-ups. However, increasing the batch size further would worsen accuracy
larger mini-batch size

80GoogleBrain
Facebook

7570

PFNTencent
Sony2 GPUs

8 GPUs2048 GPUs
64 GPUs651,024 GPUs

We achieved accuracy
with up to 84k mini-batchusing 2048 GPUs

60

Source: Accurate, Large Minibatch SGD: Training
ImageNet in 1 Hour, P. Goyal (Facebook) et al, 2017
6

Copy right 2019 FUJITSU LABORATORIES LIMITEDKey points for distributed training Data Parallel Method Based on Synchronous-SGD using Allreduce Optimal mini-batch size
 Allreduce algorithms

7

Copy right 2019 FUJITSU LABORATORIES LIMITEDData Parallel Method
 We have continued to accelerate training using a data parallel method based onsynchronous-SGD using Allreduce#0Training
Iteration

ForwardBackward
UpdateNumber of Images for Training is 1,281,167 (ImageNet-1k)

mini-batch size
8Copy right 2019 FUJITSU LABORATORIES LIMITEDData Parallel Method Since 2016, we have continued to accelerate training using a data parallel method
based on synchronous-SGD using Allreduce#0#1

Training
Iteration#2#3

ForwardBackward
Allreduce
UpdateNumber of Images for Training is 1,281,167 (ImageNet-1k)

mini-batch size

Therefore, a single GPU will process less datawhen using a large number of GPUs for training.
9Copy right 2019 FUJITSU LABORATORIES LIMITED

Optimal mini-batch size
 The performance gets worse as the amount of data to process for eachGPU decreases

Speed [arb. unit]

Numberof images per GPU
系列1100系列2系列3系列4系列5The performance gets worse as the amount ofdata to process for each GPU decreases
1032- TSUBAME2.5 system- Tesla K20X (using 1GPU per node)

64

128

256

Number of GPUs
10

Copy right 2019 FUJITSU LABORATORIES LIMITED

Optimal mini-batch size
 We selected the optimal mini batch size for enough accuracy
Weak scale
Increases the amount of imagesper iteration in proportion to thenumber of accelerators
Pros; good scalability
Cons; accuracy down in a large
mini batch

Speed [arb. unit]

Numberof images per GPU
系列1100系列2系列3系列4系列5Strong scale
1032- TSUBAME2.5 system- Tesla K20X (using 1GPU per node)

64

128

256

Number of GPUs
11

Execute the same process bydividing it with an accelerator
Pros; get the same result
Cons; Bad scalability- Reduced parallelism of GPU- Communication overhead
Copy right 2019 FUJITSU LABORATORIES LIMITEDComparison of Allreduce algorithms

algorithm

Data size to transfer#stepsRemarks

Recursive
Halving/Doubling
Ring
Double Tree

12

Copy right 2019 FUJITSU LABORATORIES LIMITEDRecursive Halving/Doubling Algorithm
0+1+2+3

rank 00

rank 11

rank 2

2

rank 3

3

Recursive
Halving/Doubling

0+1+2+3

0+1

2+3

step1

algorithm

0+1+2+3

step2

step3

step4

step

Data size
to transfer#stepsRemarks

𝑁 𝑁~
2 𝑚

2 × 𝑙𝑜𝑔2 𝑚

We Implemented※ N, m are the Data size and the number of ranks, respectively.
13

Copy right 2019 FUJITSU LABORATORIES LIMITEDRing algorithm
0+1+2+3

rank 00rank 1

1

rank 2

2

rank 33Ring
2D – Ring

0+1+2+3
0+1+2+3

1+2
1+2+3

step1

algorithm

0+1+2+3

step2

step3

step4

step5

Data size
to transfer#steps𝑁𝑚2 × (𝑚 − 1)

𝑁 𝑁,
𝑚𝑥 𝑚𝑦2 × (𝑚𝑥 + 𝑚𝑦 − 2)

step6

step

Remarks

NCCL 2.3※ N, m, mx, my are the Data size, the number of ranks, the x-ring size and y ring size, respectively
14

Copy right 2019 FUJITSU LABORATORIES LIMITEDDouble Tree algorithm

source: NCCL v2.4 commentsalgorithm
Double Tree

Data sizeto transfer#stepsRemarks

𝑁22 × 𝑙𝑜𝑔2 𝑚

NCCL 2.4※ N, m are the Data size and the number of ranks, respectively.
15

Copy right 2019 FUJITSU LABORATORIES LIMITEDComparison of Allreduce algorithms
algorithm
Recursive
Halving/Doubling
Ring
2D – Ring

Double Tree

Data size to transfer#stepsRemarks

𝑁 𝑁~
2 𝑚

2 × 𝑙𝑜𝑔2 𝑚

We Implemented

2 × (𝑚 − 1)

𝑁𝑚

NCCL 2.3

2 × (𝑚𝑥 + 𝑚𝑦 − 2)𝑁2

2 × 𝑙𝑜𝑔2 𝑚

NCCL 2.4※ N, m, mx, my are the Data size, the number of ranks, the x-ring size and y ring size, respectively
16

Copy right 2019 FUJITSU LABORATORIES LIMITEDEvaluate the Allreduce algorithms
100

System: ABCI
Number of nodes: 128
Number of GPUs: 512
Latency [ms]10

2D Ring (NCCL2.3)

1

Recursive Halving/Doubling(Our work)

The data size regions often usedby Allreduce of distributed training

0.1
16K

32K

64K

128K

256K

512K

1M

2M

4M

8M

16M

32M

Data Size [Byte]
17

Copy right 2019 FUJITSU LABORATORIES LIMITEDEvaluate the Allreduce algorithms
100

System: ABCI
Number of nodes: 128
Number of GPUs: 512
Latency [ms]10

2D-Ring (NCCL2.3)
1

Recursive Halving/Doubling(Our work)
Double Tree (NCCL2.4)

The data size regions often usedby Allreduce of distributed training

0.1
16K

32K

64K

128K

256K

512K

1M

2M

4M

8M

16M

32M

Data Size [Byte]
18

Copy right 2019 FUJITSU LABORATORIES LIMITED

Optimization Points
 Accelerate training speed Optimization of initialization and unnecessary processing
 Changes for MLPerf v0.6

19

Copy right 2019 FUJITSU LABORATORIES LIMITEDAccelerate Training Speed
 Overlapping Allreduce communication with backward computationforwardupdatebackward

Allreduce Allreduce

Allreduce

overheadAllreduce

Allreduce

ForwardUpdateBackward

Allreduce

The processing speed increased to over 1.5 M images/sec.
However we could not reach our goal for the overall training time.
20

Copy right 2019 FUJITSU LABORATORIES LIMITED

Optimization of initialization and unnecessary processes
Initialization

Validation

Training

1 epoch

①②

③

③

③

Time

①generate common initial-weights using common random-seed in each GPU instead ofbroadcasting initial weights from one GPU②Overlapping NCCL initialization with framework initialization③Eliminating unnecessary processes after each epoch （0.1 ～ 0.2 sec. / epoch）
Reduce overall training time by 45 seconds (120s → 75s)
21

Copy right 2019 FUJITSU LABORATORIES LIMITEDChanges when submitting MLPerf v0.6
 In MLPerf v0.6 rules, the validation accuracy was increased from 74.9% to 75.9%.
① Database for training
256 pixel
256 pixel
training image(before Data augmentation)

Clopped image for April 1 result

Original image for MLPerf v0.6

② Learning Rate scheduling
LearningRate

系列1

We tuned LR scheduling, and changed it tofollow MLPerf v0.6 rules

20151050

系列202250

Epoch

Copy right 2019 FUJITSU LABORATORIES LIMITEDEvaluation and results Hardware
 Software Results• Validation Accuracy in training• Accuracy improvement• Scalability
23

Copy right 2019 FUJITSU LABORATORIES LIMITEDEvaluation environment
 Hardware

ABCI compute node configuration

 compute nodes• 4-GPUs / node
• 2-HCA / node IB Interconnect• The intra-rack network has topology
of full bi-section fat-tree• The inter-rack network has topology
of fat-tree with 1/3 over subscription

24

Copy right 2019 FUJITSU LABORATORIES LIMITEDEvaluation environment
 Software
 MXNet / Horovod• Original source is NVIDIA tuned MXNet• We customized and tuned

 Other libralies• CUDA, cuDNN v7.5, NCCL v2.4
• OpenMPI• GCC 7.3
• Python 3.6

25

Copy right 2019 FUJITSU LABORATORIES LIMITEDResult; Validation Accuracy in training
100%

141060%

8640%

420%
20%

0020

406080

Learning Rate

Validation accuracy

1280%

We achieved 76%
accuracy with 84k
mini-batch using
2k GPUs

Epoch26

Copy right 2019 FUJITSU LABORATORIES LIMITEDResult; Accuracy improvement

Extended mini-batch size

ImageNet Validation accuracy

80%75%

70%

Improve accuracy
65%128k 256k

27

Copy right 2019 FUJITSU LABORATORIES LIMITEDResult; Scalability
 The number of computation images per GPU is the same (Weak scale)

Image / second

1.0E+07

Parallel efficiency is77% on 2k GPUs

1.0E+06

1.0E+05

系列1系列2

1.0E+04

1.0E+0312481632

64

128256512 1,024 2,048

Number of GPUs
28

Copy right 2019 FUJITSU LABORATORIES LIMITEDConclusion
 We achieved 70 sec. training time (world record †) and 84K mini-batch size
(world record††) of ResNet-50/ImageNet under MLPerf v0.6 rules††† Using ABCI 512 compute nodes (2,048 GPUs)
Mini-batchsize(max)Facebook

8,192

Google16,384Preferred Networks

32,768Tesla P100

Tencent

65,536

Google×76.3 %TensorFlow

30 min.

76.1 %

× 1,024

Chainer15 min.

74.9 %

Tesla P40

× 2,048

TensorFlow

6.6 min.

75.8 %

65,536

TPU v3

× 1,024

TensorFlow

1.8 min.

75.2 %

Sony55,296

Tesla V100

× 3,456

NNL2.0 min.

75.3 %

Fujitsu Labs.

86,016

Tesla V100

× 2,048

MXNet

1.17 min.

76.1 %?

Ascend A910

× 1,024

MindSpore

0.997 min.

>75.9%††256

Validation
accuracy

1 hour

Our investigation in March, 2019†††

Tesla P100

TimeCaffe2

Huawei†DL
SoftwareProcessor
full TPU Pod

Our investigation under the conditions of SGD and fixed mini-batch size in March, 2019

Used closed Division rules, except for tuning six hyper parameters
29

Copy right 2019 FUJITSU LABORATORIES LIMITEDCopy right 2019 FUJITSU LABORATORIES LIMITED