Performance Improvement of Deep Learning Trainingon Large-scale Manycore Cluster
1,22*Toshihiro Hanawa , Kohei Tamura

1 Information Technology Center, 2 Department of Electrical Engineering and Information Systems
The University of Tokyo (* Now in NTT DATA Corp.)

Overview
To shorten a large-scale training for deep learning, the distributed deep learning are widely applied to the massive clusters using accelerators such as GPUs. In contrast, manycore
processor such as Intel Xeon Phi is also suitable for computing deep learning operation and it is easy to expand to large-scale cluster. In this study, to utilize deep learning training onlarge-scale many core cluster, we conduct performance evaluation of large-scale deep learning framework ChainerMN on Oakforest-PACS system operated by JCAHPC, and optimize theAllreduce communication latency. As a result, the improved communication of ChainerMN is 2.1x faster than the original one on Oakforest-PACS system.

ChainerMNResults• About• Loader Process and Core Affinity

ChainerMN is a scalable distributed deep learning framework developed by PreferredNetworks [1]. It is an add-on package to Chainer and written in Python. Recently,ChainerMN have been merged into Chainer v5.
ü Scalable
It makes full use of the latest technologies such as cuPy for GPU, MKL-DNN for CPU, andmpi4py for multi-node execution.
ü Flexibility
Even dynamic neural networks can be trained in parallel.
ü EasyMinimal changes to existing code by Chainer are required.

ü Data parallelØ Divide minibatchesØ Copy a modelØ Average gradients

CPU+GPUGPUCPU

Minibatch#2

Forward

Minibatch#3

Backward

ForwardBackward

Save DataCopy

Create Model

2

ProcessingProcessing

Return01HT: Loader process on the same core
2HT: Loader process on the same physical core, butdifferent logical core (HyperThreading)

3.5
364 thread / process,excluding the busy core for OS services

2.5
2
1.5
10.5

Figure 4: Break down of Execution Time per Iteration••

2HT

Figure 3: Affinity of Loader Process• Performance Analysis ofImageNet Training on OFP

Optimize

1Load Data

CPU+GPU: GPU computes asynchronously and CPU is dedicatedfor data load.
CPU-only: CPU have to manage both of load and compute.
=> Separate loader process from compute process

Optimize

1.5

0.5

Figure 2: Execution Model in CPU-only andCPU+GPU

Optimize

AllReduce

Save Model

20% better

1HTü Model parallelØ Divide a modelØ Use a portion of the modelØ Calculate for one minibatch

Backward

Load DataCopy

Minibatch#1Forward

Create Model[sec/iter]CPU2.5[sec/iter]• Data Parallel and Model Parallel

CPU-only0

Weak scaling problem, well scaling for the computation.
The communication time by All-reduce is dominant in eachiteration.
=> Improvement for All-reduce is required.

163264128

256

Number of Nodes

Forward+Backward+UpdateAll-Reduce

Figure 1: Process of Synchronous Data-parallel Deep Learning
The four steps of synchronous data-parallel deep learning, which is the standard method ofparallelism, is illustrated. In All-Reduce step, workers communicate with each other to find theaverage of gradients. Each worker optimizes the model by the average of gradients.

Oakforest-PACS

1.Reduce-Scatter in the horizontal direction

0Theoretical Computation Performance
Number of Core
Memory Capacity
MemoryBandwidth3046.4 GFLOPS

Physical68

Logical272

MCDRAM

16 GB

DDR496 GB

MCDRAM

490 GB/s

DDR484.5 GB/s

23Add45867

ReduceScatter

Rank:39 10 11AddRank:0

46Rank:1

2

345 8

10Rank:3

20 22 10 11

12 13 24 26

Table 2 : Specification of Oakforest-PACS

Parallel
FileSystem
HighSpeed
FileCache

All-Gather

2. All-Reduce in the vertical direction

Rank:212 13 14 15

3.All-Gather in the horizontal direction

4.Completed

Spec

Rank:1

Rank:0

Number of Node8208

Interconnect

Intel Omnipath Architecture(100Gbps)
Full-bisection BW Fat-tree

Type

Lustre File System

Storage Capacity

26 PB

Data Transfer Rate

500 GB/sec

Type

DDN Infinite Memory Engine (IME)

Capacity

940 TB

Data Transfer Rate1,560 GB/sec

24 28 32 36

24 28 32 36

Rank:224 28 32 36

Rank:024 28 2

Rank:13Copy

45 32 36

Rank:2

Rank:324 28 10 11

24 28 32 36

3Copy 12 13 32 36

All-reduce

Replace “All-gather”
communication to “Put”
operation to reduce thecost by synchronizationand memory copy

2.91

2.5

2.1x faster thanoriginal !!

Experiment

AllGather

Rank:3[sec/iter]

1.40 GHz

1Rank:2

Table 1: Specification of KNL
Operation Frequency

Rank:1

Rank:0ItemSpec

Place as 𝑀x𝑁# of process 𝑃
Reduce-Scatter
All-Reduce

Introduction of 2D-Torus All-Reduce [4]Oakforest-PACS is a supercomputer
which is made up 8,208 nodes using
Intel® Xeon Phi ™ 7250 processors(Codename: Knights Landing=KNL) [2, 3].

Item𝑁

ü Huge Overheadü Synchronization
ü Memory copy
ü Average calculation after reduction• Overview

68 cores/node, 3 TFLOPS x 8,208= 25 PF𝑀• Optimization of Communication

TOP 500 #6, HPCG #3,Green 500 #6 @Nov. 2016
IO 500 #1 @Jun. 2018

512

2

1.70
1.38

1.5

2.04

1.89
1.58

1.44

10.5
0RSAA

• Dataset

All-ReduceOnlyImagenet is a large visual database which has over 1,400,000 pictures. Each picture is handannotated to indicate what objects are pictured.

RSAP2, 128

RSAARSAP

16, 16

RSAARSAP

64, 4

Figure 6: Elapsed Time of “All-Reduce” Process (Fig. 4) per Iteration• All-Reduce Only：Original pure-allreduce• RSAA：Reduce-Scatter, All-Reduce，All-Gather (original 2D Torus-AllReduce)

DatasetModel

Framework

SoftwareVersionImageNet

Resnet-50

ChainerMNIntel Python

3.6.3

Intel MPI

2018.1.163

MPI4py3.0.0

Chainer

5.0.0

iDeep4py

2.0.0

http://www.image-net.org• Implementation

https://arxiv.org/abs/1512.03385

https://chainer.org

From Chainer v4, iDeep(Intel Deep Learning Package) is added as a backend. iDeep enables us
to parallelize threads with OpenMP and generates AVX-512 instructions for KNL automaticallyby JIT compiler technology.

• RSAP：Reduce-Scatter, All-Reduce，Put (Proposed)

References[1] T. Akiba, K. Fukuda, and S. Suzuki, “ChainerMN: Scalable Distributed Deep Learning
Framework,” Proceedings of Workshop on ML Systems in The Thirty-first Annual
Conference on Neural Information Processing Systems (NIPS), 2017.
[2] Joint Center for Advanced HPC, “Oakforest-PACS”, http://jcahpc.jp/eng/ofp_intro.html .
[3] A. Sodani, “Knights Landing (KNL): 2nd Generation Intel® Xeon PhiTM Processor,” IEEE Hot
Chips 27 Symposium, 2015.
[4] H. Mikami, et al. “ImageNet/ResNet-50 Training in 224 Seconds,” arXiv preprint
arXiv:1811.05233, 2018.