Performance Improvement of Deep Learning Training onLarge-scale Manycore Cluster
Toshihiro Hanawa
Kohei Tamura∗hanawa@cc.u-tokyo.ac.jp
The University of Tokyo
Tokyo, JapanABSTRACT

3.5
3KEYWORDS
Deep Learning, Intel Xeon Phi, Knights Landing, MPI, OmniPath,
Oakforest-PACS
ACM Reference Format:
Toshihiro Hanawa and Kohei Tamura. 2019. Performance Improvementof Deep Learning Training on Large-scale Manycore Cluster. In ICPP ’19:
International Conference on Parallel Processing, August 05–08, 2019, Kyoto,Japan. ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/1122445.
11224561INTRODUCTION

Today, Deep Learning (hereafter, DL) has been utilized in many
fields. DL requires more complex and deeper network model and alarge amount of dataset for supervised training to achieve high accuracy. GPU is widely used for such trainings, however the memory
capacity is limited up to 16 GB or 32 GB at most, and the communication among GPUs has larger latency due to the bottleneck by
PCIe I/F.
Manycore CPU can address these problems thanks to the self
bootable feature and larger memory capacity. We are developing∗ Currently he is in NTT DATA Corporation.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
ICPP ’19, August 05–08, 2019, Kyoto, Japan© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-9999-9/18/06. . . $15.00
https://doi.org/10.1145/1122445.1122456[sec/iter]

To shorten a large-scale training for deep learning, the distributed
deep learning are widely applied to the massive clusters using accelerators such as GPUs. In contrast, manycore processor such asIntel Xeon Phi is also suitable for computing deep learning operation and it is easy to expand to large-scale cluster. In this study,to utilize deep learning training on large-scale many core cluster,
we conduct performance evaluation of large-scale deep learning
framework ChainerMN on Oakforest-PACS system operated byJCAHPC, and optimize the Allreduce communication latency. As aresult, the improved communication of ChainerMN is 2.1x fasterthan the original one on Oakforest-PACS system.

2.5
21.5
10.5
0163264

128256512

Number of Nodes

Forward+Backward+Update

All-Reduce

Figure 1: Breakdown of Execution Time per Iteration
large-scale deep learning environment using Oakforest-PACS supercomputer system (OFP) with 8,208 nodes of Intel Xeon Phi processors (KNL)[4]. The performance of KNL processer is similar to
NVIDIA P100 in terms of the peak FLOPS and memory bandwidthof MCDRAM, in addition, KNL has DDR4 memory as well [5]. OFP
also employs high performance interconnect and storage system,
and as the result, these features of OFP are helpful for large-scale
DL training. In this study, we target ChainerMN, which is the multinode version of Chainer [1, 2].

2

PERFORMANCE ANALYSISFirst, we investigate the performance on OFP using ImageNet. We
used ResNet-50 as the neural network model, and iDeep developed
by Intel is adopted as the backend of the chainer for KNL’s AVX-512
instructions. The parallelization can be specified as the combination
between multithreading by OpenMP and multiprocessing by MPI.
Fig. 1 shows the breakdown of the execution time per iteration
with various number of nodes. The learning process is performed using the fixed number of mini-batch size, therefore, it is weak-scaling
problem. As the result, the elapsed time of Allreduce procedure
exceeds half of the entire execution time, and Allreduce becomesbottleneck as the number of nodes increases.

3OPTIMIZATIONTo reduce the communication time of Allreduce part, we propose thenew communication algorithm instead of Allreduce. This method
is based on the 2-D Torus allreduce proposed by Mikami for theGPU cluster[6]. As shown in Fig. 2, the number of processes isdecomposed by M × N . In this algorithm, first, Reduce-scatter isperformed in column-direction, then, Allreduce is done in rowdirection, and finally, Allgather collects from and shares with M
members in column-direction.

ICPP ’19, August 05–08, 2019, Kyoto, JapanToshihiro Hanawa and Kohei TamuraCPU
Freq., # of cores, Peak Perf.
MemoryMem. BW
Interconnect
Parallel File System
Capacity
Fast File Cache
Capacity
Data Transfer Rate

Oakforest-PACS (OFP)
Xeon Phi 7250
1.4 GHz, 68c, 3.04 TF
16 GB (MCDRAM)+ 96 GB (DDR4)
490.0 GB/s (effective), 115.2 GB/s
OmniPath 100 Gbps
Lustre File System
26 PB
DDN Infinite Memory Engine (IME)
940 TB
1.56 TB/s

indicates the example of the communication using Put as one-sided
communication instead of Allgather. In this case, Put is naturally
suitable for the data placement rather than Allgather operation.
Fig. 4 shows the elapsed time of each Allreduce process. As theresult, in the case of 256 processes, we achieved over 2.1 times
speedup by the communication algorithm using Reduce-scatter,
Allreduce, and Put.
3

Table 2: Software Environment
Software
PythonIntel MPI
MPI4pyChaineriDeep4py

2.91

2.5

[sec/iter]

Table 1: Specification of Oakforest-PACS

2

1.38

1.5

2.04

1.89

1.70

1.58

1.44

10.5

version
3.6.3
2018.1.163
3.0.0
5.0.0
2.0.00RSAA
All-Reduce
OnlyIn this approach, the advantage is the average calculation ofeach element can be performed in parallel manner with M after theAllreduce on the second step.

RSAP2, 128

RSAARSAP

16, 16

RSAARSAP

64, 4

Figure 4: Comparison of Elapsed Time of Allreduce Processwith 256 processes (Allreduce only: original MPI Allreduce,
RSAA: Reduce-scatter, Allreduce, Allgather, RSAP: Reducescatter, Allreduce, Put)

REFERENCES
𝑀
𝑁Place as 𝑀x𝑁# of process 𝑃
Reduce-Scatter
All-Reduce

All-Gather

Figure 2: Communication Algorithm of 2-D Torus Allreduce

1.Reduce-Scatter in the horizontal direction
Rank:1

Rank:00123Add

4Rank:2

8

9 10 11Add

5624 28 32 36

Rank:224 28 32 36

Rank:04

Rank:3

Rank:2

6Rank:12345 8

10Rank:3

20 22 10 11

12 13 24 26

3.All-Gather in the horizontal direction

4.Completed
Rank:02. All-Reduce in the vertical direction

712 13 14 15[1] S. Tokui, K. Oono, S. Hido, and J. Clayton: “Chainer: a next-generation open source
framework for deep learning”, NIPS, 2015[2] Akiba, T., Fukuda, K. and Suzuki, S., “ChainerMN: Scalable Distributed Deep
Learning Framework,” Proc. of Workshop on ML Systems in The Thirty-first
Annual Conference on Neural Information Processing Systems (NIPS), 2017[3] T. Kurth, J. Zhang, N. Satish, et al.: “Supervised and Semi-Supervised Classificationfor Scientific Data”, Supercomputing, 2017.
[4] Joint Center for Advanced HPC: “Oakforest-PACS,” http://jcahpc.jp/eng/ofp_intro.
html .
[5] A. Sodani: “Knights Landing (KNL): 2nd Generation Intel® Xeon PhiTM Processor,”IEEE Hot Chips 27 Symposium, 2015.
[6] H. Mikami, et al., “ImageNet/ResNet-50 Training in 224 Seconds,” arXiv preprintarXiv:1811.05233, 2018.

Rank:124 28 32 36

Rank:324 28 32 36

Rank:024 28 2

Rank:1

3

Rank:224 28 10 11

Copy

4

5 32 36

Rank:3

Copy 12 13 32 36

Replace “All-gather”
communication to “Put”
operation to reduce thecost by synchronizationand memory copy

Figure 3: Proposed Communication Algorithm Based on 2-DTorus Allreduce
Moreover, we propose the new method using one-sided communication mechanism instead of collective communications. Fig. 3