Extremely Large Minibatch SGD:
Training ResNet-50 on ImageNet in 15 Minutes

Takuya Akiba
Preferred Networks, Inc.
akiba@preferred.jp

Shuji Suzuki
Preferred Networks, Inc.
ssuzuki@preferred.jp

Keisuke Fukuda
Preferred Networks, Inc.
kfukuda@preferred.jp

Abstract
We demonstrate that training ResNet-50 on ImageNet for 90 epochs can be achievedin 15 minutes with 1024 Tesla P100 GPUs. This was made possible by using alarge minibatch size of 32k. To maintain accuracy with this large minibatch size,
we employed several techniques such as RMSprop warm-up, batch normalization
without moving averages, and a slow-start learning rate schedule. This paper also
describes the details of the hardware and software of the system used to achieve
the above performance.

1

Introduction

Training deep neural networks is computationally expensive. Acceleration by distributed computing isrequired for higher scalability (larger datasets and more complex models) and for higher productivity(shorter training time and quicker trial and error). This paper demonstrates that highly-parallel training
is possible with a large minibatch size without losing accuracy on carefully-designed software andhardware systems.
We used the 90-epoch, ResNet-50 [5] training on ImageNet as our benchmark. This task has been
extensively used in evaluating performance of distributed deep learning [3, 4, 10]. Table 1 shows thesummary of these previous attempts along with our new results. We achieved a total training time of
15 minutes while maintaining a comparable accuracy of 74.9%.
The technical challenge is two-fold; On the algorithm side, we have to design training methods thatcan prevent loss of accuracy with large minibatch sizes, while on the system side, we have to design
stable and practical combinations of available hardware and software components.
Table 1: 90-epoch training time and single-crop validation accuracy of ResNet-50 for ImageNet reported bydifferent teams.

2TeamHardware

Software

Minibatch size

TimeAccuracy

He et al. [5]Goyal et al. [4]Codreanu et al. [3]You et al. [10]Tesla P100 × 8
Tesla P100 × 256
KNL 7250 × 720
Xeon 8160 × 1600

Caffe
Caffe2
Intel CaffeIntel Caffe

256
8,192
11,52016,00029 hr
1 hr
62 min31 min75.3 %
76.3 %
75.0 %75.3 %This work

Tesla P100 × 1024

Chainer

32,76815 min74.9 %

Training Procedure for Large Minibatches

We build on the training procedure proposed by [4], and the same settings are used unless otherwise specified. We briefly highlight the differences in this section. For further details, please see
Appendix A.

RMSprop Warm-up. We found that the primary challenge is the optimization difficulty at the startof training. To address this issue, we start the training with RMSprop [7], then gradually transition toSGD.
Slow-Start Learning Rate Schedule. To further overcome the initial optimization difficulty, we use
a slightly modified learning rate schedule with a longer initial phase and lower initial learning rate.
Batch Normalization without Moving Averages. With the larger minibatch sizes, the batch normalization moving averages of the mean and variance became inaccurate estimates of the actual
mean and variance. To cope with this problem, we only considered the last minibatch, instead of themoving average, and used all-reduce communication on these statistics to obtain the average over all
workers before validation.

3

Software and Hardware Systems

Software. We used Chainer [8] and ChainerMN [1]. Chainer is an open-source deep learning
framework featuring the define-by-run approach. ChainerMN is an add-on package for Chainer
enabling multi-node distributed deep learning with synchronous data-parallelism. We used development branches based on versions 3.0.0rc1 and 1.0.0, respectively. As the underlying communication
libraries, we used NCCL version 2.0.5 and Open MPI version 1.10.2. While computation wasgenerally done in single precision, in order to reduce the communication overhead during all-reduce
operations, we used half-precision floats for communication. In our preliminary experiments, weobserved that the effect from using half-precision in communication on the final model accuracy wasrelatively small.
Hardware. We used MN-1, an in-house cluster owned by Preferred Networks, Inc. designed tofacilitate research and development of deep learning. It consists of 128 nodes, where each node has
two Intel Xeon E5-2667 processors (3.20 GHz, eight cores), 256 GB memory and eight NVIDIA
Tesla P100 GPUs. The nodes are interconnected by Mellanox Infiniband FDR.

Experimental Results

For running time and accuracy, the mean and standard deviation from five independent runs are reported. The per-worker
minibatch size was 32, and the total minibatch size was 32k
with 1024 workers.

0.30
0.25

Training iteration
Communication

0.20

Time [s]40.15

Training Time. Using 1024 GPUs, the training time was0.10
897.9 ± 3.3 seconds for 90 epochs, including validation after
0.05
each epoch. Figure 1 illustrates the average communication
0.00
time (i.e., all-reduce operations) and time to complete a whole
1 2 4 8 16 32 64 128 256 512 1024
Number of GPUsiteration (i.e., forward and backward computation, communication, and optimization) over 100 iterations. Our scaling Figure 1: Iteration and communication
efficiency when using 1024 GPUs is 70% and 80% in compar- time for different numbers of GPUs.
ison to single-GPU and single-node (i.e., 8 GPUs) baselines,respectively.
Accuracy. After training on 90 epochs using 1024 GPUs with the training procedure designed inSection 2, the top-1 single-crop accuracy on the validation images was 74.94% ± 0.09. As we
can observe from Table 1, this accuracy is comparable to that of previous results using ResNet50. Therefore, it shows that ResNet-50 can be trained on ImageNet with a minibatch size of 32k
without severely degrading the accuracy, which validates our claim that training of ResNet-50 can be
successfully completed in 15 minutes.

Acknowledgements
The authors thank Y. Doi, G. Watanabe, R. Okuta, T. Kikuchi, and M. Sakata for help on experiments, T. Miyato
and S. Tokui for fruitful discussions, and H. Maruyama, R. Calland, and C. Loomis for helping to improve themanuscript.

2References[1] T. Akiba, K. Fukuda, and S. Suzuki. ChainerMN: scalable distributed deep learning framework. CoRR,abs/1710.11351, 2017.
[2] D. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential
linear units (ELUs). CoRR, abs/1511.07289, 2015.
[3] V. Codreanu, D. Podareanu, and V. Saletore.
Achieving deep learning training in less than 40 minutes on imagenet-1k.
https://blog.surf.nl/en/
imagenet-1k-training-on-intel-xeon-phi-in-less-than-40-minutes/, 2017.
[4] P. Goyal, P. Dollár, R. B. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He.
Accurate, large minibatch SGD: training ImageNet in 1 hour. CoRR, abs/1706.02677, 2017.
[5] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages
770–778, 2016.
[6] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
[7] T. Tieleman and G. Hinton. Divide the gradient by a running average of its recent magnitude. COURSERA:
Neural Networks for Machine Learning, 2012.
[8] S. Tokui, K. Oono, S. Hido, and J. Clayton. Chainer: a next-generation open source framework for deeplearning. In LearningSys, 2015.
[9] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey,J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens,G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes,and J. Dean. Google’s neural machine translation system: Bridging the gap between human and machinetranslation. CoRR, abs/1609.08144, 2016.
[10] Y. You, Z. Zhang, C. Hsieh, J. Demmel, and K. Keutzer. ImageNet training in minutes. CoRR,abs/1709.05011, 2017.

ADetails of Training Procedure

A.1

RMSProp Warm-up

Our update rule is a simple combination of momentum SGD and RMSprop [7] (a variant withmomentum), defined as follows:
mt = µ2 mt−1 + (1 − µ2 )gt2 ,αRMSprop∆t = µ1 ∆t−1 − αSGD + √gt , andmt + εθt = θt−1 + η∆t .
Here, t denotes the current index of iteration. The weights, gradients, momentum, and moving
average of the second moment of the gradient at the i-th iteration are represented by θi , gi , ∆i ,and mi respectively. The inputs are gt , θt−1 , ∆t−1 , and mt−1 , and the outputs are θt , ∆t , and mt .
Hyperparameters are η, µ1 , µ2 , ε, αSGD and αRMSprop : η is the learning rate, µ1 determines the amountof momentum, µ2 is the coefficient for the moving average of the gradient second moment, and ε is asmall number added for numerical stability. We used µ1 = 0.9, µ2 = 0.99, and ε = 10−8 throughout
our experiments. Parameters αSGD and αRMSprop determine the balance between momentum SGD andRMSprop: when αRMSprop = 0, it corresponds to the standard momentum SGD, and when αSGD = 0,
it matches RMSprop.
We start with RMSprop (i.e., αSGD ≈ 0), and then smoothly switch to SGD (i.e., αSGD = 1). For thetransition schedule, we use a function that is similar to the exponential linear unit (ELU) activation
function [2] defined as follows:1 2 exp(2(epoch − βcenter )/βperiod ) (epoch < βcenter )αSGD = 21 + 2(epoch − βcenter )/βperiod(epoch < βcenter + 12 βperiod ) .
1(otherwise)3

Here, βcenter and βperiod are hyperparameters. First, αSGD increases exponentially. At the βcenter -thepoch, αSGD reaches 12 . After that, it increases linearly until the βcenter + 21 βperiod -th epoch. At theβcenter + 12 βperiod -th epoch, αSGD becomes 1, and we set αSGD = 1 for the remainder of the training.
We set βcenter = 10 and βperiod = 5 throughout our experiments.
We used ηRMSprop = 0.0003 for the learning rate of RMSprop. Let ηSGD be the learning rate of SGD,
which will be discussed in the next subsection. To incorporate different learning rates of SGD andRMSprop, we set η = ηSGD and αRMSprop = (1 − αSGD )ηRMSprop /ηSGD . One might think that the rulewould be simpler if we multiply ηSGD to αSGD beforehand, but we should make ∆t independent fromvarying learning rates for momentum correction proposed by Goyal et al. [4].
A method similar to our RMSprop warm-up is used by Wu et al. [9] for a machine translation
task. They use the Adam [6] optimizer at the beginning, then switch to SGD. In our preliminary
experiments, we found that RMSprop performs better for our task. In addition, Wu et al. suddenly
switches from Adam to SGD. However, we found that sudden transition severely impacts training andhas a negative effect on the final results. Therefore, we designed a smooth transition from RMSprop
to SGD. We examined a few transition functions including linear and sigmoid functions. Linear
functions have a similar problem at the beginning of the transition. ELU and sigmoid performed
similarly, but ELU performs slightly better, so we opted for ELU.
A.2

Slow-Start Learning Rate ScheduleLet ηbase be the initial learning rate under the linear rule by Goyal et al. [4]. Specifically, ηbase =total
local
0.1 · b256
= 0.1 · nb256 , where n is the number of workers, blocal is the local batch size for each worker,
and btotal is the total batch size among all workers (i.e., btotal = nblocal ). In our experiments, n = 1024
and blocal = 32, and thus ηbase = 12.8. Goyal et al.’s learning rate schedule is as follows: ηbasefor first 30 epochs, 0.1 · ηbase for the next 30 epochs, 0.01 · ηbase for the following 20 epochs, and0.001 · ηbase for the last 10 epochs.
To overcome the initial optimization difficulty, we used a slow-start schedule; our learning rate for
SGD was 0.5 · ηbase for the first 40 epochs, 0.075 · ηbase for the next 30 epochs, 0.01 · ηbase for thefollowing 15 epochs, and 0.001 · ηbase for the last 5 epochs.

4