Training Deep Networks with Stochastic Gradient Normalized by Layerwise Adaptive Second Moments
Authors: Boris Ginsburg
Patrice Castonguay
Oleksii Hrinchuk
Oleksii Kuchaiev
Vitaly Lavrukhin
Ryan Leary
Jason Li
Huyen Nguyen
Yang Zhang
Jonathan M. Cohen
Venue:  (2020)
Keywords: deep learning
optimization
SGD
Adam
NovoGrad
large batch training

Abstract: We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with momentum and Adam/AdamW. 
Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has two times smaller memory footprint than Adam.

Under review as a conference paper at ICLR 2020
TRAINING DEEP NETWORKS WITH STOCHASTIC GRA- DIENT NORMALIZED BY LAYERWISE ADAPTIVE SEC- OND MOMENTS
Anonymous authors Paper under double-bling blind review
ABSTRACT
We propose NovoGrad, an adaptive stochastic gradient descent method with layer- wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs par or better than well-tuned SGD with mo- mentum, Adam, and AdamW. Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in large batch setting, and (3) has half the memory footprint of Adam.
INTRODUCTION
The most popular algorithms for training Neural Networks (NNs) are Stochastic Gradient Descent (SGD) with momentum (Polyak] 1964; Sutskever et al. 2013) and Adam (Kingma & Ba 2015). SGD with momentum is the preferred algorithm for computer vision, while Adam is the most com- monly used for natural language processing (NLP) and speech problems. Compared to SGD, Adam perceived as safer and more robust to weight initialization and learning rate. However, Adam has certain drawbacks. First, as noted in the original paper (Kingma & Ba 2015), the second moment can vanish explode, especially during the initial phase of training. To alleviate this problem, learning rate (LR) warmup (Goyal et al. 2017) is typically used. Adam often leads to solutions that generalize worse than SGD (Wilson al. 2017). , and to improve Adam regularization, Loshchilov & Hutter 2019) proposed Adam W with decoupled weight decay.
Our motivation for this work was to find an algorithm which: (I) performs equally well for im- age classification, speech recognition, machine translation, and language modeling, (2) is robust to learning rate and weight initialization, (3) has strong regularization properties. We start with Adam, and then (1) replace the element-wii wise second moment with the layer-wise moment, (2) compute the first moment using gradients normalized by layer-wise second moment, (3) and decouple weight de- cay (similar to AdamW) W) from normalized gradients. The resulting algorithm, NovoGrad, combines SGD's and Adam's 's strengths. We applied NovoGrad to a variety of large scale problems - image classification, neural machine translation, language modeling, and speech recognition and found that in all cases, it performs as well or better than Adam/ Adam/ and SGD with momentum.
2 RELATED WORK
NovoGrad belongs to the family of Stochastic Normalized Gradient Descent (SNGD) optimiz- ers (Hazan et al. 2013; Nesterov] 1984). SNGD uses only the direction of the stochastic gradient update the weights, and the step size does depend on the magnitude of that gradient. Hazan jet al. (2015) proved that the direction of the gradient was sufficient for convergence. Ignoring the gradient magnitude makes SNGD robust to vanishing and exploding gradients.
SNGD with layer-wise gradient normalization was introduced by Singh et al. (2015). The method scales up small gradients, while keeping large gradients unchanged:
g. gf. (1 log(1
|lg/

---

Under review as a conference paper at ICLR 2020
where gi is the gradient for the layer step t. A similar approach was proposed by Yu et al. (018)), who used layer-wise gradient normalization for SGD and Adam to alleviate vanishing and exploding gradients. They divide the stochastic gradient g by its norm ||g||:
g
g
19
NovoGrad is also closely related to the Normalized Prrection-preservvi Adam (ND-Adam), proposed by Zhang et al. (2017). For each layer, ND-Adam first removes the projection of gradient g on the layer's weights wt: g. - rو (g': w/) wi
Then, of is used to compute Adam 1s and 2nd (scalar) moments:
m; = B1 m (1 31) gg.
of = B2 2/-1 + (1 - (2) 119.11?
Similar to Adam, the weights are updated with the 1st moment re-scaled by the 2nd moment:
m
w/ 41 w d Vol +e
Adaptive methods like Adam generalize worse than SGD with momentum Wilson et al.] (2017). For example, Keskar Socher (2017) proposed to use Adam during the initial stage only and then switch to SGD. Luo al. (2019) suggested improve Adam regularization by limiting the factor 7 to a certain range: limiting from above helps to decrease the training
shown in
loss while limiting from below helps to generalize better. Loshchilov & Hutter (2019) showed that Adam's weak regularization is due to the fact that the 2nd moment normalization effectively turns off L2 12-reuularization. They proposed Adam W, which decouples the weight decay d w, from the gradient and uses directly in the weight update:
me:
d w.)
0/
+
Adam needs to store the 2nd moment, and this doubles the optimizer memory compared to SGD with momentum. This affects large models like GPT-2 (Radford et al. 2019) with 1.5 billion parameters. Shazeer & Stern (2018) proposed the AdaFactor algorithm, which replaced the full 2nd moment with moving averages of the row and column sums the squared gradients. For a layer defined by an matrix, this would reduce memory from O(n x m) to Onn + m.) NovoGrad consumes the same amount memory as SGD with momentum.
ALGORITHM
NovoGrad is based on ideas: (1) layer-wise 2nd moments instead of 2nd moment per each parame- ter, (2) gradients normalization with layer-wise 2nd moments, (3) decoupled weight decay.
Let of be the stochastic gradient for layer step NovoGrad first computes the layer- wise 2nd
moment of using the norm ||g!
B2 =
1/11 + - (1- |!
where 0 S B2 < 1. We use much smaller B2 than in Adam, usually in the range [0.2, 0.5
The moment of is used to normalize the gradient g before calculating the first moment m/ Sim- ilarly to AdamW, we decouple weight decay d w, from stochastic gradient, but we add it to normalized gradient before computing moment m
g Vot+t +E
m = 31 mm +
d w.)
We use La-norm || to compute v/. would be interesting to see how Li or L.s norms perform. FI 32 0, then of || and NovoGrad becomes layer-wise NGD with decoupled weight decay.
2

---

Under review as a conference paper at ICLR 2020
Algorithm 1 NovoGrad
Parameters:
Initial learning rate λo moments B1, $2, weight decay d, number of steps T Weight initialization: = O, Initialize wo-
Moment initialization: t = 1, for each layer set of = [|g]|l?: = gi Vol
+ d w/-
while T do
X Learning Raterninggateuudduueceud pdote(do Au,t, t) (compute the global learning rate)
for each layer I do
g (.L(w))
U B2 vi-1 + (1 11-33 32) -19//12
m + B1 m + (- Vutte
+ d. w!)
w/+1 t wi - d mi
end for end while
where 0 < 31 < 1 is the momentum, typically in the same range as in SGD or Adam [0.9 0.95j. The first moment can be also computed via an exponential moving average in Adam-like style:
( gi + d w!)
m = 31 m + -1)
+
Finally, weights are updated the same way as SGD with momentum.
Similar to Adam, one construct counter-example for NovoGrad in the stochastic convex opti- mization settings (Wilson et al. 2017). However, the "AMS-Graa fix (Reddi et al.) 2018) for Adam can also be applied in this case to guarantee NovoGrad convergence:
0 B2 o 1 + (1 B2) 1!
6. max 1/
m = 31 m + (- g w00)
+
EXPERIMENTS WITH DEEP LINEAR NETWORKS
Following (Andrew M. Saxe & Ganguli 2013 lan J. Goodfellow & Saxe 2013) will use Novo- Grad to train linear model composed of two linear layers W1, without any non-linearity. The model y = (w) - -22) x should output when
1. This model linear with respect to the inputs, but it is non-linear with respect to the weights, since they are factorized into the product of layers' weights. Training the model is equivalent to the minimization of the loss L(w1, w2) = (wi W2 1)2 (Ian J. Goodfellow & Saxe 2015). The loss is not convex, and its minima are located on the hy- perbola W/ = (see Figure 1). Minima close to the points (-1, -1) and (1, 1) are good "flat" minima which generalize well. Minima close to the axes are "sharp" minima (Keskar et al. 2016).
We trained the model with SGD with momentum, Adam, AdamW, and NovoGrad, using the same fixed learning rate r weight decay, and weights initialization. The model was trained for 500 steps. Figure 2 shows the training trajectory and the zoomed-out area near the final point, All algorithms behave in similar way: first the trajectory goes the curve wy 1/w1, and then follows the hyperbola towards 11, or (-1, 11) During the first phase, training loss decreases, and during the second phase, generalization improves. SGD converges nicely toward (1, 1) but its trajectory is still slightly off of the optimal solution. Adam oscillates wildly around hyperbola w2 = 1/w1, while AdamW behaves much better since weight decay decoupling significantly reduces oscillations.
NovoGrad is the most stable out of four algorithms. exhibits better generalization and closely follows the minima curve because normalized gradients prevent trajectory from going far from it. We also found that NovoGrad is more robust than other algorithms to the choice of learning rate, weight decay, and weight initialization (see for details Appendix Ap.
We will use the gradient averaging for SGD first moment in Adam: B mc-1 + (1 -9) to the same LR for all optimizers.
3

---

Under review as a conference paper at ICLR 2020
y = W W2 - 1)2
4
global solutions
3
flat minima
sharp minima
2
1
W2
-1
-2
-3
-4
4
-3
-2
4
W
Figure 1: 2D-contour plot of surface y (w) W2 - 1)2 of linear model with layers. The loss functions has many global minima are located on hyperbola we 1/w1. Solutions near (-1, 1) and (1, 1) are good "flat" minima, and solutions near axes are "sharp" minima.
SGD
Adam
AdamW
NovoGrad
-2
3
33
-2
12
1,1
1.1
1.1
1.3
1.0
1.0
1.0
1.0
6.9
0.9
0.9
0.9
0.9
0.6
1.1
1.2
0.9
0.3
0.9
1.2
1.0
1.0
1.0
0.8
0.5
0.5
0.1
0.1
0.1
0.1
MARAAAAAAAA
0.0
#steps
Hsteps
/steps
#stops
Figure 2: Deep Linear Network with two layers: training with SGD, Adam, and NovoGrad
4

---

Under review as a conference paper at ICLR 2020
5 EXPERIMENTS WITH LARGE DNNS
We train four deep models: ResNet-50 He ct al. 2016 for ImageNet classification, Transformer- big (Vaswani et al. 2017) for WMT 20 14 translation. Jasper (Li et al. 2019 - for LibriSpeech speech recognition, and Transformer-X Daa et al. 2019) for Wiki Text- 103 word-level language modeling, with NovoGrad, SGD with momentum, and Adam/Adam Each a single DGX-1 machine with 8 NVIDIA V100 GPUs with gradient accumulation used for large batch training. In all the experiments, NovoGrad performed par better than other algorithms.
Each model was trained on
5.1 IMAGE CLASSIFICATION
We used ResNet-50 v2 (He et al., 2016) for ImageNet classification task (Russakovsky et al. 2015 We trained this model with three optimizers: SGD with momentum (SGD), AdamW, and NovoGrad. All models have been trained with the batch size of 1024 for 100 epochs. We used quadratic LR decay for SGD with momentum and cosine decay (Loshchilov & Hutter 2016) for AdamW and NovoGrad. We could not find any training recipe for ResNet-50 with Adam W. so we report the best accuracy we achieved after extensive syper-paramete search. We used only standard data augmen- tation methods: re-size, flip, random crop, and did not employ any additional training tricks (He ct al. 2018). The single-crop validation accuracy for each algorithm is reported in Table]
Table 1: ImageNet classification
ResNet-50 v2, batch 1024, top-1 and top-5 accuracy
Optimizer
LR policy
Init LR
Weight decay
Epochs
top-1,% 1,
top-5, %
100
76.38
93.08
SGD
poly (2)
0.400
0.0001
200
76.33
92.96
100
76.36
93.01
AdamW
cosine
0.002
0.120
200
76.48
92.94
100
76.94
93.41
NovoGrad
cosine
0.007
0.002
200
77.74
93.70
300
77.65
93.62
NovoGrad outperformed both AdamW and SGD with the top-1 accuracy of 76.94% after 100 epochs. SGD and Adam accuracy remained under 76.5% if we trained for 200 epochs instead, while NovoGrad accuracy improved to 77.74%. NovoGrad demonstrated powerful regularization capabilities: training for 100 additional epochs kept 1=776_77.65% without overfitting. Note that this is "vanilla" ResNet-50, without sophisticated data augmentation or model tweaking.
5.1.1 LARGE BATCH TRAINING FOR IMAGE CLASSIFICATION
Hazan et al. 2015) showed that large batch size is beneficial for SNGD convergence, which moti- vated explore NovoGrad for large batch training. We trained ResNet-50 v2 with batch sizes of 8K and 32K. To compare with the previous methods, we train the model for 90 epochs using cosine LR decay. To emulate large batch, we used a mini-batch of 128 per GPU and accumulated gradients from several mini-batches before each weight update.
Table Large batch training with NovoGrad ImageNet, ResNet-50 v2, epochs, accuracy(%).
Batch
Init LR
Weight decay
Top-1, %
Top-5,%
0.070
0.002
76.86
93.31
8K
0.016
0.006
76.64
93.14
32K
0.026
0.010
75.78
92.54
To establish the baseline for NovoGrad training with batch 32K we first used the method similar proposed in Goyal et al. (001)) scaling the learning rate linearly with the batch size and using
'Training done in OpenSeq2Seq Kuchaiev et al. 2018 https: com I
5

---

Under review as a conference paper at ICLR 2020
LR warmup. This method gives 10p-1%75.09% and top-5=92.27%% We found that get much better results when we increase both the learning rate À and the weight decay d to improve the regularization (see 2).
For comparison, we took 3 methods, which (1) use fixed batch size during training and (2) do not modify the original model. All 3 methods employ SGD with momentum. The first method (Goyyl et al. (017)) scales LR linearly with batch size and uses the LR warmup to stabilize the initial training phase. The second method (You al. (2018)) combines warmup with Layer-wise Adaptive Rate Scaling (LARS) (You et al., 2017). The last method (Codreanu et al. (2017)) uses warmup and dynamic weight decay (WD). NovoGrad outperformed all other methods without using any additional techniques like LR warmup (Goyal et al., (017). dynamic weight decay, special batch normalization initialization, etc. Using warm-up (500 steps) we slightly improved top1 accuracy to 75.99% and top5 to 92.72%.
Table 3: Large batch training comparison ImageNet, ResNet-50v top-1 accuracy(%))
Optimizer
Reference
Bag of Tricks
Epochs
B=8K
B=32K
SGD
Goyal al. 2017
warmup
90
76.26
72.45
SGD
You et al. 2018)
warmup, LARS
90
75.30
75.40
SGD
Codreanu et
2017
warmup, multi-step WD
100
76.60
75.31
90
76.64
75.78
NovoGrad
warmup
90
75.99
5.2 SPEECH RECOGNITION
We conducted experiments with Jasper-10x: 10x5 (Li et al. (2019)), a very deep convolutional neural acoustic model, on the LibriSpeech speech recognition task (Panayotov et al.) (0015). Jasper was trained with SGD with momentum (SGD), Adam and NovoGrad for 400 epochs with a batch of 256, polynomial LR decay, and Layerwise Adaptive Rate Clipping (LARC) [ We found that NovoGrad
yields lower Word Error Rates (WER) comparing to SGD, especially for the long runs. The model and training parameters described in Li al. (2019).
Table 4: Speech recognition Jasper- LibriSpeech, 400 epochs, WER (%).
Dev
Test other
Optimizer clean
other
clean
Adam
13.20
31.71
13.36
32.71
SGD
3.91
12.77
3.98
12.79
NovoGrad
3.64
11.89
3.86
11.95
5.2.1 LARGE BATCH TRAINING FOR ASR
We trained Jasperl0x: 0x5 with batch sizes 512, 4K, 8K, 16K and 32K on LibriSpeech. all cases, we trained the model for 400 epochs. For batch size up to 8K we scaled LR linearly with the batch size and used LR warmup. To scale batch to 16K and 32K we also increased weight decay (see Table 5). The batch 16K leads to WER comparable to the baseline. Batch 32K has higher WER due to the smaller number of training steps (9 weights updates per epoch). Figure shows WER on dev-clean during training for different batch sizes.
5.3 LANGUAGE MODELING
We trained Transformer-X XL (Dai al. 2019), the state-of-the-an LM architecture the word- level Wik (Merity et al. 2016) benchmark. For all the experiments we used a 16-layer

---

Under review as a conference paper at ICLR 2020
Table 5: Large batch training with NovoGrad hasper-10x5, LibriSpeech, 400 epochs, WER (%).
Dev
Test
Batch
Init LR
Warmup
Weight decay
clean
other
clean
other
512
0.015
0.001
3.58
11.30
3.85
11.29
4K
0.03
0.05
0.001
3.66
11.67
3.92
11.68
8K
0.06
0.05
0.001
3.69
11.76
3.96
11.75
16K
0.06
0.05
0.003
3.67
11.03
3.94
11.19
32K
0.06
0.08
0.004
4.01
11.73
4.14
11.89
100
batch-512-basellin
batch-4K
batch-8K
dev-clean WER
batch-16K
batch-32K
10-1
0
50
100
150
200
250
300
350
400
# of epochs
Figure 3: ASR, large batch training. Jasper- 10x5 trained with NovoGrad on LibriSpeech.
model with 191M parameters (dmodel 512, dr = 2048, = Parp 0.15). other hyper- parameters were taken from the original Transformer-XI. paper, the source code was based on publicly available implementation 6 Each configuration was trained for 12 billion tokens which is approximately 117 epochs and 366K training iterations. Figure 4 shows that NovoGrad exhibits much smaller gap between training and validation perplexity compared to Adam, which results in better performance the test set. Longer training for 20B tokens does not lead to overfitting the resulting validation and test perplexities improve even further.
Table LM. Transformer-Xi trained on WikiText-103 with batch size 256, sequence length 512.
Optimizer
Tokens
Init LR
Weight decay
Val PPL
Test PPI.
Adam
12B
0.00025
23.84
25.40
AdamW
12B
0.00025
0.001
23.64
25.06
12B
0.01
20.53
21.26
NovoGrad
20B
0.01
19.89
20.65

---

Under review as a conference paper at ICLR 2020
50
adam train
45
adam valid
novograd train
40
novograd valid
perples eerllexity
35
30
25
20
50000
150000
250000
350000
# of gradient updates
Figure 4: LM. Transformer-Xl. trained with Adam and NovoGrad WikiText-103.
5.4 NEURAL MACHINE TRANSLATION
We trained Transformer Vaswani et al. 2017) on WMT 2014 English-to-German benchmark. For all the experiments, we used a 12-layer Transformer-big model with 185M parameters (dmodet 1024, dr 4096, h 16) with the vocabulary of 8192 tokens based joint source-target byte- pair-encodings (Sennrich et al. 2015). For Adam and AdamW we used dropout of Parop 0.3 and for NovoGrad used Parop 0.2. We trained all algorithms with mixed precision (Micikevicius et al. 2017) for 100K steps (approximately 150 epochs) with 4K steps warmup on batches of up 490K source and target tokens obtained via gradient accummulation (Ott et al., (0188) with cosine learning rate annealing. We did not use checkpoint averaging, all the results are reported for the last checkpoint in the corresponding run.
Table 7: WMT' 14 English-to -errmmn translation, Transformer-big, batch 490K tokens, 150 epochs, no checkpoint averaging. Tokenized BLEU and detokenized SacreBLEU on WMT 14 (newstestl4)
Optimizer
Init LR
Weight decay
SacreBLEU
TokenBLEU
Adam
0.0006
28. 28.26
28.71
AdamW
0.0006
0.005
28.24
28.72
NovoGrad
0.04
0.0001
28.80
29.35
CONCLUSION
We propose NovoGrrd an adaptive SGD method with gradients normalized by the layer-wise second moment and with decoupled weight decay. We tested NovoGrad on deep models for image classification, speech recognition, translation, and language modeling. In all experiments, NovoGrad performed equally better than SGD and Adam/AdamW. NovoGrad is more robust to the initial learning rate and weight initialization than other methods. For example, NovoGrad works well without LR warm-up, while other methods require it. NovoGrad performs exceptionally well for large batch training, e.g. it outperforms other methods for ResNet-50 for all batches up to 32K. In addition, GovoGrad requires half the memory compared to Adam.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
James L. McClelland Andrew M. Saxe and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural network. In ICLR, 2013.
Valeriu Codreanu, Damian Podareanu, and Vikram Saletore. Scale out for large minibatch sgd: Residual network training on imagenet- kk with improved accuracy and reduced time to train. 19tp://11.44911 2017.
Zihang Dai, Zhilin Yang, Yiming Yang, William Cohen, Jaime Carbonell, Quoe Le, and Rus- lan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv:1901.02860, 2019.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jía, and Kaiming He. Accurate, large minibatch SGD: training ima- genet in hour. (tti::/70..000000 2017.
E. Hazan, K. Levy, and S. Shalev-Shwarke Beyond convexity: Stochastic quasi-convex optimiza- tion. In NIPS. pp. 15851593, 2015.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. arXiv: 05000, 2016.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. arXiv:1812.00187 2018.
Oriol Vinyals lan J. Goodfellow and Andrew M. Saxe. Qualitatively characterizing neural network optimization problems. In ICLR, 2015.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to SGD.
2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv:11609.044 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Oleksii Kuchaiev, Boris Ginsburg, Igor Gitman, Vitaly Lavrukhin, Carl Case, and Paulius Micike- vicius. Openseq2seq: extensible toolkit for distributed and mixed precision training of sequence- to-sequence models. arXiv:/805. J0387, 2018.
Li, V. Lavrukhin, Ginsburg, Leary, O. Kuchaiev, J. M. Cohen, H. Nguyen, and R. T. Gaddei. Jasper: An end-to-end convolutional neural acoustic model. arXiv:1904.03288 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. In ICLR, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv:1609.07844 2016.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. ICLR, 2017.
Y. E. Nesterov. Minimization methods for nonsmooth convex and quasiconvex functions. Matekon, 29:519531, 1984.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. arXiv preprint arXiv: arXiv:1866.00 180.00018. 2018

---

Under review as a conference paper at ICLR 2020
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In ICASSP, pp. 5206-520622 5210. IEEE, 2015.
B.T Polyak. Some methods of speeding up the convergence of iteration methods. In USSR Compu- tational Mathematics and Mathematical Physics, 1964.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Sashank J. Reddi, Stayen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In ICLR, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211 1-222, 2015.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv: 1508.07909 2015.
Noam Shazeer and Mitchell Stem. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv:1804.04235 2018.
Bharat Singh, Soham De, Yangmuzi Zhang, Thomas Goldstein, and Gavin Taylor. Layer-specific adaptive learning rates for deep networks. In ICML, 2015.
llya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initializa- tion and momentum in deep learning. In ICML, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv: 1706.03762, 2017.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stem, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In NIPS, 2017.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv: 03888, 2017.
Yang You, Zhao Zhang, Cho-Jui Hsich, and James Demmel. 100-epoch imagenet training with alexnet in 24 minutes. arXiv:1709.0501. 2018.
Adams Wei Yu, Qihang Lin, Ruslan Salakhutdinov, and Jaime Carbonell. Block-normalized gradient method: An empirical study for training deep neural network. /77.42277. 2018.
Zijun Zhang, Lin Ma. Zongpeng Li, and Chuan Wu. Normalized direction prentrion-r adam. 17Xv::179.04446. 2017.
10

---

Under review as a conference paper at ICLR 2020
A TRAINING OF DEEP LINEAR NETWORKS
Following (Andrew M. Saxe Ganguli 2013 an Goodfellow Saxe 2013) we will use Novo- Grad to train linear NN model composed of two linear layers W1, without any non- non-linearity. The model y = (w) W2) z should output 1 when x The model is linear function of the imputs, but it is non-linear function of the weights, since they are factorized into the product of layers' weights. Training the model is equivalent the minimization of the loss [[w1, w2) (wh W22 1)². The loss is not convex, and its minima are located on the curve: Wi W2 = Minima close to the points (1) and (1, 1) are good "flat" minima which generalize well. Minima close to axes are "sharp"
minima with bad generalization (see (Keskar et al. 2016)). The 2D-contour plot of the loss function shown on Figure
(Wi W2
1)2
global solutions
3
flat minima
sharp minima
2
0
-1
4
3
-2
0
2
W1
Figure 5: 2D-contour plot of surface (Wi W2 1)2 of linear model with two layers. The loss functions has many global minima located on hyperbola w2 = 1/w, Solutions near (-1,-1) and 11, are good "flat" minima, and solutions near axes are "sharp" minima.
We will study how the behavior of each algorithm depends on learning rate, weight decay and initialization. We will train the model with each optimizer for 500 steps using the rate, weight decay, and weights initialization. To use the same learning rate for all optimizers, we will use the "gradient averaging" for NovoGrad. We will also use the version of SGD with "gradient averaging" (similar to Adam): m, = 3-m-1+(11 3)-9t. For fixed learning rate this SGD version equivalent the regular SGD with momentum.
learning
Training trajectories for the baseline (fixed learning rate 0.2, weight decay 0.1, and 31 0.95, B2 0.5.) are shown on the Figure 6 All algorithms first go to the curve w2 1/w1, and then slide along hyperbola towards 1) or (-1,- -1). SGD is slightly off with respect to the optimal solution. Adam oscillates wildly around line w2 1 1/w1- AdamW behaves better since weight decay decou- pling significantly reduces osculations. NovoGrad is the most stable out of four algorithms, it also shows much better generalization than other algorithms and converges to (1, 1) closely following the minima curve.
Next, we increased learning rate from 0.2 to 1.0 while keeping weight decay equal to 0.1. Training trajectories are shown the Figure 7 SGD and Adam diverge. Only AdamW and NovoGrad converge.
Similarly, when increased weight decay from 0.1 0.5 while keeping learning rate 0.2, all algorithms except NovoGrad diverge, while NovoGrad demonstrates high robustness to the weight decay choice (see Figure 8)
Finally, we started training from different initial point. SGD and NovoGrad are most robust with respect the initialization, while A diverge (see Figure
11

---

Under review as a conference paper at ICLR 2020
NovoGrad
10
12
85
0.8
0.88
0.8
0.9
1.1
0.9
0.8
1.2
1.0
1.0
0.6-
0.6
0.6
0.6
1000%
6.2
0.2
0.2-
0.3
0.0
100
200
300
400
500
100
200
300
400
500
0.0
10n
200
500
400
500
0.0
100
200
300
400
500
i'steps
risteps
H
nssteps
#stops
Figure 6: DLN training- baseline: learning 0.2, weight decay 0.1.
SGD
Adamn
AdamW
NovoGrad
mill
8 1.0
1.0
1.0
1.0
0.9
0.9
0.9
0.9
0.8
09.8
0.8
10%
102
2.5
1.0
2.0
0.6
0.44
1.0-
0.0
0.0
0.0
300
400
100
400
200
300
Asteps
Hsteps
#steps
#stops
Figure 7: DLN training increased learning rate 0.2 - 1.0: SGD and Adam diverges, AdamW and NovoGrad converge.
To summarize our experiments with linear neural network: NovoGrad is more robust than other algorithms to the choice of learning rate, weight decay, and weight initialization.
12

---

Under review as a conference paper at ICLR 2020
SGD
Adam
AdamW
NovoCrrad
18
10
100
0.9
0.9
0.9
0.8
0.8
0.8
0.8
10%
10%
1.0-
3.0-
66-
0.6
0.6
M/0
0.2
0.00
0.0
200
300
400
200
400
200
400
#steps
#steps
#steps
#steps
Figure 8: DLN training increased weight decay 0.1 + 0.5: All algorithms except NovoGrad diverge.
SGD
Adam
Adam
NovoGrad
-2
101
1.2
1.2
1.2
1.2
1.1
1.0
0.5
0.9
1.2
0.8
1.0
1.1
1.2
0.5
0.9
1.0
1.3
1.2
0.5
0.0
1.1
1.2
3.5
80
140
120-
3.0
60
2.5
50
1.5
30-
1.0
20
0.5
10
100
200
300
400
500
100
200
300
400
300
100
200
500
400
500
100
200
300
400
500
#steps
#stops
#stops
#steps
Figure 9: DLN training "bad" initialization: AdamW diverges.
13


Reviews:
[{"review_id": "rJeCpuqnFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a variation of Adam that combines layerwise normalization and weight decay decoupling. They test quite extensively their algorithm against other optimization methods on varied tasks.\n\nFrom a theoretical point of view, the contribution seems very incremental. The authors acknowledge that weight decay decoupling is already present in the literature, so it appears that the only contribution is the layer-wise normalization. No justification is proposed as to why this kind of normalization would either accelerate the convergence or leads to better generalization. Proof of convergence even in a deterministic convex setting are missing, so the reader has to extrapolate correctness from previous work on adaptive gradient descent.\nOn the other hand, the proposed algorithm is tested on a great variety of tasks, using state of the art models. The reported performance for the benchmarks (checked for Resnet and Transformer-XL) are on par with what can be found in the literature. The proposed method outperform consistently the other optimizers in terms of generalization performance.\n\nI have some concerns regarding section 4:\n “SGD converges nicely toward (1, 1) but its trajectory is still slightly off of the optimal solution”. It is unclear to me what the reader should understand. Does it converge to the optimal solution? If yes, why should we expect the trajectory to follow the hyperbola?\nUsing the same learning rate for all methods is a bit odd. Why not search for the best learning rate for each optimizer and report its performance? It seems that the oscillation of some of the optimizers could be fixed by using a smaller learning rate. Also, it can be seen in section 5 that Adam consistently needs a smaller learning rate than Novograd.\nIf the difference between NovoGrad and AdamW lies in the layer-wise second moment, it is unclear to me why their performance should differ on this task, as each layer has only one weight. It would be great if the authors could clarify this point.\n\nAs a conclusion, I am a bit conflicted regarding this paper. The motivation for this modified version of AdamW are unclear, but the empirical results are convincing and rigorous. The authors made a great effort in testing in a variety of different settings. I’m leaning toward accepting this paper, to give the community a chance of testing and, maybe, adopting it.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgnbu7RYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the paper, the authors propose a novel optimization method for training deep learning models. The idea is from the LARS and AdamW. The authors then test the proposed method on multiple experiments, results showing that the proposed method works better than other compared methods.  The following are my concerns:\n\n1) No convergence guarantee in the paper. There are too many papers claiming faster convergence these days, proof of convergence guarantee is always preferred. \n2) The proposed method is straightforward and easy to understand. It is just a combination of AdamW and LARS. I am worried about the novelty of this paper. \n3) In the experiments, why the compared methods are usually different. For example, compared methods are Adam, SGD, and NovoGrad in table 4 and compared methods are Adam, AdamW, and NovoGrad in table 6.  Why not compare all these methods?\n4) When the batch size varies,  is it required to tune beta_2 accordingly? I didn't find it clearly mentioned in the paper, could authors explain how to set it? \n5)  I am confusing that NovoGrad method works much better than Adam or AdamW in Table 6 with no weight decay, more explanations are required. \n6) It is unclear why NovoGrad is better than LARS. LARS normalizes learning rate through |w|_2/|g|_2. The authors should explain why normalizes using layerwise |g|_2 is better.\n\nAlthough the idea is straightforward, the proposed method may be helpful for the community.   I will consider increasing the score if authors can address my concerns.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1gFrcwAFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am not sure about my assessment of this paper. \nThe authors propose their approach as a mixture of two other approaches. Then, they proceed with an illustrative 2-D example where they apply the same hyperparameters for all tested methods. This does not seem appropriate especially given that later in the paper they show that SGD with momentum and Adam use very different hyperparameter settings such as learning rate and weight decay. Basically, SGD and NovoGrad perform very similar in Figure 6 and the difference might be due to a too large learning rate for SGD (and other approaches). \nIncreasing it further does not help as Figure 7 suggests. It is not the case for NovoGrad because of its gradient normalization. However, what the authors don't show us is a different experimental setup with some trivial objective function where it would take NovoGrad an enormous amount of steps to converge to the optimum IF the algorithm is initialized far enough from the optimum. Since NovoGrad is based on normalized gradients, its steps are pretty much just signs of the original gradient multiplied by learning rate, especially in 2D. Thus, with some initial learning rate of 0.1 and when initialized in say (1, 10^10), it would take NovoGrad about 10^9 steps to converge independently on the scale of the objective function. This is not the case for SGD whose gradients are not normalized. This is actually a strong counterargument to the main claim of the paper that \"NovoGrad is robust to the choice of learning rate and weight initialization\". This argument in the paper is based solely on the results of the toy problem where other methods used large learning rates. \n\nThe paper includes a set of experiments containing different methods and their hyperparameters. \nWhen the authors use some cosine function to schedule learning rate in Table 1, they do it for AdamW and NovoGrad but they use polynomial schedule for SGD. Why? Weight decay and learning rate values for SGD and NovoGrad are very different in the same table, why? Do the authors explain why this difference may happen, they don't. I believe that there is a better answer than just \"this is the result of hyperparameter tuning\". \n\nThe authors note that when beta=0, NovoGrad becomes layer-wise NGD with decoupled weight decay. They suggest beta to be in [0.2, 0.5] and I am wondering which beta were used in different experiment (the default suggested value is not given). \n\nThe experimental results suggest that NovoGrad performs slightly better than SGD with momentum but it has more hyperparameters and we don't know whether the results are due to different computation efforts in hyperparameter tuning. Figure 5 compares Adam and Novograd on WikiTex-103 and shows that Adam converges faster in terms training perplexity. However, Adam's test perplexity is worse than the one of NovoGrad. Interestingly, NovoGrad's test perplexity is better than its training perplexity especially in the beginning. It seems that NovoGrad is good here because it does not converge well, i.e., the results are likely due to regularization which is problematic in the original Adam. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presented an adaptive stochastic gradient descent method with layer-wise normalization and decoupled weight decay and justified it on a variety of tasks. The main concern for this paper is the novelty is not sufficient. The method is a combination of LARS and AdamW with slight modifications. Although the paper has good empirically evaluations, theoretical convergence proof would make the paper more convincing. 
Decision: Reject