Training Instability and Disharmony Between ReLU and Batch Normalization
Authors: Inyoung Paik
Jaesik Choi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep learning
Gradient Exploding
ReLU
Batch normalization
Training instability
LARS
WarmUp

Abstract: Deep neural networks based on batch normalization and ReLU-like activation functions experience instability during early stages of training owing to the high gradient induced by temporal gradient explosion. ReLU reduces the variance by more than the expected amount and batch normalization amplifies the gradient during its recovery. In this paper, we explain the explosion of a gradient mathematically while the forward propagation remains stable, and also the alleviation of the problem during training. Based on this, we propose a Layer-wise Asymmetric Learning rate Clipping (LALC) algorithm, which outperforms existing learning rate scaling methods in large batch training and can also be used to replace WarmUp in small batch training.

Under review as a conference paper at ICLR 2023
TRAINING INSTABILITY AND DISHARMONY BETWEEN RELU AND BATCH NORMALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks based on batch normalization and ReLU-like activation functions experience instability during early stages of training owing to the high gradient induced by temporal gradient explosion. ReLU reduces the variance by more than the expected amount and batch normalization amplifies the gradient during its recovery. In this paper, we explain the explosion of gradient mathe- matically while the forward propagation remains stable, and also the alleviation of the problem during training. Based on this, we propose a Layer-wise Asymmet- ric Learning rate Clipping (LALC) algorithm, which outperforms existing learn- ing rate scaling methods large batch training and can also be used to replace WprmUp in small batch training.
INTRODUCTION
The success of deep neural networks is based on their great expressive power, which increases exponentially with depth (Chatziafratis et al. 2019). However, the deep hierarchical architecture of a deep neural network may induce the exploding/vanishin gradient problem (Pascanu et al. 2013), which degrades performance and may even make training impossible.
Several techniques have been suggested to address the aforementioned problem, including better weight initialization (Glorot & Bengio 2010: He al. | (0015), activation functions (Nair & Hinton
2010 Klambauer et al. (211)). residual connections (He et al. 2016), normalization methods (loffe
& Szegedy 2015] Ulyanov al. 2016 Nam & Kim 2018 Ba al., 2016 Wu & He 2018 Qiao
et al., 2019), etc. However, Contrary to popular belief, gradient explosion is present in various
modern deep learning architectures that use batch normalization and ReL activation functions (Figure D. The stable flow of forwarding activation does not guarantee the stable flow of backward gradient when an entropy difference exists between the layers (Philipp et al.) (017)).
A gradient explosion in modern deep learning architecture has been reported in number of papers (Philipp et al. 2017 Frankle et al., 2020 You et al. 2017). but its cause and solution have not been sufficiently researched. (Philipp et al. 2017) discussed the context and presented an intuitive un- derstanding of the problem, considering the modality of gradient explosion while activation remains stable, the increase in entropy induced by the activation function, the alleviation of the problem ing residual learning, etc. However, although the authors observed that gradient explosion occurs only in the case of batch normalization, they did not explain this phenomenon. They speculated that the sampling етгоr of the normalization procedure amplifies the problem- no clear corre- lation has been reported between the gradient explosion rate and the expected sampling error (batch size, difference normalization scheme, etc.). To the best of our knowledge, this is the first attempt to demonstrate how disharmony between activation function and batch normalization causes gradient explosion and training instability during the early stages of neural network training mathematically. The alleviation of the problem during training also discussed.
Exploding/vanishien gradient is a well-known problem in deep learning. It may seem unnatural that it still cxists but is not widely known. One reason is that the problem is not as severe as before. The exploding rate is approximately / - 1) ~ 1.21 per effective depth. is tolerable in networks with tens of layers those with hundreds of layers and dense residual connections. Moreover, the exploding gradient at the initialization state is rapidly relieved during training, and even vanishing gradient state be approached. Thus, this problem has been referred to by different names in

---

Under review as a conference paper at ICLR 2023
IRC Mothe-l Riiacci NaT
Ierpout
10P
307
107
Crudant Eigtoaan
Crodent (toahnt-ghhraaa
Saah
207
V0032
100
201-
109
Revers Order)
LayerSttewrren Oidee
(b)
Figure 1: Gradient explosion rate (v "aarrggh /Varrgg)//Varii (V)) of deep neural network models at the initialization state corresponding to (a) different architectures and (b) activation functions. The explosion rate is upproximately / 1) vanilla networks with batch normalization; but is lower in architectures with residual connections (He et al. 2016). which reduces the effective depth. Moreover, gradient explosion does not occur in architectures with layer normalization (Ba et al.] 2016), including transformer-based architectures (Dosovitskiy al. 2020 Liu et al. (021). Figure (b) is plotted using 51 51-layer VGG-like (Simonyan & Zisserman, 2014) architecture. Smoother variants of ReLU (Maas al., 2013 Clevert al.) 2015: Ramachandran al. 2017 Hendrycks
& Gimpel 2016) exhibit lower exploding rates as they exhibit flatter behavior near zero and lower
signal blockage the initialization state. In the extreme case, gradient explosion does not occur
the activation function is not used. Note that it also does not occur with DropOut (Srivastava et al.,
2014), which can be regarded as ReLU that blocks signals randomly.
the literature, such as 'large gradient magnitude at the early stage of training' or 'instability of deep neural network at the early phase of training' (Frankle et al. 2020 Goyal et al., (0175)) It is also often treated as an 'instability problem of large batch training' (Goyal et al.) 201 since the problem becomes severe when a large batch size (and high learning rate, accordingly) is used during training. Although training instability occurs only during early stages of training. can disrupt the balance between the layers and degrade the final performance (Goyal et al., 2017b You et al. 2017
2 How GRADIENT EXPLOSION OCCURS
As weights repeatedly multiplied during forward/backward propagation, the exploding ishing gradient problem is commonly believed to be caused by excessively large or small parameters (Bengio et al. 1994 Pascanu et al. 2013). Thus, it has been largely treated the problem of ini- tialization and maintenance of optimal weight scales. In that case, the problem would have been 'solved' with the advent of (batch) normalization (loffe & Szegedy 2015). which automatically cor- rects suboptimal choices of weight scales. However, maintaining the norm of weights, and thereby the norm of forwarding propagation, does not automatically maintain the norm backward propa- gation.
Rectified Linear Unit (ReLU) Nair & Hinton 2010) (as well as its smoother variants (Maas al. P017/lo Clevert el., 2015 Ramachandran et al. 2017] Hendrycks & Gimpel 2016 blocks approximately half of the activation each instance. this sense, (He et al. 2015) assumed that it halves the output variance based on some zero-centered assumptions. Therefore, the authors concluded that initializing weights with NO0, /2 2/Rout) maintains similar variances in both forward and backward propagation.
2013 Klambauer et al.) 2017
However, the relationship of the activation function with the input should also be considered. As depicted in Figure 2 using the positive part of the activation different from blocking randomly selected activations. In short, (He et al. 2015) described the situation assuming DropOut (Srivastaval (et al.) 2014)) to be the activation function instead if ReLU. In that case, both forward and backward signals roughly halved, and neither exploding vanishing gradient occur. This is verified in
2

---

Under review as a conference paper at ICLR 2023
1.0
1.0
1.0
0.8
0.8
0.8
Poquency
0.6
0.6
0.6
0.4
0.4
0.4
o²-1
0² 0.5
0.34
0.2
0.2
0.2
0.0
0.0
0.0
2
(a) M(O,
(b) 0.5
ReLUD00
Figure 2: (a) Sample of normally distributed input. (b) Hypothetical output distribution assuming that ReLU randomly drops half of the input. (c) Real distribution after ReLU activation, which exhibits smaller variance than that in (b). (He et al. (015)) assumed the post-ReLU distribution to be similar that depicted in (b), in which half the variance is compared the input if the input is zero-centered. Since ReLU also halves the gradient during backpropagation, the authors concluded that selecting optimal weights to maintain input variance also guarantees stability of the variance of the gradient. However, the correlation between the input and the activation function may affect input variance and gradient variance differently. Thus, in batch normalization, which recovers the variance decreased by the activation function, the backpropagating gradient may increase exponentially while the forward propagation remains stable.
Figure hb). However, since the real activation functions are highly dependent on the input, it may affect the variance differently during forward and backward propagation.
Proposition 1. Let X = RRLUUYY = max (Y,)). where Y ~ Nu, o²). Then, we have:
E(X) = 1(1 - 0l ())
a
aaa
20
22T
Var(X) = (0² (1))) - o1- )( + e
2² (H(1 o( (0=
σ
e
20 )
(2)
22n
2xx
where d(x) = -0 V2n
12 is the cumulative distribution function of the standard normal dis- x2
tribution.
Proof. See Appendix A.1
Theorem 1. Consider the repetitive neural network architecture, f Rdd m Rda-1 where CoreeNNoo(W""(eOOOO"
a1+1 =f"(("")
b²) (W"(ReU((a" "+6")) + gr+1,
9 where n0+1, 1g e 12d.+1 are affine transformation parameters and μ. Bdd+t are the es- timated mean and standard deviation. Assume that inputs (x²) ) are independent and follow the zevo-centered normal distribution and that the weights (W3) are sampled from the zero-centered distribution. Moreover, let the gradient of the output (9)+) = dL/dx where L denotes the loss) be not correlated the input. Ignoring the sampling error of batch normalization, we have:
E > Var(z,)) Var(2 (Warl,, ++
π
(3)
-
where the equality holds when the activation before the normalization layer exhibits identical vari- ance.
3

---

Under review as a conference paper at ICLR 2023
Proof. See Appendix A.2
size of activation remains similar, the gradient can be expected to increase exponentially at
rate of V x-1 1.21
Corollary 1. In addition to the assumption of Theorem 1, let x=+1 and gh be zero-centered. Then, we have:
E E (a))
(4)
--1
where the equality holds when the activation before the normalization layer exhibits identical vari- ance.
Proof. Trivial, since Var(X) = E(X2) for zero-centered distributions.
The theorem can be roughly explained by the fact that the ReLU activation function reduces the variance by approximately (# 1 1)/22 instead of 1/2 under the zero-centered assumption. Without loss of generality, let the weight be initialized such that it maintains the variance after the transfor- mation layer. Then, the sample variance before the normalization layer becomes (æ 1)/2x and the forward activation is divided by this value. During backpropagation, the gradient is multiplied by 2n/(n= ( 1) at the normalization layer, and half are dropped at the activation function. Therefore the gradient is amplified at rate of T/(TT 1). The formula is much more complex without the zero-centered assumption, but the structure remains essentially identical. Since every activation function decreases variance (or increases entropy) by a greater amount than it blocks the gradient, always amplifies the gradient if the assumptions hold.
2.1 THEN WHY DOES GRADIENT EXPLOSION EXIST ONLY DURING EARLY STAGES OF TRAINING?
By Theorem 1, gradient explosion in a deep neural network seems to be unavoidable. However, in reality, gradient explosion exists only during early stages of training. The dynamics of a deep neural network change rapidly during the early stages of training.
Various factors contribute to gradient explosion or vanishing. For example, gradient explosion as- signs a high gradient to low (near-input) layers and induces high variance in the distribution of weights of low layers, resulting in a mismatch of weight norms between layers. This contributes to the direction of the vanishing gradient. (Imagine that y = WyW1,, where w1 = 100 and w2 1.) On the other hand, gradient explosion is essentially induced by the interaction between batch nor- malization and activation function. Thus, does not occur when the activation functions are lincar or 'sseudo-linear' (Philipp et al. 2017). ReLU-likee activation functions a node only outputs values with identical signs in most cases or |p/a| >> 1. In the extreme case, the invalidated activation functions can be replaced with scalar values (0 or 1). Such layer does not generate gradient explosion, although it does not function as valid layer contributing to the effective depth either.
be (partially) disabled
Corollary 2. Under the assumptions of Theorem let us consider inputs that no longer zero- centered and x. ~ N a At the limit where ReLU becomes completely pseudo-lineur, we have:
lim df/P" DOVE
E Var(a"))
)
(5)
where the equality holdy when the activation before the normalization layer exhibits identical vari- ance.
Proof. See Appendix A.3
4

---

Under review as a conference paper at ICLR 2023
10-2
Varlg)
10-6
10-10
var(x)
100
10-1
10-3
|Mean(x)/Std(x)] Var(w)
step 0
step 10
step 100
10-4
0.01
0.00
11
[Corrix
14
[Cor1(x
20
40
60
80
100
120
140
Layers(Reverse Order)
Figure 3: Sample chart of the variance of gradien//aciivaiion/vegg, mean-std ratio, correlation be- tween activation and gradient (E |Corr (z,, g.)l/C), and the average correlation between activa- tions (Ei [Crrr(x, x (/C((CCCC -1)C during early stages of training. The order is reversed the left side represents the layer near the output. All weights are initialized from (00. 0.013) for better visualization. Var(x) is normalized near at every batch normalization layer. However, the variance observed to increase after the ReLUU and convolution layers during training, even when the norm of weights remains almost constant. Gradient explosion at the initialization state may be reduced by various factors, including an increase in the norm of weights near the input, pseudo- linearization of the activation functions, the correlation between the input and the gradient, etc. However, the correlation between activations seems to contribute to the decrease of the gradient primarily. It alters the variance after linear transformation and decreases the norm of the gradient flow while the forward flow remains stable since the variance of the sum of dependent variables different from that of independent variables (See Theorem 2.)

---

Under review as a conference paper at ICLR 2023
However, the most fundamental factor that decreases gradient seems to be the correlation between input activations (Figure 33. Most of the assumptions in Theorem 1 seem to remain valid even after the neural network trained. Internal variables such as weight, activation, and gradient, tend to remain zero-centered. And the correlation between activation and gradient also tends to remain small. However, neural network tends to generate multiple redundant and duplicated signals during training (Anwar et al. 2017), which necessitates 'neural network pruning'. Since the variance of the sum of correlated random variables different from that of independent random variables, even gradient vanishing may occur if a high correlation exists between the inputs.
Theorem 2. Under the assumptions of Theorem 1, let us consider the case in which inputs are perfectly correlated. Then, we have:
E E
Var(s)) Va
π
(6)
2
where the equality holds when the activation before the normalization layer exhibits identical vari- ance.
Proof. See Appendix A.4
SOLUTIONS
A large amount of the gradient induced by gradient exploding is not an error of calculation. A change in the lower (near-input) layer actually creates significant impact on the upper layers. Ac- tivation functions cannot be designed avoid decreasing entropy as compromises their function. Residual connections (He et al. 2016) decrease the effective depth and ameliorate the problem sig- nificantly, but not completely.
Fortunately, the problem is temporal. Although vanishing gradient may occur in the middle stages of training,
less harmful than gradient explosion and practical observations suggest that no measures need to be taken to resolve it at this stage. In this sense, WarmUp (Goyal et al.) (2077b) is an efficient and effective method to bypass the problem. assigns a very small learning rate during the early stages of training, preventing the assignment of excessive noise to the neural network until the learning is stabilized.
But in more extreme cases, such large batch training (and large learning rate, accordingly). better solution required. Layer-wise Adaptive Rate Scaling (LARS, Algorithm 1) (You et al, 2017) maintains the ratio between gradient and weight as constant for each layer. It is powerful and convenient tool for large-batch training and has been successfully applied in several large-scale experiments (Mathuriya al. 2018 8
Narayanan al., 2019 Chen al. 2020a by Yu et al. 2022). Certain modifications have also been suggested. (You et al. 2019) modified the formula and applied the technique the Adam (Kingma & Bal 2014) optimizer to train a large-scale language model (Algorithm 2). (Huo et al. (022) suggested using the average of the norm of the per-sample gradient instead of the minibatch gradient.
However, LARS and similar techniques have hardly been applied in small-batch experiments, since they restrict training and induce minor performance degradation. Even for large-batch training, (Nado et al., (02T) showed that we may obtain better results without using LARS if we can control the instability using WarmUp and careful hyperparameter tuning.
A large amount of the gradient means that a small change in that parameter affects the output (and near-output layers) significantly. But the gradient descent algorithm even assigns larger step sizes to that parameter, assuming the parameters differentials are negligible. However, since the neural network is highly hierarchical, this can make the training process noisy and unstable. The ideal solution would be higher-order optimization algorithm, e.g., Newton's method (Newton 1711), but is not practical to obtain a Hessian matrix for deep neural network with millions billions of parameters.
roughly connected in parallel and (joint) second-ordes
Therefore, an engineering solution capable of assigning a smaller learning rate corresponding to instances of gradient explosion is required. Existing methods may suffice for this purpose, but they

---

Under review as a conference paper at ICLR 2023
functions only during the early stages of training, but it reduces the learning rate for every layer with a fixed schedule. LARS reduces the learning rate when gra- dient explosion occurs, but it affects every layer during the whole training procedure. Moreover, amplifies the learning rate when the gradient is excessively small, which may not be necessary.
not very specific-Waate
We don't want single step of gradient update causes more than certain level of change in the output (and near-output layers). The amount of change would be proportional to the gradient norm times the learning rale, therefore the learning rate should be bounded by a value that inversely proportional to the norm of the gradient. We found that the local learning rate of LARS Yoou et al., 2017) with a higher coefficient can serve suitable upper bound, better than the simple inverse of gradient norm.
Therefore we propose Layer-wise Asymmetric Learning rate Clipping (LALC), which sets an upper bound on the learning rate instead of scaling it. Algorithm 3 describes the algorithm, and Appendix presents the PyTorch (Paszke et al., 2019) implementation. The changes are simple but it has desired properties-LAre -LLC does not amplity the gradient even when the gradient is significantly lower than the weight. Moreover, LALC is naturally disabled as learning stabilizes and global learning rate decreases since it uses fixed upper bound using ratio between the weight and the gradient. The norm the gradient after applying optimization techniques (c.g., momentum, weight decay, etc.) is calculated- the obtained value is slightly better than that obtained using the norm of the original gradient in our experiment.
4 EXPERIMENTS
The ResNet50 (He et al. 2016) model is used in both CIFARIO (Krizhevsky & Hinton 2009) and ImageNet (Deng et al. 2009) experiments. For LARS (You et al. (017) and CLARS (Huo et al. 2021 algorithm, the coefficient 7) between 10-2 and 10-4 is used, where smaller y is used for larger batch sizes. We performed a hyperparameter search with an order of 10. Therefore careful hyperparameter tuning like (Nado et al., (02T) may improve the performance. On the other hand, 10-2 is used for LALC in all cases, which yields satisfactory results. A learning rate of 0.1 is used for batch size of 128 and it linearly scaled with batch size (Goyal et al. 2017a). Please see Appendix B for further details.
LAMB You et al. 2019). which (modified) LARS based on the Adam optimizer (Kingma & Ba 2014) is not considered in these experiments. The Adam optimizer requires an additional hyper- parameter have different characteristics, complicating its fair comparison with an SGD optimizer. For example, it is known to exhibit very fast convergence, but basic SGD optimizer with momen- tum often outperforms it with a longer training schedule. Although the whole experiment can be repeated with an Adam-family optimizer, we do not deem it to be necessary.
The experimental results are depicted Figure 4 and 5 LALC outperforms learning rate scaling methods in large batch training, and exhibits similar performance compared to WarmUp (Goyal) et al. 2017b) in small batch training. Application of WarmUp to the LALC optimizer does not
Algorithm LARS (You et al., 2017)
Require: w/ Weight matrix of layer
Require: " Learning rate at step
Require: 17 LARS coefficient
Require: 8: Weight decay rate
Require: f: Gradient optimization algorithm (SGD, Adam, etc.)
Require: e: Small constant avoid numerical error
while < 'T' for each layer I do
gi & dL
dw!
h/ f(g w4,9)
>/ Ml|Wo|l
+ 11 11+
wo <- w 36 2
end while
7

---

Under review as a conference paper at ICLR 2023
Algorithm 2 LARS/LAMB (You et al. 2019)
Require: Φ: ::A scaling function
while < 'T' for each layer do 9 f dr-
dw!
X + Al1lll1
w < w N,h
end while
A Algorithm 3 LLLC (ours)
while < 'I' for each layer I do
g - dL.
dw!
h4 f(g', 3)
X mllw'll
li5T++
w++1 < w min(ni; N/)
end while
make significant difference. This is attributed to the fact that the primary function of Warm is the reduction of training instability during early stages of training, which is already handled by LALC. To the best of our knowledge, there is general optimization theory that suggests using much smaller learning rate at the beginning training compared to that in the middle of training.
Nesterov momentum (Botev al. 2016 improves the performance slightly but stably in variety of setups. However, we could not find evidence that particularly overcomes training instability. (Lin) et al. 2020) proposed method to improve Nesterov momentum but it is targeted for distributed learning and no improvement is observed in the single-node case. Huu et al. (2021) is also targeted for distributed learning, but it also suggests using per-sample gradient instead of minibatch gradient. Some benefits are observed in certain cases, but the computational cost of calculating the norm of the per-sample gradient is not negligible.
DISCUSSION & FUTURE WORKS
In this paper, we analyze gradient explosion that occurs in general neural network architecture with ReLU activation and batch normalization. However, further theoretical analysis is required. For example, Theorem states that gradient vanishing may occur if a high correlation exists between the inputs. Experiments the simple MLP model (Figure[T[an may suggest that the equality dition (identical variance before batch normalization layer) is not very tight-but an upper bound is required guarantee the gradient decrease theoretically. The problem is that there is probability of divergence when the variance approaches zero. Thus, a small constant should be added to the estimated variance avoid a numerical error (denoted by "e' in (loffe & Szegedy 2015)). However, this makes the equation mathematically tricky-we we intend to explore it in a Future work. The con- volutional layer is observed exhibit smaller explosion rate compared fully connected layers (Figure ( This may be attributed to the correlation between inputs in the CNN since spatially adjusted pixels tend to exhibit similar activation.
A local learning rate of LARS (You et al. 2017) is used to select upper bound of the learning rate. This is an engineering solution, which is not theoretically grounded. Although exact calculation practically unrealistic, practical approximation of the second derivative the noise induced by the update may be estimated to determine a more appropriate upper bound.
Networks using layer normalization, including transformer-basee models (Dosovitskiy et al. 2020 Liu et al. 2021), do not exhibit gradient explosion. This may partially contribute to the success of large-scale transformer models in various domains. However, it does not necessarily imply that batch normalization should be replaced in models of every type, since layer normalization may suffer from greater sampling noise depending the network architecture.
8

---

Under review as a conference paper at ICLR 2023
91
Volccation Accuresy
Aicloursi
80
LAUGEOUSS
25
LALC
ARS
ARC440mmipp
IXPC-Vlamilp
CLARS
CL/RS5
75
Viammjp
SOD+NNrmUp
Vestorov WarmUp
Nesteriy vtomuu
70
2042
Epochs
(a)
(b)
Figure 4: CIFARIO experiment for (a) different batch sizes and (b) training curves for 4k batch size. LALC outperforms existing methods in large batch training, and exhibits similar performance com- pared to WarmUp (Goyal al. 2017b) in small batch training. Application of WarmUp LALC does not have a significant effect. We assume that one of the primary functions of WarmUp is to relieve the training instability, which already handled by LALC. We could not observe any evi- dence that Nesterov momentum overcomes training instability. LARS and CLARS are sufficiently effective handle the training instability, but they are applied throughout training every layer and degrade performance slightly. Using excessively large batch size in the CIFAR10 experiment degrades performance irrespective of the algorithm since CIFARIO only contain 50k training data points and the parameter can only be updated number of times per epoch with batch sizes of 8k or 16k.
Tops Acuuraty
80
LALClours!
20
LALC WammUp
Warmip
SCD WCC+WWemUp
10
30
50
Figure 5: ImageNet experiment with 8k batch size. We obtained similar results and training curves the CIFAR experiment. Both LALC and LARS effectively handled training instability the early stage of training, but we also found minor performance degradation in LARS. We also could not observe a significant difference when WarmUp is applied to LALC.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jazefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg. Martin Wicke, Yuan Yu, and Xiaogiang Zheng. TensorFlow: Lcree-scale machine learn- ing on heterogencous systems, 2015. URL hitps: /ww. www.ttennsooooom org/ Software available from tensorflow.org.
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruming of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13(3):1-18, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1o0n::606555555 2016.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions neural networks, 5(2):157-166, 1994.
Aleksandar Botev, Guy Lever, and David Barber. Nesterov's accelerated gradient and momentum as approximations to regularised update descent, 2016. URL https://arxiv.org/ /
1607. 01981
Vaggos Chatziafratis, Sai Ganesh Nagarajan, Ioannis Panageas, and Xiao Wang. Depth-width trade- off's for relu networks via sharkovsky's theorem. arXiv preprint arXiv:1912.04378 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information pro- cessing systems, 01-22243-222555 2020b.
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 15 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint UrXiv:2010.11922 2020.
Jonathan Frankle, David Schwab, and Ari Morcos. The early phase of neural network training. arXiv preprint 20X.:20022.13365. 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski. Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint /rXiv://06.0000 1706.02677, 2017a.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangging Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 1706.0270..0267, 2017b.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). ar preprint
(etv:660608115 2016.
Zhouyuan Huo, Bin Gu, and Heng Huang. Large batch optimization for deep learning using new complete layer-wise adaptive rate scaling. In Thirty-Fifth AAAI Conference on Artificial Intelli- gence, AAAI, pp. 2-9, 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Johan Ludwig William Valdemar Jensen. Sur les fonctions convexes et les inégalités entre les valeurs moyennes, Acta mathematica, 30(1):175-193, 1906.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. International Conference on Learning Representations, 12 2014.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances neural information processing systems, 30, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Tao Lin, Lingjing Kong, Sebastian Stich, and Martin Jaggi. Extrapolation for large-batch training in deep learning. In International Conference Machine Learning, pp. 6094-6104. PMLR, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10012-10022, 2021.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 1008.033,,, 2016.
Andrew L Maas, Awni Y Hannun, Andrew Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, pp. 3. Citeseer, 2013.
Amrita Mathuriya, Deborah Bard, Peter Mendygral, Lawrence Meadows, James Arnemann, Lei Shao, Siyu He, Tuomas Kärnä, Diana Moise, Simon J Pennycook, et al. Cosmoflow: Using deep learning to learn the universe at scale. In SC18: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 819 829. IEEE, 2018.
Zachary Nado. Justin M Gilmer, Christopher J Shallue, Rohan Anil, and George E Dahl. A large batch optimizer reality check: Traditional, generic optimizers suffice across batch sizes. arXiv preprint arXiv:2 102.0000.666 2021.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
Hyeonseob Nam and Hyo-Eun Kim. Batch-instance normalization for adaptively style-invariant neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gre- gory R Ganger, Phillip Gibbons, and Matei Zaharia. Pipedream: generalized pipeline par- allelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pp. 2019.
Isaac Newton. De analysi per aequationes numero terminorum infinitas. 1711.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318. PMLR, 2013.

---

Under review as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vitito Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http: /ptptess/ppppsssp.ppesst http:appeerreeerrss....ccppppeee
ibbrar.
pdf
George Philipp, Dawn Song, and Jaime G Carbonell. The exploding gradient problem demystified- definition, prevalence, impact, origin, tradeoffs, and solutions. arXiv preprint arXiv:1712.05577, 2017.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch- channel normalization and weight standardization. arXiv preprint arXiv:1903. 10520, 2019.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05%4 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitiing The Journal of Machine Learning Research, 15(1):1929-1))) 1958, 2014.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint
2016.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 730.0088888888 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint 1rXXvv:904000966 2019.
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in PyTorch. arXiv preprint arXiv:2109. 120.299,..2298, 2021.
Yang Yu. Zixu Zhao, Yueming Jin, Guangyong Chen, Qi Dou, and Pheng-Am Amn Heng. Pseudo-label guided cross-video pixel contrast for robotic surgical scene segmentation with limited annotations. arXiv preprint arXiv:2207.0066 2022.
A APPENDIX
A.1 PROOF OF PROPOSITION 1
We the moment generating function and its properties.
12

---

Under review as a conference paper at ICLR 2023
Mx(1) E
=c = 0) / c²z
1
2)
20
dx
V2T0
P(N(H, (00,,,000000 +0)+ +
25 dx.
00
V2mT
= o x(- ( + elll+ x2,2
1
(0-p-a21f Zo
dx
V2TO
e /2 is the cumulative distribution function of the standard normal dis- where (bx) 1
v2= tribution. Substituting x'
(x μ 0²t)/o.
Φ - + eat+ x
e
da'
(8)
= M eat 222
0²t
We calculate the first and second moment from the moment generating function.
d dt
Mx(1) = (11 + o .2,3
(1 Φ0 1 1 /
+ qult 22
1 e- 2 20
(-1)
V2T
(2++ -²,
(1 - 00 1²
1 3 Peoot 2202 (ntt 2
+
20
(9)
2T
222
(1 0l u+
= =
+
2n2
v 2m
d d1'
/ u o2t
Mx(1) (o² (u (20+0))))e
(1-
+ (μ + 0²1) eat c 2422 d
O - )
J
(10)
0
= (o² ( (
(1 - -1 + -
)) + (u+ 021)
o
e
20
22T
Now, we finally obtain:
B(X) di Mx = p(l - )1
(1I)
2
d²
d
Var(X)
.M (f)) 1=1
My (2)
dt²
dt
(12)
(a² + 2 (1
06 -) + H 22
- ,
-wm(1 - x( +
2²
)
22
which proves the proposition.
A.2 PROOF OF THEBREM 1
Let us denote ReLU(x") = W" + = gn+1, Batch.Norm(a*t)) x+1
First, we need to estimate the distribution of a1+1 which is estimated by the batch normalization layer.
13

---

Under review as a conference paper at ICLR 2023
E
(o) Wij) + 6
(B(a) (W?) +
(13)
of V2T
Wi))
+ 63
This equation describes the sampling process of the batch normalization layer-the statistics of model parameters(W??, ij. b?) are not considered. Thus, 'E" denotes the expectation related to the input distribution. Similarly,
(05 = Var(a)t))
Var
(I. f (z)++W)) + b9)
Var(o W?))
E(wg)
(14)
[(w))(o))((-0000 ((0)) (o7)²
2T
1 [o7))(w7))
2 2n
Recall that 25"" is given by:
29 27t1 BatchNorm(2att
+n+1
125-1 V3 + 83
(15)
0gtl
where n 1,0+1 R are affine transform parameters of batch normalization.
Now, we calculate the gradient of the input using the backpropagation algorithm. Since ReLU operation not only changes the activation but also blocks the gradient, they are treated separately. For indices such that x. < 0, the gradient g is just zero. Then, for each that is not blocked by ReLU (i.e. for E (i xx > 0}), we have:
dL dr
Σ dL day
gf
dr dxy
9j n+1 dap+t dān+1
(16)
din de?
n+1
Σ oJ 1n+l W
14

---

Under review as a conference paper at ICLR 2023
n+1 35 n+1
Varlg,) = Var Σ 9j o7 W')
Σ Var(g) n+1 11 a+1
g Wiy)
(17)
n+1
Evar(ott) (E
(W?))
Ei| 11 - w393
(6
Σ Var(g] n+1 (or())t))())
E
10 [E
Since f(x) 1/2 is convex at x O, we can apply Jensen's inequality (Jensen 1906).
Σ Varlg Varls,t)()
1
B((W
E((G)ti32]
E((W )
Σ (a
EIC 2x) (o
(18)
2T
E(ggth
a E|Σn 2o
2m (o
π 1 En(op)²
The equality holds when oz+ identical for all j. Note that this formula no longer dependent on the input index if æ, is not blocked by ReLU, Moreover, as the input distribution is zero zero-centered by assumption, the probability of passing ReLU is 1/2 for all i, independent of o.. Therefore, can arrange the equation:
(ar E(ott)ieVar(1
E
2xT
1(o Var(9)
1/ (ar())) 1+1
2
2
π - 1 Ex(ot)²
3
(19)
1 2m
(gg)
2
π
This proves Theorem 1.
A.3
PROOF OF COROLLARY
Using the same notations as Theorem 1, let 1+ == (ilp 0}. We have:
15

---

Under review as a conference paper at ICLR 2023
lim 2//p2 -00
(0
lim Var(a)=
ot /P2 -00
lim
of /P2 -00
"//PF +0V-
lim
[(W)) + -
x( 14
o7
+ kr
v 2T
0 (,,
21 - o.
(1 o( o(-
0:
Σ (WF)
(44) 0) + u. o
0 - (p
0) + 1 o.P
0)3)
2T
V
(WT"(((')+++ + 1
0
0 (1 - + 14 0
0)3)
2
22
IET
Σ (W")) (KT) (4+))) + Σ (W7)².0
Σ (W)))
SE
(20)
The remainder of the proof is similar to that of Theorem 1.
A.4 PROOF OF THEOREM 2
Since inputs(æ") are perfectly correlated, let b ~ NO0, 1) and x² bl, where E R is treated as a constant. Let ReTU = -* "**** + f=+1, and BatchhNorm first calculate the mean and variance of fn-1, which are estimated by the batch normalization layer. As the sign of xn reversed corresponding to positive and negative t, they are treated separately.
= x=+1 We
1 = Elist1
E 0j 2 0) Ela < 0.P(b 0)
-E
W ReLR (teeLU(())))))) (616 2 0] + _ E W)BReU((-t)(-bbbb55
(21)
2
Σ (R) + ReLU(-1:))
2m
v 2m
The conditional expectation of b calculated using Proposition Then, we directly calculate the variance before normalization.
12 =
(1 - P(bjab
WT -15 122 1
db
V2T
(22)
1
+
W Re (t,) - 2/2
$ db
/ 22
1 e db +
/
000
(P,b - 1n+1 1 1
=
(-N,b- - 11
db
/2T
2
16

---

Under review as a conference paper at ICLR 2023
where N, Σ, Re and P, := E, W. Using the formula f(ar
0)²e-=01 /2 = 2l(a2 + eerf v22 ae -a" (2 - 2b) + C, where erf(x) = 20(xv2) - 1 is
an enror function, obtain:
V 221 (N] + (A5 (3rr
+ NNe (-N - 2 1
2m
V2
IV (P3 + (0) )9) er f(
Pe & (Pzz 2/0 |] %)/ 20
2m
(23)
V2 122 1
1
T (N3 + (1 (2) + v [
pp 11
2 [N3 + P3) + (0 V 2 1- (P3 N2)
1
Since } are independent of each other and E|N,] E|P3] EE we obtain:
E E(
E11 (N3 PJ) (n3
V 2
1
(P, (P,+NN) Nz)1
1 (F[N3) F(P3)) + [inn +1
v
12 /
Fl N2)]
2
π
2 (Var(())) + (ar((P)) + Var(u,+)) -0
(24)
2 202
Σ (e)U()) +o Σ RELUU(-())
0 Σ |t,12
+ 2x
= 2 1
2 2x
where or denotes the standard deviation of the weights. Let I = {i: x. > 0}. For E I+, we have:
n+1 3j 1+1
Var(g") = Var Σ 9 g W?)
n+1 35 n+1
Σ (f
ox WT)
7j n+1
Vvr(o (EE
(0a (Wz)²)
E
on W7y1)
( = + )
+ 1 E
( [
(25)
Σ Var(o) n (6
E(
Ello |2
2m
22 Σx Ro0z
+
2x
1
Σ
+ 2
12 Σn
The equality holds when 03 are identical for all j. Finally, we obtain:
17

---

Under review as a conference paper at ICLR 2023
E
o
Var(s)))
E 2
2π
Elotth&var((tt)) /
+ 2 Ex k
(26)
1 2T Σ, B
2 + 2 1h
π
π + 2
This proves the theorem.
B EXPERIMENTAL DETAILS
SGD with momentum 0.9 and weight decay with rate of 10 are used in all experiments. In the CIFARIO experiment, ResNet50 (He al. 20166) architecture is used, in which the first pooling layer and stride operation of the first convolution layer are removed to handle a smaller image size. The input normalized using the mean (0.4914, 0.4822, 0.4465) and std (0.2023, 0.1994, 0.2010). Random cropping with padding size of 4 is used and random horizontal flips are applied. When WarmUp (Goya et al., 2017b) is used, the learning rate linearly increased over the first 10 (CIFAR10) or two epochs (ImageNet). After that, cosine learning rate decay (Loshchilov & Hutter 2016) without restarts used.
In the ImageNet experiment, input is normalized using the mean (0.485, 0.456, 0,406) and std (0.229, 0.224, 0.225). RandomResizedCrop (He al. (2116) with an image size of 224 is used as augmentation. Then, random horizontal flip is applied. In the evaluation step, images resized to the size 256 and center-cropped to size 224.
In experiments with very large batch sizes, computational trick is used updates the parameter every n steps to increase the batch size by n times effectively. It is almost identical to using a hatch size that is times larger, except that it involves slightly higher sampling errors from normalization layers.
The CLARS algorithm requires per-sample gradient, which cannot be easily obtained in popular deep learning frameworks, such as PyTorch (Paszke et al. (2019) or Tensorflow (Abadi et al. (015)) Thus, module is borrowed to obtain the per-sample gradient from Opacus (Yousefpour et al. 2021], which was originally developed for differential privacy. This is the best option to the best of our knowledge, but it requires twice the time and memory.
Batch size
LARS
CLARS
LALC(OurS)
128
0.01
0.01
0.01
2048
0.001
0.001
0.01
4096
0.001
0.001
0.01
8192
0.001
0.0001
0.01
Table 1: Coefficient 7) used in experiment. A hyperparameter search in the order of 10 is per- formed.
18

---

Under review as a conference paper at ICLR 2023
for in group[" p::rams]:
Optimizer part. It can be any optimizer want. if p-grad is None:
continue
d_p p.grad.data
if weight_dece I= 0:
d_p.add (weight_decay, p.data)
if momentun I= 8:
param_ state self.state[p]
if "mommnntmn buffer' not param_sterl state:
param__staii state[ momentum buffer') ] torch.clome(d.)) dettach
buf param state[ momentum buffer'] buf (nomentum). dampening, d_p)
if nesterov:
p_p. p... buf)
else:
d_p buf
LALC part
w_norm torch. # L2 norm in default
e_norm torch.norm(d_p)
if w_nore 8_norm 8:
local_ @ocal_rr= = eta w_norm / (e_normkepsilon)
local_Ir = llcalllllllee det_._..cp(.ddtcc(()_a__a
Becomes very similar to LARS if this part "Ir=local_lr"rrupp 'Ir'}"
Ir min(loca__r,, group 'lr'])
else:
Ir group [rr'
Update the parameter p.data.add_(IIr d_p)
Figure 6: Py Torch (Paszke et al. 2019) implementation of LALC. It can easily be implemented by adding few lines of code to the existing optimizer.
Batch size
Method
128
2048
4096
8192
SGD
94.97
93.17
91.26
63.12
SGD+ WarmUp
95.35
93.7
91.66
67.67
Nesterov+ Up
95.19
93.63
89.19
60.24
LARS
94.32
93.34
92.92
87.5
LARS+WarmUp
94.84
93.21
93.62
92.98
CLARS
92.58
93.42
93.75
86.89
LALC(ours)
95.3
94.45
94.35
93.68
LALC+ WarmUp
95.37
94.77
94.14
93.71
LAI.C+ 95.22
94.67
94.39
93.71
Table 2: The validation accuracy in the CIFAR10 experiment (Figure 4).
19


Reviews:
[{"review_id": "txZRc1-Pb0Y", "reviewer": "Reviewer_dvc8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a new interpretations on why gradient explosion still can exist during the early training of normalized model. Their theoretical results can match the empirical observations to some extents. Based on their method, they propose a new optimization method, LALC, which can obtain good performance in both small batch and large batch training settings.", "summary_of_the_review": "Overall speaking, the result is novel and interesting especially the theoretical part. But the theoretical part and solution part has weak connection. I suggest the author to compensate the gap by combining the theorems in the missing literature. Besides, re-examine the experiment setting.", "strength_and_weaknesses": "Strength:\n\n1) The theoretical result is interesting and novel, and match the empirical observations to some extents;\n\n2) Experimental results prove the effectiveness of the proposed method;\n\nWeakness:\n\n1) As far as I concern, there is no apparent connection between the theoretical results (Sec 1-2) and proposed solution (Sec 3-4), which severely damage the integrity of the whole paper. I suggest the author to refer to a missing literature [1], in which another relationship between weights' norm and gradients' norm is derived. I think combining with the theoretical results in [1], the theoretical part can well support the reasonableness of the proposed method.\n\n2) It is wired figure 5 shows SGD+warmup totally failed on imagenet with 8K batch size. In [2], the performance of Resnet50 on imagenet with 8K batch only degrades slightly. Can author explain why?\n\n[1] Wan, Ruosi, et al. \"Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay.\" Neurips (2021): 6380-6391.\n\n[2] Goyal, Priya, et al. \"Accurate, large minibatch sgd: Training imagenet in 1 hour.\" arXiv preprint arXiv:1706.02677 (2017).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gNo1Nh5AIK", "reviewer": "Reviewer_eGDm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper shows that mathematically the gradient explosion of Relu + BN network approximately grows in the speed of \\sqrt{\\pi/(\\pi + 1)} w.r.t. to the layers. They also proposed a simple variant of LARS/LAMB that upperbounds the learning rate but not scale it. The authors conducted several experiments to show that the proposed method is effective. ", "summary_of_the_review": "Despite that this paper provides certain insights in terms of theoretical analysis of the \"gradient explosion\" pattern for RELU + BN, it is not stated clearly, and the novelty seems not being properly \"positioned\", in that 1) no clear reasoning leading to the empirical algorithm; 2) it's limited to a restricted scenario. ", "strength_and_weaknesses": "Strength:\n\n1. This paper contains a certain level of insights: it theoretically analyzes the instability property of BN + RELU, and reveals that the order of speed of gradient explosion (defined by Var(X_l)Var(g_l)/Var(x_{l+1})Var(g_{l+1}). Its mathematical proof looks accurate and rigorous. \n\n2. The proposed method is simple which is a good property for real-world applications, especially in industry level dataset/scenarios. Also given the simpleness, I'm in favor of its good reproducibility. \n\nWeakness:\n\n1. The written of this paper could be non-trivially improved, in particular the \"logic flow\" of it seems vague (detailed comments see below);\n\n2. The mathematical conclusion of this paper (as its main technical contribution) seems narrow in that a) it limits to BN + RELU + no residual connection; b) it seems not closely related with the derived algorithm. Again check below for the details. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IuBx3CqGxd", "reviewer": "Reviewer_EVtb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": " An important research question in neural computing is the interplay between Batch Normalization (BN) and learning. In this vein, this paper studies the influence of BN on gradient norm across the layers. BN causes an exponential growth with depth for the norm of a gradient in the first layer. When studying this phenomenon, the paper establishes theoretical and practical contributions: \n- *Theory.* It is claimed that theorem 1. establishes an exponential growth in the norm of the gradient\n- *Practice.\" Paper proposes a layer-wise stepsize clipping to compensate for the gradient expansion. The proposed method is effective in the classification of CIFAR10 and Imagenet datasets.", "summary_of_the_review": "In short, I have some concerns regarding the soundness of theoretical contributions, and also some questions regarding the novelty of the proposed algorithm. ", "strength_and_weaknesses": "**Strength**\n\nThe paper well-motivates the gradient explosion in presence of BN; then, it scientifically studies the cause of the explosion. The extensive references and linking of the results well to the literature are very well done. \n\n**Weakness** \n\nI have two main concerns about the theoretical and practical aspects of the results. \n- *Theoretical.* \n  - I am not sure the expectation in Thm.1 is taken w.r.t which random variables. What is the random variable in Eq. (3)? \n  - I have not checked the proof but suppose that Thm.1. is soundly proven.  I am not sure Thm 1. obtains exponential growth. The  Thm. 1 states $E[a_n/a_{n+1}]>1$ where $a_n$ is a factor of gradient variance in layer $n$. Although it is stated such lower-bound in expectation leads to an exponential growth in $a_n$, I am not sure about this. If there was no expectation, it would be possible to derive exponential growth. However, the expectation makes the recurrence complicated. Would the authors mind if I ask to elaborate on this? \n  - My experiments show the norm of the gradient in early layers (close to input) grows at an exponential rate with depth. While the paper suggests an exponential growth in the norm of the gradient in deep layers (close to outputs). \n- *Practical.* The proposed algorithm is very similar to adaptive gradient clipping proposed by Brock, A., De, S., Smith, S. L., and Simonyan, K. in a paper titled \"High-performance large-scale image\nrecognition without normalization\". \nWhat is the difference between the proposed method and the adaptive gradient clipping?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FwnEY1_eB6", "reviewer": "Reviewer_NA9i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the incompatibility of two very commonly found components of deep learning, namely the ReLU and Batch Normalization. This incompatibility is manifested e.g., in exploding gradients during the early part of training. The authors hypothesize that the source of the incompatibility is the fact that the variance of the activations and their gradients is affected differently by the combination of ReLU and Batch Normalization. The authors show this analytically, then explore it experimentally. Finally, they propose a solution to alleviate the problem.\n", "summary_of_the_review": "Decent work but key references missing.\n", "strength_and_weaknesses": "Strengths (S):\n\n----------------\n\nS: The studied problem of exploding gradients problem is a very important one in deep learning, expecially since it happens with the most typical combination of normalization and activation functions, namely ReLU and BN. \n\nS: The paper proposes a simple extension (LALC) to LARS optimizer that seems to fix at least partially the exploding gradients problem by clipping the learning rate. \n\nS: If LALC is shown to be better than LAMB and the key reference comment (see below) is correct, the paper could be revised to focus on LALC.\n\nWeaknesses (W):\n\n----------------\n\nW: The main weakness of the paper seems to be missing key references. Could the authors discuss how the analytical part of the work is different from this paper Yang, Greg, et al. \"A mean field theory of batch normalization.\" arXiv preprint arXiv:1902.08129 (2019). https://arxiv.org/abs/1902.08129 and this posting: https://kyleluther.github.io/2020/02/18/batchnorm-exploding-gradients.html. \n\nW: Regarding the clipping method (LALC), the paper seems to fail to cite Fong, Jeffrey, Siwei Chen, and Kaiqi Chen. \"Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clipping.\" arXiv preprint arXiv:2011.13584 (2020). https://arxiv.org/abs/2011.13584 , which also contains clipping. Please compare both analytically and experimentally to this method.\n\nW: The experimental part should contain trainings with LAMB, since in many cases it improves over LARS without expensive tuning of LAMB hyperparams.\n\nW: The results with LALC could be compared with the 3 proposed solutions in https://arxiv.org/abs/1902.08129\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject