POP-Norm: A Theoretically Justified and More Accelerated Normalization Approach
Authors: Hanyang Peng
Shiqi Yu
Venue:  (2020)
Keywords: Batch Normalization
Optimization
Accelerate Training

Abstract: Batch Normalization (BatchNorm) has been a default module in  modern deep networks  due to its effectiveness for accelerating training deep neural networks.  It is widely accepted that the great success of BatchNorm is owing to reduction of internal covariate shift (ICS), but recently it is demonstrated that the link between them is fairly weak. The intrinsic reason behind effectiveness of BatchNorm is still unrevealed that limits it to be made better use. In light of this,  we propose a new normalization approach,  referred to as Pre-Operation Normalization (POP-Norm), which is theoretically ensured to speed up the training convergence. Not surprisingly, POP-Norm and BatchNorm are largely the same. Hence the similarities  can help us to theoretically interpret the root of BatchNorm's effectiveness. There are still some significant distinctions between the two approaches. Just the distinctions make POP-Norm achieve faster convergence rate and better performance than BatchNorm, which are validated in extensive experiments on benchmark datasets: CIFAR10, CIFAR100 and ILSVRC2012.

Under review as a conference paper at ICLR 2020
POP-NORM: A THEORETICALLY JUSTIFIED AND MORE ACCELERATED NORMALIZATION APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
Batch Normalization (BatchNorm) has been a default module in modern deep net- works due to its effectiveness for accelerating training deep neural networks. It is widely accepted that the great success of BatchNorm is owing to reduction of internal covariate shift (ICS), but recently it is demonstrated that the link between them is fairly weak. The intrinsic reason behind the effectiveness of BatchNorm is still unrevealed that limits it to be made better use. In light of this, we pro- pose new normalization approach, referred to Pre-Operation Normalization (POP-Norm), which is theoretically ensured to speed up the training convergence. Not surprisingly, POP-Norm and BatchNorm are largely the same. Hence the similarities can help us to theoretically interpret the root of BatchNorm's 's effec- tiveness. There still some significant distinctions between the two approaches. Just the distinctions make POP-Norm achieve faster convergence rate and better performance than BatchNorm, which are validated in extensive experiments on benchmark datasets: CIFAR1O, CIFAR 100 and ILSVRC2012.
INTRODUCTION
The last decade has witnessed significant advances in deep neural networks (DNN), which brought substantial improvements for many real-world tasks, such (Sject(Szegedy al. 2015 He al.
2015), scene et al.) 2016 Guo et al. 2017) and action (roggnitoo((Fichtenhhe
et al. [017]]
object detection Ren et al. [2015] He et al. 2017b; Reemmon & Farhadi] (2018) and image segmenta- (onnneheerrr et al., 2015 He et al.) 7a). BatchNorm (loffe & Szegedy 2015) is milestone DNN, and nowadays it has become a default component to con- struct modern deep networks. BatchNorm is widely-used in DNN due to that it is able to reduce the sensitivity to initialization, significantly raise the learning rate, substantially speed up the train- ing process and considerably improve the performance. Actually, along with this line variety of works, such as layer normalization (Ba et al., 2016), instance normalization (Ulyanov et al. 2016), weight caaimalizationnSaliaann & Kingma 2016) and group normalization (Wu & He 2018). batch renormalization, (Toffe 2017) were sequently proposed and also have made great success.
|
technique in the development
Nowadays, the practical success of BatchNorm is mainly attributed to reduction of internal covari- ate shift (ICS) by controlling the first two moments (mean and variance) of the distributions of layer inputs. Recently, this point of view challenged by (Santurkar et al., 2018). It points out that the link between reduction of ICS and performance improvement of BatchNorm is tenuous, at best. However, the success of BatchNorm indisputable, therefore there may exist a profound mathe- matical mechanism behind BatchNorm. In fact, from the perspective of loss landscape, (Santurkar al. 2018) demonstrates BatchNorm will make the optimization landscape smoother, and then this smoothness induces more predictive and stable behavior of the gradients.
As stated above, the original theoretical foundation of the vanilla BatchNorm not so solid and the exact reason for its effectiveness is poorly understood, hence BatchNorm maybe not optimal in the- ory and we might not make better use of it. In light of this, we try to propose a theoretically justified normalization approach for accelerating training deep neural networks, so that we can continuously further algorithmic progress. To achieve this goal, we firstly demonstrate that lowering gradient Lipschitz constant and gradient variance is the key to boost the convergence rate of optimization algorithms with stochastic gradient methods. And then we construct a normalization approach - removing the mean of inputs and dividing by the scaled l norm of inputs before conducting the

---

Under review as a conference paper at ICLR 2020
main operation (convolution inner- (iner-production) of layer, which can be theoretically ensured to reduce the gradient Lipschitz constant and gradient variance.
The new normalization approach and BatchNorm similar. Hence utilize their similarity to interpret the effectiveness of the vanilla BatchNorm, along the line of lowering gradient Lipschitz constant and gradient variance, which is different from the view of the loss landscape (Santurkar et al. 2018). According to our theory, we can easily explain why BatchNorm with ReLU is powerful with but ineffective with Sigmoid and Tanh activation functions, which may help better understand the underlying complexity of neural networks.
The main contributions of this paper are summarized follows:
From scratch a theoretically justified normalization approach is deduced for accelerating the convergence speed of training deep neural networks.
We theoretically explain the root of BatchNorm's effectiveness and the weakness from the perspective of lowering the gradient variance and gradient Lipschitz constant.
With the help of the new normalization approach, extensive experiments the performance for Sigmoid and Tanh achieve massive improvement, and they are even competitive to ReLU.
Our paper is organised follows. In Section stochastic optimization. Then, built on the premise, we will propose a new normalization approach in Section In Section 4, we will analyze the exact 5, we will implement extensive experiments to demonstrate the superiorly of the new normalization approach. We will summarize our work and discuss further work in Section
will deduce the theoretical premises accelerate
of BatchNorm's effectiveness. Section
THEORETICAL PREMISES
In this section, we will theoretically deduce that reduction of gradient Lipschitz constant and gradi- ent variance plays a vitally important role in convergence when applying stochastic gradient descen- ((SGD) algorithms for machine learning tasks.
In machine learning task, given a set of samples { the empirical risk as the average of all the samples loss is typically treated as the optimizing objective, i.e.,
1 n Σ (w;z)).
F(w) = F (w; {a:)==))
(I)
where fx( is the loss incurred by the parameter vector w with respect the i-th sample or mini-batch. Noth that sometimes F (w:; (x:b) and f((w;,;) may be abbreviated as F(w) and f((w))
In scage machine learning tasks, such as deep learning, we usually choose SGD to minimize the empirical risk Eq. (T) since its computational cost is cheap. Applying SGD, the parameter
update at k-th iteration is
Wk+1 + wt 0-++1 V fez (wys Tic)
(2)
where the k-tth iteration Fi, is randomly chosen from the full sets [a and ac+1 is the learning rate.
Convergence guarantee of most gradient-basee based algorithms including SGD is built upon the assump- tion of Lipschitzz-contiuuuu objective gradients, which is state as follow.
The function F(w) is continuously differentiable and the gradient function of F(w), namely V F(c),, is Lipschitz continuous with gradient Lipschitz constant T, > i.e.,
VF(w))
(F(w))) & L||w] - w2l1-
(3)
An important equality directly induced from Eq.
F(wz) < F(w1) + (V (w1). w2 - w1) + L
= 1| - will2
(4)
2

---

Under review as a conference paper at ICLR 2020
The proof for it can be found in (Neterrvv 2004 Bottou et al. 2016).
Under the assumption in Eq. we provide convergence analysis in the following for the general objective function Eq. when applying SGD, no matter whether it is convex not.
Theorem Suppose the assumption in Eq. 4 is satisfied and the gradient of F(wa) in Eq. is bounded (i.e., FF()s) M oc), SGD algorithm is applied optimize F(wa).
(1) If the leaning rate satisfies Σ 1=1 Ok 00 and Ex=ı of < oc, and then
Ek=o 04 04+1 F
F( Fint + ax+1
1/2 5E 44 BI|
V (wx)
K K+000
0,
K Ek=o a++1
Ex=o
K
Σk= ax+1
(5)
where Finy min F(w). Then, [[F((w))] K-00 in probability.
(2) If the leaning rate satisfies L 2 L and Σk=1 OKk 00, we have
ΣK 2k= O E [IIIFF B[F(wo)) Fing Ex=o ox++ C ViV (wre)
(6)
Ek=O 0x++1
Ek=0 06++1
Σ ak+1
where V(V Sin (wx)] = BlIV Su (wx) V lix (wx)]l that the gradient variance of Súg (wx).
When K- + 0C.
inf (BIIVF(*>)]11) s sup (wx)))
k
According to the first conclusion of Theorem when the learning rate satisfies k=1 co and
Σx=1 oz < oo, we know that SGD algorithm will guarantee the empirical risk function in Eq. { converges stationary point in probability. The conditions in the second conclusion of Theorem 1 is casier to be met. In this case we should make the the gradient variance V(V fi, (wx) as smallest as possible ensure the empirical risk function in Eq. ①1) can converge to good point.
Theorem also implies that increasing learning rate will directly speed up convergence. Since larger B[F(wo)] FFnn in Eq.
Og will make the term E[F(wo)]- Fent in Eq. (5) (or Σk
(6) faster to diminish.
3 ΣR
However, when the learning rate becomes large, the term >E QE (w=))) in Eq. 3 (or ΣK 0K+1 ViVSi, (wx)) in Eq. (6) may also become large and the condition
2 may not be met. Hence reducing the gradient variance ViV (wrx). and the value of Lipschitz constant T. is the key to increase the learning rate and accelerate the convergence. It is worth noting that the conclusion that gradient variance reduction is beneficial accelerate convergence has been demonstrated in (Johnson & Zhang 2013) when the objective function is convex.
Therefore, if normalization approach effective make Eq. T converge a good point and accelerate the convergence process, it needs to reduce gradient variance V(V fi, (wx)) and gradient Lipsehitz constant L.
THE PROPOSED APPROACH
In this section, we will present new normalization approach of which each step will be theoretically proved to be beneficial to reduce the gradient variance V V/V Sú, (wx)] and the gradient Lipschitz constant simultancously. therefore the new approach will accelerate the convergence of training and improve the performance.
IIt is known V/ (wx) = VIV (wx)) + (B[I S4 (we) 2 decreasing V(V fa. (wx) will also lower E||| fi, [.
3

---

Under review as a conference paper at ICLR 2020
When applying mini-batch SGD, at the k-th iteration the empirical risk loss with respect to the weight/bias the I-th conneccd layer of DNN can be expressed as
(im)(ukk (1)
1 m
Ji (w 11 win. 1 - E9
(10h
(1) wk
+ 0/%
G(m) ((x(m)))
w/ + B (1)
(m))
(8)
where m is the size of mini-batch, and x X(mm)e = [ot, 11 22
(1-1) D1 ;... amm mm and B00
=
192 22.. bingl Fiz (1-1) is randomly choose from the entire train set f (= li-1 with e- qual probability, hence F(wgg)
, ii ( is the expectation of F(n)) (wr) ( (LF(m) F(w()), where n the total number of train set. Note that Eq. also implies that F(w)) = G ((X) For clarity, henceforth we will omit the layer index.
, Ex= fin (ut&
(
+ BB) where x |,,, 222, 1..99 i...5 2 and B 11 Ub
1-1
Ungll
We construct a new normalization approach, referred to as Pre-Operation Normalization (POP- Norm), of which one main characteristic is putting the normalization technique before operation. For a layer fed with a d-dimensional input Ti, = (2 Vi, "......,,,, (..)) we ideally normalize each dimension with the global estimation of entire samples, Le.,
[2]
n
T 4/
17
(Mean Removal)
(9)
1/
1=1
s/1
a
[i]
Dig.
pl v
51 1x
lla
(Division by sclaed l2 Norm)
(10)
plll
1 V
where 0 S 1pbl 2, pbl
2 1 and m/61 = (aff)
U1 22
The normalization approach consists of two parts mean removal and division by le norm. The both steps are actually not interdependent and can individually take effects for decreasing gradient variance and gradient Lipschitz constant. We will theoretically demonstrate it in the following.
Theorem 2 (Justification of Mean Removal in Eq. Eq. O is adopted to handle each input
Tiy of the model in Eq. (8) Suppose assumption in Eq. (3) is satisfied and the gradient and Hessian
function of g(2) and G(z) in Eq. 8 is bounded, i.e. [9((2)lz M1 < oo and [ S
M2 < oc, we have
(1) the upper bound of the gradient variance VVV fue aw)] will be lower than that of Ji, x.))
(2) the upper bound of the minimal gradient Lipschitz constant of F (wwk (Ti, ) will be lower than that of (wys (2ia Mmm);
Theorem (Justification of Division by l2 Norm in Eq.(0)) Eq. (T) is adopted to handle each input dig from Eq. (.. Suppose assumption in Eq. (3) is satisfied and the gradient and Hessian
function of g(z) and G(z) in Eq. { is bounded, Lee, Vg(2)]l2 S Mi < xc and ||>2(=))]| S
M2 < Cc. we have
(1) the upper bound of the gradient variance VV (wi, f.)) will be lower than that of VV ;;))
(2) the upper bound of the minimal gradient Lipschitz constant of F (wni (Fig Fir) will be lower than that of F (was Ya.)
(3) if (w)) is locally convex and max, maxx((lli12)) maxy (E (EEn (eli)))) (ming(pld1y)
miny (Ei (E (2l()aa) the lower bound of conditional
number of 2 Fww: (f Hir) will be lower than that of 2 (wk: (Ti, }im) in the local space.
2 A convolution layer can be also reformulated to he matrix form hence the analysis for convolution layers is similar, but it is still sotewhat complicated and discuss it in the appendix.
4

---

Under review as a conference paper at ICLR 2020
BetchNorm
POP-Norm
Mozn Nemovel
Main Operation
Operation
Divison
Act. Fume
Shift
Division
Scale
Act Funs
(h)
Figure 1: (a) The structure of standard layer with BatchNorm. (b) The structure of layer with POP-Norm.
Due to the limited space, the proofs of Theorem 2 and Theorem 3 are provided in the appendix.
The computation of Σn Fig and Σi= 2 in Eq. 9 10 over the entire training set is im- practical when adopting SGD. Therefore, just like the vanilla BatchNorm, each mini-batch produces L -> Tig and n0 2im1 x², the estimates instead. This implementation will also make all the inputs fully participate in the gradient pack-propagation The vanilla BatchNorm introduces the s- cale and shift parameter to be learned restore the presentation power of the network. To make POP-Norm more like the vanilla BatchNorm, factorize rf5ر) and pbl into fixed parameter and parameter to be learned. In practice POP-Norm is presented as follows:
m
+ +,31,,
(Mean Removal)
(11)
111
y/1
15l
(Division by scaled l2 Norm)
(12)
i=1
where (ااو and 7ا6 are the shift parameter and the scale parameter to be learned, and (2 i)
hyperparameter. If ||851] 2 Σ" oi and |لار| my and Theorem 3 will still hold.
E
(5
bi
satisfied, Theorem
12 are similar to the vanil- la BatchNorm, they still slight different, shown in Figure For BatchNorm, abll gاا (a اهب 1 + +1bi)//// a Em (a ) where μ = Em 2 while for POP-Norm, a 4g
Remark 1. Although the main steps of POP-Norm in Eq. 11
اi (=11 - اار + s/1))// Em (a - nb + =b)) Besides, there are other two significant differ- ences between the proposed POP-Norm and the vanilla BatchNorm placing normalization before the main operation(convolluion and inner- production) and adding new scale parameter к. Actu- ally, we deliherately make POP-Norm more like the the vanilla BatchNorm. POP-Norm has other
forms, for example, mean removal Eq.(11 can be a = 1+. Σi= 2 and the norm to divide can be substituted by the other norm, such as l norm and 1-x nrom. Mean removal and di- vision by le norm of POP-Norm are actually not interdependent and can individually take effects for decreasing gradient variance and gradient Lipschitz constant, and their order can be also exchanged. From the proof Theorem 2 and Theorem 3, we also know the division by l2 norm will lowering the upper bound of V(V f., (wx)) and L more substantially than mean removal. Additionally, the theoretical analysis and construction of POP-Norm are generalized, and we can also apply them normalization approach to compete with instance normalization (Ulyanov al.
m
to construct 2016) and group normalization (Wu & He 2018).
Remark 2. Adding new parameter and setting > is more beneficial decrease the gradient
variance and the gradient variance V (w*)] and the gradient Lipschitz constant L. When look- ing into the proof of Theorem and Theorem 3, can conclude the upper bound of V[V fú, (wx) and L will be proportional to 2. which means the larger λ will be quickly smaller V/V f., (wx)] and L so that speeds up the convergence. However, & can not too large, otherwise the value after the operation will be much small, and then it may harm the representation capability of the layer,
5

---

Under review as a conference paper at ICLR 2020
since the non-linear activation function will mainly work in the linear regime. Moreover, when is too large that will directly make gradient variance much small, and then it is difficult for train loss to escape the trap of early local minimums. In our practices, when is moderately large, it will substantially improve the performance, which is demonstrated in the experimental section. Not strictly speaking, adding k: is just make f(w) change to s(W). It seems the optimization trajectory will be the same if we magnify the initial value of w by an exact factor x and scale the corresponding learning rate However, we do not exactly change the initial value and the learning rate, and the optimum is not unique. On the contrary, the number of local minimums of deep neural network is much large and they scatter throughout all the space, Therefore, adding in the network quickly converge to local minimum near the initial value according to Theorem 3 rather than along with a far trajectory converge to the original local minimum.
more likely to help w
As proved in Theorem 3, adopting the POP-Norm will be conducive to reduction of the condition number of 2 F(wx) if F(wrs) locally convex. Smaller condition number will make the training algorithm avoid the danger of running into sudden change of the loss such as flat region or a steep slope, which also enable us to adopt a larger learning rate. In other words, the smaller condition number will make the landscape or convergence trajectory of the loss is more smooth and substantially speed up the training process.
Remark 3.
4 JUSTIFYING EFFECTIVENES, AND WEAKNESS OF BATCHNORM
The vanilla BatchNorm has been a standard toolkit for modern deep networks, but the root of Batch- Norm's effectiveness is still vague. The motivation of BatchNorm built on ICS is somewhat heuris- tic. Actually. in (Santurkar et al. 2018) it demonstrates the link between the performance gain and reduction of ICS is fairly weak, which means the success of BatchNorm may be not owing to ICS. (Santurkar et al.] 2018) owes faster training with BatchNorm to the improvement of loss landscape. In this section, will analyze not only BatchNorm's is effectiveness but also its weakness from the perspective of lowering the gradient variance and gradient Lipschitz constant above.
BatchNorm and POP-Norm similar, but there still some significant distinctions between BatchNorm and POPNorm. One of them is putting the normalization before the main operation (convolution and inner-production) for POPNorm. As analyzed in the above subsection, each step POP-Norm has been theoretically justified. Therefore, we can analyze the influence of differences between BatchNorm and POPNorm to verify the effectiveness and weakness of BatchNorm.
BatchNorm is placed behind the operation of convolution or inner-production and before the non- linear activation function which is displayed in Figure 1, hence BatchNorm actually takes effect on the operation of the next layer. For simplicity, we omit the shift and scale parameter here. Batch- Norm and POP-norm can be expressed as:
(13)
44
71
1=1
(14)
|لارار 2
1
Σn
h(al)))
(15)
where h(-) is a activation function.
Due to nonlinearity of the activation function, the benefits of normalization will be impaired. When it is ReL.U, the damage will be not so serious. Suppose all samples { ) are symmetric about zero that can be easy to be satisfied in practice, we have
* bi 4 = i
= /(
2 Σ h2 (1
(16)
hence when adopting ReLU as the activation function for BatchNorm, is equal to taking the step division by la norm with respect to h(=bl) for Porm. In other words, no matter we put nor- malization before or behind ReI.U, division by 12 norm will be effective. From Remark 2, we know

---

Under review as a conference paper at ICLR 2020
in normalization mean removal and division by norm can take effects individually and division by l2 norm plays a major role and can still substantially reduce the gradient variance and gradient Lipschitz constant. Therefore, the effectivness of BatchNorm with ReLU will be still guaranteed. On the contrary, when the activation function is Tanh, it will damage division by 22 norm and reserve mean removal, but the performance of BatchNorm with Tanh will be still undermined. When the activation function is Sigmoid, it will destroy both of mean removal and division by la norm, hence performance of BatchNorm with Sigmoid will be seriously deteriorated.
As we analyze in Remark 2, moderately increase the scale of the l2 norm will be beneficial to decrease the gradient variance and gradient Lipschitz constant. It is known most of activation func- tions will shrink the value of inputs, and then the value of normalization will become relatively larger, hence from this perspective of view, activation function may beneficial, but the deteri- oration brought by distortion of activation functions may be still dominated for BatchNorm with Sigmoid, but BatchNorm with ReLU will be benefited from the declined value of imputs.
5
EXPERIMENTS
5.1 GENERAL SETTINGS
In the experiments, we first modify each layer in Figure ((a) to Figure 1(b) for widely-used VGG- nets (Karen Simonyan 2016) and ResNets (He et al. 2016) with POP-Norm, and all other network architecture and parameters were exactly the same with the standard VGG-nets and ResNets, and then we evaluate the standard VGG-nets and ResNets and the new ones on benchmark dataset- s: CIFRRIO, CIFARIO0 and ILSVRC2012. It is worth noting that we just want to evaluate the performance and effectiveness of the proposed POPNorm, compared with BatchNorm, rather than pursue the highest accuracy on benchmark datasets, hence we do not adopt much deep networks and complicated tricks.
For the experiments on the datasets CIFARIO and CIFARI00, the batch size is set as 128. We use a weight of 0.0005 and SGI) with momentum of 0.9. To simplify the tuning process and compare fairly, we identically start with a learning rate of 0.1, divided by 10 at 16k and 24k iterations, and finally terminate at 30k iterations. For the experiments on ILSVRC2012, standard SGD with weight decay of 0.0001 and momentum of 0.9 is adopted. the number of samples in batch size is set as 256 uniformly distributed on 8 GPUs The learning rate starts from 0.05 and is divided by 10 at 250k and 450k and 550k iterations, and finally terminate at 600k iterations. The images are first simply resized to 256 x 256, and then randomly cropped to 224 x 224. No other data augmentation except simple horizontal image flipping is employed.
5.2 PARAMETER SENSITIVITY ANALYSIS
We add a scaling factor k to magnify the value of normalization for POP-Norm. k of POP-Norm will greatly influence the effectiveness of POP-Norm. In this subsection, will evaluate the perfor- mance of VGG16 Embedding POP-Norm with different values of scaling factor k from the candidate set of {1, 4, 16).
As shown in Figure 2(a-c), when k is moderately large (k = 4), it is not only beneficial to speed up the training but also help train loss to finally converge to smaller value. However, if we further increase k to 16, although the relative convergence speed is accelerated, the training loss is more like converge
larger local minimum. The reason for it is just what we analyzed in Remark 2 of Section 3. Overlarge will make activation functions (Tanh and Sigmoid), work in the linear regime, which will further harm the representative ability of neural networks, so that is difficult for training loss to find better local minimum. Overlarge x will directly lead to small gradient variance, which will make train loss hard to escape the trap of an early local minimum. k in POP-Norm will be uniformly set to 4 in the following experiments for better performance.
POP-Norm speed up the convergence through reducing gradient Lipschitz constant (increasing learning rate) and lowering gradient variance. We fix the learning rate in the experiments in this subsection, hence it also experimentally validates reduction of gradient variance is conducive to faster training convergence.
7

---

Under review as a conference paper at ICLR 2020
Fonseer 14 31
Por-iom X Poe-sem
Porstorm Ic 31
PoP-tom (c
That Lama
Thin Lone
The
23000
Number Itanatinn
Numbbr tarattona
Namber tanationa
(a) Sigmoid
(b) Tanh
(c) ReLU
Figure 2: Train Loss for POP-Norm embedded VGG16 with different activation functions on CIFAR100: (a)
Sigmoid, (b) Tanh and (c) ReLU.
Baith Seri
Un Neit
No torm
Datchtorm
Ace sarn joaly Oaiiae Koh-tern
Fron savrr (2illy Nustoee
Tram Lose
Mamber
(a) Sigmoid
(b) Tanh
(c) ReLU
Figure 3: Train Loss for different normalization approaches embedded VGG16 with different activation func- tions on CIFAR 100: (a) Sigmoid, (b) Tanh and (c) ReI.U.
5.3 ABLATION EXPERIMENTS
As analyzed in Remark I in Section 3, mean removal and division by 12-norm of POP-Norm can work individually, in this section we will evaluate their performance of VGG16 on CIFAR100 in- dividually and collectively. To better access their performance, we also add VGG with no nor- malization and BatchNorm as anchors. We set the initial learning rate 0.00005 and 0.005 for no normalization and POP-norm only with mean removal since a higher learning rate will make train- ing diverge. Specifically, when the activation function is Sigmoid, no matter what value the initial learning rate is. no normalization, POP-norm only with mean removal and POP-norm only with division cannot make loss decrease. This phenomenon for no normalization was also reported in (loffe & Szegedy 2015). We guess this may be owing gradient vanishing, and the mean removal and division together can relief it but each individual component can not. We will explore the exact reason in the future.
As shown in Figure 3, comparing with no normalization, mean removal and division by 1n-norm works can individually accelerate convergence mainly due increasing learning rate, and it also implies that division by 12-norm is more influential than mean removal in POP-Norm, which can validate our analysis in Section 3. Moreover, when adopting ReLU as activation function, the per- formance of BatchNorm is close to POP-Norm only with division. And when adopting Tanh and Sigmoid, the convergence rate BatchNorm much slower than POP-Norm. These phenomenons just follow the analysis in Section 4.
5.4 CLASSIFICATION COMPARISON
In this subsection, we will access the classification performance of BatchNorm and POP-Norm on widely-used CIFAR10, CIFARI00 and ILSVRC2012. The results are reported in Table [
As displayed in Table D the classification accuracy for POP-Norm is consistently higher ( (average more than 3%) than Narch Specifically, when the activation functions are Sigmoid and Tanh,
8

---

Under review as a conference paper at ICLR 2020
the improvement of POP-Norm is more obvious, gaining about 8% and 4% respectively, which demonstrates the effectiveness of POP-Norm. It is worth noting that with the help of the proposed POPNorm the performance of the networks with Sigmoid and Tanh is fairly close to the networks with ReLU, which also validates the power of the proposed approach.
Comparing to BatchNorm with ReLU, the performance for BatchNorm with Sigmoid and Tanh is heavily discounted. The reason for it is what we analyzed in Section 4. When the activation func- tion is ReLU, placing BatchNorm behind convolution and inner-production and before activation functions will less influence the performance of normalization, but when adopting Sigmoid or Tanh, this position will have a great impact for the performance of normalization. Therefore, from another aspect. it verifies the importance of putting normalization before convolution and inner- umerrproddction.
Table I: Mean test accuracy 96) and standard deviation of 10 trials for BatchNorm and POPNorn embedded VGGI6 and ResNer20 on CIFARIO and CIFAR100.
CIFARIO
CIFAR100
Model
Act. Fun.
BatchNorm
POP-Norm
BatchNorm
POP-Norm
Sigmoid
84.21 + 0.46
90.69 + 0.15
62.49 + 0.56
70.29 + 0.14
VGG16
Tanh
88.69 + 0.15
90.63 + 0.09
66.63 = 0.12
70.59 + 0.09
ReLU
90.89 1 0.13
92.15 + 0.11
70.83 1 0.25
71.38 + 0.12
Sigmoid
83.08 + 0.14
89.68 + 0.14
54.10 + 0.37
66.91 + 0.32
ResNet20
Tanh
88.08 0.11
89.63 + 0.09
62.42 + 0.36
67.02 + 0.10
ReLU
90.23 0.13
90.90 0.12
66.70 0.16
67.53 + 0.12
We also implement classification experiments on ILSVRC2012. As shown in Tabel 2 POP-Norm achieves about 0.8% higher classification accuracy than BatchNorm, which demonstrates POP- Norm is also effective in large-scale datasets. Due to the limited time, we did not utilize deeper networks to evaluate BatchNorm and POP-Norm. We will report more results of deeper networks in future.
Table 2: Test accuracy(%) for BatchNorm and POP- Nor- embedded ResNet18 on ILSVRC2012. The activa- tion function is ReLU.
BatchNorm
POP-Norm
Model
top-1
top-5
top-1
top-5
ResNet18
66.81
87.32
67.53
87.95
CONCLUSIONS AND DISCUSSIONS
In this paper we try to propose theoretically justified and effective normalization approach. To achieve this goal, we first deduce that lowing gradient variance and gradient Lipschitz constant is the key to speed up the convergence when adopting SGD. And then we present a new normalization approach, of which each step theoretically proven to be beneficial reduce gradient variance and gradient Lipschitz constant. POP-Norm and BatchNorm are largely the same, which just help us to explain the effectiveness of BatchNorm. Besides the form of POP-Norm slightly different from BatchNorm, there are two other differences between the two normalization methods. One difference that the proposed method place the normalization ahead the main operation of a layer rather than behind it. Another is that the proposed method adds scaling factor to increase the value of normalization. Just the differences make our normalization approach more powerful, which is then benchmark datasets. Moreover, POP-Norm
validated by extensive experiments forms that are much different from BatchNorm, and the principle of POP-Norm can be also applied instance normalization and group normalization.
have other
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E, Hinton. Layer normalization. arXiv preprint: 1607.064500 06450v1 2016.

---

Under review as a conference paper at ICLR 2020
Lon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. arXiv preprint: 04838, 2016.
Christoph Feichtenhofer, Axel Pinz, and Richard P. Wildes. Spatiotemporal multiplier networks for video action recognition. In JEEE Conference on Computer Vision and Pattern Recognition, pp. 7445-7454 7454, 2017.
Sheng Guo, Weilin Huang, Limin Wang, and Qiao Yu. Locally supervised deep hybrid model for scene recog- nition. IEEE Transactions on Image Processing, 26(2):808 820, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In IEEE Conference on Computer Vision and Puttern Recog- nition, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-can. IEEE Transactions Pattern Analysis and Machine Intelligence, P1, 2017a.
Wenhao He, Xu Yao Zhang, Fei Yin, and Cheng Lin Liu. Deep direct regression for multi-oriented scene text detection. In IEEE International Conference on Computer Vision, pp. 745-753, 2017b.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in noah-nnomalized models. In Advances Neural Information Processing Systems, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on International Conference on Machine Learning, 2015.
R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in Neural Information Processing Systems, 2013.
Andrew Zisserman Karen Simonyan. Very deep convolutional networks for large-scale image recognition. arXiv preprint: 1409.1556, 2016.
Khan, M Hayat, M Bennamoun, Sohel, and R Togneri. discriminative representation convolutional features for indoor scene recognition. IEEE Transactions on Image Processing, 25(7):3372-3383, 2016.
Yurii Nesterov. Introductory lectures convex optimization: A hasic course. Springer Science Business Media, I.C,, 2004.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint: 1804.02767, 2018
S. Ren, K. He, R Girshick, and J. Sun. Faster r-can: Towards real-time object detection with region proposal networks, IEEE Trans Pattern Anal Mach Intell, 39(6): 137 1149, 2015.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisee Interven- tion, pp. 234-241, 2015.
Tim Salimans and Diederik P. Kingma. Weight normalization simple reparameterization accelerate training deep neural networks. Advances in Neural Information Processing Systems, 2016.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? (no, it is not about internal covariate shift). In Advances in Neural Information Processing Systems, 2018.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-9, 2015.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint: 1607.08022, 2016.
Yuxin Wu and Kaiming He. Group normalization. In European Conference on Computer Vision, 2018.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
A. POP-NORM FOR CONVOLUTION
When the layer is convolutional layer, the matrix equivalence of sample with respect to the learning parameter wp is
[1.1]
0
0
0
[1.1]
0
[4
[1,1]
[1.2]
[d.1]
[d.-
2 [d.rl
Fia
[1.2]
[1.3]
[1,++]]
[1.2]
drrl
[d.++1]
[1.--1] 1]
[1.4]]
[1,8-1) 1] 0
0
0
[1,8]
dd.a]
20
9/
0
0
0
(17)
where the number of channels is d, the number of spatial dimensions is 8, the size of kernel is To make the expression legible, we present the matrix equivalence of input with one-dimensional spatial locations. Actually, when and 1, the convolution layer is degraded to fully-connected layer.
Theorem 2 and Theorem 3 require each row vector of The in a convolutional should be considered as a atom that is like a sample in a fully-connected layer, so POP-Norm for convolution layer is
1,61)
T
biel b4
(Mean Removal)
(18)
128
i=1
9=1
3:
3/
(Division by sclaed Norm)
(19)
pli)
1
Σn
Ei=
(zlinl)
ns
B. PROOF OF THEOREM
Proof (1) At the k-th iteration, we have
L wxll 11.
F(WN+1) S F(wx) + (VF(Ww); wk+1 wk)
|
(20)
2
We know wh+1 = Wi 0.+1 V (wx) in Eq. so Eq. 20 can be formulated as
La
F(WR+1)+ & F(wx) - a-- (F(w)) (wr) +
(21)
2
Since each iteration the sample mini-batch is randomly chose from the full set, V fix (wx) is an unbias estimate of F(w), i.e. f. (w)] = FF(w). Hence taking expectation on the k-th sample or mini-batch, we have
E[F( F(wx) - au [V + La E
B fr. (wx))..
(22)
Rearranging terms yields
01 0'+1 [V+W)] < F(wx) [[[((+++]]] + La
E V (wx)
(23)
Taking the total expectation and summing over 0 to K, obtain
K
au V F(F(w))] -E(F(K++1)) +
Σ
a
(24)
k=0
11

---

Under review as a conference paper at ICLR 2020
Dividing both sides of Eq. 24 with Σx=o ak+1 and combining Fin F(wk). we have
E 0k++ VF
[[ Pint + 1/2 EE a BIll V (wx)
(25)
Ek-oak+1 ak+1
Σk= ax+1
Σk=o ax+1
Since (F(w) is bounded, V fic (w) will be bounded, and then E fie (wx) |l should be also
bounded. known Ek=1 Or = oo and E of < co, so when K tends to infinity, we abtain
Σ ak+1 VF K 0.
(26)
0 02+1
It is known that Ek=1 Og 0o, hence for any > 0, there should exists number N N. so only sparse numbers k > N satisfies that EE] V F ((wx))l > otherwise when K - 0c.
DK 041131111 (( + do that contradicts Eq. 26). Therefore almost everywhere
Ex-o ax41
B[][FF(W;]11) K-000 0.
(27)
And then for any Ç <>0 O,
E V K-000 0
P(IVF((:)]]) = c2
(28)
C²
where the inequality holds due to Markov's Inqquality, which means ability.
||FF(w)]]
K+20 prob
(2) Since
2 L and Eq. 22 holds, we have
/V+W// 0k+11
E ((F(w++1)) S F(wk) a441111/111
(29)
It is known fi,, ( [/
so Eq. (29) can be reformulated as:
Lo
ak++ [V < F(wx) E[F(w++1)) +
Vi. (wx))l
(30)
2
Taking the total expectation and summing over O to K, we have
K
K
B E[F((K([W)))
a++
-EF
ak+1
w fu (wx)ll]-
(31)
2
k=
k=
Dividing both sides of Eq.
31 with
Σ
K k= ax+1 and combining Fins S F(W)), we have
Σx a2+
[[F((w)] - Fing + Ek a++1 V (wx)))
<
(32)
Ekz 01++1
+ ax+1
E ax+1
Exe = co. hence when K + 00 we have [[F(wo)] -F(w) + O. We know
It is known
$ Σi 0:11
E|ll F(ww)l1 > infa (B())VF(()x)(11) and VIll fi, (ww)ll s sup, (VIlV (wx)ll), and then
we obtain
inf (F[[)F((m))1))) sup (VIIV (wx)]l)),
(33)
&
and finally we arrive at the desired result.
12

---

Under review as a conference paper at ICLR 2020
C. PROOF OF THEOREM 2
Proof 2 (I). From Eq. we know fie (Ww: ((Fre) + bb), and then
Via, (1n3 Tiin Vg (Irs,) Yun + bk),
(34)
hence
V [Il (ww) (. III =E [1 V fix (wx: F/a ) (E |I| fue (wxx III)
E V fix (wxt 3.a )
[1 Vg (zxz)) "uk + bx) [1
(35)
E2 [1] |1 V ((Tx,) (oxx,)) + bk) [|a]
S MM E|| 1]
where the second inequality holds due to the fact ||AB||2 || and the third inequality follows ([ & M1 < co.
Similarly, conclude that
V (wai 24, Ill & ME +
(36)
Recalling Eq. 1..
we have
E
3=1
11
E
v
(37)
E
1/ (2 - 111ر1
[E
[])
2)
1=1
d
=Bil |1
Σ 5=1
15 - 1155))
From 0 S 1/51 2 in Eq.
9
we know 0 S 1)li((2
nlll) S 1, and then
EE a E Iic
11)
Ei
121
SI
(38)
E
Dig
Hence we arrive the conclusion that mean removal in Eg. gradient variance V(V frg (wrz [] is lower than that of V
9 will make the upper bound of of
VVV
I (wys R..)).
(2). If Eq. 33 holds, the following inequality can be obtained due to Lagrange Mean Theorem, i.e.
Linf = 1 (wii ( 1|2,
(39)
where Linf is the minimal gradient Lipschitz constant.
Following Eq. 88 we know F (wk; (0 ) = G GXtwx + BR), and then the Hessian function of F(Wx) is
V2 (wos (ra 1n 1i = X 2 (X + B&) xi.
(40)
13

---

Under review as a conference paper at ICLR 2020
And then obtain
2 (wxi (rig)==)) - |l2 X + B&) X
V²G (X + Bk) X
S G (Xiwa B&) l12 x
VAG (X) Bu) 11 |
(41)
- M2 || Xxx
<M2d (H(
where d the dimension of each sample Ii, and H are the j-th diagonal element of (XgX the second and fourth equalities hold due to the fact [B |B |BAl22 the third inequality follows ||AB]|2 S ||/l the fifth inequality is owing to G[)l(= S My, and the last inequality is due to the fact ||Al|2 S d dmmxx where E Rdxd is positive definite and Abdsl the j-th diagonal element.
Similarly, we can derive that
| (wii VEL) ll2 M2d max (Mb.d)).
(42)
where F1(2 is j-th diagonal element of (XxX)) in which Xx is constructed by following X in Eq. (8). According to the definition of X in Eq. 99.
can be reformulated as
nbl
[i
[3]
#
n1
n/51 (2 - 111)
Σ
Σ
(43)
Zip
iu
1=1
115ر (2 و6
2
=
n
From 0 S 7/1l < 2 in Eq.
99)
we know 0 5 151501(2 -n(b1) S 1, and then
H [il
(44)
H
Therefore the upper bound of the gradient Lipschitz constant Ling of F (wki HEr) is lower that of F (wki {x,, 1n 1.
D. PROOF OF THEOREM 3
Proof (1)Similar to Eq 34
35 it can be also concluded that
V V (wn;
Mi B(lla,,
(45)
14

---

Under review as a conference paper at ICLR 2020
According to Eq.
10
we have
|
B
[19]
E
Σ
E
(46)
f=1
pكام V
1
n
1
وراماي
Σ
(I!(2))
n
Hence,
E
1
1
S
(47)
E
Fig
{pbl)²
2 1
[())
where the inequality holds since (gbا))]
(1) 2
Ei=1
I
2 1.
Therefore the upper bound of the gradient variance V(V (wy; will be lowered than that of V[V füg (wy, F.
(2). Similar Eg.
35
40
we conclude
2 (wai 12 Mzdmax (#)
(48)
where Ar(d-51 is j-th diagonal element of (xx in which X2 is constructed by following N in Eq.
(8). According to T0 we have
2
n
T .
A
Σ
(49)
pbll
n
Hence,
1
1
1
(50)
اbااا (pbi))
(2(11) 1x
where the inequality holds since (obl)}
(1)
2 1.
Hence the upper bound of the minimal gradient Lipschitz constant of F (wxi (6r Pi ) will be fur- ther lowered than that of F (wys (
(3). If (wys Yo is locally strongly convex, the condition num G (z) is also strong convex, and then
Xmaz (G(z)) Amin (G(z)) > M > 0,
(51)
where denase (.) is the maximal eigenvalue and Amin (.) is the minimal eigenvalue.
15

---

Under review as a conference paper at ICLR 2020
the local space the condition number of 4 Flo) fze, 1 is
cond(V2 (wxi (Tx))==)) =IV (urks 1 ||al| (V (wki (Tin )i=1)) 112
=|XX = (X + (G (X/X))
uo + B&) X | (Xx (X B&) x -1 1l2
Amin (G)
Amnax (oxxe(Xx XX)
mmax (G)
Amin (
Amax (XxXi)
2M
Amin (X
maxj Al
2M
(H
minj
(52)
where H is the j-th diagonal element of (Xxxf);: The third equality follows ||AB||2 ||
the forth inequality holds due to the fact ||BB] 2 (T((AB) Z Amin (A)dmax (B) if A and B are positive definitive.
Similarly, we have
max, (Ap
cond F (wxi (+r Bo-)) 2 M
(53)
ming [
Recalling Eq.
10 we obtain
2
A
(54)
m
pbll V
Σズ=
[1]
It is known max, {(ab)))
max Σm (((ا))
max, (B1--1)
hence we have
(min, [2b2)y2)
min;
Ein (a!b1)1²
miny (
(
(M
max,
max;
(55)
(fr(
(M(
miny
miny
Finally we arrive the conclusion that the lower bound of conditional number of 2 (wks (āi, will be lowered than that of 2 F (wii (5., Ln 1 1) in the local space. O
Pr
16


Reviews:
[{"review_id": "rJl0UqjjYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary: This paper proposes an alternative approach to batch normalization called POP-Norm. The authors motivate their approach theoretically and suggest that this also helps explain BatchNorm’s success due to the similarity between the two approaches. They then empirically examine their method on CIFAR10, CIFAR100 and ImageNet classification.\n\nComments: I find the paper hard to read and and plagued with major technical flaws. The paper also shows unfamiliarity with standard literature on optimization. Some of my major concerns are discussed below:\n\nA. Theoretical Results (Section 2):\n\nThese results (the connections between convergence and gradient Lipshictzness (beta-smoothness) and gradient variance) are extremely well-known (an example reference is Section 3 in arXiv:1909.03550). In fact, the effects of gradient variance on convergence in optimization have prompted a whole sub-field on variance reduced gradient descent methods such as SVRG. Moreover, prior work by Santurkar et al. identifies precisely the same effect (better beta-smoothness) with BatchNorm as one of the factors responsible for its success. \n\nB. Proposed approach:\n\nThe authors claim that one of the key differences between their approach and BatchNorm is dividing by the l2 norm instead of the gradient variance. However, it seems that they take the l2 norm of the random variable after mean subtraction not before (Equations 10 and 12). This is actually the same as the variance of the variable before mean subtraction, which makes this part of their approach identical to BatchNorm. Then the only difference in this aspect of their approach is the way the learned scale and shift is introduced, which seems somewhat arbitrary. \n\nEven more concerning are the claims in Theorems 2 and 3 which justify why mean subtraction and dividing by the l2 norm (variance) help reduce gradient Lipschitz constant/variance. The authors are comparing two upper (or two lower) bounds which is insufficient to draw any meaningful conclusions. In particular, given two random variables X1 and X2, one cannot claim that X1 < X2 based on (arbitrary) upper bounds; especially since there is no evidence that these bounds are tight. \n\nMoreover, all the theoretical analysis in this Section is performed for a single layer of the network assuming other layers are fixed. Specifically, as per their proofs, the gradient of the weights at a particular layer is not dependent by any of the previous/layer layers, thus completely ignoring the effects that come from simultaneously updating all the layers. Thus, the theoretical analysis in this Section does not really tell us about the effects of BatchNorm in a deep network. Their setting is actually identical to studying the effects of input normalization on a single network layer. In this setting, it is well-known that input normalization improves the conditioning of the problem and thus helps optimization ([LeCun et al., 1998 and Wiesler et al., 2011])---in fact, this was actually one of the motivations of the original BatchNorm paper. \n\nC. Experimental Results:\n\nBased on this, it seems to me that the proposed approach is essentially the same as BatchNormalization except for (1) the positioning of the layer (before the weights instead of after), (2) the exact way the learned scale and shift is done and (3) a hyperparameter kappa that is used to scale down the variance of all the layers and hence essentially scales down the learning rate by a constant. \n\nFrom an experimental point of view, I find (3) particularly concerning because it just means that in their learning rate for the POP-Norm experiments are 1/kappa times those for the BatchNorm baseline. Since the authors do not really tune over hyperparameters (they use a fixed learning rate in all their experiments) when comparing the different approaches, it is not clear whether the chosen learning rate is optimal for BatchNorm. To perform a fair comparison of their approach to BatchNorm, the authors should:\n1. Introduce the kappa scaling into BatchNorm layers and see the effect on performance.\n2. Perform a grid search for both methods and report the best performance over hyperparameters.\n3. Also include comparison to other SOTA normalization alternatives (including those that the authors mention in the introduction)\n\nD. Other comments:\ni. The discussion in Section 4 is hard to parse and I am not convinced of its correctness (I am not sure how the authors arrived at (16)).\nii. The paper writing is poor with several typos and spelling mistakes. In addition, the inline math makes the paper very hard to read at times.\n\nOverall, I think there are significant concerns with the paper, both in terms of writing and the soundness/novelty of the technical results and experiments. Thus, I recommend rejection.\n\n------------------------------------------\nReferences:\n\nLeCun, Y., Bottou, L., Orr, G., and Muller, K. Efficient backprop. In Orr, G. and K., Muller (eds.), Neural Networks: Tricks of the trade. Springer, 1998.\n\nWiesler, Simon and Ney, Hermann. A convergence analysis of log-linear training. In Shawe-Taylor, J., Zemel, R.S., Bartlett, P., Pereira, F.C.N., and Weinberger, K.Q. (eds.), Advances in Neural Information Processing Systems 24, pp. 657–665, Granada, Spain, December 2011.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "S1x3wWGXtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Pre-OPeration Normalization (POP-norm) as a theoretically grounded modification of batch normalization (Batch-norm). The authors provide an analysis of stochastic gradient descent (SGD) showing that the convergence can be accelerated by lowering the gradient Lipschitz constant and gradient variances. Based on this observation, the authors show that by applying POP-norm which is quite similar to the Batch-norm applied before the activation functions, the gradient Lipschitz constants and the gradient variances can be lowered. \n\nI must first confess that I’m not an expert in the relevant area. I’m don’t know why the reviewing system assigned this paper to me, but I will try my best to judge the value of the paper. \n\n Having said that, I don’t have enough background knowledge to evaluate the novelty of the paper. For instance, the Theorem 1 is stated as if it was originally introduced in this paper, without any citation. But I doubt that because I could easily find several results relating the convergence rate of SGD to the gradient Lipschitz constant. Are there any papers or standard textbooks to be noted? If so, please cite them properly.\n\nThe theorem 2 and 3 are based on the full-dataset statistics, but because they are infeasible in practice, the authors propose to use the batch statistics. How may this affect the theory? Intuitively, one could easily imagine that the smaller batch-size would lead to higher variances for batch statistics estimators, and thus impeding the regularization effect of the normalization. Also, the theory just states that the upper bound of (Lipschitz constant, gradient variance) after normalization will be lower than the lower bound without normalization, so it is not clear how much the normalization would reduce them. If possible please state the detailed results in the main text. \n\nThe differences between POP-norm and Batch-norm are, 1) whether the normalization is applied before or after the operations, and 2) minor difference in the normalization procedure, such as the additional scaling parameter kappa. The experiments provide results to demonstrate the effectiveness of 1), but not clear about 2). It would be interesting to compare POP-norm to the Batch-norm applied before the operations to see the net effect of 2). The very idea of applying Batch-norm before the convolution operation has already been demonstrated and widely used in the case of ResNet with ReLU activation functions, so if the contribution from 2) is insignificant, I think one may ask the novelty of the proposed normalization scheme.\n\nOverall, despite the lack of background knowledge, I find this paper to be borderline, but more close to the reject, because of some unclear points. I would like to hear back from the authors and other reviewers.\n\nI don’t understand why the matrix B in eq.8 gathering the biases has additional indices 1_k, …, m_k. Shouldn’t it be the repetition of b_k for m times?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Bke5lxKoYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new normalization method for accelerating the training of deep neural networks. Specifically, the proposed method put the normalization step before the linear transformation step. Then, the authors show that the normalization step can reduce the variance of gradients. The experimental results show some improvement. \n\n1. The proof of Theorem 1 is the regular convergence analysis of SGD. No new contributions.\n\n2. If my understanding is right, Theorem 2 and 3 actually can also show that other normalization techniques have these effects. So, what's the difference between POP-Norm and other normalization techniques in reducing gradient variance and  Lipschitz  constant?\n\n3. This paper claims that the effect of batch normalization will be destroyed by Tanh and Sigmoid activation function. How about POP-Norm? It's better to show the performance of POP-Norm for the network with Tanh or Sigmoid.\n\n4. How about its performance for large networks, such as ResNet-50?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an alternative to batch norm, which they call POP-norm, and provide theoretical justification for POP-norm in nonconvex optimization on the basis of variance reduction. They then present empirical arguments.

One of the most cogent reviewers believed the theoretical results were known and the empirical arguments unconvincing because the method is similar to batch norm up to a change in learning rate and some minor differences.

Unfortunately, the reviewers did not engage with the author rebuttals at all. The authors seem to have addressed most points. However, if the reviewers are unwilling to engage, despite multiple emails, there's not much I can do, short of redoing the whole process from scratch. And I'll take the lack of engagement as lack of interest by the reviewers. Not being an expert in optimization myself, I'm not going to override the scores. I do know enough to know that there are standard bounds for both convex and nonconvex optimization that improve with decreased variance. 
Decision: Reject