A spherical analysis of Adam with Batch Normalization
Authors: Simon Roburin
Yann Dubois de Mont-Marin
Andrei Bursuc
Renaud Marlet
Patrick Perez
Mathieu Aubry
Venue:  (2021)
Keywords: Deep Learning
Machine Learning
Adam
Batch Normalization

Abstract: Batch Normalization (BN) is a prominent deep learning technique. In spite of its apparent simplicity, its implications  over  optimization  are  yet  to  be  fully  understood. While previous studies mostly focus on the interaction between BN and stochastic gradient descent (SGD), we develop a geometric perspective which allows us to precisely characterize the relation between BN and Adam. More precisely, we leverage the radial invariance of groups of parameters, such as filters for convolutional neural networks, to translate the optimization steps on the $L_2$ unit hypersphere. This formulation and the associated geometric interpretation shed new light on the training dynamics. Firstly, we use it to derive the first effective learning rate expression of Adam. Then we show that, in the presence of BN layers, performing SGD alone is actually equivalent to a variant of Adam constrained to the unit hypersphere. Finally, our analysis outlines phenomena that previous variants of Adam act on and we experimentally validate their importance in the optimization process.

Under review as a conference paper at ICLR 2021
A SPHERICAL ANALYSIS OF ADAM WITH BATCH NORMALIZATION
Anonymous authors Paper under double-bling review
Batch Normalization (BN) is a prominent deep learning technique. In spite of its apparent simplicity, its implications over optimization are yet to be fully understood. While previous studies mostly focus the interaction between BN and stochastic gradient descent (SGD), we develop geometric perspective which allows us to precisely characterize the relation between BN and Adam. More precisely, we leverage the radial invariance of groups of parameters, such as filters for convolutional neural networks, to translate the optimization steps on the L2 unit hypersphere. This formulation and the associated geometric interpretation shed new light on the training dynamics. Firstly, we use it to derive the first effective learning rate expression of Adam. Then we show that, in the presence of BN layers, performing SGI) alone is actually equivalent to a variant of Adam constrained to the unit hypersphere. Finally, our analysis outlines phenomena that previous variants of Adam act on and we experimentally validate their importance in the optimization process.
INTRODUCTION
The optimization process of deep neural net- works is still poorly understood. Their train- ing involves minimizing a high-dimensional non-convex function, which has been proved to be a NP-hard problem (Blum& & Rivest, 1989). Yet, elementary gradient- based methods show good results in practice. To improve the qual- ity of reached minima, numerous methods have stemmed in the last years and become common practices. One of the most prominent Batch Normalization (BN) (Ioffe & Szegedy) (2015). which improves significantly both the optimiza- tion stability and the prediction performance; it now used in most deep learning architectures. However, the interaction of BN with optimiza- tion and its link to regularization remain open research topics. Previous studies highlighted mechanisms of the interaction between BN and SGD, both empirically (Santurkar et al. 2018) and theoretically (Arora et al. 2019] Bjorck
2h V(x())
u1++1
[(4)
Fk
0 (HEII)))))))
S
Figure 1: Illustration of the spherical perspective for SGD. The loss function L of a NN w.r.t. the parame- ters X E R" of neuron followed by BN is radially invariant. The neuron update x X in the origi- nal space, with velocity 7æ VL(x,), corresponds an update u2 -> us+11 of its projection through an expo- nential map on the unit hypersphere Sa-1 with velocity vy V || at order 2 (see details in Section 2.3).
al. 20188] Hoffer et al. 2018b). None of them studied the interaction between BN and one of the most common adaptive schemes for Neural Networks (NN), Adam (Kingma & Ba, 2015), except van Laarhoven (2017), which tackled only in the asymptotic regime. In this work, we provide an extensive analysis of the relation between BN and Adam during the whole training procedure.
One of the key effects of BN to make NNs invariant to positive scalings of groups of parameters. The core idea of this paper is precisely to focus on these groups of radially-invariant parameters and analyze their optimization projected on the T.2 unit hypersphere (see Fig. [T. which topologically equivalent to the quotient manifold of the parameter space by the scaling action. One could directly optimize parameters on the hypersphere as Cho Lee 2017). 7), yet, most optimization methods are still performed successfully in the original parameter space. Here we propose to study optimization scheme for a given group of andiall-invarian parameters through its image scheme on the unit hypersphere. This geometric perspective sheds light on the interaction between normalization layers and Adam, and also outlines an interesting link between standard SGD and a variant of Adam adapted and constrained to the unit hypersphere: AdamG (Cho & Lee 2017). We believe this kind of analysis

---

Under review as a conference paper at ICLR 2021
is an important step towards better understanding of the effect of BN on NN optimization. Please note that, although our discussion and experiments focus on BN, our analysis could be applied to any madiall--ivarrian invariant model.
The paper is organized as follows. In Section we introduce our spherical framework to study the optimization of radially-invariant models. We also define a generic optimization scheme that encompasses methods such as SGD with momentum (SGD-M) and Adam. We then derive its image step on the unit hypersphere, leading definitions and expressions of effective learning rale and effective learning direction. This new definition is explicit and has a clear interpretation, whereas the definition of van Laarhoven (2017) asymptotic and the definitions ofooa al. (2019) and of Hoffer et al. (2018b) are variational. In Section we leverage the tools of our spherical framework to demonstrate that in presence of BN layers, SGD has an adaptive behaviour. Formally, we show that SGD is equivalent to AdamG, variant of Adam adapted and constrained to the hypersphere, without momentum. In Section we analyze the effective learning direction for Adam. The spherical framework highlights phenomena that previous variants of Adam (Loshchilov & Hutter, 2017 Cho & Lee 2017 act on. We perform an empirical study of these phenomena and show that they play significant role in the training of convolutional neural networks (CNNs). In Section 5 these results are put in perspective with related work.
Our main contributions are the following:
A framework to analyze and compare order-1 optimization schemes of radially-invariant models;
The first explicit expression of the effective learning rate for Adam;
The demonstration that, in the presence of BN layers, standard SGD has an adaptive behaviour;
The identification and study of geometrical phenomena that occur with Adam and impact signifi- cantly the training of CNNs with BN.
2 SPHERICAL FRAMEWORK AND EFFECTIVE LEARNING RATE
In this section, we provide background on radial invariance and introduce a generic optimization scheme.
Projecting the scheme update on the unit hypersphere leads to the formal definitions of effective learning rate and learning direction. This geometric perspective leads the first explicit expression of the effective learning rate for Adam. The main notations are summarized in Figure] [
2.1 RADIAL INVARIANCE
We consider family of parametric functions dx TR - ouu parameterized by a group of radially- invariant parameters omitted for clarity), a dataset D 18 x Rout a loss function É IRou x Ran + and training loss function L: _d + E defined as: def 1
E Rd < {0), i.c., Yp 0, 0px = Øx (possible other parameters of Øx
((x)
Σ (s) (s.t)E
(1)
D
It verifies: Vp > O, C(px) -(x). In the context NNs, the group of radially parameters x can be the parameters of a single neuron in a linear layer or the parameters of a whole filter in convolutional layer, followed by BN (see Appendix A for details, and Appendix for the application to other normalization schemes such as InstanceNorm (Ulyanov et al. 2016), LayerNorm (Ba et al. 2016) or GroupNorm (Wu He| 2018)).
The quotient of the parameter space by the equivalence relation associated to radial invariance is topologically equivalent to sphere. We consider here the T-2 sphere S2- = (u E Rd /||uu|/2 = 1} whose canonical metric corresponds to angles: ds (lj u2) arccos ((u1, uz)). This choice of metric relevant to study NNs since filters in CNNs or neurons MLPs are applied through scalar product input data. Besides, normalization in BN layers is also performed using the L2 norm.
Our framework relies on the decomposition of vectors into radial and tangential components. During optimization, write the radially invariant parameters at step & 2 Oas X r'uu where T& X and u₁ = X x ||. For any quantity qk E Rd at step k, we write q = q - -(g Ug un its tangential component relatively to the current direction Ugg.
2

---

Under review as a conference paper at ICLR 2021
The following lemma states that the gradient of rudially-invarian loss function is tangential and -1 homogeneous:
Lemma 1 (Gradient of function with radial invariance). If C R - R is radially invariant and almost everywhere differentiable, then, for all p >> 0 and all x E IE where & is difserentiable:
xV) =0 and VL(x) = V V((x)).
2.2 GENERIC OPTIMIZATION SCHIEME
There a large body of literature on optimization schemes (Sutskever et al. 2013 Duchi et al. 2011
Tieleman & Hinton 2012 Kingma & Ba 2015 Loshchilov & Hutter 2019). We focus here on two
of the most popular ones, namely SGD and Adam (Kingma & Baj 2013). Yet, to establish general
results that may apply to variety of other schemes, introduce here generic optimization update:
X X; a). @ bj,
Ag Bax + V((xx) + AXk=
(4)
where X₁ E Ted the group of radially-invariant parameters at iteration k. L. the group's loss estimated on a batch of input data, ag E Rd is a momentum, by E R2d is a division vector that can depend on the trajectory (x,, ([(x)))sE(o.]] T IR is the scheduled learning rate, O denotes the Hadamard element-wise division, 3 is the momentum parameter, and À is the -e-regularriztion parameter. We show how it encompasses several known optimization schemes.
inndeeednn
Stochastic gradient descent (SGD) has proven to be effective optimization method in deep learning. can include L2 regularization (also called weight decay) and momentum. Its updates are:
X = X m)
(5)
m 8m:-1 V ((xs) AXE,
(6)
where my is the momentum, 3 the momentum parameter, and the Læ-regularization parameter. It corresponds to our generic scheme (Eqs. 3 4 with mg and b2 [1....]
Adam likely the most common adaptive scheme for NNs. updates are:
m
Vk
X = Xx
TE
1
g
V 1
3 gl+1 tc,
m = -3 (VL(x) AXx);
3 + (1233) + AX:)? (8)
where my is the momentum with parameter 31, Vk is the second-order moment with parameter Bg, and prevents division by zero. (Here and in the following, the square and the square root of a vector be understood as elseent-wise.) It corresponds to generic scheme (Eqs. with 8=81 and:
gh+t +1
Vk
ag
b
+c.
31
1 8
g
D2
2.3 IMAGE OPTIMIZATION ON THE HYPERSPHERE
The radial invariance implies that the radial part of the parameter update does not change the function Φx encoded by the model, nor does it change the loss L(x). The goal of training is to find the best possible function encodable by the network. Due to radial invariance, the parameter space projected on the unit hypersphere is topologically closer to the functional space of the network than the full parameter space. hints that looking at optimization behaviour on the unit hypersphere might be interesting. Thus, we need to separate the quantities that can (tangential part) and cannot (radial part) change the model function. Theorem formulates the spherical decomposition (Eqs. in simple terms. It relates the update of radially-invariant parameters in the parameter space Rd and their update on S through an exponential map.
Theorem (Image step Sa-1). The update of group of radially-invariee parameters X at step l corresponds to an update of its projection un on Sa-1 through an exponential map at u, with velocity Th c at order 3:
[1++0(nnllctt n
U
Expax
(10)
3

---

Under review as a conference paper at ICLR 2021
where ExPu, is the exponential map on Sa-1. and with
def
bk
def
7K
(1
1k (Ck, up
C₁
2
d.
b&ll
1k
rt 1/2 |b&l
r/ 1/2//111b/
(II)
More precisely:
Uk
Uk
(12)
V1 (ng 11)
The proof is given in Appendix C.I.I and the theorem is illustrated in the case SGD in Figure] Note that with typical values in CNN training we have 1- To (o(c..un)
// O, which is property
needed for the proof. Another hypothesis is that steps on the hypersphere are shorter than IT These hypotheses are discussed and empirically verified in Appendix C.1.2
2.4 EFFECTIVE LEARNING RATE FOR ADAM
In Theorem 2 the normalized parameters update in Eq. 10 can be read Ug+11 R ExPux (-n
where níé and oj can then be respectively interpreted as the learning rate and the direction of an optimization step constrained to S since ag is the momentum and, with Lemma [ the quantity ax:ag in C2 can be seen as a momentum on the hypersphere. Due to the radial invariance, only the change of parameter on the unit hypersphere corresponds to change of model function. Hence can interpret 7yg and cj as effective learning rate and effective learning direction. In other words, these quantities correspond to the learning rate and direction on the hypersphere that reproduce the function update of the optimization step.
derive actual effective learning rates for any optimization scheme that fits our generic framework. These expressions, summarized in Table are explicit and have a clear interpretation, in contrast learning rates in (van Laarhoven, 2017), which are approximate and asymptotic, and in (Hoffer et al. 2018a; Arora et al., 2019), which are variational and restricted to SGD without momentum only.
Using Theorem
Table 1: Effective learning rate and direc- tion for optimization schemes (k omitted), with v = rd r2- [b]l
Scheme
SGD
VL(u)
SGD T
VL(u)
> > (1-nA)
In particular, we provide the first explicit expression of the effective learning rate for Adam:
SGD-M
2
Adam
(1
1(c.u)
c
77k (1 TIKE
[Ck u
FP
12
Th
(13)
TVk
where V/2 = d / || is homogeneous to the norm of a gradient on the hypersphere and can be related to second-order moment the hypersphere (see Appendíx C.1.3 for details). This notation also simplifies the in-depth analysis in Section 4 allowing a better interpretation of formulas.
The expression of the effective learning rate of Adam, i.e., the amplitude of the step taken the hypersphere, reveals dependence on the dimension d (through v) of the considered group of radially- invariant parameters. In the case of an MLP CNN that stacks layers with neurons or filters of different dimensions, the learning rate is thus tuned differently from one layer to another.
We also that for all schemes the learning rate is tuned by the dynamics of radiuses rg. which follow:
Tk+1
(1
-2d- bb ) V (nālle) II)a
77 (Ck, uz)
(14)
Fk
In contrast to previous studies (Arora al. 2019 van Laarhoven 2017), this result demonstrates that for momentum methods, (cy. ug). which involves accumulated gradients terms in the momentum as well as La regularization, tunes the learning rate (cf. Fig
4

---

Under review as a conference paper at ICLR 2021
3 SGD IS A VARIATION OF ADAM ON THE HYPERSPHERE
We leverage the tools introduced in the spherical framework to find a scheme constrained to the hypersphere that is equivalent to SGD. It shows that for radially-invariant models, SGD is actually an adaptive optimization method. Formally SGD is equivalent to a version of AdamG, a variation of Adam adapted and constrained to the unit hypersphere, without momentum.
3.1 EQUIVALENCE BETWEEN TWO OPTIMIZATION SCHEMES
Due to the radial invariance, the functional space of the model is encoded by Sa-1- In other words, two schemes with the same sequence of groups of radially-invariant parameters on the hypersphere (ux)x20 encode the same sequence of model functions. Two optimization schemes and S are equivalent iff Vk: 2 0, it = ügg- By using Eq. 12 we obtain the following lemma, which useful to prove the equivalence of two given optimization schemes:
Lemma (Sufficient condition for the equivalence of optimization schemes).
{
Un = Ug Ay 0, nf T CF
Ak 2
0.
Up
(15)
oz
3.2 WETERRSPHERE-CONSSTRR
SCHEME EQUIVALENT TO SGD
We now study, within our spherical framework, SGD with L2 regularization, i.e., the update X X - The (VL(xx) - ). From the effective learning rate expression, we know that SGD yields an adaptive behaviour because it is scheduled by the radius dynamic, which depends fact, the tools in our framework allow us to find a scheme constrained to the unit hypersphere that is equivalent SGD: AdamG (Cho & Lee 2017). More precisely, it is AdamG with null momentum factor 31 = 0, an non-null initial second-order moment vp. an offset of the scalar second-order moment A + 1 + & and the absence of the bias correction term 1 -/++1 Dubbed * this scheme reads:
gradients. In
v((x))
& = Xx-- 7k
Vok
(AdamG*):
X = 2:
Uk+1 BUk + [
Starting from SGD, we first use Lemma 3 to find an equivalence scheme with simpler radius dynamic. We resolve this radius dynamic with a Taylor expansion at order 2 (7ı [/ll) second use of Lemma finally leads to the following scheme equivalence in Theorem (see proof in Appendix C.1.4). If we call a equivalent at order 2 in the step > a scheme equivalence that holds when we for Tyy expression that satisfies the radius dynamic with Taylor expansion order we have the following theorem:
A
Theorem (SGD equivalent scheme on the unit hypersphere). For any 0, 0, > we have the following equivalence when using the radius dynamic at order 2 in (1) VL un (l) /r2:
(AdamG*)
(SGD)
Xo Un I (1 17/211 T = (28) (28)-1/2 112 vo r2 3112)-1
Xn Tn
scheme-equivalen order 2 in step with
A X
7 = 7
This result unexpected because SGD, which is not adaptive by itself. is equivalent second order moment adaptive method The scheduling performed by the radius dynamics actually replicates the effect of dividing the learning rate by the second-order moment of the gradient norm: 14- First, the only assumption for this equivalence is to neglect the approximation in the Taylor expansion at order 2 of the radius which is highly verified in practice (order magnitude of 1e 4 isee Appendix C.I.5). Second, with standard values of the hyper-parameters: learning rate 77 < and weight decay A<L we have 3 S 1 which corresponds to a standard value for a moment factor. Interestingly, the L2 regularization parameter À controls the memory of the past gradients norm. If 3 1 (with X = 0) there is no attenuation, each gradient norm has the same contribution in the order of two moments. 0, there is decay factor (3 < 1) on past gradients norm in the order 2 moment.
5

---

Under review as a conference paper at ICLR 2021
4 GEOMETRIC PHENOMENA IN ADAM
Our framework with its geometrical interpretation reveals intriguing behaviors occurring in Adam. The unit hypersphere is enough to represent the functional space encoded by the network. From the perspective of manifold optimization, the optimization direction would only depend on the trajectory on that manifold. In the case of Adam, the effective direction not only depends on the trajectory on the hypersphere but also on the deformed gradients and additional radial terms. These terms are thus likely to play a role in Adam optimization.
In order to understand their role, we describe these geometrical phenomena in Section Inter- estingly, previous variants of Adam, Adam W (Loshchilov & Hutter (017) and AdamG (Cho & Lee 2017) are related to these phenomena. To study empirically their importance, we consider in Section variants of Adam that first provide a direction intrinsic to the unit hypersphere, without deformation of the gradients, and then where radial terms are decoupled from the direction. The empirical study of these variants over variety of datasets and architectures suggests that these behaviors do play a significant role in CNNs training with BN.
4.1 IDENTIFICATION OF GEOMETRICAL. PHENOMENA IN ADAM
Here, we perform an in-depth analysis of the effective learning direction of Adam.
(a) Deformed gradients. Considering the quantities defined for a generic scheme in Eq. I1 b has a deformation effect on ay. due to the Hadamard division by
b, and a scheduling effect d d/ b.
d' V/ b&l on the effective learning rate. In the case where the momentum factor is null 31 = 0, the direction of the update at step i is VZ((u)) Ø
(Eq. 11 and the deformation by d' may push the direction of the update outside the tangent space of S at ug. whereas the gradient itself lies in the tangent space. This deformation is in fact not isotropic: the displacement of the gradient from the tangent space depends on the position of ug on the sphere. We illustrate this anisotropy in
by
Fig. b).
(b) Additional radial terms. In the momentum on the sphere Ck. quantities that are radial (resp. orthogonal) a point on the sphere may not be radial (resp. orthogonal) at another point. To clarify the contribution of C₁ in the effective learning direction C₁ we perform the following decomposition (cf. Appendix(..l
by
C₁ = (cck grad
+ ArREC 2
with:
(16)
d' b&ll
1-1
k-1
grad del (L(u))
gk Tk VC(u:)
and
of def
u
gk- P2
(17)
r;
Fkk
1=0
1=0
1. Contribution of grad At step k. the contribution of each past gradient corresponds to the orthogonal part VL(u.) - (V,,,, ug ug. . It impacts the effective learning direction depending on its orientation relatively to ugg Two past points, although equally distant from u1 on the sphere and with equal gradient amplitude may thus contribute differently in of due to their orientation (ef. Fig. ())).
2. Contribution of cf Naturally, the current point us does not contribute to the effective learning direction of unlike the history of points in >k-1 gh-i Ii u, which does. This dependency can be avoided if we decouple the La regularization, which case we do not accumulate L2 terms in the momentum. This shows that the decoupling proposed in Adam (Loshchilov & Hutter 2019) actually removes the contribution of L2 regularization in the effective learning direction.
P2
C and La2 (in inverse proportion) impacts the effective learning direction c it can differ for identical sequences (u:)εęk the sphere but with distinet radius histories (re) Since the radius is closely related to the effective learning rate, it means that the effective learning direction cf is adjusted according to the learning rates history.
(c) The radius ratio 2%k present in both grad
Note that AdamG (Cho & Lee 2017), by constraining the optimization the unit hypersphere and thus removing L2 regularization, neutralizes all the above phenomena. However, this method has no scheduling effect allowed by the radius dynamics (ef. T4p since it is kept constant during training.

---

Under review as a conference paper at ICLR 2021
X/-1
Ke 5.0m iee Mel
ve(on)
19g mgg 0-
u
((V2(()))
Sa-1
St-1
S6-1
(a) Radial scheduling
(b) Anisotropy of deformation
(c) Gradient-history contribution
Figure 2: (a) Effect of the radial part of ea on the displacement on St-1; (b) Example of anisotropy and
sign instability for the deformation & ((VCCuu)) = V v Ø 3-1;]VE<(,,]] (where | is the clement-wise optimization step: (c) Different contribution eă of two past gradients
/VC(u))
absolute value) occurring in Adam's V, and V2 of equal norm, depending on their orientation. Illustration of the transport of Va from ug-1 to ug
Tu, (V1) (ef. Appendix D.2 for details)
4.2 EMPIRICAL STUDY
To study empirically the importance of the identified geometric phenomena, we perform an ablation study: compare the performance (accuracy and training loss speed) of Adam and variants that neutralize each of them. We recall that Adam neutralizes (b2) and that AdamG neutralizes all of above phenomena but loses the scheduling effect identified in Eq. [4] To complete
analysis, use geometrical tools to design variations of Adam which neutralizes sequentially each phenomenon while preserving the natural scheduling effect in Theorem 2 We neutralize (a) by replacing the clement-wise second-order moment, (b1) and (b2) by transporting the momentum from current point the new one, (c) by re-scaling the momentum at step The details Appendix, D.2 The final scheme reads:
Xk+1
Xk
17
+ C,
(18)
mg
B]
Tk-1 pu
rk
W
- (V)(E(( + AXE);
(19)
02
8g
Kk
Uk
+ (2)d (|L((x) 1x
(20)
rx
where l'ue is the hypersphere canonical transport from u to ug. Implementation details are in
Appendix D.3
Protocol. For evaluation, conduct experiments on two architectures: VGG16 (Simonyan & Zisserman 2015) and ResNet (He et al. 2016) - more precisely ResNer20, a simple variant designed for small images (He al. 20166), and ResNet18, popular variant for image classification. We consider three datasets: SVHN (Netzer et al. 2011). CIFAR1O and CIFAR100 (Krizhevsky et al. 2009).
Since our goal is to evaluate the significance of phenomena on radially-invariant parameters, i.c., the convolution filters followed by BN, only apply variants of Adam including AdamG and AdamW on convolution layers. For comparison consistency, we keep standard Adam on the remaining parameters. We also use fixed grid hyperparameter search budget and frequency for each method and each architecture (see Appendix D.3 for details).
Results. Table we report quantitative results of Adam variants across architectures and datasets. In addition, we compare the evolution of the training loss in Fig. 3 We observe that each phenomenon displays specific trade-off between generalization (accuracy on the test set) and training speed, as following. Neutralizing (a) has little effect on the speed over Adam, yet achieves better accuracy. Although slows down training, neutralizing (ab) leads to minima with the overall best accuracy on test set. Note that Adam T neutralizes (b2) with its decoupling and is the fastest method, but finds minima with overall worst generalization properties. By constraining the optimization to the hypersphere, A speeds up training over the other variants. Finally, neutralizing (c) with Adam
7

---

Under review as a conference paper at ICLR 2021
resnet20.oo on cifar10
resnet20 cifar10
Adam
600
Adam
(a)
Adam
(ab)
Gebr)
AdainG
0.00
AdbmGG
los5
1.0
Adaamti
Adaom
mean
640
641
epoch
egoch
Figure 3: Training speed comparison with ResNet20 on CIFARIO. Left: Mean training loss over all training epochs (averaged across seeds) for different Adam variants. Right: Zoom-in the last epochs. Please refer Table 2 for the corresponding accuracies.
w/o (abc) brings slight acceleration, though reaches lower accuracy than Adam w/o (ab). We can see that the revealed geometrical phenomena impact substantially training of BN-equipped CNNs.
Table 2: Accuracy of Adam and its variants. The figures in this table the mean top1 accuracy + the standard deviation over seeds on the test set for CIFARIO, 10, CIFAR 100 and on the validation set for SVHN, indicates that the original method is only used on convolutional filters while Adam is used for other parameters.
CIPARIO ResNet10
CIFARIO0
SVHN
Method
20
VGS16
18
V8G16
Re
V
Adam
90.98 0.06
93,77 + 0.20
92.83 +: 0.17
71.30 4: 0.36
68.43 + 0.16
95.32 0.23
95.57 + 0.20
Adamw
90.19 0.24
93.61 + 0.12
92.53
4: 0.25
67.39
4 0.27
71.37 - 0.22
95.13 d: 0.15
94.97 +: 0.08
AdamG1
91.64
0.12
93.41 0.17
73.76
0.34
70.17
95.73
95,70 0.25
Adam wlo (a)
91.15 0.11
93.95 0.25
92.92
0.11
74.44 + 0.22
68.73
95.75 0.00
95.66 + 0.09
Adam wio (ab)
91.92
0.10
93.89
0.25
71.53 0:19
96.05
0.12
0.09
Adam wis (sibe)
91.81 + 0.20
94.92 = 0:05
93.75 2: 0.06
75.28 4: 0:35
71.45 2 0.13
05.84 1: 0.07
95.82 + 0.05
5 RELATED WORK
Understanding Batch Normalization. Albeit conceptually simple, BN has been shown to have complex implications over optimization. The argument of Internal Covariate Shift reduction (Toffe) Szegedy. 2015) has been challenged and shown to be secondary to smoothing of optimization landscape (Santurkar et al. 2018 Ghorbani et al.] 2019) or its modification by creating a different objective function (Lian & Liu| 2019). or enabling of high learning rates through improved condition- ing (Bjorek et al., 2018). Arora et al. (2019) demonstrate that (S)GD with BN is robust to the choice of the learning rate, with guaranteed asymptotic convergence, while a similar finding for GD with
BN is made by Cai et al. (2019).
Invariances in neural networks. Cho & Lee (017) propose optimizing over the Grassmann mani- fold using Riemannian GD. [.D.Liuu et al. (2017) project weights and activations on the unit hypersphere and compute a function of the angle between them instead of inner products, and subsequently generalize these operators by scaling the angle (Liu et 2018). 3 In (Li & Arora (2020) the radial in- variance is leveraged to prove that weight decay (WD) can be replaced by an exponential learning-rate scheduling for SGD with without momentum. Arora et al. (2019) investigate the radial invariance and show that radius dynamics depends on the past gradients, offering an adaptive behavior to the learning rate. Here we go further and show that SGD projected on the unit hypersphere corresponds to Adam constrained to the hypersphere, and we give an accurate definition of this adaptive behavior.
Effective learning rate. Due to its scale invariance, BN can adaptively adjust the learning rate (van Laarhoven 2017 Cho & Leel 2017 Arora et al.] 2019 Li & Arora (020). van Laarhoven (2017) shows that in BN-equipped networks, WD increases the effective learning rate by reducing the norm of the weights. Conversely, without WD, the norm grows unbounded (Soudry et al.) 2018), decreasing the effective learning rate. Zhang et al. (2019) brings additional evidence supporting hypothesis in van Laarhoven (2017). while Hoffer et al. 2a) finds an exact formulation of the effective learning rate for SGD in normalized networks. In contrast with prior work, we find generic definitions of the effective learning rate with exact expressions for SGD and Adam.
8

---

Under review as a conference paper at ICLR 2021
6 CONCLUSION
The spherical framework introduced in this study provides a powerful tool to analyse Adam optimiza- tion scheme through its projection the L2 unit hypersphere. It allows us give precise definition and expression of the effective learning rate for Adam, to relate SGD to a variant Adam, and to identify geometric phenomena which empirically impact training. The framework also brings light to existing variations of Adam, such as Lz-regularization decoupling. This approach could be extended other invariances in CNNs such as filter permutation.
REFERENCES
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization. In International Conference on Learning Representations (ICLR), 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv: 1607.06400. 2016.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch nor- malization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 7694-7705, 2018.
Avrim Blum and Ronald L Rivest. Training 3-node neural network is ap-molee In Advances in Neural Information Processing Systems (NeurIPS), pp. 494-501, 1989.
Yongqiang Cai, Qianxiao Li, and Zuowei Shen. A quantitative analysis of the effect of batch normalization gradient descent. In 36th International Conference on Machine Learning (ICML), volume 97, 2019.
Minhyung Cho and Jachyung Lee. Riemannian approach batch normalization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5225-5235, 2017.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal Machine Learning Research (JMLR), 12(ul):2121-2159, 2011.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In 36th International Conference on Machine Learning (ICML), volume 97, pp. 2232-2241, 2019.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances Neural Information Processing Systems (NeurIPS), pp. 2160-2170, 2018. 8a.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: the marginal value of training the last weight layer. arXiv preprint 1rXiv:1801.04540. 2018b.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In 32nd International Conference Machine Learning (ICML), volume 37, pp. 448-456, 2015.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. In Interna- tional Conference on Learning Representations (ICLR), 2020.
Xiangru Lian and Ji Liu. Revisit batch normalization: New understanding and refinement via composition optimization. In The 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 3254-3263, 2019.

---

Under review as a conference paper at ICLR 2021
Weiyang Liu, Yan-Ming Zhang, Xingguo Li, Zhiding Yu, Bo Dai, Tuo Zhao, and Le Song. Deep hyperspherical learning. In Advances Neural Information Processing S ystems (NeurIPS), pp. 3950-3960, 2017.
Weiyang Liu, Zhen Liu, Zhiding Yu, Bo Dai, Rongmei Lin, Yisen Wang, James M Rehg, and Le Song. Decoupled networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2771-2779, 2018.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
lyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Shibani Santurkar, Dimitris Tsipras, Andrew Hyas,, and Aleksander Madry. How does batch normal- ization help optimization? In Advances Neural Information Processing Systems (NeurIPS), pp. 2483-2493, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research (JMLR), 19 (1):2822-288) 2018.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum deep learning. In 30th International Conference on Machine Learning (ICML), volume 28, pp. 1139-11477 1147, Atlanta, Georgia, USA, 17-19 Jun 2013.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 1607.08020 2016.
Twan van Laarhoven. L2 regularization versus batch and weight normalization, 2017. arXiv preprint arXiv:1706.08
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
10


Reviews:
[{"review_id": "ejILp5uQkK", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper attempts to analyze Adam with BN from a spherical perspective. \n\n1. Though the authors derived the effective learning rate for Adam in Eq. 13, the analysis still needs to be more thorough. For example, how this effective learning rate is affected by the hyperparameters (i.e. learning rate, batch size). And in Eq. 13, since this term is complicated, involving ck, uk and b, a detailed and intuitive analysis should be provided. \n\n2. About the second contribution, the paper claimed that SGD with BN behaves like a variant of Adam, AdamG*. However, this AdamG* has significant difference from original Adam: in AdamG* the division is by a *scalar* (not an element-wise division in Adam), this does not change the radially-invariant property of models. Thus, the connection between SGD with BN and AdamG* is obvious. I expect the authors provide some analysis on the connection between SGD with BN and AdamG*. \n\nSo I think is the contributions in this work are slightly below the acceptance bar of ICLR before these concerns are solved appropriately. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "VFhRjIVeXIi", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work studies optimization dynamics for neural network models that are scaling invariant with respect to parameters. A general formulation of optimization algorithms is considered, covering many widely used algorithms like SGD and Adam. The projected dynamics (to the unit sphere) is studied, and the effective learning rate and update direction on the unit sphere are derived. Focusing on the projected dynamics, the equivalence is built between SGD and a type of \"Adam\". Then, different factors in the Adam dynamics that can potentially influence the optimization performance are identified, and empirically studied. \n\nThe paper is overall clear and easy to follow. My major concern is that the results are weaker than that is claimed in the abstract. Specifically,\n1. The image optimization on the hypersphere is just a projected dynamics, it is not a close dynamics with respect to u. The radial part, although does not contribute to the loss directly, still plays a role in the dynamics. In this sense, the projected dynamics is not simplified compared to the original one. The effective learning rate is not easy to control. Moreover, the loss function of u is still highly non-convex and hard to study. No evidence is shown that L(u) is simpler than L(x).\n2. The adaptive gradient methods are adaptive in at least two ways: a) the learning rate is adaptive; b) the learning rate is different for each parameter. The AdamG* considered in the paper does not use element-wise learning rate. It is more like an SGD with a learning rate schedule, instead of an adaptive gradient algorithm. Hence, the sentence \"performing SGD alone is actually equivalent to a variant of Adam constrained to the unit hypersphere\" in the abstract is misleading and conveys over-optimistic information. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "H-WjODdMW6V", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-response update: I thank the authors for their response. I updated my score, but still think the paper needs improvement to be of interest to the ICLR community.\n\n---\n\n\nThe submission analyzes the behavior of gradient descent and adaptive variants for scale-invariant models, including batch-normalization, by looking at the trajectory of the iterates when projected on the unit sphere. It contributes a formula for the equivalent learning rate if the gradient step was to be taken on the unit sphere for SGD and Adam and shows an approximate equivalence between gradient steps and normalized gradient steps taken on the unit sphere.\n\nThe spherical perspective is an interesting and insufficiently explored aspect of modern machine learning models. As the magnitude of the weights is essentially an irrelevant free parameter, understanding its behavior is likely to yield insights on how we train those overparametrized models.\n\nThe submission, however, does not provide a strong motivation for the presented results. The stated contribution that \"in the presence of BN layers, standard SGD behaves like Adam without momentum\" is also slightly over-selling the results of section 3. The main issue is that the significance of the explicit learning rates derived in section 2 or the equivalence between gradient descent and a normalized variant in section 3 is unclear. The manuscript does not provide sufficient context to understand what the derived formulas change about our theoretical understanding of those methods or what they imply for applications. \n\nAs it stands, I am worried that the submission would have little impact. I believe the manuscript would benefit from major revisions to be of interest to the community and my initial recommendation is a rejection.\n\nThe major issues that need to be adressed are:\n- The technical definitions of the terms introduced in the work, like \"equivalence of order 2\", and the significance of the results, need to be stated in the main text. That two methods produce similar updates in terms of the overall model is interesting. But the significance of the result depends on what information it gives for the study or the application of the method, and this is currently unclear from the manuscript. For example; \n  - why does the memory $v_k$ in the AdamG* update increase over time (with $\\beta > 1$) if $\\eta\\lambda >2$? Or is it a setting that is not supported by the assumptions in Thm. 2? \n  - What is the effect of L2 regularization if the complexity of the model is independent of the magnitude of the weight vector? \n  - Why is $\\beta = 1$ if there is no L2 regularization but $\\beta < 1$ if $\\eta\\lambda < 2$? \n  - Does it tell us anything about the difference between an Adagrad-style complete sum and Adam-style exponential moving average?\n- The claimed contribution that SGD behaves like Adam without momentum is over-selling the results of section 3. Those results show that a gradient step is approximately a normalized gradient step projected on the unit ball. While this version might have similarities with Adam, this is not what the stated contribution would imply to most readers.\n- The message of the paper is obscured by excessive formality. It might of course useful for a subset of readers to state the result in terms of topological equivalence to quotient manifolds, isomorphisms or to relate the results to the canonical metric on the sphere. But it makes the paper harder than necessary for the average reader, especially when technical terms, like the canonical metric, do not appear after their introduction. I strongly advise the authors to keep the main message of the paper accessible. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers recommend rejecting or weak reject. The studied problem is interesting, but as one reviewer pointed out, it is not that clear how this work changes our theoretical understanding of those methods or what they imply for applications. Overall, I feel this work is on the borderline (probably it deserves higher score than the current score), but probably below the acceptance bar at the current form. 
Decision: Reject