AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
Authors: Byeongho Heo
Sanghyuk Chun
Seong Joon Oh
Dongyoon Han
Sangdoo Yun
Gyuwan Kim
Youngjung Uh
Jung-Woo Ha
Venue: ICLR 2021 Poster (2021)
Keywords: momentum optimizer
scale-invariant weights
normalize layer
effective learning rate

Abstract: Normalization techniques, such as batch normalization (BN), are a boon for modern deep learning. They let weights converge more quickly with often better generalization performances. It has been argued that the normalization-induced scale invariance among the weights provides an advantageous ground for gradient descent (GD) optimizers: the effective step sizes are automatically reduced over time, stabilizing the overall training procedure. It is often overlooked, however, that the additional introduction of momentum in GD optimizers results in a far more rapid reduction in effective step sizes for scale-invariant weights, a phenomenon that has not yet been studied and may have caused unwanted side effects in the current practice. This is a crucial issue because arguably the vast majority of modern deep neural networks consist of (1) momentum-based GD (e.g. SGD or Adam) and (2) scale-invariant parameters (e.g. more than 90% of the weights in ResNet are scale-invariant due to BN). In this paper, we verify that the widely-adopted combination of the two ingredients lead to the premature decay of effective step sizes and sub-optimal model performances. We propose a simple and effective remedy, SGDP and AdamP: get rid of the radial component, or the norm-increasing direction, at each optimizer step. Because of the scale invariance, this modification only alters the effective step sizes without changing the effective update directions, thus enjoying the original convergence properties of GD optimizers. Given the ubiquity of momentum GD and scale invariance in machine learning, we have evaluated our methods against the baselines on 13 benchmarks. They range from vision tasks like classification (e.g. ImageNet), retrieval (e.g. CUB and SOP), and detection (e.g. COCO) to language modelling (e.g. WikiText) and audio classification (e.g. DCASE) tasks. We verify that our solution brings about uniform gains in performances in those benchmarks. Source code is available at https://github.com/clovaai/adamp

Published as a conference paper at ICLR 2021
ADAMP: SLOWING DOWN THE SLOWDOWN FOR MO- MENTUM OPTIMIZERS ON SCALE-INVARIANT WEIGHTS
Byeongho Heo*:
Sanghyuk Chun 1.2
Seong Joon Oh
Dongyoon Han
Sangdoo Yun!
Gyuwan Kim
Youngjung Uh
Jung-Woo Hal.2 1.2
Naver AT Lab", Naver Clova? Applied Information Engineering, Yonsei University Equal contribution", " Works done at Naver Al Lab
ABSTRACT
Normalization techniques, such batch normalization (BN), boon for mod- ern deep learning. They let weights converge more quickly with often better gener- alization performances. has been argued that the normalization-induce scale in- variance among the weights provides an advantageous ground for gradient descent (GD) optimizers: the effective step sizes are automatically reduced over time, sta- bilizing the overall training procedure. It is often overlooked, however, that the additional introduction of momentum in GD optimizers results in a far more rapid reduction in effective step sizes for scale-invariant weights, phenomenon that has not yet been studied and may have caused unwanted side effects in the current practice. This is crucial issue because arguably the vast majority of modern deep neural networks consist of (1) momentum-barn
GD (e.g. SGD or Adam) and (2) scale-invariant parameters (e.g. more than 90% of the weights in ResNet are scale-invariant due to BN). In this paper, we verify that the widely- com- bination of the two ingredients lead to the premature decay of effective step sizes and sub-optimal model performances. We propose a simple and effective rem- edy, SGDP and AdamP: get rid of the radial component, or the norm -ere-incraasing direction, at each optimizer step. Because of the scale invariance, this modifi- cation only alters the effective step sizes without changing the effective update directions, thus enjoying the original convergence properties of GD optimizers. Given the ubiquity of momentum GD and scale invariance in machine learning, we have evaluated our methods against the baselines on 13 benchmarks. They range from vision tasks like classification (e.g. ImageNet), retrieval (e.g. CUB and SOP), and detection (e.g. COCO) to language modelling (e.g. WikiText) and audio classification (e.g. DCASE) tasks. We verify that our solution brings about uniform gains in performances in those benchmarks. Source code is available https:/ / thub. com/ c olaav addmp..
INTRODUCTION
Normalization techniques, such as batch normalization (BN) (loffe & Szegedy, 2015), layer nor- malization (LN) (Ba et al., 2016), instance normalization (IN) (Ulyanov et al., 2016), and group normalization (GN) (Wu & He. 2018), have become standard tools for training deep neural network models. Originally proposed to reduce the internal covariate shift (Ioffe & Szegedy, 2015), normal- ization methods have proven to encourage several desirable properties in deep neural networks, such as better generalization (Santurkar et al., 2018) and the scale invariance (Hoffer et al., 2018).
Prior studies have observed that the cermalizationinndueee scale invariance of weights stabilizes the convergence for the neural network training (Hoffer et al., 2018; Arora et al., 2019; Kohler et al., 2019; Dukler al., 2020). We provide sketch of the argument here. Given weights w and an input ær, we observe that the normalization makes the weights become cale-invariian:
Norm(w" xx) Norm(cw> Tx)) Ve 0.
(1)
The resulting equivalence relation among the weights lets us consider the weights only in terms of their l no-nommalized vectors w = 2
the sphere gdd-1 {v Ed ||l|| 1}. We refer to

---

Published as a conference paper at ICLR 2021
Sd-1 as the effective space, opposed to the nominal space R where the actual optimization algo- rithms operate. The mismatch between these spaces results in the discrepancy between the gradient descent steps on Rd and their effective steps on S Specifically, for the gradient descent updates, the effective step sizes | [Ai == || [i,+1 - mell2 are the scaled versions of the nominal step | w. 2|2 by the factor w. la (Hoffer et al., 2018). Since || ||2 increases during training (Soudry et al., 2018; Arora et al., 2019), the effective step sizes | ||2 decrease as the optimization progresses. The automatic decrease in step sizes stabilizes the convergence of gra- dient descent algorithms applied on models with normalization layers: even if the nominal learning rate is set a constant, the theoretically optimal convergence rate is guaranteed (Arora al., 2019).
sizes ||
In this work, we show that the widely used momentum-based gradi- ent descent optimizers (e.g. SGD and Adam (Kingma & Ba, 2015)) decreases the effective step size Aw; even more rapidly than the momentum-less counterparts considered in (Arora al., 2019).
Wi
Unit sphere
This leads to a slower effective convergence for Wú and poten- tially sub-optimal model performances. We illustrate this effect on 2D toy optimization problem in Figure 1. Compared to "GD", "GD+momentum" much faster in the nominal space R², but the norm growth slows down the effective convergence in S1, reducing the acceleration effect of momentum. This phenomenon is not con- fined to the toy setup, for example, 95.5% and 91.8% of the param- eters of the widely- used ResNet18 and ResNet50 (He et al., 2016) scale-invariant due to BN. The majority of deep models nowa- days are trained with SGD or Adam with momentum, And yet, our paper first delve into the issue in the widely-used combination of seale-invarian parameters and momentum based optimizers.
Gradient descent (G)
GD momentum projection
(ours)
Optimal solution
We propose simple solution slow down the decay of effective step sizes while maintaining the step directions of the original opti- mizer in the effective space. At each iteration of momentum noomentum-based based gradient descent optimizer, we propose to project out the radial component (i.e. component parallel to w)) from the update, thereby reducing the increase in the weight norm over time. Because of the scale invariance, the procedure does not alter the update direction in the effective space; it only changes the effective step sizes. We can observe the benefit of our optimizer in the toy setting in Fig- ure 1. "Ours" suppresses the norm growth and thus slows down the effective learning rate decay, allowing the momentum-accee -cccleeraed convergence in 2 to be transferred to the actual space St, "Ours"
GD momentum
Itorations
300 400
500
Optimizer tra- jectories. Shown is the w, for the optimization problem w=" max [w Trajectories start from wo towards the opti- mal solution w The problem is invariant to the scale of w. Video version the attached code.
Figure
converges most quickly and achieves the best terminal objective value. We do not discourage the use of momentum-batte based optimizers; momentum often an indispensable ingredient that enables best performances by deep neural networks. Instead, we propose to use our method that helps momen- tum realize its full potential by letting the acceleration operate on the effective space, rather than squandering it on increasing norms to no avail.
The projection algorithm simple and readily applicable to various optimizers for deep neural net- works. We apply this technique on SGD and Adam (SGDP and AdamP, respectively) and verify the slower decay of effective learning rates as well as the resulting performance boosts over diverse set of practical machine learning tasks including image classification, image retrieval, object detection, robustness benchmarks, audio classification, and language modelling.
As side note, we have identified certain similarities between our approaches and Cho & Lee (2017), (). Cho & Lee (2017) have considered performing the optimization steps for the scale- ccale-invariant param- eters the spherical manifold. We argue that our approaches operate on the ambient Euclidean space, and are more practical. See Appendix $G.1 for a more detailed argumentation based conceptual and empirical comparisons.
conceptually different, ours
PROBLEM
Widely-used normalization techniques (Ioffe & Szegedy, 2015; Salimans Kingma, 2016; Ba et al., 2016; Ulyanov al., 2016; Wu & He, 2018) in deep networks result in the scale invariance for
2

---

Published as a conference paper at ICLR 2021
weights. We show that the introduction of momentum in prddient-deseent (GD) optimizers, when applied on such scale-inve invariant parameters, decreases the effective learning rate much more rapidly. This phenomenon has not yet been studied in literature, despite its ubiquity. We suspect the resulting early convergence may have introduced sub-optimality in many SGD and Adam-trained models across machine learning tasks. The analysis motivates our optimizer in g3.
2.1 NORMALIZATION LAYER AND SCALE INVARIANCE
For tensor z E | of rank we define the normalization operation along the axes & e {0, 1) 1)(2..)+)) as (o)m --#+(2) where jAk: o are the mean and standard deviation functions along the axes k. without axes reduction (to allow broadcasted operations with æ). Depending on k,
mp(a)
Norm, includes special cases like batch normalization (BN) (Ioffe & Szegedy, 2015).
For function g(u), we say that g is scale invariant if g(cu) g(u) for any с > 0. We then observe that Norm(-) is scale invariant. In particular, under the context of neural networks,
Norm(w x) Norm((cw)
for any c > O, leading to the scale invariance against the weights w preceding the normaliza- tion layer. The norm of such weights ||w/|2 does not affect the forward fuo (ax) or the backward V Vwlu fw (a:) computations of neural network layer lw parameterized by w. We may represent the
scale-invariant weights via their fz-ormallized vectors v w Sd-1 (i.e.
Tw
2.2 NOTATIONS FOR THE OPTIMIZATION STEPS
See the illustration on the right for the summary of notations describing an optimization step. We write a gradient descent (GD) algorithm as:
Awr+1ll2
W <- w: W/P:
(3)
Wi+1
1/pt
where 1) > 0 is the user-defined learning rate. The norm of the difference ||Aw, ll2 == ||w,+1 well2 = 17 p: 1l is referred to as the step size. When p V w. (w)) equation 3 is the vanilla GD algorithm. Momentum- based variants have more complex forms for p.
| ll
W
W
In this work, we study the optimization problem in terms of the & nor- malized weights in gd-1, as opposed to the nominal space Rd, As the result of equation 3, an effective optimization step takes place in Sd-1; Aw == W wt We refer to the effective step size AA8++1l12.
2.3 EFFECTIVE STEP SIZES FOR VANILLA GRADIENT DESCENT (GD)
We approximate the effective step sizes for the scale-invariant w under the vanilla GD algorithm. We observe that the scale invariance (cun) = f(w) leads to the orthogonality:
0f(cw) w T fw dc
(4)
For example, the vanilla GD update step p V we establish the effective step size for w on gd
w f (w) is always perpendicular to w. Based on this,
A +
W
W
A l|a
(5)
[
|| l12
[w0+1 ll
|| llz
2
where the approximation assumes [ 1l2
wwll2 o(n), which holds when Pt L as in the
vanilla GD. We have thus derived that the effective step size on gd-1 is inversely proportional to the weight norm, in line with the results in Hoffer et al. (2018).
Having established the relationship between the effective step sizes and the weight norm a scale- invariant parameters (s.i.)), derive the formula for its growth under the vanilla GD optimization.
Lemma 2.1 (Norm growth by GD, Lemma 2.4 in Arora et al. (2019)). For s.i.p. w and the vanilla GD, where p. V (w..) |0
| " 11pp:
3

---

Published as a conference paper at ICLR 2021
The lemma follows from the orthogonality in equation 4. It follows that the norm of ocale-invarian parameter | 12 is monotonically increasing and consequently decreases the effective step size for w. Arora al. (2019) has further shown that GD with the above adaptive step sizes converges to stationary point at the theoretically optimal convergence rate O(T-1/2) under fixed learning rate.
2.4 RAPID DECAY OF EFFECTIVE STEP SIZES FOR MOMENTUM BASED GD
Momentum is designed to accelerate the convergence of gradient-based optimization by letting w escape high-curvature regions and cope with small and noisy gradients. It has become an indispens- able ingredient for training modern deep neural networks. A momentum update follows:
1 w; T,,t,
Pt 3p-- V L f(w::
for steps 2 O, where 3 E 1) and p-1 is initialized at O. Note that the step direction Pt and the parameter we may not be perpendicular anymore. We show below that momentum increases the weight norm under the scale invariance, even more so than does the vanilla GD.
Lemma 2.2 (Norm growth by momentum). For a s.i.p. w updated via equation 7, we have
| 11 = 1/ + 11 - 2m
3r *0
(8)
R=
Proof is in the Appendix $A. Comparing Lemma 2.1 and 2.2, we notice that the formulation is identi- cal, except for the last term the right hand side of Lemma 2.2. This term is not only non-negative, but also is an accumulation of the past updates. This additional term results the significantly accel- erated increase of weight norms when the momentum used. We derive more precise asymptotic ratio the weight norms for the GD with and without momentum below.
Corollary 2.3 (Asymptotic norm growth comparison). Let [|nn and |m be the weight norms at step I. 2 0, following the recursive formula in Lemma 2.1 and 2.2, respectively. We assume that the norms of the updates ||Pelle for GD with and without momentum are identical for every Z O. We further assume that the sum of the update norms is non-zero and bounded: O < Ei [P:l13 Then, the asymptotic ratio between the two norms is given by:
GDM
2 B
1
as t 20.
(9)
2 GD
200
Proof in the Appendix ŠA. While the identity assumption for |pel12 between GD with and without momentum is strong, the theory is designed to illustrate approximate norm growth ratios between
the algorithms. For popular choice of B = 0.9, the factor is as high as 1 + 23/(1 - 33) = 19. Our observations also applicable Nesterov momentum and momentum-based adaptive optimizers like Adam. We later verify that the momentum induce the increase in weight norms and thus rapidly reduces the effective learning rates in many realistic setups of practical relevance ($3.2 and 84).
METHOD
We have studied the accelerated decay of effective learning rates for scale-invariant weights (e.g. those preceding a normalization layer) under the momentum. In this section, we propose a projection-based solution that prevents the momentum-induced effective step size decreases while not changing the update directions in the effective weight space Sd-1
3p
Momentum
3.1 OUR METHOD: PROJECTED UPDATES
Pr
Projection 1h.
We remove the accumulated error term in Lemma 2.2, while retain- ing the benefits of momentum, through simple modification. Let (.) be a projection onto the tangent space of w:
Ours
I (p.)
Gradient V (wr,)
Ih (x:
(w x)
(10)
We apply (.). to the momentum update (equation 7) to remove the radial component, which accumulates the weight norms without
Figure 2. Vector directions of the gradient, momentum, and ours.
4

---

Published as a conference paper at ICLR 2021
contributing to the optimization. Our modified update rule is:
u+++1 tt
79t,
ww. (p:)
if cos wi fwf(w)) < ( dd)(
(11)
q:
Pt
otherwise
where cos(a, b) bl is the cosine similarity. Instead of manually registering weights pre- ceding normalization layers, our algorithm automatically detects scale invariances with the cosine similarity for user convenience. In all experiments considered, we found ā = 0.1 to be sufficiently small to precisely detect orthogonality and sufficiently large to recall all scale-invariant weights (Appendix (C): we suggest future users to use the same value. The proposed update rule makes scale-invariant parameter w perpendicular to its update step q. It follows then that the rapid weight norm accumulation shown in Lemma 2.2 is alleviated back to the vanilla gradient descent growth rate in Lemma 2.1 due to the orthogonality:
||l||||||
|| 112 = w. 12 17 11 - 1 1 + r/ 1p:12
(12)
where inequality follows from the fact that g: IT., (p:) and IT. projection operation. Al- though the updates pt are not identical between equation 8 and equation 12, we observe that after our orthogonal projection, the updates no longer get accumulated as in the last term of equation equa- tion 8. We emphasize that this modification only alters the effective learning rate while not changing the effective update directions, as shown in the below proposition.
Proposition 3.1 (Effective update direction after projection). Let w == un - TpP: and w w; m, (p:) be original and projected updates, respectively. Then, the effective update after the projection w lies on the geodesic on gd-1 defined by Wr and wi+1-
:
Proof in Appendix. As such, we expect that our algorithm inherits the convergence guarantees of GD. As the convergence rate, conjecture that similar analysis by Arora et al. (2019) holds.
The proposed method is readily adaptable to existing gradient- based optimization algorithms like SGD and Adam. Their modifications, SGDP and AdamP, shown in Algorithms 1 and 2, respec- tively (Modifications are colorized). In practice, we consider two types of scale-invariance: layer- wise (e.g.. by LN) and channel-wise (e.g., by BN, IN) invariance. Hence, the weight projection in equation 1 are performed either channel- or layer-wise. Our optimizers incur only 8% extra training time on top of the basclines (ResNet18 on ImageNet classification).
Algorithm 1: SGDP
Algorithm 2: AdamP
Require: Learning rate 1) > 0, momentum
Require: Learning rate 1) O, momentum
B > O, thresholds 5, e >> 0.
0 < 31, B2 < 1. thresholds 6, > 0.
1: while u, not converged do
1: while u. not converged do
2: Pt +- .8p--1 + Jew,))
2: m, + 3,m,-1 +
31 V wen( ((
3: Compute qt with equation 11.
Ut 1 -20011111
4: Wt+11 f- W; - 7/
4: P: + +-m//(y ( 02 +)
5: end while
5: Compute Jt with equation 11.
6: Wi+1 t-
77
end while
3.2 EMPIRICAL ANALYSIS OF EFFECTIVE STEP SIZES AND THE PROPOSED PROJECTION
So far, we have studied the problem ($2), namely that the momentum accelerates effective step sizes for scale-invarian weights, as well as the corresponding solution ($3.1) only at conceptual level. Here, we verify that the problem does indeed exist in practice and that our method successfully addresses on both synthetic and real-world optimization problems.
Synthetic simulation. While examples of scale-invariant optimization problems abound in mod- em deep learning BN), they are scant in popular toy optimization objectives designed for sanity
checks. We use Rosenbrock function (Rosenbrock, 1960): h(11,2)) (1 21)² 3 + 300(x;2
xf)² (Figure 3). As our verification requires scale invariance, we define a 3D Rosenbrock function
by adding redundant radial axis r, while treating the original coordinates (2)) as the polar
5

---

Published as a conference paper at ICLR 2021
Gradient Descent (GD)
GD weight
GD projection (ours)
30g
2.5
2.5
2.5
2.0
2.0
2,0
1.0
1.0
1.0
0.5
0.5
0.0
0.0
0.0
-0.5
-109
-1.03
-10g
Figure 3. scale-invariant Rosenbrock. Three optimization algorithms are compared. Upper row: loss sur- face and optimization steps. Lower row: norm r of parameters over the iterations. Results for Adam variants in Appendix I..I.
angles (vh, Φ) of the spherical coordinates, resulting in the function h(r, rb, d) h(t,, x)) h is opti- mized in the 3D space with the Cartesian coordinates. We describe the full details in Appendix JB.
In Figure 3, we compare the trajectories for optimizers the spherical coordinates (v, Φ). We com- pare the baseline momentum GD and our projection solution. We additionally examine the impact of weight decay (WD), since a careful choice of WD another way regularize the norm growth. We observe that the momentum GD does not converge sufficiently to the optimum. The slowdown explained by the decreased effective step sizes on & due to the increase the parameter norm (r = 1.0 - 3.2). Careful tuning of WD partially addresses the slow convergence (momentum GD WD trajectory) by regularizing the norm growth, but WD) is still unable preclude the initial surge in the weight norm (r 1.0 2.53). In practice, addressing the problem with WD is even less attractive because WD is often a sensitive hyperparameter (see the following experiments). On the other hand, our projection solution successfully subdues the weight norm growth (r 1.0 1.12), ensuring undiminished effective step sizes and a faster convergence to the optimum.
Real-world experiments. We ver- ify the surge of weight norm and sub- optimality of model performances in momentum-traiine deep networks on real-world datasets: ImageNet clas- sification with ResNet-18 and mu- sic tagging (Law et al., 2009) with Harmonic CNN (Won al., 2020a). See Table 1 for the analysis. In all experiments, our projection solutions (SGDP, AdamP) restrain the weight norm growth much better than the vanilla momentum methods. For ample, Adam-induced norm increase (+4.21) is 13.2 times greater than that of AdamP (+0.32) in the music tagging task. In ImageNet SGD ex- periments, we observe that careful choice of weight decay (WD) mitigates the norm increases, but the final norm values and perfor- mances are sensitive to the WD value. On the other hand, our SGDP results in stable final norm values and performances across different WD values, even at WD= 0. We observe that under the same learning setup models with smaller terminal weight norms tend to obtain improved perfor- mances. Though is difficult to elicit a causal relationship, we verify in $4 that SGDP and AdamP bring about performance gains in diverse set of real-world tasks. More analysis around the norm growth and the learning curves are in Appendix $F. We also conduct analysis on the momentum coefficient in Appendix HH.3
Table 1. Analysis of optimizers on real-world tasks. The norm values and final performances for different tasks and optimizers are shown. Norm:: : norm first epoch. Norm ust norm at last epoch. Score: accuracy for ImageNet and AUC for music tagging.
Task
Optimizer
WD
Norm'
Norm
Score (1)
SGD
0
1.35
4.34 (+2.99)
67.94
SGD
10-6
1.35
4.08 (42.73)
67.89
SGD
10-5
1,33
2.70 (+1.36)
69.06
ImageNet ResNet-18 18
SGD
10-4
1.21
1.02 (90.1)
70.43
SGDP
0
1.25
2.10 (40.85)
70.21
SGDP
10-6
1.25
1.96 (40.72)
70.27
SGDP
10
1.25
1.19 (-0.04)
70.57
AdamW
0
1.70
10.41 (48.71)
68.38
AdamP
1.29
3.02 (+1.73)
70.55
Music tagging
AdamW 0
7.35
11.56
(+4.21)
90.86
HCNN
AdamP
7.30
7.62
(40.32)
91.19

---

Published as a conference paper at ICLR 2021
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of our projection module for training scale- invariant weights with momentum-based optimizers. We experiment and datasets. From the image domain, we show results on ImageNet classification (64.1, 5D.2, $D.3), object detection (84.2), and robustness benchmarks ($4.3, SD.1). From the audio domain, we study music tagging, speech recognition, and sound event detection ($4.4). We further show the results when the scale invariance is artificially introduced to a network with no ccale-invariant parameters (e.g. Transformer-Xi (Dai et al., 2019)) in $4.5. To diversify the root cause of scale invariances, we consider the case where it stems from the f2 projection of the features, as opposed to the statistical normalization done in e.g. BN, in the image retrieval experiments ($4.6). In the above set of experi- ments totaling > 10 setups, our proposed modifications (SGDP and AdamP) bring about consistent performance gains against the baselines (SGD (Sutskever et al., 2013) and AdamW (Loshchilov & Hutter, 2019)). We provide the implementation details in the Appendix JE and the standard deviation values for the experiments in Appendix HH.2.
various real-world tasks
4.1 IMAGE CLASSIFICATION
Batch normalization (BN) and momentum-based optimizer are standard techniques to train state- of-the-art image classification models (He al., 2016; Han al., 2017: Sandler et al., 2018; Tan & Le. 2019; Han et al., 2020). We evaluate the proposed method with ResNet (He et al., 2016), one of the most popular and powerful architectures on ImageNet, and MobileNetV2 (Sandler al., 2018), relatively lightweight model with ReLU6 and depthwise convolutions, on the ImageNet- IK benchmark (Russakovsky et al., 2015). For ResNet, we employ the training hyperparameters in (He et al., 2016). For MobileNetV2, we have searched for the best hyperparameters, as it is generally difficult to train it with the usual settings. Recent researches have identified better training setups (Cubuk et al., 2020) where the cosine-annealed learning rates and larger training epochs (100 epochs or 150 epochs) than 90 epochs (He et al., 2016) are used. We use those setups for all experiments in this subsection.
Our optimizers are compared against their corresponding baselines in Table 2. Note that AdamP is compared against AdamW (Loshchilov & Hutter, 2019), which has closed the gap between Adam and SGD) performances on large-scale benchmarks. Across the spectrum of network sizes, our opti- mizers outperform the baselines. Even when the state-of-the-art CutMix (Yun et al., 2019) regular- ization is applied, our optimizers introduce further gains. We provide three additional experiments on EfficientNet (Tan & Le. 2019) ($D.2), the large-batch training scenario (GD.3) and the compar- ison in the same computation cost ($H.4) to demonstrate the benefit of AdamP on diverse training setups. There, again, our methods outperform the baselines.
Table 2. ImageNet classification. Accuracies of state-of-the-as networks trained with SGDP and AdamP.
Architecture
params
SGD
SGDP (ours)
Adam
Adam W
AdamP (ours)
MobileNetV2
3.5M
71.55
72.09 (+0.54)
69.32
71.21
72.45 (+1.24)
ResNet18
11.7M
70.47
70.70 (40.23)
68.05
70.39
70.82 (+0.43)
ResNet50
25.6M
76.57
76.66 (+0.09)
71.87
76.54
76.92 (40.38)
ResNet50 + CutMix
25.6M
77.69
77.77 (40.08)
76.35
78.04
78.22 (40.18)
4.2 OВJECT DETECTION
Object detection is another widely-used real-world task where the models often include normaliza- tion layers and are trained with momentum-ber based optimizers. We study the two detectors Center- Net (Zhou et al., 2019) and SSD (Liu et al., 201 to verify that the proposed optimizers are also ap- plicable to various objective functions beyond the classification task. The detectors are either initial- network (offi-
Table MS-COCO object detection. Average pre- cision (AP) scores CenterNet (Zhou al., 2019) and SSD (Liu et al., 201 6a) trained with Adam and AdamP optimizers.
Model
Initialize
Adam
AdamP (ours)
CenterNet
Random
26.57
27.11 (+0.54)
CenterNet SSD SSD
ImageNet
28.29
29.05 (+0.76)
ized with the ImageNet-pretraing cial Py Torch models) or trained from scratch, in order to separate the effect of our method from that
Random
27.10
27.97 (+0.87)
ImageNet
28.39
28.67 (40.28)
7

---

Published as a conference paper at ICLR 2021
of the pretraining. & (He et al., 2016) and VGG16 BN (Simonyan & Zisserman, 2015) are used for the CenterNet and SSD backbones, respectively. In Table 3, we report average precision per- formances based the MS-COCO (Lin et al., 2014) evaluation protocol. We observe that AdamP boosts the performance against the baselines. demonstrates the versatility of our optimizers.
4.3 ROBUSTNESS
Model robustness an emerging problem in the real-world applications and the training of robust models often involve complex optimization prob-
adversarial
Test-wourrcy
lems (e.g. minimax). We examine how our opti- mizers stabilize the complex optimization. We con- sider two types of robustness: adversarial (below)
19
Adam Istan acouracyl
Adam jatracked accurscyl
and cross-bias (Bahng et al., 2020) (Appendix 3D.I).
AdamP spannar acturacy)
20-
AdamP getracced sccuracyl
Adversarial training alternatively optimizes min- imax problem where the inner optimization is an adversarial attack and the outer optimization is the standard classification problem. Adam is commonly employed, in order handle the complexity of the optimization (Tsipras et al., 2019; Chun et al., 2019).
75
150
125
150
175
Epoch
PGD Lint adversarial training
We consider solving the minimax optimization prob- lem with our proposed optimizers. We train Wide- ResNet (Zagoruyko & Komodakis, 2016) with the projected gradient descent (PGD) (Madry et al., 2018) on CIFAR-10. We use 10 inner PGD iterations and 80/255 for the L2 PGD and & 4/255 for the Lo PGD. Figure 4 shows the learning curves of Adam and AdamP. By handling the effective step sizes, AdamP achieves faster convergence than Adam (less than half the epochs required). AdamP brings more than +9.3 pp performance gap in all settings. We also performed 8/255 and the results are reported in Appendix $E.4.1.
50
40
Adam lottokked accurscyl
acturacy)
30-
(atiacedd accuracy!
25
50
75
100
125
150
175
200
Epoch
Figure 4. Adversarial training, Learning curves by Adam and AdamP.
4.4 AUDIO CLASSIFICATION
We evaluate the proposed optimizer on three audio classification tasks with different physical prop- erties: music clips, verbal audios, and acoustic signals. For automatic music tagging, we the Magna'TagATine ATune (MTAT) benchmark (Law et al., 2009) with 21k samples and 50 tags. Each clip contains multiple tags. We use the Speech Commands dataset (Warden, 2018) for the keyword spot- ting task (106k samples, 35 classes, single label). For acoustic signals, we use the DCASE sound event detection benchmark (Mesaros et al., 2017) (53k samples, 17 tags, multi-labeled).
We train the Harmonic CNN (Won et al., 2020a) on the three benchmarks. Harmonic CNN consists of data-driven harmonic filters and stacked convolutional filters with BN. Audio datasets are usually smaller than the image datasets and are multi-labeled, posing another set of challenges for the opti- mization. Won et (2020a) have trained the network with a mixture of Adam and SGD (Won al., 2019b). Instead of the mixture solution, and AdamP on a validation set. The results are given in Table 4. AdamP shows better performances than the baselines, without having to adopt the complex mixture solution. The results signify the superiority of AdamP for training
have searched the best hyperparameters for the Adam
weights on the audio domain.
Table 4. Audio classification. Results on three audio tasks with Harmonic CNN (Won et al., 2020a).
Music Tagging
Keyword Spotting
Sound Event Tagging
Optimizer
ROC-AUC
PR-AUC
Accuracy
Fl score
Adam + SGD (Won et al., 2019b)
91 2.
45.67
96.08
54.60
AdamW
91.12 91.35 (+0.23)
45.61
96.47
55.24
AdamP (ours)
45.79 (+0.18)
96.89 (+0.42)
56.04 (+0,80)
8

---

Published as a conference paper at ICLR 2021
4.5 LANGUAGE MODELING
Models in language domains, such as the Trans- former (Vaswani et al., 2017), often do not have any scale-invariant weight. The use of layer nor- malization (LN) in the Transformer does not result in the scale invariance because LN is applied right after the skip connection f(w, x) := LN(gwr) (x)+ a) (note the difference from equation? 2). The skip
Table Language Modeling. Perplexity on Wiki-
Text103. Lower is better.
Model
AdamW
AdamP (ours)
Transformer-Xi Transformer-Xl.
23.38
23.26 (-0.12)
23,96
22.77 (-1.19)
connection makes I scale-vuriunt with respect to w. To allow our optimizers to effectively op-
crate on Transformer, we introduce scale invariance artificially through the weight normalization (WN) (Salimans & Kingma, 2016).
We have trained Transformer- XL (Dai et al., 2019) on WikiText- 103 (Merity et al., 2016), As shown in Table 5, AdamP does not significantly outperform AdamW enforcement of scale invariance. WN on its own harms the performance (23.33-233 23.90 for Adam). When AdamP is applied on Transformer-XL. WN, it significantly improves over the AdamW baseline 233.90>220 23,99,, beating the original performance 23.33 by Adam on the vanilla network.
without the explicit
4.6 RETRIEVAL
In the previous experiments, we have examined the scale invariances induced by statistical normal- ization methods (e.g. BN). Here, we consider another source of scale invariance, l projection of features, which induces a scale invariance in the preceding weights. It is widely used in retrieval tasks for more efficient distance computations and better performances. We fine-tune the ImageNet- pretrained ResNet-50 network on CUB (Wah et al., 2011), Cars-196 (Krause et al., 2013), In-Shop Clothes (Liu al., 2016b), and Stanford Online Products (SOP) (Oh Song et al., 2016) benchmarks with the triplet (Schroff et al., 2015) and the Proxy ProxyAnchor (Kim et al., 2020) losses. In Table 6, we observe that AdamP outperforms Adam over all four image retrieval datasets. The results support the superiority of AdamP for networks with & normalized embeddings.
Table 6. Image retrieval. Optimizers are tested on networks with fz-induced scale invariant weights. Recall @ 1.
CUB
Cars- 196
InShop
SOP
Optimizer
Triplet
PA
Triplet
PA
Triplet
PA
Triplet
PA
AdamW
57.9
69.3
59.8
86.7
62.7
85.2
62.0
76.5
AdamP (ours)
58.2 (+0.3)
69.5 (+0.2)
59.9 (+0.2)
86.9 (+0.2)
62.8 (+0.0)
87.4 (+2.2)
62.6 (+0.6)
78.0 (+1.5)
5 CONCLUSION
Momentum-baan based optimizers induce an excessive growth of the scale-invariant weight norms. The growth of weight norms prematurely decays the effective optimization steps, leading to sub-optimal performances. The phenomenon is prevalent in many commonly- mizers (e.g. SGD and Adam) used for training the vast majority of deep models. The widespread use of normalization layers make a large proportion of network weights scale-invariant (e.g. ResNet). We propose a simple and effective solution: project out the radial component from the optimization updates. The resulting SGDP and AdamP successfully suppress the weight norm growth and train a model at an unobstructed speed. Empirically, our optimizers have demonstrated their superiority over the baselines on more than 10 real-world learning tasks.
setup. Momentum-based opti-
ACKNOWLEDGEMENT
We thank NAVER AI LAB colleagues for discussion and advice, especially Junsuk Choe for the internal review. Naver Smart Machine Learning (NSML) platform (Kim et al., 2018) has been used in the experiments.

---

Published as a conference paper at ICLR 2021
REFERENCES
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by hatch normalization. In International Conference on Learning Representations (ICLR), 2019. URL https: wpwpreviewwww.ee/////p/.. net forum? i drrrk---aaee--ee--eee- -A9FX.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint vrXiv:1677006450 2016.
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning, 2020.
Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. In Advances in Neural Information Processing Systems, pp. 5--555225555 2017.
Sanghyuk Chun, Scong Joon Oh, Sangdoo Yun, Dongyoon Han, Junsuk Choe, and Youngjoon Yoo. An empirical evaluation on robustness and uncertainty of regularization methods. ICML Work- shop on Uncertainty and Robustness in Deep Learning, 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.099 09501, 2018
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pp. 702-703, 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: 1901.00000286, 2019.
Timothy Dozat. Incorporating nesterov momentum into adam. In International Conference on Learning Representations (ICLR) workshop, 2016.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(11))2121-2159, 2011.
Yonatan Dukler, Quanquan Gu, and Guido Montúfar. Optimization theory for relu neural networks trained with normalization layers. In International Conference Machine Learning, 2020.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference Learning Representations (ICLR), 2019. URL / opwnreeeeeeeeeeeee/e/ee/eee net wttp:/ew.eeeeeoooodddddddaaea Bocrrmm?dd-Byhhh
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256, 2010.
Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 706.02677 2017.
Dongyoon Han, Jiwhan Kim, and Jummo Kim. Deep pyramidal residual networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Dongyoon Han, Sangdoo Yun, Byeongho Heo, and YoungJoon Yoo. Rexnet: Diminishing represen- tational bottleneck on convolutional neural network. arXiv preprint arXiv:2007.0009: 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv: 1907.000 2019.
10

---

Published as a conference paper at ICLR 2021
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems, 2018.
Gao Huang, Yu Sun. Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646 646-661. Springer, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015.
Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang. Youngkwan Kim, et al. NSML: Meet the MLLaS platform with a real-world case study, arXiv preprint 181.997810.09957, 2018.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, Ming Zhou, and Klaus Neymeyr. Exponential convergence rates for batch normalization: The power of angth-direction decoupling in non-convex optimization. In International Conference on Artificial Intelligence and Statistics, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. Proceedings of the IEEE International Conference on Computer Vision work- shops, 2013.
A. Krizhevsky. Learning multiple layers of features from tiny images. In Tech Report, 2009.
Edith Law, Kris West, Michael Mandel, Mert Bay, and Stephen Downie. Evaluation of algorithms using games: The case of music tagging. In International Society for Music Information Retrieval Conference (ISMIR), 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-222 22224, 1998.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations (ICLR), 2020.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. SSD: Single shot multibox detector. In European Conference on Computer Vision (ECCV), 2016a.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016b.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
11

---

Published as a conference paper at ICLR 2021
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018. URL https:/ https::/openzeview. www.:/ww.eeeeeeeeeeeeeeeeeeeeeee wwt/forummieie IJZIBfZAb.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 16X9.06609.7770 2016.
Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Emmanuel Vincent, Bhiksha Raj, and Tuomas Virtanen. DCASE 2017 challenge setup: Tasks, datasets and baseline system. In Detection and Classification of Acoustic Scenes and Events (DCASE), 2017.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations (ICLR), 2018.
HoHo Rosenbrock. An automatic method for finding the greatest or least value of a function. The Computer Journal, 3(3):175-184, 1960.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211- 252, 2015.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances neural information processing systems, pp. 901- 909, 2016
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chinh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? In Advances in Neural Information Processing Systems. 2018.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19 (1), 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, lla Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958, 2014.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initializa- tion and momentum in deep learning. In International Conference on Machine Learning (ICML), 2013.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint 1mXh:1105..19966 2019.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representa- fions, 2019. URL htttps://pppeeeeeeeeeee net/ wtt.://eooooommmoddsssaaaae
12

---

Published as a conference paper at ICLR 2021
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.0802,, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd dataset. 2011.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:/804.03209% 2018.
Minz Won, Sanghyuk Chun, Oriol Nieto, and Xavier Serra. Automatic music tagging with harmonic cnn. In Late Breaking Demo in the International Society for Music Information Retrieval Conf, 2019a.
Minz Won, Sanghyuk Chun, and Xavier Serra. Toward interpretable music tagging with self- attention. arXiv preprint arXiv: 1906.04972, 2019b.
Minz Won, Sanghyuk Chun, and Xavier Serra. Visualizing and understanding self-attention based music tagging. ICML Workshop on Machine Learning for Music Discovery. 2019c.
Minz Won, Sanghyuk Chun, Oriol Nieto, and Xavier Serra. Data-driven harmonic filters for audio representation learning. In International Conference Acoustics, Speech, and Signal Processing, 2020a.
Minz Won, Andres Ferraro, Dmitry Bogdanov, and Xavier Serra. Evaluation of cnn-based automatic music tagging models. arXiv preprint arXiv:2006.0075/1 2020b.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 03888, 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learn- ing: Training bert in 76 minutes. urXiv preprint arXiv: arXiv:1904.0000 1904.00962 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Interna- tional Conference on Computer Vision (ICCV), 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. The British Machine Vision Conference (BMVC), 2016.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. In International Conference on Learning Representations (ICLR), 2019a.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey Hinton. Lookahead optimizer: k steps forward, step back. In Advances in Neural Information Processing Systems, 2019b.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. In arXiv preprint arXiv:1904.07850, 2019.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvomek,, Xenophon Pa- pademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in Neural Information Processing Systems, 33, 2020.
13

---

Published as a conference paper at ICLR 2021
APPENDIX
This document provides additional materials for the main paper. Content includes the proof' (5A), detailed experimental setups ($B and {E), and the additional analysis on the learning rate scheduling and weight decay ($F).
A PROOFS FOR THE CLAIMS
We provide proofs for Lemma 2.2, Corollary 2.3, and Proposition 3.1 in the main paper.
Lemma A.1 (Monotonic norm growth by the momentum). For scale-invariant parameter w up- dated via equation 7, we have
we++ = / + 1p111 - 2n² Σ
3 1x
(A.1)
k=
Proof. From equation 7, we have
| 1|2 ||2x + 1/=1 p: l1 21yw Pt.
(A.2)
remains to prove that w, P. = -1 21 gr-k [px 1.. We prove by induction on 2 0.
First, when = 0, we have wn Po u70
V ( = 0 because of equation 4.
Now, assuming that w, p, - Σ r k=0 Br
pp l/
have
w,-1 Pr+1 Wr+1 (Bp, + V
Bw
Dr B(w,, -mpp) Pr
(A.3)
-n Σ
8 1 B1llp+12
3r gr-k+1
|
(A.4)
k=0
which completes the proof.
Corollary A.2 (Asymptotic norm growth comparison). Let mm GD [l2 and || w GDDD2
the weight norms at step t 2 0, following the vanilla gradient descent growth (Lemma 2.1) and momentum- based gradient descent growth (Lemma 2.2), respectively. We assume that the norms of the updates l for GD with and without momentum are identical for every t > 0. We further assume that the sum of the update norms is non-zero and bounded: O
Σizo 1|pell < 00. Then, the asymptotic
ratio between the two norms is given by:
GDM
12
We
2
+
as +0000 00.
(A.5)
| GD
W
12
1
B
Proof. From Lemma 2.1 and Lemma 2.2, we obtain
w
GD
1
||oo| 1
1pxl12
(A.6)
1-1
1-1-k
GDM
l
|1 12 = n²
Σ
1px1
+ 2n7
Σ
gd
1ipx122.
(A.7)
k=0
k=
I=1
Thus, the corollary boils down to the claim that
Eieo k= Et= A
B
F
1
(A.8)
Elzo Ab
B
where A
|pal12
Let > O. We will find large-enough that bounds Fr around
T-3 by constant multiple of
14

---

Published as a conference paper at ICLR 2021
We first let T be large enough such that
Σ
S
(A.9)
42741
which is possible because E120
A, < c0. We then let T be large enough such that
T
8
(A.10)
3
T max A&
1=1
which is possible due to the convergence of the geometric sum and the boundedness of A (because its infinite sum bounded).
We then define i = T + T and break down the sums in Fe as follows:
Σ
E
-k g0) Ak
T+T"
Et=r"-k 3)) A
kK=TI
F
(A.11)
Σ AK
Ek=T41 AB
T+T
Ck=0 k=
T-B + ri(e))
A + r2(e)
(A.12)
Ek-o A + r3(()
9
T
Ak + T max; (cri(e) r2(e)
kk=
(A.13)
Σ A + ra(e)
where T1, r2, and 13. are the residual terms that are bounded as follows:
[n(()) - maxx A
(A.14)
by equation A. and
|n2(e)] < (1 11-B) -) and [rs(e)] Se
(A.15)
3
by equation A.9. It follows that
B
( + TT max: (Eri(e)
+ r2(e)
Fg
<
(A.16)
Σ Aa +r((c)
1
8
1-8
+
+1
(A.17)
Σ k= A4
1 3
3
8
1-8 8
<
+1)
(A.18)
M
8
due to the triangular inequality and the positivity of r3. M > 0 is suitable constant independent of T.
Proposition A.3 (Effective update direction after projection). Let wi+ := w. - Tppt and w == w: nTI, (pt) be original and projected updates, respectively. Then, the effective update after the projection wf,, lies on the geodesic on Sd-1 defined by wh and wi+
Proof. The geodesic defined by wn and witt can be written as $d-1 span(w;, Pt). Thus, w. w,-lll,, (pt)
suffices to show that wot1 E span(ww, p:). Indeed, we observe that wi = Iw, - gll (Pelll span(w:, P:) because TTo, (p:) = Pt (w) Pe) ur)
15

---

Published as a conference paper at ICLR 2021
B TOY EXAMPLE DETAILS
2D toy example in Figure We describe the details of the toy example in Figure 1. We solve the following optimization problem:
w
min 1
(B.1)
11w11 12
where w and w* are 2-dimensional vectors. The problem is identical to the maximization of the cosine similarity between and w² We set the w* to (0, -1) and the initial (0.001, 1).
This toy example has two interesting properties. First, the normalization term makes the optimal w for the problem not unique: if w" is optimal, then cw* is optimal for any c > 0. In fact, the cost function is scale-invariant, Second, the cost function is not convex.
As demonstrated in Figure and videos attached our submitted code, the momentum gradient method fails to optimize equation B.1 because of the excessive norm increases. In particular, our simulation results show that a larger momentum induces larger norm increase (maximum norm 2.93 when momentum is 0.9, and 27.87 when momentum is 0.99), as shown in the main paper 5 2.4. On the other hand, our method converges most quickly, among the compared methods, by taking advantage of the momentum-induct accelerated convergence, while avoiding the excessive norm increase.
3D spherical toy example in Figure 3 We employ 3D Rosenbrock function (Rosenbrock, 1960):
h(r, v 6) (1 w² + 300( 2 in the spherical coordinate (r, th, ø). Since we are mapping
2D space to spherical surface, (v, Φ) is defined on the hemisphere. Thus, / 2 I (w). d) S 2 Based on the required range of problem f. scalar e can multiplied each angles f(e)) oΦ) to adjust angle scale to problem scale, where we choose
1.5 in the experiments. The initial point is (cp, cΦ) = (-2, 2) above the unit sphere r = 1, and the minimum point is (co) cØ) = (1,
Instead of optimizing h in the spherical coordinate, we optimize the toy example in the Cartesian coordinate (1r, 2) by computing minz,y.2 M(r Ø) = min.,y, M(T(x,,,,)) 2)) We employ the spherical transform T (I, U, (r, 1/, ()) as follows:
+ cos (x/=), sin
(B.2)
For all optimizers, we set the momentum to 0.9, and we have exhaustively searched the optimal initial learning rates between 0.001 and 0. 1. The learning rates are decayed by linearly at every iteration.
C 8 SENSITIVITY
Table B.I. 5 sensitivity. We measured scale-variant and scale-invariant parameter detection performance AdamP, AdamP consistently shows high perfor- mance over a wide range of values.
Based on ImageNet training with ResNet18, we analyzed the cosine similarity of scale-invariant parameter and scale-variant parameter, and veri- fied the sensitivity of 8. We first measured the co- sine similarity between the gradient and weights (Eq. 11). As results, the cosine similarities for scale-variant weights are 10.0028, 0.5660] com- pared |5.5 10 10 4.4 x 10 10-6] for scale-
Scale-variant
Scale-invatien detection
5
Accuracy
detection
1
86.64%
100%
70.74
invariant ones (99% confidence intervals). Be-
0.2
100%
100%
70.83
0.1
100%
100%
70.81
cause of this large gap, our methods are stable on
0.05
100%
100%
70.78
a wide range of 8 values. We also measured scale-
0.02
100%
99.944
70.81
variant and scale-invariant parameter detection ac-
0.01
100%
97.73%
70.66
curacy based Eq. 11 with various 8 values. The results are shown in Table B. In a wide range of delta values, AdamP perfectly discriminates the variant and scale-invare inve--nvariant parameters. Also, AdamP consistently shows high performance. Therefore, AdamP is not sensitive to the value and 0.1 suitable to separate scale- variant and scale-invariant parameters.
16

---

Published as a conference paper at ICLR 2021
D ADDITIONAL EXPERIMENTS
D.1 CROSS-BIAS ROBUSTNESS
Cross-bias generalization problem (Bahng et al., 2020) tackles the scenario where the training and test distributions have different real-world biases. This often occurs when the training-se set biases provide an easy shortcut to solve the problem. Bahng et al. (2020) has proposed the ReBias scheme based on the minimax optimization, where the inner problem maximizes the independence between an intentionally biased representation and the target model of interest and the outer optimization solves the standard classification problem. As for the adversarial training, Bahng et al. (2020) has employed the Adam to handle the complex optimization.
We follow the two cross-bias generalization benchmarks proposed by Bahng et al. (2020). The first benchmark is the Biased MNIST, the dataset synthesized by injecting colors on the MNIST background pixels. Each sample colored according to a pre-defined class-color mapping with probability p. The color is selected at random with 1 P chance. For example, a completely biased dataset and p = 0.1 leads to an unbiased dataset. Each model is trained on the p-biased MNIST and tested on the unbiased MNIST. We train stacked convolutional network with BN and ReLU. The second benchmark is the 9-Class ImageNet representing the real-world biases, such as textures (Geirhos et al., 2019). The unbiased accuracy is measured by pseudo-labels generated by the texture clustering. We also report the performance on ImageNet-A (Hendrycks et al., 2019), the collection of failure samples of existing CNNs.
1.0 leads
In Table D.I, we observe that AdamP outperforms Adam in all the benchmarks. AdamP a good alternative to Adam for difficult optimization problems applied on scale-invariant parameters.
Table D.I. Real-world bias robustness. Biased MNIST and 9-Class ImageNet
benchmarks with Re-
Bias (Bahng et al., 2020).
Biased MNIST Unbiased acc.
9-Class ImageNet
Optimizer
.999
.977
.955
.990
avg.
Biased
UnBiased ImageNet-A
Adam
22.9
63.0
74.9
87.0
61.9
93.8
92.6
31.2
AdamP (ours)
30.5 (+7.5)
70.9 (+7.9)
80.9 (+6.0)
89.6 (+2.6)
68.0 (+6.0)
95.2 (+1.4)
94.5 (+1.8
32.9 (+1.7)
D.2 TRAINING WITH VARIOUS TECHNIQUES
EfficientNet (Tan & Le, 2019) and ReXNet (Han al., 2020) recently proposed high- performance networks that trained using various techniques such as data augmentation (Cubuk al., 2018; 2020), stochastic depth (Huang et al., 2016), and dropout (Srivastava et al., 2014). We measured the performance of AdamP on EfficientNets and ReXNets to verify it can be used with other training techniques. The experiment were conducted on the well-known image classification codebase' Table D.2 shows performance of the original paper, our reproduced results, and AdamP. The results show that AdamP is still effective in training with various techniques, and can contribute to improving the best performance of the network.
Table D.2. Training with various techniques. EfficientNet and ReXNet were trained using the latest training techniques, and the result shows that AdamP can also contribute learning with these techniques.
Network
Image size
of params
Paper
Reproduce
AdamP
EfficientNet-Bo
224
5.3M
77.1
77.7
78.1 (+0.4)
EfficientNet-Bi
240
7.8M
79.1
78.7
79.9 (+1.2)
EfficientNet-B3
260
9.1M
80.1
80.4
80.5 (+0.1)
EfficientNet-B3
300
12.2M
81.6
81.5
81.9 (+0.4)
ReXNet-x1.0
224
4.8M
77.9
77.9
78.1 (+0.2)
1.
224
6.4M
79.5
79.5
79.7 (40.2)
https////gggthtb..cc.aragaaaaaaa httpbb/..oo/eiigggttmn/gggtrrrre ptteonnnno. mtg.://mmmmmmme...aoaoooamaaaaaa mage-modals
'https://gi
17

---

Published as a conference paper at ICLR 2021
Table E.1. Dataset statistics. Summary of the dataset specs used in the experiments. ImageNet-Ik 1k (Rus- sakovsky et al., 2015), MS-COCO (Lin et al.. 2014), CIFAR-10 (Krizhevsky. 2009), Biased-MNIST (LeCun et al., 1998; Bahng et al., 2020), ImageNet-A (Hendrycks et al., 2019). 9-Class ImageNet (Bahng et al., 2020), 9-Class ImageNet-A (Bahng et al., 2020), Magna TagAaaae (Law et al., 2009), Speech Commands (Warden, 2018), DCASE 2017 task 4 (Mesaros et al., 2017), CUB (Wah et al., 201 1), In-Shop Clothes (Liu et al., 2016b) and SOP (Oh Song al., 2016) are used in experiments.
Task
Dataset
#classes
#samples
Note
Image classification
ImageNet-1k
1,000
R 1.33M
Ohject detection
MS-COCO
80
R3 123k
CIFAR-10
10
60k
Thased-MNIST
10
Rs 60k
colors are injected to be biased
Robustness
9-Class ImageNct
9
Ro 57k
a subset of ImageNet-Ik subset of ImageNct-A
9-Class ImageNet-A A
9
617
MagnaTTagATTune
50
2 21k
mutl-labeled datasct
Audio classification
Specch Commands
35
106k
DCASE 2017 task 4
17
53k
mutl-labeled dataset
Language Modeling
WikiText-100
103M tokens
vocabulary size (267,735)
CUB Cars- 196
200
R3 12k
tr classes (100), te classes (100)
196
16k
classes (98), te classes (98)
Image retrieval
In- In-Shop Clothes SOP
7.982
classes (3,997). te classes (3985)
22,634
R 120k
tr classes (11,318), te classes (11,316)
D.3 LARGE-BATCH TRAINING
Table D.3. Large-batch training. ResNet50 (He
In order to efficiently use multiple machines and huge computational resources, large-batch train- ing is essential. However, a general optimizer suf- fers from a significant performance decrease in a large-hatch training, so large-batch training is an- other challenge for a deep learning optimizer. (You al., 2017: 2019; Goyal et al., 2017) We con- ducted experiments to verify the performance of AdamP in such large-batch training, and the re- sults are shown in Table D.3. The performance im- provement of AdamP in large-batch training is greater than that of regular batch-size (Table 2). Therefore, the decrease of effective learning rate due to momentum can be considered as one of the causes of performance degradation in large-batch training. However, AdamP does not show as much performance as the large-batch optimizers (You et al., 2017: 2019; Goyal et al., 2017), and therefore, applying AdamP to the large-batch optimizer should be studied as a future work.
et al., 2016) is trained over 100 epochs with various batch-size on ImageNet (Russakovsky et al., 2015). AdamW (Loshchilov & Hutter, 2019) and AdamP are used the optimizer.
Batch-size
Adam
AdamP (ours)
4k
72.41
73.75 (+1.34)
8k
69.74
71.36 (+1.62)
16k
63.79
66.89 (43.1)
E EXPERIMENTS SETTINGS
We describe the experimental settings in full detail for reproducibility.
E. COMMON SETTINGS
All experiments are conducted based on PyTorch. SGDP and AdamP are implemented to han- dle channel-wise (e.g. batch normalization (loffe & Szegedy, 2015) and instance normaliza- tion (Ulyanov et al., 2016)) and layer-wise normalization (Ba et al., 2016). Based on the empirical measurement of the inner product between the weight vector and the corresponding gradient vec- tor for scale-invarian parameters (they are supposed to be orthogonal), we set the in Algorithms and to 0.1. We use the decoupled weight decay (Loshchilov & Hutter, 2019) for SGDP and AdamP in order to separate the gradient due to the weight decay from the gradient due to the loss function. Please refer the attached codes: sgdp. py and adamp. py for further details.
18

---

Published as a conference paper at ICLR 2021
E.2 IMAGE CLASSIFICATION
Experiments on ResNet (He et al., 2016) are conducted based on the standard settings learning rate 0.1, weight decay 10-1 1, batch-size 256, momentum 0.9 with Nesterov (Sutskever al., 2013) for SGD and SGDP. For Adam series, we use the learning rate 0.001, weight decay 10-1, batch-size 256, 91 0.9, 32 0.999, 10
For training MobileNetV2 (Sandler et al., 2018), we have additionally used label-smoothing and large batch size 1024, and have searched the best learning rates and weight decay values for each optimizer.
The training sessions are run for 100 epochs (ResNet18, ResNet50) or 150 epochs (MobileNetV2, ResNet50 + CutMix) with the cosine learning rate schedule (Loshchilov & Hutter, 2016) on ma- chine with four NVIDIA V100 GPUs.
E.3 OBJECT DETECTION
Object detection performances have been measured the MS-COCO dataset (Lin et al., 2014) with two popular object detectors: CenterNet (Zhou et al., 2019) and SSD (Liu et al.. 2016a). We adopt the CenterNet with ResNet18 (He et al., 2016) backhone and the SSD with VGG16 BN (Simonyan & Zisserman, 2015) backbone as baseline detectors. CenterNet has been trained for 140 epochs with learning rate 2.5 10-4, weight decay 10-5, batch size 64, and the cosine learning rate schedule. SSD has been trained for 110 epochs with learning rate 10
10-4, -4, weight decay 10 batch size 64, and the step learning rate schedule which decays learning rates by 1/ /110 at 70% and 90% of training.
E.4 ROBUSTNESS
E.4.1 ADVERSARIAL TRAINING
Adversarial robustness benchmark results have been reproduced using the unofficial PyTorch im- plementation the adversarial training of Wide-ResNet (Zagoruyko & Komodakis, 2016)² for the CIFAR-10 attack challenge'. Projected gradient descent (PGD) attack variants (Madry et al., 2018) have been used as the threat model for the all the experiments. We employed 10 inner PGD iterations and 80/ /25 for the L2 PGD attack and test stronger threat models, Loc PGD with Following Madry et al. (2018), we employ 7 inner PGD iterations for the training threat model, and 20 inner PGD iterations for the test threat model. In all the experiments, Wide-ResNet-33 been trained with the PGD threat model. The models have been trained for 200 epochs with learning rate 0.01, weight decay 0.0002, batch size 128, and the step learning rate schedule which decays learning rates by /11 at epochs 100 and 150. Table E.2 shows the detailed results.
4/255 for the Loo PGD attack. We additionally 8/255 and 20 iterations as Madry et al. (2018).
have
Table E.2. Adversarial training. Standard and attacked accuracies PGD-adversarially trained Wide-ResNet on CIFAR-I0. 10. For each threat model scenario, we report the perturbation size e and the number of PGD itera- tions n. * denotes results from the original paper.
Attack Method (train)
Attack Method (test)
Optimizer
Standard Ace
Attacked Acc
Adam
80.12
56.58
Eo 4/235, 10)
AdamP
89.85 (+9.73)
66.28 (+9.70)
foo (5 = 4/255, n 10)
Adam
80.
29.00
ta 8/235, n 20)
AdamP
89.85 (+9.73)
35.76 (+6.76)
Adam
84. 14
70.33
&a (e 80/255, n 10)
Ea (e 80/255, n 10)
AdamP
93.46 (+9.32)
83.59 (+13.26)
Adam
69.76
39.48
loc (8 8/255,
foc (5 8/255, 20)
AdamP
85.74 (+15.98)
42.42 (+2.94)
Madry et al. (2018)
45.8"
1__challengeeeee
19

---

Published as a conference paper at ICLR 2021
E.4.2 ROBUSTNESS AGAINST REAL-WORLDD BIASES
We follow the two cross-bias generalization benchmarks proposed by (Bahng et al., 2020). We refer (Bahng et al., 2020) for interested readers. For all experiments. the batch size is 256 and 128 for Biased MNIST and 9-Class ImageNet, respectively. For Biased MNIST, the initial learning rate is 0.001, decayed by factor 0.1 every 20 epochs. For 9-Class ImageNet, the learning rate is 0.001, decayed by cosine annealing. We train the fully convolutional network and ResNet18 8 for 80 and 120 epochs, respectively. The weight decay 10-4 for all experiments.
E.5 AUDIO CLASSIFICATION
Dataset.
Three datasets with different physical properties are employed as the audio benchmarks. We illustrate the statistics in Table E.I. The music tagging is a multi-label classification task for the prediction of user-generated tags, e.g., genres, moods, and instruments. We use subset of Mag- naTagATune (MTAT) dataset (Law et al., 2009) which contains =21k audio clips and 50 tags. The average of tag- wise Area Under Receiver Operating Characteristic Curve (ROC-AUC) and Area Under Precision-Recal Curve (PR-AUC) are used as the evaluation metrics. Keyword spotting is a primitive speech recognition task where an audio clip containing keyword is categorized among list of limited vocabulary. We use the Speech Commands dataset (Warden, 2018) which contains R 106k samples and 35 command classes such as "yes". "no". "left", "right". The accuracy metric is used for the evaluation. Acoustic sound detection is a multi-label classification task with non-music and non-verbal audios. We use the "large-scale weakly supervised sound event detection for smart cars" dataset used for the DCASE 2017 challenge (Mesaros et al., 2017). It has =53k audio clips with 17 events such as "Car", "Fire truck", and "Train horn". For evaluation, we use the FI-score by setting the prediction threshold as 0.1.
Training setting. We use the 16kHz sampling rate for the all experiments, and all hyperparam- eters, e.g., the number of harmonics, trainable parameters, are set to the same as in (Won et al., 2020a). The official implementation by (Won et al., 2020a)* is used for all the experiments. We compare three different optimizers, Adam, AdamP (ours), and the complex mixture of Adam and SGD proposed by (Won et al., 2019b).
For the mixture of Adam and SGD, we adopt the same hyperparameters as in the previous pa- pers (Won et al., 2019b;c:a; 2020a). The mixed optimization algorithm first runs Adam for 60 epochs with learning rate 10-1, After 60 epochs, the model with the best validation performance is selected as the initialization for the second phase. During the second phase, the model is trained using SGD) for 140 epochs with the learning rate 10-4, decayed by 1/10 at epochs 20 and 40. We use the weight decay 10-4 for the optimizers. Using the hyperparameters, we reproduce the ROC-AUC score on MTAT dataset by the recent clean-up paper Won et al. (2020b), 91.27.
To show the effectiveness of our method, we have searched the best hyperparameters for the Adam optimizer on the MTAT validation dataset and have transferred them to AdamP experiments. As the result of our search, we set the weight decay as 0 and the initial learning rate as 0.0001 decayed by the cosine annealing scheduler. The number of training epochs are set to 100 for MTAT dataset and 30 for SpeechCommand and DCASE dataset. As a result, we observe that AdamP shows superior performances compared to the complex mixture, with fewer number of training epochs (200- - 30).
F.6 RETRIEVAL.
Dataset. We use four retrieval benchmark datasets. For the CUB (Wah et al., 2011) dataset which contains bird images with 200 classes, we use 100 classes for training and the rest for evaluation. For evaluation, query every test image to the test dataset, and measure the recall@1 I metric. The same protocol is applied to Cars-196 (Krause et al., 2013) (196 classes) and SOP (Oh Song et al., 2016) (22,634 classes) datasets. For InShop (Liu et al., 201 experiments, follow the official benchmark setting proposed by (Liu et al., 2016b). We summarize the dataset statistics in Table E.
hhttps://gi thw.t com www.///nwww....occcooo.o.oooo..t driita--riiie wwrriveehhaamee harmoni e- lters
20

---

Published as a conference paper at ICLR 2021
Training setting. For the all experiments, the same backbone network and the same train- ing setting excepting the optimizer and the loss function. The official implementation by (Kim et al,, 2020)* is used for the all experiments.
We use the Pytorch official ImageNett-reetraieed ResNet50 model as the initialization. During the training, we freeze the BN statistics as the ImageNet statistics (eval mode in PyTorch). We replace the global average pooling (GAP) layer of ResNet with the summation of GAP and global max pooling layer as the implementation provided by (Kim al., 2020). Pooled features are linearly mapped to the 512 dimension embedding space and fz-normalized.
We set the initial learning 10-1, decayed by the factor 0.5 for every 5 epochs. Every mini-batch contains 120 randomly chosen samples. For the better stability, we train only the last linear layer for the first 5 epochs, and update all the parameters for the remaining steps. The weight decay set to 0.
F ANALYSIS WITH LEARNING RATE SCHEDULE AND WEIGHT DECAY
We analyze the norm growth of inveriinvariant parameters and the corresponding change in effective step-size. We provide extended results of this experiment by measuring norm growth and effective step-size for SGD, SGDP, Adam and AdamP under various weight decay values. The experiment based on ResNet18 trained on ImageNet dataset, and the network was trained for 100 epoch in the standard setting in E.2. We have analyzed the impact of learning rate schedule and weight decay for the scale-invariant parameters. Figure F.I and Figure F.2 show the results of SGD and SGDP under the step-decay and cosine-anncaling learning rate schedules, respectively, The same results for Adam and AdamP are shown Figures F.3 and Figure F4. We have used the optimal weight decay value of the baseline the reference point and changed the weight decay values. We write the weight decay in each experiment in relative values with respect to the corresponding optimal values.
In all considered settings, SGDP and AdamP effectively prevent the norm growth, which prevents the rapid decrease of the effective step sizes. SGDP and AdamP shows better performances than the baselines. Another way to prevent the norm growth is to control the weight decay. However, this way of norm adjustment is sensitive to the weight decay value and results poor performances as soon non-optimal weight decay values are used. Figure F.1 and F.3 shows that the learning curves are generally sensitive to the weight decay values even showing abnormalities such as the gradual increase of the effective step sizes. On the other hand, SGDP and AdamP prevent rapid norm growth without weight decay, leading to smooth effective step size reduction. SGDP and AdamP are not sensitive to the weight decay values.
SGD (wd 0)
SGD (wd = 1e-5)
SGDP (wd = 0)
SGDP (wd = 1e-5)
SGD (wd 1e-6)
SGD (wd 1e-4)
SGDP (wd 1e-6)
70
Effective step size
0.004
Tap-1 Accuracy
65
0.000
Weight
2
0.002
60
0.001
55
0
20
40
60
80
100
0.000
0
20
40
60
80
100
50
20
40
60
100
Epoch
Epoch
Epoch
Figure F.I. Norm value analysis: SGD step learning rate decay.
F.1 ANALYSIS AT HIGH WEIGHT DECAY
In the previous Figures, we only showed the case where the weight decay less than le-4. This be- cause when the weight decay is large, the scale of the graph changes and it is difficult to demonstrate the difference small weight decay. Therefore, we separately report the large weight decay cases through Figure F.5 and F6. The high weight decay further reduces the weight norm and increases
Shttps://g/ thub. http:///ccc....9//////////o/ee// http://tw........//..//////e///m
21

---

Published as a conference paper at ICLR 2021
the effective step-size, but it does not lead improvement in performance. Also, this result shows that the weight decay used in Section 4.1 (Ie-4) is the best value for the baseline optimizers.
SGD (wd 0) 5GD (wd 1e-6)
SGD [wd 1e-5)
SGDP (wd 0)
SGDP (wd 1e-5)
SGD [wd 1e-4)
SGDP (wd 1e-6)
70
Top-1 Accuracy
Effective stop
0.003
65
Weight
2-
0.002
60
0.001
40
60
80
100
20
40
60
80
100
500
20
40
60
100
Epoch
Epach
Epoch
Figure F.2. Norm value analysis: SGD cosine learning rate decay. SGD (wd=10-1) and SGDP (wd=10" - are the same setting the reported numbers in Table
Adam (wd 0) Adam (wd 1e-6)
Adam (wd = 1e-5)
AdamP (wd = 0)
AdamP (wd 1e-5)
Adam (wd 1e-4)
AdamP (wd 1e-6)
Effecive step size
0.000
Top-1 Accuracy
Weight Nor
65
0.004
-4
60
0.002
80
100
0.000
20
80
100
50
20
Epoch
Epoch
Epoch
Figure F33. Norm value analysis: AdamW step learning rate decay.
Adam (wd 0)
Adam (wd 1e-5)
AdamP (wd
AdamP (wd 1e:5)
Adam (wd 1e-6)
Adam (wd 1e-4)
AdamP (wd 1e-6)
Effective step size
0.0006
70
Tap-1 Accuracy
0.004
Weight
60
55
0
0.000
50
20
40
60
80
100
0
20
40
60
80
1.00
20
40
60
80
100
Epoch
Epoch
Epoch
Figure F.4. Norm value analysis: AdamW + cosine learning rate decay. AdamW (wd=10-1) and AdamP
(wd=10-Fv 9) are the same setting as the reported numbers in Table 2
0.032
522 (ext 1m-4)
0.010
S2D (edt 30
SG0 (wd Be-43
0.208
0.008
10
0.004
0.802
0.000
Epach
Esoch
Epocch
Figure F.5. Norm value analysis for high weight decay: SGD cosine learning rate decay.
22

---

Published as a conference paper at ICLR 2021
2.5
Adammit (wed 20-41
Adtemii (wed 30-1)
Adammit (wed 1e-3)
0.205
4.5
Weggh
Efferive
1.2
0.5
20
4////
0:000
100
20
100
Esech
Epoch
Figure F.6. Norm value analysis for high weight decay: AdamW + cosine learning rate decay.
G RELATED WORK
We provide a brief overview of related prior work. A line of work is dedicated to the development general and effective optimizers, such as Adagrad (Duchi al., 2011), Adam (Kingma & Ba, 2015), and RMSprop. Researchers have sought strategies to improve Adam through e.g- improved conver- gence (Reddi et al., 2018), warmup learning rate (Liu et al., 2020), moving average (Zhang et al., 2019b), Nesterov momentum (Dozat, 2016), rectified weight decay (Loshchilov & Hutter, 2019), and variance of gradients (Zhuang et al., 2020). Another line of researches studies existing opti- mization algorithms in greater depth. For example, (Hoffer et al., 2018; Arora et al., 2019; Zhang et al., 2019a) have delved into the effective learning rates on scale-invariant weights. This paper at the intersection between the two. We study the issues when momentum- based optimizers are applied on scale-invariant weights. We then propose a new optimization method to address the problem.
G.1 COMPARISON WITH CHO & LEE (2017)
Cho & Lee (2017) have proposed optimizers that are similarly motivated from the scale invariance of certain parameters in neural network. They have also propose solution that reduces the dial component of the optimization steps. Despite the apparent similarities, the crucial difference between Cho & Lee (2017) and ours is in the space where the optimization is performed. Cho & Lee (2017) performs the gradient steps on the Riemannian manifold. Ours project the updates on the tangent planes of the manifold. Thus, ours operates on the same Euclidean space where SGD and Adam operate on. From a theory point of view, Cho & Lee (2017) has made contributions to the optimization theory on a Riemannian manifold. Our contributions are along different axis: we focus on the norm growth when the updates are projected onto the tangent spaces ($2.4, $3.1, and $3.2). We contribute present theoretical findings that are not covered by Cho & Lee (2017).
From the practicality point of view, we note that changing the very nature of space from Euclidean to Riemannian requires users to find the sensible ranges for many optimization hyperparameters again. For example, Cho & Lee (2017) has "used different learning rates for the weights in Euclidean space and on Grassmann [Riemannian] manifolds" (page 7 of (Cho & Lee, 2017)), while in our case hyper- parameters are largely compatible between scale-invariant and scale-variant parameters, for they both accommodated the same kind of space. We have shown that SGDP and AdamP outperform the SGD and Adam baselines with exactly the same optimization hyperparameters (Section E.2). The widely used Xavier (Glorot & Bengio, 2010) or Gaussian initializations are no longer available in the spherical Riemannian manifold, necessitating changes in the code defining parameters and their initialization schemes: e.g. Cho & Lee (2017) has employed dedicated initialization based on truncated normals. Finally, Cho & Lee (2017) requires users to manually register icale-invariant hyperparameters. This procedure is not scalable, as the networks nowadays are becoming deeper and more complex, and the architectures are becoming more machine-designed than handcrafted. Our optimizers automatically detect scale invariances through the orthogonality test (Equation 1 I), and users do not need to register anything by themselves. The shift from linear to curved coordinates introduces non-trivial changes in the optimization settings; our optimizers does not introduce such shift and it is much easier to apply our method on a new kind of model, from a user's 's perspective.
In addition to the above conceptual considerations, the Grassmann optimizers (Cho & Lee, 2017). We first compare them in the 3D ecale-invariant Rosenbrock example (see $3.2 for description). Figure G.I and G.2 show the results for the Grass-
compare the performances between ours and
23

---

Published as a conference paper at ICLR 2021
86D3
SCDG wa gradient diipping
6D projection (ours)
2.5
20
200
20
10
1.
16
0.5
0.
0.0
0.0
0.
-00
-00
-00
120
swatfone
Figure G.I. 3D scale-invariant Rosenbrock with SGDG optimizers. 3D toy experiments based on SGDG
optimizers. Upper row: loss surface and optimization steps. Lower row: of parameters over the iterations.
gradiert cippirg
projecion
2.0
1.0
1.6
1.6
1,0
1.0
1.0
Lun
0.0
-05
2
Figure G.2. 3D scale-invariant Rosenbrock with AdamG optimizers. toy experiments based on AdamG row: loss surface and optimization steps. Lower row: norm of parameters over the iterations.
optimizers. Upper row:
mann optimizers: SGDG and AdamG, respectively. It can be seen that SGDG and AdamG optimizer do not introduce any norm increase by definition: they operate on spherical space. However, rate of convergence seems slower than our SGDP and AdamP (the rightmost plots in each figure). We note that SGDG and AdamGi include a gradient clipping operation to restrict the magnitude of pro- jected gradients on the Grassmann manifold v, where v is empirically set to 0.1 in Cho & Lee (2017). We have identified adverse effect of the gradient clipping, at least As the clipping is removed, SGDG and AdamG come closer to the fast convergence of ours (the middle plots in each figure). We may conclude for the toy example that the Grassmann optimizers also address the unnecessary norm increase and converge as quickly as our optimizers do.
toy example.
For more practical setup, we present experiments ImageNet with the Grassmann optimizers (SGDG and AdamG) and report the top-1 accuracies. We have conducted the experiments with ResNet18 following the settings in Table 2. In addition to the learning rate of optimizers (Irguciddean) Cho & Lee (2017) introduces three more hyperparameters: (1) learning rate on the Grassmann man- ifold (IrGossmannn) (2) degree of the regularization on the Grassmann manifold (a), which replaces the Le regularization in Euclidean space, and (3) the gradient clipping threshold (V). Since Cho & Lee (2017) have not reported results on ImageNet training, we have tuned the hyperparameters ourselves, following the guidelines of Cho & Lee (2017). Table G.1 shows the exploration of the hyperparameters that are considered above. The first rows show the performance with the recom- mended hyperparameters in the paper (o v = 0.1 and IrEuclidem 0.01 for SGDG and AdamG; IrGrassmano 0.2 for SGDG and IrGrassmann = 0.05 for AdamG). We first tested the effects of reg- ularization (a) and gradient clipping (V), which were additionally introduced in the Grassmann optimizers. The first block of the Table G.1 shows the result. The gradient clipping (v) did not have significant effect, but the regularization (ai decrease the performance. Therefore,
turned off regularization (a = 0) in the optimizers and started learning rate search. In Table 2, the baseline optimizers and optimizers compared in the fixed learning rate (SGD: 0.1, Adam: 0.001)
24

---

Published as a conference paper at ICLR 2021
Table G.I. Hyperparameter search of the Grassmann optimizers. This talbe shows the performance of ResNet18 ImageNet trained with Grassmann optimizers (SGDG and AdamG) with various hyperparameters. Except for the hyperparameters specified in the table, the setting is the same as the experiment in Table
(a) SGDG (SGDP accuracy: 70.70)
(b) AdamG (AdamP accuracy: 70.82)
Search
Seurch larget
Accuracy
Accuracy
target
Euchideam
Grasmmmn
Euchddaa
Grussmann
0.1
0.01
0.2
66.00
0.1
0.1
0.01
0.05
67.40
OL,V
0.1
0.01
0.2
67.85
0,V
0.1
0.01
0.05
68.43
0.1
None
0.01
0.2
66.35
0.1
None
0.01
0.05
67.36
0.1
0.01
0.2
67.85
0.1
0.01
0.05
68.43
0
0.1
0.1
2
63.28
0.1
0.001
0.005
66.97
0.1
0.005
0.1
68.73
0.1
0.0002
0.001
55.9
0.1
0.1
0.1
69.74
0.1
0.001
0.001
59.36
0.1
0.1
0.1
69.74
0.1
0.01
0.05
68.43
68.33
Ir
63.96
tuning
0.1
0.03
0.03
70.39
tuning
0.1
0.003
0.015
69.45
0.1
0.01
0.01
69.21
0.1
0.1
0.5
59.39
setting. So, for fair comparison, we design four learning rate (Ir) candidates for grassmann optimiz- ers: Use the learning rates of the paper (Cho & Lee, 2017). 2) & 3) Use the fixed learning rates for Ireuctidean or Iggassmannn and adjust the other following the learning rate ratio of the paper. 4) Use baseline learning rates for both learning rates (Ireuclidean and Ir Iggrassmann). The result reported in the second block of Table G. 1. SGDG shows the best performance at option 4) and AdamG the best in option 1). However, the performances of Grassmann optimizers
still much lower than our optimizers: SGDP (70.70) and AdamP (70.82). We further tuned the learning rate of the Grassmann optimizer, which is shown the last block of the Table G.I. After learning rate tuning, Grassmann optimizers show comparable performance with the baseline optimizer (SGD: 70.47, Adam: 68.05, AdamW: 70.39). However, learning rate tuning is essential for the performance, and our optimizer has higher performance. So, it can be said that our optimizer is more practical and effective for ImageNet training than the Grassmann optimizer.
H ADDITIONAL EXPERIMENTS
H. 3D TOY EXPERIMENTS FOR ADAM
We also evaluated our 3D toy experiments for Adam optimizer. The results shown in Fig. H.1 Adam optimizer shows quiet different steps in early stage. However, the fact that norm growth reduces the rate of late convergence is the same SGD. The weight decay and our projection mitigate the norm growth and helps fast convergence.
optimizer
whicht decae
projection (ours)
25
2.5
2.5
L//
1.0
1.0
1.0
0.5
0.6
0.6
25
30
75
100
150
25
30
75
100
122
150
25
100
125
130
Figure H.I. 3D scale-invariant Rosenbrock with Adam optimizers. 3D toy experiments based on Adam optimizers. Upper row: loss surface and optimization steps. Lower row: norm of parameters over the iterations.
25

---

Published as a conference paper at ICLR 2021
H.2 STANDARD DEVIATION OF EXPERIMENTS
The most experiments in the paper were performed three times with different seed. The mean value was reported in the main paper and the standard deviation value is shown in the Table H.1, H.2, H.3, H.4, H.S accordingly. In case of ImageNet classification, mean values are shown in Table 2 and standard deviation values are shown in Table H. In most cases, the improvement of our optimizer is significantly larger than the standard deviation. Even in the worse case, (SGDP on ResNet50), the performance increases are at the level of the standard deviations. For the audio classification results (Table 4 and H.2), the performance increases by AdamP in Music Tagging (ROC-AUC) and Keyword Spotting are much greater than the standard deviation values; for the other entries, the performance increases are at the level of the standard deviations. In the language model, all experiments have similar standard deviation values as shown in Table H.3. In all cases, AdamP's improvement (Table 5) is significantly larger than the standard deviation. We observe different tendency for the image retrieval tasks (Table 6 and H.4). In many cases, the standard deviation values large, the performance boost is not clearly attributable to AdamP. However, the performance increases for InShop-PA and SOP-PA are still greater than the standard deviation values. Finally, in the results for robustness against real-world biases (Table D.I and H.5), our optimization algorithm brings about far greater performance gains than the randomness among trials. In most experiments, the performance improvement of our optimizers is much greater than the standard deviation values.
Table 1. Standard deviation for ImageNet classification. Standard deviation of accuracy of statc-of-the-an networks trained with SGDP and AdamP (Table 2).
Architecture
params
SGD
SGDP (ours)
Adam
Adam
AdamP (ours)
MobileNetV2
3.5M
= 0.14
t 0.11
#.04
t 0.16
t 0.16
ResNet18
11.7M
* 0.16
t 0.06
# 0.06
t 0.07
t 0.06
ResNet50
25.6M
* 0.06
# 0.08
* 0.112
* 0.06
* 0.02
ResNet50 CutMix
25.6M
0.11
0.05
0.08
0.09
0.09
Table H.2. Standard deviation for audio classification. Standard deviation for results on the audio tasks with
Harmonic CNN (Won et al., 2020a) (Table 4).
Music Tagging
Keyword Spotting
Sound Event Tagging
Optimizer
ROC-AUC
PR-AUC
Accuracy
Fl
Adam SGD (Won et al., 2019b)
0.06
1 0.03
AdamW
1 0.19
0.06
0.39
AdamP (ours)
= 0.27
* 0.06
0.83
Table H.3. Standard deviation for Language Modeling. Standard deviation of the perplexity values Wiki- Text103 (Table 5).
Model
AdamW
AdamP (ours)
Transformer-XL Transformer-XI.+ + WN
* 0.05
* 0.05
0.06
0..6
Table H.4. Standard deviation for image retrieval. Standard deviation of Recall @1 on the retrieval tasks (Table 6).
CUB
Cars 196
InShop
SOP
Optimizer
Triplet
PA
Triplet
PA
Triplet
PA
Triplet
PA
Adam W
* 0.91
* 0.24
0.82
0.09
0.52
* 0.08
+ 0.71
* 0.04
AdamP (ours)
* 0.62
+ 0.77
+ 1.35
* 0.19
* 0.82
* 0.02
+ 0.74
+ 0.09
H.3 ANALYSIS WITH MOMENTUM COEFFICIENT
Since our optimizer is deeply involved with the momentum of optimizers, we measure and analyze the effect of our optimizer on several momentum coefficients. The experiment was conducted using ResNet18 in ImageNet with the setting of Section 4.1, and weight decay not used to exclude norm decrease due to weight decay. The results are shown in Table H.6. According to the differ- ence between equation 8 and equation 12, the effect of preventing norm growth of our optimizer is
26

---

Published as a conference paper at ICLR 2021
Table H.5. Standard deviation for robustness against real-world biases. Standard deviation for Re-
Bias (Bahng al., 2020) performances on Biased MNIST and 9-Class ImageNet benchmarks (Table D.I).
Biased MNIST Unbiased acc. at
9-Class ImageNet
Optimizer
.999
.997
.995
.990
avg.
Biased
UnBiased
ImageNet-A
Adam
- 6.04
= 0.48
0.70
0.48
I 0.27
0.40
#1.06 # 0.82
AdamP (ours)
= 1.73
= 1.31
2.16
+ 0.40
= 0.21
0.27
affected by the momentum coefficient, It can be observed in this experiment. The larger the momen- tum coefficient, the greater the norm difference between SGD and SGDP. In addition, it can be seen that the improvement accuracy SGDP also increases the momentum coefficient increases. The experiment shows that our optimizer can be used with most of the momentum coefficients, espe- cially when the momentum coefficient large, the effect of our optimizer is significant and essential for the high performance.
Table H.6. Analysis with momentum coefficient. We measured the diffcrence between our SGDP and SGD with various momentum coefficients.
SGD
SGDP
Difference
Momentum
Norm asst
Accuracy
Norm lter
Accuracy
Norm aatt
Accuracy
0.5
7.60
67.57
67.75
-0.7
0.18
0.8
8.31
67.64
6.57
68.55
-1.7
0.91
0.9
8.53
67.42
6.33
68.95
-2.2
1.53
0.95
8.76
67.67
6.23
69.12
-2.5
1.45
0.975
8.73
67.56
6.14
69.17
-2.6
1.60
H.4 COMPARISON
SAME COMPUTATION COST
As specified in Section 3.1, 1, our optimizers requires an additional computation cost, which increases the training cost by 8%. general, the optimizer's performance compared and we followed this convention in other experiments. However, the training cost is also important issue, so we conduct further verification of our optimizer through comparison at the same training budget. The experimental setup is simple. We performed imagenet classification in Section 4.1 with only 92% epochs for our optimizers (SGDP and AdamP) and set the training budget of our opti- mizer and baseline optimizer to be the same. The results are shown in Table H.7. Training iteration reduced, so the performance timizer. Thus, it can be seen that our optimizer outperforms the baseline optimizer not only on the same iteration, but also on the same training budget.
same iteration,
our optimizer is reduced, but it still outperforms the baseline op-
Table H.7. ImtgeNet classification comparison at the same computation cost. Accuracies of state-of-the-ae networks trained with SGDP and AdamP. We also conduct training over 92% epochs with SGDP and AdamP for the comparison in the same computation cost.
Architecture
SGD
SGDP (92%)
SGDP
Adam W
AdamP (92%)
AdamP
MobileNerV2
71.55
71.85 (+0.30)
72.09 (+0.54)
71.21
72.34 (40.13)
72.45 (+1.24)
ResNet18
70.47
70.54 (10.07)
70.70 (+0.23)
70.39
70.64 (+0.25)
70.82 (+0.4))
ResNet50
76.57
76.58 (+0.01)
76.66 (+0.09)
76.54
76.84 (40.30)
76.92 (40.38)
27


Reviews:
[{"review_id": "aBNDGVAVq0o", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###################################################################\n\nSummary:\n\nThis paper shows that momentum-based gradient descent optimizers reduce the effective step size in training scale-invariant models including deep neural networks normalized by batch normalization, layer normaliztion, instance normalization and group normalization. The authors then propose a solution that projects the update at each step in gradient descent onto the tangent space of the model parameters. Theoretical results are provided to show that this projection operator only adjusts the effective learning rate but does not change the effective update directions. Empirical results on various tasks are provided to justify the advantage of the proposed method over the baseline momentum-based (stochastic) gradient descent and Adam.\n\n###################################################################\n\nReason for the Score:\n\nOverall, this paper could be an interesting algorithmic contribution. However, there are relevant points needed to be clarified on the theory and experiments. My first main concern is that theoretically it is hard to justify that the proposed projection-based update yields smaller parameter norms than the baseline momentum-based update. My second main concern is some baseline results in the experiments do not match those in existing literature, and no error bars are provided in the empirical results even though the improvements of the proposed methods over the baseline methods are small. \n\nCurrently, I am leaning toward rejecting the paper. However, given additional clarifications on the two main concerns above in an author response, I would be willing to increase the score.\n\n###################################################################\n\nStrong points:\n\n1. The paper points out a relevant issue in using normalization techniques such as batch normalization together with momentum-based optimization algorithms in training deep neural networks. \n\n2. The paper provides experimental results on various tasks and datasets to demonstrate the advantage of the proposed method.\n\n3. The paper is well-written with illustrative figures.\n\n###################################################################\n\nWeak points:\n\n1. It is not clear to me that the proposed update in equation (12) yields smaller norms ||w_{t+1}|| than the momentum-based update in equation (8).  The parameters of the model evolve differently under these two update rules. Throughout the training, the update p_t in equation (11) is different from the update p_t in equation (8). As a result, it is hard to compare ||q_t|| in equation (12) and ||p_t|| plus all the terms ||p_k|| in equation (8). \n\n2. The improvements of the proposed SGDP and AdamP over the baseline SGD and Adam are small across experiments, and thus error bars are needed to validate that these improvements are not due to randomness. However, no error bars are provided for the empirical results in the paper.\n\n3. The reported baseline results for audio classification are worse than those reported in (Won et al., 2019).\n\n4. The baseline results for adversarial robustness seems to be much higher than reported results in (Madry et al., 2018). Also why are the values of epsilon used in the paper quite small (80/255 and 4/255 vs. 8 in (Madry et al., 2018)).\n\n###################################################################\n\nAdditional Concerns and Questions for the Authors:\n\n1. Adam normalizes the gradient by its cumulative norm. This can help eliminate the small step size issue since the norms of the gradients become smaller during training. Can you provide a similar simulation as in Figure 3 but using Adam, AdamW, and AdamP?\n\n2. What are the baseline results, reported in existing literature, on ImageNet for ResNet18 and ResNet50 trained with the cosine learning rate schedule in 100 epochs? Can you please link me to the previous papers that report those results? The paper you cite, (Loshchilov & Hutter, 2016), does not report those results.\n\n3. In section 4.1, the authors say “For ResNet, we employ the training hyperparameters in (He, 2016)”. However, the training hyperparameters for ResNet used in the paper are not from (He, 2016). In (He, 2016), the models are trained for only 90 epochs without using cosine learning rate.\n\n4. The proposed update is more expensive than the baseline momentum-update. The paper also reports that the proposed update incurs 8% extra training time on top of the baselines for ResNet18 on ImageNet classification while resulting in only small improvements over the baselines. It is needed to compare the proposed optimizer with the baseline momentum-based optimizer trained with more epochs and potentially with an additional learning rate decay.\n\n###################################################################\n\nMinor Comments that did not Impact the Score:\n\n1. The paper proposes not only AdamP, but also SGDP. It is better if the authors remove AdamP in the title.\n\n2. In figure 4, is the y-axis the test or train accuracy?\n\n###################################################################\n\nReferences:\n\nMinz Won, Sanghyuk Chun, and Xavier Serra. Toward interpretable music tagging with self- attention. arXiv preprint arXiv:1906.04972, 2019. \n\nAleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018. URL https://openreview.net/forum?id= rJzIBfZAb. \n\nIlya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. \n\nKaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.\n\n\n###################################################################\n\nPost Discussion Score:\n\nAfter reading the rebuttal from the author and the comments from other reviewers, I am still not clear if the proposed update in equation (12) yields smaller norms ||w_{t+1}|| than the momentum-based update in equation (8). However, the authors have addressed all of my other concerns. I decided to increase my score for this paper from 4 to 5. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QnPQsdqu9jZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\nThis paper studies the hurtful effect of momentum on scale-invariant parameters (such as weight matrices of linear layers followed by Batch Normalization (BN)): Indeed, momentum tend to increase the norm of the parameters, but the effective update size of scale-invariant parameters is inversely proportional to their norm, which leads to \"_premature decay of effective step size_\". The authors propose the improved SGDP and AdamP to reduce this issue, and show how it improves the performances of a wide variety of models (ResNets, MobileNets, Transformers, etc...) on many different tasks (ImageNet classification, adversarial training, audio classification, etc...)\n\n### Strengths:\n+ This paper addresses the optimization of scale-invariant parameters, which can be found in pretty much all the state-of-the-art models.\n+ The paper is well written and contains a good balance of illustrative examples, theoretical analysis and experiments. Good job!   \n+ The empirical evaluation of the proposed algorithms is quite large, and many tasks and architectures are considered.\n+ I also really like the \"cos(w, Grad_w)\" hack to automatically detect scale invariant parameters in the model. This is a very neat trick!\n\n### Concern:\nMy main concern is with the statement that \"_our paper is first to delve into the issue in the widely-used combination of scale-invariant parameters and momentum-based optimizers\"_ . Indeed the authors have missed [1], which also adapts SGD with momentum and Adam to work on scale-invariant parameters, and showed performance improvement over vanilla SGD and Adam. Now, the approach of [1] is slightly different, in the sense that they propose to keep the scale-invariant parameters on the unit sphere S^1. Also, [1] performs the projection onto the tangent space before computing the momentum term (and second order moment in Adam), while this projection is done after in SGDP and AdamP.\n\nIn any case, I do believe proper comparison with [1] throughout the paper is required, due to the conceptual similarity of both methods. It would be particularly relevant  to show how the angles between SGDP and the SGD of [1] compare in Figure 1 (I think they would be identical, although not 100% certain). Also, I'm curious to see if the proposed SGDP and AdamP work better than the algorithms proposed in [1], as the nice automatic learning rate decay of BN disappears when keeping the scale-invariant parameters on the unit sphere (which would be an advantage of the proposed SGDP and AdamP).\n\n### Reasons for score:\nOverall I really liked that paper, but I don't think it would be fair to accept it without an in-depth comparison with [1].\n\n### Questions / Comments:\n- I like the toy example in Figure 1, although I'm not sure I understand what you mean by: \"_Compared to GD, GD+Momentum speeds up the norm growth, resulting in a slower effective convergence in S^1, though being faster in the nominal space R^2_\". It seems to me that the angle between the optimal solution and GD+momentum is smaller than the angle between the optimal solution and GD. I'm not sure to understand how do you observe a slower effective convergence in  S^1?\n- (3.1, real-world experiments and F) What happens if you increase WD when using SGD? It seems that the best values reported for SGD in Table 1 and Figures F.* are with WD=1e-4, which is also the highest value reported. One can imagine even better results could be obtained using WD=3e-4 or 1e-3. It would be nice to have this result for completeness.\n- Reporting mean and standard deviation across several seeds would have been nice, but I'm going to let this one slide, since results are reported on a lot of different tasks and architectures.\n\n[1] Cho, Minhyung and Lee, Jaehyung, _Riemannian approach to batch normalization_, NIPS 2017\n\n___\n\n### After author response and paper revision\n\nFirst, I would like to congratulate the authors for their amazing work during the rebuttal period.\n\nMy main concern, the comparison against [1], has been perfectly addressed in appendix G (both \"conceptually\", by highlighting the differences between both methods, and showing the advantages of the proposed SGDP and AdamP, but also empirically on ImageNet, where a fair comparison with proper hyper-parameter tuning has been performed).\n\nThe authors also reinforced their empirical investigation by reporting standard deviation of the results, which allows to better appreciate the performances of SGDP and AdamP. Finally, they also added the experiments with higher weight decay, showing that indeed 1e-4 was the best value.\n\nWith all these changes, I think the paper is good and I recommend its acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NZl1e-u4U7_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: Based on the assumption that the rapidly decrease step size \\delta \\omega leads to a solve effective convergence of \\omega. This paper proposes to use  a projection step to remove the radial component, and thus reduce the norm of training parameters, and faster the training procedure. The experiments look good to me while the derivation of this paper based on many assumptions and conjectures.   I tend to accept this paper at this time. However, I am not an expert at this area I will not be sad if this paper is reject by other reviews.\n\nminor problems:\n1) Can the author explain more on how to derive eq.(4) ?", "extra_scores": {}, "extra_text": {}}, {"review_id": "5Zsn_YEJpFR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper points out that momentum in GD optimizers results in a far more rapid reduction in effective step sizes for scale-invariant weights.  To solve the problem, two algorithms called SGDP and AdamP are proposed, which project the updates to tangent space of the parameter. Experiments on several tasks including image classification, language modeling, etc show the effectiveness of the proposed algorithms. The idea to study the integration of BN and momentum is interesting. The analyses and proposed algorithms provide guidance to practitioners. \n\nQuestions:\n\n(1) What is the difference between the proposed algorithm and the algorithm in paper \"Cho & Lee, Riemannian approach to batch normalization\"? Eq.(10) in your paper is exactly the same with Eq.(6) in [Cho & Lee]. It is an important related work which should be cited and compared with.\n\n(2) What does \"lies on the geodesic\" mean in Proposition 3.1? What is the relation between Proposition 3.1 and the convergence of the algorithm?\n\n(3) It is not clear which results are proposed in related work and which results are proposed in this paper. Is Lemma 2.1 a new result? If not, it needs a citation.\n  \n(4) The theory part analyzes the effect of momentum, while the experiments shows the effect of weight decay. How does weight decay influence the norm growth theoretically? Why not conduct experiments with varying momentum coefficient? \n\n(5) The theory part shows the negative effect of momentum. Does it mean that \"SGD is a better choice than momentum SGD\"? Momentum SGD is a standard algorithm to train deep neural networks with BN in practice but not vanilla SGD. How to explain it?\n\n(6) What is the additional computational cost of the proposed algorithms compared with the baselines?  \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Clarity: The paper is well-written with illustrative figures.

Originality: The originality of the paper is relatively restricted, mainly due to the resemblance with the work [1]. However, there are important differences, that the authors nicely pointed out, and we encourage them to include these in the final version of the paper.

Significance: The paper points out a relevant issue in using normalization techniques such as batch normalization together with momentum-based optimization algorithms in training deep neural networks. While the paper could be considered "another algorithms for training NNs", the papers illustrates nicely the main arguments, and is backed up with more than sufficient experimental results.

Main pros:
- In the main pros, AC and reviewers admit the phenomenal job in responding to reviewers' questions and requests
- The paper provides experimental results on various tasks and datasets to demonstrate the advantage of the proposed method.
- After the reviews, The authors also reinforced their empirical investigation by reporting standard deviation of the results, which allows to better appreciate the performances of SGDP and AdamP. Finally, they also added the experiments with higher weight decay, showing that indeed 1e-4 was the best value.

Main cons:
- One reviewer requires more explanation why the proposed update in equation (12) yields smaller norms ||w_{t+1}|| than the momentum-based update in equation (8).
Decision: Accept (Poster)