Understanding AdamW through Proximal Methods and Scale-Freeness
Authors: Zhenxun Zhuang
Mingrui Liu
Ashok Cutkosky
Francesco Orabona
Venue: ICLR 2022 Submitted (2022)
Keywords: Optimization of Deep Neural Networks
Scale-free
AdamW

Abstract: Adam has been widely adopted for training deep neural networks due to less hyperparameter tuning and remarkable performance. To improve generalization, Adam is typically used in tandem with a squared $\ell_2$ regularizer (referred to as Adam-$\ell_2$). However, even better performance can be obtained with AdamW, which decouples the gradient of the regularizer from the update rule of Adam-$\ell_2$. Yet, we are still lacking a complete explanation of the advantages of AdamW. In this paper, we tackle this question from both an optimization and an empirical point of view. First, we show how to re-interpret AdamW as an approximation of a proximal gradient method, which takes advantage of the closed-form proximal mapping of the regularizer instead of only utilizing its gradient information as in Adam-$\ell_2$. Next, we consider the property of ``scale-freeness'' enjoyed by AdamW and by its proximal counterpart: their updates are invariant to component-wise rescaling of the gradients. We provide empirical evidence across a wide range of deep learning experiments showing a correlation between the problems in which AdamW exhibits an advantage over Adam-$\ell_2$ and the degree to which we expect the gradients of the network to exhibit multiple scales, thus motivating the hypothesis that the advantage of AdamW could be due to the scale-free updates.

Under review as a conference paper at ICLR 2022
UNDERSTANDING ADAMW THROUGH PROXIMAL METHODS AND SCALE-FREENESS
Anonymous authors Paper under double-blind review
ABSTRACT
Adam has been widely adopted for training deep neural networks due to less hyperparameter tuning and remarkable performance. To improve generalization, Adam is typically used in tandem with a squared &g regularizer (referred to as Adam-l2). However, even better performance can he obtained with Adam W, which decouples the gradient of the regularizer from the update rule of Adam- Yet, we are still lacking a complete explanation of the advantages of AdamW. In this paper, we tackle this question from both optimization and an empirical point of view. First, we show how to re-interpret Adam W as an approximation of a proximal gradient method, which takes advantage of the closed-form proximal mapping of the regularizer instead of only utilizing its gradient information as in Adam-E2. Next, we consider the property of "scale-freeness" enjoyed by AdamW W and by its proximal counterpart: their updates are invariant to component wisee rescaling of the gradients. We provide empirical evidence across wide range of deep learning experiments showing a correlation between the problems in which Adam W exhibits an advantage over Adam-&2 and the degree to which we expect the gradients of the network to exhibit multiple scales, thus motivating the hypothesis that the advantage of AdamW could be due to the scale-free updates.
INTRODUCTION
Recent years have seen surge of interest in applying deep neural networks (LeCun et al. 2015) to myriad of areas. While Stochastic Gradient Descent (SGD) (Robbins & Monro 1951) remains the dominant method for optimizing such models, its performance depends crucially on the step size hyperparameter which controls how far the algorithm proceeds along the negative stochastic gradient in each step to update the model parameters. To alleviate this problem, people have developed fruitful line of research on adaptive gradient methods (e.g. Duchi et al. 201 2010a McMahan & Streeter 2010 Tieleman & Hinton, 2012] Zeiler 2012] Luo et al. 2018 Zhou et al.) 2018). These methods provide mechanisms automatically stepsizes, and have been shown to greatly reduce the tuning effort while maintaining good performance. Among those adaptive algorithms, one of the most widely used is Adam (Kingma & Ba, 2015) which achieves good results across variety of problems even by simply adopting the default hyperparameter setting.
In practice, improve the generalization ability, Adam is typically combined with a Eg regularization which adds the squared E2 norm of the model weights on top of the loss function (which we will call Adam-C2 hereafter). This technique is usually referred to weight decay because when using SGD, the & regularization works by first shrinking the model weights by constant factor in addition to moving along the negative gradient direction in each step. However, as pointed out in Loshchilov & Hutter (2019) for Adam, there is no fixed regularization that achieves this same effect. To address this, they provide method called Adam W that decouples the gradient of the & regularization from the update of Adam and directly decays the weights. The two algorithms are shown in Algorithm [ Although Adam frequently outperforms Adam-l2, the approach is primarily motivated empirically without clear understanding of why it works so well.
Recently, however, Bjorck et al. (020) applied Adam in Natural Language Processing (NLP) and Reinforcement Learning problems and found no improvement of performance over sufficiently tuned Adam-lz. Considering the huge popularity of AdamW (Kuen et al.) al. 2020), investigate when and why AdamW has significant improvement over Adam-
et al., 2019 Caaion

---

Under review as a conference paper at ICLR 2022
In this paper, focus understanding the training and testing dynamics of the AdamW update in contrast to 12. We consider this contrast from the lens of optimization theory rather than directly investigating generalization over multiple epochs. First, we unveil the surprising connection between Adam W and proximal updates. In partieular, we show that Adam W is an approximation of the latter and also confirm such similarity with an empirical study. Noticing that AdamW and the proximal update are both scale-free while Adam-l2 is not, we derive a theorem showing that scale-free optimizers enjoy an automatic acceleration with respect to the condition number on certain cases. This gives AdamW& concrete theoretical advantage in training over Adam-
Next, we empirically identify the scenario of training very deep neural networks with batch normal- ization (BN) switched off as case in which AdamW substantially outperforms Adam-&2 in both testing and training. Note that the setting of removing BN is not our invention: indeed, there is already active research in this (De & Smith 2020 Zhang et al. (019). The reason is that BN has
many disadvantages (Brock et al. 2021 b including added memory overhead (Bulò et al. 2018) and
training time (Gitman & Ginsburg 2017 and discrepancy between training and inferencing (Singh
& Shrivastava| 2019). BN has also been found to be not suitable for many cases including distributed
computing with small minibatch per GPU (Wu & He 2018 Goyal et al., 2017), sequential modeling
tasks (Ba et all 2016), and contrastive learning algorthmss (Chen et al. 2020p. Moreover, there are
already SOTA architectures that do not BN including the Vision transformer (Dosovitskiy et al. 2021 and the BERT model (Devlin et al. 2019
For such settings of removing BN, observe that the magnitudes of the coordinates of the updates during training are much more concentrated about a fixed value for AdamW than for Adam- which an expected property of scale-free algorithms. Further, as depth increases, we expect greater diversity of gradient scalings, scenario that should favor scale-free updates. Our experiments support this hypothesis: deeper networks have more dramatic differences between the distributions of update scales between Adam-l2 and Adam W, and larger accuracy ad advantages for AdamW.
Specifically, the contributions of this paper are:
1. We show that AdamW can be seen as an approximation of the proximal updates, which utilize the closed-form proximal mapping of the regularizer instead of only its gradient.
2. We point out the scale-firenness property enjoyed by AdamW and show the advantage of such a property on concrete problem classes.
3. We find scenario where AdamW W is significantly better than Adam-l2 in both training and testing performance and report an empirical observation of the correlation between such advantage and the scale-freeness property of AdamW.
The rest of this paper is organized as follows: In Section 2 we discuss the relevant literature. The connection between Adam and the proximal updates as well as its freeness are explained in Section 3 We then report the empirical observations in Section 4 Finally, we conclude with a discussion of the results and point out some potential future directions.
2 RELATED WORK
By enforcing the magnitude of the model weights to be small, weight decay has long been standard technique to improve the generalization ability in machine learning (Krogh & Hertz 1991 Bos &
Chug. 1996) and is still widely employed in training modern deep neural networks (Devlin et al. [019
Tan & Le19)) Here, we do not attempt to explain the generalization ability of Adam W. Rather, we assume that the regularization and the topology of the network guarantee good generalization
performance. Instead, we study algorithms from the perspective of convergence rate.
The use of proximal updates in the batch optimization literature dates back at least to 1965 (Moreau 1965 Martinet] 1970 [Rckafellar 1976; Parikh & Boyd 2014), and more recently used even in the stochastic setting (Toulis & Airoldi 2017 Asi & Duchi 2019). We are not aware of any previous paper pointing out the connection between AdamW and proximal updates.
The scale-free property was first proposed in the online learning field (Orabona & Pal 2018). There, they do not need to know a priori the Lipschitz constant bounding the gradient norms while still able to achieve the optimal rates. To the best of our knowledge, ecale-freeness has not been explored as an explanation for the efficiency of deep learning optimization algorithms.
2

---

Under review as a conference paper at ICLR 2022
Algorithm Adam with 1.2 regularization (Adam-E2) and Adam with decoupled weight decay (Adam W)
Loshchilov & Hutter (2117) (Note that these two are exactly the same when X = 0, namely no weight decay.)
1: Given a, 31, 82, À E IR, (n0020 All operations on vectors are element-wise.
2: Initialize: 2% E Rd, first moment vector mg +- 0, second moment vector vo <- 0
3: for l- 1,2,... T do
4: Compute the stochastic gradient V (2((x=1)) evaluated on a mini-batch of samples
5: 9: ( +
6: mr - 3,m + (1
-11g..
82 - 8- + (1- 11--Ba 82)9
7: m. - m,/(1 81),
Ú,
0 (1- 33)
8:
It f- -
- --
-namm // +
9: end for
3 THEORETICAL INSIGHTS ON THE MERITS OF ADAMW
A and Proximal Updates: Here, we show that Adam approximates a proximal update with squared l2 regularization. This provides first theoretical motivation for Adam
Consider that we want to minimize the objective function
F F(x)) = Σे || 2 + f(x),
(1)
where 0 and f(æ) F2 R is function bounded from below. We could use stochastic optimization algorithm that updates in the following fashion
C2 #t-1 M.Pe
where M, is a generic matrix containing the learning rates and denotes the update direction. Specifically, consider M, nJ la where тe learning rate schedule, e-g., the constant the cosine annealing (Loshchilov & Hutter, (017). This update covers many cases:
1. Pt 9t gives us the vanilla SGD;
2. P. VE tite gives the AdaGrad algorithm (Duchi et al. 2011);
Pt / e) recovers the Adam algorithm (see Algorithm
Note that in the above we use 9: to denote the stochastic gradient of the entire objective function:
f + Ax,-1 (if the regularizer is not present 0), with mn, and i, both updated using 9,-
This update rule 2 is given by the following online mirror descent update (Nemirovsky & Yudin 1983: Warmuth & Jagota 1997 Beck & Teboulle 2003):
Kt = argmin =ER4
x 12 + f(x,-1) p. (a x1-1) + (x - xx-1) M, (x x1-1))
This approximates minimizing first-order Taylor approximation of F centered in #;-1 plus a term that measures the distance between the a and 1;-1 according to the matrix M,-1. The approximation becomes exact when P; = V (x) + A
However, this is not the only way to construct first-order updates for the objective function 1 An alternative route is to linearize only f and to keep the squared la norm in its function form:
C3 argmin 2े | + f(x(-1) + P. (x x1-1) + 1 (x - - xI-1) M, (xx x1-1).
=ER
This update rule using the proximal operator (Moreau 1965
Parikh & Boyd
2014] of |ll 1|3 with
respect the norm || lIa It is intuitive why this would better update: We directly minimize the squared lg norm instead of approximating From the first-order optimality condition, we have
2; (Ia + AM:)- M.p:).
When À 0, the update in 2 and this coincide. Yet, when À & 0, they longer the same.
3

---

Under review as a conference paper at ICLR 2022
We now show how the update in generalizes the one in AdamW. The update of AdamW is
It (1 Amx xt-1 - naam///// 0, + +c). )
(4)
On the other hand, using M,- 1/: la and P: wwt.//vvv +e) in (3) gives:
3t = (1 + Am)
naam ( 8r + +)),
(5)
which we will call AdamPron afterward. Its first-order Taylor approximation around 7/. is
2, (1 - - /.omm./( 0b + e).
which is exactly the AdamW update (4) as in the original paper.
The careful reader might notice that the approximation of AdamW to AdamProx becomes less accurate when '/ becomes too large, and so be concerned whether this approximation is practical at all. Fortunately, in practice, ?) never large enough for this to be an issue. Most practical learning rate schedules, c.g., cosine. exponential, polynomial, and step decay, all decrease from 770 1, or some even smaller value. Thus, the remainder term of this approximation is O(vn²) S O(Ar)) which we should always expect to be small as both À and π) are small. Consequently, we can expect AdamW W and AdamProx to perform similarly for learning rate schedules 1)e commonly employed in practice, and we will indeed confirm this empirically in Section 4.3
While perhaps not widely known, the theory of proximal methods is a deep and beautiful branch of optimization (see Parikh & Boyd (2014) for survey). On the other hand, although Adam is very popular practical algorithm (c.g., for training BERT (Devlin et al. 2019) and vision transformer- architectures DDsovvisskiy al. 202 ). is still unclear how it works so well. So, this connection opens the door to new ways to design optimization algorithms. For example, the proximal update gives an immediate theoretical advantage: the convergence rate, at least in the convex case, will depend on | V fzx) l2,4, rather than on [Vz)( + 1 Duchi et al. (2010b).
A Adam W is Scale-Free An optimization algorithm is said to be scale-free if its iterates do not change when one multiplies any coordinate of all the gradients by positive constant (Orabona & Pal 201 8). turns out that the update 44) of Adam and the update (5) of AdamProx are both scale-free free when e= O. This is evident for AdamW, since the scaling factor for any coordinate of the gradient is kept in both mi and V and will be canceled out when dividing them. In contrast, for Adam- the addition of the weight decay vector to the gradient (see Line 5 of Algorithm I destroys this property. Note that in practical applications c is very small but not zero, yet we empirically verify in Section 4.2 that it is small enough to still approximately guarantee the scale-free property.
We want to emphasize the comparison between Adam-lz and AdamW: once Adam- adopts non- zero X, it loses the scale-freeness property; in contrast, Adam W still enjoys this for arbitrary X. The same applies to any AdaGrad-type and Adam-type algorithms that employ weight decay strategy in the same way as Adam-&2 by simply adding the gradient of the E regularizer directly to the gradient of f (as implemented in Tensorflow and Pytorch). Such algorithms are scale-free only when they do not use weight decay. In fact, this is one of our main observations: ccale-freeness helps optimizers in deep learning and Adam preserves this feele-freeness property even with an Eg regularizer.
We stress that the scale-freeness is an important but largely overlooked property of an optimization algorithm. It has already been utilized to explain the success of AdaGrad (Orabona & Páll 2 (018). Recently, /Agaraal et al. (2020) also advocates for setting the e in the denominator of AdaGrad to be O thus making the update indeed scale-free, and they also provide an NLP experiment where such choice yields the best performance.
Now, we show the effect of craee-frenness from a theoretical point of view. Consider twice contin- uously differentiable function f. is well-known that the convergence rate of many optimization algorithms, like gradient descent, minimizing depends on the condition number k 2 f(r)) i... the ratio of its largest eigenvalue to its smallest cigenvalue of the Hessian (sec, c.g. Nesterov 2004).
It turns that scale-freeness can effectively reduce the effects of the condition number, as detailed by the following theorem whose proof can be found in the appendix.
Theorem 1. Let I be twice continuously differentiable function such that x² E argmin f(x). Then, let fa be the family of functions such that x" E argmin fa(x), and V² = AV f(x) where A = -dia((X...... ( >a) 2 O. Consider a scale-free algorithm whose convergence rate can be bounded depending on the condition number of the objective function. Then, the convergence of such algorithm used to minimize I will only depend on the smallest condition number among all functions sx-
4

---

Under review as a conference paper at ICLR 2022
To give example of when this is advantageous, consider when f(x) is diagonal matrix:
v² = (2),
dd. Denote j == arg max; M,/u4. Choose X; s.t. My Xij4 S (z) S A,M, S M, and we have that AV2f(z) has condition number x'
Assume 1 S #4 S (x(x) S M, < M for E [I....,d]].
My Mz. This gives scale-free algorithms a big advantage when max; M//u: < M/H. Specifically:
Corollary 1. For quadratic problems f(z) 1 Hx + 6" x with diagonal and positive definite, any scale-free algorithm will not differentiate between minimizing f and f(x) = 1
distancc minimizer
10a
H H- x + As the condition number of f is 1, the operation, and most importantly, the convergence, of a scale-free algorithm will not be affected by the condition number of J at all.
10-1
Adam/ GD
10
10
Reration
Figure Tillustrates Corollary 1. Here, we compare GD (non-scale- free) with A (scale-free) on optimizing two quadratic func- tions with the same minimizer, but one hessian matrix being rescaled version of the other, resulting in different condition num- bers. Even after tuning the learning rates, the figure clearly shows that the updates of AdamW (starting from the same point) and thus its convergence the minimizer, is completely unaffected by the condition number, while GD's s updates change drastically and its performance deteriorates significantly when the condition number is large. Moreover, note that poor training performance mostly likely will also lead to poor testing performance.
(a) Condition number 1.
distarnce minimzer
10a
AdamW
GD
10
(b) Condition number 100000.
Figure 1: Non-scale-free GD v.s. scale-free Adam W on quadratic functions with differ- ent condition numbers.
This also explains AdaGrad's improvements over SGD in certain scenarios. As example, in Appendix B analyze variant AdaGrad with restarts and show an improved convergence on strongly convex functions due to scale-ficeness. Note that the folklore justification for such improve- ments is that the learning rate of AdaGrad approximates the inverse of the Hessian matrix, but this is incorrect: AdaGrad does not compute Hessians and there is no reason to believe it approximates them in general.
More importantly, another scenario demonstrating the advantage of scale-freeness is training deep neu- ral networks. Neural networks are known suffer from the notorious problem of vanishing/epploding
gradients (Bengio et al.) 1994; Glorot & Bengio, 2010 Pascanu et al. 2013). This problem leads to the gradient scales being very different across layers, especially between the first and the last layers. The problem is particularly severe when the model is not equipped with normalization mechanisms like Batch Normalization (BN) (Toffe & Szegedy 2015). In such cases, when using non-scale-fres free optimization algorithm (e.g. SGD), the first layers and the last layers will proceed at very different speeds, whereas a scale-free algorithm ensures that each layer is updated at a similar pace.
4 EMPIRICAL FINDINGS OF ADAMW ON DEEP LEARNING EXPERIMENTS
In this section, we empirically compare Adam- with AdamW. W. First (Section 4.1 we report experiments for deep neural networks on image classification tasks (CIFARI0/100). Here, AdamW enjoys significant advantage over Adam-&, when the batch normalization is switched off on deeper neural networks. We also report the correlation between this advantage and the scale-freeness property of AdamW. Next (Section 4.2). we show that AdamW is still almost scale-free even when the used in practice not O, and how, contrary to AdamW, Adam- not invariant of the loss function being multiplied by positive constant. Finally (Section 4.3 we show that AdamW performs similarly to AdamProx in practice, thus supporting the observations in Section 3
Data Normalization and Augmentation: We consider the image classification task on CIFAR- 10/100 datasets. Images are normalized per channel using the means and standard deviations computed from all training images. We adopt the data augmentation technique following Lee et al. (2015) (for training only): 4 pixels are padded on each side of an image and a 32 x 32 crop is randomly sampled from the padded image its horizontal flip.
5

---

Under review as a conference paper at ICLR 2022
Adom Bei.eBBoo.oe tox emo 0.000
MlnmN Rez croy
Jiby Reg tor cre 0.024
Minol Redt tec one
0.05
04
6.55
04
94
04
chpole (0.2011
.011
Iritel ctepita (o..m
0.3
01
0.
0.2
02
02
0.1
0.
WWeghht deray (n0.0m)3
Weight decay (S.001)
Weight decay (oc.00))
Weight decay (55.001)
(a) 20 Layer Resnet CIFARIO
(b) 56 Layer Resnet CIFARIO
Alumti-Be:
0.4
04
1900 1nttl stopize 0000
Jdaam 0.0765
Adamil: toct 0.0770
0.5
0.5
1502003
InD strep zo (150003
Initisl shapsir
0.2
0.5
0.1
01
100
0.0
100
00
0.0
00
Wi g14 desay 1"0.0001
g16 desay 1"00001
Vi-ght decay (N0.001)
Weigh docay (M5,001)
(c) 110 Layer Resnet CIFARIO
(d) 218 Layer Resnet CIFARIO
sropde 20.001
8.66
0.8
305
Ademmt---Seet
05
30
0,333 010.1 0.5 Wesghe deday (*3.003)
C.R. Weight decky 10.001)
(c) 100 layer DenseNet-BCC on CIFAR100
Figure 2: The final T test error on using Adam vs. Adam-E2 on training a Resnet/DenseNet with Batch Normalization on CIFAR10/100 (where the black circle denotes the best setting).
Models: For the CIFAR-10 dataset, we employ the Residual Network model (He et al. 20/44/56/1 10/218 layers; and for CIFAR-100, we utilize the DenseNet-Boe model (Huang al. 2017) with 100 layers and growth-rate of 12. The loss is the cross-entropy loss.
2016 of
Hyperparameter tuning:
For both Adam-l2 and Adam W. we set 31 = 0.9, 82 = 0.999, e 10-8 as suggested in the original Adam paper Kingma & Ba (215)). To set the initial step size and weight decay parameter X, we grid scarch over {0.00005, 0.0001, 0.0005, 0.001, 0.005) for a and {0, 0.00001, 0.00005, 0.0001, 0.0005, 0.001} for X. Whenever the best performing hyperparameters li in the boundary of the searching grid, we always extend the grid to ensure that the final best- performing hyperparameters fall into the interior of the grid.
Training: For each experiment configuration (e.g. 110-layer Resnet batch normalization), we randomly select initialization of the model to hyperparameter settings. We use a mini-batch of 128, and train 300 epochs unless otherwise specified.
fixed starting point for all optimizers and
4.1 ADAMW vs. ADAM-&2 ON IMAGE CLASSIFICATION TASKS, THE INFLUENCE OF BATCH NORMALIZATION, AND THE CORRELATION WITH SCALE-FREENESS
With BN, Adam-L2 is on par with Adam W: Recently, Bjorek al. (2200) found that AdamW has no improvement in absolute performance over sufficiently tuned Adam-&z in some reinforcement learning experiments. We also discover the same phenomenon in several image classification tasks, see Figure 2 Indeed, the best weight decay parameter is 0 for all cases and Adam W coincides with Adam-l, in these cases. Nevertheless, AdamW W does decouple the optimal choice of the weight decay parameter from the initial step size much better than Adam- in all cases.
Removing BN: Notice that the models used Figure all employ Batch Normalization (BN) (Ioffe
Szegedy 2015). BN works by normalizing the input each layer across the mini-bated to make each
http:///thhhhoooo//kaasssssrrrrr ci far

---

Under review as a conference paper at ICLR 2022
ARY Eaet
Man
Boch
Good
(a) 20 Layer Resnet on CIFAR1O
02000000000 Orter 0000 9721
of mag Back of
(b) 44 Layer Resnet on CIFARIO
Adter Eaett
5700711818112 6-3
Epon
(c) 56 Layer Resnet on CIFAR1O
13
Adtrn
(d) 110 Layer Resnet on CIFARIO
Adtnr East
(e) 218 Layer Resnet on CIFARIO
(f) 100 layer DenseNet-BC on CIFAR100
Figure 3: On using Adam vs. Adam-l2 on training a Resnet/DenseNet without Batch Normalization on CIFAR10/100, (Left two) The final Top-1 test error (where the black circle denotes the best setting). (Middle two) The training loss and test accuracy curve when employing the initial step size and the weight decay parameter that gives the smallest test error. (Right two) The histogram of the magnitude of corresponding updates of all coordinates of the network near the end of the training when employing the initial step size and the weight decay parameter that gives the smallest test error.
coordinate have zero-mean and unit-variance. Without BN, deep neural networks are known to suffer from gradient explosion and vanishing (Schoenholz al., 2017). This means each coordinate of the gradient will have very different scales, especially between the first and last layers. For non-scale-fiee free algorithms, the update to the network weights will also be affected and each coordinate will proceed at different pace. In contrast, scale-free optimizers are robust to such issues as the scaling of any single coordinate will not affect the update. Also, we already mentioned the introduction some other reasons for considering removing BN. Thus, we consider the case where BN is removed as that is where AdamW and Adam- will show very different patterns due to ccale-freeness.
Without BN, AdamW W Outperforms Adam- In fact, without BN, AdamW outperforms Adam- even when both are finely tuned, especially on relatively deep neural networks (see Figure 3). AdamW not only obtains much better test accuracy but also trains much faster.
Correlation between Adam W's Advantage and Scale-freeness: We also observe that the advantage of Adam W becomes more evident the network becomes deeper. Recall that as the depth grows, without BN, the gradient explosion and vanishing problem will become more severe. This means that for the non-scale-free Adam-l2, the updates of each coordinate will be dispersed on wider range of
7

---

Under review as a conference paper at ICLR 2022
Adanny--ee vest aroon 1055
Adamii- Bist lod ertr 1122
04
Inba ptopates (0. (Q.003)
Adom. Best lest emon 0 0,1452
0.03
0.1
Adamt-Ba Buok bedt ener 0.1389
C2 C
03
puopsies
0.5
02
0.05
011
0.002/55.11 0.5
Weight decay (.30011)
Weight dacay (P5.001)
0.0
0.0
0.0
0.
Weight decay (90,00)
Weight decay (70.001)
(a) Loss multiplied by 10
(b) Loss multiplied by 100
Figure 4: The final top-1 test error of A vs. Adam- on optimizing a 110-layer Resnet with batch normalization removed optimized by AdamW Adam with the loss function multiplied by 10 (left two figures) and 100 (right two figures).
scales even when the same weight decay parameter is employed. In contrast, the scales of the updates of AdamW will much more concentrated in a smaller range. This exactly verified empirically as illustrated in the 5th & 6th columns of figures in Figure 3 There, we report the histograms of the absolute value of updates of Adam-l2 l2 vs. Adam of all coordinates near the end of training (for their comparison during the whole training process please refer to the Appendix).
This correlation between the advantage Adam over Adam-l2 and the different spread cor- responding update scales which is induced by the ccale-freeneee property of AdamW provides an empirical justification on when and why Adam W excels over Adam-
SGD and craae-freenness The reader might wonder why SGD is known to provide state-of-thee performance on many deep learning architectures (e.g., He et al.) 2016 Huang et al. 2017) without being scale-free. At first blush, this seems to contradict claims that freenese correlates with good performance. In reality, the good performance of SGD in very deep models is linked to the use of BN that normalizes the gradients. Indeed, we verified empirically that SGD fails spectacularly when BN is not used. For example, on training the 110 layer Resnet without BN using SGD with momentum and weight decay of 0. 0.0001, even a learning rate of 10 will lead to divergence.
4.2 VERIFYING SCALE-FREENESS ADAMW VS. ADAM- UNDER LOss MULTIPLICATION
In the previous section, we elaborated on the scale-freeness property of AdamW and its correlation with AdamW's advantage over Adam- However, one may notice that practice, the e factor in the AdamW update is typically small but not O, in case 1e-8, thus preventing from completely free. In this section, we verify that the effect of such an on the ceale-freeness is negligible.
As simple empirical verification of the fcaeesfre,ness, we consider the scenario where we multiply the loss function by a positive number. Note that any other method to test scale-freeness would be equally good. For feed-forward neural network without BN, this means the gradient would also be scaled up by that factor. In this case, the updates of a scale-free optimization algorithm would remain exactly the same, whereas they would change for an optimization algorithm that is not scale-free.
Figure 4 shows results of the loss function being multiplied by 10 and 100 respectively on optimizing 110-layer Resnet with BN removed. For results of the original loss see Figure [3]] We can that AdamW has almost the same performance across the range of initial learning rates and weight decay parameters, and most importantly, the best values of these two hyperparameters remain the same. This verifies that, even when employing (small) oonzzero- Adam is still approximately scale-free. In contrast, Adam-&z is not scale-free and we can see that its behavior varies drastically with the best initial learning rates and weight decay parameters in each setting totally different.
4.3 ADAMW AND ADAMPROX BEHAVE VERY SIMILARLY IN PRACTICE
In Section we showed theoretically that AdamW is the first order Taylor approximation of AdamProx (update rule (5)) Beyond this theoretical argument, also verified empirically that the approximation is good. In Figure 5 we consider the case when 7) = for all t relatively large constant learning rate schedule. In such cases, AdamW and AdamProx still behave very similarly. This suggests that for most learning rate schedules, e.g., cosine, exponential, polynomial, and step decay, which all monotonously decrease from To 1, AdamProx will remain a very good approximation to AdamW. Thus, it is reasonable to try to understand AdamW by instead understanding the more classically-linked AdamProx.
8

---

Under review as a conference paper at ICLR 2022
Alrnd Del entor 1:112
04
Albmitae Bew.-Booooooo coo 1022
Adamwll first Tost croon 3899
0.00
Adomhrow ted or
2055
0.00
copplie (0.000)
9.05
Cas 01
Inbtet expptte 00/011
0.55
0.1
0.3
9.5
6.5
0.1
Intral
0 0.1 ads O.L.
8.0 D.06, 6.5. 0.5
0.01 0:03 B.J. B.
6.81 0.05 B.1. D.S.
Beight
Weight decay
(a) ResNet on CIFAR-10
(b) DenseNet-BC on CIFAR-100
Figure The final T test error of using Adam vs. AdamProx on training (where the black circle denotes the best setting). (Top row) 110-layer ResNet with Batch aatch-normaliztioon removed on CIFAR- 10 (trained for 300 epochs). (Bottom row) 100-layer DenseNet-BC with Batch-normalization removed on CIFAR-100 (trained for 100 epochs).
CONCLUSION AND FUTURE WORKS
In this paper, we provide explanations for the merits of AdamW from two points of view. We first show that A is an approximation of the proximal updates both theoretically and empirically, We then identify the setting of training very deep neural networks without batch normalization which Adam W substantially outperforms Adam-& in both training and testing, and provide an empirical explanation through the ccale-freeness property of AdamW. Nevertheless, there still many directions worth exploring and we name a few here.
Update for no-square & regularization. In- stead of using the squared &g regularization, we the lg regularization, that
04
04
Adunii-Buu Bidd lasi ere 0.100004444
6:144
Alumheea22- Buok texi ener: 0.1 0.11
0.05
03
0.05
03
might think
0.3
without the square. This is known to have
better statistical properties than the squared &
(Oraoona 2014), but is not smooth so it is harder to be optimized. However, with proxi- mal updates, we don't tt have to worry about its mon-smoothness. Hence, we can consider the objective function
Weight decary (OLLC1)
0.0
Weight detay (P0.001)
0.0
Figure 6:
The final Top-1 test eгror of using AdamW vs. AdamProxL2 train 110-layer ResNet with BN removed on CIFAR10 (where the black circle denotes the best setting).
F(z) = A||=||2 + f(x).
The corresponding prox-SGD update was derived in Duchi & Singer 2009) for scalar learning rates and it easy that generalizes to our setting for M. 11:
Any m.P (I, - 7.P.)
E++1
max
Its performance, named AdamProxL2, as shown in Figure can be on a par with AdamW.
Update for é regularization. The E regularization is widely used for achieving the sparsity of the solution. Recently, Neyshabur 2020)) proposed B-lasso, variant of the LASSO algorithm namely regression with & regularization, and showed significant improvements on training fully-connected networks. Thus, we are curious see how the variant of Adam W derived from using & regularization instead will behave. Specifically, we consider the objective function:
F(x) A||2|11 f(x),
for which the proximal update is obtained by solving:
argmin [(x (æ: M,P.)) M (x, M.P.)) A||=|111
When M, a diagonal matrix with positive eigenvalues, the above objective function is decomposable and we can solve for each dimension. The solution for 1-d case was derived Duchi & Singer (2009) and it is easy to see that it generalizes to our setting as
1,+1.
sign( M.p.); max(la;
M.p,le XM,,
Distributed Training As already noted in Section 4.1 batch normalization not very compatible with distributed training. Since AdamW outperforms Adam-E, significantly in settings with BN switched off, at least in ferd-forward neural networks, we can apply Adam in distributed training see if it still enjoys the same merits.

---

Under review as a conference paper at ICLR 2022
REFERENCES
N. Agarwal, R. Anil, E. Hazan, T. Koren, and C. Zhang. Disentangling adaptive gradient methods from learning rates. arXiv preprint arXiv:2002.11803, 2020.
H. Asi and J. C. Duchi. Stochastic (approximate) proximal point methods: Convergence, optimality, and adaptivity. SIAM Journal on Optimization, 29(3):2257-2290, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06445 2016.
A. Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167-175, 2003.
Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Bjorek, K. Q. Weinberger, and C. Gomes. Understanding decoupled and early weight decay. ArXiv, abs/2012. 13841, 2020.
S. Bos and E. Chug. Using weight decay to optimize the generalization ability of perceptron. Proceedings of International Conference on Neural Networks (ICNN'96) '96, 1:241-244 vol.1, 1996.
Andrew Brock, Soham De, Samuel L.. Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In Proc. of the International Conference Machine Learning (ICML), pp. 1059-1071, 2021.
Samuel Rota Bulò, Lorenzo Porzi, and Peter Kontschieder, In-place activated batchnorm for memory- optimized training of dnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5639-5647, 2018.
N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. Hal Daumé, III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020.
Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks. Advances in Neural Information Processing Systems, 33, 2020.
J. Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Duchi and Y. Singer. Efficient online and batch learning using forward backward splitting. The Journal of Machine Learning Research, 10:2899-2934, 2009.
J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. In COLT, 2010a.
C. Duchi, S. Shalev-Shwartz, Y. Singer, and A. Tewari. Composite objective mirror descent. In COLT, pp. 14-26, 2010b.
J. C. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, 2011.
Igor Gitman and Boris Ginsburg. Comparison of batch normalization and weight normalization algorithms for the large-scale image classification. arXiv preprint 1709v 0414, 2017.
10

---

Under review as a conference paper at ICLR 2022
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proc. of the thirteenth international conference on artificial intelligence and statistics, pp- 249-256. JMLR Workshop and Conference Proceedings, 2010.
P. Goyal, P. Dollár, R. B. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch SGD: Training ImageNet in hour. ArXiv, 2017.
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In Proc. of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International conference on machine learning, pp. 148-45.. PMLR, 2015.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
A. Krogh and J. Hertz. A simple weight decay can improve generalization. NIPS, 1991.
Kuen, F. Perazzi, Z. Lin, Zhang, and Y.-P. Tan. Scaling object detection by transferring classification weights. In Proc. of the IEEE/CVF International Conference Computer Vision, pp. 6044-6053, 2019.
Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
C. Lee, Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply-supervised nets. In Proc. of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38, pp. (62-570. PMLR, 2015.
Y. Lifchitz, Y. Avrithis, S. Picard, and A. Bursuc. Dense classification and implanting for few-shot learning. In Proc. of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9258-9267, 2019.
I. Loshchilov and F. Hutter. SGDR:
Stochastic gradient descent with warm restarts. International Conference on Learning Representations, 2017.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
L. Luo, Y. Xiong, Y. Liu, and X. Sun. Adaptive gradient methods with dynamic bound of learning rate. In International Conference on Learning Representations, 2018.
B. Martinet. Régularisation d' d'inéquations variationnelles par approximations successives. rev. française informat. Recherche Opérationnelle, 4:154-158, 1970.
H. B. McMahan and M. J. Streeter. Adaptive bound optimization for online convex optimization. In COLT, 2010.
J.-J Moreau, Proximité et dualité dans un espace hilbertien. Bulletin de Société mathématique de France, 93:273-299, 1965.
A. S. Nemirovsky and D. Yudin. Problem complexity and method efficiency in optimization. Wiley, New York, NY, USA, 1983.
Y. Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer, 2004.
B. Neyshabur. Towards learning convolutions from scratch. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 8078-8083 Curran Associates, Inc., 2020.
11

---

Under review as a conference paper at ICLR 2022
F. Orabona. Simultaneous model selection and optimization through parameter- free stochastic learning. Advances in Neural Information Processing Systems 27, 2014.
F. Orabona and D. Pál. Scale-free online learning. Theoretical Computer Science, 716:50-69, 2018. Special Issue on ALT 2015.
N. Parikh and S. Boyd. Proximal algorithms. Foundations and Trends optimization, 1(3):127-339 2014.
R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318, PMLR, 2013.
H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22:400-407, 1951.
R. Rockafellar. Monotone operators and the proximal point algorithm. SIAM journal on control and optimization, 14(5):877-898, 1976.
S. Schoenholz, J. Gilmer, S. Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations (ICLR), 2017.
Saurabh Singh and Abhinav Shrivastava. Evalnorm: Estimating batch normalization statistics for evaluation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3633-3641, 2019.
M. Tan and Quoe V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. ArXiv, abs/1905.1 1946, 2019.
Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
P. Toulis and E. M. Airoldi. Asymptotic and snite-sample properties of estimators based stochastic gradients. The Annals of Statistics, 45(4): 1694-177777 2017.
M. K. Warmuth and A. K. Jagota. Continuous and discrete-time nonlinear gradient descent: Relative loss bounds and convergence. In Electronic proceedings of the 5th International Symposium on Artificial Intelligence and Mathematics, 1997.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp. 3-19, 2018.
M. D. Zeiler. ADADELTA: an adaptive learning rate method. arXiv preprint arXiv: 1212.5001, 2012.
H. Zhang, Y. N. Dauphin, and T. Ma. Fixup initialization: Residual learning without normalization. In International Conference on Learning Representations, 2019.
D. Zhou, Y. Tang. Z. Yang, Y. Cao, and Q Gu. On the convergence adaptive gradient methods for nonconvex optimization, 2018, arXiv: 808.05671.
12

---

D Under review as a conference paper at ICLR 2022
Appendices
A PROOF OF THEOREM
Proof. From the Fundamental Theorem of Calculus we have:
=f(z)= L. f2f(2* =t(x==²)){== dt L. f(2'2=+t(2==" (x)))= d'
Thus, for any function (x) whose Hessian AV f(x) and fa(=*) = 0.
Now, from the definition of scale-free algorithm, the iterates of such algorithm do not change when one multiplies any coordinate of all the gradients by a positive constant. Thus, a scale-free algorithm optimizing f behaves the same as if optimizing fa-
B A SCALE-FREE ALGORITHM WITH DEPENDENCY ON THE CONDITION NUMBER
Algorithm 2 AdaGrad (Duchi et al. 2011: McMahan & Streeter 2010) (All operations vectors are element-wise.)
Input: #lterations T, set IC, 2 E K. stepsize !
for = 1...I do
Receive: fx()
Set: 17:
VELL((
Update: x++1 = IIx (I, 77 f(x)))
where II is the projection onto K.
end for
Output: . = + Et=
Algorithm 3 AdaGrad with Restart
Input: #Rounds N, æb IRd, upper bound on ||200 x= [|> as Doc, strong convexity smoothness M
Set: Ey Zy N do
for i = 1...N.
Run Algorithm to get with 32d 1 end for Output: ØN-
for: | 5. 11-1 S 01 = 5;-1,
122
Doc/vI
2
Theorem 2. Let K. be hypercube with ||x - yllxc Doo for any n, E K. For convex function f, set D=s then Algorithm guarantees for any x E K.:
v2
T
T
Σ f(x)) f(x)
2dD2 Σ [V)())
(6)
Theorem 3. For μ strongly convex and M smooth function f. denote its unique minimizer x² E Rdd Given Bo E Rd, assume that | 1lo 5 Doe. then Algorithm guarantees:
| xx" 113
4N
Thus, get a such that || x0. S need most 32d M a
log, (D2 gradient calls.
Proof of Theorem 3 Consider round i and assume X. passed to Algorithm 2 bounded w.r.t. loo
norm by Dx... When f is u-strongly convex and M smooth, let æ = x", Equation (6) becomes:
f(x) - fx=)
2dD2 20:
V 11
1MdD2 8 Σ (s(s)) - f(x)))
13

---

Under review as a conference paper at ICLR 2022
where the second inequality is by the M smoothness of f. This gives:
-(x))-((a - 4M
Let 8; = + EiT De we have by the j-strongg-onvvextty that:
||:; x² 1l a. x01 2 (
f(x)) S (((m)) f(x)))
2 p P
8MdD KT
(7)
Put T' 32d in Equation 77) we have that || x 1|3 I D2 Thus, after each round, the Loo distance between the update ã. and is shrinked by half, which turn ensures that x still inside the K. passed to Algorithm in the next round with D Do0.4
Pas, This concludes the proof. m
2
Proof of Thherrmm
T
Σ f(x1)= - f(x)
T
V (xx)) xt x))
T
05 (It) * (Dr - 2z))
dmi,j
2
(Fes - -,,)² - -(x13 Te. aor.)
0f (xx) - x3/)
0f (xi)
+
Σ
2m
2
dzi.
t=1
1=1
(Fr. - 22,)² (5c+1.d - =,,)²
T
d
Th
0f
(xi1)
2
+
2m
2
(=
,=1
(Ir - x,,)
nes
0f (ai)
2
m1-1.5
2
j=1
j=1
D2
as Oais
(a::)
as (ax.)
11
as Oer.
(a:))
2
2n
Orys
2/ Σi=1
0s رon,
(2:))
D2
0s
2
0s (xx1)
2
Σ
(Ix)
+ 7
Σ
21
for,
d
T Σ
0f (xI1
2
2D2
deri
1=1
1=1
0f Fos,,
2dD2
(xx)
T (.
2dD2 0
where the first inequality is by convexity, the second one by the projection lemma the projection onto a hypercube equals performing the projection independently for each coordinate, the fifth one by Lemma 5 in (MeMahan & Streeter 2010), and the last one by the concavity of V. n
14

---

Under review as a conference paper at ICLR 2022
C THE HISTOGRAMS OF THE MAGNITUDE OF EACH UPDATE COORDINATE DURING THE ENTIRE TRAINING PHASE
In this section, report the histograms of the absolute value of updates of Adam- AdamW of all coordinates divided by a during the whole training process. From the figures shown below, we can clearly see that AdamW's updates remain in much more concentrated scale range than Adam- during the entire training. Moreover, as the depth of the network grows, Adam-&z" 's updates become more and more dispersed, while AdamW's updates are still concentrated. (Note that the leftmost bin contains all values equal or less than 2 27 = 10 10-8.1 8.1 and the rightmost bin contains all values equal to or larger than 1.)
15

---

Under review as a conference paper at ICLR 2022
Adam Epoch
AdamW Epoch
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
22-24221-1815-12
9 - -3 0
-27-24-2--8-115-12 -9
-6
-3
0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
27-2422118-1512--
-6 -3 0
-27-24-21-8-115-12
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
22-22421-1815:122-
-6 -3
0
-27-24-21-18-15-12
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-21-2421-88551--9 -6 -3 0
-27-24-21-1815--22 99 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-
(b) AdamW
Figure 7: The histograms of the magnitudes of all updates (without a) of a 20-layer Resnet with BN removed trained by AdamW or Adam-l, on CIFARIO.
16

---

Under review as a conference paper at ICLR 2022
Adam Epoch
AdamW Epoch
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-2-24421-8815-12
9 - -3 0
-27-24-2--88115-12 -9
-6
-3
0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
27-24221-8-15-12-9
-6 -3 0
-27-24-21-88115-12 -9
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
27-2422118-15:12--
-6 -3
0
-27-24-21-18-15-12
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-21-2421-81512-99 -6 -3 0
-27-24-21-1815--22 99 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-
(b) AdamW
Figure 8: The histograms of the magnitudes of all updates (without a) of a 44-layer Resnet with BN removed trained by AdamW or Adam-l, on CIFARIO.
17

---

Under review as a conference paper at ICLR 2022
Adam Epoch
AdamW Epoch
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
22-24221-8815-12
9 - -3 0
-27-24-2--8-155-12 -9
-6
-3
0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
27-2422118-1512--
-6 -3 0
-27-24-21-88115-12 -9
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
27-2422118-15-12-
-6 -3
-27-24-21-18-15-12
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-21-2421-81512-99 -6 -3 0
-27-24-21-1815-122 99 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-
(b) AdamW
Figure 9: The histograms of the magnitudes of all updates (without a) of a 56-layer Resnet with BN removed trained by AdamW or Adam-l, on CIFARIO.
18

---

Under review as a conference paper at ICLR 2022
Adam Epoch
AdamW Epoch
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
22-24221-8815-12
9 - -3 0
-27-24-2--8-155-1 -9
-6
-3
0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
27-24221-18155112
99 -6 -3 0
-27-24-21-18-15-12 -9
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
22-22421-18155-122
-6 -3
-27-24-21-18-15-12
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-27242211815-1--9 -6 -3
-27-24-21-18151122 9 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-
(b) AdamW
Figure 10: The histograms of the magnitudes of all updates (without a) of 110-layer Resnet with BN removed trained by Adam W or Adam-l, on CIFAR10.
19

---

Under review as a conference paper at ICLR 2022
Adam Epoch
AdamW Epoch
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
22-24221-1815-12
9 - -3 0
-27-24-2--8-155-12 -9
-6
-3
0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
27-2422118-15-12-9
-6 -3 0
0.0
-27-24-21-1815-112 -9
-6
-3
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
27-24-21-1-11-12--
-6 -3 0
-27-24-21-18-15122
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-21242211815-12-9 5
-27-24-21-1815-122 -9 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-
(b) AdamW
Figure 11: The histograms of the magnitudes of all updates (without a) of 218 218-layer Resnet with BN removed trained by Adam W or Adam-l, on CIFAR10.
20

---

Under review as a conference paper at ICLR 2022
Adam Epoch
AdamW Epoch
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
22-2422118815-12
9 - -3 0
-27-24-21-8811--12 -9
-6
-3
0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
27-2422118-15122-
-6 -3 0
-27-24-21-88115-12 -9
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
22-24221-8-15:12-9
-6 -3
-27-24-21-18-15-12
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-2122221-81512-99 -6 -3 0
-27-24-21-18-5-122 99 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-
(b) AdamW
Figure 12: The histograms of the magnitudes of all updates (without a) of a 100-layer DenseNet-BC with BN removed trained by Adam or Adam-l, on CIFAR100.
21


Reviews:
[{"review_id": "uD2QLshNmq", "reviewer": "Reviewer_xwd7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The main contribution comes from providing the connection between the proximal operator and the AdamW updating, which is interesting.\n2. The submission further shows that scale-freeness is an essential property of AdamW compared to Adam-l2. Moreover, scale-freeness can help mitigate the impact of condition number, which significantly affects the convergence speed.\n\nWeaknesses:\n1. One problem is that the submission shows that scale-freeness is essential for the optimizer. However, many optimizers share this property, such as adaGrad; this submission does not show why AdamW is generally better than the others \n2. For Line 5 in Alg.1, if we scale the whole gradient $g_t$, we also get the scale-freeness? Do the author mean only scale the gradient of $f$. ie.e,$\\nabla f$? It seems that all the adam-type optimizers all have this property. Why is AdamW special?\n\nMinor Problem:\n1. 'AdaGrad does not compute Hessians, and there is no reason to believe it approximates them in general.', which confused me. The quasi-Newton method also does not compute Hessians, but it indeed approximates the inv of Hessians. Moreover, scale-freeness certainly is superior in the scenario that the identity matrix can reasonably approximate the hessian. Why deny such a view here?", "summary_of_the_paper": "An interesting finding shows that the advantages of AdamW compared to the vanilla Adam-l2.", "summary_of_the_review": "An interesting paper provides a good finding.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ZI9kVPy-D3G", "reviewer": "Reviewer_rZ6j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Authors show that AdamW can be seen as an approximation of the proximal updates, utilizing the closed form of proximal mapping of the regularizer. It is nice to see the reasonable theoretical explanations to the AdamW optimizer.\n2. Authors analyze the scale-freeness property of AdamW, and further show the advantages of the scale-freeness property which is convincing to explain why AdamW gets better performance when the network is deeper or not equipped with batch normalization.\n\nWeakness:\nAuthors conduct experiments in small-scale datasets with convolutional neural networks. Actually, a more realistic setting is to validate the vision transformer-based architectures, which is more meaningful because (1) transformer is equipped with AdamW as a standard setting (2)  vision transformers are not equipped with the batch normalization. So I thought if the theoretical analysis can be validated in the transformer-based architectures, and with larger dataset such as ImageNet, the results will be more convincing.\n\nTypos: In section 4 the last sentence of the first paragraph, \"??\" should be modified.", "summary_of_the_paper": "The paper give an analysis to the AdamW optimizer. This paper gives a theoretical view of the AdamW optimizing algorithm and connect it to the proximal gradients methods. Authors also explore in depth the advantages of AdamW over AdaM with l2 regularizer, from the perspective of scale-freeness property. By analyzing the scale-freeness property, authors find that the training is more stable in certain scenarios with large conditional numbers. Authors conduct experiments to validate their conclusions. ", "summary_of_the_review": "I lean to accept the paper. The paper analyze the AdamW optimizer well, and the experiments on small dataset support their conclusions. It will be more convincing to conduct experiments in more realistic scenarios such as transformer-based architectures trained on ImageNet.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NmwAgl8Ofm", "reviewer": "Reviewer_Vxer", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n(1)\tIt interprets AdamW as an approximation of a proximal gradient method, and shows the difference between AdamW and Adam-L2. \n(2)\tThe authors also show that AdamW is “scale-freeness” algorithm which may enjoys faster convergence speed\n\nWeaknesses: \n(1)\tIn the abstract, the authors aim to solve the problem: why AdamW generalizes better than Adam-L2. However, in the paper, they actually did not solve or try to solve this problem. \n\n(1.1)\tThey show the difference between AdamW and Adam-L2 from an approximation of a proximal gradient method. But they do not explain why different approximations give different generalizations? Why the approximation of AdamW is better than Adam-L2 in terms of generalization. \n\n(1.2)\tFor the second contribution, AdamW is a “scale-freeness” algorithm that may enjoy faster convergence speed. First convergence speed does not have a necessary correlation to generalization, since one can converge to a very sharp minimum and may generalize worse. Secondly, For the “scale-freeness” algorithm, it is only possible to achieve faster convergence rate. But for AdamW, the authors do not show how/why adamw can achieve a faster convergence rate. This is a big gap. \n\n(1.3) adam is  “scale-freeness”, but its performance is usually worse than adam-l2. So how “scale-freeness” explains this?\n\n(2)\tFor interpreting AdamW as an approximation of a proximal gradient method and showing the difference between AdamW and Adam-L2, this seems to be very trivial in my view. What is the novelty or contribution of this? \n", "summary_of_the_paper": "This work first re-interprets AdamW as an approximation of a proximal gradient method, which takes advantage of the closed-form proximal mapping of the regularizer instead of only utilizing its gradient information as in Adam-l2.  Then it further shows that AdamW is “scale-freeness” algorithm which may enjoys faster convergence speed.  In this way, this work may explain why AdamW generalizes better than Adam-L2.", "summary_of_the_review": "Overall, this work does not provide very new insights for improving adamw or showing the better generalization of adamw over adam-l2. Moreover, it also does not have new proof techniques/frameworks.   ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "IfXJDFERzUL", "reviewer": "Reviewer_tVBU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n- Their work is well-motivated. An insightful interpretation or explanation of AdamW can benefit the whole deep/machine learning community in a way of guiding people on what scenario to use it and how.\n\nWeakness & major concerns:\n- The manuscript definitely need robust editing by a native speaker. To me sometimes it is really hard to follow. \n- The math is not self-contained and the logical flow is disconnected. For example, the line above eq(3) \"This results in the following update rule ...\". But I don't think it is trivial to derive it from previous formulations. TBH, I fail to see how to derive eq(3). I would appreciate if the author could help me to see the connections between eq(3) and previous formulations. \n- I assume eq(5) is the core updating rule of AdamProx, since there is no description of your algorithm. Are you claim that when $\\eta_t$ is small enough, then AdamW is an approximation of eq(5)? Is this assumption too strong? When $\\eta$ is too small, then the overall actual learning rate is almost zero which results in ineffective update.\n- I feel the experiment section just carried out a comparison between AdamW and Adam-$\\ell$2. It supposes to prodive more experiments with AdamProx. \n", "summary_of_the_paper": "In this paper, the authors first interpret AdamW as an approximation of proximal mapping, and then proposed their own algorithm AdamProx. They aim to unravel the connection between AdamW and AdamProx from both theoretical and empirical perspective. They delicately designed some probing experiments to verify their hypothesis. ", "summary_of_the_review": "I recommend to reject the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I agree that reviewer Vxer was confrontational and abusive, especially in the response to the author's rebuttal, and believe that some form of sanction or reprimand is appropriate. That said, I do think that "performance" should be evaluated on both convergence rate and generalization.  Figure 3 does suggest some improvement on generalization for deep versions of resnet without batch normalization.

The three less offensive reviewers all indicated weak acceptance.  One reviewer pointed out the weakness of only getting positive results on deep versions of resnet with batch normalization removed.  Results on transformers, where Adam is typically used, would be more compelling.  This is my primary issue with the paper.  It has not demonstrated improvement in the standard practice of resnet (with batch normalization) and has not presented experiments on transformers.  The theoretical analysis is not aimed at explaining why the improvement is only observed on deep resnets with batch normalization removed or why L2 regularization seems to be of no value when batch normalization is present.  I understand that these are very difficult questions.

The paper has no champion and I am personally concerned about the significance of the contribution.
Decision: Reject