Understanding AdamW through Proximal Methods and Scale-Freeness
Authors: Zhenxun Zhuang
Mingrui Liu
Ashok Cutkosky
Francesco Orabona
Venue: Accepted by TMLR ()

Abstract: Adam has been widely adopted for training deep neural networks due to less hyperparameter tuning and remarkable performance. To improve generalization, Adam is typically used in tandem with a squared $\ell_2$ regularizer (referred to as Adam-$\ell_2$). However, even better performance can be obtained with AdamW, which decouples the gradient of the regularizer from the update rule of Adam-$\ell_2$. Yet, we are still lacking a complete explanation of the advantages of AdamW. In this paper, we tackle this question from both an optimization and an empirical point of view. First, we show how to re-interpret AdamW as an approximation of a proximal gradient method, which takes advantage of the closed-form proximal mapping of the regularizer instead of only utilizing its gradient information as in Adam-$\ell_2$. Next, we consider the property of "scale-freeness" enjoyed by AdamW and by its proximal counterpart: their updates are invariant to component-wise rescaling of the gradients. We provide empirical evidence across a wide range of deep learning experiments showing a correlation between the problems in which AdamW exhibits an advantage over Adam-$\ell_2$ and the degree to which we expect the gradients of the network to exhibit multiple scales, thus motivating the hypothesis that the advantage of AdamW could be due to the scale-free updates.

Published in Transactions on Machine Learning Research (August/2022)
Understanding AdamW through Proximal Methods and Scale-Freeness
Zhenxun Zhuang Boston University
zxzhuang @ bu. edu
Mingrui Liu George Mason University
mingruil @ gmu. edut
Ashok Cutkosky
ashok @ cutkosky.com
Boston University
Francesco Orabona Boston University
crabessco@rrabbbaaccom
Reviewed on OpenReview:
Abstract
Adam has been widely adopted for training deep neural networks due to less hyperparameter tuning and remarkable performance. To improve generalization, Adam typically used in tandem with a squared f2 regularizer (referred to as Adam-E2). However, even better performance can be obtained with AdamW, which decouples the gradient the regularizer from the update rule of Adam-l2. Yet, are still lacking a complete explanation of the advantages of AdamW. In this paper, tackle this question from both optimization and empirical point of view. First, show how to re-interpret Adam W as approximation of a proximal gradient method, which takes advantage of the closed-form proximal mapping of the regularizer instead of only utilizing its gradient information as in Adam- Next, we consider the property of proximal counterpart: their updates are invariant to component-win rescaling of the gradients. We provide empirical evidence across wide range of deep learning experiments showing a correlation between the problems in which AdamW exhibits an advantage over Adam-l2 and the degree which expect the gradients of the network to exhibit multiple scales, thus motivating the hypothesis that the advantage of Adam W could be due to the scale-free updates.
enjoyed by Adam and by its
1 Introduction
Recent years have seen surge of interest in applying deep neural networks (LeCun al. 2015) to myriad of areas (Krizhevsky et al. 2012 Goodfellow et al. 2014 Vaswani et al. 2017 Wu et al. 2020). While Stochastic Gradient Descent (SGD) (Robbins & Monro 1951 remains the dominant method for optimizing such models, its
performance depends crucially on the step size hyperparameter. To alleviate this problem, there has been a significant
amount of research on adaptive gradient methods (e.g. Duchi al. 201Oa, McMahan Streeter 2010 Tieleman &
Hinton 2012 Zeiler 2012, Luo al., 2018 Zhou et al. 2018 Zhang al. 2018 Li & Orabona, 2019 2020 Li al.
2021b. These methods provide mechanisms automatically set stepsizes and have been shown greatly reduce the
tuning effort while maintaining good performance. Among these adaptive algorithmss, one of the most widely used is
Adam (Kingma & Ba 2015), which achieves good results across variety of problems even by simply adopting the default hyperparameter setting. Motivated by its huge successes, there has been much follow-up research addressing the
theoretical convergence of Adam and its variants (Reddi et al. 2018 De et al. 2018 Zhou et al. 2018 Wang et al.
2020 Chen et al. 2019).
On the other hand, practice, to improve the generalization ability, Adam is typically combined with squared & regularization, which will call Adam-&, hereafter. Yet, pointed out by Loshchilov Hutter (2019), the gradient

---

Published in Transactions on Machine Learning Research (August/2022)
of the regularizer does not interact properly with the Adam update rule. To address this, they provide method called AdamW that decouples the gradient of the &2 regularization from the update of Adam. The two algorithms are shown in Algorithm I Although AdamW is very popular (Kuen et al.) 2019 Lifchitz et al., 2019; Carion et al. 2020) and it frequently outperforms Adam- 2, it is currently unclear why it works so well. Recently, however, Bjorek et al. (2021) applied AdamW in Natural Language Processing and Reinforcement Learning problems and found no improvement of performance over sufficiently tuned Adam-l2.
In this paper, we focus on understanding how the AdamW update differs from Adam-&2 from optimization point of view. First, unveil the surprising connection between AdamW and proximal updates (Parikh & Boyd 2014). In particular, we show that Adam is an approximation of the latter and confirm such similarity with an empirical study. Moreover, noticing that AdamW W and the proximal update are both scale-free while Adam- is not, we also derive a theorem showing that scale-free optimizers enjoy an automatic acceleration w.r.t. the condition number on certain cases. This gives AdamW a concrete theoretical advantage in training over Adam-
Next, empirically identify the scenario of training very deep neural networks with Batch Normalization switched off as case in which AdamW substantially outperforms Adam-L2 in both testing and training. In such settings, observe that the magnitudes of the coordinates of the updates during training are much more concentrated about fixed value for Adam W than for Adam-éz, which is an expected property of scale-free algorithms. Further, as depth increases, we expect greater diversity of gradient scalings, a scenario that should favor scale-free updates. Our experiments support this hypothesis: deeper networks have more dramatic differences between the distributions of update scales between Adam-l2 and AdamW and exhibit larger accuracy advantages for AdamW.
To summarize, the contributions of this paper are:
We show that AdamW can be seen as an approximation of a proximal update, which utilizes the entire regularizer rather than only its gradient.
2. We point out the fceleefseeness property enjoyed by AdamW and show the advantage of such property on a class of functions.
3. We find scenario where AdamW is significantly better than Adam-& in both training and testing performance and report empirical observation of the correlation between such advantage and the ccaleffreenes, property of AdamW. W.
The rest of this paper is organized as follows: In Section 2 we discuss the relevant literature. The connection between AdamW and the proximal updates as well as its scale-freeness are explained in Section B We then report the empirical observations in Section Finally, we conclude with a discussion of the results, some limitations of this work, and future directions.
2 Related Work
Weight decay By biasing the optimization towards solutions with small norms, weight decay has long been standard technique to improve the generalization ability in machine learning (Krogh & Hertz 1992 Bos & Chug 1996) and is still widely employed in training modern deep neural networks (Devlin et al. 2019 Tan & Lc 2019). Note that here we do not attempt to explain the generalization ability of weight decay or AdamW. Rather, we assume that the regularization and the topology of the network guarantee good generalization performance and study training algorithms from an optimization point of view. In this view, we are not aware of other work on the influence of regularization on the optimization process.
Proximal updates The use of proximal updates in the batch optimization literature dates back at least to 1965 Morcau
1965: Martinet 1970, Rockafellar 1976. Parikh & Boyd 2014) and they were used in the online setting (Kivinen &
Warmuth 1997; Campolongo & Orabona 2020), and also in the stochastic one (Toulis & Airoldi 2017 Asi & Duchi 2019). We are not aware of any previous paper pointing out the connection between Adam and proximal updates.
Scale-free algorithms The scale-free property was first proposed in the online learning field (Cesa-Biande et al.. 2007 Orabona Pál, 22015] Orabona & Pal 2018). There, they do not need to know a priori the Lipschitz constant of the functions, while obtaining optimal convergence rates. To the best of our knowledge, the connection between
2

---

Published in Transactions on Machine Learning Research (August/2022)
Algorithm Adam with fa regularization (2dam-62) and Adam
Loshchilov & Hutter 2017
All operations on vectors are element-wise.
1: Given a, 81, $2,
E R, Ir schedule h70-
2: Initialize: Zo E RR, my f- 0, Vo +
for ..., do
Compute stochastic evaluation of the true gradient V fx(r-i) denoted as Vi
5: 9. f (f(( +Ax,-
m. 3 + (1
91) vt 820 (1 B2)gr
m. m1/(1 (8)),
de 01/(1 (1 85)
34. 5- 11-1
-1/:
-mamm (itia/(Vii (v +
9: end for
scale-freeness and the condition number explain in Section 3 is novel, as is the empirical correlation between scale-freeness and good performance.
Removing Batch Normalization (BN) The setting of removing BN is not our invention: indeed, there is already active research this (De & Smith 2020 Zhang et al. 2019). The reason that BN has many disadvantages (Brock et al. 2021) including added memory overhead (Bulô al. 2018)) and training time (Gitman & Ginsburg (017)) and discrepancy between training and inferencing (Singh & Shrivastava 2019). BN has also been found to be unsuitable for many cases including sequential modeling tasks (Ba et al.
2016) and contrastive learning algorithms (Chen et al. 2020
Also, there are SOTA architectures that do not use BN including the Vision transformer (Dosovitskiy et al.
2021 and
the BERT model (Devlin et al. 2011)).
3 Theoretical Insights on Merits of AdamW
Adam W and Proximal Updates Here, we show that Adam W approximates a proximal algorithm (Moreau 1965
Parikh Boyd, 2014). A proximal algorithm is an algorithm for solving a convex optimization problem that uses the proximal operators of the objective function. The proximal operator prox,, .d Rd of a convex function h is defined for any y E I2 as prox, (z) arg mi (h(x) + 1/2/|x -
Consider that want to minimize the objective function
F(x) | + f(x),
(1)
where À > O and f(x) IR - R is a function bounded from below. We could use stochastic optimization algorithm that updates in the following fashion
-t-1 T/PPt
(2)
where is learning rate schedule, e.g., the constant one or the cosine annealing (Loshchilov & Hutter 2017) and p; denotes any update direction. This update covers many cases, where denotes the initial step size:
p: = agg gives us the vanilla SGD;
2. P: og./ (V o gives the AdaGrad algorithm Duchi et al. 2010a);
P: = am. (Voo, + recovers Adam (Kingma & Ba 2015), where m, denotes the bias corrected first moment of past gradients and o, denotes the bias corrected second moment of past gradients as updated in Line 6-7 in Algorithm
Note that in the above we use 9, to denote the stochastic gradient of the entire objective function: g. = (x;--)))) + Ax (A = 0 if the regularizer is not present), where ( is stochastic evaluation of the true gradient (((
This update rule (2 is given by the following online mirror descent update (Nemirovsky Yudin 1983 Warmuth & Jagota 1997 Beck Teboulle 2003):
Ft argmin 2 11 f(x,-1) + P. (x xz-1) + 2m
x 24-1
3

---

Published in Transactions on Machine Learning Research (August/2022)
Update (2)
U Update (5)
f (ft-1) + 1
Ve((r, 1)
Pt = o9:
34 = 1t-1 (17):9
3t Anj
(2ie-1 (x1):1r))
Pt
9,
Dt = F2 1 (17)
9,
21
1+Am
1
3t-1 aT/p
9.
9?
-i g7+c
im1
Pt Vi
Dt 21-1
027)s
Voite
my
1+
2(-1
Cr7]t
mi /oi Voi-e
Table 1: Comparison of Update
2 and ( for different Pt, where m. and og are defined in Line 7 in Algorithm
This approximates minimizing a first-order Taylor approximation of F centered in æ;-1 plus term that measures the distance between the and z;-1 according to the lz norm. The approximation becomes exact when P; (((++-1) + --
Yet, this is not the only way to construct first-order updates for the objective f and to keep the squared lg norm in its functional form:
An alternative route is to linearize only
Zz = argmin
DERR
A 1 + f(x(-1) + p. (x xx-1) + 1 || xt-1 112 2 prox Am 2
24y,
2
1-113 (2 - n.P.).
(4)
which uses the proximal operator of the convex function Ame 11 1|.. is intuitive why this would be a better update: directly minimize the squared &g norm instead of approximating it. We also would like to note that, similar to (3). the proximal updates of (4) can be shown to minimize the objective under appropriate conditions. However, we do not include the convergence analysis of 4 this is already well-studied in the literature. For example, when Pt (=x=1) in ( and is convex and smooth, the update becomes a version of the (non-accelerated) iterative shrinkage-thresholdiie algorithm. This algorithm guarantees F(x;)) F < O1)/t), which is in the same order as obtained by gradient descent on minimizing f alone (Beck & Teboulle 2009
From the first-order optimality condition, the update is
xt = (1 + (x - -
(5)
When = 0, the update in (2) and this one coincide. Yet, when A * O, they are no longer the same. For easier comparison between (2) and (). we listed in Table thh detailed update formulas of them.
We now show how the update in (5) generalizes the one AdamW. The update of Adam is
- x maaiim//aibbeeeeeeeeeeeee +c)
(6)
On the other hand, using P. amm (Vobn/((e + c) in gives:
x = (1 + Ame) (xI-1 nami/(Vo + c)),
(7)
Its first-order Taylor approximation around V/- = 0
2t (1 Am))a n)amm, (V c),
exactly the AdamW update 66. Hence, AdamW is first-order approximation of proximal update.
The careful reader might notice that the approximation from Adam W to the update in (7) becomes less accurate when Th becomes too large, and so be concerned whether this approximation is practical at all. Fortunately, in practice, is never large enough for this to be an issue. The remainder term of this approximation is O which we should always expect to be small as both and 1/, are small. So, we can expect AdamW and the update in 7 to perform similarly for learning rate schedules 7/t commonly employed practice, and we will indeed confirm this empirically in Section 4.3
Let's now derive the consequences of this connection with proximal updates. First of all, at least in the convex case, the convergence rate of the proximal updates will depend on | [V: rather than on [V)((;)) Aax (Duchi al.
4

---

Published in Transactions on Machine Learning Research (August/2022)
2010b). This could be significant improvement: the regularized loss function is never Lipschitz, the regularized gradients (mx) + Xæ, could be much larger than (m)) when ʃ itself is Lipschitz.
More importantly, proximal updates are fundamentally better at keeping the weights small. Let us consider a couple of simple examples to see how this could be. First, suppose the weights are already zero. Then, when taking an update according to ( we increase the weights to - P- In contrast, update (5) clearly leads to a smaller value. This is because it computes an update using the regularizer rather than its gradient. As an even more disturbing, yet actually more realistic example, consider the case that - 1 is non-zero, but 9, O. In this case, taking update using 2 may actually increase the weights by causing He to overshoot the origin. In contrast, the proximal update will never demonstrate such pathological behavior. Notice that this pathological behavior of (2) can be mitigated by properly tuning the learning rate. However, one of the main attractions of adaptive optimizers is that we should not need to tune the learning rate as much. Thus, the proximal update can be viewed as augmenting the adaptive methods with an even greater degree of learning--ra rate robustness.
Adam W is Scale-Free We have discussed what advantages the proximal step hidden in AdamW can give but have not yet taken into consideration the specific shape of the update. Here instead we will look closely at the pa used in AdamW to show its scale-freenes Our main claim is: the lack of scale-fieeness seems to harm Adam-lz f2 's performance in certain scenarios in deep learning, while Adam W preserves the peale-freeness even with an Eg regularizer. We will motivate this claim theoretically in this section and empirically in Section 4
An optimization algorithm is said to be scale-free if its iterates do not change when one multiplies any coordinate of all the gradients of the losses ft by a positive constant (Orabona & Pál 2018). turns out that the update (6) of AdamW and the update ( are both scale-free when e O. This is evident for Adam W the scaling factor for any coordinate of the gradient is kept in both m. and Vo, and will be canceled out when dividing them. (In practical applications, though, is very small but not zero, so we empirically verify in Section 4.2 that it is small enough to still approximately ensure the scale-free property.) In contrast, for Adam-L2, the addition of the weight decay vector to the gradient (Line 5 of Algorithm|] TT destroys this property.
We want to emphasize the comparison between Adam-f2 and AdamW: once Adam-&z adopts a non-zero X, it loses the scale-freeness property: in contrast, AdamW enjoys this property for arbitrary X. The same applies to any AdaGrad-type and Adam-type algorithm that incorporates the squared Eg regularizer by simply adding the gradient of the eg regularizer directly to the gradient of f. in Adam-ly (as implemented in Tensorflow and Pytorch). Such algorithms are scale-free only when they do not use weight decay.
Also, as we wrote above, AdamW can be seen as the first-order Taylor approximation on t/- = 0 of 7 in turn, the scale-freeness of 7 directly comes from the proximal updates Of course, there may be other ways to design scale-free updates solving [ yet, for AdamW, its scale-free property derives directly from the proximal update.
We stress that the scale-freeness an important but largely overlooked property of an optimization algorithm. It has already been utilized to explain the success of AdaGrad (Orabona & Pál. 2018). Recently, (Agawal et al. (2020) also provides theoretical and empirical support for setting the e in the denominator of AdaGrad to be O, thus making the update scale-free.
Below, we show how scale-freeness can reduce the condition number of a certain class of functions.
Scale-Freeness Provides Preconditioning For a twice continuously differentiable function f. its Hessian matrix is symmetric and its condition number ε is defined as the ratio of its largest absolute value eigenvalue to its smallest one. It is well-known that the best convergence rate when minimizing such f using first-order optimization algorithm (e.g.. gradient descent) must depend on the condition number (Theorem 2. 1. 1.1. 3.1.13, Nesterov| 2004). In particular, problem with a small x can be solved more efficiently than one with a big k. One way to reduce the effect of the condition number is to use a preconditioner (Nocedal & Wright 2006). While originally designed for solving systems of linear equations, preconditioning can be extended to the optimization of non-linear functions and it should depend on the Hessian of the function (Boyd & Vandenberghe, 2004, Li 2018). However, is unclear how to set the preconditioner given that the Hessian might not be constant (Section 9.4.4 Boyd & Vandenberghe 2004) and in stochastic optimization
the Hessian cannot be easily estimated (Li 2018).
In the following theorem, we show that scale-freeness gives similar advantages to the use of an optimal diagonal preconditioner,,fl for (proof in the Appendix). Specifically, a scale-free algorithm can automatically transform solving
5

---

Published in Transactions on Machine Learning Research (August/2022)
12 distance to minimizer
12 distance to minimizer
100
100
10-3
10-3
AdamW
AdamW
GD
GD
10
10-6
10
100
10a Iteration
102
100
10
102
Iteration
(a) Condition number 1.
(b) Condition number 100000.
Figure 1: Non-scale-free GD v.s. scale-free Adam on quadratic functions with different condition numbers.
the original problem into solving a problem with a potentially much smaller x. and thus could provide substantial improvements over non-scale- free ones, as shown in Figure inn the paper.
Theorem 3.1. Let f be a twice continuously differentiable function and x² such that (fxx=) = 0. Then, let fA be the family of functions such that ffa(=*)= = O, and V f²()x = f where A = diag(>1,...... running any scale-free optimization algorithm on J and J'x will result exactly the same iterates, assuming the same noise on the gradients. Moreover, any dependency on the condition number of the scale-free algorithm will be reduced to the smallest condition number among all the functions fa-
Ž O. Then,
To give example of when this is advantageous, consider when V2 (xx()) is diagonal matrix:
V
Assume 0 1 S 14 S (0)) I M2 < M for i E (1....,d). A144 S (x) A,M, S M, then A has condition number K' = M2/Uj. This gives scale-free algorithms big advantage when max; Mi/ps & M/U.
d). Denote j arg max; Ms/p-- Choose X s.t. 14
Another example is one of the quadratic functions.
Corollary 3.2. For quadratic problems f(x) 2 Hx b x + with H diagonal and positive definite, any scale-free algorithm will not differentiate between minimizing I and f(xx) = Lar +H--b) x x As the condition number of 1, the operation, and most importantly, the convergence, of a scale-free algorithm will not be affected by the condition number of all.
Figure illustrates Corollary 3.2. we compare GD (non-scale-free) with AdamW (scale-free) on optimizing two quadratic functions with the same minimizer, but one's Hessian matrix being a rescaled version of the other's, resulting in different condition numbers. The figure clearly shows that, even after tuning the learning rates, the updates of AdamW (starting from the same point) and thus its convergence to the minimizer, is completely unaffected by the condition number, while GD's updates change drastically and its performance deteriorates significantly when the condition number is large. is not hard to imagine that such poor training performance would likely also lead to poor testing performance.
This can also explain AdaGrad's improvements over SGD certain scenarios. As an additional example, in Appendix B analyze variant of AdaGrad with restarts and show an improved convergence on strongly convex functions due to scale-freeness. Note that the folklore justification for such improvements is that the learning rate of AdaGrad approximates the inverse of the Hessian matrix, but this is incorrect: AdaGrad does not compute Hessians and there is no reason to believe it approximates them in general.
More importantly, another scenario demonstrating the advantage of scale-freeness training deep neural networks. Neural networks are known to suffer from the notorious problem of vanishing/exploding gradients (Bengio et al. 1994 Glorot & Bengio 2010 Pascanu et al. 2013). This problem leads the gradient scales being very different across layers, especially between the first and the last layers. The problem particularly severe when the model is not

---

Published in Transactions on Machine Learning Research (August/2022)
equipped with normalization mechanisms like Batch Normalization (loffe & Szegedy 2015). In such cases, when using non-scale-free optimization algorithm (e.g., SGD), the first layers and the last layers will proceed at very different speeds, whereas a scale-free algorithm ensures that each layer is updated at similar pace, We will investigate these effects empirically in the next section.
4 Deep Learning Empirical Evaluation
In this section, we empirically compare Adam-l2 with AdamW. First (Section 4.1) we report experiments for deep neural networks on image classification tasks (CIFAR10/100). Here, AdamW enjoys a significant advantage over Adam-&2 when BN switched off on deeper neural networks. We also report the correlation between this advantage and the scale-freeness property of AdamW. Next (Section 4.2). we show that AdamW is still almost scale-free even when the used in practice is not 0, and how, contrary to AdamW, Adam-&2 is not scale-free. Finally (Section 4.3) we show that AdamW performs similarly to the update in which we will denote by AdamProx below, thus supporting the observations in Section g
Data Normalization and Augmentation: We consider the image classification task on CIFAR-10/100 datasets. Images are normalized per channel using the means and standard deviations computed from all training images. We adopt the data augmentation technique following[ et al. (2015) (for training only): pixels are padded on each side of an image and 32 32 crop is randomly sampled from the padded image its horizontal flip.
Models: For the CIFAR- 10 dataset, we employ the Residual Network model (He et al.
2016 of 20/44/56/1 I 10/218 layers; and for CIFAR-100, we additionally utilize the DenseNet-BCC model (Huang et al. growth-rate of 12. The loss is the cross-entropy loss.
2017 with 100 layers and
Hyperparameter tuning: For both Adam- l2 and AdamW, we set 8 = 0.9, 3/ = 0.999, e = 10-8 as suggested
in the original Adam paper Kingma Ba (2015). To set the initial step size a and weight decay parameter A, we grid search over {0.00005, 0.0001, 0.0005, 0.001, 0.005] for a and {0, 0.00001, 0.00005, 0.0001, 0.0005, 0.0013 for X. Whenever the best performing hyperparameters lie in the boundary of the searching grid, we always extend the grid to ensure that the final best-performing hyperparameters fall into the interior of the grid.
Training: For each experiment configuration (e.g., 110-layer Resnet without BN), we randomly select an initialization of the model to use as a fixed starting point for all optimizers and hyperparameter settings. We use a mini-batch of 128, and train 300 epochs unless otherwise specified.
4.1 AdamW vs. Adam-/2: Influence of Batch Normalization and Correlation with Scale-freeness
With BN, Adam-l2 is on par with Adam Recently, Bjorck et al. (202)) found that AdamW has no improvement in absolute performance over sufficiently tuned Adam- in some reinforcement learning experiments. We also discover the same phenomenon in several image classification tasks, see Figure Indeed, the best weight decay parameter 0 for all cases and Adam W coincides with Adam-f2 in these cases. Nevertheless, AdamW does decouple the optimal choice of the weight decay parameter from the initial step size much better than Adam-&2 in all cases.
Removing BN Notice that the models used in Figure 2 all employ BN. BN works by normalizing the input to each layer across the miti-batch to make each coordinate have zero-mean and unit-variance. Without BN, deep neural networks are known to suffer from gradient explosion and vanishing (Schoenholz et al.) 2017). This means each coordinate of the gradient will have very different scales, especially between the first and last layers. For non-scale-free algorithms, the update to the network weights will also be affected and each coordinate will proceed a different pace. In contrast, scale-free optimizers we consider the case where BN is removed as that is where AdamW and Adam- will show very different patterns due to scale-freeness.
robust such issues as the scaling of any single coordinate will not affect the update. Thus,
Without BN, A Outperforms Adam-Ez In fact, without BN, AdamW outperforms Adam- even when both are finely tuned, especially on relatively deep neural networks (see Figure and 4). AdamW not only obtains a much better test accuracy but also trains much faster.
httpe///ttaab.cmm///aasssrrrrrge resnet, cifarl0 assi catfen
7

---

Published in Transactions on Machine Learning Research (August/2022)
Setom-Uesttee test error 0.098
0.4
Intial stepotre (4.0001
Adamii Best test emon: 0.0992
0.4
0.05
Adom- deett-etetttt test error: 0.0844
0.4
Inbal t1 (000001)
0.55
Sdemt-WJest test eror: 0.0911
0.4
0.3
0.5
0.2
0.2
0,7
0.1
Inzall
Incal
n.: Ban 8.1. 05
0.0
0.01 0.05 81 0.5 6
0.0
c.11 0g0 9. 05
0.0
0.01 aun 6. B5
0.0
Weight decay (40.001)
Weight decay (00.003]
Weight decal (*0.001)
Weight decay (50.001)
(a) 20 Layer Resnet CI
(b) 44 Layer Resnet CIFARIO
Adem-Best test error 0.0834
AdemiW--Bestt emoon 0808
Adam-Best test enror
AdemW-BBst test emer 0.0819
Inbil steasice (00001))
6.05
Inttal steasha ('0.001)
Initial stossin (0000))
0.05
Intial steastes ('0 (O.001)
0.2
0.2
0.2
0.1
0.1
0.7
i5
0.1
0.
0.0
(10.003)
0.85
05
(00.001)
Weight dacay (90001))
unh U5
0.0
(c) 56 Layer Resnet CIFARIO
(d) 110 Layer Resnet
CIFARIO
Adam-Best
0.0766
AdlanW-Bess 0799
0.3180
AdauW-BBrd 0.3079
stappize ("00003)
6.05
Initaal stepsine (0.000))
stapaioo (10.000011000)) (M0,003)
0.05
Inibal stepsice P0,001)
0.1:
0.3
0.2
0.3
0.1
03
0.5
0.5
0.2
0.5
0.2
0.4
0.0
Weight docay (.0.001)
Weight decay (+0.001)
Weight decay (.0.001)
Weight docay ('0.00))
(e) 218 Layer Resnet on CIFARIO
(1) 100 layer DenseNet-BC on CIFAR100
Figure 2: The final Top-1 I test error on using AdamW vs. Adam-l2 on training a Resnet/DenseNett with Batch Normalization on CIFAR10/100 (the black circle denotes the best setting). Note how close are the best performing hyperparameter combinations and the lowest testing error each optimizer obtains between Adam-&z and AdamW for each setting suggesting they perform similarly when BN is turned on.
Adam W's Advantage and Scale-freeness We also observe that the advantage of AdamW becomes more evident as the network becomes deeper. Recall that as the depth grows, without BN, the gradient explosion and vanishing problem becomes more severe. This means that for the non-scale-free Adam-&z, the updates of each coordinate will be dispersed on a wider range of scales even when the same weight decay parameter is employed. In contrast, the scales of the updates AdamW will be much more concentrated a smaller range. This is exactly verified empirically as illustrated in the 5th & 6th columns of figures in Figure and There, we report the histograms of the absolute value of updates of Adam-lz vs. AdamW W of all coordinates near the end of training (for their comparison over the whole training process please refer to the Appendix
This correlation between the advantage of AdamW over Adam- and the different spread of update scales which is induced by the fele--freenes property of AdamW provides empirical evidence on when Adam W excels over Adam-Ez.
SGD and Scale-freeness The reader might wonder why SGD is known to provide state-of-the-art performance on many deep learning architectures (e.g., He et al., 2016 Huang et al. 2017) without being scale-free. At first blush, this seems to contradict our claims that freee-ff correlates with good performance. In reality, the good performance of SGD in very deep models linked the use of BN that normalizes the gradients. Indeed,
verified empirically that SGD fails spectacularly when BN is not used. For example, training the 1 110 layer Resnet without BN using SGD with momentum and weight decay of 0.0001, even a learning rate of le 10 will lead to divergence.
4.2 Verifying Scale-Freeness
In the previous section, we elaborated on the ccale-freence property of AdamW and its correlation with AdamW's advantage over Adam-lz. However, one may notice that practice, the factor in the AdamW update typically small but not O, in our case 1e-8, thus preventing from completely scale-free. this section, we verify that the effect of such on the cale-freeness is negligible.
8

---

Published in Transactions on Machine Learning Research (August/2022)
1.5
Acars Spech
Acerite Souch 200
30.
20.4
044
11
3736353515322 4-33 o
60
23:30.33 15-15
(a) 20 Layer Resnet
Acier Sooth 200
200
12
3136358515122 4-3 O
23:35 1833-22 hude
(b) 44 Layer Resnet
Spoch
Acernta Spoch 200
5a.
3354-3 13
Couch
(c) 56 Layer Resnet
Spoch
1.0
Acerrtt Epoch 300
Allin
5ae
06-
22
Aegl Baay
Couch
(d) 110 Layer Resnet
Acert Spoch
1.0
Acerrti Epach 300
2oc
0.
20.4 0.2
0.2
Wed (OOO POCEI
23 ROW CRA0
Ppan 0,
Older of magggttta Ioowe o22)
10-0335111111 4
Order of mage maak (power of
(c) 218 Layer Resnet
Figure 3: On using AdamW vs. Adam-l2 on training a Resnet without Batch Normalization on CIFAR10. (Left two) The final Top-1 1 test error (the black circle denotes the best setting). (Middle two) The training loss and test accuracy curve when employing the initial step size and the weight decay parameter that gives the smallest test error. (Right two) The histogram of the magnitude of corresponding updates of all coordinates of the network near the end of the training when employing the initial step size and the weight decay parameter that gives the smallest test error. Note that as the depth of the neural network increases, Adam- 's updates scatter more evenly over the entire spectrum while AdamW's updates are still concentrated in a small range, and AdamW's advantage in both training and testing over Adam- becomes more significant.
As a simple empirical verification of the ccale-freeness we consider the scenario where we multiply the loss function by a positive number. Note that any other method to test scale-freeness would be equally good. For a feed-forward neural network without BN, this means the gradient would also be scaled up by that factor. In this case, the updates of scale-free optimization algorithm would remain exactly the same, whereas they would change for an optimization algorithm that is not scale-free.
Figure 5 shows results of the loss function being multiplied by 10 and 100 respectively on optimizing a 1 110-layer Resnet with BN removed. For results of the original loss see Figure We can see that AdamW has almost the same performance across the range of initial learning rates and weight decay parameters, and most importantly, the best values of these two hyperparameters remain the same. This verifies that, even when employing a (small) non-zero c, AdamW is still approximately freee-free. In contrast, Adam-lz is not scale-free and we can see that its behavior varies drastically with the best initial learning rates and weight decay parameters in each setting totally different.
9

---

Published in Transactions on Machine Learning Research (August/2022)
1.5
Acart Spech
Acerite Souch 200
0.8
30.
30.4
044
330
320
Adarms
330
S136353515322 0-33 0
60
23:30.33
15
(a) 20 Layer Resnet
Acier South 200
Adarts
20.
20.4
04-
3136358515122
4-3 O
made
(b) 44 Layer Resnet
Spoch
Acerrte Spoch 300
Adarni
Bou
Adorit
5354-3 13
dede Pocall
tpwch
(c) 56 Layer Resnet
Spoch
1.0
Acerrtt Epoch 300
Ba. 06-
5ae
Adarm
1002335101222 i g
22 E8
Angld sway
tpach
(d) 110 Layer Resnet
Acert Spoch
1.0
Acerntii Epach 300
Adia vil
2oc
0.
20.4 022
0.2
8.2g
055055011111 4.5
Wed lolk Pocall
Apacs
Tacch
Older of maggtttaa Ioww of
Onder mage malk spower of
(c) 218 Layer Resnet
Acari Spech
Acarrte Spach 200
Me 2.812
50.-
046
30.4
Adum
62)
Epach
Eauch
Dider of www.nnii (anwe (1%)
Order of badd (powsr 21
(f) 100 layer DenseNct-BC
Figure 4: On using AdamW vs. Adam-l2 training Resnet/DenseNet without Batch Normalization CIFAR100. (Left two) The final Top-1 test error (the black circle denotes the best setting). (Middle two) The training loss and test accuracy curve when employing the initial step size and the weight decay parameter that gives the smallest test error. (Right two) The histogram of the magnitude of corresponding updates of all coordinates of the network near the end of the training when employing the initial step size and the weight decay parameter that gives the smallest test error. Note that as the depth of the neural network increases, Adam-Ez's 's updates scatter more evenly over the entire spectrum while AdamW's updates are still concentrated in a small range, and AdamW's advantage in both training and testing over Adam-lz becomes more significant.
4.3 AdamW and AdamProx Behave very Similarly
In Section 3 we showed theoretically that AdamW is the first order Taylor approximation of AdamProx (update rule 7 Beyond this theoretical argument, here we verify empirically that the approximation is good. In Figure 6 we consider the case when 7) = for all t a relatively large constant learning rate schedule. In such cases, AdamW and AdamProx still behave very similarly. This suggests that for most learning rate schedules, e.g., cosine, exponential, polynomial, and step decay, which all monotonously decrease from fju- = 1, AdamProx will remain very good approximation to AdamW. Thus, it is reasonable use the more classically-linkes AdamProx to try understand AdamW.
10

---

Published in Transactions on Machine Learning Research (August/2022)
0.05
Adam-Ueett test error 0.1452
0.4
Intiall stepotie (-0,001
055
Adammi/ Best test emon: 0.1053
0.4
0.05
Adom- -esttteettttte test error: 0.1.899
0.4
Inbal steosize th0 0000003
0.55
AdamWW-Jest test eror:
emor:-0.112 0.1122
0.4
033
Intial stopstre
0.2
0,7
0.3
00:0.0 61 U5
10
0.0
0.01 0.05 0.5 Weight decay (0.001)
0.0
00.88 0M0 10.10. 1 5018360
0.0
0.01 ais 6a Weight decay (50.001)
0.0
Weight decay (40.001)
Weight decay (*0.001)
(a) Loss multiplied by 10
(b) Loss multiplied by 100
Figure 5: The final top-1 test error of AdamW Adam-l2 optimizing 100-layer Resnet with BN removed CIFAR-10 with the loss function multiplied by 10 (left two figures) and 100 (right two figures). Note how the best performing hyperparameter combinations of AdamW remain the same for different loss multiplication factors as well as the shape of the heatmap being very similar. In contrast, Adam- 's performance as well as the best performing hyperparameter combinations vary dramatically for different loss multiplication factors.
(100003)
AdamW--Bstt test emer:: 0.1147
0.4
AdamProx- Best tast emon 0.1072
0.4
Indial stapsize (100001)
AdamWW BeenttW--estttte test error: 0,3899
0.8
Iniial Intial (M.0000 M0.0013
AdamProx- Best test erron: 0.3965
0.8
0.05
0.05
8.1
0.1
0.3
Inital stepstre (M0
0.
0.3
0.5
0.7
000 aus 0.1. 05 Weight decay (+0.001)
0.0
B.L 0,55
0.01 0.55
0.01 0.05
Weight decay (10.003)
Weight decay (M0.001)
Weight decay (+0.001)
(a) ResNet on CIFAR-10
(b) DenseNet-Bi BCC on CIFAR-100
Figure The final Top-1 test error of using AdamW vs. AdamProx on training (the black circle denotes the best setting). (Top row) a 110-layer ResNet with BN removed on CIFAR-10 (trained for 300 epochs). (Bottom row) 100-layer DenseNet-BC with BN removed CIFAR-100 (trained for 100 epochs). Note how similar the shapes of the heatmaps, the best performing hyperparameter combinations, and the test errors between AdamW and AdamProx.
5 Conclusion and Future Work
In this paper, provide insights for understanding the merits of AdamW from two points of view. We first show that AdamW is approximation of the proximal updates both theoretically and empirically. We then identify the setting of training very deep neural networks without batch normalization in which AdamW substantially outperforms Adam- in both training and testing and show its correlation with the scale-freeness property of AdamW. Nevertheless, we are aware of some limitations of this work as well as many directions worth exploring.
Limitations First, we only focus on investigat- ing the effects of scale-freeness on Adam-L2 and AdamW, but it would be interesting to study scale- freeness more generally. Also, what showed is just correlation instead of causality thus we did not rule out other possible causes beyond scale- freeness for the success of AdamW. Indeed, rigor- ously proving causality for any such claim is ex- tremely difficult even in the hard sciences. Note that there papers claiming that adaptive updates have worse generalization (Wilson et al.) (2017); however, such claims have been recently partly con- futed (see, e.g., Agarwal et al. 2020). On this note, despite its empirical success, we stress that Adam will not even converge on some convex functions (Reddi et al. 2018), thus making it hard to prove formal theoretical convergence and/or generalization guarantees.
Initial stepsize P 10.0011
AdamW- error:
Intia stepsize (00001)
AdamProxt22 test
0.4
0.05
0.05
0.3
0.3
0.1
0.2
0.2
05
0.1
0.1
0.0
Weight decay (.0.001)
Weigit decay (.0.001)
Figure 7: Final Top-1 test error of using AdamW vs. AdamProxL2 to train 110-layer ResNet without BN on CIFAR1O (the black circle denotes the best setting).
Update for no-square l regularization. Instead of using the squared & regularization, we might think use the & regularization, that is without the square. This is known to have better statistical properties than the squared &a (see,

---

Published in Transactions on Machine Learning Research (August/2022)
e.g., Orabona 2014), but it is not smooth so is harder to be optimized. However, with proximal updates, don't
have to worry about its non-smoothness. Hence, we can consider the objective function F(æ) > l2 + ((x)
The corresponding prox-SGD update was derived in Duchi & Singer (2009) for scalar learning rates and it is easy to generalize to our setting as
(1
Ane [ 7iP.
0) (xt 7:P.)
1++1
max
Its performance, named AdamProxL2, as shown in Figure can be on par with AdamW.
Distributed Training Batch normalization is not user-friendly in distributed training requires each machine to collect batch of statistics to update the model which may be inaccurate when machines do not communicate frequently with each other Goyal et al. (2017). Since AdamW outperforms Adam-lz significantly in settings without BN, at least in feed-forward neural networks, we can apply AdamW in distributed training to see if it still enjoys the same merits.
Broader Impact Statement
The main contribution of this paper the study of known optimization algorithm AdamW from the theoretical angles of proximal updates and scale-freeness, while the experiments are done to empirically validate and support the theoretical findings. It is general algorithm and we do not specify in which applications should it be employed, thus we do not foresee any direct negative societal impact our work might cause.
Acknowledgments
Francesco Orabona is supported by the National Science Foundation under the grants no. 1908111 "AF: Small: Collaborative Research: New Representations for Learning Algorithms and Secure Computation", no. 2022446 "Foundations of Data Science Institute", and no. 2046096 "CAREER: Parameter-free Optimization Algorithms for Machine Learning". Mingrui Liu supported in part by grant from George Mason University.
References
N. Agarwal, R. Anil, E. Hazan, T. Koren, and C. Zhang. Disentangling adaptive gradient methods from learning rates. arXiv preprint arXiv:2002.1 1803, 2020.
H. Asi and J. C. Duchi. Stochastic (approximate) proximal point methods: Convergence, optimality, and adaptivity. SIAM Journal on Optimization, (013))225-22990 2019.
J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: 2016.
Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 3163):66-177,, 2003.
A. Beck and M. Teboulle. A fast iterative shrinkagg-ttreeshokdng algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183 202, 2009.
Y. Bengio, P. Simard, and Frasconi. Learning long-term dependencies with gradient descent is difficult. JEEE transactions on neural networks, 5(2):157-166, 1994.
J. Bjorck, K. Q Weinberger, and C. Gomes. Understanding decoupled and early weight decay. Proceedings of the AAAI Conference on Artificial Intelligence, 35())66777-6785, 2021.
S. Bos and E. Chug. Using weight decay optimize the generalization ability of perceptron. Proceedings of International Conference on Neural Networks (ICNN'96), 1:241-246 vol.1, 1996.
S. Boyd and L. Vandenberghe. Convex optimization. Cambridge University Press, New York, NY, USA, 2004. ISBN 0521833787.
A. Brock, S. De, S. L. Smith, and K. Simonyan. High-pun seghhperformance large-scale image recognition without normalization. In Proc. of the International Conference Machine Learning (ICMI)), pp. 1059-1071, 2021.
12

---

Published in Transactions on Machine Learning Research (August/2022)
S. R. Bulò, L. Porzi, and P. Kontschieder. In-place activated BatchNorm for memory-optimized training of DNNSs In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5639-5647, 2018.
N. Campolongo and F. Orabona. Temporal variability in implicit online learning. In Advances in Neural Information Processing Systems, volume 33. Curran Associates, Inc., 2020.
N. Carion, F. Massa, G. Synnaeve, N. Usumier, A. Kirillov, and Zagoruyko. End-to-end object detection with Transformers. European Conference on Computer Vision, pp. 213-229. Springer, 2020.
N. Cesa-Bianchi, Y. Mansour, and G. Stoltz. Improved second-order bounds for prediction with expert advice. Machine Learning, 66(2):321-352 2007.
T. Chen, S. Korblithh M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé, IIII and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020.
X. Chen, S. Liu, R. Sun, and M. Hong. On the convergence of class of Adam-type algorithms for non-convex optimization. In 7th International Conference Learning Representations, ICLR 2019, 2019.
S. De and S. Smith. Batch normalization biases residual blocks towards the identity function in deep networks. Advances in Neural Information Processing Systems, 33, 2020.
S. De, A. Mukherjee, and E. Ullah. Convergence guarantees for RMSProp and Adam non-convex optimization and an empirical comparison to nesterov acceleration. arXiv preprint arXiv: /807.06766. 2018
J. Devlin, Ming-Waiee Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani. M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR, 2021.
J. Duchi and Y. Singer. Efficient online and batch learning using forward backward splitting. The Journal of Machine Learning Research, 10:2899-299 2934, 2009.
Duchi, Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. In COLT, 2010a.
J. Duchi, S. Shalev-Shwarke Y. Singer, and A. Tewari. Composite objective mirror descent. In COLT, pp. 14-26, 2010b.
I. Gitman and B. Ginsburg. Comparison of batch normalization and weight normalization algorithms for the large-scale image classification. arXiv preprint
2017.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proc. of the thirteenth international conference on artificial intelligence and statisties, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Goodfellow, J. Pouget-Abadie, M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
P. Goyal, P. Dollár, R. B. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch SGD: Training ImageNet hour. ArXiv, abs/ 02667, 2017.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the HEEE conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift.
In International conference machine learning, pp. 448 456. PMLR, 2015.
13

---

Published in Transactions on Machine Learning Research (August/2022)
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
J. Kivinen and M. Warmuth. Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132(1):1-63, January 1997.
Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
A. Krogh and Hertz. A simple weight decay can improve generalization. In Advances in neural information processing systems, pp. 250-957, 1992.
J. Kuen, F. Perazzi, Z. Lin, J. Zhang, and Y.-P. Tan. Scaling object detection by transferring classification weights. In Proc. of the IEEE/CVF International Conference on Computer Vision, pp. 6044-6053, 2019.
Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
C. Lee, Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply-supervised nets. In Proc. of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38, pp. 562-570. PMLR, 2015.
X. Li and F. Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In Proc. of the 22nd International Conference on Artificial Intelligence and Statistics, AISTATS, 2019.
X. Li and Orabona. high probability analysis adaptive SGD with momentum. In ICMI, 2020 Workshop on Beyond First Order Methods in ML Systems, 2020.
X. Li, Z. Zhuang, and F. Orabona. A second look at exponential and cosine step sizes: Simplicity, adaptivity, and performance. In International Conference on Machine Learning, pp. 6553-6564. PMLR, 2021.
X.-L Li. Preconditioned stochastic gradient descent. IEEE Transactions on Neural Networks and Learning Systems, 29 1411454-4466, 2018.
Y. Lifchitz, Y. Avrithis, S. Picard, and A. Bursuc. Dense classification and implanting for few-shot learning. In Proc. of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 00599928800 2019.
Loshchilov and Hutter. SGDR: Stochastic gradient descent with warm restarts. International Conference on Learning Representations, 2017.
Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
L. Luo, Y. Xiong, Y. Liu, and X. Sun. Adaptive gradient methods with dynamic bound of learning rate. In International Conference on Learning Representations, 2018.
B. Martinet. Régularisation d' d'inéquations variationnelles par approximations successives. rev. française informat. Recherche Opérationnelle, 4: 4:854-158, 1970.
H. B. McMahan and M. J. Streeter. Adaptive bound optimization for online convex optimization. In COLT, 2010.
J.JJ. Moreau. Proximité et dualité dans un espace hilbertien. Bulletin de Société mathématique France, 93: 273-299, 1965.
A. S. Nemirovsky and D. Yudin. Problem complexity and method efficiency in optimization. Wiley, New York, NY, USA, 1983.
Y. Nesterov, Introductory lectures on convex optimization: A basic course, volume 87. Springer, 2004.
J. Nocedal and S. J. Wright. Numerical Optimization. Springer Series in Operations Research and Financial Engineering. Springer, New York, 2006.
F. Orabona. Simultaneous model selection and optimization through parameter-free stochastic learning. Advances in Neural Information Processing Systems 27, 2014.
14

---

Published in Transactions on Machine Learning Research (August/2022)
F. Orabona and D. Pál. Scale-free algorithms for online linear optimization. In International Conference Algorithmic Learning Theory, pp. 287-301. Spriner, 2015.
F. Orabona and D. Pál. Scale-free online learning. Theoretical Computer Science, 716:50-69, 2018. Special Issue ALT 2015.
N. Parikh and S. Boyd. Proximal algorithms. Foundations and Trends in optimization, 1(3):127-239, 2014.
R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318. PMLR, 2013.
S.J. Reddi, S. Kale, and S. Kumar. On the convergence of Adam and beyond. In International Conference on Learning Representations, 2018.
H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22:400-407, 1951.
R. T. Rockafellar. Monotone operators and the proximal point algorithm. SIAM journal on control and optimization, 14 (5):877-898, 1976.
S. Schoenholz, J. Gilmer, S. Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations (ICLR), 2017.
S. Singh and A. Shrivastava. EvalNorm: Estimating batch normalization statistics for evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3633-3641, 2019.
M. Tan and Q. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 6105-6114. PMLR, 2019.
T. Tieleman and G. Hinton. Lecture 6.5 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
P. Toulis and M. Airoldi. Asymptotic and finite-sample properties of estimators based on stochastic gradients. The Annals of Statistics, 45(4):1694-172) 2017.
A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, N Gomez, E. Kaiser, and Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
G. Wang. S. Lu, Q. Cheng, W. Tu, and Zhang L. SAdam: A variant of Adam for strongly convex functions. International Conference Learning Representations, 2020.
M. K. Warmuth and A. K. Jagota. Continuous and discrete-time nonlinear gradient descent: Relative loss bounds and convergence. In Electronic proceedings of the 5th International Symposium on Artificial Intelligence and Muthematics, 1997.
A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
N. Wu, J. Phang, J. Park, Y. Shen, Z. Huang, M. Zorin, S. Jastrzębski, T. Févry, J. Katsnelson, E. Kim, S. Wolfson, U. Parikh, S. Gaddam, L. L. Y. Lin, K. Ho, J. D. Weinstein, B. Reig, Y. Gao, H. Toth, K. Pysarenko, A. Lewin, J. Lee, K. Airola, E. Mema, S. Chung, F. Hwang, N. Samreen, S. G. Kim, L. Heacock, L.. Moy, K. Cho, and K. J. Geras. Deep neural networks improve radiologists' performance in breast cancer screening. IEEE Transactions on Medical Imaging, 18)::1184-1194 2020.
M. D. Zeiler. ADADELTA: an adaptive learning rate method. arXiv preprint arXiv: 1212.570,, 2012.
Zhang, Y. N. Dauphin, and T. Ma. Fixup initialization: Residual learning without normalization. International Conference on Learning Representations, 2019.
L. Zhang, S. Lu, and Z. Zhou. Adaptive online learning in dynamic environments. Advances in neural information processing systems, 31, 2018.
D. Zhou, J. Chen, Y. Cao. Y. Tang, Z. Yang, and Q. Gu. On the convergence of adaptive gradient methods for nonconvex optimization. arXiv preprint 1808: 2018.
15

---

Published in Transactions on Machine Learning Research (August/2022)
Appendices
A
Proof of Theorem 3.1
Proof. From the Fundamental Theorem of Calculus have:
= ([(=)=\(xx')) + / V2f(2" + t(x x x²))(x x² x²)dt = / f2 + t((x x²))( x: xd)dt.
Thus, for any function (x) whose Hessian is f(x) and (x*) = 0, we have ffx(z) AV)(
Now, from the definition a scale-free algorithm, the iterates of such an algorithm do not change when one multiplies each coordinate of all the gradients by a positive constant. Thus, scale-free algorithm optimizing / behaves the same as if it is optimizing fA-
B A Scale-free Algorithm with Dependency on the Condition Number
Algorithm AdaGrad (Duchi et al. 2010aa McMahan & Streeter 2010) (All operations on vectors are dlement-wise
Input: #lterations T, a set K., æ1 IC, stepsize y
for 1...T do
Receive: (xxx)
Set: n,
V Ei_,
Update: 1t-1 = Ic (x+ n, ff())) where Ilg is the projection onto K.
end for Output: x T EE= St.
Algorithm 3 AdaGrad with Restart
Input: #Rounds N, an E Rd, upper bound on |/10 x² b0 as Doo. strong convexity μ4, smoothness
Set: T. ZD N do
for 1...N
Run Algorithm to get with T 32d B.-1. fx: || 2.-1 4-1 J. n= Dw/vI
D2
end for
Output: NN-.
Theorem B.1. Let K. be hypercube with |æ yl1 I Dor any y, E K. For convex function f. set D.
V
then Algorithm 2 guarantees for any K:
[ax) [(x) <
2dD
(8)
t=1
Theorem B.2. For J strongly convex and M smooth function f. denote its unique minimizer x² Rd. Given Rdd, assume that ||z x= lloc I Doo. then Algorithm guarantees:
D2
| x 11
4N
Thus, to get x such that || - x= 113c S 6, we need at most 32d M log, (D2o/e) gradient calls.
Proof of Theorem B.2 Consider round i and assume K. passed to Algorithm is bounded w.r.t. loo norm by D Do
When f is p-strongly convex and M smooth, let æ = x", Equation becomes:
Σ f(x:) f(0') 2dD2
[2:))
4Md
E((o)) f(x)))
t=1
16

---

Published in Transactions on Machine Learning Research (August/2022)
where the second inequality is by the M smoothness of f. This gives:
r Σ (mx) - f(x") & AMdD
Let 0; = Et= at have by the p-strong--oovvexity that:
| x= 11 xi x²1 (f(x) f(x=))
2 Σ p
8MdD - f(x²))
nT
D 4
Put T = 32dd M in Equation 9 we have that | x 11. < the update I; and x" is shrinked by half, which in turn ensures that x is still inside the K. passed to Algorithm 2 in the next round with D10,11
Thus, after each round, the foo distance between
Dey This concludes the proof.
2
Proof of Theorem B.
T
Σ [(xx) - 1(20)
V f(xe), xt xx)
0f (wx) (Zt. - xz))
Orij
2
T
(xit. - x²/²² -Irt. [t..
This Tom.
as (x:) - Tij)
T
d
Mh.
af
(xI))
ax
1=1
f=1
(xt.j - xz1)² (11415 - x2,)
T
d
Tt.j
0f (zi)
27035
2
iz
t=1
j=1
(Di.. x2
1
1
d
T
nt.j
0f (2i)
2
2
T7..5
axis
j=1
t=1
D2
of (xi)
af (a:))
2
0f Fwi
2
Σ
17
(xx))
21
trij
dois
t=1
2 Ei=1
( as (a:)) (Dil.
2
T Σ
as
as (mt) dres
(Ix1)
+ m
Σ
277
azes
T
Df (2te)
Σ
2
Jern
1=1
1=1
T
d
0f (x01) Oere
2
2d
1=1
T
2d
V VV (2
1=1
17

---

Published in Transactions on Machine Learning Research (August/2022)
where the first inequality is by convexity, the second one by the projection lemma as the projection onto hypercube equals performing the projection independently for each coordinate, the fifth one by Lemma 5 in (McMahan & Streeter 2010), and the last one by the concavity of V-.
C The Histograms of the Magnitude of each Update Coordinate during the Entire Training Phase
In this section, we report the histograms the absolute value updates of Adam-l2 vs. AdamW of all coordinates divided by a during the whole training process. From the figures shown below, we can clearly see that AdamW's updates remain in much more concentrated scale range than Adam-&2 during the entire training. Moreover, as the depth of the network grows, Adam- 's updates become more and more dispersed, while AdamW's updates still concentrated. (Note that the leftmost bin contains all values equal to or less than 2 2-27 R 10- and the rightmost bin contains all values equal to larger than 1.)
18

---

Published in Transactions on Machine Learning Research (August/2022)
Adam Epoch 1
AdamW Epoch 1
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-2-24221-8-1512--
-6 -3
0
-27-24-21-18-15-12 -9
-6
-3
0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-21-2421-1815112-9
-6 -3
0
0.0
-2724--1--8-15-12 -9
-6
-3
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-181512-
-6 -3
0.0
-27-24-21-88155-12 -
-6
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0 -27-24-21-1-15--1299 -6 -3 0
-2124211181512- -6 -3
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-62
(b) AdamW
Figure& 8: The histograms of the magnitudes of all updates (without o:) a 20-layer Resnet with BN removed trained by
AdamW W or Adam-&, on CIFARIO.
19

---

Published in Transactions on Machine Learning Research (August/2022)
Adam Epoch 1
AdamW Epoch 1
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-2-24221-8-1512--
-6 -3
0
0.0
-27-24-21-18-15-22 -9
-6
-3 0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-1815112-9
-6 -3
0
0.0
-2724--1-18-15--2 -9
-6
-3
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-1815-2-
-6 -3
0.0
-27-24-21-88115-12 -
-6
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0 -27-24-21-1-15-12299 -6 -3 0
-272421-181512-9 -6 -3
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-62
(b) AdamW
Figure 9: The histograms of the magnitudes of all updates (without o:) a 44-layer Resnet with BN removed trained by
AdamW W or Adam-&, on CIFARIO.
20

---

Published in Transactions on Machine Learning Research (August/2022)
Adam Epoch 1
AdamW Epoch 1
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-2-24221-8-1512--
-6 -3
0
-27-24-21-18-15-22 -9
-6
-3 0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-21-2421-1815112-9
-6 -3
0
0.0
-2724--1--8-15--2 -9
-6
-3
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-1815-22
-6 -3
0.0
-27-24-21-18115-12 99
-6
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-272421-181512-9 -6 -3
-27-24-21-1-15-12299 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-62
(b) AdamW
Figure 10: histograms of the magnitudes of all updates (without a) of a 56-layer Resnet with BN removed trained by AdamW or Adam-&, on CIFARIO.
21

---

Published in Transactions on Machine Learning Research (August/2022)
Adam Epoch 1
AdamW Epoch 1
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-2-24221-8-1512--
-6 -3
0
-27-24-21-18-15-22 -9
-6
-3
0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-27-2421-18-5--1-99
-6 -3
0
-27-24-21-8815--22
-9
-6
-3
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-1815-22-
-6 -3
0.0
-27-24-21-18-5--22 -9
-6
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-2122221-8-1512--9 - -3
22724221-1815-12299 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-
(b) AdamW
Figure 11: The histograms the magnitudes of all updates (without a:) a 110-layer Resnet with removed trained by AdamW or Adam-&, on CIFARIO.
22

---

Published in Transactions on Machine Learning Research (August/2022)
Adam Epoch 1
AdamW Epoch 1
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-2-24221-8-1512--
-6 -3
0
-27-24-21-18-15-22 -9
-6
-3 0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-1815-12-9
-6 -3 0
0.0
-2722-21-18-15-11 -9
-6
-3 0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
227-24-2-18-15--22
-6 -3
0.0
-27-24-21-18-1--22 -9
-6
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-212421118-1512-- 6 -3
-27-2421-18-1--12-9 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-
(b) AdamW
Figure 12: The histograms the magnitudes of all updates (without a:) a 218-layer Resnet with removed trained by AdamW or Adam-&, on CIFARIO.
23

---

Published in Transactions on Machine Learning Research (August/2022)
Adam Epoch 1
AdamW Epoch 1
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-2-24221-8-1512--
-6 -3
0
-27-24-21-18-15-22 -9
-6
-3 0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-18-5--2-99
-6 -3 0
0.0
-2724--1--8-15--2 -9
-6
-3
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-1815122-
-6 -3
0.0
-27-24-21-88115-12 -
-6
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0 -27-24-21-1815-12-99 -6 -3 0
-212421-181512-99 -6 -3
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-
(b) AdamW
Figure 13: histograms of the magnitudes of all updates (without a) of a 20-layer Resnet with BN removed trained by AdamW or Adam-&, on CIFAR100.
24

---

Published in Transactions on Machine Learning Research (August/2022)
Adam Epoch 1
AdamW Epoch 1
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-2-24221-8-1512--
-6 -3
0
0.0
-27-24-21-18-15-22 -9
-6
-3 0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-18-5-12-99
-6 -3 0
0.0
-2724--1--8-15--2 -9
-6
-3
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-1815-22-
-6 -3
0.0
-27-24-21-18-55-2 -
-6
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-272421-181512-9 -6 -3
-27-24-21-1:15-12-99 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-
(b) AdamW
Figure 14: The histograms of the magnitudes of all updates (without a) of a 44-layer Resnet with BN removed trained by AdamW or Adam-&, on CIFAR100.
25

---

Published in Transactions on Machine Learning Research (August/2022)
Adam Epoch 1
AdamW Epoch 1
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-2-24221-8-1512--
-6 -3
0
-27-24-21-18-15-22 -9
-6
-3 0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-21-2421-1815112-9
-6 -3
0
0.0
-2724--1--8-15--2 -9
-6
-3
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-1815-22
-6 -3
0.0
-27-24-21-18115-12 -
-6
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-212421-181512- -6 -3
-27-24-21-1-15-12299 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-62
(b) AdamW
Figure 15: The histograms of the magnitudes of all updates (without a) of a 56-layer Resnet with BN removed trained by AdamW or Adam-&, on CIFAR100.
26

---

Published in Transactions on Machine Learning Research (August/2022)
Adam Epoch 1
AdamW Epoch 1
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-2-24221-8-1512--
-6 -3
0
-27-24-21-18-15-22 -9
-6
-3
0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-27-2421-18-5--1--9
-6 -3
0
-27-24-21-8815--12
-9
-6
-3
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
227-2421-1815-12-
-6
-3
0.0
-27-24-21-18-5--22 -9
-6
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-272421118.112-99 6 -3
22-24221-1815-12-99 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-62
(b) AdamW
Figure 16: The histograms the magnitudes of all updates (without a:) a 110-layer Resnet with removed trained by AdamW or Adam-&, on CIFAR100.
27

---

Published in Transactions on Machine Learning Research (August/2022)
Adam Epoch 1
AdamW Epoch 1
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-2-22421-8-1512--
-6 -3
0
-27-24-21-18-15-22 -9
-6
-3 0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-1815112-9
-6 -3
0.0
-27-2-21-18-15-112 -9
-6
-3 0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-18-5-122-
-6 -3
0.0
-27-24-21-18-5--2
-9
-6
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-272421-181512-99 6 -3
22-24221-8:15-12-99 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-62
(b) AdamW
Figure 17: The histograms the magnitudes of all updates (without a:) a 218-layer Resnet with removed trained by AdamW or Adam-&, on CIFAR100.
28

---

Published in Transactions on Machine Learning Research (August/2022)
Adam Epoch 1
AdamW Epoch 1
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-2-24221-8-1512--
-6 -3
0
-27-24-21-18-15-22 -9
-6
-3 0
Adam Epoch 100
AdamW Epoch 100
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-21-2421-1815112-9
-6 -3
0
0.0
-2724--1--8-15--2 -9
-6
-3
0
Adam Epoch 200
AdamW Epoch 200
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-27-2421-1815-22
-6 -3
0.0
-27-24-21-18-5--2 -
-6
0
Adam Epoch 300
AdamW Epoch 300
1.0
1.0
0.8
0.8
Proportion
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-212421118:512- -6 -3
-27-24-21-1815-12-99 -6 -3 0
Order of magnitude (power of 2)
Order of magnitude (power of 2)
(a) Adam-62
(b) AdamW
Figure 18: The histograms of the magnitudes of all updates (without a) 100-layer DenseNet-BC with BN removed trained by AdamW Adam- on CIFAR100.
29


Reviews:
[{"review_id": "dKQjgPWGFL", "reviewer": "Reviewer_enES", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed idea is novel and interesting.\n2. The authors have conducted numerous experiments to verify their claims.\n3. The paper has nice comparisons between Adam, AdamW, and AdamProx(as proposed in the paper)\n\nWeaknesses:\n1. One part of the paper is very unclear (see below)\n2. The \"theory\" part of this paper is rather intuitive than explanatory, only the strongly convex setting is considered\n3. I do not see the impact of this work, or how it can help us design the optimizers. Since scale-freeness is a desired property, how about we design a scale-free version of SGD, e.g., scale the gradients by their norms. How is that algorithm going to perform on neural networks without BN? Since the authors only compare Adam with AdamW, and I guess SGD will likely fail in those networks, I wonder how the scaled-free version of SGD is going to perform.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the advantage of AdamW over Adam, which is a well-known phenomenon especially in computer vision tasks, where people typically argue that AdamW is better because its implementation of weight decay is correct while Adam's is not. This paper provides a different perspective of viewing the two methods in terms of whether scale-freeness is satisfied, and show both theoretically and empirically that this property helps AdamW to outperform Adam. Extensive experimental results are provided by the authors. The proposed ideas are, as far as I know, novel and interesting. However, I kind of have the feeling that scale-freeness is another way of saying \"weight decay\" is better than \"l_2 regularization\" in adaptive methods.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall, this paper is a sound empirical work and I would recommend for publication if the following problems are fixed or addressed properly.\n\nPage 4, the paragraph before \"Adam is Scale Free\" is very unclear to me. The authors refer to different cases of $x_t$, different cases of $g_t$ and many update rules including (2) and (5). Moreover the term $p_t$, as in the paper, can have different forms. Therefore, I am really confused when the author says \"when $g_t$ is zero\". Even if $g_t$ is zero, what is $p_t$ in these cases? I would strongly suggest that the author make a table for the two cases they mention, and what the consequences will be in different cases. With Eqn. (2) and (5) in different pages, I have to go back and forth when reading this paragraph and I am still confused by what the authors really mean.\n\nSome notations are never defined. For example, what is $m_t$ and what is $f_t$ in page 3/4? As someone who studies optimization, I guess the author means the \"first order momentum\" and the \"stochastic approximation\" to the loss function $f$. However, please define them clearly if theoretical analysis is provided. \" with $m_t$ and $v_t$ both updated using $g_t$\" is very vague. Please write out the update rules of Adam directly.\n\nFor the experiments, I am surprised that when BN is turned on, AdamW and Adam behaves the best when the regularization is zero since it will lead to overfitting when I run them. However, I am even more surprised by Figure 2(b). When both Adam and AdamW have zero weight decay, shouldn't them be the same? Why is the best learning rate 1 for Adam and 5 for AdamW? Could the authors explain it to me?\n\nAs I mentioned above, it is better if scale-free SGD is added to the comparison, but this is less important.\n\nMinor errors: some citations are used incorrectly, e.g., Duchi et al. 2010b in page 4, Nesterov 2004 in page 5. I believe the authors want to use \\citep instead of \\citet there. \n\n\n\n ", "broader_impact_concerns": "This paper studies optimization algorithms and I don't see any concerns with the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2R8n4Xw8Wy", "reviewer": "Reviewer_ZnRb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength is the perspective is new to the best of my knowledge.\nWeakness is the limited theoretical depth.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides some theoretical and empirical insights on AdamW, especially on why it can outperform Adam l2. The key messages can be summarized in two points. 1) AdamW can be viewed as a first-order approximation of a special proximal method. 2) The advantage of AdamW over Adam l2 might be brought by its scale-freeness property. \n\nOverall I enjoyed reading the paper and the idea is interesting. But I do have a few questions for the authors.\n\n1. What is the main further insight brought by interpreting AdamW as an approximation of proximal methods? \n\n2. Another intuitive understanding of the advantages of AdamW over Adam l2 is treating AdamW as assigning different l2 regularization parameters to different coordinates, according to the gradient magnitudes of different coordinates. The advantage then would be AdamW is regularizing the l2 norm of different weights according to their importance in changing the objective function. I believe this is a more common and easily understood perspective than scale-freeness. I wonder how does the authors view this perspective and are there connections between scale-freeness and this perspective?", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "It seems the paper is mainly on scale-freeness and the connection to proximal methods does not provide much insight. Also, it feels to me that scale-freeness and the connection to proximal methods are separated results without much connection to each other.  Maybe it is better to restructure the paper to highlight scale-freeness. Or add some transition between the connection to proximal methods and the scale-freeness part.", "broader_impact_concerns": "I do not have concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lHbPVXuchz", "reviewer": "Reviewer_i3NP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "For the most part, I found the paper quite easy to follow and enjoyable to read. The theoretical contributions are quite interesting, and have the potential (as evidenced by the planned future work) to lead to new training algorithms with similarly beneficial properties. The experimental work did a good job of demonstrating the practical implications of scale-freeness in the deep learning setting.\n\nThe main weakness is that the term \"proximal update\" is not always used in a precise manner, so it is at times unclear which of the existing proximal methods the manuscript is referring to. For example, the manuscript states that Eq 4 \"uses the proximal operator\", but it seems to me that there are additional terms compared to a conventional proximal operation definition, and a different Bregman divergence is used. Making this clear is important, as the manuscript does make some claims about how AdamW inherits properties that pertain to specific proximal methods. It is not clear to me how the momentum terms from Adam(W) interact with this proximal update interpretation. E.g., the convergence guarantees of Duchi et al. (2010b) are invoked, but it is not obvious to me that they apply in the case where momentum is present.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript investigates the benefits of AdamW by deriving connections to proximal methods and showing it exhibits a scale-freeness property. The paper also provides empirical validation of some approximations made in the derivation and demonstrates that the effects hypothesised by the theoretical analysis are observed in practice.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Some of the figures in Section 4 have plots that have inconsistent sizes. It would be nice (but not required for acceptance) if these could be fixed, and perhaps slightly more guidance could be given to the reader about what features of the plots can be used to draw the conclusions the authors have drawn.\n\nThe formalisation at the beginning of Section 3 should be improved before acceptance. A touch more scene-setting would be useful; e.g., introduce the definition of a proximal operation, and use it to set up some notation that allows for compact and easily understandable derivations. Doing this may also alleviate what I thought was the main weakness of the paper, by making the relationship to existing proximal methods more clear.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors provide new theoretical understandings of AdamW from the optimization viewpoint. First, they demonstrate that AdamW can be seen as an approximation of a proximal update. Second, they prove that AdamW enjoys the scale-freeness property, which leads to an automatic acceleration on certain cases.

This paper made significant progress towards understanding the advantage of AdamW. It is well-written, and the theoretical findings are supported by empirical studies.

Decision: Accept (poster)