Adam-family Methods with Decoupled Weight Decay in Deep Learning
Authors: Kuangyu Ding
Nachuan Xiao
Kim-chuan Toh
Venue: Accepted by TMLR ()

Abstract: In this paper, we investigate the convergence properties of a wide class of Adam-family methods for minimizing quadratically regularized nonsmooth nonconvex optimization problems, especially in the context of training nonsmooth neural networks with weight decay. Motivated by AdamW, we propose a novel framework for Adam-family methods with decoupled weight decay. Within our framework, the estimators for the first-order and second-order moments of stochastic subgradients are updated independently of the weight decay term. Under mild assumptions and with non-diminishing stepsizes for updating the primary optimization variables, we establish the convergence properties of our proposed framework. In addition, we show that our proposed framework encompasses a wide variety of well-known Adam-family methods, hence offering convergence guarantees for these methods in the training of nonsmooth neural networks. More importantly, compared to the existing results on the choices of the parameters for the moment terms in Adam, we show that our proposed framework provides more flexibility for these parameters. As a practical application of our proposed framework, we propose a novel Adam-family method named  Adam with Decoupled Weight Decay (AdamD), and establish its convergence properties under mild conditions. Numerical experiments demonstrate that AdamD outperforms Adam and is comparable to AdamW, in the aspects of both generalization performance and efficiency.

Published in Transactions on Machine Learning Research (04/2025)
Adam-family Methods with Decoupled Weight Decay in Deep Learning
Kuangyu Ding
kuangyndSo.nus.cd
Edwardson School of Industrial Engineering
Purdue Universily
Nachuan Xiao
merryBchkk edu.cr
School Data Science
The Chinese University of Hong Kong, Shenzhen
Kim- Chuan Toh Department of Mathematics and Institute of Operations Research and Analytics National University of Singapore
mattohkeo@uss www.ohkkkenuuuedu.... edu.sg
Reviewed on OpenReview: https: // openreview. net/ forum? id- a AVEHA
Abstract
In this paper, we investigate the convergence properties of a wide class of Adam- meth- ods for minimizing quadratically regularized nonsmooth nonconvex optimization problems, especially in the context of training nonsmooth neural networks with weight decay. Moti- vated by AdamW, we propose a novel framework for Adam-family methods with decoupled weight decay. Within our framework, the estimators for the first-order and second-order moments of stochastic subgradients Under mild assumptions and with non-diminishing stepsizes for updating the primary op- timization variables, we establish the convergence properties of our proposed framework. In addition, we show that our proposed framework encompasses wide variety of well- known Adam-family methods, hence offering convergence guarantees for these methods in the training of nonsmooth neural nelworks. More importantly, compared to the existing results on the choices of the parameters for the moment terms in Adam, we show that our proposed framework provides more flexibility for these parameters. As practical appli- cation of our proposed framework, propose a novel Adam-family method named Adam with Decoupled Weight Decay (AdamD), and establish its convergence properties under mild conditions. Numerical experiments demonstrate that AdamD) outperforms Adam and comparable to AdamW, in the aspects of both generalization performance and efficiency.
updated independently of the weight decay term.
1 Introduction
We consider the following unconstrained optimization problem (UOP):
min ZER
x=x)== ||=||²
(UOP)
where the function f:R" + IR is assumed to be locally Lipschitz continuous and possibly nonsmooth over R" Moreover, the constant > 0 is the penalty parameter for the quadratic regularization term. Such a regularization term is also known as the weight decay term, which is widely employed to enhance the generalization performance in training neural networks (Bos & Chug 1996 Krogh & Hertz 1991].
Stochastic Gradient Descent (SGD) one of the most fundamental methods for solving (UOP). SGD, all coordinates of the variable are updated using the same stepsize (ie.., learning rate). To accelerate SGD,

---

Published in Transactions on Machine Learning Research (04/2025)
KKigmma & Ba 2015) develops the widely used Adam method, which adjusts coordinate-wise stepsizes based on first-order and second-order moments of the stochastic gradients. Due to its high efficiency in training neural networks, Adam has become one of the most popular choices for various neural network optimization tasks.
Motivated by Adam, numerous efficient Adam-family methods have been developed, such as AdaBelief (Zhuang et al.) 2020), AMSGrad (Reddi et al. 2018), Yogi (Zaheer et al. 2018), etc. From a theoretical
perspective, the majority of existing works (Barakat & Bianchi 2021 Défossez et al. 2022 Guo et al., 2021:
Shi et al. 2021 Wang et al., 2022 Zaheer et al. 2018 Zhhng et al. 2022 Zhuang et al.
2022 Zou et al,
2019) establish convergence properties for these Adam-family methods, based on the assumption that f is continuously differentiable over R". However, as emphasized in (Bolte et. al., 2021 Bolte & Pauwels 2021
Bolte et al. 2022b), nonsmooth activation functions, including ReLU and leaky ReLU, are popular choices in building neural networks. For any neural network built from these nonsmooth activation functions, its loss function is usually nonsmooth and lacks Clarke regularity (e.g., differentiability, weak convexity, etc.). Consequently, these existing works are unable to provide convergence guarantees for their analyzed met in the training of nonsmooth neural networks.
1.1 Existing works on training nonsmooth neural networks
In nonsmooth optimization, it has been demonstrated in (Daniilidis & Drusvyatskiy, 2020 that a general Lip- schitz continuous function f can exhibit highly pathological properties, leading to the failure of subgradient descent method to find a critical point of f. Moreover, the chain rule may fail for the Clarke subdifferential (Clarke| 1990) of the loss function of nonsmooth neural network. Specifically, when we differentiate the loss function of nonsmooth neural network using automatic differentiation (AD) algorithms, the outputs may not be contained in the Clarke subdifferential of f (Bolte & Pauwels 2020).
Consequently, most of the existing works restrict their analysis to the class of palt-differentiable functions (Bolte & Pauwels, 2021 Definition 3). For any datf-differentiable function f, there exists a graph-closed set-valued mapping Dy, called conservative field for f, such that for any absolutely continuous mapping 7: [0 + R", it holds that f(r(t)) f(r(0)) = f6 max (f(s), d) ds for any t 2 0. It is worth mentioning that the most important choice of the conservative field D, is the Clarke subdifferential of f. Morcover, as discussed in (Bolte & Pauwels 2021 Castera et al.) 2021 Davis et al.) 2020), the class of pail--iifeerentiable functions are general enough to cover a wide range objective functions neural network training tasks, especially when the neural networks employ nonsmooth building blocks, such as the ReLU activation function. In addition, Bolte & Pauwels 22020 2021) show that the outputs of AD algorithms in differentiating nonsmooth neural networks contained in a conservative field of the loss function. Therefore, the concept of the conservative field is capable of characterizing the outputs of AD algorithms, which are implemented in training nonsmooth neural networks in practice.
Based on the stochastic approximation frameworks (Benaim 2006 Benaim et al. 2005 Borkar 2009 Davis
et al. 2020 several existing works have investigated the convergence properties of stochastic subgradient
methods in training nonsmooth neural networks. In particular, [Bolte & Pauwels (2021): ); Davis et al. (2020)
study the convergence properties of SGD and proximal SGD for minimizing nonsmooth path-differentiable
functions. Moreover, (Castera et al. 2021) proposes the inertial Newton algorithm (INNA), which can be
regarded as a variant of momentum-accolorattee SGD method. Additionally, Le
22023 Ruszczyński (2020):
Xiao et al. (023bb establish the convergence properties of SGD with heavy-ball momentum, while Ding & Toh (2024) investigates the convergence of Bregman-type methods. Furthermore, Hu et al. (2022a b) apply
these mothods to solve manifold optimization problems basod on the constraint dissolving approach (Xiao)
et al. 2023c). In addition, Gürbüzzalabaa et al. (2022): Ruszczynski (2021) design stochastic subgradient methods for solving mult.i-level nested optimization problems.
2

---

Published in Transactions on Machine Learning Research (04/2025)
1.1.1 Challenges from non-diminishing stepsizes in Adam
With the concept of conservative field, Adam utilizes the following framework when applied to solve UOP
Is d2 + E6+1,
m (1 -00)mm + (x( + a.n),
(Adam)
Uk++1 = -61-Be)vx + Be + o.r))
1k+1 = Ik - mm (VT+T + + @ mg+11
Here, 9h a stochastic subgradient of ʃ at. Ik. in the sense that dg represents a possibly inexact evaluation of Dy(x)), Ek+1 is a random vector characterizing the evaluation noise and ørą is the weight decay term. The operators O and (+) denote clement-wise multiplication and clement-wise p-th power of given vector, respectively. The sequences {ma} and {*x), referred to as momentum terms and estimators respectively, are updated to track the first-order and second-order moments of {ge + ozx). The sequences {ma), {0k), and { represent the stepsizes for the primal variables {æ the parameters for the momentum terms {mah. and the parameters for the estimators {uk}, respectively.
In the framework (Adam), the weight decay term is integrated with the function f throughout the iterations. As a result, we can directly apply the existing convergence results on Adam to analyze the convergence properties of the framework {Adamp. In particular, when f is nonsmooth path-differentiable function, (Xaao et al.) 2023a) investigates the convergence of a class of idam-family methods based on the frameworks proposed by [Benaim et al. 2005 Bianchi et al. 2022 Davis et al. 2020). However, in the analysis of (Niao et al. 2023a the stepsizes and parameters sequences are assumed to be diminishing and single-timescale, in the sense that ]m }0x} and { converge to 0 at the same rate as k goes to infinity.
Boyond the single-timescale scheme, some existing works (Reddi et al., 2018 Zhang et al. 2022 Jin et al. 2024) establish the convergence of Adam for continuously differentiable I with {0s} and (3x fixed constants. In particular, (Zhang et al. 2022) proves that for any 0 E (0, 1) and m7 O(1/VB) there exists sufficiently small B that forces } to stabilize within neighborhood of the critical points of g. In addition, (Jin et al. 2024) establishes the asymptotic convergence convergence of Adam with relaxed conditions on the hneo of f. However, their analyses are restricted to continuously differentiable objectives. Therefore, these results are not capable of explaining the convergence of Adam in a wide range practical settings, where the neural networks are built from nonsmooth blocks. It worth noting that (Xiao et al.] 2023b) establishes the convergence of stochastic subgradient methods with two-timescale stepsizes. However, their analysis is restricted with SGD-type methods. To the best of our knowledge, this analysis cannot be extended to Adam-family methods.
Furthermore, in establishing convergence properties for stochastic subgradient methods, diminishing step sizes are commonly assumed as they facilitate the almost sure convergence the itcrates {Ix } to critical points under various conditions (Benaïm el al. 2005 Bolte et al., 2022a Bolte & Pauwels 2021 Castera Xiao et al. 2023a b). However, for Adam, the results in (Reddi et al.] 2018 Zhang et al.) 2022) indicate that even with diminishing step sizes, the iterates { } are only guaranteed to converge to neighborhood of critical points. Moreover, Bianchi et al.) (2 ; Josz et al. (2023) show that for nonsmooth, path-diiferenniabble objective functions and fixed step size, SGD converges almost surely only to a neighborhood of the Dy-stationary points of f. Their analysis, however, is limited to SGD and SGD with heavy-ball momentum and does not extend to Adam. Despite the widespread use of diminishing step size schemes in practice, it is both theoretically and practically important to investigate the convergence properties of Adam-family methods in the non-diminishing regime (i.e., when lim infx-+00 > 0).
let al.] 2021: Davis et al.] 2020 Le] 2023) Ruszezyíski 2020;
1.1.2 Challenges from decoupling the weight decay term in Adam
Another challenge in solving (UOP) by Adam is related to the incorporation of the weight decay term. The conventional approach is to directly minimize g by Adam, as implemented in various computational frameworks. That is, the weight decay is coupled with the stochastic subgradients of f, in the sense that f and the weight decay term z ||rr|| are treated as an integrated function to be minimized. As demonstrated
3

---

Published in Transactions on Machine Learning Research (04/2025)
in Loshchilov & Hutter 2019 Adam with coupled weight decay usually exhibits worse generalization performance than SGD. To address this issue, Loshchilov & Hutter (2019) suggests novel method named AdamW, which decouples the weight decay term from the stochastic subgradients of f. The update schemes of AdamW can be summarized by the following framework:
d% + Ex+1,
m = (1 - 0x)mx + 0x:Kk.
(Adamw)
uk+1 = - (1=)J) + Bx(
#k++1 2 Va(VeE++ + +) m 7kTZk-
(2019) demonstrates that the weight decay decoupled from the momentum terms {ma} and the estimators {ua), the sense that the update schemes for {m²} and {ux} are independent of the weight decay parameter Ø. Moreover, unlike (Adam), the weight decay term ozy is not scaled by the preconditioner (vTE+I e)- in AdamW.
Here, Loshchilov & Hutter
The AdamW, recognized for its superior generalization performance over Adam with coupled weight decay (i.e., the method in (Adam ), has become a popular choice in the training of neural networks (Loshchilov & Hutter 2019). particularly in tasks such as image classification and langnage modeling. As shown in (Schaipp) 2023 Zhuang et al. 2022 AdamW can be interpreted as a proximal approximation Adam, with regularization handled via proximal operators. However, compared with Adam, the convergence properties of AdamW and its proximal variants (Schaipp 2023 Zhuang et al.) 2022) remain relatively unexplored. As suggested in (Loshchilov & Hutter 2019 Zhou et al. 2024), AdamW iterates by taking descent step towards dynamically adjusted surrogale function [(z) 2 (x,, (vrx-1 + e) O x) in the k-th iteration, thereby lacking clearly defined objective function to minimize. As a result, only the paper by (Zhou et al. 2024 has established the convergence properties of AdamW for contimuuuly differentiable f. In (Zhou et al. 2024). the stationarity AdamW is measured by [|Vf(z) (y/k++1 + E) @ =l|- As the estimators { evolves over iterations and may not converge, the proposed stationarity measure is at best an approximation of the standard notion of stationarity. More importantly, the analysis in (Zhou et al. (024) relies on the differentiability of f, and cannot be extended to analyze the convergence of AdamW for nonsmooth cases. Consequently, the results presented in (Zhou et al.. 2024) do not sufficiently explain the convergence of AdamW in real-world training tasks, where the neural networks are typically nonsmooth.
Given that Adam-family methods with coupled weight decay usually perform less effectively than AdamW, and considering that AdamW lacks convergence guarantees in training nonsmooth neural networks, we are driven to raise the following question:
Can we design Adam-family methods with decoupled weight decay that have convergence guarantees with non-diminishing stepsizes, in the context of training nonsmooth neural networks?
1.2 Contributions
The contributions our paper are summarized as follows.
A novel framework with decoupled weight decay
In this paper, motivated by AdamW, we propose a novel framework for Adam-fan Adam-family methods with decoupled weight decay (AFMDW),
gh = dy + Ek+11
mi+1 (1 0x)m² + 0
(AFMDW)
Choose the estimator Uk+1,
[k++1 Ix 7) (vx+1)) @ (m++1 + axx)).
Here, dy is an approximated evaluation of (*x), while Ex+1 is the corresponding evaluation noise of dr. Therefore, gk represents the stochastic subgradients of f at Ta- Moreover, the sequences
4

---

Published in Transactions on Machine Learning Research (04/2025)
{me) and }0k} are stepsizes for the variables {2x} and parameters for the momentum terms {m²): respectively. Furthermore, H IR" + R" is the mapping that determines how we construct the preconditioner based on Uk+1- As the framework (AFMDW) is designed to minimize (UOP), both the momentum term m and the weight decay term ozk are scaled by (U in (FFMDW) distinguishing it from AdamW.
Convergence analysis
We establish the global convergence of the framework (AFMDW under mild conditions with non- diminishing stepsizes. When the noises { correspond to random reshufiling (RR), and the es- timator {uk} is updated as in (AdamW) with non-diminishing (ne and {3x), we prove that with sufficiently small but non-diminishing {0x}, ) the sequence { could stabilize within a neighborhood of the critical points of (COP). In addition, when we further assume {0x} + O, we prove that the sequence { converges to the critical points of (UOP) almost surely. Moreover, by employing single-timescale scheme in (AFMDW), we prove that with sufficiently small {nk J, the sequence { stabilizes within a neighborhood of the critical points of (UOP).
Furthermore, we extend the convergence analysis of the framework (AFMDW] with diminishing stepsizes and with replacement sampling (WRS), and establish the almost sure convergence to critical points of (UOP). Table ] presents a brief comparison of our results with existing works on the convergence of stochastic subgradient methods.
Table 1: A brief comparison of our results and existing works on the convergence of stochastic subgradient methods.
Result
Sampling method
Update scheme
Stepsizes
Convergence
Guaranteed stability
Theorem 10 & 3.22
WRS
Adam
Diminishing
Almost sure
Theorem
3. 13
RR
Adam
Constant
Almost sure
Y
Josz et
2023
RR
SGD
Constant
Almost sure
Bianchi et
2022
WRS
SGD
Constant
High probability
Y
Xiao et al.
2023a
WRS
Adam
Diminishing
Almost sure
Advantages in incorporating weight decay into Adam
We demonstrate that the framework (AFMDW) encompasses (see Table 2 for details) a wide range of Adam-famii methods, including SGD, Adam, AMSGrad, AdaBelief, AdaBound, Yogi. Therefore, our analysis provides convergence guarantees for these Adam-family methods in training nonsmooth neural networks.
coo-conveeeence conveevveeeeece analysis of Adam in (Reddi et al. 2018 Zhang et al.
Moreover, compared with the non 2022), our analysis illustrates that the incorporation of weight decay term grants more flexibility on the choices of the parameters {0x) and {B²} } for the framework (AFMDW) These results illustrate the great theoretical advantages of a weight decay term in the framework (AFMDW).
Numerical experiments
Based on our proposed framework (AFMDW) , we develop a novel method named Adam with Decou- pled Weight. Decay (AdamD) and establish its convergence guarantees training nonsmooth neural networks. We conduct numerical experiments in both image classification and language modeling tasks to assess the performance of our proposed AdamD. The results show that in image classification tasks, AdamD) outperforms Adam and performs comparably to AdamW in both generalization and efficiency. In language modeling tasks, it demonstrates similar effectiveness to Adam and outper- forms AdamW, highlighting its versatility and effectiveness across different tasks. Additionally, our numerical experiments illustrate that the sequence {llyx - æxl1} tends to 0, where y& is an auxiliary variable that approximates the dynamics of SGD. This validates our theoretical analysis that the proposed AdamD asymptotically approxinates the SGD method. These results further demonstrate the promising potential of our proposed framework (FMDWW)
5

---

Published in Transactions on Machine Learning Research (04/2025)
1.3 Organization
The rest of this paper is organized as follows. In Section 2, we define the notations used (hroughout the paper and present some basic concepts related to nonsmooth analysis and stochastic approximation. Section 3 presents the convergence properties of our proposed framework (AFMDW) with non-diminishing stepsizes }m Moreover, we extend these convergence properties to the framework (AFMDW) with timesslle stepsizes. As an application of our theoretical analysis, we propose a new Adam-family method named Adam with Decoupled Weight Decay (AdamD) and establish its convergence properties in Section 4. In Section 5, present the results of our numerical experiments that investigate the performance of the proposed AdamD in training nonsmooth neural networks. Some further discussions on AdamD are also presented in Section 5. Finally, we conclude the paper in the last section.
2
Preliminaries
2.1 Notations
For any vectors and y in R2 and 8 E R, we denote I T y, x², 2/v, |z], x+6 6, v as the vectors whose i-th entries given by Zill; zf, 2s/bu Va: |aal, H, 6, and vIa, respectively. We denote R == fa R z; 2 O for any 1 < S n). Moreover, for any subsets X, R", we denote X' o y := f X, y V). X = [|0] E } and || [Iw]] w X}. In addition, for any IR" we denote 2 + {z) +X'
and = o X = {>} o X.
Furthermore, for any positive sequence (0t), we define d == 0, )₂ = Ex= Og for i 2 1, and A(t) = ( > 0:62 2 AaJ. More explicitly, A(I) = if X S t < Apt1 for any p 2 0. In particular, A(A))
2.2
Noise model
In this subsection, we present some essential concepts from probability theory, which are necessary for the proofs in this paper.
Definition 2.1. Let FP be probability space. We soy that (Falkes is filtration if {Fa} is collection of aallebbaa that salisfies Fo F .... C Frx F.
Definition 2.2. We say that stochastic series {Ex} is a martingale difference sequence if the following conditions hold,
The sequence of random vectors {Ex} is adapted the filtration {Fx),
For each 0, almost surely, it holds that E[[Ex]] < and E [x 0.
Moreover, we say that a martingale difference sequence {Ex] uniformly bounded if there exists constant Mç such that sups20 ||Ex|l < Me.
In the following, we present the results in (Benaim 2006
Proposition 4.4), which controls the weighted
summation of any uniformly bounded martingale difference sequence, and plays a crucial role in establishing the convergence propertics for our proposed framework (AFMDW).
Proposition 2.3 (Proposition 4.4 in (Benaïm] 20066). ). Suppose { is a diminishing positive sequence of real numbers that satisfy lima 02 log(k) = O. Then for any T > O, and any uniformly bounded martingale difference sequence {Ex), almost surely holds that
lim SSISAAA
sup
Σ Ox5N41
(1)
300
2.3
Nonsmooth analysis
In this subsection, we introduce some basic concepts in nonsmooth optimization, espocially those related the concept of conservative field (Bolte & Pauwels 2021). Interested readers could refer (Bolte & Pauwels, 2021 Davis et al. 2020 for more details.

---

Published in Transactions on Machine Learning Research (04/2025)
We begin introduction on the concept of Clarke subdifferential (Claree 1990). which plays essential role in characterizing stationarity and the development of algorithms for nonsmooth optimization problems.
Definition 2.4 (Clarkke 1990 For any given locally Lipschitz continuous function f IR R and any R", the Clarke subdifferential 0 is defined as
0f(z) conv ((d R : 24 - Vf((x) + d]).
(2)
Next we present a brief introduction on the concept of conservative field, which can be applied to characterize how nonsmooth neural networks are differentiated by automatic diffcrentiation (AD) algorithms.
Definition 2.5. A set-valued mapping D IR" = R is mapping from I" to a collection of subsets of RR. D is said to have a closed graph, or graph-closed if the graph of D, defined by
graph(D) = [(w, 2) E R2 x R² w E R", D(w)),
is closed subset of R x R
Definition 2.6. A valudd mapping D IR" R is said be locally bounded if, for any æ R", there is neighborhood V2 of such that Uyev D(y) is bounded.
Next, we present the definition of conservative field and its corresponding potential function.
Definition 2.7. An absolutely continuous curve is a continuous mapping " R4 - R whose derivative Y erists almost everywhere in 1 and (()) - 7(0) equals the Lebesque integral of belween O and for all E R.,, i.e.
>0)=)00)- + / 7(()dd, for all E R.
Definition 2.8 (Definition 1 in (Bolte & Pauwels 2021 ). Let D be a graph closed set- val-valudd mapping from T subsets of Rr. We call D conservative field whenever it has nonempty compact values, and for ony absolutely continuous curve 7: [0, 1] + R. satisfying 7(0) = (1), holds that
max (t((t), dt 0.
(3)
vED()(t)
Here the integral is understood in the Lebesgue sense.
It is important to note that any conservative field is locally bounded (Bolte & Pauwels 2021 Remark 3). We now introduce the definition of potential function corresponding to conservative field. Definition 2.9 (Definition in (Botte & Pauwels 2021)) Let be a conservative field in I8 Then with any given 2a R", define function f R IR through the path integral
f(x)= f
dt = f(xo) ['' min dED(TT))
())
(4)
max
dED()())
for any absolulely continuous curve y that salisfies 70)) = To and 7(1) = x. The function J is called a potential function for 'D. We also say that D admits I as its potential function, or that D is a conservative field for f.
The following two lemmas characterize the relationship between conservative field and Clarke subdifferential.
Lemma 2.10 (Theorem 1 in (Bolue & Pauwels, 2021 ). Let f RD - R be polentåal funclion that admils Dj as its conservative field. Then Dz(x) = {Vf(z) almost everywhere.
Lemma 2.11 (Corollary 1 in (Bolte & Pauwels ). Let f R - IR be potentiol function that admits D, as its conservative field. Then Of is a conservative field for f, and for all x E R it holds that
Of(x) C conv (Dy(x)).
(5)

---

Published in Transactions on Machine Learning Research (04/2025)
From the above two lemmas, eralization of Clarke subdifferential. Therefore, conservative field can be applied to characterize stationarity, as illustrated in the following definition.
conclude that the concept of conservative field can be regarded as gen-
Definition 2.12. Let f : In + IR a potential function that admits Dj as its conservative field. We say that x is a Dy-stationory point of f if 0 E conv (Dy(z)). In particular, we say x is a ff--ssftionayy point of f if e 0f(z).
generalization of Clarke subdifferential. Therefore, function is differentiable in the sense of conservative field if it admits a conservative field for which Definition 2.9 holds true. Such functions are called path-differentiable (Bolte & Pauwels 2021 Definition 3), which is given below.
As demonstrated in (Bolte & Pauwels 2021
conservative field can be regarded
Definition 2.13. Given a locally Lipschitz continuous function f R" + R, we say that f is path- differentiable if f the potential function of a conservative field on R
It is worth mentioning that the class of path-differentiable functions is general enough to cover the objectives
in a wide range of real-world problems. As shown in (Davis et al., 2020 Section 5.1), any Clarke regular func- tion is path-differentiabll. Beyond Clarke regular functions, another important class of path-differentiable fanctions are functions whose graphs are definable in an o-minimal structure (Davis et al. 2020 Definition 5.10). Usually, the o-minimal structure fixed, and we simply call these functions definable. As demon- strated in (Van den Drics & Miller 1996). any definable function admits a. Whitney C* stratification (Davis 2020 Delinition 5.6) for any 8 8 1, hence is path-differentiable (Bolte & Pauwels 2021 Davis et al.
ee al.
2020 To characterize the class of definable functions, (Davis et al. 2020; Bolte & Pauwels] 2021 Bolte
et al. b022b) shows that numerous common activation functions and dissimilarity functions are all definable.
Furthermore, since definability is preserved under finite summation and composition (Bolte & Pauwels 2021
Davis et al. 2020), for any neural network built from definable blocks, its loss function is definable and thus belongs to the class of path-differentiabll functions.
Moreover, (Bolte et al. 2007) shows that any Clarke subdifferential of definable functions is definable, Consequently, for any neural network constructed from definable blocks, the conservative field corresponding to the AD algorithms can be chosen as a definable set-valued mapping formulated by compositing the Clarke subdifferentials of all its building blocks (Bolte & Pauwels 2021 The following proposition shows that the definability of f and Dj leads to the nonsmooth Morse Sard property (Bolte et al. (007) for (UOP).
Proposition 2.14 (Theorem 5 in (Bolte & Pauwels 2021)). Let f be a potential function that admits Dy as its conservative field. Suppose both f and D are definable over R,,", then the set {f(x) conv (D)(
is finite.
2.4 Differential inclusion and stochastic subgradient methods
In this subsoction, we introduce some fundamental concepts related the stochastic approximation tochnique that are essential for the proofs presented in this paper. The concepts discussed in this subsection mainly
from [Benaim et al., 2005). Interested readers could refer to (Benaim 2006 Benaim et al. 2005 Borkar
2009
Davis et al.] 2020 for more details on the stochastic approximation technique.
Definition 2.15. For any locally bounded set-valued mapping D R" = I2 that is nonempty compact convex valued and has closed graph, we say that an absolutely continuous path ztt) in R is a solution for the differential inclusion
dx
E D((),,
(6)
dt
with initial point No if x0) no, and a:(t D(x(()) holds for almost every t. 0.
Definition 2.16. For any given sel-valued mapping D R" R and any conslant 8 2 0, the sel-valued mapping DE is defined as
D²(2) := ( 32 Bz(x), dist(w, D(2)) < 8)
(7)
Definition 2.17. Let B R be a closed set. continuous function ò R. - is referred to as a Lyapunov function for the differential inclusion
( with the stable set B, if il salisfies the following conditions:
8

---

Published in Transactions on Machine Learning Research (04/2025)
1. For any that is solution for 6 with 7(0) E B, it holds that d()(t)) <(>(()) for any t 2 0. For any y that solution for with 7(0) & B, it holds that d()(t)) < ((7(0)) for any >
The following proposition illustrates that is a Lyapunov function the differential inclusion at E (D)
The proof of the following proposition directly follows from (Bolte & Pauwels 2021). hence is omitted for simplicity.
Proposition 2.18. Suppose f is aath-differentiable function f that admits Dj as its conservative field. Then f is Lyapumov function for the differential inclusion dx -Dy(x) with the stable set {xr R2 Dy(x))-
Definition 2.19. We say that an absolutely continuous function is perturbed solution to 6 if there evisls locully integrable function R2 + R", such that
1t u(s) ds = O.
For any T > O, it holds that lim sup 10000 0S1ST
There crists 8 R4 + R such that lim 5(t) and 1(t) u(t) ()W))))))
Now consider the sequence } generated by the following updating scheme,
#k+1 2k 7h (dx + Ek),
(8)
where } is diminishing positive sequence of real numbers. We define the (contimnous-timc) interpolated process of { generated by as follows.
Definition 2.20. The (continuous-time interpolated process of { generated by 8 is the mapping w R4 - IR" such that
w(); + 5) == 1; 7
(I141 zi), E [0,
Here Au 0, and Eizh Th for i
The following lemma an extension of (Benaim evaluations of the set-valued mapping D. It shows that the interpolated process of {Ix} from is perturbed solution of the differential inclusion 6
2005 Proposition 1.3), which allows for inexact
Lemma 2.21. Let D Rn = Rn be locally bounded set-valued mapping that is nonempty compact conver valued with closed graph. Suppose the following condilions hold in
Ei TREK 0.
1. For any T > 0, it holds that lim sup SSISA(A++)
There erist posilive sequence {ōx} such that lime-soo 8u O and de DO DDx())).
3. sup:20 |xa||| < 0o, sup+2o ||dx|l < <c.
Then the interpolated process of { defined in 19 is a perturbed solution for
The following theorem summarizes the results in Benaïm et al. 2005), which illustrates the convergence of
{xx}
generated by 8 is worth mentioning that Theorem 2.22 is directly derived from putting (Benaim
et al.
2005 Proposition 3.27) and (Benaïm et al. 2005 Theorem 3.6) together. Therefore, we omit the
proof of Theorem 2.22 for simplicity.
Theorem 2.22. Let D R = R be a locally bounded set-valued mapping that is nonemply compact conver valued with closed graph. For any sequence (xx), suppose there erist. continuous funclion o R - R and closed subset B of g such that
1. is bounded from below, and the set (Ø(x) B) has emply interior in R.

---

Published in Transactions on Machine Learning Research (04/2025)
2. is Lyapunov function for the differential inclusion (6) that admits B as its stable set.
3. The interpolated process of { I is a perturbed solution of
Then any cluster point of { lies in B, and the sequence {d(x)) converges.
Similar results under slightly different conditions can be found in [Brrkrr 2009 Davis et al. 2020 Duchi &
Ruan 2018). Moreover, towards the convergence properties of (8) with potentially non-diminishing stepsizes,
several recent works (Bianchi ct al. 2022 Josz et al.) 2023 Xiao et al. 2023b) provide convergence guarantees
under more relaxed conditions. Interested readers could refer to those works for details.
3 Global Convergence
In this section, we prove the convergence properties of the framework (FFMDW) even though the sequence of stepsizes } is assumed to be non-diminishing. The proofs are provided in the Appendix.
3.1 Basic assumptions
We first make the following assumptions the quadratically regularized optimization problem (UOP). Assumption 3.1.
f a path-differentiable function that admits conver valued set-valued mapping Dj as ils conservative field.
2. There exists a constant L > 0 and v E [0, such that for any x E TR" il holds that [D s
L(1 + +|=||"").
3. The set {9(x): :0 E (T(x)) ax) has empty interior in R.
As discussed in Section 2.3, the class of path-differentiable functions covers a wide variety of objective functions in real-world applications. In particular, for wide range of common neural networks, their loss functions are definable and thus path-differentiabll, ct al. 2021 Davis et al. (020). As result, Assumption 3.1 (1) is mild in practice. Morcover, Assumption
demonstrated in (Bolte & Pauwels 2021 Castera
3.122 (2 imposes a growth condition on the conservative field. Furthermore, Assumption (3) is referred to
as the nonsmooth weak Sard's property, which is commonly observed in various existing works Bianchi & Castera et al.
Rios-Zertuche] 2021 Bolte et al.) 2022a Bolte & Pauwels 2021
2021 Davis et al., 2020 Le
2023) and is demonstrated to be mild in (Bolte & Pauwels 2021 Castora ct al.
2021 Davis et al. 2020
Notice that the chain rule holds for conservative fields (Bolte & Pauwels
2021 Lemma 5), and it is easy to
verify that is path-differentiable function that admits D, (x(x) + its conservative field. Therefore, in
the rest of the paper, we fix the conservative field Dy : R = R² for the objective function g in (UOP) as:
Dy(x) = ((((x) + or.
(10)
In the following lemma, present some basic properties of Dg. The proof of Lemma 3.2 follows straight- forwardly from (Bolte & Pauwels 2021 Corollary 4), hence it is omitted for simplicity.
Lemma 3.2. Suppose Assumption 3.1 holds. Then g is path- dith-differenttaabl function, and D, is conver- valued graph- closhe conservative field that admits g its potential function.
We also need the following assumptions the framework (AFMDW) to establish its convergence properties. A Assumption 3.3.
There erist constants S, and M, with E Me, such that E, S (1 S M, holds for any k 2 0.
2. There exists non-negalive sequence {dx} such that lime- oz = 0 and ds E DJ (Tx).
The sequence of noises {Ex} is a uniformly bounded martingale difference sequence. That is, there erists constant Mg such that almost surely, supe20 [|Ex]] S Mg, and B(Ex+1 [Fa] 0 for any k. 2 0.
10

---

Published in Transactions on Machine Learning Research (04/2025)
assumes the uniform boundedness of (H(m)), which is satisfied in various existing works as shown in Table Tn addition, later in Section 3.2, we provide some sufficient conditions that guarantee the uniform boundedness
Here, we make some comments the assumptions in Assumption 3.3
Assumption 3.3
of {zz}. Assumption 3.3 2) characterizes how dg approximates (x(x)).
Furthermore, Assumption 3.3(3) assumes that the evaluation noises {Ex} is a uniformly bounded martingale difference sequence. Consider the setting where f(x) = + Σn fi(æ)) for locally Lipschitz continuous and path-differentiable functions {fi} that admit { } as their conservative fields (i.e., f follows a finite-sum
formulation), {ara) uniformly bounded, and gs E fa, (r&) with randomly chosen is E [N] by WRS. Then as demonstrated in (Bolte & Pauwels 2021; Castera al. 2021), there exists {Ex} such that g Ex+1 E D (2x) and {Ex } is uniformly bounded martingale difference sequence. Therefore,
conclude that Assumption 3.3 3) characteizes the evaluation noises when f follows finite-sum formulation and the indexes sampled by WRS.
3.2 Uniform boundedness of {ax} and {
In this subsection, present some sufficient and easy-to-verify conditions that guarantee the validity of uniform boundedness of {xx). The following proposition illustrates that under some mild global growth conditions for f and the uniform boundedness of (Hx)) the sequence {mR} is uniformly bounded.
Proposition 3.4. Suppose Assumption 3.1 and Assumption 3.3 hold, and supk2u S - Then for any initial point (Io, mo, vo), there erists a constant Q > 0 such that sup=20 ||0X||| S Q.
Next, we discuss the uniform boundedness of the sequence {H(u))) Apart from Assumption 3.1 and Assumption 3.3 we make the assumption on the global Lipschitz contimuity of f, in the sense that
[Df(,)]] s My for some constant Mj > 0. =
sup
(11)
Such assumption is standard in varions existing works. Table 2 lists some Adam-family methods, where
the sequence (+R) remains uniformly bounded under Assumption 3.1 Assumption 3.3 and equation
Table 2:
Different update schemes for } in the framework (AFMDW) under Assumption 3.1
Assumption
and 11
Here c, Ct, Cu. > 0 are hyper-paramcters for these Adam-family methods.
Method
Update scheme for [vx}.
Formulation for H())
Choice of (Ev, Mo) (1,1)
SGDW Looshhhlovv & Hutter
2019
vk+1 = (1 1o Bigh
1
Adam
(Kiggua & Ba
2015
VE+1 (1 81) 819
(e)e))
(Mg++
AMSGrad
Reddi et al.
2018
Ph-1 max(ue: (1 Bi + (19
(Vo+e)-1
[My MM
Adamax
K Kingma& & Ba 2015
Mk+1 maxx/11114 vk. laa| [e)
(v)
(aky + Me M
RAAam
Liu al.
2019
Mk+1 - Bi u $199
(V
(M, Me
AdaBelief
Zmang et 2020
#141
91 )o2 B1 (gx- m241)
(V++e)-1
2(0r M )+e
AdaBound
Luo et al. 2019
V (1 31 v B19
min min v 21}
(q,, ew) (M
Yogi (aheer al. 2018
Ub+1 U/ - sign (v% 9
(Vo+s)-1
3.3 Convergence with non-diminishing stepsizes(na)
Assumption 3.5.
The sequences of stepsizes {na and momentum parameters {0x} satisfy
Munx sup Mk < min oMo oe Muiin == inf m > 0, and Σ 0x = oc. #20
(12)
k20
We bogin our theoretical analysis with Lemma 3.6 which shows that the sequence {m²} and {gt} are uniformly bounded. Lemma 3.6 directly follows from the uniform boundedness of {ax} in Proposition 3.4 and (Ex) in Assumption 3.3 and the fact that Dy is locally bounded, hence we omit its proof for simplicity.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Lemma 3.6. Suppose Assumption 3.1 and Assumption 3.3 hold. Then there crists constant Ma > 0 such that sup=zo (1 [Iaxe] + [|m=l))) & Ma holds almost surely.
Lemma 3.7 illustrates that I|or= + mæ - 0 the momentum parameter f0x) diminishes.
Lemma 3.7. Suppose Assumption 3.1 Assumption 3.3 and Assumption 3.5 hold. Then for any (0k)
salisfying lim=++o O2 0, we have that ling-++o lork m& = 0 holds almost surely.
From the proof of Lemma 3.7 it follows that the asymptotic behavior of | + mxll be controlled by }0x} as & + 20, Specifically, from equation (32), we have limg- 8k+1 0 when limg- Og Consequently, for any ε 0, there exists threshold mmaz 0 such that, if lim supx-soo Og fmarx it follows that lim supx- loor + mm S &. Moreover, the convergence of | + mm is faster as {0x} decreases more rapidly.
Based on the Lemma 3.7 let the auxiliary sequence } be defined as
1 yk m,, for any 2 0.
(13)
Then we can conclude that limg llyk scheme for {m,} in (AFMDW), we arrive at the following relation
I& More importantly, substituting into the update
O% (d+ + tyv + Ex+1).
(14)
Uk+1 = Uk
In the following lemma, prove that dy + ay, can be regarded approximated evaluation for D, (yx).
Lemma 3.8. Suppose Assumption 3.1 Assumption 3.3. and Assumption 3.3 hold. Then let 8% (1
d6 + 8c, holds that
dg oyk DJ (vk),
(15)
where & defined in equation
32
We can conclude from Lemma 3.8
that the auxiliary sequence {yk} follows the differential inclusion,
Ok (D. (yx) + EA+1).
yk-1 yv
(16)
This fact illustrates that the sequence {yn} can be viewed as sequence generated by the SGD method for minimizing g. Therefore, in the following proposition, prove that the interpolated process of the sequence } is perturbed solution of the following differential inclusion:
dy dt
E (D))).
(17)
We first present the results for the case where the noise is induced by with-replacement sampling.
Proposition 3.9. Suppose Assumption 3.1 Assumption 3.3 and Assumption 3.5 hold, and lim 0x log(k) = 0. Then the interpolated process of the sequence {yk} is a perturbed solution for the differential inclusion
In the following theorem, we prove the convergence properties of the framework (FFMDW).
Thcorem 3.10. Suppose Assumption 3.1 Assumption 3.3 and Assumption 3.5 hold, and
limg-++00 0g log(k) = 0. Then almost surely, any cluster point of the sequence {rx} is a D Dy-stationary point of g, and {g(x+)} converges.
In the rest of this subsoction, we aim to establish the global stability of the framework (FMDDW), where the noises {Ex} correspond to random reshuffling. Therefore, we make the following assumptions on the momentum parameters (0x) and noises (Ex).
12

---

Published in Transactions on Machine Learning Research (04/2025)
Assumption 3.11. There crists integer N > 0 such that
For any nonnegative integers i,j N, it holds that OKN4i - OEN+S for any k E N... For ony j E N almost surely, it holds that Σ k=jN
G+1NN-1 Ek+1= =0
Here we make several remarks on Assumptions 3.11 Assumption 3.11 demonstrates that the momentum parameters {0x} remain constant within each epoch, which is standard setting in neural network training
tasks.
Morcover, Assumption 3.11 2) characterizes the random reshuffling sampling technique for computing the
stochastic subgradients {gx} in AFMDW] For example, consider the case where f(x) = x Σi", fi(2), with convex-valued conservative field D (z)). At each iteration, sample is from [N] : (1, .1, NN such that {ix J < k < (j + 1)N} = [N] holds for any j 2 O. We then compute E p, (Ix). This setup corresponds to the standard random reshuffling framework.
each Si being path-diffeerentabble and admitting
For each 2 0, let jk 2 0 be such that k E [i (ix + 1)M), it follows that Ox E ED(( + Es+1, where Ea+1 represents the evaluation noise. Defining
6k =2N
Σ
| - =x1l,
(18)
in SIN
have Ok E Dir,
(#*))
(2N) frx
(IJuN)-
Consequently, it holds that
1
Σ
Σ
Djn (xjeN) S ( S DDf
gi E
N
N
Furthermore, it holds that N #ENnSk<(HHD) Ex+1 =. Therefore, we conclude that Assumption 3.11 2) corresponds to settings where the stochastic subgradients {ar} are generated by the random reshufflng sampling technique. Furthermore, as shown later Theorem 3.13 establishing the global stability of the iterates only requires } be sufficiently small rather than diminishing. This condition ensured when }mR} is chosen sufficiently small, given the definition of {6x} in (18
Lemma 3.12. Suppose Assumplion 3.3(3) and Assumption 3.11 hold for the sequence of noises (Ex) and momentum parameters {0k}. Then for any > 0 and T > 0, there erists Og > such that for any {0k} satisfying lim su O2 < Oe, almost surely, it holds that
Σ 0RCkt1 Se. kms
limsup sup +00 ((SAA(())T)
(19)
Then we have the following theorem illustrating the global stability of the framework non-diminishing }
AFMDW with
Theorem 3.13. Suppose Assumption 3.1 Assumption 3.3 (1))) Assumption 3.5 and Assumption 3.11 hold. Then for any s O, there exists 0 0 and > O, such that for any {0x} and {dx} satisfying lim su 6x < 8 and lim sup Og 0, almost surely, it holds that
limsup dist (Ix, {x E RR. -. Se. k-++00
(20)
implics that as long as the momentum paramcters {0x} } are sufficiently small, the sequence {ax} maintains stability, regardless of how the est istimo vs is updated. When the estimator i& is updated in the manner of Adam, ie., as a parameter 0g is kept small, irrespective of the choice of parameters for updating the socond-moment estimator Ug.. Therefore, our proposed AdamD offers greater flexibility in selecting momentum parameters and those associated with updating vg- For numerical illustration, refer to Figure in Section 5.1.3
Thcorem 3.13
estimator, AdamD consistently works as long as the momentum
13

---

Published in Transactions on Machine Learning Research (04/2025)
Moreover, from the results in Theorem 3.13 can asymptotically find the stationary points of (UOP). The result is presented in the following corollary and is omitted for simplicity.
can prove that with diminishing [0x]. the sequence {a*
hold, and Corollary 3.14. Suppose Assumption 3.1 Assumption 3.5 Assumption 3.5 and Assumption 3.11
lime-+40 O2 = 0. Then almost surely, any cluster point of the sequence { is a Pg-slationary point of and {g(z))} converges.
3.4 Convergence with a single- tingle-tmmescal in } and {0x}
In this subsection, we investigate the convergence of the framework (FMMDW ) when the sequences of stepsizes { and momentum parameters } are singlo-timescale in the sense that they diminish at the same rate.
The convergence properties presented in Section 3 suggest that the sequence {yk asymptotically approx- imates the trajectory of the differential inclusion 17 One may conjecture that this phenomenon at- tribulable to the involvement of non-diminishing stepsizes {mk] in the framework (AFMDW).
However, in this section, we aim to show that when single-timescale stepsizes and momentum parameters are employed in the framework (AFMDW), the interpolated process of }ya is still a perturbed solution of the differential inclusion (17). These theoretical results suggest that it is the decoupled weight decay that leads the asymptotic approximation of the differential inclusion 17 in the framework (AFMDW). regardless of the timescale of the employed sequences {nx and {0x}-
The proof techniques in this section are motivated by the techniques in (Xiao et al. 2023a Section 3). To prove the convergence of (AFMDW) with single-timescale sequences {mk) and [0x]. we make the following assumptions.
Assumption 3.15.
1. There erists locally bounded mapping W R R + R2 and prefized constand T2 > O such that the sequence of estimalors {wR} follows the update scheme uk+1 = Uk 72m((ex - ( mx+1))-
2. The mapping H RT R is fired as H(v) = (max - =) for prefixed constant > 0.
The sequences } and {0a} are positive and satisfies
02 7l
Σ Th oc,
Σ
0₂
lim
(21)
k=
k=U
for prefized positive constant 1 o0).
4- There erists non-negalive sequence {d2} such that limg- 5z = and de E DD)
The sequence of noises { is a uniformly bounded martingale difference sequence.
Here we make some comments on Assumption 3.15 Assumption 3.15 4)(5) are identical to Assumption 3.3 (3), respectively. Assumption 3.15 characterizes how the estimators (vx) are updated. As discussed in Barakat & Bianchi 2021 Xiao et al. 2023a). Assumption 3.15 1) is general enough to include the update
schemes for Adam, AdaBeliel, AMSGrad, and Yogi. Moreover, Assumption 3.15 () fixes the formulation of the mapping H, and Assumption 3.15 3) assumes that the stepsizes {nk} and momentum parameters {0x}
in the framework (AFMDW) are single-timescale.
We begin our analysis with the following lemmaa, which shows the uniform boundedness of {ma} and (gx} directly from the uniform boundedness of {œx} in Proposition 3.4 As a result, we omit its proof for simplicity.
Lemma 3.16. Suppose Assumption 3.1 and Assumption 3.15 hold. Then there erists a constant Ma > such that supx20 ||9+|| [|m=]] < Ma holds almost surely.
Next we present the following auxiliary lemma, which follows directly from the uniform boundedness of {#x), {ma} and {ge} in Lemma 3.16 together with the local boundedness of the mappings Dj and W.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Lemma 3.17. Suppose Assumption 3.1 and Assumption 3.15 hold. Then there erists constant Mw >> 0 such that su [|W mx+1)ll I Mw holds almost surely.
Let P+ = max 0), and (xx = {d E R2
||d|| & Mw). Consider the set-valued mapping
9: R. x R" _" I2" x I x I= defined by
(P+(v) + =) @ (m + σz) 7 1D
0(x,,,0)= =
(22)
T2V ,2,
and the following differential inclusion:
dæ dm du dt dt dt
(23)
In the following lemma, we prove that the set-valued mapping is capable of characterizing the update direction of {(Tk, mg, [&]} in the framework (AFMDW). The proof straightforwardly follows from Lemma 3.17 hence we omit it for simplicity.
Lemma 3.18. Suppose Assumption 3.1 and Assumption
3.15 hold. Then the inclusion
Uk+1 Ub (Vx U(a,, mx))
(24)
holds for any k 2 0. Furthermore, supezo ||u44a|| < 3o holds almost surely.
Let OP+ be the generalized Jacobian of the mapping P+, and define the function h: :R" x R
R"
R
Me, m, v) = f(x) + | +
m aw, (P+ + -) @ (m + ox)).
(25)
2T
The next Lemma 3.19 presents the formulation of the conservative field of h.
Lemma 3.19. Suppose Assumplion 3. and Assumplion admits
3.15 hold. Then h is a polential funclion that
D2(x) +z +s) - :mmm +oz) - (P+ +e) @ (m+or)
m,,
(26)
(m + ox)² o (P+ (P)
its conservative field.
Proposition 3.20. Suppose Assumption 3.1 and Assumption 3.15 hold. Then h is a Lyapunou function for the differential inclusion 23 with the stable set {(x, m, v) E R" x R² x IR" : :0 E (x)x), m + ox 0}.
In the next proposition, we show that the interpolated process of the sequence ( my, un) is a perturbed solution to the differential inclusion 23
Proposition 3.21. Suppose Assumplion 3.1 and Assumption 3. 2.15 hold, and limg Mr log(k) = O. Then
almost surely, the interpolated process of (Zks my, wx)} is perturbed solution for the differential inclusion 23
In the following theorem, we present. the convergence properties of the sequence {(x,, my vx)}, and prove that lim [[ Gxvll almost surely.
Theorem 3.22. Suppose Assumption 3.1 and Assumption 3.15 hold, and limg Ti log(k) ([)= 0. Then for the sequence { mR,vk). generated by the framework (FFMDW), almost surely, it holds that
any cluster point of the sequence {xu} -tttttiiinr point of g;
2. limp- [l + axxll== 00
9. the sequence of function values (g(xx)} converges.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Theorem 3.22
illustrates that limg-+20 || - ykll 0. Therefore, substituting the formulation of {yk in 13 into the update scheme of {m²} in the framework (AFMDW).
conclude that {ue} follows the same Together with the fact that lim 1 - y& = 0, based on the same proof techniques as
scheme as 14
in Lemma
3.8
WC can conclude that there exists a sequence of non non-negative random variables {Tk} such that
kk
holds almost surely, and
On (Dz (yx) + (k+1)-
uk+1 E yk
Then we have the following corollary showing that the interpolated process of the sequence {yk} is perturbed
solution of the differential inclusion [17 The proof of Corollary 3.23 is the same as Proposition 3.9 hence is omitted for simplicity.
Corollary 3.23. Suppose Assumption 3.1 and Assumplion 3.15
hold.
Then the interpolated path of the
sequence } is perturbed solution of the differentiol inclusion 17
4 Application: Adam with Decoupled Weight Decay
In this section, we propose a novel variant of Adam, which is named Adam with decoupled weight decay (AdamD). As application of our theoretical analysis in Section 3, we show the convergence properties of AdamD directly from the results in Theorem 3.10 and Theorem 3.22
Throughout this section, we focus on the settings where f in UOP takes the following finite-sum formula- tion:
1 N
f(x)
f((x).
(27)
Here we make the following assumptions on the functions {fe : [N]] in (27 Assumption 4.1. 1. For each [N], Js is desinable function that admils desinable sel-valued mapping D, as its conservative field.
SUPie[N), ZER" (x)]l 00.
3. is bounded from below.
As demonstrated in (Bolte & Pauwels, 2021 for any neural network that is built from definable blocks, the conservative field corresponds the AD algorithms is definable set-valued mapping. Hence, conclude 4.1 1) can be satisfied in a wide range of training tasks. Assumption 4.1 assumes the Lipschitz continuity of the function f, which is common in varions existing works (Barakat & Bianchi 2021
that Assumption
Guo et al. 2021 Shi et al. 2021 Zhang et al. 2022).
Moreover, [Bolte al., 2021 Corollary 4) illustrates that f a path-differentiable function and admits N ΣN Dr, as its consorvative field. Therefore, in the rest of this section, we choose the conservative field Dy as
N
Dy(x) conv
Σ P1,
(28)
N
The detailed AdamD mothod presented in Algorithm In our proposed AdamD method, the weight decay term aee decoupled from the update schemes for {mx) and {ua). In particular, the estimators {vx} are updated introduced in (Schaippl (2023) employs a proximal operator to handle the L2 regularizer term, derived from a first-order Taylor expansion with a variable metric. In practice, its updates yield approximately the same coefficients for 2& and m,; as our AdamD method. However, to the best of our knowledge, no convergence result is available for AdamP, even in the smooth case.
exponential moving average over {92} with parameter
(0, We note that AdamP
presented in Theorem 3.10 the following Based on the convergence properties of the framework (FMMDW) theorem establishes the convergence properties of Algorithm with non-diminishing {mk)
16

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm 1 Adam with decoupled weight decay (AdamD) for nonsmooth problem (UOP).
Algorithm 2 AdamW (Loshchllov & Hutter 2019
Require: Initial point zo R", mo E R", up E
Require: Initial point 30 E IR", mg RR vo R7, weight decay parameter > 0, safeguard parameter > O, stepsize S ;, and [ E (0,
R weight decay parameter o > 0, safeguard parameter ε 0, E (), and (0,1);
Set k= 0;
1: Set ==
2: while not. terminated do
2: while not terminated do
3: Independently sample in from [N], and com- pute 9k E Dr, (æx);
Independently sample in from [N], and com- pute gh Dr, (ax);
4: Update momentum:
Update momentum:
m++1 = (1 (1--0)) 0ym, + gki
m (1 02)ma + O2 gki
5:
Update estimator:
Update estimator:
Uk+ = -1 + 3
Uk+1 = (1 B)0k 33
6: Update:
6:
Update:
((VTkTT+ + 3 m + 03%
Fk+1 = xx
Th+11 = Ihk -1 (voK+i =)= (mx+1 + 024)
7: Set k + +
Set + +
8: end while return Tg--
end while return Zk.
Suppose Assumption 3.5 and Assumption 4.1 hold. Moreover, we assume that the momentum parameters {6g} is positive sequence that satisfies limg-+oo Ox log(k) = 0. Then almost surely, any cluster point of { in Algoritlmm is a Dy--saaiinary point of 9, and the sequence {g(øx)} converges.
Theorem 4.2.
In the following theorem, we establish the convergence properties for Algorithm when is equipped with timesalt stepsizes. The results in Theorem 4.3 are direct consequences Theorem 3.22 Hence, we omit its proof for simplicity.
Theorem 4.3. Suppose Assumplion 4.1 holds. Moreover, we assume that
1. The slepsizes 1) and replaced by mh and Br respectively in Algorithm 7
2. There exists constants T2 > 4r such that O2 TiTT and 8g. T27k hold for any k 2 0. Moreover, the sequence {m satisfies E km Uk oo and lim log(k) 0.
3. In Step 6 of Algorithm the sequence } follows the update scheme
Øb+1 = 23 Tı (1x+1 + (m++1 + ozR).
Then almost surely, any cluster point of {xx} in Algorithm 1 is a Dy-stationary point of g, and the sequence {g(øx)} converges.
5 Numerical Experiments
In this section, we conduct numerical experiments to demonstrate the effectiveness AdamD in the context of image classification and language modeling tasks. We compare AdamD with the most popular adaptive algorithms used for training neural networks, i.c. Adam and AdamW. All experiments are conducted using NVIDIA RTX 3090 Ti GPU and are implemented in Python 3.9 with PyTorh 1.12.0.
17

---

Published in Transactions on Machine Learning Research (04/2025)
5.1 Implementations of AdamD
In our numerical experiments, we focus on two key lasks: image classification employing Convolutional Neural Networks (CNNs) and language modeling using Long Short-Term Memory (LSTM) networks (Hochreiter & Schmidhuber 1997). Specifically, our image classification experiments include the deployment of well- established architectures, namely Resnet34 (He et al., 2016) and Densenet12/ 121 (Huang et al.) 2018), to train the CTFAR-10 and CIFAR-100 datasets (Krizhevsky et al. (2009). Our language modeling experiments focus on LSTM networks applied to the Penn Treebank dataset (Mareus et al.) 1993). It is worth noting that AdamW typically demonstrates superior generalization performance when used to train CNNs for image classification tasks. For training LSTMs, prior studies such as (Ding et al. 2023 Loshchilov & Hutter 2019:
Zhuang et al.
2020) have observed that Adam exhibits better generalization capacity than AdamW.
5.1.1 CNNs on image classification
In all our experiments on image classification, we train the models consistently for 200 epochs, employing a batch size of 128. At the 150th epoch, we reduce the step size by factor of 0.1. This step size reduction schedule is a prevalent practice in contemporary deep neural network training. It is helpful to accelerate the convergence of the optimization algorithm, and to enhance generalization capacity. Similar stralegies can be observed in previons works, such as (He et al. 2016 Zhuang et al. 2020). We adopt the following hyperparameter settings for the tested algorithms. For the weight decay parameter, we consider values in a E (5x 10-3 -30000330000 1003,030000 3x100-444.. 10- By fixing o first, we ensure that all methods solve the same minimization problem. With fixed, we then perform a grid search over the learning rate v for AdamD, Adam, and AdamW using 1) E (5 10-5, 10-4, 5x 10-4, 10-3,0.. 10-3, 10-2, xx 10-2, 20-2,10-1} Other parameters are set as follows:
Adam/AdamWi We set E = 10 02 = 10-1 and 8 = 10-3 as the default sctting in Pytorch.
AdamD: We set 0g = 0g
(l)g(++2)) with representing the epoch number. Within the s-th epoch, Og
lakes the constant value 0,. Under this setting, we can easily verify that 02 = o log Here, we set the initial momentum parameter to 0g- 10-1, the second moment parameter to B 10-3 and the regularization parameter to e = 10-8, which are the same as the default settings in Py Porch for Adam/Adamik
In Step 6 of Algorithm 1 the coefficient associated with 2& is given by 1--o (Vi++1 + +c) Note that as training progresses, Vu++1 + E lends to decrease, which can make the coellicient excessively small when the weight decay a is relatively large. In our experiments, larger weight decay parameters (exceeding 10-5) enhance gencralization in image classification, whereas smaller weight decay paramcters (not excceding 10-5) yield better performance for LSTM training, as discussed a later subsection. This trend is also observed in (Luo et al. 2019 Zhuang et al. 2020 Li et al.] (022).. To enhance AdamD's performance image classification, where larger weight decay typically yields better generalization, we adopt a clipping strategy for the preconditioner similar to that used in Adabound (Luo et al., 2019). Specifically, we define ,,, cu), thereby balancing the coefficient, now expressed as 1 - 11ōg- Here, the clipping function is defined as Clip(z, q,, cu) == max(min c.), and we choose (ct,cu) c.) = (10-2, 10²)..
ōe == Clip (o(vi++ + e)-1, 44
Figure I presents numerical results obtained with a fixed weight decay parameter O = 5 x 10-3 and the best-tuned learning rate 7) that achieves the highest test accuracy, ensuring that all methods minīmize the same objective function. Additional experimental results and further details are provided in Appendix B As shown in the first row, all methods can achieve 100% training accuracy across four tasks. In the second row, AdamD consistently attains the highest test accuracy compared with Adam and AdamW, while Adam's 's generalization performance is inferior. This may be due to AdamD's asymptotic approximation to SGD with weight decay, which has been shown to perform better in image classification (Luo et al. 2019 Zhuang et al. 2020). These findings underscore the importance of weight decoupling when solving the quadratically regularized problem defined in (UOP).
we also present a plot of || + ommll as shown in Figure 2 When 6 adheres to decay schedule described by O (k-7), (32 and basic calculus imply that || + mel exhibits
To verify the results in Lemma 3.7
18

---

Published in Transactions on Machine Learning Research (04/2025)
accuracy Training epoch
Train accuracy Training epoch
Train accuracy Training epoch
accuracy Training epoch
Accuracy
Accurscy
Accuracy
Alan
Adain
Training Epoch
Training Epoch
Training Epoch
Training Epoch
Test accuracy Training epoch
Test accurscy Training epoch
Test accuracy Training epoch
Test accuracy Training epoch
Accuracy
Accurasy
Accurary
Alavo
Gere Adxem
AdamWW
Training
Training
Training
Train
Training epoch
Train loss
Training apach
Train loss
Training epoch
Training opech
Adamo
Ada10
Adand
Alam
Man
LO55
Lost
Training Epoch
Training Epoch
Training Epoch
Training Epoch
Test loss Traniing epoch
Test loss Training apach
Test loss Training epoch
Test loss
Taining opoch
Adam/
Adari
Adare!
Adxmm
LOSs
LOIS
Trainina
Training Epoch
Training Epoch
(a) ResNet34 on CIFAR10
(b) DenseNet on CIFARIO
(c) ResNet34 on CIFAR100
(d) DenseNet CIFAR100
Figure 1: ResNet34 and DenseNet121 on CIFAR10 and CIFAR100 datasets. Stepsize is reduced to 0.1 times of the original value at the 150th epoch.
Train residual
Training epoch
Train residual
Training epach
10²
12'
1
1//K
1:/
1.k
1.k
1oxx mm]
30-1
10'1
31
10300 2000 30006 48300 soice 6000 7900 coie
1008 3000 3008 420c 5000 5009 r00e 1909 Training Iterationss
Training Iterations
(a) ResNet34 on CIFAR10
(b) ResNet34 on CIFAR100
Figure 2: [mm + axxll under different decay rates of (0c). The stepsizes for updating {ax} are fixed.
an asymptotic behavior of (k->). The results Figure are consistent with our thcoretical analysis that fllmæ + ozxll ll} converges to 0, or equivalently flla Mell converges Notably, larger values of y correspond to a more rapid decline in ||ar + mm]l-
19

---

Published in Transactions on Machine Learning Research (04/2025)
5.1.2 LSTMs on language modeling
In all our language modeling experiments, we train our models for 200 epochs using a batch size of 128. We employ step size reduction strategy that decreases the learning rate to 0.1 times its previous value twice during training, specifically the 75th and 150th epochs. These settings follow the commonly used experimental setup for training LSTMs, as demonstrated in previous works (Chen et al. 2021 Zhuang et al. 2020). This strategy accelerates convergence and enhances the generalization capacity of the optimization algorithm. We adopt the following hyperparameter scttings for the tested algorithms. For the weight decay parameter, we consider values in a E {10-*, 110-5,10=$, 100-). By first fixing 0, we ensure that all methods minimize the same objective function. With a fixed, we perform a grid search over the learning rate n for AdamD, Adam, and AdamW using values in 1 E {10-1,5 x 10-2, 10-2,5 5 x 10-3, 1010,035 x 10-4). In the training of LSTMs, we do not employ the clipping strategy since smaller weight decay parameters already yield good generalization performance. All other hyperparameter settings are identical to those used in Section 5.1.1 Figure 3 displays the numerical results obtained with fixed o 10-5, ensuring that all methods minimize the same objective function.
As shown in Figure 3 both AdamD) and Adam outperform AdamW terms of generalization. For 1-layer LSTMs, AdamD and Adam achieve comparable performance, whereas for deeper 2-layer and 3-layer LSTM models, AdamD reduces the test perplexity by at least. 0.6 units compared to Adam. We summarize the tuned stepsizes in the experiments in Figure and 3 in Table 3 Further experimental details can be found
in Appendix B
: 5e-03
a : 1e-05
Task 1
Task 2
Task 3
Task
Task
Task
Task 7
AdamD
5e-03
5c-03
5e-03
5e-03
5e-03
5e-03
5e-03
Adam
1e-04
1e-04
le-04
1e-04
5e-03
5e-03
5e-03
AdamW
le-02
5e-04
5e-04
5e-04
1e-03
1e-03
le-03
Table 3: Tuned stepsizes for varing tasks. Task 1: ResNet31 on CIFAR10; Task 2: DenseNet121 on CIFAR10;
Task 3: ResNet34 on CIFAR100;; Task 4: DenseNet121 CIFAR100; Task 5: 1-layer LSTM; Task 6: 2-layer LSTM; Task 7: 3-layer LSTM.
5.1.3 Performance with different choices of {0s} and {Bx}
Finally, we investigate the performance the AdamD method with different choices of its step sizes {0&} and {8&] for the momentum terms {m²} } and {ux ), respectively. In our numerical experiments, both sequences are fixed as constants, i.e., 02 and the step size 17 is tuned among {10 2 10-3 10 10-4) to maximize the proportion of training loss below 0.6. The regularization parameter for the second moment is 1 following the default settings in PyTorch for Adam.
[0, and 84 3 [0, 1]. The weight decay parameter is set to 5 10-4
Figure 4 shows that the light-colored regions (indicating lower training loss) for AdamW and AdamD are larger than that for Adam. As demonstrated in (Zhang et al. 2022). Adam can fail when is small and / is large, as indicated by the red area the bottom right portion of Figure 4la). In contrast, the corresponding region for AdamD in Figure c) remains light. These observations suggest that incorporating weight decay in AdamD enhances its robustness to the choices of 9 and / compared to Adam with coupled weight decay, Furthermore, the results support the theoretical findings in Theorem 3.13 which imply that if the momentum parameters 0g are kept sufficiently small, the sequence 2y remains stable. This stability is maintained with relaxed requirements on the updates of the estimator Uk, provided that the preconditioner ((x)) is bounded both below and above. An additional observation that, while AdamD and AdamW exhibit similar overall robustness, the favorable regions for 0 and 3 differ between the two methods. Specifically, compared with AdamW, AdamD is more robust than AdamW when small, but less robust when 0 is large, as shown in Figures (b) and (c).
20

---

Published in Transactions on Machine Learning Research (04/2025)
Train set perplexity training epoch
Train set perplexity training epoch
Train set perplexity training epach
Perplexity
Perplexity
Perplexity
201
020
130
Adam
Adam AdamW
Adam AdamW
Adami!
Training Epoch
Training Epoch
Training Epoch
Test set perplexity training epoch
Test set perplexity training epoch
Test set perplexity training epoch
130
202
Perplexity
Perplexity
Perplexity
AdamD
AdamD
AdamD AdamW
Adami
2
303
us
150
20
Us
125
100
200
303
335
2
Training Epoch
Training Epoch
Training Epoch
(a) L-layer LSTM
(b) 2-layer LSTM
3-layer LSTM
Figure 3: Training and test perplexity (lower is better) LSTMs on Penn Treebank dataset with stepsize
reduced to 0.1 times of the original value at the 75th epoch and 150th epoch.
train loss
0/1
6
1-
(a) Adam
(b) AdamW
(c) AdamD
Figure 4: Performance comparison of Adam, AdamW, and AdamD for training a three-layer CNN the MNIST dataset. Here, the parameters {6x} and {Bx} are set to fixed constants f and B, respectively, The proportion of values below 0.55 is 15.50% for Adam, and 80.25% for both AdamW and AdamD.
5.2 Further discussions on the AdamD
5.2.1
Asymptotic approximation to SGD sequence helps generalization
the term ||ar, + mm&ll converges to 0 as tends to infinity. Then as discussed in Lemma 3.8 the sequence {yk (defined by y/ := - -mm) approximatcly follows the update scheme (14] which asymplotically approximates SGD method. Together with the fact that limg- || yyk = 0, we can conclude that the sequence (xx) in AdamD is controlled by an SGD sequence {y as goes to infinity. Morcover, the interpolated process of {yk} is perturbed solution of the differential inclusion 11 i.c.,
As demonstrated in Lemma 3.7
dy ( dt
(29)
21

---

Published in Transactions on Machine Learning Research (04/2025)
On the other hand, in the early stage of the iterations of AdamD, the term |azz m&Il is large, and the ratio of Og and Th usually remains nearly unchanged. Then as illustrated in the discussion in Section 4, the sequence {TT,, my, o4)} jointly tracks the trajectories of the differential inclusion
E (P+(t) + ()) (m + tax) 7im ( T2 - (z)(x))
dr dm de dt
(30)
dt dt
Here U(z)= == 2 Di=nddo (d : d E (z)), Similar results are also exhibited in (Bianchi et al. 2022.
Xiao
ct al. 2023a As the differential inclusion (30) imposcs proconditioners to the update directions of t based on the second-order moments of the stochastic subgradients, the sequence could quickly converge to a neighborhood of the stationary points.
These thcoretical properties explain the fast convergence of AdamD in the carly stage of the training and its lower generalization error than Adam with coupled weight decay. Based on the numerical experiments and our theoretical analysis, we believe the ability to asymptotically track SGD sequence in AdamD helps to explain its superior generalization performance over Adam.
5.2.2 Decoupled weight decay is equivalent to quadratic regularization
It conjectured in (Loshchilov & Hutter 2019) that the quadratic regularization term contributes to the low generalization error training neural networks. Moreover, the authors in (Loshchilov & Hutter 20199) develop AdamW, showing that weight decay is not equivalent to quadratic regularization. As result, the term drk in AdamW is not scaled by the preconditioner (vTK+I + e)- Therefore, AdamW does not have clear objective function and lacks convergence guarantees in training nonsmooth neural networks.
In our AdamD method, the objective function is exactly the g(z) in (UOP). Hence the weight decay parameter is exactly the penalty parameter for the quadratic penalty term & ||||| in (UOP). More importantly, we provide theoretical guarantees for AdamD in training nonsmooth neural networks. The stationarity of the iterates } is characterized by D (z4) tork, hence has clearer meaning when compared with AdamW.
Furthermore, our numerical experiments demonstrate the superior performance of AdamD, illustrating that employing the quadratic regularization term in (UOP) does not undermine the generalization error. Based these results, we can conclude that within our framework (AFMDW), the weight decay can be interpreted as the quadratic regularization, which is different from the demonstrations in (Loshchilov & Hutter 2019 regarding AdamW.
6 Conclusion
In this paper, motivated by AdamW, propose novel framework [AFMDW] for Adam-family methods with decoupled weight decay. We prove that under mild assumptions with non-diminishing stepsizes {me H
and diminishing momentum parameters [0z], any cluster point of { is a DD-sttatonary point of (UOP).
When { is also non-diminishing, the sequence {x*} eventually stabilizes around the critical points of
the Dy-stationary point of (UOP). Moreover, when employing a single-timescale scheme, any cluster point of (ax] a Dy-stationary point of (UOP). Compared with AdamW, our proposed framework (AFMDW) enjoys convergence guarantocs training nonsmooth neural networks and yields solutions that have clearer meanings. More importantly, we prove that the decoupled weight decay grants more flexibility of the choices of the parameters {0x} and { in (AFMDW) than Adam. This fact theoretically illustrates the advantages of the employment of the decoupled weight decay.
As an application of our proposed framework (AFMDW), 1 we develop a novel Adam-family method named Adam with decoupled weight decay (AdamD), and prove its convergence properties under mild conditions. Numerical experiments on image classification and language modeling demonstrate the effectiveness of our proposed method. To conclude, we believe that our work has enriched the thcoretical understanding of weight decay and explained its practical utility in the field of deep learning applications.
22

---

Published in Transactions on Machine Learning Research (04/2025)
It is worth mentioning that the formulation of (AFMDW) is slightly different from the widely employed AdamW (Loshchilov & Hutter 2019). However, it is still challenging to find Lyapunov function for AdamW to establish its convergence guarantees under general nonsmooth nonconvex settings. We believe that exploring the convergence properties of AdamW in nonsmooth nonconvex optimization is an interesting topic for future research.
Acknowledgments
The rescarch of Kim-Chuan Toh is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 grant (MOE- T00-02222022)--022))
References
Anas Barakat and Pascal Bianchi. Convergence and dynamical behavior of the ADAM algorithm for non- convex stochastic optimization. SIAM Journal on Optimization, 31(1):244-274, 2021.
Michel Benaïm. Dynamics of stochastic approximation algorithms. Seminaire de probabilites XXXIII, pp. 1 68. Springer, 2006.
Michel Benaïm, Josef Hofbauer, and Sylvain Sorin. Stochastic approximations and differential inclusions.
SIAM Journal on Control and Oplimizzation, 44(1):328 348, 2005.
Pascal Bianchi and Rodolfo Zersuuer A closed-measure approach to stochastic approximation. arxiv preprint ar Xrx:::2112.0544 2021.
Pascal Bianchi, Walid Hachem, and Sholom Schechtman. Convergence of constant step stochastic gradient descent for non-smooth non-convex functions. Set- Valued and Variational Analysis, pp. 1-31, 2022.
Jérôme Bolte and Edouard Pauwels. A mathematical model for automatic differentiation in machine learning. Advances in Newral Information Processing Systems, 33:10809-10811 10819, 2020.
Jérôme Bolte and Edouard Pauwels. Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming, 188(1):19 51, 2021.
Jérôme Bolte, Aris Daniilidis, Adrian Lewis, and Masahiro Shiota. Clarke subgradients of stratifiable func- lions. STAM Journal on Oplimization, 18(2):556 572, 2007.
Jérôme Bolte, Tam Le, Edouard Pauwels, and Tony Silveti-Falls. Nonsmooth implicit iationnntiation for
machine-learning and optimization. Advances in Neural Information Processing Systems, 34, 2021.
Jérôme Bolte, Tam Le, and Edouard Pauwels. Subgradient sampling for nonsmooth nonconvex minimization. arXiv preprint 2r0: 19744, 2022a.
Jérôme Bolte, Edouard Pauwels, and Antonio José Silveti-Falls. Differentiating nonsmooth solutions to parametric monotone inclusion problems. arXie preprint arXiv:2212.0788 2022b.
Vivek S Borkar. Stochastic approzimation: dynamical systems vieupoint, volume 48. Springer, 2009.
Siegfried Bos and E Chug. Using weight docay to optimize the generalization ability of a perceptron. In Proceedings of International Conference on Neural Networks (ICNN'96) volume 1, pp. 241 246. IEEE, 1996.
Camille Castera, Jérôme Bolte, Cédric Févotte, and Edouard Pauwels. An inertial Newton algorithm for deep learning. The Journal of Machine Learning Research, (2(1):5977-6007 2021.
Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu. Closing the gen- cralization gap of adaptive gradient methods in training deep neural networks. In Proceedings of the T Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3267-322 32775, 2021.
23

---

Published in Transactions on Machine Learning Research (04/2025)
Frank H. Clarke. Optimization and nonsmooth analysis, volume 5. SIAM, 1990.
Aris Daniilidis and Dmitriy Drusvyatskiy. Pathological subgradient dynamics. SIAM Journal on Optimiza- lion, 30(2):1327 1338, 2020.
Damek Davis, Dmitriy Drusvyatskiy, Sham Kakade, and Jason D Lec. Stochastic subgradient method converges on lame functions. Foundalions of Compulational Malhemalics, 20(1):119 1et1119-14,, 2020.
Alexandre Défossez, Leon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of adam and adagrad. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL Intps:// (penrevies..et/foourrf wwt.:eeoorrttdd-222211000.
Kuangyu Ding and Kim-Chuan Toh. Stochastic bregman subgradient methods for nonsmooth nonconvex optimization problems. arXio preprint orXiv:2404. 17986, 2024.
Kuangyu Ding, Jingyang Li, and Kim-Chuan Toh. Nonconvex stochastic Bregman proximal gradient method with application to deep learning. orXiv preprint Xrxi::2306. 14522, 2023.
John C Duchi and Feng Ruan. Stochastic methods for composite and weakly convex optimization problems. SIAM Journal on Optimization, 28( (8(4 (1::3222-3222 3259, 2018.
Zhishuai Guo, Yi Xu, Wotao Yin, Rong Jin, and Tianbao Yang. A novel convergence analysis for algorithms of the Adam family. NewrIPS OPT Workshop, 2021.
Mert Gürbüzbalaban, Andrzej Ruszozyíski, and Landi Zhu. A stochastic subgradient method for distribu- tionally robust non-convex and non-smooth learning. Journal of Optimization Theory and Applications, 194(3):1014 1041, 2022.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735 1780, 1997.
Xiaoyin Hu, Nachuan Xiao, Xin Liu, and Kim-Chuan Toh. A constraint dissolving approach for nonsmooth optimization over the Stiefel manifold. arXiv preprint arXiv:2205. 10500, 2022a.
Xiaoyin Hu, Nachuan Xiao, Xin Liu, and Kin-Chuan Toh. An improved unconstraained approach for bilevel optimization, arviv preprint XrXix:2208. 00732, 2022b.
Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Q Weinberger. Condensenct: An efficient densenet using learned group convolutions. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 2752-2761, 2018.
Ruinan Jin, Xiao Li, Yaoliang Yu, and Bacxiang Wang. A comprehensive framework for analyzing the convergence of adam: Bridging the gap with sgd. arXie preprint arXiv:2410.0444 04458, 2024.
Cédric Josz, Lexiao Lai, and Xiaopeng Li. Convergence the momentur method for semialgebraic functions with locally lipschitz gradients. SIAM Journal on Optimization, 33(4):3012-33 10372 2023.
Diederik P Kingmaa and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
Tam Le. Nonsmooth nonconvex stochastic heavy ball. orXiv preprint arXio:2304.13225 2023.
Jingyang Li, Pan Zhou, Kuangyu Ding, Kim-Chuan Toh, and Yinyu Ye. Cemension-reducee adaptive gradient method. 2022.
24

---

Published in Transactions on Machine Learning Research (04/2025)
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arxiv preprint orxi 03263, 2019.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Represeniations, 2019.
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint 1xXi::1022.00 09843, 2019.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of english: The penn trecbank. 1993.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In 6th International Conference on Learning Representations (ICLR), 2018.
Andrzej Ruszczyúski.. Convergence of a stochastic subgradient method with averaging for nonsmooth non- convex constrained optimization. Optimization Letters, 14(7):1615-1622, 2020.
Andrzej Ruszezynski. stochastic subgradient method for nonsmooth nonconvex multilevel composition optimization. STAM M Journal on Control and Oplimization, 59(3):2301 22020, 2021.
Fabian Schaipp. Decay no more. In ICLR Blogposts 2023, 2023. URL https//ickl-blloopatte gittub. io/203/blog/2223/ddm
blog/ 2023 /22/////////.....
Naichen Shi, Dawei Li, Mingyi Hong, and Ruoyu Sun. Risprrp converges with proper hyperparameter, In International Conference on Learning Representation, 2021.
Lou Van den Dries and Chris Miller. Geometric categories and o-minimal structures. Duke Mathematical Journal, (2))44997- 1996.
Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Zhi-Ming Ma, Tie- Yan Liu, and Wei Chen. Provable adaptivity in adam. arXiv preprint arXiv:2208. 09900, 2022.
Nachuan Xiao, Xiaoyin Hu, Xin Liu, and Kim-Chuan Toh. Adam-family methods for nonsmooth optimization with convergence guarantees. arXie preprint or (tl:2305003934 2023a.
Nachuan Xiao, Xiaoyin Hu, and Kim-Chuan Toh. Convergence guarantees for stochastic subgradient methods in nonsmooth nonconvex optimization. arXiv preprint arXio:2307.1055. 2023b.
Nachuan Xiao, Xin Liu, and Kim-Chual Toh. Dissolving constraints for Riemannian optimization. Mothe- matics of Operations Research, 2023c.
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. Advances in Neural Information Processing Systems, 31, 2018.
Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Tuo. Adam can converge without any modification update rules. Advances in Neural Information Processing Systems, 35:28386-28399 2022.
Pan Zhou, Xingyu Xie, Zhouchen Lin, and Shuicheng Yan. Towards understanding convergence and gener- alization of AdamW. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 2024.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabclicf optimizer: Adapting stepsizes by the belief in obscrved gradients. Advances in neural information processing systems, 33:18795-18800 2020.
Zhenxun Zhuang, Mingrui Liu, Ashok Cutkosky, and Francesco Orabona. Understanding adamw through proximal methods and scale-freeness. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL httpps://operrevier net//Forumiid--KHPPWGaa Expert Certification.
Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. A sufficient condition for convergences Adam and RMSProp. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11127-11135, 2019.
25

---

Published in Transactions on Machine Learning Research (04/2025)
A Proofs
Proof of Proposition 3-4 As illustrated in Assumption 3.3 dg + (xx) and {&x} is uniformly bounded.
Then it is casy to verify that there exists constant L such that || = I|dæ + Ex+all S L(1 + || (rxl"") holds for any & 2 0.
Let the constant Q be defined as
2MpL EDO
F
My |mall EgO
Q 2 max
| + 1
(31)
In the following, for any sequence } generated from (AFMDW). we aim to prove that the set {/ 2 0
|| 2 Q} is an empty set by contradiction. Therefore, assume that the set (k 2 0 ||
2 QQ is non-
empty and set r= (k 2 0: || 2 Q} 1. Then from the definition of T, we have || N >> ||
On the other hand, from the update scheme (FFMDW) for any < T, we have
|m++i|| S max mo: sup osisk+1
| < max(| L( + Q*) s
06e Q M
where the last incquality follows from the definition of Q and the fact that
ge.
2M,L
Q" S σee
Q1-"Q""
= 065
Q.
M. occ
M
Mp
Then it holds that
|x++1 || = [1(1
2 ( 3 m
< (1 m)))) ||>:|| + 7/. Mr || || < MH0E QQ + M.
Q Q.
M,
But |xr++||| < Q contradicts to the definition of r. Thus, the set (k 2 0 ||لا||| > Q) is empty. Therefore.
we have that supk2o ||=||||| < Q holds almost surely. This completes the proof.
Proof of Lemma 3. From Assumption 3.5 there exists constant ý E (0, 1) such that max{|1-7aoM4 medeol> I - holds for any & 2 0. Then from the update scheme of {øx} in the framework (AFMDW) almost surely, it holds that
loob+1 + m++1ll
[[(1 - H @ (ar, + m²) + 0₂(1 - m @ (gh mz)ll
max M, mm (1 + -m + 0s 11ga - mxll)
(32)
< (1 -i) |aox mnIl 2M,0 s (1 M1 k+1 | + moll + 2M4
(1
h-i0.
1=0
(1 (1--n)++1g ( Ma) + 2M, Σ (1 i1) k-i 0 =: 8
i=0
Since lim 0g 0, we have lim (1- k-00 0. Thus we get limp 8, 0, and | m& s å;
k0+00
holds for any & > This completes the proof.
Proof of Lemma 3.8
As illustrated in Assumption 3.3 3.3(2), 2), there exists åt E Bs, (xx) and de (z(
such that dd de
< &g and limg-soo & 0. Combining with equation
32
it holds that Ilve Fx S
26

---

Published in Transactions on Machine Learning Research (04/2025)
llyx xxll + 12 Txll S de + 6g. As a result,
dist (dy oy,, (āx))
| + oyk - (dx oix))l
x du - dz +o live - Tull S d2 + o 8 + +6x.
Since Fx 15; (yk) and dist(d+ + Ty (ū())) - 6% we get 15
Proof of Proposition 3.9 Based on Lemma 2.21 by verifying its conditions, we can prove that the interpolated process of {Me) is a perturbed solution for the differential inclusion (17).
Condition (1) of Lemma 2.21 directly follows from Assumption 3.3 3) and Proposition 2.3 by choosing the
stepsizes in (8 as { (0) Morcover, Lemma 3.8 guarantees the validity the condition (2) in Lemma 2.21 by
noting that limg==oo 0z = 0. Furthermore, condition (3) of Lemma 2.21 follows from Assumption 3.3 2) and
Lemma 3.6 As result, directly from Lemma [.21 we can conclude that almost surely, the interpolated
process of {1 is a perturbed trajectory of the differential inclusion [7
Proof of Theorem 3.10 From Lemma 3.2 and Proposition 2.18 we can conclude that g a Lyapunov
function for the differential inclusion
17 with the stable set {a R" :0 (z(x). Moreover, Proposition
(399 illustrates that almost surely, the interpolated process of the sequence {yk } in (13) is perturbed
solution of the differential inclusion
17 As a result, it follows from Theorem 2.22 that any cluster point of
} lies in the set (i E IR" 0 E Dg( and the sequence {g(m)) converges.
Since lim 0g = 0, Lemmaa 3.7 implies that limg-+00 | - yull = 0 holds almost surely. Then from the continuity of g and the convergence properties of {yk}, we can conclude that any cluster point of {æ*} lies in the set. (x IR" E (x)) and the sequence {9(xx)} converges. This completes the proof.
Proof of Lemma 3.12 From Assumption 3.11 holds for all s 2 0 and any satisfying s SiiA(((s)) that
N- 1-1
N-
N-
N-L
O GKEN+1
5
OKER+I
+
Σ
OKEK-11
+
OKEK+1
K=N-T&1
A=N-L+
(33)
N
OgEx+1
+
OKEK+I
K=N-L L
Therefore, for any any e > O, choose 0g
MN guarantees that
lim sup +0 ((ISA(A()++T)
sup
Σ BREN+I k=8
S M
limsup 000 (SIAA((+)+T))
0,5c.
This completes the proof.
Proof of Theorem 3.13. For the update scheme illustrate that for any & > 0, there exists f.. 0 T such that whenever lim sup Ths S î) lim limsup 0g < è and (Ex) is (T, (0+))-comtttll controlled, we have
14
Lemma 3.12 and (iaoo et al. 2023b Theorem 3.7)
lim sup dist (yk, f RR :0 (,(x))) < k-0000
(34)
Then by equation
332 in Lemma 3.7 we have that there exists O such that whenever lim sup 0g S 62. Therefore, whenever lim sup 0x S min 02), we have that
lim sups-soo ||
< 5
lim sup dist (Ir {ar R. e (g(r))) < lim
limsup sup dist (un, for RR" 0 Dy(x))) + E Se. 1-+000
(35)
k-0000
27

---

Published in Transactions on Machine Learning Research (04/2025)
This completes the proof.
Proof of Lemma 3.19 Notice that f is potential function that admits Dj its conservative field, and the function (z, m, v) + + oz, toz (P+(v) + + @ (m + +ox) is semi-algebraic and thus definable. Then by the chain rule for conservative field (Bolte & Pauwels, 2021). we can conclude that h is a potential function that admits Dr as its conservative field. Moreover, as Dj and öP+ are convex-valued over IR", it holds that Dr. is convex-valued over I x R x R". This completes the proof.
Proof of Proposition 3.20 For any trajectory the differential inclusion 23 there exists 4 R, + R" and lu : R.+ + R such that (s) E (x((s)) and (1()) (x((s) m(s)) for almost every s 2 0, and
((P+(u(s) + + (m((s ozx(s))
(s)
- mm) + (s)
(36)
-T2P+(((s)) (s)
Then from the formulation of h, we have
(Da(((s), (()) (()), (ss),
(P+(v(s) + ( -m))))))))))))))))))))))))))-- + (m(s) + +ox(s))
( - +1y(s))
22 (Im(s) (P (m(s)) lu(s))
1n
$ (P+(n(s))+ 0 (m(s) (3)) +ox(s))
(m(s) (
22
((()))
4n
s (P+( + (m
ozx(s))
12
(p-()()) +)) (m)(s) oz) m(s) + ox(s))
4m
S - _ (P+ + =) の (m(s)) m(s) oz(s))
71
Here the first inequality follows from the fact that (s) > 0 and (p+ @r= P+)). The third inequality follows from the fact that 1 22 2 0 in Assumption 3.. Therefore, can conclude that for any initial point m(0), v()) E R x R x R it holds for any 2 0 that h(x(t), m(t), u(t)) - h(=(0), m0)) v())
min 1'" IEDM (elo) (*.(s. m(s), ((s)) ds
(37)
< 7. - (p(s) +e) (m(s) + oz(()), m(s) + az(s)) ds.
-
As a result, we can conclude that for any trajectory of the differential inclusion 23 that m()) < h(r(0), m0), v00). Now consider the case when (z(0), m()), x(0) & ((x, u,)) E Ra R x R O E Dy(r), m ox 0).
it holds for any > 0
Suppose there exists some T > 0 such that
h(x(T), mT)) x(T) = (x((0)))))))))))))(())))))) (0), v(0)).
(38)
Then 37 implics that ms)) + (x(ss) = 0 holds for almost every & E T). Therefore, ri((s) + 0i((s) = 0 and implies that i(s) = hold for almost every 8 E [o, TJ. As a result, we have
23
+ 0 mss) E (s) + 7. Dy - (z)ss()(s)
28

---

Published in Transactions on Machine Learning Research (04/2025)
holds for almost every 8 [0. Together with the fact that (e(t), a(t)) is absolutely continuous and Dy is graph-closed and locally bounded, we have that
m(0) ax((0) 0. 0 (e + ax(() = D)(x(()).
But the above contradicts the condition that (z(0), m() (0)) & {(x, m. result, can conclude that for any T> 0, whenever (z(00, m()), e()) & ((x,,,)) (y(x), -), it holds that
u) m...m,)). : m- 0}. As a
h(r(T) o(T) h(z(0), v()).
This completes the proof.
Proof of Proposition 3.21 From the uniform boundedness of {m²), {ux} and {ox} in Lemma 3.16 and
Lemma 3.17 and Assumption 3.15(4). 4), can conclude that limg-soo || -mall + || Therefore, there exists sequence of random variables {Tx} such that almost surely, limg-+oo 0 holds and [| m& + ||06+1 - vzll |l S 7k.
0.
Then from the formulation of the framework inclusion
AFMDW), the sequence {(Ik, mg, 2R)} satisfies the following
(TK+1, (6+11, E (Ir, my, vx) g (IE, may un) - (0, -TI Ex++
Then directly follows from Assumption 3.15 4) and Proposition 2.3 that
lim 50000
sup <iSA(A.+T)
(0, Ti 1 0) =.
Therefore, can conclude that the conditions (1) and (2) in Lemma 2.21 hold. Moreover, condition (3) in Lemma 2.21 directly follows from Assumption 3.15] 1), Lemma 3.16 and Lemma 3.17 Therefore, from
Lemma 2.21 we can conclude that the interpolated process of {(ZK, my v.) is perturbed solution for the This completes the proof.
differential inclusion
23
Proof of Theorem 3.22 From Proposition 3.21 we can conclude that the interpolated process of (ar,,, my, {&)} is a perturbed solution for the differential inclusion (23 Moreover, Proposition 3. illustrates that is a Lyapunov function for the differential inclusion (23) with stable set {(2, m, v) E R" x R" x R"
= 0}. Then we conclude that any cluster point of ((r,, ma, ux)} lies in the set [(x, m, v) E R x R x _ 0 (x)x), + oz 0}, and the sequence [h(Ik, mp, vk)} converges.
E (,(x),
Thus, we can conclude that any cluster point of { lies in the set {ar R : 0 E Dz(x)), and any cluster point of ((Tx, me)} lies in {(x, m) E Ih" x R ax ax++m m = 0}. As a result, noting that far + mm is bounded in R", it holds that lim |ook + m = O. Furthermore, since
lim [1 ma, mx) vk) - g(xx)l lim t00
look +mxll O,
k 2nive
we can deduce that the sequence {g(#x)} converges. This completes the proof.
The definability of fs and D,, implics the definability of and D,, hence from (Bolte & Pauwels 2021 hhorem 5), f is path-differentiable and ensures the validity of
Proof of Theorem 4.2 We first verify the validity of Assumption
3.
the set {f(x) 0 Dz())} is finite subset of R. Additionally, Assumption 4.1
Assumption 3.1 2). This verifies the validity of Assumption 3.1
Moreover, let {Fe} be a sequence of o-algebras generated by {xy, dj. : j S k), de Efg and {x+1 = gk dR. Then we can conclude that de e (x4) and E[5++1 [x] = 0. Moreover, Assumption 1.1 (2) illustrates that there exists a constant M, such that supse(N), =ER= [D) I My. Thus we can conclude that sup#2o II S My and supxzo [|dx|| $ M, hold almost surely. Then sup420 ||E++1l s 2M, holds almost surely. This verifies the validity of Assumption 3.3 3).
29

---

Published in Transactions on Machine Learning Research (04/2025)
Furthermore, from the update scheme in Step 5 of Algorithm we conclude that sup220 l supk20 |ل - Mf. This illustrates that Assumption 3.3(1) holds with Ev My+E and My = Therefore, from Theorem 3.10 we can conclude that any cluster point of the sequence { is a Py-stationary point of 9, and the sequence {g(x))} converges. This completes the proof.
B Experiments details
In this section, provide additional experimental results to further illustrate the performance and robust- ness of our methods. The following tables report the outcomes various tasks under different hyperparam- eter sottings.
B.1 Image Classification:
Tables present the performance of ResNet34 and DenseNet121 on CIFAR10 and CIFAR100 datasets. these experiments, we vary the learning rate (LR) and weight decay parameter for the optimizers AdamD, Adam, and AdamW. The best performance for each fixed weight decay parameter highlighted in bold.
a: 5e-03
a:
5e-04
1c-04
AdamD
Adam
AdamW
AdamD
Adam
Adama
Adam
Adam
AdamW
AdamD
Adam
Adamm
1c-01
77.05
17.84
92.01
91.23
59.13
93.39
93.67
62.71
93.19
94.41
65.09
92.22
5e-02
85.03
55.27
93.28
93.68
59.33
94.31
95.22
63.57
93.93
94.96
76.80
93.72
1e-02
94.08
62.33
94.54
95.10
79.67
94.16
94.54
81.43
94.22
94.28
88.96
94.09
5c-03
95.13
73.20
94.28
94.67
84.46
94.19
94.53
86.81
94.33
93.69
91.47
94.27
1e-03
94.57
89.81
94.40
93.05
92.48
94.34
92.54
93.31
94.18
92.12
94.06
94.56
5e-04
93.42
92.25
94.19
91.60
93.46
94.40
91.24
93.83
94.26
90.67
94.29
94.39
le-04
87.64
94.09
94.09
87.22
94.04
93.86
87.47
94.10
94.58
86.62
93.89
93.71
5e-05
85,91
93.63
93.11
85.90
93.64
92.65
84.68
93.23
92.86
85.03
92.78
92.96
Table 4: ResNet34 CIFAR10.
d: 5e-03
d: 1e-03
d:: 5e-04
d: 1e-04
7
Adamm
Adam
AdamW
Adaml)
Adam
AdamW
AdamD
Adam
Adamm
Aclam
Adam
AdamW
Te-01
68.39
46.88
89.47
90.13
59.13
93.55
93.80
59.58
93.26
94.52
67. 28
93.24
5e-02
81.32
51.89
93.21
94.02
93.96
95.47
65.89
93.72
94.73
76.94
93.1
1e-02
94.09
66.60
94.21
95.68
79.67
93.94
95.25
84.27
93.51
94.48
88.40
93.58
5c-03
95.22
71.17
94.44
95.04
84.46
94.01
94.75
87.19
93.68
93.99
90.51
94.07
1c-03
95.04
89.91
94.29
92.65
92.48
94.54
91.94
93.57
94.32
91.55
94.10
94.49
5e-04
93.57
92.37
94.69
90.88
93.46
94.83
90.62
94.11
94.41
90.48
94.32
94.388
1e-04
86.08
94.14
94.16
85.16
94.10
84.60
94.10
93.93
84.53
93.88
93.58
5c-05
81,22
93.64
02.47
92.29
81.53
92.35
91.82
81.34
92.38
Table 5: DenseNet121 on CIFAR10
5e-03
0:
5e-04
1e-04
A
Adam
AdamW
AdamD
Adam
AdamW
Adam
Adam
Adamn
AdamD
Adam
Adamm
1e-01
44.78
1.71
55.48
71.37
14.12
70.27
75.40
18.67
71.21
73.80
28.14
69.27
5e-02
60.58
8.19
72.09
74.71
21.98
72.39
77.22
25.05
69.82
72.79
37.
69.70
1c-02
76.64
19.20
70.68
77.26
41.02
70.65
76.16
42.35
67.47
74.74
56.91
68.50
5c-03
77.74
30.09
68.44
76.47
54.23
67.86
75.73
58.16
67.71
74.11
67.05
67.10
le-03
75.11
60.86
72.69
72.18
71.26
72.73
70.86
73.25
72.33
70.01
74.55
72.83
5c-04
73.15
71.67
73.99
68.11
73.71
73.82
66.
74.36
73.79
67.31
74.65
73.78
lc-04
60.71
74.31
73.18
59.29
75.00
72.99
58.83
75.42
73.09
58.85
74.53
73.17
5e-05
55.21
73.84
71.11
54.67
73.31
70.77
53.93
72.54
70.47
54.45
71.32
70.90
Table 6: Res on CIFAR100
30

---

Published in Transactions on Machine Learning Research (04/2025)
a: 5e-03
a: 1e-03
5e-04
a: 1e-04
?
AdamD
Adam
AdamW
AdamD
Adam
AdamW
Adam
Adam
AdamW
Adamm
Adam
AdamW
1c-01
34.09
9.07
48.00
68.93
17.65
68.67
75.90
26.13
64.31
75.66
34.29
61.80
5e-02
51.52
15.52
69.43
77.81
22.10
69.12
78.93
30.30
66.41
77.05
39.90
65.60
1e-02
78.38
23.09
70.97
79.28
49.81
71.15
77.65
70.57
76.06
64.45
70.63
5c-03
79.42
28.61
71.90
77.54
52.05
70.82
76.45 76.42
59.47
70.25
74.42
69.14
70.01
1c-03
76.27
64.95
75.08
71.94
72.93
74.79
70.27
73.85
75.42
70.10
74.86
75.11
5e-04
73.77
73.25
76.20
67.19
75.12
76.34
65.47
76.28
76.10
67.17
75.61
75.98
1o-04
54.30
74.29
73.82
52.55
74.44
73.23
51.84
73.97
74.01
52.09
73.62
73.46
5c-05
42.13
71.99
69.23
42.55
70.03
68.29
42.10
69.56
68.23
42.47
69.40
68.12
Table 7: DenseNet121 on CIFAR100
Additionally, Tables and present experiments on CIFAR100 using the AdamW-D variant, which employs a diminishing momentum parameter 0g for AdamW. The results indicate that using diminishing momentum parameter AdamW effective, achieving performance comparable to that of the original AdamW.
50-03
a:
1c-03
a:
5c-04
a:
1c-04
A
AdamW-D
Adam
AdamW-D
Adamw
AdamW-D
AdamW
AdamW-D
lc-01
55.48
63.90
70.27
70.03
71.21
70.90
69.27
66.50
5c-02
72.09
70.24
72.39
69.95
69.82
70.06
69.70
70.72
1e-02
70.68
72.02
70.65
70.46
67.47
70.10
68.50
69.71
5c-03
68.44
69.77
67.86
69.72
67.71
68.43
67.10
67.23
le-03
72.69
73.19
72.73
72.58
72.33
72.83
72.83
72.81
5e-04
73.99
74.26
73.82
73.99
73.79
73.85
73.78
73.61
1c-04
73.18
73.38
72.99
73.08
73.09
73.49
73.17
73.66
5e-05
71.11
70.51
70.77
70.47
70.47
70.20
70.90
70.71
Table 8: ResNet34 on CIFAR100: diminishing 02 for AdamW-D
5c-03
a: 1c-03
a: 5c-04
a: 1c-04
AdamW
AdamW-D
Adamn
Adamn-D
Adamn
AdamW-D
AdamW
AdamW-D
le-01
48.00
55.29
68.67
68.57
64.31
63.80
61.80
65.40
5e-02
69.43
67.20
69.12
66.55
65.41
67.23
65.60
68.11
1e-02
70.97
71.04
71.15
70.92
70.57
68.98
70.63
70.48
5e-03
71.90
72.20
70.82
71.29
70.25
70.67
70.01
70.80
le-03
75.08
75.06
74.79
74.95
75.42
75.73
75.11
74.74
5e-04
76.20
76.32
76.34
75.79
76.10
75.85
75.98
76.45
le-04
73.82
74.02
73.23
73.77
74.01
73.18
73.46
73.23
5e-05
69.23
68.70
68.29
69.20
68.23
69.38
68.12
68.55
Table 9: 1enseNet121 on CIFAR100: diminishing 0, for AdanW-D
B.2
Language Modeling with LSTMs:
Tables 10 summarize the performance of 1-layer, 2-layer, and 3-layer LSTM models under various learn- ing rate and weight decay configurations. Figure illustrates the training and test perplexity curves for LSTM models with different numbers of layers, highlighting the convergence behavior and generalization performance of the optimizers. In Figure 5 the weight decay parameter is fixed at a = 10-5 and the
learning rates are tuned among {10-2,5 10 10-3, 3,10-3 based on observations from Tables 10 and 12
From Figure we observe that AdamD consistently outperforms Adam and AdamW in terms of test per- formance for models with more than two layers. Moreover, the performance gap between AdamD and the
31

---

Published in Transactions on Machine Learning Research (04/2025)
other methods widens the number of layers increases. Note that for models with more than five layers, overfitting may occur, as indicated by an increase in test perplexity.
a: le-04
c: le-05
a: le-06
7
A
Adam
AdamW
A
Adam
Adam
AdamD
Adam
AdamW
lc-01
166.1
172.8
560.6
582.9
88.2
582.9
128.5
173.0
572.8
5o-02
165.2
167.6
361.9
381.3
85.3
381.3
95.1
95.8
372.3
1e-02
163.2
165.2
139.1
85.8
85.6
139.7
92.0
90.9
138.9
50-03
163.5
163.9
102.9
85.2
85.0
103.2
95.3
85.8
103.3
lc-03
171.1
165.9
87.8
90.1
89.7
87.8
85.4
85.1
87.8
5e-04
182.6
169.3
88.7
96.2
94.8
88.7
88.5
88.0
88.7
Table 10: 1-layer LSTM
a: 1c-04
σ: 1c-05
0: 1c-06
7
Adamn)
Adam
AdainW
AdamD
Adam
Adamn
AdamD
Adam
AdamW
1e-01
602.1
158.5
602.1
623.9
75.0
623.9
92.4
93.3
623.8
5e-02
489.3
197.1
502.5
507.2
71.8
507.2
70.2
71.2
446.5
le-02
145.1
147.2
93.0
70.9
71.4
93.3
68.4
68.1
93.5
5e-03
146.8
147.0
83.6
70.0
70.7
82.9
83.4
67.9
83.2
le-03
154.4
147.5
73.2
74.9
74.5
73.3
71.2
71.0
73.3
5e-04
170.5
151.2
73.3
79.9
78.5
73.3
72.4
72.4
73.3
Table 11: 2-layer LSTM
σ: 1e-04
σ: 1e-05
σ: 1e-06
7
AdamD
Adam
AdamW
AdamD
Adam
AdamW
AdamD
Adam
AdamW
1c-01
141.7
154.6
642.1
638.9
72.9
638.9
69.8
69.7
638.9
5e-02
138.4
145.9
587.1
573.5
67.9
573.5
65.2
65.0
591.4
1c-02
135.0
191.2
106.0
64.9
66.7
99.4
64.9
65.2
99.1
5c-03
134.5
138.8
81.0
64.4
65.4
80.9
73.5
65.9
81.1
1e-03
144.7
137.8
69.0
69.6
67.1
69.2
67.3
66.6
69.2
5c-04
164.1
139.8
69.2
69.2
70.7
69.2
68.3
68.1
69.2
Table 12: 3-layer LSTM
Train Perplexity Layers
Test Perplexity vs LSTM Layers
Adam
Train Perplexity
Test Perplexiis lexiity
LSTM Layers
LSTM Layers
(a) Training perplexity
(b) Test perplexity
Figure 5: Comparison of training and test perplexity curves across LSTM models with different numbers of layers and three optimizers.
32


Reviews:
[{"review_id": "Yu4nOwoN12", "reviewer": "Reviewer_7Xvo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. I particularly appreciate the clarity of Section 1.1, and the subsequent clarity of a contextualized Section 1.2 (literature review and contribution statements)\n2. I am happy with the presentation of the convergence arguments (note: I did not verify each step of the proofs in the appendix)\n3. A good array of networks are included in the empirical analysis\n\n## Weaknesses\nNo significant weaknesses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors analyze a class of Adam-family methods for quadratically regularized *nonsmooth* nonconvex optimizations. Convergence proofs are provided for path-differentiable functions under reasonable assumptions, extending the literature on convergence results toward the practical implementations of neural networks. From their analysis, an extension of Adam is derived, AdamD. Empirical studies ranging from relatively small to $10^8$ scale models compare Adam, AdamW, and AdamD, which shows the promise of AdamD.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Requested Changes\n1. Please actually define \"UOP\". Unconstrained Optimization Problem? \n2. Please use consistent algorithmic naming, i.e. renaming framework (1) to (Adam)\n3. section 5.1.1: \"stepszie\" <-- \"stepsize\"\n4. Please provide justification within Section 5.1.1 for using 1/10 stepsizes for AdamD compared to Adam(W).\n5. section 5.1.1 p18, \"noticessly\" <- \"notably\" ?\n6. As a colorblind person I suggest using a dark-to-light colorbar for fig 7 (right now the highest and lowest scores are both dark)\n7. I believe it would present nicer to combine panels from Fig 1-4 into a single 4x4 , or at least two 2x4's (also to be consistent with Fig 6 format)\n8. [optional] There is an argument that this would be obvious for someone reading past the abstract, but you may consider explicitly introducing \"the weight decay term\" after (1) to better clarify the last sentence in p3,P5 after (AdamW), and to clarify the \"integrated with the function f\" statement, and potentially define \"coupled\" to better clarify what you mean by \"decoupled\" later on. E.g.\np3,P1,s1: \"In the framework (1), the weight decay term, $\\sigma x_k$, is integrated with the momentum and estimator terms. This comes from a direct interpretation of differentiating the loss function, where f and the optimization momentum are coupled.\"\n9. Why no AdamW visualization in Figure 7?\n10. [optional]: A (more complete?) extension of Fig 6 would be: x-axis is the number of LSTM layers (1:100, say), and y-axis is the final test-set perplexity. I think this would showcase the limits of performance characterization. By the looks of the initial trend, AdamD is doing better for deeper autoregressive nets, which is a great property to showcase. Assuming a standard LSTM implementation, I am guessing they are much smaller (in terms of # parameters) than the models in 5.1.1? \n\n### thoughts (no need to respond)\n11. Re: fig 7, it may be interesting to try computing the \"area\" of good performance as a novel metric. Why no AdamW visualization?\n12. It _may_ be worth it to also put AdamW into an algorithm block, right next to Algorithm 1, for direct comparison .\n13. Perhaps Section 2.2 could be named (for example) \"noise model\" instead of probability theory, to contextualize those def'ns", "broader_impact_concerns": "None is given nor is one needed. This work uses modest computational resources and a wide array of models that would be accessible to anyone implementing optimization algorithms.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bpuF9meEsx", "reviewer": "Reviewer_Y8UK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- **Broad framework**: Their framework AFMDW covers all Adam and Adam-family algorithms except AdamW. Therefore, their convergence results are applicable to all these algorithms.\n- **Experimental verification** : Even though their framework does not cover AdamW, their proposed AdamD obtains similar performance to AdamW.\n- **Convergence of AFMDW with Non-diminishing $\\eta_k$**: Their theoretical results under bounded noise assumptions obtain a.s. convergence to stationary points for non-diminishing $\\eta_k$, however with diminising $\\theta_k$.\n- **Extensive Literature Survey**\n\n### Weaknesses\n- **Noise Assumptions**: In Table 1, the authors claim that their methods have WRS (weighted random sampling) and RR (randomized reshuffling) noise assumptions. This is not the case.\n    - **WRS** : The noise assumptions corresponding to WRS, Assumptions 3.3 and 3.15, require the noises to be uniformly bounded martingale difference sequence. For a finite-sum objective ($f(x) = \\frac{1}{N}\\sum_{i=1}^N f_i(x)$) with weighted random sampling (consider weights $\\{w_i\\}_{i\\in [N]}$) with a given batch size $B$, additional assumptions, for instance Lipschitz-ness of $f_i$'s are required for this WRS sampling to satisfy Assumptions 3.3 or 3.15. Assumption 4.1 requires Lipschitz $f_i$'s, so this might imply a certain WRS sampling result as well, however, the authors should either explicitly show that this implies WRS or provide a reference for this. Additionally, the noise assumptions corresponding to WRS should be referred to as bounded martingale difference noise or something equivalent as these are the exact conditions required.\n    - **RR**: Assumption 3.11(2) is the additional noise assumption, so I'm considering this as the randomized reshuffling assumption. If $\\xi_k$ is the noise at iteration $k$, this assumption requires $\\sum_{i=jN}^{(j+1)N-1}\\xi_{i+1}=0$ for some $N\\in \\mathbb{N}$. First of all, this is a very strong assumption as the sum of noise is exactly $0$ on top of being bounded martingale difference. Second, this does not correspond to a random reshuffling arrangement for finite sum objective as the sum of gradient noises in one permutation does not sum to $0$, unless each $f_i(x) = f(x) + c(i)$, where $c$ is a function independent of $x$. I'd recommend the authors to not use the term RR, and provide justification for this noise assumption possibly via examples. Note that this noise assumption is required for global stability of AFMDW.\n\n- **Typos**: Please fix these typos.\n    - Assumption 3.11 (2), $\\sum_{i=jN}{(j+1)N-1} \\xi_{i+1} = 0$ instead of $\\sum_{i=jN}{(j+1)N-1} \\xi_{k+1} = 0$. \n    - Page 13, 2nd paragraph, 2nd line : \"can asymptotically find\" instead of \"can asymptotically finds\".\n    - Page 26, \"Proof of Proposition 3.20\" instead of  \"Proof ofof Proposition 3.20\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider minimize the objective $ g(x)\\coloneqq f(x) + \\frac{\\sigma^2}{2}\\|x\\|^2$ over $x\\in \\mathbb{R}^n$, denoted by (UOP).\nHere, $f$ is possibly non-smooth and locally Lipschitz. \n\nThe authors consider path-differentiable functions $f$ defined in (Bolte \\& Pauwels, 2021). First, note that output of automatic differentiation programs on non-smooth neural network modules like ReLU result in path-differentiable functions. Further, path-differentiable functions admit a conservative field $D_f$ such that for any abs. cont.  mapping $\\gamma:[0,\\infty]\\to \\mathbb{R}^n$, $f(\\gamma(t)) - f(\\gamma(0)) = \\int_0^t \\max_{d\\in D_f(\\gamma(s))} \\langle \\dot \\gamma(s), d \\rangle ds$.\n\nThey consider the Adam-type optimizers to minimize (UOP). Existing works(Xiao et al 2023) that analyze convergence of Adam on path differentiable non-smooth functions require all step-sizes decaying to $0$ to show convergence to stationary points. Additionally, in practice AdamW, where the weight decay ($\\sigma$) is decoupled and added only at the update step with step size $\\eta_k$, performs much better than Adam. However, the only convergence analysis for AdamW (Zhou et al 2024) requires smoothness and uses an approximate stationarity measure as the convergence metric. \n\n\nTheir main contributions are --\n-  **Algorithms**: They propose a larger class of Adam-family of algorithms denoted by (AFMDW), that covers several common adam variants, for instance Adam, Adagrad, Yogi etc that minimizes UOP. Note that AFMDW does not contain AdamW, but these also decouple weight decay and Adam updates like AdamW, with the key difference being that AFMDW applies the preconditioner to the weight decay term as well.\n- **Theory**: \n    - The proof sketch follows that of (Benaim et al 2005, Davis et al 2020). Here, the interpolated update of AFMDW lies in a perturbed solution of $-D_f$. The perturbation magnitude decreases to $0$ with iterations, so that asymptotically, a stationary point is reached. The magnitude of perturbation depends on noise and step sizes.\n    - When the gradient noise is a uniformly bounded martingale difference sequence (Assumptions 3.3(3), 3.15(5)), for decreasing step sizes, the authors show a.s. convergence to stationary points for AFMDW when the step size $\\eta_k$ used for updating the iterates is non-diminishing, but the step size $\\theta_k$, used to update momentum terms is $o(\\frac{1}{\\log k})$ (Theorem 3.10). \n    - Under a stronger noise assumption and parameterization of $\\theta_k$ (Assumption 3.11), the authors show that AFMDW is globally stable, i.e., for every $\\epsilon>0$, there exists $\\theta_{\\max}>0$ with $\\lim \\sup_{k\\to+\\infty}\\theta_k \\leq \\theta_{\\max}$ such that the distance to stationary points is $\\epsilon$ a.s. (Theorem 3.13).\n    - For a single time-scale, $\\lim_{k\\to\\infty}\\frac{\\theta_k}{\\eta_k} = \\tau_1 $, a specific instance of AFMDW reaches the stationary point of UOP a.s (Theorem 3.22).\n- **Experiments**: The authors propose one instantiation of their AFMDW scheme, AdamD, that performs as well as AdamW for several NN tasks involving image classification on ResNets and LSTMs for language modeling.\n\n**References**\n- (Bolte \\& Pauwels 2021) Conservative set-valued fields, automatic differentiation, stochastic gradient methods and deep learning. Math Programming.\n- (Xiao et al 2023) Adam-family Methods for Nonsmooth Optimization with\nConvergence Guarantees. JMLR. \n- (Zhou et al 2024) Towards Understanding Convergence and Generalization of AdamW. TPAMI.\n- (Benaim et al 2005) Stochastic approximations and differential inclusions. SIAM Journal on Control and Optimization.\n- (Davis et al 2020) Stochastic subgradient method\nconverges on tame functions. FoCM.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please fix the weakneses. The noise assumptions are critical to my recommendation.\n\nAdditionally, consider these minor changes.\n- **Differences over (Xiao et al 2023):** The current work looks like an extension of (Xiao et al 2023), with a focus on two-timescale analysis of Adam with regularization instead of single-timescale analysis of Adam with gradient clipping. As these papers are very similar in their proof techniques and defining a generic Adam-family, I'd recommend the authors to highlight concrete differences.\n- **Possible parallel work (Jin et al 2024)**: This seems to be a parallel work on Adam's convergence guarantees released on arxiv which the authors should consider mentioning in their related works. They consider smooth losses, so the comparison to this paper is direct.\n- **Difference between AdamW and AFMDW**: From what I understand, the key reason to introduce AFMDW instead of AdamW is to have the term $\\sigma x + m$ which goes to $0$. AdamW instead might have resulted in a weighted version of the same equation with one of the weights depending on $v$ and thus requiring convergence of $v$, as the authors have argued in Page 3 last paragraph. Comparing the surrogate loss function from this paragraph, i.e., $f(x) + \\frac{\\sigma}{2}\\langle x, (\\sqrt{v} + \\epsilon) \\odot x \\rangle$, we see that there is a similar term in AFMDW's Lyapunov function in third term of Eq(25), only AFMDW weighs the additional term by $\\sigma x + m$. I think the fact that the possible weighted version of $\\sigma x$ and $m$ would depend on $v$ and would thus be harder to handle should be discussed to motivate using AFMDW instead of AdamW. Also, this term itself might not be that hard to handle, so the proof of AFMDW could be extended to AdamW. \n\n**References**\n\n- (Xiao et al 2023) Adam-family Methods for Nonsmooth Optimization with\nConvergence Guarantees. JMLR.\n- (Jin et al 2024) A comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap With SGD. arXiv:2410.04458", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xcxRO9gRvd", "reviewer": "Reviewer_tSGb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper analyzes Adam-type methods in a very general nonsmooth nonconvex setting. In particular, the analysis is valid for a subdifferential that includes the \"gradients\" that are usually computed from automatic differentiation. This keeps the analysis close to what is actually done in practice. Another strength is that the analysis in fact includes many different methods (see Table 2) within a unifying proof technique.\n\nWeaknesses: \n* The convergence results are asymptotic. No non-asymptotic guarantees or convergence rates are provided. \n\n* A TMLR paper by Zhuang et al., 2022 (https://openreview.net/forum?id=IKhEPWGdwK) shows how AdamW can be seen as an approximation of proximal Adam (that is, handling the regularization with proximal operators). This important reference is missing in the discussion. For example, on page 3 bottom, it says: \"AdamW lacking a clearly defined objective function to minimize\". In the light of Zhuang et al, 2022, this is not entirely true: AdamW can be seen as minimizing a regularized objective with a proximal method (up to a first-order Taylor approximations).\n\n* The proposed method AdamD seems to have been proposed almost identically in this (peer-reviewed) blog post: https://iclr-blogposts.github.io/2023/blog/2023/adamw/. If one looks at the method called AdamP therein, then doing the same first-order Taylor approximation as mentioned above on the coefficients, one obtains AdamD. To be clear, the contributions of this submission still go beyond the blog post as (i) a theoretical analysis of AdamD is provided in a very general setting and (ii) the numerical evaluation in this paper is much more detailed and extensive than the one in the blog post. However, a discussion and reference to these previous works should be included.\n\n* Several details on the experimental section are unclear or not provided (see *Requested Changes*)\n\n* Page 2: \"Given the fact that non-diminishing step-sizes are widely employed in computational frameworks, ...\". Are non-diminishing step sizes really the standard practice for training? To my knowledge, a common choice is to use step size schedulers with some form of annealing in the end (cosine, WSD, linear decay); step sizes of the form 1/sqrt(t) are also widely established in theory and practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper analyzes a family of Adam-type methods in the nonsmooth, nonconvex setting. A global asymptotic convergence and stability  result is shown for a general notion of the subdifferential. Finally, one specific member of this family of methods is proposed and analyzed in experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please add discussions and references related to the prior works mentioned in Weaknesses. Also, a comparison to the this paper https://openreview.net/forum?id=ZPQhzTSWA7 is missing, which also analyses Adam.\n\n**Changes for experiments:**\n\n1) I understand that the effect of weight-decay is different for AdamW vs AdamD due to the fact that the term $\\sigma x_k$ is multiplied by $H_k$ or not. But in consequence, why is the weight decay value $\\sigma$ fixed, while the range of learning rates $\\eta$ is different for Adam(W) and AdamD by one order? Wouldn't it make more sense to compensate by changing the value of $\\sigma$, while doing exactly the same grid search over $\\eta$ for all methods (as $\\eta$ also impacts the gradient term)?\n\n2) Please state for each method and problem setting the tuned value of $\\eta$. Possible performance differences might be due to the tuning procedure, and it is currently incomprehensible for the reader as these details are missing. In general, it is also helpful to see what is the normal variation of a method only due to tuning $\\eta$. For example, one could show the K best step sizes/runs for each method in the appendix.\n\n3) Was the choice of $\\theta_s$ used for AdamD also tried for Adam(W)? If yes, how does it compare?\n\n**Minor:**\n\n* There are many citations that should use \"\\citet\" instead of \"\\citep\", for example last line of page 1. Or \"Kingma and Ba (Kingma & Ba 2015)\" on page 1.\n* The paper cited as \"Loshchilov & Hutter 2017\" was published in 2019 at ICLR. As these two authors have another widely known paper from 2017, this can lead to confusion (so it should be \"Loshchilov & Hutter 2019\" instead).\n* Page 26: \"Proof ofof\" should be \"Proof of\".", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the convergence properties of a class of Adam-family methods with decoupled weight decay for minimizing quadratically regularized nonsmooth nonconvex optimization problems. The paper highlights AdamD, a novel version of Adam with weight decay that resembles AdamW (and achieves similar practical performance). However, unlike AdamW, AdamD falls within the studied class, allowing its convergence properties to be derived from the proposed framework.

This paper develops new convergence theory for Adam-type methods in nonsmooth settings. The reviewers noted that including non-asymptotic convergence guarantees could have further strengthened the contribution.

The authors did an excellent job revising the manuscript based on the reviewers' recommendations. All reviewers unanimously recommend acceptance of this work.
Decision: Accept (poster)