Cautious Optimizers: Improving Training with One Line of Code
Authors: Kaizhao Liang
Lizhang Chen
Bo Liu
qiang liu
Venue: ICLR 2026 Poster (2026)
Keywords: Optimizer
AdamW
Primary area: foundation or frontier models, including LLMs
TL;DR: Improving Training with One Line of Code

Abstract: AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcomes. In this work, we propose a \textbf{single-line modification in Pytorch} to any momentum-based optimizer, which we rename cautious optimizer, e.g. C-AdamW and C-Lion.  Our theoretical result shows that this modification preserves Adam's Hamiltonian function and it does not break the convergence guarantee under the Lyapunov analysis. In addition, a whole new family of optimizers is revealed by our theoretical insight. Among them, we pick the simplest one for empirical experiments, showing not only consistent speed-up on LLM pretraining and post-training tasks, but also better results in MAE pretraining, with minimum extra tuning on hyperparameters.

Published as a conference paper at ICLR 2026
CAUTIOUS OPTIMIZERS: IMPROVING TRAINING WITH ONE LINE OF CODE
Kaizhao Liang*..f Lizhang Chen Bo Liu', Qiang Liu' 'The University of Texas at Austin
ABSTRACT
AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcomes. In this work, we propose a one-line modífication in Pytorch to any momentum-bant based optimizer, which we rename cautious optimizer, c.g. C- AdamW and C-Lion. Our theoretical result shows that this modification preserves Adam's 's Hamiltonian function and it does not break the convergence guarantee under the Lyapunov analysis. In addition, whole new family of optimizers is revealed by our theoretical insight. Among them, we pick the simplest one for empirical experiments, showing not only consistent speed-up on LLM pretraining, but also image classification, with minimum extra tuning on hyperparameters.
INTRODUCTION
Algorithm 1 Caution an Optimizer (OPT) in PyTorch
param P, update u from OPT, grad g
m (u * g to (g. dpe) p. adddd_ (u m/ mm. () +eps), alpha=-1r)
Optimization is an important and constantly evolving field in modern machine learning. Undoubtedly. Adam (Kingma, 2014) and AdamW (Loshchilov, 2017) are the most consequential optimizers proposed almost a decade ago. Since then, many efforts (Zhang et al., 2021; Loshchilov et al., 2017) have been made to discover better and faster optimizers beyond these two. However, until now, AdamW remains the dominant workhorse for applications, from pre-training Large Language Models (LLMs) (Touvron et al., 2023) to fine- fine-tuning text to image diffusion (Rombach et al., 2022), with no real challenges to their ruling status.
In the dawn the era of LLMs, the arms race of model scaling intensifies (Achiam et al., 2023). A faster optimizer means more training tokens can be consumed within the same amount of time. Ultimately. this leads to more capable models (Kaplan et al., 2020). Hence, the interest in scarching for optimizer beyond AdamW W is re-kindled. Recent progress in new AdamW alternatives such as Lion (Chen et al., 2024; 2023a), SHAMPOO (Gupta et al., 2018), SOAP (Vyas et al., 2024), ADOPT (Taniguchi et al., 2024), and Schedule-Free (Defazio et al., 2024), all claim substantial improvement over AdamW.
However, these methods normally require non-trivial efforts obtain optimal results, especially hyperparameter tuning, which greatly limits their potential and wide adoption. In light of this dilemma, we propose cautious optimizers, an exceptionally simple performance booster of any momentum-basse optimizer that only requires one line of modification (see Algorithm 1). The change simple: do not update unless the proposed update direction and the current gradients are aligned. With this minor change, we obtain consistent improvement over the base optimizer without modification of the original optimal hyperparameters.
To provide an overview of the idea, let us consider general optimizer for minimizing the loss L(w):
w <- w: 6
'Equal contribution by KZ. and I.Z. 1 Conresonndence: kai zhaol, 1zchen@utexas.edd

---

Published as a conference paper at ICLR 2026
where uy is the negative update direction of the parameter w, iteration t, and 6 0 is the step size. We will assume that the update represents a generic momentum- based optimizer, including, for example, Polyak and Nesterov momentum, Adam, and Lion. In all these momentum- based optimizers, uy does not necessarily align with the gradient direction 9: = VL(w)) which can result in temporary increase in the loss function and slow down convergence.
Cautious optimizers avoid this issue by adding a simple mask function based on the sign consistency of u, and gr:
wr+1 + we t, o pbu gt),
where denotes element-wise product, and ø is map that reweights the update based on the product u o gt- We simply take it as
d(vr) ( 0),
so that the update is zeroed out for coordinates on which the sign of u, and gt. are inconsistent. Here axx) is positive scaling factor, introduced to compensate the decrease of update magnitude due to masking. A simple choice of is
dim( vz))
a(x) nnz (er > 0) +
where dim(-) and anz(-) represent the total number of elements and the number of non-zero elements of the input vector, respectively. Here, & 0 is a positive constant, which we set to & by default. See Algorithms 1 and 2 for more details.
This modification ensures the new negative update have a non-negative inner product with the gra- dient, and hence decreases the loss monotonically when the step size is sufficiently small. Specifically, Taylor approximation shows
((w++1)
ce o g (uu g:) S.
This ensures decrease of the loss, i.e., [(w++1)= < C(w,), when the step size is sufficiently small. In comparison, typical momentum-based optimizers do not always guarantee monotonic decrease in loss even with infinitesimal step sizes. The nature of momentum dynamics introduces oscillations due to inertia-like effects.
Our theoretical analysis shows that the modified algorithm converges to local optima under mild conditions on the base optimizers. An interesting aspect of this algorithm that it does not get stuck at non-stationary points of the loss, even if u, can be temporarily completely conflicting with gr and is therefore entirely masked out. This is because, in typical momentum methods, the update direction u, continues to accumulate the gradients and will eventually be updated to have a positive inner product with g. if is stuck at non-stationary point.
Our theoretical analysis encompasses both continuous-time and discrete-time cases. In the continuous- time setting, our theory shows that the modified algorithm guarantees convergence to local optima for optimizers that admit DamiltonianDDeseen structure (Chen et al., 2023a; Liang et al., 2024), which broadly includes almost all existing popular algorithms, such as Adam, Lion, heavy ball, and Nesterov momentum. For these algorithms, we demonstrate that the cautious optimizers, with ø satisfying x0(x) 2 max(x,, 0), retains the monotonic decreasing properties of the original Lyapumov (or Hamiltonian) functions of these algorithms while additionally minimizing the loss function. In the discrete-time setting, we analyze the behavior of various variants of mask functions and establish general conditions under which updates from cautious optimizers yield larger local descents compared to their base optimizers.
To summarize our contributions, we present the following:
We propose Cautious Optimizers, a simple performance booster for any momentum-based optimizer, implemented with just a single line of code.
We theoretically demonstrate that cautious optimizers preserve the convergence guarantees of the base optimizer while also accelerating the decrease of the loss function.
We show consistent improvements across various tasks, from pretraining L.M to image classification.
2

---

Published as a conference paper at ICLR 2026
Algorithm 2 Cautious AdamW (C-AdamW)
Require: parameter w, step sizes { dampening factors A1, B2 E [0, 1), O, weight decay
y 2 0.
1: Initialize & 0, and mo, tnp-
2: while wy not converged do
3: t f 1+1
4: - V (wrr(())1))
5:
m + 3.m m,m,- 1 +1 91
6:
f 120 + -11 82)gi
7:
m, t m. (1 -81)
8:
U. 0z/( (1 12)
9:
u - m (v +e)
Φ - T(ue o 9 > 0)
I/ Compute alignment mask Scale dimension db
E a 1
12:
Wt + W 0u
// Masked update
w; wi Yw;
|/ Add weight decay
14: end while
THEORY
We start with introducing a general Hamiltonian descent framework for the continuous-timg forms of general momentum algorithms (Section 2.1). We then introduce the cautious optimizers in the continuous time form and discuss its theoretical properties (Section 2.2). Finally, discuss in Section 2.3 theoretical properties of cautious optimizers in discrete time forms.
2.1 HAMILTONIAN+DESCHNI
In the continuous-time form, most momentum-based algorithms can be viewed as variants of the damped Hamiltonian system, which admit Lyapunov (or Hamiltonian) function that certifies their convergence towards the stationary points.
The Lyapunov function is typically augmented loss function H(w, 8) defined over both the weights w and an optimization state vector & (which includes the momentum). It must satisfy min, H(w, s) L(w), so that minimizing [(w) is equivalent to minimizing H(w, s). This is typically achieved using a separable Hamiltonian of the form
H(w, 8) [(w) + K(s),
where KC(.) is any lower-bounded function. Consulting physical intuitions, we can think H as the total energy of system parameterized by (w, s), with L and K. representing the potential energy and kinetic energy, respectively.
The continuous-time form of common momentum-based algorithms can be unified into:
d
di W: -KK(s.) (.(VL(w.))
(2)
d
v((w)) V. (VK(ss).
dt
where (.).) and (-(-) are monotonic mappings satisfying
| = (z, (x)))) 2 0.
| = (x, V (z 2 0,
for any x. With (xx) = L (x) = 0, the system in (2) reduces to the standard Hamiltonian system, which preserves H(w,, s,) const along the trajectory. When the descending components 4, and V, are added, the system ensures that H(w, s) becomes monotonically non-decreasing:
d
H(w,, 9t) A (ww,, US!) S O,
(3)
dt
where
A (w vSs) = |V((w: 12. +IVK(s)))1.
3

---

Published as a conference paper at ICLR 2026
On the other hand, [(w), which is the true objective, is not necessarily decreasing monotonically. There can be cases where [(w) increases temporarily in exchange for large decrease in the kinetic energy (()s,, while still ensuring decrease in the total energy H. C + K. Specifically,
[(w)) = - 8c);
(4)
dt
Ac(w,, St) V.((w) VVC(s:) + [VV((w))) [V
Here, Ac(m, 8t) may be negative due to the cross term.
See Appendix for the Hamiltonian of common optimizers including Adam (Kingma, 2014) and Lion-K (Chen al., 2023b;a).
2.2 CAUTIOUS DYNAMICS
Our idea change the dynamics to make it simultaneously decrease both H(w s) and L(w) We do this with modified system:
T VL(W.) VK((3t)
d
W. ( VK(3.) (V(((()
dt
(5)
d
8, VL(ww: - (.K(())))).
dt
where denotes the element-wise product and is a vector vector mapping. Here we weigh each element of the update direction VK((s; based on the product of VK((s) with the gradient (L(w). Note that we do not need to apply mask on the (VL(), term since it is always non-increasing by definition of 4,,
The following conditions on the choice of function & ensure that the system decreases both H. and L simultaneously.
Theorem 2.1. Following the dynamics in (5) in IR,, we have
d
H(w, (z; (1 (Tx)) An. (1 3t),
dt
d
di ((w:)
- Φ T V [V
2
[ (1 )x))) Dc, (wi, 3t).
Here, An. (w,, 8t) and Ac, (w,, 8t), defined in (3) and (4), respectively, represent the decreasing rates of H. and C in accordance with the original system (2), Hence:
xf T (1 ((x)) S for any Rd, then both H and L decreases faster than the original system:
H(w, 3.) 5 - An. WT. 3e) S 50
dt
((w,) S -
di
T ((x) 0 for any Rd, then L decreases monotonically, de [(w.) S
One sufficient condition for & to satisfy both conditions in Theorem 2.1 is enforce the following element- wise constraint:
d(vx); iff 0,
and
d(v)), Of x; < 0.
where σ(vr); denotes the i-th element of o(uz). Under this condition, both H. and L decrease monotonically following the cautious dynamics, at a rate faster than the original systems. In particular, the default choice ((vx) ((vx))((v 0) with a(ux) 2 satisfies the conditions above.
4

---

Published as a conference paper at ICLR 2026
Convergence to Stationary Points In addition to monotonically decreasing the loss function, want to ensure that the algorithm does not get stuck unless the solution reaches a stationary point, which in practice is typically local optimum, of the loss function. The following result demonstrates that this property holds under the same conditions as the original Hamiltonian descent system.
Corollary 2.2. Assume that the norm || 1l is positive definite, (0) O, and that H(w, s) [(w) + K(s) is differentiable. Then, the bounded solutions of the original system (2) converge to stationary point of H(w, 8). Similarly, the bounded solutions of (5) also converge to stationary point of (H(,,s)) s).
2.3 DISCRETE- TIME ANALYSIS
We analyze the discrete time case, demonstrating that each step of cautious optimizers are at least as good as the step of the original optimizers under mild conditions.
We will consider generic update of form
W+++1 W) C1 (Wk, sk),
8k-1 8i: - 2% (ers 8k);
where Uk, Uh are vector fields that define the updates, and is the step size. We write the cautious variants
W k++1 wkk C₁ Ug UOR
UR us (Kk.sk)
(8)
8k++1 8% + [1,, 8k),
where wal is mask vector determined by the algorithm, voly = alu [ o dij > where Vg = ((
0),
The following is comparison result showing that each step of the cautious optimizer yields larger loss decrease than the original optimizer under mild conditions.
Theorem 2.3. Consider (7) and (8) with p-smoooh loss function L(-)- Assume the element-wise operator 6 satisfies
A(Ux) == - -x T (1 ([((x)) 2 0.
Starting from (wi, (wi St), have
((W++1) S 1(w;-1);
which holds for step size <
2 oug
where UT. ū,- and Vg. = VL(W.)
[12
This result works only for range of step sizes due the need of Taylor approximation. In Appendix A.5, we show a case when the comparison holds for all step sizes when using a different mask function based on inner products and when L is convex. The following is another result showing that when imposing more restrictive conditions on the mask to the step size that ensure that the cautious optimizer is guaranteed to decrease loss at each step.
Theorem 2.4. Consider updates (7) and (8). Assuming L(.) is p-smooth, and consider the following mask finction:
vΦ =og (VL(W))) oug na vūg (ukg)
where } is any sequence and 2 EyOC Starting from (w,, 8t) (We, Be), we have
[( & L(Wx).
The results above demonstrate that cautious optimizers reduce the loss more efficiently than the original optimizers at each single step. natural question whether these comparison results can be extended to multiple steps. This, however, becomes challenging because, after the first step, the two optimizers explore different regions of the loss landscape, making it easy to construct counterexamples where one method outperforms the other. This is consistent with the no free lunch theorems, which state that no single optimizer can dominate another across all possible loss functions (Wolpert & Macready, 1997). Nevertheless, it is reasonable expect that the advantage observed in a single step would naturally extend to multiple steps for the practical loss functions encountered in deep learning, evidenced by the experiments presented.
5

---

Published as a conference paper at ICLR 2026
GDM
C-GDM
Optimum
b)
6)
Optimization Trajectories
Optimization Trajectories
((w))
N(W,; Sc]
1.0
100
0.02
10:
10-1-
Obyective
10"13
10
0.00
10
10-21
10-22
10
-0.5
-0.01
10-16
10-35
-1.0
20
Step
Step
Figure 1: We compare gradient descent with Polyak momentum (GDM) and element-wise cautious variant (C-GDM), using gradient descent (GD) as a haseline. The step size for GD and the hyperparameters of GDM (including step size and momentum coefficients) are chosen to achieve the optimal convergence rates, which be analytically derived (sce, c.g., Goh (2017)) For cautious optimizers, step sizes and momentum coefficients are empirically tuned, shown in Figure Detailed experimental settings are described in Section 3.1. Plot (a), we visualize the optimization trajectories of the three methods, starting from the initial point (1, 1) with cero-initiaizzed momentum. Notably. C-GDM converges the optimum with significantly reduced overshooting and oscillation, Plot (b) zooms in the trajectories from Plot (a), focusing smaller region (0.02 0.02) for enhanced clarity. Furthermore, Plots and (d) show that C-GDM consistently monotonically decreases both the objective and the Hamiltonian associated with the original GDM, highlighting its superior performance
in minimizing these metrics compared to GDM.
GDM
C-GDM
00-001
Pc-0-1
inner
C2
0-2
0.2
82
02-
B
B
Figure 2: Convergence rate heatmaps for the objective function with condition number (same setup in 1). The heatmaps illustrate convergence rates, where values greater than indicate divergence for the corresponding (c, 3) configuration. Smaller convergence rates correspond to faster convergence. From left to right, we show heatmaps for d, with 0.01, 0. and snnew as shown in (13). Note that do represents constant function, reducing gradient descent with momentum (GDM), whose optimal convergence rate given in closed form (Goh, 2017). From the hcatmaps, clear that all cautious momentum variantsć, with c 0.01, 0.1 and Pimer demonstrate superior convergence rates compared to GDM.
EXPERIMENTS
In this section, evaluate the performance of cautious optimizers compared to their standard counterparts, highlighting the benefits introduced by the cautious masking mechanism. We begin with a 2D toy experiment to provide visual demonstration of how cautious masking improves optimization. Subsequently, we extend the evaluation to large-scale pretraining tasks for both language and vision models, comparing the performance of standard optimizers and their cautious variants.
3.1 2D OPTIMIZATION ToY
We consider a 2D optimization problem with objective [(w) = x(wr)² + (wz)², where w (wi, W2) E 2² is the parameter. Obviously, the optimum is at w" (0, 0). We set in our experiments. We apply gradient descent (GD), gradient descent with Polyak momentum (GDM), and cautious gradient descent with momentum (C-GDM) on this toy example, starting from wh = (1, 1). Specifically, for GDM, we adopt the conventional momentum update:
St - 38,-1-1 + vL(w:),
W. f w EBt,
where 3 E [0, 1) and is the learning rate.

---

Published as a conference paper at ICLR 2026
Observation: On the right of Figure I, we compare GDM and C-GDM with the same hyperparam- eters (e, 3). We ablate over different combinations of (3,e) E {00001 0.5), (0.01, (0.01, 0.99), (0.01, 0.999), (0.1, 0.99), (0.001, 0.99)). Across all settings. C-GDM outperforms GDM, confirming the importance of cautious masking. Given the same (e, B), cautious is always not worse than mo- mentum, and often it gives significant improvement, especially when the choice (c, (3) is suboptimal for momentum, meaning that cautious masking makes it more robust. From the left of Figure I, one can see that GDM, due to the momentum, has fluctuating [(w,), while C-GDM ensures that ((w,) monotonically decreases. In addition, C-GDM achieves a faster drop in terms of GDM's s Hamiltonian.
In Figure I, we compare GDM and C-GDM, each using their optimal (c. 3). For GDM, the optimal values are derived theoretically (e.g., (Goh, 2017)), while for C-GDM, they are obtained through a grid search. Despite using the optimal configuration, GDM exhibits significant overshooting, oscillations, and slower loss convergence. In contrast, C-GDM achieves smoother trajectories, reduced overshooting, and faster convergence, demonstrating its superior stability and efficiency.
We estimate an algorithm's 's convergence rate as the slope of log ((w,) over time via linear regression and plot heatmaps over (e, 9) -space, where and 3 are the learning rate and momentum. Figure shows that cautious methods achieve lower optimal convergence rates compared to the momentum method VR-1 red dot). The heatmaps highlight that all cautious momentum variants outperform v#+1* GDM in convergence rates.
3.2 PRETRAINING LARGE LANGUAGE MODELS (LLMS)
We begin by investigating the language modeling task using a 100M LLaMA (Touvron et al., 2023) model as the foundational architecture. The models are trained on the C4 (Colossal Clean Crawled Corpus) dataset (Raffel et al., 2020), a large-scale web-crawled text corpus containing billions of tokens. We provide results from the following settings: we take a 100M model and train with batch size up to 2 million tokens for 50 billion tokens (25x Chinchilla Optimal). For optimization, we employ Adam W (Loshchilov, 2017) and Lion (Chen et al., 2023c), two popular optimizers in modern language modeling, as baselines. These are compared with their cautious counterparts, which we term Cautious AdamW (C-AdamW) W) and Cautious Lion (C-Lion).
Ir
le-4
3e-4
1e-3
3e-3
le-2
2e-2
3e-2
1e-1
AdamW
85.050
24.384
19.249
19.007
18.965
19.609
**
**
C-AdamW
19.065
18.771
18.684
18.821
Ir
3e-5
le-4
3e-4
6e-4
le-3
le-2
Lion
28.250
21,401
21.937
**
C-Lion
21.354
19.795
20.403
20.977
**
Table 1:
We report final evaluation perplex, the lower the better. nee" are runs that did not converge due to small learning rates. "_" stands for runs we skip due to lack of baseline comparison.
either too large Each model trained with batch size up to 2 million tokens for 50 billion tokens in total (25 x Chinchilla Optima((Hoffmam et al., 2022)). We use 31 = 0.9, B2- = 0.95 and weight decay 0.1 on Adam W; Sequence lengths of all models are 1024. For Lion experiments, we follow the recommendation from the (Chen et al., 2023a) and use 3i = 0.95 and 32 = 0.98. For scheduler, we use CosineAnnealing with warmup and the learning rate is decayed to 10% of the initial learning rate. Gradient accumulation is set to 8 to increase the global batch size.
Observation: As shown in Table 1, Cautious Optimizers demonstrate consistent improvements in both evaluation perplexity and sample efficiency. Table shows the cautious modification robust across learning rates and Cautious doesn't it change the optimality of hyperparameter search done on the base optimizers. Surprisingly, Cautious can also tolerate a higher learning rate in the Lion experiments and achieve stable training even when the baseline diverges.
To further confirm our finding, we also include a scaling experiment with C-AdamW on Fibe -duu (Penedo et al., 2024), more recent and higher quality scale text dataset. After rigorous and thorough hyperparameter search, we found that C-Adam is consistently outperforming baseline AdamW. In table 2, we report optimal results for each scale.
7

---

Published as a conference paper at ICLR 2026
Table Perplexity comparison between AdamW and C-AdamW across different model scales at 1x Chinchilla (Hoffimann et al., 2022), hyperparameters are extensively searched with coordinate descent over discrete grid. Details can be found in appendix
Scale
Adam
C-AdamW
Improvement (%)
130M
27.39
27.30
0.33
300M
18.30
18.28
0.10
520M
15.07
14.92
1.00
1,2B
11.36
11.32
0.32
Furthermore, we perform downstream evaluation on the produced 1.2B checkpoints with 1x Chin- chilla (20x tpp) on 7 downstream tasks, where the checkpoint trained by cautious optimizer wins in of them (MMLU, OpenBookQA, Are Easy as well as HellaSwag and Are Challenge).
Task Group
Metric
Adam
C-AdamW
Are Easy (Clark et al., 2018)
0.6082 = 0.0100
0.6090 = 0.0100
Are Challenge (Clark et al., 2018)
_norm
0.2875 0.0132
0.2978 = 0.0134
Hellaswag (Zellers et al., 2019)
acc_norm
0.4169 0.0049
0.4193 + 0.0049
Lambada OpenAl (Radford et al., 2019)
acc
0.3311 + 0.0066
0.3229 + 0.0065
OpenBookQA (Mihaylov et al., 2018)
0.2340 + 0.0190
0.2360 + 0.0190
PIQA (Bisk et al., 2020)
0.6774 0.0109
0.6768 + 0.0109
MMLU (Hendrycks et al., 2021)
0.2529 + 0.0037
0.2535 + 0.0037
Table Comparison of benchmark results between C-AdamW and AdamW across multiple tasks and MMLU groups. All evaluations are done with LM wval-Harness(Gare et al., 2024). Bold indicates better score.
3.3
IMAGE CLASSIFICATION
We also include classic classification task on Mini-ImageNet ViT (Dosovitskiy, 2020) with two additional optimizers MARS (Yuan et al., 2024) and LaProp (Ziyin et al., 2020), both are momentum-based optimizers and their cautious variants perform consistently better as shown in the table 4.
Method
Eval Topl
C-AdamW
73.52
AdamW
72.
C-LaProp
73.92
LaProp (Ziyin et al., 2020)
71.73
C-MARS
74.91
MARS (Yuan et al., 2024)
74.06
Table Top-1 evaluation accuracy on Mini- Imageen, higher better. can see the cautious variant better across base optimizer options. Hyperparameters can be found in appendix D
4 RELATED WORK
We provide a brief overview of existing efforts on designing Adam-like optimizers, and the related works on Hamiltonian dynamics.
Adam and Its Variants A plethora of Adam variants have been developed to address different aspects of optimization challenges (Kingma, 14; Loshchilov & Hutter, 2017). Adam (Loshchilov & Hutter, 2017) introduced key improvement by decoupling weight decay from optimization steps, restoring the original formulation of weight decay regularization. NAdam (Dozat, 2016) incorporated Nesterov updates into Adam, while AdaBelief (Zhuang et al., 2020) refined the second momentum
8

---

Published as a conference paper at ICLR 2026
0t track the EMA of (9t m,) improving generalization. Adan (Xie et al., 2024) added an extra momentum term for better training performance, albeit at the cost of additional memory usage. More recently, ADOPT (Taniguchi et al., 2024) innovated by folding normalized updates into first- order momentum updates. From an efficiency perspective, several approaches target memory cost reduction. AdaFactor (Shazeer & Stern, 2018) factorizes second-order statisties into row-column outer product, enabling sub-linear memory usage. K-Fac (Martens & Grosse, 2015) approximates the Fisher information matrix with a Kronecker-factored representation, supporting sublinear natural gradient updates. Techniques like fused gradient computation (Lv et al., 2023) further minimize memory costs. (Wang et al., 2024) concurrent work that focus on continuous learning setting. In contrast to these approaches, our proposed C-AdamW W introduces single-line modification to the widely used Adam optimizer. This modification not only retains the simplicity and efficiency of AdamW but also eliminates the need for hyperparameter tuning, the default parameters of AdamW suffice. Furthermore, while many of the aforementioned methods focus on optimizing or extending specific aspects of the Adam algorithm, C-Adam W is more generalit seamlessly integrates with all momentum-hee -asse optimizers, offering a general solution with minimal implementation effort.
Hamiltonian Dynamics Hamiltonian dynamics, rooted in classical mechanics, provides pow- erful mathematical framework for analyzing the motion of systems in continuous spaces. This perspective has gained traction in optimization, where the introduction of Hamiltonian principles sheds light on momentum-basee algorithms (Sutskever et al., 2013; Nesterov, 1983; Nguyen et al., 2024; Anonymous, 2024). Unlike Gradient Descent (GD), which ensures monotonic decrease in the objective function, momentum-based methods often follow non-monotonic trajectories, posing unique analytical challenges (Jin al., 2018). To address this, researchers have developed multiple Lyapunov functions for convex settings (Krichene et al., 2015; Wilson et al., 2016), providing a structured approach analyze convergence. (Sutskever et al., 2013) offered physical interpretation of momentum in optimization, linking it to the dynamics described by Hamiltonian mechanics, and demonstrated how these principles underpin classical methods like those of Nesterov and Polyak (Nes- terov, 1983). Furthermore, Hamiltonian dynamics have been instrumental in deriving convergence rates for accelerated methods (Jin et al., 2018) and, more recently, for advanced optimizers like Lion (Chen et al., 2023a) and its distributed variant (Liu al., 2024). In related vein, (Maddison et al., 2018) explored optimization methods from the perspective of continuous sizing their Hamiltonian structure. Mirror Descent, related framework, has been shown maintain efficiency estimates with a mild dependence on the dimensionality of decision variables, making particularly suitable for large-scale optimization problems (Tzen al., 2023; Krichene et al., 2015). These advancements highlight the versatility and depth of Hamiltonian formalism in bridging the gap between optimization theory and practical algorithm design.
ODEs, empha-
CONCLUSION
In summary, we introduce Cautious Optimizers, an enhancement for momentum-based optimizers that can be implemented with single line of code. Our theoretical analysis demonstrates that Cautious Optimizers not only preserve the convergence guarantees of the base optimizers but also accelerate the reduction of the loss function. Empirically, it delivers faster LLM pretraining and better accuracy on image classification. Finally, we suggest a few promising future directions: (1) Apply cautious optimizers to more settings such reinforcement learning and continuous learning; (2) masking in the eigenspace rather than the parameter space; (3) rigorous analyses how cautious optimizers strictly improve the convergence rate (empirically shown in 1).
6 ACKNOWLEDGEMMEN
This work was supported by the Amazon AI PhD Fellowship. and in part by the Institute for Foundations of Machine Learning (IFML) and the Office Naval Research (ONR) under Grant No. N00014-25-1-2354.

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.088774 2023.
Anonymous. Improving adaptive moment optimization via preconditioner diagonalization. In Submitted to The Thirteenth International Conference on Learning Representations, 2024. URL https: wwwpr/eeeeeeeeeeeeeeeeeeeeeeeeea net htt.:/eeettttttte///////e
under review.
Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Straight to zero: Why linearly decaying the learning rate to zero works best for llms. arXiv preprint arXiv:2502. 15938, 2025.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Lizhang Chen, Bo Liu, Kaizhao Liang, and Qiang Liu. Lion secretly solves constrained optimization: As Iyapunov predicts. arXiv preprint arXiv:2310.05898, 2023a.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al. Symbolic discovery of optimization algorithms. arXiv preprint arXiv: 06675, 2023b.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2302.06675. 2023c.
Xiangning Chen, Chen Liang. Da Huang, Esteban Real, Kaiyuan Wang, Hicu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. (rXiv:1803.05457:/1 2018.
Aaron Defazio, Xingyu Alice Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The road less scheduled. arXiv preprint arXiv:2405. 15682, 2024.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXhv:2010.11929 2020.
Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 2024. URL https:/ /ww.eeeodoooooooooooooooooo. www.oodo.ogg///coeooeooooo
2608602.
Gabriel Goh. Why momentum really works. Distill, 2017. doi: 2/2991///iisill.000006 URL
http://distili hut.:iil.ppbbplabe pub/2011 //mmmm/mmnuuummm
Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimiza- tion. In International Conference on Machine Learning, pp. 1842-1850, PMLR, 2018.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEFF/CV conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference Learning Representations (ICLR), 2021.
10

---

Published as a conference paper at ICLR 2026
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXhv:
2022.
Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The n+ implementation details of rlhf with ppo: A case study on tl; dr summarization. arXiv preprint arXhv:2403.1703/ 2024.
Chi Jin, Praneeth Netrapalli, and Michael I Jordan. Accelerated gradient descent escapes saddle points faster than gradient descent. In Conference On Learning Theory, pp. 1042-1085. PMLR, 2018.
Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https:
//kellerjordan. giierddnnnitthhs wttp://..pposssssssooss
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.698, 2014.
Walid Krichene, Alexandre Bayen, and Peter L. Bartlett. Accelerated mirror descent in continuous and discrete time. Advances in neural information processing systems, 28, 2015.
Kaizhao Liang, Bo Liu, Lizhang Chen, and Qiang Liu. Memory-efficient Ilm training with online subspace descent. arXiv preprint arXiv:2408. 12857, 2024.
Bo Liu, Lemeng Wu, Lizhang Chen, Kaizhao Liang, Jiaxu Zhu, Chen Liang, Raghuraman Krish- namoorthi, and Qiang Liu. Communication efficient distributed training with distributed lion. arXiv preprint urXiv:2404.00438, 2024.
Loshchilov. Decoupled weight decay regularization. arXiv preprint arXi:: 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05001, 2017.
Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101.5 2017.
Kai Lv, Yuqing Yang. Tengxiao Liu, Qi jie Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources. In Annual Meeting of the Associa- tion for Computational Linguistics, 2023. URL / httpa/pp.p..aammemmmm semanticscholar org/ CorpusID:2591999
Chris Maddison, Daniel Paulin, Yee Whye Teh, Brendan O' O'Donoghue, and Amaud Doucet. Hamiltonian descent methods. arXiv preprint 1rXiv:1809.05000 2018.
James Martens and Roger Baker Grosse. Optimizing neural networks with kronecker-factoree approximate eurvature. In International Conference Machine Learning, 2015. URL https: nt T cschola aareehorrrrrooro worg..oggooeoe Coarg//oopusDD...o 11D::::144455 480464.
//api. semaatt
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Yurii Evgenevich Nesterov. A method for solving the convex programming problem with convergence rate o (1/K² 2). In Dokl. akad. nauk Sssr, volume 269, pp.
1983.
Son Nguyen, Lizhang Chen, Bo Liu, and Qiang Liu. H-fac: Memory-efficient optimization with factorized hamiltonian descent. arXiv preprint arXiv::4006 2099988,0 2024.
Guilherme Penedo, Hynek Kydlek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information P Processing Systems, 37:308 1-30849, 2024.

---

Published as a conference paper at ICLR 2026
Powerinfer. Powerinfer/qwq---ogooo---- 2024.
Qwen. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https: /ttpw//m.gitttieee//ee.......... github. io/ /bllo////////ooooooooooo.o.ooooo / qwqq 32b poloppprevwww.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684- 10695, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal computer vision, 115:211-22 1522, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp.
PMLR, 2018.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139-1147. 147. PMLR, 2013.
Shohei Taniguchi, Keno Harada, Gouki Minegishi, Yuta Oshima, Scong Cheol Jeong, Go Nagahara, Tomoshi liyama, Masahiro Suzuki, Yusuke Iwasawa, and Yutaka Matsuo. Adopt: Modified adam can converge with any \beta.2 with the optimal rate. arXiv preprint arXiv:241 1.02853, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Belinda Tzen, Anant Raj. Maxim Raginsky, and Francis Bach. Variational principles for mirror descent and mirror langevin dynamics. IEEE Control Systems Letters, 2023.
Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.1332,, 2024.
Shaowen Wang, Anan Liu, Jian Xiao, Huan Liu, Yuckui Yang, Cong Xu, Qianqian Pu, Suncong Zheng, Wei Zhang, and Jian Li. Cadam: Confidence-based optimization for online learning. arXiv preprint ar X1x:22411.19644 2024.
Ashia C Wilson, Benjamin Recht, and Michael I Jordan, A Iyapumov analysis momentum methods in optimization. arXiv preprint arXiv:161 (00::::61.000000 1.0665,, 2016.
David H Wolpert and William G Macready. No free lunch theorems for optimization. 1EEE transactions on evolutionary computation, 1(1):67-82 1997.
Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1511 2024.
Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. Mars: Unleashing the power of variance reduction for training large models. arXiv preprint arXiv:2411.10438 2024.
12

---

Published as a conference paper at ICLR 2026
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Guoqiang Zhang, Niwa Kenta, and W Bastiaan Kleijn. Extending adamw by leveraging its second moment and magnitude. arXiv preprini arXiv:21 2021.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Pa- pademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-18806, 2020.
Liu Ziyin, Zhikang T Wang, and Masahito Ueda. Laprop: Separating momentum and adaptivity in adam. arXiv preprint arXiv:2002. 2020.
A
APPENDIX
The appendix is organized into the following key components:
We provide conditions 2.1 1 on the choice of function ensure that the system decreases both H and L simultaneously and Corollary 2.2 asserts that with a positive definite norm and differentiable Hamiltonian, bounded solutions of the discussed systems converge to stationary point the Hamiltonian.
Cautious Hamiltonian Descent:
Applications to Optimizers: In section A.3, the cautious framework applied to derive cautious variants of popular optimizers, including Adam, Signed Momentum, and Lion, highlighting their theoretical properties and practical advantages.
Discrete-Time Time Analysis: rigorous analysis connects the continuous- -tmme dynamics to discrete-time updates, providing convergence guarantees and bounds on loss reduction for cautious optimizers.
Pseudocode: We present the implementation details for cautious optimizers, focusing on the cautious Lion optimizer representative example.
Comprehensive details on the experimental setup, hyperparameters. and hardware configurations are provided, demonstrating the effectiveness of cautious optimizers in accelerating convergence and improving performance on large-scale tasks such as language modeling and masked autoencoder pretraining.
Experimental Details:
A.1 HAMILTONIAN OF COMMON OPTIMIZERS
We introduce the Hamiltonian functions of the common optimizers. Example A.1. Adam (Kingma, 2014) yields the following continuous-time form and Hamiltonian,
d um
d
ww:
di vm = a(V ((ow,) - wm:),
dl
Vou
vt: by b(VL(vw,)) vvz),
dt
1 um
with H(vw, vm, vv) = [(ww) +
vma
2a /ou+ +e'
We can show that d H(ww, om,, vvv) & 0 when a 2 b/4. dt Example A.2. The Lion-K optimizer (Chen et al., 2023b;a) (without weight decay) can be written into
d
vw, VKC((1 b)um, bVL(vw))).
dt d
di umn -a ( (L(ww)) vm))
13

---

Published as a conference paper at ICLR 2026
and K(vx) is any convex function that attains the minimum 0. One of its Hamiltonians that yields the Kamiltonian+deseee structure (Eq (13) in Chen et al. (Chen et al., 2023a]) is
where e [0.1]
H(ww, vm) aC(vw) + -
See Chen et al. (2023a) for other Hamiltonian functions. Lion-K includes a large family algorithms as special cases, including Polyka momentum, Nesterov momentum, signed momenium, mirror descent, Frank- Wolfe, etc.
A.2 CAUTIOUS DYNAMICS
We establish conditions in Theorem 2.1 for the function ø that ensure simultaneous decreases in both H. and L. Corollary 2.2 further asserts that, with positive definite norm and differentiable Hamiltonian, bounded solutions of the systems converge to a stationary point of the Hamiltonian.
Theorem 2.1. Following the dynamics in (5) in F. we have
d H(w,, 5.) ( (1 ((T.)) Ane 31),
dl
and
d [(w.) = Φ( || VV
dt
= = (1 - ((x)) dc. (w. 3t),
Here, AH, (wr, 8t) and &c, (w,, as defined in (3) and (4), respectively, represent the decreasing
rates of H and C in accordance with the original system (2). Hence:
If æ [11 <(x)) < 0 for any E Rd, then both H and L decreases faster than the original system:
d
H( H(m,, 81) -A, (we, 81) S O,
dt
d
[(w,) S -cc, 3t).
dt
If T [((x)) 2 for any Rd, then L decreases monotonically, _ ((w:) S... 0.
dl
Proof. For simplicity, write a V.(ww, v(((s.). Recall the definition of AH, (w,, 8,) and Ac, (w,, sz)):
Ax (w,, 8:) = [VV ()) + [[
Az 3,) = V.(w,) V + VL(w,)
Following the dynamics in (5), let us see the derivation of H. w.r.t. t:
d
H(W,, 31)
dt
VL(W.) w. + VK(33) 3t
VL(w.) ( (-VK((B)) -. + VK(3.) -V [|))
o - 1r -V<
-1 ((VL(W:) (1-
- | VVC(
..7 ((x:)) Au (1,,
14

---

Published as a conference paper at ICLR 2026
Given the fact that @ is an element- operator, it is noteworthy that if x (1- x(x)) 0,
then the first term in (9) 1 ((VL(w) o (3(3)) o (1 (VV((w) VK(3))) I 0 since we see V (K((3) as æ.
Next, let us look into the derivative of ((w,) w.I.t. t:
d
dt ((w,)
v((w,) ) VK( - (VL(())
VL(w,) (VK((3)) (VL( o VK( V((w)) T 2. (((())
- x. o(x))) - [V 12
(10)
(a: (1 (ba::)) A c. (wi, 8t).
(11)
evident that T ((x) 2 for any x E R then the first term in (10), E. (T,). satisfies
zz (TTe)) 2 0. Consequently, at ( S 0. This condition holds when x ((x) 2 0 for all æ E Idd.
It is noteworthy that if T (1-((x)) for any IRd, then the first term iI1), x. (1- (bax,)))
holds. Consequently, have ((w,) s -c. 8.).
Corollary 2.2. Assume that the norm || || is positive definite, ((()) 0, and that H(w s)
L(w) + iC(s) is differentiable. Then, the bounded solutions of the original system (2) converge to a stationary point of H(w, s). Similarly, the bounded solutions of (5) also converge to a stationary point of (ww, 8).
Proof. First, Let us look at system (2), we use LaSalle's invariance principle to find the conditions that the accumulation points (positive limit points) (w", s*) satisfy:
| )l 0. VVL w 1|ă
By the assumption that IR is positive definite and (0 0, have VK((s) 0. For positive limit points of systems (2), if VL(w") * 0, then the point (w*, s²)) ) is not positive limit point since
vL(w") V (VC((( v V((w") ) +
Thus, VL(w") = 0. Together with VIC(s") = 0, we conclude that (w", )*) is a stationary point of H(w,s) 8).
For system (5), the proof follows the same reasoning.
A.3 EXAMPLES OF CAUTIOUS DYNAMICS
In this section, we instantiate results Adam, Signed Momentum, and Lion, along with their cautious variants. While the standard methods are widely used, our focus is on introducing and analyzing the cautious versions of these algorithms to better understand their dynamics and stability.
A.3.1 CAUTIOUS ADAM
Adam: We begin by recalling the continuous-time dynamics of the Adam optimizer:
w. Vo, +E
dt
dt m, B1 (VV((w) - m.).
di V. B2 (V vt),
15

---

Published as a conference paper at ICLR 2026
where the associated Hamiltonian is given by:
1
Hl(w,, m, v,) = [(w,) +
m,
(12)
2
v +
The time evolution of the Hamiltonian can be derived as: d'H(wr, M&s vi) dt
32
m 02
VL(w,))
1
B2
me V + m&
43
3/2
4,31
+
For stability, we require
dH dt
S O, which leads to 31 7.
We now introduce a cautious variant of Adam. In this case, the update for is modified by introducing an indicator function that enforces alignment between the gradient and the momentum:
Cautious Adam:
d wi I(VL((r) m, 0) m.
dt
vu +
The dynamics for and remain the same. It is straightforward to verify that the loss function [(w,) serves as Lyapunov function:
dt ((w)) = ((ww)) IVLL(W: m, 0) m
S 0.
V +
Meanwhile, by Theorem 2.1, H(w my vy) = L(w,) + decreasing. Therefore, H. remains valid Hamiltonian for Cautious Adam.
28, Ve.te
m
my is also monotonically
A.3.2 CAUTIOUS SIGNED (POLYAK) MOMENTUM
Signed Momentum:
The update rule for Signed Momentum is:
w. -iign m.),
m: V((w.) - m..
The Hamiltonian for this system is:
H(w, m,) = L(w,) + |mm/ll,
with time evolution:
dH -I
di
Cautious Signed Momentum:
The cautious variant modifies the update rule for w, by introducing
an indicator:
ui - sign(m.) @ T(VL(un) m, 0).
The Lyapunov function remains the loss L(w:), with:
dl(w.)) VL(w.) @ T(VL(m)) m. 0)]l-
dt
By Theorem 2.1, H(w,, m.) = L(w,) + m is also monotonically decreasing. Therefore, H remains a valid Hamiltonian for Cautious Signed Momentum.
A.3.3
CAUTIOUS LION
Lion: The dynamics of Lion Chen al. (2023a) are given by:
m, a [((w) 7m,,, w; sign (,:),
where m, m. -aees-laaVii ((ww)) + mm,). The associated Hamiltonian is: H(w, m.) = aC(w.) + (1 e7)]|m
The time derivative is:
Mun (11 87) m || - milhi
16

---

Published as a conference paper at ICLR 2026
Cautious Lion: In the cautious version, the update for w, is modified:
Wr -iiggnn..) @ (())( m. 0).
By Theorem 2.1, M w,, my) = (((ww + (1 Ey) mella is also monotonically decreasing. Therefore, H. remains a valid Hamiltonian for Cautious Lion.
Meanwhile, the Lyapunov function remains (())), with:
d((ww, VL(w,, o [VV[(w;) m. > 0)]l
dl
A.4 DISCRETE TIME ANALYSIS
We analyze the discrete time case, demonstrating that cautious optimizers are at least as good as the original optimizers under mild conditions.
Theorem 2.3. Consider (7) and (8) with the loss function L(-) p--smooth Assume the element-wise operator ø satisfies
A(vx) == -ex Te - (-((v)) O.
starting from (w, 8:) = (W;, Bt), we have
((W++1) S L(w;-1),
which holds for step size e - 07 [[2 ||2
2.
where यτ, = u,0 and Vg. = VL(WW,)
Proof. First, we calculate the difference between wi+11 and We+1, then we use p-smooth condition bound the difference between L( ((m)-1) and [(W++1))
W T Wi+1 = -2 Th vØ,- un) -Er Uh vØ, Up.) = -T
Let us use smothh condition to bound the L difference -(w++1)-L(wx))
11 & |
< 6, (L(w+1)) 0F, + 2
= 6 (VL(w.) + V((W+++) - vL((e))) T
wT: + 11
2" or. 1|
= 6, V.W,)) vr, + l (VL(w,+1) vL(w;) vr, + 4 or 1l
2
पे. |l + H 2 cा, 1|
I e, VLW,)) प.. + | ((w++1) v<(w:)lI
//Cauchy Schwarz inequality
2
= e. UT: + 2 14.
+
/ 2 | 1
//g.. गे. S 0.
2
= - o v + E |म.
pe, || + - |
0. //By the choice of
Theorem A.3. Assume L(w) p-smooth and differentiable, and the element-wise operator
satisfies Φ(x) 2 0 as shown in Theorem 2.1. With (w, 32) following the update in (8) with constant step size E:
u. = u. (wv, 8t)
Wi+1 = we - Cth o VØ
8:41 8, U. (w,; 8t).
Assume have:
L(w) L(w"
[(..) Uzlla
BT,
T
Te
2T
where Br =E" |/2x.11 u0 argmin ((w)) and for notions, we write ||00|1 xx d(x)) Vnn
17

---

Published as a conference paper at ICLR 2026
Proof. Using the y-smoothness of [(w), expand ((w;-1) ((w)):
((w++1) ((w,) < ( Wt+1 - we) + "
Wt+1 - well
Substitute W:+1 W = - o v to get:
((w++1) - ((w)) $ c v((w)) (üe o voo) + 1c2
u2 a w
2
Simplify using the definition of |1-11 and I'llo,;
V (((w)) (U o vΦ.) = [[((w) o uull
which gives:
[(w(+1) [(w,, S - (I o oulllo + mc²
Summing over t I,..., T, we obtain telescoping sum:
T
([((w)) o U. W.lls) + ne T
[(wr+1) - ((w,)
[7,
112
2 t=1
Rearranging, dividing by Te, and noting L(WT+1)2 2 L(W*), we get:
L(w L(w
T [((w,) u.lls
BT,
Te
2T
where BT ΣT
This concludes the proof.
Theorem A.4. Consider updates (7) and (8), Assuming f(-) is j-smooth and the scaled step size EgOrK S and consider the following mask function:
vΦR ax.((L(wx)) ouk 2 vug vuk),
we have
[(Wk+1) S L(Wx).
which holds for any step size c 2
Proof. By smoothness, we have
[(w0++1)- -f(wwk)))))))))))
+ /
< [L(wR) (w++1- w
[1+11
Wx
2
- VAVVL(W ( (UOA T. +
k
T
2
2
(V((w)) (LL( ( T4K
μ1
Tu + VūR
I(VL(ww) Vüg
oür vū:)
2
2
2
Seaa
(VV((w)) VVV[TT."T[VVVVVVVV (I(VL(T))) :य
Uyk
T
I(VL(W)) FH 2 na
Tü TUg) o
2
0.
18

---

Published as a conference paper at ICLR 2026
A.5 INNER PRODUCT MASKS
Out of theoretical interest, let us consider a case of using an inner product mask:
UØ I 4Ok 0),
egk V ((()))
which yields a scalar that applies to the entire update vector. In comparison, the demenntwwiee mask in the paper treats each element separately.
This case is interesting because, on convex functions, the cautious optimizers always no worse than the base optimizers, regardless of the step size choices.
Theorem A.5. Assume L(-) is convex, and vox [ 2 0). Then, starting from the same point (wk, 8k) = (w): 3k), we have
[(w++1) & ((wx+1))
which holds for any step size Ek 2
Proof. If ug v ((wx)) > we have vo, 1, and Wk+1 = We+1, and [(Wx+1) = f(wx+1).
u VL(ws) S O, we have voy- = and Wx+1 = wi. By the convexity of L, we have C(wx+1) [(wx+1) f(ww euR) ((w))
2 -Eu VL(w)) >
This proves the result.
Corollary A.6. Consider the elementary test function:
1 ((w)
2 w
where Rdd is non-zero vector. This is commonly used as the testbed for optimization algorithms in theoretical analysis.
Assune uk and Uy are element-wise mappings. We have [(Wx+11 I ((wx+1) given (wk+ Sk)
(wk, 8t), with either the inner product mask vock T( ((( 2 0), or the element-wic wise mask
v = I(u V((w)) 2 0).
Proof. The inner product case is implied by Theorem A.5. For element-wise mask, since the loss an element- wise sum, and the update functions are element-wise mappings, which can apply Theorem A.5 on each element.
This argument, of course, can be extended to general convex and separable loss functions of form [(w) = Σ, (ws)..
Theorem A.7. Consider update (8). Assuming C(-) is y-smooth, and consider the following mask function:
VΦL aa.(VVL(W,. Tū 2 2
Tū 11
where og is scaling factor
We have
[(Wx+1) C(wx).
which holds for any step size 2 0.
19

---

Published as a conference paper at ICLR 2026
Ablation 6: To analyze the impact of cautious masking, conduct ablation studies different choices of ø. Our results show that the performance not sensitive to the specific choice of o, as all cautious variants incorporating these functions consistently outperform GDM. This underscores the robustness and effectiveness of cautious masking, as shown in Figure 3.
The tested functions are defined follows:
Φe = [( > 0) cI S 0),
Qinner T(x: y 0).
(13)
These results highlight that cautious masking enhances optimization stability and performance across diverse settings.
(w,)
109
10-8
GDM
Objective
10
C-GDM
10
cc= 0.01
10-32
c 0.1
10-40
Φ iinner
20
40
Step
Figure 3: This plot presents an ablation study evaluating the performance of various configurations. We compare the loss curves for different & choices. Across all tested configurations, the cautious GDM variants, de with 0.01, 0.1, and @rner as defined in (13), consistently outperform standard GDM with optimal hyper-parameter configration (Goh, 2017).
20

---

Published as a conference paper at ICLR 2026
A.6 PESUDO CODE
See Algorithm: 3 for the Cautious variant of the Lion optimizer.
Algorithm C-ILion Optimizer
Require: learning rate e, momentum coefficient 31,32 E [0, 1), weight decay factor ?
1: Initialize parameter vector un
2: Initialize t = O, my = =
3: while w, not converged do
4: I - 1 ++1
5: gt V Ww (w-(((w=1)
(Get gradients at timestep / {get the signed update) {update momentum I/ Compute alignment mask I/ Scale Ir, d is dimension of db
6:
+ (3gn(31 m + (1 (1)-+ .9)
7:
my 82m (1 (22 9t
8:
Φ: - T(u 9: 0) d
9:
& = c "pobl
we w;-1 0u
// Masked update
11: Wr f W; E
WWeghtt decay)
end while
B CAUTIOUS MASK STUDY
Figure 4 shows the ratio of active dimensions Ta = nnz(oug o Agg > 0)/dim(uu)) o vg.) across the training iteration & on LLaMA 100M. It see that it decreases from and stabilizes around 0.55. This suggests that the scaling factor generally lies within the range [1, 1.55). The stable ratio does not change significantly across different models.
Figure 4: The sparsity
ratio r(x) nazi x=
dix(k]
during pretraining of
LLaMA 100M the
Ratio
0.65
C4 dataset using the
C-Adam W optimizer.
The ratio quantifics the proportion of elements in representations over training steps.
2.55
500
1000
1500
Stops
C ADDITIONAL DETAILS ON LLM SCALING
The training sequence length is set to be 4096 and we perform coordinate-descent over discrete grids for all optimizer hyperparameters (Ir, weight decay, warmup, 31, B2, c, max-grad-none batch size). The following tables 5, 6 show the optimal set of parameters for each scale.
D ADDITIONAL DETAILS ON MINI-IMAGENET
Here you can find the hyperparameter search of the Mini- Mini-ImageNe experiments. The findings are consistent with our LIM experiments.
21

---

Published as a conference paper at ICLR 2026
Adam
C-AdamW W
Hyperparameter
130M
300M
520M
1.2B
130M
300M
520M
1.2B
Learning rate (Ir)
0.008
0.008
0.004
0.002
0.008
0.008
0.008
0.006
Weight decay (wd)
0.1
0.1
0.2
0.2
0.1
0.1
0.1
0.1
Warmup steps
2000
2000
1000
1000
2000
2000
2000
2000
0.9
0.9
0.9
0.9
0.95
0.98
0.98
0.98
32
0.98
0.98
0.98
0.98
0.98
0.98
0.98
0.98
1e-20
1c-10
1c-10
1c-05
1c-15
lc-25
1e-25
1c-16
Max-grad-norm
1
2
1
2
Batch size
128
128
256
256
128
128
256
256
Table 5: Optimal hyperparameters (from coordinate-desseett over discrete grids) for AdamW and C-Adam across model scales.
Hyperparameter
Range
Learning rate (Ir)
{0.001, 0.002, 0.004, 0.006, 0.008, 0.010, 0.012, 0.014, 0.016, 0.018, 0.020)
Weight decay (wd)
{0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4)
Warmup steps
(500, 1000, 2000, 4000, 6000)
81 $2
(0.85, 0.90, 0.92, 0.95, 0.98, 0.99)
(0.96, 0.98, 0.99, 0.995, 0.9995]
[le-30, le-25, 1e-20, 1e-15, le-10, le-5) {0.5, 1, 2, 4}
Max-grad gak-grad--omm norm
Batch size
{32, 64, 128, 256, 512]
Table Diserete grid for hyperparameter search
Eval Topl Comparison
adamwi I-50 04
cadamn I::e-004
ademm I-1w00
cademny I=1e-00 cip grads
Exd Tog
25
75
200 kpoch
125
150
175
200
Figure S: AdamW/C-AdamW learning search Mini-ImageNNt
22

---

Published as a conference paper at ICLR 2026
Evel
up op. In-20 04
captop, Ir=2e 04
Inite-0d
cappop
uprop.
capiip, Ir:le
aprop. rm:2--03
Erd 10g1
Figure LaProp/C-L.aProp learning rate search on Mini-ImageNet
Comparison
mars I::e- 04
cmars In:ee 04
In:1-:00
mars,
SITrr, In:e
End teg3
75
102
125
150
175
Figure 7: MARS/C-MARS learning rate search on Mini-ImageNNtt
23

---

Published as a conference paper at ICLR 2026
@ial_eval__e loss
c_ adamw 0.011 linear _iiearrdeecay _eecay
adamw. 0.011 _.0r__inearrrr__.._llla.a Jinear_decay
2.935
2.925
2.92
2.915
Step
150k
160k
180k
190k
200k
Figure 8: This experiment follows the same setup as the previous 100M Llama experiment in
E ADDITIONAL EXPERIMENTS
E. LEARNING RATE SCHEDULER ABLATION
We notice a concurrent work (Bergsma et al., 2025) suggests that reducing learning rate straight to zero with linear decay is beneficial. To preliminarily validate the robustness of cautious optimizer, we took the 100M model setup in our LLM pretraining experiment and swap in the new scheduler. We found cautious optimizers stably outperform baselines.
E.2 BATCH SIZE ABLATION ON 60M
We take the 60M parameter model and perform sweep on Batch Sizes and train for 1.2 billion tokens (1x Chinchilla) C4 and reporting perplexity (lower the better).
Batch Size (tokens)
24K
120K
600K
AdamW
38.5
37.2
41.7
C-Adam
37.2
36.2
40.6
Table 7: Perplexity of AdamW vs. C-Adam W across different batch sizes.
E.3 MUON (JORDAN ET AL., 2024)
We tested preliminary version of C-Muon (Fig 9) and found that results encouraging as follows. Although at the time of submission we are not able to verify it at larger scale, it could potentially be very interesting future direction.
E.4 POST-TRAINING LLM
To further test cautious optimizers in language modeling tasks, we perform two post-training experi- ments. First, we instruction-tune Qwen2.5-1..B-Innstte (Yang et al., 2024) the Powerlnfer/QWQQ LONGCOT-500K (PowerInfer, 2024), a dataset focusing on enhancing the model's 's reasoning abil- ities by data distilled from QwQ-32B-PP 3-B-Prvveew (Qwen, 2024). Then follow the experiment setting in (Huang et al., 2024) to perform RLHF alignment with PPO (Schulman al., 2017) on FleetheerA//prn AI/pythia-
A Ib-deduped.
24

---

Published as a conference paper at ICLR 2026
final_ Leee__elassss_
muon_ 0.001
c c_muon, 0.001
2.96
2.958
2,956
2.954
2.952
Step
150k
160k
180k
190k
200k
Figure 9: Following the same 100M model training setup in 1, we also preliminarily test the latest popular optimizer Muon Jordan et al. (2024) and find cautious improves upon the baseline.
Observation: In both experiments, find under same training steps and PPO episodes, cautious optimizers obtain lower training loss as well as higher rewards. This indicates potential of our proposed method beyond pretraining tasks.
Figure 10: (a) shows training loss on the task instruction finetuing Swee2.5-1.SB-instteee
Instruction
0.815
0.533
Acemel
on the dis- tilled dataset from QwQ for 500K question and answer pairs. (b) shows RLHF reward with PPO million episodes. In both cases, C-Adam W outperforms AdamW.
0.45
Adamte
0.62
Tra 0-1 Seis
Tuiling Gaboocs
E.5 PRETRAINING MASKED AUTOENCODERS (MAES)
Masked Autoencoders (MABs) (He al., 2022) have emerged powerful approach for pretraining Vision Transformers (ViTs) (Dosovitskiy, 2020) on lcrge-scale datasets like ImageNet-1K KK (Rus- sakovsky et al., 2015). This task involves reconstructing 75% of randomly masked image patches, challenging objective that requires extensive training over hundreds of epochs and millions of images. The primary goal is to learn robust visual representations that are generalizable across downstream vision tasks. The quality of these representations is typically measured by the final evaluation loss, which reflects how accurately the model reconstructs masked test images; lower evaluation loss indicates higher-quality representations. The results of our experiments summarized in Figure 11, where we compare the performance of the cautious optimizer against the Adam baseline.
AdamiN C-AdemiW
Figure 11: Evaluation loss pretrained MAEs on ImageNet on ViT backbone for 50 epochs, using AdamW and C- AdamW. Hyperparame- ters can be found in Ta- ble 8.
0.70
0.85
0.55
0.50
2000
4000
6000
8000
Global Stea
Observation: From figure 11, we observe that the cautious optimizer achieves lower evaluation loss faster compared to AdamW. This result highlights the effectiveness of the cautious approach
25

---

Published as a conference paper at ICLR 2026
Params
$1
Learning rate
weight decay
Batch Size
110 M
0.9
0.999
1.5x 10-4
0.05
4096
Table 8: Hyperparameters for MAE experiment. This experiment follows the optimal setup provided He al. (2022)
in improving the precision of reconstruction and, consequently, the quality of the learned visual representations.
F MISCELLANEOUS
The masking and scaling would incur some additional costs. However, in practice we observe that the impact minimum overall throughput in the Distributed Data Parallel setting. For our 100M model runs 16 GPUs, AdamW has token throughput of 579383 token/s, whereas C-AdamW has 551483 tokens/s, which is around 3% difference training throughput. Note that this is comparing our naive cautious implementation against the fused pytorch implementation.
As for other distributed training setting, such as tensor parallel, masking operation is element-wise hence not communication- bound, whereas scaling would require global statistics. However, the communication also minimum, since only a single floating-poin number (the local mean) needs to be all-gathered by other workers. Given modern GPU bandwidth, this should incur only minor overhead.
Table 9: Training efficiincy comparison between AdamW and C-AdamW.
Method
Wall-Clock (h)
Throughput (token/s)
AdamW 10.547 10.567
571,839.76 560,488.63
C-AdamW
26


Reviews:
[{"review_id": "6gkbYSUave", "reviewer": "Reviewer_TurQ", "summary": "This paper proposes a plug-in method for improving momentum-based optimization methods (such as AdamW) named as Cautious optimizers. This method applies a coordinate-wise mask according to if the proposed update direction aligns in sign with the current gradient. Theoretically, the authors analyze the method in a proposed Hamiltonian/Lyapunov framework. With smoothness-based arguments, they show that cautious optimizers preserve the base optimizer’s Hamiltonian descent, can ensure monotonic decrease of the loss and further accelerates it. Empirically, they evaluate the proposed method first on a 2D toy objective function, then on language and vision tasks. Consistent improvements on convergence rates and training performances are shown.", "questions": "Why is weight decay not included in cautious optimizers but performed after it? What if one uses other regularizers instead of the L2 norm, such as the KL regularizer in RL?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes cautious optimizers, a single yet effective trick for improving momentum-based gradient methods. The method is lightweight and easy to implement.  \n2. The authors argue the soundness of their proposed approach from a Hamiltonian/Lyapunov perspective, which might be of individual interest for future optimizer design.  \n3. Performance gains are shown empirically across different tasks, demonstrating the effectiveness and wide applicability of the method.", "weaknesses": "1. The LLM pretraining experiments use the same learning rate when comparing with original AdamW and Lion. This might not be a fair comparison since cautious optimizers use a rescaling factor $\\\\alpha$. Although the rescaling factor can be normalized, it is still hard to say if the same learning rate means the same thing for different optimizers.  \n2. The reported performance gains in Table 3 are modest, making interpretation sensitive to data and random noise. Reporting the number of random seeds tried would strengthen the claims.\n\nMinor comments: Typo in line 390: “catuious”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EZgGx3ZY2v", "reviewer": "Reviewer_CSiD", "summary": "The authors introduce cautious optimizers which is a modification applicable to any momentum-based optimizer where the main idea is to update parameters mainly when the proposed update direction and the gradient have the same sign. The authors show theoretically that it preserves the optimizer’s Hamiltonian structure and guarantees monotonic decrease of the loss under sufficiently small steps. Empirically, C-AdamW, C-Lion, etc. yield performance improvements in large-scale pretraining and image classification.", "questions": "Please refer to the comments in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea is quite elegant can be seamlessly applied to all momentum-based optimizers without introducing new hyperparameters. \n\n- The authors situate the method within the Hamiltonian Descent framework, which were used to analyze Adam and Lion. Theorem 2.1 and Corollary 2.2 offers a clear interpretation of cautious masking as energy-preserving damping.\n\n- The discrete-time results confirm that each cautious step decreases the loss more efficiently than the base optimizer under µ-smoothness.\n\n- The experiments show improvements without re-tuning hyper-parameters — showing that the cautious modification preserves hyperparameter stability.\n\n- Overall, writing in the paper is clear.", "weaknesses": "- The claim that cautious optimizers “do not get stuck at non-stationary points even when the update is fully masked out” is only partially justified. The authors state that momentum dynamics will eventually realign updates and gradients, but did not provide no empirical analysis of how long this alignment requires. Maybe exploring the failure cases near saddle points or flat regions (regimes that dominate the modern deep learning optimization) could help?\n\n- Empirically, the improvements are modest and sometimes fall within the noise range of large-scale training. For example, Table 2 reports ≤ 1 % perplexity gains at max. The comparison is also relatively limited i.e., Section 4 mainly discusses related optimizers like AdamW, Lion, but omits several recent and directly relevant baselines such as AdaBelief, Adan and SOAP even when these methods employ simple directional or normalization modifications. \n\n- the effect of the scaling factor introduced in Eq. (1) on the magnitude of updates and on the effective learning-rate distribution is not thoroughly analyzed. Since α directly scales updates in proportion to the ratio, could it change the convergence behavior in anisotropic curvature regions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yBQaInyo57", "reviewer": "Reviewer_4Ez6", "summary": "The paper proposes “Cautious Optimizers,” a one-line masking of momentum updates that zeros coordinates where the proposed update and current gradient have opposite signs, optionally rescaled by the active-mask ratio (Alg. 1). The theory is framed via a Hamiltonian+Descent view (continuous time) and per-step comparison results (discrete time). Experiments include a 2-D toy, Mini-ImageNet, and LLM pretraining up to 1.2B parameters.", "questions": "See comments 2 of weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes a simple mechanism (coordinate-wise sign check) that is easy to implement; clear statement that it promotes monotone decrease of the loss for small steps. \n\n\n2. Empirical results cover both vision and language, and generally show small but consistent improvements", "weaknesses": "1. The key idea of masking coordinates where the gradient and velocity have opposite signs is meant to promote descent. But requiring sign consistency on every coordinate feels too strict — usually, it's enough for the update direction and gradient to have a positive inner product. Also, in stochastic settings with noisy gradients, enforcing per-coordinate alignment could hurt rather than help. Plus, the paper motivates this with “monotonic decrease,” but per-step monotonicity isn’t necessary for faster convergence — e.g., Nesterov's method is non-monotone. This raises the question: is this motivation really essential?\n\n2. In Theorem 2.3 and 2.4, the discrete-time analysis relies on certain properties of the masking function (like Δ(vₓ) ≥ 0). But Algorithm 1 uses a non-smooth hard indicator (1(uᵢgᵢ > 0)) and a heuristic rescaling. These look inconsistent. Is this difference purely due to the hard masking being non-smooth? If so, it would be good to clarify the gap between what’s proved and what’s implemented.\n\n3. The toy experiment in §3.1 is just 2D and too simple. The LLM results in §3.2 only go up to 1.2B parameters, which is relatively small. To claim relevance to large-scale pretraining, results on models at 7B scale or above would be much more convincing.\n\n\n4. It feels odd to postpone related work to the very end (§4). This makes §2.1 hard to follow since many readers won’t be familiar with the Hamiltonian perspective. I'd recommend moving related work earlier or giving at least a brief summary in §2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fek5DrH77f", "reviewer": "Reviewer_G1tn", "summary": "The paper proposes a simple modification to momentum-based optimizers called Cautious Optimizers, implemented by masking update directions that disagree in sign with the current gradient. The authors argue that this one-line change ensures monotonic decrease in loss for small step sizes, preserves the Lyapunov/Hamiltonian structure of momentum dynamics, and empirically improves convergence speed and training stability. Experiments on toy problems, LLM pretraining (100M LLaMA on C4 and FineWeb-Edu), and vision tasks (Mini-ImageNet with ViT) confirm that the proposed modification can improve the performance of some popular optimizers.", "questions": "1. Could the authors address the concerns above?\n\n2. In Table 1, why were C-AdamW runs with learning rates 1e-4 and 3e-4 omitted?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a practically relevant problem.\n\n- The proposed method is very simple (a one-line change in PyTorch) and broadly applicable to many optimizers.\n\n- The experiments show consistent (but modest) gains in LLM pretraining, image classification, and toy settings, with improved stability and slightly better downstream performance.\n\n- The theoretical analysis tries to conceptually connect the masking heuristic to theoretical convergence guaranties.", "weaknesses": "1. Theoretical claims are not fully convincing. The results demonstrate that cautious optimizers can reduce the loss more than the original optimizers in a *single step*, but not throughout the full optimization trajectory. In line 84, the authors claim that \"Our theoretical analysis shows that the modified algorithm converges to local optima under mild conditions on the base optimizers\", but the presented results do not establish such convergence.\n\n2. Lack of stochastic analysis (the paper only considers deterministic gradient setting).\n\n3. Empirical improvements are modest. Most improvements (e.g., Table 2) are around $0.1–1$%, which may fall within variance across training runs.\n\n4. The abstract claims \"consistent speed-up\", but training time or throughput overhead is not reported (the results focus on perplexity and accuracy).\n\n5. No results are shown on other architectures (CNNs, RNNs) or non-transformer tasks.\n\n6. Some equations in the appendix overflow the page margins.\n\n7. Minor issues: several typos and grammatical errors (e.g., 'methods normally requires' in line 43, 'The follow is a comparison result' in line 240, 'catuious' in line 390, 'generalit' in line 444),  inconsistent boldface notation (e.g., equation (1)), formatting issues in References.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XxwGEevfyk", "reviewer": "Reviewer_uTNE", "summary": "The paper presents \"Cautious Optimizers,\" a simple, one-line modification for momentum-based optimizers like AdamW. The method only applies parameter updates when their direction aligns with the sign of the current gradient, preventing counter-productive steps. This change is claimed to accelerate model training (including for LLMs and computer vision) with minimum extra tuning on hyperparameters, all while preserving theoretical convergence guarantees under the Lyapunov analysis.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The change is trivial (a single line of code) and applicable to existing momentum-based optimizers.\n\n- The method boosts performance without the need for costly and time-consuming hyperparameter retuning.\n\n- The intuitive idea is supported by theoretical analysis, ensuring that convergence guarantees are maintained.\n\n- It demonstrates consistent speed-ups in various high-impact domains, including LLMs and image classification.", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)