A Physics-Inspired Optimizer: Velocity Regularized Adam
Authors: Pranav Vaidhyanathan
Lucas Schorling
Natalia Ares
Michael A Osborne
Venue: ICLR 2026 Poster (2026)
Keywords: Optimization in deep learning
physics-inspired
edge of stability
Primary area: optimization
TL;DR: We introduce Velocity-Regularized Adam (**VRAdam**), a velocity penalizing optimizer for deep neural networks.

Abstract: We introduce Velocity-Regularized Adam (VRAdam), a physics-inspired optimizer for training deep neural networks that draws on ideas from quartic terms for kinetic energy with its stabilizing effects on various system dynamics. 
Previous algorithms, including the ubiquitous Adam, operate at the so-called adaptive edge of stability regime during training, leading to rapid oscillations and slowed convergence of loss.
However, VRAdam adds a higher order penalty on the learning rate based on the velocity such that the algorithm automatically slows down whenever weight updates become large. In practice, we observe that the effective dynamic learning rate shrinks in high-velocity regimes, and damping oscillations. By combining this velocity‑based regularizer for global damping with Adam’s per‑parameter scaling, we create a powerful hybrid optimizer. For this optimizer, we provide rigorous theoretical analysis of operation at the edge of stability from a physical and control perspective for the momentum. Furthermore, we derive convergence bounds with the rate $\mathcal{O}(\ln(N)/\sqrt{N})$ for a stochastic non‑convex objective under mild assumptions. We demonstrate that VRAdam exceeds the performance against standard optimizers including AdamW. We benchmark various tasks such as image classification, language modeling, and generative modeling using diverse architectures and training methodologies including Convolutional Neural Networks (CNNs), Transformers, and GFlowNets.

Published as a conference paper at ICLR 2026
A PHYSICS-INSPIRED OPTIMIZER: VELOCITY REGU- LARIZED ADAM
Pranav Vaidhyanathan 1.3x Lucas Schorling Natalia Ares Michael A. Osborne
University of Oxford. United Kingdom 3 ((pranav, mcsb)&robots. ox.ootsssox.aa ox. ac.uk
o [lucas. schorling, natalia ares @eng. ac.
ABSTRACT
Adam (VRAdam), a physscc--isspeee inspired opti- mizer for training deep neural networks that draws on ideas from quartic terms for kinetic energy with its stabilizing effects on various system dynamics. Previ- ous algorithms, including the ubiquitous Adam, operate at the so-called adaptive edge stability regime during training, leading to rapid oscillations and slowed convergence of loss. However, VRAdam adds higher order penalty the learn- ing rate based on the velocity such that the algorithm automatically slows down whenever weight updates become large. In practice, observe that the effective dynamic learning rate shrinks in high-velocity regimes, and damping oscillations. By combining this velocity-based -based regularizer for global damping with Adam's ss per- per-parameter scaling, we create a powerful hybrid optimizer. For this optimizer, we provide rigorous theoretical analysis of operation at the edge of stability from a physical and control perspective for the momentum. Furthermore, we derive convergence bounds with the rate (In(N)/VN for a stochastic non-convex ob- jective under mild assumptions. We demonstrate that VRAdam exceeds the per- formance of standard optimizers including AdamW. W. We benchmark various tasks such as image classification, language modeling, and generative modeling using diverse architectures and training methodologies including Convolutional Neural Networks (CNNs), Transformers, and GFlowNets.
We introduce
INTRODUCTION
Optimizing the parameters of deep neural networks remains comnertonn of progress in machine learning. Improving on the core idea Stochastic Gradient Descent (SGD) (Sutskever et al. 2013), adaptive methods like Adam (Adaptive Moment Estimation) (Kingma & Bal 2015) have become ubiquitous due to their practical effectiveness across diverse tasks and architectures. Despite its suc- cess, the performance of Adam can be sensitive to hyperparameter choices and its training dynamics can exhibit instabilities (Reddi et al. 2019). Furthermore, fully understanding the training dynam- ies of deep neural networks remains an open challenge (Wang & Choromanska 2025). and even small improvements to existing optimization algorithms can often lead to significant reductions in resource consumption.
One line of work, observed empirically, is that training often occurs at the edge of stability (Co- hen et al., 2022; 2024p, a regime for which the largest eigenvalue, also called sharpness, of the loss Hessian equilibrates around a fixed value proportional to the inverse of the learning rate (LR). This seems in contrast with common presumptions in classical optimization theory and has profound im- plications for convergence speed, stability, and generalization. In classical optimization, higher LRs lead to faster convergence at the cost of oscillations or divergence if stability constraints (depending on the loss landscape) violated (Boyd & Vandenberghe 2004). This pathological behavior of optimizers, like AdamW, leads to instabilities and a slowed decrease of the loss.
"qqual Contribution
1 "ooresponniig author

---

Published as a conference paper at ICLR 2026
These challenges motivate the exploration of alternative optimization strategies. In line with the ori- gins of machine learning itself (Hopfield 1982 Ackley et al., 1985), one promising avenue draws inspiration from physics. For that, the optimization trajectory is conceptualized as discretized motion of a particle within the high-dimensional loss landscape. Both from the structure of the "po- tential" landscape and the discretization, instabilities may arise from excessive "velocity" or overly large step sizes. This perspective suggests that mechanisms from high-energy and non-classical physics applied optimization can improve aspects of stability. Building upon the established suc- cess of momentum, which incorporates velocity into gradient updates, recent ideas have explored maximal velocity, drawing parallels to the speed of light in the theory of special relativity (França et al. 2020
This work introduces class of physics-inspired optimizers, termed (VRAdam), designed to improve upon the stability and performance of standard adaptive methods (Kingma & Ba. 2015] [oshchilov & Hutter 2017). Inspired by quartic terms used to model kinetic energy in more stable systems such as classical time crystals (Shapere & Wilczek] 2012) and heavy quark modeling using non-relativistic quantum chromodynamics (NRQCD) (Braaten 1997) known for their unique stability properties, VRAdam adapts this heuristic and introduces novel reg- ularization mechanism. This mechanism controls the effective learning rate via penalizing high velocity, namely V) o0/(1 min(8s||o:112:.0
Adam
Equipped with this new optimizer, VRAdam, we probe its dynamics at the adaptive edge of stability, observe faster convergence, and analyze its sharpness empirically against AdamW and Sharpness Aware Minimization (SAM) (Foret et al. (021). We also introduce rigorous theoretical analysis of the global uniform exponential stability of momentum (Weber et al., 2024 as well a physics- inspired Lyapunov candidate derived from our Lagrangian that demonstrates stability properties. With well-tuned hyperparameters, we benchmark VRAdam against AdamW, RAdam (Liu et al., 2020). SGD with Nesterov momentum and RMSProp (Ruder 2017) on image classification with CIFAR-10 dataset and the convolutional neural network architecture, on language modeling with transformers on the WikiTexx2 dataset, and a generative modeling task with GFlowNets and report improved performance on all tasks. We also report increased performance against AdamW on large scale training for language models such GPT (Brown al. 2020) with marginal computational increase in overhead.
With this work, we contribute:
VRAdam, a physics-inspired and interpretable modification to AdamW,
Adaptive edge of stability analysis of VRAdam with faster convergence and associated empirical and theoretical evidence from momentum analysis,
Convergence bound for non-convex stochastic objective,
VRAdam outperforms A and other optimizers on wide range of benchmarks.
2 BACKGROUND
Edge stability. The training of deep neural networks does not follow classical optimization trajec- tories when trained with full-batch gradient descent (GD). However, during training, these models experience a surprising phase called the edge of stability (EoS). In this phase, the loss Hessian's 's largest eigenvalue (Aaax) rises to approximately 2/ 2/7,, the numerical stability limit determined by the learning rate At EoS, the eigenvalue persists at this threshold, causing short-term, non-monotonic oscillations in the loss function. Despite these oscillations, the model still achieves long-term de- scent in the loss, though at the cost of slower convergence (Arora et al. 2022 Cohen et al. 2022).
More recently, empirical bounds on the adaptive edge of stability (AEoS) have been observed for adaptive optimizers such as Adam (Cohen et al. 2024). Here, the relevant stability threshold in- volves preconditioning the Hessian He. where the precondition is constructed from the exponential moving average (EMA) of past dementt-wsee squared gradients my:
P- P. diag (ymi E),
2

---

Published as a conference paper at ICLR 2026
This adaptive preconditioning coincides with the learning rate scaling in Adam (compare Alg. and scales down the step size in high-variance (typically high-curvature) directions as well as scales up in lawvacri ones. Since the local stability of an optimizer around minimizer depends on the eigenvalues of the quadratic Taylor approximation [(x) R 2 Hx, Adam's 's dynamics are shown to be stable (Cohen al. 2024) as long
+ 2
38 7
max H.) (1 B1)
(31 = 0.9).
However, as this threshold is attained, the adaptive oscillatory regime slow final convergence. as the optimizer continually adjusts its preconditioner to maintain stability as well (Song & Yun 2023).
Physical origins of exotic Lagrangians. To better comprehend and navigate the edge of stability regime, we can draw inspiration from the physics governing complex optimization scenarios. The deep insights provided by the interplay of physics and machine learning frameworks have been demonstrated in various scenarios, such the improved interpretation of Neural Tangent Kernels (NTK) through Langevin dynamics (Avidan et al. 2025). One central concept in physics is the Lagrangian C(x, v) T(v)- (z)) of a system, which is a function of position and velocity v and (lypically) defined as the difference between kinetic energy 7 and potential energy V from which the equation of motion can be derived via the Euler-Lagrange equation. In this work, we investigate non-standard Lagrangian formulations of physical phenomena with excellent stability conditions.
For example, the stability of seemingly disparate quantum systems like heavy quarkonia (described by NRQCD) and classical time crystals share conceptual parallels rooted in higher-order velocity terms. These terms fundamentally reshape energy landscapes by creating non-standard dispersion relations, establishing invariant submanifolds in phase space where stable configurations emerge as attractors or limit cycles. Further information regarding these systems are found Appendix
METHOD
To translate this physics insight into an optimizer design, we identify the stabilizing aspects of such phenomena such as the heavy-quark momentum with the optimizer's global momentum buffer v. Accordingly, we posit kinetic energy of the form: TvRAdam (v) ||o the mass and A3 is a tunable parameter. The Lagrangian then becomes
+ es ||=]| where m is
C(x, e)
m 2
B3
V(x),
(3)
4
for which solve the Fuler-Lagrange equation 8c network with parameters x can be thought of as the potential landscape (Holderrieth et al.) 2024).
8C
We know that the loss of neural
0z
such that 8c (z) - Lioss (z).
Dz
The Euler- Lagrange equation then becomes
[m + Balle|13) e| (Loss di
(4)
which can be rearranged to
V LLoss (aouss(x)/// (x)/ 3Bg||e113),
(5)
where the dot corresponds to the time derivative. Rather than explicitly constructing an optimizer based ordinary differential equation solver for Eq. 5 (via discretization and introduction of dissipation, e.g. |rança et al. (2220)), we utilize the term, (1/(m + 3,3 [o](2), and embed it as dynamic learning rate into AdamW. W, enhance the successfully proven properties. Note that this expression from Eq. iSis derived for the 1-D case, where the dynamics é collinear with v. In particular, this approach avoids choosing specific forms associated with various different integrators. The full velocity regularized Adam is given in Alg. where we highlight in blue the changes to AdamW.
In Fig. [ the vector field in Eq. 5 is plotted for the case of the loss function being a simple quadratic. Compared to kinetic energy term without quartic velocity, the vector field is "squeezed"
3

---

Published as a conference paper at ICLR 2026
in direction, and the resulting trajectories not circular. In this idealized setting, alize the performance of VRAdam and Adam simplified to VRMomentum and Momentum, which corresponds to Alg. I without second-order moment estimates or bias corrections (setting m, and dropping lines 6, 8, and 9). After the first step, VRMomentum has a lower step size than Mo- mentum, which leads to fewer oscillations. Through reparameterization and modification, we obtain
visu-
Figure 1: Vector field i -1 +1++1+++))) 3v²) and derived by solving the Euler-Lagrange equation for L 2² /2 v /4 x² Black lines contin- uous trajectories; blue/orange show VRMomentum Momentum steps.
Algorithm VRAdam optimizer. f(0): objective function; B1, B2 E [0, 1); uz: velocity estimate; my: second-moment estimate; Te: dynamic learning rate at step t; ag: maximal learning rate; a0o(1 + ):): minimal learning rate; 83: velocity penalizer. Code available
Phase Space Diagram with quadratic potential
Vector fowat, dvieti
Vaidhyanathan et al. (2026).
Continunus Trajectonees
Komeetur
1: Input: f(), Po. On. 01, 31, 82, A3.
Vilkamerzuri
e,
2: Initialize uo 0, mo + 0
3: for t = 1....,, do
4: 9i <- Vf(
5: Ct + 310;-1 + (1 (1- 31)gt my 32m4-1
Welcctty
6:
B2) TIt + 00/(1 (1 +minn(3alln an))
7:
8:
& + u2/ 10 -11))
9:
mi my -35)
0t
10: 8g + 1 )1 -
Vm=
end for
12:
Output: 0T
for timestep for VRAdam, where Og and (11 control the maximal and minimal LR respectively, and 83 controls the strength of the velocity penalty. This is inspired by the bound introduced to v² in physical setting as discussed in Appendix[A] The parameterization of LR, compared to the physically derived one, clips the velocity avoid getting stuck gradients and therefore velocity become large. Weight decay applied in the traditional manner. An implementation of VRAdam is available at Vaidhyanathan et al. (2026).
the dynamic learning rate 7), an/(1 +
4 ANALYSIS
4.1 EMPIRICAL ANALYSIS
For this analysis, following Cohen et al., (2024), we train ResNet-32 architecture CIFAR-10 for an image classification task, with VRAdam, Adam and SAM. The training is stopped as soon as the training loss falls below 0.1 or an accuracy of 0.97 reached. In Fig. [ (a) and (b), the training curves of VRAdam indicate faster convergence in both minimizing training loss as well as maximizing training accuracy as compared to Adam and SAM, known for sharpness minimization. When juxtaposed with the sharpness comparison depicted in Fig. 2 (c), we observe that the maxi- mum eigenvalue (sharpness) of the preconditioned Hessian of the loss, remains adaptable to faster convergence due to the dynamic learning rate adjustments induced by VRAdam.
The effective learning rate of VRAdam is shown in Fig. 2 (d). During the first 25 iterations, the learning rate dynamically decreases before increasing close to the maximal value allowed LR we converge to the minima. The bound on the minimal LR not active in this example, while the base LR of Adam stays constant throughout training. As described in the seminal work of Schmidhuber et al., we note that minima with lower sharpness are associated with better generalization (Hochreiter & Schmidhuber) 1997 Foret et al.) 2021). We can also observe, that VRAdam's 's dynamic learning rate quickly moves to the maximal LR to exploit the loss landscape optimally, while exploiting the trade-off between the adaptive edge of stability and faster convergence.
4

---

Published as a conference paper at ICLR 2026
(a)
(b)
Training Loss Comparison
Training Accuracy Comparison
0.6
VRAdam SAM
VRAdam
SAM
0.5
0.3
0.2
125
175
130
(c)
Iteration
(d)
Iteration
Sharpness Comparison
Learning Rate Comparison
1e4
1e-3
2000
175
Eigenvelue Magnitute
1:3
1:2
Leaainn
103
0.73
Adam foxd
Adamm max eigenvalue max eigenvalue SAN max elgenvelue
VRAdam dynamic
025
min
SAM rixed
-1
25
75
125
350
175
25
50
100
125
150
175
Iteration
Iteration
Figure 2: (a) Training loss curves for VRAdam, Adam, and SAM Forret et al. 2021 of ResNet-32 on CIFAR-10 (b) training accuracy curves (c) plot of maximal eigenvalues of the loss Hessian (d) effective learning rate during training. Hyperparameters for these plots are provided Appendix
E.4
4.2 STABILITY OF VRADAM
We analyze the behavior of VRAdam in the adaptive edge of stability regime compared to that of Adam in momentum ablation setting without parameter-scaling based on second order moments estimates, bias corrections or weight decay. By simply adding weight decay, the admissible base step is constrained, the rest of the proof remains follows. We first give quadratic warm-up (Theorem (.1) as the main result of this section; immediately after, we state a nonconvex corollary that replaces the global convexity assumption by trajectory-level curvature bound together with an analytical shift (decoupled weight decay).
Objective function with minimal curvature assumption. Along the realized trajectory, the Hes- sian satisfies a two-sided spectral bound
-m I 3 f(,.)) = LT for all iterates 62, for some m 2 0, L > O. We include decoupled weight decay of strength À 2 0 in the update (Alg. 1, line 10). As a standard simplification, we analyze first the quadratic model
(6)
S(0) - 10 H(O 1000-000000000 0')
where H is a positive definite Hessian matrix. Let It = 0g - 0" be the error. The gradient is, Pr (f( Hx,-1- Let f Amin (H) and L == Amax (H). From Algorithm 1 we know
that 7/ E H = [Tmin, (oo] and Mmnn = a0/ (1 a1) > O. Let B1 from Algorithm be 3 for this analysis. The following theorem and proof provides sufficient but not necessary conditions for stability in discrete time. Thus proving a more restrictive case of the optimizer.
Theorem 4.1 (Uniform exponential stability of VRMomentum). Consider f(0 with 0 H LI. Let VRMomentum be 3 E 10, B3 > O, ao > 0, On E (0, oc), and set X = 0 in this warm-up (no weight decay in the matrix recursion). If Fo < B(3) 2(++9) or Mrain L < B(3) if the LR clip 1-3
is active, then for any realization of {ne) generated by the gate n = av (1 + min( 1l the origin is a globally uniformly exponentially stable equilibrium. Moreover, there exists Com- mon Quadratic Lyapunov Function (CQLF), V(z) = 2 Pz with P > such that V(2)) S
(1 - <) for some E (0,
5

---

Published as a conference paper at ICLR 2026
Proof. Define the 2d-state 24 (zt, U1) and the parameterized update matrix
A(r) (
T - 7 3)
-n3T) 31
A(m))=11
(8)
(1-
Let H TT diag(h:) with 0 μ S h S L. In the eigenbasis of H, the dynamics split into identical 2 x subsystems that share the same scalar and differ only by the curvature / E [a,, LJ:
& 0/
{t-1 Ut-1
An = (1 (1 71 3)h
-3
- (n))
(9)
(1-8)h
Hence suffices to build a CQLE for the family {Aa(n) : E [M, L), n E HJ.
For fixed (h, n),, the characteristic polynomial is
x - (1 n11 8)h)A = 0, so the Schur criterion gives stability iff 17(1 B3h < 2(1 + 3). In particular, if
(10)
og < B = 2(1 () 3)
(11)
1-3
then every A, (n) with r E H Schur-stable. (This is the Adam/AEoS bound specialized to P (Cohen et al. 2024))
If, instead of the quadratic model, we assume the trajectory-level curvature range h E [-m, LJ and include decoupled weight decay À > 0 in the update, the same 2 2 calculation yields the uniform sufficient condition
m and
ao - B)L + (1+ () 2(1 (11+1) B).
(12)
0 recovers eq. Setting À 0 and
We now produce such that
An(r)) T PA,(T) - 1 -el2 Vn E H, vh E Ju, L.,
(13)
for some O. Take the block-diagonal, curvature-agnostic form
P = diag(pi I2!)), P1 > 0. 0.
(14)
which lifts to Pa = (Pi Ja. P2 P2)) 2d dimensions.
For fixed (h, 17), set
AA
A12
T Ann,h A PAN()) - P
(15)
A
Az2
with
(n (1 (1-8)h 22p + (1- 31 (p1r)² + +p2)
(16)
(22 h) pin2 32 33 1),
(17)
A (t) -Pim3(1 7 3)h) +pp3(1 - -3),.
(18)
We first force strict negativity on the diagonal, uniformly over 1J H and [u, L). Using the
bounds E [/min, oo] and h E [u, L). sufficient pair of conditions for An S -ò1 and A22 S -
2mmin Pi
CO P2)L
(19)
upper hound for the h -temm
lower bound for the linear h-term
P2 - B2)
(20)
negative term magnitude
pusitive term bound
(Here used the worst cases 1) = og and h
I. wherever they make the expression largest.)

---

Published as a conference paper at ICLR 2026
Intersecting eq. 19 20 with P1 1 gives nonempty interval iff
0g.32
2//in 2
0 < 2(1 + 3) Muiin
<
(21)
1 32
(1
B)L
Indeed, set pt and choose any
0g 32 (1- B)L
2/min
P2 E
(22)
Next, to pass from diagonal negativity to matrix negativity, we use the Schur complement:
A
A i -12 + A1 S -, 22
-6.
(23)
A
Under eq. 19
20
Au and A22 are uniformly < - for some ô > 0. A direct (but routine) bound
gives, for all 17 E M. h E [,, L).
ppica + p2(1 3)
(24)
A
21minP1 - (oăP) + p2) -3)
and the right-hand side is strictly smaller than - A22 when eq.
19 20
hold with slack. Thus there
exists é > 0 such that eq. 13 satisfied.
For fixed h and P 0, the map
(n) PA 7ー An)))
(Ao nA1) PPAo + nA1)
(25)
matrix-convex in y (its second derivative is 2 PA1 č 0). Therefore, if the inequality
A (1) PAxx)) P 3 -el2
(26)
holds at the endpoints 17 = Mmin and r = ag, it holds for all rj [Mmin an). Under eq. 21 the choice eq. 22 does exactly that.
The block choice Pa diag(p: P2Ta) certifies
A(1) Pa Pa 3 eeTaa An H,
(27)
hence the quadratic Lyapunov V(2) = = T yields
Paz
V(+) S (1 (x-1), c= Amax (Pa)
E
E (0, 1),
(28)
which implies global uniform exponential stability and concludes the proof.
Corollary for nonconvex stability via analytical shift. Assume along the VRAdam trajectory that -mI = V2
LI and include decoupled weight decay À If eq. 12 holds (with do or 7mmn), then every x 2 eigen-direction block is Schur-stable uniformly over h E [-m, L] and " H. hence the origin is globally uniformly exponentially stable. (Setting À 0 and m reduces to eq.
II)
In the Appendix an alternative Lyapunov candidate based Lagrangian physics is presented.
4.3 GLOBAL PER- PARAMETER LR ADAPTIVITY
We contrast two ways of modulating the step magnitude, namely, (i) per-parameter control via time-varying diagonal matrix (element-wise scaling), and (ii) a global scalar gate ne that multiplies the entire (preconditioned) step in Alg. [ The global gate introduces key advantages in the edge-of-stability regime.
Uniform Lyapunov stability under arbitrary scalar switching. In the momentum ablation
quadratic, the global-gated dynamics decouple into identical 2 x blocks Ax(1) in the eigenbasis of the Hessian (see the construction surrounding eq. 13 eq. 14 and the Schur bound eq. ITI. Theo-
rem 4.1 produces a pervature-agnostie Common Quadratic Lyapunov Function V = Pz with P
independent of h and the time-varying 7/. [1/inn certifying global uniform exponential

---

Published as a conference paper at ICLR 2026
stability whenever agL 2(1+81)
1-8 (or the same with Mmmn when the clip is active), i... the Schur condition eq. 11 Hence stability holds for any scalar gate sequence generated by Alg. 1 (See
Sec.
Global, rotation-invarian control of AEoS with bounded steps. The velocity- based gate in
Alg. [ derived from the quartic Lagrangian eq. 3 and the collinear 1-D reduction eq. [ implies
dimension-free bound on the update norm and raises the instantaneous stability threshold when- ever the measured velocity grows:
|00 10e-00-10 - 1/ mell - Or
2(1 +1)) = LESs(T) (1 31)m (1- (1-31)ay
2(1+ (1+min(3sloxal,aai)
2/B3'
(29)
Eqs. (45)-46). Thus the method automatically retreats from instability velocities spike near AEoS and prevents runaway steps. Because the gate is scalar, these guarantees are or- thogonally invariant and do not require the adaptive preconditioner to commute with the Hessian. This mechanism aligns with the reduced ringing and sharpness observed empirically.
see App. B
Avoiding switched-anisee anissoropp instabilities. Allowing the step to vary per coordinate corresponds to a switched linear system with state matrix A(D.), where D, is a time-varying diagonal scaling. Even if each fixed D yields Schur-stable map (spectral radius 1), products such A(D2) ((D2))(())) can be unstable because the maps generally do not commute and the contraction directions rotate across steps. A scalar 1); eliminates this failure mode: all directions are scaled identically, and the CQLF from Theorem 4.1 guarantees contraction under arbilrary scalar switching, again under the same Schur bound eq. eq. cq.
4.4 CONVERGENCE ANALYSIS
In this section, we provide convergence analysis of VRAdam by extending the work of Défosse7 al.] (022) on Adam. The main result probabilistic bound that VRAdam converges expec- tation to stationary point for nonconvex (and therefore also convex) objective function under set of regularity assumptions and simplifications.
Setup. The objective function is given by F(0): Rd + R. V are gradient estimates of F such that f
VF. The random variable TN describes an index with probabilities given by F/[ = jl x 1 - for j E {0, L,.,,. N - 1}. The constants amii ao/(1 + a1) > 0 and Omax 0g > 0 correspond the minimal and maximal possible learning rate every step. F is bounded from below, namely V0, F(0) > F.. The gradient estimates are almost surely bounded in the l namely ||
R Ve. V is L-Lipschitz-comminuuus in the Ez-norm, namely v0, σb F(O) ( l| < | 2122. The weight decay parameter À = 0 is set zero. The bias correction for the velocity 0t +- v1 - -8)) is not included. The hyperparameter 31, 32 fulfill 0 < 32 < 1,0 0 < B1 32 and the number of iterates satisfies N > 31/(1 - 31).
Theorem 4.2 (Convergence of VRAdam). Given the six assumptions above, the iterates 0g of Algo- rithm fulfill:
[F(0))] S 28 F(06) - F. amin
)- + B 1 In(11 (1- (1-1330 N
N In(32))
(30)
with Ñ
8 T-81
and
maa dRL(1 Bi)
12 am dR² /I- Bi
2a d dL?
E
+
+
(31)
amin (1 B1/82)(1 32)
Omin - 81/32) -1-12 Ominn (1-in1-841 81/32) -1- 3//3/22
A proof can be found in the appendix C The right-hand side of the bound converges zero in the limit of N - if OminnN] N-11/2 Omazx( N-1 and 1 (2NN N to leading order. For this scaling of hyperparameters, VRAdam has the same convergence rate of (In(N//VN as Adam. While this convergence result provides a strong theoretical foundation, benchmarking is key to understanding performance under realistic conditions.
8

---

Published as a conference paper at ICLR 2026
5 BENCHMARKS
We benchmark VRAdam against various optimizers on diverse datasets, tasks, and architectures. These benchmarks include image classification using CNN on the CIFAR-10 10 dataset (Krizhevsky
Hinton 2009), language modeling with Transformer architecture (Vaswani et al.
2023) the
WikiText2 dataset (Merity et al., 2016), Generative Flow Networks (GFlowNets) (Bengio et al.,
2021 on grid world for sequence generation tasks (Chevalier-Boisver
al. 2023 and training
large language model (LLM) such as the Generative Pretrained Transformer (GPT) (Brown et al.
2020j. These benchmarks represent broad variety of deep learning architectures and tasks, ranging from older techniques for image classification much newer and actively emerging models and training techniques. We focus on the comparison with Adam since its exceeding popularity and consistent performance but also report the performance of stochastic gradient descent (SGD) with momentum QQia 1999). root mean square propagation (RMSProp) (Ruder) 2017) and RAdam (Liu et al., 2020 for all tasks except for the LLM benchmark. In particular, demonstrate the excellent convergence properties as well as performance improvements in most benchmarks for VRAdam (Table D Note that we run Bayesian optimization for hyperparameter sweeps for the image classification, and language modeling tasks, with several evaluations in close proximity to the found minimum, and the results are statistically significant. We report the optimal value from this optimization procedure, however, we include shaded error bars in the appendix. The Bayesian optimization for the hyperparameters was conducted with the objective to minimize validation loss, however, we also report test loss. Hyperparameters for the task involving GFlowNets were picked at random due to compute constraints. All details regarding model training, setup and datasets along with error envelopes using multiple runs are provided in Appendices and
Method
WikiTeex--2 2 Loss
CIFAR- 10 Loss
GridWorld Flow Matching Loss
Validation
Test
Validation
Test
Validation
Test
VRAdam
5.99
6.00
0.476
0.469
1.25
1.33
AdamW
6.47
6.50
0.522
0.565
2.41
3.60
RAdam
7.511
7.554
2.300
4.005
1,407
2.290
SGD+Nesterov
NaN
NaN
0.625
0.620
2.71
2.61
RMSProp
NaN
NaN
0,801
0.813
25.0
25.0
Table Comparison of optimizer performance across three tasks: language modeling on WikiText- 2, image classification on CIFAR-10, and flow matching on GridWorld.
Method
Training time (s)
Validation Loss
AdamW
48549.56
3.511
VRAdam
48522.40
3.447
Table 2: Comparison of training time and validation loss for GPT-2 training.
LLM benchmark. In order to demonstrate the effectiveness of VRAdam for training large net- works, such as large language models (Brown al., (0200), present the validation loss results of VRAdam against that of AdamW and Lion when training a 124M parameter GPT-2 model on the Fine FineWebEdu-100 dataset from scratch (Penedo et al., 2024). In Table 2 we report the validation loss after around 57000 steps or 2 epochs as well as the average training time per epoch on H100 GPU using hyperparameters found in the Appendix.
Setting
Model
Dataset
AdamW PPL
VRAdam PPL
Lion PPL
4-bit QLoRA
LLaMA-2-7B GPT-2 Large (774M)
OASST2
3.84
3.55
3.56
Full model
GSMSK
4.12
3.53
3.67
Table 3: Additional LLM benchmarks comparing Adam W, VRAdam, and Lion (Chen et al. 2023 in challenging fine-tuning regimes for OASST2 (instruction following) and GSM8K (math reasoning).

---

Published as a conference paper at ICLR 2026
Beyond the 124M GPT-2 pretraining setup, also evaluate VRAdam in two more challenging language-moded modeling regimes summarized Table First, we fine-tune a 7B LLaMA-2 model using 4-bit NF4 QLoRA on the OASST2 cnntrructionfoollwwiii dataset Touvron et al. (202)). This setting lies close to the adaptive edge-of-stability: gradients are both low-precision and strongly stochas- tic Hayou et al. (2024). We also perform full-parameter fine-tuning of GPT-2 Large (774M) on GSM8K. These experiments indicate that VRAdam scales favorably from medium-sized to larger LLMs, both in quantized low-rank adaptation and full-model fine-tuning settings. On GSM8K, fine-tuning GPT-2-Large with VRAdam improves exact-match accuracy from 35% (AdamW) to 42% under the same training budget. On OASST2, in 4-bit QLoRA setup for LLaMA-2-7B, VRAdam improves an automatic instruction-followinn quality score from 72.3/100 to 78.5/100 compared AdamW. W. This score combines lexical diversity, repetition, and response length over 50 held-out instructions.
6 RELATED WORK
Follow-up work on Adam. The Adam optimizer was introduced in 2015 by KKngmaa & Ba (2015) by combining the previously existing concepts of momentum and scaling a base LR for each param- eter based on second-order moment estimates. The base learning rate, however, remains hard-coded (potentially chosen through a learning rate scheduler) throughout training. Since then, several mod- ifications to Adam have been introduced, such as NAdam (Dozat) 2016). RAdam (Liu et al.) 2020), Adabelief (Zhuang et al., 2020), and in particular Adam W (Loshchilov & Hutter (0177). which rein- troduces weight decay in its original intention. LARS (You et 2017) and LAMB You et al., 2020) compute learning rates for layers individually. More recent optimization techniques include LION, an automatically discovered alternative to signed momentum (Chen et al.) 2023), Sophia (Liu et al. 2023). which uses estimated diagonal entries of the Hessian as precondition, sharpness- aware minimization methods (Foret et al., 2021). and a modified LR in Adam for reinforcement learning (Ellis et al. 2024).
Understanding training dynamics, convergence analysis, and edge of stability. Another line of work focuses on understanding the dynamics of the training of deep neural networks as well as de- rive convergence properties and guarantees for commonly used optimizers. [Wang Choromanska 2025) provides a recent survey over the later and Reddi al. (019) provide an explicit convex example for which Adam does not converge. A popularized framework for understanding training dynamics in the continuous training flow and infinite network width limit was introduced in Jacot et al. (2018), extended by finite width corrections in Huang & Yau (2020). and developed for graph neural networks in Du et al. (2019) Lastly, the role of the edge of stability regime offers empiri- cally view and was analyzed in Cohen et (2024), Arora et al. (2022); Cohen et al. (2022); Damian al. (0022); Song & Yun (2023): Wang et al. (2022)
Symplectic optimization. This research area derives discrete-time optimization algorithms by dis- cretizing continuous-time Hamiltonian or Lagrangian flows using symplectic integrators, which exactly preserve the underlying geometric (symplectic) structure of the dynamical system. This approach guarantees long-term stability,
properties (or controlled energy dissi- pation in the dissipative case). and can provide valuable insights into existing optimizers. Notable work includes Betancourt al. (2018); França et al. (2020); Maddison et al. (2018): Duruisseaux & Yuan & Zhang 2023)
Leok 20033)
7 DISCUSSION
L Limitations. The edge of stability regime remains not fully understood and questions regarding generalization capabilities remain. We also have constrained compute resources.
Summary and future of work. Motivated by physical perspectives for complex optimization sce- narios and stability conditions along with the adaptive edge of stability, we developed a new opti- mizer VRAdam based quartic kinetic energy terms. We analyzed its performance at the adaptive edge of stability and benchmark it against several optimizers in particular Adam on image clas- sification, language modeling, and generative task using GFlowNets where report improved performance and robustness to hyperparameters. We hope that this work leads to further develop- ment of interpretable optimizers using concepts from physics.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltz- mann machines. Cognitive Science, (1):147-169 1985.
Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on edge of stability in deep learning, 2022. http:::ttssssttt
org/ abs/ 2200. 09745
Benoît Assi, Bernd A. Kniehl, and Joan Soto. Matching the standard model to heavy -owyk- ef- fective theory and nonrelativistic ged. Nucleur Physics B, 992: 16173, July 2023, ISSN 0550- 3213. doi: 10. 10.110166..mll wttp://..mccppyyy.2.32... nuclphysb. 2023. 116173
URL http:/ doii org 101100000.000011116611111111 1
Yehonatan Avidan, Qianyi Li, and Haim Sompolinsky. Connecting ntk and nngp: A unified theo- retical framework for wide neural network learning dynamics, 2025. URL hhttps://arxi:t
exc/abs/ 2309. 04522
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in neural information processing systems, 34:27381-27394, 2021.
Michael Betancourt, Michael I. Jordan, and Ashia C. Wilson. On symplectic optimization. arXiv preprint arXiv:/802.036:: 2018.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Eric Braaten, Introduction to the nrqed factorization approach to heavy quarkonium, 1997. URL
https://arxiv.. wrw. abs /hepeeeppppe http:hhhppphhhh22/// ph/9702225
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL bttps://arxiie org/ abs 2005 14165
Xiangning Chen, Chen Liang. Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems,
49233, 2023.
Maxime Shevalier-Boisverr Bolun Dai, Mark Towers, Rodrigo VicentVi Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Mod- ular & customizable reinforcement learning environments for gral-oriented tasks. In Advances in Neural Information Processing Systems 36, New Orleans, LA, USA, December 2023.
Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge stability, 2022. URL https://arxiv.
org/ hts//bb//2111 00065
Jeremy M. Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze, Zachary Nado, George E. Dahl, and Justin Gilmer.
Adaptive gradient methods the edge of stability, 2024. URL httpss//arxxv..rr///a
0707.14455
Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization:: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209.15594 2022.
Aaron Defazio, Xingyu Alice Yang. Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The road less scheduled, 2024. URL htl.https///aaxi.ooo///////
15682
Timothy Dozat. Incorporating nesterov momentum into adam. In /CLR 2016 Workshop, 2016. https: http://wwww.a...e..//ooo/ooo/o/a
2016. URL

---

Published as a conference paper at ICLR 2026
Simon S Du, Kangcheng Hou, Russ R Salakhutdinov, Barnabas Poczos, Ruosong Wang, and Keyulu Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kernlls. Advances in neural information processing systems, 32, 2019.
Valentin Duruisseaux and Melvin Leok.
Practical perspectives on symplectic accelerated optimiza- 2023. doi: 10.1080/10555788.222..
tion. Optimization Methods and Software, 2214837.
Alexandre Défossez, Léon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of adam and adagrad, 2022. URL httpp:t/axxxxii..ooo wwwp:/wwwwwww..aa////a////////// 02395
Benjamin Ellis, Matthew T Jackson, Andrei Lupu, Alexander D Goldie, Mattie Fellows, Shimon Whiteson, and Jakob Foerster. Adam on local time: Addressing nonstationarity rl with relative adam timesteps. Advances in Neural Information Processing Systems, 37:134567-134590, 2024.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2021.
Guilherme França, Jeremias Sulam, Daniel Robinson, and René Vidal. Conformal symplectic and relativistic optimization. Advances in Neural Information Processing Systems, 33:16916-16924 2020.
Partha Guha and A Ghose-Choudeee Choudhury. Construction of the classical time crystal lagrangians from sisyphus dynamics and duality description with the liénard type equation, 2019. URL. https: wwarxiv.crg/abs///1
11626
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization on lora finetuning dynamics, 2024. URL htttt::t///aaat
org/ http/oggg/aaaaaeeeeeeeee awwp///6....... 08447
Thomas M Helliwell and Vatche V Sahakian. Modern classical mechanics. Cambridge University Press. 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1): 42, 1997. doi: 10.1166//eco..199.....
Peter Holderrieth, Yilun Xu, and Tommi Jaakkola. Hamiltonian matching and generative flows, 2024.
20470
John J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 7965):2554-2558, 1982.
Jiaoyang Huang and Horng-Tzer Yau. Dynamics of deep neural networks and neural tangent hier- archy. In International conference machine learning, pp. 4542-4555 PMLR, 2020.
H Iro and K Anderson. Modern approach to classical mechanics. Appl. Mech. Rev, 56(6):880 B81,, 2003.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Ryoichi Kawai, Juan MR Parrondo, and C Van den Broeck. Dissipation: The phase-space perspec- tive. Physical review letters, 9888)0000 2007.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA, wrwia, org/ haspg//bb//1111 6980 arXiv:1412.6980.
May 2015. URL https: http:s://arxvv.voo
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny im- ages. Technical Report TR-2009-003, University of Toronto, 2009. URL https:/ / www.
eurontoo.du/-kriizz. kwia/eeariiigggg----e.eaaaa features 2000-000--000 TR. pd.
Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305. 14342, 2023.
12

---

Published as a conference paper at ICLR 2026
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
Chris Maddison, Daniel Paulin, Yee Whye Teh, Brendan O' O'Donoghue, and Arnaud Doucet. Hamiltonian descent methods. arXiv preprint (rXiv:/809005042 2018
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Antti J. Niemi. Time crystals: From schrödinger to sisyphus, 2021. URL.ttps://axxi..orr
abs/2109. 06091
Guilherme Penedo, Hynek Kydlížek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirt Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URLhttps:/ URUlhttps://peerevalti /pwp///epeeeeeeeeeee/eeeeeeeeeee net/ id 2d=n66Ckk2Qab
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural Nei-
works, 12(1):145-151, 1999. ISSN 0893-6080. doi: http://daiann////000006501111199 00116-6. URL
$089360809800110
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv: 092377, 2019.
Sebastian Ruder. An overview of gradient descent optimization algorithms, 2017. URL https:
//aaxiv..rq/aaata 04747
Alfred Shapere and Frank Wilezek. Classical time crystals. Physical Review Letters, 109(16),
October 2012. ISSN 1079-7114. doi: 10.1 10.100//tyyeviiet..00..... URL http:// /ttp://dx.
doi org/ 10. 1103/ 1113///ysseeeeeecccc
109.06
Minhak Song and Chulhee Yun. Trajectory alignment: understanding the edge of stability phe- nomenon via bifurcation theory. arXiv preprint N.ti:22377.022044 2023.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International conference on machine learning. pp- 1139-1147, PMLR, 2013.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet. Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. UtLpttps: http:///wwwwvvooo/aabb/ httpo////aab/00...3333 09288
Pranav Vaidhyanathan, Lucas Schorling, Natalia Ares, and Michael A. Osborne. Implementation of vradam. https: http::://gittus. wttp/////ww/a////aaaaaaaaaa
2026.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez,
Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https: ://arxiv.
org//bb/ ab./////.........// 03762
13

---

Published as a conference paper at ICLR 2026
Jing Wang and Anna Choromanska. A survey of optimization methods for training dl models: The- oretical perspective on convergence and generalization, 2025. URL /tt.ttps////xxt...orrr /ttp://xxxvv//////..
abs a.ss/2000 02//2250..144455
Zixuan Wang, Zhouzi Li, and Jian Li. Analyzing sharpness along gd trajectory: Progressive sharp- ening and edge of stability. Advances in Neural Information Processing Systems, 35:9983-9994 2022.
Mare Weber, Bahman Gharesifard, and Christian Ebenbauer. Inferring global exponential stability properties using lie-bracket approximations, 2024.
03871
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks, 2017. abs/1
URL http:t://///xx..coomm wrxxiv.org/bbb///78... 03888)
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Jui Hsieh. Large batch optimization for deep learn- ing: Training bert 76 minutes, 2020. URL Stctp:://aaaiv..og/aas//1000000 00962
Ya-xiang Yuan and Zhang. Symplectic discretization approach for developing new proximal point algorithm. arXiv preprint arXiv:2308 S.0i:::2000003986, 2023.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Pa- pademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-18806, 2020.
A BOUNDING NRQCD
By explicitly breaking certain symmetries- Lorentz invariance in NRQCD and time-translation in time crystals- higher-order kinetic terms paradoxically enhance stability through topological pro- tection mechanisms and the generation of emergent length/time scales (Niemi 202 Guha & Ghose- Choudhury 2019).
As demonstration of this phenomenon, can consider Nonrelativistic QCD (NRQCD), which is an effective field theory that expands full quantum chromodynamies in inverse powers of a heavy- quark mass m (Assi et al. 2023). The bilinear heavy-quark Lagrangian C up to O(1/m²) reads.
LNRQCD bt (Do +
D2
D
+...)
(32)
2m
8m3
The corresponding Hamiltonian density derived via Legendre transformation:
D
D
HNRQCD 25 (
+
(33)
2m
Sm
For rigorous analysis of the dispersion relation, work in momentum space:
( () - /
d'p s (2m)3 (5 (7,1)
(34)
In momentum space, the operators transform as:
Do - iE (time evolution operator)
(35)
D - ip (spatial covariant derivative)
(36)
22 - p² (squared spatial derivative)
(37)
4 pd (quartic spatial derivative)
(38)
The energy eigenvalue equation derived from the Hamiltonian is:
p²
F = +
(())
(39)
2m
8m3
m
14

---

Published as a conference paper at ICLR 2026
This gives the NRQCD dispersion relation to order 1/mm.
1¹
p
ENROCD (p) = m +
O
(40)
2m
8m3
172
A.1 ANALYSIS OF BOUNDEDNESS
The exact relativistic energy-momentum relation:
Eya(y) mm2 p2
(41)
Using Taylor series expansion for vI + x where a = p²/m²²
1
5
-+ _ x
r
x²
+
O
(42)
16
128
Applying this to the relativistic energy:
1 p²
pz
p6
Exe(p) = m 1
+
(43)
2 m
8 m
m
Simplifying:
p²
p
p
Epe(p) = m +
+
(44)
2m
8m3
m5
For the non-relativistic approximation without the quartic term:
ENK(P)
+ 2m
lim
(45)
p00
(p)
p 100
Vm² + p2
p² + 2m
lim
(46)
p0000
Pv + m
0
(47)
This shows that the non-relativistic approximation without the quartic term diverges from the true relativistic behavior.
The quartic term introduces a negative contribution to the energy that precisely cancels the fourth- order term in the relativistic expansion:
p²
ENROCD (p) = +
O
(48)
2m
8m3
m
Let's define a parameter À = p² /²/m² (proportional to o²). For the NRQCD expansion to be valid, we require X 1.
The second derivative of the NRQCD energy with respect to X is:
dl ENROCD
O())
(49)
dxa
the truncated dispersion is concave and has finite maximum, so does not increase unboundedly.
B PHYSICS ANALYSIS
In this section, we provide a stability analysis from physical perspective. Starting from the quartic Lagrangian in E we derive the one-dimensional dynamics and discretize them implicitly through the state-dependent step
ag
= + min (3g v. 01) = -18 + (1-
= Tt-1 U/ (50)
15

---

Published as a conference paper at ICLR 2026
For quadratic potential (a)) 2 Hæ with 0 < H LI. work in the eigenbasis H Q (.. Writing coordinates (Ia, 14) and fixing h (0, T.), the one-dimensional update reads
y nh, where a 1 and 17 E [Tmmnn aol with Mmin ao/ + a1).
V Biu + (1 1-31,
x+ x 1---111 1ju,
a: 1- B1,
(51)
We introduce an energy that mirrors the quartic kinetic term and couples consistently to the curva- ture:
5 +
(x, v) 2 hr + +ax +
4 2
(52)
2h
and set
B1 " Mmnin
7 5 Mwin
8 82
EE (0,
(53)
33
A direct expansion of (x, v) (x, u) yields a quadratic form in (æ, u) plus the quartic dif- ference (u*-u*)/a w²//4. Because the coefficients of the quadratic form are convex polynomials in nh, the worst case over E h agh) occurs at the endpoints. Evaluating there, and using the adaptive edge-of-stability bound for momentum,
2(1 + B1)
ag
(54)
1-31
one obtains constants Cx, Cu > (depending only on 31, oz Mmin)) such that, uniformly for all h [n, I.] and 1j [Amin: (r)):
Ex(x .u) - Ex( -caha² cu u²
-((00= 1o---u*) u²).
(55)
The gate implies S a0/(1 + 3gr2) and hence
ag 2 TP 8,33
The choice 2 // therefore ensures that whenever |0] > 1/vB3 the quartic contribution dominates any potential increase from the discretization term and contributes strict dissipation; when |u| & 1/VB3 the negative quadratic terms already control the step.
Summing coordinates yields the global energy
Exx,
Σ
En, (2it U));
(56)
and constants Ki, K4 >> such that
4 E(x,, wx) E( -x12T Hz Nov, H1 U1-1 max 0,
33
(57)
This inequality formalizes the physical picture in Fig. The first two terms express curvature- weighted exchange between potential and quadratic kinetic energy with net dissipation; the quartic term brake that activates in high-velocity regimes created near the adaptive edge of stability. The same mechanism explains the reduction in ringing and lower sharpness observed in Fig. when |v,l grows, the gate reduces 1); and the quartic channel increases dissipation, pushing the dynamics back to low-velocity regime.
Two immediate consequences follow. First, the instantaneous adaptive stability threshold increases with the measured velocity:
LEos(t) = 2(1 31) 2(1+ 31),
(1 31 - 11-Bi)on (1 + min(3a ax)),
(58)
so the method moves away from instability as oscillations grow.
16

---

Published as a conference paper at ICLR 2026
Second, each parameter update is uniformly bounded:
0gg
Uv
00
aate -4-1 = 1/p (eel
(59)
2/D3
which prevents runaway steps and is not available to classical momentum. These properties are consistent with the design of [agorithm[aand the empirical behavior reported in the analysis section.
GENCE PROOF FOR VRADAM C
The proof is based on the work of the proof in Défossez et al. 2002
There, the authors derive equation A.37 the bound
N
3 B
< F( - F.
2R
A 0
N
+ 2 L
U4
n=1
B
n-1
03y 1,2
E
un- 11
Σ
4/2 11-11111 31
30N R 1 31
1: TE [[
By bounding an with Ominn in expression A, and aN with Omax in expressions B, C, D obtain the final result. Although not included in the proof, the numerical impact of the velocity bias correc- tion becomes negligible after the first few update steps and asymptotic behavior remains unchanged (Défossez et al., 2022). is well established that weight decay contributes to improved robust- ness and generalization in practice and is therefore included here, despite challenging theoretical analysis.
D ABLATION ON KINETIC ENERGY POTENTIAL
Table 4 displays the test loss for higher powers k in the learning rate ak/(1 + min(3slluu |u.1* az)), which correspond to fifth- to eighth-ordes terms in the kinetic energy. Training performed on CIFAR- 10 for 10 Epochs with ag = 0.005.
Table 4: Test Loss on CIFAR-10 for different kinetic energies
Power
Test Loss
0.932
1.155
4
0.974
1.004
1.036
E HYPERPARAMETERS, DATASETS AND MODEL ARCHITECTURES
We recommend 33 1.0 and on 10 the default setting for VRAdam.
17

---

Published as a conference paper at ICLR 2026
E.1
IMAGE CLASSIFICATION
This section details the comprehensive sweep for the CNN, on the CIFAR-10 dataset.
Model:
Convolutional Neural Network
Dataset: CIFAR-10
Hyperparameter sweep method: Bayesian optimization
Optimization metric: validation loss
Table 5: Dataset Splits during CNN Sweep
Dataset Split
Configuration Description
Training
80% of the original CIFAR-10 training set (50,000 images).
Validation
20% the original CIFAR-10 training set.
Test
Full CIFAR-10 10 test set (10,000 images).
Table 6: Fixed Hyperparameters during CNN Comprehensive Sweep
Parameter
Value
Model Architecture
Convolutional Neural Network
Dataset
CIFAR-10
Epochs
100
Batch Size
1024
Scheduler Type (AdamW)
WammpCossineAmeellngg
Warmup Epochs
5
Warmup Factor
0.1
Scheduler 17 min
1 10-5
VRAdam 31
0.9
VRAdam 32
0.999
VRAdam power
2
VRAdam weight decay
1 10-5
VRAdam
1 10-8
AdamW 31
0.9
AdamW $2
0.999
AdamW weight decay
x 10-5
SGD momentum
0.9
SGD nesterov
True
SGD weight decay
x 10-5
RMSProp
0.99
RMSProp
1 x 10-8
RMSProp weight decay
1 10-5
Table 7: Swept Hyperparameters during CNN Comprehensive Sweep
Optimizer
Parameter
Sweep Configuration
Optimal Parameter
VRAdam
Ok
Log-uniform, Min: x 10-4, Max: 0.1
0.0846
VRAdam
83
Uniform, Min: 0.1, Max: 1.5
1.015
VRAdam
Q1
Integer Uniform. Min: 3, Max: 30
29
AdamW
7
Log-uniform. Min: 1 X 10-5
Max: 1 10
0.0625
SGD
n
Log-uniform, Min: 1 x 10-5
Max: 1 10-1
0.00784
RMSProp
Log-uniform. Min: 1 x 10-6
Max: 0.1
1.78e-4
E.2 LANGUAGE MODELING
This section details the comprehensive sweep for the Transformer, the WikiText-2 dataset.
18

---

Published as a conference paper at ICLR 2026
Model: Transformer
Dataset: Wikitext-2
Hyperparameter sweep method: Bayesian optimization
Optimization metric: validation loss
Table 8: Dataset Splits during Transformer Comprehensive Sweep
Dataset Split
Configuration Description
Training
Full WikiText-2 predefined training set.
Validation
Full WikiText-2 predefined validation set.
Test
Full WikiText-2 predefined test set.
19

---

Published as a conference paper at ICLR 2026
Table Fixed Hyperparameters during Transformer Comprehensive Sweep
Parameter
Value
Model Architecture
TransformerModel
Epochs
100
Batch Size
32
Seed
Scheduler Type
NamupCossineAnnaalig
Warmup Epochs
Warmup Factor
0.1
Scheduler 1
x 10-5
Model sequence length
64
Model embed dimension
128
Model hidden dimension
256
VRAdam B1
0.9
Ad 83
0.999
VRAdam power
VRAdam weight decay
x 10
VRAdam
x 10
Adam W 31
0.9
Adam W
82
0.999
AdamW W weight decay
x 10-5
SGD sgd momentum
0.9
SGD sgd nesterov
True
R.SPPrpp
0.1
R.MSProp e
10
Table 10: Swept Hyperparameters during Transformer Comprehensive Sweep
Optimizer
Parameter
Sweep Configuration
Optimal Parameter
VRAdam
Log-uniform, Min: x 10-5, Max: 0.I Uniform, Min: 0.1, Max: 5.0
1.55e-05
VRAdam
Sg
3.35
VRAdam
normgrad
Values: [True, False]
False
VRAdam
on
Integer Uniform, Min: 5, Max: 30
Adam
Log-uniform. Min: 1 x 10
Max: 0.1
1.661c-05
SGD
7
Log-uniform. Min: 1 X 10-5. -5-5
Max: 10
RMSProp
"
Log-uniform, Min: 1 10-5,
Max: 0.1
E.3
GENERATIVE MODELING WITH GFLOWNETS
Here include the hyperparameters used for reporting the performance of the optimizers for the GFlowNet.
Table 11: Hyperparameters GFlowNets
Optimizer
Parameter
Value
VRAdam
Og
0.01
VRAdam
A
VRAdam
normgrad
False
VRAdam
19
VRAdam
weight decay
x 10-5
AdamW
weight decay
10
AdamW W
0.01
SGD
0.01
RMSProp
0.01
20

---

Published as a conference paper at ICLR 2026
E.4
EDGE OF STABILITY ANALYSIS
Table 12: Hyperparameters edge of stability analysis
Parameter
Value
Model Architecture
ResNet-32
Max iterations
20000
Batch Size
1000
Seed
0
Loss criterion
Mean squared error
VRAdam og
0.002
VRAdam 31
0.9
VRAdam $3
0.999
VRAdam 33
VRAdam power
VRAdam normgrad
False
VRAdam 0n
19
VRAdam
x 10-?
Adam $1
0.9
Adam $2
0.999
Adam
x 10
F COMPUTE RESOURCES
All experiments run Lambda cloud instances the Google cloud platform (GCP). Exper- iments were conducted either using a NVIDIA L4 GPU with 24 GB GPU memory and 31 GB of system memory larger experiments were performed on NVIDIA A10 with 24 GB of GPU memory and 200 GB of system memory. The GPT benchmark was run on an NVIDIA H100 GPU.
G LOSS CURVES WITH ERROR ENVELOPES
Stop
Stop
20 40
60
100
Figure 3: Train (left) and validation (right) loss curves with error envelopes calculated using different run values for language modeling using AdamW.
Train and validation loss curves calculated using different run values for language modeling using SGD Nesterov with momentum all generate NaN values. Visualization is not included.
21

---

Published as a conference paper at ICLR 2026
Train Loss with Error Envelope
Validation Loss with Error Envelope
10
Stop
20 40
60
80
100
100
Figure 4: Train (left) and validation (right) loss curves with error envelopes calculated using different run values for language modeling using VRAdam.
Train Loss with Error Envelope
Validation Loss with Error Envelope
8.5
78
72
Step
Figure 5: Train (left) and validation (right) loss curves with error envelopes calculated using different values for language modeling using RMSProp. The dots the top indicate NaN values.
Test Loss across Different Runs
Train Loss with Error Envelope
5
3
Step
Figure 6: Train (right) curves with error envelopes calculated using different run values for image classification using VRAdam. Test loss values (left) over different runs.
22

---

Published as a conference paper at ICLR 2026
Test Loss across Different Runs
Train Loss with Error Envelope
Stop
0.0
0.2
0.4
0.6
08
10
20
40
8D
80
100
Figure 7: Train (right) curves with error envelopes calculated using different run values for image classification using AdamW, W. Test loss values (left) over different runs.
Test Loss across Different Runs
Train Loss with Error Envelope
15
08
Figure 8: Train (right) curves with error envelopes calculated using different run values for image classification using SGD Nesterov with Momentum. Test loss values (left) over different runs.
Test Different Runs
Train Error Envelope
100
Step
40
Figure 9: Train (right) curves with error envelopes calculated using different run values for image classification using RMSProp. Test loss values (left) over different runs.
23

---

Published as a conference paper at ICLR 2026
H BACKGROUND ON PHASE SPACE, LAGRANGIANS, AND HAMILTONIANS
H.1 PHASE SPACE
In classical mechanics, the state dynamical system completely described by point in a high- dimensional abstract space called phase space Kawai et al. (0077) For system with N degrees of freedom (e-g.. the positions of multiple particles), the phase space is 2.V -.V-dimnsional space. Its axes correspond to the generalized coordinates zy (representing positions) and their corresponding generalized momenta p:- Each point (a, p) phase space represents a unique, instantaneous state of the system. The evolution of the system over time is then visualized as trajectory traced out by this point moving through phase space.
H.2 LAGRANGIAN FORMALISM
The Lagrangian formulation of classical mechanics describes the dynamics of system using gen- eralized coordinates and their time derivatives (velocities) Iro & Anderson (2003). The central quantity in this formalism is the Lagrangian, C, which is function of the system's 's coordinates and velocities. It is typically defined as the difference between the system's 's kinetic energy, T, and its potential energy, V:
C(x, v) = T(u) - V(x)
(60)
where e = . represents the velocity. The system's 's path through its configuration space is determined by the Principle of Least Action. This principle states that the actual trajectory taken by the system between a starting time & and an ending time t2 is the one that minimizes the action integral, S:
s s= 2(x,0,1)
(61)
Applying the calculus of variations to find the path that minimizes this action integral yields the fundamental Euler-Lagrange equations of motion:
d OL Oc dt du dr =0
(62)
Evaluating derivatives requires treating & and a = v as independent variables. This set of second- order differential equations fully defines the system's trajectory. In the context of deep learning optimization, the potential energy V(x) is analogous to the loss function Liss with the model parameters @ serving as the coordinates æ. Optimizing the loss function then corresponds to solving the system of differential equations with numerical integrator.
H.3 HAMILTONIAN FORMALISM
The Hamiltonian formalism offers an alternative, often more powerful, description of system dynam- ies that is within phase space, using coordinates x: and momenta p as its fundamental variables. The transition from the Lagrangian to the Hamiltonian framework is achieved via a mathematical procedure known as a Legendre transformation Helliwell & Sahakian (2020). First, the general- ized (or canonical) momentum p is defined as the partial derivative of the Lagrangian with respect the velocity:
dc
(63)
8v
The Hamiltonian, H. is then defined as:
H(z,p) Σ P.us - L(I,v)
(64)
For many standard physical systems where kinetic energy is a quadratic function of velocity and potential energy is a function of position only, the Hamiltonian is equivalent to the total energy of the system, 7. = T + V. The dynamics are then described by pair of first-order differential equations known as Hamilton's equations:
07
(65)
0p
24

---

Published as a conference paper at ICLR 2026
p 0H Fx
(66)
These equations provide an elegant description of the flow of system states through phase space. A key feature of this formalism is that it naturally describes systems that conserve energy (if H. is time-independent) and preserve the volume of phase space. This makes the Hamiltonian framework exceptionally well-suited for analyzing the long-term stability of dynamical systems and serves as the foundation for symplectic optimization methods.
I FURTHER DISCUSSION
Recent work by Defazio et al. (2024) introduces Schedule-Fre optimizers (e.g.. Schedule- SGD and AdamW), which remove the need for explicit learning rate schedules by utilizing specific weighting the iterate sequence. While both VRAdam and Schedule- methods aim to simplify hyperparameter tuning by modifying how the step size and updates are handled, they operate through fundamentally different mechanisms and target distinct dynamical regimes.
Mechanism: Averaging vs. Feedback Control. The core innovation of Schedule-Free methods an interpolation between primal and Polyak averaging, where the effective learning rate is de- termined by deterministic, dime-dependent weighting scheme derived from online-to-batcl con- version guarantees. The effective step size in these methods a function of the iteration index t, simulating decay schedule without fixed horizon.
In contrast, VRAdam employs a state-dependent feedhack mechanism. Our gating term, = 01/(1 Bs I12), is not explicit function of time, but strictly function of the global momentum buffer norm. Consequently, VRAdam reacts dynamically to the optimization trajectory: it actively damps updates during periods of high oscillatory kinetic energy (common the early training phase or high-curvatury regions) and relaxes the gate when the trajectory stabilizes. This creates closed- loop feedback control system rather than pre-computed averaging schedule.
Theoretical Focus. The theoretical underpinning of Defazio et al. (2024) focuses on achieving worst-case optimal convergence rates for convex Lipschitz problems. Our analysis focuses on the Adaptive Edge of Stability (AFoS) (Cohen et al. 2024). We derive our update rule from quartic Lagrangian to ensure global exponential stability by enforcing a uniform bound on the parameter update norm, ||0, 0.-1]l |l. effectively raising the stability threshold in response to gradient bursts.
Finally, we note that these approaches are mechanistically orthogonal. The velocity-regulariized gate of VRAdam operates the preconditioned gradient and could, in principle, be combined with the iterate averaging schemes of Schedule- FFee methods, though we leave such hybrid explorations to future work.
25


Reviews:
[{"review_id": "AZMlmX1MRj", "reviewer": "Reviewer_SPPq", "summary": "This manuscript challenges gradient descent optimization techniques. Motivated by physics, the authors present VRAdam, which automatically controls the learning rate based on the momentum. Experiments on CIFAR, Wikitext, GFlowNet, and GPT-2 demonstrate improvements.", "questions": "Please see the weaknesses above. My score is based on the assumption that all typos are corrected in the revised manuscript.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The motivation, especially the physics-inspired optimizer, is interesting. Indeed, the momentum optimizer itself is derived from Newtonian dynamics.\n- The theoretical analysis looks solid and promising for the improved convergence.\n- Source code is available, which eases deployment in practice.", "weaknesses": "- The norm of $v_t$ of the VRAdam looks to compute the global norm, but it may be appropriate to use the parameter-wise norm to allow parameter-wise learning rate control. This choice is not sufficiently discussed.\n- VRAdam brings three hyperparameters of \\alpha_0, \\alpha_1, and \\beta_3. I think these additional hyperparameters make it difficult to adopt the VRAdam in practice.\n- Accuracy of 80% for ResNet-32 with CIFAR-10 is a weak baseline.\n- Table 1 is not convincing enough. These results should be supplemented with more quantitative and qualitative analysis.\n- The experimental results, such as Table 2, are focused on the final validation loss. Is it possible to demonstrate other indices, such as practical ones? I think certain practitioners may want to capture the performance more practically, but the value of loss is difficult to understand on an absolute scale. It is also difficult to understand whether it corresponds to sufficient convergence or is still far from convergence.\n- LLM results were only trained for 2 epochs, which I think is insufficient for convergence.\n- I think Eq. 36 would be -m/4 + O(\\lambda), not -3m \\lambda/4 + O(\\lambda^2). Is it possible to provide an exact derivation?\n- Writing should be improved.\n    - Kingma & Ba (2017) → Kingma & Ba (2015)\n    - “the, global” → “the global”\n    - “d” → “(d)” for the caption of Figure 2.\n    - “physical inspired” → “physics-inspired” at Line 70.\n    - “Note, that” → “Note that” at Line 784.\n- The manuscript writes to compute velocity norm, whereas the source code computes gradient norm by default (normgrad=True). To be compatible with the description in this manuscript, normgrad=False may be correct for default.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LAr1Ftl18c", "reviewer": "Reviewer_G5iq", "summary": "The authors propose a physics-inspired variant of AdamW, where the effective learning rate is regularized via the magnitude of the momentum. Some theoretical and empirical results are provided, including a convergence rate and evaluation on CIFAR.", "questions": "1. Could the authors elaborate on how the kinetic energy on line 129 was chosen? What happens if we choose something else? Could this lead to a better optimizer than the proposed method?\n2. See Weaknesses.", "limitations": "", "rating": null, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "1. The proposed method is novel, and the inspiration from physics is motivated.\n2. The theoretical results appear to be solid, and the experimental results seem promising.", "weaknesses": "1. A major part of the theoretical analysis assumptions a convex quadratic objective, which the authors claim to be \"[traditional] for stability analysis\". However, this is far too strong of an assumption to be practically useful in any deep learning context. Furthermore, the convergence rate is a trivial corollary of the work of Defossez et al., providing little theoretical insight or novelty.\n2. The provided experimental results are conducted with small (124M) models and evaluated on simple tasks such as CIFAR. This makes it difficult to establish the scalability of the proposed method for practical deep learning.\n3. The proposed method introduces a hyperparameter $\\beta_3$, with no discussion on tuning or practical recommendations.\n4. The writing is sometimes very difficult to read. While inspiration from other fields is quite common in machine learning, the extended use of physical analogies may not be easily understood by the ICLR community (it was certainly lost on me).\n5. Minor nits: ensure the displays on page 3 are normal sized font. Use `\\citep` for citations that should be parenthetical.\n\nAs there are serious limitations with both the theoretical and empirical contributions of this work, I recommend rejection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kVSTudlffo", "reviewer": "Reviewer_JcRv", "summary": "This paper proposes a physics-inspired optimizer, VRAdam, designed to address the oscillation and convergence slowdown problems that occur when adaptive optimizers (such as Adam) train near the stability boundary. Its core idea originates from kinetic models in classical physical systems. By introducing a velocity regularization term, the method imposes a higher-order penalty on the momentum (velocity) during optimization, thereby dynamically adjusting the learning rate and suppressing instability.\n\nContributions\n1. Proposes a physics-inspired optimizer, VRAdam, which introduces a velocity regularization term into Adam to dynamically adjust the learning rate and suppress training oscillations.\n\n2. Theoretically proves the stability of a simplified model and establishes a convergence bound under non-convex settings.\n\n3. Demonstrates the effectiveness of the method on image classification, language modeling, generative flow networks, and GPT-2 training tasks.", "questions": "1. The visualization of results could be improved. Why is Figure 5 in the appendix blank, and why are some figure fonts too small?\n\n2. The theoretical proofs are based on a simplified model, while the actual implementation includes all modules. The authors are encouraged to explicitly discuss how this simplification affects the stability and convergence conclusions.\n\n3. The reproducibility and statistical significance of the results are difficult to assess.\n\n4. Did the authors conduct any study or experiments regarding the choice of $\\beta_3$?\n\n5. Have the authors attempted experiments on more complex tasks?\n\n6. Does VRAdam provide any advantage in training time, and does it introduce additional computational overhead?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is not a minor modification of Adam but introduces an intuition derived from a non-standard Lagrangian containing a $v^4$ term. This physics-to-algorithm mapping is uncommon in existing literature and demonstrates novelty.\n\n2. Theoretically, the paper provides a proof of uniform exponential stability for VRMomentum under quadratic objectives (Theorem 4.1) and establishes a convergence rate bound under stochastic non-convex settings comparable to that of Adam (Theorem 4.2).\n\n3. The experimental tasks cover a wide range of applications.\n\n4. If VRAdam proves to be stable and effective under broader settings (including larger-scale models and more repeated runs), it could be a strong improvement over existing optimizers.", "weaknesses": "1. The theoretical analysis is based on a simplified model, while the actual implementation includes all modules. This may lead to inconsistencies between theory and practice.\n\n2. The tables lack explanations regarding reproducibility and do not include error bars, raising questions about the reliability of the results.\n\n3. The authors are advised to discuss the robustness of the results to hyperparameters, as all experiments use a single configuration. In all experiments, β₃ is fixed to 1, but the rationale behind this choice is not provided.\n\n4. The baseline methods are relatively outdated and insufficient in number. Although the related work section mentions more recent optimization techniques, these newer methods are not included in the experiments.\n\n5. The authors only compared the training time between AdamW and VRAdam on GPT-2, and the results did not show a clear time advantage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YtfPNtleeV", "reviewer": "Reviewer_uYkK", "summary": "This paper proposes Velocity-Regularized Adam (VRAdam), a new optimizer for deep neural network training that introduces a physics-inspired regularization mechanism. Based on the classical Adam optimizer, VRAdam incorporates a quartic kinetic energy term to dynamically regulate the effective learning rate based on the velocity (momentum) of parameter updates. The resulting learning rate shrinks automatically in high-velocity regimes, reducing oscillations and improving convergence stability. The paper provides a theoretical framework, proving uniform exponential stability via Lyapunov analysis for stochastic non-convex objectives. Extensive experiments on CIFAR-10, WikiText-2, GridWorld, and GPT-2 fine-tuning show that VRAdam achieves faster convergence, smoother training curves, and better generalization compared to AdamW and other optimizers.", "questions": "1. See weakness.\n\n2. The author choose the quartic kinetic term NRQCD system as T(v), is there any other choice? Can you report the ablation study since it deserves to be the key insights of the improvement against AdamW.\n\n3. How sensitive is VRAdam to the choice of the velocity penalizer β₃? Does it generalize well across tasks without tuning?\n\n4. Does the quartic kinetic term introduce any bias that could affect convergence to flatter minima or generalization in practice? Can it be integrated with other techiques to improve generalization like in ‘Improving Generalization of Deep Neural Networks by Optimum Shifting , AAAI25’?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The introduction of a quartic kinetic energy term as a stabilizing mechanism is a fresh and interesting physics-based perspective on optimizer design.\n\n2. The analogy between optimization trajectories and particle dynamics adds an intuitive understanding of how the method moves away from instability near the edge of stability.\n\n3. The paper rigorously proves global uniform exponential stability and convergence under mild conditions, supported by clear mathematical derivations.\n\n4. VRAdam consistently outperforms AdamW, RAdam, RMSProp, and SGD across diverse tasks (CNNs, Transformers, GFlowNets, and LLMs).", "weaknesses": "1. The presentation is not friendly to readers not familiar with optimization techniques using Langevin dynamics.\n\n2. Lots of hyperparameters are used. Although $\\beta_3$ is claimed to be robust, the practical sensitivity of VRAdam to its hyperparameters $like (\\alpha_0,\\alpha_1,\\beta_3)$ is not fully explored. Are they same with Adam or will be influenced by $\\beta_3$ ?\n\n3. While AdamW(2017) is a strong baseline, the study omits comparisons with newer optimizers (e.g., LION, AdaHessian), which are relevant for modern deep learning tasks.\n\n4. The heavy use of physical analogies (NRQCD, Lagrangians) brings difficulties to understand the motivation and improvement of the algorithm for readers unfamiliar with physics.\n\n5. The experiments only report the validation and test loss, the improvement against AdamW seems modest, whether VRAdam improves the task’s accuracy is not reported. Besides, no clear ablation on the contribution of the quartic term versus standard momentum damping—this would help isolate the true effect of velocity regularization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)