Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime
Authors: Beomhan Baek
Minhak Song
Chulhee Yun
Venue: ICLR 2026 Poster (2026)
Keywords: Adam
Signum
implicit bias
separable data
adaptive algorithms
mini-batch
Primary area: optimization
TL;DR: We characterize the implicit bias of mini-batch Adam on separable data, showing that it can deviate from the $\ell_\infty$-max-margin classifier, whereas Signum consistently preserves the $\ell_\infty$-bias for any batch size.

Abstract: Adam [Kingma & Ba, 2015] is the de facto optimizer in deep learning, yet its theoretical understanding remains limited. Prior analyses show that Adam favors solutions aligned with $\ell_\infty$-geometry, but these results are restricted to the full-batch regime. In this work, we study the implicit bias of incremental Adam (using one sample per step) for logistic regression on linearly separable data, and show that its bias can deviate from the full-batch behavior. As an extreme example, we construct datasets on which incremental Adam provably converges to the $\ell_2$-max-margin classifier, in contrast to the $\ell_\infty$-max-margin bias of full-batch Adam. For general datasets, we characterize its bias using a proxy algorithm for the $\beta_2 \to 1$ limit. This proxy maximizes a data-adaptive Mahalanobis-norm margin, whose associated covariance matrix is determined by a data-dependent dual fixed-point formulation. We further present concrete datasets where this bias reduces to the standard $\ell_2$- and $\ell_\infty$-max-margin classifiers. As a counterpoint, we prove that Signum [Bernstein et al., 2018] converges to the $\ell_\infty$-max-margin classifier for any batch size. Overall, our results highlight that the implicit bias of Adam crucially depends on both the batching scheme and the dataset, while Signum remains invariant.

Published as a conference paper at ICLR 2026
IMPLICIT BIAS OF PER-SAMPLE ADAM ON SEPARABLE DATA: DEPARTURE FROM THE FULL-BATCH REGIME
Beomhan Baek*t Seoul National University bhbaek2001 (at.k20000@euu..e @ssuuac.ke
Minhak Song, Chulhee Yun
KAIST
{mi nhaksong, chulhee, yun) kr
ABSTRACT
Adam (Kingma & Ba, 2015) is the de facto optimizer in deep learning, yet its theoretical understanding remains limited. Prior analyses show that Adam favors solutions aligned with fox -geometry, but these results are restricted to the full-batch regime. In this work, we study the implicit bias of incremental Adam (using one sample per step) for logistic regression on linearly separable data, and show that its bias can deviate from the full-batch behavior. As extreme example, we construct datasets on which incremental Adam provably converges to the 1a-max-marge classifier, in contrast to the loe max- margin bias of full-batch Adam. For general datasets, we characterize its bias using proxy algorithm for the 32 + 1 limit. This proxy maximizes data-adaptive Mahalanobis-non norm margin, whose associated covariance matrix is determined by data-dependent dual fixed-point formulation. We further present concrete datasets where this bias reduces to the standard lz- and classifiers. As counterpoint, et al., 2018) converges to the foe max--argin classifier for any batch size. Overall, results highlight that the implicit bias of Adam crucially depends on both the batching scheme and the dataset, while Signum remains invariant.
max-margin
prove that Signum (Bernstein
INTRODUCTION
The implicit bias of optimization algorithms plays crucial role in training deep neural net- works (Vardi, 2023). Even without explicit regularization, these algorithms steer learning toward solutions with specific structural properties. In wee-pparameterized models, where the training data can be perfectly classified and many global minima exist, the implicit bias dictates which solutions are selected. Understanding this phenomenon has become central to explaining why over-parameterized models often generalize well despite their ability to fit arbitrary labels (Zhang et al., 2017).
canonical setting for studying implicit bias is linear classification separable data with logistic loss. In this setup, achieving zero training loss requires the model's weights to diverge to infinity, making the directional convergence-whie defines the decision boundary- the key object of study, Seminal work by Soudry et al. (2018) establishes that gradient descent (GD) directionally converges to the l- f--max- marginn solution. This foundational result has inspired extensive research extending the analysis to neural networks, alternative optimizers, and other loss functions (Gunasekar et al., 2018 8b; & Telgarsky, 2019; 2020; Lyu Li, 2020; Chizat & Bach, 2020; Yun et al., 2021). this work, we revisit the simplest setting linear classification separable data examine how the choice of optimizer shapes implicit bias.
Among modern optimization algorithms, Adam (Kingma & Ba. 2015) is one of the most widely used, making its implicit bias particularly important to understand. Zhang et al. (2024a) show that, unlike solution. This behavior is closely
GD, full-batch Adam converges in direction to the loo related to sign gradient descent (SignGD), which can be interpreted as normalized steepest descent in the -nor and is also known to converge to the maax-aa direction (Gunasekar et al., 201 8a; Fan et al., 2025). Xie et al. (2025) further attribute Adam's empirical success in language model training its ability exploit the favorable lo -geometry of the loss landscape.
Authors contributed equally to this paper. Work done undergraduate intern KAIST

---

Published as a conference paper at ICLR 2026
Cosine Similarity MyMax
Similarity Margin
1.00
0.95
0.05
Cosine
0.80
GD
GD
0.75
botch Adam
Full batch Adam
With eppereeeeeeentt.cco Sizdhestic Adam
With comment Stochastic Adam
0.70
Random Reshufling ing Stochastic Azum
Random Reshuffiing Stochastic Adam
Adam
Adem
0.65
10a 10a
10'
10"
Iteratinss
Itarations
Figure 1: Mini-batch Adam loses the Lo max-margin bias of full-batch Adam. Cosine similarity between the weight vector and the f2-max-margin (left) and Ex-mma-marghn (right) solutions in linear classification task 10 data points drawn from the 50-dimensional standard Gaussian. Full-batch Adam with ($1, 32) = (0.9, 0.95) converges to the 1k-maa-marrge solution, whereas mini-batch variants with batch size of converge different direction (see Section 4 for the detailed characterization). See Appendix C. for experimental details.
Yet, prior work on implicit bias linear classification has almost exclusively focused on the full-batch setting. In contrast, modern training relies stochastic mini-batches, regime where theoretical understanding remains limited. Notably, Nacson et al. (2019) show that SGD with an arbitrary batch size preserves the same optimizer's 's implicit bias. But does this extend to adaptive methods such as Adam?
bias GD, suggesting that mini- bati-bathhing may not alter an
Does Adam's characteristic fox -biis persist under the mini-batch setling?
Perhaps surprisingly, we find that the answer is no. Our experiments (Figure illustratee that when trained on Gaussian data, full-batch Adam converges to the foc variants with a batch size of converge margin solution. To explain this phenomenon, we develop a theoretical framework for analyzing the implicit bias of mini-batch Adam and focus on the batch size case as a representative contrast to the full-batch regime. To the best of our knowledge, this work provides the first theoretical evidence describing the data-dependent characterization of the implicit bias of Adam with a batch size 1.
direction, whereas Adam different direction, which is even closer the f--max-
Our contributions are summarized follows:
We analyze incremental Adam, which processes one sample per step in a cyclic order. Despite its mamnentum-bsee updates, show that its epoch-wise dynamics can be approximated by recurrence depending only on the current iterate, which becomes a key tool in our analysis (see Section 2).
We demonstrate sharp contrast between full-batch and mini-batch Adam using a family of structured datasets, Scaled Rademacher (SR) data, SR data, we prove that incremental Adam converges to the f2-max-margen solution, while full-batch Adam converges to the loo -ea--max-aaaaaa solution (see Section 3).
For general dataset, we introduce a uniform-avvrrggnng proxy that characterizes the limiting behavior of incremental Adam as 32- - 1. We identify its convergence direction as the solution of data- adaptive margin- margin-maximinn maximization problem, induced by a Mahalanobis norm whose covariance matrix is determined by a data- depa-dee dual fixed-poin point equation. We further present concrete datasets where this bias reduces to the standard E2- and foo max-margin classifiers (see Section 4).
Finally, we prove that Signum (SignSGD with momentum; Bernstein et al. (2018)), unlike Adam, maintains its bias toward the fa--max-margin solution for any batch size when the momentum parameter is sufficiently close (see Section 5).
How CAN WE APPROXIMATE WIHOUT-REPLACEMENT ADAM?
For vector v, let vik] denote its k-th entry. For matrix M. let Mi. denote its (i. entry. We use -i to denote the probability simplex in R. Let [N] = {0, 1....... . N -1 denote the set of the first N non-negative integers. For PSD matrix M. define the Mahalanobis norm
Notation.
2

---

Published as a conference paper at ICLR 2026
Algorithm 1 -et--Adat -Adam
Algorithm Inc-Adam
Hyperparams: Learning rate schedule {n 1
Hyperparams: Learning rate schedule {m 12
momentum parameters B1, B2 E [0,
momentum parameters 31, 32 E [0,
Input: Initial weight Wo. dataset bx]
Input: Initial weight Wg. dataset {x, [ie
I: Initialize momentum m- v =
I: Initialize momentum m_1 = V 0
2: for 2,..2..... T do
for 0.1,2.... T 1 do
3: g: V[((w.)
3: g: <- VLi, (Wi), i, = [ mod N
4: m, f- 31m;- +
31)g
4: m; < f1m,- + +11 (1- (1-33)). g+
5: V: - -1 1 + (1 $2)gi
5: Vi f B2 + (1- B2)gi
6: Wi+1 f W; - 710 VVi
my
6: Wr+1 f We 1)t vVi
m.
7:
end for
7: end for
8: return WT
return WT
as | A Vx Mx. For vectors, v, ()), and operations applied entry-wise unless stated
otherwise. Given two functions f(t). we denote f(t) = O(g(t)) if there exist C,T > 0 such that / 2 T implies [ s Clg(()). For two vectors and w, we denote w if c w for positive scalar c > 0. Let = mod b denote the remainder when dividing by b, i.e., 0 < <
Algorithms. We focus on incremental Adam (Inc-Adam), which processes mini-batch gradients sequentially from indices 0 to N - 1 in each epoch. Studying Inc-Adam provides a tractable way to understand the implicit bias of mini-batch Adam under various sampling schemes: our experiments show that Enc-Adamt directional convergence closely aligns with that of Adam with a batch size of under both sampling with replacement and manomm-eshufffing. Sharing the same mini-bater accumulation mechanism, Inc-Adam serves as a faithful surrogate for theoretical analysis. Pseudocodes for Algorithms and 2.
and full-batch deterministic Adam (Det-Adam) are given in
Stability Constant c. In practice, we often consider an additional term for numerical stability and update with We+1 = W, 7) /v. In fact, when investigating the asymptotic behavior of Adam, the stability constant significantly affects the converging direction, since V. - 0 as ( -> oo and dominates vz. Wang et al. (2021) investigate RMSprop and Adam with the stability constant, yielding their directional convergence to fe-max-margin solution. More recent approaches, however, point out that analyzing Adam without the stability constant is more suitable for describing its intrinsic behavior (Xie & Li, 2024; Zhang et al., 2024a; Fan et al., 2025). We adopt this view and consider the version of Adam without
m.
Problem Settings. We primarily focus on binary linear classification tasks. To be specific, training data are given by {(x, ke)]e where x, E Rd E (-1,+11) and N is the number of data points. We aim to find linear classifier w which minimizes the loss
[(w)
1 N
Σ VE[N]
((w((w
1 N
Σ Ci(w), EE[M]
where E F - R is surrogate loss for classification accuracy and L:(W) Ely) (ly:(w, (w)))) denotes the loss value on the i-th data point. Without loss of generality, we assume y/i = +1, since we can newly define x, = y:x;- In this paper, consider two loss functions l E { ling). where fexp(z)= = exp(-22 denotes the exponential loss and (z) = log(1 + e-²) denotes the logistic loss. Given sequence of vectors {v. for notational simplicity, let vē VrN+s denote the s-th element the r-th group of N consecutive terms.
To investigate the implicit bias of Adam variants, we make the following assumptions. Assumption 2.1 (Separable data). There exists w E Rd such that w T x; > 0, Vi E [N]. Assumption 2.2 (Nonzero coordinates). For all E [N] and & E [d], x, [ + 0. Assumption 2.3 (Learning rate schedule). The sequence of learning rates, {1/ p satisfies
(a) {n )m is decreasing in E N7 oo, and lim,++00 T) = O.
(b) For all / E (0, 1), e > 0, there exist t E N4,02 > 0 such that Et=o (e Dimmne 1) 627) for all / 2 4.
3

---

Published as a conference paper at ICLR 2026
Assumption 2.1 guarantees linear separability of the data. Assumption 2.2 holds with probability 1 the data is sampled from a continuous distribution. Assumption 2.3 originates from Zhang et al. (2024a) and it takes crucial role to bound the error from the movement of weights. We note that a polynomial decaying learning rate schedule 7)2 = (t (t++3 2) E (0, satisfies Assumption 2.3, which is proved by Lemma C.I 1 in Zhang et al. (2024a).
The dependence of the Adam update on the full gradient history makes its asymptotic analysis largely intractable. We address this challenge with the following propositions, which show that the epoch-wise updates of Inc Inc-Adam and the updates of Det Adam can be approximated by function that depends only on the current iterate. Detailed proofs are deferred to Appendix D.
Proposition 2.4. Let {wo)i=o be the iterates of Det-Adam with B1 S 32. Then, under Assump- tions 2.2 and 2.3, if limm-+00 can be represented by
1//3 [(wr) [[Wws]][]
0, then the update of k-th coordinate We+i[k] - w.
We+1 [k] wolk) = 1/ (V..)(( +ex,
for some lim;- Et = 0.
Proposition 2.5. Let {wa) be the iterates of Adam with B1 S 32. Then, under Assump- tions 2.2 and 2.3, the epoch-wise update wh+1 w.. can be represented by
Σ EjEN 3/ V (w.))
w - w =-NN (Cme(B1, 32)
+ +
(2)
iE[N] V VEjE(N] (L,
where g((23 g(i-)) mad N
9,82 (.f(t))
1- 2N =32 (((-i)) mod N Cine (31,32) = I- T- is a function of
31, f2, and lim 6, 0. 1f = (t 2) for some E (0, 1]. then || = a1-
Discrepancy between Det -dda and Inc-Adam Propositions 2.4 and 2.5 reveal a fundamental discrepancy between the behavior of Det-Adam and one Inc-Adam. Proposition 2.4 demon- strates that bet-Adan can be approximated by SignGD, which has been reported by previous works (Balles Hennig, 2018; 8; Zou et al., 2023). Note that the condition not satisfied when V.(ww)) [k] decays at rate the order of 1/. 1/2 [(w.), which often calls for more detailed analysis (see Zhang et al. (2024a, Lemma 6.2)). Such an analysis establishes that Det ad asymptotically finds an Eoo- max-margin solution, property that holds regardless of the choice of momentum hyperparameters satisfying 31 S B2 (Zhang et al., 2024a).
In stark contrast, our epoch-wise wise analysis illustrates that Inc-Adam's updates more closely follow a weighted, preconditioned GD. This makes its behavior highly dependent on both the momentum parameters and the current iterate. The discrepancy originates from the use of mini-batch gradients; the preconditioner tracks the sum of squared mini-batch gradients, which diverges from the squared full-batch gradient. This discrepancy results in the highly complex dynamics of Inc-Adam, which are investigated in subsequent sections.
3 WARMUP: STRUCTURED DATA
Eliminating Coordinate-Adaptiiity. To highlight the fundamental discrepancy between Det-Adam and Inc-Adam, we construct scenario that completely nullifies the coordinate-wise adaptivity of Inc-Adam's preconditioner by introducing the following family of structured datasets. Definition 3.1. We define Scaled Rademacher (SR) data as set of vectors [xiNe which satisfy [x; [ = x [4],, Vk, E [d]. for each i [N]. We also assume that SR data satisfy Assumptions 2.1 and 2.2, unless otherwise specified.
Applying Proposition 2.5 to the SR dataset, we obtain the following corollary.
Corollary 3.2. Consider Inc Adam iterates {w, J on SR data. Then, under Assumptions 2.2 and 2.3, the epoch-wise update w w w.. can be approximated by weighted normalized GD, i.e.
as(r) VL(w2)]
w r+1
w.. -1rN
Σ NE(N]
[.s + Er
4

---

Published as a conference paper at ICLR 2026
Cosine Similarity MyMax
Similarity z. Mergin
1.00
0.96
0.94
0.92
0.90
GD
GD
botch Adam
Full batch Adam
With eppeecementttt Sizdhestic Adam
With Stochastic Adam
Random Reshuffingg ing Stochastic Azum
Random Reshuffiing Stochastic Adam
Adam
Adem
10a
10a
Itarations 10'
10
Ieeratonss
Figure 2: Mini-batch Adam converges to the ma- solution on the SR dataset. We train on the dataset X6 = (1, 1, 1, 1), X1 = 22 22,, X2 = (3,3,-3, -3), and x = (4, -4, 4, -4). Variants of mini-batch Adam with batch size of consistently converge to the direction, while full-batch Adam converges to the &o --a--mamaaaa -max-margin direction.
where lim, 6, = 0 and C1 S (r(r)) S C2 for some positive constants C1, only depending on 31, 82, (xilie(N) If = (t +++22 2) -M for some E (0, 1). then ||e, = O(r-a/2/
Although using structured dataset simplifies the denominator in Equation (2), the dynamics are still governed by weighted GD, which requires careful analysis. Prior work studies the implicit bias of weighted GD, particularly in the context of importance weighting (Xu et al., 2021; Zhai et al., 2023), but these analysis typically assume that the weights are constant convergent. In our setting, the weight (r) varies with the epoch count r. We address this challenge and characterize the implicit bias of Lnc-Adaa on the SR data as follows.
Theorem 3.3. Consider Inc-Adam iterates (weht with $1 S .2 SR data under Assump- tions 2.1 to 2.3. If (a) [(w,) + as I + oc and (b) = + 2)-a for a E (2/3, 1], then satisfies
lim W
Weg
where We, denotes the (unique) ma-max-margin solution of SR data {xI]ee[N)-
The analysis in Theorem 3.3 relies on Corollary 3.2, which ensures that the weights (r(r) are bounded by two positive constants, and This condition is crucial to prevent any individual data from having a vanishing contribution, which could cause the Inc-Adam iterates to deviate from the ma-max-margin direction. Furthermore, the controlled learning rate schedule is key to bounding the e, term in our analysis. The proof and further discussion are deferred to Appendix E. As shown in SR data confirm that bini-batch Adam with batch size of 1 converges
Figure 2, our experiments in direction to the & 2y-max ge--ma-marrgae margin classifier, in contrast to the Eor -bas of full-batch Adam.
Notably, Theorem 3.3 holds for any choice of momentum hyperparameters satisfying 81 S B.2 Figure 9 in Appendix B for empirical evidence. This invariance of the bias arises from the structure of SR data, which removes the coordinate adaptivity that momentum hyperparameters would normally affect. For general datasets, the invariance no longer holds; the adaptivity persists and varies with the choice of momentum hyperparameters, as discussed in Appendix A. the next section, we introduce a proxy algorithm to study the regime where 32 is close to I and characterize its implicit bias.
4 GENERALIZATION: ADAMPROXY
Uniform-Averagee Proxy. key challenge characterizing the limiting predictor of Enc-Adar for general datasets is that its approximated update (Proposition 2.5) is difficult to analyze directly. To address this, we study a simpler uniform-averaae proxy, derived in Proposition 4.1 under the limit B2 + 1. This approximation is well-motivated, 33 is typically chosen close to 1 in practice.
5

---

Published as a conference paper at ICLR 2026
Proposition 4.1. Let Ti be the iterates of Inc-Adam with 81 S 32. Then, under Assump- tions 2.2 and 2.3, the epoch-wise update w r++ w.. can be expressed as
w 0 r+1 - w. = -Pr (1
32
VL(w.)
+ t
B2
ΣN
VLi(w.)2
where lim sup, [r) E(32) and lim ((22)
Definition 4.2. We define update of AdamP roxa roxy
VL(w.)
6, Prx(ww (w,) A
i= VL,(w,)²
W++1 Wt 17:00-
Proposition 4.3 (Loss convergence). Under Assumptions 2.1 and 2.2, there exists positive constant > 0 depending only on the datasel {Xi)EIN), such that if the learning rate schedule satisfies S. and Et=o Tt = cc. then adamProx) iterates minimize the loss, i.e., lim L(w,) = 0.
To characterize the convergence direction of AdamProxy, we further assume that the weights [wr and the updates { converge in direction.
Assumption 4.4. We assume that: (a) learning rates { satisfy the conditions in Proposition 4.3, (b) 3 liim
[will w. and (c) 3 alim ||6,112
5,
8.
Lemma 4.5. Under Assumptions 2.1, 2.2 and 4.4, there exists с (Cu, CCN-1) E N such that the limit direction w of AdamP roxy satisfies
EieNy GX,
EIEM x?x
and c. 0f & S, where S arg min w x; is the index set of support vectors of w.
Prior research the implicit bias of optimizers has predominantly focused characterizing the convergence direction through the formulation of corresponding optimization problem. For example, the solution the ep-max-margin problem,
1/w1p subject to w T 1 2 0, Vi E [N]. 2
max
describes the implicit bias of the steepest descent algorithm with respect the fp-norm in linear classification tasks (Gunasekar et al., 20). However, Equation (5) does not correspond to the KKT conditions of conventional optimization problem. To address this, we introduce novel framework to describe the convergence direction, based on parametric optimization problem combined with fixed-point analysis between dual variables.
Definition 4.6. Given A-1, we define parametric optimization problem PAdam as
|w/ M(() subject to w x, - 2 0, Vi [N], Padam (c) min 2
(6)
where M(c) = diag( ESEIN
Iddxd We define p(c) the set of global optimizers of
PAsam(C) (c) and ddcc) as the set of corresponding dual solutions. Let 5(w) {i [N] w x; 1) denote the index set for the support vectors for any w E p(c).
Assumption 4.7 (Linear Independence Constraint Qualification). For any c N-1 and w E p(c). the set of support vectors (x.)ies(w) is linearly independent.
Assumption 4.7 ensures the uniqueness of the dual solution for (Aaam(C), which is essential for framework. This assumption naturally holds in the overparameterized regime where the dataset (x: (E)NE(N) consists of linearly independent vectors.

---

Published as a conference paper at ICLR 2026
Algorithm 3 Fixed-Point Iteration
Input:
Dataset {x initializatiio eo E A threshold Eür
1: repeat
2: Solve PAdan (Co) min gllw/ll(c.)) subject w X; 0,00000 e
3: f- Primal( PAdam)
4:
< Dual PAdam)
5: 8 ||c| coll2
6:
7: until 5 S Ethr
8:
return
Theorem 4.8. Under Assumptions 2.1 and 4.7, PAdam(C) admits unique primal and dual solutions, that p(c) and (cc) be regarded as vector-valued functions. Moreover, under Assumptions 2. 2.2, 4.4 and 4.7, the following hold:
(a) A N-N-- - Rd is continuous.
(b) d: A N-1 + R MO} is continuous. Consequently, the map T(c) A d(c)
is continuous.
[d(c]]
(c) The map T N N-1 admits least one fixed point.
(d) There exists c² E {c E A-11 T(c) = =c} such that the convergence direction w of AdamP roXy is proportional to p(c").
Theorem 4.8 shows how the parametric optimization problem Padam (c) captures the characterization from Lemma 4.5. The central idea is to treat the vector с from Equation (5) in a dual role: as both the parameter of P' I'Adam( (c) and its corresponding dual variable. The convergence direction is then identified at the point where these two roles coincide, leading naturally to the fixed-poine seed-point formulation. Detailed proofs are deferred Appendix F.
To computationally identify the convergence direction of AdamP roxy based on Theorem 4.8, we introduce the fixed-point iteration deseribed in Algorithm Numerical experiments confirm that the resulting solution accurately predicts the limiting directions of both AdamP roxy and Inc-Ade Adam (see Example 4. 4.10). However, the complexity of the mapping T makes it challenging to establish formal convergence guarantee for Algorithm 3. A rigorous analysis is left for future work.
Data-dependent Limit Directions. We illustrate how structural properties of the data shape the limit direction of AdamProx roxy through three case studies. These examples demonstrate that both AdamProxy and Inc Anc-Addam converge to directions that are intrinsically data-dependent.
Example 4.9 (Revisiting SR data). For SR data (xi [e]]) the matrix M(c) reduces scaled identity for every E N-1, Hence, the parametric optimization problem PAdam (c) narrows down to the standard SVM formulation
wll2 subject to w WTx, 0, Vi E [N].
Therefore, Theorem 4.8 implies that AdamP roxy converges to the l ma-max-margin solution. This finding is consistent with Theorem 3.3, which establishes the directional convergence of Inc-Adam on SR data. Together, these results indicate that the structural property of SR data that eliminates coordinate adaptivity persists in the limit 32 + 1.
Example 4.10 (Revisiting Gaussian data). We next validate the fixed-point characterization in Theorem 4.8 using the Gaussian dataset from Figure 1. The theoretical limit direction is given by the fixed point of ''' defined in Theorem 4.8, which we compute via the itcration in Algorithm 3. As shown Figure 3, both AdamE roxy and mini-batch Adam variants with a batch size of converge to the predicted solution, confirming the fixed- fixed-point formulation and the effectiveness of Algorithm 3. Furthermore, this demonstrates that, depending on the dataset, the limit direction of mini-bate Adam may differ from both the conventional (z- and fxc max-margin solutions.
7

---

Published as a conference paper at ICLR 2026
Cosine Similarity
Cosine Similarity Solution
1,00
0.95
Cosine Smilar
0.90
0.05
GD
63
botch Adam
FJll batch Adam
0.75
With riplacoment Studhestic Adam
Witn relibceme Stuchastic Adam Adam
Randomn Reshufl ing Stochastic Azam
Random Rushuftinng Stochastic Adam Incrumencal Adam
0.70
incrementa: Adam
Adammraayy
AdamPrrsyy
0.65
10a
10a
10'
10"
Reeations
Itarations
Figure 3: Mini-batch Adam converges to the fixed-point solution on Gaussian data. We train on the same Gaussian data as in Figure and plot the cosine similarity of the weight vector with the 1--max-marghh solution (left) and the fixed-point solution (right). The results show that variants of mini-batch Adam with batch size of converge to the fixed- point solution obtained by Algorithm 3, consistent with our theoretical prediction (Theorem 4.8).
Similartyy with Margin Solution,
Cosine Similartty L, Margin Solution,
1.0
Ohete Simberky
0.8
GD
GD
Fulibatch Adam
Pull-betch
Wit epphemepteetntt Sizdhestic Adam
With ment Stechastic Adam
Random Reshuffiing Sinchaelic
Random Reshuffing Stocheric
0.6
Adam
Adem
AdomFroxyy
AdomProky
10
Y0
Y6
10
10
10"
Rerations
Iterrtionns
Figure 4: Mini-batch Adam converges to the f-x-max-margnn solution shifted-diagonal dataset. We train on the dataset Xy = (1,8, 6, 6), x1 = (6, 2, 6)8, X2 = (6, 5, 4, 5), and x3 = (6, 6, 8) with d 0.1. Variants of mini-batch Adam with batch size of converge the -ma- margin direction.
Example 4.11 (Shifted-diagonal data). Consider N and {xi C Rd with x; x,e; 8 E e, for some and xo <... xd-1- Then, the -----------mm -max-margin problem
min 2 w subject to w x; 2 1, [
has the solution we -o- 1j 3o tid 1J8 Rd. Notice that c* 01, E d-1 is fixed point of T in Theorem 4.8, and w. p(c"); detailed calculations are deferred to Appendix Consequently, the lo -----axxmaaaaaa direction of AdamP roxy
solution serves a candidate for the convergence predicted by Theorem 4.8. To verify this, we run AdamP roxy and mini-batch Adam variants with a batch size of 1 on shifted-diagonal data given by X₂ = (1, (1,6, 8),
X1 (6, 2, 8, 5), X2 (8, 6, 4, d,,), and x; 5,, 5, 8) with 8 0.1. As shown in Figure 4, all mini-batch Adam variants converge to the l max-mnnn solution, consistent with the theoretical prediction.
A key limitation of our analysis is that it assumes 32 1 and batch size of 1. In Appendix A, provide a preliminary analysis of how batch size and momentum hyperparameters affect the implicit bias of mini-batch Adam. In particular, Appendix A.2 explains why our fixed-point framework does not directly extend to general B2
5 SIGNUM CAN RETAIN Ex -BIAS UNDER MINI-BATCH REGIME
In the previous section, showed that Adam loses its toc -ww-mma-mmmmaaa mar-mnrgiin bias under mini-batch updates, drifting toward data-dependent solutions. This motivates the search for a SignGD-rype algorithm that preserves loc -geometyy even in the mini-batch regime. We prove that Signum (Bernstein et al.,
8

---

Published as a conference paper at ICLR 2026
Similarity Margin
Similarity with Margin
Oavine Samllarty
0.8
GO
GD
0.7
Pullowee Sippum
Fall-betch Spnum
Incrementon memttal Signum (betch size
herr mentel Signum (betch dine
Indremento Sigumm (batch dize
Increme mentall Signum (batch dae
Indreme rtal Signum (batch dize 1I
Incre me stal Signum (batch dize
ementtal Adam (batch size
Increme Adam teatch size
0.5
16
10
10
16
10
10
10
107
Iterations
Iterations
Figure 5: Mini-batch Signum converges to the &oo max-m-aaariin solution. We train the same Gaussian data (NV = 10, d == 50) as in Figure 1, using full-batch Signum and incremental Signum with 3 0.99, for batch sizes b E (5, }). Across all batch sizes, incremental Signum consistently converges to the foc se--max-maa maxx-mrgiin solution, in sharp contrast to incremental Adam.
2018) satisfies this property: with momentum close to 1, its iterates converge to the -ax-max-arggin direction for arbitrary mini-batch sizes.
Theorem 5.1. Let å > O. Then there exists 0 such that the iterates {wo) of Inc- Signum (Algorithm 4) with batch size and momentum 3 E (1 (1-68 c, c, under Assumptions 2.1 and 2.3, satisfy
minge[N]
x; We
lim inf
7o0 - 6
Ww 00
where
Yoc 2
min w TTxx X4, | Si [
D 2 max ||x||1,, iE N
max
and
=
N min 5
e=1 if N. fo, 2 i,
2D
Theorem 5.1 demonstrates that, unlike Adam, Signum preserves fae -ax marginn bias for any batch size, provided momentum sufficiently close to 1. This generalizes the full-batch result of Fan et al. (2025). Moreover, the requirement 3 is merely technical but necessary in the mini-batch setting to ensure convergence to the fa--max-margin solution; see Figure 10 in Appendix for empirical evidence, As shown in Figure 5, our experiments on the Gaussian dataset from Figure I show that Inc- Sigumm (8 = 0.99) maintains -bias, regardless of the choice of batch size. Proofs and further discussion are deferred to Appendix G.
RELATED WORK
Understanding Adam. Adam (Kingma & Ba, 2015) and its variant AdamW (Loshchilov Hutter, 2019) are standard optimizers for large-scale models, particularly in domains like language modeling where SGD often falls short. A significant body of research seeks to explain this empirical success. One line focuses on convergence guarantees. The influential work of Reddi et al. (2018) 8) demonstrates Adam's 's failure to converge on certain convex problems, which motivates numerous studies establishing its convergence under various practical conditions (Défossez al., 2022: Zhang et al., 2022; Li et al., 2023; Hong & Lin, 2024; Ahn & Cutkosky, 2024; Jin et al., 2025). Another line investigates why Adam outperforms SGD, attributing its success to robustness against heavy-tailed gradient noise (Zhang et al., 2020), better adaptation to ill-conditioned landscapes (Jiang et al., 2023; Pan & Li, 2023), and effectiveness in contexts of heavy-tailed class imbalance gradient/Hessian heterogeneity (Kunstner et al., 2024; Zhang et al., 2024b; Tomihari & Sato, 2025). Ahn et al. (2024) further observe that this performance gap arises even in shallow linear Transformers. Recent works investigate how the choice of momentum hyperparameters (Orvieto & Gower, 2025) and the rotation operation (Zhang et al., 2025) affect the performance of Adam.
Implicit Bias and Connection to loc Geommetr A growing body of work examines Adam's implicit bias and its connection to &ec -ee-meeeery. This link is motivated by Adam's similarity to SignGD (Balles & Hennig, 2018; Bernstein et al., 2018), which performs normalized steepest descent under the Ex norm. Kunstner et al. (2023) show that the performance gap between Adam and SGD increases with batch size, while SignGD achieves performance similar to Adam in the full-batch

---

Published as a conference paper at ICLR 2026
regime, supporting this connection. Zhang et al. (2024a) prove that Adam without stability constant converges to the -mx maax-mrriin solution in separable linear classification, later extended to multi- class classification by Fan et al. (2025). Tsilivis et al. (2025) investigate implicit bias of steepest descent homogeneous neural networks, supporting that SignGD describes a typical dynamics of Adam. Complementing these results, Xie & Li (2024) show that AdamW implicitly solves fx--oomm-contrrinee optimization problem, connecting its dynamics to the Frank- Wolfe algorithm. Exploiting this fo -geometry is argued to be key factor in Adam's advantage over SGD, particularly for language model training (Xie et al., 2025). Vasudeva et al. (2025) examine how Adam and GD show different implicit biases when training two-layer ReLU networks, describing Adam's 's richer and more diverse decision boundary.
7 DISCUSSION AND FUTURE WORK
We studied the convergence directions of Adam and Signum for logistic regression on linearly separable data the mini-batch regime. Unlike full-batch Adam, which always converges to the lo -o--ma--arggnn solution, mini-batch Adam exhibits data-dependent behavior, revealing a richer implicit bias, while Signum consistently preserves the lo 1e-maa-marge bias across all batch sizes.
Toward understanding the Adam-SGD gap. Empirical evidence shows that Adam's advan- tage over SGD is most pronounced in large-batch training, while the gap diminishes with smaller batches (Kunstner al., 2023; Srećković al., 2025; Marek al., 2025). Our results suggest possible explanation: the fx -ddapiiiy of Adam, proposed as the source of its advantage (Xie et al., 2025), may vanish in the mini-batch regime. An important direction for future work investigate whether this loss of foo -adaptiviy extends beyond linear models and how it interacts with practical large-scale training.
Limitations. Our analysis for general dataset relies on the asymptotic regime 82 incremental Adam as a tractable surrogate. Extending the framework to incorporate general 32 < 1, larger batch sizes, and common sampling schemes (e.g., random reshuffling) would make the theory more complete. See Appendix A for further discussion. Developing additional theoretical tools that can be applied under weaker assumptions also remains an important direction.
and
ACKNOWLEDGMENNTS
This work supported by the National Research Foundation of Korea (NRF) grants funded by the Korean government (MSIT) (No. RS- AS-2023-0021 1352; No. RS-2024-00421203) and the InnoCORE program of the Ministry of Science and ICT (No. N10250156).
REPRODUCIBILITY STATEMENT
All assumptions and theorems for our theoretical results are stated the main paper, with their complete proofs deferred to Appendices D) to G. The primary experimental setups are described in the main paper and Appendix C, while details for supplementary experiments are provided in Appendices B and C.
DECLARATION OF LLM USAGE
The authors utilized LLMs to improve the grammar and readability of this manuscript. The core conceptualization, analysis, and writing of the content were performed exclusively by the authors.
REFERENCES
Kwangjun Ahn and Ashok Cutkosky. Adam with model exponential moving average is effective for nonconvex optimization. In The Thirty-ighth Annual Conference Neural Information Processing Systems, 2024. URL https: http:////ppeeeee.ee.c.mmm ottpre/rwweeeeettooooooooooaaao
10

---

Published as a conference paper at ICLR 2026
Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Ali Jadbabaie, and Suvrit Sra. Linear attention (maybe) all you need (to understand transformer optimization). In The Twelfth International Conference Learning Representations, 2024. URL httts::/oppereeiiee net. fert/forum? d=0uu5415
Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients, 2018. URL https: / openreview.eeeeeeeeeeeeeee www.:/.eewo.ueieieeoeeemmmmm wtt.:iddddswwwwwt000oo.. 2,
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadeneshelii and Animashree Anandkumar.
signSGD: Compressed optimisation for non-convex problems. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 560-569, PMLR, 10-15 Jul 2018. URL www.iinggsm........ www.m.rrr.ess.. vv0 htt0///ensteein.ncmmemee 8a. html. (document), 5,
https: https///pocceeenegeeeee
Lénaïc Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Jacob Abernethy and Shivani Agarwal (eds.), Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 1330--338 PMLR. 09-12 Jul 2020. URL https: ppw.eedingg......eeeeee........p mml..
press/v1255 chi aa html.
Alexandre Défossez, Leon Bottou, Francis Bach, and Nicolas Usunier. simple convergence proof of adam and adagrad. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https: hwtp//eeeeeeeeeeeeeeeeeeeeeeeeep net wwwuummdddPPhTTSSt00
Steven Diamond and Stephen Boyd. CVXPY: Python-embeddeed modeling language for convex optimization. Journal of Machine Learning Research, 17(83)11-5, 2016. C
Chen Fan, Mark Schmidt, and Christos Thrampoulidis. Implicit bias of spectral descent and muon on multiclass separable data, 2025. URL https:://aaaeee org/ abs/2502. 04 646. 1, 2, G, 1.1
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In Jennifer Dy and Andreas Krause (, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1832-1841. PMLR, 10-15 Jul 2018a. URL https: / http//poooceigggeeoee mlr.
veeas/v80// press/ v80 guw.sekaraaaaaaaamam Ba. html. 1, 4, G
Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018b. URL https: http:/::////ccceeeeeeeeee nwwripe. wteurip..ccppppprrrr saapper_fieesppaaee /tt__fess//aape//11..... http:///eBBeeeeeeeeceeeeebeeeeee Sacpe9s1cc--aappe..eeeeee pat.e.paa.p.........ee.
Yusu Hong and Junhong Lin. On convergence of adam for stochastic optimization under relaxed assumptions. In The eiihty- Annual Conference on Neural Information Processing Systems, 2024. URL https::tp//oc........ http//ppnvvvveeeeee net. / forum? d=x 7usmi dzx 6
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations, 2019. URL https: / /pwp//veee..e..eeeeeeee
net/ ftt::/ooummm//o/m/ o
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 17176-17186. Curran Associates, Inc., 2020. URL https: https//////coceeeooesseoo neurips, wctpappr_fii.... httprrrrriilleeape/2/0000 f
Ziwei Ji, Miroslav Dudík, Robert E. Schapire, and Matus Telgarsky. Gradient descent follows the regularization path for general losses. In Jacob Abernethy and Shivani Agarwal (eds.). Proceedings of Thirty Third Conference Research, pp. 2109 2139-2136. PMLR. 09-12 Jul 2020. URL https: http:///ooccedddeeeee mlr. preas/v1 22 20 html. E.2, F.2, E.2, E.3
Learning Theory, volume 125 of Proceedings of Machine Learning

---

Published as a conference paper at ICLR 2026
Kaiqi Jiang, Dhruv Malik, and Yuanzhi Li. How does adaptive optimization impact local neural net- work geometry? In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. -305-8384, Curran Asso- ciates, Inc., 2023. URL httpss::p ptpppppceeeeeeerr.rrropo.ppppepp 2023
neur
Ruinan Jin, Xiao Li, Yaoliang Yu, and Baoxiang Wang. A comprehensive framework for analyzing the convergence of adam: Bridging the gap with sgd. 2025. URL htLLt:t/////aa...ooo ors/abs/ 2410. 04458. 6
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, 2015. URL httttp:://axxi..orggg
1412.6980 (document), 1,
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. In The Eleventh International Conference on Learning Representations, 2023. URL https: / //openrevi ew net/ forum? d=a d= 65YK0cqH88. 6.7
Frederik Kunstner, Robin Yadav, Alan Milligan, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why adam outperforms gradient descent language models. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:
// //pppprreveww.rr wtw.eewwnee//ooroie wea.mmiddd66jjjf 1o j6aV800
Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. Convergence of adam under relaxed assumptions. In Thirty-seventh Conference Neural Information Processing Systems, 2023, URL https: //ppp/eeeiwww.... net/ forum? Eww.dddw.e.i.i......
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer-
ence Learning Representations, 2019. URL https: /ttttp/e///penevieew.. htw.:eewweee/fformmeee
Bkg6RiCqY7. 6
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2020. URL https:://ppenreeiee. net / ferum?ormmm Sed=SJJeLgBBK S.
Martin Marek, Sanae Lotfi, Aditya Somasundaram, Andrew Gordon Wilson, and Micah Goldblum. Small batch size training for language models: When vanilla SGD works, and why gradient accumulation is wasteful. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: wpw.reveeeeeeeeeeeeeeeeeeeeeeeee httpeleeeeeeeeeeeeeeeeeeeeeeeeee fwwmmmidummssr 5-52hpee o
Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent on separable data: Exact convergence with fixed learning rate. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty-Secome International Conference on Artificial Intelligence and Statisties, volume 89 of Proceedings of Machine Learning Research, pp. 3051-3059, PMLR, 16-18 Apr 2019. URL https: http:///ooeeeeeeeeeee/eeee mml. /eeess/v89.. wtw/////////nnooeemeee 9a html. setpeeiiinnmm...sse..e..
Antonio Orvieto and Robert M. Gower. In search of adam's secret sauce. The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025, URL /tttps://openrrvee/ net bett/ooze forum? 1d=CH72Xy2s4y. 6
Yan Pan and Yuanzhi Li. Toward understanding why adam converges faster than sgd for transformers. arXiv preprint arXiv:2306.00204 2023. 6
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018. URL https::/openreevee:
net/ (urt//Eoum?2 fiaimmidd--yyoo 1ywiddurrr----a- RZ. 6
Daniel Soudry. Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1-57, 2018. URL http: / jm1 r. org/ wwt.g//aprs//eeee vl 9 8- 188 html. 1, F.3, F.3
12

---

Published as a conference paper at ICLR 2026
Teodora Srećković, Jonas Geiping, and Antonio Orvieto. Is your batch size the problem? revisiting the adam-sgd gap in language modeling. arXiv preprint 2ttt::2506.125433 2025.
Akiyoshi Tomihari and Issei Sato. Understanding why adam outperforms sgd: Gradient heterogeneity in transformers. arXiv preprini urXiv:2502.00211, 2025. 6
Nikolaos Tsilivis, Gal Vardi, and Julia Kempe. Flavors of margin: Implicit bias steepest descent in homogeneous neural networks. In The Thirteenth International Conference on Learning Representations, 2025. URL https: htt.tttps///ppneeeeeeeeeeeeee
wtw.:eeee.oorrmmiiiaasaaaaaaaa 9r.
Gal Vardi. On the implicit bias in deep-learning algorithms. Commun. ACM, (6(6))66-633) May 2023. ISSN 0001-0782. doi: 10.1145/3571070. URL /
Bhavya Vasudeva, Jung Whan Lee, Vatsal Sharan, and Mahdi Soltanolkotabi. The rich and the simple: On the implicit bias of adam and SGD. The T'hirty-iith Annual Conference on Neural Information Processing Systems, 2025. URL pectps://openreee net forum? id- XL-.LLLwnzzxxx...
Bohan Wang, Qi Meng, Wei Chen, and Yan Liu. The implicit bias for adaptive optimization algorithms on homogeneous neural networks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine URL https: : / http//poooee.g....//o////
Learning Research, pp. 00849-10858. PMLR, 18-24 Jul 2021.
mlr. pwresppess/oo.mm. / v1 13 www//aan111 g html.
Shuo Xie and Zhiyuan Li. Implicit Bias of AdamW: & Constrained Optimization. In Forty-first International Conference on Machine Learning, 2024. URL httphtttss///peerreee/
net/ forum? ?
dtte:/oommmmmmmmmamaammaa d=CmXkd106613 2,
Shuo Xie, Mohamad Amin Mohamadi, and Zhiyuan Li. Adam Exploits -geomery of Loss Land- scape via Coordinate-wise Adaptivity. In The Thirteenth International Conference on Learning Representations, 2025. URL https:: http:////peeeeeeeeeeeeee ret. forum?i SerrmmmidPUUUDBUUEEE 1, 6,
Da Xu, Yuting Ye, and Chuanwei Ruan. Understanding the role of importance weighting for deep learning. In International Conference on Learning Representationss 2021. URL https: //openreview.o / forum?id= wttriiiidddttteeeee.b.eee 3, E.2
Chulhee Yun, Shankar Krishnan, and Hossein Mobahi. unifying view on implicit bias in training linear neural networks. In International Conference on Learning Representations, 2021. URL 4MMKK..
/ttps:////ppeeeeeeeeee net
forum?id=Zs2m 2
Runtian Zhai, Chen Dan J Zico Kolter, and Pradeep Kumar Ravikumar. Understanding why generalized reweighting does not improve over ERM. In The Eleventh International Conference on Learning Representations, 2023. URL httph:tt///c/ h/tp:::ss///p//evviiiie ew. net/ Forum? d=ashPcee_ W 3.2
Chenyang Zhang, Difan Zou, and Yuan Cao. The implicit bias of adam on separable data. In The ehirth-eighth Annual Conference Neural Information Processing Systems, 2024a. URL htttps: //pperevveww... ieweiewwneee net www.nrewwwww.nee/orummmm forum? di= 1, 2, 2, 6, D.1, G, L1, (b), 1.2,
I.3, 1.4, I.6
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017. URL https:
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, San- jiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 15383-15393. Curran Associates, Inc., 2020. URL https: httppp:://ppcceedinge.. neurips. cttpeeerreeepeeeeeeeeeeeeeeeeeee firrffiiees.. files/paperr2222/ f #illebb5tt57ffadde 0ag.aa7776aaa1000000000606660000
pddf.
13

---

Published as a conference paper at ICLR 2026
Tianyue H. Zhang, Lucas Maes, Alan Milligan, Alexia Selicoeur-Martinee Ioannis Mitliagkas, Damien Scieur, Simon Lacoste-Julien, and Charles Guille-Escuret. Understanding adam re- quires better rotation dependent assumptions. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: /openrvveeeeeveeee/eeeeeeee ew. net/ forum? d= KDD4gunbhh. 6
Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi thi-Quan Luo. Adam can converge without any modification on update rules. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp- 28386-28399. Curran Associates, Inc., 2022. URL https::/prroeeedinggg neurips. wttp:ppsscppppppppppppep http:p///poocccci......orrcprppp
wttpppppe_f.lleeppppeee////////e file file/b6260aee hlleeeD000sseeeaaasaaaaa66022222 Papenaaaaapppaaaaaaa Conference pdf. 6
Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why trans- formers need adam: A hessian perspective. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https:/ https::/openrevaae wpwpppppenneeeeeeeeeeee net forum?id- X6rqEpbnj3. 6
Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. Understanding the generalization of adam in learning neural networks with proper regularization. The Eleventh International Confer- ence on Learning Representations, 2023. URL /atpss//pppeeeeeeeeeeeee ew net heetfforu?iid
1UYpN1 2, D. D.I,
14

---

Published as a conference paper at ICLR 2026
Appendix
Further Discussion
16
A.I Effect of Hyperparameters Mini-batch Adam
16
A.2 Can We Directly Analyze Inc-Adam for General 3g?
16
Additional Experiments
18
C Experimental Details
18
Missing Proofs in Section
21
D.I Proof of Proposition 2.4
21
D.2 Proof of Proposition 2.5
22
Missing Proofs in Section
27
E.1 Proof of Corollary 3.2...
27
E.2 Proof of Theorem 3.3
28
F Missing Proofs in Section
32
F.1 Proof of Proposition 4.1
32
F2 Proof of Proposition 4.3..
32
F3 Proof of Lemma 4.5
34
F.4 Proof of Theorem 4.8
38
F.5 Detailed Calculations of Example 4.
39
G Missing Proofs in Section
40
Missing Proofs in Appendix A
48
I Technical Lemmas
48
L11 Proxy Function
48
I.2 Properties of Loss Functions
49
L.3 Auxiliary Results
50
15

---

Published as a conference paper at ICLR 2026
A FURTHER DISCUSSION
A.1 EFFECT OF HYPERPARAMETERS ON MINI-BATCH ADAM
The scope of our analysis does not fully encompass the effects of batch sizes and momentum hyperparameters on the limit direction of mini-batch Adam. To motivate further investigation, this section presents preliminary empirical evidence that shows the sensitivity of the limit direction to these choices.
Effect of Batch Size. To investigate the effect of batch size on the limiting behavior of mini-batch Adam, we run incremental Adam the Gaussian data with N = 10, 50, varying batch sizes among 1, 2, 5, and 10. Figure 6 shows that as the batch size increases, the cosine similarity between the iterate and max-maa-nr solution increases. This result suggests that the choice of batch size does affect the limiting behavior of mini-batch Adam, wherein larger batch sizes yield dynamics that converge towards those of the full-batch regime. A formal characterization of this dependency presents compelling direction for future research.
Costne Simllantt with 4 Max Margin Solution
Similarity with Max Solution
1.0
Coatre Simllarky
0.9
0.8
0.7
Full batch Alaam
Full sotich Adam
mnneomenta Adam (batch dize 51
Incrementa Asia (batch dee
0.6
Incrementa Adam (bstch slaw 21
Increments Adam (batch sice
Incramenta Adam (batch slaw 11
Incrementa Adam (batch sice
0.5
10
10
10
10a
10"
10
10
10
Reeations
Iterations
Figure 6: The choice of batch size influences the limit direction of mini-batch Adam. We train on the same Gaussian data (N = 10, d = 50) as in Figure and plot the cosine similarity of the weight vector with the lg fa-max-margin solution (left) and the - maa-mmagiin solution (right), varying batch sizes in {1. 2, 5, 10). As the choice of batch size becomes closer to 10 (full-batch), the limit direction aligns closer to be --a--ma--maaa max-manginn solution.
Effect of Momentum Hyperparameters. Theorem 4.8 characterizes the limit direction of AdamProxy, which approximates mini batch Adam with batch size of one in the high-3g regime. We investigate how this approximation fails in the different choice of momentum hyperparameters. Revisiting the Gaussian data with N = 10, d = 50, we run mini-batch Adam with batch size of 1 (including Inc-Adam) using LR schedule тソ = ((-00.8) varying the momentum hyperparameters (B1,8)) {0.11, 0.95), (0.5, 0.95), (0.9, 0.95), (0.1, 0.1), (0.1, 0.5), 0.9).
The first experiment investigates the influence of 3, by varying 31 E {0.1, 0.9} while maintaining
high choice of (3 0.95. The results, presented in Figure 7, demonstrate that 31 does not affect the convergence direction. This finding validates Proposition 4.1, which posits that our AdamP roxy framework accurately models the high-82 regime, regardless of the choice of 31.
Conversely, the choice of 32 shows to be critical. We sweep 82 E {0.1, 0.5, 0.9} while maintain- ing 81 0.1 and plot the cosine similarities in Figure 8. The results illustrate that for choices of B2 E {0.1, 0.5), the trajectory of mini-batch Adam deviates from the fixed-point solution of Theorem 4.8. It indicates that the high-3, condition is crucial for the approximation via AdamP roxy and characterizing the limit direction of mini-batch Adam in the low-32 regime remains an important future direction.
A.2 CAN WE DIRECTLY ANALYZE INC-ADAK FOR GENERAL /32
As empirically demonstrated in Appendix A. the selection of 32 alters the limiting behavior of Inc-Adam. This observation motivates an inquiry into whether our exxed-point formulation can be directly generalized to accommodate general choices of B2, based a more general proxy algorithm. We proceed by outlining the technical challenges that prevent such a direct application of our framework, even under stronger assumption on B1 and the behavior of w,.
16

---

Published as a conference paper at ICLR 2026
Cosine Simlanly. (0..0.55)
Cosine Samilarky, 10.55.05
Cowine Simierky, By. Pal (2.11
1.000
11:22
1:300
Witherp soamer Sccrest
Mopooppeemmet Noom
Wicheallomm (Stncharts Adim Hncuamnted Ade
125
10 Joam
Ioveerid Adam
Herrations
Baralions
Figure 7: B1 does not affect the convergence direction of mini-batch Adam for large 82. We train on the same Gaussian data as in Figure 1, varying 31 E 0.59,0.5, 0.1) with fixed |3 = 0.95, and plot the cosine similarity between the weight vector and the fixed-point solution (Algorithm 3). All mini-batch Adam variants with a batch size of consistently converge to the fixed-point solution.
Cosine Siilarty.co (1..8)1(8...0
Cosine Smiariy, 18.01.00...51
Cosine Similarty,
m4
Weveeraceent Steheskk Adar lorda Liiiing Socreate
moneeceemme Stachalic
fandor bestuliigg Stataricc
Ttar otions
Iterations
Ittrationa
Figure B2 affects the convergence direction mini-batch Adam. We train on the same Gaussian data as in Figure 1, varying 32 E {0.9, 0.5, 0.1} with fixed 31 0.1, and plot the cosine similarity between the weight vector and the fixed-point solution (Algorithm 3). Mini-batch Adam variants with batch size of 1 deviate increasingly from the fixed-point solution B3 decreases.
Let [w:] } be the Aac-Adaam iterates with 31 O. For simplicity, we only consider the epoch-wise update and denote w, wh. 1/7 Cine (0, 82 D abuse of notation. By Proposition 2.5, W, can be written by
(.,(w,)
8,
Σ
+e,
Σ
SEENI a
(W,
NE
Wr+1 W, = -7, 5,
for some E, - O. Note that replaces SalamProxy in Section 4, incorporating the rich behavior induced by general 32. Then, we provide a preliminary characterization of the limit direction of Inc-Adam as follows.
Lemma A.1. Suppose that (a) [(w,) + and (b) w, = ||w, 12w + p(r) for some w with Blim, p(r). Then, under Assumptions 2. and 2.2, there exists e (co, ... ", .CN-1) N such
that the limit direction W of Inc-Adam with 31 = O satisfies
Σ
RE(N
EjE[N
Σ
and c 0. & S, where S arg min w is the index set of support vectors of w.
We recall that the fixed-pont point formulation in Theorem 4.8 arises from constructing an optimiza- tion problem whose KKT conditions are given by Equation (5) fixing the c's in the denomi- nator; the convergence direction is then characterized when the dual solutions of the KKT con- ditions coincide with the q's in the denominator. Therefore, to establish an analogous fixed- point type characterization, d.x;
should construct an optimization problem whose solution is given with dual variables d; > 0 satisfying that d, for ex
by w EiE[ VEJENN
j S = arg min w X,
However, this cannot be formulated via KKT conditions of an optimization problem. The in- dex set S indicates support vectors with respect x,, while our dual variables multiplied to
17

---

Published as a conference paper at ICLR 2026
x.(c). A notable direction for future work is to generalize the proposed
v Ex 8/2 (1 c cfx
methodology for arbitrary values of $2-
B ADDITIONAL EXPERIMENTS
Supplementary Experiments in Section 3. To investigate the universality of Theorem 3.3 with respect to the choice of the momentum hyperparamettrs. We run mini-batch Adam (with batch size on SR dataset Xo (1, 1, 1, 1). xi = 2, 2, -2), X2 (3,3 -3, -3) and X3 (4, 4,,44,44 4, -4), varying the momentum hyperparameters (B1,8g) E demonstrates that its limiting behavior toward 1a-max-margin solution consistently holds on the broad choices of (2).
(0.9, 0.95) (.5,, Figure
Supplementary Experiments in Section 5. Theorem 5.1 demonstrates that Inc-St Siguum main- tains its bias to maxx-mann solution, while the momentum hyperparameter 3 should be close enough to depending on the choice of batch size; the gap between and 1 should decrease as run Inc-Siggnt
batch size b decreases. To investigate this dependency, sian data as in Figure 1, varying batch size E {1, 2, 5, 10) and the momentum hyperparameter (0.5, 0.9, 0.95, 0.99). Figure 10 shows that maintain the -bias, the choice of /3 should be closer to 1 as the batch size decreases.
the same Gaus-
C EXPERIMENTAL DETAILS
This section provides details for the experiments presented in the main text and appendix.
We generate synthetic separable data follows:
Gaussian data (Figures 1, 3, 5, 6 to 8 and 10): Samples are drawn from the standard Gaussian distribution (0,, /). We set the dimension margin that the data is linearly separable.
50 and sample N 10 points, ensuring a positive
Scaled Rademacher (SR) data (Figures 2 and 9): We use (1,1,1,1), 2,, -2).
X₂ = (3.3, -3, -3), and X3 (4, -4, 4, -4).
Shifted-diagonal data (Figure 4): We use xo (1,6,8,8), (6,2,5,0), x2 66,6,4,0) and
X3 = (ó, 6, 6, 8) with = -.1.
We minimize the exponential loss using various algorithms. Momentum hyperparameters are (81, B2) = (0.9, 0.95) for Adam and [ 0.99 for Signum unless specified otherwise. For Adam and Signum variants, we use a learning rate schedule π)- = no 2)-a with r/ = 0.1 and a = 0.8, following theoretical analysis. Gradient descent uses fixed learning rate π) nlu = 0.1. Margins with respect to different norms are computed using CVXPY (Diamond & Boyd, 2016).
The fixed-point solution (Theorem 4.8) is obtained via fixed-point iteration (Algorithm 3) for Figures 7 and 8. We initialize co (1/N..... 1/N) E N-1, set the threshold <hr 10-8, and converge to the fixed-point solution within 20 iterations in all settings.
18

---

Published as a conference paper at ICLR 2026
Cosine Similarity with byMaaa Margin Solution
Cosine Similarity with Lu Maraaa-aarain Solution
1.00
Cosnn Siilarts
0.92
0.30
0.88
rpfacoment
resladement Stachastic
incrementa Stochastic
Reshuffing Stochostic
0.86
Incrumental Adam
10"
10
Reeations
Iterations
(81, 82) (0.1,0.1)
Cosine
Solution
Casina Similarity
1.00
0.38
Cosnee Simbarks
0.96
0.94
0.92
GD
63
Fullbbtch Adem
null-wttch Adem
With replacoment Studhastic Adam
With resibcement Stochastic Adam
Random Reshuffiing Stedhastic Adam
Random Reshufting Stochastic Matm
incrementa: Azam
Incrumencal Adam
16a
10a
10
16
10a
10a
10
16"
Iterations
Iterations
(b) (3,, = (0.5, 0.5)
Cosine Simllarity with J.Max. Solution
Casine Similarity with I. Max Margin Sclution
1.00
Sinlarr
0.94
0.92
0.90
0.88
wethesplaccment
0.86
10
(c) (81, 32) (0.9, 0.95)
Figure 9: Mini-batch Adam converges to the max l mm-man solution for SR data. We train on SR dataset X 1,, x1 2, -2), x2 (3, -3 -3), and X3 (4, -4, 4, -4). varying the momentum hyperparameters In all tested configurations, the family of mini- batch Adam algorithms with a batch size of 1 converge to the & max-mare max-margin solution, which deviate significantly from the loo bias of full-batch Adam.
19

---

Published as a conference paper at ICLR 2026
Casine Similarity MyMax Margin
Similarity with Margin
Cosine Sievlarit
0.9
0.8
0.7
hull batch Signum
Full bbtth Signum
Incrementon nial Signum (bstri size 5)
tocre me stal
Signum (batch die
Incremental Signum (betch sizw 21
There mentrll Signum (betch dine
Indrementol Signum (batch dite 11
Incremerte noa Signum (batch dbe
0.5
10
10
10
16
10
10
10
10"
Iterations
Iterations
(a)
8 = 0.5
Cosine Similarity with Max MaaM-Marrgn Margin Solution
Cosine Similarity with I. Max Maraa-Marinn Solution
1.0
Coliree Similart'
0.8
0.7
botch
0.6
Signum
Signum
Signum
Sigum
Signum
Rerations
Iterations
(b) 0.9
Casine Similarity with Mangin Solution
Casina Similarity with Margin Solution
1.0
0.9
0.8
Rul batch Sgrum
batch Signum
Incremienaal Signum (batch dize 51
Incrementt stal Signum (batch dice
Indremental rtal Signum (batch daz 21
Increme stal Sipnum (batch doc
Incrementall Signum (batch sizw 11
The me- stal Signum (beich siee
0.5
10
10
10
10
10a
10
10
10"
Ieeations
Iterations
(c) B
0.95
Costne Simllartt with Solution
Casine Similarity with I. Max Marain Sclution
1.0
Casire Samilertty
0.8
0.7
Rullbwecc Sqgum
Full ail-batch Stonum
Indromettal
(batch (batch
0.6
Incremetta
Signum
Signum
10
Rerations
Iterations
(d) B = 0.99
Figure 10: Effect of Batch Size on Inc-Signum. We run Inc- gnuummmmmmmm on the same Gaussian data (N 10, d 50) as in Figure 1 and plot the cosine similarity of the weight vector with the ma-gann solution (left) and the Lox -max-marriin solution (right), varying batch size b E (1, 5, 10} and the momentum hyperparameter 8 {0.5, 0.9, 0.95. 0.99). As the batch size decreases, we should choose 3 closer to I to maintain the limit direction toward lo -mx margin solution.
20

---

Published as a conference paper at ICLR 2026
D MISSING PROOFS IN SECTION 2
In this section, we provide the omitted proofs in Section 2, which describes asymptotic behaviors of Det-Adam and Inc-Adam, We first introduce Lemma D.1 originated from Zou al. (2023, Lemma A.2), which gives a coordinate-wise upper bound of updates of both Det -ddam and Inc-Adam. Then, prove Propositions 2.4 and 2.5 by approximating two momentum terms.
Notation. In this section, we introduce the proxy function G TRd IR defined as
((w)
Σ ('(w') xz).
N
iEIN]
Lemma D.1 (Lemma A.2 in Zou et al. (2023)). Assume B7 B2 and let V Bz(1- B,)3 Then, for both Det- Adam and Eno-Adam iterates, m, [k] S a (i for all E [d].
(1-Baa(B---p 9f)
Proof. Following the proof of Zou et al. (2023, Lemma A.2), we can easily show that the given upper bound holds for both Det- -ddam and Inc-Adan, Ada.. We prove the case of Inc-Adam, while it naturally extends Det- By Cauchy-Schwartz inequality, we get
m, [
(1 - V. (w.,))
( 8.)IV [b]]
1/2
1/2
BPT - B1)²
(5 B2) V..
(CS inequality)
55 B2)
S volk.
The last inequality is from
31-133) 31)
3
B₂ 81) 1122 (8))
Σ
1 1 32
B2
==0
where the infinite sum is bounded from of < 82.
D.1 PROOF OF PROPOSITION 2.4
Proposition 2.4. Let [wr) be the iterates of -et-Adam with B1 S 82. Then, under Assump- tions 2.2 and 2.3, if limim+4o can be represented by
W. [(w,) = 0, then the update of k-th coordinate W [k] w, [k]
[k welk) = -7 (sign( (VL(()))))) +
(1)
for some Doc E
O.
Proof. We recall Lemma 6.1 in Zhang et al. (2024a), stating that
m, [k]
(1 - 8 VL S cmnll(we).
V1 3 [L(:)) s cv coVT)
ve k
for all t > t and l E [d]. Based on these results, we can rewrite mi[k] and vv as
m [k] (1 (
= 1- +++l (L(w))()))))
21

---

1 1 from Lemma 1.1 and Published as a conference paper at ICLR 2026
where Cm
O
(v)))
Note that (( w.)
ata
bic2
b
b+ca
9
b
btcz
+
for positive numbers e1, 6., b. Therefore,
if Tim
1
[(we)
0, then get
[W:(w:
m, k
81+1
sign (VL(w,)[k)
[.]k]]
cm (t)g (we)
g++1
cv (t(
sign (V [V,(())()))
32+
V (7(w [A:]]
32
3551 [V[(w,)[b]
bounded
30
+
From 13,88 ++ 0, we get W w.[k] m. [
(sign (VL(w,)(k) + 4.) for some
Vv A
lim;-xc Et O.
D.2 PROOF OF PROPOSITION 2.5
To prove Proposition 2.5, start by characterizing the first and second momentum terms m., Ve inc-Adam, which track the exponential moving averages of the historical mini-batch gradients and square gradients. As mentioned before, key technical challenge of analyzing Adam is its dependency in the full gradient history. The following lemma upproximates momentum terms with respect a function of the first iterate in each epoch w., which is crucial for our epoch-wise analysis.
Lemma D.2. Under Assumptions 2.2 and 2.3, there exists t only depending Bi, 32 and the datasel, such that
1- B1
m [k]
Σ
((0.) VL, (W.)(A))
x 6m (t) max JE[N]
V (w (w.))
1 3N
JE[M]
B2
vi[k]
Σ
a(a.)) (w)"))
I (v)())
max JE
[L
1 1 3N
JE[N]
for all r, satisfying rN + la and lo [d], where
cm (t) A (1 Bi)caN 02 + (CaNDarN - 1) 81+1
(1) 3(1 32 20 DWEN DONN dglte + 3 DoNDN - 1) + 8241
D maxxe[N] [|x;||ı, and 022 eg are constants only depend on B1, 32, and the dataset.
22

---

Published as a conference paper at ICLR 2026
Proof. Consider = rN + s and the gradient at time is sampled from data with index s in r-th epoch. Then we can decompose the error between m [k] and 1-8,
VL as EjE(N]
1- 31 1 B
m
JE[N]
1- 31
31 - 3,) V (WL-))1
Σ [[,
1 B JE[N]
Σ
(T - (w
Σ 7=0
(1- V.)V
7=0
(4): error from movement of weights
(wi)kk) Σ
7=0
(:3): error between we and w.
+
B1 - BI)VL, (w
1- B
Σ
[[,
((85)
1- B
JE[M]
(C): error from infinite sum approximation
Note that
(A) < Σ BT(1-B1)le [w]](w,_,,,))) l' ((w, ([1_,))/X(,_11
(((WT_ xxi_r)
Σ BT B1)
16 1e'(w, X,, xx,..) |x1_.. [3:]]
('w, x
0=0
(1 31)
max JE[N]
VL, (wr) [k]]
Σ Ei -1)
7=0
(1 B1)c2T max
VC, (w.) [k]],
JE[N]
(18))
(1 - (1-B))a B.)e N Call max
JE[M]
for some 02 > 0 and t 1> 11. Here, (+) is from Lemma 1.3 and
w(w:-w...) Xi, .1 1 < e ||w. lle 1x., Il - 1 S D 1.
Also, (**) is from Assumption 2.3, and (***) is from
VC, [k V (w.) [k:
max jE ]E(N]
[w.((w))<]]]
S max VC, (w JE[N]
max JE(N
Ll ['(wixx xx, (((w" x,)
max VC, (w.)))) JE[N]
max JE[N]
A N
max jE[N]
VL (w (w [x]b
where the last inequality is from Lemma 1.3 and
l'(w.Tx x,) (((wo" x;))
< max (w) < a DueN. JE[N]
max JE[N]
23

---

Published as a conference paper at ICLR 2026
Also, observe that
(B) -
(1 - Bi)le((w xix_.) - (w."" Xi.- 11x [b]]
7=0
8'(w, Xic_r)
Σ 7=0
BT B1)
((w@" xi._,) 1 (e'ww Xi. )1(x) [K]]
31) max JE[N]
(C - 1) Σ BT
T=U
(CaNDON 1) max [V,(w,))M]]
JE[N]
where (+) is from Lemma 1.3 and
((w,xi.-. - 1 S el(we-w)) Tx - 1 S ellw. 1.ll S CaNDDiNN -1,
((w] x,,-,)
and (**) from Furthermore,
$7 S I-3. ST-
1
1
80 8[(1 V.),
(0)
Σ
Σ ( 8,)V (w:))[k]
Σ (1 B1) Vc, (w9)(a|
+=
S g JE[M]
max
Therefore, we can conclude that
m [
1 - 31
Σ
3(5
JEIN]
(11 - aa DOON 02 + (GNND
-1)+ - 37+11
max [L,
JE(N
(1)
Similarly,
[1[K] 1 32
Σ
3 (ff VL,
1- BN
JE[N]
Σ 83 VC,V
1- B2
86 (w.))))
1 3 JE(N
B5 V (we- [k]]
Σ 33 (1- V..
(D): crror from movement of weights
+1
Σ
(1 VL. (w.)
Σ
7=0
7=0
(E): error between w. and w..
Σ
(1 B2) VL. (w
1- B2
(د ((ا [L
B2
I BN
JE[N]
(P): error from infinito-som approximation
24

---

Published as a conference paper at ICLR 2026
Observe that
(D) (1 (w) Xi . (w, xs. )
T=U
Σ (1 B2)
(e(mi
(((w) xi
11(ww, xix 11 [k]]
(9)
2 3(1 (2)
Σ B5 (e -1) [L.(w.))k]]]
max JE
(+*)
2 3(1 32)c2m max JE[N]
[w,
(w.*)
S 3(1 32) 2aN Dr
22)
egT max
[w]]
for some c2 > 0 and t > Ej. Here, (+) is from Lemma 1.4 and
(((w (( - C(w,, Xi, 3 -1 S 3(e2l(w.-ww)) x., - 1) S 3(E 2aD Ei 9i- 1),
(**) is from Assumption 2.3, and (***) can be derived similarly. Also, get
(B) < 35 B2)le((w Xi, )²- ('w Xi )21|x; [k]]
0
$ 2aN DUIN 1) max [W,(w")
JE(N
(F)
Σ
35 V2) (w.)
Σ
-1-1-88) 32)1-B)VL [w])
<
Σ (5 B2) (w91411
7=
S
g++1
max (w.' JE[N]
which can also be derived similarly to the previous part. Therefore, we can conclude that
- 32
vi[k|
(W
1- B
JE[N]
(3(1 B2)e 20. DrNN En + 3( 2aN 1) 52+1) JE(N]
max
Notice that (1) and (4) defined in Lemma D.2 converge to 0 oc, implying that each coordinate of two momentum terms can be effectively approximated by weighted sum of mini-batch gradients and gradient squares, which emphasizes the discrepancy with bet-Adam and Inc-Adam. We also mention that the bound depends on maxje[N] [V, ke], which converges to 0 as L(w!) + 0. Such approaches provide tight bounds, which enables the asymptotic analysis of Inc Inc-Adam.
Proposition 2.5. Let )00 be the iterates of Inc-Adam with B1 S 32. Then, under Assump- ++1 w can be represented by
tions 2.2 and 2.3, the epoch-wise update w u
EjEN g( V VC, (w
w 0 r1
w -7rN CCnc(31, 32)
Σ
+ tc)).
(2)
ESE[N] 2f VL (C,(W))
$E[M]
25

---

Published as a conference paper at ICLR 2026
where 8/6h1 0/5-1) mod N 1B 3/ (/(-5) mod N Cine (31, 32)
1-92" I-31 1-3, is a function of
1-3, /
82 and lim Cr 0. 1f 1)- (t + 2 for some E (0, 1). then ||c,11>0 = ar
Proof. Since both [i[k] and 1- EJE[N] 82 I-
are positive and |a² - 62| = la
blla + bl 2 la - b/2 holds for two positive numbers a and b, Lemma D.2 implies that
32
Σ JEIN]
3/ (W)(
(W)
max JE[N]
32
Therefore, we can rewrite m [:] and v k as
mf(k)
1 B1 1
8
(1)
max
V
(w.
3
JE(M]
JE[N]
(a)
(a)
v
Bg
alo.) (. + (C(()
max ME[M
VC, (w])
BN
JEIM]
(ea)
for some error terms
<m (t), (t)
such that I (t) S (m (4( S (t). Note that
a+tl
a b b+ez
for positive numbers E1, 62, b. Thus, we
b+t22
b
b+e2
can conclude that
m
(EL)
(0)
(c2)
VA
+
+ 0,
(9)
v
since
(E1)
1
Fm (1) + 0,
(b)
1- 32 O
32
1-8
(a)
VN,
8
(c2)
(1) + 0.
Now consider the epoch- wise update. From above results, we get
w.l +
[k]
w.[k]]
N
Σ
7
m
[k]
v=lk|
0
N
ESEIN 8(th3) VC,((w?) + C [k]
Σ
Cin 32)
Tex
VEJENN ((..)) (W,
(10)
for some E + 0. Since lim ni = O, the difference between T)-N for different E [N]
converges to O, which proves the claim.
Next, we consider the case 17. (t + 2) for some E (0, 1]. Then it is clear that
(t) = (1 3.)eaN DONN c2 + (DoNDrr - 1) + 8/+1 = O(t-a) cv 3(1 B2)e 2oN DnEN
DnoN EATe 3(c 2aN2NDD - 1) + g6 O
26

---

Published as a conference paper at ICLR 2026
where D [MX X Therefore, from Equation (9), get
m [k:]
EJENNI (L.
v kk Cine (31
01
VEJEIN 32 ( V [k]
which implies (x(A] = (((-a/2) in Equation (10). Note that
N-1
EjE[M BL
+s
Cine(Bi, (2)
N SISENN (Lg
Sp(s)
N-1
Σ
p(s) + T+ ANN p(s) +
TrN+s
VoN
Furthermore,
TNN - M1++1)N
-1-(1+
=1-
N
0(r-1)
rN r2+2
MNN
from Lemma 1.7. Since p(s) is upper bounded by a constant from CS inequality, we get (x Orr-)/2,, which ends the proof.
E MISSING PROOFS IN SECTION 3
In this section, we provide the omitted proof's in Section 3. We first introduce the proof of Corollary 3.2 describing how SR datasets eliminate addppiittta of Inc-Adam. Then, we review previous literature the limit direction of weighted GD and prove Theorem 3.3.
PROOF OF COROLLARY 3.2
Corollary 3.2. Consider Inc-Adam iterates {wr] on SR data. Then, under Assumptions 2.2 and 2.3, the epoch-wise update w w can approximated by weighted normalized GD, iee.
a,(r) /L(w9))| V..(wW)) + +
0 r+1
w.
-
(E(N]
where limm 6, and ei < a:(r) S c2 for some positive constants cı, 02 only depending on 81,82, 82, (xi)ie[N) 1fTk 2) a for some (0, 1), then |lylloo O(r-a/2).
27

---

Published as a conference paper at ICLR 2026
Proof. Given SR data fxe (x let 2,
|x [0]. Notice that EJENN B VL,
IIENN VL, (w
Σ
KE[N] V EJEIN 31
VL, (W
EIENN 1.
((W.. xi)) 2x9
ME|M]
Σ E
(w)(
VIIENN B((D) (ee 1
SE(M] V
= E
(w
EIEINI (e (WW)) xx)
JEIN]
3 VL((W))l
(w,(
Σ
Σ EIE[N| 32 ((ed))
((w(. xi)) 129
VL(w:))12
JE[N]
(s(r)
Therefore, it is enough to show that (r) is bounded. Note that
ar(r) S N [V((w))l2
EiE(N] (e(( xx))
3 1N-11 v EiE(N ((w
V 1/2 VEEENN) (e'(WWP, 2x)
Vd
EIEIN) (6' xx
VdN
VBN N VIIEIN| (x))
v 1/28"-
To find lower bound of az(r), we use Assumption 2.1. Take v E Rd such that ||v||2 = 1 and
v > 0, Vi E [N]. Let 7 - minselN] v x; > 0. Note that
VL()) N
Σ (-e((w). xi)) x;
(e((w,.
N
[E[M]
LE[N]
and by CS inequality,
|V = -ttp://lllllllll/aeeveee- (
Σ
(e((wW xi))
(II)
N
[E|M]
Therefore, we can conclude that
LVC(W%)l2
(+)
EMEN (((WP.x)))
ag(r) 2 N
13 1
EIelN] (e((wW. (x))
VIELNM (u(( xi))
7 7NN
maXIEIN IT
where (+) is from Equation (11). Now we take e max ži on B1, B2, (x:)-
70
and 02
JN only depending
L
E.2 PROOF OF THEOREM 3.3
Related Work. We now turn to the proof of Theorem 3.3, building upon the foundational work of Ji et al. (2020), who characterized the convergence direction of GD via its regularization path. Sub- sequent research has extended this characterization to weighted GD, which optimizes the weighted empirical risk (q(t)(w) = EN (t)(((w xx). Xu et al. (2021) proved that weighted GD con- verges to fg-max-margin direction on the same linear classification task when the weights are fixed during training. This condition was later relaxed by Zhai et al. (2023), who demonstrated that the same convergence guarantee holds provided the weights converge limit, i.e., 3 lime-zo q(t) = q.
28

---

Published as a conference paper at ICLR 2026
Our setting, however, introduces distinct technical challenges. First, the weights are bounded but not guaranteed to converge. The most relevant existing result is Theorem in Zhai et al. (2023), which establishes the same limit direction but requires the stronger combined assumptions of lower-bounder weights, loss convergence, and directional convergence of the iterates. A further complication in analysis is additional error term, 6, in Corollary 3.2, which must be carefully controlled. Our linengraieed analysis overcomes these issues by extending the methodology of Ji et al. (2020), enabling us manage the error term under the sole, weaker assumption of loss convergence.
Definition E.1. Given
(a1,-.. ,an) E RN, we define a-weighted loss as [*(w) - EiE[N] (w). We denote the regularized solution as w ( arg Birgmin]w/l <B (w)).
By introducing a-weighted loss, we can regard weighted GD as vanilla GD with respect to weighted loss. To follow the line of Ji et al. (2020), show that the regularization path converges in direction to f2-max-marre solution, regardless of the choice of the weight vector a if it is bounded by two positive constants, and such convergence uniform; can take sufficiently large B to be close the l solution for any a E [c1, c2]
E.2 (Adaptation of Proposition 10 in Ji et al. (2020)). Let ú arg Iw min4e(N] (v, xx) be the (unique) ba-max-margin solution and c1,02 C2 be two positive constants. Then, for any a (2),
Lemma
wa W B
lim
ú.
Furthermore, given O, there exists M (c1, C2, C, only depending C1 C2 N such that B > M implies B
(B) ûll e for any a E [2
Proof. We first have to show the uniqueness of fz-max-margin solution. This proof was introduced by Ji et al. (2020, Proposition 10), but we provide it for completeness. Suppose that there exist two distinct unit vectors u1 and u2 such that both of them achieve the max-margin 5. Take U3 422 as middle point of and ug. Then we get
ug x, [
u x2) 4,
for all E [N], which implies that min ug x; 9. Since in * u2, we get |us|| 1, implying that us achieves larger margin than 7. This makes contradiction. |ual
Now prove the main claim. Let 9 min [N] (ū,x,) be the margin of ú. Then, it satisfies
mmim min (w (w(B) (B), x < L (w (B)) < L (Bá) < Ne2((BA).
(12)
KE[N]
For = Eesps get [ ( (B), x;) By - loog
Neg e,
which implies
wa
Ne
min VE(N]
log
(13)
B
C1
Since fy-max-margin solution is unique, Equation (13) does not depend depends on C1. C22 e, N.
(B( converges to ú. Note that the lower bound in [ei, N Therefore, the choice of M in Lemma E.2 only
et For = log. Equation (12) implies that (wa x,)) S New (Cx(( Notice that Noa > and minggNN (B), > B4 > hold for sufficiently large B from Lemma 1.2. From Lemma I.5, we get
(
1
min SE[N
2
log ` - -1).
B
B
Following the proof of the previous part, we can easily show that the statement also holds in this case.
Lemma E.3 (Adaptation of Lemma Ji et al. (2020)), Let a, C1. C2 > be given. Then, there
exists p(a) 0 such that ||l]|2 > p(a) = ( + a) w/ 2 & L (w) for any E (cr, N2)
29

---

Published as a conference paper at ICLR 2026
Proof. Let ú be the fg-max-margin solution and & [ (ū,x,) be its margin. From the uniform convergence in Lemma F.2, we can choose p(a)) large enough so that
w w l2)
[w] > p(a)
S añ,
|wl2
for any [21. For si i n, we get
(
(1/wl)) w.l.2,,xx, +
S oollwlll + <[/w]/2ú, x,)
- + (1) w|2(,,,)
This implies that
C² + ((]++a)]lwlla)) [ - (w)),
for any a 02].
Theorem 3.3. Consider Inc-Adam iterates {wo) with 81 S 32 SR data under Assump-
tions 2.1 to 2.3. If (a) [(w,) + as t - oc and (b) = (t 2)- for a E (2/3, 11, then it satisfies
lim t
Wez
Well2
where We, denotes the (unique) f2-max-margin solution of SR data {xi]EENN-
Proof.
From Corollary 3.2, we can rewrite the update as
w r+ w
V((r)VL,(w")) TT.EE
VL
KE(M
MNN
vL (w.)) Tr,EE,,
where 6 S ar(r) S 02 for some positive constants C1, C2 and lim E, = 0.
w First, we show that lim [w Wiz Let be given. Then, that Tto 1 - Since || + co, we choose To such that 2 ro
take -
T so ||wel22 >
max (p(ar) 1}. where p(a)) is given by Lemma E.3. Then for any r 2 ro. get
(VL w. (1 +))) [l 2 L (w:)) - ( + a) 1 2 0,
which implies
(VL (w.), 2 (1 [| ú).
Therefore, we get
(w. r+
w íi)
VLa(r) ( ()) + (-m,eer û)
[VL(W9)) ll2
2 (1 a) w
JrN
[aalr) w + (-moNe,, à)
VL(ww))
1
wh, w
w0ll2 (n:NG, w (1m,Ne,, ú)
(1 + a)
w
(1 +
a
ww
1
1
1
Il -
ww +(-ThNEr, á
w. (1 + ax) w
(1 + woll2
112
1
1
w0112
-2 Dnn.llerllz.
(1 + w2ll2
2
2
where the last inequality is from (y.NEr, á
(I+a)]|w!]aa
wb
S n ||a ja
[[
w
5
2 1/2/
30

---

Published as a conference paper at ICLR 2026
Note that
wwll2
|w.+1ll2
-|[w
[w[]]
Furthermore,
|| ++1 w011 2(1 + <3) w
w.11
1
V [Ca(r) !1
vN 1rr12 VVL w
2
2
I
Cgr
2a
for some > 0 and sufficiently large since 7/,N 0,,=)),1E,11 =
and
upper bounded from
(cXVdmaxiE(M) Is EVEIN) [e'(WW, ((W))))))))
V w w (+) VV(( W
c2d.22 (max}E[N) xz))
72
EiEMN ((ww,,)))))
with y = min (We,, x.) Note that (*) from
2
[VL(W))113 - W/ VL (Wez) N
Σ
(((wf)xx))))
x1) xx)
7
Σ
(((((,,))))
N
KE(N)
iE(N)
Therefore, we get
|wPll 1| ll2
(w w ro:
ú)
Σ
038 2 Σ
1
==
s=Tp
2 (1 ([ww)]l 1/
C33
2a
C48
$a
8=
s=To
=0g
since ||e||| = 0r733/3) and E (2/3, 1]. As result, we can conclude that
w
(1 (I]
[w% (lll)) +
(w Pg
ú)
+
ú) 2
wwll
|w 12
which implies
w
lim inf
û)
> 1
|
Since choose 0 arbitrarily, we get lim,-
w w
Weg-
Second, we claim that lim 0 for all E [N]. Note that
w. l|wella
We..
It suffices to show that lim,
wwll2
w. Iwella
11
w
w
w
w.
w
w
9l
1w
2
|w
|w
2
[w
w
||
Iw
w.l
12
|well2
w:lle
$
0.
|w
||
which ends the proof.
31

---

Published as a conference paper at ICLR 2026
F MISSING PROOFS IN SECTION 4
F.1 PROOF OF PROPOSITION 4.1
2 be the iterates of Inc-Adam with &1 S 32. Then, under Assump- tions 2.2 and 2.3, the epoch-wise update w 1+ w can be expressed as
Proposition 4.1, Let {w:)
(1
1 32
VL(w.)
w r41 w. -NN
+ t
B2
E (L,(2
where lim so [16%) (r) S ((3)) and lim ((2)) = O.
Proof. Note that
ΣJENN 8f (1 (w.)
EJE(NI
EJEINI [19
ESEIN WWW
=E[N]
1 AN
VC(W2)[k)
1 B1
E -= VL,(w9)]6}
Furthermore,
ENEN 31 (()))) (w
EjENN 21 v (w
V EJEM Bl (w
(w.) [
ESENN B (15))
VC, (w:))K
ViIEINI Bl (w]((\)
EJEIN (.
VC C, (W))
v ESELNI (W2)())
Σ 8((2
1 )
Σ JE(N] 3(6
1 11 1-VBNN1
JELN]
(c((2)
where lim c(32) = O. Substituting to Equation (2), we get
1 - 3N
VL(W-)((
w w [k:] -w 7-N
Cine (31,
+ EB (r)
1 B1 Di=1 (w.))al
v ([(
- Cproy (82)
N i=ll VL,
where Cproxy (32)
NN = V 9-
lim limsup, (T)lo & &Ne(32), and lim E(B2) = O.
F.2 PROOF OF PROPOSITION 4.3
To prove Proposition 4.3, we begin with identifying AdamP roxy as normalized steepest descent with respect to an energy norm, where the inducing matrix depends on the current iterate and the dataset. The following lemma shows that the matrix is always non-degenerate; the energy norm is bounded above and below with respect to l f--morm multiplied by two constants only depending on the dataset. This result takes a crucial role to make the convergence guarantee of AdamProxy.
Lemma F.1. Consider AdamP roxy iterates (w.) under Assumptions 2 and 2.2. Then, it satisfies
32

---

Published as a conference paper at ICLR 2026
(a) Prx(w)= = arg min (VZ(w), v), where P(w) = diag (VENEIN) VL,(W)) and P(w) = [
P(w).
I ((w)
(b) There exist positive constants C1 0g depending only on the dataset (xi)iEiN) such that ei |lv122 z
|Vllpw> S 12 for all v, w Rd.
Proof. (a) Note that Prx(w)- = -P(w) VL(w) arg (VL(w), v) + B There-
fore, normalizing by IIVL(w) (w) we get (rx(w)= arg min (VL(w),,) v)
|
(b) enough to show that every element P(w) is bounded for some C1 Cg > 0. For simplicity. we denote [e'(w x;)] = rú. [Njje(N |x, = B₁ > 0 and [NXie(N].je( |x, B₂ O.
Note that
Σ iE(NI
ΣjEld C
2 B1
??
EiEN B2)2
VE[N]
Σje(4 ViNEIMI B²
Bi 1 EiELNI r?
B²
B3 d (Eiein) r:)3 Nd B3
Let v RRd s.t. ||v||2 and vxx > 0, Vi E [N] (since {x, } is linearly separable). Let minle[N] v x; 7 > 0. Then, we get VL(() = EiE(N) TAv x;
Tx. 2 7 Eie(N) Ta which
implies | ((w) | (VL(W) (v. VL(W))? 21² (EIe(N)Ts)
Note that || Ejela 1 get
13 x111122 v((l² s dB2 EiEIN r?. To wrap up,
72 (EIENN) r;)² r:)²
[V[(W]
dB2 v /EEELM 1.
and therefore,
EiE[ (2x;[k]2
dB2 VinE 17
72 (Eie(N) r:)3 S dB3 72
P(w)[k,A]
T2x: [k|2
VL(w) P(w)
NE[N]
As result, we can conclude that
B? dB3N
dB3 72 |vv. Vv,
|vv| S llvllp(w) S
and take C1
F2 dB.
and
C2
dn3 7
Proposition 4.3 (Loss convergence). Under Assumptions 2.1 and 2.2, there exists a positive constant
n 0 depending only on the dataset {x))ie[N) such that if the learning rate schedule satisfies We n and Eee it 0o, then AdamP roxy iterates minimize the loss, i.e., lime-soo L(w.) 0.
33

---

Published as a conference paper at ICLR 2026
Proof. First, start with the descent lemma for AdamProxy, following the standard techniques in the analysis of normalized steepest descent.
Let D suPwerd [aXie[N] ||x; || p-w()-- Notice that D S 02 maxie[[]] |xx/|2 oo by Lemma F.1.
Also, we define
max min v X; ||vllp $ ie[N]
be the || [|p(()) - max-margin. Also notice that wuP Tw since
max min v x, max min v x; II I iE|N]
[| PMW NE[N]
for any w E Rd by Lemma F.I. Then, we get [(W++1) [(we) + The (VV.C.C.)) ( Prx))) nh (rx(w)) T V/L(W, 3(wz+1 -w:+4-1ww))) (rx(
(2) S ((w;) - t/ V W (W( +
D2
sup(l(w,),
2
(+0)
n/ DRD g(w,)
3 [(w.) V
2
(0))) ((wi) 1)c o D
(V((w)ll--(ww)
Ywo
2
< ((w.)
[[ ((w)))
2
for 1/ 1 en
17. Note that (x) is from
Prx(w:) TV2 Prx(w)) = Σ (wW(((W)((W)) xx)²
1 N
EE[N
Σ
(w))
Prx |x:l| D'g((w,,
N
<E[N]
where the last inequality is from Lemma 1.1, and (**), (***) are also from Lemma I.1. Telescoping this inequality, we get
Th VV 1(w)) L((:o) -(WT) S L(Wea),
which implies Ee 1h WVV((w)ll [ (W.()))pp-/( < X0. Since
Σl= = oc, we get
lim inf VV((w;) I(w.) = 0. From Lemma F.1, we get lim inf VL((w;)] 0, also
implying lim inf, ((w,)= = O. Since L(w,) is monotonically decreasing, we get ((w,) + 0. 0
F.3 PROOF OF LEMMA 4.5
Intuition. Before we provide rigorous proof of Lemma 4.5, first demonstrate its intuitive explanation motivated by Soudry et al. (2018). For simplicity, assume = fexp and let w, g(f)w + p(t) where g(t) = ||wll + 0o, p(t) e Rd, and p(t) + 0. Then, the mini-batch gradient can be represented by
VL:(w) = exp(-w x,)x; - exp(
T xx)exx(-p(() x:)x;
As (() - oo, the coefficient exponentially decays to O. implies that only terms with the smallest
w x; will contribute to the update of AdamProxt) Therefore, the limit direction w will be described
by ΣLEIN Cox:
where c, is the contribution of the i-th sample to the update and it vanishes for Tx;
EiENN xxx
i É S where S = arg min w x;.
Building upon this intuition, we first establish the following technical lemma, characterizing limit points of sequence in form of AdamP roxy.
34

---

Published as a conference paper at ICLR 2026
Lemma F.2. Let (a(t))(20 be sequence of real vectors in B and fx, hes IR be the dataset with nonzero entries for an index set S S [M]. Suppose that b, Eies a,())x.
satisfies ||b,ll2 2
VEnes a.(0)x?
C> for all 2 0. Then every limit point of [
b
is positively proportional to
EiEny for
VE ox2]
some с A N-1 satisfying G = =0forr i $ S.
Proof. Define a function F A TRd as
F(d) = Eies dixi /Eies
Since {xi)ies has nonzero entries, F is continuous. Let A = {d A [[F(d)]2 2 C). Since F is continuous, A is closed subset of A|81-1, Furthermore, since ||6. 2 C for all 2 0. {a(t))(2 S A.
Now let be a limit point of 8 Define a function G ACCAA ACCA
S1 A
Rd as
T
1
Eies dxx
G(d)= = Encs dex;
Viessffe xfx
V2ree dix
Notice that G is continuous on and lim, G(a(t)). Since As bounded and closed, Bolzano- Weierstrass Theorem tells us that there exists a subsequence a(tn) such that 3 lim, a(tx) E A. Therefore, we get
lim G(a(t,)) G( lim att,)) G(c). n-000
2-000
Hence, the limit point å is proportional for & S.
Exes eix:
Then we regard e N-1 by taking &
VEis ox
Lemma 4.5. Under Assumptions 2.1, 2.2 and 4.4, there exists (co, .CN-1) E A-1 such that the limit direction w of AdamP roxy satisfies
W α EiENN GX,
(5)
EvEMN exx
and c; 0f & S, where S arg [in w x; is the index set of support vectors of w.
Proof. We start with the case of é= tep. First step is to characterize 8, the limit direction of 6, To begin with, introduce some notations.
From Assumption 4.4, let = g(t) + p(t) where g(t) p(t) 0.
-|wellz + oo, p(t) E Rd,, and
Let 7 min; (x,, w). 7: (x,, w).7 mingss w). Then it satisfies S { E [N]
(x,, w) = 7). Here, note that 7 > 7 0.
Let ax(1) E N be az (4) exp(- (()) Tx:)-
Let By = max; |xx Bj mingg(N]je(d] xx|]]| > and By max,e[M]je(d] [x]]]
Since [Ip(t)]/g((t) + 0 and 7, 7 > 0, there exist te teg such that
p(t) x; < 11((t) I (rg(t)) VI > to Vi E [N], p(t) x; -1)(())] -c) Vt tor, Vi [N],
35

---

Published as a conference paper at ICLR 2026
for all 61,62 > 0. Then, decompose dominant and residual terms in the update rule.
Eies exp(==g(())) exp(-p(t)) x;)x;
E Eieso exp( (pp(-5:((1)) exp(-p(t)) xx))))))x)x)x;;
6:
V VIEIN [xpi)exp(--a1 (xp(-227:0()) exp(-2((() x,)x3 EIEIN exp( exp(-2xppp))) 27x(())) exp(-2p(( -2p(1) xx)xx d(() + r(l).
To investigate the limit direction of ö,, we first show that (t) dominates r(f),,i.. lim,
|r(6]
[d(0]l2
Let M, diag (V EIEIN) exp( exp(-27ho(t)) exp((2p(t) expp(-22)) xx? Notice that
[M.illald(())) 2 (M,,d(()) = Σ
exp( exp(-ng(tt) exp((p(t) x.).
iES
Since the diagonals of M, are upper bounded by B2 EiE(M) exp( exp( T xx)) we
get
Eies exp(- ex) x:)
|d(t)l22
B2 Bav Exermeep((- exp(( exx
expl
Also, notice that
B2 Eies exp((xe()))xxxppppppp exp(()))))))ppp)))))))))) x:)
[r(()]l2 S
Bi B EiEiN exp(-22epppp2000000 exp(-27:9(11) exp) 2p((1)
From the following inequalities
Σ
exp())(()xpp(()) 2xx((==(()))xx(((ar)()
SS
=c))(((++)))g(t)),
Σ RESo
xx) exp( (o))= exp((p(t)) (xx)) s N eex((-59(t)) exp(er7g(()
= -NNcx(((1-c2)dofl (2))g(t))
we conclude that
1r(0122
B3
Σ Si exp(-rg(tt) (p((( x.)
d())]]
BB
EiEs exp(-7g(t)) exp( -p(t) x;)
NB
exp(
(7 - -))(()) + 0.
BB,
2
Next, we claim that every limit point of d(t) is positively proportional |d(
EieN ax. VEreNN ox?
for some
=Coos" ,CN-1) N satisfying c; 0 for i & S. Notice that
d(t)[k] = Expsex((-20 (t)) exp(-p(t)
V ExeMeexp(-2211
exp( exp((27:g(t)) exp(-2p(t)) xx, [x]
Eies exp( (t)())gg(t)) exp(-p(t) x, xx)x, [k]
V Eieseeep exp(-22g(t)) exp(- xx)xx [k] Eieso exp(-27:g(t)) exp (2p(() T Txx xx)x([M)
Eies exp((cx(((t) exp(- Tx;) xx;x;{{
V EiEs E (00p-222())) xx))>
Σ Eicoo (xp(-22:g((0) exp( (p())) xx)x
Eses exp(-2y9((e) exp(-2p(() [.
36

---

Published as a conference paper at ICLR 2026
Let b, = En expl 70(44) oxp( p(t) xx,xx
Enes expl (xp(-p(t) x,)x,
Since
Viess exp( 2ng(()) expl -2p(() x()x²
Viies expl -2p(t) x,|x
E Ciss (pp(-) (xp(-2p(t) T xx)xx [k]
0,
Eies exp( (2ggg exp( 2p((t) x; x [k]
every limit point of d(t) Ta(r)) is represented by a limit point of Tb.
b
Notice that b, is an update
of AdamP roxy under the dataset (xi which implies ||b/|2 2 lower bounded by positive constant from Lemma F.1. Therefore, Lemma F.2 proves the claim.
Hence, we can characterize as
d(t) + r(t) [[d(t) r(1]22
8 lim
lim +0c
|
d(t)
r(t) [d(t) + r(t)ll
lim
+ lim
||d(t) + r(t)ll2 40
lim d(()
EiE|N] cX,;
||d(t)]l2 x
EiEEM xx
for A N-1 satisfying C, 0 for i & S.
Second step is to connect the limiting behavior of ò, to the limit direction w using Stolz-Cesaro theorem. From the first step, we can represent
8, = = + ot),
where h(()) = ||,, l| and
K(E) (1) - O. Notice that w. wo Σ (x)(()(( ([ड)
-
( Since
+ h(s) (t) is bounded, we get Ex= == Nch(s) + 00. Then we take
a. Wt Wo Σ ( + h(s)
1
a()))
b, Σ nsh(s).
Then, {br is strictly monotone and diverging. Also, lim Cesaro theorem, we get
a1-1-a: be-1-b.
= 8. Then, by Stolz-
a 8.
lim 80000
b
This implies w, b.6 + (t)) where 7(0) g()), we get
O.
Also notice that W. = g(t) p(t). Dividing by
be
g(t) + p(t)
be g(t)
8
T(t)
w lim
lim 100
1-000
g(t)
b,
Since & norm is continuous, we get
b
T(t) b,
b
= ww/22 lim
8 8+
lim
g(()
g())
which implies w = 8.
Then we move on to the case of l= 4og. This kind extension is possible since the logistic loss has similar tail behavior of the exponential loss, following the line of Soudry al. (2018). We adopt the same notation with previous part, and we decompose dominant and residual terms as follows:
8, Eies le (1g(1) + p()) x, x,){x;
Eieso (T(g(())) + p()) x;)
V EiEIN (5(gg(( + p(t) 1xx
V EIEINN (e(Tig(t) + p(t) xx, 1x2
A d(t) + +r()..
37

---

Published as a conference paper at ICLR 2026
16 (=) limg-+00 I+e- 1. Therefore, the limit behavior of d(t) and r(() is Notice that limg-+o0
[=](=]] [=]
identical to the previous Lesp case. This implies the same proof also holds for the logistic loss. which ends the proof.
F.4 PROOF OF THEOREM 4.8
Theorem 4.8. Under Assumptions 2.1 and 4.7, (C) admits unique primal and dual solutions, so that p(c) and d(c) can be regarded as vector-valued functions. Moreover, under Assumptions 2.1, 2.2, 4.4 and 4.7, the following hold:
(a)
p A N-1 + R is continuous.
(b) A N-1 + RAY
MO) is continuous. Consequently, the map T(c)
d(c) [[[[e]]l
is continuous.
20
(c) The map T ::ANC N-N - N-1 admits at least one fixed point.
(d) There exists c¹ E {c E AN-1; T(c) =c} such that the convergence direction w of AdamP roxy is proportional to p(c*).
Proof. We first show that (Adam(c) has a unique solution and (cc) can be identified as a vector-valued function. Since M(c) positive definite for every A 2 || [ strictly convex. Since
the feasible is convex, there exists unique optimal solution of PAdam (c) and we can redefine pc) as vector-valued function.
Since the inequality constraints are linear, PAdam (c) satisfies Slater's condition, which implies that there exists dual solution. From Assumption 4.7, such dual solution is unique.
(a) Let /(w, c) = ||ll// be the objective function (c) and F {w E Rdd x; Vi [N]} be the feasible set. It is clear that such continuous on w and c. Let A N-1 and assume is not continuous on č. Then there exists {cx} C A N-1 such that We denote W = p(c²) and
lim C è but [p((;) p(c)ll2 2 c for some
w w CCC.
First, construct {us} c F such that limg- UR w. Then we get a natural relationship between We and
1 2 w/ Mccc) S 1 uu (
Second, consider the case when { is bounded. Then we can take a subsequence Wk. Wg- Since {w F and F is closed, we get F. Also, since f is continuous, ʃ (Wkn, CK - f(Wo, c). Therefore.
f(WK,, C...) S f(w, Ck..)
S(wo, c) S f(w,c),
22000
which implies wo = w. This makes a contradiction to ||p(c=) - p())l22 = l|wk-wlla 2
Lastly, consider the case when {Wx} is not bounded. By taking subsequence, we can assume that ||wallz > oo without loss of generality. Define v Tw Since Vy is bounded, we can take convergent subsequence and consider limp = v without loss of generality. Then,
wi
wT M[cM)c S Ju
T M(c))u = 2
1 [ S
ul
M(c²)
Wk
|
||
Since f is continuous and {u,} is bounded, we get
1 A JTMM(eJ f(v.a) lim s(VE, ck)
cx)=
lim k 2
Vv MMcc)
k-
lim lim k-000
1
U
M(Cx)
u
2
[w
|[
Note that M(c) is positive definite and contradiction.
1v M(c) = implies
0. which makes
38

---

Published as a conference paper at ICLR 2026
(b) Let Co E A NN be given and take w" p(cu). From KKT conditions of (Cu) the dual solution d(co) is given by
M(C,)w"
dc(co)x;
(ES(W*)
and such dc(c)) 2 is uniquely determined since (XiliEs(w)) is set of linearly independent vectors by Assumption 4.7.
Cp- Notice that minlgg((wr) w* 1. Since p is continuous at Co, there exists > such that p(c) x; - > for i & S(w*) and
Now claim that (cc) is continuous
E A N n Bx(co). Therefore, S(p(c) S S(w") on c E A N-1 ( B,(c)))
Let X be matrix whose columns are the support vectors of w On A W-1 (c))). KKT conditions tells us that
M(c)p(c)=
Σ di(c)x;
(+)
Σ
d,(c)x== = Xd(c)
(ES(P(e)
iES(W*)
d(c) =x (mm
M(C))((), I
where (+) is from S(p(c) C S(w)) and (**) is from the linear independence of columns of X. Notice that M(c) and w wc(c) are continuous on Co. which implies that d(c) is continuous on Co.
Since at least one of the dual solutions is strictly positive, is a continuous map from A N-1 to B. [0). This implies that continuous, since + EveNn) continuous on R. {0).
(c) Since A N-1 is a nonempty convex compact subset of R there exists fixed point of T by Brouwer fixed-point theorem.
(d) From Lemma 4.5, there exists A N-1 such that w
c with of 0 for i $ S"
Ei
Tx; Then we take w Eins kets, where S = arg min]e[v] w x;
for some k > 0. We claim
Vins ox
that such becomes fixed point and x p(c").
Consider the optimization problem PAdam (c*) and its unique primal solution w' = p(c)). Notice that minkelN) w X; y > since AdamProxy minimizes the loss. Therefore, w and di(c*) kgt satisfy the following KKT conditions
M(c")w" Σ d,x;, de 2
SES
w' x; - 1 0, Vi [N].
where 5* = {i E [N] w" x, 1= 0} is the index set of support vectors of w*. This implies that T(c*) c* and w = 7w x w- = =(c)), which proves the claim.
F.5 DETAILED CALCULATIONS OF EXAMPLE 4.11
Consider N = d and {x Jeea C Rd where x; = æ,e; + ④ Efjz ej for some 0 < and < 20 <
Id-1- 1- maxx-mannn problem is given by
min ||wloo subject to w x, 2 1, [N].
39

---

Published as a conference paper at ICLR 2026
(For the convenience of calculation, use the objective ||w| rather than 1 //ll..)) Its KKT conditions are given by
a ///llo0 3 Σ \
iE[M]
A x; - 1) =0
\ 2 0, w x; 1 2 0, Vi E [N]]
Note that w - (no 1)ø the KKT conditions since
R and x
(oo (d- 138 1000-115000... 0,...,,.,.......... E Rd satisfy
no t 138
(d=-
0||
Ad-1 d-1
1
Xo
E Xix;,
Iu + (d - 1)8
=E|NV
x; - 1) = Ai{ 30 + (d 1)6
Σ KE[M]
\5(w*1
1) = 0,
+ (d - 1)0
x 220, 1 2 0, Vi E [N].
Now we show that c* = (1, 0, ... .0) A is a fixed point of T in Theorem 4.8 and w - =p(c).
Note that for 1o+(d-175 1)8 > satisfies
M(c")w" = diag(ro, 6, ... ,0w = -xo =k Σ x;
iE(N]
Σ G((w
= 0,
iE[M
C 2 0, w
x 11 0, Vi E [N].
which implies T(c*) = c* and w = p(c*).
G MISSING PROOFS IN SECTION
Algorithm 4 Inc-Signum
Learning rate schedule {m momentum parameter [0, 1), batch size b Input: Initial weight Wo. dataset {x, ]EE[N] Initialize momentum m
Hyperparams:
2: for t = 0, 0, 1.2.....7 T 1 do
3: B, + [(1 b + i) (mod N)
4: g: + VLg, (w,) = 1 EieB, E' ('(wTxx)) x))x
5: m, + -mm-1 + (1-33 (1-3)g.
6: Wi-1 1-+-,, W. 7/ (m;)
7:
end for
8: return WT
Related Work. Our proof of Theorem 5.1 builds on standard techniques from the analysis of the implicit bias normalized steepest descent on linearly separable data (Gunasekar et al., 2018a; Zhang et al., 2024a; Fan et al., 2025). The most closely related result is due to Fan et al. (2025), who showed that full-batch Signum converges in direction to the maximum foe -margin solution. Theorem 5.1 extends this result to the mini-batch setting, establishing that the mini-batch variant of PiccSSignur (Algorithm 4) also converges in direction to the maximum foc -margin solution, provided the momentum parameter is chosen sufficiently close to 1.
40

---

Published as a conference paper at ICLR 2026
Technical Contribution. The key technical contribution enabling the mini- analysis is Lemma G.2. Importantly, requiring momentum parameter close to 1 is not merely a techni- cal convenience but intrinsic to the mini-batch setting (b < N), as formalized in Lemma G.2 and supported empirically in Figure 10 of Appendix B.
Implicit Bias of SignSGD. We note that as an extreme case, Inc-St Inc-Signune Signum with 0 and batch size of 1 (i.e., SignSGD) has simple implicit bias: its iterates converge in direction to EieNy sig.(x;). which corresponds to neither the E-- nor the -a---maaaappaee solution.
Notation. introduce additional notation to analyze Inc Sigmum (Algorithm 4) with arbitrary mini-batch size b. Let B, S [N] denote the set of indices in the mini-batch sampled at iteration i. The corresponding mini-batch loss CB. (w) is defined as
L8 (w)
A
Σ .EB
E(ww ((wT);). x;).
B
We define the maximum normalized fo
-mariin as
Yoo A max min W x; > 0,
|| le [N[
and again introduce the proxy G : Rd - IR defined as
((w)
1 N
E SE|NI
l(w x:).
As before, we consider & to be either the logistic loss (z)
= log(1 + exp(=2)) or the exponential
loss Eexep(2 (2) exp(-2). Finally, let D be upper bound on the f--nrrm of the data, i.e., ||x||| D for all E [N].
gnum Lemma G.1 (Descent inequality). Inc-Si gnn-mignum iterates
[we] satisfy
[(W241) < L(w:) Th (VLL).)) A.) + CCHRf((w:), A sign(m:);
where Ch = DD
Proof. By Taylor's theorem,
[(W1+1) L(W, m. ( " -C
for some (0,1). Note that for any w Rd.
A ((W)
Σ 8" (xTx( xx)
( (w II & D2g(w), Σ
N
N
KE[N]
SE[N]
where we used g(w): 2 EIE[N] ('(w" (wTx,) from Lemma L.1. Then,
[(w(1)) & C(w)) nu (VC(W.).A)) +
2 1. A
s L(w:) T. (VC(w,), + D'g(ww - -(nA)
< L(w.) 7/. (VC(w.), A.) + 2 nr De g(w).
where we used (ww') < Dlw' (ww) for all w, w' from Lemma L.I Finally, choosing CH = DoD obtain the desired inequality.
Lemma G.2 (EMA misalignment). We denote e; == m, VL(w.). Suppose that 3 E ( NN Then, there exists to E N such that for all t Z to.
1).
N
| = || - VL(w:)ll < [(1 B)D 1) Cin.. + C2.3 ((w.)
where C1, C2 > 0 are constants determined by 3, N, b, and D.
41

---

Published as a conference paper at ICLR 2026
Proof. The momentum m, can be written as:
m, = (1 (1-3) 3) T=0
B - (1 (1-3) Σ
Br V
7=0
and the full-batch gradient V ((w,) can be written as:
V((w)) = 8 1 + (1- 3) Σ Bd (w.),
Consequently, the misalignment e2 m V[((:) can be decomposed as:
B) Σ gr (VLs (We->) VLB (we))
0=0
(1 8) Σ 8r 3'VVL., (wo) -V((w.)
(((z)),
and thus
| 11 =
(1 - 3) Σ
8r LVB....We-- V CB.... (wa))
T=0
$ (A)
(1 - B) Σ 8r (VLL. (wi) (w))))()))))
+
T=0
(B)
3 v L(w:)l
(C)
We upper bound each term separately.
First, the term (A) represents the misalignment by the weight movement, which can be bounded as:
(A) =
(1 3)
BT BVVLB.. - v VL.... (w:))
7=
$ $ - 3) Σ
BT / LB. (w V (w,)ll
7=0
=(1 3) Σ9 fr
1
Σ
(C(W, ('w, )x))
b
7=
VEB
D
S 8) Σ
8
If' (w[,xx)) xi) ((wTxx))
iEB
(1 B)D
E' C'(WT_xx xi) ((w, x;)
3
1e(wTxxi) xx)
b 7=0
iEB
(1 3)DN
[((TT,x))
g(w,)
B
b
C((,, xx)
T=0
iEB
where we used NG(W) = -IEEN e'(w xx)) = EiE[Ny 1e'(w Tx,)l 2 max (((TTxx)) in the last inequality. For all i E [N].
El E'(wT,x 1 < <w xil - 1 sell
11x 15 DE -1.
El (w, xz)
42

---

Published as a conference paper at ICLR 2026
By Assumption 2.3, there exists N and constant e1 0 determined by 3 and D such that Er=o B (c De nh-!" - 1) I c for all 2 to. Then, for all t 2 to. we have
(A) 2 (1 (1-B)DD B)DN
((we)
Σ
BTe DebeeDi: Di
b
T=0
(1 B)DNG(W.)
Σ
Be Re-r Me-r! -1
==
$ (1 B)DNeri((w.).
Second, the term (B) represents the misalignment by mini-batch updates. Denote the number of mini-batches in a single epoch as m := H. Since B, = ((tbb+)) (mod N) note that B; - B; and only if j (mod m). Now, the term (B) can be upper bounded as
(B)
(1 B)
Σ
8T (5(V (V ((D)-V(((W)) V.(ww))
T=0
(1 B)
gr
VLB.,, (w:)
V Lis, (w:)
r=0
j=1
(1 3)
Σ
gr
1
gr
V LB, (w:)
j=1
(t-(()))))
(mod m)
T=0
1
(1 B)m max jE(m]
8
gr
max jE /E/mJ
V LB, (wr)l1
TSh (t-T)=J (mod m)
+=0
< (1 D)
max jE/ml
Σ
g
gr
(t- 7)
(mod m)
T=0
where the last inequality holds since
1 max
DN
max JE[m]
LBB (w)ll
Σ KBB
('(w") xx)
(((w xx)l D g(w)) Dmg(w).
b
e m
for all R
43

---

Published as a conference paper at ICLR 2026
1 remains to upper bound maxje[m] Erst (t-r)=) (mod m) - Ex= Sr [m]. Note that
Fix arbitrary e
(1-8)
8"
m
TSt (t-r)=j (mod m)
0=0
$ (1 - 3)
gmik (1 - 3)
k=0
T=0
(1 - 8)
gmn
(1 - B)
m
3mk
Σ
8r
(1 3) 1
8
k=0
T=0
7=m
1)+1
1
(1--3)8 gm
gmk
(1- (3
- (1 -mm
m
gmn (m 1)(1 - 3)
(1 B)B m +
2
k=0
1
(m 1)(1 - -3
<1 - 3) m
8m
2
(40)
2
(m 1)(1 B)²
S (1 8) m
m(1 3)
2
- 3) m
m
(1
3),
where the inequalities (+) and (**) hold since (1 e)m < m(m-1)
2 < 1
2 for all
0 -- and choose 3.
44

---

Published as a conference paper at ICLR 2026
Similarly, we have
(1 3)
1
8
Σ
Br
T=0
TSt: (t-r)= =)= (mod m)
++1
|-1
m
s
3)
8r (1 3)
gm((+1)-1
++1
m
1-1
m-1
i
[+1
= -1- B)
gmk
Σ T=0
3r
+ (1
3)
Σ
Br - (1 3)
gm(k+1)
m1
m
k=0
[+]
m=m|
1 (1
$ (1 3)
Σ
gmk
3mh
(1- - -333m-1
m
3
(1 - 1m-
gmk
(1 3mm) - (1 - -3)
m
1 8 21-m+2(1 gm-
gmkk (m - 1)(1 - 9)
m
2
g1-m++
11
1
(m 1)
m
8
(1 3) m
m
(1 3).
m
Combining the bounds, we get
(B) < (1 - Dm (aamm((--mmm0) mm mm 11)(ww))..
Finally,
(C) || V((,)ll S g ((+)D)(W.).
Therefore, we conclude
||ell S [(1 mm 1) + Cin. + C2.4 G(w()
where C1 C2 > 0 are constants determined by B, m, and D.
Corollary G.3. Suppose that 3 E (N 1). Then, there exists to E N such that for all t 2 to Inc-Signun iterates {we} satisfy
[(We+1) S L(we) - 11 2(1 - 1 - (2C; + Ca) - 2C where CH. C₁, C2 > 0 are constants in Lemmas G.1 and G.2.
Proof. By Lemma I.1, we get
(VL(w)), A.) (m, A)) (e,, Ae)
2 |mm]]] - ||e Arll oo
(I/V[(w:)]l - |||e/| ||e:|l
[VL((w))l (0ll--22lc.. -2
(w. -2/ee_l11-
Now using Lemma G.1 and Lemma G.2, conclude
[(we+1) S [(w.) n. (VV(V. A.) + (Hn)g((w))
$ ((w)) (a(77c0(w)) - 2||e:l1) + Cnn)g(w.)
< L(w)) Tee (T 2(1 3) N ( 1) - (2C1 Ch) ((()))
which ends the proof.
45

---

Published as a conference paper at ICLR 2026
Proposition G.4 (Loss convergence). Suppose that B E (1 b = N, where Co == D' (5 1). Then, f(w,) - 0 as t 00.
1) if < N and 3 (0, if 20.
40%
Proof. Note that 3 E ( (N= 1) since Yoo- = maxx////>>> Min w x, S D. By Corollary G.3, there exists to E such that for all t 2 to.
7T 2C B) (2C1 Ch) ) 2C23") S [(w.) ((we+1).
Since Th, 8 + 0 & oc, there exists t 2 to such that for all t 2 t1
(2C + CH)n. + +0223* < 7o0 4
Then,
7/0
((())) -
71
$ (ww))L((w-))) < co.
4
1=1
t=
t=
Thus, Ei=to (we) oo and since Eisto Ths co, this implies G(we) and therefore [(w.) 0 oc.
Proposition G.5 (Unnormalized margin lower bound). Suppose that 3 e (1
Yee
1) ifb < N and
4Co
E (0, i = N, where Co = D ( 1). Then, there exists t, E N such that for all t 2 t,
g(w,) (2C1 CH) [(w,)
2C270
min w x; S (7/x -2Co(1 - 3)) Σ NE(N]
Σ n
The T=t,
T=T.
where Co = D 1) and Cu, C1 C2 > 0 are constants in Lemmas G.I and G.2.
Proof. By Proposition G.4, there exists time step to N such that C(w,) I log2 for all t 2 t.. Then, l(w, x:) S ((we) S log 2 < 1, and thus minkEN] w. x; 2 0 for all i lg. Then, for all I 12 bn
N
exp(- min W x;) max exp( W. x:) S max log(1 + exp(-w Teppp-Txxo xx)) N(w)) RE(N]
[E[N]
log 2 xE[N]
log 2
for logistic loss, and exp(- min W, x;) S (W))) S
NC(w)) for exponential loss. log2
Using Corollary G.3 and G(w) - [(w) from Lemma I.1, we get
(%o0 - 2Co(1 - 3))m g(w,-1) + (2C1 + C + 202, 30228'-111--) (1- (7.00
((w,) < L(W(-1)
[((,-1)
9(W,-1)
W <((w-1) exp
-(7co -2o - 8)) L(W(-1)
+ +2C + Cn 2C200=In=- 11-2
1-1
((w,)
1-1
1-1 Σ 8 T=te
< L((r,) exp
- - -C0((1 (3)) Σ
1)
+ (20 + CH)
Σ
nh + 2Cs
L(
r=te
T=M
log2 N
t-1
g(w,)
[-1
2C2100 1- 3
cxp
(77o0 - 2Co (3)) Σ
17 [(w,)
(2C1 + CCH
Σ
7/ +
T=h.
T=t.
Thus, we get
exp(- min w. x:) 5 NL(w)
iE[N]
log
G(w,) [(w,)
2C'700
< cxp
(1%c 2Cc611
Σ 7T r=t,
(2C1 + CH)
Σ n
3
T=t.
which gives the desired inequality.
46

---

Published as a conference paper at ICLR 2026
Theorem 5.1. Let å >> O. Then there exists such that the iterates {woli of Inc-Signus
(Algorithm 4) with batch size and momentum B E (1 (1-88 e, e, under Assumptions 2.1 and 2.3, satisfy
[ x; We
lim inf 1-000
2 700 - 6,
W, 1000
where
A
min w T X D 2 max ||x:|
max
|w $1 S
ie Ne
and
min {o, Nx. ifb 6=1 ib= N.
2D.
N
h (b - 1)
Proof. Let Co == D - 1) so that e == min
Tse
if < and == iff N. Note
20h 3Co
that Co = 0 if = N. Suppose that 3 E (1 - c,
Let to be time step that satisfy Corollary G.3. By Proposition G.4, there exists t" 2 to such that and L(w,) log for all t 2 t². Then, for each t", get
(2C1 CH )m + 2C2
g(w.) 2 Ne(wo) [(w,)
2 2. By Corollary G.3, for all t 2 t*,
2
g(w,-1) + (2C1 + 1 + 2 mm-1 1 - (% 2Co(1 -))mm-1 [(w(-1)
[(w,) - 2(w(-1)
-1211
S 1(
Yoo
8 70071--1
$ ((Wi-1) exp
8 70o7
L-1
< [(W.-)exp exp
700
log2 N
Toc
1-1
exp
TJr
8
Consequently, by Lemma I.1, we have
((w,) [(we)
N(())
(-120
1-1
Σ 1/ 7=tt
exp
2
for all /. 1 1* Finally, using Proposition G.5, we get
Yoc- 2Co(1 - B3) minse[M] w,. x;
/eeloo (700 2Co(1 (3) (lwoll Et'-1 of + Er DEe" 280 ): + (2C1 Ch) E 7i +
2Ca
1-9
|ool Et- 1/7
Etird 71 + Dr nhe Jho D1-1 Tr E 1h
Σ 17r
Therefore, we conclude
minic[N] w. x;
lim inf
2 700 3 27--8
Wz 100
47

---

Published as a conference paper at ICLR 2026
H MISSING PROOFS IN APPENDIX A
Lemma A.1. Suppose that (a) f(wr) - and (b) w, = ||,,
12w + p(r) for some w with
- p(r). Then, under Assumptions 2.1 and 2.2, there exists (co. , ,CN-1) A NN11 such
that the limit direction w of Inc-Ace Adam. with 31 = 0 satisfies
W x
Σ
CxXi
(8)
B (( ox EjE(N
2x3
N
and c 0.r & S. where S arg [[N] w X4 is the index set of support vectors of w.
Proof. We start with the case of = loxp. First step is to characterize d the limit of 8,.. Notice that (b)
a strictly stronger assumption than Assumption 4.4 and simplifies the analysis, while maintaining the intuition that the terms of support vectors dominate the update direction. Let lim, p(r) = p.
We recall previous notations as = min; (x, w), 9i (x,, w), 7 = minggs (X,, w). Then
satisfies S {i E [N] (x:, w) = 7 and γ 7 We can decompose dominant and residual terms in the update rule as follows.
exp(- exp exp(-p(r) x;)x;
6, =
Σ JE(N] 3(433) expl - 27, exp( (2p())
IES
exp(-5:g(r) exp( p(r) xx)xx;
+ Er
ΣJE[N] ((u))) 1xp(-227,00 exp(
exp(-2p(r) (2p(r))xx Txy)x3
KESD
p(r) xx,)x;
Σ EJE(N) (6433) exp(-2((7) (r)) exp( (ecppxp(pppp))) -2p(r) (xp(r)Txp Tx2))x))))))) xx3
iES
exp(- - ()g(r)) (1)()))))(())))(p(p((( x;)x;;
Cr
EjE(N] (b,)) exp( - ())(()) exp(22p(r) (((1-2((()xx))))) ((((r))xx)))) x2)
YESt
d(r) + rr)) + cr.
Since 7, > y and g(r) - xc, r(r) converges to O. Therefore, we get
exp( (p()) x;)x;;
8 A lim 6, = lim d(r) = lim Σ
+00 r-000
r0000
EjEs 3 a(1,)) exp(- (2ppr)Txx)))
iES
exp(-p) x; )x;
Σ Cjes B (1.d)
expl 12p xx,xx3
iES
Gi
E
8 Ex
for some с E A N-1 satisfying 4 = 0 for i $ S. Using the same technique based on Stolz-Cesan theorem, we can also deduce that w = 8. Since we can extend this result to é = llog following the proof of Lemma 4.5, the statement is proved.
I TECHNICAL LEMMAS
I.1 PROXY FUNCTION
Lemma L1 (Proxy function). The proxy function 9 satisfy the following properties: for any given weights w, w' E d and any norm -
(a) (W)) [VL(()]l- < DG((w, where D max ||x:|| and 71-11 max minie[N] w x; is the ||. ||.||oormallied max margin,
48

---

Published as a conference paper at ICLR 2026
(b) 1 Ne(w) I [(w) 1
((w)
(c) g(w) 2 N EiEIAN ("(w") T x:).
(d) g(w') S S ((w)) where D = maXie[N] ||x ll.
Proof. This lemma (or a similar variant) is proved in Zhang et al. (2024a) and Fan et al. (2025).
Below, we provide a proof for completeness.
(a) First, by duality we get
[V = max (g. VL(())
max lgll
(((w') x:)g
X;
gglli
N
2 ((w)
max min g X; lg iE[N]
(ww))
Second, we can obtain the lower bound as
[|V[())l.
Σ ME[N]
('(w'
x;)x;ll- S
([wTxx)llxlll S DG(w). SEIM
Σ
N
N
(b) For exponential loss, ((w) = 1. For logistic loss, the lower bound ((w) from Zhang al. (2024a, Lemma C.7). The upper bound follows from the elementary inequality Moo (2)
Z(w) 2 NLww) follows
1+exp(-2) < log(1 + exp( = flg(()) for all 2 E R.
expl exp(=2)
(c) For exponential loss, the equality holds. For logistic loss, the elementary incquality -Mlog (2) = exp(=2)
1+exp(-2) ([xxp(-=2)) (log(2) for all E TR, which results in
2
exp(-2)
((w) = N
Σ e'(w' x.)
N
1 N
Σ
(w.x).
(w
tE[N]
KE[N]
(d) First, for exponential loss, -leep (z') = expl( - -exp (z) S expp(1= - -2)(ee (=), and
for logistic loss, -Moog(23 exp(2 xxp(=')+1 (log (2) 5 exp(1=' =2))(ng( (z) hold for any z, a E R. By duality, we get
exp(z)+1
G(w') N
Σ L' ('(w") (wxx,)
x,)
1
Σ
(' x; + (w' ww'-w) xx,)
N
SE[N]
KE(N]
1
Σ
f'(w') xx) exp([(w'- x)
N
VE(N)
Σ
('(w") xx) exp([lw' w||lxx]1l.)
N
VE(N]
1
Σ
El ('(w" (x:) Cxp(DIlw' - -wl)
N
VEIN)
Bllw' wolll'-wllg(v. g(w).
I.2 PROPERTIES OF LOSS FUNCTIONS
Lemma L.2 (Lemma C.4 in Zhang et al. (2024a)). For & E (lep, either G(w) < [(w) < n implies w T x; > for all i [N].
In or
1
log2
49

---

Published as a conference paper at ICLR 2026
Lemma 1.3 (Lemma C.5 in Zhang et al. (2024a)). For & E {leap blog) and any 21, 22 R, we have
(=1)
S e -.
(=2)
Lemma 1.4 (Lemma C.6 in Zhang et (2024a)). For l (lesp kloa) and any 21: 22, 23, 24 E E. have
(z1) (32))
1) + (elx-t20-22=21) - -1) 1 (ela-=al (elzo-zal -1)
Lemma I.5. For 1 and 21, O, if (z1) (2)o)(22). then 2 22 -log(22 1).
Proof. Note that
log(1 +6) & a log(11 te t0) = e S (1+e
and define f(x) = (1+z)--1 Since f is an increasing function on the interval (0, 1). we get suPze(0.1) f(x) = f(1) = 2 1. This implies (1 z) 1 s (2" 1)x for (0, 1). Since
21.22 22 > 0, it satisfies e e= - 22 (0, 1). Therefore, we get
e (1 =2)0 (2 1)e 12
By taking the natural logarithm of both sides, we get the desired inequality.
1.3 AUXILIARY RESULTS
Lemma L6 (Lemma C. in Zhang et al. (2024a)). The learning rate T/e = (f + 2) with a E (0,
satisfies Assumption 2.3.
Lemma 1.7 (Beroulli's Incquality). (a) Ifr and 1,, then (1 + x)) 2 rx.
(b) 1 S r S and x N -1, then (1 + x)) 11+ rz.
Lemma 18 (Stolz-Cesaro Theorem). Let (an)n21 and (bn)n21 be the two sequences of real numbers. Assume that (b,)n21 is strictly monotone and divergent sequence and the following limit exists:
lim Ox+1 On b bn
L.
Then it satisfies that
an bn
lim 7
Lemma 1.9 (Brouwer Fixed-point Theorem). Every continuous function from nonempty convex compact subset of R. to itself has a fixed point.
50


Reviews:
[{"review_id": "nIY6BVhoOn", "reviewer": "Reviewer_f1uy", "summary": "This paper studies the implicit bias of Adam for binary classification on linearly separable data, particularly the effect of batch size on the learned solution. Building on prior work that shows that iterates of full-batch Adam directionally converge to the $\\ell_\\infty$-max-margin solution, this work shows empirically, and theoretically, under some conditions, that for batch size 1, this is not the case: incremental Adam (using one sample every step) converges to the $\\ell_2$-max-margin solution for some dataset class, while for general datasets, it characeterizes the convergence of proxy-Adam (limit $\\beta_2\\to 1$) using a fixed-point framework. Additionally, the paper shows that other variants of signGD may not share this behaviour: Signum, in the limit of momentum parameters tending to 1, converges to the $\\ell_\\infty$-max-margin solution for any batch size.", "questions": "Some suggestions to further improve clarity are as follows:\n\n1. It should be clarified that for vectors, the division is element-wise (e.g., Eq. 2). Relatedly, it can be mentioned in the proof of Cor. 3.2, that shared elements in the denominator allow moving to common division rather than element-wise.  \n\n2. Omit ‘the’ in line 211.\n\n3. More pointers can be added to the Appendix in the main body. For instance, references to proofs, and the additional experiments would be helpful.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow, overall. \n\n2. It studies an important problem (what is the effect of batch size on the implicit bias of Adam?), characterizes the implicit bias of per-sample Adam under some conditions and provides interesting insights about how can be different compared to standard max-margin solutions .", "weaknesses": "There are a few weaknesses that the paper should address:\n\n1. The paper is missing discussion on some related works that study the implicit bias of Adam in neural networks [1-2], and effect of momentum parameter values [3] and rotations [4]. There is also another paper [5] on effect of batch sizes on SGD and Adam that should be cited in Section 7.\n\n2. In Fig. 1 (left), the cosine similarity of Adam iterates with the $\\ell_2$-max-margin solution does not converge to 1. This is later clarified in Fig. 3 (and Section 4). However, it could be useful to either i) add another panel in Fig. 1 (similar to Fig. 3 (right)), or ii) add a pointer to Fig. 3 or Section 4 in Fig. 1 caption. Relatedly, the Introduction mainly discusses how per-sample Adam converges “closer to $\\ell_2$-max-margin solution”, and it could be helpful to further clarify this early on, e.g., adding something similar to line 361 in point 3 of the Contributions. \n\n**References:**\n\n[1] Tsilivis et al., “Flavors of margin: Implicit bias of steepest descent in homogeneous neural networks”, ICLR 2025.\n\n[2] Vasudeva et al., “The Rich and the Simple: On the Implicit Bias of Adam and SGD”, NeurIPS 2025.\n\n[3] Orvieto and Gower, “In Search of Adam’s Secret Sauce”, NeurIPS 2025.\n\n[4] Zhang et al., “Understanding Adam Requires Better Rotation Dependent Assumptions”, NeurIPS 2025.\n\n[5] Marek et al., “Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful”, NeurIPS 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ugJChd5pZ", "reviewer": "Reviewer_v5Xn", "summary": "This paper studies the implicit bias of Adam in the noisy setting and showed it may not converge to $\\ell_\\infty$-max-margin as deterministic Adam. The converged direction depends on the dataset structure and batch size.", "questions": "1. Do you think it is possible to get similar result without assumption 4.4? What is the necessary role of it if we can't remove it?\n2. I only want to point out some possible experiments to explore, which might make the story more complete. But it won't affect my rating and doesn't need to be finished in the rebuttal process. I feel the batch size decides the balance between noise variance and deterministic gradient in $v_t$. When the batch size is small and noise dominates $v_t$, the update rule of Adam will become more similar with SGD in the sense that the effective learning rate is divided by $\\sigma$. When the batch size is large and deterministic gradient square dominates $v_t$, the update rule will be more similar to deterministic setting. Therefore, the shape of the noise may be as important as batch size. Then it's worth to try the following experiments. Fix deterministic gradient function $g(x)$ and manually inject noise $\\delta(t)$ so that the stochastic gradient is $g(x_t)+\\delta(t)$. $\\delta(t)$ can a random variable from Gaussian distribution and we can adjust the covariance matrix. We can multiply it by large constant or small constant and see how the converged direction of Adam changes under different scenario. For example, for what range of signal to noise ratio will the stochastic Adam converge closer to $\\ell_2$-max-margin or $\\ell_\\infty$-max-margin? We can also try covariance matrix that is different from identity matrix. When the eigenvalues are heterogeneous, will incremental Adam still converge to $\\ell_2$-max-margin?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper explicitly points out that previous literature only considers the implicit bias of Adam under deterministic setting and suggest that noise can have different influence on Adam and signum. It can expand the study of Adam. \n2. There is one rigorous theoretical example on the Generalized Rademacher data that explicitly shows that Adam will converge $\\ell_2$-max-margin. \n3. There are many synthetic experiments that show the solution that stochastic Adam converges to is closer to $\\ell_2$-max-margin, supporting the theoretical example and more general conjecture. The experiments also show that the converged solution of signum is never close to $\\ell_2$-max-margin. \n4. The experiments also show the key role of batch size in deciding the implicit bias.", "weaknesses": "1. The theory for stochastic Adam is not complete on general data. After the stage of theorem 4.8, the fixed point analysis can only be done by simulation of algorithm 3, which is less satisfying. Also the proposed AdamProxy almost reduces to AdaGrad. So it is not sure whether the analysis really suits for Adam, which works with a fixed $\\beta_2<1$. \n2. The assumption 4.4 is too strong, which requires the existence of the converged direction. In comparison, Zhang et al., 2024 doesn’t need to assume the existence but directly shows the convergence. \n3. The example of Generalized Rademacher data is not general enough. I feel the result in section 3 can hold for any dataset such that the stochastic gradient $g_t$ always satisfying $|g_t[k]|=|g_t[l]|$ for any $k,l \\in [d]$. However, this case is also too special in the sense that $v_t$ in Adam always satisfies $|v_t[k]|=|v_t[l]|$. The authors intentionally built the example to eliminate the coordinate-adaptivity of Adam but it also makes me wonder whether it can really help understand Adam in the general case. \n4. It would be better to include batch size or noise scale in the framework. Since the figure 6 already shows that batch size 5 can already make the solution deviate from the incremental Adam, it is better to understand the effect the batch size intuitively through theoretical result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AwL3J3hJtU", "reviewer": "Reviewer_yBsj", "summary": "In this paper, the authors study the implicit bias of Adam under stochastic mini-batch settings. Under certain assumptions, authors propose that stochastic Adam has an implicit bias toward a maximum margin solution w.r.t. a specific margin, determined by a data-dependent diagonal matrix. This result is vastly different from the previous work of implicit bias of full-batch Adam [1], and of great importance to the understanding of Adam.\n\n[1] Zhang et al. The implicit bias of Adam on separable data. NeurIPS 2024", "questions": "1. In fact, both the construction of the GR data and the assumption that $\\beta_2 \\to 0$ for general data ensure a critical point: within each epoch, all updates $w_{t+1} - w_t$ corresponding to different data points share the same adaptive term—namely, the denominator (please correct me if this observation is inaccurate). Based on this property, we can interchange the order of summations so that all data points contribute equally to the total update of one epoch. I am curious whether this property still holds if we consider randomly shuffling the data within each epoch, rather than processing them in a fixed order.\n2. Moreover, I suggest that the authors revise their presentation regarding the implicit bias on the GR data, at least in the abstract. This result heavily depends on the specific construction of the data distribution, and I am somewhat skeptical whether it can be legitimately described as an implicit bias of Adam. Therefore, stating the $\\ell_2$-margin conclusion in the abstract without explicitly mentioning its strong reliance on the particularly trivial data setup could be misleading, especially for readers who do not examine the full details.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Considering that the real practice of Adam is usually mini-batch instead of full-batch, studying the implicit bias of stochastic Adam is of great importance. \n2. The conclusions of this paper are fairly complete and abundant, covering the stochastic Adam and Signum.", "weaknesses": "1. **Strong technical assumptions**: The critical assumption that $\\beta_2\\to 1$ essentially makes the denominator of Adam's updates remain invariant for each data. From the theoretical perspective, such a simplification directly renders that stochastic Adam has the same updating direction as a specific preconditioned full-batch GD, which appears to be the direct reason for the derived implicit bias. The central question, however, is whether this assumption is justifiable for drawing conclusions about the true implicit bias of stochastic Adam in practical settings. This is particularly controversial because, even under the simple incremental setting, for any fixed $\\beta_2 < 1$, the term $\\beta_2^N$ decays dramatically with a large sample size $N$. Consequently, toy experiments with a minimal $N=10$ are insufficient and fail to substantiate the generalizability of the claimed conclusion. In addition, the assumption that Adamproxy has a convergent direction is also unsatisfactory. To some extent, combined with the connection between Adamproxy and preconditioned GD, the final conclusion is somewhat trivial, following the intuitive idea that only support vectors will contribute to the updates like [2]. I know that a similar assumption is also utilized in [3], but [3] does not study the simple linear classification settings, and hence acceptable.  Furthermore, many assumptions could be improved. For example, the assumption (a) in Theorem 3.3. should be rigorously established. In addition, could the author explain why the theoretical derivation for general data has to be built upon Adamproxy, instead of Adam, given that the remainder will converge to 0? \n\n2. **Lack of convergence rate**: The mathematical derivation and calculations do not provide an exact convergence rate for the directional convergence. \n\n3. **Potential issues in the proof**: In the proof of Proposition 4.3, authors finally conclude that $\\\\|\\nabla L(w_t)\\\\|\\_{P^{-1}(w_t)} \\to 0$ since $\\sum \\eta_t \\\\|\\nabla L(w_t)\\\\|\\_{P^{-1}(w_t)}$ is bounded. However, the boundedness of $\\sum a_n b_n$ and the divergence of $\\sum a_n$ do not, in general, guarantee that $b_n \\to 0$ (There exist many counter-examples such that $\\lim\\sup b_n >0$).  Authors should provide more details on how they derive such a conclusion. \n\n[2]. Soudry et al. The implicit bias of gradient descent on separable data. JMLR. \n\n[3]. Xie and Li. Adam exploits $\\ell_\\infty$-geometry of loss landscape via coordinate-wise adaptivity. ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)