Title: Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold
Venue: NeurIPS 2025
Accepted: yes
Keywords: Adam

adaptive gradient methods

implicit bias

regularization

deep learning theory
Abstract: Despite the popularity of Adam optimizer in practice, most theoretical analyses study SGD as a proxy and little is known about how the solutions found by Adam differ. In this paper, we show that Adam reduces a specific form of sharpness measure shaped by its adaptive updates, leading to qualitatively different solutions from SGD.
    When the training loss is small, Adam wanders around the manifold of minimizers and takes semi-gradients to minimize this sharpness measure in an adaptive manner, a behavior we rigorously characterize via a continuous-time approximation using stochastic differential equations.
    We further illustrate how this behavior differs from that of SGD in a well-studied setting: when training overparameterized models with label noise, SGD has been shown to minimize the trace of the Hessian matrix, $\text{tr}(\textbf{H})$, whereas we prove that Adam minimizes $\text{tr}(\text{diag}(\textbf{H})^{1/2})$ instead. In solving sparse linear regression with diagonal linear networks, Adam provably achieves better sparsity and generalization than SGD due to this difference.
    Finally, we note that our proof framework applies not only to Adam but also to a broad class of adaptive gradient methods, including but not limited to RMSProp, Adam-mini, and Adalayer. This provides a unified perspective for analyzing how adaptive optimizers reduce sharpness and may offer insights for future optimizer design.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo analyze the optimization trajectories of Adam after reaching the minimizer manifold, the authors propose the *slow sde* framework, which applies to a general class of adaptive optimizers, including Adam. Authors prove that such a continuous SDE solution can approximate the iterates of Adam within an error lower than $O(\\eta^{1/4})$. In addition, they also provide a convergence guarantee for Adam to reach the minimizer manifold. Based on such an SDE approximation, the authors analyze the fixed points of the SDE and provide the implicit bias of Adam and AdamE-$\\lambda$, respectively.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The analysis regarding the continuous approximation of optimizers plays a key role in helping to understand their behavior. Therefore, such a result is of great significance, especially given that there are few related works for Adam due to its complicated updating rules compared to SGD.\n2. The theoretical findings and illustration of this paper are relatively complete and solid. Almost all the theoretical conclusions are established with convincing evidence and clear illustrations. For example, the authors provide both the convergence guarantee to reach the minimizer manifold, then the SDE approximation inner the neighborhood of such a minimizer manifold. I would, however, suggest that authors consider relocating subsection 4.3.1, the convergence of Adam, to the beginning of Section 4. \n\n**Weakness:**\nSome of the following points might be more suitable to state in the question part, but I still state them here since I'm not very familiar with the SDE and differential geometry. Due to the same reason, I have set my confidence level of this rating to 2. Authors could correct me if I have any misunderstandings regarding their results.\n\n1. The requirement that $1-\\beta_2$ is comparable with $\\eta^2$, coupled with the $O(\\eta^{1/4})$  approximation error established in Theorem 4.1, implies that the \"slow ODE\" presented in this paper might not adequately capture Adam's trajectories. For example, taking $\\beta_2=0.9999$, a value already larger than common settings, still yields a convergence guarantee on the scale of $1/3$.  Consequently, this approximation might not provide a strong guarantee. Is such a requirement raised as you use the $\\eta^2$ as the interpolation scale to establish the ode form of $v_t$? Similarly, I wonder whether there are some typos in the formula of Theorem 4.1 of the term $\\eta^2$. I also wonder why there exists a factor $\\sqrt\\eta$ before the diffusion term of the conventional SDE in Line 129. From my perspective, the noise of the stochastic gradient might be uncorrelated with the learning rate. \n\n2.  While the conclusion of the implicit bias regarding the trace of Hessian is convincing to me, some explanations of such an result would be much better. For example, what are the connections or comparisons between this result and the direct conclusion that Adam can maximize the $\\ell_\\infty$-margin in a recent work [1]. At least to me, the conclusion in [1] is easier to understand its implication. I know you have provided an example in subsection 5.2. However, its extremely specific settings ($z_i \\in \\\\{1, -1\\\\}$) make me feel that the conclusion related to the implicit regularization of different forms of norm lacks sufficient generalization. In addition, I suggest that the authors could design some experiments to directly support their main theorems on the trace of the Hessian matrix. \n\n[1] Zhang et al. The implicit bias of Adam on separable data. NeurIPS.\n\nQUESTIONS:\nCould the authors explain more about ''slow sde\" for both the SGD and Adam? For me, the conventional SDE is clear. However, the \"slow sde\" is relatively hard to follow. It's difficult to understand why the trajectories of SGD can be characterized in this form. What do the diffusion term and the drift term indicate? Why is the current updating direction of SGD (from my understanding, this should depend on its local information) related to its convergence point (as reflected in the definition of $\\Phi$)? \n\nBesides, could you please explain how you deal with the second term in line 306 of the proof sketch? Since the definition of the calculation $\\nabla (\\cdot) [H]$ is defined for a Hessian matrix, it's a little hard for me to follow this calculation.\n\nIn addition, although you have mentioned that the first momentum would not affect the approximation result, I wonder whether you can also add it as a part of your \"slow sde\" system? This is not a major concern, just for better illustration.\n\nLIMITATIONS:\nNo negative societal impact. Potential technical weaknesses have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-26"}

comments: SUMMARY:
This paper studies the Adam optimizer and shows that it reduces a specific form of sharpness that leads to qualitatively different solutions from SGD. The paper shows that when using Adam to train over-parameterized models with label noise, Adam minimizes $tr(Diag(H)^{1/2})$ a measure of "sharpness" which differs from the $tr(H)$ that is minimized with SGD. Finally, they show that for the specific case of solving sparse linear regression with diagonal linear networks Adam achieves better sparsity and generalization than SGD.

STRENGTHS AND WEAKNESSES:
## Strengths:
- The paper is written clearly and the results bring new insights into how the implicit bias of Adam differs from SGD.
- The paper provides concrete examples that illustrate the qualitative difference between the kinds of solutions Adam learns vs. SGD (such as Fig. 1 and the sparse linear regression example).
- The paper introduces the AdamE-$\lambda$ optimizer to directly tune the implicit bias of Adam.
- The theoretical contributions apply broadly to many AGM and is not specific to Adam.

## Weaknesses:
- The entire Appendix is missing which makes it impossible to verify the correctness of the proofs.
- Adam is typically used to train ReLU neural networks which are non-smooth and non-convex. It is not clear whether the analysis here translates to this more realistic setting.

QUESTIONS:
None.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
This paper investigates the behavior of the Adam optimizer under the assumption that the set of minimizers forms a manifold. By analyzing the corresponding slow stochastic differential equation (SDE), the authors demonstrate that Adam  promotes sparsity in the solution.

STRENGTHS AND WEAKNESSES:
I'm not familiar with the field. Here are some strengths and weaknesses:

Strengths:

This paper generalizes the slow SDE to more general adaptive optimizers, like Adam. The result also tells us the difference between SGD and Adam minimizers they find.

Weaknesses:

1.The assumption implies that the trajectory x(t) of the optimizer converges to a single point, which can be quite strong. In practice, especially in non-convex settings or under noisy dynamics (like SDEs), only subsequence convergence may be guaranteed.

2. Why Assume the Minimizer Lies on a Manifold? This is often based on empirical observation as said in the paper. However, from a theoretical standpoint, assuming the set of minimizers is a manifold requires strong structural properties.


minor comments:
Line 159: better to use the terminology: in the normal space.

line 202: preconditioner flow: lacks blanket.

Definition 4.3: Typo Correct: given → Given.

QUESTIONS:
See the weaknesses above.

Moreover, Diag(H)^{1/2} is Diag(H^{1/2}) or (Diag(H))^{1/2}?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper theoretically analyzes the implicit bias of Adam. One of the main results shows that in the setting with label noise, Adam and SGD minimize different things: Adam minimizes $\\mathrm{tr}(\\mathrm{Diag}(H)^{1/2})$ and (as has been previously known) SGD minimizes $\\mathrm{tr}(H)$. Furthermore, the authors propose a tunable version of Adam which minimizes $\\mathrm{tr}(\\mathrm{Diag}(H)^{\\lambda})$ for any choice of $\\lambda \\in (0,1)$. This allows more control over the desired implicit bias of the algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nUnfortunately, I could not check all the math, but the paper appears very interesting to me.\n\nStrengths:\n* Adam is considered far less in theoretical studies than GD is despite the fact that Adam is, used more in many practical settings. This paper adds to the understanding of Adam.\n* The tools presented in the paper capture other adaptive gradient methods and should provide useful for further works in the future. It is almost more like a framework than a single result.\n* The authors make an effort to try to illustrate what the difference in behaviour between SGD and Adam mean intuitively.\n* Proving Adam's bias towards sparsity in the presence of label noise is very interesting.\n\nWeakness:\n* There are a few assumptions (but I found them mostly mild).\n* The analysis only works for specific parameter settings: $\\beta_2$ and the learning rate have to follow a specific relation.\n\nQUESTIONS:\n* In the experiments, all version of AdamE either behave more or less like SGD or like Adam. Can you narrow down at which $\\lambda$ the change happens? Or, if we tried many more intermediate values of $\\lambda$, we would see a gradual shift requiring more and more training data as $\\lambda$ decreases? If it is sudden, do you have an explanation for this phase transition?\n* I assume \"Aappendix\" in line 302 should say Appendix A?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}