Title: Theoretical Investigation of Adafactor for Non-Convex Smooth Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Adafactor

stochastic optimization

non-convex smooth optimization

convergence rate
Abstract: Adafactor is an early memory-efficient optimization algorithm proposed as an alternative to Adam. By eliminating first-order momentum and employing a
rank-$1$ matrix factorization to approximate the second-moment matrix, Adafactor achieves near-zero memory overhead compared to traditional gradient descent methods.
Despite its practical suitability for large-scale training tasks where memory efficiency is critical, its theoretical convergence analysis remains unexplored, largely due to the challenges posed by its matrix factorization and update clipping mechanisms. In this work, we provide a convergence analysis of Adafactor for non-convex smooth optimization.
We establish optimal convergence rates (up to logarithmic factors) for finding stationary points in both deterministic and stochastic settings, the latter under sub-Gaussian noise.
Central to our analysis is viewing Adafactor as an approximation of Adam, and the use of a new proxy step-size to approximate the unique
adaptive step-size induced by Adafactor's matrix factorization and update clipping, along with an induction argument to control the gradient magnitude. 
Our findings may theoretically suggest that involving rank-$1$ matrix approximation of the second-moment matrix in Adam does not fundamentally hinder the convergence.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work establishes convergence guarantees for the Adafactor algorithm in non-convex smooth optimization. Adafactor is analyzed in the deterministic case, the stochastic case without clipping, and in the stochastic case with clipping under a slight modification from a fixed clipping threshold to a time-dependent tuned threshold. Experiments on language tasks are provided to demonstrate the relationship between a theory-related parameter ($c$) and the training loss.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. It is of value to have proper convergence guarantees for practical methods, in order to better understand the proposed methods and to support future improvements.\n2. An analysis of a method such as Adafactor requires handling several technical challenges, which the authors detailed in their work. It is often the case that techniques introduced in one method can help in the analysis of other closely related methods.\n\n**Weaknesses**\n\n3. Beyond establishing the convergence of Adafactor (which is of value of its own), I do not see how the analysis leads to new and meaningful conclusions. This is in addition to the fact that Adafactor, or variants of it, is not a widely used method.\n4. A weakness of this work, which often appears in the analysis of such methods, is the sub-optimal bounds (in terms of problem parameters) compared to vanilla SGD. It would be beneficial to demonstrate a somewhat general setting (under additional but reasonable assumptions) where Adafactor has improved convergence compared to SGD. This is minor as this weakness is shared across several works on the topic of adaptive methods, but solving it can contribute to the significance of this work.\n\nQUESTIONS:\n1. While this work is mainly focused on the dependence on $T$, can the authors compare the established result to other adaptive methods (Such as AdaGrad, Adam, and Shampoo. Not necessarily all of them) in terms of the dependence on the dimension? From what I gather, the dependence is linear in the deterministic case and super-linear in the stochastic case.\n2. Can the authors conjecture of other methods whose theoretical analysis might benefit from the tools developed in this work?\n\nOverall, this work contributes to our understanding of adaptive methods by providing convergence guarantees for the Adafactor method, while the established guarantees are yet to indicate why such methods perform so well in practice.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

comments: SUMMARY:
The paper provides the theoretical analysis of Adafactor for nonconvex objectives under standard smoothness and stochastic gradient assumptions. The proof techniques are extended from those used to analyze the convergence of earlier adaptive stepsize algorithms such as Adam. The convergence rates are comparable to those of Adam.

STRENGTHS AND WEAKNESSES:
Strengths

- Adafactor is an important memory-saving adaptive stepsize algorithm, and its theoretical analysis would be important for practitioners and theoreticians as well. 

- The writing is clear. Assumptions and theoretical results are clearly stated. The new techniques are properly highlighted in Section 8.

Weakness

- The value of $c_0$, which is related to $\epsilon_1$ according to Theorem 6.1, should be the order of $10^{-30}$ based on [37]. This may result in super large constants $H$ and $I$. Authors may also discuss the theoretical implications of small $\epsilon_1$.

-  One motivation of Adafactor is reducing memory cost from $O(mn)$ to $O(m+n). Therefore, it is better to also make the dependence on $m$ and $n$ explicit in Theorem 5.1, Theorem 6.1, and Theorem 7.1.

QUESTIONS:
- The introduction before formulation (1) is slightly too narrow. Adafactor and other adaptive stepsize algorithms are proposed to optimize neural networks with multiple matrices instead of only one matrix. Could authors revise the introduction?

- In Figure 1, could authors also compare Adam's training curve with Adafactor's? This would convince the reader about the effectiveness of the implementation of Adafactor.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-01

comments: SUMMARY:
The paper studies the convergence of the Adafactor algorithm for smooth non-convex functions. It provides convergence rates under sub-Gaussian noise that match the optimal rate up to log factors for this class of functions.

STRENGTHS AND WEAKNESSES:
Strengths: 
- The paper is clearly written and does a good job introducing the algorithm for readers who might not be super familiar, and it also does a good job explaining the challenges that were faced in obtaining such theoretical results.  
- The theoretical analysis is interesting for the community: ideas used here for dealing with the one rank approximation and the clipping of Adafactor can be potentially used for other methods.

Weaknesses (minor):
- The theory assumes smoothness and bounds the noise in terms of the Frobenius norm, this norm hides (in the worst case) a factor that is proportional to the dimension compared to operator norms (associated to $\ell_2$ for example). 
- I can also go on about smoothness that is known not to hold for NNs, but we have to start somewhere. It would be nice if this could be extended to more general smoothness assumptions.

QUESTIONS:
-  Can your approach work with smoothness with operator norms instead of Frobenius?

- What about other smoothness assumptions like $(L_0,L_1)$?

LIMITATIONS:
Maybe more general smoothness assumptions and operator norms.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
The paper analyzes the convergence rate to a stationary point of Adafactor.

STRENGTHS AND WEAKNESSES:
# Strengths
1. For the deterministic case, the paper proves an optimal convergence rate $\mathcal{O}(1/T)$ for Adafactor.
2. For the stochastic case, the paper proves a near-optimal convergence rate $\mathcal{O}(1/\sqrt{T})$ for Adafactor.

# Weaknesses
1. Some things are not written in the most clear way. More details are written in the suggestions.

QUESTIONS:
# Suggestions
1. Some symbols look too close to each other, I suggest changing some of them.
    1. $c$ and $c_0$ have different purposes, but almost identical symbols.
    2. $G_k$ and $\mathbf{G}_k$ looks almost identical. I think that bold text is not enough to differentiate between symbols.
2. The corollaries are not a direct result of the theorem as they use different $\beta_{2,k}$. I think the paper should clarify that the proofs of the corollaries are in the appendix.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-14