ICML 2025

STACEY: Promoting Stochastic Steepest Descent via Accelerated
ℓp-Smooth Nonconvex Optimization
Xinyu Luo * 1 Cedar Site Bai * 1 Bolian Li * 1 Petros Drineas 1 Ruqi Zhang 1 Brian Bullins 1
Abstract
While popular optimization methods such as
SGD, AdamW, and Lion depend on steepest de-
scent updates in either ℓ2 or ℓ∞norms, there re-
mains a critical gap in handling the non-Euclidean
structure observed in modern deep networks train-
ing. In this work, we address this need by intro-
ducing a new accelerated ℓp steepest descent al-
gorithm, called STACEY, which uses interpolated
primal-dual iterate sequences to effectively nav-
igate non-Euclidean smooth optimization tasks.
In addition to providing novel theoretical guar-
antees for the foundations of our algorithm, we
empirically compare our approach against these
popular methods on tasks including image classi-
fication and language model (LLM) pretraining,
demonstrating both faster convergence and higher
final accuracy. We further evaluate different val-
ues of p across various models and datasets, un-
derscoring the importance and efficiency of non-
Euclidean approaches over standard Euclidean
methods.
Code can be found at https://
github.com/xinyuluo8561/Stacey.
1. Introduction
Stochastic first-order methods have proven essential for effi-
ciently training modern deep learning models. Beyond the
basic stochastic gradient descent (SGD) algorithm (Robbins
& Monro, 1951) and its momentum-based variants (Nes-
terov, 1983; Polyak, 1964), a variety of adaptive methods
have been developed, such as AdaGrad (Duchi et al., 2011a),
Adam (Kingma & Ba, 2015), and AdamW (Loshchilov &
Hutter, 2019), which incorporate second-moment gradient
information to provide per-coordinate scaling. Meanwhile,
*Equal contribution
1Department of Computer Science, Pur-
due University, Indiana, USA. Correspondence to: Xinyu Luo
<luo466@purdue.edu>, Cedar Site Bai <bai123@purdue.edu>,
Bolian Li <li4468@purdue.edu>.
Proceedings of the 42 nd Internat...