ICML 2025

Revisiting Convergence: Shuffling Complexity Beyond Lipschitz Smoothness
Qi He 1 Peiran Yu 2 Ziyi Chen 1 Heng Huang 1
Abstract
Shuffling-type gradient methods are favored in
practice for their simplicity and rapid empirical
performance. Despite extensive development of
convergence guarantees under various assump-
tions in recent years, most require the Lipschitz
smoothness condition, which is often not met in
common machine learning models. We highlight
this issue with specific counterexamples. To ad-
dress this gap, we revisit the convergence rates of
shuffling-type gradient methods without assuming
Lipschitz smoothness. Using our stepsize strategy,
the shuffling-type gradient algorithm not only con-
verges under weaker assumptions but also match
the current best-known convergence rates, thereby
broadening its applicability. We prove the conver-
gence rates for nonconvex, strongly convex, and
non-strongly convex cases, each under both ran-
dom reshuffling and arbitrary shuffling schemes,
under a general bounded variance condition. Nu-
merical experiments further validate the perfor-
mance of our shuffling-type gradient algorithm,
underscoring its practical efficacy.
1. Introduction
Gradient-based optimization has always been a critical area
due to its extensive practical applications in machine learn-
ing, including reinforcement learning (Sutton and Barto,
2018), hyperparameter optimization (Feurer and Hutter,
2019), and large language models (Radford et al., 2018).
While numerous gradient-based algorithms have been devel-
oped for convex functions (Nemirovskij and Yudin, 1983;
Nesterov, 2013; d’Aspremont et al., 2021), research on non-
convex functions has become particularly active in recent
years, driven by advances in deep learning. Notably, with
unbiased stochastic gradients and bounded variance, SGD
1Department of Computer Science, University of Maryland,
College Park. 2Department of Computer Science and Engineer-
ing, University of Texas Arlington. Correspo...