SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 2 (Individual convex+interpolation). Assume that  $f$  is convex, each  $f_{i}$  is convex and  $L$ -smooth, and that  $\mathrm{err}_{f,B}^{2} = \sigma_{f,B}^{2} = 0$ , by setting  $c_{p} = \frac{c_{p}^{scale}}{\sqrt{f_{i_{0}}(\mathbf{x}_{0}) - f_{i_{0}}^{\star}}}$  and  $c_{l} = \frac{c_{l}^{scale}}{\rho\sqrt{\gamma_{0}\|\nabla f_{i_{0}}(\mathbf{x}_{0})\|^{2}}}$  with constants  $c_{p}^{scale} \geq 1$  and  $c_{l}^{scale} \geq 1$ , then for any  $T \geq 1$ , SGD (no projection) with AdaSPS or AdaSLS converges as:
$$
(A d a S P S) \quad \mathbb {E} [ f (\bar {\mathbf {x}} _ {T}) - f ^ {\star} ] \leq \left(4 L \left(c _ {p} ^ {s c a l e}\right) ^ {2} \mathbb {E} _ {i _ {0}} \left[ \frac {\left\| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \right\| ^ {2}}{f _ {i _ {0}} (\mathbf {x} _ {0}) - f ^ {\star}} \right]\right) \frac {L \left\| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \right\| ^ {2}}{T}, \tag {7}
$$
$$
\left. \left(A d a S L S\right) \quad \mathbb {E} \left[ f \left(\bar {\mathbf {x}} _ {T}\right) - f ^ {\star} \right] \leq \left(\frac {\left(c _ {l} ^ {\text {s c a l e}}\right) ^ {2}}{\rho^ {3} L \min  ^ {2} \left\{\frac {1 - \rho}{L} , \gamma_ {\max } \right\}} \mathbb {E} _ {i _ {0}} \left[ \frac {\left| \left| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \right| \right| ^ {2}}{\gamma_ {0} \left| \left| \nabla f _ {i _ {0}} (\mathbf {x} _ {0}) \right| \right| ^ {2}} \right]\right) \frac {L \left| \left| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \right| \right| ^ {2}}{T}. \right. \tag {8}
$$
where  $\bar{\mathbf{x}}_T = \frac{1}{T}\sum_{t=1}^{T}\mathbf{x}_t$ .
The result implies that the bounded iterates assumption is not needed if we have both individual convexity and interpolation by picking  $c_p$  and  $c_l$  to satisfy certain conditions that do not depend on unknown parameters. To our knowledge, no such result exists for stepsizes from the AdaGrad family. It is worth noting that the min operator defined in AdaSPS or AdaSLS is not necessary in the proof.
Remark 3. We note that for non-interpolated problems, AdaSPS only requires the knowledge of  $\ell_{i_t}^{\star}$  while the exact  $f_{i_t}^{\star}$  is needed under the interpolation condition. We argue that in many standard machine learning problems, simply picking zero will suffice. For instance,  $f_{i_t}^{\star} = 0$  for over-parameterized logistic regression and after adding a regularizer,  $\ell_{i_t}^{\star} = 0$ .
Strongly convex. We now present two algorithmic behaviors of AdaSPS and AdaSLS for strongly convex functions. In particular, We show that 1) the projection step can be removed as shown in DecSPS, and 2) if the interpolation condition holds, the min operator is not needed and the asymptotic linear convergence rate is preserved. The full statement of Lemma 4 can be found in Appendix B.2.
Lemma 4 (Bounded iterates). Let each  $f_{i}$  be  $\mu$ -strongly convex and  $L$ -smooth. For any  $t = 0, \dots, T$ , the iterates of SGD with AdaSPS or AdaSLS satisfy:  $||\mathbf{x}_t - \mathbf{x}^\star ||^2 \leq D_{\max}$ , for a constant  $D_{\max}$  specified in the appendix in Equation (B.16).
Corollary 5 (Individual strongly convex). Assume each  $f_{i}$  is  $\mu$ -strongly convex and  $L$ -smooth, Theorem 1 holds with PSGD and  $D$  replaced by SGD and  $D_{\mathrm{max}}$  defined in Lemma 4.
Although it has not been formally demonstrated that AdaGrad/AdaGrad-Norm can relax the assumption on bounded iterates for strongly convex functions, we believe that with a similar proof technique, this property still holds for AdaGrad/AdaGrad-Norm.
We next show that AdaSPS and AdaSLS achieve linear convergence under the interpolation condition.
Theorem 6 (Strongly convex + individual convex + interpolation). Consider SGD with AdaSPS (AdaSPS) or AdaSLS (AdaSLS) stepsize. Suppose that each  $f_{i}$  is convex and  $L$ -smooth,  $f$  is  $\mu$ -strongly convex and that  $\sigma_{f,B}^{2} = \mathrm{err}_{f,B}^{2} = 0$ . If we let  $c_{p} = \frac{c_{p}^{\text{scale}}}{\sqrt{f_{i_{0}}(\mathbf{x}_{0}) - f_{i_{0}}^{*}}}$  and  $c_{l} = \frac{c_{l}^{\text{scale}}}{\rho \sqrt{\gamma_{0}||\nabla f_{i_{0}}(\mathbf{x}_{0})||^{2}}}$  with constants  $c_{p}^{\text{scale}} \geq 1$  and  $c_{l}^{\text{scale}} \geq 1$ , then AdaSPS or AdaSLS converges as:
$$
(A d a S P S) \quad \mathbb {E} [ | | \mathbf {x} _ {T + 1} - \mathbf {x} ^ {\star} | | ^ {2} ] \leq \mathbb {E} _ {i _ {0}} \left[ \left(1 - \frac {\left(f _ {i _ {0}} (\mathbf {x} _ {0}) - f ^ {\star}\right) \mu}{\left(2 c _ {p} ^ {\text {s c a l e}} L \| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \|\right) ^ {2}}\right) ^ {T} \right] | | \mathbf {x} _ {0} - \mathbf {x} ^ {\star} | | ^ {2}, \tag {9}
$$
$$
(A d a S L S) \quad \mathbb {E} [ \| \mathbf {x} _ {T + 1} - \mathbf {x} ^ {\star} \| ^ {2} ] \leq \mathbb {E} _ {i _ {0}} \left[ \left(1 - \frac {\mu \rho^ {3} \min  ^ {2} \left\{\frac {1 - \rho}{L} , \gamma_ {\max } \right\} \gamma_ {0} \| \nabla f _ {i _ {0}} (\mathbf {x} _ {0}) \| ^ {2}}{\left(c _ {l} ^ {\text {s c a l e}} \| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \|\right) ^ {2}}\right) ^ {T} \right] \| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \| ^ {2}. \tag {10}
$$
The proof of Theorem 6 is presented in Appendix B.3. We now compare the above results with the other step sizes. Under the same settings, DecSPS has a slower  $\mathcal{O}(1 / \sqrt{T})$  rate due to the usage of  $\Theta (1 / \sqrt{t})$  decay step size. While AdaGrad-Norm does have a linear acceleration phase when the accumulator grows large, to avoid an  $\mathcal{O}(1 / \varepsilon)$  slow down, the parameters of AdaGrad-Norm have to satisfy  $c_{g} < b_{0} / L$ , which requires the knowledge of Lipschitz constant [56]. Instead, the conditions on  $c_{p}$  and  $c_{l}$  for AdaSPS and AdaSLS only depend on the function value and gradient norm at  $\mathbf{x}_0$  which can be computed at the first iteration. SPS, SLS, and Vannilia-SGD with constant stepsize achieve faster linear convergence rate of order  $\mathcal{O}\big(\exp (-\frac{\mu}{L} T)\big)$ . It is worth noting that Vannila-SGD can further remove the individual convexity assumption.
Discussion. In non-interpolation regimes, AdaSPS and AdaSLS only ensure a slower  $\mathcal{O}(1 / \sqrt{T})$  convergence rate compared with  $\mathcal{O}(1 / T)$  rate achieved by vanilla SGD with  $\Theta (1 / t)$  decay stepsize when optimizing strongly-convex functions [7]. To our knowledge, no parameter-free adaptive stepsize exists that achieves such a fast rate under the same assumptions. Therefore, developing an adaptive algorithm that can adapt to both convex and strongly-convex functions would be a significant further contribution.
4 AdaSPS and AdaSLS with variance-reduction
Combining variance-reduction (VR) with adaptive Polyak-stepsize and line-search to achieve acceleration is a natural idea that has been explored in the last decade [49, 36]. However, it remains an open challenge as no theoretical guarantee has been proven yet. Indeed, Dubois-Taine et al. [14] provide a counter-example for intuitive line-search methods. In Appendix E we provide counter-examples of the classical SPS and its variants. The reason behind the failure is the biased curvature information provided by  $f_{i_t}$  that prevents global convergence. In this section, we introduce a novel framework to address this issue. Since there exists many variance-reduced stochastic gradient estimators, we focus on the classical SVRG estimator in this section, and our framework also applies to other estimators such as SARAH [40].
4.1 Algorithm design: achieving variance-reduction without interpolation
It is known that adaptive methods such as SPS or SLS converge linearly on problems where the interpolation condition holds, i.e.  $f(\mathbf{x})$  with  $\sigma_{f,B} = 0$ .
For problems that do not satisfy the interpolation condition, our approach is to transition the problem to an equivalent one that satisfies the interpolation condition. One such transformation is to shift each individual function by the gradient of  $f_{i}(\mathbf{x})$  at  $\mathbf{x}^{\star}$ , i.e.  $F_{i}(\mathbf{x}) = f_{i}(\mathbf{x}) - \mathbf{x}^{T}\nabla f_{i}(\mathbf{x}^{\star})$ . In this case  $f(\mathbf{x})$  can be written as  $f(\mathbf{x}) = \frac{1}{n}\sum_{i = 1}^{n}F_{i}(\mathbf{x})$  due to the fact that  $\frac{1}{n}\sum_{i = 1}^{n}\nabla f_{i}(\mathbf{x}^{\star}) = 0$ . Note that  $\nabla F_{i}(\mathbf{x}^{\star}) = \nabla f_{i}(\mathbf{x}^{\star}) - \nabla f_{i}(\mathbf{x}^{\star}) = 0$  which implies that each  $F_{i}(\mathbf{x})$  shares the same minimizer and thus the interpolation condition is satisfied ( $\sigma_{f,1}^{2} = 0$ ). However,  $\nabla f_{i}(\mathbf{x}^{\star})$  is usually not available at hand. This motivates us to design the following algorithm.
Algorithm 1 (Loopless) AdaSVRPS and AdaSVRLS
Require:  $\mathbf{x}_0\in \mathbb{R}^d$ $\mu_F > 0$ $c_{p} > 0$  or  $c_{l} > 0$    
1: set  $\mathbf{w}_0 = \mathbf{x}_0$ $\eta_{-1} = +\infty$    
2: for  $t = 0$  to  $T - 1$  do   
3: uniformly sample  $i_t\subseteq [n]$    
4: set  $F_{i_t}(\mathbf{x}) = f_{i_t}(\mathbf{x}) + \mathbf{x}^T (\nabla f(\mathbf{w}_t) - \nabla f_{i_t}(\mathbf{w}_t)) + \frac{\mu_F}{2} ||\mathbf{x} - \mathbf{x}_t||^2$    
5:  $\eta_{t} = \min \Bigl {\{}\frac{F_{i_{t}}(\mathbf{x}_{t}) - F_{i_{t}}^{*}}{c_{p}||\nabla F_{i_{t}}(\mathbf{x}_{t})||^{2}}\frac{1}{\sqrt{\sum_{s = 0}^{t}F_{i_{s}}(\mathbf{x}_{s}) - F_{i_{s}}^{*}}},\eta_{t - 1}\Bigr \}$  (AdaSVRPS)   
6:  $\eta_t = \min \left\{\gamma_t\frac{1}{c_l\sqrt{\sum_{s = 0}^t\gamma_s||\nabla F_{i_s}(\mathbf{x}_s)||^2}},\eta_{t - 1}\right\}$  (AdaSVRLS) $^2$    
7:  $\mathbf{x}_{t + 1} = \Pi_{\mathcal{X}}\bigl (\mathbf{x}_t - \eta_t\nabla F_{i_t}(\mathbf{x}_t)\bigr)$    
8:  $\mathbf{w}_{t + 1} = \left\{ \begin{array}{ll}\mathbf{w}_t & \text{with probability} 1 - p_{t + 1}\\ \mathbf{x}_t & \text{with probability} p_{t + 1} \end{array} \right.$    
9: return  $\bar{\mathbf{x}}_T = \frac{1}{T}\sum_{t = 0}^{T - 1}\mathbf{x}_t$
4.2 Algorithms and convergence
Inspired by the observation, we attempt to reduce the variance of the functions  $\sigma_{f,B}^{2}$  by constructing a sequence of random functions  $\{F_{i_t}(\mathbf{x})\}$  such that  $\sigma_{\frac{1}{n}\sum_{i = 1}^{n}F_{i_t}(\mathbf{x}),B}\to 0$  as  $\mathbf{x}_t\rightarrow \mathbf{x}^\star$ . However, directly applying SPS or SLS to  $\{F_{i_t}(\mathbf{x})\}$  still requires the knowledge of the Lipschitz constant to guarantee convergence. This problem can be solved by using our proposed AdaSPS and AdaSLS. The whole procedure of the final algorithm is summarized in Algorithm 1.
At each iteration of Algorithm 1, we construct a proxy function by adding two quantities to the minibatch function  $f_{i_t}(\mathbf{x})$ , where  $\frac{\mu_F}{2} ||\mathbf{x} - \mathbf{x}_t||^2$  is a proximal term that helps improve the inherent stochasticity due to the partial information obtained from  $f_{i_t}(\mathbf{x})$ . The additional inner product quantity is used to draw closer the minimizers of  $f_{i_t}(\mathbf{x})$  and  $f(\mathbf{x})$ . Following [27, 33], the full gradient is computed with a coin flip probability. Note that Algorithm 1 still works with  $\eta_t$  replaced with SVRG and AdaSVRG stepsize since  $\nabla F_{i_t}(\mathbf{x}_t) = \nabla f_{i_t}(\mathbf{x}_t) - \nabla f_{i_t}(\mathbf{w}_t) + \nabla f(\mathbf{w}_t)$ , and thus this framework can be seen as a generalization of the standard VR methods. A similar idea can also be found in the works on federated learning with variance-reduction [32, 2, 24, 37, 50].
Theorem 7. Assume each  $f_{i}$  is convex and  $L$  smooth and  $\mathcal{X}$  is a convex compact feasible set with diameter  $D$ . Let  $p_t = \frac{1}{at + 1}$  with  $0 \leq a < 1$ . Algorithm 1 converges as:
$$
(A d a S V R P S) \quad \mathbb {E} [ f (\bar {\mathbf {x}} _ {T}) - f ^ {\star} ] \leq \frac {1 + \frac {2 L}{(1 - a) \mu_ {F}}}{T} \left(2 c _ {p} (L + \mu_ {F}) D ^ {2} + \frac {1}{c _ {p}}\right) ^ {2}, \tag {11}
$$
$$
(A d a S V R L S) \mathbb {E} [ f (\bar {\mathbf {x}} _ {T}) - f ^ {\star} ] \leq \frac {1 + \frac {2 L}{(1 - a) \mu_ {F}}}{T} \left(\max  \left\{\frac {L + \mu_ {F}}{(1 - \rho) \sqrt {\rho}}, \frac {1}{\gamma_ {\operatorname* {m a x}} \sqrt {\rho}} \right\} c _ {l} D ^ {2} + \frac {1}{c _ {l} \sqrt {\rho}}\right) ^ {2}, \tag {12}
$$
where  $\bar{\mathbf{x}}_T = \frac{1}{T}\sum_{t = 0}^{T - 1}\mathbf{x}_t$
Suppose  $\gamma_{\mathrm{max}}$  is sufficiently large, then picking  $\mu_F^\star = \mathcal{O}(L)$ ,  $c_p^\star = \mathcal{O}\left(\frac{1}{\sqrt{LD^2}}\right)$  and  $c_l^\star = \mathcal{O}\left(\frac{\sqrt{1 - \rho}}{\sqrt{LD^2}}\right)$  yields an  $\mathcal{O}\left(\frac{LD^2}{T}\right)$  rate which matches the  $\mathcal{O}\left(\frac{L||\mathbf{x}_0 - \mathbf{x}^\star||^2}{T}\right)$  rate of full-batch gradient descent except for a larger term  $D^2$  due to the lack of knowledge of the Lipschitz constant.
Corollary 8. Under the setting of Theorem 7, given an arbitrary accuracy  $\varepsilon$ , the total number of gradient evaluations required to have  $\mathbb{E}[f(\bar{\mathbf{x}}_T) - f^{\star}] \leq \varepsilon$  in expectation is  $\mathcal{O}(\log(1/\varepsilon)n + 1/\varepsilon)$ .
The proved efficiency of stochastic gradient calls matches the optimal rates of SARAH [40]/SVRG and AdaSVRG [14] but removes the artificially designed inner and outer loop size. However, note that Algorithm 1 requires an additional assumption on individual convexity. Unfortunately, we believe this
where  $\gamma_{t}$  is obtained via the Armijo backtracking line-search (Algorithm 4) which satisfies:  $F_{i_t}(\mathbf{x}_t - \gamma_t\nabla F_{i_t}(\mathbf{x}_t))\leq F_{i_t}(\mathbf{x}_t) - \rho \gamma_t||\nabla F_{i_t}(\mathbf{x}_t)||^2$  and  $\gamma_{t}\leq \gamma_{\mathrm{max}}$
Figure 2: Illustration of the accelerated convergence of AdaSVRPS and AdaSVRLS on quadratic loss without interpolation. Both algorithms require less gradient evaluations than AdaSPS and AdaSLS for optimizing non-interpolated problems. However, they are less efficient for solving interpolated problems. (Repeated 3 times. The solid lines and the shaded area represent the mean and the standard deviation.)
assumption is necessary for the VR methods to work for the algorithms in the Polyak and line-search family since SPS/SLS has to assume the same condition for the proof in the interpolation settings.
Discussion. The classical SVRG with Armijo line-search (presented as Algorithm 6 in [14]) employs the same gradient estimator as SVRG but chooses its stepsize based on the returning value of line-search on the individual function  $f_{i}$ . Similarly, SVRG with classical Polyak stepsize uses the individual curvature information of  $f_{i}$  to set the stepsize for the global variance-reduced gradient. Due to the misleading curvature information provided by the biased function  $f_{i}$ , both methods have convergence issues. In constraint, Algorithm 1 reduces the bias by adding a correction term  $\mathbf{x}^T (\nabla f(\mathbf{w}_t) - \nabla f_{i_t}(\mathbf{w}_t))$  with global information to  $f_{i}$  and then applying line-search or Polyak-stepsize on the variance-reduced functions  $F_{i_t}$ . This difference essentially guarantees the convergence.
5 Numerical evaluation
In this section, we illustrate the main properties of our proposed methods in numerical experiments. A detailed description of the experimental setup can be found in Appendix F. We report the theoretically justified hyperparameter  $c_p^{\mathrm{scale}}$  or  $c_l^{\mathrm{scale}}$  as defined in Theorem 6 rather than  $c_p$  or  $c_l$  in the following.
Synthetic data. We illustrate the robustness property on a class of synthetic problems. We consider the minimization of a quadratic of the form:  $f(\mathbf{x}) = \frac{1}{n}\sum_{i=1}^{n}f_{i}(\mathbf{x})$  where  $f_{i}(\mathbf{x}) = \frac{1}{2} (\mathbf{x} - \mathbf{b}_{i})^{T}A_{i}(\mathbf{x} - \mathbf{b}_{i})$ ,  $\mathbf{b}_{i} \in \mathbb{R}^{d}$  and  $A_{i} \in \mathbb{R}^{d\times d}$  is a diagonal matrix. We use  $n = 50$ ,  $d = 1000$ . We can control the convexity of the problem by choosing different matrices  $A_{i}$ , and control interpolation by either setting all  $\{\mathbf{b}_{i}\}$  to be identical or different. We generate a strongly convex instance where the eigenvalues of  $\nabla^2 f(\mathbf{x})$  are between 1 and 10, and a general convex instance by setting some of the eigenvalues to small values close to zero (while ensuring that each  $\nabla^2 f_{i}(\mathbf{x})$  is positive semi-definite). The exact procedure to generate these problems is described in Appendix F.
For all methods, we use a batch size  $B = 1$ . We compare AdaSPS and AdaSLS against DecSPS [44], SPS [34] and SLS [52] to illustrate the robustness property. More comparisons can be found in Appendix F.1. We fix  $c_p^{\mathrm{scale}} = c_l^{\mathrm{scale}} = 1$  for AdaSPS/AdaSLS and use the optimal parameters for DecSPS and SPS. In Figure 1, we observe that SPS does not converge in the non-interpolated settings and DecSPS suffers from a slow  $\mathcal{O}(1 / \sqrt{T})$  convergence on the two interpolated problems. AdaSPS and AdaSLS show the desired convergence rates across all cases which matches the theory. When the problems are non-interpolated, AdaSVRPS and AdaSVRLS illustrate faster convergence, which can be seen in Figure 2.
Binary classification on LIBSVM datasets. We experiment with binary classification on four diverse datasets from [10]. We consider the standard regularized logistic loss:  $f(\mathbf{x}) = \frac{1}{n}\sum_{i=1}^{n}\log(1 + \exp(-y_i \cdot \mathbf{a}_i^T\mathbf{x})) + \frac{1}{2n}||\mathbf{x}||^2$  where  $(\mathbf{a}_i, y_i) \in \mathbb{R}^{d+1}$  are features and labels. We defer the study of variance-reduction methods to Appendix F.2 for clarity of presentation. We benchmark against popular optimization algorithms including Adam [26], SPS [34], DecSPS [44], SLS [52] and AdaGrad-Norm [15]. We fix  $c_p^{\mathrm{scale}} = c_l^{\mathrm{scale}} = 1$  for AdaSPS/AdaSLS and pick the best learning rate from  $\{10^i\}_{i=-4,\dots,3}$  for SGD, Adam and AdaGrad-Norm. We observe that Adam, SPS and SLS have remarkable performances on duke with  $n = 48$  and  $d = 7129$ , which satisfies interpolation. AdaSPS
Figure 3: Comparison of AdaSPS/AdaSLS against six other popular optimizers on four LIBSVM datasets, with batch size  $B = 1$  for duke,  $B = 64$  for rcv1,  $B = 64$  for ijCNN and  $B = 128$  for w8a. AdaSPS and AdaSLS have competitive performance on rcv1, ijCNN, and w8a while SPS, SLS, and Adam converge fast on duke. (Repeated 3 times. The solid lines and the shaded area represent the mean and the standard deviation.)
CIFAR10-ResNet34
Figure 4: Comparison of the considered optimizers on multi-class classification tasks with CIFAR10 and CIFAR100 datasets using ResNet34 with softmax loss.
R100-ResNet34
and AdaSLS consistently perform reasonably well on the other three larger datasets. It is worth noting that the hyper-parameters  $c_p^{\mathrm{scale}}$  and  $c_l^{\mathrm{scale}}$  are fixed for all the datasets, which is desired in practice.
Deep learning task. We provide a heuristic extension of AdaSPS to over-parameterized non-convex optimization tasks to illustrate its potential. We benchmark the convergence and generalization performance of AdaSPS (DL) 5 for the multi-class classification tasks on CIFAR10 [28] and CIFAR100 [29] datasets using ResNet-34 [21]. More experimental details can be found in Appendix G. We demonstrate the effectiveness of AdaSPS (DL) in Figure 4.
Discussion. AdaSPS and AdaSLS consistently demonstrate robust convergence across all tasks and achieve performance on par with, if not better than, the best-tuned algorithms. Consequently, it is reliable and convenient for practical use.
6 Conclusion and future work
We proposed new variants of SPS and SLS algorithms and demonstrated their robust and fast convergence in both interpolated and non-interpolated settings. We further accelerate both algorithms for convex optimization with a novel variance reduction technique. Interesting future directions may include: accelerating AdaSPS and AdaSLS with momentum, developing effective robust adaptive methods for training deep neural networks, designing an adaptive algorithm that gives a faster rate  $\mathcal{O}(1 / T)$  under strong convexity, extensions to distributed and decentralized settings.
We appreciate the fruitful discussion with Anton Rodomanov.
[48] Mark Schmidt and Nicolas Le Roux. Fast convergence of stochastic gradient descent under a strong growth condition. arXiv preprint arXiv:1308.6370, 2013.
[49] Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162:83-112, 2017.
[50] Ohad Shamir, Nati Srebro, and Tong Zhang. Communication-efficient distributed optimization using an approximate newton-type method. In International conference on machine learning, pages 1000-1008. PMLR, 2014.
[51] Matthew J. Streeter and H. Brendan McMahan. Less regret via online conditioning. CoRR, abs/1002.4862, 2010. URL http://arxiv.org/abs/1002.4862.
[52] Sharan Vaswani, Aaron Mishkin, Issam H. Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien. Painless stochastic gradient: Interpolation, line-search, and convergence rates. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3727-3740, 2019. URL https://proceedings.neurips.cc/paper/2019/bit/2557911c1bf75c2b643afb4ecbfc8ec2-Abstracht.html.
[53] Sharan Vaswani, Issam Laradji, Frederik Kunstner, Si Yi Meng, Mark Schmidt, and Simon Lacoste-Julien. Adaptive gradient methods converge faster with over-parameterization (but you should do a line-search). arXiv preprint arXiv:2006.06835, 2020.
[54] Xiaoyu Wang, Mikael Johansson, and Tong Zhang. Generalized polyak step size for first order optimization with momentum. arXiv preprint arXiv:2305.12939, 2023.
[55] Rachel Ward, Xiaoxia Wu, and Leon Bottou. AdaGrad stepsizes: Sharp convergence over nonconvex landscapes. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6677-6686. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/ward19a.html.
[56] Yuege Xie, Xiaoxia Wu, and Rachel Ward. Linear convergence of adaptive stochastic gradient descent. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1475–1485. PMLR, 26–28 Aug 2020. URL https://proceedings.mlr.press/v108/xie20a.html.
A Technical Preliminaries
A.1 Basic Definitions
We use the following definitions throughout the paper.
Definition 1 (convexity). A differentiable function  $f: \mathbb{R}^d \to \mathbb{R}$  is convex if  $\forall \mathbf{x}, \mathbf{y} \in \mathbb{R}^d$ ,
$$
f (\mathbf {y}) \geq f (\mathbf {x}) + \left\langle \nabla f (\mathbf {x}), \mathbf {y} - \mathbf {x} \right\rangle . \tag {A.1}
$$
Definition 2 (strong convexity). A differentiable function  $f: \mathbb{R}^d \to \mathbb{R}$  is  $\mu$ -strongly convex if  $\forall \mathbf{x}, \mathbf{y} \in \mathbb{R}^d$ ,
$$
f (\mathbf {y}) \geq f (\mathbf {x}) + \left\langle \nabla f (\mathbf {x}), \mathbf {y} - \mathbf {x} \right\rangle + \frac {\mu}{2} \| \mathbf {x} - \mathbf {y} \| ^ {2}. \tag {A.2}
$$
Definition 3 ( $L$ -smooth). Let function  $f: \mathbb{R}^d \to \mathbb{R}$  be differentiable.  $f$  is smooth if there exists  $L > 0$  such that  $\forall \mathbf{x}, \mathbf{y} \in \mathbb{R}^d$ ,
$$
\left| \left| \nabla f (\mathbf {x}) - \nabla f (\mathbf {y}) \right| \right| \leq L \| \mathbf {x} - \mathbf {y} \|. \tag {A.3}
$$
A.2 Useful Lemmas
We frequently use the following helpful lemmas for the proof.
Lemma 9 (Nesterov [39], Lemma 1.2.3). Definition 3 implies that there exists a quadratic upper bound on  $f$ :