SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaowen Jiang
CISPA*
xiaowen.jiang@cispa.de
Sebastian U. Stich
stich@cispa.de
The recently proposed stochastic Polyak stepsize (SPS) and stochastic line-search (SLS) for SGD have shown remarkable effectiveness when training overparameterized models. However, two issues remain unsolved in this line of work. First, in non-interpolation settings, both algorithms only guarantee convergence to a neighborhood of a solution which may result in a worse output than the initial guess. While artificially decreasing the adaptive stepsize has been proposed to address this issue (Orvieto et al. [44]), this approach results in slower convergence rates under interpolation. Second, intuitive line-search methods equipped with variance-reduction (VR) fail to converge (Dubois-Taine et al. [14]). So far, no VR methods successfully accelerate these two stepsizes with a convergence guarantee. In this work, we make two contributions: Firstly, we propose two new robust variants of SPS and SLS, called AdaSPS and AdaSLS, which achieve optimal asymptotic rates in both strongly-convex or convex and interpolation or non-interpolation settings, except for the case when we have both strong convexity and non-interpolation. AdaSLS requires no knowledge of problem-dependent parameters, and AdaSPS requires only a lower bound of the optimal function value as input. Secondly, we propose a novel VR method that can use Polyak stepsizes or line-search to achieve acceleration. When it is equipped with AdaSPS or AdaSLS, the resulting algorithms obtain the optimal rate for optimizing convex smooth functions. Finally, numerical experiments on synthetic and real datasets validate our theory and demonstrate the effectiveness and robustness of our algorithms.
Stochastic Gradient Descent (SGD) [46] and its variants [7] are among the most preferred algorithms for training modern machine learning models. These methods only compute stochastic gradients in each iteration, which is often more efficient than computing a full batch gradient. However, the performance of SGD is highly sensitive to the choice of the stepsize. Common strategies use a fixed stepsize schedule, such as keeping it constant or decreasing it over time. Unfortunately, the theoretically optimal schedules are disparate across different function classes [8], and usually depend on problem parameters that are often unavailable, such as the Lipschitz constant of the gradient. As a result, a heavy tuning of the stepsize parameter is required, which is typically expensive in practice.
Instead of fixing the stepsize schedule, adaptive SGD methods adjust the stepsize on the fly [15, 26]. These algorithms often require less hyper-parameter tuning and still enjoy competitive performance in practice. Stochastic Polyak Stepsize (SPS) [34, 6, 45] is one of such recent advances. It has received rapid growing interest due to two factors: (i) the only required parameter is the individual optimal function value which is often available in many machine learning applications and (ii) its adaptivity
$^{*}$ CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
utilizes the local curvature and smoothness information allowing the algorithm to accelerate and converge quickly when training over-parametrized models. Stochastic Line-Search (SLS) [52] is another adaptive stepsize that offers exceptional performance when the interpolation condition holds. In contrast to SPS, the knowledge of the optimal function value is not required for SLS, at the cost of additional function value evaluations per iteration.
An ideal adaptive stepsize should not only require fewer hyper-parameters but should also enjoy robust convergence, in the sense that they can automatically adapt to the optimization setting (interpolation vs. non-interpolation). This will bring great convenience to the users in practice as they no longer need to choose which method to use (or runnin both of them at double the cost). Indeed, in many real-world scenarios, it can be challenging to ascertain whether a model is effectively interpolating the data or not [11]. For instance, the feature dimension of the rcv1 dataset [10] is twice larger than the number of data points. A logistic regression model with as many parameters as the feature dimension may tend to overfit the data points. But the features are actually sparse and the model is not interpolated. Another example is federated learning [23] where millions of clients jointly train a machine learning model on their mobile devices, which usually cannot support huge-scale models. Due to the fact that each client's data is stored locally, it becomes impractical to check the interpolation condition.
While SPS and SLS are promising adaptive methods, they are not robust since both methods cannot converge to the solution when interpolation does not hold. Orvieto et al. [44] address this issue for SPS by applying an artificially decreasing rule and the resulting algorithm DecSPS is able to converge as quickly as SGD with the optimal stepsize schedule. However, the convergence rates of DecSPS in interpolation regimes are much slower than SPS. For SLS, no solution has been proposed.
If the user is certain that the underlying problem is non-interpolated, then applying variance-reduction (VR) techniques can further accelerate the convergence [22, 40, 13, 27, 49, 33]. While gradient descent with Polayak stepsize and line-search perform well in the deterministic settings, there exists no method that successfully adapt these stepsizes in VR methods. Mairal [36] and Schmidt et al. [49] proposed to use stochastic line-search with VR. However, no theoretical guarantee is shown. Indeed, this is a challenging open question as Dubois-Taine et al. [14] provides a counter-example where classical line-search methods fail in the VR setting. As such, it remains unclear whether we can accelerate SGD with step sizes from Polyak and line-search family in non-interpolated settings.
1.1 Main contributions
In this work, we provide solutions to the aforementioned two challenges and contribute new theoretical insights on Polyak stepsize and line-search methods. We summarize our main contributions as follows:
- In Section 3, we propose the first robust adaptive methods that simultaneously achieve the best-known asymptotic rates in both strongly-convex or convex and interpolation or non-interpolation settings except for the case when we have strongly-convexity and non-interpolation. The first method called AdaSPS, a variant of SPS, requires only a lower bound of the optimal function value as input (similar to DecSPS) while AdaSLS, the second method based on SLS, is parameter-free. In the non-interpolated setting, we prove for both algorithms an  $\mathcal{O}(1 / \varepsilon^2)$  convergence rate for convex functions which matches the classical DecSPS and AdaGrad [15] results, whereas SPS and SLS cannot converge in this case. In the interpolated regime, we establish fast  $\mathcal{O}(\log(1 / \varepsilon))$  and  $\mathcal{O}(1 / \varepsilon)$  rates under strong convexity and convexity conditions respectively, without knowledge of any problem-dependent parameters. In contrast, DecSPS converges at the slower  $\mathcal{O}(1 / \varepsilon^2)$  rate and for AdaGrad, the Lipschitz constant is needed to set its stepsize [56].
- In Section 4, we design a new variance-reduction method that is applicable to both Polyak stepsizes or line-search methods. We prove that to reach an  $\varepsilon$ -accuracy, the total number of gradient evaluations required in expectation is  $\widetilde{\mathcal{O}}(n + 1/\varepsilon)$  for convex functions which matches the rate of AdaSVRG [14]. With our newly proposed decreasing probability strategy, the artificially designed multi-stage inner-outer-loop structure is not needed, which makes our methods easier to analyze.
Our novel VR-framework is based on proxy function sequences and can recover the standard VR methods [22] as a special case. We believe that this technique can be of independent interest to the optimization community and may motivate more personalized VR techniques in the future.
Stepsize	Interpolation	Non-interpolation
strongly-convex	convex	required input	strongly-convex	convexa	required input
SPS/SPSmax[34]	O(log(1/ε))	O(1/ε)	fist	ε≥Ω(σ2f,B)	ε≥Ω(σ2f,B)	fist
SLS[52]	O(log(1/ε))	O(1/ε)	None	ε≥Ω(σ2f,B)	ε≥Ω(σ2f,B)	None
DecSPS[44]	O(1/ε2)	O(1/ε2)	lstar	O(1/ε2)	O(1/ε2)	lstar
AdaSPS (this work)	O(log(1/ε))	O(1/ε)	fist	O(1/ε2)	O(1/ε2)	lstar
AdaSLS (this work)	O(log(1/ε))	O(1/ε)	None	O(1/ε2)	O(1/ε2)	None
aThe assumption of bounded iterates is also required except for SPS and SLS.
Table 1: Summary of convergence behaviors of the considered adaptive step sizes for smooth functions. For  $\mathrm{SPS} / \mathrm{SPS}_{\mathrm{max}}$  and SLS in non-interpolation settings,  $\Omega(\cdot)$  indicates the size of the neighborhood that they can converge to. In the other cases, the  $\mathcal{O}(\cdot)$  complexity provides the total number of gradient evaluations required for each algorithm to reach an  $\mathcal{O}(\varepsilon)$  suboptimality. For convex functions, the suboptimality is defined as  $\mathbb{E}[f(\bar{\mathbf{x}}_T) - f^{\star}]$  and for strongly convex functions, the suboptimality is defined as  $\mathbb{E}[||\mathbf{x}_T - \mathbf{x}^{\star}||^2]$ .
Figure 1: Illustration of the robust convergence of AdaSPS and AdaSLS on synthetic data with quadratic loss. SPS and SLS have superior performance on the two interpolated problems but cannot converge when the interpolation condition does not hold. DecSPS suffers from a slow convergence on both interpolated problems. (Repeated 3 times. The solid lines and the shaded area represent the mean and the standard deviation.)
1.2 Related work
Line-search procedures has been successfully applied to accelerate large-scale machine learning training. Following [52], Galli et al. [16] propose to relax the condition of monotone decrease of objective function for training over-parameterized models. Kunstner et al. [30] extends backtracking line-search to a multidimensional variant which provides better diagonal preconditioners. In recent years, adaptive stepsizes from the AdaGrad family have become widespread and are particularly successful when training deep neural networks. Plenty of contributions have been made to analyze variants of AdaGrad for different classes of functions [15, 51, 43, 55, 56], among which Vaswani et al. [53] first propose to use line-search to set the stepsize for AdaGrad to enhance its practical performance. More recently, variance reduction has successfully been applied to AdaGrad stepsize and faster convergence rates have been established for convex and non-convex functions [14, 25].
Another promising direction is the Polyak stepsize (PS) [45] originally designed as a subgradient method for solving non-smooth convex problems. Hazan and Kakade [20] show that PS indeed gives simultaneously the optimal convergence result for a more general class of convex functions. Nedic and Bertsekas [38] propose several variants of PS as incremental subgradient methods and they also discuss the method of dynamic estimation of the optimal function value when it is not known. Recently, more effort has been put into extending deterministic PS to the stochastic setting [47, 6, 42]. However, theoretical guarantees of the algorithms still remain elusive until the emergence of SPS/SPS $_{\mathrm{max}}$  [34]. Subsequently, further improvements and new variants such as DecSPS [44] and SPS with a moving target [17] have been introduced. A more recent line of work interprets stochastic Polyak stepsize as a subsampled Newton Raphson method and interesting algorithms have been designed based on the first-order local expansion [17, 18] as well as the second-order expansion [31]. Wang et al. [54] propose to set the stepsize for SGD with momentum using Polyak stepsize. Abdukhakimov et al. [1] employ more general preconditioning techniques to SPS.
There has been a recent line of work attempting to develop methods that can adapt to both the interpolation setting and the general growth condition beyond strong convexity. Using the iterative halving technique from [5], Cheng and Duchi [11] propose AdaStar-G which gives the desired property if the Lipschitz constant and the diameter of the parameter domain are known. How to remove these requirements is an interesting open question for the future research.
2 Problem setup and background
In this work, we consider solving the finite-sum smooth convex optimization problem:
$$
\min  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \left[ f (\mathbf {x}) = \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} (\mathbf {x}) \right]. \tag {1}
$$
This type of problem appears frequently in the modern machine learning applications [19], where each  $f_{i}(\mathbf{x})$  represents the loss of a model on the  $i$ -th data point parametrized by the parameter  $\mathbf{x}$ . Stochastic Gradient Descent (SGD) [46] is one of the most popular methods for solving the problem (1). At each iteration, SGD takes the form:
$$
\mathbf {x} _ {t + 1} = \mathbf {x} _ {t} - \eta_ {t} \nabla f _ {i _ {t}} \left(\mathbf {x} _ {t}\right), \tag {2}
$$
where  $\eta_{t}$  is the stepsize parameter,  $i_t\subseteq [n]$  is a random set of size  $B$  sampled independently at each iteration  $t$  and  $\nabla f_{i_t}(\mathbf{x}) = \frac{1}{B}\sum_{i\in i_t}\nabla f_i(\mathbf{x})$  is the minibatch gradient.
Throughout the paper, we assume that there exists a non-empty set of optimal points  $\mathcal{X}^{\star}\subset \mathbb{R}^{d}$ , and we use  $f^{\star}$  to denote the optimal value of  $f$  at a point  $\mathbf{x}^{\star}\in \mathcal{X}^{\star}$ . We use  $f_{i_t}^{\star}$  to denote the infimum of minibatch function  $f_{i_t}(x)$ , i.e.  $f_{i_t}^{\star} = \inf_{\mathbf{x}\in \mathbb{R}^d}\frac{1}{B}\sum_{i\in i_t}f_i(\mathbf{x})$ . We assume that all the individual functions  $\{f_i(\mathbf{x})\}$  are  $L$ -smooth. Finally, we denote the optimal objective difference, first introduced in [34], by  $\sigma_{f,B}^{2} = f^{\star} - \mathbb{E}_{i_t}[f_{i_t}^{\star}]$ . The definitions for the interpolation condition can be defined and studied in various ways [48, 9, 4, 11]. Here, we adopt the notion from [34]. The problem (1) is said to be interpolated if  $\sigma_{f,1}^{2} = 0$ , which implies that  $\sigma_{f,B}^{2} = 0$  for all  $B\leq n$  since  $\sigma_{f,B}^{2}$  is non-increasing w.r.t  $B$ . Note interpolation implies that the global minimizer of  $f$  is also a minimizer of each individual function  $f_{i}$ .
2.2 SGD with stochastic polyak stepsize
Loizou et al. [34] propose to set the stepsize  $\eta_{t}$  as:  $\eta_{t} = 2\frac{f_{i_{t}}(\mathbf{x}_{t}) - f_{i_{t}}^{\star}}{||\nabla f_{i_{t}}(\mathbf{x}_{t})||^{2}}$ , which is well known as the Stochastic Polyak stepsize (SPS). In addition to SPS, they also propose a bounded variant  $\mathrm{SPS}_{\mathrm{max}}$  which has the form  $\eta_{t} = \min \left\{2\frac{f_{i_{t}}(\mathbf{x}_{t}) - f_{i_{t}}^{\star}}{||\nabla f_{i_{t}}(\mathbf{x}_{t})||^{2}},\gamma_{b}\right\}$  where  $\gamma_{b} > 0$ . Both algorithms require the input of the exact  $f_{i_t}^\star$  which is often unavailable when the batch size  $B > 1$  or when the interpolation condition does not hold. Orvieto et al. [44] removes the requirement for  $f_{i_t}^\star$  and propose to set  $\eta_{t}$  as:  $\eta_{t} = \frac{1}{\sqrt{t + 1}}\min \left\{\frac{f_{i_{t}}(\mathbf{x}_{t}) - \ell_{i_{t}}^{\star}}{||\nabla f_{i_{t}}(\mathbf{x}_{t})||^{2}},\sqrt{t}\eta_{t - 1}\right\}$  for  $t\geq 1$  (DecSPS), where  $\eta_0 > 0$  is a constant and  $\ell_{i_t}^{\star}$  is an input lower bound such that  $\ell_{i_t}^{\star}\leq f_{i_t}^{\star}$ . In contrast to the exact optimal function value, a lower bound  $\ell_{i_t}^{\star}$  is often available in practice, in particular for machine learning problems when the individual loss functions are non-negative. We henceforth denote the estimation error by:
$$
\operatorname {e r r} _ {f, B} ^ {2} := \mathbb {E} _ {i _ {t}} \left[ f _ {i _ {t}} ^ {\star} - \ell_ {i _ {t}} ^ {\star} \right]. \tag {3}
$$
For convex smooth functions, SPS achieves a fast convergence up to a neighborhood of size  $\Omega(\sigma_{f,B}^2)$  and its variant  $\mathrm{SPS}_{\mathrm{max}}$  converges up to  $\Omega(\sigma_{f,B}^2\gamma_b / \alpha)$  where  $\alpha = \min\{\frac{1}{L},\gamma_b\}$ . Note that the size of the neighborhood cannot be further reduced by choosing an appropriate  $\gamma_b$ . In contrast, DecSPS converges at the rate of  $\mathcal{O}(1/\sqrt{T})$  which matches the standard result for SGD with decreasing stepsize. However, the strictly decreasing  $\Theta(1/\sqrt{t})$  stepsize schedule hurts its performance in interpolated settings. For example, DecSPS has a much slower  $\mathcal{O}(1/\sqrt{T})$  convergence rate compared with the fast  $\mathcal{O}(\exp(-T\mu/L))$  rate of SPS when optimizing strongly-convex objectives. Therefore, both algorithms do not have the robust convergence property (achieving fast convergence guarantees in both interpolated and non-interpolated regimes) and we aim to fill this gap. See Figure 1 for a detailed illustration of the non-robustness of SPS and DecSPS.
3 Adaptive SGD with polyak stepsize and line-search
In this section, we introduce and analyze two adaptive algorithms to solve problem (1).
3.1 Proposed methods
AdaSPS. Our first stepsize is defined as the following:
$$
\eta_ {t} = \min  \left\{\frac {f _ {i _ {t}} (\mathbf {x} _ {t}) - \ell_ {i _ {t}} ^ {\star}}{c _ {p} | | \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) | | ^ {2}} \frac {1}{\sqrt {\sum_ {s = 0} ^ {t} f _ {i _ {s}} (\mathbf {x} _ {s}) - \ell_ {i _ {s}} ^ {\star}}}, \eta_ {t - 1} \right\}, \quad \mathrm {w i t h} \eta_ {- 1} = + \infty , \quad \mathrm {(A d a S P S)}
$$
where  $\ell_{i_t}^{\star}$  is an input parameter that must satisfy  $\ell_{i_t}^{\star} \leq f_{i_t}^{\star}$  and  $c_p > 0$  is an input constant to adjust the magnitude of the stepsize (we discuss suggested choices in Section 5).
AdaSPS can be seen as an extension of DecSPS. However, unlike the strict  $\Theta(1/\sqrt{t})$  decreasing rule applied in DecSPS, AdaSPS accumulates the function value difference during the optimization process which enables it to dynamically adapt to the underlying unknown interpolation settings.
AdaSLS. We provide another stepsize that can be applied even when a lower bound estimation is unavailable. The method is based on line-search and thus is completely parameter-free, but requires additional function value evaluations in each iteration:
$$
\eta_ {t} = \min  \left\{\frac {\gamma_ {t}}{c _ {l} \sqrt {\sum_ {s = 0} ^ {t} \gamma_ {s} | | \nabla f _ {i _ {s}} (\mathbf {x} _ {s}) | | ^ {2}}}, \eta_ {t - 1} \right\}, \qquad \text {w i t h} \eta_ {- 1} = + \infty , \qquad \mathrm {(A d a S L S)}
$$
where  $c_{l} > 0$  is an input constant, and the scale  $\gamma_{t}$  is obtained via standard Armijo backtracking line-search (see Algorithm 4 for further implementation details in the Appendix D) such that the following conditions are satisfied:
$$
\left. f _ {i _ {t}} \left(\mathbf {x} _ {t} - \gamma_ {t} \nabla f _ {i _ {t}} (\mathbf {x} _ {t})\right) \leq f _ {i _ {t}} (\mathbf {x} _ {t}) - \rho \gamma_ {t} \| \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) \| ^ {2} \quad \text {a n d} \quad \gamma_ {t} \leq \gamma_ {\max }, 0 <   \rho <   1, \right. \tag {4}
$$
for line-search parameters  $\gamma_{\mathrm{max}}$  and  $\rho$ . By setting the decreasing factor  $\beta \geq \frac{1}{2}$  defined in Algorithm 4, one can show that  $\gamma_t \geq \min \left( \frac{1 - \rho}{L}, \gamma_{\mathrm{max}} \right)$ . We give a formal proof in Lemma 16 in Appendix A.2.
Discussion. Our adaptation mechanism in AdaSPS/AdaSLS is reminiscent of AdaGrad type methods, in particular to AdaGrad-Norm, the scalar version of AdaGrad, that aggregates the gradient norm in the denominator and takes the form  $\eta_{t} = \frac{c_{g}}{\sqrt{\sum_{s=0}^{t}||\nabla f_{i_{s}}(\mathbf{x}_{s})||^{2} + b_{0}^{2}}}$  where  $c_{g} > 0$  and  $b_{0}^{2} \geq 0$ .
The primary distinction between AdaSPS and AdaSLS compared to AdaGrad-Norm is the inclusion of an additional component that captures the curvature information at each step, and not using squared gradient norms in AdaSPS. In contrast to the strict decreasing behavior of AdaGrad-Norm, AdaSPS and AdaSLS can automatically mimic a constant stepsize when navigating a flatter region.
Vaswani et al. [53] suggest using line-search to set the stepsize for AdaGrad-Norm which takes the form  $\eta_{t} = \frac{\gamma_{t}}{\sqrt{\sum_{s=0}^{t}||\nabla f_{i_{s}}(\mathbf{x}_{s})||^{2}}}$  where  $\gamma_{t} \leq \gamma_{t-1}$  is required for solving non-interpolated convex problems. While this stepsize is similar to AdaSLS, the scaling of the denominator gives a suboptimal convergence rate as we demonstrate in the following section.
3.2 Convergence rates
In this section, we present the convergence results for AdaSPS and AdaSLS. We list the helpful lemmas in Appendix A. The proofs can be found in Appendix B.
General convex. We denote  $\mathcal{X}$  to be a convex compact set with diameter  $D$  such that there exists a solution  $\mathbf{x}^{\star} \in \mathcal{X}$  and  $\sup_{\mathbf{x}, \mathbf{y} \in \mathcal{X}} \| \mathbf{x} - \mathbf{y} \|^{2} \leq D^{2}$ . We let  $\Pi_{\mathcal{X}}$  denote the Euclidean projection onto  $\mathcal{X}$ . For general convex stochastic optimization, it seems inevitable that adaptive methods require the bounded iterates assumption or an additional projection step to prove convergence due to the lack of knowledge of problem-dependent parameters [12, 15]. Here, we employ the latter solution by running projected stochastic gradient descent (PSGD):
$$
\mathbf {x} _ {t + 1} = \Pi_ {\mathcal {X}} \left(\mathbf {x} _ {t} - \eta_ {t} \nabla f _ {i _ {t}} \left(\mathbf {x} _ {t}\right)\right). \tag {5}
$$
Theorem 1 (General convex). Assume that  $f$  is convex, each  $f_{i}$  is  $L$ -smooth and  $\mathcal{X}$  is a convex compact feasible set with diameter  $D$ , PSGD with AdaSPS or AdaSLS converges as:
$$
\left(A d a S P S\right): \mathbb {E} \left[ f \left(\bar {\mathbf {x}} _ {T}\right) - f ^ {\star} \right] \leq \frac {\tau_ {p} ^ {2}}{T} + \frac {\tau_ {p} \sqrt {\sigma_ {f , B} ^ {2} + \operatorname {e r r} _ {f , B} ^ {2}}}{\sqrt {T}}, \tag {6}
$$
$$
(A d a S L S): \mathbb {E} [ f (\bar {\mathbf {x}} _ {T}) - f ^ {\star} ] \leq \frac {\tau_ {l} ^ {2}}{T} + \frac {\tau_ {l} \sigma_ {f , B}}{\sqrt {T}},
$$
where  $\bar{\mathbf{x}}_T = \frac{1}{T}\sum_{t = 0}^{T - 1}\mathbf{x}_t$ ,  $\tau_p = (2c_pLD^2 +\frac{1}{c_p})$  and  $\tau_l = \max \left\{\frac{L}{(1 - \rho)\sqrt{\rho}},\frac{1}{\gamma_{\mathrm{max}}\sqrt{\rho}}\right\} c_lD^2 +\frac{1}{c_l\sqrt{\rho}}$ .
As a consequence of Theorem 1, if  $\mathrm{err}_{f,B}^2 = \sigma_{f,B}^2 = 0$ , then PSGD with AdaSPS or AdaSLS converges as  $\mathcal{O}\left(\frac{1}{T}\right)$ . Suppose  $\gamma_{\max}$  is sufficiently large, then picking  $c_p^\star = \frac{1}{\sqrt{2LD^2}}$  and  $c_l^\star = \frac{\sqrt{1 - \rho}}{\sqrt{LD^2}}$  gives a  $\mathcal{O}\left(\frac{LD^2}{T}\right)$  rate under the interpolation condition, which is slightly worse than  $\frac{L||\mathbf{x}_0 - \mathbf{x}^\star||^2}{T}$  obtained by SPS and SLS but is better than  $\mathcal{O}\left(\frac{LD^2}{\sqrt{T}}\right)$  obtained by DecSPS. If otherwise  $\sigma_{f,B}^2 > 0$ , then AdaSPS, AdaSLS, and DecSPS converge as  $\mathcal{O}(1 / \sqrt{T})$  which matches the rate of Vanilla SGD with decreasing stepsize. Finally, AdaGrad-Norm gives a similar rate in both cases while AdaGrad-Norm with line-search [53] shows a suboptimal rate of  $\mathcal{O}\left(\frac{L^3D^4}{T} + \frac{D^2L^{3/2}\sigma}{\sqrt{T}}\right)$ . It is worth noting that SPS, DecSPS and SLS require an additional assumption on individual convexity.