Title: Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reduction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Convex Optimization

SGD

Adaptive Methods

Variance Reduction

Polyak Stepsize

Line-Search
Abstract: The recently proposed stochastic Polyak stepsize (SPS) and stochastic line-search (SLS) for SGD have shown remarkable effectiveness when training over-parameterized models. However, two issues remain unsolved in this line of work. 

First, in non-interpolation settings, both algorithms only guarantee convergence to a neighborhood of a solution which may result in a worse output than the initial guess. While artificially decreasing the adaptive stepsize has been proposed to address this issue (Orvieto et al.), this approach results in slower convergence rates under interpolation. Second, intuitive line-search methods equipped with variance-reduction (VR) fail to converge (Dubois-Taine et al.). So far, no VR methods successfully accelerate these two stepsizes with a convergence guarantee.

In this work, we make two contributions:
Firstly, we propose two new robust variants of SPS and SLS, called AdaSPS and AdaSLS, which achieve optimal asymptotic rates in both strongly-convex or convex and interpolation or non-interpolation settings, except for the case when we have both strong convexity and non-interpolation. AdaSLS requires no knowledge of problem-dependent parameters, and AdaSPS requires only a lower bound of the optimal function value as input. Secondly, we propose a novel VR method that can use Polyak stepsizes or line-search to achieve acceleration. When it is equipped with AdaSPS or AdaSLS, the resulting algorithms obtain the optimal rate
for optimizing convex smooth functions. Finally, numerical experiments on synthetic and real datasets validate our theory and demonstrate the effectiveness and robustness of our algorithms.

Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reduction

Abstract

The recently proposed stochastic Polyak stepsize (SPS) and stochastic linesearch (SLS) for SGD have shown remarkable effectiveness when training overparameterized models. However, two issues remain unsolved in this line of work. First, in non-interpolation settings, both algorithms only guarantee convergence to a neighborhood of a solution which may result in a worse output than the initial guess. While artificially decreasing the adaptive stepsize has been proposed to address this issue (Orvieto et al. [44]), this approach results in slower convergence rates under interpolation. Second, intuitive line-search methods equipped with variance-reduction (VR) fail to converge (Dubois-Taine et al. [14]). So far, no VR methods successfully accelerate these two stepsizes with a convergence guarantee. In this work, we make two contributions: Firstly, we propose two new robust variants of SPS and SLS, called AdaSPS and AdaSLS, which achieve optimal asymptotic rates in both strongly-convex or convex and interpolation or noninterpolation settings, except for the case when we have both strong convexity and non-interpolation. AdaSLS requires no knowledge of problem-dependent parameters, and AdaSPS requires only a lower bound of the optimal function value as input. Secondly, we propose a novel VR method that can use Polyak stepsizes or line-search to achieve acceleration. When it is equipped with AdaSPS or AdaSLS, the resulting algorithms obtain the optimal rate for optimizing convex smooth functions. Finally, numerical experiments on synthetic and real datasets validate our theory and demonstrate the effectiveness and robustness of our algorithms.

1 Introduction

Stochastic Gradient Descent (SGD) [46] and its variants [7] are among the most preferred algorithms for training modern machine learning models. These methods only compute stochastic gradients in each iteration, which is often more efficient than computing a full batch gradient. However, the performance of SGD is highly sensitive to the choice of the stepsize. Common strategies use a fixed stepsize schedule, such as keeping it constant or decreasing it over time. Unfortunately, the theoretically optimal schedules are disparate across different function classes [8], and usually depend on problem parameters that are often unavailable, such as the Lipschitz constant of the gradient. As a result, a heavy tuning of the stepsize parameter is required, which is typically expensive in practice.

Instead of fixing the stepsize schedule, adaptive SGD methods adjust the stepsize on the fly [15, 26]. These algorithms often require less hyper-parameter tuning and still enjoy competitive performance in practice. Stochastic Polyak Stepsize (SPS) [34, 6, 45] is one of such recent advances. It has received rapid growing interest due to two factors: (i) the only required parameter is the individual optimal function value which is often available in many machine learning applications and (ii) its adaptivity

CISPA ∗

utilizes the local curvature and smoothness information allowing the algorithm to accelerate and converge quickly when training over-parametrized models. Stochastic Line-Search (SLS) [52] is another adaptive stepsize that offers exceptional performance when the interpolation condition holds. In contrast to SPS, the knowledge of the optimal function value is not required for SLS, at the cost of additional function value evaluations per iteration.

An ideal adaptive stepsize should not only require fewer hyper-parameters but should also enjoy robust convergence, in the sense that they can automatically adapt to the optimization setting (interpolation vs. non-interpolation). This will bring great convenience to the users in practice as they no longer need to choose which method to use (or runinning both of them at double the cost). Indeed, in many real-world scenarios, it can be challenging to ascertain whether a model is effectively interpolating the data or not [11]. For instance, the feature dimension of the rcv1 dataset [10] is twice larger than the number of data points. A logistic regression model with as many parameters as the feature dimension may tend to overfit the data points. But the features are actually sparse and the model is not interpolated. Another example is federated learning [23] where millions of clients jointly train a machine learning model on their mobile devices, which usually cannot support huge-scale models. Due to the fact that each client's data is stored locally, it becomes impractical to check the interpolation condition.

While SPS and SLS are promising adaptive methods, they are not robust since both methods cannot converge to the solution when interpolation does not hold. Orvieto et al. [44] address this issue for SPS by applying an artificially decreasing rule and the resulting algorithm DecSPS is able to converge as quickly as SGD with the optimal stepsize schedule. However, the convergence rates of DecSPS in interpolation regimes are much slower than SPS. For SLS, no solution has been proposed.

If the user is certain that the underlying problem is non-interpolated, then applying variancereduction (VR) techniques can further accelerate the convergence [22, 40, 13, 27, 49, 33]. While gradient descent with Polayak stepsize and line-search perform well in the deterministic settings, there exists no method that successfully adapt these stepsizes in VR methods. Mairal [36] and Schmidt et al. [49] proposed to use stochastic line-search with VR. However, no theoretical guarantee is shown. Indeed, this is a challenging open question as Dubois-Taine et al. [14] provides a counter-example where classical line-search methods fail in the VR setting. As such, it remains unclear whether we can accelerate SGD with stepsizes from Polyak and line-search family in non-interpolated settings.

1.1 Main contributions

In this work, we provide solutions to the aforementioned two challenges and contribute new theoretical insights on Polyak stepsize and line-search methods. We summarize our main contributions as follows:

In Section 3, we propose the first robust adaptive methods that simultaneously achieve the bestknown asymptotic rates in both strongly-convex or convex and interpolation or non-interpolation settings except for the case when we have strongly-convexity and non-interpolation. The first method called AdaSPS, a variant of SPS, requires only a lower bound of the optimal function value as input (similar to DecSPS) while AdaSLS, the second method based on SLS, is parameter-free. In the non-interpolated setting, we prove for both algorithms an O (1 /ε 2 ) convergence rate for convex functions which matches the classical DecSPS and AdaGrad [15] results, whereas SPS and SLS cannot converge in this case. In the interpolated regime, we establish fast O (log(1 /ε )) and O (1 /ε ) rates under strong convexity and convexity conditions respectively, without knowledge of any problem-dependent parameters. In contrast, DecSPS converges at the slower O (1 /ε 2 ) rate and for AdaGrad, the Lipschitz constant is needed to set its stepsize [56].

In Section 4, we design a new variance-reduction method that is applicaple to both Polyak stepsizes or line-search methods. We prove that to reach an ε -accuracy, the total number of gradient evaluations required in expectation is ˜ O ( n +1 /ε ) for convex functions which matches the rate of AdaSVRG [14]. With our newly proposed decreasing probability strategy, the artificially designed multi-stage inner-outer-loop structure is not needed, which makes our methods easier to analyze.

Our novel VR-framework is based on proxy function sequences and can recover the standard VR methods [22] as a special case. We believe that this technique can be of independent interest to the optimization community and may motivate more personalized VR techniques in the future.

a The assumption of bounded iterates is also required except for SPS and SLS.

0

1000

2000

# mini-batch gradient evaluations

10

8

10

5

10

2

10 1

10 4

10 7

f(x)-

f

strongly-convex+interpolation

0

2000

4000

# mini-batch gradient evaluations

10 2

10 4

10 6

f(x)-

f

strongly-convex+non-interpolation

0

2500

5000

7500 10000

# mini-batch gradient evaluations

10 0

10 2

10 4

10 6

f(x)-

f

convex+interpolation

0

10000

20000

# mini-batch gradient evaluations

10 1

10 2

10 3

10 4

10 5

10 6

f(x)-

f

convex+non-interpolation

DecSPS

SLS

AdaSLS

SPS

1.2 Related work

Line-search procedures has been successfully applied to accelerate large-scale machine learning training. Following [52], Galli et al. [16] propose to relax the condition of monotone decrease of objective function for training over-parameterized models. Kunstner et al. [30] extends backtracking line-search to a multidimensional variant which provides better diagonal preconditioners. In recent years, adaptive stepsizes from the AdaGrad family have become widespread and are particularly successful when training deep neural networks. Plenty of contributions have been made to analyze variants of AdaGrad for different classes of functions [15, 51, 43, 55, 56], among which Vaswani et al. [53] first propose to use line-search to set the stepsize for AdaGrad to enhance its practical performance. More recently, variance reduction has successfully been applied to AdaGrad stepsize and faster convergence rates have been established for convex and non-convex functions [14, 25].

Another promising direction is the Polyak stepsize (PS) [45] originally designed as a subgradient method for solving non-smooth convex problems. Hazan and Kakade [20] show that PS indeed gives simultaneously the optimal convergence result for a more general class of convex functions. Nedi´ c and Bertsekas [38] propose several variants of PS as incremental subgradient methods and they also discuss the method of dynamic estimation of the optimal function value when it is not known. Recently, more effort has been put into extending deterministic PS to the stochastic setting [47, 6, 42]. However, theoretical guarantees of the algorithms still remain elusive until the emergence of SPS/SPS max [34]. Subsequently, further improvements and new variants such as DecSPS [44] and SPS with a moving target [17] have been introduced. A more recent line of work interprets stochastic Polyak stepsize as a subsampled Newton Raphson method and interesting algorithms have been designed based on the first-order local expansion [17, 18] as well as the second-order expansion [31]. Wang et al. [54] propose to set the stepsize for SGD with momentum using Polyak stepsize. Abdukhakimov et al. [1] employ more general preconditioning techniques to SPS.

There has been a recent line of work attempting to develop methods that can adapt to both the interpolation setting and the general growth condition beyond strong convexity. Using the iterative halving technique from [5], Cheng and Duchi [11] propose AdaStar-G which gives the desired property if the Lipschitz constant and the diameter of the parameter domain are known. How to remove these requirements is an interesting open question for the future research.

2 Problem setup and background

2.1 Notations

In this work, we consider solving the finite-sum smooth convex optimization problem:

This type of problem appears frequently in the modern machine learning applications [19], where each f i ( x ) represents the loss of a model on the i -th data point parametrized by the parameter x . Stochastic Gradient Descent (SGD) [46] is one of the most popular methods for solving the problem (1). At each iteration, SGD takes the form:

where η t is the stepsize parameter, i t ⊆ [ n ] is a random set of size B sampled independently at each iteration t and ∇ f i t ( x ) = 1 B ∑ i ∈ i t ∇ f i ( x ) is the minibatch gradient.

Throughout the paper, we assume that there exists a non-empty set of optimal points X ⋆ ⊂ R d , and we use f ⋆ to denote the optimal value of f at a point x ⋆ ∈ X ⋆ . We use f ⋆ i t to denote the infimum of minibatch function f i t ( x ) , i.e. f ⋆ i t = inf x ∈ R d 1 B ∑ i ∈ i t f i ( x ) . We assume that all the individual functions { f i ( x ) } are L -smooth. Finally, we denote the optimal objective difference, first introduced in [34], by σ 2 f,B = f ⋆ -E i t [ f ⋆ i t ] . The definitions for the interpolation condition can be defined and studied in various ways [48, 9, 4, 11]. Here, we adopt the notion from [34]. The problem (1) is said to be interpolated if σ 2 f, 1 = 0 , which implies that σ 2 f,B = 0 for all B ≤ n since σ 2 f,B is non-increasing w.r.t B . Note interpolation implies that the global minimizer of f is also a minimizer of each individual function f i .

2.2 SGD with stochastic polyak stepsize

Loizou et al. [34] propose to set the stepsize η t as: η t = 2 f i t ( x t ) -f ⋆ i t ||∇ f i t ( x t ) || 2 , which is well known as the Stochastic Polyak stepsize (SPS). In addition to SPS, they also propose a bounded variant SPS max which has the form η t = min { 2 f i t ( x t ) -f ⋆ i t ||∇ f i t ( x t ) || 2 , γ b } where γ b > 0 . Both algorithms require the input of the exact f ⋆ i t which is often unavailable when the batch size B > 1 or when the interpolation condition does not hold. Orvieto et al. [44] removes the requirement for f ⋆ i t and propose to set η t as: η t = 1 √ t +1 min { f i t ( x t ) -ℓ ⋆ i t ||∇ f i t ( x t ) || 2 , √ tη t -1 } for t ≥ 1 (DecSPS), where η 0 > 0 is a constant and ℓ ⋆ i t is an input lower bound such that ℓ ⋆ i t ≤ f ⋆ i t . In contrast to the exact optimal function value, a lower bound ℓ ⋆ i t is often available in practice, in particular for machine learning problems when the individual loss functions are non-negative. We henceforth denote the estimation error by:

For convex smooth functions, SPS achieves a fast convergence up to a neighborhood of size Ω( σ 2 f,B ) and its variant SPS max converges up to Ω( σ 2 f,B γ b /α ) where α = min { 1 L , γ b } . Note that the size of the neighborhood cannot be further reduced by choosing an appropriate γ b . In contrast, DecSPS converges at the rate of O (1 / √ T ) which matches the standard result for SGD with decreasing stepsize. However, the strictly decreasing Θ(1 / √ t ) stepsize schedule hurts its performance in interpolated settings. For example, DecSPS has a much slower O (1 / √ T ) convergence rate compared with the fast O (exp( -Tµ/L )) rate of SPS when optimizing strongly-convex objectives. Therefore, both algorithms do not have the robust convergence property (achieving fast convergence guarantees in both interpolated and non-interpolated regimes) and we aim to fill this gap. See Figure 1 for a detailed illustration of the non-robustness of SPS and DecSPS.

3 Adaptive SGD with polyak stepsize and line-search

In this section, we introduce and analyze two adaptive algorithms to solve problem (1).

3.1 Proposed methods

AdaSPS. Our first stepsize is defined as the following:

where ℓ ⋆ i t is an input parameter that must satisfy ℓ ⋆ i t ≤ f ⋆ i t and c p > 0 is an input constant to adjust the magnitude of the stepsize (we discuss suggested choices in Section 5).

AdaSPS can be seen as an extension of DecSPS. However, unlike the strict Θ(1 / √ t ) decreasing rule applied in DecSPS, AdaSPS accumulates the function value difference during the optimization process which enables it to dynamically adapt to the underlying unknown interpolation settings.

AdaSLS. We provide another stepsize that can be applied even when a lower bound estimation is unavailable. The method is based on line-search and thus is completely parameter-free, but requires additional function value evaluations in each iteration:

where c l > 0 is an input constant, and the scale γ t is obtained via stardard Armijo backtracking line-search (see Algorithm 4 for further implementation details in the Appendix D) such that the following conditions are satisfied:

for line-search parameters γ max and ρ . By setting the decreasing factor β ≥ 1 2 defined in Algorithm 4, one can show that γ t ≥ min( 1 -ρ L , γ max ) . We give a formal proof in Lemma 16 in Appendix A.2.

Discussion. Our adaptation mechanism in AdaSPS/AdaSLS is reminiscent of AdaGrad type methods, in particular to AdaGrad-Norm, the scalar version of AdaGrad, that aggregates the gradient norm in the denominator and takes the form η t = c g √ ∑ t s =0 ||∇ f i s ( x s ) || 2 + b 2 0 where c g > 0 and b 2 0 ≥ 0 .

The primary distinction between AdaSPS and AdaSLS compared to AdaGrad-Norm is the inclusion of an additional component that captures the curvature information at each step, and not using squared gradient norms in AdaSPS. In contrast to the strict decreasing behavior of AdaGrad-Norm, AdaSPS and AdaSLS can automatically mimic a constant stepsize when navigating a flatter region.

Vaswani et al. [53] suggest using line-search to set the stepsize for AdaGrad-Norm which takes the form η t = γ t √ ∑ t s =0 ||∇ f i s ( x s ) || 2 where γ t ≤ γ t -1 is required for solving non-interpolated convex problems. While this stepsize is similar to AdaSLS, the scaling of the denominator gives a suboptimal convergence rate as we demonstrate in the following section.

3.2 Convergence rates

In this section, we present the convergence results for AdaSPS and AdaSLS. We list the helpful lemmas in Appendix A. The proofs can be found in Appendix B.

General convex. We denote X to be a convex compact set with diameter D such that there exists a solution x ⋆ ∈ X and sup x , y ∈X || x -y || 2 ≤ D 2 . We let Π X denote the Euclidean projection onto X . For general convex stochastic optimization, it seems inevitable that adaptive methods require the bounded iterates assumption or an additional projection step to prove convergence due to the lack of knowledge of problem-dependent parameters [12, 15]. Here, we employ the latter solution by running projected stochastic gradient descent (PSGD):

Theorem 1 (General convex) . Assume that f is convex, each f i is L -smooth and X is a convex compact feasible set with diameter D , PSGD with AdaSPS or AdaSLS converges as:

As a consequence of Theorem 1, if err 2 f,B = σ 2 f,B = 0 , then PSGD with AdaSPS or AdaSLS converges as O ( 1 T ) . Suppose γ max is sufficiently large, then picking c ⋆ p = 1 √ 2 LD 2 and c ⋆ l = √ 1 -ρ √ LD 2 gives a O ( LD 2 T ) rate under the interpolation condition, which is slightly worse than L || x 0 -x ⋆ || 2 T obtained by SPS and SLS but is better than O ( LD 2 √ T ) obtained by DecSPS. If otherwise σ 2 f,B > 0 , then AdaSPS, AdaSLS, and DecSPS converge as O (1 / √ T ) which matches the rate of Vanilla SGD with decreasing stepsize. Finally, AdaGrad-Norm gives a similar rate in both cases while AdaGradNorm with line-search [53] shows a suboptimal rate of O ( L 3 D 4 T + D 2 L 3 / 2 σ √ T ) . It is worth noting that SPS, DecSPS and SLS require an additional assumption on individual convexity.

Theorem 2 (Individual convex+interpolation) . Assume that f is convex, each f i is convex and L -smooth, and that err 2 f,B = σ 2 f,B = 0 , by setting c p = c scale p √ f i 0 ( x 0 ) -f ⋆ i 0 and c l = c scale l ρ √ γ 0 ||∇ f i 0 ( x 0 ) || 2 with constants c scale p ≥ 1 and c scale l ≥ 1 , then for any T ≥ 1 , SGD (no projection) with AdaSPS or AdaSLS converges as:

and

The result implies that the bounded iterates assumption is not needed if we have both individual convexity and interpolation by picking c p and c l to satisfy certain conditions that do not depend on unknown parameters. To our knowledge, no such result exists for stepsizes from the AdaGrad family. It is worth noting that the min operator defined in AdaSPS or AdaSLS is not necessary in the proof.

Remark 3. We note that for non-interpolated problems, AdaSPS only requires the knowledge of ℓ ⋆ i t while the exact f ⋆ i t is needed under the interpolation condition. We argue that in many standard machine learning problems, simply picking zero will suffice. For instance, f ⋆ i t = 0 for over-parameterized logistic regression and after adding a regularizer, ℓ ⋆ i t = 0 .

Strongly convex. We now present two algorithmic behaviors of AdaSPS and AdaSLS for strongly convex functions. In particular, We show that 1) the projection step can be removed as shown in DecSPS, and 2) if the interpolation condition holds, the min operator is not needed and the asymptotic linear convergence rate is preserved. The full statement of Lemma 4 can be found in Appendix B.2.

Lemma 4 (Bounded iterates) . Let each f i be µ -strongly convex and L -smooth. For any t = 0 , . . . , T , the iterates of SGD with AdaSPS or AdaSLS satisfy: || x t -x ⋆ || 2 ≤ D max , for a constant D max specified in the appendix in Equation (B.16) .

Corollary 5 (Individual strongly convex) . Assume each f i is µ -strongly convex and L -smooth, Theorem 1 holds with PSGD and D replaced by SGD and D max defined in Lemma 4.

Although it has not been formally demonstrated that AdaGrad/AdaGrad-Norm can relax the assumption on bounded iterates for strongly convex functions, we believe that with a similar proof technique, this property still holds for AdaGrad/AdaGrad-Norm.

We next show that AdaSPS and AdaSLS achieve linear convergence under the interpolation condition.

Theorem 6 (Strongly convex + individual convex + interpolation) . Consider SGD with AdaSPS (AdaSPS) or AdaSLS (AdaSLS) stepsize. Suppose that each f i is convex and L -smooth, f is µ -strongly convex and that σ 2 f,B = err 2 f,B = 0 . If we let c p = c scale p √ f i 0 ( x 0 ) -f ⋆ i 0 and c l = c scale l ρ √ γ 0 ||∇ f i 0 ( x 0 ) || 2 with constants c scale p ≥ 1 and c scale l ≥ 1 , then AdaSPS or AdaSLS converges as:

and

The proof of Theorem 6 is presented in Appendix B.3. We now compare the above results with the other stepsizes. Under the same settings, DecSPS has a slower O (1 / √ T ) rate due to the usage of Θ(1 / √ t ) decay stepsize. While AdaGrad-Norm does have a linear acceleration phase when the accumulator grows large, to avoid an O (1 /ε ) slow down, the parameters of AdaGrad-Norm have to satisfy c g < b 0 /L , which requires the knowledge of Lipschitz constant [56]. Instead, the conditions on c p and c l for AdaSPS and AdaSLS only depend on the function value and gradient norm at x 0 which can be computed at the first iteration. SPS, SLS, and Vannilia-SGD with constant stepsize achieve faster linear convergence rate of order O ( exp( -µ L T ) ) . It is worth noting that Vannila-SGD can further remove the individual convexity assumption.

√

Discussion. In non-interpolation regimes, AdaSPS and AdaSLS only ensure a slower O (1 / T ) convergence rate compared with O (1 /T ) rate achieved by vanilla SGD with Θ(1 /t ) decay stepsize when optimizing strongly-convex functions [7]. To our knowledge, no parameter-free adaptive stepsize exists that achieves such a fast rate under the same assumptions. Therefore, developing an adaptive algorithm that can adapt to both convex and strongly-convex functions would be a significant further contribution.

4 AdaSPS and AdaSLS with variance-reduction

Combining variance-reduction (VR) with adaptive Polyak-stepsize and line-search to achieve acceleration is a natural idea that has been explored in the last decade [49, 36]. However, it remains an open challenge as no theoretical guarantee has been proven yet. Indeed, Dubois-Taine et al. [14] provide a counter-example for intuitive line-search methods. In Appendix E we provide counter-examples of the classical SPS and its variants. The reason behind the failure is the biased curvature information provided by f i t that prevents global convergence. In this section, we introduce a novel framework to address this issue. Since there exists many variance-reduced stochastic gradient estimators, we focus on the classical SVRG estimator in this section, and our framework also applies to other estimators such as SARAH [40].

4.1 Algorithm design: achieving variance-reduction without interpolation

It is known that adaptive methods such as SPS or SLS converge linearly on problems where the interpolation condition holds, i.e. f ( x ) with σ f,B = 0 .

For problems that do not satisfy the interpolation condition, our approach is to transition the problem to an equivalent one that satisfies the interpolation condition. One such transformation is to shift each individual function by the gradient of f i ( x ) at x ⋆ , i.e. F i ( x ) = f i ( x ) -x T ∇ f i ( x ⋆ ) . In this case f ( x ) can be written as f ( x ) = 1 n ∑ n i =1 F i ( x ) due to the fact that 1 n ∑ n i =1 ∇ f i ( x ⋆ ) = 0 . Note that ∇ F i ( x ⋆ ) = ∇ f i ( x ⋆ ) -∇ f i ( x ⋆ ) = 0 which implies that each F i ( x ) shares the same minimizer and thus the interpolation condition is satisfied ( σ 2 f, 1 = 0 ). However, ∇ f i ( x ⋆ ) is usually not available at hand. This motivates us to design the following algorithm.

Algorithm 1 (Loopless) AdaSVRPS and AdaSVRLS

Require: x 0 ∈ R d , µ F > 0 , c p > 0 or c l > 0

1: set w 0 = x 0 , η -1 = + ∞

2: for t = 0 to T -1 do

3: uniformly sample i t ⊆ [ n ]

9: return ¯ x T = 1 T ∑ T -1 t =0 x t

4.2 Algorithms and convergence

Inspired by the observation, we attempt to reduce the variance of the functions σ 2 f,B by constructing a sequence of random functions { F i t ( x ) } such that σ 2 1 n ∑ n i =1 F i t ( x ) ,B → 0 as x t → x ⋆ . However, directly applying SPS or SLS to { F i t ( x ) } still requires the knowledge of the Lipschitz constant to guarantee convergence. This problem can be solved by using our proposed AdaSPS and AdaSLS. The whole procedure of the final algorithm is summarized in Algorithm 1.

At each iteration of Algorithm 1, we construct a proxy function by adding two quantities to the minibatch function f i t ( x ) , where µ F 2 || x -x t || 2 is a proximal term that helps improve the inherent stochasticity due to the partial information obtained from f i t ( x ) . The additional inner product quantity is used to draw closer the minimizers of f i t ( x ) and f ( x ) . Following [27, 33], the full gradient is computed with a coin flip probability. Note that Algorithm 1 still works with η t replaced with SVRG and AdaSVRG stepsize since ∇ F i t ( x t ) = ∇ f i t ( x t ) -∇ f i t ( w t ) + ∇ f ( w t ) , and thus this framework can be seen as a generalization of the standard VR methods. A similar idea can also be found in the works on federated learning with variance-reduction [32, 2, 24, 37, 50].

Theorem 7. Assume each f i is convex and L smooth and X is a convex compact feasible set with diameter D . Let p t = 1 at +1 with 0 ≤ a < 1 . Algorithm 1 converges as:

Suppose γ max is sufficiently large, then picking µ ⋆ F = O ( L ) , c ⋆ p = O ( 1 √ LD 2 ) and c ⋆ l = O ( √ 1 -ρ √ LD 2 ) yields an O ( LD 2 T ) rate which matches the O ( L || x 0 -x ⋆ || 2 T ) rate of full-batch gradient descent except for a larger term D 2 due to the lack of knowledge of the Lipschitz constant.

Corollary 8. Under the setting of Theorem 7, given an arbitrary accuracy ε , the total number of gradient evaluations required to have E [ f (¯ x T ) -f ⋆ ] ≤ ε in expectation is O (log(1 /ε ) n +1 /ε ) .

The proved efficiency of stochastic gradient calls matches the optimal rates of SARAH [40]/SVRG and AdaSVRG [14] but removes the artificially designed inner and outer loop size. However, note that Algorithm 1 requires an additional assumption on individual convexity. Unfortunately, we believe this

0

1000

2000

# mini-batch gradient evaluations

10

8

10

5

10

2

10 1

10 4

10 7

f(x)-

f

strongly-convex+interpolation

0

2000

4000

# mini-batch gradient evaluations

10

8

10

5

10

2

10 1

10 4

10 7

f(x)-

f

strongly-convex+non-interpolation

0

5000

10000

# mini-batch gradient evaluations

10 1

10 3

10 5

f(x)-

f

convex+interpolation

0

10000

20000

# mini-batch gradient evaluations

10

2

10 0

10 2

10 4

10 6

f(x)-

f

convex+non-interpolation

AdaSVRLS

AdaSVRPS

AdaSLS

assumption is necessary for the VR methods to work for the algorithms in the Polyak and line-search family since SPS/SLS has to assume the same condition for the proof in the interpolation settings.

Discussion. The classical SVRG with Armijo line-search (presented as Algorithm 6 in [14]) employs the same gradient estimator as SVRG but chooses its stepsize based on the returning value of line-search on the individual function f i . Similarly, SVRG with classical Polyak stepsize uses the individual curvature information of f i to set the stepsize for the global variance-reduced gradient. Due to the misleading curvature information provided by the biased function f i , both methods have convergence issues. In constrast, Algorithm 1 reduces the bias by adding a correction term x T ( ∇ f ( w t ) -∇ f i t ( w t )) with global information to f i and then applying line-search or Polyakstepsize on the variance-reduced functions F i t . This difference essentially guarantees the convergence.

5 Numerical evaluation

In this section, we illustrate the main properties of our proposed methods in numerical experiments. A detailed description of the experimental setup can be found in Appendix F. We report the theoretically justified hyperparameter c scale p or c scale l as defined in Theorem 6 rather than c p or c l in the following.

Synthetic data. We illustrate the robustness property on a class of synthetic problems. We consider the minimization of a quadratic of the form: f ( x ) = 1 n ∑ n i =1 f i ( x ) where f i ( x ) = 1 2 ( x -b i ) T A i ( x -b i ) , b i ∈ R d and A i ∈ R d × d is a diagonal matrix. We use n = 50 , d = 1000 . We can control the convexity of the problem by choosing different matrices A i , and control interpolation by either setting all { b i } to be identical or different. We generate a strongly convex instance where the eigenvalues of ∇ 2 f ( x ) are between 1 and 10 , and a general convex instance by setting some of the eigenvalues to small values close to zero (while ensuring that each ∇ 2 f i ( x ) is positive semi-definite). The exact procedure to generate these problems is described in Appendix F.

For all methods, we use a batch size B = 1 . We compare AdaSPS and AdaSLS against DecSPS [44], SPS [34] and SLS [52] to illustrate the robustness property. More comparisons can be found in Appendix F.1. We fix c scale p = c scale l = 1 for AdaSPS/AdaSLS and use the optimal parameters for DecSPS and SPS. In Figure 1, we observe that SPS does not converge in the non-interpolated settings and DecSPS suffers from a slow O (1 / √ T ) convergence on the two interpolated problems. AdaSPS and AdaSLS show the desired convergence rates across all cases which matches the theory. When the problems are non-interpolated, AdaSVRPS and AdaSVRLS illustrate faster convergence, which can be seen in Figure 2.

Binary classification on LIBSVM datasets. Weexperiment with binary classification on four diverse datasets from [10]. We consider the standard regularized logistic loss: f ( x ) = 1 n ∑ n i =1 log(1 + exp( -y i · a T i x )) + 1 2 n || x || 2 where ( a i , y i ) ∈ R d +1 are features and labels. We defer the study of variance-reduction methods to Appendix F.2 for clarity of presentation. We benchmark against popular optimization algorithms including Adam [26], SPS [34], DecSPS [44], SLS [52] and AdaGradNorm [15]. We fix c scale p = c scale l = 1 for AdaSPS/AdaSLS and pick the best learning rate from { 10 i } i = -4 ,.., 3 for SGD, Adam and AdaGrad-Norm. We observe that Adam, SPS and SLS have remarkable performances on duke with n = 48 and d = 7129 , which satisfies interpolation. AdaSPS

0

500

1000

1500

# mini-batch gradient evaluations

10

3

10

2

10

1

10 0

10 1

Gradient Norm

duke

0

5000

10000 15000

# mini-batch gradient evaluations

10

6

10

5

10

4

10

3

10

2

Gradient Norm

rcv1

0

10000

20000

30000

# mini-batch gradient evaluations

10

3

10

2

10

1

Gradient Norm

ijcnn

0

10000

20000

30000

# mini-batch gradient evaluations

10

4

10

3

10

2

10

1

Gradient Norm

w8a

DecSPS

SPS

SGD

SLS

Adam

AdaSLS

AdaNorm

0

50

100

150

200

Epoch

0.800

0.825

0.850

0.875

0.900

0.925

0.950

Test Accuary

CIFAR10-ResNet34

0

50

100

150

200

Epoch

10

3

10

2

10

1

10 0

Train loss

0

50

100

150

200

Epoch

0.50

0.55

0.60

0.65

0.70

0.75

Test Accuary

CIFAR100-ResNet34

0

50

100

150

200

Epoch

10

3

10

2

10

1

10 0

Train loss

SPS

Adam

AdaGrad

SGD-M

DecSPS

and AdaSLS consistently perform reasonably well on the other three larger datasets. It is worth noting that the hyper-parameters c scale p and c scale l are fixed for all the datasets, which is desired in practice.

Deep learning task. We provide a heuristic extension of AdaSPS to over-parameterized non-convex optimization tasks to illustrate its potential. We benchmark the convergence and generalization performance of AdaSPS (DL) 5 for the multi-class classification tasks on CIFAR10 [28] and CIFAR100 [29] datasets using ResNet-34 [21]. More experimental details can be found in Appendix G. We demonstrate the effectiveness of AdaSPS (DL) in Figure 4.

Discussion. AdaSPS and AdaSLS consistently demonstrate robust convergence across all tasks and achieve performance on par with, if not better than, the best-tuned algorithms. Consequently, it is reliable and convenient for practical use.

6 Conclusion and future work

We proposed new variants of SPS and SLS algorithms and demonstrated their robust and fast convergence in both interpolated and non-interpolated settings. We further accelerate both algorithms for convex optimization with a novel variance reduction technique. Interesting future directions may include: accelerating AdaSPS and AdaSLS with momentum, developing effective robust adaptive methods for training deep neural networks, designing an adaptive algorithm that gives a faster rate O (1 /T ) under strong convexity, extensions to distributed and decentralized settings.

Acknowledgments

We appreciate the fruitful discussion with Anton Rodomanov.

References

Farshed Abdukhakimov, Chulu Xiang, Dmitry Kamzolov, and Martin Takáˇ c. Stochastic gradient descent with preconditioned polyak step-size. arXiv preprint arXiv:2310.02093 , 2023.

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263 , 2021.

Larry Armijo. Minimization of functions having lipschitz continuous first partial derivatives. Pacific Journal of Mathematics , 16(1):1-3, 1 1966.

Hilal Asi and John C Duchi. Stochastic (approximate) proximal point methods: Convergence, optimality, and adaptivity. SIAM Journal on Optimization , 29(3):2257-2290, 2019.

Hilal Asi, Daniel Lévy, and John C Duchi. Adapting to function difficulty and growth conditions in private optimization. Advances in Neural Information Processing Systems , 34:19069-19081, 2021.

Leonard Berrada, Andrew Zisserman, and M. Pawan Kumar. Training neural networks for and by interpolation. In Proceedings of the 37th International Conference on Machine Learning , ICML'20. JMLR.org, 2020.

Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review , 60(2):223-311, 2018. doi: 10 . 1137/16M1080173. URL https://doi . org/10 . 1137/ 16M1080173 .

Sébastien Bubeck. Convex optimization: Algorithms and complexity. Found. Trends Mach. Learn. , 8(3-4): 231-357, nov 2015. ISSN 1935-8237. doi: 10 . 1561/2200000050. URL https://doi . org/10 . 1561/ 2200000050 .

Karan Chadha, Gary Cheng, and John Duchi. Accelerated, optimal and parallel: Some results on modelbased stochastic optimization. In International Conference on Machine Learning , pages 2811-2827. PMLR, 2022.

Chih-Chung Chang and Chih-Jen Lin. Libsvm: A library for support vector machines. ACM Trans. Intell. Syst. Technol. , 2(3), may 2011. ISSN 2157-6904. doi: 10 . 1145/1961189 . 1961199. URL https: //doi . org/10 . 1145/1961189 . 1961199 .

Gary Cheng and John Duchi. adastar: A method for adapting to interpolation. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop) , 2022. URL https://openreview . net/forum?id= dMbczvwk6Jw .

Ashok Cutkosky and Kwabena Boahen. Online convex optimization with unconstrained domains and losses. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, page 748-756, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.

Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014. URL https://proceedings . neurips . cc/paper_files/ paper/2014/file/ede7e2b6d13a41ddf9f4bdef84fdc737-Paper . pdf .

Benjamin Dubois-Taine, Sharan Vaswani, Reza Babanezhad, Mark Schmidt, and Simon Lacoste-Julien. SVRG meets adagrad: painless variance reduction. Mach. Learn. , 111(12):4359-4409, 2022. doi: 10 . 1007/s10994-022-06265-x. URL https://doi . org/10 . 1007/s10994-022-06265-x .

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. , 12(null):2121-2159, jul 2011. ISSN 1532-4435.

Leonardo Galli, Holger Rauhut, and Mark Schmidt. Don't be so monotone: Relaxing stochastic line search in over-parameterized models, 2023.

Robert M. Gower, Aaron Defazio, and Michael G. Rabbat. Stochastic polyak stepsize with a moving target. CoRR , abs/2106.11851, 2021. URL https://arxiv . org/abs/2106 . 11851 .

Robert M Gower, Mathieu Blondel, Nidham Gazagnadou, and Fabian Pedregosa. Cutting some slack for sgd with adaptive polyak stepsizes. arXiv preprint arXiv:2202.12328 , 2022.

Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference and prediction . Springer, 2 edition, 2009. URL http://www-stat . stanford . edu/~tibs/ ElemStatLearn/ .

Elad Hazan and Sham Kakade. Revisiting the polyak step size, 2019. URL https://arxiv . org/abs/ 1905 . 00313 .

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016. doi: 10 . 1109/CVPR . 2016 . 90.

Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013. URL https://proceedings . neurips . cc/paper_files/paper/2013/file/ ac1dd209cbcc5e5d1c6e28598e8cbbe8-Paper . pdf .

Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konecný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi, Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Found. Trends Mach. Learn. , 14 (1-2):1-210, jun 2021. ISSN 1935-8237. doi: 10 . 1561/2200000083. URL https://doi . org/10 . 1561/ 2200000083 .

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 5132-5143. PMLR, 13-18 Jul 2020. URL https://proceedings . mlr . press/v119/karimireddy20a . html .

Ali Kavis, Stratis Skoulakis, Kimon Antonakopoulos, Leello Tadesse Dadi, and Volkan Cevher. Adaptive stochastic variance reduction for non-convex finite-sum minimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 23524-23538. Curran Associates, Inc., 2022. URL https://proceedings . neurips . cc/paper_files/paper/2022/file/ 94f625dcdec313cd432d65f96fcc51c8-Paper-Conference . pdf .

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv . org/abs/ 1412 . 6980 .

Dmitry Kovalev, Samuel Horváth, and Peter Richtárik. Don't jump through hoops and remove those loops: Svrg and katyusha are better without the outer loop. In Aryeh Kontorovich and Gergely Neu, editors, Proceedings of the 31st International Conference on Algorithmic Learning Theory , volume 117 of Proceedings of Machine Learning Research , pages 451-467. PMLR, 08 Feb-11 Feb 2020. URL https://proceedings . mlr . press/v117/kovalev20a . html .

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). . URL http://www . cs . toronto . edu/~kriz/cifar . html .

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). . URL http://www . cs . toronto . edu/~kriz/cifar . html .

Frederik Kunstner, Victor S. Portella, Mark Schmidt, and Nick Harvey. Searching for optimal per-coordinate step-sizes with multidimensional backtracking, 2023.

Shuang Li, William J Swartworth, Martin Takáˇ c, Deanna Needell, and Robert M Gower. Sp2: A second order stochastic polyak method. arXiv preprint arXiv:2207.08171 , 2022.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Inderjit S. Dhillon, Dimitris S. Papailiopoulos, and Vivienne Sze, editors, Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020 . mlsys.org, 2020. URL https://proceedings . mlsys . org/book/316 . pdf .

Zhize Li, Hongyan Bao, Xiangliang Zhang, and Peter Richtarik. Page: A simple and optimal probabilistic gradient estimator for nonconvex optimization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6286-6295. PMLR, 18-24 Jul 2021. URL https://proceedings . mlr . press/v139/ li21a . html .

Nicolas Loizou, Sharan Vaswani, Issam Hadj Laradji, and Simon Lacoste-Julien. Stochastic polyak stepsize for SGD: an adaptive learning rate for fast convergence. In Arindam Banerjee and Kenji Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event , volume 130 of Proceedings of Machine Learning Research , pages 1306-1314. PMLR, 2021. URL http://proceedings . mlr . press/v130/loizou21a . html .

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations , 2017. URL https://openreview . net/forum?id= Skq89Scxx .

Julien Mairal. Optimization with first-order surrogate functions. In International Conference on Machine Learning , pages 783-791. PMLR, 2013.

Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning , pages 15750-15769. PMLR, 2022.

Angelia Nedi´ c and Dimitri Bertsekas. Convergence Rate of Incremental Subgradient Algorithms , pages 223264. Springer US, Boston, MA, 2001. ISBN 978-1-4757-6594-6. doi: 10 . 1007/978-1-4757-6594-6{ \ _}11. URL https://doi . org/10 . 1007/978-1-4757-6594-6_11 .

Yurii Nesterov. Introductory Lectures on Convex Optimization: A Basic Course . Springer Publishing Company, Incorporated, 1 edition, 2014. ISBN 1461346916.

Lam M. Nguyen, Jie Liu, Katya Scheinberg, and Martin Takáˇ c. SARAH: A novel method for machine learning problems using stochastic recursive gradient. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 2613-2621. PMLR, 06-11 Aug 2017. URL https: //proceedings . mlr . press/v70/nguyen17b . html .

Jorge Nocedal and Stephen J. Wright. Numerical Optimization . Springer, New York, NY, USA, 2e edition, 2006.

Adam M Oberman and Mariana Prazeres. Stochastic gradient descent with polyak's learning rate. arXiv preprint arXiv:1903.08688 , 2019.

Francesco Orabona and Dávid Pál. Scale-free algorithms for online linear optimization. In Kamalika Chaudhuri, CLAUDIO GENTILE, and Sandra Zilles, editors, Algorithmic Learning Theory , pages 287-301, Cham, 2015. Springer International Publishing. ISBN 978-3-319-24486-0.

Antonio Orvieto, Simon Lacoste-Julien, and Nicolas Loizou. Dynamics of sgd with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 26943-26954. Curran Associates, Inc., 2022. URL https://proceedings . neurips . cc/paper_files/paper/2022/file/ ac662d74829e4407ce1d126477f4a03a-Paper-Conference . pdf .

B. T. Polyak. Introduction to optimization . Translations series in mathematics and engineering. Optimization Software, Publications Division, New York, 1987. ISBN 0911575146; 9780911575149.

Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics , 22(3):400 - 407, 1951. doi: 10 . 1214/aoms/1177729586. URL https://doi . org/10 . 1214/ aoms/1177729586 .

Michal Rolinek and Georg Martius. L4: Practical loss-based stepsize adaptation for deep learning. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018) , pages 6434-6444. Curran Associates, Inc., 2018. URL http://papers . nips . cc/paper/7879-l4-practical-loss-basedstepsize-adaptation-for-deep-learning . pdf .

Mark Schmidt and Nicolas Le Roux. Fast convergence of stochastic gradient descent under a strong growth condition. arXiv preprint arXiv:1308.6370 , 2013.

Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming , 162:83-112, 2017.

Ohad Shamir, Nati Srebro, and Tong Zhang. Communication-efficient distributed optimization using an approximate newton-type method. In International conference on machine learning , pages 1000-1008. PMLR, 2014.

Matthew J. Streeter and H. Brendan McMahan. Less regret via online conditioning. CoRR , abs/1002.4862, 2010. URL http://arxiv . org/abs/1002 . 4862 .

Sharan Vaswani, Aaron Mishkin, Issam H. Laradji, Mark Schmidt, Gauthier Gidel, and Simon LacosteJulien. Painless stochastic gradient: Interpolation, line-search, and convergence rates. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 3727-3740, 2019. URL https://proceedings . neurips . cc/paper/2019/hash/ 2557911c1bf75c2b643afb4ecbfc8ec2-Abstract . html .

Sharan Vaswani, Issam Laradji, Frederik Kunstner, Si Yi Meng, Mark Schmidt, and Simon Lacoste-Julien. Adaptive gradient methods converge faster with over-parameterization (but you should do a line-search). arXiv preprint arXiv:2006.06835 , 2020.

Xiaoyu Wang, Mikael Johansson, and Tong Zhang. Generalized polyak step size for first order optimization with momentum. arXiv preprint arXiv:2305.12939 , 2023.

Rachel Ward, Xiaoxia Wu, and Leon Bottou. AdaGrad stepsizes: Sharp convergence over nonconvex landscapes. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 6677-6686. PMLR, 09-15 Jun 2019. URL https://proceedings . mlr . press/v97/ward19a . html .

Yuege Xie, Xiaoxia Wu, and Rachel Ward. Linear convergence of adaptive stochastic gradient descent. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 1475-1485. PMLR, 26-28 Aug 2020. URL https://proceedings . mlr . press/v108/xie20a . html .

Appendix

A Technical Preliminaries

A.1 Basic Definitions

We use the following definitions throughout the paper.

Definition 1 (convexity) . A differentiable function f : R d → R is convex if ∀ x , y ∈ R d ,

Definition 2 (strong convexity) . A differentiable function f : R d → R is µ -strongly convex if ∀ x , y ∈ R d ,

Definition 3 ( L -smooth) . Let function f : R d → R be differentiable. f is smooth if there exists L > 0 such that ∀ x , y ∈ R d ,

A.2 Useful Lemmas

We frequently use the following helpful lemmas for the proof.

Lemma 9 (Nesterov [39], Lemma 1.2.3) . Definition 3 implies that there exists a quadratic upper bound on f:

Lemma 10 (Nesterov [39], Theorem 2.1.5) . If a convex function f satisfies (A.4) , then it holds that:

Lemma 11 (Ward et al. [55]) . For any non-negative sequence a 0 , ..., a T , the following holds:

If a 0 ≥ 1 , then the following holds:

Proof. To show equation (A.6), we proceed with the proof by induction. For t = 0 , (A.6) holds trivially since √ a 0 ≤ √ a 0 ≤ 2 √ a 0 . Assume equation (A.6) holds for T -1 . For RHS, we have:

where the last inequality is due to the fact that 2 √ x -y + y √ x ≤ 2 √ x for any x ≥ y ≥ 0 . For LHS, we have:

where the last inequality is due to the fact that √ x -y + y √ x ≥ √ x for any x ≥ y ≥ 0 .

We next show equation (A.7) by induction. For t = 0 , equation (A.7) trivially holds since 1 ≤ log( a 0 ) + 1 . Assume (A.7) holds for T -1 , we have:

where the last inequality is due to the fact that log( x -y ) + y x ≤ log( x ) for any x ≥ y ≥ 0 since e y x ≤ 1+ y x 1 -y 2 x 2 .

Lemma 12 (Dubois-Taine et al. [14, Lemma 5]) . If x 2 ≤ a ( x + b ) for a ≥ 0 and b ≥ 0 , then it holds that:

The following Lemma is an extension of Lemma 5 in [44].

Lemma 13. Let z t +1 ≤ (1 -aη t ) z t + η t b and z t ≥ 0 where a > 0 , b > 0 and η t > 0 , η t +1 ≤ η t , ∀ t ≥ 0 . It holds that:

Proof. Since η t is non-increasing, 1 -aη t ≤ 0 is non-decreasing. For any t ≥ 0 such that 1 -aη t ≤ 0 , we have z t +1 ≤ η t b ≤ η 0 b . If 1 -aη t ≤ 0 for all t ≥ 0 , then the proof is done. Otherwise, let us assume there exists a first index j such that 1 -aη j > 0 and we have z j ≤ max { z 0 , η 0 b } := ˜ z 0 . We proceed with the proof starting with the index j by induction. For t = j , the lemma trivially holds. Let us assume z t ≤ max { b a , ˜ z 0 } for t > j . If b a ≥ ˜ z 0 , then by induction, we have:

If instead b a ≤ ˜ z 0 , then by induction, we have:

Combining the above cases concludes the proof.

The following lemma is commonly used in the works on Polyak stepsize [34, 20].

Lemma 14. Suppose a function f is L -smooth and µ -strongly convex, then the following holds:

The following lemma provides upper and lower bounds for the stepsize of AdaSPS.

Lemma 15. Suppose each f i is L -smooth, then the stepsize of AdaSPS (AdaSPS) satisfies:

Proof. The upper bound follows from the definition of the stepsize (AdaSPS). To prove the lower bound, we note that the stepsize (AdaSPS) is composed of two parts where the first component f i s ( x s ) -ℓ ⋆ i s c p ||∇ f i s ( x s ) || 2 ≥ 1 2 c p L for all 0 ≤ s ≤ t due to (A.15), and the second component is always decreasing. Finally, recall that η -1 = + ∞ and thus the proof is completed.

The following lemma provides upper and lower bounds for the stepsize of AdaSLS. We refer to Appendix D for details of the line-search procedure.

Lemma 16. Suppose each f i is L -smooth, then the stepsize of AdaSLS (AdaSLS) satisfies:

Proof. The upper bound is due to the definition of the stepsize (AdaSLS). We next prove the lower bound. From the smoothness definition, the following holds for all γ t :

For any 0 < γ t ≤ 2(1 -ρ ) L , we have:

which satisfies the line-search condition (4). From the procedure of Backtracking line-search (Alg. 4), if γ max ≤ 1 -ρ L , then γ t = γ max is accepted. Otherwise, since we require the decreasing factor β to be no smaller than 1 2 in Algorithm 4, we must have γ t ≥ 2(1 -ρ ) 2 L . Therefore, γ t is always lower bounded by min { 1 -ρ L , γ max } . The second component of AdaSLS is always decreasing, and recall that η -1 = + ∞ . The proof is thus completed.

B Proofs of main results

B.1 Proof of Theorem 1

Proof. We follow a common proof routine for the general convex optimization [15, 14, 44]. Using the update rule of PSGD (5), we have:

Dividing by 2 η t and rearranging gives:

Summing from t = 0 to t = T -1 , we get:

where in the second inequality, we use the decreasing property of the stepsize η t which guarantees 1 2 η t -1 2 η t -1 ≥ 0 , and we use the fact that || x t -x ⋆ || 2 ≤ D 2 because of the projection step in (5). For clarity, we next separate the proof for AdaSPS and AdaSLS.

AdaSPS: We upper bound the last two terms by using Lemma 15 and we obtain:

and

Using || x 0 -x ⋆ || 2 2 η 0 ≤ D 2 2 η T -1 and plugging (B.4) and (B.5) back to (B.3) gives:

Taking the expectation on both sides, we have:

Using the convexity assumption of f and applying Jensen's inequality to the square root function, we get:

where err 2 f,B = E i s [ f ⋆ i s -ℓ ⋆ i s ] . Let τ := 2 c p LD 2 + 1 c p . Taking the square gives:

We next apply Lemma 12 with x = ∑ T -1 t =0 E [ f ( x t ) -f ( x ⋆ )] , a = τ 2 and b = T ( σ 2 f,B +err 2 f,B ) :

We conclude by dividing both sides by T and using Jensen's inequality:

where ¯ x T = 1 T ∑ T -1 t =0 x t .

AdaSLS: The proof is almost the same as AdaSPS. We omit procedures with the same proof reasons for simplicity. We first use Lemma 16 to obtain:

and

Inequality (B.3) can then be further bounded by:

where we used line-search condition (4) and the fact that f i s ( x t -γ t ∇ f i s ( x s )) ≥ f ⋆ i s .

Let τ := max { L (1 -ρ ) √ ρ , 1 γ max √ ρ } c l D 2 + 1 c l √ ρ . We arrive at:

where ¯ x T = 1 T ∑ T -1 t =0 x t .

B.2 Full statement and proof for Lemma 4

Lemma 17 (Bounded iterates) . Let each f i be µ -strongly convex and L -smooth. For any t ∈ N , the iterates of SGD with AdaSPS or AdaSLS satisfy:

where σ 2 max := max i t { f i t ( x ⋆ ) -ℓ ⋆ i t } , b := 1 / ( 4 c 3 p √ f i 0 ( x 0 ) -ℓ ⋆ i 0 ) for AdaSPS and σ 2 max := max i t { f i t ( x ⋆ ) -f ⋆ i t } , b := 1 / ( 4 c 3 l ρ 2 √ γ 0 ||∇ f i 0 ( x 0 ) || 2 ) for AdaSLS.

Proof. By strong convexity of f i t , the iterates generated by SGD satisfy:

We next separate the proofs for clarity.

AdaSPS : Plugging in the upper bound of η t in Lemma 15, we obtain:

where σ 2 max := max i t { f i t ( x ⋆ ) -ℓ ⋆ i t } .

We now split the proof into two cases. Firstly, if c p √ ∑ t s =0 f i s ( x t ) -ℓ ⋆ i s ≤ 1 2 , then it follows that:

Plugging in the above bounds, we have:

We conclude by applying Lemma 13 with z t = || x t -x ⋆ || 2 , a = µ , b = (2 σ 2 max + 1 4 c 3 p √ f i 0 ( x 0 ) -ℓ ⋆ i 0 ) . Secondly, if instead c p √ ∑ t s =0 f i s ( x t ) -ℓ ⋆ i s ≥ 1 2 , then we have:

and consequently we can apply Lemma 13 with z t = || x t -x ⋆ || 2 , a = µ , b = 2 σ 2 max AdaSLS : Similarly, by plugging the upper bound of η t in Lemma 16, we obtain:

.

where σ 2 max = max i t { f i t ( x ⋆ ) -f ⋆ i t } . We can then compare c l ρ √ ∑ t s =0 γ s ∥|∇ f i s ( x s ) || 2 with 1 2 and apply Lemma 13 correspondingly.

B.3 Proofs for Theorem 2 and 6

Proof. For clarity, we separate the proofs for AdaSPS and AdaSLS. AdaSPS: Plugging in the upper bound of η t in Lemma 15, we have:

Since c p √ f i 0 ( x 0 ) -f ⋆ ≥ 1 , (B.23) can be reduced to:

By convexity of f i t , we get:

Note that ⟨∇ f i t ( x t ) , x t -x ⋆ ⟩ ≥ 0 and η t is non-increasing, we thus get:

We first show that η T -1 is always lower bounded. From equation (B.25) and using convexity of f i t , we get:

Summing from t = 0 to t = T -1 , we get:

Using the lower bound of η t , we get:

This reveals that:

Plugging in the lower bound of η T -1 to (B.26), we obtain:

Plugging in c p = c scale p √ f i 0 ( x 0 ) -f ⋆ , we get

case 1: f is convex .

For any t ≥ 1 , we take expectation conditional on i 0 on both sides and get:

Summing up from t = 1 to t = T and dividing by T , we obtain:

Note that E [ || x 1 -x ⋆ || 2 | i 0 ] = || x 1 -x ⋆ || 2 ≤ || x 0 -x ⋆ || 2 due to (B.31). We thus get:

Taking expectation w.r.t i 0 on both sides, we arrive at:

case 2: f is strongly-convex

For any t ≥ 1 , we take expectation conditional on i 0 on both sides of (B.32) and get:

Note that f ( x t ) -f ⋆ ≥ µ 2 || x t -x ⋆ || 2 due to strong convexity, we obtain:

For any T ≥ 1 , unrolling gives:

The claim of Theorem 6 follows by taking expectation w.r.t i 0 on both sides.

AdaSLS : We only highlight the difference from AdaSPS. Plugging in the upper bound of η t from Lemma 16 and using the line-search condition, we get:

Since c l ρ √ ∑ t s =0 γ s ||∇ f i s ( x s ) || 2 ≥ 1 , we obtain the same equation as (B.26). To find a lower bound of η T -1 , we rearrange (B.25) as:

the left-hand-side of which can be lower bounded by:

Summing over t = 0 to t = T -1 gives:

This implies that:

After plugging the above into (B.26), the remaining proof follows from the same routine as shown for AdaSPS.

Proofs for Loopless Variance-Reduction

B.4 Statement and Proof of Lemma 18

The following Lemma provides us with the guarantee that as w t , x t → x ⋆ , E i t [ F i t ( x t ) -F ⋆ i t ] → 0 , which implies diminishing variance.

Lemma 18. Assume each f i is convex and L -smooth, for any t ≥ 0 , the iterates generated by Algorithm 1 satisfy:

Proof. By definition of F i t ( x ) , we have:

By µ F -strong convexity of F i t ( x ) , we obtain:

Plugging (B.47) into (B.46), taking expectation w.r.t the randomness i t on both sides gives:

B.5 Proof of Theorem 7

Proof. Recall the proof technique that gives equation (B.6) and (B.14) in Theorem 1. Following the same routine, we arrive at:

where τ = (2 c p ( L + µ F ) D 2 + 1 c p ) for AdaSVRPS and τ = max { L + µ F (1 -ρ ) √ ρ , 1 γ max √ ρ } c l D 2 + 1 c l √ ρ for AdaSVRLS. The difference is due to the fact that F i t ( x ) is ( L + µ F ) -smooth. Taking the expectation, using the fact that E [ ∇ F i t ( x t )] = E [ ∇ f i t ( x t ) + ∇ f ( w t ) -∇ f i t ( w t )] = ∇ f ( x t ) and applying Lemma 18, we end up with:

Taking the square gives:

Define the Lyapunov function: Z t +1 = 1 2(1 -a ) τ 2 p t +1 µ F ||∇ f i t +1 ( w t +1 ) -∇ f i t +1 ( x ⋆ ) || 2 . It follows that:

Adding ∑ T -1 t =0 E [ Z t +1 ] to both sides of (B.51) and substituting the above upper bound, we get:

Rearranging and dropping E [ Z T ] ≥ 0 gives:

By our choice of p t , we have:

Therefore, it holds that:

Further, by L -smoothness and convexity of f , we have

Hence. we obtain:

It follows that:

Dividing both sides by T and applying Jensen's inequality concludes the proof.

B.6 Proof of Corollary 8

Proof. Algorithm 1 calls the stochastic gradient oracle in expectation O (1 + p t n ) times at iteration t . Therefore, the total number of gradient evaluations is upper bounded by O ( ∑ T -1 t =0 p t n + T ) . By our choice of p t , it holds that ∑ T -1 t =0 p t ≤ 1 a ∑ T -1 t =0 1 t +2 ≤ 1 a (log( T ) + 1 -1) = 1 a log( T ) . Due to the sublinear convergence rate of Algorithm 1, we conclude that the total number of stochastic gradient calls is O (log(1 /ε ) n +1 /ε ) .

C Pseudo-codes for AdaSPS and AdaSLS

In this section, we provide formal pseudo codes for AdaSPS (AdaSPS) and AdaSLS (AdaSLS).

To implement AdaSPS, a lower bound of optimal function value for each minibatch function is required. For machine learning problems where the individual loss functions are non-negative, we can use zero as an input. Apart from that, we need to provide a constant c p to adjust the magnitude of the stepsize. Theoretically suggested c p for robust convergence satisfies c scale p = c p √ f i 0 ( x 0 ) -ℓ ⋆ i 0 ≥ 1 2 . Therefore, a common choice is to set

Algorithm 2 AdaSPS

To implement AdaSLS (AdaSLS), a line-search sub-solver 4 and an input constant c l > 0 are required. Similar to (AdaSPS), we can set c l = 1 ρ √ γ 0 ||∇ f i 0 ( x 0 ) || 2 according to the theory.

Algorithm 3 AdaSLS

D Line-search procedure

In this section, we introduce the classical Armijo line-search method [3, 41]. Given a function f i t ( x ) , the Armijo line-search returns a stepsize γ t that satisfies the following condition:

where ρ ∈ (0 , 1) is an input hyper-parameter. If f i t ( x ) is a smooth function, then backtracking line-search 4 is a practical implementation way to ensure that D.1 is satisfied.

Algorithm 4 Backtracking line-search

Require: β ∈ [ 1 2 , 1) , ρ ∈ (0 , 1) , γ max > 0 (We fix β = 0 . 8 and ρ = 0 . 5 for AdaSLS)

1: γ = γ max

3: γ = βγ return γ t = γ

To implement Algorithm 4, one needs to provide the decreasing factor β , the maximum stepsize γ max , and the condition parameter ρ . Starting from γ max , Algorithm 4 decreases the stepsize iteratively by a constant factor β until the condition D.1 is satisfied. Note that checking the condition requires additional minibatch function value evaluations. Fortunately, note that the output γ cannot be smaller than 1 -ρ L (Lemma 16), and thus the number of extra function value evaluations required is at most O ( max { log Lγ max / (1 -ρ ) 1 /β , 1 } ) . In practice, Vaswani et al. [52] suggests dynamic initialization of γ max to reduce the algorithm's running time, that is, setting γ max t = γ t -1 θ 1 /n where a common choice for θ is 2 . This strategy initializes γ max by a slightly larger number than the last output and thus is usually more efficient than keeping γ max constant or always using γ max t = γ t -1 . In all our experiments, we use the same γ max at each iteration for AdaSLS to show its theoretical properties.

Goldstein line-search is another line-search method that checks D.1 and an additional curvature condition [41]. We do not study this method in this work and we refer to [52] for more details.

E Counter examples of SPS and its variants for SVRG

We provide two simple counterexamples where SVRG with the SPS stepsize and its intuitive variants fail to converge. For simplicity, consider the update rule x t +1 = x t -η t ∇ f ( x t ) , i.e. w t = x t for all t ≥ 0 . Consider the function f ( x ) = f 1 ( x )+ f 2 ( x ) 2 where f 1 ( x ) = a 1 ( x -1) 2 and f 2 ( x ) = a 2 ( x +1) 2 with a 1 , a 2 > 0 .

Example 19. Individual curvature is not representative. Consider the standard stochastic Polyak stepsize: η t = f i ( x t ) -f ⋆ i ||∇ f i ( x t ) || 2 where i is randomly chosen from { 1 , 2 } . We now let a 1 = 1 and a 2 < 1 . Note that ∇ 2 f ( x ) = a 1 + a 2 ∈ (1 , 2) while E i [ η t ] = 1 8 + 1 8 a 2 → + ∞ as a 2 → 0 , which leads to divergence. The reason behind this is that individual curvature does not match the global curvature.

̸

Example 20. Mismatching quantity. Consider a variant of stochastic Polyak stepsize: η t = f i ( x t ) -f ⋆ i ||∇ f i ( x t ) -∇ f i ( w t )+ ∇ f ( w t ) || 2 where i is randomly chosen from { 1 , 2 } . Let a 1 = a 2 = 1 . We note E i [ η t ∇ f ( x t )] = x 2 t +1 2 x t = 0 and thus no stationary point exists. Similar reasoning can exclude a number of other variants such as: η t = f i ( x t ) -f i ( w t )+ f ( w t ) -f ⋆ i ||∇ f i ( x t ) -∇ f i ( w t )+ ∇ f ( w t ) || 2 . Indeed, the numerator is not the proper function value difference of a valid function with the gradient defined in the denominator.

F Experimental details and additional experiment results

In this section, we provide a detailed setup of the experiments presented in the main paper.

In practice, we can use a lower bound of F ⋆ i t for running AdaSVRPS since convergence is still guaranteed thanks to the property of AdaSPS. By default, we use ℓ ⋆ i t +min x { x T ( ∇ f ( w t ) -∇ f i t ( w t )) + µ F 2 || x -x t || 2 } for all the experiments, where ℓ ⋆ i t is a lower bound for f ⋆ i t .

F.1 Synthetic experiment

We consider the minimization of a quadratic of the form: f ( x ) = 1 n ∑ n i =1 f i ( x ) where f i ( x ) = 1 2 ( x -b i ) T A i ( x -b i ) , b i ∈ R d and A i ∈ R d × d is a diagonal matrix. We use n = 50 , d = 1000 . We control the interpolation by either setting all { b i } to be identical or different. Each component of { b i } is generated according

to N (0 , 10 2 ) . We control the complexity of the problems by choosing different matrices A i . For the stronglyconvex case, we first generate a matrix A N = clip (   a 11 ... a 1 d ... a n 1 ... a nd   , 1 , 10) where each a ij ∼ N (0 , 15 2 ) and the clipping operator clips the elements to the interval between 1 and 10. Then we compute:

where ⊙ denotes the Hadamard product. We set the diagonal elements of each A i using the corresponding row stored in the matrix A . Note that ∇ 2 f ( x ) = 1 n ∑ n i =1 A i has the minimum and the largest eigenvalues being 1 and 10. For the general convex case, we use the same matrix A N to generate a sparse matrix A M such that A M = A N ⊙ M where M is a mask matrix with M ij ∼ B (1 , p ) and ( 1 ... 1 ) · M : j ≥ 1 , ∀ j ∈ [1 , d ] . We then compute the matrix A and set each A i in the same way.

Through the construction, the smallest eigenvalues of ∇ 2 f ( x ) are clustered around zero, and the largest eigenvalue is 10. Additionally, each ∇ 2 f i ( x ) is positive semi-definite.

We set the batch size to be 1 and thus we have f ⋆ i t = 0 with interpolation and ℓ ⋆ i t = 0 without interpolation.

For AdaSPS/AdaSVRPS we fix c scale p = 1 , and for AdaSVRPS we further use µ F = 10 and p t = 1 0 . 1 t +1 . We compare against DecSPS [44], SPS [34] and SVRG [22] and tune the stepsize for SVRG by picking the best one from { 10 i } i = -4 ,.., 3 .

In addition to these optimizers, we further evaluate the performance of SLS [52], AdaSLS, SGD, SPS max [34] and AdaSVRLS. We fix c scale l = 1 , γ max = 10 , β = 0 . 8 and ρ = 1 / 2 for both algorithms and for AdaSVRLS, we further use µ F = 10 and p t = 1 0 . 1 t +1 . For SGD, we use the best learning rate schedules in different scenarios. Specifically, for both interpolation problems, we keep the stepsize constant and for non-interpolation problems, we apply Θ(1 / √ t ) and Θ(1 /t ) decay schedules for convex and strongly-convex problems respectively. We further pick the best stepsize from { 10 i } i = -4 ,..., 3 . For SPS max , we use γ b = 10 -3 and we only showcase its performance in non-interpolated settings. We report the results in Figure F.1. We observe that AdaSLS is comparable if no faster than the best-tuned vanilla SGD. SPS max is reduced to the vanilla SGD with constant stepsize. AdaSVRLS provides similar performance to AdaSVRPS but due to the cost of additional function evaluations, it is less competitive than AdaSVRPS.

0

1000

2000

# mini-batch gradient evaluations

10

8

10

5

10

2

10 1

10 4

10 7

f(x)-

f

strongly-convex+interpolation

0

2000

4000

# mini-batch gradient evaluations

10

8

10

5

10

2

10 1

10 4

10 7

f(x)-

f

strongly-convex+non-interpolation

0

2500

5000

7500

10000

# mini-batch gradient evaluations

10 0

10 2

10 4

10 6

f(x)-

f

convex+interpolation

0

10000

20000

# mini-batch gradient evaluations

10

2

10 0

10 2

10 4

10 6

f(x)-

f

convex+non-interpolation

0

1000

2000

# mini-batch gradient evaluations

10

6

10

5

10

4

10

3

10

2

10

1

Step size

0

2000

4000

# mini-batch gradient evaluations

10

3

10

2

10

1

10 0

Step size

0

2500

5000

7500

10000

# mini-batch gradient evaluations

10

3

10

2

10

1

10 0

Step size

0

10000

20000

# mini-batch gradient evaluations

10

3

10

2

10

1

Step size

AdaSVRPS

AdaSVRLS

SVRG

SPS

AdaSLS

SPS_max

SGD

DecSPS

SLS

F.2 Binary classification

Following the binary classification experiments presented in the main paper, we provide additional experiments for VR algorithms. The chosen hyper-parameters for each algorithm can be found in Table F.3. In particular, we fix c scale l = 1 , γ max = 10 3 , β = 0 . 8 and ρ = 1 / 2 for AdaSLS and AdaSVRLS. We report the best µ F ∈ { 10 -4 , 10 2 } . In Figure F.2, we observe that AdaSVRLS/AdaSVRPS provides similar performance to the other two variance-reduction methods. The details of the four considered datasets are summarized in Table F.2.

We next investigate the impact of the probability schedule on the convergence behaviours. We pick w8a as the dataset and run AdaSVRPS (Alg. 1) with and without probability decay. Specficially, we set p t = B/n and p t = 1 0 . 1 t +1 to separate the cases. We control the level of the interpolation by using B = 32 and B = 128 since σ f, 128 ≤ σ f, 32 . From Figure F.3, we observe that decreasing probability schedule is more efficient when the problem is more non-interpolated. This is because for interpolated problems, the frequent computation of the full gradients at the beginning provides no additional convergence benefits.

0

500

1000

1500

# mini-batch gradient evaluations

10

3

10

2

10

1

10 0

10 1

Gradient Norm

duke

0

5000

10000

15000

# mini-batch gradient evaluations

10

7

10

5

10

3

Gradient Norm

rcv1

0

10000

20000

30000

# mini-batch gradient evaluations

10

7

10

5

10

3

10

1

Gradient Norm

ijcnn

0

10000

20000

30000

# mini-batch gradient evaluations

10

5

10

4

10

3

10

2

10

1

10 0

Gradient Norm

w8a

0

500

1000

1500

# mini-batch gradient evaluations

10

4

10

2

10 0

10 2

Step size

0

5000

10000

15000

# mini-batch gradient evaluations

10 0

10 1

10 2

10 3

10 4

10 5

Step size

0

10000

20000

30000

# mini-batch gradient evaluations

10

1

10 0

10 1

10 2

10 3

Step size

0

10000

20000

30000

# mini-batch gradient evaluations

10

1

10 1

10 3

10 5

Step size

SVRG

AdaSVRLS

AdaSVRPS

AdaSVRG

DecSPS

SPS

SGD

SLS

AdaSLS

AdaNorm

0

100000

200000

300000

400000

# mini-batch gradient evaluations

10

6

10

5

10

4

10

3

10

2

10

1

10 0

Gradient Norm

w8a (B=32)

0

10000 20000 30000 40000 50000

# mini-batch gradient evaluations

10

5

10

4

10

3

10

2

10

1

10 0

Gradient Norm

w8a (B=128)

AdaSVRPS with decreasing p

AdaSVRPS with constant p

a

γ

=

1

µ

F

G Deep learning task

In this section, we provide a heuristic extension of AdaSPS to over-parameterized non-convex optimization tasks. When training modern deep learning models, Loshchilov and Hutter [35] observe that a cyclic behaviour of the stepsize, i.e., increasing at the beginning and then decreasing up to a constant, can help with fast training and good generalization performance. Since AdaSPS is a non-increasing stepsize, it excludes such a cyclic behaviour. To address this issue, we provide a non-convex version of AdaSPS which incorporates a restart mechanism that allows an increase of the stepsize according to the local curvature. The full algorithm is summarized in Algorithm 5. In practice, we can set u = B n . Algorithm 5 updates the stepsize and c p at the beginning of each epoch and uses AdaSPS (AdaSPS) for the rest of the epoch.

Following [34, 52], we benchmark the convergence and generalization performance of AdaSPS (DL) 5 for the multi-class classification tasks on CIFAR10 [28] and CIFAR100 [29] datasets using ResNet-34 [21]. We compare against SPS [34], Adam [26], AdaGrad [15], DecSPS [44] and SGD with momentum. We use the smoothing technique and pick c = 0 . 02 for SPS as suggested in [34]. We use the official implementations of Adam, AdaGrad, and SGD with momentum from https://pytorch.org/docs/stable/optim.html. We choose lr = 10 -3 , β 1 = 0 . 9 and β 2 = 0 . 999 for Adam. We choose lr = 0 . 01 for AdaGrad. We choose lr = 0 . 01 and β = 0 . 9 for SGD with momentum. Finally, we pick c scale p = 0 . 02 for Algorithm 5. In Figure G.1, AdaSPS (DL) shows competitive performance on both datasets. We leave the study of its theoretical properties to future work.

0

50

100

150

200

Epoch

0.75

0.80

0.85

0.90

0.95

Test Accuary

0

50

100

150

200

Epoch

10

3

10

2

10

1

10 0

Train loss

CIFAR10-ResNet34

0

50

100

150

200

Epoch

10

5

10

4

10

3

10

2

10

1

Step size

0

50

100

150

200

Epoch

0.50

0.55

0.60

0.65

0.70

0.75

Test Accuary

0

50

100

150

200

Epoch

10

3

10

2

10

1

10 0

Train loss

CIFAR100-ResNet34

0

50

100

150

200

Epoch

10

4

10

2

10 0

Step size

SGD-M

AdaGrad

DecSPS

SPS

Adam

Algorithm 5 AdaSPS (DL)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents two step-sizes, AdaSPS and AdaSLS, and theoretical analyses of PSGD with AdaSPS/AdaSLS. It also presents numerical results to support the analyses. The contribution of the paper is to provide AdaSPS and AdaSLS step-sizes.\n\nSTRENGTHS:\nThe strength of the paper is to provide two step-sizes, AdaSPS and AdaSLS, and to apply them to PSGD. In practice, the modifications, AdaSVRPS and AdaSVRLS, accelerate the existing methods.\n\nWEAKNESSES:\n- This paper considers only convex optimization, although optimization problems in deep learning are nonconvex.\n- There seem to be some doubts in Lemmas and Theorems. I would like to have solutions for the doubts (Please see Questions).  \n- Numerical results are insufficient since it does not consider optimization problems in deep neural networks.\n\nQUESTIONS:\n- AdaSLS and Lemma 17 (AdaSPS_main+appendix.pdf): Using the Armijo condition (4) in AdaSLS is interesting. Here, I check Lemma 17 (AdaSPS_main+appendix.pdf). However, I doubt that there exists a lower bound $(1-\\rho)/L$ of the step-size $\\gamma_t$ satisfying (4). In  fact, we can provide counter-examples such that the step-size $\\gamma_t$ satisfying (4) does not have any lower bounds. Consider $f(x) = x^2$ and apply it to the Armijo condition (4). Then, we can set a sufficiently small $\\gamma_t$.\n- Armijo condition (4): Related to the above comment, Figure 5 in [32] indicates that there is a possibility such that $\\gamma_t$ satisfying (4) converges to 0. Hence, we have counter-examples of Lemma 17 such that there does not exist a lower bound of the step-size $\\gamma_t$ satisfying (4).\n- AdaSPS: AdaSPS uses $\\ell_{i_t}^\\star$. It would be unrealistic in deep learning. Can the authors provide some practical examples of $\\ell_{i_t}^\\star$ and $\\eta_t$ defined by AdaSPS in deep learning?\n- Assumptions: Can the authors provide some examples of satisfying the interpolation condition? \n- Convex optimization: Since optimization problems in deep learning are nonconvex, considering only convex optimization is limited in machine learning society. Can the authors rebut this? \n- Can the authors provide numerical results for training DNNs on the benchmark datasets (CIFAR-100 and ImageNet)? Using only LIVSBM datasets is insufficient to show usefulness of the proposed algorithms.\n\nLIMITATIONS:\nThere is no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to propose robust methods that achieves optimal rates in both\nstrongly-convex or convex and interpolation or non-interpolation settings.\nSpecifically, they propose AdaSPS, a modification of $SPS_{max}$ with\nan AdaGrad-like denominator (replacing the gradient norms with function values) in\nthe stepsize, and AdaSLS, a combination of AdaGrad-Norm and line search\nmethods.\n\n- For convex functions,\nAdaSPS and AdaSLS achieve a $O(1/\\epsilon^2)$ convergence rate assuming individual\nsmoothness and bounded domain.\nWhen interpolation is assumed, both methods get rid of the bounded domain\nassumption (but additionally require individual convexity and AdaSPS requires\nthe exact minimal function values) and achieve a $O(1/\\epsilon)$ convergence\nrate.\n- For strongly-convex functions, individual strong-convexity and smoothness\nare required. AdaSPS and AdaSLS achieve a $O(1/\\epsilon^2)$ rate without interpolation.\nWith interpolation, they achieve linear convergence rates.\n- Furthermore, the author combine the proposed methods with variance reduction techniques,\nand improves the rate for strongly-convex and convex settings to $O(1/\\epsilon)$\nwithout interpolation.\n\nThe proposed methods are also evaluated by numerical experiments.\n\nSTRENGTHS:\n1. The paper has clear logic and is well-written, making it easy to follow.\n2. In all considered settings (convex/strongly-convex and interpolation/non-interpolation),\nthe proposed AdaSPS and AdaSLS match the rate of well-tuned SGD in all settings\nexcept for the case when we have strongly-convexity and non-interpolation.\nImportantly, AdaSLS achieves this without requiring knowledge of any problem-dependent parameters.\n3. The intuition behind the proposed variance reduction framework is interesting,\nand could potentially motivate new algorithms.\n\nWEAKNESSES:\n1. The related work on interpolation is somehow insufficient. One very\n  relevant work [1] is missing. Importantly, [1] also proposed combining AdaGrad\n  with line search, similar to AdaSLS, and provided theoretical guarantees in convex setting.\n  The difference is that AdaSLS uses a minimum operator for all past stepsizes.\n  Usually, non-diminishing stepsizes work better in practice, so it would also\n  make sense to compare them in the experiments.\n  2. I cannot find any theorems or proofs that show the results of AdaSVRPS/AdaSVRLS \n  in strongly-convex and convex settings with interpolation,\n  as listed in Table 1. This is the reason why I give\n  a 1 in the soundness assessment.\n  3. Without variance reduction, the paper claims that their algorithms match the convergence rate of SGD in\n  many settings, however, the assumptions are different. Individual smoothness\n  and individual (strongly-)convexity are additionally assumed in many theorems in this work,\n  making it hard to compare them to classical results of SGD (also AdaGrad-Norm).\n  I wonder if there are lower bounds for the individual (strongly-)convexity\n  settings. With variance reduction, Theorem 8 also requires individual convexity,\n  while the compared AdaSVRG does not.\n\n  Minor issues:\n\n  4. Table 1 could contain results of AdaSVRG and SARAH for a more comprehensive comparison\n  of adaptive variance reduction methods. For the interpolation strongly-convex setting\n  where we have linear convergence, the dependence on $\\kappa$, the condition number,\n  is also important.\n  5. In Theorem 5, $T_p$ and $T_l$ are not defined. According to the results in\n  the appendix, they depends on $\\epsilon$ (of order $1/\\log(1+\\epsilon)$, or nearly $1/\\epsilon$,\n  if $c_p$ is chosen poorly).\n  6. Usually the scalar version of AdaGrad is referred as AdaGrad-Norm. Calling\n  it AdaNorm is not common, especially when there are other algorithms called AdaNorm.\n\n\n  **References:** \n\n  [1] Vaswani, Sharan, et al. \"Adaptive gradient methods converge faster with over-parameterization (but you should do a line-search).\" arXiv preprint arXiv:2006.06835 (2020).\n\nQUESTIONS:\nI would appreciate it if the author could address the concerns listed\n  as weaknesses 1-3. \n\nAdditionally, I wonder if we could change the added function value\n  gaps (under the square root) in the denominator of AdaSPS stepsize to the\n  squared norm of gradients (like AdaGrad) and obtain a similar result. Is the use\n  of function value gap necessary?\n\nLIMITATIONS:\nI think the main limitation is the sub-optimality in the non-interpolation\n  strongly-convex setting, which is also a known hard problem for adaptive methods.\n  This is metioned in the Conclusion and future work section of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Adagrad-norm type update into stochastic line search (SLS) and stochastic Polyak step size (SPS) (namely AdaSLS and AdaSPS) that guarantee convergence in non-interpolating convex scenario with a $\\mathcal{O}(\\frac{1}{\\epsilon^2})$ rate. Then it shows that AdaSLS and AdaSPS  converge linearly when interpolation holds under strong convexity. Finally, it combines variance reduction with AdaSLS/AdaSPS to improve the convergence to $\\mathcal{O}(n+\\frac{1}{\\epsilon})$ for convex losses.\n\nSTRENGTHS:\n1. Overall, the paper is well-written. The comparisons with existing literature results are thorough. \n\n2. The introduction of loopless variance reduction, together with AdaGrad and SPS/SLS into a single framework is novel.\n\nWEAKNESSES:\n1. There is no improvement in the convergence rates. DecSPS was introduced to make SPS converge in non-interpolating settings by forcing the step size to be monotonic-decreasing. The proposed AdaSPS essentially replaces the sequence $c_k$ in DecSPS [1] with Adagrad-Norm, and the step size is also monotonically decreasing, resulting in the same rate as DecSPS. In the interpolating settings, SLS  [2] and SPS [3] already converge without the step size being monotonic decreasing. I don’t see the improvements in AdaSPS/AdaSLS over previous methods under interpolation or non-interpolation. \n\n2. In the strongly-convex interpolating case, AdaSPS/AdaSLS converges linearly with some requirements given in Corollary 6. SPS and SLS have the same convergence rate without those additional requirements. In fact, the constants $c_p$ and $c_l$ associated with AdaSPS and AdaSLS respectively depend on the sample at first iteration. I don’t see how this is always satisfied. Does it mean that $c_p$  and $c_l$ need to change every time when running the algorithm? In addition to this, AdaSPS/AdaSLS assumes that each function is strongly convex, this is a very strong assumption that does not appear in the analysis of SPS and SLS. Does it mean only optimizing one function is sufficient as they all share the same unique global optimum under interpolation? \n\n3. The experiments are not promising. Figure 1 just shows that AdaSPS performs worse than SPS under interpolation, and performs worse than or similarly to DecSPS under non-interpolation. If there already exists some better alternatives in either setting, what is the benefit of using AdaSPS?\n\nQUESTIONS:\nI vote for rejection due to the following reasons:\n\n1. No improvements in rates over previous works in either interpolation or non-interpolation scenario.\n\n2. Weak empirical results. The gain in the proposed method is insignificant.\n\n3. The analysis requires some strong assumptions and additional requirements to obtain the same rate as previous works.\n\n[1] Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solution\n\n[2] Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates\n\n[3] Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two new variants of SPS and SLS, called AdaSPS and AdaSLS, which provide convergence in non-interpolation settings for convex and strongly convex functions when training over-parameterized models. AdaSLS requires no knowledge of problem-dependent parameters, and AdaSPS requires a lower bound of the optimal function value as input. In addition, the paper studies a new variance reduction technique for AdaSPS and AdaSLS and proves the gradient complexity for convex functions, which improves upon the rates of AdaSPS and AdaSLS without variance reduction in the non-interpolation settings. This matches the fast rates of AdaSVRG and SARAH without the inner-outer-loop structure, which is easier to implement and analyze. The authors provides numerical experiments on synthetic and binary classification on LIBSVM datasets.\n\nSTRENGTHS:\nThe two algorithms AdaSPS and AdaSLS are interesting. The authors provide the convergence analysis for them, especially in non-interpolation setting the algorithms attained the classical convergence rate for convex functions. The variance reduction technique has the same complexity in expectation for convex functions as the rate of AdaSVRG and SARAH.\n\nWEAKNESSES:\nThe theoretical rate has comparison with SARAH but it is not included in the experiment. The variance technique and the one loop structure is not entirely new since it already appeared in the PAGE algorithm. The authors may consider discussing that as well. A natural question is why practitioners should use the new variance reduction algorithms while it is more complicated than the well established methods in convex settings. Thus more extensive experiments would help to demonstrate the effectiveness of the algorithms.\n\nQUESTIONS:\nSince the algorithm requires more parameter than classical methods, could you explain how they are chosen in the experiments/ suggested in practical settings? e.g. $\\mu_F$, $c_p$, $c_l$, $\\gamma_t$ , $p_t$? \n\nHow the lower bound information of 0 may affect the performance of the algorithm (theoretically and empirically), since the theory uses the correct lower optimal values. \n\n---\nI thank the authors for your rebuttal. My recommendation remains the same.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}