Title: Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction
Venue: NeurIPS 2024
Accepted: yes
Keywords: convex optimization

stochastic optimization

adaptive methods

universal algorithms

acceleration

variance reduction

AdaGrad

SVRG

weakly smooth functions

Hölder condition

inexact oracle

complexity estimates
Abstract: We present adaptive gradient methods (both basic and accelerated) for solving
convex composite optimization problems in which the main part is approximately
smooth (a.k.a. $(\delta, L)$-smooth) and can be accessed only via a
(potentially biased) stochastic gradient oracle.
This setting covers many interesting examples including Hölder smooth problems
and various inexact computations of the stochastic gradient.
Our methods use AdaGrad stepsizes and are adaptive in the sense that they do
not require knowing any problem-dependent constants except an estimate of the
diameter of the feasible set but nevertheless achieve the best possible
convergence rates as if they knew the corresponding constants.
We demonstrate that AdaGrad stepsizes work in a variety of situations
by proving, in a unified manner, three types of new results.
First, we establish efficiency guarantees for our methods in the classical
setting where the oracle's variance is uniformly bounded.
We then show that, under more refined assumptions on the variance,
the same methods without any modifications enjoy implicit variance
reduction properties allowing us to express their complexity estimates in
terms of the variance only at the minimizer.
Finally, we show how to incorporate explicit SVRG-type variance reduction into
our methods and obtain even faster algorithms.
In all three cases, we present both basic and accelerated algorithms
achieving state-of-the-art complexity bounds.
As a direct corollary of our results, we obtain universal stochastic gradient
methods for Hölder smooth problems which can be used in all situations.

Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction

Abstract

We present adaptive gradient methods (both basic and accelerated) for solving convex composite optimization problems in which the main part is approximately smooth (a.k.a. ( δ, L ) -smooth) and can be accessed only via a (potentially biased) stochastic gradient oracle. This setting covers many interesting examples including Hölder smooth problems and various inexact computations of the stochastic gradient. Our methods use AdaGrad stepsizes and are adaptive in the sense that they do not require knowing any problem-dependent constants except an estimate of the diameter of the feasible set but nevertheless achieve the best possible convergence rates as if they knew the corresponding constants. We demonstrate that AdaGrad stepsizes work in a variety of situations by proving, in a unified manner, three types of new results. First, we establish efficiency guarantees for our methods in the classical setting where the oracle's variance is uniformly bounded. We then show that, under more refined assumptions on the variance, the same methods without any modifications enjoy implicit variance reduction properties allowing us to express their complexity estimates in terms of the variance only at the minimizer. Finally, we show how to incorporate explicit SVRG-type variance reduction into our methods and obtain even faster algorithms. In all three cases, we present both basic and accelerated algorithms achieving state-of-the-art complexity bounds. As a direct corollary of our results, we obtain universal stochastic gradient methods for Hölder smooth problems which can be used in all situations.

1 Introduction

Motivation. Gradient methods are among the most popular and efficient optimization algorithms for solving machine learning problems. To achieve the best convergence speed for these algorithms, their stepsizes needs to be chosen properly. While there exist various theoretical recommendations, dictated by the convergence analysis, on how to select stepsizes based on various problem-dependent parameters, they are usually impractical because the corresponding constants may be unknown or their worst-case estimates might be too pessimistic. Furthermore, every applied problem usually belongs to multiple problem classes at the same time, and it is not always evident in advance which of them better suits the concrete problem instance one works with. For classical optimization algorithms, this problem is typically resolved by using a line search. This is a simple yet powerful mechanism which automatically chooses the best stepsize by checking at each iteration a certain condition involving the objective value, its gradient, etc.

However, the line-search approach is usually unsuitable for problems of stochastic optimization, where gradients are observed with random noise (unless some extra assumptions are made, see [57]). For these problems, it is common instead to apply so-called adaptive methods which set up their

∗

stepsizes by simply accumulating on-the-fly certain information about observed stochastic gradients. The first such an algorithm, AdaGrad [17, 39], was obtained from theoretical considerations but quickly inspired several other heuristic methods like RMSProp [56] and Adam [32] that are now at the forefront of training machine learning models.

Excellent practical performance of adaptive methods on various applied problems naturally sparked a lot of theoretical interest in these algorithms. An important observation was done by Levy, Yurtsever, and Cevher [34] who showed that AdaGrad possesses a certain universality property, in the sense that it works for several problem classes simultaneously. Specifically, they showed that AdaGrad converges both for nonsmooth problems with bounded gradient and also for smooth problems with Lipschitz gradient, without needing to know neither the corresponding Lipschitz constants, nor the oracle's variance but enjoying the rates which are characteristic for algorithms which have the knowledge of these constants. They also presented an accelerated version of AdaGrad with similar properties. An independent version of the accelerated AdaGrad including diagonal scaling was proposed by Deng, Cheng, and Lan [12]. Further improvements and generalization of these ideas were considered in [18, 28, 30].

Nonsmooth and smooth problems are the extremes of the more general Hölder class of problems. The fact that AdaGrad methods simultaneously work for these two extreme cases does not seem to be a coincidence and suggests that these algorithms should work more generally for any problem with intermediate level of smoothness. Some further confirmations to this were recently provided in [48] although in a rather restricted setting of deterministic problems and only for the basic AdaGrad method. The stochastic case and acceleration were constituting an open problem which was recently resolved in [49] for a slightly modified AdaGrad stepsize (see (4)).

All the previously discussed results were proved only for the classical stochastic optimization setting where the variance of stochastic gradients is assumed to be uniformly bounded. In a recent work, Attia and Koren [2] showed that the basic AdaGrad method for smooth problems works under the more general assumption when the variance is bounded by a constant plus a multiple of the squared gradient norm. On a related note, it was also shown recently that AdaGrad stepsizes can be used inside gradient methods with SVRG-type variance-reduction. The first such an algorithm was proposed in [16]. The accelerated SVRG method enjoying optimal worst-case oracle complexity for smooth finite-sum optimization problems was later presented in [36].

Contributions. In this work, we further extend the results mentioned above by demonstrating that AdaGrad stepsizes are even more universal than was shown previously in the literature. Specifically, we consider the composite optimization problem where the main part is approximately smooth (a.k.a. ( δ, L ) -smooth) and can be accessed only via a (potentially biased) stochastic gradient oracle. This setting is more general than typically considered in the literature on adaptive methods and covers many interesting examples, including smooth, nonsmooth and, more generally, Hölder smooth problems, problems in which the objective function is given itself as another optimization problem whose solution can be computed only approximately, etc.

Our contributions can be summarized as follows:

We start, in Section 3, with identifying the key property of AdaGrad stepsizes, which allows us to apply these stepsizes, in a unified manner, in a variety of situations we consider later. We present our two mains algorithms, UniSgd and UniFastSgd which are the classical stochastic gradient method (SGD) and its accelerated version, respectively, equipped with AdaGrad stepsizes.

We then establish, in Section 4, efficiency guarantees for these methods in the classical setting where the oracle's variance is assumed to be uniformly bounded.

In Section 5, we complement these results by showing that, under additional assumptions that the variance is itself approximately smooth w.r.t. the objective function, the same UniSgd and UniFastSgd without any modifications enjoy implicit variance reduction properties allowing us to express their complexity estimates in terms of the variance only at the minimizer.

Under the additional assumption that one can periodically compute the full (inexact) gradient of the objective function, we show, in Section 6, how to incorporate explicit SVRG-type variance reduction into our methods, obtaining new UniSvrg and UniFastSvrg algorithms which enjoy even faster convergence rates by completely eliminating the variance.

Our results are summarized in Table 1 (in the BigO-notation). In all the situations, we present both basic and accelerated algorithms whose only essential parameter is an estimate D of the diameter

of the feasible set; the methods automatically adapt to all other problem-dependent constants. In a number of special cases, our algorithms achieve known state-of-the-art complexity bounds, but not restricted to those special cases. In Section 7, we illustrate the significance of our results by demonstrating that complexities for our methods on stochastic optimization problems with Hölder smooth components can be obtained as simple corollaries from our main results.

2 Preliminaries

Notation. We work in the space R d equipped with the standard inner product ⟨· , ·⟩ and a certain Euclidean norm: ∥ x ∥ := ⟨ Bx,x ⟩ 1 / 2 , where B is a fixed positive definite matrix. The dual norm is defined in the standard way: ∥ s ∥ ∗ := max ∥ x ∥ =1 ⟨ s, x ⟩ = ⟨ s, B -1 s ⟩ 1 / 2 .

For a convex function ψ : R d → R ∪ { + ∞} , its (effective) domain is the following set: dom ψ := { x ∈ R d : ψ ( x ) < + ∞} . By ∂ψ ( x ) , we denote the subdifferential of ψ at a point x ∈ dom ψ ; the specific subgradients are typically denoted by ∇ ψ ( x ) .

A convex function f : R d → R is called ( ν, H ) -Hölder smooth for some ν ∈ [0 , 1] and H ≥ 0 iff ∥∇ f ( x ) - ∇ f ( y ) ∥ ∗ ≤ H ∥ x -y ∥ ν for all x, y ∈ R d and all ∇ f ( x ) ∈ ∂f ( x ) , ∇ f ( y ) ∈ ∂f ( y ) . Apart from the special case of ν = 0 , such a function f is differentiable at every point, i.e., ∂f ( x ) is a singleton. A (1 , L ) -Hölder smooth function is usually called L -smooth.

For a convex function ψ : R d → R ∪{ + ∞} , point x ∈ R d , vector g ∈ R d , and coefficient M ≥ 0 , by Prox ψ ( x, g, M ) := argmin y ∈ dom ψ {⟨ g, y ⟩ + ψ ( y ) + M 2 ∥ y -x ∥ 2 } , we denote the proximal mapping. When M = 0 , we allow the solution to be chosen arbitrarily.

For a convex function f : R d → R , points x, y ∈ R d and ∇ f ( x ) ∈ ∂f ( x ) , we denote the Bregman distance by β ∇ f ( x ) f ( x, y ) := f ( y ) -f ( x ) -⟨∇ f ( x ) , y -x ⟩ ( ≥ 0) . When the specific subgradient ∇ f ( x ) is clear from the context, we use the simplified notation β f ( x, y ) .

The positive part of t ∈ R is [ t ] + := max { t, 0 } . For τ > 0 , we also use log + τ := max { 1 , log τ } .

Problem Formulation. In this paper, we consider the composite optimization problem

where f : R d → R is a convex function, and ψ : R d → R ∪{ + ∞} is a proper closed convex function which is assumed to be sufficiently simple in the sense that the proximal mapping Prox ψ can be easily computed. We assume that this problem has a solution which we denote by x ∗ .

To quantify the smoothness level of the objective function, we use the following assumption:

Algorithm 1 ( s x N , x N , M N ) ∼ = UniSgd p g,ψ ( x 0 , M 0 , N ; D )

Input: Oracle p g , comp. part ψ , point x 0 ∈ dom ψ , coefficient M 0 , iteration limit N , diameter D .

1: g 0 ∼ = p g ( x 0 ) .

2: for k = 0 , . . . , N -1 do

3: x k +1 = Prox ψ ( x k , g k , M k ) , g k +1 ∼ = p g ( x k +1 ) .

4: M k +1 = M + ( M k , D 2 , x k , x k +1 , g k , g k +1 )

5: return ( s x N , x N , M N ) , where s x N := 1 N ∑ N i =1 x i .

Assumption 1. The function f in problem (1) is approximately smooth: there exist constants L f , δ f ≥ 0 and s f : R d → R , s g : R d → R d such that, for any x, y ∈ R d , β f, s f, s g ( x, y ) := f ( y ) -s f ( x ) -⟨ s g ( x ) , y -x ⟩ satisfies the following inequality: 0 ≤ β f, s f, s g ( x, y ) ≤ L f 2 ∥ x -y ∥ 2 + δ f .

Assumption 1 is well-known in the literature under the name ( δ, L ) -oracle and was originally introduced in [15]. It covers many interesting examples. For instance, if f is L -smooth, then Assumption 1 is satisfied with s f = f , s g = ∇ f , δ f = 0 and L f = L . More generally, if the function f is ( ν, H f ( ν )) -Hölder smooth, then Assumption 1 is satisfied with s f = f , s g = ∇ f (arbitrary selection of subgradients), any δ f > 0 and L f := [ 1 -ν 2(1+ ν ) δ f ] 1 -ν 1+ ν [ H f ( ν )] 2 1+ ν (see Theorem 13). If f can be uniformly approximated by an L -smooth function ϕ , i.e., ϕ ( x ) ≤ f ( x ) ≤ ϕ ( x )+ δ , then Assumption 1 is satisfied with s f = ϕ , s g = ∇ ϕ and δ f = δ . If f represents another auxiliary optimization problem with a strongly concave objective, e.g., f ( x ) = max u Ψ( x, u ) , whose solution s u ( x ) can only be found with accuracy δ , then f satisfies Assumption 1 with s f ( x ) = Ψ( x, s u ( x )) , s g ( x ) = ∇ u Ψ( x, s u ( x )) and δ f = δ . For more details and other interesting examples, we refer the reader to [15].

In what follows, we assume that we have access to an unbiased stochastic oracle p g for s g . Formally, this is a pair p g = ( g, ξ ) consisting of a random variable ξ and a mapping g : R d × Im ξ → R d (with Im ξ being the image of ξ ). When queried at a point x , the oracle automatically generates an independent copy ξ of its randomness and then returns p g x = g ( x, ξ ) (notation: p g x ∼ = p g ( x ) ). We call g and ξ the function component and the random variable component of p g , respectively. At this point, we only assume that our stochastic oracle p g is un unbiased estimator of s g , and later make various assumptions on its variance.

Another important assumption on problem (1), that we need in our analysis, is the boundedness of the feasible set dom ψ .

Assumption 2. There exists D > 0 such that ∥ x -y ∥ ≤ D for any x, y ∈ dom ψ .

Assumption 2 is rather standard in the literature on adaptive methods for stochastic convex optimization (see [16, 18, 30, 34, 36, 49]) and can always be ensured with D = 2 R 0 whenever one has the knowledge of an upper bound R 0 on the distance from the initial point x 0 to the solution x ∗ . To that end, it suffices to rewrite the problem (1) in the following equivalent form: min x ∈ dom ψ D [ f ( x ) + ψ D ( x )] , where ψ D is the sum of ψ and the indicator function of the ball B 0 := { x ∈ R d : ∥ x -x 0 ∥ ≤ R 0 } . Note that this transformation keeps the function ψ D reasonably simple as its proximal mapping can be computed via that of ψ by solving a certain one-dimensional nonlinear equation, which can be done very efficiently by Newton's method (at no extra queries to the stochastic oracle); in some special cases, the corresponding nonlinear equation can even be solved analytically, e.g., when ψ = 0 , the proximal mapping of ψ D is simply the projection on B 0 .

Throughout this paper, we refer to D from Assumption 2 as the diameter of the feasible set, and assume that its value is known to us. This will be the only essential parameter in our methods.

3 Main Algorithms and Stepsize Update Rules

We now present our two main algorithms for solving problem (1): UniSgd (Algorithm 1), and its accelerated version, UniFastSgd (Algorithm 2). Except the specific choice of the stepsize coefficients M k , both algorithms are rather standard: the first one is the classical SGD method, and the second one is the classical accelerated gradient method for stochastic optimization [33], also known as the Method of Similar Triangles (see, e.g., Section 6.1.3 in [46]).

Both methods are expressed in terms of a certain abstract stepsize update rule M + ( · ) defined as follows. Given the current stepsize coefficient M ≥ 0 , constant Ω > 0 (the scaled squared diameter),

▷

e.g.,

(3)

=

√

M

2

k

+

1

D

2

∥

g

k

+1

-

g

k

∥

2

∗

.

Algorithm 2 UniFastSgd p g,ψ ( x 0 ; D )

Input: Stochastic oracle p g , composite part ψ , point x 0 ∈ dom ψ , diameter D .

1:

v

0

=

x

0

,

M

0

=

A

0

= 0

.

2: for k = 0 , 1 , . . . do

3: a k +1 = 1 2 ( k +1) , A k +1 = A k + a k +1 .

4: y k = A k A k +1 x k + a k +1 A k +1 v k , g y k ∼ = p g ( y k ) .

5: v k +1 = Prox ψ ( v k , g y k , M k a k +1 ) .

current point x ∈ dom ψ with the stochastic gradient p g x ∼ = p g ( x ) , next iterate p x + = x + ( p g x ) ∈ dom ψ (which is the result of the deterministic function applied to p g x ), and the corresponding stochastic gradient p g x + ∼ = p g ( p x + ) , the update rule computes x M + = M + ( M, Ω , x, p x + , p g x , p g x + ) (deterministic function of its arguments) such that x M + ≥ M and the following inequality holds for any Ď M > c 2 L f :

where p ∆( x M + ) := β f, s f, s g ( x, p x + ) + ⟨ s g ( x ) -p g x , p x + -x ⟩ -x M + 2 ∥ p x + -x ∥ 2 , c 1 , c 2 , c 3 , c 4 > 0 are some absolute constants, and Var p g ( x ) := E ξ [ ∥ g ( x, ξ ) -s g ( x ) ∥ 2 ∗ ] is the variance of p g . The expectations in (2) are taken w.r.t. the randomness ( ξ, ξ + ) coming from p g x ≡ g ( x, ξ ) , p g x + ≡ g ( p x + , ξ + ) .

The main example is the following AdaGrad rule:

For this rule, we have c 1 = 5 2 , c 2 = 4 , c 3 = 6 , c 4 = 2 (see Lemma 20). Another interesting example recently suggested in [49] is x M + found from the equation

This equation admits a unique solution which can be easily written down in closed form (see Lemma E.1 in [49]). For this rule, we have c 1 = 1 , c 2 = 2 , c 3 = 6 , c 4 = 2 (see Lemma 21).

Inequality (2) is the only property we need from the stepsize update rule to establish all forthcoming results. This inequality is exactly what is typically used inside the convergence proofs for stochastic gradient methods with predefined stepsizes M k ≡ Ď M (in which case M = x M + = Ď M ), where Ď M depends on problem-dependent constants. The key property of AdaGrad stepsizes (either (3) or (4)) is that they ensure the same inequality but now Ď M is the virtual stepsize existing only in the theoretical analysis. The price for this is the extra error term [min { x M + , Ď M } -M ] + Ω appearing in the right-hand side of (2). The crucial property of this error term is that it is telescopic, ∑ k i =0 [min { M i +1 , Ď M } -M i ] + Ω = [min { M k +1 , Ď M } -M 0 ] + Ω (see Lemma 18) and therefore its total cumulative impact is always bounded by the controllable constant Ď M Ω . Although a number of other works on theoretical analysis of AdaGrad methods for smooth optimization use some similar ideas about the virtual stepsize (e.g., [30, 34, 36]), this is the first time one has abstracted away all the technical details and identified the specific inequality (2) responsible for the universality of AdaGrad.

4 Uniformly Bounded Variance

In this section, we assume that the variance of our stochastic oracle is uniformly bounded.

Assumption 3. For the stochastic oracle p g , we have σ 2 := sup x ∈ dom ψ Var p g ( x ) < + ∞ , where Var p g ( x ) := E ξ [ ∥ g ( x, ξ ) -s g ( x ) ∥ 2 ∗ ] .

Under this assumption, we can establish the following efficiency estimates for our UniSgd and UniFastSgd methods (the proofs are deferred to Appendix C).

Theorem 4. Let Algorithm 1 with M 0 = 0 be applied to problem (1) under Assumptions 1-3. Then, for the point s x N generated by the algorithm, we have

Theorem 5. Let Algorithm 2 be applied to problem (1) under Assumptions 1-3. Then, for any k ≥ 1 ,

We see that, in contrast to UniSgd , the accelerated algorithm UniFastSgd is not robust to the oracle's errors: it accumulates them with time at the rate of O ( kδ ) . This is not surprising since the same phenomenon also occurs in the classical accelerated gradient method, even when the oracle is deterministic and the algorithm has the knowledge about all constants (see [15]).

The complexity results from Theorems 4 and 5 are similar to those from [13]. However, it is important that our methods are adaptive and do not require knowing the constants L f and σ .

In the specific case when δ f = 0 , we recover the same convergence rates as in [30, 34], although our methods work for the more general composite optimization problem and, in contrast to [34], do not require that ∇ f ( x ∗ ) = 0 .

5 Implicit Variance Reduction

The assumption of uniformly bounded variance may not hold for some problems, or the corresponding constant σ 2 might be quite large, which is why there has recently been a growing interest in various alternative variance bound assumptions [5, 22, 24, 29, 42, 54, 59]. One interesting option is expressing complexity bounds via the variance at the minimizer, σ 2 ∗ := Var p g ( x ∗ ) , assuming that the stochastic oracle p g satisfies some extra smoothness conditions. Let us show that, for our Algorithms 1 and 2, we can also establish such bounds, moreover, this can be done without any modifications to the algorithms .

In this section, we study problem (1) under Assumptions 1 and 2 and also under the following additional smoothness assumption on the variance:

Assumption 6. There exist δ p g , L p g ≥ 0 such that Var p g ( x, y ) ≤ 2 L p g [ β f, s f, s g ( x, y ) + δ p g ] for any x, y ∈ R d , where Var p g ( x, y ) := E ξ [ ∥ [ g ( x, ξ ) -g ( y, ξ )] -[ s g ( x ) -s g ( y )] ∥ 2 ∗ ] .

Note that Var p g ( x, y ) is the usual variance of the estimator g ( x, ξ ) -g ( y, ξ ) which uses the same randomness ξ for both arguments. Hence, Var p g ( x, y ) ≤ E [ ∥ g ( x, ξ ) -g ( y, ξ ) ∥ 2 ∗ ] for any x, y . Furthermore, if p g b is the mini-batch version of p g of size b (i.e., the average of b i.i.d. samples of p g ( x ) at any point x ), then Var p g b ( x, y ) = 1 b Var p g ( x, y ) for any x, y .

For instance, if f ( x ) = E ξ [ f ξ ( x )] , where each function f ξ is convex and ( δ ξ , L ξ ) -approximately smooth with components ( s f ξ , s g ξ ) , then, the stochastic gradient oracle p g , defined by g ( x, ξ ) := s g ξ ( x ) satisfies Assumption 6 with s f ( x ) = E ξ [ s f ξ ( x )] , s g ( x ) = E ξ [ s g ξ ( x )] , and δ p g = 1 L max E ξ [ L ξ δ ξ ] ( ≤ E ξ [ δ ξ ]) , L p g = L max , where L max := sup ξ L ξ (see Lemma 16). Furthermore, if p g b is the mini-batch version of p g of size b , then p g b satisfies Assumption 6 with the same δ p g b = δ p g but L p g b = 1 b L p g = 1 b L max which can be much smaller than L max when b is large enough.

Under the new assumption on the variance, UniSgd enjoys the following convergence rate (see Appendix D.1 for the proof).

Theorem 7. Let Algorithm 1 with M 0 = 0 be applied to problem (1) under Assumptions 1, 2 and 6, and let σ 2 ∗ := Var p g ( x ∗ ) . Then, for the point s x N produced by the method, we have

Comparing the above result with Theorem 4, we see that we have essentially replaced the uniform bound σ with the more refined one σ ∗ at the cost of replacing L f with L f + L p g and δ f with δ f + δ p g .

This corresponds to classical results on the usual SGD for which we know all problem dependentconstants. However, our method is universal and works automatically under both assumptions from the previous section and the current one, and therefore enjoys the best among the rates given by Theorems 4 and 7.

For the accelerated algorithm, we have the following result (whose proof is located in Appendix D.2).

Theorem 8. Let Algorithm 2 be applied to problem (1) under Assumptions 1, 2 and 6, and let σ 2 := Var p ( x ∗ ) . Then, for any k ≥ 1 , we have

∗ g

Comparing our previous complexity bound for UniFastSgd under the assumption on uniformly bounded variance (Theorem 5) with the bound from Theorem 8, we see that, instead of simply replacing σ with σ ∗ , L f with L f + L p g and δ f with δ f + δ p g , which was the case for the basic method, the situation is now not that simple. Specifically, the L f and L p g terms now converge at different rates: O ( 1 k 2 ) and O ( 1 k ) , respectively. While this may seem strange at first, this behavior is actually unavoidable, at least in the case when δ f = δ p g = 0 (see, e.g., Section E in [59]). For the case when δ f = δ p g = 0 , the complexity result from Theorem 8 is similar to the results for the Accelerated SGD algorithm from [59]. However, the latter paper studies a specific setting where f ( x ) = E [ f ξ ( x )] , where each component f ξ is L max -smooth and then assumes that f is also L max -smooth, instead of working with the constant L f which can be much smaller than L max . A similar separation of the constants L f and L p g , which we do, was recently considered in [24], where the authors obtained some similar rates to our Theorem 8. However, it is important that, unlike the algorithms considered in [24, 59], our UniFastSgd is universal and does not require knowing any problem-dependent constants except D . Furthermore, our results are more general because we allow the oracle to be inexact.

6 Explicit Variance Reduction with SVRG

Let us now show that we can also incorporate explicit SVRG-type variance reduction into our methods. In this section, we consider problem (1) under Assumptions 1, 2 and 6. All the proofs are deferred to Appendix E.

In addition to the stochastic oracle p g , we now assume that we can also compute the (approximate) full-gradient oracle s g . This allows us to define the following auxiliary SVRG oracle induced by p g with center ˜ x ∈ R d (notation p G = SvrgOrac p g, s g (˜ x ) ) as the oracle with the same random variable component ξ as p g and the function component given by G ( x, ξ ) = g ( x, ξ ) -g (˜ x, ξ ) + s g (˜ x ) .

Our UniSvrg method is presented in Algorithm 3. This is the classical epoch-based SVRG algorithm which can be seen as the adaptive version of the SVRG++ method from [1]. A similar scheme was suggested in [16], however, instead of accumulating gradient differences as in (3), their method accumulates gradients and therefore does not work without the additional assumption of ∇ f ( x ∗ ) = 0 (which may not hold for constrained optimization).

Let us now present the complexity guarantees. To do so, we first need to introduce, one more assumption we need in our analysis.

Assumption 9. The variance of p g satisfies Var p g ( x, y ) ≤ 4 L p g [ β ∇ f ( x ) f ( x, y ) + 2 δ p g ] for any x, y ∈ R d and any ∇ f ( x ) ∈ ∂f ( x ) .

Assumption 9 is very similar to Assumption 6. The only difference between them is that the former contains the standard Bregman distance in the right-hand side, while the latter contains its approximation β f, s f, s g ( x, y ) involving the approximate function value s f ( x ) and the approximate gradient s g ( x ) . Nevertheless, both assumptions are actually satisfied for the main examples we discussed after introducing Assumption 6 (see Lemma 16).

Algorithm 4 UniFastSvrg p g, s g,ψ ( x 0 , N ; D )

Input: Oracles p g , s g , composite part ψ , point x 0 ∈ dom ψ , epoch length N , diameter D .

1: ˜ x 0 = Prox ψ ( x 0 , s g ( x 0 ) , 0) , v 0 = x 0 , M 0 = 0 , A 0 = 1 N .

3: a t +1 = A t , A t +1 = A t + a t +1 .

2: for t = 0 , 1 , . . . do √

4: (˜ x t +1 , v t +1 , M t +1 ) ∼ = UniTriSvrgEpoch p g, s g,ψ (˜ x t , v t , M t , A t , a t +1 , N ; D ) .

Algorithm 5 (˜ x + , v + , M + ) ∼ = UniTriSvrgEpoch p g, s g,ψ (˜ x, v 0 , M 0 , A, a, N ; D )

Input: Oracles p g , s g , comp. part ψ , points ˜ x, v 0 , coefficients M 0 , A, a , epoch length N , diameter D .

1: A + = A + a , x 0 = A A + ˜ x + a A + v 0 , p G = SvrgOrac p g, s g (˜ x ) , G x 0 ∼ = p G ( x 0 ) .

2: for k = 0 , . . . , N -1 do

3: v k +1 = Prox ψ ( v k , G x k , M k a ) .

4: x k +1 = A A + ˜ x + a A + v k +1 , G x k +1 ∼ = p G ( x k +1 ) .

6: return ( s x N , v N , M N ) , where s x N := 1 N ∑ N k =1 x k .

Theorem 10. Let UniSvrg (as defined by Algorithm 3) be applied to problem (1) under Assumptions 1, 2, 6 and 9. Then, for any t ≥ 1 and s c 3 := max { c 3 , 1 } , we have

To construct ˜ x t , the algorithm needs to make O (2 t ) queries to p g and O ( t ) queries to s g .

We now present an accelerated version of UniSvrg , see Algorithm 4. As UniSvrg , this method is also epoch-based, and its epoch is very similar to UniFastSgd (Algorithm 4) in the sense that it also iterates similar-triangle steps. However, the triangles in UniTriSvrgEpoch are of the form (˜ x, v k , v k +1 ) , i.e., they always share the common vertex ˜ x , in contrast to the triangles ( x k , v k , v k +1 ) in UniFastSgd (in UniTriSvrgEpoch , the role of the average points y k is played by x k ). We note that our UniFastSvrg is essentially the primal version of the VRADA method from [53], but equipped with AdaGrad stepsizes. Alternative accelerated SVRG schemes with AdaGrad stepsizes (3) were recently proposed in [36]; however, they seem to be much more complicated.

The special choice of the initial reference point ˜ x 0 at Line 1 is rather standard and motivated by the desire to keep the initial function residual appropriately bounded: F (˜ x 0 ) -F ∗ ≤ 1 2 L f D 2 + δ f ; the simplest way to achieve this is to make the full gradient step from any feasible point (see Lemma 34).

Theorem 11. Let UniFastSvrg (Algorithm 4) be applied to problem (1) under Assumptions 1, 2 and 6, and let N ≥ 9 . Then, for any t ≥ t 0 := ⌈ log 2 log 3 N ⌉ -1 ( ≥ 0) , it holds that

To construct ˜ x t , the algorithm needs to make O ( Nt ) queries to p g and O ( t ) queries to s g . Assuming that the complexity of querying s g is n times bigger than that of querying p g and choosing N = Θ( n ) , we get the total stochastic-oracle complexity of O ( nt ) .

Note that Theorem 11, unlike Theorem 10, does not require the extra Assumption 9. This suggests that Assumption 9 might be somewhat artificial and could potentially be removed from Theorem 10 as well. However, we do not know how to do it, even in the simplest case when δ f = δ p g = 0 and the algorithm has the knowledge of the constants L f and L p g from Assumptions 1 and 6.

7 Application to Hölder Smooth Problems

To illustrate how powerful our results are, let us quickly consider the specific example of solving the stochastic optimization problem with Hölder smooth components.

Example 12 . Suppose that the function f in problem (1) is the expectation of other functions, f ( x ) = E ξ [ f ξ ( x )] , where each function f ξ is convex and ( ν, H ξ ( ν )) -Hölder smooth. Consider the

standard mini-batch stochastic gradient oracle p g b of size b , defined by g b ( x, ξ [ b ] ) = 1 b ∑ b j =1 ∇ f ξ j ( x ) , where ξ [ b ] := ( ξ 1 , . . . , ξ b ) with b i.i.d. copies of ξ , and ∇ f ξ ( x ) ∈ ∂f ξ ( x ) is an arbitrary selection of subgradients for each ξ . We define H f ( ν ) as the Hölder constant for the function f and H max ( ν ) := sup ξ H ξ ( ν ) as the worst among Hölder constants for each f ξ . Note that we always have H f ( ν ) ≤ E ξ [ H ξ ( ν )] but H f ( ν ) can, in principle, be much smaller than the right-hand side. Also, define σ 2 := sup x ∈ dom ψ Var p g 1 ( x ) ≡ sup x ∈ dom ψ E ξ [ ∥∇ f ξ ( x ) - ∇ f ( x ) ∥ 2 ∗ ] and σ 2 ∗ := Var p g 1 ( x ∗ ) ≡ E ξ [ ∥∇ f ξ ( x ∗ ) - ∇ f ( x ∗ ) ∥ 2 ∗ ] . We assume that the computation of p g b can be parallelized and the computation of ∇ f is n b times more expensive than that of p g b .

To solve the above problem, we can apply any of the methods we presented before. The resulting oracle complexities (in terms of the BigO-notation) are summarized in Table 2; the precise statements the corresponding results and their proofs are deferred to Appendix F.

Note that our problem is characterized by a large number of parameters, ν , H f ( ν ) , H max ( ν ) , σ , σ ∗ . For each combination of these parameters, we get a certain complexity guarantee for each of our methods, and it is impossible to say in advance which combination results in the smaller complexity bound. However, it is not important for our methods since none of them needs to know any of these constants to ensure the corresponding bound. This means that our algorithms are universal : they automatically figure out the best problem class for a specific problem given to them.

8 Experiments

Let us illustrate the performance of our methods in preliminary numerical experiments 2 on solving

where a i , b i ∈ R d , q ∈ [1 , 2] and R > 0 .

This test problem covers several interesting applications. Indeed, if q = 2 , we get the classical Least squares problem. If q = 1 , this is the well-known Support-Vector Machines (SVM) problem. In both cases, the ball-constraint ∥ x ∥ ≤ R acts as a regularizer, and problem (5) is, in fact, equivalent to min x ∈ R d [ f ( x ) + µ 2 ∥ x ∥ 2 ] for a certain µ ≥ 0 (this follows, e.g., from the KKT optimality conditions) such that µ decreases when R increases.

Another interesting application of (5), which we consider in this section, is the polyhedron feasibility problem : find x ∗ ∈ R d , ∥ x ∗ ∥ ≤ R , inside the polyhedron P = { x : ⟨ a i , x ⟩ ≤ b i , i = 1 , . . . , n } . Such a point exists iff f ∗ = 0 . Note that (5) is a problem with Hölder smooth components of degree ν = q -1 . By varying q in (5), we can therefore check the adaptivity of different methods to the unknown to them Hölder characteristics of the objective function.

The data for our problem is generated randomly. First, we generate x ∗ uniformly from the sphere of radius 0 . 95 R centered at the origin. Then, we generate i.i.d. vectors a i with components uniformly distributed on [ -1 , 1] . We then make sure that ⟨ a n , x ∗ ⟩ < 0 by inverting the sign of a n if necessary. Next, we generate positive reals s i uniformly in [0 , -0 . 1 c min ] , where c min := min i ⟨ a i , x ∗ ⟩ < 0 , and set b i = ⟨ a i , x ∗ ⟩ + s i . By construction, x ∗ is a solution of our problem with f ∗ = 0 , and the origin x 0 = 0 lies outside the polyhedron since there exists j (corresponding to c min ) such that b j = c min + s j ≤ 0 . 9 c min < 0 .

0

50000

100000 150000 200000 250000

number of stochastic oracle calls

10

1

10 0

10 1

10 2

10 3

10 4

10 5

f(x)-

f

q = 1

0

25000 50000 75000 100000125000150000

number of stochastic oracle calls

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 7

f(x)-

f

q = 1.3

0

20000

40000

60000

number of stochastic oracle calls

10 2

10 4

10 6

10 8

f(x)-

f

q = 1.6

0

5000

10000

15000

number of stochastic oracle calls

10

6

10

3

10 0

10 3

10 6

10 9

10 12

f(x)-

f

q = 2

UniSgd (ours)

UniFastSgd (ours)

AdaSVRG

UniSvrg (ours)

AdaVRAG

AdaVRAE

FastSvrg

UniFastSvrg (ours)

0

5000

10000

15000

20000

number of stochastic oracle calls

10 5

10 6

10 7

10 8

f(x)-

f

UniSgd

batchsize=64

batchsize=512

batchsize=4096

0

2000

4000

6000

8000

10000

12000

14000

number of stochastic oracle calls

10 1

10 3

10 5

10 7

f(x)-

f

UniFastSgd

batchsize=64

batchsize=512

batchsize=4096

0

5000

10000

15000

20000

25000

30000

35000

number of stochastic oracle calls

10 4

10 5

10 6

10 7

10 8

f(x)-

f

UniSvrg

batchsize=64

batchsize=512

batchsize=4096

0

2500

5000

7500

10000

12500

15000

17500

20000

number of stochastic oracle calls

10 1

10 3

10 5

10 7

f(x)-

f

UniFastSvrg

batchsize=64

batchsize=512

batchsize=4096

We compare UniSvrg (Algorithm 3) against AdaSVRG [16] (with parameters K = 3 and η = D = 2 R ). We next compare UniFastSvrg (Algorithm 4) against AdaVRAE and AdaVRAG [36]. We also compare it with the FastSvrg method with constant stepsize, which is the primal version of the VRADAmethod from [53]; the stepsize is selected by doing a grid search over { 10 j : j = -3 , . . . , 4 } and choosing the best value in the sense that the algorithm is neither too slow nor has a large error. We report UniSgd (Algorithm 1) and UniFastSgd (Algorithm 2) together with these methods. For UniFastSvrg , contrary to the theoretical recommendation of choosing ˜ x 0 as the result of the full gradient step, we found it slightly more useful to simply set ˜ x 0 = x 0 . For all our methods, we use the AdaGrad stepsize (3); the other stepsize (4) works very similarly (see Appendix H.2 for a detailed comparison). For all methods, we use the standard mini-batch stochastic oracle of size b = 256 .

The results are shown in Fig. 1, where we fix n = 10 4 , d = 10 3 , R = 10 6 and consider different values of q ∈ { 1 , 1 . 3 , 1 . 6 , 2 } . We plot the total number of stochastic oracle calls against the function residual. We treat one mini-batch oracle computation as one stochastic oracle call. If we compute the full gradient, we count this as n/b stochastic oracle calls where n is the total number of samples and b denotes the mini-batch size.

We see that, except the AdaSVRG method, all SVRG algorithms typically converge much faster than the usual SGD methods without explicit variance reduction, at least after a few computations of the full gradient. Among the non-accelerated SVRG methods, UniSvrg converges consistently faster than AdaSVRG, while UniFastSvrg performs the best across the accelerated ones. Note that FastSvrg with constant stepsize is not converging when the problem is not Lipschitz smooth ( q < 2 ), in contrast to our universal methods.

In Fig. 2, we also illustrate the impact of the mini-batch size b on the convergence of our methods. We consider the same values of n , d , R as before and fix q = 1 . 5 . As we can see, in the idealized situation, when one can implement the mini-batch oracle computations by perfect parallelism, there is a significant speedup in convergence when increasing the mini-batch size, as predicted by our theory.

For additional experiments, including the discussion of implicit variance reduction, see Appendix H.

9 Conclusions

In this paper, we showed that AdaGrad stepsizes can be applied, in a unified manner, in a large variety of situations, leading to universal methods suitable for multiple problem classes at the same time. Note that this does not come for free. We still need to know one parameter, the diameter D of the feasible set. While it is not necessary to know this parameter precisely, the cost of underestimating or overestimating it, can be high (all complexity bounds would be multiplied by the ratio between our guess and the true D ). At the same time, there already exist some parameter-free methods which are based on AdaGrad and aim to solve precisely this problem [6, 11, 25, 31, 41]. It is therefore interesting to consider extensions of our results to these more advanced algorithms. Another interesting direction is, of course, nonconvex problems.

References

Z. Allen-Zhu and Y. Yuan. Improved SVRG for Non-Strongly-Convex or Sum-of-Non-Convex Objectives. In International Conference on Machine Learning , pages 1080-1089, 2016.

A. Attia and T. Koren. SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variance. In International Conference on Machine Learning , pages 1147-1171, 2023.

R. Babanezhad Harikandeh, M. O. Ahmed, A. Virani, M. Schmidt, J. Koneˇ cn` y, and S. Sallinen. StopWasting My Gradients: Practical SVRG. Advances in Neural Information Processing Systems , 28, 2015.

D. P. Bertsekas. Stochastic optimization problems with nondifferentiable cost functionals. Journal of Optimization Theory and Applications , 12(2):218-231, 1973.

L. Bottou, F. E. Curtis, and J. Nocedal. Optimization Methods for Large-Scale Machine Learning. SIAM Review , 60(2):223-311, 2018.

Y. Carmon and O. Hinder. Making SGD Parameter-Free. In Conference on Learning Theory , pages 2360-2389, 2022.

C.-C. Chang and C.-J. Lin. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology , 2:27:1-27:27, 3, 2011.

A. Cotter, O. Shamir, N. Srebro, and K. Sridharan. Better mini-batch algorithms via accelerated gradient methods. Advances in Neural Information Processing Systems , 24, 2011.

A. Cutkosky and K. Boahen. Online Learning Without Prior Information. In Conference on Learning Theory , pages 643-677, 2017.

A. Cutkosky and F. Orabona. Black-Box Reductions for Parameter-free Online Learning in Banach Spaces. In Conference On Learning Theory , pages 1493-1529, 2018.

A. Defazio and K. Mishchenko. Learning-Rate-Free Learning by D-Adaptation. In International Conference on Machine Learning , pages 7449-7479, 2023.

Q. Deng, Y. Cheng, and G. Lan. Optimal Adaptive and Accelerated Stochastic Gradient Descent. arXiv preprint arXiv:1810.00553 , 2018.

O. Devolder. Stochastic first order methods in smooth convex optimization. CORE Discussion Papers, 2011/70 , 2011.

O. Devolder. Exactness, Inexactness and Stochasticity in First-Order Methods for Large-Scale Convex Optimization . PhD thesis, Université catholique de Louvain (UCL), 2013.

O. Devolder, F. Glineur, and Y. Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming , 146:37-75, 2013. DOI: 10.1007/s10107013-0677-5 .

B. Dubois-Taine, S. Vaswani, R. Babanezhad, M. Schmidt, and S. Lacoste-Julien. SVRG meets AdaGrad: painless variance reduction. Machine Learning , 111(12):4359-4409, 2022.

J. Duchi, E. Hazan, and Y. Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research , 12(7), 2011.

A. Ene, H. L. Nguyen, and A. Vladu. Adaptive Gradient Methods for Constrained Convex Optimization and Variational Enequalities. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35 of number 8, pages 7314-7321, 2021.

E. Gorbunov, F. Hanzely, and P. Richtárik. A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent. In International Conference on Artificial Intelligence and Statistics , pages 680-690, 2020.

R. M. Gower, P. Richtárik, and F. Bach. Stochastic quasi-gradient methods: variance reduction via Jacobian sketching. Mathematical Programming , 188(1):135-192, 2021.

R. M. Gower, M. Schmidt, F. Bach, and P. Richtárik. Variance-Reduced Methods for Machine Learning. Proceedings of the IEEE , 108(11):1968-1983, 2020.

R. M. Gower, N. Loizou, X. Qian, A. Sailanbayev, E. Shulgin, and P. Richtárik. SGD: General Analysis and Improved Rates. In International Conference on Machine Learning , pages 52005209, 2019.

G. N. Grapiglia and Y. Nesterov. Regularized Newton Methods for Minimizing Functions with Hölder Continuous Hessians. SIAM Journal on Optimization , 27(1):478-506. DOI: 10.1137/ 16M1087801 .

S. Ilandarideva, A. Juditsky, G. Lan, and T. Li. Accelerated stochastic approximation with state-dependent noise. arXiv preprint arXiv:2307.01497 , 2023.

M. Ivgi, O. Hinder, and Y. Carmon. DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule. In International Conference on Machine Learning , pages 14465-14499, 2023.

A. Jacobsen and A. Cutkosky. Unconstrained Online Learning with Unbounded Losses. In International Conference on Machine Learning , pages 14590-14630, 2023.

R. Johnson and T. Zhang. Accelerating Stochastic Gradient Descent using Predictive Variance Reduction. Advances in Neural Information Processing Systems , 26, 2013.

P. Joulani, A. Raj, A. Gyorgy, and C. Szepesvári. A Simpler Approach to Accelerated Optimization: Iterative Averaging Meets Optimism. In International Conference on Machine Learning , pages 4984-4993, 2020.

A. Juditsky, A. Kulunchakov, and H. Tsyntseus. Sparse recovery by reduced variance stochastic approximation. Information and Inference: A Journal of the IMA , 12(2):851-896, 2023.

A. Kavis, K. Y. Levy, F. Bach, and V. Cevher. UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimization. Advances in Neural Information Processing Systems , 32, 2019.

A. Khaled, K. Mishchenko, and C. Jin. DoWG Unleashed: An Efficient Universal ParameterFree Gradient Descent Method. Advances in Neural Information Processing Systems , 36:67486769, 2023.

D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR) , 2015.

G. Lan. An optimal method for stochastic composite optimization. Mathematical Programming , 133(1):365-397, 2012.

K. Y. Levy, A. Yurtsever, and V. Cevher. Online Adaptive Methods, Universality and Acceleration. Advances in Neural Information Processing Systems , 31, 2018.

C. Liu and M. Belkin. Accelerating SGD with momentum for over-parameterized learning. arXiv preprint arXiv:1810.13395 , 2018.

Z. Liu, T. D. Nguyen, A. Ene, and H. Nguyen. Adaptive Accelerated (Extra-)Gradient Methods with Variance Reduction. In International Conference on Machine Learning , pages 1394713994, 2022.

S. Ma, R. Bassily, and M. Belkin. The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning. In International Conference on Machine Learning , pages 3325-3334, 2018.

M. Mahdavi, L. Zhang, and R. Jin. Mixed Optimization for Smooth Functions. Advances in Neural Information Processing Systems , 26, 2013.

H. B. McMahan and M. Streeter. Adaptive Bound Optimization for Online Convex Optimization. arXiv preprint arXiv:1002.4908 , 2010.

Z. Mhammedi and W. M. Koolen. Lipschitz and Comparator-Norm Adaptivity in Online Learning. In Conference on Learning Theory , pages 2858-2887, 2020.

K. Mishchenko and A. Defazio. Prodigy: An Expeditiously Adaptive Parameter-Free Learner. arXiv preprint arXiv:2306.06101 , 2023.

E. Moulines and F. Bach. Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning. Advances in Neural Information Processing Systems , 24, 2011.

I. Necoara, Y. Nesterov, and F. Glineur. Linear convergence of first order methods for nonstrongly convex optimization. Mathematical Programming , 175:69-107, 2019.

D. Needell, R. Ward, and N. Srebro. Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz Algorithm. Advances in Neural Information Processing Systems , 27, 2014.

Y. Nesterov. Universal gradient methods for convex optimization problems. Mathematical Programming , 152:381-404, 2015. DOI: 10.1007/s10107-014-0790-0 .

Y. Nesterov. Lectures on Convex Optimization , volume 137. Springer, 2nd edition, 2018.

F. Orabona. Simultaneous Model Selection and Optimization through Parameter-free Stochastic Learning. Advances in Neural Information Processing Systems , 27, 2014.

F. Orabona. Normalized Gradients for All. arXiv preprint arXiv:2308.05621 , 2023.

A. Rodomanov, A. Kavis, Y. Wu, K. Antonakopoulos, and V. Cevher. Universal Gradient Methods for Stochastic Convex Optimization. arXiv preprint arXiv:2402.03210 , 2024.

M. Schmidt, N. Le Roux, and F. Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming , 162:83-112, 2017.

M. Schmidt and N. L. Roux. Fast Convergence of Stochastic Gradient Descent under a Strong Growth Condition. arXiv preprint arXiv:1308.6370 , 2013.

S. Shalev-Shwartz and T. Zhang. Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization. Journal of Machine Learning Research , 14(1), 2013.

C. Song, Y. Jiang, and Y. Ma. Variance Reduction via Accelerated Dual Averaging for Finite-Sum Optimization. In Advances in Neural Information Processing Systems , volume 33, pages 833-844, 2020.

S. U. Stich. Unified Optimal Analysis of the (Stochastic) Gradient Method. arXiv preprint arXiv:1907.04232 , 2019.

M. Streeter and H. B. McMahan. No-Regret Algorithms for Unconstrained Online Convex Optimization. In Proceedings of the 25th International Conference on Neural Information Processing Systems , volume 2, pages 2402-2410, 2012.

T. Tieleman. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. Coursera: Neural Networks for Machine Learning , 4(2):26, 2012.

S. Vaswani, A. Mishkin, I. Laradji, M. Schmidt, G. Gidel, and S. Lacoste-Julien. Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates. Advances in Neural Information Processing Systems , 32, 2019.

C. Wang, X. Chen, A. J. Smola, and E. P. Xing. Variance Reduction for Stochastic Gradient Optimization. Advances in Neural Information Processing Systems , 26, 2013.

B. E. Woodworth and N. Srebro. An Even More Optimal Stochastic Optimization Algorithm: Minibatching and Interpolation Learning. Advances in Neural Information Processing Systems , 34:7333-7345, 2021.

L. Zhang, M. Mahdavi, and R. Jin. Linear Convergence with Condition Number Independent Access of Full Gradients. Advances in Neural Information Processing Systems , 26, 2013.

A General Auxiliary Results

A.1 Approximately Smooth Functions

Theorem 13 (Lemma 2 in [45]) . Let f : R d → R be a convex ( ν, H ) -Hölder smooth function for some ν ∈ [0 , 1] and H ≥ 0 . Then, for any δ > 0 , any x, y ∈ R d and any ∇ f ( x ) ∈ ∂f ( x ) , it holds that β ∇ f ( x ) f ( x, y ) ≤ L 2 ∥ x -y ∥ 2 + δ with L = [ 1 -ν 2(1+ ν ) δ ] 1 -ν 1+ ν H 2 1+ ν (with the convention that 0 0 = 1 ).

Theorem 14. Let f : R d → R be a ( δ, L ) -approximately smooth convex function with components ( s f, s g ) , i.e., for any x, y ∈ R d and β f, s f, s g ( x, y ) := f ( y ) -s f ( x ) -⟨ s g ( x ) , y -x ⟩ , we have

Then, for any x, y ∈ R d and any ∇ f ( x ) ∈ ∂f ( x ) , the following inequalities hold:

Proof. Inequality (7) follows immediately from (6) by substituting y = x .

To prove (8), we rewrite

and then apply (7).

Using the first part of (8) and applying (6) twice, we obtain (9).

To prove (10) and (11), let us fix some s f 1 ( x ) ∈ R and s g 1 ( x ) ∈ R d such that β f, s f 1 , s g 1 ( z ) := f ( z ) -s f 1 ( x ) -⟨ s g 1 ( x ) , z -x ⟩ ≥ 0 for any z ∈ R d . Note that we can choose either ( s f 1 , s g 1 ) = ( s f, s g ) or ( s f 1 , s g 1 ) = ( f, ∇ f ) . In view of (6), for any z ∈ R d , we can write the following inequalities:

Minimizing the right-hand side in z ∈ R d and rearranging, we conclude that

where the final inequality is due to (7). Substituting now either ( s f 1 , s g 1 ) = ( s f, s g ) or ( s f 1 , s g 1 ) = ( f, ∇ f ) , we obtain either (10) or (11), respectively.

Inequality (12) follows immediately from (6) and (10).

Inequality (13) follows from (11):

To prove (14), we proceed as follows using first (6), then (7), and then (11):

where the final inequality is ( a + b ) 2 ≤ 2 a 2 +2 b 2 , a, b ≥ 0 .

Remark 15 . Some of the inequalities from Theorem 14, namely, (7), (10) and (12), were established already in [15]. We nevertheless prefer to present the corresponding proofs since they are rather simple, and we use the associated ideas for proving the other new inequalities.

Lemma 16. Let f : R d → R be the function f ( x ) := E ξ [ f ξ ( x )] , where each f ξ : R d → R is convex and ( δ ξ , L ξ ) -approximately smooth with components ( s f ξ , s g ξ ) . Further, let p g be the stochastic oracle defined by g ( x, ξ ) := s g ξ ( x ) , and let s f ( x ) := E ξ [ s f ξ ( x )] , s g ( x ) := E ξ [ s g ξ ( x )] . Then, p g is an unbiased oracle for s g and, for any x, y ∈ R d , L max := sup ξ L ξ and s δ := 1 L max E ξ [ L ξ δ ξ ] , it holds that

Furthermore, for any x, y ∈ R d and any ∇ f ( x ) ∈ ∂f ( x ) , it also holds that

Proof. According to our definition of s g , we have E ξ [ s g ξ ( x )] = s g ( x ) for any x , so p g is indeed an unbiased oracle for s g . Further, for any x, y ∈ R d , we can estimate

where s δ is as defined in the statement; the second inequality follows from Theorem 14 (inequality (10)), and the final identity is due to the linearity of β f, s f, s g ( x, y ) in ( f, s f, s g ) and the fact that, by our definitions, E ξ [ f ξ ( x )] = f ( x ) , E ξ [ s f ξ ( x )] = s f ( x ) , E ξ [ s g ξ ( x )] = s g ( x ) for any x . This proves (15).

The proof of (16) is similar but now we apply (13) instead of (10):

where we have used the fact that ∂f ( x ) = E ξ [ ∂f ξ ( x )] (see Proposition 2.2 in [4]), meaning that, for any ∇ f ( x ) ∈ ∂f ( x ) , we can find a selection of ∇ f ξ ( x ) ∈ ∂f ξ ( x ) such that ∇ f ( x ) = E ξ [ ∇ f ξ ( x )] .

A.2 Miscellaneous

Lemma 17. Let ψ : R d → R ∪ { + ∞} be a proper closed convex function, x ∈ dom ψ , g ∈ R d , M ≥ 0 , and let

Then, for any y ∈ dom ψ , we have

Proof. Indeed, by definition, x + is the minimizer of the function h : R d → R ∪ { + ∞} given by h ( y ) := ⟨ g, y ⟩ + ψ ( y ) + M 2 ∥ x -y ∥ 2 , which is strongly convex with parameter M (or simply convex if M = 0 ). Hence, for any y ∈ dom ψ ( = dom h ), we have h ( y ) ≥ h ( x + ) + M 2 ∥ y -x + ∥ 2 , which is exactly the claimed inequality.

Lemma 18. Let N ≥ 1 be an integer, ( M k ) N k =0 be a nondecreasing nonnegative sequence of reals, and let Ď M ≥ 0 . Then,

Proof. It suffices to prove the identity only in the special case when N = 2 , i.e., to show that γ 0 + γ 1 = Γ , where γ 0 := [min { M 1 , Ď M }-M 0 ] + , γ 1 := [min { M 2 , Ď M }-M 1 ] + , Γ := [min { M 2 , Ď M }-M 0 ] + . The general case then easily follows by induction.

To prove the identity, we use our assumption that M 0 ≤ M 1 ≤ M 2 and consider three possible cases. If M 1 ≥ Ď M , then γ 0 + γ 1 = [ Ď M -M 0 ] + + 0 = [ Ď M -M 0 ] + = Γ . If M 1 < Ď M ≤ M 2 , then γ 0 + γ 1 = ( M 1 -M 0 ) + ( Ď M -M 1 ) = Ď M -M 0 = Γ . Finally, if M 2 < Ď M , then γ 0 + γ 1 = ( M 1 -M 0 ) + ( M 2 -M 1 ) = M 2 -M 0 = Γ .

Lemma 19. Let p g be a stochastic oracle in R d . Then, for any x, y, z ∈ R d and any τ > 0 , we have

Proof. Both inequalities are direct consequences of the standard inequality ∥ s 1 + s 2 ∥ 2 ∗ ≤ (1 + τ ) ∥ s 1 ∥ 2 ∗ + (1 + τ -1 ) ∥ s 2 ∥ 2 ∗ which is valid for any s 1 , s 2 ∈ R d and any τ > 0 . Indeed, let g and ξ be, respectively, the function and the random variable components of p g , and let ∆( x, ξ ) := g ( x, ξ ) -E [ g ( x, ξ )] for any x ∈ R d . Then, for any x, y, z ∈ R d and τ > 0 , we can estimate

Similarly,

B Omitted Proofs for Section 3

Lemma 20 (AdaGrad stepsize) . Let function f satisfy Assumption 1. Consider the stepsize update rule x M + = M + ( M, Ω , x, p x + , p g x , p g x + ) defined by

Then, this stepsize update rules satisfies (2) with c 1 = 5 2 , c 2 = 4 , c 3 = 6 , c 4 = 2 .

Proof. Let p ∆( Ď M ) := β f, s f, s g ( x, p x + ) + ⟨ s g ( x ) -p g x , p x + -x ⟩ -Ď M 2 ∥ p x + -x ∥ 2 . From our Assumption 1 and Theorem 14 (inequality (8)), it follows that β f, s f, s g ( x, p x + )+ β f, s f, s g ( p x + , x ) ≤ ⟨ s g ( p x + ) -s g ( x ) , p x + -x ⟩ +2 δ f . Hence,

where p ∆ 1 ( x M + ) := ⟨ p g x + -p g x , p x + -x ⟩ -x M + 2 ∥ p x + -x ∥ 2 . Hence,

From the definition of x M + , it follows that ∥ p g x + -p g x ∥ 2 ∗ = ( x M 2 + -M 2 )Ω = ( x M + + M )( x M + -M )Ω . Since x M + ≥ M , this means that

Let us now upper bound p Γ := p ∆ 1 ( x M + ) + ( x M + -M )Ω . For this, let us fix an arbitrary constant Ď M ≥ 0 and consider two cases. If x M + ≥ Ď M , we can bound

If x M + ≤ Ď M , we can bound

Combining the two cases, we get p Γ ≤ [ p Γ( Ď M )] + +2[min { x M + , Ď M } -M ] + Ω . Thus,

Let us now estimate the first term. Denote p S := p g x -s g ( x ) and p S + := p g x + -s g ( p x + ) . Then,

Using now our Assumption 1 and Theorem 14 (inequalities (9) and (12)), we can continue as follows:

Consequently,

In particular, for Ď M > 4 L f , we can estimate 2 Ď M + 1 2[ Ď M -2 L f (1+ 2 L f Ď M )] ≤ 2 Ď M + 1 2( Ď M -4 L f ) ≤ 5 2( Ď M -4 L f ) . Therefore, for any Ď M > 4 L f ,

where the final identity follows from the fact that E [ ∥ p S + -p S ∥ 2 ∗ ] = E [ ∥ p S + ∥ 2 ∗ ] + E [ ∥ p S ∥ 2 ∗ ] = E [Var p g ( p x + )] + Var p g ( x ) (because p S + , conditioned on the randomness ξ defining p g x ≡ g ( x, ξ ) , has zero mean).

Combining everything together, we get

This is exactly (2) with c 1 = 5 2 , c 2 = 4 , c 3 = 6 , c 4 = 2 .

Lemma 21. Let function f satisfy Assumption 1. Consider the stepsize update rule x M + = M + ( M, Ω , x, p x + , p g x , p g x + ) defined as the solution of the following equation:

Then, this stepsize update rules satisfies (2) with c 1 = 1 , c 2 = 2 , c 3 = 6 , c 4 = 2 .

Proof. Let us define p ∆( Ď M ) := β f, s f, s g ( x, p x + ) + ⟨ s g ( x ) -p g x , p x + -x ⟩ -Ď M 2 ∥ p x + -x ∥ 2 . Starting as in the proof of Lemma 20, we see that

with the same p ∆ 1 ( · ) as defined in the statement.

Let us now upper bound p Γ := p ∆ 1 ( x M + ) + ( x M + -M )Ω . For this, let us fix an arbitrary constant Ď M ≥ 0 and consider two cases. If x M + ≥ Ď M , we can bound, using the monotonicity of p ∆ 1 ( · ) ,

If x M + ≤ Ď M , we can bound

Combining the two cases, we get p Γ ≤ 2[ p ∆ 1 ( Ď M )] + +2[min { x M + , Ď M } -M ] + Ω , and hence

Let us now estimate the first term. According to our Assumption 1 and Theorem 14 (inequality (9)), we have ⟨ s g ( p x + ) -s g ( x ) , p x + -x ⟩ ≤ L f ∥ p x + -x ∥ 2 + 2 δ f . Hence, denoting p S := p g x -s g ( x ) and p S + := p g x + -s g ( p x + ) , we can estimate, for any Ď M > 2 L f ,

Hence,

where the final identity follows from the fact that E [ ∥ p S + -p S ∥ 2 ∗ ] = E [ ∥ p S + ∥ 2 ∗ ] + E [ ∥ p S ∥ 2 ∗ ] = E [Var p g ( p x + )] + Var p g ( x ) (because p S + , conditioned on the randomness ξ defining p g x ≡ g ( x, ξ ) , has zero mean).

Thus, which is exactly (2) with c 1 = 1 , c 2 = 2 , c 3 = 6 , c 4 = 2

C Omitted Proofs for Section 4

C.1 Universal SGD

Lemma 22 (Stochastic Gradient Step) . Consider problem (1) under Assumption 1. Let p g be an unbiased oracle for s g . Let x ∈ dom ψ be a point, M ≥ 0 be a coefficient, p g x ∼ = p g ( x ) , and let

If further Assumption 2 is satisfied, and x M + ≥ M is a random coefficient (possibly dependent on p g x ), then, we also have

Proof. From Lemma 17, it follows that

Passing to expectations and rewriting

and

we obtain the first of the claimed inequalities.

To prove the second one, we simply add to both sides of the already proved first inequality the expected value of

and then bound ∥ p x + -x ∗ ∥ ≤ D , ∥ p x + -x ∥ ≤ D using our Assumption 2 and the fact that x, p x + , x ∗ ∈ dom ψ .

Lemma 23 (Universal Stochastic Gradient Step) . Consider problem (1) under Assumptions 1 and 2. Let p g be an unbiased oracle for s g . Further, let x ∈ dom ψ be a point, M ≥ 0 be a coefficient, p g x ∼ = p g ( x ) , and let

Then, for any Ď M > c 2 L f , it holds that

Proof. According to Lemma 22,

where p ∆( x M + ) := β f, s f, s g ( x, p x + )+ ⟨ s g ( x ) -p g x , p x + -x ⟩-x M + 2 ∥ p x + -x ∥ 2 . At the same time, according to the main requirement (2) on the stepsize update rule, for any Ď M > c 2 L f ,

Combining the two displays, we get the claim.

Lemma 24 (Universal SGD: General Guarantee) . Consider problem (1) under Assumptions 1 and 2. Let p g be an unbiased oracle for s g . Further, let x ∈ dom ψ be a point, M ≥ 0 be a coefficient, N ≥ 1 be an integer, and let

as defined by Algorithm 1, and let x 0 , . . . , x N be the corresponding points generated inside the algorithm. Then, for any Ď M > c 2 L f , it holds that

Proof. Each iteration k of the algorithm, when conditioned on x k , follows the construction from Lemma 23 (with x = x k , p g x = g k , M = M k , p x + = x k +1 , p g x + = g k +1 , x M + = M k +1 ). Hence, we can write, after passing to full expectations, that, for each k = 0 , . . . , N -1 ,

where Ď M > 2 L f is an arbitrary constant. Telescoping the above inequalities (using Lemma 18) and then bounding N [ F ( s x N ) -F ∗ ] ≤ ∑ N k =1 [ F ( x k ) -F ∗ ] (using the convexity of F and our choice of s x N = 1 N ∑ N k =1 x k ), we get the claim.

Theorem 4. Let Algorithm 1 with M 0 = 0 be applied to problem (1) under Assumptions 1-3. Then, for the point s x N generated by the algorithm, we have

Proof. Applying Lemma 24, substituting our choice of M 0 = 0 , estimating Var p g ( · ) ≤ σ 2 and dropping the nonnegative β f, s f, s g ( · , · ) terms, we obtain

where Ď M > 2 L f is an arbitrary constant. The optimal Ď M which minimizes the right-hand side is Ď M = c 2 L f + σ D √ 2 c 1 c 4 N . Substituting this choice into the above display, we get

C.2 Universal Fast SGD

Lemma 25 (Stochastic Triangle Step) . Consider problem (1) under Assumption 1. Let p g be an unbiased oracle for s g , let x, v ∈ dom ψ be points and M,A ≥ 0 , a > 0 be coefficients. Further, for A + := A + a , let

Denote p ∆( M ) := β f, s f, s g ( y, p x + ) + ⟨ s g ( y ) -p g y , p x + -y ⟩ -MA + 2 a 2 ∥ p x + -y ∥ 2 . Then,

If further Assumption 2 is satisfied, and x M + ≥ M is a random coefficient (possibly dependent on p g y ), then we also have

Proof. Denoting θ := Aβ f, s f, s g ( y, x ) + aβ f, s f, s g ( y, x ∗ ) and using the fact that E [ p g y ] = s g ( y ) , we can rewrite

Further, by the definition of p v + and Lemma 17,

This means that

where the second inequality is due to the definition of p x + and the convexity of ψ , and

since p x + -y = a A + ( p v + -v ) (by the definitions of y and p x + ). Substituting the above inequality into the first display and rearranging, we get the first of the claimed inequalities.

To prove the second one, we simply add to both sides of the already proved first inequality the expected value of

and then bound, using the fact that p x + -y = a A + ( p v + -v ) together with our Assumption 2,

Lemma 26 (Universal Stochastic Triangle Step) . Consider problem (1) under Assumptions 1 and 2, and let p g be an unbiased oracle for s g . Let x, v ∈ dom ψ be points, M,A ≥ 0 , a > 0 be coefficients. Further, for A + := A + a , let

Then, for any Ď M > c 2 L f a 2 A + , it holds that

Proof. According to Lemma 25 (together with the fact that x M + ≥ M which is guaranteed by the requirement on the stepsize update rule), we have

where p ∆( x M + ) := β f, s f, s g ( y, p x + )+ ⟨ s g ( y ) -p g y , p x + -y ⟩-x M + A + 2 a 2 ∥ p x + -y ∥ 2 . Further, according to the main requirement (2) on the stepsize update rule (applied in the variables M ′ := A + a 2 M , Ω := a 2 A 2 + D 2 , x M ′ + := A + a 2 x M + , Ď M ′ := A + a 2 Ď M for which we have M ′ Ω = M D 2 A + , x M ′ + Ω = x M + D 2 A + , Ď M ′ Ω = Ď M D 2 A + ), it holds that

where Ď M > c 2 L f a 2 A + is an arbitrary constant. Multiplying both sides of the above display by A + and adding the result to the first display, we obtain the claim.

Lemma 27 (Universal Fast SGD: General Guarantee) . Consider Algorithm 2 applied to problem (1) under Assumptions 1 and 2. Then, for any k ≥ 1 and any Ď M > c 2 L f , it holds that

where a k = 1 2 k , A k = 1 4 k ( k +1) , ∑ k i =1 a 2 i = 1 24 k ( k +1)(2 k +1) , ∑ k i =1 A i = 1 12 k ( k +1)( k +2) for each k ≥ 1 .

Proof. Each iteration k of the algorithm, when conditioned on ( x k , v k ) , follows the construction from Lemma 26 (with x = x k , v = v k , M = M k , A = A k , a = a k +1 , A + = A k +1 , y = y k , p g y = g y k , p v + = v k +1 , p x + = x k +1 , p g x + = g x k +1 , x M + = M k +1 ), where A k and a k are the following coefficients: a k = 1 2 k , A k = ∑ k i =1 a i = 1 4 k ( k +1) . Applying Lemma 26 (dropping the nonnegative β f, s f, s g ( y, x ) term) and passing to full expectations, we therefore obtain, for each k ≥ 0 ,

where Ď M is an arbitrary constant such that Ď M > c 2 L f a 2 k +1 A k +1 . Note however that, for our sequences a k and A k , we have a 2 k A k = 1 4 k 2 1 4 k ( k +1) = k k +1 ≤ 1 . Therefore, we can replace c 1 a 2 k +1 Ď M -c 2 L f a 2 k +1 A k +1 in the above display with c 1 a 2 k +1 Ď M -c 2 L f under the requirement that Ď M > c 2 L f . Doing this and then telescoping the above inequalities (applying Lemma 18), and using the fact that M 0 = A 0 = 0 , we get the claimed inequality.

It remains to do some standard computations to see that ∑ k i =1 a 2 i ≡ 1 4 ∑ k i =1 i 2 = 1 24 k ( k +1)(2 k +1) and ∑ k i =1 A i ≡ 1 4 ∑ k i =1 i ( i +1) = 1 4 ( 1 6 k ( k +1)(2 k +1)+ 1 2 k ( k +1)) = 1 12 k ( k +1)( k +2) .

Theorem 5. Let Algorithm 2 be applied to problem (1) under Assumptions 1-3. Then, for any k ≥ 1 ,

Proof. Let k ≥ 1 be arbitrary and F k := E [ F ( x k )] -F ∗ . Applying Lemma 27, dropping the nonnegative β f, s f, s g ( · , · ) terms and bounding Var p g ( · ) ≤ σ 2 , we obtain, for an arbitrary constant Ď M > c 2 L f ,

where δ k := c 3 3 ( k +2) δ f . We now choose Ď M > c 2 L f which minimizes the right-hand side. This is Ď M = c 2 L f + σ 2 D √ c 1 3 c 4 k ( k +1)(2 k +1) , for which we get

D Omitted Proofs for Section 5

D.1 Universal SGD

Theorem 7. Let Algorithm 1 with M 0 = 0 be applied to problem (1) under Assumptions 1, 2 and 6, and let σ 2 ∗ := Var p g ( x ∗ ) . Then, for the point s x N produced by the method, we have

Proof. Let x 0 , . . . , x N be the points generated inside the method and let F N := E [ F ( s x N )] -F ∗ . Using Lemma 19 and Assumption 6, we can estimate, for any 0 ≤ k ≤ N -1 ,

Substituting this bound into the general guarantee given by Lemma 24 (and taking into account the fact that M 0 = 0 ), we obtain

where Ď M > c 2 L f is an arbitrary constant and α := 4 c 1 L p g Ď M -c 2 L f . Requiring now that 3 α ≤ 1 or, equivalently, that Ď M ≥ c 2 L f +12 c 1 L p g =: Ď M min , we can cancel the nonnegative β f, s f, s g ( · , · ) terms on both sides and obtain

where δ := c 3 δ f + 4 3 δ p g . The optimal coefficient Ď M ∗ minimizing the right-hand side is Ď M ∗ = c 2 L f + σ ∗ D √ 6 c 1 N c 4 . However, we still need to respect the constraint Ď M ≥ Ď M min . Choosing Ď M = c 2 L f +12 c 1 L p g + σ ∗ D √ 6 c 1 N c 4 , we conclude that

D.2 Universal Fast SGD

Theorem 8. Let Algorithm 2 be applied to problem (1) under Assumptions 1, 2 and 6, and let σ 2 ∗ := Var p g ( x ∗ ) . Then, for any k ≥ 1 , we have

Proof. Let k ≥ 1 be arbitrary and F k := E [ F ( x k )] -F ∗ . Using Lemma 19 and Assumption 6, we can estimate, for each i ,

Substituting this bound into the guarantee given by Lemma 27, we obtain

where α i +1 := 4 c 1 L p g a 2 i +1 Ď M -c 2 L f , a i = 1 2 i , A k = 1 4 k ( k +1) , ∑ k i =1 a 2 i = 1 24 k ( k +1)(2 k +1) , ∑ k i =1 A i = 1 12 k ( k +1)( k +2) . Requiring now that 3 α i +1 ≤ a i +1 for all i = 0 , . . . , k -1 or, equivalently, that Ď M ≥ c 2 L f +12 c 1 L p g a k ≡ c 2 L f +6 c 1 L p g k , we can cancel the nonnegative β f, s f, s g ( · , · ) terms on both sides and obtain

where δ k := c 3 3 ( k +2) δ f + 4 3 δ p g .

The minimizer of the right-hand side is Ď M ∗ = c 2 L f + σ ∗ 2 D √ c 1 c 4 k ( k +1)(2 k +1) . However, recall that we also need to satisfy the constraint Ď M ≥ c 2 L f +6 c 1 L p g k . Choosing Ď M = c 2 L f +6 c 1 L p g k + σ ∗ 2 D √ c 1 c 4 k ( k +1)(2 k +1) , we obtain

E Omitted Proofs for Section 6

Lemma 28 (Basic property of SVRG oracle) . Let p g be a stochastic oracle in R d , and let p G = SvrgOrac p g (˜ x ) for some ˜ x ∈ R d . Then, for any x ∈ R d , the mean value of p G at x is the same as that of p g at x , while Var p G ( x ) = Var p g ( x, ˜ x ) .

Proof. Let g and ξ be, respectively, the function and the random variable components of p g , and let g ( x ) := E ξ [ g ( x, ξ )] , g (˜ x ) := E ξ [ g (˜ x, ξ )] . Then, by definition, p G is the oracle with the same random variable component ξ and the function component G defined by G ( x, ξ ) = g ( x, ξ ) -g (˜ x, ξ ) + g (˜ x ) . Consequently, E ξ [ G ( x, ξ )] = g ( x ) , and

E.1 Universal SVRG

Lemma 29 (Universal SVRG Epoch) . Consider problem (1) under Assumptions 1, 2, 6 and 9. Let x, ˜ x ∈ dom ψ be points, M ≥ 0 be a coefficient, N ≥ 1 be an integer, p G = SvrgOrac p g (˜ x ) , and let

as defined by Algorithm 1. Then, for any Ď M ≥ c 2 L f +12 c 1 L p g , α := 4 c 1 L p g Ď M -c 2 L f , and any ∇ f ( x ∗ ) ∈ ∂f ( x ∗ ) , it holds that

Proof. Since p g is an unbiased oracle for s g , so is p G (Lemma 28). Therefore, we can apply Lemma 24 to get

where Ď M > c 2 L f is an arbitrary constant and x k are the points generated inside UniSgd . Applying now Lemmas 19 and 28 and Assumptions 6 and 9, we can estimate, for each k ,

where ∇ f ( x ∗ ) ∈ ∂f ( x ∗ ) is arbitrary. Denoting α := 4 c 1 L p g Ď M -c 2 L f , we thus obtain

Requiring now Ď M ≥ c 2 L f + 12 c 1 L p g , we get α ≤ 1 3 which allows us to cancel the nonnegative β f, s f, s g ( · , · ) terms on both sides. The claim now follows.

Theorem 10. Let UniSvrg (as defined by Algorithm 3) be applied to problem (1) under Assumptions 1, 2, 6 and 9. Then, for any t ≥ 1 and s c 3 := max { c 3 , 1 } , we have

To construct ˜ x t , the algorithm needs to make O (2 t ) queries to p g and O ( t ) queries to s g .

Proof. The algorithm iterates (˜ x t +1 , x t +1 , M t +1 ) ∼ = UniSgd p G t ,ψ ( x t , M t , 2 t +1 ; D ) for t ≥ 0 , where p G t = SvrgOrac p g (˜ x t ) . Applying Lemma 29 with Ď M := c 2 L f +48 c 1 L p g (for which α = 1 12 so that 6 α 2 t +1 = 2 t ) and passing to full expectations, we obtain, for any t ≥ 0 ,

where β t := β ∇ f ( x ∗ ) f ( x ∗ , ˜ x t ) and ∇ f ( x ∗ ) ∈ ∂f ( x ∗ ) can be chosen arbitrarily. Rewriting F t +1 := F (˜ x t +1 ) -F ∗ as F t +1 = β t +1 + ( F t +1 -β t +1 ) and telescoping the above inequalities (using, Lemma 18), we get, for any t ≥ 1 ,

where the final inequality is due to the fact that M 0 = 0 , while ∑ t i =1 2 i = 2(2 t -1) . According to Lemma 30, we can choose ∇ f ( x ∗ ) ∈ ∂f ( x ∗ ) such that β i ≤ F (˜ x i ) -F ∗ for all i ≥ 0 . Dropping now various nonnegative terms from the left-hand side of the above display, we conclude that

Let us estimate Φ 0 . Using our Assumptions 1 and 2 and Theorem 14 (inequality (14)), we can bound β 0 ≤ L f ∥ ˜ x 0 -x ∗ ∥ 2 +2 δ f ≤ L f D 2 +2 δ f . Therefore,

Φ 0 ≤ L f D 2 +2 δ f + c 4 Ď MD 2 +2(2 t -1)( c 3 δ f + 4 3 δ p g ) ≤ LD 2 +2( s c 3 δ f + 4 3 δ p g ) · 2 t where L := L f + c 4 Ď M ≡ ( c 2 c 4 +1) L f +48 c 1 c 4 L p g and s c 3 := max { c 3 , 1 } . Thus, E [ F t ] ≤ Φ 0 2 t ≤ LD 2 2 t +2 s c 3 δ f + 8 3 δ p g , which proves the claimed convergence rate.

Let us now estimate the number of oracle queries. At each iteration t , the algorithm first queries s g to construct the SVRG oracle p G t (by precomputing s g (˜ x t ) ). All other queries are then done only to p G t or, equivalently, to p g inside UniSgd p G t ,ψ which is run for N t +1 = 2 t +1 iterations and thus requiring O ( N t +1 ) queries to p g . Summing up, after T iterations, we obtain the total number of ∑ T t =1 O ( N t ) = ∑ T t =1 O (2 t ) = O (2 T ) queries to p g , and T queries to s g .

Helper Lemmas

Lemma 30. Let F : R d → R ∪{ + ∞} be the function F ( x ) := f ( x ) + ψ ( x ) , where f : R d → R is a convex function, and ψ : R d → R ∪{ + ∞} is a proper closed convex function. Let x ∗ be a minimizer of F and let F ∗ := F ( x ∗ ) . Then, there exists ∇ f ( x ∗ ) ∈ ∂f ( x ∗ ) such that, for any x ∈ dom ψ ,

Proof. Since x ∗ is a minimizer of F , we have 0 ∈ ∂F ( x ∗ ) = ∂f ( x ∗ ) + ∂ψ ( x ∗ ) . In other words, there exists ∇ f ( x ∗ ) ∈ ∂f ( x ∗ ) such that ∇ ψ ( x ∗ ) := -∇ f ( x ∗ ) ∈ ∂ψ ( x ∗ ) . Consequently, for any x ∈ dom ψ ,

E.2 Universal Fast SVRG

Lemma 31 (Universal Triangle SVRG Step) . Consider problem (1) under Assumptions 1, 2 and 6. Let ˜ x, v ∈ dom ψ be points, M ≥ 0 and A,a > 0 be coefficients, p G := SvrgOrac p g (˜ x ) . Further, let, for A + := A + a ,

Then, for Ď M := c 2 L f a 2 A + +6 c 1 L p g a 2 A , it holds that

Proof. Since p g is an unbiased oracle for s g , so is p G (Lemma 28). Therefore, we can apply Lemma 26 to obtain

where Ď M > c 2 L f a 2 A + is an arbitrary coefficient. Using Lemmas 19 and 28 and Assumption 6, we can further bound

Denoting α := 2 c 1 L p g a 2 Ď M -c 2 L f a 2 A + , we thus obtain

Choosing now Ď M = c 2 L f a 2 A + +6 c 1 L p g a 2 A , we get α = 1 3 A ( ≤ 1 3 A + ) , which allows us to drop the nonnegative β f, s f, s g ( · , · ) terms from both sides. The claim now follows.

Lemma 32 (Universal Triangle SVRG Epoch) . Consider problem (1) under Assumptions 1, 2 and 6. Let ˜ x, v ∈ dom ψ be points, M ≥ 0 and A,a > 0 be coefficients, N ≥ 1 be an integer, and let

as defined by Algorithm 5. Then, for A + := A + a and Ď M := c 2 L f a 2 A + +6 c 1 L p g a 2 A , it holds that

Proof. Each iteration k of the algorithm, when conditioned on v k , follows the construction from Lemma 31 (with v = v k , M = M k , A = A k , a = a k +1 , A + = A k +1 , x = x k , p G x = G x k , p v + = v k +1 , p x + = x k +1 , p G x + = G x k +1 , x M + = M k +1 ). Hence, we can write, after passing to full expectations, for each k = 0 , . . . , N -1 ,

where δ := c 3 A + δ f + 5 3 Aδ p g . Telescoping the above inequalities (using Lemma 18), we get

The claim now follows from the convexity of F and our definitions ˜ x + = s x N = 1 N ∑ N k =1 x k , v + = v N , M + = M N , M 0 = M , v 0 = v .

Theorem 11. Let UniFastSvrg (Algorithm 4) be applied to problem (1) under Assumptions 1, 2 and 6, and let N ≥ 9 . Then, for any t ≥ t 0 := ⌈ log 2 log 3 N ⌉ -1 ( ≥ 0) , it holds that

To construct ˜ x t , the algorithm needs to make O ( Nt ) queries to p g and O ( t ) queries to s g . Assuming that the complexity of querying s g is n times bigger than that of querying p g and choosing N = Θ( n ) , we get the total stochastic-oracle complexity of O ( nt ) .

Proof. By our definition, the algorithm iterates for t ≥ 0 :

where A t and a t +1 are deterministic coefficients satisfying the following equations:

In particular, for any t ≥ 0 , we have Ď M ′ t := c 2 L f a 2 t +1 A t +1 +6 c 1 L p g a 2 t +1 A t ≤ c 2 L f +6 c 1 L p g =: Ď M , and hence [min { M t +1 , Ď M ′ t } -M t ] + ≤ [min { M t +1 , Ď M } -M t ] + (because, for any fixed a and b , the function [min { a, ·} -b ] + is nondecreasing as the composition of two nondecreasing functions). Applying now Lemma 32 and passing to full expectations, we therefore obtain, for any t ≥ 0 ,

Telescoping the above inequalities (using, in particular, Lemma 18), we obtain, for any t ≥ 1 ,

where, for the last inequality, we have used the fact that M 0 = 0 and denoted S t := ∑ t i =1 A i . Thus, for any t ≥ 1 ,

At the same time, according to (17), A t +1 -A t = √ A t for any t ≥ 0 . Hence, by Lemma 33 (and our assumption on A 0 ), we can estimate A t ≥ 1 9 ( t -t 0 +1) 2 for any t ≥ t 0 := ⌈ log 2 log 3 1 A 0 ⌉-1 ( ≥ 0) . Further, since the sequence A t is increasing, we can estimate S t ≡ ∑ t i =1 A i ≤ tA t , so that S t A t ≤ t . Substituting these bounds into the above display and using our formula for A 0 , we obtain, for any t ≥ t 0 ,

where ρ t := 9 N ( t -t 0 +1) 2 ≤ 1 . By our choice of ˜ x 0 , it holds that F (˜ x 0 ) -F ∗ ≤ 1 2 L f D 2 + δ f (see Lemma 34). Denoting L := 1 2 L f + c 4 Ď M ≡ ( c 2 c 4 + 1 2 ) L f +6 c 1 c 4 L p g , we get

which is exactly the claimed convergence rate bound.

Let us now estimate the number of oracle queries. At the beginning, the algorithm makes 1 query to s g to compute ˜ x 0 . All other queries to the oracles are then done, at each iteration t , only inside the call to UniTriSvrgEpoch (Algorithm 5). Each such a call needs only one query to s g to construct the SVRG oracle p G (by precomputing s g (˜ x ) ), and O ( N ) queries to p g (which implements each query to p G ). Summing up, we get, after t iterations, the total number of O ( Nt ) queries to p g and O ( t ) queries to s g .

Helper Lemmas

Lemma 33 (c.f. Lemma 1.1 in [23]) . Let A t be a positive sequence such that

for all t ≥ 0 , where γ > 0 , and let A 0 ≤ 1 9 γ . Then, for any t ≥ 0 , we have

where t 0 := ⌈ log 2 log 3 γ A 0 ⌉ -1 ( ≥ 0) .

Proof. By replacing A t with A ′ t = A t /γ , we can assume w.l.o.g. that γ = 1 .

For any t ≥ 0 , we have A t +1 ≥ √ A t , and hence

In particular, for t 0 (as defined in the statement), we get t 0 ≥ log 2 log 3 1 A 0 -1 , so 2 t 0 ≥ 1 2 log 3 1 A 0 , and hence

(recall that A 0 ≤ 1 9 ≤ 1 ).

On the other hand, for any t ≥ t 0 , we have

where we have used the fact that A t ≥ A t 0 ≥ 1 9 since A t is monotonically increasing. Telescoping these inequalities and rearranging, we get, for any t ≥ t 0 ,

Lemma 34. Consider problem (1) under Assumptions 1 and 2. Let x ∈ dom ψ , and let x + := Prox ψ ( x, s g ( x ) , 0) . Then, F ( x + ) -F ∗ ≤ 1 2 L f D 2 + δ f .

Proof. From the first-order optimality condition for the point x + (see Lemma 17), it follows that

Combining the above inequality first with f ( x + ) ≤ s f ( x ) + ⟨ s g ( x ) , x + -x ⟩ + L f 2 ∥ x + -x ∥ 2 + δ f and then with s f ( x ) + ⟨ s g ( x ) , x ∗ -x ⟩ ≤ f ( x ∗ ) (which are both due to our Assumption 1), we obtain

It remains to bound ∥ x + -x ∥ ≤ D .

F Omitted Proofs for Section 7

We start with the observation that for our specific example all our main assumptions are satisfied. Remark 35 . Under the setting from Example 12, Assumptions 1, 6 and 9 are satisfied with s f = f , s g ( x ) = ∇ f ( x ) := E ξ [ ∇ f ξ ( x )] , any δ f , δ p g > 0 and

Further, the oracle p g b satisfies Assumption 3 with σ 2 b := sup x ∈ dom ψ Var p g b ( x ) = 1 b σ 2 , and σ 2 ∗ ,b := Var p g b ( x ∗ ) = 1 b σ 2 ∗ .

Proof. For b = 1 , this follows from Theorem 13 and Lemma 16 and our definitions of σ 2 and σ ∗ . The general case b ≥ 1 follows from the fact that the standard mini-batching of size b reduces each of the variances Var p g 1 ( · ) and Var p g 1 ( · , · ) in b times.

The following auxiliary result will be useful throughout this section:

Lemma 36. Let a, b, p > 0 be real. Then,

Proof. The expression inside the min is a convex function in t > 0 . Differentiating and setting its derivative to zero, we see that the minimum is attained at t ∗ = ( ap b ) 1 p +1 . Hence,

F.1 Uniformly Bounded Variance

Corollary 37. Consider problem (1) under the setting from Example 12 and also under Assumption 2. Let Algorithm 1 be applied to this problem with the oracle p g = p g b and initial coefficient M 0 = 0 . Then, for the point s x N generated by the algorithm, we have

To reach E [ F ( s x N )] -F ∗ ≤ ϵ for any ϵ > 0 , it suffices to make O ( [ H f ( ν ) ϵ ] 2 1+ ν D 2 + σ 2 D 2 bϵ 2 ) queries to p g b .

Proof. Denote for brevity H f := H f ( ν ) . Taking into account Remark 35 and applying Theorem 4, we get, for any δ f > 0 ,

where σ N := 2 σ b D √ 2 c 1 c 4 N = 2 σD √ 2 c 1 c 4 bN . Minimizing the right-hand side in δ f (using Lemma 36 with p = 1 -ν 1+ ν for which p +1 = 2 1+ ν ), we obtain

This proves the claimed convergence rate, and the oracle complexity bound easily follows since each iteration of the algorithm requires only 1 query to p g b .

Corollary 38. Consider problem (1) under the setting from Example 12 and also under Assumption 2. Let Algorithm 2 be applied to this problem with the oracle p g = p g b . Then, for any k ≥ 1 , we have

To reach E [ F ( x k )] -F ∗ ≤ ϵ for any ϵ > 0 , it suffices to make O ( [ H f ( ν ) D 1+ ν ϵ ] 2 1+3 ν + σ 2 D 2 bϵ 2 ) queries to p g b .

Proof. Let k ≥ 1 be arbitrary and denote for brevity H f := H f ( ν ) . Taking into account Remark 35 and applying Theorem 5, we get, for any δ f > 0 ,

where σ k := 4 σ b D √ 2 c 1 c 4 3 k = 4 σD √ 2 c 1 c 4 3 bk . Minimizing the right-hand side in δ f (using Lemma 36) and estimating k +2 ≤ 2( k +1) , we obtain

This proves the claimed convergence rate, and the oracle complexity bound easily follows since each iteration of the algorithm requires only O (1) queries to p g b .

Remark 39 . The efficiency guarantees given by Corollaries 37 and 38 are exactly the same as those from [49], up to absolute constants.

F.2 Implicit Variance Reduction

Corollary 40. Consider problem (1) under the setting from Example 12 and also under Assumption 2. Let Algorithm 1 be applied to this problem with the oracle p g = p g b and initial coefficient M 0 = 0 . Then, for the point s x N generated by the algorithm, we have

where c f ( ν ) := (2 c 2 c 4 ) 1+ ν 2 c 1 -ν 2 3 1+ ν = O (1) and c p g ( ν ) := (24 c 4 ) 1+ ν 2 ( 4 3 ) 1 -ν 2 1+ ν = O (1) . To reach E [ F ( s x N )] -F ∗ ≤ ϵ for any ϵ > 0 , it suffices to make O ( [ H f ( ν ) ϵ ] 2 1+ ν D 2 + 1 b [ H p g ( ν ) ϵ ] 2 1+ ν D 2 + σ 2 ∗ D 2 bϵ 2 ) queries to p g b .

Proof. Denote for brevity F N := E [ F ( s x N )] -F ∗ , H f := H f ( ν ) and H max := H max ( ν ) . Taking into account Remark 35 and applying Theorem 7, we get, for any δ f , δ p g > 0 ,

where σ N := 2 σ ∗ ,b D √ 6 c 1 c 4 N = 2 σ ∗ D √ 6 c 1 c 4 bN . Minimizing the right-hand side in δ f and δ p g (using Lemma 36 twice), we get

This proves the claimed convergence rate, and the oracle complexity bound easily follows since each iteration of the algorithm requires only 1 query to p g b .

Corollary 41. Consider problem (1) under the setting from Example 12 and also under Assumption 2. Let Algorithm 2 be applied to this problem with the oracle p g = p g b . Then, for any k ≥ 1 , we have

where c f ( ν ) := (8 c 2 c 4 ) 1+ ν 2 ( 2 3 c 3 ) 1 -ν 2 1+ ν = O (1) and c p g ( ν ) := (48 c 1 c 4 ) 1+ ν 2 ( 4 3 ) 1 -ν 2 1+ ν = O (1) . To reach E [ F ( x k )] -F ∗ ≤ ϵ for any ϵ > 0 , it suffices to make O ( [ H f ( ν ) D 1+ ν ϵ ] 2 1+3 ν + 1 b [ H max ( ν ) ϵ ] 2 1+ ν D 2 + σ 2 ∗ D 2 bϵ 2 ) queries to p g b .

Proof. Let k ≥ 1 be arbitrary and denote for brevity F k := E [ F ( x k )] -F ∗ , H f := H f ( ν ) and H max := H max ( ν ) . Taking into account Remark 35 and applying Theorem 8, we get, for any δ f , δ p g > 0 ,

where σ k := 4 σ ∗ ,b D √ 2 c 1 c 4 k = 4 σ ∗ D √ 2 c 1 c 4 bk . Minimizing the right-hand side in δ f and δ p g (using Lemma 36 twice) and estimating 1 3 ( k +2) ≤ 2 3 ( k +1) , we obtain

This proves the claimed convergence rate, and the oracle complexity bound easily follows since each iteration of the algorithm requires only O (1) queries to p g b .

Remark 42 . In the proof of Corollary 41, it was important that δ f and δ p g were allowed to be two separate constants. If we were not paying attention to such a separation and simply used the same δ everywhere, we would end up with the much weaker rate of O ( H max ( ν ) D 1+ ν b 1+ ν 2 k ν ) for the second term.

F.3 Explicit Variance Reduction with SVRG

Corollary 43. Consider problem (1) under the setting from Example 12 and also under Assumption 2. Let UniSvrg (as defined by Algorithm 3) be applied to this problem with the stochastic oracle p g = p g b and the full-gradient oracle s g = ∇ f . Then, for any t ≥ 1 ,

where c f ( ν ) := [2( c 2 c 4 +1)] 1+ ν 2 (2 s c 3 ) 1 -ν 2 1+ ν = O (1) , c p g ( ν ) := (96 c 1 c 4 ) 1+ ν 2 ( 8 3 ) 1 -ν 2 1+ ν = O (1) , s c 3 := max { c 3 , 1 } . To get E [ F (˜ x t )] -F ∗ ≤ ϵ , it suffices to make O ( N ν ( ϵ )) queries to p g b and O (log + N ν ( ϵ )) queries to ∇ f , where N ν ( ϵ ) := [ H f ( ν ) ϵ ] 2 1+ ν D 2 + 1 b [ H max ( ν ) ϵ ] 2 1+ ν D 2 . Assuming that the complexity of querying s g b is n b times bigger than that of querying ∇ f , we get the total stochastic-oracle complexity of O ( N ν ( ϵ ) + n b log + N ν ( ϵ )) .

Proof. Let t ≥ 1 be arbitrary and denote for brevity F t := E [ F (˜ x t )] -F ∗ , H f := H f ( ν ) and H max := H max ( ν ) . Taking into account Remark 35 and applying Theorem 10, we get, for any δ f , δ p g > 0 ,

Minimizing the right-hand side in δ f , δ p g (using Lemma 36 twice), we obtain

where c f := [2( c 2 c 4 +1)] 1+ ν 2 (2 s c 3 ) 1 -ν 2 1+ ν and c p g := (96 c 1 c 4 ) 1+ ν 2 ( 8 3 ) 1 -ν 2 1+ ν . This proves the claimed convergence rate.

Let us now estimate the oracle complexity. From the already proved convergence rate bound, we see that F t ≤ ϵ once 2 t ≥ O (1) N ( ϵ ) , where N ( ϵ ) := [ H f ϵ ] 2 1+ ν D 2 + 1 b [ H max ϵ ] 2 1+ ν D 2 . At the same time, according to Theorem 10, to generate the corresponding ˜ x t , the algorithm needs to make O (2 t ) queries to p g b and O ( t ) queries to ∇ f . Combining these two facts together, we get the claimed O ( N ( ϵ )) queries to p g b and O (log 2 N ( ϵ ) + 1) = O (log + N ( ϵ )) queries to ∇ f .

Corollary 44. Consider problem (1) under the setting from Example 12 and also under Assumption 2. Let UniFastSvrg (Algorithm 4) be applied to this problem with the stochastic oracle p g = p g b , the full-gradient oracle s g = ∇ f , and the epoch length N ≥ 9 . Then, for any t ≥ 2 t 0 , where t 0 := ⌈ log 2 log 3 N ⌉ -1 ( ≥ 0) , it holds that

where c f ( ν ) := [72( c 2 c 4 + 1 2 )] 1+ ν 2 s c 1 -ν 2 3 1+ ν = O (1) , c p g ( ν ) := (432 c 1 c 4 ) 1+ ν 2 ( 5 3 ) 1 -ν 2 1+ ν = O (1) , s c 3 := max { c 3 , 1 } . To get E [ F (˜ x t )] -F ∗ ≤ ϵ , it suffices to make O ( NT ν ( ϵ )) queries to p g b and O ( T ν ( ϵ )) queries to ∇ f , where T ν ( ϵ ) := [ H f ( ν ) D 1+ ν N 1+ ν 2 ϵ ] 2 1+3 ν +[ H max ( ν ) D 1+ ν ( bN ) 1+ ν 2 ϵ ] 2 1+3 ν +log 2 log 3 N . Assuming that the complexity of querying s g b is n b times bigger than that of querying ∇ f and choosing N = Θ( n b ) , we get the total stochastic-oracle complexity of O ( [ n ν b H f ( ν ) D 1+ ν ϵ ] 2 1+3 ν + [ n ν b H max ( ν ) D 1+ ν b (1+ ν ) / 2 ϵ ] 2 1+3 ν + n b log log n b )

Proof. Let t ≥ 2 t 0 be arbitrary, F t := E [ F (˜ x t )] -F ∗ , H f := H f ( ν ) , H max := H max ( ν ) . Taking into account Remark 35 and applying Theorem 11, we get, for any δ f , δ p g > 0 ,

Since t ≥ 2 t 0 , we can estimate t -t 0 +1 = 1 2 t + 1 2 t -t 0 +1 ≥ 1 2 ( t +1) , which gives us

Minimizing the right-hand side in δ f , δ p g (using Lemma 36 twice), we obtain

where c f := [72( c 2 c 4 + 1 2 )] 1+ ν 2 s c 1 -ν 2 3 1+ ν and c p g := (432 c 1 c 4 ) 1+ ν 2 ( 5 3 ) 1 -ν 2 1+ ν . This proves the claimed convergence rate.

Let us now estimate the number of oracle queries. In view of the above convergence rate bound, we have F t ≤ ϵ once t ≥ T ( ϵ ) := T 1 ( ϵ ) + 2 t 0 = O ( T 1 ( ϵ ) + log log N ) , where T 1 ( ϵ ) := [ H f D 1+ ν N (1+ ν ) / 2 ϵ ] 2 1+3 ν +[ H max D 1+ ν ( bN ) (1+ ν ) / 2 ϵ ] 2 1+3 ν = T 2 ( ϵ ) N 1+ ν 1+3 ν , where T 2 ( ϵ ) := [ H f D 1+ ν ϵ ] 2 1+3 ν +[ H max D 1+ ν b (1+ ν ) / 2 ϵ ] 2 1+3 ν does not depend on N . Combining this with Theorem 11 saying that, to generate the corresponding ˜ x t , the algorithm needs to make O ( Nt ) queries to p g b and O ( t ) queries to ∇ f , we get the claimed O ( NT ( ϵ )) queries to p g b and O ( T ( ϵ )) queries to ∇ f .

Assuming now that the complexity of querying ∇ f is n b times bigger than that of querying p g b , we get the total stochastic-oracle complexity of O ( ( N + n b ) T ( ϵ ) ) = O ( ( N + n b ) [ T 2 ( ϵ ) N (1+ ν ) / (1+3 ν ) + log log N ]) . Ignoring the doubly-logarithmic term, we get the expression of the form ( N + n b ) 1 N q = N 1 -q + n b N q with q := 1+ ν 1+3 ν ∈ [0 , 1] , whose minimal value is achieved at N = Θ( n b ) . Substituting this value into our complexity bound, we get the stochastic-oracle complexity of O ( n b ( T 2 ( ϵ ) n (1+ ν ) / (1+3 ν ) b +

G Additional Discussion of Related Work

Inexact Oracle and Approximate Smoothness. Devolder, Glineur, and Nesterov [15] introduced the notion of the inexact first-order oracle and analyzed the behaviour of several first-order methods for smooth convex optimization using such an oracle. Although their work was motivated by the desire to present the general definition of an inexact oracle covering many different applications, it was also observed that this oracle model is suitable for studying weakly smooth problems. This insight was later used in [45] to develop universal gradient methods for Hölder smooth problems. First stochastic gradient methods for approximately smooth functions with inexact oracle were proposed in [13]. These algorithms however are not adaptive and require the knowledge of problem-dependent constants. For more details on the subject, see [14].

0

20000

40000

60000

number of stochastic oracle calls

10

13

10

10

10

7

10

4

10

1

f(x)-

f

mushrooms

0

5000

10000

15000

20000

number of stochastic oracle calls

10

13

10

10

10

7

10

4

10

1

f(x)-

f

w8a

0

20000

40000

60000

number of stochastic oracle calls

10

5

10

4

10

3

10

2

10

1

10 0

f(x)-

f

leu

0

25000

50000

75000

100000

number of stochastic oracle calls

10

7

10

5

10

3

10

1

f(x)-

f

colon-cancer

0

20000

40000

number of stochastic oracle calls

10

6

10

5

10

4

10

3

10

2

10

1

f(x)-

f

0

100000

200000

number of stochastic oracle calls

10

6

10

5

10

4

10

3

10

2

10

1

f(x)-

f

0

50000 100000150000200000

number of stochastic oracle calls

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f(x)-

f

0

100000 200000 300000

number of stochastic oracle calls

10

6

10

4

10

2

10 0

f(x)-

f

UniSgd (ours)

UniFastSgd (ours)

AdaSVRG

UniSvrg (ours)

AdaVRAG

AdaVRAE

UniFastSvrg (ours)

Parameter-Free Methods. Parameter-free algorithms originating from the literature on online learning [9, 10, 26, 40, 47, 55] is another popular type of adaptive methods. They are usually endowed with mechanisms helping achieving efficiency bounds that are almost insensitive (typically, with logarithmic dependency) to the error of estimating certain problem parameters, such as the diameter of the feasible set [6, 11, 25, 31, 41].

Variance Reduction. Variance reduction techniques encompass a set of strategies that enhance the convergence speed of SGD when multiple passes are possible over the training dataset. Various researchers simultaneously introduced methods to reduce variance around the same period [27, 38, 50, 52, 58, 60]. The consideration of mini-batching in the context of these methods is documented in [3], while, in [20], it is shown that the convergence rate is influenced by both the average and the maximum smoothness of individual components. For further details, see [21] and the references therein.

Sometimes, it is even not necessary to use an explicit variance reduction mechanism. SGD may converge fast in the so-called over-parameterized regime, or when the stochastic noise is small at the optimal solution [8, 35, 37, 43, 44, 51]. In this work, we call this effect implicit variance reduction. Such a situation is also considered in [19, 54] and, more recently, Woodworth and Srebro [59] proposed an accelerated SGD algorithm for this setting, under the assumption that the smoothness and noise constants are known.

H Additional Experiments

H.1 Logistic Regression with Real-World Data

In this section, we present experiments on the logistic regression problem :

where a i ∈ R d and b i ∈ {-1 , 1 } are features and labels taken from diverse real-world datasets from LIBSVM [7]: mushrooms ( d ≪ n ), w8a ( d ≪ n ), leu ( d ≫ n ) and colon-cancer ( d ≫ n ). The dataset leu is quite special because it satisfies the so-called interpolation condition, meaning that the variance at the optimum is zero. We fix R = 1 and use the mini-batch size of b = 32 for the first two datasets and b = 1 for the last two.

Figure 3 shows the results of our experiments. The solid lines and the shaded area for each method represent, respectively, the mean and the region between the minimum and the maximum values after three independent runs of the algorithm. We see that, on the leu dataset, UniSgd and UniFastSgd converge as fast as the best non-accelerated and accelerated SVRG methods, respectively, which

0

50000

100000 150000 200000 250000

number of stochastic oracle calls

10 3

10 4

10 5

f(x)-

f

q = 1

0

25000 50000 75000 100000125000150000

number of stochastic oracle calls

10 4

10 5

10 6

10 7

f(x)-

f

q = 1.3

0

20000

40000

60000

number of stochastic oracle calls

10 5

10 6

10 7

10 8

10 9

f(x)-

f

q = 1.6

0

5000

10000

15000

number of stochastic oracle calls

10

1

10 1

10 3

10 5

10 7

10 9

10 11

f(x)-

f

q = 2

0

50000

100000 150000 200000 250000

number of stochastic oracle calls

10 4

10 5

10 6

Step size

0

25000 50000 75000 100000125000150000

number of stochastic oracle calls

10 3

10 4

Step size

0

20000

40000

60000

number of stochastic oracle calls

10 2

Step size

0

5000

10000

15000

number of stochastic oracle calls

10 0

4×10 1

6×10 1

2×10 0

Step size

UniSgd-AdaGrad

UniSgd-Other

UniSvrg-AdaGrad

UniSvrg-Other

0

50000

100000 150000 200000 250000

number of stochastic oracle calls

10 0

10 1

10 2

10 3

10 4

10 5

f(x)-

f

q = 1

0

25000 50000 75000 100000125000150000

number of stochastic oracle calls

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 7

f(x)-

f

q = 1.3

0

20000

40000

60000

number of stochastic oracle calls

10 2

10 4

10 6

10 8

f(x)-

f

q = 1.6

0

5000

10000

15000

number of stochastic oracle calls

10

6

10

3

10 0

10 3

10 6

10 9

10 12

f(x)-

f

q = 2

0

50000

100000 150000 200000 250000

number of stochastic oracle calls

10

1

10 0

10 1

10 2

10 3

10 4

10 5

10 6

Step size

0

25000 50000 75000 100000125000150000

number of stochastic oracle calls

10

1

10 0

10 1

10 2

10 3

10 4

10 5

Step size

0

20000

40000

60000

number of stochastic oracle calls

10

2

10

1

10 0

10 1

10 2

10 3

Step size

0

5000

10000

15000

number of stochastic oracle calls

10

2

10

1

10 0

Step size

UniFastSgd-AdaGrad

UniFastSgd-Other

UniFastSvrg-AdaGrad

UniFastSvrg-Other

confirms our theory on implicit variance reduction. Otherwise, these two SGD methods are typically much slower than the SVRG algorithms. Our UniSvrg method performs consistently better than AdaSVRG across all the datasets. Overall, all adaptive accelerated SVRG methods demonstrate comparable performance for solving these smooth problems.

H.2 Comparison between Stepsize Update Rules

In this section, we compare the AdaGrad stepsize rule (3) with the other rule (4) for UniSgd (Algorithm 1), UniFastSgd (Algorithm 2), UniSvrg (Algorithm 3), and UniFastSvrg (Algorithm 4). We consider the polyhedron feasibility and logistic regression problems under the same setups as in Section 8 and Appendix H.1.

The results are shown in Figs. 4 and 5, where we plot the function residual and the stepsize (inverse of M ) against stochastic oracle calls. We see that the two stepsize rules work very similarly across all test cases, which was not evident from the theory alone.

0

10000

20000

30000

40000

50000

number of stochastic oracle calls

10

14

10

12

10

10

10

8

10

6

10

4

10

2

10 0

f(x)-

f

q = 1

0

20000

40000

60000

80000

number of stochastic oracle calls

10

14

10

12

10

10

10

8

10

6

10

4

10

2

10 0

f(x)-

f

q = 1.3

0

20000

40000

60000

number of stochastic oracle calls

10

5

10

4

10

3

10

2

10

1

10 0

10 1

f(x)-

f

q = 1.6

0

20000

40000

60000

80000

100000

number of stochastic oracle calls

10

7

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f(x)-

f

q = 2

0

10000

20000

30000

40000

50000

number of stochastic oracle calls

10

1

10 0

10 1

Step size

0

20000

40000

60000

80000

number of stochastic oracle calls

10

1

10 0

10 1

Step size

0

20000

40000

60000

number of stochastic oracle calls

10

2

10

1

Step size

0

20000

40000

60000

80000

100000

number of stochastic oracle calls

10

2

10

1

Step size

UniSgd-AdaGrad

UniSgd-Other

UniSvrg-AdaGrad

UniSvrg-Other

0

10000

20000

30000

40000

50000

number of stochastic oracle calls

10

6

10

5

10

4

10

3

10

2

10

1

f(x)-

f

q = 1

0

50000

100000

150000

200000

250000

number of stochastic oracle calls

10

6

10

5

10

4

10

3

10

2

10

1

f(x)-

f

q = 1.3

0

25000 50000 75000100000125000150000

number of stochastic oracle calls

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f(x)-

f

q = 1.6

0

50000100000 150000 200000 250000 300000

number of stochastic oracle calls

10

7

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f(x)-

f

q = 2

0

10000

20000

30000

40000

50000

number of stochastic oracle calls

10

5

10

4

10

3

10

2

10

1

10 0

10 1

Step size

0

50000

100000

150000

200000

250000

number of stochastic oracle calls

10

6

10

5

10

4

10

3

10

2

10

1

10 0

10 1

Step size

0

25000 50000 75000100000125000150000

number of stochastic oracle calls

10

4

10

3

10

2

Step size

0

50000100000 150000 200000 250000 300000

number of stochastic oracle calls

10

7

10

6

10

5

10

4

10

3

10

2

10

1

Step size

UniFastSgd-AdaGrad

UniFastSgd-Other

UniFastSvrg-AdaGrad

UniFastSvrg-Other

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Each claim is justified by a theorem.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Each theorem is followed by a discussion. For the numerical experiments in Section 8 we carefully discuss the performance of the method. Additional experiments are presented in Appendix H.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes. This is a theory paper. All proofs are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the pseudocode and the hyperparameter settings for all algorithms.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the GitHub link to our code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Everything is carefully described in Section 8 and Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Different runs of the same algorithm for the same hyperparameter settings result in almost the same performance in our experiments. Therefore, we plot only one curve for each method, focusing more on highlighting the (more significant) differences between various methods.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: This is not important for the results we present since the algorithms are compared in terms of oracle calls and not the actual time of execution. All experiments are run on a standard MacBook Pro laptop and do not require a lot of computational power.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This is a theory paper.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This is a theory paper. We do not expect immediate societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Does not apply.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the LIBSVM dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any new assets in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No such experiments or research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Does not apply.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the adaptive gradient methods for solving convex composite optimization problems. The authors propose two main algorithms, UniSgd and UniFastSgd, equipped with AdaGrad stepsizes. They establish efficiency guarantees under various conditions, demonstrating implicit and explicit variance reduction properties. The results are also extended to incorporate SVRG-type variance reduction, leading to faster convergence rates.\n\nSTRENGTHS:\n1. This paper extends existing results by showing the universality of AdaGrad stepsizes in a more general setting, providing efficiency guarantees and state-of-the-art complexity bounds.\n2. The proposed methods are adaptive, not requiring problem-specific constants, making them practical for real-world applications.\n3. Numerical experiments validate the theoretical findings, showing the practical effectiveness of the proposed methods.\n\nWEAKNESSES:\n1. Although the authors call their algorithm adaptive methods, the proposed algorithm still requires knowing the domain diameter, which is an obvious drawback. Also, the reliance on the boundedness of the domain might also limit the application of the methods in more general or practical scenarios.\n2. The main body of this paper is difficult to understand. It seems that the algorithms are all rather complicated and the authors do not explain the methods clearly. Also, I think the authors should further emphasize the novelty and the challenge of this paper. \n3. More extensive empirical evaluations across a wider range of practical problems on real-world datasets would strengthen the experiments part of this paper.\n4. The authors claim in the abstract that “the main part can be accessed only via a (potentially **biased**) stochastic gradient oracle”, but it seems that the paper assumes an unbiased stochastic oracle. Maybe I missed something, but I do not find where we use the biased oracle. Can you explain this more clearly?\n\nQUESTIONS:\nSee the weaknesses part.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proves new convergence rates for stochastic gradient methods with\nAdaGrad step-sizes. The authors' build on the fact that AdaGrad step-sizes\nadapt to both the smooth and non-smooth settings and extend these results to\nshow convergence with biased stochastic gradient oracles on functions which are\nonly approximately smooth. Because Hölder-smooth functions satisfy this class,\nthis paper shows that AdaGrad step-sizes lead to universal methods which only\nrequire knowledge of the diameter, $D$. As part of their analysis, the authors\nalso show how to relax the bounded variance assumption and combine AdaGrad\nstep-sizes with acceleration and variance reduction.\n\nSTRENGTHS:\n- The theoretical setting is extremely general, covering composite optimization\n    with Hölder smooth functions with biased stochastic gradient oracles\n    satisfying a relaxed variance bound.\n\n- The analysis covers SGD as well as accelerated SGD and (accelerated) SGD \n    with variance reduction.\n\n- The synthetic experiments show that the proposed methods are faster than\n    existing methods and adapt to different degrees of smoothness in the\n    objective function.\n\nWEAKNESSES:\n- The contributions are fairly niche. Universality of (accelerated) SGD with\n    modified AdaGrad step-sizes is known [14], Adagrad is known to converge\n    without bounded variance [2], and AdaGrad step-sizes have been used with\n    (accelerated) SVRG previously [14, 34]. Thus, this paper fits into a small\n    gap by proving universality with a relaxed variance condition and covering\n    both acceleration and variance reduction.\n\n- ~The actually degree to which bounded variance is relaxed is not clear since\n    the connections between Assumption 3 and Assumption 6 are not developed.~\n\n- ~Variance reduce SGD with AdaGrad step-sizes (UniSvrg) only converges to a \n    neighbourhood of the optimal solution even when using an unbiased stochastic\n    gradient oracle.~\n\n- Experiments are only for synthetic problems and do not show the actual relationship\n    between oracle calls and optimization error. \n\n### Detailed Comments\n\n**Note**: The authors have clarified the connection between Assumptions 3 and 6 as well the role of $\\delta_g$ in Theorem 10. \n\n**Assumption 3 vs Assumption 6/9**: \n\nAn important and unaddressed issue is the comparison between between\nAssumption 3 and Assumption 6/9. Clearly Assumption 3 implies Assumption 6/9\nup to constant factors. But, in what circumstances do Assumptions 6/9 hold\nbut the variance is not bounded, i.e. Assumption 3 fails? In comparison to this\nwork, Nguyen et al. [2] give results for (strongly-convex) SGD that depend\nonly on the stochastic gradient variance at the minimizer and without \nany variance assumption. Indeed, the typical assumption in this setting is only\nthat $\\mathbb{E}[\\|g(x^*, \\xi)\\|_2^2]$ is finite.\n\n\n**Theorem 10**\n\nI'm somewhat disturbed that SVRG with Adagrad step-sizes doesn't converge\nexactly to a minimizer $x^*$ under your assumptions. For example, suppose\n$\\nabla f$ is $L$-Lipschitz and $\\bar g = \\nabla f$ so that $\\delta_f$ is zero.\nIn this setting, I expect variance-reduced methods like SVRG to converge\nexactly to a minimizer, but Theorem 10 shows UniSVRG converges to a\nneighbourhood of size $O(\\delta_{\\hat g})$. In comparison, standard SVRG, which\ndoes not require any variance bound (Allen-Zhu and Yuan, [1]), converges\nexactly. Can you please give sufficient conditions for $\\delta_{\\hat g}$ to be\nzero and explain how they compare to standard SVRG assumptions?\n\nAlso, why are both Assumptions 6 and 9 required simultaneously for this\ntheorem? They provide similar bounds on the variance of the gradient\nestimators, so I would have thought that Assumption 9 supplants Assumption 6.\n\n**Experiments**:\n\nI would have liked to see more realistic experiments. I understand that finding\nreal-world problems for which the diameter $D$ is bounded can be difficult, but\none synthetic regression problem is not sufficient to judge the performance of\nthe methods analyzed in this paper. In comparison, Dubois-Taine et al. [14]\ninclude a large number of experiments on real-world data.\n\nIn addition, I strongly dislike the choice to treat one mini-batch gradient\ncomputation as a single oracle call instead of $b$ calls.  While the choice has\nno effects in Figure 1, it is actually quite misleading in Figure 2, where it\nmakes the total SFO complexity of all the algorithms appear to decrease as\nbatch-size increases. In reality, the SFO complexity should decrease until a\nthreshold and then $b$ and then begin to increase again. That is, there is some\noptimal batch-size (usually less than $n$) which minimizes total complexity\n\n\n### Minor Comments\n\n- \"Gradient methods are the most popular and efficient optimization algorithms\n    for solving machine learning problems.\" --- They are not the most efficient\n    methods depending on the problem class and desired sub-optimality, so I\n    wouldn't say this.\n\n- \"However, the line-search approach is unsuitable for problems of stochastic\n    optimization, where gradients are observed with random noise.\" --- you may\n    want to mention that line-search works with stochastic gradients when\n    interpolation is satisfied [1].\n\n- So the difference between the results in Section 4 and [47] is that [47]\n    considers a modification of the Adagrad step-size? You might want to \n    state this modification so the difference is clear. \n\n- Contributions, Bullet 3: I don't like calling this variance reduction. It is\n    well-known that SGD with a constant step-size depends only on the variance\n    of the stochastic gradients at the minimizer [2], but no one would call SGD\n    variance reduced. \n\n- Table 1: \n    - Can you make the font-size larger? It's a little hard to read this\n    table. \n    - If you are reporting Big-O complexities, then can you please put them\n        in Big-O notation?\n    - You should explain in the caption why UniSgd and UniFastSgd have two \n        convergence rates each (e.g. variance assumptions).\n\n- Line 117: If it's satisfied for any $\\delta_f > 0$ then taking limits as\n    $\\delta_f \\rightarrow 0$ would imply that it's also satisfies for \n    $\\delta_f = 0$.\n\n- Line 136: I would say the classical method is due to Nesterov [3], while you\n    are referring to more recent acceleration methods for composite\n    optimization.\n\n- Eq. (4): How you solve this implicit equation, given that $\\hat x_+$ depends\n    on $\\hat M_+$ through the stochastic gradient update?\n\n- Assumption 6: In line 126, you say that the oracle outputs are assumed to be\n    independent. However, from the notation in Assumption 6, it seems that\n    in this case the oracle is evaluated with the same randomness $\\xi$.\n    Indeed, this must be the case else Assumption 6 reduces to Assumption 3.\n    Maybe you can add a remark clarifying this in the text?\n\n- Theorems 4,7: You should comment somewhere in the text that these rates \n    (according to Algorithm 1) are not anytime, but require knowledge of $N$.\n\n- Line 230: This sentence isn't finished.\n\n- Page 7: The spacing between the algorithm blocks is quite weird. You\n    should probably fix this. \n\n- Theorem 11: I would prefer if you stated this without assuming \n    $N \\in \\Theta(n)$. \n\n- Table 2, row 3: I think there's a typographic issue here since you appear to \n    be adding the definition of $N_\\nu(\\epsilon)$ with the log. \n\n- Figures 1, 2: \n    - The font sizes are much too small to read. As a rule of thumb,\n        the size of text in a figure should be at least as large as the size of\n        text in paragraph mode.\n    - Figure 1: It would be nice if you reminded the reader that $q$ is the\n        power in the test problem in Eq. (5).\n\n- Line 916: You should also reference SDCA [4] and SAG [5], which were very\n    early (if not the earliest) variance reduction methods.\n\n### References\n\n\n[1] Vaswani, Sharan, et al. \"Painless stochastic gradient: Interpolation, line-search, and convergence rates.\" Advances in neural information processing systems 32 (2019).\n\n[2] Nguyen, Lam, et al. \"SGD and Hogwild! convergence without the bounded gradients assumption.\" International Conference on Machine Learning. PMLR, 2018.\n\n[3] Nesterov, Yurii. \"A method for solving the convex programming problem with convergence rate O (1/k2).\" Dokl akad nauk Sssr. Vol. 269. 1983.\n\n[4] Shalev-Shwartz, Shai, and Tong Zhang. \"Stochastic dual coordinate ascent methods for regularized loss minimization.\" Journal of Machine Learning Research 14.1 (2013).\n\n[5] Schmidt, Mark, Nicolas Le Roux, and Francis Bach. \"Minimizing finite sums with the stochastic average gradient.\" Mathematical Programming 162 (2017): 83-112.\n\nQUESTIONS:\nTo reiterate, my questions are:\n\n- Why doesn't SVRG with AdaGrad step-sizes converge exactly and under what\n    conditions is $\\delta_{\\hat g} = 0$ satisfied?\n\n- Why are both Assumptions 6 and 9 required to show convergence of UniSVRG? \n\n- Under what conditions are Assumptions 6/9 weaker than Assumption 3?\n\nLIMITATIONS:\nThe contributions fit into a fairly small gap given previous work and experiments are not actually sufficient to justify the methods. See main review for details.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper demonstrates the universality of AdaGrad in stochastic optimization, presenting adaptive algorithms that converge efficiently without prior knowledge of problem-specific constants.  The research contributes novel variance reduction techniques, theoretical proofs, and empirical evidence, showcasing robust performance across optimization scenarios.  It advances the field by offering a versatile approach to stochastic optimization, hinting at potential extensions to more complex problem sets.

STRENGTHS:
The paper's strength lies in its theoretical depth, the algorithms are designed to be universally applicable to a wide range of optimization problems, including those with Hölder smooth components, showcasing a high level of adaptability. They also illustrate the impact of the mini-batch size on the convergence of proposed methods, which further helps to verify the theorem.

WEAKNESSES:
Rather than pointing out shortcomings, I would like to discuss some issues with the authors.

QUESTIONS:
The prior knowledge is also not required in UniXGrad[28] and AcceleGrad[32]. What is the difference between them and the proposed algorithm?

LIMITATIONS:
It is mentioned on line 246 of page 7 that "Alternative accelerated SVRG schemes with AdaGrad stepsizes (3) were recently proposed in [34]; however, they seem to be much more complicated. " I think it would be beneficial if the author discussed the algorithm in [34] in detail to help readers understand the advantages of the algorithm in this paper more clearly.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied how to apply AdaGrad type stepsize for stochastic convex optimization in a unified way. It proposed different algorithms and a general rule for the stepsize. Later, the authors presented different convergence rates under different settings, all of which match the existing best results. Finally, numerical experiments are given to show the better performance of these new algorithms.\n\nSTRENGTHS:\nThe paper also extends the result to the Hölder smooth case.\n\nWEAKNESSES:\n1. The major weakness is still Assumption 2. Though I understand this condition is widely used in the prior works with AdaGrad stepsize, this doesn't mean we should always accept it. \n\n    **i.** From the practical perspective, a bounded domain means one either has some prior information on the solution or puts some artificial constraints on the problem. However, both of them may not be realistic. A simple example is logistic regression with separable data (but this is unknown in advance). Then clearly, putting a bounded domain can guarantee the existence of $x^*$. However, one can notice that a better solution always exists but not in the domain.\n\n    **ii.** From the analysis perspective, with a bounded domain, one can immediately bound $\\frac{M_k\\\\|x_k-x^*\\\\|^2-M_k\\\\|x_{k+1}-x^\\*\\\\|^2}{2}\\leq\\frac{M_k\\\\|x_k-x^*\\\\|^2-M_{k+1}\\\\|x_{k+1}-x^\\*\\\\|^2}{2}+O((M_{k+1}-M_{k})D^2)$ to get a telescoping sum, which significantly simplify the analysis. In my experience, this is the major hardness. Without a bounded domain, one has to pay more effort to control this term or may meet other difficulties if one chooses to divide $M_k$ in the analysis.\n\n    **iii.** More importantly, I think different works with AdaGrad-like stepsize are trying to relax it recently as mentioned by the authors.\n    \n    **iv**. To summarize, under Assumption 2, the paper is more like a combination of different existing results. Nothing particularly new I can find in both algorithms and proofs. Even under Assumption 1 (or any inexact oracle), the proofs still follow the classical steps and one only needs to put $\\delta$ in the R.H.S. of every inequality.\n\n2. If I am not wrong, all results (except Theorems 7 and 8) in the paper hold for any $F(u)$ but not only $F^*$. If so, the authors can add a remark to state it to improve the paper. But feel free to correct me if I missed something.\n\n3. For some plots, I cannot find the confidence interval. Please add it according to the checklist.\n\nQUESTIONS:\nSee **Weaknesses**.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}