Title: Optimal Time Complexities of Parallel Stochastic Optimization Methods Under a Fixed Computation Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: nonconvex optimization

convex optimization

parallel methods

asynchronous methods

lower bounds
Abstract: Parallelization is a popular strategy for improving the performance of methods. Optimization methods are no exception: design of efficient parallel optimization methods and tight analysis of their theoretical properties are important research endeavors. While the minimax complexities are well known for sequential optimization methods, the theory of parallel optimization methods is less explored. In this paper, we propose a new protocol that generalizes the classical oracle framework approach. Using this protocol, we establish minimax complexities for parallel optimization methods that have access to an unbiased stochastic gradient oracle with bounded variance. We consider a fixed computation model characterized by each worker requiring a fixed but worker-dependent time to calculate stochastic gradient. We prove lower bounds and develop optimal algorithms that attain them. Our results have surprising consequences for the literature of asynchronous optimization methods.

Optimal Time Complexities of Parallel Stochastic Optimization Methods Under a Fixed Computation Model

Abstract

Parallelization is a popular strategy for improving the performance of iterative algorithms. Optimization methods are no exception: design of efficient parallel optimization methods and tight analysis of their theoretical properties are important research endeavors. While the minimax complexities are well known for sequential optimization methods, the theory of parallel optimization methods is less explored. In this paper, we propose a new protocol that generalizes the classical oracle framework approach. Using this protocol, we establish minimax complexities for parallel optimization methods that have access to an unbiased stochastic gradient oracle with bounded variance. We consider a fixed computation model characterized by each worker requiring a fixed but worker-dependent time to calculate stochastic gradient. We prove lower bounds and develop optimal algorithms that attain them. Our results have surprising consequences for the literature of asynchronous optimization methods.

1 Introduction

We consider the nonconvex optimization problem

where f : R d × S ξ → R , Q ⊆ R d , and ξ is a random variable with some distribution D on S ξ . In machine learning, S ξ could be the space of all possible data, D is the distribution of the training dataset, and f ( · , ξ ) is the loss of a data sample ξ. In this paper we address the following natural setup:

(i) n workers are available to work in parallel,

(ii) the i th worker requires τ i seconds 1 to calculate a stochastic gradient of f .

The function f is L -smooth and lower-bounded (see Assumptions 7.1-7.2), and stochastic gradients are unbiased and σ 2 -variance-bounded (see Assumption 7.3).

1.1 Classical theory

In the nonconvex setting, gradient descent (GD) is an optimal method with respect to the number of gradient ( ∇ f ) calls (Lan, 2020; Nesterov, 2018; Carmon et al., 2020) for finding an approximately stationary point of f . Obviously, a key issue with GD is that it requires access to the exact gradients

Peter Richt´ arik

∇ f of the function f. However, in many practical applications, it can be infeasible to calculate the gradient of E [ f ( · ; ξ )] analytically. Moreover, even if this is possible, e.g., if the distribution D is described by m possible samples, so that E ξ ∼D [ f ( · ; ξ )] = ( 1 / m ) ∑ m i =1 f ( · ; ξ i ) ,m can be huge (Krizhevsky et al., 2017), and gradient evaluation can be arbitrarily expensive.

Stochastic Gradient Descent. Due to the above-mentioned problem, machine learning literature is preoccupied with the study of algorithms that can work with stochastic gradients instead (Lan, 2020; Ghadimi and Lan, 2013). For all x ∈ R d , we assume that the n workers have access to independent, unbiased, and σ 2 -variance-bounded stochastic gradients ̂ ∇ f ( x, ξ ) (see Assumption 7.3), where ξ is a random sample from D . Under such assumptions, with one worker , stochastic gradient descent (SGD), i.e., the method x k +1 = x k -γ ̂ ∇ f ( x k ; ξ k ) , where ξ k are i.i.d. random samples from D , is known to be optimal with respect to the number of stochastic gradient calls (Ghadimi and Lan, 2013; Arjevani et al., 2022). SGD guarantees convergence to an ε -stationary point in expectation after O ( L ∆ / ε + σ 2 L ∆ / ε 2 ) stochastic gradient evaluations, where ∆ := f ( x 0 ) -f ∗ and x 0 ∈ R d is a starting point.

1.2 Parallel optimization methods

Using the bounds from Section 1.1, one can easily estimate the performance of these algorithms in real systems. For instance, if it takes τ 1 seconds to calculate a stochastic gradient with one worker , then SGD guarantees to return a solution after

seconds. If instead of a single worker we can access n workers that can calculate stochastic gradients in parallel, we can consider the following classical parallel methods:

Minibatch SGD. The minibatch SGD method (Minibatch SGD), i.e., the iterative process

where γ is a stepsize, ξ k i are i.i.d. samples from D , and the gradients ̂ ∇ f ( x k ; ξ k i ) are calculated in parallel. This method converges after O ( L ∆ / ε + σ 2 L ∆ / nε 2 ) iterations (Cotter et al., 2011; Goyal et al., 2017; Gower et al., 2019) and after

seconds, where τ max := max i ∈ [ n ] τ i is the processing time associated with the slowest machine 2 .

Although the time complexity (2) of Minibatch SGD improves with the number of workers n , in general, this does not guarantee better performance due to the delay τ max . In real systems, parallel computations can be very chaotic, e.g., they can be slow due to inconsistent network communications, or GPU computation delays (Dutta et al., 2018; Chen et al., 2016).

Asynchronous SGD. We now consider the asynchronous SGD method (Asynchronous SGD) (Recht et al., 2011; Nguyen et al., 2018; Arjevani et al., 2020; Feyzmahdavian et al., 2016) described by

Receive ̂ ∇ f ( x k -δ k ; ξ k -δ k ) from a worker ,

Ask the worker to calculate ̂ ∇ f ( x k +1 ; ξ k +1 ) ,

where ξ k are i.i.d. samples from D , and δ k are gradient iteration delays. This is an asynchronous method: the workers work independently, finish calculations of stochastic gradients with potentially large and chaotic delays δ k , and the result of their computation is applied as soon as it is ready, without having to wait for other workers. Asynchronous SGD was also considered in the heterogeneous setting (see details in Section A.2).

Cohen et al. (2021); Mishchenko et al. (2022); Koloskova et al. (2022) provide the current state-ofthe-art analysis of Asynchronous SGD. In particular, they prove that Asynchronous SGD converges after O ( nL ∆ / ε + σ 2 L ∆ / ε 2 ) iterations. To show the superiority of Asynchronous SGD, Mishchenko et al. (2022) consider the following fixed computation model : the i th worker requires τ i seconds to calculate stochastic gradients. In this setting, Asynchronous SGD converges after

seconds (we reprove this fact in Section L). Thus, Asynchronous SGD can be ( 1 / n ) ∑ n i =1 τ max / τ i times faster than Minibatch SGD.

Besides Asynchronous SGD, many other strategies utilize parallelization (Dutta et al., 2018; Woodworth et al., 2020; Wu et al., 2022), and can potentially improve over Minibatch SGD.

2 Problem and Contribution

In this paper, we seek to find the optimal time complexity in the setting from Section 1: our goal is to provide a lower bound and a method that attains it. Our main contributions are:

(i) Lower bound. In Sections 4 & 5, we define new Protocols 2 & 3, and a new complexity measure m time (see (6)), which we believe are more appropriate for the analysis of parallel optimization algorithms. In Section 6, we prove the time complexity lower bound for (nonconvex) functions and algorithms that work with parallel asynchronous oracles.

(i) Optimal method. In Section 7, we develop a minimax optimal method-Rennala 3 SGDthat attains this lower bound.

In addition, we investigate several other related questions. As an independent result, in Section 8 we prove that all methods which synchronize workers in each iteration (e.g., Minibatch SGD) have provably worse time complexity than asynchronous methods (e.g., Rennala SGD (see Method 4), Asynchronous SGD). In Section A, we extend our theory to the heterogeneous case, in which the workers have access to different distributions (datasets), and provide a lower bound and a new method that attains it. In Section B, we provide the optimal time complexities in the convex setting.

3 Classical Oracle Protocol

Let us recall the classical approach to obtaining lower bounds for optimization algorithms. We need to define a function class F , an oracle class O , and an algorithm class A . We then analyze the complexity of an algorithm A = { A k } ∞ k =0 ∈ A , using the following protocol:

Protocol 1 Classical Oracle Protocol

1: Input: function f ∈ F , oracle and distribution ( O, D ) ∈ O ( f ) , algorithm A ∈ A

2: for k = 0 , . . . , ∞ do

3: x k = A k ( g 1 , . . . , g k )

4: g k +1 = O ( x k , ξ k +1 ) , ξ k +1 ∼ D

5: end for

More formally, in first-order stochastic optimization, the oracle class O returns a random mapping O : R d × S ξ → R d based on a function f ∈ F and a distribution D ; we use the notation ( O, D ) ∈ O ( f ) . An algorithm A = { A k } ∞ k =0 ∈ A is a sequence such that

Typically, an oracle O returns an unbiased stochastic gradient that satisfies Assumption 7.3: O ( x, ξ ) = ̂ ∇ f ( x ; ξ ) for all x ∈ R d and ξ ∈ S ξ . Let us fix an oracle class O . Then, in the nonconvex first-order stochastic setting, we analyze the complexity measure

where the sequence { x k } k is generated by Protocol 1. Virtually all previous works are concerned with lower bounds of optimization problems using Protocol 1 and the complexity measure (5) (Nemirovskij and Yudin, 1983; Carmon et al., 2020; Arjevani et al., 2022; Nesterov, 2018).

4 Time Oracle Protocol

In the previous sections, we discuss the classical approach to estimating the complexities of algorithms. Briefly, these approaches seek to quantify the worst-case number of iterations or oracle calls that are required to find a solution (see (5)), which is very natural for sequential methods. However, and this is a key observation of our work, this approach is not convenient if we want to analyze parallel methods. We now propose an alternative protocol that can be more helpful in this situation:

Protocol 2 Time Oracle Protocol

1: Input: functions f ∈ F , oracle and distribution ( O, D ) ∈ O ( f ) , algorithm A ∈ A

2: s 0 = 0

3: for k = 0 , . . . , ∞ do

4: ( t k +1 , x k ) = A k ( g 1 , . . . , g k ) , 5: ( s k +1 , g k +1 ) = O ( t k +1 , x k , s k , ξ k +1 ) , ξ k +1 ∼ D

6: end for

Protocol 2 is almost identical to Protocol 1 except for one key detail: Protocol 2 requires the algorithms to return a sequence { t k +1 } ∞ k =1 such that t k +1 ≥ t k ≥ 0 for all k ≥ 0 . We assume that t 0 = 0 . We also assume that the oracles take to the input the states s k and output them (the role of these states will be made clear later). In this case, we provide the following definition of an algorithm.

Definition 4.1. An algorithm A = { A k } ∞ k =0 is a sequence such that

▷

x

0

=

A

0

for

k

= 0

.

and, for all k ≥ 1 and g 1 , . . . , g k ∈ R d , t k +1 ≥ t k , where t k +1 and t k are defined as ( t k +1 , · ) = A k ( g 1 , . . . , g k ) and ( t k , · ) = A k -1 ( g 1 , . . . , g k -1 ) .

Let us explain the role of the sequence { t k } k . In Protocol 1, an algorithm outputs a point x k and then asks the oracle: Provide me a gradient at the point x k . In contrast, in Protocol 2 an algorithm outputs a point x k and a time t k +1 , and asks the oracle: Start calculating a gradient at the point x k at a time t k +1 . We have a constraint that t k +1 ≥ t k for all k ≥ 0 , which means that the algorithm is not allowed to travel into the past.

Using Protocol 2, we propose to use another complexity measure instead of (5):

where the sequences t k and x k are generated by Protocol 2. In (5), we seek to find the worst-case number of iterations k required to get E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε for any A ∈ A . In (6), we seek to find the worst-case case time t required to find an ε -stationary point for any A ∈ A .

We now provide an example, considering an oracle that calculates a stochastic gradient in τ seconds. Let us define the appropriate oracle for this problem:

and ̂ ∇ f is a mapping such that ̂ ∇ f : R d × S ξ → R d . Further, we additionally assume that ̂ ∇ f is an unbiased σ 2 -variance-bounded stochastic gradient (see Assumption 7.3).

Note that the oracle O ̂ ∇ f τ emulates the behavior of a real worker. Indeed, the oracle can return three different outputs. If s q = 0 , it means that the oracle has been idle, then 'starts the calculation' of the gradient at the point x , and changes the state s q to 1 . Also, using the state, it remembers the time moment t when the calculation began and the point x. Next, if s q = 1 and t < s t + τ, it means the oracle is still calculating the gradient, so if an algorithm sends time t such that t < s t + τ, then it receives the zero vector. Finally, if s q = 1 , as soon as an algorithm sends time t such that t ≥ s t + τ, then the oracle will be ready to provide the gradient. Note that the oracle provides the gradient calculated at the point x that was requested when the oracle was idle. Thus, the time between the request of an algorithm to get the gradient and the time when the algorithm gets the gradient is at least τ seconds.

In Protocol 2, we have a game between an algorithm A ∈ A and an oracle class O , where algorithms can decide the sequence of times t k . Thus, an algorithm wants to find enough information from an oracle as soon as possible to obtain ε -stationary point.

Let us consider an example. For the oracle class O that generates the oracle from (7), we can define the SGD method in the following way. We take any starting point x 0 ∈ R d , a step size γ = min { 1 / L , ε / 2 Lσ 2 } (see Theorem D.8) and define A k : ( R d ×··· × R d ) ︸ ︷︷ ︸ k times → R ≥ 0 × R d such that

for all k ≥ 1 , and A 0 = (0 , x 0 ) . Let us explain the behavior of the algorithm. In the first step of Protocol 2, when k = 0 , the algorithm requests the gradient at the point x 0 at the time t 1 = 0 since

A 0 = (0 , x 0 ) . The oracle O changes the state from s 0 q = 0 to s 1 q = 1 and remembers the point x 0 in the state s 1 x . In the second step of the protocol, when k = 1 , the algorithm calls the oracle at the time τ ( ⌊ k / 2 ⌋ +1) = τ. In the oracle, the condition t 2 ≥ s 1 t + τ ⇔ τ ≥ 0 + τ is satisfied, and it returns the gradient at the point x 0 . Note that this can only happen if an algorithm does the second call at a time that is greater or equal to τ.

One can see that after τK seconds, the algorithm returns the point x 2 K = x 0 -γ ∑ K -1 j =0 ̂ ∇ f ( x 2 j ; ξ 2 j +1 ) , where ξ j ∼ D are i.i.d. random variables. The algorithm is equivalent to the SGD method that converges after K = O ( L ∆ / ε + σ 2 L ∆ / ε 2 ) steps for the function class F ∆ ,L (see Definition 6.1) for x 0 = 0 . Thus, the complexity m time ( { A } , F ∆ ,L ) equals O ( τ × ( L ∆ / ε + σ 2 L ∆ / ε 2 )) .

Actually, any algorithm that was designed for Protocol 1 can be used in Protocol 2 with the oracle (7). Assuming that we have mappings A k : R d ×··· × R d → R d for all k ≥ 1 , we can define mappings A k : R d ×··· × R d → R × R d via

For k = 0 , we define ̂ A 0 = (0 , A 0 ) .

5 Time Multiple Oracles Protocol

The protocol framework from the previous section does not seem to be very powerful because one can easily find the time complexity (6) by knowing (5) and the amount of time that oracle needs to calculate a gradient. In fact, we provide Protocol 2 for simplicity only. We now consider a protocol that works with multiple oracles:

Protocol 3 Time Multiple Oracles Protocol

̸

Compared to Protocol 2, Protocol 3 works with multiple oracles, and algorithms return the indices i k +1 of the oracle they want to call. This minor add-on to the protocol enables the possibility of analyzing parallel optimization methods. Also, each oracle O i can have its own distribution D i .

Let us consider an example with two oracles O 1 = O ̂ ∇ f τ 1 and O 2 = O ̂ ∇ f τ 2 from (7). One can see that a 'wise' algorithm will first call the oracle O 1 with the time t 0 = 0 , and then, in the second step, it will call the oracle O 2 also with the time t 1 = 0 . Note that it is impossible to do the following steps: in the first step an algorithm calls the oracle O 1 with the time t 0 = 0 , in the second step, the algorithm calls the oracle O 1 with the time t 1 = τ 1 and receives the gradient, in the third step, the algorithm calls the oracle O 2 with the time t 2 = 0 . Indeed, this can't happen because t 2 < t 1 .

An example of a 'non-wise' algorithm is an algorithm that, in the first step, calls the oracle O 1 with the time t 0 = 0 . In the second step, the algorithm calls the oracle O 1 with the time t 1 = τ 1 and receives the gradient. In the third step, the algorithm calls the oracle O 2 with the time t 2 = τ 1 . It would mean that the 'non-wise' algorithm did not use the oracle O 2 for τ 1 seconds. Consequently, the 'wise' algorithm can receive two gradients after max { τ 1 , τ 2 } seconds, while the 'non-wise' algorithm can only receive two gradients after τ 1 + τ 2 seconds.

We believe that Protocol 3 and the complexity (6) is a better choice for analyzing the complexities of parallel methods than the classical Protocol 1. In the next section, we will use Protocol 3 to obtain lower bounds for parallel optimization methods.

6 Lower Bound for Parallel Optimization Methods

Considering Protocol 3, we define a special function class F , oracle class O , and algorithm class A . We consider the same function class as Nesterov (2018); Arjevani et al. (2022); Carmon et al. (2020):

Definition 6.1 (Function Class F ∆ ,L ) . We assume that function f : R d → R is differentiable, L -smooth, i.e., ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ ∀ x, y ∈ R d , and ∆ -bounded, i.e., f (0) -inf x ∈ R d f ( x ) ≤ ∆ . A set of all functions with such properties we denote by F ∆ ,L .

In this paper, we analyze the class of 'zero-respecting' algorithms, defined next.

̸

Definition 6.2 (Algorithm Class A zr ) . Let us consider Protocol 3. We say that an algorithm A from Definition 4.1 is a zero-respecting algorithm, if supp ( x k ) ⊆ ⋃ k j =1 supp ( g j ) for all k ∈ N 0 , where supp ( x ) := { i ∈ [ d ] | x i = 0 } . A set of all algorithms with this property we define as A zr .

A zero-respecting algorithm does not try to change the coordinates for which no information was received from oracles. This family is considered by Arjevani et al. (2022); Carmon et al. (2020), and includes SGD, Minibatch and Asynchronous SGD, and Adam (Kingma and Ba, 2014).

Definition 6.3 (Oracle Class O σ 2 τ 1 ,...,τ n ) . Let us consider an oracle class such that, for any f ∈ F ∆ ,L , it returns oracles O i = O ̂ ∇ f τ i and distributions D i for all i ∈ [ n ] , where ̂ ∇ f is an unbiased σ 2 -variance-bounded mapping (see Assumption 7.3). The oracles O ̂ ∇ f τ i are defined in (7). We define such oracle class as O σ 2 τ 1 ,...,τ n . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n .

We take O σ 2 τ 1 ,...,τ n because it emulates the behavior of workers in real systems, where workers can have different processing times (delays) τ i . Note that O σ 2 τ 1 ,...,τ n has the freedom to choose a mapping ̂ ∇ f. We only assume that the mapping is unbiased and σ 2 -variance-bounded. We are now ready to present our first result; a lower bound:

Theorem 6.4. Let us consider the oracle class O σ 2 τ 1 ,...,τ n for some σ 2 > 0 and 0 < τ 1 ≤ · · · ≤ τ n . We fix any L, ∆ > 0 and 0 < ε ≤ c ′ L ∆ . In view Protocol 3, for any algorithm A ∈ A zr , there exists a function f ∈ F ∆ ,L and oracles and distributions (( O 1 , . . . , O n ) , ( D 1 , . . . , D n )) ∈ O σ 2 τ 1 ,...,τ n ( f ) such that E [ inf k ∈ S t ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] > ε, where S t := { k ∈ N 0 ∣ ∣ t k ≤ t } , and

The quantities c ′ and c are universal constants.

Theorem 6.4 states that

The interpretation behind this complexity will be discussed later in Section 7.3. No algorithms known to us attain (9). For instance, Asynchronous SGD has the time complexity (3). Let us assume that σ 2 / ε ≤ p and p ∈ [ n ] . Then (lower bound from (9)) = O (( 1 p ∑ p i =1 1 τ i ) -1 ( L ∆ ε )) . In this case, the lower bound in (9) will be at least ( 1 n ∑ n i =1 1 τ i ) -1 / ( 1 p ∑ p i =1 1 τ i ) -1 times smaller. It means that either the obtained lower bound is not tight, or Asynchronous SGD is a suboptimal method. In the following section we provide a method that attains the lower bound. The obtained lower bound is valid even if an algorithm has the freedom to interrupt oracles. See details in Section F.

6.1 Related work

For convex problems, Woodworth et al. (2018) proposed the graph oracle, which generalizes the classical gradient oracle (Nemirovskij and Yudin, 1983; Nesterov, 2018), and provided lower bounds

for a rather general family of parallel methods. Arjevani et al. (2020) analyzed the delayed gradient descent method, which is Asynchronous SGD when all iteration delays δ k = δ are a constant.

As far as we know, Woodworth et al. (2018) provide the most suitable and tightest prior framework for analyzing lower bound complexities for problem (1). However, as we shall see, our framework us more powerful. Moreover, they only consider the convex case. In Section M, we use the framework of Woodworth et al. (2018) and analyze the fixed computation model, where i th worker requires τ i seconds to calculate stochastic gradients. In Section B, we consider the convex setting and show that the lower bound obtained by their framework is not tight and can be improved. While the graph oracle framework by Woodworth et al. (2018) is related to the classical oracle protocol (Section 3) and also calculates the number of oracle calls in order to get lower bounds, our approach directly estimates the required time. For more details, see Section B and the discussion in Section B.1.1.

Method 4 Rennala SGD

7 Minimax Optimal Method

We now propose and analyze a new method: Rennala SGD (see Method 4). Methods with a similar structure were proposed previously (e.g., (Dutta et al., 2018)), but we are not aware of any method with precisely the same parameters and structure. For us, in this paper, the theoretical bounds are more important than the method itself.

Let us briefly describe the structure of the method. At the start, Method 4 asks all workers to run Method 5. Method 5 is a standard routine: the workers receive points x k from the server, calculate stochastic gradients, and send them back to the server. Besides that, the workers receive and send the iteration counter k of the received points x k . At the server's side, in each iteration k, Method 4 calculates g k and performs the standard gradient-type step x k +1 = x k -γg k . The calculation of g k is done in a loop. The server waits for the workers to receive a stochastic gradient and an iteration index. The most important part of the method is that the server ignores a stochastic gradient if its iteration index is not equal to the current iteration index. In fact, this means that g k = ( 1 / S ) ∑ S i =1 ̂ ∇ f ( x k ; ξ i ) , where ξ i are i.i.d. samples. In other words, the server ignores all stochastic gradients that were calculated at the points x 0 , · · · , x k -1 .

It may seem that Method 4 does not fully use the information due to ignoring some stochastic gradients. That contradicts the philosophy of Asynchronous SGD, which tries to use all stochastic gradients calculated in the previous points. Nevertheless, we show that Rennala SGD has better time complexity than Asynchronous SGD, and this complexity matches the lower bound from Theorem 6.4. The fact that Rennala SGD ignores the previous iterates is motivated by the proof of the lower bound in Section 6. In the proof, any algorithm, on the constructed 'worst case' function, does not progress to a stationary point if it calculates a stochastic gradient at a non-relevant point. This suggested to us to construct a method that would focus all workers on the last iterate.

7.1 Assumptions

Let us consider the following assumptions.

Method 5 Worker's Infinite Loop

Assumption 7.1. f is differentiable & L -smooth, i.e., ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ , ∀ x, y ∈ R d .

Assumption 7.2. There exist f ∗ ∈ R such that f ( x ) ≥ f ∗ for all x ∈ R d .

Assumption 7.3. For all x ∈ R d , stochastic gradients ̂ ∇ f ( x ; ξ ) are unbiased and σ 2 -variancebounded, i.e., E ξ [ ̂ ∇ f ( x ; ξ ) ] = ∇ f ( x ) and E ξ [ ∥ ∥ ∥ ̂ ∇ f ( x ; ξ ) -∇ f ( x ) ∥ ∥ ∥ 2 ] ≤ σ 2 , where σ 2 ≥ 0 .

7.2 Analysis of Rennala SGD

Theorem 7.4. Assume that Assumptions 7.1, 7.2 and 7.3 hold. Let us take the batch size S = max {⌈ σ 2 / ε ⌉ , 1 } , and γ = min { 1 L , εS 2 Lσ 2 } = Θ( 1 / L ) in Method 4. Then after

iterations, the method guarantees that 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

In the following theorem, we provide the time complexity of Method 4.

Theorem 7.5. Consider Theorem 7.4. We assume that i th worker returns a stochastic gradient every τ i seconds for all i ∈ [ n ] . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n . Then after

seconds, Method 4 guarantees to find an ε -stationary point.

This result with Theorem 6.4 state that

for Protocol 3 and and the oracle class O σ 2 τ 1 ,...,τ n from Definition 6.3.

7.3 Discussion

Theorem 7.5 and Theorem 6.4 state that Method 4 is minimax optimal under the assumption that the delays of the workers are fixed and equal to τ i . Note that this assumption is required only in Theorem 7.5, and Theorem 7.4 holds without it.

In the same setup, the previous works (Cohen et al., 2021; Mishchenko et al., 2022; Koloskova et al., 2022) obtained the weaker time complexity (3). We do not rule out that it might be possible for the analysis, the parameters or the structure of Asynchronous SGD to be improved and obtain the optimal time complexity (10). We leave this to future work. However, instead, we developed Method 4 that has not only the optimal time complexity, but also a very simple structure and analysis (see Section D.4.1). Our claims are supported by experiments in Section J.

The reader can see that we provide the complexity in a nonconstructive way, as the minimization over the parameter m ∈ [ n ] . Note that Method 4 automatically finds the optimal m in (7.5), and it does not require the knowledge of the delays τ i to do so! Let us explain the intuition behind the complexity (10). Let m ∗ be the optimal parameter of (10) with the smallest index. In Section D.4.3, we show that all workers with the delays τ i for all i > m ∗ can be simply ignored since their delays are too large, and their inclusion would only harm the convergence time of the method. So, the method automatically ignores them! However, in Asynchronous SGD, these harmful workers can contribute to the optimization process, which can be the reason for the suboptimality of Asynchronous SGD.

In general, there are two important regimes: σ 2 / ε ≪ n ('low noise/large # of workers') and σ 2 / ε ≫ n ('high noise/small # of workers'). Intuitively, in the 'high noise/small # of workers' regime, (11) is minimized when m is close to n. However, in the 'low noise/large # of workers', the optimal m can be much smaller than n.

8 Synchronized Start of Workers

In the previous sections, we obtain the time complexities for the case when the workers asynchronously compute stochastic gradients. It is important that the complexities are obtained assuming that the workers can start their calculations asynchronously. However, in practice, it is common to train machine learning models with multiple workers/GPUs, so that all workers are synchronized after each stochastic gradient calculation (Goyal et al., 2017; Sergeev and Balso, 2018). The simplest example of such a strategy is Minibatch SGD (see Section 1.2). We want to find an answer to the question: what is the best time complexity we can get if we assume that the workers start simultaneously? In Section G, we formalize this setting, and show that the time complexity equals to

for Protocol 2 and the oracle class O σ 2 , sync τ 1 ,...,τ n from Definition G.1. Comparing (11) and (12), one can see that methods that start the calculations of workers simultaneously are provably worse than methods that allow workers to start the calculations asynchronously .

9 Future Work

In this work, we consider the setup where the times τ i are fixed. In future work, one can consider natural, important, and more general scenarios where they can be random, follow some distribution, and/or depend on the random variables ξ from Assumption 7.3 (be correlated with stochastic gradients).

References

Arjevani, Y., Carmon, Y., Duchi, J. C., Foster, D. J., Srebro, N., and Woodworth, B. (2022). Lower bounds for non-convex stochastic optimization. Mathematical Programming , pages 1-50.

Arjevani, Y., Shamir, O., and Srebro, N. (2020). A tight convergence analysis for stochastic gradient descent with delayed updates. In Algorithmic Learning Theory , pages 111-132. PMLR.

Aytekin, A., Feyzmahdavian, H. R., and Johansson, M. (2016). Analysis and implementation of an asynchronous optimization algorithm for the parameter server. arXiv preprint arXiv:1610.05507 .

Carmon, Y., Duchi, J. C., Hinder, O., and Sidford, A. (2020). Lower bounds for finding stationary points i. Mathematical Programming , 184(1):71-120.

Chen, J., Pan, X., Monga, R., Bengio, S., and Jozefowicz, R. (2016). Revisiting distributed synchronous sgd. arXiv preprint arXiv:1604.00981 .

Cohen, A., Daniely, A., Drori, Y., Koren, T., and Schain, M. (2021). Asynchronous stochastic optimization robust to arbitrary delays. Advances in Neural Information Processing Systems , 34:9024-9035.

Cotter, A., Shamir, O., Srebro, N., and Sridharan, K. (2011). Better mini-batch algorithms via accelerated gradient methods. Advances in Neural Information Processing Systems , 24.

Dutta, S., Joshi, G., Ghosh, S., Dube, P., and Nagpurkar, P. (2018). Slow and stale gradients can win the race: Error-runtime trade-offs in distributed SGD. In International Conference on Artificial Intelligence and Statistics , pages 803-812. PMLR.

Feyzmahdavian, H. R., Aytekin, A., and Johansson, M. (2016). An asynchronous mini-batch algorithm for regularized stochastic optimization. IEEE Transactions on Automatic Control , 61(12):3740-3754.

Ghadimi, S. and Lan, G. (2013). Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368.

Gower, R. M., Loizou, N., Qian, X., Sailanbayev, A., Shulgin, E., and Richt´ arik, P. (2019). SGD: General analysis and improved rates. In International Conference on Machine Learning , pages 5200-5209. PMLR.

Goyal, P., Doll´ ar, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. (2017). Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 .

Khaled, A. and Richt´ arik, P. (2020). Better theory for SGD in the nonconvex world. arXiv preprint arXiv:2002.03329 .

Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .

Koloskova, A., Stich, S. U., and Jaggi, M. (2022). Sharper convergence guarantees for asynchronous SGD for distributed and federated learning. arXiv preprint arXiv:2206.08307 .

Koneˇ cn´ y, J., McMahan, H. B., Yu, F. X., Richt´ arik, P., Suresh, A. T., and Bacon, D. (2016). Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 .

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2017). Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90.

Lan, G. (2020). First-order and stochastic optimization methods for machine learning . Springer.

LeCun, Y., Cortes, C., and Burges, C. (2010). Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2.

Mishchenko, K., Bach, F., Even, M., and Woodworth, B. (2022). Asynchronous SGD beats minibatch SGD under arbitrary delays. arXiv preprint arXiv:2206.07638 .

Mishchenko, K., Iutzeler, F., Malick, J., and Amini, M.-R. (2018). A delay-tolerant proximal-gradient algorithm for distributed learning. In International Conference on Machine Learning , pages 3587-3595. PMLR.

Nemirovskij, A. S. and Yudin, D. B. (1983). Problem complexity and method efficiency in optimization.

Nesterov, Y. (2018). Lectures on convex optimization , volume 137. Springer.

Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Malek, M., and Huba, D. (2022). Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics , pages 3581-3607. PMLR.

Nguyen, L., Nguyen, P. H., Dijk, M., Richt´ arik, P., Scheinberg, K., and Tak´ ac, M. (2018). SGD and hogwild! convergence without the bounded gradients assumption. In International Conference on Machine Learning , pages 3750-3758. PMLR.

Recht, B., Re, C., Wright, S., and Niu, F. (2011). Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Advances in Neural Information Processing Systems , 24.

Sergeev, A. and Balso, M. D. (2018). Horovod: Fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799 .

Woodworth, B., Patel, K. K., Stich, S., Dai, Z., Bullins, B., Mcmahan, B., Shamir, O., and Srebro, N. (2020). Is local SGD better than minibatch SGD? In International Conference on Machine Learning , pages 10334-10343. PMLR.

Woodworth, B. E., Wang, J., Smith, A., McMahan, B., and Srebro, N. (2018). Graph oracle models, lower bounds, and gaps for parallel stochastic optimization. Advances in Neural Information Processing Systems , 31.

Wu, X., Magnusson, S., Feyzmahdavian, H. R., and Johansson, M. (2022). Delay-adaptive step-sizes for asynchronous learning. arXiv preprint arXiv:2202.08550 .

Contents

A Heterogeneous Regime

Up to this point, we discussed the regime when all workers calculate i.i.d. stochastic gradients. In distributed optimization and federated learning (Koneˇ cn´ y et al., 2016), it can be possible that the workers hold different datasets. Let us consider the following optimization problem:

where f i : R d × S ξ i → R d and ξ i are random variables with some distributions D i on S ξ i . Problem (13) generalizes problem (1). Here we have the same goals as in the previous sections. We want to obtain the minimax complexities for the case when the workers contain different datasets.

A.1 Lower bound

For the heterogeneous case, we modify Definition 6.3:

Definition A.1 (Oracle Class O σ 2 , heterog τ 1 ,...,τ n ) .

Let us consider an oracle class such that, for any f = 1 n ∑ n i =1 f i ∈ F ∆ ,L , it returns oracles O i = O ̂ ∇ f i τ i and distributions D i for all i ∈ [ n ] , where ̂ ∇ f i is an unbiased σ 2 -variance-bounded mapping for all i ∈ [ n ] (see Assumption 7.3). The oracles O ̂ ∇ f i τ i are defined in (7). We define such oracle class as O σ 2 , heterog τ 1 ,...,τ n . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n .

2

Theorem A.2. Let us consider the oracle class O σ , heterog τ 1 ,...,τ n for some σ 2 > 0 and 0 < τ 1 ≤ · · · ≤ τ n . We fix any L, ∆ > 0 and 0 < ε ≤ c ′ L ∆ . In the view Protocol 3, for any algorithm A ∈ A zr , there exists a function f = 1 n ∑ n i =1 f i ∈ F ∆ ,L and oracles and distributions (( O 1 , . . . , O n ) , ( D 1 , . . . , D n )) ∈ O σ 2 , heterog τ 1 ,...,τ n ( f 1 , . . . , f n ) such that E [ inf k ∈ S t ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] > ε, where S t := { k ∈ N 0 ∣ ∣ t k ≤ t } , and

The quantity c ′ and c are universal constants.

Theorem A.2 states that

One can see that the lower bound for the heterogeneous case is larger than (9). In Section A.3, we provide a method that attains the lower bound.

A.2 Related work and discussion

The optimization problem (13) is well-investigated by many papers, including (Aytekin et al., 2016; Mishchenko et al., 2018; Nguyen et al., 2022; Wu et al., 2022; Koloskova et al., 2022; Mishchenko et al., 2022). There were attempts to analyze Asynchronous SGD in the heterogeneous regime. For instance, Mishchenko et al. (2022) proved the convergence to a neighborhood of a solution only. In general, it is quite challenging to get good rates for Asynchronous SGD without additional assumptions about the similarity of the functions f i (Koloskova et al., 2022; Mishchenko et al., 2022).

In the non-stochastic case, when σ 2 = 0 , Wu et al. (2022) analyzed the PIAG method in the nonstochastic heterogeneous regime and showed convergence. Although the performance of PIAG can be good in practice, in the worst case PIAG requires O ( τ n ̂ L ∆ / ε ) seconds to converge, where τ n is the time delay of the slowest worker, ̂ L := √∑ n i =1 L 2 i , and L i is a Lipschitz constant of ∇ f i . Note that the synchronous Minibatch SGD (see Section 1.2) method has the complexity O( τ n L ∆ / ε ) , which is always better. 4

Our lower bound in Theorem A.2 does not leave hope of breaking the dependence on the worst straggler in the heterogeneous case. In the stochastic case, the lower bound is slightly more optimistic in the regimes when the statistical term (the second term in (14)) is large. If the stragglers do not have too large delays, then their contributions to the arithmetic mean can be small. Note that in Theorem 6.4 in the homogeneous case, we have the harmonic mean of the delays instead.

A.3 Minimax optimal method

In this section, we provide Malenia 5 SGD (see Method 6) that is slightly different from Rennala SGD (Method 4). There are two main differences: the first one is that Method 6 has different gradients estimators g k i for each worker, and the second one is the constraint ( 1 n ∑ n i =1 1 / B i ) -1 < S / n in the inner loop 6 . The more gradients we get from the workers, the larger the term ( 1 n ∑ n i =1 1 / B i ) -1 .

As in Section 7.2, we can provide the convergence theorems.

Theorem A.3. Assume that Assumptions 7.1 and 7.2 hold for the function f . Assumption 7.3 holds for the function f i for all i ∈ [ n ] . Let us take the parameter S = max {⌈ σ 2 / ε ⌉ , n } , and

Method 7 Worker's Infinite Loop

1: Init g = 0 , k ′ = -1 , and worker's index i

2: while True do

3: Send ( g, k ′ , i ) to the server

4: Receive ( x k , k ) from the server

5: k ′ = k

6: g = ̂ ∇ f i ( x k ; ξ ) , ξ ∼ D

7: end while

γ = min { 1 L , εS 2 Lσ 2 } = Θ( 1 / L ) in Method 6, then after K ≥ 24∆ L / ε iterations the method guarantees that 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

Theorem A.4. Let us consider Theorem A.3. We assume that i th worker returns a stochastic gradient every τ i seconds for all i ∈ [ n ] . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n . Then after

seconds, Method 6 guarantees to find an ε -stationary point.

Comparing Theorem A.2 and Theorem A.4, one can see that the complexity (15) is optimal. Note that Theorem A.3 holds without assumptions that the delays τ i are fixed.

A.4 Discussion

Unlike Asynchronous SGD and PIAG, Malenia SGD ignores all stochastic gradients that were calculated in the previous iterations, which appears to be counterproductive. Nevertheless, we show that Malenia SGD converges, and the time complexity is optimal with respect to all parameters. Note that Malenia SGD does not require the Lipschitz smoothness of the local functions f i , does not depend on the time delays τ i , does not need any similarity assumptions about the functions f i , and can be applied to problems where the function is not Lipschitz (does not have bounded gradients). The analysis of the method is elementary and does not go far away from the theory of the classical SGD method. When the ratio σ 2 / ε is large, Malenia SGD is better than Minibatch SGD (see (2)) by Θ ( τ n / ( 1 n ∑ n i =1 τ i )) times.

B Convex Case

B.1 Lower Bound

Let us consider the optimization problem (1) in the case when the function f is convex. For the convex case, using Protocol 2, we propose to use another complexity measure instead of (6):

where the sequences t k and x k are generated by Protocol 2. Let us consider the following class of convex functions:

Definition B.1 (Function Class F conv R,M,L ) .

Let us define B 2 (0 , R ) := { x ∈ R d | ∥ x ∥ ≤ R } . We assume that a function f : R d → R is convex, differentiable, L -smooth on the set B 2 (0 , R ) , i.e.,

and M -Lipschitz on the set B 2 (0 , R ) , i.e.,

A set of all functions with such properties we define as F conv R,M,L .

For the convex case, we analyze the following class of algorithms:

Definition B.2 (Algorithm Class A R zr ) .

Let us consider Protocol 3. We say that an algorithm A from Definition 4.1 belongs to a class A R zr iff A ∈ A zr and x k ∈ B 2 (0 , R ) for all k ≥ 0 .

We also define an oracle class:

Definition B.3 (Oracle Class O conv ,σ 2 τ 1 ,...,τ n ) .

Let us consider an oracle class such that, for any f ∈ F conv R,M,L , it returns oracles O i = O ̂ ∇ f τ i and distributions D i for all i ∈ [ n ] , where ̂ ∇ f is an unbiased σ 2 -variance-bounded mapping on the set B 2 (0 , R ) . The oracles O ̂ ∇ f τ i are defined in (7). We define such oracle class as O conv ,σ 2 τ 1 ,...,τ n . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n .

For this setup, we provide the lower bound for the class of convex functions in the next theorem.

Theorem B.4. Let us consider the oracle class O conv ,σ 2 τ 1 ,...,τ n for some σ 2 > 0 and 0 < τ 1 ≤ · · · ≤ τ n . We fix any R,L,M,ε > 0 such that √ LR > c 1 √ ε > 0 and M 2 R 2 > c 2 ε 2 . In the view Protocol 3, for any algorithm A ∈ A R zr , there exists a function f ∈ F conv R,M,L and oracles and distributions (( O 1 , . . . , O n ) , ( D 1 , . . . , D n )) ∈ O conv ,σ 2 τ 1 ,...,τ n ( f ) such that

where S t := { k ∈ N 0 ∣ ∣ t k ≤ t } , and

The quantities c 1 , c 2 and c are universal constants.

B.1.1 Discussion

We improve the lower bound obtained by (Woodworth et al., 2018) (see Table 3). Woodworth et al. (2018) try to reduce any optimization problem to an oracle graph. Then, they get a lower bound using the depth and the number of nodes in a graph. Our approach is different, as we directly estimate the required time and avoid the reduction to an oracle graph. One can think that our 'oracle graph' is always linear in Protocol 3, but every node in an 'oracle graph' is associated with a timestamp and an index. Unlike the oracle in (Woodworth et al., 2018), which always returns a stochastic gradient, our oracle (7) returns a stochastic gradient only if the conditions are satisfied. Also, Woodworth et al. (2018) construct different 'worst case' functions and oracles for the 'optimization' and 'statistical' terms. While our construction consists only of one function and one oracle.

B.2 Minimax optimal method

B.2.1 Assumptions

Additionally to some assumptions from Section 7.1, we use the following assumptions in the convex case.

Assumption B.5. The function f is convex and attains the minimum at some point x ∗ ∈ R d .

Assumption B.6. The function f is M

-Lipschitz, i.e.,

Assumption B.7. For all x ∈ R d , stochastic gradients ̂ ∇ f ( x ; ξ ) are unbiased and have σ 2 -variancebounded, i.e., E ξ ∼D [ ̂ ∇ f ( x ; ξ ) ] ∈ ∂f ( x ) and E ξ ∼D [ ∥ ∥ ∥ ̂ ∇ f ( x ; ξ ) -E [ ̂ ∇ f ( x ; ξ ) ]∥ ∥ ∥ 2 ] ≤ σ 2 , where σ 2 ≥ 0 .

B.2.2 Analysis of Rennala SGD and Accelerated Rennala SGD in convex case

Theorem B.8. Assume that Assumptions B.5, B.6 and B.7 hold. Let us take the batch size S = max {⌈ σ 2 / M 2 ⌉ , 1 } , and γ = ε M 2 + σ 2 /S = Θ( ε / M 2 ) in Method 4, then after K ≥ 2 M 2 R 2 / ε 2 iterations the method guarantees that E [ f ( ̂ x K ) ] -f ( x ∗ ) ≤ ε, where ̂ x K = 1 K ∑ K -1 k =0 x k and R = ∥ ∥ x ∗ -x 0 ∥ ∥ .

Theorem B.9. Let us consider Theorem B.8. We assume that i th worker returns a stochastic gradient every τ i seconds for all i ∈ [ n ] . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n . Then after

seconds Method 4 guarantees to find an ε -solution.

Let us provide the theorems in the smooth convex case. We consider the accelerated version of Rennala SGD. In particular, we assume that instead of Line 13 in Method 4, we have

We refer to such method as Accelerated Method 4 or Accelerated Rennala SGD. The acceleration technique is based on (Lan, 2020).

Theorem B.10. Assume that Assumptions B.5, 7.1 and 7.3 hold. Let us take the batch size S = max {⌈ ( σ 2 R ) / ( ε 3 / 2 √ L ) ⌉ , 1 } , and γ = min { 1 4 L , [ 3 R 2 S 4 σ 2 ( K +1)( K +2) 2 ] 1 / 2 } in Accelerated Method 4, then after K ≥ 8 √ LR √ ε iterations the method guarantees that E [ f ( x K ) ] -f ( x ∗ ) ≤ ε, where R ≥ ∥ ∥ x ∗ -x 0 ∥ ∥ .

Theorem B.11. Let us consider Theorem B.10. We assume that i th worker returns a stochastic gradient every τ i seconds for all i ∈ [ n ] . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n . Then after

seconds Accelerated Method 4 guarantees to find an ε -solution.

C Table of Notations

D Proofs for Homogeneous Regime

D.1 The 'worst case' function

In this section, we recall the 'worst case' function that we use to prove our lower bounds. This is the standard function that is used in nonconvex optimization. Let us define

̸

In our proofs, we use the construction from (Carmon et al., 2020; Arjevani et al., 2022). For any T ∈ N , the authors define

where

The main property of the function F T ( x ) is that its gradients are large unless prog ( x ) ≥ T. Lemma D.1 (Carmon et al. (2020); Arjevani et al. (2022)) . The function F T satisfies:

F T (0) -inf x ∈ R T F T ( x ) ≤ ∆ 0 T, where ∆ 0 = 12 .

The function F T is l 1 -smooth, where l 1 = 152 .

For all x ∈ R T , ∥∇ F T ( x ) ∥ ∞ ≤ γ ∞ , where γ ∞ = 23 .

For all x ∈ R T , prog ( ∇ F T ( x )) ≤ prog ( x ) + 1 .

For all x ∈ R T , if prog ( x ) < T, then ∥∇ F T ( x ) ∥ > 1 .

We use these properties in the proofs.

D.2 Proof of Theorem 6.4

Theorem 6.4. Let us consider the oracle class O σ 2 τ 1 ,...,τ n for some σ 2 > 0 and 0 < τ 1 ≤ · · · ≤ τ n . We fix any L, ∆ > 0 and 0 < ε ≤ c ′ L ∆ . In view Protocol 3, for any algorithm A ∈ A zr , there exists a function f ∈ F ∆ ,L and oracles and distributions (( O 1 , . . . , O n ) , ( D 1 , . . . , D n )) ∈ O σ 2 τ 1 ,...,τ n ( f ) such that E [ inf k ∈ S t ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] > ε, where S t := { k ∈ N 0 ∣ ∣ t k ≤ t } , and

The quantities c ′ and c are universal constants.

Before we prove the theorem, let us briefly explain the idea. In Steps 1 and 2 of the proof, we construct the appropriate scaled function and stochastic oracles using the function (19). These steps are almost the same as in (Carmon et al., 2020; Arjevani et al., 2022).

In Step 3, we use the zero-chain property of the function (19) and the zero-respecting property of algorithms that would guarantee us that unless oracles send us a non-zero coordinate, an algorithm would not be able to progress to a new coordinate. The oracles send a non-zero coordinate with some probability p . We have n parallel oracles that flip random coins in parallel . With a large probability, we show that will not get a new coordinate earlier than

seconds, where τ i are the delays of the oracles. So, with a large probability, we will not be able to solve the optimization earlier than

where T is the dimension of the problem.

Proof. ( Step 1 : f ∈ F ∆ ,L )

Let us fix λ > 0 and take a function f ( x ) := Lλ 2 /l 1 F T ( x λ ) , where the function F T is defined in Section D.1. Note that the function f is L -smooth:

Let us take then

We showed that the function f ∈ F ∆ ,L .

( Step 2 : Oracle Class)

In the oracles O i , we have the freedom to choose a mapping ̂ ∇ f ( · ; · ) (see (7)). Let us take

and D i = Bernouilli ( p ) for all i ∈ [ n ] , where p ∈ (0 , 1] . We denote [ x ] j as the j th index of a vector x ∈ R T . It is left to show this mapping is unbiased and σ 2 -variance-bounded. Indeed, for all i ∈ [ T ] , and

because the difference is non-zero only in one coordinate. Thus

where we take

( Step 3 : Analysis of Protocol)

We choose

to ensure that ∥∇ f ( x ) ∥ 2 = L 2 λ 2 l 2 1 ∥ ∥ ∇ F T ( x λ ) ∥ ∥ 2 > 2 ε ✶ [ prog ( x ) < T ] for all x ∈ R T , where we use Lemma D.1. Thus and

Protocol 3 generates a sequence { x k } ∞ k =0 . We have

Using Lemma D.2 with δ = 1 / 2 and (20), we obtain for

D.3 Auxillary lemmas

D.3.1 Proof of Lemma D.2

Lemma D.2. Let us fix T, T ′ ∈ N such that T ≤ T ′ , consider Protocol 3 with a differentiable function f : R T ′ → R such that prog ( ∇ f ( x )) ≤ prog ( x ) + 1 for all x ∈ domain ( f ) , delays 0 < τ 1 ≤ · · · ≤ τ n , distributions D i = Bernouilli ( p ) and oracles O i = O ̂ ∇ f τ i for all i ∈ [ n ] , mappings

and an algorithm A ∈ A zr . With probability not less than 1 -δ, for

where the iterates x k are defined in Protocol 3.

Proof. (Part 1): Comment: in this part, we formally show that if inf k ∈ S t ✶ [ prog ( x k ) < T ] < 1 holds, then we have the inequality ∑ T i =1 ̂ t η i ≤ t , where ̂ t η i are random variables with some known 'good' distributions. If inf k ∈ S t ✶ [ prog ( x k ) < T ] < 1 , then it means that exists k such that prog ( x k ) = T. Note that the algorithm is zero-respecting, so it can not progress to T th coordinate unless the oracles generate stochastic gradients with non-zero 1 st , 2 nd , ..., T th coordinates. The oracles flip coins in parallel, so the algorithm should wait for the moment when the oracles flip a success. At the same time, it takes time to generate a coin (calculate a stochastic gradient), and the oracles can not flip more than k coins before some time ̂ t k . So if the η i is an index of the first success to generate a non-zero i th coordinate, then the algorithm should wait at least ̂ t η i seconds. Next, we give a formal proof.

Let us fix t ≥ 0 and define the smallest index k ( i ) of the sequence when the progress prog ( x k ( i ) ) equals i :

If inf k ∈ S t ✶ [ prog ( x k ) < T ] < 1 holds, then exists k ∈ S t such that prog ( x k ) = T, thus, by the definition of k ( T ) , t k ( T ) ≤ t k ≤ t, and k ( T ) < ∞ . Note that t k ( T ) is the smallest time when we make progress to the T th coordinate.

Since x 0 = 0 and A is a zero-respecting algorithm, the algorithm can return a vector x k with the non-zero first coordinate only if some of returned by the oracles stochastic gradients have the first coordinate not equal to zero. The oracles O i are constructed in such a way (see (21) and (7)) that they zero out a coordinate based on i.i.d. Bernoulli trials.

Definition D.3 (Sequence k ξ j ) . Let us consider a set

We order this set and define the result sequence as { k ξ j } j m =1 , where m ∈ [0 , ∞ ] is the size of the sequence. The sequence k ξ i is a subsequence of iterations where the oracles use the generated Bernouilli random variables in the third output of (7). The sequence s k -1 i k is defined in Protocol 3.

Let k success be the first iteration index when the oracles use a draw ξ = 1 , i.e.,

Since the algorithm A is a zero-respecting and the function f is a zero-chain function, i.e., prog ( ∇ f ( x )) ≤ prog ( x ) + 1 for all x ∈ domain ( f ) , then prog ( g k ) = prog ( x k ) = 0 for all k < k success . If inf k ∈ S t [ prog ( x k ) < T ] < 1 holds, then k success < ∞ , and t k success ≤ t k (1) .

✶ The oracles use the generated Bernoulli random variables { ξ k | k ∈ { k ξ j } j m =1 } . Let us denote the index of the first successful trial as η 1 , i.e.,

The i th worker can generate the first Bernoulli random variable not earlier than after τ i seconds, the second Bernoulli random variable not earlier than after 2 τ i seconds, and so forth.

Definition D.4 (Sequence ̂ t k ) . Let us consider a multi-set of times

We order this multi-set and define the result sequence as { ̂ t k } ∞ k =1 , and ̂ t ∞ := lim k →∞ ̂ t k = ∞ .

Then η 1 th Bernoulli random variable can not be generated earlier than ̂ t η 1 because ̂ t η 1 is the earliest time when the oracles can generate η 1 random variables. Therefore, if inf k ∈ S t ✶ [ prog ( x k ) < T ] < 1 holds, then ̂ t η 1 ≤ t k success ≤ t k (1) .

Using the same reasoning, t k ( j +1) ≥ t k ( j ) + ̂ t η j +1 , where η j +1 is the index of the first successful trial of Bernouilli random variables when prog ( · ) = j in the sequence x k . More formally:

Definition D.5 (Sequence k ξ j,i ) . Let us consider a set

We order this set and define the result sequence as { k ξ j,i } m j +1 i =1 , where m j +1 ∈ [0 , ∞ ] is the size of the sequence. The sequence k ξ j,i is a subsequence of iterations where the oracles use the generated Bernouilli random variables in (7) when prog ( s x ) = j.

Then

By the definition of k ( j ) , x k ( j ) is the first vector of the sequence, that contains a non-zero j th coordinate. Thus the oracles will start returning stochastic gradients that potentially have a non-zero j +1 th coordinate starting only from the iteration k ( j ) . Therefore,

Combining the observations, if inf k ∈ S t ✶ [ prog ( x k ) < T ] < 1 holds, then ∑ T i =1 ̂ t η i ≤ t k ( T ) ≤ t. Thus

In Section D.3.2, we prove the following inequality that we use in Part 2 of the proof.

Lemma D.6. Let us take l j +1 ∈ N . Then

for all j ∈ { 0 , . . . , T -1 } .

(Part 2): Comment: in this part, we use the standard technique to bound the large deviations of the sum ∑ T i =1 ̂ t η i .

Let us fix t ′ ≥ 0 . Recall Definition D.4 of { ̂ t k } ∞ k =1 . If the number of workers n = 1 , then ̂ t k = kτ 1 for all k ≥ 1 . For n > 1 , the sequence { ̂ t k } ∞ k =1 has more complicated structure and depends on the delays τ 1 , . . . , τ n .

For any k ≥ 1 , if ̂ t k ≤ t ′ , then k ≤ ∑ n i =1 ⌊ t ′ τ i ⌋ . Indeed, let us assume that k > ∑ n i =1 ⌊ t ′ τ i ⌋ . The sequence ̂ t k is constructed by the ordering the multi-set { jτ i | j ≥ 1 , i ∈ [ n ] } . The number of elements, which are less or equal to t ′ , equals ∑ n i =1 ⌊ t ′ τ i ⌋ . Thus, we get a contradiction.

It means that

Using Lemma D.6, we have

If 0 ≤ t ′ < τ 1 , then ∑ n i =1 ⌊ t ′ / τ i ⌋ = 0 , and

Otherwise, if t ′ > τ 1 , then ∑ n i =1 ⌊ t ′ / τ i ⌋ ≥ 1 , and

where we use the fact that 1 -(1 -p ) m ≤ pm for all p ∈ [0 , 1] and m ∈ N . For all t ′ ≥ 0 , we have

Let us define then

Let us fix s ≥ 0 and ̂ t ≥ 0 . Using the Chernoff method, we have

Let us bound the expected value separately:

Since ̂ t η i ≥ 0 , we have

Thus and

Let us take s = 1 / t ′ , and get

Let us recall the definition of p ′ :

Now, we have to take the right t ′ . We will take it using a nonconstructive definition. Assume that t ′ = 1 4 p ( ∑ j ∗ i =1 1 τ i ) -1 , where

This set is not empty because n belongs to it. Using the definition of j ∗ , we have

Substituting this inequality to (24), we obtain

Recall that t ′ = 1 4 p ( ∑ j ∗ i =1 1 τ i ) -1 . Using Lemma D.7, we have

Finally, we obtain for

D.3.2 Proof of Lemma D.6

In the following lemma, we use notations from Part 1 of the proof of Lemma D.2.

Lemma D.6. Let us take l j +1 ∈ N . Then

In this lemma, we want to bound the probability for the random variable η j +1 from (22). η j +1 is the index of the first successful trial of the sequence of Bernouilli random variables. At first sight, this is a trivial task since η j +1 has a distribution similar to the geometric distribution. But the main problem here is that the sequence k ξ j,i and the quantity m j +1 are also random variables. Therefore, we must be careful with this.

For ̂ t ≤ t ′ ( T 2 +log δ ) , we have

Proof. Since the image of the random variables η j , . . . , η 1 is in N ∩ {∞} . Let us take l 1 , . . . , l j ∈ N ∪ {∞} , and prove the theorem for a probability conditioned on an event ⋂ j i =1 { η i = l i } such that P ( ⋂ j i =1 { η i = l i } ) > 0 . Therefore, it is enough to prove that

First, assume that exists i ∈ [ j ] such that l i = ∞ . It means that, for all k ≥ 0 , prog ( x k ) < j, thus

for all l j +1 ∈ N . Let us explain this step. if exists i ∈ [ j ] such that l i = ∞ , then an algorithm never get a progress to j th coordinate, thus η j +1 = inf { i | ξ k ξ j,i = 1 and i ∈ [1 , m j +1 ] } = ∞ a.s. because m j +1 = 0 and k ξ j,i is an empty sequence.

Assume that l i < ∞ for all i ∈ [ j ] . By the definition of η j +1 , we have m j +1 ≥ l j +1 and ξ k ξ j, 1 = · · · = ξ k ξ j,l j +1 -1 = 0 and ξ k ξ j,l j +1 = 1 . Thus

Since k ξ j,s < k ξ j,i a.s. for all s < i ∈ [ m j +1 ] , using the law of total probability, we have

where ∑ ∞ k 1 < ··· <k l j +1 =1 is a sum over a set { ( k 1 , . . . , k l j +1 ) ∈ N l j +1 | ∀ i < p ∈ [ l j +1 ] : k i < k p } . Next, if the event

holds, then an event ⋂ l j +1 i =1 A k i holds, where

At the same time, if ⋂ l j +1 i =1 A k i holds, then { m j +1 ≥ l j +1 } holds. Therefore,

and

Let us define σ ( ξ 1 , . . . , ξ k l j +1 -1 ) as a sigma-algebra generated by ξ 1 , . . . , ξ k l j +1 -1 . Note that, for all i ∈ [ l j +1 -1] , the event { ξ k i = 0 } ∈ σ ( ξ 1 , . . . , ξ k l j +1 -1 ) . Also, for all i ∈ [ l j +1 ] , the event { k ξ j,i = k i } ⋂ A k i ∈ σ ( ξ 1 , . . . , ξ k l j +1 -1 ) . Finally, since k ξ i -1 ,l i < k ξ j,l j +1 , the event

for all i ∈ [ j ] . Therefore, the event { ξ k l j +1 = 1 } is independent of the event

because ξ k are i.i.d. random variables. Using the independence and the equality P ( ξ k l j +1 = 1 ) = p, we have

Since the events { k ξ j,l j +1 = k l j +1 } ⋂ A k l j +1 do not intersect, we can use the additivity of the probability. If l j +1 = 1 , we get

and prove the lemma for l j +1 = 1 . Otherwise, if l j +1 > 1 , we obtain

For any events A and B, we have P ( A,B ) ≤ P ( A ) , thus

Let us continue for l j +1 > 1 and rewrite the last inequality:

Note that, for all i ∈ [ l j +1 -2] , the event { ξ k i = 0 } ∈ σ ( ξ 1 , . . . , ξ k l j +1 -2 ) . Also, for all i ∈ [ l j +1 -1] , the event { k ξ j,i = k i } ⋂ A k i ∈ σ ( ξ 1 , . . . , ξ k l j +1 -2 ) . Finally, since k ξ i -1 ,l i < k ξ j,l j +1 -1 , the event

for all i ∈ [ j ] . Therefore, the event { ξ k l j +1 -1 = 0 } is independent of the event

Thus, we have

Since the events { k ξ j,l j +1 -1 = k l j +1 -1 } ⋂ A k l j +1 -1 do not intersect, we use the additivity of the probability. If l j +1 = 2 , we get

and prove the lemma for l j +1 = 2 . Otherwise, if l j +1 > 2 , we obtain

where we use P ( A,B ) ≤ P ( A ) for any events A and B . Using mathematical induction, we can continue and get that

D.3.3 Lemma D.7

This is a technical lemma that we use in the proof of Lemma D.2.

Lemma D.7. Let us consider a sorted sequence 0 < τ 1 ≤ · · · ≤ τ n ≤ τ n +1 = ∞ and a constant S ≥ 1 4 . We define

where and

Then

Proof. Additionally, let us define

where j ∗ 2 is the smallest index. For j ∗ 2 = 1 , we have

For j ∗ 2 > 1 , we have

From this inequality, we get

Thus τ j ∗ 2 < t 2 for all j ∗ 2 ≥ 1 .

Then either j ∗ 2 ≤ j ∗ 1 and

or j ∗ 2 > j ∗ 1 and and

where we used the definition of j ∗ 1 . It concludes that t 2 ≥ t 1 .

Assume that j ∗ 1 > S +1 . Since the harmonic mean of a sequence less or equal to the maximum, we have

This inequality contradicts the definition of j ∗ 1 . It means that j ∗ 1 ≤ S +1 and

D.4 Proof of Theorems 7.4 and 7.5

D.4.1 Proof of Theorems 7.4

Theorem 7.4. Assume that Assumptions 7.1, 7.2 and 7.3 hold. Let us take the batch size S = max {⌈ σ 2 / ε ⌉ , 1 } , and γ = min { 1 L , εS 2 Lσ 2 } = Θ( 1 / L ) in Method 4. Then after

iterations, the method guarantees that 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

Proof. Note that Method 4 is just the stochastic gradient method with the batch size S. Method 4 can be rewritten as x k +1 = x k -γ 1 S ∑ S i =1 ̂ ∇ f ( x k ; ξ i ) , where the ξ i are independent random samples. It means that we can use the classical SGD result (see Theorem D.8). For a stepsize

we have if

Using the choice of S, we showed that Method 4 converges after

D.4.2 The classical SGD theorem

We reprove the classical SGD result (Ghadimi and Lan, 2013; Khaled and Richt´ arik, 2020).

Theorem D.8. Assume that Assumptions 7.1 and 7.2 hold. We consider the SGD method:

steps with where

For a fixed x ∈ R d , g ( x ) is a random vector such that E [ g ( x )] = ∇ f ( x ) ,

and g ( x k ) are independent vectors for all k ≥ 0 . Then

Proof. From Assumption 7.1, we have

We denote G k as a sigma-algebra generated by g ( x 0 ) , . . . , g ( x k -1 ) . Using unbiasedness and (26), we obtain

Since γ ≤ 1 / L , we get

We subtract f ∗ and take the full expectation to obtain

Next, we sum the inequality for k ∈ { 0 , . . . , K -1 } :

Finally, we rearrange the terms and use that E [ f ( x K ) -f ∗ ] ≥ 0 :

The choice of γ and K ensures that

for

D.4.3 Proof of Theorems 7.5

Theorem 7.5. Consider Theorem 7.4. We assume that i th worker returns a stochastic gradient every τ i seconds for all i ∈ [ n ] . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n . Then after

seconds, Method 4 guarantees to find an ε -stationary point.

Proof. In this setup, the method converges after K ×{ time required to collect a batch of the size S } . Without loss of generality, we assume that τ 1 ≤ · · · ≤ τ n .

Let us define time that is enough to collect a batch of the size S as t ′ . Obviously, one can always take t ′ = 3 τ n ⌈ S n ⌉ and guarantees that every worker calculates at least ⌈ S n ⌉ stochastic gradients, but we will provide a tighter t ′ .

We define B i as the number of received gradients with an iteration index equals to k. 7 For each worker, there are two options: either the i th worker does not send a gradient with an iteration index k and B i = 0 , or it sends at least once and B i > 0 .

In the worst case, for i th worker, the time required to calculate B i gradients equals

because either B i > 0 and, in the worst case, a worker finishes the calculation of a gradient from the previous iteration (that we ignore) and only then starts the calculation of a gradient of the current iteration k, or B i = 0 and the server does not receive any gradients from a worker.

Note that all workers work in parallel, so our goal is to find feasible points t ′ ∈ R and B 1 , · · · , B n ∈ N 0 such that

First, we relax an assumption that B i ∈ N 0 and assume that B i ∈ R for all i ∈ [ n ] :

If B i ∈ R are feasible points of (28), then

It means that if t ′ ∈ R and B 1 , · · · , B n are feasible points of (28), then 2 t ′ and ⌈ B 1 ⌉ , · · · , ⌈ B n ⌉ are feasible points of (27).

Let us define

and take j ∗ = arg min j ∈ [ n ] t ′ ( j ) , j ∗ is the smallest index from all minimizers of t ′ ( j ) . Let us show that t ′ ( j ∗ ) and

are feasible points of (28). First, we have

Next, we show that B i > 0 for all i ≤ j ∗ . If j ∗ = 1 , then t ′ (1) = τ 1 ( S +1) , thus B 1 = S > 0 . If j ∗ > 1 , then, by its definition, we have t ′ ( j ∗ ) < t ′ ( j ∗ -1) , thus

From this inequality, we get

From the last inequality, we get that τ j ∗ < t ′ ( j ∗ ) , thus B i ≥ B j ∗ > 0 for all i ≤ j ∗ . It is left to show that

Finally, we can conclude that Method 4 returns a solution after

and seconds.

E Proofs for Heterogeneous Regime

E.1 Proof of Theorem A.2

Theorem A.2. Let us consider the oracle class O σ 2 , heterog τ 1 ,...,τ n for some σ 2 > 0 and 0 < τ 1 ≤ · · · ≤ τ n . We fix any L, ∆ > 0 and 0 < ε ≤ c ′ L ∆ . In the view Protocol 3, for any algorithm A ∈ A zr , there exists a function f = 1 n ∑ n i =1 f i ∈ F ∆ ,L and oracles and distributions (( O 1 , . . . , O n ) , ( D 1 , . . . , D n )) ∈ O σ 2 , heterog τ 1 ,...,τ n ( f 1 , . . . , f n ) such that E [ inf k ∈ S t ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] > ε, where S t := { k ∈ N 0 ∣ ∣ t k ≤ t } , and

The quantity c ′ and c are universal constants.

The structure of the following proof is similar to the proof of Theorem 6.4. In the heterogeneous regime, the main difference is that we have more freedom to choose the functions f i .

Proof. In (14), we have the sum of two terms. We split the proof in two parts for each of the terms. ( Part 1 )

( Step 1 : f ∈ F ∆ ,L )

Let us fix λ > 0 . We consider the following functions f i :

Let us show that the function f is L -smooth:

Let us take then

We showed that the function f ∈ F ∆ ,L .

( Step 2 : Oracle Class)

In the oracles O i , we have the freedom to choose a mapping ̂ ∇ f i ( · ; · ) (see (7)). In this part of the proof, we simply take non-stochastic mappings ̂ ∇ f i ( x ; ξ ) := ∇ f i ( x ) that are, obviously, unbiased and σ 2 -variance-bounded. We can take an arbitrary distribution, for instance, let us take D i = Bernouilli(1) for all i ∈ [ n ] .

( Step 3 : Analysis of Protocol)

We take to ensure that

for all x ∈ R T such that prog ( x ) < T. Thus

Only the n th worker contains a nonzero function and can provide a gradient every τ n seconds. Since A is a zero-respecting algorithm and the function f n is a zero-chain function, for all k ≥ 0 such that

we have because we need at least T oracle calls to obtain prog ( x k ) ≥ T. It means that

for

We now prove the second part of the lower bound.

( Part 2 )

( Step 1 : f ∈ F ∆ ,L )

Let us fix λ > 0 . We assume that x = [ x 1 , . . . , x n ] ∈ R nT . We define x i ∈ R T as the i th block of a vector x = [ x 1 , . . . , x n ] ∈ R nT . We consider the following functions f i :

The function f i depends only on a subset of variables x i from x. Let us show that the function f is L -smooth. Indeed, we have

and

Let us take then

We showed that the function f ∈ F ∆ ,L .

( Step 2 : Oracle Class)

In the oracles O i , we have the freedom to choose a mapping ̂ ∇ f i ( · ; · ) (see (7)). Let us take

D i = Bernouilli ( p i ) , and p i ∈ (0 , 1] for all i ∈ [ n ] . Let us show it is unbiased and σ 2 -variancebounded:

for all j ∈ nT, and

because the difference is non-zero only in one coordinate. Thus

where we take

( Step 3 : Analysis of Protocol)

We fix η > 0 and choose to ensure that

Protocol 3 generates the sequence { x k } ∞ k =0 ≡ { [ x k 1 , . . . , x k n ] } ∞ k =0 . From (29), we have

Further, we require the following auxillary lemma. See the proof in Section E.2.

Lemma E.1. For η = 4 , with probability not less than 1 / 2 , for

Using Lemma E.1 and (31), we have for

This finishes the proof of Part 2.

E.2 Proof of Lemma E.1

In the following lemma, we use notations from the proof of Theorem A.2.

Lemma E.1. For η = 4 , with probability not less than 1 / 2 , for

for all x = [ x 1 , . . . , x n ] ∈ R T . Thus and

Proof. Let us fix t ≥ 0 . Our goal is to show that the probability of an inequality is small.

We now use the same reasoning as in Lemma D.2. We use a notation { x } i is i th block of the vector x . Let us fix a worker's index i ∈ [ n ] .

Definition E.2 (Sequence k ξ i,j,l ) . Let us consider a set

We order this set and define the result sequence as { k ξ i,j,l } m i,j +1 i =1 , where m i,j +1 ∈ [0 , ∞ ] is the size of the sequence. The sequence k ξ i,j,l is a subsequence of iterations where the i th oracle use the generated Bernouilli random variables in (7) when prog ( { s x } i ) = j. The sequence s k -1 i k is defined in Protocol 3.

Then

The quantity η i,j +1 is the index of the first successful trial, when prog ( · ) = j in the i th block of the sequence x k . Since the algorithm A is a zero-respecting algorithm, for all k < k ξ i,j,η i,j +1 , the progress prog ( x k i ) < j +1 .

As in Lemma D.2 (we skip the proof since the idea is the same. It is only required to use the different notations: x k → x k i , ̂ t η j → ̂ t i,η i,j ), for all i ∈ [ n ] , one can show that if inf k ∈ S t ✶ [ prog ( x k i ) < T ] < 1 holds, then ∑ T j =1 ̂ t i,η i,j ≤ t, where ̂ t i,k := kτ i for all k ≥ 1 . The time ̂ t i,k is the smallest possible time when the i th oracle can return the k th stochastic gradient. Thus

Using (25) with n = 1 (and the different notations p → p i , τ 1 → τ i , and ̂ t η j → ̂ t i,η i,j ), we have

We now rearrange the terms and use Markov's inequality to obtain

for η > 2 . Using the choice of p i in (30), we have

for all i ∈ [ n ] . The last term does not depend on i. Therefore, we can use (33) with

to get

Finally, for η = 4 and δ = 1 / 4 , we have

E.3 Proof of Theorem A.3

Theorem A.3. Assume that Assumptions 7.1 and 7.2 hold for the function f . Assumption 7.3 holds for the function f i for all i ∈ [ n ] . Let us take the parameter S = max {⌈ σ 2 / ε ⌉ , n } , and γ = min { 1 L , εS 2 Lσ 2 } = Θ( 1 / L ) in Method 6, then after K ≥ 24∆ L / ε iterations the method guarantees that 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

Proof. Note that Method 6 can be rewritten as x k +1 = x k -γ 1 n ∑ n i =1 1 B i ∑ B i j =1 ̂ ∇ f i ( x k ; ξ i,j ) , where the ξ i,j are independent random samples. The variance of the gradient estimator equals

where we use the inequality ( 1 n ∑ n i =1 1 B i ) -1 ≥ S n , We can use the classical SGD result (see Theorem D.8). For a stepsize

we have if

Using the choice of S, we obtain that Method 6 converges after

steps.

E.4 Proof of Theorem A.4

Theorem A.4. Let us consider Theorem A.3. We assume that i th worker returns a stochastic gradient every τ i seconds for all i ∈ [ n ] . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n . Then after

seconds, Method 6 guarantees to find an ε -stationary point.

Proof. The method converges after K ×{ time required to collect batches with the sizes B i such that ( 1 n ∑ n i =1 1 / B i ) -1 ≥ S n holds } .

In the worst case, for i th worker, the time required to calculate B i gradients equals

because it is possible that a worker finishes the calculation of a gradient from the previous iteration (that we ignore) and only then starts the calculation of a gradient of the current iteration.

Our goal is to find feasible points t ′ ∈ R and B 1 , · · · , B n ∈ N such that

Using the same reasoning as in Theorem 7.5, we relax the assumption that B i ∈ N and assume that B i ∈ R for all i ∈ [ n ] :

If t ′ ∈ R and B 1 , · · · , B n are feasible points of (35), then 2 t ′ and ⌈ B 1 ⌉ , · · · , ⌈ B n ⌉ are feasible points of (34). Let us show that t ′ = 2 ( τ n + ( 1 n ∑ n i =1 τ i ) S n ) and B i = t ′ τ i -1 are feasible points. Indeed, for all i ∈ [ n ] ,

Next, we have and

where we use t ′ -τ i ≥ t ′ / 2 + τ i -τ i = t ′ / 2 for all i ∈ [ n ] . Finally, it means that Method 6 returns a solution after

seconds.

F Interrupt Oracle Calculations

Let us define a protocol and an oracle where an algorithm can stop the oracle anytime. If an algorithm stops the oracle, its current calculations are canceled and discarded.

Protocol 8 Time Multiple Oracles Protocol With Control

1: Input: functions f ∈ F , oracles and distributions (( O 1 , . . . , O n ) , ( D 1 , . . . , D n )) ∈ O ( f ) , algorithm A ∈ A

2: s 0 i = 0 for all i ∈ [ n ]

3: for k = 0 , . . . , ∞ do

6: end for

In Protocol 8, we allow algorithms to output the control variables c k that can be used in the following oracle.

We take an oracle

such that

and ̂ ∇ f is a mapping such that

The oracle (36) generalizes the oracle (7) since an algorithm can send a signal c to the oracle (36) and interrupt the calculations. Note that if c = 1 , then (36) has the same behavior as (7). But, if c = 0 , then the oracle (36) discards all previous information in the state, and changes s q to 0 .

Let us define an oracle class:

Definition F.1 (Oracle Class O σ 2 , stop τ 1 ,...,τ n ) .

Let us consider an oracle class such that, for any f ∈ F ∆ ,L , it returns oracles O i = O ̂ ∇ f τ i and distributions D i for all i ∈ [ n ] , where ̂ ∇ f is an unbiased σ 2 -variance-bounded mapping (see Assumption 7.3). The oracles O ̂ ∇ f τ i are defined in (36). We define such oracle class as O σ 2 , stop τ 1 ,...,τ n . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n .

For the oracle class O σ 2 , stop τ 1 ,...,τ n , we state that

The lower bound is the same as for the oracle class from Definition 6.3. We do not provide a formal proof, but a close investigation can reveal that the proof is the same as in Theorem 6.4.

Indeed, in Part 1 of the proof of Lemma D.2, we reduce the the inequality inf k ∈ S t ✶ [ prog ( x k ) < T ] < 1 to the inequality ∑ T i =1 ̂ t η i ≤ t , where ̂ t η i is the shortest time when the oracles can draw a successful Bernouilli random variable. The fact that an algorithm can interrupt the oracles can not change the quantity ̂ t η i .

̸

G Time Complexity with Synchronized Start

In this section, we continue and fill up the discussion in Section 8.

Let us design an oracle for the synchronized start setting. We take an oracle

such that

and ̂ ∇ f is a mapping such that

We assume that ξ k +1 is a tuple in Protocol 2: ξ k +1 ≡ ( ξ k +1 1 , . . . , ξ k +1 n ) ∼ D .

The oracle (37) with Protocol 2 emulates the behavior of a setting where we broadcast an iterate x to all workers, and they start calculations simultaneously. The workers have different time delays, hence some finish earlier than others. An algorithm can stop the procedure earlier and get calculated stochastic gradients, but other non-calculated ones will be discarded.

For this oracle, we define an oracle class:

Let us consider an oracle class such that, for any f ∈ F ∆ ,L , it return an oracle O = O ̂ ∇ f τ 1 ,...,τ n and a distribution D , where ̂ ∇ f is an unbiased σ 2 -variance-bounded mapping (see Assumption 7.3). The oracle O ̂ ∇ f τ 1 ,...,τ n is defined in (37). We define such oracle class as O σ 2 , sync τ 1 ,...,τ n . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n .

We now provide the lower bound for Protocol 2 and the oracle class O σ 2 , sync τ 1 ,...,τ n .

Theorem G.2. Let us consider the oracle class O σ 2 , sync τ 1 ,...,τ n for some σ 2 > 0 and 0 < τ 1 ≤ · · · ≤ τ n . We fix any L, ∆ > 0 and 0 < ε ≤ c ′ L ∆ . In the view Protocol 2, for any algorithm A ∈ A zr , there exists a function f ∈ F ∆ ,L and an oracle and a distribution ( O, D ) ∈ O σ 2 , sync τ 1 ,...,τ n ( f ) such that E [ inf k ∈ S t ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] > ε, where S t := { k ∈ N 0 ∣ ∣ t k ≤ t } , and

The quantity c ′ and c are universal constants.

G.1 Minimax optimal method

In this section, we analyze the m -Minibatch SGD method (see Method 9). This method generalizes the Minibatch SGD method from Section 1.2. Unlike Minibatch SGD, the m -Minibatch SGD method only asks for stochastic gradients from the first m ∈ [ n ] (fastest) workers. Later, we show that optimal

Method 9 m -Minibatch SGD

1: Input: starting point x 0 , stepsize γ , number of workers m ∈ [ n ]

2: for k = 0 , 1 , . . . , K -1 do

3: Send the point x k to the first m workers

4: Receive i.i.d. stochastic gradients ̂ ∇ f ( x k , ξ 1 ) , . . . , ̂ ∇ f ( x k , ξ m ) from the workers

6: end for

5: x k +1 = x k -γ 1 m ∑ m i =1 ̂ ∇ f ( x k , ξ i )

m is determined by (38). And with this parameter, m -Minibatch SGD method is minimax optimal under the setting from Sections 8 and G. Note that m -Minibatch SGD is Minibatch SGD if m = n.

We now provide the convergence rate and the time complexity.

Theorem G.3. Assume that Assumptions 7.1, 7.2 and 7.3 hold. Let us take the step size

iterations the method guarantees that 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

Theorem G.4. Let us consider Theorem G.3. We assume that i th worker returns a stochastic gradient every τ i seconds for all i ∈ [ n ] . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n . Let us take

seconds Method 9 guarantees to find an ε -stationary point.

Despite the triviality of the m -Minibatch SGD and it analysis, we provide it to show that the lower bound in Theorem G.2 is tight.

G.2 Proof of Theorem G.2

Theorem G.2. Let us consider the oracle class O σ 2 , sync τ 1 ,...,τ n for some σ 2 > 0 and 0 < τ 1 ≤ · · · ≤ τ n . We fix any L, ∆ > 0 and 0 < ε ≤ c ′ L ∆ . In the view Protocol 2, for any algorithm A ∈ A zr , there exists a function f ∈ F ∆ ,L and an oracle and a distribution ( O, D ) ∈ O σ 2 , sync τ 1 ,...,τ n ( f ) such that E [ inf k ∈ S t ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] > ε, where S t := { k ∈ N 0 ∣ ∣ t k ≤ t } , and

The quantity c ′ and c are universal constants.

Step 1 and Step 2 mirrors the corresponding steps from the proof of Theorem G.2.

Let us fix λ > 0 . We take the same function f ∈ F ∆ ,L as in the proof of Theorem 6.4. We define

in Method 9, then after

Then after

with

( Step 2 : Oracle Class)

Following the proof of Theorem 6.4, in the oracle O, we take the following stochastic estimator

and D = ( Bernouilli ( p ) , . . . , Bernouilli ( p )) ︸ ︷︷ ︸ n times , where p ∈ (0 , 1] . The stochastic gradient is unbiased and σ 2 -variance-bounded if

( Step 3 : Analysis of Protocol)

We choose to ensure that ∥∇ f ( x ) ∥ 2 = L 2 λ 2 l 2 1 ∥ ∥ ∇ F T ( x λ ) ∥ ∥ 2 > 2 ε ✶ [ prog ( x ) < T ] for all x ∈ R T , where we use Lemma D.1. Thus

and

Protocol 2 generates a sequence { x k } ∞ k =0 . We have

Further, we require the following auxillary lemma. See the proof in Section D.3.

Lemma G.5. With probability not less than 1 -δ, for

Using Lemma G.5 with δ = 1 / 2 and (41), we obtain

for

G.2.1 Proof of Lemma G.5

Lemma G.5. With probability not less than 1 -δ, for

Proof. (Part 1): Comment: in this part, we mirror the proof of Lemma D.2. We also show that if inf k ∈ S t ✶ [ prog ( x k ) < T ] < 1 holds, then we have the inequality ∑ T i =1 ̂ t η i ≤ t , where ̂ t η i are random variables with some known 'good' distributions. However, in this lemma the quantities ̂ t η i are different.

In Protocol 2, the algorithm A consequently calls the oracle O. If inf k ∈ S t ✶ [ prog ( x k ) < T ] < 1 holds, then exists k ∈ S t such that prog ( x k ) = T. Since the algorithm A is zero-respecting, the mappings A k will not output a non-zero vector (a vector with a non-zero first coordinate) unless the oracle returns a non-zero vector.

Let us define the smallest index k ( i ) of the sequence when the progress prog ( x k ( i ) ) equals i :

Definition G.6 (Sequence ( k ξ p , i ξ p ) ) . Let us consider a set

We order this set lexicographically and define the result sequence as { ( k ξ p , i ξ p ) } m p =1 , where m ∈ [0 , ∞ ] is the size of the sequence. The sequence s k -1 is defined in Protocol 2.

Note that the algorithm A is only depends on the random samples { ξ k ξ p i ξ p } m p =1 since the sequence

{ ( k ξ p , i ξ p ) } m p =1 are the indices of the random samples that are used in the oracle (37).

Let us denote the index of the first successful trial as η 1 , i.e.,

If inf k ∈ S t ✶ [ prog ( x k ) < T ] < 1 holds, then η 1 < ∞ . Using the times t k , the algorithm A consequently requests the gradient estimators from the oracle (37).

In each round of the oracle's calculation, the algorithm can get the vector ̂ ∇ f ( s x ; ξ 1 ) in not less than τ 1 seconds, the vector ∑ 2 i =1 ̂ ∇ f ( s x ; ξ i ) in less than τ 2 seconds, and so forth (see (37)). The algorithm can repeat any number of these rounds sequentially.

The algorithm A one by one requests m 1 , . . . , m i , · · · ∈ { 0 , . . . , n } gradient estimators from the oracle (37). It takes at least τ m i seconds ( τ 0 ≡ 0 ) to get a vector ∑ m i i =1 ̂ ∇ f ( s x ; ξ i ) . Let us consider that k ∈ N is the first index of gradient estimators such that is depends on some ξ i = 1 . Necessarily, we have ∑ k j =1 m j ≥ η 1 . Also, since the A is zero-respecting, we have ∑ k j =1 τ m j ≤ t k (1) . Note that k, and m 1 , . . . , m k . depend on the algorithm's strategy. Let us find 'the best possible' quantities k, and m 1 , . . . , m k that are independent of an algorithm.

Let us assume that k ∗ , and m ∗ 1 , . . . m ∗ k minimize the quantity

Then, we have

Note that if exists j ∈ [ k ∗ ] such that m ∗ j > η 1 , then k ∗ , and m ∗ 1 , . . . , m ∗ j -1 , η 1 , m ∗ j +1 . . . , m ∗ k are also minimizers of (42) since the sequence τ k is not decreasing. Therefore, (42) is equivalent to

Then, using the simple algebra, we have

̸

̸

̸

In the first inequality, we use that m ∗ j ∈ { 0 , . . . , η 1 } for all j ∈ [ k ∗ ] . Next, using Lemma G.9, we get

Using the same reasoning, for j ∈ { 0 , . . . , T -1 } ,

where η j +1 is the index of the first successful trial of Bernouilli random variables when prog ( · ) = j. More formally, for all j ∈ { 0 , . . . , T -1 } :

Definition G.7 (Sequence ( k ξ j,p , i ξ j,p ) ) . Let us consider a set

We order this set lexicographically and define the result sequence as { ( k ξ j,p , i ξ j,p ) } m j +1 p =1 , where m j +1 ∈ [0 , ∞ ] is the size of the sequence. The sequence s k -1 is defined in Protocol 2.

Then,

By the definition of k ( j ) , x k ( j ) is the first iterate such that prog ( · ) = j. Therefore, the oracle can potentially start returning gradient estimators with the non-zero j +1 th coordinate from the k ( j ) th iteration.

Thus, if inf k ∈ S t ✶ [ prog ( x k ) < T ] < 1 holds, then

Finally, we can conclude that

As in Lemma D.6, we show that

Lemma G.8. Let us take l j +1 ∈ N . Then

for all j ∈ { 0 , . . . , T -1 }

See the proof in Section G.2.3. Intuitively, the algorithm A can not increase the probability of getting a successful Bernouilli random variable earlier with its decisions.

Let us temporally define for all i ∈ [ T ] .

(Part 2): Comment: in this part, we use the standard technique to bound the large deviations of the sum ∑ T i =1 ̂ t η i .

Note that a function g : R → R such that g ( x ) := 1 2 min m ∈ [ n ] τ m ( 1 + x m ) is continuous, stronglymonotone and invertible. For t ′ ≥ 0 , we have

Using (45), we obtain

Let us define then

Using the Chernoff method, as in Lemma D.2, one can get

Let us take then

Therefore,

Using (44), for we have

The last inequality concludes the proof.

G.2.2 Lemma G.9

Lemma G.9. Let us consider a sorted sequence 0 < τ 1 ≤ · · · ≤ τ n and a constant η ∈ N . We define

and

Then

Proof. Additionally, let us define and

Then, using m 1 ≤ η, we have

If m 2 ≤ η, then

Otherwise, if m 2 > η,

G.2.3 Proof of Lemma G.8

In the following lemma, we use notations from Part 1 of the proof of Lemma G.5.

Lemma G.8. Let us take l j +1 ∈ N . Then

The idea of the following proof repeats the proof of Lemma G.8. But Protocol 2 with the oracle (37) differ, so we present the proof for completeness.

Proof. We prove that

for all l 1 , . . . , l j ∈ N ∪ {∞} such that P ( ⋂ j i =1 { η i = l i } ) > 0 .

If exists i ∈ [ j ] such that l i = ∞ , then

for all l j +1 ∈ N (see details in the proof of Lemma D.6).

Assume that l i < ∞ for all i ∈ [ j ] . By the definition of η j +1 , we have m j +1 ≥ l j +1 and ξ k ξ j, 1 i ξ j, 1 =

Let us define a set

S l j +1 := { (( k 1 , i 1 ) , . . . , ( k l j +1 , i l j +1 )) ∈ ( N × [ n ]) l j +1 | ∀ p < j ∈ [ l j +1 ] : ( k p , i p ) < ( k j , i j ) } ∀ l j +1 ≥ 1 . Using the law of total probability, we have

where we take the sum over all (( k 1 , i 1 ) , . . . , ( k l j +1 , i l j +1 )) ∈ S l j +1 . If the event

holds, then an event ⋂ l j +1 p =1 A ( k p ,i p ) holds, where

At the same time, if ⋂ l j +1 p =1 A ( k p ,i p ) holds, then { m j +1 ≥ l j +1 } holds. Therefore,

and

Let us define σ i j as a sigma-algebra generated by ( ξ 1 1 , . . . , ξ 1 n ) , ( ξ 2 1 , . . . , ξ 2 n ) , . . . , ( ξ i 1 , . . . , ξ i j ) for all i ≥ 0 and j ∈ { 0 , . . . , n } . Then, we have

and

since, for all p ∈ [ l j +1 ] , the event A ( k p ,i p ) is only determined by s k p -1 and t k p that do not depend on { ξ j } ∞ j = k l j +1 . And, for all p ∈ [ l j +1 ] , the fact that ( k ξ j,p , i ξ j,p ) = ( k p , i p ) does not depend on { ξ j } ∞ j = k l j +1 . Note that k ξ i -1 ,l i < k ξ j,l j +1 (a.s.) for all i ∈ [ j ] . Thus

since, for all i ∈ [ j ] , this event implies that k ξ i -1 ,l i < k l j +1 . All in all, we have that

Since { ξ k l j +1 i l j +1 = 1 } is independent of σ k l j +1 -1 n ⋃ σ k l j +1 -1 i l j +1 -1 , 8 we get

If l j +1 = 1 , observe that the events { ( k ξ j,l j +1 , i ξ j,l j +1 ) = ( k l j +1 , i l j +1 ) } ⋂ A ( k l j +1 ,i l j +1 ) do not intersect for all ( k l j +1 , i l j +1 ) ∈ N × [ n ] . Thus, we can use the additivity of the probability, and obtain

Otherwise, if l j +1 > 1 , we also use the fact that the events do not intersect and get

where we used an inequality P ( A,B ) ≤ P ( A ) for any events A and B. We take the sum over all (( k 1 , i 1 ) , . . . , ( k l j +1 -1 , i l j +1 -1 )) ∈ S l j +1 -1 , where S l j +1 -1 = { (( k 1 , i 1 ) , . . . , ( k l j +1 -1 , i l j +1 -1 )) ∈ ( N × [ n ]) l j +1 -1 | ∀ p < j ∈ [ l j +1 -1] : ( k p , i p ) < ( k j , i j ) } . Using the same reasoning, one can continue and get that

G.3 Proof of Theorem G.3

Theorem G.3. Assume that Assumptions 7.1, 7.2 and 7.3 hold. Let us take the step size

iterations the method guarantees that 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

Proof. Note that in Method 9, then after

we have if

G.4 Proof of Theorem G.4

Theorem G.4. Let us consider Theorem G.3. We assume that i th worker returns a stochastic gradient every τ i seconds for all i ∈ [ n ] . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n . Let us take

seconds Method 9 guarantees to find an ε -stationary point.

Proof. In this setup, the method converges after K × τ m seconds because the delay of each iteration is determined by the slowest worker. Thus, the time complexity equals

τ m ( 12∆ L ε + 12∆ Lσ 2 ε 2 m ) = 12∆ L ε τ m ( 1 + σ 2 εm ) = 12∆ L ε min m ′ ∈ [ n ] τ m ( 1 + σ 2 m ′ ε ) , where we use the choice of the number of workers m.

Then after

where the stochatsic gradients are i.i.d. Therefore, we can use the classical SGD result (see Theorem D.8). For a stepsize

H Proofs for Convex Case

H.1 The 'worst case' function in convex case

In this proof, we use the construction from (Woodworth et al., 2018). Let us define B 2 (0 , R ) := { x ∈ R T +1 : ∥ x ∥ ≤ R } .

Let us take functions f l,η : R T +1 → R and ˜ f l,η : R T +1 → R such that

where l, η > 0 are free parameters. Let us define

For ths function f l,η , we have the following properties:

Lemma H.1 (Woodworth et al. (2018)) . The function f l,η satisfies:

(Lemma 4) The function f l,η is convex, l -Lipschitz, and η -smooth.

(eq. 75)

and

(Lemma 6) For all x ∈ B 2 (0 , 1) , prog ( ∇ f l,η ( x )) ≤ prog ( x ) + 1 and prog ( y ( x )) ≤ prog ( x ) + 1 .

H.2 Proof of Theorem B.4

Theorem B.4. Let us consider the oracle class O conv ,σ 2 τ 1 ,...,τ n for some σ 2 > 0 and 0 < τ 1 ≤ · · · ≤ τ n . We fix any R,L,M,ε > 0 such that √ LR > c 1 √ ε > 0 and M 2 R 2 > c 2 ε 2 . In the view Protocol 3, for any algorithm A ∈ A R zr , there exists a function f ∈ F conv R,M,L and oracles and distributions (( O 1 , . . . , O n ) , ( D 1 , . . . , D n )) ∈ O conv ,σ 2 τ 1 ,...,τ n ( f ) such that

where S t := { k ∈ N 0 ∣ ∣ t k ≤ t } , and

The quantities c 1 , c 2 and c are universal constants.

In Step 1, we almost repeat the proof from Woodworth et al. (2018). Steps 2 and 3 are very close to Steps 2 and 3 of the proofs for the nonconvex case.

Proof. ( Step 1 : f ∈ F conv R,M,L ) Following Woodworth et al. (2018), we assume that R = 1 . Otherwise, one can rescale the parameters of the construction. Let us take the function f l,η from Section H.1 with parameters

Using Lemma H.1, one can see that f l,η is convex, M -Lipschitz, and L -smooth. Therefore, f l,η ∈ F conv R,M,L . Further, we use the notation f := f l,η .

( Step 2 : Oracle Class)

Let us take and D i = Bernouilli ( p ) for all i ∈ [ n ] , where p ∈ (0 , 1] . We denote [ x ] j as the j th index of a vector x ∈ R T +1 . It is left to show this mapping is unbiased and σ 2 -variance-bounded. Indeed,

for all i ∈ [ T +1] , and

because the difference is non-zero only in one coordinate. Thus

where we take

( Step 3 : Analysis of Protocol)

Protocol 3 generates the sequence { x k } ∞ k =0 . Then, we have

Assume that prog ( x k ) < T. Using Lemma H.1, if prog ( x k ) < T, then prog ( y ( x k )) ≤ T and [ y ( x k )] T +1 = 0 . Therefore, f ( x k ) ≥ -5 l 2 T η ≥ -5 l 2 ( T +1) η and

if prog ( x k ) < T. Let us take

to ensure that

if prog ( x k ) < T. It is left to use Lemma D.2 with δ = 1 / 2 :

for

we have if

Wecan take universal constants c 1 and c 2 equal to 8 · 64 to ensure that T ≥ 6 . Therefore, 16 ε ≥ l 2 √ T +1 and (46) holds for

for and for

H.3 Proof of Theorem B.8

Theorem B.8. Assume that Assumptions B.5, B.6 and B.7 hold. Let us take the batch size S = max {⌈ σ 2 / M 2 ⌉ , 1 } , and γ = ε M 2 + σ 2 /S = Θ( ε / M 2 ) in Method 4, then after K ≥ 2 M 2 R 2 / ε 2 iterations the method guarantees that E [ f ( ̂ x K ) ] -f ( x ∗ ) ≤ ε, where ̂ x K = 1 K ∑ K -1 k =0 x k and R = ∥ ∥ x ∗ -x 0 ∥ ∥ .

Proof. Using the same reasoning as in the proof of Theorem 7.4, one can see that Method 4 is just the stochastic gradient method with the batch size S. Method 4 can be rewritten as x k +1 = x k -γ 1 S ∑ S i =1 ̂ ∇ f ( x k ; ξ i ) , where the ξ i are independent random samples. It means that we can use the classical SGD result (Theorem H.2). For a stepsize

H.3.1 The classical SGD theorem in convex optimization

We reprove the classical SGD result (see, for instance, (Lan, 2020)) for convex functions.

Theorem H.2. Assume that Assumptions B.5 and B.6 hold. We consider the SGD method:

For a fixed x ∈ R d , g ( x ) is a random vector such that E [ g ( x )] ∈ ∂f ( x ) ( ∂f ( x ) is the subdifferential of the function f at the point x ),

where

and g ( x k ) are independent vectors for all k ≥ 0 . Then for

Proof. We denote G k as a sigma-algebra generated by g ( x 0 ) , . . . , g ( x k -1 ) . Using the convexity, for all x ∈ R d , we have

Note that

and

E [ ∥ ∥ g ( x k ) ∥ ∥ 2 ∣ ∣ ∣ G k ] = E [ ∥ ∥ g ( x k ) -E [ g ( x k ) ∣ ∣ G k ]∥ ∥ 2 ∣ ∣ ∣ G k ] + ∥ ∥ E [ g ( x k ) ∣ ∣ G k ]∥ ∥ 2 ≤ σ 2 + M 2 .

Therefore, we get

By taking the full expectation and summing the last inequality for t from 0 to K -1 , we obtain

Let divide the last inequality by K, take x = x ∗ , and use the convexity:

The choices of γ and K

H.4 Proof of Theorem B.9

Theorem B.9. Let us consider Theorem B.8. We assume that i th worker returns a stochastic gradient every τ i seconds for all i ∈ [ n ] . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n . Then after

seconds Method 4 guarantees to find an ε -solution.

Proof. The proof is the same as in Theorem 7.5. It is only required to estimate the time that is required to collect a batch of size S . Method 4 returns a solution after

seconds.

H.5 Proof of Theorem B.10

Theorem B.10. Assume that Assumptions B.5, 7.1 and 7.3 hold. Let us take the batch size S = max {⌈ ( σ 2 R ) / ( ε 3 / 2 √ L ) ⌉ , 1 } , and γ = min { 1 4 L , [ 3 R 2 S 4 σ 2 ( K +1)( K +2) 2 ] 1 / 2 } in Accelerated Method 4, then after K ≥ 8 √ LR √ ε iterations the method guarantees that E [ f ( x K ) ] -f ( x ∗ ) ≤ ε, where R ≥ ∥ ∥ x ∗ -x 0 ∥ ∥ .

Proof. One can see that Accelerated Method 4 is just the accelerated stochastic gradient method with the batch size S. It means that we can use the classical SGD result (Proposition 4.4 in Lan (2020)). For a stepsize

if

where we use the choice of S.

H.6 Proof of Theorem B.11

Theorem B.11. Let us consider Theorem B.10. We assume that i th worker returns a stochastic gradient every τ i seconds for all i ∈ [ n ] . Without loss of generality, we assume that 0 < τ 1 ≤ · · · ≤ τ n . Then after

seconds Accelerated Method 4 guarantees to find an ε -solution.

Proof. The proof is the same as in Theorem 7.5. It is only required to estimate the time that is required to collect a batch of size S . Accelerated Method 4 returns a solution after

seconds.

we have

Therefore,

I Construction of Algorithm for Rennala SGD

In this section, we provide the formal construction of the algorithm from Definition 4.1 for Rennala SGD. We consider the fixed computation model, where i th worker requires τ i seconds to calculate stochastic gradients. We now define the corresponding sequence { A k } ∞ k =0 . Let us fix a starting point x 0 ∈ R d , a stepsize γ ≥ 0 , and a batch size S ∈ N .

First, let us consider the sequence { ̂ t k } ∞ k =1 from Definition D.4 that represents the times when the workers would be ready to provide stochastic gradients. Additionally, let us define ̂ t 0 := 0 and a sequence of the workers' indices { ̂ i k } ∞ k =1 that are corresponding to the times { ̂ t k } ∞ k =1 . We can define

for all k ≥ 1 , and A 0 = ( ̂ t 0 , ̂ i 1 , x 0 ) . For the fixed computation model, one can use this algorithm in Protocol 3 with the oracle (7) to get an equivalent procedure to Method 4.

J Experiments

In this section, we compare Rennala SGD with Asynchronous SGD and Minibatch SGD on quadratic optimization tasks with stochastic gradients. The experiments were implemented in Python 3.7.9. The distributed environment was emulated on machines with Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz.

J.1 Setup

We consider the homogeneous optimization problem (1) with the convex quadratic function

We take d = 1000 ,

Assume that all n workers has access to the following unbiased stochastic gradients:

where ξ ∼ D i = Bernouilli ( p ) for all i ∈ [ n ] , where p ∈ (0 , 1] . We denote [ x ] j as the j th index of a vector x ∈ R d . In our experiments, we take p = 0 . 01 and the starting point x 0 = [ √ d, 0 , . . . , 0] ⊤ . We emulate our setup by considering that the i th worker requires √ i seconds to calculate a stochastic gradient. In all methods, we fine-tune step sizes from a set { 2 i | i ∈ [ -20 , 20] } . In Rennala SGD, we fine-tune the batch size S ∈ { 1 , 5 , 10 , 20 , 40 , 80 , 100 , 200 , 500 , 1000 } .

J.2 Results

In Figures 1, 2, and 3, we present experiments with different number of workers n ∈ { 100 , 1000 , 10000 } . When the number of workers n = 100 , Rennala SGD with Asynchronous SGD converge to the minimum at almost the same rate. However, when we start increasing the number of workers n , one can see that Asynchronous SGD 9 starts converging slower. This is an expected behavior since the maximum theoretical step size in Asynchronous SGD decreases as the number of workers n increases (Koloskova et al., 2022; Mishchenko et al., 2022).

K Experiment with Small-Scale Machine Learning Task

We also consider the methods in a more practical scenario. We solve a logistic regression problem with the MNIST dataset (LeCun et al., 2010). We take n = 1000 workers that hold the same subset of MNIST of the size 3000 . Each worker samples stochastic gradients of size 4 . In Figures 4 and 5, we provide convergence rates and a histogram of the time delays from the experiment. As in (Mishchenko et al., 2018), we can observe that asynchronous methods converge faster than Minibatch SGD. Unlike Section J where Rennala SGD converges faster than Asynchronous SGD, these methods have almost the same performance in this particular experiment.

0

200

400

600

800

1000

times (seconds)

10

2

10

1

10 0

10 1

10 2

10 3

f

(

x t

)

f

(

x

*

)

Minibatch SGD: Step size: 1.0

Minibatch SGD: Step size: 0.5

Minibatch SGD: Step size: 0.25

0

1000

2000

3000

4000

5000

times (seconds)

10

2

10

1

10 0

10 1

10 2

10 3

f

(

x t

)

f

(

x

*

)

Minibatch SGD: Step size: 1.0

Minibatch SGD: Step size: 0.5

Minibatch SGD: Step size: 0.25

0

200

400

600

800

1000

times (seconds)

10

2

10

1

10 0

10 1

10 2

f

(

x t

)

f

(

x

*

)

Minibatch SGD: Step size: 1.0 (Timeout)

Minibatch SGD: Step size: 2.0 (Timeout)

Minibatch SGD: Step size: 0.5 (Timeout)

0

250

500

750

1000

1250

1500

1750

2000

time

10

2

10

1

10 0

10 1

f

(

x t

)

f

(

x

*

)

Minibatch SGD: Step size: 4.0

Minibatch SGD: Step size: 2.0

Minibatch SGD: Step size: 8.0

1

2

3

4

5

time

0

10

20

30

40

50

60

number of delays

L Time Complexity of Asynchronous SGD

The works (Mishchenko et al., 2022; Koloskova et al., 2022) state that Asynchronous SGD convereges after

iterations. This result directly does not reveal the time complexity of Asynchronous SGD. Let us provide the time complexity for the case when the workers require exactly τ i seconds to compute stochastic gradients. Let us fix a time t ≥ 0 . Then the workers will calculate at most

stochastic gradients. To get ε -stationary point, we have to find the minimal t such that

where the quantity c is a numerical constant, since the number of iterations can not be larger than the number of calculated gradients. Note that

Therefore, the minimal time t that satisfies (49) is greater or equal to (50).

M Analysis of Fixed-Computation Model Using Graph Oracle Models

In this section, we analyze the fixed computation model, where i th worker requires τ i seconds to calculate stochastic gradients. Without loss of generality, let us assume that 0 < τ 1 ≤ · · · ≤ τ n . We use the graph oracle framework by Woodworth et al. (2018). Let us fix some t > τ n . Then, in the fixed computation model, the depth of the graph oracle D is at least Ω( t / τ 1 ) . This is the number of gradients that the first node can calculate after t seconds. The size of the graph N equals Θ ( ∑ n i =1 t τ i ) since all workers calculate in parallel. Applying these estimates to Theorem 1 of (Woodworth et al., 2018), one can see that, for convex, L -smooth, M -Lipschitz problems with unbiased and σ 2 -variance-bounded stochastic gradients on the ball B 2 (0 , R ) , the lower bound is at most equals

From this estimate, we can conclude that in order to get an ε -solution, it is required

seconds.

M.1 Example when the lower bound from (Woodworth et al., 2018) is not tight

Let us provide an example when the lower bound from Theorem B.4 is strictly higher. Let us take τ i = √ i for all i ∈ [ n ] . Then, it is sufficient to compare

and

Let us divide both formulas by the term with minimum and obtain

The time t ′ that satisfies is

and

for some a ≥ 0 . Since ∑ m i =1 1 τ i = ∑ m i =1 1 √ i = Θ( √ m ) , then we get

and

if a ≤ n. For instance, let us additionally assume that a = √ n, then the first term equals Θ ( n 1 / 4 ) , while the second equals Θ(1) . Theorefore, the lower bound from Theorem B.4 is tighter.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConsider the classical statistical risk minimization framework with stochastic gradients, and propose new lower bounds on the optimal time complexity. Also propose a new method Rennala SGD that attains the lower bound. This work makes the case for asynchronous methods.\n\nIn particular, this work proposes a new oracle-based lower bound on the optimal convergence rate in time, rather than iterations, and develops an algorithm (essentially straggler-based SGD) that achieves this bound in the shared-system/parameter-server setting.\n\nSTRENGTHS:\n**Significance**\nThis work takes a step towards developing more practical asynchronous optimization methods, and providing convergence rates and complexity bounds that more closely reflect practical considerations of convergence time, rather than iterations.\n\n**Originality**\nThe proposed method is not entirely novel (e.g., see Dutta et al., 2018 or related works on straggler-based SGD methods that ignore computation from stragglers), however the new lower bounds, problem formulation, and analysis are of sufficient novelty, to the best of my knowledge.\n\n**Clarity**\nThe work is well written and easy to follow.\n\nWEAKNESSES:\nLacking empirical evaluations, the only such experiments are on quadratics, buried deep inside the appendix. While these plots are compelling, I would encourage the authors to consider small-scale numerical results on other machine learning workloads.\n\nAnother weakness of this work is on assuming a fixed computation time for each worker. Such models are highly restrictive and impractical. I would encourage the authors to consider ways for extending their framework to arbitrary, but bounded worker delays. I would appreciate a discussion on how such lower-bounds (or the optimal method), may change if these bounds were overly conservative and long-tailed.\n\nQUESTIONS:\nPlease see my questions provided in-line above, in the weaknesses section.\n\nLIMITATIONS:\nLimitations of the work are not adequately addressed. There are for instance, many assumptions that are taken for granted in the analysis that may limit the applicability of the findings to practical settings... namely the fixed delay model per worker, and the relationship between the batch-size S and the number of workers n. I am curious how findings would change in settings where S >> n or S << n.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a protocol that generalizes the classical oracle framework approach. Using this protocol, it establishes minimax complexities for parallel optimization methods that have access to an unbiased stochastic gradient oracle with bounded variance.

STRENGTHS:
- The motivation of this paper is clear. 
- The theoretical analysis seems sound.

WEAKNESSES:
I am not an expert on this topic of this paper.

1. The challenge of theoretical analysis should be made more clear, which highly relates to the contribution of this paper.
2. The empirical study lacks.
3. The organization of this paper is strange. E.g., related work is located in 6.1, and conclusion section lacks, etc.

QUESTIONS:
Please cf. weaknesses part.

LIMITATIONS:
none
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies time complexity of parallel stochastic optimization, establishes lower bounds, and proposes algorithms achieving matching complexity results.\n\nSTRENGTHS:\nBased on my reading of the main paper (I haven't read the proofs), this is a solid theoretical work. The theoretical results improve widely used asynchronous SGD, and are even a bit surprising, especially the fact that ignoring some computations actually helps.\n\nWEAKNESSES:\nGeneral comments:\n- The organization of the paper is a bit reader unfriendly. The assumptions are mentioned at multiple places, starting with page 1, but are formally introduced almost at the end (page 8). This makes the reading tedious.\n\nMinor comments:\n- line 16, shouldn't the codomain of $f$ be $\\mathbb R$ rather than $\\mathbb R^d$?\n- line 39: $L, \\Delta$ have not been introduced so far in the paper\n- Async. SGD (after line 52): step 2 gives the impression that only homogeneous data distribution across devices is considered in the paper. This is true for the main paper, but since the authors do have heterogeneous case results as well, they might consider reflecting that here.\n- Lines 98-99: \"this approach is not convenient\" Maybe giving a short intuition why (which becomes clear on the next page), would help the reader.\n- Typo in line 212\n\nQUESTIONS:\nDoubts:\n- In (6), S_t is the set of indices that started computing gradients before time t. Shouldn't we consider k's which \"finished\" computation before t?\n- Line 205: what does it mean by the freedom to interrupt oracles?\n\nQuestions:\n- As the authors themselves remark in line 237, Rennala SGD goes contradictory to the idea of using all stochastic gradients. In that case, have the authors studied how it performs in practice in comparison to async SGD, especially with heterogeneity?\n- In Theorem 7.4, we need a large batch size in the presence of sizable noise. Since in a lot of analyses, the ratio of learning rate and batch size is the crucial quantity, is it possible to choose $S=1$, and $\\gamma = \\Theta (\\epsilon)$, and still get convergence? One would expect this to be closer to asynch SGD while achieving better rates, and performing better in practice.\n- I haven't looked at Appendix A, but looking at the heterogeneous result in Table 1, the lower bound and Malenia SGD complexity have no dependence on the heterogeneity term. How come?\n- The bound in (12) for when the workers start simultaneously: What is the explanation for this approach being slower?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the minimax complexities of distributed asynchronous stochastic optimization methods. By extending the oracle framework previously used in the literature, it establishes new (lower) lower bounds for parallel optimization methods. Based on the insights from their proof, they propose a new algorithm meeting these bounds. Notably, it is provably faster than previous synchronous and asynchronous methods in the homogeneous case, which is verified experimentally.\n\nSTRENGTHS:\n* This paper naturally extends the oracle protocol of previous work to cater for analysing parallel methods instead of sequential ones, which allows for a finer analysis of distributed methods.\n* Efforts have been made in the writing to explain the introduced novelties step by step, which greatly helps understand and position the contributions.\n* The insight taken from their new complexity proof that “ignoring stale gradients might actually help to converge faster” is interesting and highly relevant to the community, as it led to the development of a new state-of-art optimal method, which has provably and experimentally faster convergence rates than previous asynchronous algorithms in the homogeneous case.\n\nWEAKNESSES:\n* Many notations are introduced, which makes the paper a bit cluttered and cumbersome to read at times.\n* No experiments comparing Renata & Asynchronous SGD to minibatch (synchronized) SGD are made to confirm the “provably better rates” claimed Section 8.\n\nQUESTIONS:\n* Your model assumes fixed delays per worker: could your analysis also hold with stochastic ones?\n* You experimented with a fixed delay of $\\sqrt i$ for worker $i$ (line 1163). Did you also experiment with real-life delays experienced naturally in the distributed asynchronous setting (as in Mischenko 2022 [1])? (For example, this would hint that your method also work with stochastic delays)\n* Did you experiment with other convex functions other than the one introduced line 1159, and did you consistently observe that Renata leads to better results than Asynchronous SGD, or is this highly dependent on the convex functions used?\n* Why Goyal et al 2017 [2] is cited line 45?\n\n**Typos :**\n\n* Line 212: our framework *is*.\n\n**References :**\n\n[1] Mishchenko, K., Bach, F., Even, M., andWoodworth, B, *Asynchronous SGD beats minibatch SGD under arbitrary delays*, In NeurIPS, 2022.\n\n[2] Goyal, P., Dolla´r, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K., *Accurate, large minibatch SGD: Training imagenet in 1 hour.*\n\nLIMITATIONS:\n*", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-22"}