Title: On the Optimal Time Complexities in Decentralized Stochastic Asynchronous Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: nonconvex and convex optimization

stochastic optimization

asynchronous methods

decentralized optimization

lower bounds
Abstract: We consider the decentralized stochastic asynchronous optimization setup, where many workers asynchronously calculate stochastic gradients and asynchronously communicate with each other using edges in a multigraph. For both homogeneous and heterogeneous setups, we prove new time complexity lower bounds under the assumption that computation and communication speeds are bounded by constants. After that, we developed a new nearly optimal method, Fragile SGD, and a new optimal method, Amelie SGD, that converge with arbitrary heterogeneous computation and communication speeds and match our lower bounds (up to a logarithmic factor in the homogeneous setting). Our time complexities are new, nearly optimal, and provably improve all previous asynchronous/synchronous stochastic methods in the decentralized setup.

On the Optimal Time Complexities in Decentralized Stochastic Asynchronous Optimization

Abstract

We consider the decentralized stochastic asynchronous optimization setup, where many workers asynchronously calculate stochastic gradients and asynchronously communicate with each other using edges in a multigraph. For both homogeneous and heterogeneous setups, we prove new time complexity lower bounds under the assumption that computation and communication speeds are bounded. We develop a new nearly optimal method, Fragile SGD , and a new optimal method, Amelie SGD , that converge under arbitrary heterogeneous computation and communication speeds and match our lower bounds (up to a logarithmic factor in the homogeneous setting). Our time complexities are new, nearly optimal, and provably improve all previous asynchronous/synchronous stochastic methods in the decentralized setup.

1 Introduction

We consider the smooth nonconvex optimization problem

where f : R d × S ξ → R , and D ξ is a distribution on a non-empty set S ξ . For a given ε > 0 , we want to find a possibly random point ¯ x, called an ε -stationary point, such that E [ ∥∇ f (¯ x ) ∥ 2 ] ≤ ε. We analyze the heterogeneous setup and the convex setup with smooth and non-smooth functions in Sections C and D.

1.1 Decentralized setup with times

Weinvestigate the following decentralized asynchronous setup. Assume that we have n workers/nodes with the associated computation times { h i } , and communications times { ρ i → j } . It takes less or equal to h i ∈ [0 , ∞ ] seconds to compute a stochastic gradient by the i th node, and less or equal ρ i → j ∈ [0 , ∞ ] seconds to send directly a vector v ∈ R d from the i th node to the j th node (it is possible that h i = ∞ and ρ i → j = ∞ ). All computations and communications can be done asynchronously and in parallel. We would like to emphasize that h i ∈ [0 , ∞ ] and ρ i → j ∈ [0 , ∞ ] are only upper bounds, and the real and effective computation and communication times can be arbitrarily heterogeneous and random. For simplicity of presentation, we assume the upper bounds are static; however, in Section 5.5, we explain that our result can be trivially extended to the case when the upper bounds are dynamic.

We consider any weighted directed multigraph parameterized by a vector h ∈ R n such that h i ∈ [0 , ∞ ] , and a matrix of distances { ρ i → j } i,j ∈ R n × n such that ρ i → j ∈ [0 , ∞ ] for all i, j ∈ [ n ] and

Peter Richtárik ∗

1

2

3

4

5

6

2

1

1

1

3

4

7

3

2

3

1

2

3

4

5

6

̸

ρ i → i = 0 for all i ∈ [ n ] . Every worker i is connected to any other worker j with two edges i → j and j → i. For this setup, it would be convenient to define the distance of the shortest path from worker i to worker j :

̸

̸

is the set of all possible paths without loops from worker i to worker j for all i, j ∈ [ n ] . One can easily show that the triangle inequality τ i → j ≤ τ i → k + τ k → j holds for all i, j, k ∈ [ n ] . Note that τ i → j ≤ ρ i → j for all i, j ∈ [ n ] . It is important to distinguish τ i → j and ρ i → j because it is possible that τ i → j < ρ i → j = ∞ if workers i and j are connected by an edge ρ i → j = ∞ , and there is a path through other workers (see Fig. 1).

We work with the following standard assumption from smooth nonconvex stochastic optimization literature.

Assumption 1. f is differentiable and L -smooth, i.e., ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ ∀ x, y ∈ R d .

Assumption 2. There exist f ∗ ∈ R such that f ( x ) ≥ f ∗ for all x ∈ R d .

Assumption 3. For all x ∈ R d , stochastic gradients ∇ f ( x ; ξ ) are unbiased and σ 2 -variancebounded, i.e., E ξ [ ∇ f ( x ; ξ )] = ∇ f ( x ) and E ξ [ ∥∇ f ( x ; ξ ) - ∇ f ( x ) ∥ 2 ] ≤ σ 2 , where σ 2 ≥ 0 . We also assume that computation and communication times are statistically independent of stochastic gradients.

2 Previous Results

2.1 Time complexity with one worker

For the case when n = 1 and τ 1 → 1 = 0 , convergence rates and time complexities of problem (1) are well-understood. It is well-known that the stochastic gradient method ( SGD ), i.e., x k +1 = x k -γ ∇ f ( x k ; ξ k ) , where { ξ k } are i.i.d. from D ξ , has the optimal oracle complexity Θ( L ∆ / ε + σ 2 L ∆ / ε 2 ) (Ghadimi and Lan, 2013; Arjevani et al., 2022). Assuming that the computation time of one stochastic gradient is bounded by h 1 , we can conclude that the optimal time complexity is

seconds in the worst case.

2.2 Parallel optimization without communication costs

Assume that n > 1 and τ i → j = 0 for all i, j ∈ [ n ] , and computation times of stochastic stochastic gradients are arbitrarily heterogeneous. The simplest baseline method in this setup is Minibatch SGD ,

where { ξ k i } are i.i.d. from D ξ and the gradient ∇ f ( x k ; ξ k i ) is calculated in worker i in parallel. This method waits for stochastic gradients from all workers; thus, it is not robust to 'stragglers' and in the worst case the time complexity of such an algorithm is

which depends on the time max i ∈ [ n ] h i of the slowest worker. There are many other more advanced methods including Picky SGD (Cohen et al., 2021), Asynchronous SGD (e.g., (Recht et al., 2011; Nguyen et al., 2018; Mishchenko et al., 2022; Koloskova et al., 2022)), and Rennala SGD (Tyurin and Richtárik, 2023) that are designed to be robust to workers' chaotic computation times. Under the assumption that the computation times of the workers are heterogeneous and bounded by { h i } , Tyurin and Richtárik (2023) showed that Rennala SGD is the first method that achieves the optimal time complexity

where π is a permutation that sorts h i : h π 1 ≤ · · · ≤ h π n . For instance, one can see that T τ =0 Rennala ≤ T τ =0 mini for all parameters.

2.3 Parallel optimization with communication costs τ i → j

We now consider the setup where workers' communication times can not be ignored. This problem leads to a research field called decentralized optimization . This setup is the primary case for us. All n workers calculate stochastic gradients in parallel and communicate with each other. Numerous works consider this setup, and we refer to Yang et al. (2019); Koloskova (2024) for detailed surveys. Typically, methods in this setting use the gossip matrix framework (Duchi et al., 2011; Shi et al., 2015; Koloskova et al., 2021) and get an iteration converge rate that depends on the spectral gap of a mixing matrix. However, such rates do not give the physical time of algorithms (see also Section B).

Let us consider a straightforward baseline: Minibatch SGD . We can implement (4) in a way that all workers calculate one stochastic gradient (takes at most max i ∈ [ n ] h i seconds) and then aggregate them to one pivot worker j ∗ (takes at most max i ∈ [ n ] τ i → j ∗ seconds). Then, pivot worker j ∗ calculates i.e.,

a new point x k +1 and broadcasts it to all workers (takes max i ∈ [ n ] τ j ∗ → i seconds). One can easily see that the time complexity of such a procedure is 4

Wecan analyze any other asynchronous decentralized method, which will be done with more advanced methods in Section 5.4.

But what is the best possible (optimal) time complexity we can get in the setting from Section 1.1?

Unlike the setups from Sections 2.1 and 2.2 when the communication times are zero ( τ i → j = 0 for all i, j ∈ [ n ] ), the optimal time complexity and an optimal method for the case τ i → j ≥ 0 for all i, j ∈ [ n ] are not known. Our main goal in this paper is to solve this problem.

3 Contributions

We consider the class of functions that satisfy the setup and the assumptions from Section 1.1 and show that (informally) it is impossible to develop a method that will converge faster than (7) seconds. Next, we develop a new asynchronous stochastic method, Fragile SGD , that is nearly optimal (i.e., almost matches this lower bound; see Table 1 and Corollary 1). This is the first such method. It provably improves on Asynchronous SGD (Even et al., 2024) and all other synchronous and asynchronous methods (Bornstein et al., 2023). We also consider the heterogeneous setup (see Table 2 and Section C), where we discover the optimal time complexity by proving another lower bound and developing a new optimal method, Amelie SGD , with weak assumptions. The developed methods can guarantee the iteration complexity O ( L ∆ / ε ) with arbitrarily heterogeneous random computation and communication times (Theorems 4 and 8). Our findings are extended to the convex setup in Section D, where we developed new accelerated methods, Accelerated Fragile SGD and Accelerated Amelie SGD .

4 Lower Bound

In order to construct our lower bound, we consider any (zero-respecting) method that can be represented by Protocol 1. This protocol captures all virtually distributed synchronous and asynchronous methods, such as Minibatch SGD , SWIFT (Bornstein et al., 2023), Asynchronous SGD (Even et al., 2024), and Gradient Tracking (Koloskova et al., 2021).

For all such methods we prove the following theorem.

Theorem 1 (Lower Bound; Simplified Presentation of Theorem 19) . Consider Protocol 1 with ∇ f ( · ; · ) . We take any h i ≥ 0 and τ i → j ≥ 0 for all i, j ∈ [ n ] such that τ i → j ≤ τ i → k + τ k → j for all i, k, j ∈ [ n ] . We fix L, ∆ , ε, σ 2 > 0 that satisfy the inequality ε < cL ∆ for some universal constant

c. For any (zero-respecting) algorithm, there exists a function f, which satisfy Assumptions 1, 2 and f (0) -f ∗ ≤ ∆ , and a stochastic gradient mapping ∇ f ( · ; · ) , which satisfies Assumption 3, such that the required time to find ε -solution is

where, for j ∈ [ n ] , π j, · is a permutation that sorts { max { τ i → j , h i }} n i =1 , i.e., max { τ π j, 1 → j , h π j, 1 } ≤ · · · ≤ max { τ π j,n → j , h π j,n } .

Protocol 1 Simplified Presentation of Protocol 8

1: Init S i = ∅ (all available information) on worker i for all i ∈ [ n ]

2: Run the following two loops in each worker in parallel

3: while True do

4: Calculate a new point x k based on S

5: Calculate a stochastic gradient ∇ f ( x k i ; ξ ) (or ∇ f i ( x k i ; ξ ) ) ξ ∼ D ξ (takes h i seconds)

i i (takes 0 seconds)

6: Atomic add ∇ f ( x k i ; ξ ) (or ∇ f i ( x k i ; ξ ) ) to S i (atomic operation, takes 0 seconds)

7: end while

8: while True do

9: Send (a) any vector from R d based on S i to any worker j and go to the next step of this loop without waiting (takes τ i → j seconds to send; worker j adds this vector to S j )

10: end while

(a): When we prove the lower bounds, we allow algorithms to send as many vectors as they want in parallel from worker i to worker j for all i = j ∈ [ n ] .

̸

The intuition and meaning of the formula (8) is discussed in Section 5.2. Note that if we take n = 1 and τ 1 → 1 = 0 our lower bound reduces to the lower bound (3) up to a log factor. Moreover, if we take n > 1 and τ i → j = 0 for all i, j ∈ [ n ] , then (8) reduces to (5) up to a log factor. Thus, (8) is nearly consistent with the lower bounds from (Arjevani et al., 2022; Tyurin and Richtárik, 2023). We get an extra log n factor due to the generality of our setup. The reason is technical, and we explain it in Section E.5. In a nutshell, the lower problem reduces to the analysis of the concentration of the time series y T := min j ∈ [ n ] y T j and y T j := min i ∈ [ n ] { y T -1 i + h i η T i + τ i → j } , where y 0 i = 0 for all i ∈ [ n ] , and { η k i } are i.i.d. geometric random variables. This analysis is not trivial due to the min i ∈ [ n ] operations. Virtually all previous works that analyzed lower bounds did not have such a problem because they analyzed time series with a sum structure (e.g., ¯ y T := ¯ y T -1 + ϱ T , where { ϱ k } are some random variables, and ¯ y 0 = 0 ).

Let us define an auxiliary function to simplify readability.

Definition 2 (Equilibrium Time) . A mapping t ∗ : R ≥ 0 × R n ≥ 0 × R n ≥ 0 → R ≥ 0 with inputs s (scalar), [ h i ] n i =1 (vector), and [¯ τ i ] n i =1 (vector) is called the equilibrium time if it is defined as follows. Find a permutation 5 π that sorts max { ¯ τ i , h i } as max { ¯ τ π 1 , h π 1 } ≤ · · · ≤ max { ¯ τ π n , h π n } . Then the mapping returns the value

5 New Method: Fragile SGD

We introduce a novel optimization method characterized by time complexities that closely align with the lower bounds established in Section 4. Our algorithms leverage spanning trees . A spanning tree is a tree (undirected unweighted graph) encompassing all workers. The edges of spanning trees are virtual and not related to the edges defined in Section 1.1 (see Fig. 1).

Algorithm 2 Fragile SGD

1: Input: starting point x 0 , stepsize γ , batch size S , pivot worker j ∗ , spanning trees st and st bc

2: Start Process 0 (Algorithm 3) in worker j ∗

3: Start Process i (Algorithm 4) in all workers for all i ∈ [ n ] (including worker j ∗ )

Algorithm 3 Process 0 (running in worker j ∗ )

1: for k = 0 , 1 , . . . , K -1 do

2: Send x k to Process j ∗

3: Init ( g k , s k ) = (0 , 0)

4: while s < S do

5: Wait for a message ( g k j ∗ , send , s k j ∗ , send ) from Process j ∗

6: g k = g k + g k j ∗ , send ; s k = s k + s k j ∗ , send

7: end while

9: end for

8: x k +1 = x k -γ s k g k

Algorithm 4 Process i (running in worker i )

(a): To simplify the listing of the algorithm, we assume here that a worker can store K auxiliary vectors in the memory. One can see that it's not necessary, and it is sufficient to maintain only one vector g k max i, next . In particular, it is sufficient to modify the logic of Lines 12 and 19, where we run the add operations only if ¯ k = k max and ˆ k = k max . The efficient implementation has O ( d ) floats memory complexity per worker.

(b): BroadcastFurtherAndCalculateStochasticGradients and SendVectorsToNextWorker may try to send through the same edge. In this case, we can interleave their communications and decrease the speed of each line by at most two.

̸

Definition 3 (mapping next T,j ( i ) ) . Take a spanning tree T and fix any worker j ∈ [ n ] . For i = j, we define next T,j ( i ) = 0 . For all i = j ∈ [ n ] , we define next T,j ( i ) as the index of the next worker on the path of the spanning tree T from worker i to worker j.

Our new method, Fragile SGD , is presented in Algorithms 2, 4, and 3. While Fragile SGD seems to be lengthy, the idea is pretty simple. All workers do three jobs in parallel: calculate stochastic gradients, receive vectors, and send vectors through spanning trees. A pivot worker aggregates all stochastic gradients in g k and, at some moment, does x k +1 = x k -γg k . The algorithms formalize this idea.

Algorithm 2 requires a starting point x 0 , a stepsize γ , a batch size S , the index j ∗ of a pivot worker, and spanning trees st and st bc for the input. We need two spanning st bc and st trees because, in general, the fastest communication of a vector from j ∗ to i and from i to j ∗ should be arranged through two different paths. Algorithm 2 starts n +1 processes running in parallel. Note that the pivot worker j ∗ runs two parallel processes, called Process 0 and Process j ∗ , and any other worker i runs one Process i. Process 0 broadcasts a new point x k through st bc to all other processes and goes to the loop where it waits for messages from Process j ∗ . Process i starts three functions that will be running in parallel: i) the first function's job is to receive a new point, broadcast it further, and start the calculation of stochastic gradients, ii) the second function receives stochastic gradients from all previous processes that are sending vectors to worker j ∗ , iii) the third function sends vectors the next worker on the path to j ∗ . By the definition of next st,j ∗ ( · ) , all calculated stochastic vectors are sent to worker j ∗ , where they are first aggregated in Process j ∗ , and then, since next st,j ∗ ( j ∗ ) = 0 , Process j ∗ will send g k j ∗ , send to Process 0 . This process waits for the moment when the number of stochastic gradients s k aggregated in g k is greater or equal to S. When it happens, the loop stops, and Process 0 does a gradient-like step. The structure of the algorithm and the idea of spanning trees resemble the ideas from (Vogels et al., 2021; Tyurin and Richtárik, 2023). The main observation is that this algorithm is equivalent to x k +1 = x k -γ s k ∑ s k i =1 ∇ f ( x k ; ξ k i ) , where s k ≥ S and { ξ k i } are i.i.d. samples. Note that all stochastic gradients calculated at points x 0 , . . . , x k -1 will be ignored in the k th iteration of Algorithm 3.

Theorem 4. Let Assumptions 1, 2, and 3 hold. We take γ = 1 / 2 L, batch size S = max { ⌈ σ 2 / ε ⌉ , 1 } , any pivot worker j ∗ ∈ [ n ] , and any spanning trees st and st bc in Algorithm 2. For all K ≥ 16 L ∆ /ε, we get 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

The proof is simple and uses standard techniques from (Lan, 2020; Khaled and Richtárik, 2022). The result of the theorem holds even if h i = ∞ and τ i → j = ∞ for all i, j ∈ [ n ] because h i and τ i → j are only upper bounds on the real computation and communications speeds. In Algorithm 3, each iteration k can be arbitrarily slow, and still, the result of Theorem 4 holds and the method converges after O ( L ∆ /ε ) iterations. The next result gives time complexity guarantees for our algorithm.

Theorem 5. Consider the assumptions and the parameters from Theorem 4. For any pivot worker j ∗ ∈ [ n ] and spanning trees st and st bc , Algorithm 2 converges after at most

seconds, where µ i → j ∗ ( µ j ∗ → i ) is an upper bound on the times required to send a vector from worker i to worker j ∗ (from worker j ∗ to worker i ) along the spanning tree st (spanning tree st bc ).

Note that our method does not need the knowledge of { h i } and { µ i → j } to guarantee the time complexity rate, and it automatically obtains it.

Corollary 1. Consider the assumptions and the parameters from Theorem 5. Let us take a pivot worker j ∗ = arg min j ∈ [ n ] t ∗ ( σ 2 / ε , [ h i ] n i =1 , [ τ i → j + τ j → i ] n i =1 ) , and a spanning tree st (spanning tree st bc )

that connects every worker i to worker j ∗ (worker j ∗ to every worker i ) with the shortest distance τ i → j ∗ ( τ j ∗ → i ). Then Algorithm 2 converges after at most

seconds, where, for all j ∈ [ n ] , π j, · is a permutation that sorts { max { τ i → j + τ j → i , h i }} n i =1 .

This corollary has better time complexity guarantees than Theorem 5 because, by the definition of τ i → j , τ i → j ≤ µ i → j for al i, j ∈ [ n ] . However, it requires the particular choice of a pivot worker and spanning trees.

5.1 Discussion

Comparing the lower bound (7) with the upper bound (11), one can see that Fragile SGD has a nearly optimal time complexity . If we ignore the log n factor in (7) and assume that τ i → j = τ j → i for all i, j ∈ [ n ] , which is a weak assumption in many applications, then Fragile SGD is optimal.

Unlike most works (Even et al., 2024; Lian et al., 2018; Koloskova et al., 2021) in the decentralized setting, our time complexity guarantees do not depend on the spectral gap of the mixing matrix that defines the topology of the multigraph. The structure of the multigraph is coded in the times { τ i → j } . We believe that this is an advantage of our guarantees since (11) defines a physical time instead of an iteration rate that depends on the spectral gap.

5.2 Interpretation of the upper and lower bounds (11) and (7)

One interesting property of our algorithm is that some workers will potentially never contribute to optimization because either their computations are too slow or communication times to j ∗ are too large. Thus, only a subset of the workers should work to get the optimal time complexity!

Assume in this subsection that the computation and communication are fixed to { h i } and { τ i → j } . One can see that (12) is the min j ∈ [ n ] min k ∈ [ n ] over some formula. In view of our algorithm, an index j ∗ that minimizes in (12) is the index of a pivot worker that is the most 'central' in the multigraph. An index k ∗ that minimizes min k ∈ [ n ] defines a set of workers { π j ∗ , 1 , . . . π j ∗ ,k ∗ } that can potentially contribute to optimization. The algorithm and and the time complexity will not depend on workers { π j ∗ ,k ∗ +1 , . . . π j ∗ ,n } because they are too slow or they are too far from worker j ∗ . Thus, up to a constant factor, we have T ∗ = L ∆ / ε max { max { τ π j ∗ ,k ∗ → j ∗ + τ j ∗ → π j ∗ ,k ∗ , h π j ∗ ,k ∗ } , σ 2 / ε ( ∑ k ∗ i =1 1 / h π j ∗ ,i ) -1 } , where τ π j ∗ ,k ∗ → j ∗ + τ j ∗ → π j ∗ ,k ∗ is the time required to communicate with the farthest worker that can contribute to optimization, h π j ∗ ,k ∗ is the computation time of the slowest worker that can contribute to optimization, and σ 2 / ε ( ∑ k ∗ i =1 1 / h π j ∗ ,i ) -1 is the time required to 'eliminate' enough noise before the algorithm does an update of x k .

5.3 Limitations

To get the nearly optimal complexity, it is crucial to select the right pivot worker j ∗ and spanning trees according to the rules of Corollary 1, which depend on the knowledge of the bounds of times. For now, we believe that is this a price for the optimality. Note that Theorem 5 does not require this knowledge and it works with any j ∗ and any spanning tree; thus, we can use any heuristic to estimate an optimal j ∗ and optimal spanning trees. One possible strategy is to estimate the performance of workers and the communication channels using load testings.

5.4 Comparison with previous methods

Let us discuss the time complexities of previous methods. Note that none of the previous methods can converge faster than (7) due to our lower bound. First, consider (6) of Minibatch SGD . This time complexity depends on the slowest computation time max i ∈ [ n ] h i and the slowest communication times max i,j ∈ [ n ] τ i → j . In the asynchronous setup, it is possible that one the workers is a straggler, i.e., max i ∈ [ n ] h i ≈ ∞ , and Minibatch SGD can be arbitrarily slow. Our time complexities (10) and (12) are robust to stragglers, and ignore them. Assume the last worker n is a straggler and h n = ∞ , then one can take permutations with π j,n = n for all j ∈ [ n ] , and the minimum operator min k ∈ [ n ] in (12) will not choose k = n because max { τ π j,n → j + τ j → π j,n , h π j,n } = ∞ for all j ∈ [ n ] .

We now consider a recent work by Even et al. (2024), where the authors analyzed Asynchronous SGD in the decentralized setting. In the homogeneous setting, their converge rate depends on the maximum compute delay and, thus, is not robust to stragglers. For the case τ i → j = 0 , our time complexity (11) reduces to (5). At the same time, it was shown (Tyurin and Richtárik, 2023) that the time complexity of Asynchronous SGD for τ i → j = 0 is strongly worse than (5); thus, the result by Even et al. (2024) is

suboptimal in our setting even if τ i → j = 0 for all i, j ∈ [ n ] . The papers by Bornstein et al. (2023); Lian et al. (2018) also consider the same setting, and they share a logic in that they sample a random worker and wait while it is calculating a stochastic gradient. If one of the workers is a straggler, they can wait arbitrarily long, while our method automatically ignores slow computations.

5.5 Time complexity with dynamic bounds

We can easily generalize Theorem 5 to the case when bounds on the times are not static.

Theorem 6. Consider the assumptions and the parameters from Theorem 4. In each iteration k of Algorithm 3, the computation times of worker i are bounded by h k i . Let us fix any pivot worker j ∗ ∈ [ n ] and any spanning trees st and st bc . Then Algorithm 2 converges after at most

seconds, where µ k i → j ∗ ( µ k j ∗ → i ) is an upper bound on times required to send a vector from worker i to worker j ∗ (from worker j ∗ to worker i ) along the spanning tree st (spanning tree st bc ) in iteration k of Algorithm 3.

This result is more general than (10), and it shows that our method is robust to changing computation { h k i } and communication { µ k i → j } times bounds during optimization processes. For instance, worker n can have either slow computation or communication to j ∗ in the first iteration, i.e., max { h 1 n , µ 1 n → j ∗ , µ 1 j ∗ → n } ≈ ∞ , then our method will ignore it, but if max { h 2 n , µ 2 n → j ∗ , µ 2 j ∗ → n } is small in the second iteration, then our method can potentially use the stochastic gradients from worker n .

6 Example: Line or Circle

̸

Figure 2: Line with ρ i +1 → i = ρ i → i +1 = ρ for all i ∈ [ n -1] , ρ i → j = ∞ otherwise. For all i = j ∈ [ n ] , edges i → j and j → i are merged and visualized with one undirected edge.

Let us consider Line graphs where we can get more explicit and interpretable formulas for (11). We analyze ND-Mesh, ND-Torus, and Star graphs in Section A. Surprisingly, even in some simple cases like Line or Star graphs, as far as we know, we provide new time complexity results and insights. In Section J, we show that our theoretical results are supported by computational experiments.

We take a Line graph with the computation speeds h i = h for all i ∈ [ n ] , and the communication speeds of the edges ρ i → i +1 = ρ i +1 → i = ρ for all i ∈ [ n -1] and ρ i → j = ∞ for all other i, j ∈ [ n ] . One can easily show the time required to send a vector between two workers i, j ∈ [ n ] equals τ i → j = τ j → i = ρ | i -j | . See an example with n = 7 in Fig. 2. We can substitute these values to (11) and get

where π j, 1 = j, π j, 2 , π j, 3 = j +1 , j -1 or π j, 2 , π j, 3 = j -1 , j +1 (only for n -1 ≥ j ≥ 2 ) and so forth. For simplicity, assume that n is odd, then, clearly, j ∗ = n -1 2 +1 minimizes min j ∈ [ n ] and T line =

According to (15), there are three time complexity regimes: i) slow communication, i.e., √ σ 2 h / ερ ≤ 1 , this inequality means that ρ is so large, that communication between workers will not increase the convergence speed, and the best strategy is to work with only one worker!, ii) medium communication, i.e., n > √ σ 2 h / ερ > 1 , more than one worker will participate in the optimization process; however, not all of them! , some workers will not contribute since their distances τ j ∗ →· to the pivot worker j ∗ are large, iii) fast communication, i.e., √ σ 2 h / ερ ≥ n, all n workers will participate in optimization because ρ is small.

As far as we know, the result (15) is new even for such a simple structure as a line. Note that these regimes are fundamental and can not be improved due to our lower bound (up to logarithmic factors). For Circle graphs, the result is the same up to a constant factor.

7 Heterogeneous Setup

In Section C (in more details), we consider and analyze the problem

where f i : R d × S ξ i → R d and ξ i are random variables with some distributions D i on S ξ i . For all i ∈ [ n ] , worker i can only access f i . We show that the optimal time complexity is

in the heterogeneous setting achieved by a new method, Amelie SGD (Algorithm 5). Amelie SGD is closely related to Rennala SGD but with essential algorithmic changes to make it work with heterogeneous functions. The obtained complexity (16) is worse than (12), which is expected because the heterogeneous setting is more challenging than the homogeneous setting.

8 Highlights of Experiments

In Section J, we present experiments with quadratic optimization problems, logistic regression, and a neural network to substantiate our theoretical findings. Here, we focus on highlighting the results from the logistic regression experiments:

0.940

0.0

0.2

0.4

0.6

0.8

1.0

times (seconds)

1e6

2.2 × 10

1

2.4 × 10

1

2.6 × 10

1

2.8 × 10

1

3×10 1

f

(

x t

)

f

(

x

*

)

Fragile SGD: Batch Size: 120 Step size: 1.0

Fragile SGD: Batch Size: 160 Step size: 1.0

Fragile SGD: Batch Size: 100 Step size: 1.0

Minibatch SGD: Step size: 1.0

Minibatch SGD: Step size: 2.0

Minibatch SGD: Step size: 0.5

0.0

0.2

0.4

0.6

0.8

1.0

times (seconds)

1e6

0.910

0.915

0.920

0.925

0.930

0.935

Accuracy (Test)

Fragile SGD: Batch Size: 120 Step size: 1.0

Fragile SGD: Batch Size: 160 Step size: 1.0

Fragile SGD: Batch Size: 100 Step size: 1.0

Minibatch SGD: Step size: 1.0

Minibatch SGD: Step size: 2.0

Minibatch SGD: Step size: 0.5

On MNIST dataset (LeCun et al., 2010) with 100 workers, Fragile SGD is much faster and has better test accuracy than Minibatch SGD .

Acknowledgments and Disclosure of Funding

The research reported in this publication was supported by funding from King Abdullah University of Science and Technology (KAUST): i) KAUST Baseline Research Scheme, ii) Center of Excellence for Generative AI, under award number 5940, iii) SDAIA-KAUST Center of Excellence in Artificial Intelligence and Data Science. The work of A.T. was partially supported by the Analytical center under the RF Government (subsidy agreement 000000D730321P5Q0002, Grant No. 70-2021-00145 02.11.2021).

References

Arjevani, Y., Carmon, Y., Duchi, J. C., Foster, D. J., Srebro, N., and Woodworth, B. (2022). Lower bounds for non-convex stochastic optimization. Mathematical Programming , pages 1-50.

Bornstein, M., Rabbani, T., Wang, E., Bedi, A. S., and Huang, F. (2023). SWIFT: Rapid decentralized federated learning via wait-free model communication. In The 11th International Conference on Learning Representations (ICLR) .

Carmon, Y., Duchi, J. C., Hinder, O., and Sidford, A. (2020). Lower bounds for finding stationary points i. Mathematical Programming , 184(1):71-120.

Cohen, A., Daniely, A., Drori, Y., Koren, T., and Schain, M. (2021). Asynchronous stochastic optimization robust to arbitrary delays. Advances in Neural Information Processing Systems , 34:9024-9035.

Duchi, J. C., Agarwal, A., and Wainwright, M. J. (2011). Dual averaging for distributed optimization: Convergence analysis and network scaling. IEEE Transactions on Automatic control , 57(3):592606.

Even, M., Koloskova, A., and Massoulié, L. (2024). Asynchronous SGD on graphs: A unified framework for asynchronous decentralized and federated optimization. In Artificial Intelligence and Statistics. PMLR .

Floyd, R. W. (1962). Algorithm 97: shortest path. Communications of the ACM , 5(6):345-345.

Ghadimi, S. and Lan, G. (2013). Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368.

Huang, X., Chen, Y., Yin, W., and Yuan, K. (2022). Lower bounds and nearly optimal algorithms in distributed learning with communication compression. Advances in Neural Information Processing Systems (NeurIPS) .

Khaled, A. and Richtárik, P. (2022). Better theory for SGD in the nonconvex world. Transactions on Machine Learning Research .

Koloskova, A. (2024). Optimization algorithms for decentralized, distributed and collaborative machine learning. Technical report, EPFL.

Koloskova, A., Lin, T., and Stich, S. U. (2021). An improved analysis of gradient tracking for decentralized machine learning. Advances in Neural Information Processing Systems , 34:1142211435.

Koloskova, A., Stich, S. U., and Jaggi, M. (2022). Sharper convergence guarantees for asynchronous SGD for distributed and federated learning. Advances in Neural Information Processing Systems (NeurIPS) .

Lan, G. (2020). First-order and stochastic optimization methods for machine learning . Springer.

LeCun, Y., Cortes, C., and Burges, C. (2010). Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2.

Lian, X., Zhang, W., Zhang, C., and Liu, J. (2018). Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning , pages 3043-3052. PMLR.

Liu, Y., Lin, T., Koloskova, A., and Stich, S. U. (2024). Decentralized gradient tracking with local steps. Optimization Methods and Software , pages 1-28.

Lu, Y. and De Sa, C. (2021). Optimal complexity in decentralized training. In International Conference on Machine Learning , pages 7111-7123. PMLR.

Mishchenko, K., Bach, F., Even, M., and Woodworth, B. (2022). Asynchronous SGD beats minibatch SGD under arbitrary delays. Advances in Neural Information Processing Systems (NeurIPS) .

Nesterov, Y. (1983). A method of solving a convex programming problem with convergence rate o (1/k** 2). Doklady Akademii Nauk SSSR , 269(3):543.

Nguyen, L., Nguyen, P. H., Dijk, M., Richtárik, P., Scheinberg, K., and Takác, M. (2018). SGD and hogwild! convergence without the bounded gradients assumption. In International Conference on Machine Learning , pages 3750-3758. PMLR.

Recht, B., Re, C., Wright, S., and Niu, F. (2011). Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Advances in Neural Information Processing Systems , 24.

Scaman, K., Bach, F., Bubeck, S., Lee, Y. T., and Massoulié, L. (2017). Optimal algorithms for smooth and strongly convex distributed optimization in networks. In International Conference on Machine Learning , pages 3027-3036. PMLR.

Shi, W., Ling, Q., Wu, G., and Yin, W. (2015). A proximal gradient algorithm for decentralized composite optimization. IEEE Transactions on Signal Processing , 63(22):6013-6023.

Tyurin, A., Pozzi, M., Ilin, I., and Richtárik, P. (2024). Shadowheart SGD: Distributed asynchronous SGD with optimal time complexity under arbitrary computation and communication heterogeneity. arXiv preprint arXiv:2402.04785 .

Tyurin, A. and Richtárik, P. (2023). Optimal time complexities of parallel stochastic optimization methods under a fixed computation model. Advances in Neural Information Processing Systems (NeurIPS) .

Van Handel, R. (2014). Probability in high dimension. Lecture Notes (Princeton University) .

Vogels, T., He, L., Koloskova, A., Karimireddy, S. P., Lin, T., Stich, S. U., and Jaggi, M. (2021). RelaySum for decentralized deep learning on heterogeneous data. Advances in Neural Information Processing Systems , 34.

Yang, T., Yi, X., Wu, J., Yuan, Y., Wu, D., Meng, Z., Hong, Y., Wang, H., Lin, Z., and Johansson, K. H. (2019). A survey of distributed optimization. Annual Reviews in Control , 47:278-305.

Contents

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

(b) 3D-Mesh

A More Examples

A.1 ND-Mesh or ND-Torus

We now consider a generalization of Line graphs: ND-Mesh graphs. In Figures 4a and 4b, we present examples of 2D-Mesh and 3D-Mesh. For simplicity, assume that n = (2 k +1) N for some k ∈ N . The computation speeds h i = h for all i ∈ [ n ] , and the communicate speeds of the edges ρ i → j = ρ if workers i and j are connected in a mesh and ρ i → j = ∞ for all other i, j ∈ [ n ] . Using geometrical reasoning, it is clear an index j ∗ , that minimizes (11), corresponds to the worker in the middle of a graph (13 in Figures 4a and 14 in Figures 4b). Therefore,

For now, let us consider a 2D-Mesh graph. The number of workers, that have the length of the shortest path to j ∗ equals to 0 , is 1 . The number of workers, that have the length of the shortest path to j ∗ less or equal to ρ , is 5 . The number of workers, that have the length of the shortest path to j ∗ less or equal to 2 ρ , is 13 . In general, the number of workers, that have the length of the shortest path to j ∗ less or equal to √ kρ , is Θ( k ) for all k ∈ { 0 , . . . , Θ( n ) } . It means

Using the same reasoning, for the general case with a ND-Mesh graph, we get

As in Line graphs, these complexities have three regimes depending on the problem's parameters. Up to a constant factor, the same conclusions apply to ND-Torus.

̸

A.2 Star graph

Let us consider Star graphs with different computation and communication speeds. Let us fix a graph with n +1 workers, where one worker with the index n +1 is in the center, and all other n workers are only directly connected to worker n +1 . Then,

̸

Using these constraints, we can simplify (11). There are two possible best strategies: i) a pivot worker j ∗ works locally without communications, ii) a pivot worker j ∗ works with communications but it would necessary require to communicate through the central worker; in this case, the central worker n +1 is a pivot worker. Therefore, we get

since τ n +1 → π n +1 ,k = ρ n +1 → π n +1 ,k for all k ∈ [ n +1] . Let us slightly simplify the result and assume that broadcasting from the central worker is fast, i.e., ρ n +1 → i ≤ ρ i → n +1 for all i ∈ [ n +1] , then

Tyurin et al. (2024) also considered Star graphs and showed that T fast comm. is the optimal time complexity for methods that communicate through the central worker. Our result is more general since we also consider decentralized methods and capture the term T slow comm. that can be potentially smaller if communication is slow. Tyurin et al. (2024) conjectured that T star is the optimal bound, and we proved it (up to log factor) here.

A.3 General case

We show how one can use our generic result (11) to get an explicit formula in some cases. For the general case with arbitrary communication and computation times, the minimizers j and k in (11) can be found in the following way. First, we have to find τ i → j using any algorithm that solves the all-pairs shortest path problem (e.g., Floyd-Warshall algorithm (Floyd, 1962)). Once we know τ i → j , we should sort { max { τ i → j + τ j → i , h i }} n k =1 for all j ∈ [ n ] to find the permutations. Finally, we have enough information to calculate t ∗ from Definition 2.

B On the Connection to the Gossip Framework

Most of the previous methods were designed for a different setting, gossip-type communication. In fact, our setting is more general than the gossip communication. Indeed, recall that in the gossip communication, worker i is allowed to get vectors from other workers through the operation ∑ n j =1 w ij x j , where w ij ∈ { 0 , 1 } . This is equivalent to our setting for the case when the communication time ρ ij = ∞ when w ij is zero, and ρ ij = 1 if w ij is not zero, and worker i sums the received vectors. But our setting is richer since we allow different communication and computation times and allow workers to do whatever they like with vectors (not only to sum).

Moreover, the gossip framework codes the communication graph through the matrix { w ij } . We propose to code graphs through the times { ρ ij } ( { τ ij } ). Our approach is closer to real scenarios because, as we explained previously, it includes the gossip framework.

C Heterogeneous Setup

We now consider the heterogeneous setting. The only difference is instead of (1), we consider the following problem.

where f i : R d × S ξ i → R d and ξ i are random variables with some distributions D i on S ξ i . We now present our upper and lower bounds and discuss them.

C.1 Lower bound

In Section F, we prove the following lower bound.

Theorem 7 (Lower Bound; Simplified Presentation of Theorem 20) . Consider Protocol 1 with ∇ f i ( · ; · ) . We take any h i ≥ 0 and τ i → j ≥ 0 for all i, j ∈ [ n ] such that τ i → j ≤ τ i → k + τ k → j for all i, k, j ∈ [ n ] . We fix L, ∆ , ε, σ 2 > 0 that satisfy the inequality ε < cL ∆ for some universal constant c.

For any (zero-respecting) algorithm, there exists a function f = 1 n n ∑ i =1 f i , which satisfy Assumptions 1,

2 and f (0) -f ∗ ≤ ∆ , and stochastic gradient mappings ∇ f i ( · ; · ) , which satisfy Assumption 3 ( E ξ [ ∇ f i ( x ; ξ )] = ∇ f i ( x ) and E ξ [ ∥∇ f i ( x ; ξ ) -∇ f i ( x ) ∥ 2 ] ≤ σ 2 ), such that the required time to find ε -solution is

seconds.

C.2 Amelie SGD : optimal method in the heterogeneous setting

We now present a new method based on Malenia SGD from (Tyurin and Richtárik, 2023) and our Fragile SGD . As in Fragile SGD , we also have n +1 processes running in all workers. The main idea is that all workers calculate stochastic gradients in parallel and accumulate them locally. Process 0 zero waits for the moment when n b j ∗ ≥ S n . Process i calculates b i in Line 20 of Algorithm 7, and sends it to Process next st,j ∗ ( i ) . Thus, b j ∗ accumulates the sum ∑ n i =1 1 s k i , which decreases with time since s k i is the number of calculated stochastic gradients in worker i in different moments of time. Therefore, n b j ∗ ≥ S n will hold at some point of time if workers continue computing gradients. Finally, Algorithm 6 runs all reduce and does the update of x k .

Theorem 8. Let Assumptions 1 and 2 hold for the function f and Assumption 3 holds for the functions f i for all i ∈ [ n ] . We take γ = 1 / 2 L, the parameter S = max { ⌈ σ 2 / ε ⌉ , n } , any pivot worker j ∗ ∈ [ n ] , and any spanning trees st and st bc , in Algorithm 5. For all iterations number K ≥ 16 L ∆ /ε, we get 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

Note that Theorem 8 states the convergence of Amelie SGD even if the computation and communication speeds are unbounded.

Theorem 9. Consider the assumptions and the parameters from Theorem 8. For any pivot worker j ∗ ∈ [ n ] and any spanning trees st and st bc , Algorithm 5 converges after at most

seconds, where µ k i → j ∗ ( µ k j ∗ → i ) is an upper bound on times required to send a vector from worker i to worker j ∗ (from worker j ∗ to worker i ) along the spanning tree st (spanning tree st bc ) for all i ∈ [ n ] .

Algorithm 5 Amelie SGD

1: Input: starting point x 0 , stepsize γ , parameter S , pivot worker j ∗ , spanning trees st and st bc

2: Start Process 0 (Alg. 6) in worker j ∗

3: Start Process i (Alg. 7) in all workers for all i ∈ [ n ] (including worker j ∗ )

Algorithm 6 Process 0 (running in worker j ∗ )

1: for k = 0 , 1 , . . . , K -1 do

2: Broadcast x k to all workers using the spanning tree st bc

3: Init s k = 0

4: while s k < S n do

5: Wait for a message b j ∗ from Process j ∗

6: Calculate s k = n b j ∗

7: end while

8: Run all reduce with { 1 s k i g k i } n i =1 to find g k = 1 n n ∑ i =1 1 s k i g k i using the spanning tree st

9: x k +1 = x k -γg k

10: end for

Algorithm 7 Process i (running in worker i )

1: while True do

2: Get a new point x k broadcasted by Process 0

3: Init ( g k i , s k i ) = (0 , 0)

4: Init b i,p = ∞ for all p ∈ [ n ] s.t. next st,j ∗ ( p ) = i

5: Run the following three functions in parallel and go to Line 24

6: function CalculateStochasticGradients

7: while True

8:

Calculate

∇

f

i

(

x

k

;

ξ

)

,

ξ

∼ D

i

9: Run atomic add g k i = g k i + ∇ f i ( x k ; ξ ) , s k i = s k i +1

10: end while

11: end function

12: function ReceiveCountersFromPreviousWorkers

13: while True

14: Wait for a message b p from any Process p s.t. next st,j ∗ ( p ) = i

15: Run atomic update b i,p = b p

16: end while

17: end function

18: function SendCounterToNextWorker

19: while True

21: Send b i ∈ R ∪ {∞} (one float) to Process next st,j ∗ ( i ) and wait while it sending (Process j ∗ sends to Process 0 by the definition of next st,j ∗ ( · ) )

22: end while

23: end function

24: Wait for new point. If receives new point, stop all computations in functions, and continue

25: Ignore all non-received messages

26: end while

Corollary 2. Consider the assumptions and the parameters from Theorem 9. Let us take any pivot worker j ∗ ∈ [ n ] and a spanning tree st (spanning tree st bc ) that connects every worker i to worker j ∗ (worker j ∗ to worker i ) with the shortest distance τ i → j ∗ ( τ j ∗ → i ), then Algorithm 5 converges after at most

seconds.

C.3 Discussion

Corollary 2 together with Theorem 7 states that the time complexity (18) is optimal. The result is pessimistic since the time complexity depends on the 'diameter' max i,j ∈ [ n ] τ i → j and the slowest performance max i ∈ [ n ] h i . A similar dependence was observed in (Lu and De Sa, 2021; Tyurin and Richtárik, 2023). As in (Tyurin and Richtárik, 2023), the stochastic term σ 2 / nε ( 1 / n ∑ n i =1 h i ) depends on the average of { h i } if σ 2 / ε is large.

C.4 Comparison with previous methods

Let us consider Minibatch SGD described in Section 2.3. This method converges after

seconds in the heterogeneous setting. If σ 2 / ε is large, then Amelie SGD can be at least

times faster than Minibatch SGD . There are many more advanced methods (Lu and De Sa, 2021; Vogels et al., 2021; Even et al., 2024) that work in decentralized stochastic heterogeneous setting. Lu and De Sa (2021) developed similar lower and upper bounds, but there are at least three main differences: i) they derived an iteration complexity instead of a time complexity and assumed the performances of all workers are the same ii) the obtained lower bound holds only for one particular multigraph while our complexity holds for any multigraph iii) they assumed the smoothness of f i , while we consider the smoothness of f. The RelaySGD and Gradient Tracking methods by Vogels et al. (2021); Liu et al. (2024) wait for the slowest worker; thus, they depend on max i ∈ [ n ] h i in all regimes of σ 2 / ε , unlike our method. Even et al. (2024) consider the heterogeneous asynchronous setting, but their method assumes the similarity of the functions f i , which is not required in our method, and Amelie SGD converges even if there is no similarity of functions.

D Convex Functions in the Homogeneous and Heterogeneous Setups

We will be slightly more brief in the convex setting since the idea, the structure of time complexities, and the general approach do not change significantly. For instance, instead of the time complexity (11) that we get for the nonconvex case, in the nonsmooth convex case, we get (20). Thus, we will get Θ ( M 2 R 2 ε 2 min j ∈ [ n ] t ∗ ( σ 2 / M 2 , . . . ) ) instead of Θ ( L ∆ ε min j ∈ [ n ] t ∗ ( σ 2 / ε , . . . ) ) . The same idea applies to the smooth convex case. In the convex setting, we need the following assumptions.

D.1 Assumptions in convex world

Assumption 4. The function f is convex and attains a minimum at some point x ∗ ∈ R d .

Assumption 5. The function f is M -Lipschitz, i.e.,

for some M ∈ (0 , ∞ ] .

Assumption 6. For all x ∈ R d , stochastic (sub)gradients ∇ f ( x ; ξ ) are unbiased and are σ 2 -variancebounded, i.e., E ξ ∼D [ ∇ f ( x ; ξ )] ∈ ∂f ( x ) and E ξ ∼D [ ∥∇ f ( x ; ξ ) -E ξ ∼D [ ∇ f ( x ; ξ )] ∥ 2 ] ≤ σ 2 , where σ 2 ≥ 0 .

D.2 Homogeneous setup and nonsmooth case

Theorem 10. Let Assumptions 4, 5 and 6 hold. Choose any ε > 0 . Let us take the batch size S = max {⌈ σ 2 / M 2 ⌉ , 1 } , stepsize γ = ε M 2 + σ 2 /S ∈ [ ε 2 M 2 , ε M 2 ] , any pivot worker j ∗ ∈ [ n ] , and any spanning trees st and st bc in Algorithm 2. Then after K ≥ 2 M 2 R 2 / ε 2 iterations the method guarantees E [ f ( ̂ x K ) ] -f ( x ∗ ) ≤ ε, where ̂ x K = 1 K ∑ K -1 k =0 x k and R = ∥ ∥ x ∗ -x 0 ∥ ∥ .

Proof. The proof of Theorem 10 is almost the same as in Theorem 4 (see Section G). The proof of Theorem 4 states that the steps of Fragile SGD are equivalent to the classical SGD method. Thus, we can use the classical result from the literature (Lan, 2020). Using Theorem 22, we get

where we use the fact that S ≥ σ 2 / M 2 .

Theorem 11. Consider the assumptions and the parameters from Theorem 10. For any pivot worker j ∗ ∈ [ n ] and spanning trees st and st bc , Algorithm 2 converges after at most

seconds, where µ i → j ∗ ( µ j ∗ → i ) is an upper bound on times required to send a vector from worker i to worker j ∗ (from worker j ∗ to worker i ) along the spanning tree st (spanning tree st bc ).

Proof. The proof is identical to the proof of Theorem 5.

Corollary 3. Consider the assumptions and the parameters from Theorem 11. Let us take a pivot worker j ∗ = arg min j ∈ [ n ] t ∗ ( σ 2 / M 2 , [ h i ] n i =1 , [ τ i → j + τ j → i ] n i =1 ) , and a spanning tree st (spanning tree st bc ) that connects every worker i to worker j ∗ (worker j ∗ to every worker i ) with the shortest distance τ i → j ∗ ( τ j ∗ → i ). Then Algorithm 2 converges after at most

seconds.

D.3 Homogeneous setup and smooth case

In the homogeneous and smooth case, we will slightly modify Fragile SGD . Instead of Line 8 of Algorithm 3, we use the following steps:

We will call such a method the Accelerated Fragile SGD method. The idea is to use the acceleration technique from (Lan, 2020) (which is based on (Nesterov, 1983)).

if for the stepsize

Theorem 12. Let Assumptions 4, 1 and 3 hold. Choose any ε > 0 . Let us take the batch size S = max {⌈ ( σ 2 R ) / ( ε 3 / 2 √ L ) ⌉ , 1 } , γ = min { 1 4 L , [ 3 R 2 S 4 σ 2 ( K +1)( K +2) 2 ] 1 / 2 } , any pivot worker j ∗ , and any spanning trees st and st bc in Accelerated Method 2 ( Accelerated Fragile SGD ), then after K ≥ 8 √ LR √ ε iterations the method guarantees that E [ f ( x K ) ] -f ( x ∗ ) ≤ ε, where R = ∥ ∥ x ∗ -x 0 ∥ ∥ .

Proof. Using the same reasoning as in Theorem 4, Accelerated Fragile SGD is just the classical accelerated stochastic gradient method with a batch size greater or equal to S . We can use Proposition 4.4 from Lan (2020). For the stepsize

if

where we use the choice of S.

Theorem 13. Consider the assumptions and the parameters from Theorem 12. For any pivot worker j ∗ ∈ [ n ] and spanning trees st and st bc , Accelerated Algorithm 2 ( Accelerated Fragile SGD ) converges after at most

seconds, where µ i → j ∗ ( µ j ∗ → i ) is an upper bound on times required to send a vector from worker i to worker j ∗ (from worker j ∗ to worker i ) along the spanning tree st (spanning tree st bc ).

Proof. The proof is identical to the proof of Theorem 5.

Corollary 4. Consider the assumptions and the parameters from Theorem 13. Let us take a pivot worker j ∗ = arg min j ∈ [ n ] t ∗ ( σ 2 R ε 3 / 2 √ L , [ h i ] n i =1 , [ τ i → j + τ j → i ] n i =1 ) , and a spanning tree st (spanning tree st bc ) that connects every worker i to worker j ∗ (worker j ∗ to every worker i ) with the shortest distance τ i → j ∗ ( τ j ∗ → i ). Then Accelerated Algorithm 2 ( Accelerated Fragile SGD ) converges after at most

seconds.

D.4 Heterogeneous setup and nonsmooth case

Consider the optimization problem (17).

Theorem 14. Let Assumptions 4, 5 hold for the function f and Assumption 6 holds for the functions f i for all i ∈ [ n ] . Choose any ε > 0 . Let us take the batch size S = max {⌈ σ 2 / M 2 ⌉ , n } , γ = ε M 2 + σ 2 /S ∈ [ ε 2 M 2 , ε M 2 ] , any pivot worker j ∗ ∈ [ n ] , and any spanning trees st and st bc , in Algorithm 5, then after K ≥ 2 M 2 R 2 / ε 2 iterations the method guarantees that E [ f ( ̂ x K ) ] -f ( x ∗ ) ≤ ε, where ̂ x K = 1 K ∑ K -1 k =0 x k and R = ∥ ∥ x ∗ -x 0 ∥ ∥ .

we have

Therefore,

Proof. The proof of Theorem 14 is almost the same as in Theorems 8 (see Section H). The proof of Theorem 8 states that the steps of Amelie SGD are equivalent to the classical SGD method. Thus, we can use the classical result from the literature (Lan, 2020). Using Theorem 22, we get if

for the stepsize

where we use the fact that S ≥ σ 2 / M 2

Theorem 15. Consider the assumptions and the parameters from Theorem 14. For any pivot worker j ∗ ∈ [ n ] and any spanning trees st and st bc , Algorithm 5 converges after at most

seconds, where µ k i → j ∗ ( µ k j ∗ → i ) is an upper bound on times required to send a vector from worker i to worker j ∗ (worker j ∗ to worker i ) along the spanning tree st (spanning tree st bc ) for all i ∈ [ n ] .

Proof. The proof is identical to the proof of Theorem 9.

Corollary 5. Consider the assumptions and the parameters from Theorem 15. Let us take any pivot worker j ∗ ∈ [ n ] and a spanning tree st (spanning tree st bc ) that connects every worker i to worker j ∗ (worker j ∗ to worker i ) with the shortest distance τ i → j ∗ ( τ j ∗ → i ), then Algorithm 5 converges after at most

seconds.

D.5 Heterogeneous setup and smooth case

Consider the optimization problem (17). In this section, we use the same idea as in Section D.3. We will modify Amelie SGD and, instead of Line 9 from Algorithm 6, we use the lines (21). We call such a method the Accelerated Amelie SGD method.

Theorem 16. Let Assumptions 4 and 1 hold for the function f and Assumption 3 holds for the functions f i . Choose any ε > 0 . Let us take the batch size S = max {⌈ ( σ 2 R ) / ( ε 3 / 2 √ L ) ⌉ , n } ,

in Accelerated Method 5 ( Accelerated Amelie SGD ), then after K ≥ 8 LR √ ε iterations the method guarantees that E [ f ( x K ) ] -f ( x ∗ ) ≤ ε, where R = ∥ ∥ x ∗ -x 0 ∥ ∥ .

Proof. Accelerated Amelie SGD is equivalent to the accelerated stochastic gradient method with a mini-batch from Lan (2020). The proof repeats the proofs of Theorem 12 and Theorem 8.

Theorem 17. Consider the assumptions and the parameters from Theorem 16. For any pivot worker j ∗ ∈ [ n ] and any spanning trees st and st bc , Accelerated Algorithm 5 ( Accelerated Amelie SGD ) converges after at most

seconds, where µ k i → j ∗ ( µ k j ∗ → i ) is an upper bound on times required to send a vector from worker i to worker j ∗ (worker j ∗ to worker i ) along the spanning tree st (spanning tree st bc ) for all i ∈ [ n ] .

Proof. The proof is identical to the proof of Theorem 9.

Corollary 6. Consider the assumptions and the parameters from Theorem 17. Let us take any pivot worker j ∗ ∈ [ n ] and a spanning tree st (spanning tree st bc ) that connects every worker i to worker j ∗ (worker j ∗ to worker i ) with the shortest distance τ i → j ∗ ( τ j ∗ → i ), then Accelerated Algorithm 5 ( Accelerated Amelie SGD ) converges after at most

seconds.

D.6 On lower bounds

In previous subsections, we provide upper bounds on the time complexities for different classes of convex functions and optimization problems. Using the same reasoning as in Section 4 and (Tyurin and Richtárik, 2023)[Section B], we conjecture that the obtained upper bounds are tight and optimal (up to log factors in the homogeneous case).

D.7 Previous works

Let us consider the time complexities (20) and (23) (accelerated rate) in the homogeneous and convex cases. When τ i → j = 0 for all i, j ∈ [ n ] , Even et al. (2024) recovers the time complexity (nonaccelerated in the smooth case) of Asynchronous SGD (Mishchenko et al., 2022; Koloskova et al., 2022) which is suboptimal (Tyurin and Richtárik, 2023). When the communication is free, we recover the time complexity (accelerated in the smooth case) of Accelerated Rennala SGD from (Tyurin and Richtárik, 2023), which is optimal if τ i → j = 0 for all i, j ∈ [ n ] .

In the heterogeneous and convex setting, (26) improves the result from (Even et al., 2024) since (26) is an accelerated rate and does not depend on a quantity that measures the similarity of the functions f i . When σ = 0 , the result (26) is consistent with the lower bound from (Scaman et al., 2017). However, when σ > 0 , as far as we know, the time complexity (26) is new in the convex smooth setting.

E Lower Bound: Diving Deeper into the Construction

Protocol 8 Time Multiple Oracles Protocol with Communication

̸

1: Input: function f (or functions f i ) computation oracles { O i } n i =1 ∈ O ( f ) communication oracles { C p i → j } i ∈ [ n ] ,j ∈ [ n ] ,p ≥ 1 , algorithm { ( M k , L k , D k , P k 1 , . . . , P k n , V k 1 , . . . , V k n ) } ∞ k =0 ∈ A 2: s h, 0 i = s τ, 0 i,j,p = 0 for all i, j ∈ [ n ] , p ∈ N 3: for k = 0 , . . . , ∞ do 4: ( t k +1 , c k +1 ) = M k ( g 1 1 , . . . g 1 n , g 2 1 , . . . , g 2 n , . . . , g k 1 , . . . , g k n ) ▷ t k +1 ≥ t k 5: if c k +1 = 0 then 6: i k +1 = L k ( g 1 1 , . . . g 1 n , g 2 1 , . . . , g 2 n , . . . , g k 1 , . . . , g k n ) 7: x k i k +1 = P k i k +1 ( g 1 i k +1 , . . . , g k i k +1 ) 8: ( s h,k +1 i k +1 , g k +1 i k +1 ) = O i k +1 ( t k +1 , x k i k +1 , s h,k i k +1 ) ▷ ∀ j = i k +1 : s h,k +1 j = s h,k j , g k +1 j = 0 9: else 10: i k +1 , j k +1 , p k +1 = D k ( g 1 1 , . . . g 1 n , g 2 1 , . . . , g 2 n , . . . , g k 1 , . . . , g k n ) 11: v k i k +1 = V k i k +1 ( g 1 i k +1 , . . . , g k i k +1 ) 12: ( s τ,k +1 i k +1 ,j k +1 ,p k +1 , g k +1 j k +1 ) = C p k +1 i k +1 → j k +1 ( t k +1 , v k i k +1 , s τ,k i k +1 ,j k +1 ,p k +1 ) ▷ ∀ j = j k +1 : g k +1 j = 0 , ∀ i = i k +1 , j = j k +1 , p = p k +1 : s τ,k +1 i,j,p = s τ,k i,j,p , 13: end if 14: end for

̸

̸

̸

̸

In Section 4, we present a brief overview and simplified theorem for the lower bound. We now provide a formal and strict mathematical construction.

E.1 Description of Protocols 8 and 1

One way how we can formalize Protocol 1 is to use Protocol 8. Let us explain it. Using Protocol 8, we consider any possible method that works in our distributed asynchronous setting. The mapping M k of the algorithm returns the time t k +1 (ignore for now) and c k +1 . If c k +1 = 0 , then the algorithm decides to start the calculation of a stochastic gradient: it determines the index i k +1 of a worker that will start the calculation, then, using all locally available information g 1 i k +1 , . . . , g k i k +1 , it calculates a new point x k i k +1 , and passes this point to the computation oracle O i k +1 that will return a new stochastic gradient after h i k +1 seconds. If c k +1 = 1 , then the algorithm decides to communicate a vector: it returns the indices i k +1 and j k +1 of two workers that will communicate, and the index p k +1 ∈ N of a communication oracle, calculates v k i k +1 in worker i k +1 using only all locally available information g 1 i k +1 , . . . , g k i k +1 , and passes it to the communication oracle C p k +1 i k +1 → j k +1 that will send the vector after τ k +1 k +1 seconds.

i → j

The computation oracles we define as

where ξ ∼ D .

The communication oracles we define as

The idea is that the computation oracle (27) emulates the behavior of a real worker i that requires h i seconds to calculate a stochastic gradient. The communication oracle emulates the behavior of a real communication channel that requires τ i → j seconds to send a vector from worker i to worker j.

̸

One can see that, for all i = j ∈ [ n ] , an algorithm can access an infinite number of communication oracles C 1 i → j , C 2 i → j , . . . . We allow an algorithm to send as many vectors from worker i to worker j in parallel as it wants.

We now discuss the role of t k +1 . One can see that the time t k +1 is returned by the algorithm, and t k +1 is passed to the oracles O i k +1 and C p k +1 i k +1 → j k +1 . Consider that O i k +1 was called with t k +1 for the first time, then it will return zero vector because ( s h,k i k +1 ) q = 0 (see (27)) at the beginning. Then, by the construction of (27), the oracle will return a non-zero vector in the second output if only the algorithm returns a time that is greater or equal to t k +1 + h i k +1 . The same idea applies to C p k +1 i k +1 → j k +1 : if it was called t k +1 for the first time, then worker j k +1 will get a non-zero vector only if the algorithm passes a time that is greater or equal to t k +1 + τ i k +1 → j k +1 .

The oracles force the algorithm to increase the time t k +1 ; otherwise, it will not get new information ( ∇ f ( s x ; ξ ) in (27)) about the function. One crucial constraint is t k +1 ≥ t k , meaning that the algorithm can not 'cheat' and travel into the past. The idea of such a protocol was proposed in Tyurin and Richtárik (2023), and we refer to Sections 3-5 for a detailed description. A more readable and less formal version of Protocol 8 is presented in the Protocol 1.

E.2 Lower bound

Before we state the main theorem, let us define the class of zero-respecting algorithms.

Definition 18 (Algorithm Class A zr ) . Let us consider Protocol 8. We say that the sequence of tuples of mappings { ( M k , L k , D k , P k 1 , . . . , P k n , V k 1 , . . . , V k n ) } ∞ k =0 is a zero-respecting algorithm, if

̸

The set of all algorithms with this properties we define as A zr .

Constraints 1-5 define domains of the mappings. Constraint 6 is required to ensure that the time sequence t k does not decrease. Constraint 7 is a standard assumption that an algorithm is zerorespecting (Arjevani et al., 2022).

Theorem 19. Consider Protocol 8. We take any h i ≥ 0 and τ i → j ≥ 0 for all i, j ∈ [ n ] such that τ i → j ≤ τ i → k + τ k → j for all i, k, j ∈ [ n ] . We fix L, ∆ , ε, σ 2 > 0 that satisfy the inequality ε < c ′ L ∆ . For any algorithm A ∈ A zr , there exists a function f, which satisfy Assumptions 1, 2 and f (0) -f ∗ ≤ ∆ , and a stochastic gradient mapping ∇ f ( · ; · ) , which satisfy Assumption 3, such that E [ inf k ∈ S t ,i ∈ [ n ] ∥ ∥ ∇ f ( x k i ) ∥ ∥ 2 ] > ε, where S t := { k ∈ N 0 | t k ≤ t } ,

where π j, · is a permutation that sorts max { h i , τ i → j } , i.e.,

for all j ∈ [ n ] . The quantities c ′ and c are universal constants. The sequences x k and t k are defined in Protocol 8.

E.3 Proof sketch of Theorem 19

Let us provide a proof sketch that will give intuition behind the theorem. The full proof starts in Section E.4.

Part 1: Construct a function and stochastic gradient

The first part of the proof is standard (Carmon et al., 2020; Arjevani et al., 2022; Tyurin and Richtárik, 2023; Huang et al., 2022; Lu and De Sa, 2021), and we delegate it to Section E.4. For any algorithm, we construct oracles and a 'worst-case' function such that

̸

where prog ( x ) := max { i ≥ 0 | x i = 0 } ( x 0 ≡ 1) and T ≈ L ∆ / ε . This inequality says that while all points in Protocol 8 have the last coordinate equals 0 by the time t, an algorithm can not find an ε -stationary point.

Since we assume that A ∈ A zr is zero-respecting, the only way to discover the next non-zero coordinate is through stochastic gradients. They are constructed in the following way (Arjevani et al., 2022):

and ξ ∼ Bernoulli ( p ) for all i ∈ [ n ] , where p ≈ ε / σ 2 . We denote [ x ] j as the j th index of a vector x ∈ R T . The stochastic gradient equals to the exact gradient except for the last non-zero coordinate: it zeros out it with the high probability 1 -p.

Part 2: The Level Game

In essence, the protocol is equivalent to the following collaborative game. Each worker i starts with level ℓ i = 0 . The goal is to reach level T with at least one worker as fast as possible. There are two ways how a worker can increase its level: i) worker i flips one coin per time from ξ ∼ Bernoulli ( p ) , it takes h i seconds to flip one coin, and if the worker is lucky, i.e., ξ = 1 , then it moves to the next level ℓ i = ℓ i +1 ; ii) worker i can share its level with another worker j, and it takes τ i → j seconds (we are allowed to run this operation again even if the previous is not finished). Both options can be executed in parallel. What is the minimum possible time to reach the game's goal?

Since all workers have the levels equal to 0 at the beginning, they should flip coins in parallel and wait for the moment when at least one worker moves to level 1 . We define η 1 i as the number of flips in worker i to get ξ = 1 . Clearly, η 1 i ∼ Geometric ( p ) are i.i.d. geometric random variables with the probability p . With any strategy, it is necessary to wait at least

seconds to reach level 1 in worker j because once worker i flips ξ = 1 , it will take at least τ i → j seconds to share level 1 to worker j due to the triangle inequality τ i → j ≤ τ i → k + τ k → j for all i, j, k ∈ [ n ] , and we should minimize h i η 1 i + τ i → j over all workers. We now use mathematical induction to prove that it is necessary to wait at least

seconds to reach level T, where we define y 0 i := 0 for all i ∈ [ n ] and { η T i } are i.i.d. random variables from Geometric ( p ) . The base case is proven above. Assume that it is true for 1 , . . . , T -1 , then worker i requires at least y T -1 i + h i η T i seconds to flip a coin that moves to level T, it takes at least τ i → j seconds to share the level T, so it is necessary to wait at least min i ∈ [ n ] { y T -1 i + h i η T i + τ i → j } seconds in worker i to get level T. Ultimately, the minimum possible time to reach the game's goal is

From the previous result, we can conclude that if t ≤ 1 2 y T , then

Let us fix any determenistic value ¯ y ∈ R and take

for any algorithm A ∈ A zr .

Using (30), we have

Thus, it is sufficient to find any ¯ y ∈ R such that

The sequence y T is a well-define time series. In Lemma 2, we show that (32) holds with

π j, · is a permutation that sorts max { h i , τ i → j } , i.e.,

for all j ∈ [ n ] . We substitute this ¯ y to (31) and get the result of theorem.

E.4 Full proof of Theorem 19

This section considers the standard 'worst case' function that helps to provide lower bounds in the nonconvex world. Let us define

̸

For any T ∈ N , Carmon et al. (2020); Arjevani et al. (2022) define

where

We will only rely on the following facts.

Lemma 1 (Carmon et al. (2020); Arjevani et al. (2022)) . The function F T satisfies:

The function F T is l 1 -smooth, where l 1 = 152 .

For all x ∈ R T , prog ( ∇ F T ( x )) ≤ prog ( x ) + 1 .

For all x ∈ R T , if prog ( x ) < T, then ∥∇ F T ( x ) ∥ > 1 .

Theorem 19. Consider Protocol 8. We take any h i ≥ 0 and τ i → j ≥ 0 for all i, j ∈ [ n ] such that τ i → j ≤ τ i → k + τ k → j for all i, k, j ∈ [ n ] . We fix L, ∆ , ε, σ 2 > 0 that satisfy the inequality ε < c ′ L ∆ . For any algorithm A ∈ A zr , there exists a function f, which satisfy Assumptions 1, 2 and f (0) -f ∗ ≤ ∆ , and a stochastic gradient mapping ∇ f ( · ; · ) , which satisfy Assumption 3, such that

where π j, · is a permutation that sorts max { h i , τ i → j } , i.e.,

for all j ∈ [ n ] . The quantities c ′ and c are universal constants. The sequences x k and t k are defined in Protocol 8.

Proof.

Part 1: The Worst Case Function

This part of the proof mirrors the proofs from Carmon et al. (2020); Arjevani et al. (2022); Tyurin and Richtárik (2023); Huang et al. (2022); Lu and De Sa (2021). We provide it for completeness. The goal of this part to construct a hard instance.

We fix λ > 0 , T ∈ N and take the function f ( x ) := Lλ 2 l 1 F T ( x λ ) , where the function F T is defined in Section E.4. Note that the function f is L -smooth:

where l 1 -smoothness of F T (Lemma 1). Let us take

then

We showed that the function f satisfy Assumptions 1, 2 and f (0) -f ∗ ≤ ∆ .

For each worker i we take an oracle O i , from (27) with the mapping g such that

and D i = Bernouilli ( p ) for all i ∈ [ n ] , where p ∈ (0 , 1] . We denote [ x ] j as the j th index of a vector x ∈ R T . This stochastic gradient is unbiased and σ 2 -variance-bounded. We have for all i ∈ [ T ] , and

because the difference is non-zero only in one coordinate. Thus

where we use Lemma 1 and take

Let us take to ensure that

T

for all x ∈ R . From Lemma 1, we know that if prog ( x ) < T, then ∥∇ F T ( x ) ∥ > 1 . Thus, we get ∥∇ f ( x ) ∥ 2 > 2 ε ✶ [ prog ( x ) < T ] (34) Therefore, and

The inequality (34) implies where { x k i } ∞ k =0 are sequences from Protocol 8.

Part 2: Reduction to Lower Bound Time Series

We now focus on (27). In (27), when the oracle in worker i calculates a new stochastic gradient, it

samples a random variable ξ ∼ D . This is equivalent to the procedure if we had T infinite sequences of i.i.d. Bernoulli random variables

for all i ∈ [ n ] and the oracle would look at the progress of s x and take the next non-taken Bernoulli random variable in the sequence that corresponds to that progress. For instance, if prog ( s x ) = j for the first time in worker i , then the oracle will apply ξ j, 1 i in (27). The next time when it gets prog ( s x ) = j it will apply ξ j, 2 i and so on. One by one, we get i.i.d. Bernoulli random variables.

Let us define for

where

for all i ∈ [ n ] and k ∈ [ T ] . This is the first Bernoulli random variable from the sequence ξ k, 1 i , ξ k, 2 i , . . . that equals 1 . The random variables { η k i } are i.i.d. geometrically distributed random variables with the probability p.

The next steps mirror Proof Sketch from Theorem 19. The oracles constructed in such a way that it takes h i seconds to calculate a stochastic gradient, and at least τ i → j seconds to send a vector from one worker to another. Using the same reasoning as in the Level Game in Proof Sketch of Theorem 19, the first time moment when worker j can get a vector with the first non-zero coordinate greater or equal

because, at the beginning, each worker i calculates stochastic gradients with prog ( s x ) = 0 and should wait at least h i η 1 i seconds to get a stochastic gradient with the progress equals 1 . Then, it can share this vector with any other worker j , but it takes at least τ i → j seconds.

As in Proof Sketch of Theorem 19, we can use mathematical induction to prove that it is necessary to wait at least

seconds to get a point such that prog ( s x ) = T. The base case for y 1 j has been proven. Worker i requires at least y T -1 i + h i η T i seconds to wait for the moment when the corresponding oracle will return a stochastic gradient with progress T because y T -1 i is the first time possible time to get a vector with progress T -1 by the induction, and it will take at least additional h i η T i seconds to calculate η T i vectors with prog ( s x ) = T -1 in (27). Also, it takes at least τ i → j seconds to share a vector, so it is necessary to wait at least min i ∈ [ n ] { y T -1 i + h i η T i + τ i → j } seconds in worker i to get the first vector with progress T.

In the end, the fastest possible time to get a vector with progress T is at least

Part 3: Reduction to the concentration of y T

The last statement means that prog ( x k i ) < T for all i ∈ [ n ] and k such that t k ≤ 1 2 y T . Therefore, we obtain

and η j i ∼ Geometric ( p ) with

where c T = 3648 -1 and c p = 1058 are universal constants. Let us fix any determenistic value ¯ y ∈ R and take

Using (38), we have

Thus, it is sufficient to find any ¯ y ∈ R such that

In the following lemma we use the notation of this theorem. We prove it in Section E.5.

we have P ( y T ≤ ¯ y ) ≤ 1 2 , where π j, · is a permutation that sorts max { h i , τ i → j } , i.e.,

for all j ∈ [ n ] . The quantity c is a universal constant.

Using Lemma 2, we can conclude that (40) holds and

for

E.5 Proof of Lemma 2

we have P ( y T ≤ ¯ y ) ≤ 1 2 , where π j, · is a permutation that sorts max { h i , τ i → j } , i.e.,

for all j ∈ [ n ] . The quantity c is a universal constant.

Proof. Using the Chernoff method for any s > 0 , we get

We have a maximum operation that complicates the analysis. In response to this problem, we use a well-known trick that bounds a maximum by a sum.

We refer to (Van Handel, 2014)[Part II] for the explanation why it can be (almost) tight. This is the main reason why we get an extra log n factor in (41). Let us consider the last exponent separately and use the same trick again:

Next, we get

In the last equality we use the independence. We now bound E [ exp ( -sy k -1 i )] by max i ∈ [ n ] E [ exp ( -sy k -1 i )] and get

Let us fix any s j > 0 for all j ∈ [ n ] and take s = max j ∈ [ n ] s j . Then

Let us fix ¯ t j > 0 and consider

Then, we can use the following inequalities:

Since η k i ∼ Geometric ( p ) , we get

Then

For all j ∈ [ n ] , we take any permutation π j, · that sorts max { h i , τ i → j } , i.e.,

We have

Recall that ¯ t j > 0 is a parameter. In the following technical lemma, we choose ¯ t j and show that the second term in (45) is 'small.' We prove it in Section E.6.

Lemma 3. For any n ≥ 1 , h i ≥ 0 ,τ i → j ≥ 0 for all i, j ∈ [ n ] , and p ∈ (0 , 1] , we have for all j ∈ [ n ] , where

and π j, · is a permutation that sorts max { h i , τ i → j } , i.e.,

for all j ∈ [ n ] .

Using Lemma 3 and (45), we get for all j ∈ [ n ] . Let us take

to get

for all j ∈ [ n ] . Using (44) and (43), we obtain

for all j ∈ [ n ] . We can conclude that

where use the same reasoning in the recursion and y 0 j = 0 for all j ∈ [ n ] . We substitute the inequality to (42):

It is sufficient to take k = T and any

to get for all j ∈ [ n ] .

Let us define k ∈ [ n ] as the largest index that minimizes (46). (Part 1): bound ¯ t j by max { τ π j,k ∗ +1 → j , h π j,k ∗ +1 } Let us consider the case k ∗ < n. We have two options:

∗

1. Let max { τ π j,k ∗ → j , h π j,k ∗ } ≥ ( ∑ k ∗ i =1 p h π j,i ) -1 , then

since max { τ π j,i → j , h π j,i } are sorted. Then, we get

where we use the definitions s = max i ∈ [ n ] s i , (46) and (47). Recall the choice of T and p in (35) and (36): T = ⌊ c T · ∆ L ε ⌋ and p = min { c p · ε σ 2 , 1 } for some universal constants c T and c p . Since we have the condition ε < c ′ L ∆ for some universal constant c ′ in the conditions of Theorem 19, we can conclude that we can take

where c is a universal constant.

E.6 Proof of Lemma 3

Lemma 3. For any n ≥ 1 , h i ≥ 0 ,τ i → j ≥ 0 for all i, j ∈ [ n ] , and p ∈ (0 , 1] , we have for all j ∈ [ n ] , where

and π j, · is a permutation that sorts max { h i , τ i → j } , i.e.,

The first inequality follows from the fact that k ∗ is the largest minimizer.

2. Let max { τ π j,k ∗ → j , h π j,k ∗ } < ( ∑ k ∗ i =1 p h π j,i ) -1 , then it is not possible that max { τ π j,k ∗ +1 → j , h π j,k ∗ +1 } < ( ∑ k ∗ +1 i =1 p h π j,i ) -1 because it would yield the inequality

The last inequality contradicts the fact that k ∗ is the largest minimizer. Thus, if k ∗ < n and max { τ π j,k ∗ → j , h π j,k ∗ } < ( ∑ k ∗ i =1 p h π j,i ) -1 , then

In total, we have max

if k ∗ < n. Using this inequality, we get for any k ∗ .

(Part 2)

We have three options:

k

∗

-

1

1. If

2. If

p

)

i

=1

( ∑

h

π j,i

{

τ

π

j,k

∗

→

j

, h

π

j,k

∗

}

,

then, using (50) and

⌊

x

x

for all

x

0

, we get

k

i

=1

p

h

π j,i

<

{

τ

π

j,k

∗

→

j

, h

π

j,k

∗

}

and

k

= 1

,

then

( ∑

because

3. If ( ∑ k ∗ i =1 p h π j,i ) -1 < max { τ π j,k ∗ → j , h π j,k ∗ } and k ∗ > 1 , then ¯ t j = 1 8 max { τ π j,k ∗ → j , h π j,k ∗ } and

for all i ≤ k ∗ such that max { τ π j,i → j , h π j,i } = max { τ π j,k ∗ → j , h π j,k ∗ } . If this equality holds for all i ≤ k ∗ , then

∗

Otherwise, there exists ℓ < k ∗ such that max { τ π j,ℓ → j , h π j,ℓ } < max { τ π j,k ∗ → j , h π j,k ∗ } and

It is not possible that max { τ π j,ℓ → j , h π j,ℓ } ≥ ( ∑ ℓ i =1 p h π j,i ) -1 because it would yield the inequality

that contradicts the fact that ¯ t j is the minimum (see (46)). Thus, we have

and

In total, we have for ¯ t j from (46).

F Lower Bound in the Heterogeneous Setup

In this case, we consider the following oracle mappings. For all i ∈ [ n ] , we define such that

where ξ ∼ D . Unlike (27), the mapping (51) returns ∇ f i ( s x ; ξ ) .

Theorem 20. Consider Protocol 8 with the mappings (51) . We take any h i ≥ 0 and τ i → j ≥ 0 for all i, j ∈ [ n ] such that τ i → j ≤ τ i → k + τ k → j for all i, k, j ∈ [ n ] . We fix L, ∆ , ε, σ 2 > 0 that satisfy the inequality ε < c 1 L ∆ . For any algorithm A ∈ A zr , there exists a function f = 1 n ∑ n i =1 f i , which satisfy Assumptions 1, 2 and f (0) -f ∗ ≤ ∆ , and stochastic gradient mappings ∇ f i ( · ; · ) , which satisfy Assumption 3 ( E ξ [ ∇ f i ( x ; ξ )] = ∇ f i ( x ) and E ξ [ ∥∇ f i ( x ; ξ ) -∇ f i ( x ) ∥ 2 ] ≤ σ 2 ), such that E [ inf k ∈ S t ,i ∈ [ n ] ∥ ∥ ∇ f ( x k i ) ∥ ∥ 2 ] > ε, where S t := { k ∈ N 0 | t k ≤ t } ,

The quantities c 1 and c 2 are universal constants. The sequences x k and t k are defined in Protocol 8.

Proof. The last two terms in the max follow from Theorem A.2 by Tyurin and Richtárik (2023), who considered the same setup but with τ i → j = 0 for all i, j ∈ [ n ] . It is left to prove the first term. Let us fix λ > 0 . Let us take any pair ( ¯ i, ¯ j ) of workers such that max i,j ∈ [ n ] τ i → j = τ ¯ i → ¯ j . Next, we split the blocks of the function F T ( x ) from (33) and define two new functions:

and

We consider the following functions f i :

Then, we get

Let us show that the function f is L -smooth:

Let us take then

We showed that the function f satisfy Assumptions 1, 2 and f (0) -f ∗ ≤ ∆ .

In the oracles O i , we simply take the non-stochastic mappings ∇ f i ( x ; ξ ) := ∇ f i ( x ) that are unbiased and 0 -variance-bounded.

We take to ensure that

for all x ∈ R T such that prog ( x ) < T. In the last inequality, we use Lemma 1. Thus

Only workers ¯ i and ¯ j contain the information about the function f. The function f is a zero-chain function, and we split it between workers ¯ i and ¯ j. Due to this splitting, workers ¯ i and ¯ j have to communicate to find the next non-zero coordinate. Only worker ¯ i can get a non-zero value in the first coordinate through the gradient of F T, 1 . Next, this worker can not get a non-zero value in the second coordinate due to the construction of (52). Thus, it has to pass a vector with a non-zero value in the first coordinate to worker ¯ j because only this worker can get a non-zero value in the second coordinate. This communication takes at least τ ¯ i → ¯ j seconds. Using the same reasoning, worker ¯ j has to send a vector to worker ¯ i once worker ¯ j has discovered a non-zero value in the second coordinate.

̸

̸

An algorithm has to repeat such communications at least T -1 2 times to find a vector x ∈ R T such that prog ( x ) = T.

Thus, we get for

G Proof of the Time Complexity for Homogeneous Case

Theorem 4. Let Assumptions 1, 2, and 3 hold. We take γ = 1 / 2 L, batch size S = max { ⌈ σ 2 / ε ⌉ , 1 } , any pivot worker j ∗ ∈ [ n ] , and any spanning trees st and st bc in Algorithm 2. For all K ≥ 16 L ∆ /ε, we get 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

Proof. Algorithm 2 produces the sequence x k such as x k +1 = x k -γ s k g k = x k -γ ¯ g k , where ¯ g k := 1 s k g k . By the design of the algorithm,

where the ξ i are independent random samples and s k ≥ S. We do not dismiss the possibility that the computation and communication times are random, so s k can be random. Assume V k is a σ -algebra generated by all computation and communication times and g 0 , . . . , g k -1 , then s k is V k -measurable. Using the independence and Assumption 3, we have

where G k is a σ -algebra generated by ¯ g 0 , . . . , ¯ g k -1 . We can use a well-known SGD result (Ghadimi and Lan, 2013; Khaled and Richtárik, 2022). Using Theorem 21, for the stepsize

we have

if

Using the choice of S, we get that Algorithm 2 converges after

Theorem 5. Consider the assumptions and the parameters from Theorem 4. For any pivot worker j ∗ ∈ [ n ] and spanning trees st and st bc , Algorithm 2 converges after at most

seconds, where µ i → j ∗ ( µ j ∗ → i ) is an upper bound on the times required to send a vector from worker i to worker j ∗ (from worker j ∗ to worker i ) along the spanning tree st (spanning tree st bc ).

Proof. Due to Theorem 4, we know that Algorithm 3 finds an ε -stationary point after at most K = Θ ( L ∆ ε ) iterations. It is left to bound the time of one iteration to prove the theorem.

For all j ∈ [ n ] , we define π j, · as a permutation that sorts { max { µ i → j + µ j → i , h i }} n i =1 as

Let us define the index

and the set steps with

that represents a set of the 'fastest' workers that can potentially contribute to an optimization process. We take

In the following steps of the proof we show that every iteration takes at most

¯ t ︸︷︷︸ (Step 1): Calculate enough stochastic gradients + ¯ t ︸︷︷︸ (Step 2): Send stochastic gradients to Process j ∗ + ¯ t ︸︷︷︸ (Step 3): Broadcast a new point = 3 ¯ t seconds.

(Step 1): Since it takes at most h i seconds to calculate a stochastic gradient in worker i, all workers from the set A ∗ will calculate at least

stochastic gradients after ¯ t seconds at the point x k . We have

for all i ≤ k ∗ by the definition of the permutations π · , · . Therefore,

for all i ≤ k ∗ . Thus, using (54) and ⌊ x ⌋ ≥ x 2 for all x ≥ 1 , we get

Therefore, after ¯ t seconds the algorithm will calculate at least S stochastic gradients at the point x k using the workers A ∗ .

(Step 2): By the design of Algorithm 4, once a stochastic gradient ∇ f ( x k ; ¯ ξ ) is calculated, it is added to g k i, next . Then, g k i, next is assigned to g k i, send which is sent to Process next st,j ∗ ( i ) . Finally, Process next st,j ∗ ( i ) receives g k i, send and adds it to g k ( next st,j ∗ ( i )) , next . Thus, the stochastic gradient ∇ f ( x k ; ¯ ξ ) is presented in the sum of g k ( next st,j ∗ ( i )) , next of Process next st,j ∗ ( i ) . At some point, Process 0 in worker j ∗ will receive a vector g k · , send where the stochastic gradient ∇ f ( x k ; ¯ ξ ) is presented.

Let us bound the time required to transmit a stochastic gradient to Process 0 of worker j ∗ . Once a stochastic gradient ∇ f ( x k ; ¯ ξ ) is calculated in Process i from A ∗ , it is added to a vector g k i, next in Process i. It will take at most 2 ρ i → next st,j ∗ ( i ) seconds to transmit it to Process next st,j ∗ ( i ) because it takes at most ρ i → next st,j ∗ ( i ) seconds to wait for the transmission of a message g k i, send where the stochastic gradient ∇ f ( x k ; ¯ ξ ) is not presented, and an additional ρ i → next st,j ∗ ( i ) seconds to send the next g k i, send where it will present. After that, Process next st,j ∗ ( i ) will receive g k i, send , where the stochastic gradient ∇ f ( x k ; ¯ ξ ) presents, and add the vector g k i, send to g k ( next st,j ∗ ( i )) , next . Then, it will take at most 2 ρ next st,j ∗ ( i ) → next ( next st,j ∗ ( i )) seconds to send a vector, where the stochastic gradient ∇ f ( x k ; ¯ ξ ) presents, to Process next ( next st,j ∗ ( i )) and so forth. In total, after a finite number of such steps a stochastic gradient calculated in Process i will be transmitted to Process 0 of worker j ∗ . From Definition 3 of next st,j ∗ , we can conclude that the vector ∇ f ( x k ; ¯ ξ ) will be transmitted through the path between workers i and j ∗ in the spanning tree st. Thus, it will take at most 2 µ i → j ∗ seconds by the definition of µ i → j ∗ .

Using (55) and the definition of A ∗ , we have

for all i ∈ A ∗ . Therefore, it will take at most ¯ t seconds to calculate at least S stochastic gradients, and at most ¯ t seconds to send all these stochastic gradients to Process 0.

(Step 3): It is left to estimate the time of the broadcast steps (Lines 7-9 in Algorithm 4) through the spanning tree st bc . By the definition of µ j ∗ → i , the time required to broadcast x k to Process i through the spanning tree st bc is less or equal to 2 µ j ∗ → i since, in all edges from j ∗ to i, workers wait at most ρ ·→· seconds while edges are blocked by previous communications, and additional ρ ·→· seconds to send x k to next workers. Using (55) and the definition of A ∗ , we have

Thus, every worker from A ∗ will get x k after ¯ t seconds. By combining all times, we can conclude that every iteration in Algorithm 3 will take at most ¯ t + ¯ t + ¯ t = 3 ¯ t seconds.

It left to show that

¯ t = O   max    max { µ π j ∗ ,k ∗ → j ∗ + µ j ∗ → π j ∗ ,k ∗ , h π j ∗ ,k ∗ } , σ 2 ε ( k ∗ ∑ i =1 1 h π j ∗ ,i ) -1      . (57) If S > 1 , then S = max { ⌈ σ 2 / ε ⌉ , 1 } = ⌈ σ 2 / ε ⌉ ≤ 2 σ 2 / ε , so it is true. Otherwise, if S ≤ 1 , then ¯ t ≤ 2 max    max { µ π j ∗ ,k ∗ → j ∗ + µ j ∗ → π j ∗ ,k ∗ , h π j ∗ ,k ∗ } , ( k ∗ ∑ i =1 1 h π j ∗ ,i ) -1    ≤ 2 max { max { µ π j ∗ ,k ∗ → j ∗ + µ j ∗ → π j ∗ ,k ∗ , h π j ∗ ,k ∗ } , h π j ∗ ,k ∗ } = 2max { µ π j ∗ ,k ∗ → j ∗ + µ j ∗ → π j ∗ ,k ∗ , h π j ∗ ,k ∗ }

and (57) holds. Notice that the r.h.s. of (57) equals to O ( t ∗ ( σ 2 / ε , [ h i ] n i =1 , [ µ i → j ∗ + µ j ∗ → i ] n i =1 ) ) , where t ∗ is the equilibrium time defined in Definition 2.

Theorem 6. Consider the assumptions and the parameters from Theorem 4. In each iteration k of Algorithm 3, the computation times of worker i are bounded by h k i . Let us fix any pivot worker j ∗ ∈ [ n ] and any spanning trees st and st bc . Then Algorithm 2 converges after at most

seconds, where µ k i → j ∗ ( µ k j ∗ → i ) is an upper bound on times required to send a vector from worker i to worker j ∗ (from worker j ∗ to worker i ) along the spanning tree st (spanning tree st bc ) in iteration k of Algorithm 3.

Proof. The proof is almost the same as in Theorem 5. If we fix a pivot worker j ∗ , then the k th iteration will finish after at most

seconds, where c is a universal constant. According to Theorem 4, the number of iterations is at most ⌈ 16 L ∆ ε ⌉ . Therefore, the total required time is at most

H Proof of the Time Complexity for Heterogeneous Case

Theorem 8. Let Assumptions 1 and 2 hold for the function f and Assumption 3 holds for the functions f i for all i ∈ [ n ] . We take γ = 1 / 2 L, the parameter S = max { ⌈ σ 2 / ε ⌉ , n } , any pivot worker j ∗ ∈ [ n ] , and any spanning trees st and st bc , in Algorithm 5. For all iterations number K ≥ 16 L ∆ /ε, we get 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

Proof. Algorithm 5 produces the sequence x k such that

where the ξ ij are independent random samples. Using the independence and Assumption 3, we have E [ g k ∣ ∣ G k ] = ∇ f ( x k ) and

where G k is a σ -algebra generated by g 0 , . . . , g k -1 . We do not dismiss the possibility that the computation and communication times are random, so s k i can be random. Assume V k is a σ -algebra generated by all computation and communication times and g 0 , . . . , g k -1 , then s k i is V k -measurable for all i ∈ [ n ] . Using the independence of stochastic gradients and the times and the tower property, we get

Algorithm 6 waits for the moment when s k ≥ S n , which is equivalent to

The value b j ∗ is calculated in Line 20 of Algorithm 7. Due to the asynchronous nature of the algorithm, we can only conclude that

because s k j ∗ can be increased by the time when Process 0 will receive b j ∗ . Using Line 15 from Algorithm 7, we can unroll the recursion in (62) and get

Let us substitute this inequality to (61) and (60) and get

As in Theorem 4, we can use the classical SGD result. Using Theorem 21, for the stepsize

we have if

Using the choice of S, we get that Algorithm 5 converges after

steps with

Theorem 9. Consider the assumptions and the parameters from Theorem 8. For any pivot worker j ∗ ∈ [ n ] and any spanning trees st and st bc , Algorithm 5 converges after at most

seconds, where µ k i → j ∗ ( µ k j ∗ → i ) is an upper bound on times required to send a vector from worker i to worker j ∗ (from worker j ∗ to worker i ) along the spanning tree st (spanning tree st bc ) for all i ∈ [ n ] .

Proof. Due to Theorem 8, the algorithm converges after K = Θ( L ∆ /ε ) iterations. Thus, it left to bound the time of one iteration. At the beginning of every iteration Process 0 broadcasts x k , which takes at most max i,j ∈ [ n ] µ i → j seconds. Then, Algorithm 6 waits for the moment when s k ≥ S n , which is equivalent to n 2 S ≥ b j ∗ . Thus, Algorithm 6 waits for the moment when n 2 S ≥ b j ∗ . We will return to this fact later.

Let us consider the term

Let us take

where s k i is the number of stochastic gradients calculated in worker i. Let us fix any time ¯ t > 0 . Then, worker i will calculate at least ⌊ ¯ t h i ⌋ stochastic gradients by the time ¯ t. Using this, we get

Then, since ⌊ x ⌋ ≥ x 2 for all x ≥ 1 , we get ⌊ ¯ t h i ⌋ ≥ ¯ t 2 h i and

Using ¯ t ≥ 2 S n ( 1 n ∑ n i =1 h i ) , we get and

after at most ¯ t seconds.

Recall that Algorithm 6 waits for the moment when n 2 S ≥ b j ∗ . Note that by the time when Process 0 receives b j ∗ , the counter s k i can be the same or increased; thus, b j ∗ captures potentially outdated information about s k i . Weknowthat (64) holds after at most ¯ t seconds. In Line 20 of Algorithm 7, Processes recursively collect 1 s k i to b j ∗ . Such a procedure will take at most 2 max i ∈ [ n ] µ i → j ∗ ≤ 2 max i,j ∈ [ n ] µ i → j seconds. Thus, the value b j ∗ will be less or equal n 2 S after at most ¯ t +2max i,j ∈ [ n ] µ i → j seconds.

The all reduce operation in (8) will take at most max i,j ∈ [ n ] µ i → j seconds. Thus, the total time of one iteration can be bounded by

seconds.

I Classical SGD Theory

We reprove the classical SGD result (Ghadimi and Lan, 2013; Khaled and Richtárik, 2022), for completeness.

Theorem 21. Let Assumptions 1 and 2 hold. We consider the SGD method:

For all k ≥ 0 , the vector g ( x ) is a random vector such that E [ g ( x k ) ∣ ∣ G k ] = ∇ f ( x k ) ,

where G k is a σ -algebra generated by g ( x 0 ) , . . . , g ( x k -1 ) . Then

Proof. From Assumption 1, we have

We denote G k as a sigma-algebra generated by g ( x 0 ) , . . . , g ( x k -1 ) . Using unbiasedness and (65), we obtain

Since γ ≤ 1 / L , we get

where for

We subtract f ∗ and take the full expectation to obtain

Next, we sum the inequality for k ∈ { 0 , . . . , K -1 } :

Finally, we rearrange the terms and use that E [ f ( x K ) -f ∗ ] ≥ 0 :

The choice of γ and K ensures that

Theorem 22. Let Assumptions 4 and 5 hold. We consider the SGD method:

For all k ≥ 0 , the vector g ( x ) is a random vector such that E [ g ( x k ) ∣ ∣ G k ] ∈ ∂f ( x k )

where G k is a σ -algebra generated by g ( x 0 ) , . . . , g ( x k -1 ) . Then

Proof. We denote G k as a sigma-algebra generated by g ( x 0 ) , . . . , g ( x k -1 ) . Using the convexity, for all x ∈ R d , we have

Note that

where for

and

Therefore, we get

By taking the full expectation and summing the last inequality for t from 0 to K -1 , we obtain

Let divide the last inequality by K, take x = x ∗ , and use the convexity:

The choices of γ and K ensure that (66) holds.

J Experiments

We now consider Fragile SGD with Minibatch SGD on quadratic optimization tasks with stochastic gradients. The working environment was emulated in Python 3.8 with one Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz. The homogeneous optimization problem (1) is constructed in the following way. We take

d = 1000 ,

Let us define [ x ] j as the j th index of a vector x ∈ R d . All n workers calculate the stochastic gradients

where ξ ∼ Bernouilli ( p ) , p ∈ (0 , 1] . In our experiments, we take p = 0 . 001 and the starting point x 0 = [ √ d, 0 , . . . , 0] ⊤ . We emulate our setup by considering that the i th worker requires h i = 1 second to calculate a stochastic gradient. And we assume that the workers have the structure of 2DMesh (see Figure 4a) and take ρ i → j = ρ ∈ { 0 . 1 , 1 , 10 } seconds for all edges that connect workers in 2D-Mesh. We take n = 100 . In all methods we fine-tune step sizes from the set { 2 i | i ∈ [ -20 , 20] } . In Fragile SGD , we fine-tune the batch size S from the set { 10 , 20 , 40 , 80 , 120 } .

The results are presented in Figures 5, 6, and 7. The plots are fully consisted with Table 1. One can see that when the communication is fast (Fig. 5), there is no big difference between the methods because both Fragile SGD and Minibatch SGD use all workers in the optimization steps. However, when we start decreasing the communication speed, we observe that Fragile SGD converges faster.

We looked deeper into the optimization processes of Fragile SGD in Figure 7 and observed that only 13 of 100 workers contribute to the optimization process for the batch size S = 120 . Other workers are too far away from the pivot worker, and their contributions can only slow down optimization.

0

20000

40000

60000

80000

100000

times (seconds)

10

3

10

2

f

(

x t

)

f

(

x

*

)

Fragile SGD: Batch Size: 40 Step size: 2.0

Fragile SGD: Batch Size: 80 Step size: 2.0

Fragile SGD: Batch Size: 120 Step size: 2.0

Minibatch SGD: Step size: 2.0

Minibatch SGD: Step size: 0.5

Minibatch SGD: Step size: 0.125

0

20000

40000

60000

80000

100000

times (seconds)

10

3

10

2

f

(

x t

)

f

(

x

*

)

Fragile SGD: Batch Size: 40 Step size: 2.0

Fragile SGD: Batch Size: 80 Step size: 2.0

Fragile SGD: Batch Size: 40 Step size: 1.0

Minibatch SGD: Step size: 2.0

Minibatch SGD: Step size: 1.0

Minibatch SGD: Step size: 0.5

0

20000

40000

60000

80000

100000

times (seconds)

10

3

10

2

f

(

x t

)

f

(

x

*

)

Fragile SGD: Batch Size: 120 Step size: 2.0

Fragile SGD: Batch Size: 80 Step size: 2.0

Fragile SGD: Batch Size: 200 Step size: 2.0

Minibatch SGD: Step size: 2.0

Minibatch SGD: Step size: 1.0

Minibatch SGD: Step size: 0.5

J.1 Experiments with Logistic Regression: Fast vs Slow Communication

We now repeat the previous experiments but with logistic regression on MNIST dataset (LeCun et al., 2010) with 100 workers. We consider two regimes: fast and slow communication between workers. One can see that when the communication is fast, the gap between the methods is small, which is expected and compliant with the theory. However, Fragile SGD is much faster and has better test accuracy when the communication is slow.

0.950

0

50000

100000 150000 200000 250000 300000 350000 400000

times (seconds)

2.2 × 10

1

2.4 × 10

1

2.6 × 10

1

2.8 × 10

1

3×10 1

f

(

x t

)

f

(

x

*

)

Fragile SGD: Batch Size: 160 Step size: 1.0

Fragile SGD: Batch Size: 120 Step size: 1.0

Fragile SGD: Batch Size: 100 Step size: 1.0

Minibatch SGD: Step size: 1.0

Minibatch SGD: Step size: 0.5

Minibatch SGD: Step size: 0.25

0

50000

100000 150000 200000 250000 300000 350000 400000

times (seconds)

0.920

0.925

0.930

0.935

0.940

0.945

Accuracy (Test)

Fragile SGD: Batch Size: 160 Step size: 1.0

Fragile SGD: Batch Size: 120 Step size: 1.0

Fragile SGD: Batch Size: 100 Step size: 1.0

Minibatch SGD: Step size: 1.0

Minibatch SGD: Step size: 0.5

Minibatch SGD: Step size: 0.25

0.940

0.0

0.2

0.4

0.6

0.8

1.0

times (seconds)

1e6

2.2 × 10

1

2.4 × 10

1

2.6 × 10

1

2.8 × 10

1

3×10 1

f

(

x t

)

f

(

x

*

)

Fragile SGD: Batch Size: 120 Step size: 1.0

Fragile SGD: Batch Size: 160 Step size: 1.0

Fragile SGD: Batch Size: 100 Step size: 1.0

Minibatch SGD: Step size: 1.0

Minibatch SGD: Step size: 2.0

Minibatch SGD: Step size: 0.5

0.0

0.2

0.4

0.6

0.8

1.0

times (seconds)

1e6

0.910

0.915

0.920

0.925

0.930

0.935

Accuracy (Test)

Fragile SGD: Batch Size: 120 Step size: 1.0

Fragile SGD: Batch Size: 160 Step size: 1.0

Fragile SGD: Batch Size: 100 Step size: 1.0

Minibatch SGD: Step size: 1.0

Minibatch SGD: Step size: 2.0

Minibatch SGD: Step size: 0.5

J.2 Experiments with ResNet-18

We test algorithms on an image recognition task, CIFAR10 (Krizhevsky et al., 2009), with the ResNet18 (He et al., 2016) deep neural network (the number of parameters d ≈ 10 7 ). We use the torus structure and 9 workers. We run all methods with the step sizes { 0 . 025 , 0 . 25 , 2 . 5 } . Our findings from the low-scale experiments are also evident in the large-scale experiments. Fragile SGD converges faster than Minibatch SGD in terms of function values. When we compare accuracies on the test split of MNIST , the superiority of Fragile SGD is even more transparent.

0

5000

10000

15000

20000

25000

30000

times (seconds)

10

3

10

2

10

1

10 0

f

(

x t

)

f

(

x

*

)

Fragile SGD: Step size: 0.25

Fragile SGD: Step size: 0.025

Fragile SGD: Step size: 2.5

Minibatch SGD: Step size: 0.25

Minibatch SGD: Step size: 0.025

Minibatch SGD: Step size: 2.5

0

5000

10000

15000

20000

25000

30000

times (seconds)

0.70

0.75

0.80

0.85

0.90

0.95

Accuracy (Test)

Fragile SGD: Step size: 0.25

Fragile SGD: Step size: 0.025

Fragile SGD: Step size: 2.5

Minibatch SGD: Step size: 0.25

Minibatch SGD: Step size: 0.025

Minibatch SGD: Step size: 2.5

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Table 1, the main part of the paper, Section C

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 5.3

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Section1.1, and the appendix

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section J

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code in the supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section J

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the top 3 best plots for each algorithm to reduce randomness factors in Section J.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section J

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the code of ethics, and our paper does not violate it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We consider a mathematical problem for machine learning.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We consider a mathematical problem for machine learning.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code and documentation in the supplementary materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper presents new theoretical advancements in decentralized stochastic asynchronous optimization. They introduce two new methods: Fragile SGD and Amelie SGD. Fragile SGD is designed for homogeneous setups, while Amelie SGD addresses heterogeneous setups. Both methods aim to achieve near-optimal time complexities under arbitrary heterogeneous computation and communication speeds. The paper establishes new lower bounds for time complexities in these setups and proves that the proposed methods match these bounds up to logarithmic factors.

STRENGTHS:
The paper is well-written and organized, making significant theoretical contributions by establishing new lower bounds for time complexities in decentralized asynchronous optimization. The introduction and analysis of Fragile SGD and Amelie SGD add value to the field. Additionally, the proposed methods demonstrate robustness to dynamic computation and communication times.

WEAKNESSES:
1. Although the algorithms are interpretable, they may be challenging to implement in practical systems. 

2. There is limited experimental validation of the proposed methods.

QUESTIONS:
I have no questions.

LIMITATIONS:
Limitations are addressed in the paper (selecting optimal pivot worker)
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the decentralizd optimization problem, where clients have heterogeneous computation and communication times. The paper proved a lower bound on the optimal physical time complexity (wherein the physical computation and communication times are take into consideration). It proposed Fragile SGD, an asynchronous decentralized first order method that matches this lower bound up to a logarithmic factor. It further considers the data heterogenous setting and proposed Amelie SGD that matches the lower bound in such a setting.\n\nSTRENGTHS:\nUnlike most other previous works, this paper takes into consideration the physical computation time and communication time of the clients in the decentralized network. The paper then proves a lower bound for decentralized optimization in terms of the physical time, and it proposed a nearly optimal algorithm Fragile SGD with matching physical running time. This is an interesting research setting, for which one could argue is more realistic, since whereas previous works on decentralized asynchrounous algorithms mostly considered the iteration complexity without worrying about the physical computation and communication time. In addition, unlike many other previous theoretical works on decentralized optimization where message passing is typically restricted to gossip type communications, this paper considers relayed communications where communication between any nodes on the graph are relayed on certain paths (in Fragile SGD, the paths are on certain spanning trees). These are all interesting additions to discussions around decentralized asynchronous algorithms.\n\nWEAKNESSES:\n1. This is also somewhat related to the strengths part. While the paper keeps referring to and comparing against some previous works (e.g. Even et al 2024), I do not think that there is a fair comparison to start with. Ultimately, the setting in which this paper is concerned is rather different from the previous works', where previous works typically only considers Gossip type communication and measures the convergence speed in terms of, number of iterations, communication rounds etc, but not the physical time of the whole process. Such a comparison is particular strange when the authors claimed that their method's convergence time does not depend on the spectral gap while previous works do -- this is obviously the case because previous works considers gossiping which is fundamentally constrained by the spectral gap while this paper allows any pair of clients to communicate via some paths on the graphs which is not directly constrained by the spectral gap (but can be otherwise more directly constrained by, say, the diameter of the graph).\n\n2. The second point is also somewhat related. I appreciate it that the authors takes computation and communication times into consideration. This might be somewhat a more realistic setting than what some previous works considers on the surface, but I'm still not very convinced that such a setting is actually closer to reality than the others. One could argue that the works in this paper is to some extend overfitting on certain metrics which are not necessarily a good measurement of the methods' performances in reality, just like the other (seemingly) simpler setting previous works consider. \n\n3. Fragile SGD requires a pivot worker which is, in some sense, at the center of the graph, to coordinate the updates. It also requires the knowledge of a spanning tree connecting the pivot worker to and from all other workers. Now suppose we take the pivot worker as the server, and set the communication time of all other workers as the distance from the worker to the pivot on the spanning tree, and we consider a the distributed asynchronous optimization problem with a centralized server. My understanding is that, conceptually speaking, the setting that the authors consider is closer to this hypothetical setting than to the more canonical decentralized setting that previous papers consider.\n\nQUESTIONS:\nsee weaknesses.\n\nLIMITATIONS:\nsee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the decentralized stochastic asynchronous optimization setup. The authors establish new time complexity lower bounds for both homogeneous and heterogeneous setups, assuming bounded computation and communication speeds. They introduce two methods: Fragile SGD, a nearly optimal method, and Amelie SGD, an optimal method. Their methods leverage spanning trees to solve the problem. These methods achieve convergence under arbitrary heterogeneous computation and communication speeds and match the derived lower bounds, up to a logarithmic factor in the homogeneous setting.\n\nSTRENGTHS:\nOriginality: \nThe innovative use of spanning trees to address the decentralized stochastic asynchronous optimization problem represents a significant departure from existing algorithms, introducing new theoretical contributions. Furthermore, the proposed algorithm exhibits greater robustness compared to previous methods, meaning that the impact of the worst-performing nodes on the overall update speed is minimized or even eliminated.\nSignificance:\nThe contributions of this paper to the field are significant. It offers a novel approach(spanning tree) for analyzing computational time complexity, which could be beneficial for the time complexity analysis of other decentralized stochastic asynchronous optimization algorithms. Additionally, the proposed method may assist in practical applications by providing insights into the selection of central nodes.\nQuality:\nThe paper provides a detailed explanation of the prerequisite knowledge, a thorough proof process, and a comprehensive introduction to the algorithm's features. Additionally, it includes another algorithm and experimental results in the appendix, offering a well-rounded and complete presentation of the material.\n\nWEAKNESSES:\nRegarding the structure of the paper, I think there are several issues. First, one of the two algorithms mentioned in the abstract is relegated entirely to the appendix, which is inappropriate for the paper’s structure. Second, the experimental results are also exclusively placed in the appendix, which does not conform to standard layout. Finally, Section 6(Example: line or circle), which serves as the conclusion, fails to provide an effective summary of the entire paper and instead functions more as the concluding part of the theoretical analysis.\n\nQUESTIONS:\nIn practical applications, we often cannot choose which node serves as the central node and are instead constrained by existing conditions. In such scenarios, will the proposed algorithm still maintain its current advantages?\n\nLIMITATIONS:\nI believe the authors have discussed the limitations of their method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the time complexity lower bounds in decentralized stochastic asynchronous optimization. It introduces two methods, Fragile SGD and Amelie SGD, which achieve near-optimal and optimal convergence, respectively. The paper also provides convergence analysis for various settings.\n\nSTRENGTHS:\nThe topic of studying the exact communication costs is interesting since this should be a crucial problem in decentralized learning. This also brings us a new perspective on decentralized learning.\n\nWEAKNESSES:\n1. I believe it is crucial to  **provide empirical results** to demonstrate the effectiveness of the proposed method from two key aspects: the efficiency, which underscores the method’s effectiveness, and the final accuracy, which ensures overall performance guarantee. I would consider raising the score if solid empirical results were provided.\n\n2. I think the proposed methods heavily relies on the pivot worker, which represents a different concept of convergence dependency compared to the spectral gap. If I understand correctly, the framework benefits from a tree-like structure; however, this structure necessitates that the pivot worker possesses significantly greater computation and communication capabilities, almost functioning as a server. For instance, as mentioned in Line 141, “A pivot worker aggregates all stochastic gradients.” Therefore, I think the proposed method acts as a compromise between the centralized “server-client” framework and a decentralized one. Could you please discuss more on the relationship between the pivot worker and the server?\n\nQUESTIONS:\n1. Could you provide empirical results for the proposed methods?\n\n2. Could you provide more discussion about the relationship between the pivot worker and the server?\n\nLIMITATIONS:\nMost limitations are listed in the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}