Title: Shadowheart SGD: Distributed Asynchronous SGD with Optimal Time Complexity Under Arbitrary Computation and Communication Heterogeneity
Venue: NeurIPS 2024
Accepted: yes
Keywords: nonconvex optimization

parallel methods

asynchronous methods

lower bounds

compressed communication
Abstract: We consider nonconvex stochastic optimization problems in the asynchronous centralized distributed setup where the communication times from workers to a server can not be ignored, and the computation and communication times are potentially different for all workers. Using an unbiassed compression technique, we develop a new method—Shadowheart SGD—that provably improves the time complexities of all previous centralized methods. Moreover, we show that the time complexity of Shadowheart SGD is optimal in the family of centralized methods with compressed communication. We also consider the bidirectional setup, where broadcasting from the server to the workers is non-negligible, and develop a corresponding method.

Shadowheart SGD: Distributed Asynchronous SGD with Optimal Time Complexity Under Arbitrary Computation and Communication Heterogeneity

Abstract

We consider nonconvex stochastic optimization problems in the asynchronous centralized distributed setup where the communication times from workers to a server can not be ignored, and the computation and communication times are potentially different for all workers. Using an unbiassed compression technique, we develop a new methodShadowheart SGD -that provably improves the time complexities of all previous centralized methods. Moreover, we show that the time complexity of Shadowheart SGD is optimal in the family of centralized methods with compressed communication. We also consider the bidirectional setup, where broadcasting from the server to the workers is non-negligible, and develop a corresponding method.

1 Introduction

We consider the nonconvex smooth optimization problem

̸

where f ( · ; · ) : R d × S ξ → R , and D ξ is a distribution on S ξ = ∅ . Given ε > 0 , we seek to find a possibility random point ˆ x such that E [ ∥∇ f (ˆ x ) ∥ 2 ] ≤ ε. Such a point ˆ x is called an ε -stationary point. We focus on solving the problem in the following setup:

(a) n workers/nodes are able to compute stochastic gradients ∇ f ( x ; ξ ) of f, in parallel and asynchronously , and it takes (at most) h i seconds for worker i to compute a single stochastic gradient;

(b) the workers are connected to a server which acts as a communication hub;

(c) the workers can communicate with the server in parallel and asynchronously ; it takes (at most) τ i seconds for worker i to send a compressed message to the server; compression is performed via applying lossy communication compression to the communicated message (a vector from R d ); see Def. 2.1;

(d) the server can broadcast compressed vectors to the workers in (at most) τ serv seconds; compression is performed via applying a lossy communication compression operator to the communicated message (a vector from R d ); see Def. A.1.

The main goal of this work is to find an optimal optimization strategy/method that would work uniformly well in all scenarios characterized by the values of the computation times h 1 , . . . , h n and

Method

Time Complexities in Some Regimes

{

h

n

,

˙

τ

n

} → ∞

,

{

h

i

,

˙

τ

i

}

<

∞∀

i < n

(the last worker is slow)

h

i

=

h,

˙

τ

i

= ˙

τ

∀

i

∈

[

n

]

(equal performance)

Numerical Comparison

(b)

σ

2

/

ε

=

1

10

3

10

6

Minibatch SGD

(see (2))

i

∈

[

n

]

{

h

i

, d

˙

τ

i

}

(

L

∆

ε

+

σ

2

L

∆

nε

2

)

∞

(non-robust)

{

h, d

˙

τ,

d

˙

τσ

2

nε

,

hσ

2

nε

}

L

∆

ε

(worse, e.g., when

˙

τ, d

or

n

large)

×

10

3

×

10

3

×

10

4

QSGD

(see (5))

(Alistarh et al., 2017)

i

∈

[

n

]

{

h

i

,

˙

τ

i

}

( (

d

n

+1

)

L

∆

ε

+

dσ

2

L

∆

nε

2

)

∞

(non-robust)

dhσ

2

nε

L

∆

ε

(worse, e.g., when

ε

small)

×

3

×

10

2

×

10

4

(Tyurin and Richtárik, 2023c),

(e.g., (Mishchenko et al., 2022))

min

j

∈

[

n

]

{

h

¯

π j

, d

˙

τ

¯

π j

,

σ

2

ε

(

j

∑

i

=1

1

h

¯

π i

)

-

1

}

L

∆

ε

(a)

<

∞

(robust)

{

h, d

˙

τ,

hσ

2

nε

}

L

∆

ε

(worse, e.g., when

˙

τ, d

or

n

large)

×

10

2

×

10

×

1

.

5

(see (9) and Alg. 1)

(Corollary 4.3)

t

∗

(

d

-

1

,

σ

2

/

ε

,

[

h

i

,

˙

τ

i

]

n

1

)

L

∆

ε

(c)

<

∞

(robust)

{

h,

˙

τ,

d

˙

τ

n

,

√

d

˙

τhσ

2

nε

,

hσ

2

nε

}

L

∆

ε

×

1

×

1

×

1

Lower Bound

(see Section 5 and Theorem O.5)

t

∗

(

d

-

1

,

σ

2

/

ε

,

[

h

i

,

˙

τ

i

]

n

1

)

L

∆

ε

(d)

-

-

-

-

-

The time complexity of Shadowheart SGD is not worse than the time complexity of the competing centralized methods, and is strictly better in many regimes (see Section 7.1). We show that (10) is the optimal time complexity in the family of centralized methods with compression (see Section 5 and Theorem O.5).

communication times τ 1 , . . . , τ n and τ serv . Since we allow these times to be arbitrarily heterogeneous, designing a single algorithm that would be optimal in all these scenarios seems challenging.

Let us clarify what we mean by { τ i } and take, for instance, the Rand K compressor (see Def. D.1 and Sec. 2.2) (sends K random entries). Then τ i is # of seconds required to send K coordinates of a vector. Now assume that the communication is proportional to the number of coordinates/bits that a worker sends, i.e., it takes ˙ τ i seconds to send a one coordinate/bit . Then, clearly, we have τ i = K × ˙ τ i . and, τ i is a function of K. While the dependence is clear for Rand K , for all possible compressors, the dependence on the amount of sent information can be less nontrivial. We generally fix an arbitrary unbiased compressor and assume it takes τ i seconds to send the compressed message.

From the viewpoint of federated learning (Koneˇ cný et al., 2016; Kairouz et al., 2021), our work is a theoretical study of device heterogeneity. Moreover, our formalism captures both cross-silo and cross-device settings as special cases. Due to our in-depth focus on device heterogeneity and the challenges that need to be overcome, we do not consider statistical heterogeneity, and leave an extension to this setup to future work.

We rely on assumptions which are standard in the literature on stochastic gradient methods: smoothness, lower-boundedness and bounded variance.

Assumption 1.1. f is differentiable & L -smooth, i.e., ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ , ∀ x, y ∈ R d

.

Assumption 1.2. There exist f ∗ ∈ R such that f ( x ) ≥ f ∗ for all x ∈ R d . We define ∆ := f ( x 0 ) -f ∗ , where x 0 ∈ R d is a starting point of all algorithms we consider.

Assumption 1.3. For all x ∈ R d , the stochastic gradients ∇ f ( x ; ξ ) are unbiased, and their variance is bounded by σ 2 ≥ 0 , i.e., E ξ [ ∇ f ( x ; ξ )] = ∇ f ( x ) and E ξ [ ∥∇ f ( x ; ξ ) -∇ f ( x ) ∥ 2 ] ≤ σ 2 .

To simplify the exposition, in what follows we first focus on the regime in which the broadcast cost can be ignored. We describe a strategy for extending our algorithm to the more general regime in Sec. A.

2 Related Work

2.1 Communication time can be ignored

We now briefly review related work and important concepts. Consider the regime when the communication cost is negligible ( τ i = 0 for all i ), and the computation times h i are arbitrary

but fixed. It is well-known that under Assumptions 1.1, 1.2, and 1.3, the vanilla SGD method x k +1 = x k -γ ∇ f ( x k ; ξ k ) , where x 0 ∈ R d is a starting point, and γ > 0 is the step size, solves (1) using the optimal number of stochastic gradients (Ghadimi and Lan, 2013; Arjevani et al., 2022). Since the # of iterations of SGD to get an ε -stationary point is O ( L ∆ / ε + σ 2 L ∆ / ε 2 ) , SGD run on a single worker whose computation time is h 1 seconds would have time complexity O ( h 1 × ( L ∆ / ε + σ 2 L ∆ / ε 2 )) seconds. The time complexity of Minibatch SGD with n workers, i.e.,

can be shown (Gower et al., 2019) to be

where h max := max i ∈ [ n ] h i , where [ n ] denotes { 1 , . . . , n } . The dependence on h max is due to Minibatch SGD employing synchronous parallelism which forces it to wait for the slowest worker. While the stochastic part of (3) can be n times smaller than in the single worker case, (3) does not guarantee an improvement since h max can be arbitrarily large. In real systems, computation times can be very heterogeneous and vary in time in chaotic ways (Dutta et al., 2018; Chen et al., 2016).

Recently, Cohen et al. (2021); Mishchenko et al. (2022) and Koloskova et al. (2022) showed that it is possible to improve upon (3) using the celebrated Asynchronous SGD method (Recht et al., 2011; Feyzmahdavian et al., 2016; Nguyen et al., 2018) and get the time complexity O(( 1 / n ∑ n i =1 1 h i ) -1 × ( L ∆ / ε + σ 2 L ∆ / nε 2 ) ) , which improves the dependence from h max to the harmonic mean of the computation times. Subsequently, Tyurin and Richtárik (2023c) developed the Rennala SGD method whose time complexity is

where π is a permutation forwhich h π 1 ≤ · · · ≤ h π n . They also showed that the time complexity (4) is optimal by providing a matching lower bound.

2.2 Communication time is a factor

In many practical scenarios, communication times can be the main bottleneck, and can not be ignored, e.g., in distributed/federated training of machine learning models (Ramesh et al., 2021; Kairouz et al., 2021; Wang et al., 2023). There are two main techniques for reducing the communication bottleneck: local training steps (McMahan et al., 2017) and compressed communication (Seide et al., 2014; Alistarh et al., 2017). In our work, we investigate the latter technique. In particular, efficient methods with compressed communication such as DIANA (Mishchenko et al., 2019), Accelerated DIANA (Li et al., 2020), MARINA (Gorbunov et al., 2021) and DASHA (Tyurin and Richtárik, 2023b) employ unbiased compressors , defined next. Assume that S ν is a nonempty arbitrary set of samples, and D ν is a distribution on S ν .

Definition 2.1. A mapping C : R d × S ν → R d is an unbiased compressor if there exists ω ≥ 0 such that E ν [ C ( x ; ν )] = x, E ν [ ∥C ( x ; ν ) -x ∥ 2 ] ≤ ω ∥ x ∥ 2 for all x . Let U ( ω ) denote the family of such compressors 5 .

Assumption 2.2. Samples from D ξ and D ν are mutually independent.

The canonical example of an unbiased compressor is the Rand K compressor (see Def. D.1) that scales K random entries of the input vector x by d / K and zeros out the rest. Many more examples of unbiased compressors are considered in the literature (Beznosikov et al., 2020; Xu et al., 2021a; Horváth et al., 2022). One of the most straightforward methods which use compression is QSGD 6 (Alistarh et al., 2017):

where each worker calculates one stochastic gradient, compresses it using C i ∈ U ( ω ) drawn independently, and sends it to the server. The server aggregates the compressed vectors and performs step (5). With a proper stepsize choice γ, QSGD converges after O ( ( ω / n +1) × L ∆ / ε +( ω +1) × σ 2 L ∆ / nε 2 ) iterations 7 (Khaled and Richtárik, 2020). Let's assume it takes τ i seconds for worker i to send one compressed vector to the server. Since the workers act in parallel, the time complexity of QSGD is

We can go through a similar exercise with any other method that uses compressed communication (e.g., (Tyurin and Richtárik, 2023a; Gauthier et al., 2023; Jia et al., 2023)). Nevertheless, as far as we know, the optimal time complexities for asynchronous centralized distributed optimization with communication compression are not known .

3 Summary of Contributions

In the regime in which the communication time can be ignored (see Sec. 2.1), Tyurin and Richtárik (2023c) showed that (4) is the optimal time complexity. In this work we endeavor to take the next step: we wish to understand the fundamental limits of the regime in which communication time is a factor. Our main contributions are:

♠ We develop a new methodShadowheart SGD (Algorithm 1)-that guarantees to find an ε -stationary point of problem (1) with time complexity T ∗ given in (10). While the general expression we give for T ∗ is hard to parse since it involves the equilibrium time t ∗ ( · ) whose definition is implicit (see Def. 3.1), we show (see Sec. 7) that T ∗ is not worse than the time complexity of known centralized 8 methods, and also who that it can be strictly better in many regimes, even by many degrees of magnitude (see Sec. 7.1 and Table 1).

♣ In Sec. 5 we show that (10) is the optimal time complexity in the family of centralized methods with compression. This is the first such result in the literature.

♦ We also developed Adaptive Shadowheart SGD (Sec. 4.2 and M), which does not require the knowledge of the computation and communication times and can work with arbitrary changing times. Moreover, we designed Bidirectional Shadowheart SGD (Sec. A), which works in the regime when broadcast cost not negligible as well.

♥ Our theoretical study of Shadowheart SGD is supported by judiciously designed synthetic experiments and machine-learning experiments with logistic regression; see Sec. Q.

4 Development of Shadowheart SGD

Our method bears some resemblance to Rennala SGD (Tyurin and Richtárik, 2023c) and QSGD (Alistarh et al., 2017), and involves some additional algorithmic elements which play a key role. First, we adopted the main suggestion of Tyurin and Richtárik (2023c)[Sec.7] behind the design of Rennala SGD that an optimal method should calculate stochastic gradients at the last iterate. Second, QSGD served as an inspiration for how to perform gradient compression. In particular, Shadowheart SGD has the form x k +1 = x k -γg k , where

In Shadowheart SGD , worker i calculates b i stochastic gradients, adds them up to form ∑ b i l =1 ∇ f ( x k ; ξ k il ) , and compresses the result m i times using independently drawn compressors. The compressed messages are sent to the server. The first non-trivial step in the design of our method is the presence of weights w i : the server aggregates the ∑ n i =1 m i compressed messages across all workers by performing a conic combination with coefficient w i ∑ n i =1 w i m i b i for messages coming from worker i . One can easily show that (9) is equivalent to Alg. 1. Note that we recover QSGD (see (5)) as a special (suboptimal) case with w i = b i = m i = 1 for all i ∈ [ n ] .

Algorithm 1 Shadowheart SGD

1: Input: starting point x 0 ∈ R d , stepsize γ > 0 , ratio σ 2 / ε , computation times h i > 0 , and communication times τ i > 0 for i ∈ [ n ]

2:

3:

4:

5:

6:

7:

8:

Find equilibrium time

Set and

b

=

m

Find for

t

⌊

⌋

∗

i

h

S

k

A

t

∗

=

b

i

using Def. 3.1

⌊

t

⌋

∗

τ

i

for all

i

∈

∧

i

i

=

i

[

n

i

]

:

m

>

0

{

∈

}

= 0

do

,

1

, . . . , K

1

-

Run Alg. 2 in active workers

A

S

Broadcast

i

i

x

, b

, m

k

to active workers

Initialize

= 0

k

9:

g

∈

in parallel do

12:

13:

14:

15:

16:

17:

18:

S

for

i

Receive ij

g

C

(

)

from worker

g

=

k

g

k

end for end for

g

=

k

g

k

x

+

k

i

w

n

(

∑

k

=1

i

-

i

C

w

γg ij

i

m

k

k

+1

=

/

x

end for

Algorithm 2 Strategy of Worker i

Definition 3.1 (Equilibrium Time) .

with inputs ω, σ 2 / ε , h 1 , τ 1 , . . . , h n , τ n is called the equilibrium time if it is defined as follows. Find a permutation a π that sorts the pairs ( h i , τ i ) as max { h π 1 , τ π 1 } ≤ · · · ≤ max { h π n , τ π n } and find the solution s ∗ ( j ) ∈ [0 , ∞ ] in s of b

for all j ∈ [ n ] . Then the mapping returns the value

We shall use the short notation t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) .

a It is possible that a permutation is not unique. The result of the mapping does not depend on the choice of the permutation. See the proof of Property 4.1.

b For convenience, we use the projectively extended real line and define 1 / 0 = ∞ .

The weights { w i } are chosen so as to minimize the variance in the proof of Lemma H.1. However, we still need to find the right values for b i and m i . Since the computation and communication times of worker i are h i and τ i , respectively, the following strategy makes intuitive sense: the server sets some time budget t for all workers, and each worker then calculates ⌊ t / h i ⌋ stochastic gradients and sends ⌊ t / τ i ⌋ compressed vectors to the server. But what is the right way to choose t ? If t is too small, then, intuitively, some workers may not have time to calculate 'enough' gradients, or may even not have time to send any messages to the server. On the other hand, if t is too large, then the workers will eventually send information of diminishing utility which will not be worth the extra time this takes.

We find out that, and this one of the key insights of our work, that there exists an optimal time budget t ∗ which depends on the quantities ω , σ 2 / ε , h 1 , τ 1 , . . . , h n , τ n , for which we coin the name equilibrium time ; see Def. 3.1. Admittedly, the definition of the equilibrium time is implicit; we do not know if it is possible to give a more explicit formula in general. To provide for some peace of mind, we prove the following property:

Property 4.1. If all inputs of the equilibrium time are non-negative, then the equilibrium time is well defined.

More importantly, in Sec. 5 we provide a lower bound that involves the same mapping. Thus, the equilibrium time is not an 'artifact' of our method, but is of a fundamental nature. We use the equilibrium time t ∗ in Shadowheart SGD when we choose b i and m i .

(

g

)

i

b

i

k

i

)

i

[

n

S

]

A

Our first main result provides iteration complexity:

Theorem 4.2. Lett Assumptions 1.1, 1.2, 1.3, 2.2 hold. Let us take γ = 1 / 2 L in Shadowheart SGD (Alg. 1). Then as long as K ≥ 16 L ∆ /ε, we have the guarantee 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

This result guarantees that Shadowheart SGD will converge after O ( L ∆ / ε ) iterations. Our second main result provides a much more relevant complexity measure: time complexity .

Corollary 4.3. Shadowheart SGD (Alg. 1) converges after at most T ∗ seconds, where

Surprisingly, we show in Sec. 5 that our time complexity guarantee (10) is optimal for the family of centralized methods with compressed communication. Moreover, in Sec. 7 and 7.1, we show that (10) is no worse and can be significantly better than the time complexities of previous centralized methods (see also Table 1 for a summary).

4.1 Tighter result with per-iteration times h k i and τ k i

A slight modification of Alg. 1 leads to Alg. 4, which can work with iteration-dependent computation and communication times h k i and τ k i . Our main result in this setup is Theorem H.3; here we present its corollary.

Theorem 4.4. Alg. 4 converges after

seconds, where h k i > 0 and τ k i > 0 are computation and communication times for worker i in iteration k .

For presentation simplicity sake, in the main part we continue to work with static { h i } and { τ i } .

4.2 On the problem of estimating the times in Algorithms 1 and 4

One of the main features of asynchronous methods (e.g., Rennala SGD , Asynchronous SGD ) is their adaptivity to and independence from processing times. In Sec. M, we design Adaptive Shadowheart SGD (Alg. 7) with this feature. Unlike Alg. 1, it does not require the knowledge of { h i } and { τ i } (or { h k i } and { τ k i } in the case of Alg. 4), and does not calculate the equilibrium time t ∗ . However, as a byproduct of this flexibility, this method has a slightly worse time complexity guarantee. In order to present our result, we need to define an auxiliary sequence.

Definition 4.5. Assume that the workers have computation and communication times less or equal to { h i } and { τ i } . Assume that ¯ h ij is the actual time required to calculate the j th stochastic gradient by worker i , h min > 0 is the smallest possible computation time. Then

t max := 128 × t ∗ ( ω, σ 2 / ε , [max { h i , τ i } , max { h i , τ i } ] n 1 ) .

That is, r i ∈ [1 , ∞ ] is the largest ratio between the fastest and the slowest computation of stochastic gradients in local time windows. r i defines a degree of fluctuations in computation times. Note that r i describes local fluctuations; it is true that r i ≤ sup j ≥ 1 ¯ h i,j / inf j ≥ 1 ¯ h i,j for all i ∈ [ n ] and r i can be arbitrarily smaller.

A corollary of our main result in this part (Theorem M.1) is presented next.

Corollary 4.6. If the computation and communication times are positive, the time complexity of Alg. 7 is L ∆ ε × t ∗ ( ω, σ 2 / ε , [max { h i , τ i } , min { τ i r i , max { h i , τ i }} ] n 1 ) up to a constant factor, where r i is defined in Def. 4.5.

Unlike Alg. 1 and Alg. 4, Alg. 7 is more 'greedy'; it calculates stochastic gradients and sends compressed vectors in parallel, and it does not know the times h i and τ i (or h k i and τ k i ). That is why this method gets a suboptimal complexity and depends on r i . Nevertheless, if we assume that i) the computation times do not fluctuate significantly, i.e., r i = Θ(1) , and ii) τ i ≤ h i for all i ∈ [ n ] , then this complexity reduces to the optimal complexity L ∆ / ε × t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) .

5 Lower Bound

Protocol 3 Simplified Representation of Protocol 9

1: Init S = ∅ on the server (all available information)

2: while True do

3: Server calculates a new point ¯ x using S and broadcasts ¯ x and S to any worker (broadcasting does not take time)

4: end while i th Worker (in parallel):

5: while True do

6: Receives ¯ x and S, calculates as many stochastic gradients as it want at the point ¯ x (each calculation takes h i seconds), aggregates all available information, and sends compressed vectors (each dispatch takes τ i seconds), which will be added to the set S

7: end while

In Sec. 4, we stated that Shadowheart SGD converges after T ∗ seconds; with T ∗ given in (10). Our next step is to understand if it might be possible to improve this complexity. In Sec. O, we formalize our setup and show in Theorem O.5 that up to a constant factor, the result (10) is optimal . Here we present a simplified illustration of our approach.

Protocol 3 can describe all centralized methods (the server updates the iterates, and the workers calculate stochastic gradients at these points), including Minibatch SGD , Asynchronous SGD , Rennala SGD , and Shadowheart SGD . In Theorem O.5, we show that up to a constant factor, no method described by Protocol 3 can converge faster than (10) seconds. In order to use our lower bound, the workers must calculate stochastic gradients at a point that was calculated by the server.

Let us briefly explain the proof's idea. The general approach is the same as in (Nesterov, 2003; Arjevani et al., 2022; Huang et al., 2022): we take the 'difficult' function (Sec. P.1), which has large gradients while the last coordinate equals to zero. Every algorithm starts with the point x 0 = 0 , and the only way to discover the next coordinate is to calculate a stochastic gradient. Oracles associated with the workers return the next non-zero coordinate with the probability p σ ≈ ε / σ 2 . Even if the stochastic oracle returns a non-zero coordinate for some worker, the corresponding communication oracle on this worker also has to return a non-zero coordinate, which happens with probability p ω ≈ 1 / ω +1 . We fix the Rand K compressor in the lower bound theorem with K ≈ L ∆ / ε ( ω +1) , and the number of coordinates ≈ L ∆ / ε ; thus, indeed, p ω ≈ 1 / ω +1 . Since all n workers work in parallel, they can discover and send to the server the next non-zero coordinate not earlier than after min m ∈ [ n ] { h m η m + τ m µ m } seconds, where η m and µ m are i.i.d. geometric random variables with p σ and p ω . With a high probability, we show that this quantity is Ω( t ∗ ( 1 / p ω , 1 / p σ , h 1 , τ 1 , . . . , h n , τ n )) . The number of coordinates is ≈ L ∆ / ε . Therefore, the lower bound is (10) seconds up to a constant factor.

6 Equilibrium Time

Since the time complexity (10) of Shadowheart SGD is optimal, we believe that the equilibrium time is a fundamental mapping that should be investigated more deeply.

6.1 Calculation strategy

The calculation of t ∗ requires us to sort max { h i , τ i } . Next, it is sufficient to solve n equations from (7). In Property 4.1, we prove that (7) has one unique solution that can be easily found, for instance, using the bisection method. Then, is it left to find the minimum in (8).

6.2 Intuition behind the equilibrium time t ∗

Assuming we found an optimal j ∗ in (8), we have t ∗ = max { max { h π j ∗ , τ π j ∗ } , s ∗ ( j ∗ ) } . The first observation is that t ∗ does not depend on the workers that correspond to max { h π j ∗ +1 , τ π j ∗ +1 } , . . . , max { h π n , τ π n } . Since these values are greater or equal to max { h π j ∗ , τ π j ∗ } , the mapping 'decides' to ignore them because they are too slow. The following

derivations are not rigorous and are merely supposed to offer some intuition. We define α i := τ π i ω and β i := h π i σ 2 / ε . Next, using (7), we have

where M := { i ∈ [ j ∗ ] : max { α i , β i } ≥ α i β i / s ∗ ( j ∗ ) } . Solving this, one can get that

Thus, s ∗ ( j ∗ ) divides the active workers into two groups M and [ j ∗ ] \ M . Both groups contribute to (13) with a harmonic mean -like and a quadratic harmonic mean -like dependences, correspondingly. The transition between two groups is decided by the rule max { α i , β i } ≥ α i β i / s ∗ ( j ∗ ) ⇔ s ∗ ( j ∗ ) ≥ min { α i , β i } . Intuitively, the last inequality means that if τ i or h i is small (a worker can quickly compute a gradient or send a compressed vector), it belongs to M . Otherwise, if a worker's computation and communication performance are balanced, it belongs to [ j ∗ ] \ M .

6.3 Properties of the equilibrium time t ∗

We now provide some properties and particular cases to understand t ∗ better. One can find the proofs and more properties in Sec. E. The first result says that t ∗ is monotonic.

Property 6.1. If ¯ ω ≥ ω ≥ 0 , ¯ σ 2 / ¯ ε ≥ σ 2 / ε ≥ 0 , ¯ h 1 ≥ h 1 ≥ 0 , ¯ τ 1 ≥ τ 1 ≥ 0 , . . . , ¯ h n ≥ h n ≥ 0 , and ¯ τ n ≥ τ n ≥ 0 , then t ∗ (¯ ω, ¯ σ 2 / ¯ ε , [ ¯ h i , ¯ τ i ] n 1 ) ≥ t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) .

Consider the Rand K compressor. If it takes ˙ τ i sec to send one coordinate by worker i , then, up to a constant factor, Property 6.2 ensures that an optimal choice of K is 1 .

Property 6.2. For all K ∈ [1 , d ] , σ 2 / ε , h 1 , ˙ τ 1 , . . . , h n , ˙ τ n ≥ 0 , we have 24 × t ∗ ( d / K -1 , σ 2 / ε , h 1 , K ˙ τ 1 , . . . , h n , K ˙ τ n ) ≥ t ∗ ( d -1 , σ 2 / ε , h 1 , ˙ τ 1 , . . . , h n , ˙ τ n ) .

6.4 Examples

We now list several examples, starting with simple corner/extreme cases. One can find the derivations in Sec. F. For brevity, we will sometimes write t ∗ instead of t ∗ ( ω, σ 2 / ε , h 1 , τ 1 , . . . , h n , τ n ) .

Example 6.3 . [ Infinitely Fast Worker ] If exists j ∈ [ n ] such that τ j = 0 and h j = 0 , then t ∗ = 0 .

Example 6.4 . [ Infinitely Slow Workers ] If τ i = ∞ and h i = ∞ for all i ∈ [ n ] , then t ∗ = ∞ .

Example 6.5 . [ Equal Performance ] If τ i = τ and h i = h for all i ∈ [ n ] , then 9

In the next example, we consider the setting from Sec. 2.1. Example 6.6 and Corollary 4.3 restore the optimal rate (4) of Rennala SGD .

Example 6.6 . [ Infinitely Fast Communication ] If τ i = 0 for all i ∈ [ n ] , then

where π is a permutation that sorts { h i } n i =1 .

The following two examples show that t ∗ is robust to slow workers or workers that do not participate. Example 6.7 . [ Ignoring Slow Workers ] If h i and τ i are fixed and finite for all i ≤ p , and max { h i , τ i } = m ∈ R for all i > p, then, for m large enough, we have t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) = t ∗ ( ω, σ 2 / ε , [ h i , τ i ] p 1 ) .

Example 6.8 . [ Partial Participation ] If max { h i , τ i } = ∞ for all i > p ≥ 1 , then t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) = t ∗ ( ω, σ 2 / ε , [ h i , τ i ] p 1 ) .

7 Comparison with Baselines

In the previous sections, we did not invoke any assumptions about the compressors except for Def. 2.1. Inspired by Property 6.2, to make the comparisons with the baselines easier, we consider the Rand K compressor with K = 1 . Using Theorem D.2, we have ω = d -1 . We also assume that worker i takes ˙ τ i seconds to send one coordinate to the server; thus τ i = ˙ τ i , since we use Rand 1 . Also, it takes d ˙ τ i to send a non-compressed vector for all i ∈ [ n ] .

Minibatch SGD and QSGD. It is well known (Lan, 2020) that the number of iterations of Minibatch SGD required to find an ε -solution is O ( L ∆ / ε + σ 2 L ∆ / ε 2 ) . In Minibatch SGD , each worker calculates one stochastic gradient and sends a non-compressed vector. Since the server waits for the slowest worker, the time complexity of such method (up to a constant factor) is

In Sec. J, we compare (16) with (10) and show Comparison 7.1 . T ∗ = O( T MB ) .

However, there are many regimes when T ∗ ≪ T MB . For instance, if max { h i , ˙ τ i } = ∞ for some worker (Example 6.8), then T MB = ∞ and T ∗ < ∞ . Also, under the conditions of Example 6.7, if m →∞ , we get T MB →∞ whereas T ∗ is bounded. The same reasoning applies to QSGD because its time complexity (6) depends on max i ∈ [ n ] ( h i + ˙ τ i ) . Due to Theorem O.5, up to a constant factor, T ∗ is less or equal to (6); see also Table 1.

Rennala SGD and Asynchronous SGD. When the communication time is negligible, Tyurin and Richtárik (2023c) proved that the optimal time complexity is attained by Rennala SGD . When ˙ τ i → 0 for all i ∈ [ n ] , we show in Example 6.6 that (10) is the same as the time complexity of Rennala SGD obtained by Tyurin and Richtárik (2023c). Assume ˙ τ i > 0 for all i ∈ [ n ] . We can apply the result from Theorem O.5 to Rennala SGD , thus the time complexity of Rennala SGD is not better than

Note that Asynchronous SGD also has the same lower bound. In Sec. J, we compare (17) with (10) and show

Comparison 7.2 . T ∗ = O( T R ) .

7.1 Shadowheart SGD is strictly better in many regimes.

Due to the non-explicit nature, it is not transparent that the time complexity of Shadowheart SGD is universally strictly better than in all baselines in many practical regimes. Let us prove it. Take

Due to (16) and (17), the time complexities of Minibatch SGD and QSGD are ∞ , and the time complexities of Asynchronous SGD and Rennala SGD are not smaller than

From (14), the time complexity of Shadowheart SGD is at most

Thus, Shadowheart SGD can be d times faster than all previous methods if the number of workers n is large. Using the same reasoning, Shadowheart SGD can be n -1 times faster if the dimension d or ˙ τ is large. Due to the continuity of the complexities, such huge differences hold even if we take n < ∞ , and start considering more heterogenous times h i and τ i by perturbating (18).

7.2 The fastest worker works locally

Another important baseline is the vanilla SGD method, which works on the fastest worker, does not communicate with the server, and performs local steps (non-centralized method). For simplicity, assume that σ 2 / ε ≥ 1 . Then, the time complexity of such an algorithm (Lan, 2020) is T SGD :=

min i ∈ [ n ] h i × σ 2 L ∆ / ε 2 . Clearly, comparing T SGD and (10), if ˙ τ i are large enough, then T SGD can be smaller than T ∗ . However, this does not contradict our lower bounds because this method does not satisfy the conditions of Theorem O.5: it does not communicate with the server. In other words, if the communication channel is too slow, it does not make sense to communicate. One may now ask: 'Under which conditions is it beneficial to communicate?' Comparing T SGD and (10), one can see that (10) is better when t ∗ ( d -1 , σ 2 / ε , h 1 , ˙ τ 1 , . . . , h n , ˙ τ n ) ≤ min i ∈ [ n ] h i × σ 2 / ε . It is sufficient to substitute the initial parameters to this inequality and decide which method to use. For instance, in the view of Example 6.5, one should compare max { h, ˙ τ, ˙ τ ( d -1) / n , hσ 2 / nε , √ ˙ τhσ 2 ( d -1) / nε } vs. hσ 2 / ε . In the regime when n is large enough or ε is small enough, we have t ∗ < hσ 2 / ε , and Alg. 7 has better convergence guarantees. On the other hand, if ˙ τ is large enough, then it is possible that t ∗ > hσ 2 / ε .

Acknowledgments and Disclosure of Funding

The research reported in this publication was supported by funding from King Abdullah University of Science and Technology (KAUST): i) KAUST Baseline Research Scheme, ii) Center of Excellence for Generative AI, under award number 5940, iii) SDAIA-KAUST Center of Excellence in Artificial Intelligence and Data Science. The work of A.T. was partially supported by the Analytical center under the RF Government (subsidy agreement 000000D730321P5Q0002, Grant No. 70-2021-00145 02.11.2021).

References

Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. (2017). QSGD: Communicationefficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems (NIPS) , pages 1709-1720.

Arjevani, Y., Carmon, Y., Duchi, J. C., Foster, D. J., Srebro, N., and Woodworth, B. (2022). Lower bounds for non-convex stochastic optimization. Mathematical Programming , pages 1-50.

Beznosikov, A., Horváth, S., Richtárik, P., and Safaryan, M. (2020). On biased compression for distributed learning. arXiv preprint arXiv:2002.12410 .

Carmon, Y., Duchi, J. C., Hinder, O., and Sidford, A. (2020). Lower bounds for finding stationary points i. Mathematical Programming , 184(1):71-120.

Chen, J., Pan, X., Monga, R., Bengio, S., and Jozefowicz, R. (2016). Revisiting distributed synchronous sgd. arXiv preprint arXiv:1604.00981 .

Cohen, A., Daniely, A., Drori, Y., Koren, T., and Schain, M. (2021). Asynchronous stochastic optimization robust to arbitrary delays. Advances in Neural Information Processing Systems , 34:9024-9035.

Dutta, S., Joshi, G., Ghosh, S., Dube, P., and Nagpurkar, P. (2018). Slow and stale gradients can win the race: Error-runtime trade-offs in distributed SGD. In International Conference on Artificial Intelligence and Statistics , pages 803-812. PMLR.

Feyzmahdavian, H. R., Aytekin, A., and Johansson, M. (2016). An asynchronous mini-batch algorithm for regularized stochastic optimization. IEEE Transactions on Automatic Control , 61(12):3740-3754.

Gauthier, F., Gogineni, V. C., Werner, S., Huang, Y.-F., and Kuh, A. (2023). Asynchronous online federated learning with reduced communication requirements. arXiv preprint arXiv:2303.15226 .

Ghadimi, S. and Lan, G. (2013). Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368.

Gorbunov, E., Burlachenko, K., Li, Z., and Richtárik, P. (2021). MARINA: Faster non-convex distributed learning with compression. In 38th International Conference on Machine Learning .

Gower, R. M., Loizou, N., Qian, X., Sailanbayev, A., Shulgin, E., and Richtárik, P. (2019). SGD: General analysis and improved rates. In International Conference on Machine Learning , pages 5200-5209. PMLR.

Gruntkowska, K., Tyurin, A., and Richtárik, P. (2023). EF21-P and friends: Improved theoretical communication complexity for distributed optimization with bidirectional compression. In International Conference on Machine Learning , pages 11761-11807. PMLR.

Horváth, S., Ho, C.-Y., Horváth, v., Sahu, A. N., Canini, M., and Richtárik, P. (2022). Natural compression for distributed deep learning. In Mathematical and Scientific Machine Learning , pages 129-141. PMLR.

Huang, X., Chen, Y., Yin, W., and Yuan, K. (2022). Lower bounds and nearly optimal algorithms in distributed learning with communication compression. arXiv preprint arXiv:2206.03665 .

Jia, J., Liu, J., Zhou, C., Tian, H., Dong, M., and Dou, D. (2023). Efficient asynchronous federated learning with sparsification and quantization. arXiv preprint arXiv:2312.15186 .

Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. (2021). Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210.

Khaled, A. and Richtárik, P. (2020). Better theory for SGD in the nonconvex world. arXiv preprint arXiv:2002.03329 .

Koloskova, A., Stich, S. U., and Jaggi, M. (2022). Sharper convergence guarantees for asynchronous SGD for distributed and federated learning. arXiv preprint arXiv:2206.08307 .

Koneˇ cný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., and Bacon, D. (2016). Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 .

Lan, G. (2020). First-order and stochastic optimization methods for machine learning . Springer.

LeCun, Y., Cortes, C., and Burges, C. (2010). Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2.

Li, Z., Kovalev, D., Qian, X., and Richtárik, P. (2020). Acceleration for compressed gradient descent in distributed and federated optimization. In International Conference on Machine Learning .

McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communicationefficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR.

Mishchenko, K., Bach, F., Even, M., and Woodworth, B. (2022). Asynchronous SGD beats minibatch SGD under arbitrary delays. arXiv preprint arXiv:2206.07638 .

Mishchenko, K., Gorbunov, E., Takáˇ c, M., and Richtárik, P. (2019). Distributed learning with compressed gradient differences. arXiv preprint arXiv:1901.09269 .

Nemirovskij, A. S. and Yudin, D. B. (1983). Problem complexity and method efficiency in optimization.

Nesterov, Y. (2003). Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media.

Nesterov, Y. (2018). Lectures on convex optimization , volume 137. Springer.

Nguyen, L., Nguyen, P. H., Dijk, M., Richtárik, P., Scheinberg, K., and Takác, M. (2018). SGD and hogwild! convergence without the bounded gradients assumption. In International Conference on Machine Learning , pages 3750-3758. PMLR.

Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021). Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR.

Recht, B., Re, C., Wright, S., and Niu, F. (2011). Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Advances in Neural Information Processing Systems , 24.

Richtárik, P., Sokolov, I., and Fatkhullin, I. (2021). EF21: A new, simpler, theoretically better, and practically faster error feedback. arXiv preprint arXiv:2106.05203 .

Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. (2014). 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In Fifteenth Annual Conference of the International Speech Communication Association .

Tyurin, A. and Richtárik, P. (2023a). A computation and communication efficient method for distributed nonconvex problems in the partial participation setting. Advances in Neural Information Processing Systems (NeurIPS) .

Tyurin, A. and Richtárik, P. (2023b). DASHA: Distributed nonconvex optimization with communication compression, optimal oracle complexity, and no client synchronization. 11th International Conference on Learning Representations (ICLR) .

Tyurin, A. and Richtárik, P. (2023c). Optimal time complexities of parallel stochastic optimization methods under a fixed computation model. Advances in Neural Information Processing Systems (NeurIPS) .

Vogels, T., Karimireddy, S. P., and Jaggi, M. (2019). PowerSGD: Practical low-rank gradient compression for distributed optimization. In Neural Information Processing Systems .

Wang, J., Lu, Y., Yuan, B., Chen, B., Liang, P., De Sa, C., Re, C., and Zhang, C. (2023). Cocktailsgd: Fine-tuning foundation models over 500mbps networks. In International Conference on Machine Learning , pages 36058-36076. PMLR.

Xu, H., Ho, C.-Y., Abdelmoniem, A. M., Dutta, A., Bergou, E. H., Karatsenidis, K., Canini, M., and Kalnis, P. (2021a). Grace: A compressed communication framework for distributed machine learning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS) , pages 561-572. IEEE.

Xu, H., Kostopoulou, K., Dutta, A., Li, X., Ntoulas, A., and Kalnis, P. (2021b). Deepreduce: A sparsetensor communication framework for federated deep learning. Advances in Neural Information Processing Systems , 34:21150-21163.

Contents

A Bidirectional Compression

In this section, we discuss a simple way to use the Shadowheart SGD techniques in the setup when broadcasting is expensive (Line 7 in Alg. 1); i.e., when τ serv ≫ 0 . We will employ the following family of compressors.

Definition A.1. A mapping C : R d × S ν → R d is a biased compressor if there exists α ∈ (0 , 1] such that

We shall use the shortcut C ( x ; ν ) ≡ C ( x ) , and denote the family of such biased compressors as B ( α ) .

The family B ( α ) is more general than U ( ω ) in the sense that if C ∈ U ( ω ) , then ( ω + 1) -1 C ∈ B (( ω +1) -1 ) . It includes the Top K and Rank K compressors (Vogels et al., 2019; Beznosikov et al., 2020), among many others.

Let C serv ∈ B ( α ) be the compressor used by the server. We use the primal error-feedback mechanism EF21-P (Gruntkowska et al., 2023) which requires us to add the following changes to Alg. 1 and Alg. 2. We add the steps

to Alg. 1 and broadcast p k +1 instead of x k . This change leads to Bidirectional Shadowheart SGD (Alg. 5). In Alg. 2, the workers should receive p k +1 , calculate w k +1 , and use w k instead of x k in the calculations of stochastic gradients. We provide the pseudo-codes of these algorithms in Sec. K.

Our main results are:

Theorem A.2. Let Assumptions 1.1, 1.2, 1.3, 2.2 hold. Choose γ = α 16 L . Then as long as K ≥ 768 L ∆ αε , Bidirectional Shadowheart SGD (Alg. 5) guarantees to find an ε -stationary point.

Corollary A.3. If the broadcast time of C serv is not greater than τ serv , then Bidirectional Shadowheart SGD (Alg. 5) converges after at most

seconds.

Remark A.4 . If the broadcast cost can't be ignored, the time complexity of Alg. 1 changes from (10) to

where τ full serv is the time required to broadcast a full/ non-compressed vector.

We should compare (21) obtained by the unidirectional algorithm and (20) obtained by the bidirectional algorithm. Consider that C serv = Top K with K ≤ d. We can see (20) that depends on τ serv , that is much less than τ full serv because K ≪ d. At the same time, (20) is 1 / α times larger than (21). This is a standard price for the fact that we use a biased compressor (e.g. (Richtárik et al., 2021; Gruntkowska et al., 2023)). However, α is very close 1 in practice (Beznosikov et al., 2020; Vogels et al., 2019; Xu et al., 2021b). It turns out that we can always choose K in Top K (we take this compressor as an example) in such a way that Alg. 5 is never worse than Alg. 1.

Comparison A.5 . Assume that it takes ˙ τ serv seconds to send one coordinate from the server to the workers. If we take K ≥ min { d, t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) / ˙ τ serv } in Top K, then T ∗ , serv = O( T ∗ ) .

If τ full serv = d ˙ τ serv is the bottleneck in (21) with Alg. 1, i.e., d ˙ τ serv ≫ t ∗ , then one can take K = t ∗ / ˙ τ serv ≪ d in Top K with Alg. 5 and improve the time complexity.

B Frequently Used Notation

We thought a table of frequently used notation could be useful. Here it is:

C Basic Facts

Here we collect some basic facts which are used repeatedly in the proofs.

Variance decomposition. Let x ∈ R d be a random vector with finite mean and finite variance. Then for any deterministic vector c ∈ R d , we have the identity

Lemma C.1. Consider a sequence q 1 , . . . , q n ∈ [0 , 1] , then

Proof. We prove by induction. For n = 1 , is it true: 1 -∑ 1 m =1 q m = ∏ 1 m =1 (1 -q m ) . Assume that that it is true for n -1 . Then

Multiply both parts by 1 -q n ∈ [0 , 1] to obtain

since q m ∈ [0 , 1] for all m ∈ [ n ]

D Rand K Compressor

Definition D.1. Assume that S is a random subset from [ d ] , | S | = K, K ∈ [ d ] . Astochastic mapping C : R d × S ν → R d is Rand K if

where { e i } d i =1 is the standard unit basis.

Theorem D.2. If C is Rand K , then C ∈ U ( d k -1 ) .

E Proofs of the Properties of the Equilibrium Time

Property 4.1. If all inputs of the equilibrium time are non-negative, then the equilibrium time is well defined.

Proof.

(Part 1: s ∗ ( j ) is well-defined)

First, we show that s ∗ ( j ) is well-defined for all j ∈ [ n ] . We fix j ∈ [ n ] and consider the equation from Def. 3.1:

w.r.t s. The function ϕ ( s ) is a non-increasing function for all s ≥ 0 , and the function ψ ( s ) is an increasing function for all s ≥ 0 . Let us consider two cases.

1) Exists p ≤ j such that τ π p ω = 0 and h πp σ 2 ε = 0 , then

̸

for all s ≥ 0 , then the only solution to the equation is s = 0 .

2) Otherwise, we have

for all s ≥ 0 . Then ϕ (0) > 0 (can be equal to ∞ ). Using ψ (0) = 0 and the monotonicity of the functions, one can show the unique solution (greater zero) exists.

If a permutation π is unique, then the formula min j ∈ [ n ] max { max { h π j , τ π j } , s ∗ ( j ) } is well-defined and we can finish the proof.

(Part 2: non-unique permutation)

We assume that there exists i ∈ [ n ] such that max { h i , τ i } < ∞ . Otherwise, min j ∈ [ n ] max { max { h π j , τ π j } , s ∗ ( j ) } = ∞ for any permutation. Next, note that s ∗ ( j +1) ≤ s ∗ ( j ) for all j < n because

for all s ≥ 0 . We will use this property later.

Consider that there are two non-equal permutations π and ¯ π that sort the pairs ( h i , τ i ) by max { h i , τ i } , and there are two corresponding solutions s ∗ ( j ) and ¯ s ∗ ( j ) . Each permutation divides the pairs ( h i , τ i ) into the same equivalence classes:

The order within each class can be different, but the elements are the same. Next, since s ∗ ( j +1) ≤ s ∗ ( j ) for all j < n, we can conclude that the minimum in

is attained for j ∗ such that max { h π j ∗ , τ π j ∗ } < max { h π j ∗ +1 , τ π j ∗ +1 } ( max { h π n +1 , τ π n +1 } ≡ ∞ ). Since max { h π j ∗ , τ π j ∗ } < max { h π j ∗ +1 , τ π j ∗ +1 } , we have max { h π j ∗ , τ π j ∗ } = max { h ¯ π j ∗ , τ ¯ π j ∗ } . Therefore, we obtain

Also, for all j ∈ [ n ] such that max { h π j , τ π j } < max { h π j +1 , τ π j +1 } , we have

Therefore, we get s ∗ ( j ∗ ) = ¯ s ∗ ( j ∗ ) and

Using the same reasoning, we can show that

It means that min j ∈ [ n ] max { max { h ¯ π j , τ ¯ π j } , ¯ s ∗ ( j ) } = min j ∈ [ n ] max { max { h π j , τ π j } , s ∗ ( j ) } , thus the final result of the mapping does not depend on a chosen permutation.

Property 6.1. If ¯ ω ≥ ω ≥ 0 , ¯ σ 2 / ¯ ε ≥ σ 2 / ε ≥ 0 , ¯ h 1 ≥ h 1 ≥ 0 , ¯ τ 1 ≥ τ 1 ≥ 0 , . . . , ¯ h n ≥ h n ≥ 0 , and ¯ τ n ≥ τ n ≥ 0 , then t ∗ (¯ ω, ¯ σ 2 / ¯ ε , [ ¯ h i , ¯ τ i ] n 1 ) ≥ t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) .

Proof. Assume that π is a permutation that sorts the pairs ( h i , τ i ) by max { h i , τ i } , and ¯ π is a permutation that sorts the pairs ( ¯ h i , ¯ τ i ) by max { ¯ h i , ¯ τ i } , then

for all j ∈ [ n ] . It means ¯ s ∗ ( j ) ≥ s ∗ ( j ) , where s ∗ ( j ) and ¯ s ∗ ( j ) are the solutions of the equation (7) with the pairs ( h i , τ i ) and ( ¯ h i , ¯ τ i ) and corresponding permutations π and ¯ π. Also, we have max { h π j , τ π j } ≤ max { ¯ h ¯ π j , ¯ τ ¯ π j } for all j ∈ [ n ] . Therefore, we have

Property E.1. For all c ∈ (0 , 1] and ω, σ 2 / ε , h 1 , τ 1 , . . . , h n , τ n ≥ 0 , we have

Proof. Using the definition of the equilibrium time, we have

where s ∗ ( j ) is the solution of

and

where s ∗ c ( j ) is the solution of

Using simple algebra, we obtain

Thus, s ∗ c ( j ) is the solution of

Comparing (24) and (25), one can see that s ∗ c ( j ) = c × s ∗ ( j ) for all j ∈ [ n ] . Using this and c ∈ (0 , 1] , we get

Property E.2. For all c ≥ 1 and ω, σ 2 / ε , h 1 , τ 1 , . . . , h n , τ n ≥ 0 , we have

Proof. The proof of this property repeats the proof of Property E.1 up to the last inequality. Using c ≥ 1 , we get

Property E.3. For all c ∈ (0 , 1] and ω, σ 2 / ε , h 1 , τ 1 , . . . , h n , τ n ≥ 0 , we have t ∗ ( c × ω, σ 2 / ε , [ h i , τ i ] n 1 ) ≥ c × t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) and t ∗ ( ω, c × σ 2 / ε , [ h i , τ i ] n 1 ) ≥ c × t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) . For all c ≥ 1 and ω, σ 2 / ε , h 1 , τ 1 , . . . , h n , τ n ≥ 0 , we have t ∗ ( c × ω, σ 2 / ε , [ h i , τ i ] n 1 ) ≤ c × t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) and t ∗ ( ω, c × σ 2 / ε , [ h i , τ i ] n 1 ) ≤ c × t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 )

Remark E.4 . We can obtain stronger inequalities. See Properties E.1 and E.2.

Proof. For all c ∈ (0 , 1] , using Properties 6.1 and E.1, we have

For all c ≥ 1 , using Properties 6.1 and E.2, we have

Proof. For c = 0 , it it clear. Assume that c > 0 . Using the definition of the equilibrium time, we have

where s ∗ ( j ) is the solution of

and

where s ∗ c ( j ) is the solution of

For both cases, we can take the same permutation π. Using simple algebra, we obtain

Thus, s ∗ c ( j ) is the solution of

Comparing (26) and (27), one can see that s ∗ c ( j ) = c × s ∗ ( j ) for all j ∈ [ n ] . Using this, we get t ∗ ( ω, σ 2 / ε , c × h 1 , c × τ 1 , . . . , c × h n , c × τ n ) = min j ∈ [ n ] max { max { c × h π j , c × τ π j } , c × s ∗ ( j ) } = c × min j ∈ [ n ] max { max { h π j , τ π j } , s ∗ ( j ) } = c × t ∗ ( ω, σ 2 / ε , h 1 , τ 1 , . . . , h n , τ n ) .

Property E.6. We fix a nonempty subset S = { k 1 , . . . , k m } from the set [ n ] with a size m ≥ 1 . For all ω, σ 2 / ε , h 1 , τ 1 , . . . , h n , τ n ≥ 0 , we have

Proof. Using Property 6.1 with ¯ τ i = ∞ and ¯ h i = ∞ for all i ̸∈ S and ¯ τ i = τ i and ¯ h i = h i for all i ∈ S, we have

Next, using Def. 3.1, we obtain

where s ∗ ( j ) is the solution of

w.r.t s for all j ∈ [ n ] , and π is a permutation that sorts max { ¯ h i , ¯ τ i } in such a way that the set { π 1 , . . . , π m } equals to the set { k 1 , . . . , k m } (the order of elements can be different). Such permutation exists because max { ¯ h i , ¯ τ i } = ∞ for all i ̸∈ S. Using max { ¯ h π i , ¯ τ π i } = ∞ for all i > m, we have

By the construction of π, (28) and (29) depend only on the elements from S. Thus, we have

Property E.7. For all σ 2 / ε , h 1 , ˙ τ 1 , . . . , h n , ˙ τ n ≥ 0 , we have

Proof. For d = 1 , it is clear. Assume that d > 1 . Using the definition of t ∗ , we have

where ¯ π is a permutation that sorts max { h i , d ˙ τ i } . Assume that j ∗ is the minimal index that minimizes (30). Then

Let us define

Using Property E.6, we have

Using Def. 3.1 of t ∗ , we get where s ∗ is the solution of

Let us take s ′ = 12 max { ( d -1) max j ∈ [ j ∗ ] ˙ τ ¯ π j , σ 2 ε ( ∑ j ∗ i =1 1 h ¯ π i ) -1 } . Since s ′ ≥ ( d -1) max j ∈ [ j ∗ ] ˙ τ ¯ π j , we have

If there exists p ∈ [ j ∗ ] such that 1 ˙ τ ¯ πp ( d -1) < 1 h ¯ πp σ 2 ε , then

and

Otherwise, we have and

Considering both cases, we have

It means that s ∗ ≤ s ′ because s ∗ is the solution of (33). Using (32), we get

Using d ≥ 1 and d ˙ τ ¯ π j ≤ max { h ¯ π j , d ˙ τ ¯ π j } , we get

Due to max j ∈ [ j ∗ ] max { h ¯ π j , d ˙ τ ¯ π j } = max { h ¯ π j ∗ , d ˙ τ ¯ π j ∗ } and (31), we obtain

Property 6.2. For all K ∈ [1 , d ] , σ 2 / ε , h 1 , ˙ τ 1 , . . . , h n , ˙ τ n ≥ 0 , we have 24 × t ∗ ( d / K -1 , σ 2 / ε , h 1 , K ˙ τ 1 , . . . , h n , K ˙ τ n ) ≥ t ∗ ( d -1 , σ 2 / ε , h 1 , ˙ τ 1 , . . . , h n , ˙ τ n ) .

Proof.

where s ∗ ( j ) is the solution of

and π is a permutation that sorts max { h j , K ˙ τ j } . Also, assume that j ∗ is a minimizer in (34). For all j ∈ [ n ] , we get

Since K ≤ d +1 2 , we have

and

At the same time, using Property E.6, we have t ∗ ( d -1 , σ 2 / ε , h 1 , ˙ τ 1 , . . . , h n , ˙ τ n ) ≤ t ∗ ( d -1 , σ 2 / ε , h π 1 , ˙ τ π 1 , . . . , h π j ∗ , ˙ τ π j ∗ ) ≤ max { max j ∈ [ j ∗ ] max { h ¯ π j , ˙ τ ¯ π j } , s ′ ( j ∗ ) } , (36)

where s ′ ( j ∗ ) is the solution of

From (35), we can conclude that 2 × s ∗ ( j ∗ ) ≥ s ′ ( j ∗ ) . Using this and (36), we obtain

Since

we obtain s ∗ (1) = 0 . We substitute it to (37) to get t ∗ ( ω, σ 2 / ε , h 1 , τ 1 , . . . , h n , τ n ) = 0 .

Example 6.4 . [ Infinitely Slow Workers ] If τ i = ∞ and h i = ∞ for all i ∈ [ n ] , then t ∗ = ∞ .

Proof. By the definition of t ∗ , we have

Note that max j ∈ [ j ∗ ] max { h ¯ π j , K ˙ τ ¯ π j } = max { h ¯ π j ∗ , K ˙ τ ¯ π j ∗ } , thus

(Part 2: K > d +1 2 ) For all K > d +1 2 , using Property 6.1, we get

Next, using Property E.5, we have

It is left to use Property E.7 to get

F Derivations of the Examples for the Equilibrium Time

Example 6.3 . [ Infinitely Fast Worker ] If exists j ∈ [ n ] such that τ j = 0 and h j = 0 , then t ∗ = 0 .

Proof. Let us take a permutation π where π 1 = j. Such a permutation exists because max { h j , τ j } = 0 . By the definition of t ∗ , we have

where s ∗ (1) is the solution of

Example 6.5 . [ Equal Performance ] If τ i = τ and h i = h for all i ∈ [ n ] , then 10

Proof. By the definition, we have

t

∗

(

ω,

σ

/

ε

2

1

1

n

, τ

where s ∗ is the solution of

Since

we have to solve and find the non-negative solution of the quadratic equation

, h

, τ

, . . . , h

) = min

j

∈

[

n

]

The solution is

Therefore, we have

Example 6.6 . [ Infinitely Fast Communication ] If τ i = 0 for all i ∈ [ n ] , then

where π is a permutation that sorts { h i } i =1 .

Proof. By the definition, we have t ∗ ( ω, σ 2 / ε , h 1 , τ 1 , . . . , h n , τ n ) = min j ∈ [ n ] max { h π j , s ∗ ( j ) } , where s ∗ ( j ) is the solution of

Therefore, we have

Using Lemma F.1, we obtain

n

{

{

h, τ

}

∗

, s

(

j

)

}

{

{

h, τ

}

,

min

j

∈

[

n

]

s

∗

(

j

)

}

(38)

{

{

h, τ

}

∗

, s

}

Lemma F.1. Let us consider the two functions

for all j ∈ [ n ] , where h i ≥ 0 for all i ∈ [ n ] , a ≥ 0 , and h 1 ≤ · · · ≤ h n . Then

Proof. If h 1 = 0 , then p (1) = h (1) = 0 and min i ∈ [ n ] p ( i ) = min i ∈ [ n ] h ( i ) = 0 . Assume that h 1 > 0 . Using the fact that a harmonic mean is less or equal to the maximum, we have

Thus, we proved the upper bound. Next, assume that j ∗ is the smallest minimizer of p ( j ) . If j ∗ = 1 , then

Otherwise, if j ∗ > 1 , then p ( j ∗ ) ≤ p ( j ∗ -1) . Using simple algebra, we obtain

Using the last inequality, we get

Example 6.7 . [ Ignoring Slow Workers ] If h i and τ i are fixed and finite for all i ≤ p , and max { h i , τ i } = m ∈ R for all i > p, then, for m large enough, we have t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) = t ∗ ( ω, σ 2 / ε , [ h i , τ i ] p 1 ) .

Proof. By the definition, we have

For m > max i ∈ [ p ] max { h i , τ i } , we have max { h i , τ i } < m for all i ≤ p and the set { 1 , . . . , p } equals to { π 1 , . . . , π p } . Thus, we get

By taking m>t ∗ ( ω, σ 2 / ε , h 1 , τ 1 , . . . , h p , τ p ) , we obtain

Therefore, we have

Example 6.8 . [ Partial Participation ] If max { h i , τ i } = ∞ for all i > p ≥ 1 , then t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) = t ∗ ( ω, σ 2 / ε , [ h i , τ i ] p 1 ) .

Proof. By the definition, we have

where π is a permutation such that the set { π p +1 , . . . , π n } equals to the set { p +1 , . . . , n } . Such a permutation exists because max { h i , τ i } = ∞ for all i > p. Using this, we have

G Generic Lemma For Unbiased Gradient Estimators

We prove the following generic lemma that estimates the variance of the general family of unbiased gradient estimators.

Lemma G.1. Consider that Assumptions 1.3 and 2.2 hold. Let us consider the gradient estimator

where m i ≥ 0 for all i ∈ [ n ] , b ij ≥ 0 for all i ∈ [ n ] and j ∈ [ m i ] are ordered batch sizes ( b i 1 ≤ ·· · ≤ b i,m i for all i ∈ [ n ] ), w ij ≥ 0 are weights for all i ∈ [ n ] and j ∈ [ m i ] , and ∑ n i =1 ∑ m i j =1 w ij b ij > 0 , C ij ∈ U ( ω ij ) are mutually independent compressors from Def. 2.1 for all i ∈ [ n ] and j ∈ [ m i ] , and x k ∈ R d is an arbitrary point. Then E [ g k ] = ∇ f ( x k ) and

Proof. First, we show the gradient estimator is unbiased:

Using Def. 2.1 and Assumption 1.3, we have

Next, we estimate the variance

Using the independence and (22), we have

Using the independence of the compressors and Def. 2.1, we get

In the view of (22), the independence of the stochastic gradients, and Assumption 1.3, we obtain

We now consider

Let us consider the set S il := { j ∈ [ m i ] | l ≤ b ij } for all i, l ∈ N . Then we can rewrite the norm in the following way

The stochastic vectors are independent, thus

Note that

The number of appearances of the term w ij w ip in the sum equals to min { b ij , b ip } . Thus

We now substitute the bounds on I 1 and I 2 to (40), and get (39).

Algorithm 4 Shadowheart SGD (Alg. 1 is equivalent to Alg. 4 when h k i = h k and τ k i = τ k ) 1: Input: starting point x 0 , stepsize γ , the ratio σ 2 / ε 2: for k = 0 , 1 , . . . , K -1 do 3: Find the maximum computation speeds h k i > 0 and compressors' communication speeds τ k i > 0 of the workers in the current iteration 4: Find the equilibrium time t ∗ using Def. 3.1 with h k i and τ k i 5: Set b i = ⌊ t ∗ h k i ⌋ and m i = ⌊ t ∗ τ k i ⌋ for all i ∈ [ n ] ( t ∗ , b i and m i are local and can be different in every iteration) 6: Find active workers S A = { i ∈ [ n ] : b i ∧ m i > 0 } 7: Run Alg. 2 in all active workers S A 8: Broadcast x k , b i , and m i to all active workers S A 9: Init g k = 0 10: for i ∈ S A in parallel do 11: w i ( a ) = ( b i ω + ω σ 2 ε + m i σ 2 ε ) -1 12: for j = 1 , . . . , m i do 13: Receive C ij ( g k i ) from the i th worker 14: g k = g k + w i C ij ( g k i ) 15: end for 16: end for 17: g k = g k / ( ∑ n i =1 w i m i b i ) 18: x k +1 = x k -γg k 19: end for ( a ) : If ω = 0 and σ 2 ε = 0 , then w i = 1

H Proofs for Algorithms 1 and 4

In the appendix, we work with Alg. 4 instead of Alg. 1. Alg. 4 is more general and estimates all the parameters based on local per-iteration times h k i and τ k i instead of h i and τ i . All results for Alg. 1 can be easily obtained by taking h k i = h i and τ k i = τ i .

Lemma H.1. Consider that Assumptions 1.3 and 2.2 hold. Then the gradient estimator (9) with the weights w i from Alg. 4 is unbiased and

Proof. Alg. 4 implements the gradient estimator (9). We can use Lemma G.1 with b ij = b i , w ij = w i and ω ij = ω for all i ∈ [ n ] and j ∈ [ m i ] . Using (39), we have

We add nonnegative terms to the last inequality to obtain

Using the choice of the weights i

Lemma H.2. Consider two quantities ω ≥ 0 and σ 2 / ε ≥ 0 , and n ∈ N . Also, consider a sequence of positive pairs { ( h i , τ i ) } n i =1 . We take b i = ⌊ t ∗ h i ⌋ and m i = ⌊ t ∗ τ i ⌋ for all i ∈ [ n ] , where t ∗ ≡ t ∗ ( ω, σ 2 / ε , h 1 , τ 1 , . . . , h n , τ n ) is the equilibrium time from Def. 3.1. Then

Proof. Assume that j ∗ ∈ [ n ] is the smallest index that minimizes max { max { h π j , τ π j } , s ∗ ( j ) } , then

Therefore, we have t ∗ ≥ max { h π j ∗ , τ π j ∗ } ≥ max { h π j , τ π j } for all j ≤ j ∗ since max { h π j , τ π j } are sorted. It means that b π j = ⌊ t ∗ h π j ⌋ ≥ 1 and m π j = ⌊ t ∗ τ π j ⌋ ≥ 1 for all j ≤ j ∗ . Using this, we have

Since b π j = ⌊ t ∗ h π j ⌋ ≥ 1 and m π j = ⌊ t ∗ τ π j ⌋ ≥ 1 , we can also conclude that b π j ≥ t ∗ 2 h π j and m π j ≥ t ∗ 2 τ π j for all j ≤ j ∗ . Therefore, we obtain

where the last inequality follows from (42). Recall that s ∗ ( j ∗ ) is the solution of the equation (7). Thus

Theorem H.3. Assume that Assumptions 1.1, 1.2, 1.3, 2.2 hold. Let us take γ = 1 2 L in Alg. 4. Then for all iterations

x

k

)

∥

2

]

ε.

and

Alg. 4 guarantees that

1

K

K

-

1

k

=0

∑

E

[ ∥

∥

∇

f

(

∥

Proof. Let us fix any iteration k ∈ N . Consider that G k is a σ -algebra generated by g 0 , . . . , g k -1 . Then, given G k , x k is a deterministic vector. Using Lemma H.1, we have

Note that the choice of the parameters in Alg. 4 satisfy the conditions of Lemma H.2. Thus, we have

for all k ≥ 0 . It is left to use the standard SGD analysis from Theorem I.1 with B = 1 and C = ε to finish the proof.

Theorem 4.2. Lett Assumptions 1.1, 1.2, 1.3, 2.2 hold. Let us take γ = 1 / 2 L in Shadowheart SGD (Alg. 1). Then as long as K ≥ 16 L ∆ /ε, we have the guarantee 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

Proof. It immediately follows from Theorem H.3 for h i = h k i and τ i = τ k i .

Theorem 4.4. Alg. 4 converges after

seconds, where h k i > 0 and τ k i > 0 are computation and communication times for worker i in iteration k .

Proof. Let us fix an iteration index k ∈ [ n ] . In every iteration, every worker calculates b i stochastic gradients and sends m i compressed vectors. Thus, the processing time of each iteration is not greater than

Using the fact that the number of iterations equals to (43), we finally get (11).

Corollary 4.3. Shadowheart SGD (Alg. 1) converges after at most T ∗ seconds, where

Proof. It immediately follows from Theorem 4.4 for h i = h k i and τ i = τ k i .

I The Classical SGD Theorem

Let us consider a slightly modified classical SGD result from (Ghadimi and Lan, 2013; Khaled and Richtárik, 2020).

Theorem I.1. Assume that Assumptions 1.1 and 1.2 hold. We consider the SGD method:

For all k ≥ 0 , the vector g ( x ) is a random vector such that E [ g ( x k ) ∣ ∣ G k ] = ∇ f ( x k ) ,

where

where G k is a σ -algebra generated by g ( x 0 ) , . . . , g ( x k -1 ) . The quantities B and C are arbitrary nonnegative constants. Then

Proof. From Assumption 1.1, we have

We denote G k as a sigma-algebra generated by g ( x 0 ) , . . . , g ( x k -1 ) . Using unbiasedness and (45), we obtain

Since γ ≤ 1 L (1+ B ) , we get

We subtract f ∗ and take the full expectation to obtain

Next, we sum the inequality for k ∈ { 0 , . . . , K -1 } :

Finally, we rearrange the terms and use that E [ f ( x K ) -f ∗ ] ≥ 0 :

The choice of γ and K ensures that

J Comparison with Baselines

In the following proofs, we use assumptions and definitions from Sec. 7. Comparison 7.1 . T ∗ = O( T MB ) .

for

Proof. Without loss of generality, we assume that all workers are sorted by max { h i , ˙ τ i } . For the Rand 1 compressor, we have ω = d -1 . From Corollary 4.3, we know that the time complexity of Alg. 1 is

up to a constant factor. Using Def. 3.1 of t ∗ , we have

where s ∗ ( n ) is the solution of

Let us take s ′ = 12max i ∈ [ n ] max { h i σ 2 nε , ω ˙ τ i } . Using simple bounds, we have

Since s ′ ≥ ω max i ∈ [ n ] ˙ τ i , we get

It means that s ∗ ( n ) ≤ s ′ since s ∗ ( n ) is the solution of (47). Using the properties of max and (46), we get

where we use

Comparison 7.2 . T ∗ = O( T R ) .

Proof. From Sec. 7, we know that

Using Property E.7, we get T ∗ = O( T R ) since ω = d -1 for Rand 1 .

and

K Description of Alg. 5 in the Bidirectional Setting

In this section, we provide the modification of Alg. 4 with the EF21-P mechanism (Gruntkowska et al., 2023). Almost all steps are the same as in Alg. 4 except for the EF21-P mechanism (we mark the main changes with the color).

L Proofs for Alg. 5

Theorem A.2. Let Assumptions 1.1, 1.2, 1.3, 2.2 hold. Choose γ = α 16 L . Then as long as K ≥ 768 L ∆ αε , Bidirectional Shadowheart SGD (Alg. 5) guarantees to find an ε -stationary point.

Proof. In the bidirectional setting, the idea of proof is the same as in Theorem H.3. Let us fix any iteration k ∈ N . The gradient estimator has the same structure as (9) but with w k instead of x k :

Consider that G k is a σ -algebra generated by all random variables from the iterations 0 , . . . , k -1 . Then, given G k , w k is a deterministic vector. Using Lemma H.1 with x k ≡ w k and Lemma H.2, we have

for all k ≥ 0 . It is left to use Theorem E.3 from (Gruntkowska et al., 2023) with B = 2 and C = ε to ensure that min 0 ≤ k ≤ K -1 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε after 768 L ∆ αε iterations.

Corollary A.3. If the broadcast time of C serv is not greater than τ serv , then Bidirectional Shadowheart SGD (Alg. 5) converges after at most

seconds.

Proof. Let us fix an iteration index k ∈ [ n ] . In every iteration, the server broadcasts one compressed vector, every worker calculates b i stochastic gradients and sends m i compressed vectors. Thus, the processing time of each iteration is not greater than

Using the converge rate from Theorem A.2, we finally get (20).

Comparison A.5 . Assume that it takes ˙ τ serv seconds to send one coordinate from the server to the workers. If we take K ≥ min { d, t ∗ ( ω, σ 2 / ε , [ h i , τ i ] n 1 ) / ˙ τ serv } in Top K, then T ∗ , serv = O( T ∗ ) .

Proof. From the assumption, we have τ serv = K ˙ τ serv and τ full serv = d ˙ τ serv . For Top K , α ≥ K / d . Therefore, up to a constant factor, we obtain

Also, up to a constant factor, we have

Therefore, T ∗ , serv = O( T ∗ ) .

M Development of Adaptive Shadowheart SGD

Algorithm 7 Adaptive Shadowheart SGD 1: Input: starting point x 0 , stepsize γ , the ratio σ 2 / ε 2: for k = 0 , 1 , . . . , K -1 do 3: Run Alg. 8 in all workers 4: Broadcast x k to the workers 5: Init l i = 0 for all i ∈ [ n ] 6: while   ∑ i : l i > 0 ( l i ∑ j =1 ( ω l 2 i m ij + ωσ 2 l 3 i m ij ε ) + σ 2 l i ε ) -1   -1 > 1 4 do 7: Receive C i,l i ,m i,l i ( g i ) , l i and m i,l i from some worker (we indicate this worker with i ) 8: if m i,l i = 1 and l i > 1 then 9: ¯ g i = ¯ g i + 1 m i, ( l i -1) ˆ g i and ˆ g i = 0 10: end if 11: ˆ g i = ˆ g i + C i,l i ,m i,l i ( g i ) 12: end while 13: Init g k = 0 14: for i ∈ [ n ] : l i > 0 do 15: ¯ g i = ¯ g i + 1 m i,l i ˆ g i 16: w i ( a ) = ( ∑ l i j =1 ω m ij + ∑ l i j =1 ωσ 2 l i m ij ε + l i σ 2 ε ) -1 17: g k = g k + w i ¯ g i 18: end for 19: g k = g k / ( ∑ i : l i > 0 w i ∑ l i j =1 j ) 20: x k +1 = x k -γg k 21: end for a ) : If ω = 0 and σ 2 ε = 0 , then w i = 1

In this section, we design a new method that, unlike Alg. 1 and 4, does not require the bounds on computations times. It automatically understands when to stop the collection of compressed vectors in g k .

Let us consider Alg. 7, which we call Adaptive Shadowheart SGD. It implements the following gradient estimator:

The idea is that each worker calculate and send compressed vectors in parallel : while the next stochastic gradients ∇ f ( x k ; ξ k i,j +1 ) are calculating, the workers are sending C ij · ( ∑ j r =1 ∇ f ( x k ; ξ k ir ) ) to server. The main difficulty is to understand when to stop. It turns out that it is sufficient to wait for the moment when the condition in Line 6 of Alg. 7 does not hold. For this method, we can prove the following guarantees.

Theorem M.1. Let Assumptions 1.1, 1.2, 1.3, 2.2 hold. Let us take γ = 1 2 L in Alg. 7. Then for all iterations K ≥ 16 L ∆ ε , Alg. 7 guarantees that 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

Corollary 4.6. If the computation and communication times are positive, the time complexity of Alg. 7 is L ∆ ε × t ∗ ( ω, σ 2 / ε , [max { h i , τ i } , min { τ i r i , max { h i , τ i }} ] n 1 ) up to a constant factor, where r i is defined in Def. 4.5.

N Proofs for Alg. 7

Lemma N.1. Consider that Assumptions 1.3 and 2.2 hold. Then the gradient estimator (48) with the parameters from Alg. 7 is unbiased and

Proof. Alg. 7 implements the gradient estimator (48). Note that since C ijp ∈ U ( ω ) , then 1 m ij ∑ m ij p =1 C ijp ∈ U ( ω / m ij ) . Therefore, we can use Lemma G.1 with ω ij = ω / m ij , b ij = j, w ij = w i , and m i = l i , and get

Since ∑ l i j =1 ∑ l i p =1 min { j, p } ≤ l 3 i , we have

We add nonnegative terms to the last inequality to obtain

Using ∑ l i j =1 j ≥ l 2 i 2 , we obtain

In the last two sums, we bound the terms j with l i to get

It is left to use the choice of the weights w i to obtain (49).

Theorem M.1. Let Assumptions 1.1, 1.2, 1.3, 2.2 hold. Let us take γ = 1 2 L in Alg. 7. Then for all iterations K ≥ 16 L ∆ ε , Alg. 7 guarantees that 1 K ∑ K -1 k =0 E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] ≤ ε.

Proof. The proof of this theorem is very close to the proof of Theorem H.3. Let us fix any iteration k ∈ N . Consider that G k is a σ -algebra generated by g 0 , . . . , g k -1 . Then, given G k , x k is a deterministic vector. Using Lemma N.1, we have

The algorithm is constructed in such a way that the first bracket in the last inequality is less or equal to 1 (see Line 6 in Alg. 7). Thus

for all k ≥ 0 . It is left to use the standard SGD analysis from Theorem I.1 with B = 1 and C = ε to ensure that the algorithm converges after 16 L ∆ ε iterations.

Corollary 4.6. If the computation and communication times are positive, the time complexity of Alg. 7 is L ∆ ε × t ∗ ( ω, σ 2 / ε , [max { h i , τ i } , min { τ i r i , max { h i , τ i }} ] n 1 ) up to a constant factor, where r i is defined in Def. 4.5.

Proof. Let us fix an iteration and take k ∈ [ K ] . It is sufficient to find a time required to send enough compressed vectors such that the inequality

holds. As soon as this inequality holds, the algorithm stops the loop in Line 6 from Alg. 7. Then the upper bound on the time complexity equals to the number of iterations × the upper bound on the time of each iteration. The previous inequality is equivalent to

Let us show that

is a sufficient time such that (50) holds.

By the definition of the equilibrium time t ∗ , in order to apply this mapping, we first have to find a permutation π that sorts the input pairs (max { h i , τ i } , min { τ i r i , max { h i , τ i }} ) by

This term equals to max { h i , τ i } . Without loss of generality, we assume that the sequence max { h i , τ i } is sorted, thus π i = i for all i ∈ [ n ] . Therefore, we have

where s ∗ ( j ) is the solution of the equation

for all j ∈ [ n ] .

Let us define j ∗ as the smallest by index minimizer in (51). Then

Assume that l i is the number of iterations (the number of calculated stochastic gradients) that the i th worker does by the time t ′ . Since t ′ ≥ 4 max { h j ∗ , τ j ∗ } and the workers are sorted by max { h i , τ i } , we have t ′ ≥ 2 ( h i + τ i ) for all i ≤ j ∗ . Therefore, for all i ≤ j ∗ , the i th worker will have time to calculate and send at least one compressed vector, i.e., l i ≥ 1 for i ≤ j ∗ .

Next, the i th worker requires at most τ i seconds to send a compressed vector and it waits for at least one calculated gradient. Consider that the computation time of the j th stochastic gradient in the k th iteration equals to h k ij . Thus t ′ 2 ≤ ∑ l i j =1 ( h k ij + τ i ) . Indeed, if t ′ 2 > ∑ l i j =1 ( h k ij + τ i ) , then the i th worker will have time to calculate and send at least one more compressed vector because t ′ 2 ≥ 2 max { h i , τ i } ≥ h i + τ i for all i ≤ j ∗ . It would contradict the definition of l i . Therefore, we have

At the same time, by the definition of l i , we have

because h min > 0 is the smallest possible calculating time. Therefore, we have

and

where t max is defined in Def. 4.5 ( t max ≥ t ′ ). Since l i ≥ 1 for all i ≤ j ∗ , we get

Using (53), we obtain

For every j th stochastic gradient, the i th worker sends at least one compressed vector or ⌊ h k ij τ i ⌋ compressed vectors because it is possible that τ i ≤ h k ij , then the worker will have time to send more than one compressed vector. Therefore, we have

and

Using the last inequality and (53), we get

It is clear that T ≤ 8 max { max j ∈ [ l i ] h k ij ,τ i } t ′ . If τ i < min j ∈ [ l i ] h k ij , then T = 8 τ i t ′ × max j ∈ [ l i ] h k ij min j ∈ [ l i ] h k ij . If τ i ≥ min j ∈ [ l i ] h k ij and τ i < max j ∈ [ l i ] h k ij , then T = 8 max j ∈ [ l i ] h k ij t ′ ≤ 8 τ i t ′ × max j ∈ [ l i ] h k ij min j ∈ [ l i ] h k ij . If τ i ≥ max j ∈ [ l i ] h k ij , then T = 8 τ i t ′ ≤ 8 τ i t ′ × max j ∈ [ l i ] h k ij min j ∈ [ l i ] h k ij .

Thus, we have

where we use the definition of r i . Using the last inequality and l i ≥ t ′ 4 max { h i ,τ i } , we have

where we use (53) and max j ∈ [ l i ] h k ij ≤ h i . One can substitute the bounds on A and B to (54) and obtain

It is left to use the definition of s ∗ ( j ∗ ) (see (52)) to obtain that H ≥ s ∗ ( j ∗ ) × 1 s ∗ ( j ∗ ) = 1 .

It means that after at most t ′ seconds, we can ensure that the algorithm will finish the loop in Line 6 from Alg. 7. In the view of Theorem M.1, the time complexity is less or equal to K × t ′ .

O Construction of the Lower Bound

We prove the lower bound by generalizing the time multiple oracles protocol from (Tyurin and Richtárik, 2023c). Note that in the classical approaches (Nemirovskij and Yudin, 1983; Carmon et al., 2020; Arjevani et al., 2022; Nesterov, 2018), the researchers bound the number of oracle calls required to find an ε -solution. Our approach is based on the idea from (Tyurin and Richtárik, 2023c), where the authors propose to bound the time required to find an ε -solution. We refer to a detailed explanation to (Tyurin and Richtárik, 2023c)[Sections 3-6].

First, we define an oracle that emulates the process of computing stochastic gradients or the process of sending a compressed vector (Tyurin and Richtárik, 2023c)[Section 4]:

where ξ ∼ D , g is an arbitrary mapping such that g : R d × S → R d , and S is the sample space of a distribution D . Next, we define the time multiple oracles protocol with compression :

Protocol 9 Time Multiple Oracles Protocol with Compression

̸

In this protocol, the server via B k returns a new point x k , and broadcasts it to the i k +1 th worker. Then, the worker calls the oracle O i k +1 that calculates stochastic gradients. Next, the oracle returns the vector g k +1 i k +1 , and the worker processes it with N k i k +1 . Finally, the worker sends g k +1 pre to the oracle ˆ C i k +1 that sends compressed vectors to the server. Using the parameters c ∇ f,k +1 and c C ,k +1 , it can decide if it wants to start/stop the process of a gradient calculation and the process of communicating a compressed vector (See Sec. F in (Tyurin and Richtárik, 2023c)). As far as we know, all centralized distributed optimization methods can be described by Protocol 9, including Minibatch SGD , QSGD , Asynchronous SGD , Rennala SGD , and Shadowheart SGD .

We consider the standard function class from the optimization literature (Nesterov, 2018; Arjevani et al., 2022; Carmon et al., 2020):

Definition O.1 (Function Class F ∆ ,L ) . We assume that a function f : R d → R is differentiable, ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ ∀ x, y ∈ R d , ( L -smooth) and f (0) -inf x ∈ R d f ( x ) ≤ ∆ ( ∆ -bounded). The set of all functions with such properties we denote by F ∆ ,L .

Next, we define the class of algorithms that we analyze.

Definition O.2 (Algorithm Class A zr ) . Let us consider Protocol 9. We say that the sequence of tuples of mappings A = { ( B k , N k 1 , . . . , N k n ) } ∞ k =0 is a zero-respecting algorithm, if

For all k ≥ 1 and and g 1 , . . . , g k ∈ R d , t k +1 ≥ t k , where t k +1 and t k are defined as ( t k +1 , . . . ) = B k ( g 1 , . . . , g k ) and ( t k , . . . ) = B k -1 ( g 1 , . . . , g k -1 ) .

̸

The set of all algorithms with this properties we define as A zr .

The properties 1 and 3 are only required to define the domains of the mappings. The property 4 ensures that these mappings are zero-respecting (Arjevani et al., 2022). The property 2 is explained in (Tyurin and Richtárik, 2023c)[Section 4, Definition 4.1]. It ensures that our algorithm does not 'travel into the past'.

The following oracle class is the same as in (Tyurin and Richtárik, 2023c). For any f ∈ F ∆ ,L , it returns n oracles that require h 1 , . . . , h n seconds to calculate a stochastic gradient. These oracles emulate the real behavior where the workers have different processing times.

Definition O.3 (Computation Oracle Class O σ 2 h 1 ,...,h n ) . Let us consider an oracle class such that, for any f ∈ F ∆ ,L , it returns oracles O i = O ∇ f, D ∇ f i h i for all i ∈ [ n ] , where ∇ f ( x ; ξ ) is an unbiased σ 2 -variance-bounded mapping (see Assumption 1.3). The oracles O ∇ f, D ∇ f i h i are defined in (55). We define such oracle class as O σ 2 h 1 ,...,h n .

The following oracle class emulates the behavior of compressors. It returns n oracles that require τ 1 , . . . , τ n seconds to send a compressed vector to the server.

Definition O.4 (Communication Oracle Class U ω τ 1 ,...,τ n ) . Let us consider an oracle class such that, it returns oracles ˆ C i = O C , D C i τ i for all i ∈ [ n ] , where C is an unbiased compressor with a parameter ω, i.e., C ∈ U ( ω ) (see Def. 2.1). The oracles O C , D C i τ i are defined in (55). We define such oracle class as U ω τ 1 ,...,τ n .

Finally, we present our lower bound theorem:

Theorem O.5. Let us consider Protocol 9. We take any h i > 0 , τ i > 0 for all i ∈ [ n ] , ω ≥ 0 , L, ∆ , ε, σ 2 > 0 such that ε < c 1 L ∆ and ω +1 ≤ T, 11 where T = ⌊ ∆ L c 2 ε ⌋ is the dimension of the construction. For any algorithm A ∈ A zr , there exists a function f ∈ F ∆ ,L , computation oracles ( O 1 , . . . , O n ) ∈ O σ 2 h 1 ,...,h n ( f ) , and communication oracles ( ˆ C 1 , . . . , ˆ C n ) ∈ U ω τ 1 ,...,τ n , 12 such that E [ inf k ∈ S t ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] > ε, where S t := { k ∈ N 0 | t k ≤ t } and

The quantities c 1 , c 2 , and c 3 are universal constants. The sequences x k and t k are defined in Protocol 9.

P Proof of Theorem O.5

P.1 The 'Worst Case' Function

Let us consider the 'worst case' function, which is a standard function to obtain lower bounds in the nonconvex world. We define

̸

In our proofs, we use the construction from (Carmon et al., 2020; Arjevani et al., 2022). For any T ∈ N , the authors define

where

The main property of the function F T ( x ) is that its gradients are large unless prog ( x ) ≥ T. Lemma P.1 (Carmon et al. (2020); Arjevani et al. (2022)) . The function F T satisfies:

The function F T is l 1 -smooth, where l 1 = 152 .

For all x ∈ R T , ∥∇ F T ( x ) ∥ ∞ ≤ γ ∞ , where γ ∞ = 23 .

For all x ∈ R T , prog ( ∇ F T ( x )) ≤ prog ( x ) + 1 .

For all x ∈ R T , if prog ( x ) < T, then ∥∇ F T ( x ) ∥ > 1 .

Theorem O.5. Let us consider Protocol 9. We take any h i > 0 , τ i > 0 for all i ∈ [ n ] , ω ≥ 0 , L, ∆ , ε, σ 2 > 0 such that ε < c 1 L ∆ and ω +1 ≤ T, 13 where T = ⌊ ∆ L c 2 ε ⌋ is the dimension of the construction. For any algorithm A ∈ A zr , there exists a function f ∈ F ∆ ,L , computation oracles ( O 1 , . . . , O n ) ∈ O σ 2 h 1 ,...,h n ( f ) , and communication oracles ( ˆ C 1 , . . . , ˆ C n ) ∈ U ω τ 1 ,...,τ n , 14 such that E [ inf k ∈ S t ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] > ε, where S t := { k ∈ N 0 | t k ≤ t } and

The quantities c 1 , c 2 , and c 3 are universal constants. The sequences x k and t k are defined in Protocol 9.

Proof.

Without loss of generality, we assume that the workers are sorted by max { h i , τ i } : max { h 1 , τ 1 } ≤ · · · ≤ max { h n , τ n } . ( Step 1 : f ∈ F ∆ ,L )

Let us fix λ > 0 and take the function f ( x ) := Lλ 2 l 1 F T ( x λ ) , where the function F T is defined in Sec. P.1. Tyurin and Richtárik (2023c)[Sec. D.2, Proof of Thm. 6.4] show that the function f is L -smooth and f (0) -inf x ∈ R T f ( x ) ≤ ∆ if

Thus, we have f ∈ F ∆ ,L .

( Step 2 : Oracle Class) Let us construct a stochastic gradient mapping. For our lower bound, we take

and D ∇ f i = Bernoulli ( p ) for all i ∈ [ n ] , where p ∈ (0 , 1] . We denote [ x ] j as the j th index of a vector x ∈ R T . Let us take

Then Tyurin and Richtárik (2023c)[Sec. D.2, Proof of Thm. 6.4] show that this mapping is unbiased and σ 2 -variance-bounded.

( Step 3 : Compression Operator) In our construction, we take the Rand K compressor (outputs K random values of an input vector without replacement , scaled by T / K (Def. D.1)). From Theorem D.2, we know that C is unbiased and T K -1 -variance bounded, i.e.,

where

and S is an uniformly random subset of [ T ] without replacement. It is sufficient to take K = ⌈ T ω +1 ⌉ to ensure that C ∈ U ( ω ) . Let us define p ω := K T . We take mutually independent distributions D C i that generate random subsets S described above.

( Step 4 : Analysis of Protocol)

Let us take

to ensure that ∥∇ f ( x ) ∥ 2 = L 2 λ 2 l 2 1 ∥ ∥ ∇ F T ( x λ ) ∥ ∥ 2 > 2 ε ✶ [ prog ( x ) < T ] for all x ∈ R T , where we use Lemma P.1. Thus and

Protocol 9 generates the sequence { x k } ∞ k =0 . We have

Using Lemma P.2 with δ = 1 / 2 and (58), we obtain for

By the assumption of the theorem, we have ω +1 ≤ T. Therefore, we get the series of inequalities:

and, using Properties 6.1 and E.1, we have

Thus, we can take

P.1.1 Proof of Lemma P.2

Lemma P.2. Let us fix T, T ′ ∈ N such that T ≤ T ′ , consider Protocol 9 with an algorithm A ∈ A zr , a differentiable function f : R T ′ → R such that prog ( ∇ f ( x )) ≤ prog ( x )+1 for all x ∈ domain ( f ) .

We take stochastic oracles O i = O ∇ f, D ∇ f i h i with the distributions D ∇ f i = Bernoulli ( p σ ) , p σ ∈ (0 , 1] , h i > 0 , and the mappings

We take compression oracles ˆ C i = O C , D C i τ i with the distributions D C i = uniform ( K,T ′ ) (= 'uniformly random subset of [ T ′ ] of the size K without replacement') and the mappings

τ i > 0 . We define p ω := K T ′ . We assume that the workers are sorted by max { h m , τ m } : max { h 1 , τ 1 } ≤ · · · ≤ max { h n , τ n } . With probability not less than 1 -δ, the following inequality holds:

for

where S t := { k ∈ N 0 | t k ≤ t } , the iterates t k and x k are defined in Protocol 9, and t ∗ is the equilibrium time from Def. 3.1.

Proof.

(Part 1): The Construction of Random Variables.

Let us fix t ≥ 0 and define the smallest index k ( i ) of the sequence when the progress prog ( x k ( i ) ) equals i :

If inf k ∈ S t ✶ [ prog ( x k ) < 1 ] < 1 holds, then exists k ∈ S t such that prog ( x k ) = 1 , thus, by the definition of k (1) , t k (1) ≤ t k ≤ t, and k (1) < ∞ . Note that t k (1) is the smallest time when we make progress to the 1 th (first) coordinate.

Since x 0 = 0 and A is a zero-respecting algorithm, the algorithm can return a vector x k with a non-zero first coordinate only if some of returned by the stochastic gradients oracles and compression oracles have the first coordinate not equal to zero. The oracles O i and ˆ C i are constructed in such a way (see (59) and (60)) that they zero out a coordinate based on i.i.d. bernoulli and uniform trials. According to Protocol 9, even if a stochastic oracle returns a non-zero coordinate, it would not mean that the server will get a non-zero coordinate because a subsequent compression oracle also has to return a non-zero coordinate.

Every time when the oracle (55) evaluates g ( s x ; ξ ) , it draws i.i.d. random variables ξ ∼ D . Let us enumerate them:

For the stochastic/computation oracles O i = O ∇ f, D ∇ f i h i , we consider the sequence { ξ m,j } ∞ j =1 , where ξ m,j is a bernoulli random variable drawn in j th call of g ( s x ; ξ ) in the m th worker in Line 5 of Protocol 9.

For the compression oracles ˆ C i = O C , D C i τ i , we consider the sequence { S m,j } ∞ j =1 , where S m,j is a uniform random variable drawn in j th call of g ( s x ; ξ ) in the m th worker in Line 7 of Protocol 9.

Let us define the following useful random variables based on previous definitions. We define

where

For all m ∈ [ n ] , j ≥ 1 , b η m,j ∈ N ∪ {∞} is the first index of the sequence { ξ m,j } ∞ j =1 that started calculating in (55) in the stochastic oracle O m in or after the iteration k ( j -1) .

For all m ∈ [ n ] , j ≥ 1 , b µ m,j ∈ N ∪ {∞} is the first index of the sequence { S m,j } ∞ j =1 that started calculating in (55) in the compression oracle ˆ C m in or after the iteration k ( j -1) .

For all m ∈ [ n ] , j ≥ 1 , if b η m,j = ∞ , then e η m,j = ∞ . For all m ∈ [ n ] , j ≥ 1 , if b η m,j < ∞ , then e η m,j ∈ N ∪ {∞} is the first index of the sequence { ξ m,j } ∞ j =1 that started calculating in (55) in the stochastic oracle O m in or after the iteration k ( j -1) and the first moment when ξ m, ( i + b η m,j -1) = 1 for some i ≥ 1 .

For all m ∈ [ n ] , j ≥ 1 , if b η m,j = ∞ , then e µ m,j = ∞ . For all m ∈ [ n ] , j ≥ 1 , if b η m,j < ∞ , then e µ m,j ∈ N ∪ {∞} is the first index of the sequence { S m,j } ∞ j =1 that started calculating in (55) in the compression oracle ˆ C m in or after the iteration k ( j -1) and the first moment when ξ m, ( i + b η m,j -1) = 1 for some i ≥ 1 .

It possible that such indexes do not exist, then we take b η m,j = ∞ , b µ m,j = ∞ , e η m,j = ∞ , or e µ m,j = ∞ , accordingly. By the construction, e η m,j ≥ b η m,j and e µ m,j ≥ b µ m,j .

Let us clarify the definitions. At the beginning x 0 = 0 , thus k (0) = 0 . It would mean that the first index of the sequence { ξ m,j } ∞ j =1 , when the worker evaluates g ( s x ; ξ ) in (55), simply equals b η m, 1 = 1 or b η m, 1 = ∞ (by the definition, it equals to ∞ if the oracle was never called). Assume that b η m, 1 = 1 , then η m, 1 = inf { i | ξ m,i = 1 and i ∈ N } is the first time when the oracle draws a 'successful' random bernoulli trial. This random variable is distributed according to the geometric distribution. Then, e η m, 1 can be equal to η m, 1 +1 or ∞ . At some (random) iteration k (1) , the algorithm A can get the first non-zero coordinate through g k , then b µ m, 2 is the next index of the sequence { ξ m,j } ∞ j =1 that started calculating in (55). 15

The server gets a non-zero coordinate if at least one worker draws a successful bernoulli trial, and this coordinate belongs to a set generated by the uniform distribution. It takes h i seconds to generate one bernoulli trial and τ i seconds to generate one uniform trial.

Then, if := inf k ∈ S t ✶ [ prog ( x k ) < 1 ] < 1 holds, then because h m η m, 1 + τ m µ m, 1 is the time required to generate η m, 1 bernoulli and µ m, 1 uniform trials. In other words, the algorithm can not progress to the next coordinate before the moment when at least one worker generates 'successful' bernoulli and uniform trials.

Using the same reasoning, t k ( j ) ≥ t k ( j -1) + ˆ t j , where

Combining the observations, if inf k ∈ S t ✶ [ prog ( x k ) < T ] < 1 holds, then ∑ T j =1 min j ∈ [ n ] ( h i η i,j + τ i µ i,j ) ≤ t k ( T ) ≤ t. Thus

(Part 2): The Chernoff Method

Let us fix s ≥ 0 and ˆ t ≥ 0 . Using the Chernoff method, we have

Let us bound the expected value separately. For all j ∈ [ T ] , let us define G j as the σ -algebra generated by random variables

The σ -algebra G T contains all information about the random variables before the moment when prog ( x k ) = T -1 . Then, we have

Note that if the random variables from (64) are 'fixed,' then ˆ t i is deterministic for all i ∈ [ T -1] because ˆ t i is a deterministic function of (64), and does not depend on other subsequent random variables.

Let us show it using a contradiction proof. Without the loss of generality, assume that ˆ t T -1 depends on ξ 1 ,b η 1 ,T ̸∈ (64) . By the definition of b η 1 ,T , it would mean that the first time when the server can get a vector g k with a non-zero coordinate in the index T -1 is after the moment t k ( T -1) . We get a contradiction since t k ( T -1) is the first time when the algorithm return an iterate with a non-zero coordinate in the index T -1 .

Thus, ˆ t i is G T -measurable for all i ∈ [ T -1] and

Let us fix t ′ ≥ 0 , then, since ˆ t T ≥ 0 , we have

We now use the result of the following lemma that we prove separately.

Lemma P.3. Using the notations from the proof of Lemma P.2, we have

for all j ∈ [ T ] .

Let us temporarily define

We substitute (67) to (66) and (65) to obtain

Next, using (63), we get

Let us take s = 1 / t ′ , and get

Let us recall the definition of p ′ :

where we define q m := ( 1 -(1 -p ω ) ⌊ t ′ τm ⌋ )( 1 -(1 -p σ ) ⌊ t ′ hm ⌋ ) ∈ [0 , 1] . Using Lemma C.1, we have

Using the inequality 16 1 -(1 -p ) m ≤ pm for all p ∈ [0 , 1] and m ∈ N 0 , we can get the following three inequalities:

and

Therefore, we get

and

Now, we have to take the right t ′ . Assume that s ∗ ( j ) is the solution of

j

j

∗

= inf

{

j

and

∈

[

n

]

|

s

∗

(

)

<

{

h

j

+1

Then we take t ′ = 1 24 s ∗ ( j ∗ ) . If j ∗ = 1 , then

Otherwise, if j > 1 , since s ( j ) ≤ s ( j -1) , we have

By the definitions of s ∗ ( j ∗ -1) and j ∗ , we have

Therefore, we get

Using s ∗ ( j ∗ ) ≥ 1 2 max { h j ∗ , τ j ∗ } , we obtain

We use the last inequality later. Let us return to the inequality (69). Using the definition of j ∗ , we obtain ⌊ t ′ τ m ⌋ = 0 or ⌊ t ′ h m ⌋ = 0 for all m>j ∗ and

, τ

j

+1

}} ∈

[

n

]

,

{

h

n

+1

, τ

n

+1

} ≡ ∞

.

where we use ⌊ x ⌋ ≤ x for all x ≥ 0 . Using max { x, y, z } ≥ 1 3 ( x + y + z ) for all x, y, z ≥ 0 , we have

Since t ′ = 1 24 s ∗ ( j ∗ ) , we obtain

Note that s ∗ ( j ∗ ) is the solution of (70), thus, we get

Substituting this inequality to (68), we obtain

Recall that the definition of t ′ . Using (71), we have

The last term equals to the equilibrium time t ∗ ( 1 / p ω , 1 / p σ , h 1 , τ 1 , . . . , h n , τ n ) from Def. 3.1 since the pairs ( h j , τ j ) are sorted by max { h j , τ j } . Thus, we obtain

for

P.2 Proof of Lemma P.3

Lemma P.3. Using the notations from the proof of Lemma P.2, we have

for all j ∈

Proof. We prove the result for j = T. The proofs for the cases 1 ≤ j < T are the same. We consider the conditional probability

For ˆ t ≤ t ′ ( T 2 +log δ ) , we have

Finally, we obtain

Let us consider the σ -algebra H T generated by (64) with j = T and

By the construction, e η m,T ≥ b η m,T and e µ m,T ≥ b µ m,T . Since G T ⊆ H T , we have

Since G T ⊆ H T , then b µ m,T is H T -measurable. By the definition of e η m,T , η m,T are H T -measurable. Let us show it using a contradiction proof. Without the loss of generality, assume that η m,T depends on ξ 1 ,e η m,T ̸∈ (73) . It would mean that the first time, when ξ m,i = 1 after the iteration k ( T -1) , happens with i ≥ e η m,T . At the same time, by the definition of e η m,T , there exists i < e η m,T such that ξ m,i = 1 calculated after the iteration k ( T -1) . We get a contradiction. Given H T , µ m,T are mutually independent since { S m,j } ∞ j =1 are mutually independent and e µ m,T are H T -measurable. Therefore, we have

Let us consider the probability P ( h m η m,T + τ m µ m,T > t ′ |H T ) :

✶ ✶ because the event { h m η m,T ≤ t ′ } ⋂ { τ m µ m,T ≤ t ′ } follows from { h m η m,T + τ m µ m,T ≤ t ′ } . Since η m,T is H T -measurable, we have

✶ Let us consider the probability P ( τ m µ m,T ≤ t ′ |H T ) . Given H T , if e µ m,T = ∞ , then µ m,T = ∞ and P ( τ m µ m,T ≤ t ′ |H T ) = 0 . Otherwise, if e µ m,T = e < ∞ , then

and it is distributed with the geometric distribution with p ω . Thus, we have 17

17 We implicitly assume that 1 -(1 -p ω ) ⌊ t ′ τm ⌋ = 0 if p ω = 1 and ⌊ t ′ τ m ⌋ = 0 , because if t ′ < τ m , then P ( τ m µ m,T ≤ t ′ ) = 0 for the r.v. µ m,T from the geometric distribution for all p ω ∈ (0 , 1] .

because the probability that j th coordinate belongs to S m, ( i + e -1) equals to p ω . We substitute this inequality to (75) and get

Given G T , η m,T are independent because b η m,T are G T -measurable. Thus

Using the same reasoning as with µ m,T , we get

because, given G T , η m,T equals ∞ or a random variable distributed according to the geometric distribution with p σ . Therefore, we obtain

P.3 Another Construction

In the proof of Theorem O.5, we could use the following construction:

( Step 3 : Compression Operator) Let us define p ω := 1 ω +1 . In our construction, we take a compressor that outputs random coordinates of an input vector, scaled by 1 / p ω , where each coordinate is taken with the probability p ω . Each worker has access to the independent compressed realizations of a such compressor. More formally, we assume that

where S is a random subset of [ T ] , where each element from [ T ] appears with the probability p ω independently . Then

and

Thus, we have C ∈ U ( ω ) . We take mutually independent distributions D C i that generate random subsets S described above.

This construction is also valid and does not require the assumption ω +1 ≲ L ∆ / ε . However, unlike the construction from Theorem O.5, this construction can return a random number of non-zero coordinates.

Q Experiments

The experiments were prepared in Python. The distributed environment was emulated on machines with Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz and 64 cores.

Q.1 Experiments with Logistic Regression

0

10000

20000

30000

40000

50000

times (seconds)

2.6 × 10

1

2.8 × 10

1

3×10 1

3.2 × 10

1

3.4 × 10

1

f

(

x t

)

f

(

x

*

)

SGD

one

: Step size: 0.03125

Minibatch SGD: Step size: 1.0

QSGD: Step size: 0.5

2

/

: 80

(a) Experiment with computation speeds

h

i

=

√

i

and

high

communications speeds

˙

τ

i

=

√

i/d

0

100000

200000

300000

400000

500000

600000

700000

800000

times (seconds)

3×10 1

4×10 1

f

(

x t

)

f

(

x

*

)

SGD

one

: Step size: 0.015625

Minibatch SGD: Step size: 1.0

QSGD: Step size: 0.25

2

/

: 20

(b) Experiment with computation speeds h i = √ i and low communications speeds ˙ τ i = √ i/d 1 / 2

0

25000

50000

75000

100000

125000

150000

175000

200000

times (seconds)

2.4 × 10

1

2.6 × 10

1

2.8 × 10

1

3×10 1

3.2 × 10

1

3.4 × 10

1

f

(

x t

)

f

(

x

*

)

SGD

one

: Step size: 0.03125

Minibatch SGD: Step size: 1.0

QSGD: Step size: 0.25

2

/

: 20

(c) Experiment with computation speeds h i = √ i and medium communications speeds ˙ τ i = √ i/d 3 / 4

We start our experiments with a practical setup: a logistic regression problem with the MNIST dataset (LeCun et al., 2010). The optimization steps of algorithms are emulated in Python, where we fix the number of workers to n = 100 , each worker has access to the MNIST dataset and sample 4 samples when calculating a stochastic gradient. We compare Shadowheart SGD with QSGD , Asynchronous SGD (we implement the version from (Koloskova et al., 2022)), Minibatch SGD , and SGDone . SGDone is the method described in Sec. 7, where SGD is run on the fastest worker locally. In Shadowheart SGD , we fine-tune the parameter σ 2 / ε ∈ { 1 , 5 , 10 , 20 , 30 , 40 , 80 , 120 , 150 , 200 } . In all the methods, we also finetune the step sizes. The dimension of the problem in the logistic regression problem is d = 7850 . In Shadowheart SGD and QSGD , we take Rand K with K = 700 .

We assume that the computations time h i of a stochastic gradient equals to √ i seconds in the i th worker. We consider three communication time setups, where it takes ˙ τ i seconds to send one coordinate from the i th worker to the server and

˙ τ i = √ i/d (High-speed communications),

˙ τ i = √ i/d 3 / 4 (Medium-speed communications),

˙ τ i = √ i/d 1 / 2 (Low-speed communications).

In the high-speed regime, the communication between the server and the worker is relatively fast. At the same time, in low-speed regimes, communication is expensive. We are ready to present the results of our experiments in Fig. 1a, 1c, and 1b.

In Figure 1a, one can see that Shadowheart SGD , Asynchronous SGD , and Minibatch SGD are the fastest because it is not expensive to send a non-compressed vector in the 'high-speed communications' regime. SGDone is the slowest since it utilizes only one worker.

Next, we analyze Figure 1b, where Shadowheart SGD and SGDone have the best performance. SGDone improves the convergence relative to other methods because the communication speed is much slower than in Figure 1a, and it is expensive to send a non-compressed vector.

One can see that Shadowheart SGD is very robust to all regimes and has one of the best convergence rates in all experiments. Notably, in the 'medium-speed communications' regime, where it is still expensive to send a non-compressed vector, our new method converges faster than other baseline methods.

Q.2 Experiments with quadratic optimization tasks and multiplicative noise

In real machine learning tasks, it is not easy to control noise. Thus, we generated synthetic quadratic optimization tasks where we can control the noise of stochastic gradients. In particular, we consider

for all x ∈ R d and take d = 1000 ,

We consider the following stochastic gradients:

where ξ ∼ Bernoulli ( p ) for all i ∈ [ n ] , and p ∈ (0 , 1] . We denote [ x ] j as the j th index of a vector x ∈ R d . In our experiments, we take the starting point x 0 = [ √ d, 0 , . . . , 0] ⊤ and p ∈ { 10 -3 , 10 -4 } ; the smaller p the larger the noise of stochastic gradients.

Q.2.1 Discussion of the experiments from Sec. Q.2.2

Using this setup, in Figures 2, 3, 4, we fix all parameters except one that we vary to understand the dependencies. In all experiments, we observe that Shadowheart SGD is the most robust to input changes among other centralized methods ( QSGD , Asynchronous SGD , Minibatch SGD ) and can converge significantly faster. At the same time, we observe that SGDone can be faster than our method in some setups. It happens in the regimes when communication is expensive (see Figure 2a), which is expected and discussed in Sec. 7. Even if communication is expensive, SGDone starts to slow down relative to other methods when we increase the noise (compare Figures 2b and 2a). The following experiments agree with our theoretical discussion in Sec. 7.

Q.2.2 Plots

In these experiments, we take n = 10000 , p = 10 -3 , h i = √ i, ˙ τ i = √ i/d 3 / 4 as base parameters; in each plot, we vary one parameter.

0

50000

100000

150000

200000

250000

300000

times (seconds)

10

3

10

2

f

(

x t

)

f

(

x

*

)

SGD

one

: Step size: 2.0

Minibatch SGD: Step size: 2.0

QSGD: Step size: 1.0

2

/

: 75

(a)

n

= 10000

, p

= 10

-

3

, h

i

=

√

i,

˙

τ

i

=

√

i/d

3

/

4

0

50000

100000

150000

200000

250000

300000

times (seconds)

10

3

10

2

f

(

x t

)

f

(

x

*

)

SGD

one

: Step size: 2.0

Minibatch SGD: Step size: 2.0

QSGD: Step size: 2.0

2

/

: 200

(b)

n

= 10000

, p

= 10

-

4

, h

i

=

√

i,

˙

τ

i

=

√

i/d

3

/

4

0

50000

100000

150000

200000

250000

300000

times (seconds)

10

3

10

2

f

(

x t

)

f

(

x

*

)

Minibatch SGD: Step size: 2.0

QSGD: Step size: 1.0

2

/

: 150

(a)

n

= 10000

, p

= 10

-

3

, h

i

=

√

i,

˙

τ

i

=

√

i/d

0

50000

100000

150000

200000

250000

300000

times (seconds)

10

3

10

2

f

(

x t

)

f

(

x

*

)

Minibatch SGD: Step size: 2.0

QSGD: Step size: 1.0

2

/

: 75

(b)

n

= 10000

, p

= 10

-

3

, h

i

=

√

i,

˙

τ

i

=

√

i/d

3

/

4

0

50000

100000

150000

200000

250000

300000

times (seconds)

10

3

10

2

f

(

x t

)

f

(

x

*

)

SGD

one

: Step size: 2.0

Minibatch SGD: Step size: 2.0

QSGD: Step size: 1.0

2

/

: 75

(a)

n

= 10000

, p

= 10

-

3

, h

i

=

√

i,

˙

τ

i

=

√

i/d

3

/

4

0

50000

100000

150000

200000

250000

300000

times (seconds)

10

3

10

2

f

(

x t

)

f

(

x

*

)

SGD

one

: Step size: 2.0

Minibatch SGD: Step size: 2.0

QSGD: Step size: 1.0

2

/

: 40

(b)

n

= 10000

, p

= 10

-

3

, h

i

= 1

,

˙

τ

i

=

√

i/d

3

/

4

Q.3 Experiments with quadratic optimization tasks and additive noise

In this section, we consider the same problem as in Sec. Q.2. However, unlike the multiplicative noise, we consider the following additive noise:

where ζ ∼ N (0 , σ 2 ) is a sample from the normal distribution. Be default, we take n = 100 workers, the dimension d = 100 , and use the Rand 1 compressor ( ω = d/ 1 -1 = 99 ), x 0 = [1 , · · · , 1] ⊤ , σ = 10 -1 , ε = 10 -4 ; thus, the ratio σ 2 / ε = 10 2 . For all methods, we choose the step sizes in such a way that they converge to the same neighborhood of the stationary point.

In Figure 5, we sample h k i and ˙ τ k i from the uniform distribution U (0 . 1 , 1) , hence the communication and computation time vary on each iteration for each client. If we increase the number of clients n , Shadowheart SGD improves (Fig. 5) compared to other methods, confirming our theory.

In Figure 6, we can see the similar results with different ratios σ 2 / ε : Shadowheart SGD is much better when the ratio is large (Fig. 6). On the other hand, when σ 2 / ε is small (Fig. 6a) SGDone can be better because, intuitively, we only need a few workers to find the minimum with a small noise (see also Sec. 7).

Next, we perform a series of experiments with different computation time and communication times ratios. We take ˙ τ k i / h k i = c for all i ∈ [ n ] , where c > 0 .

In Figure 7, we take h k i ∼ U (0 . 1 , 1) and ˙ τ k i ∼ c · U (0 . 1 , 1) . Shadowheart SGD is better in the high and medium communication speed regimes (Fig. 7b), when the communication times are not too large. On the other hand, with large c = 10 2 , Shadowheart SGD spends much time on sending gradients to the server, whereas SGDone does not spend time on communication and does not compress (Fig. 7c). Similar to Figure 7, we obtain the results with h k i = √ i and ˙ τ k i = c · √ i in Figure 8.

Q.3.1 Plots

0

1000

2000

3000

4000

5000

Time,

t

k

10

3

10

2

10

1

f

(

x k

)

2

SGD

one

Minibatch SGD

QSGD

(a)

n

= 10

0

1000

2000

3000

4000

5000

Time,

t

k

10

3

10

2

10

1

f

(

x k

)

2

SGD

one

Minibatch SGD

QSGD

(b)

n

= 10

2

0

1000

2000

3000

4000

5000

Time,

t

k

10

3

10

2

10

1

f

(

x k

)

2

SGD

one

Minibatch SGD

QSGD

(c)

n

= 10

3

0

500

1000

1500

2000

2500

3000

Time,

t

k

10

4

10

3

10

2

10

1

f

(

x k

)

2

SGD

one

Minibatch SGD

QSGD

(a)

σ

2

/

ε

= 1

0

500

1000

1500

2000

2500

3000

Time,

t

k

10

3

10

2

10

1

f

(

x k

)

2

SGD

one

Minibatch SGD

QSGD

(b)

σ

2

/

ε

= 10

0

2000

4000

6000

8000

10000

Time,

t

k

10

3

10

2

10

1

f

(

x k

)

2

SGD

one

Minibatch SGD

QSGD

(c)

σ

2

/

ε

= 10

2

0

200

400

600

800

1000

Time,

t

k

10

3

10

2

10

1

f

(

x k

)

2

SGD

one

Minibatch SGD

QSGD

(a)

c

= 10

-

1

0

200

400

600

800

1000

Time,

t

k

10

3

10

2

10

1

f

(

x k

)

2

SGD

one

Minibatch SGD

QSGD

(b)

c

= 1

0

1000

2000

3000

4000

5000

Time,

t

k

10

3

10

2

10

1

f

(

x k

)

2

SGD

one

Minibatch SGD

QSGD

(c)

c

= 10

2

0

1000

2000

3000

4000

5000

Time,

t

k

10

2

10

1

f

(

x k

)

2

SGD

one

Minibatch SGD

QSGD

(a)

c

= 10

-

2

0

1000

2000

3000

4000

5000

Time,

t

k

10

3

10

2

10

1

f

(

x k

)

2

SGD

one

Minibatch SGD

QSGD

(b)

c

= 10

-

1

0

1000

2000

3000

4000

5000

Time,

t

k

10

3

10

2

10

1

f

(

x k

)

2

SGD

one

Minibatch SGD

QSGD

(c)

c

= 1

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Section 3 and Table 1

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Sections 4.1, 4.2, and 7.2

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Section 1 and Appendix

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section Q

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In the supplementary material

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section Q

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: While each particular experiment in every plot from Section Q was run with one seed, the amount of provided experiments and considered settings can give a high confidence in our judgments that the experimental part supports the theoretical part, which is our main contribution.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section Q

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work considers a mathematical problem from the machine learning domain.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work considers a mathematical problem from the machine learning domain.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work considers a mathematical problem from the machine learning domain.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Section Q

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code for the experiments is in the supplementary materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studied distributed asynchronous SGD with heterogeneous communication and computation times for non-convex stochastic optimization problems. In particular, the paper proposed a new algorithm called Shadowheart SGD and analyzed his time complexity. The time complexity is proven to be optimal in the famly of centralized methods with compressed communication by providing a lower bound. This time complexity is also shown to be better than those in the literature.\n\nSTRENGTHS:\n1. The paper considered a general scenario of distributed/federated learning by considering asynchronous SGD, arbitrary computation and communication time, and unbiased compression. \n\n2. The paper proposed a new and non-trivial algorithm called Shadowheart SGD, which is optimal in terms of time complexity. \n\n3. The paper introduced equilibrium time, which is a key parameter to characterize the number of computations and communications in Shadowheart SGD. \n\n4. The the paper also proposed Adaptive Showheart SGD, which does not require the knowledge of equilibrium time, the communication and the computation times.\n\nWEAKNESSES:\n1. The paper was not well written and is very hard to read. It reads like a summary of the results that are in the appendix. In addition, the intuitive explanations are very few and many notations were not well explained. \n\n2. The paper considered only IID data distribution. This almost rules out the application of the algorithm in federated learning. The results are nice contributions to the existing results but it seems minor. \n\n3. It lacks experimental results. All the experiments use either logistic regression with MNIST dataset or quadratic optimization and multiplicative noise. Since the paper proposed a new algorithm, i believe the experiments need to be much more extensive.\n\nQUESTIONS:\n1. In page 3, why $D_v$ is a distribution on $\\mathbb{S}_{\\xi}$?\n\nLIMITATIONS:\nThe paper only focuses on the IID dataset.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

comments: SUMMARY:
The paper proposes a method (Shadowheart SGD) for centralized asynchronous optimization with compression. The lower complexity bounds are proposed, as well. It is shown that Shadowheart SGD achieves the lower bounds, meaning that the method is optimal.

STRENGTHS:
The main strength is the development of an optimal method.

The paper contribution is outlined and clearly presented.

The proposed method is compared to others in several regimes of communication delays, showing how Shadowheart SGD can outperform its counterparts.

WEAKNESSES:
Personally, I did not understand the discussion of equilibrium time well. I understand that it comes from the analysis; also the examples are illustrative. But still, can equilibrium time have some interpretation like mixing time or first hit time in Markov chains? Or maybe it is possible to interpret equilibrium time using some random process, like the authors do in [1].

Also citation of MARINA on line 76, page 3 is missing.

[1] Vogels, T., Hendrikx, H., & Jaggi, M. (2022). Beyond spectral gap: The role of the topology in decentralized learning. Advances in Neural Information Processing Systems, 35, 15039-15050.

QUESTIONS:
See question on equilibrium time above.

LIMITATIONS:
The paper is mostly theoretical and does not have negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers distributed and centralized smooth non-convex optimization when workers have different computation and communication speeds. These different speeds on the workers (and even possibly the server) characterize the problem's device heterogeneity. The authors provide a new algorithm that uses unbiased compression and, more importantly, assigns a time budget for each communication round to optimize for total time complexity. This time budget is calculated based on the speeds of different workers and is the main contribution of this paper. Adapting classic lower bounds for distributed zero-respecting algorithms, and using RandK compression algorithm, the authors show their algorithm is optimal. The authors also offer extensions, such as an adaptive variant of their algorithm, and consider the setting when server communication requires some time.\n\nSTRENGTHS:\nThe paper is well written, and a reader not familiar with the literature gets to a very good understanding of the current literature in the area after reading the first few sections of the paper. The results are also presented in a transparent, easy to understand, and mathematically sound manner. The studied problem is a fundamental problem in distributed optimization, and underlies several more complicated seeming problems in Federated learning. The authors attempt at showing optimality of their procedure at least for RandK also provides context on any scope for improvement in the current result. Finally, the authors consider several important extreme limits as well as baselines, making clear comparisons with them. Overall, it is a good paper with only some flaws that can be corrected using remarks and small additional comments, that can be accommodated in the camera ready version. \n\nI support accepting the paper, and am happy to further increase my score based on author rebuttal and discussion.\n\nWEAKNESSES:\nI have the following comments in a decreasing order of importance:\n\n1. I am not sure if the lower bound actually holds for all unbiased compression schemes. For instance, if the unbiased compression scheme is to add random noise to the gradient vector before arbitrarily picking a direction to communicate, then it woun't be distributed zero-respecting. Ofcourse, with randK this does not happen because if it picks a co-ordinate it wouldn't make it non-zero if it isn't already. This is an important subtlety and should be incorporated in the discussion, but either removing the above compression scheme I mentioned through an additional constraint in the definition of distributed zero-respecting algorithms (with compression), or state that the lower bound is only for randK compression operator. Otherwise, it is not accurate to say the proposed method is optimal. \n\n2. In the introduction while specifying the scope of the problem, it would also be useful to discuss the issue of bit communication complexity somewhere. It is true that in some settings, the communication time complexity only depends on the total number of communications, but in more optimized channel communication the bit complixty is an important factor as well, and for instance in the general setup $\\tau_i$ should be a function of $K$ for say randK compressor. I ofcourse don't expect the authors to solve all problems in a single problem, but discussing this would make the survey more exhaustive in my opinion. \n\n3. The authors should comment on when (12) is a reasonable approximation, even if the calculations are not exact. \n\n4. Small typos: line 112 should be \"is not negligible\", line 134 should have $t/2$ instead of $t$, unless the authors are using some interlacing idea to compute and communicate simultaneously, in line 217 it should be \"it is left\",\n\nQUESTIONS:\n1. Do the authors have thoughts on statistical heterogeneity can be considered in this setting? I am concerned that the current algorithm might heavily bias the final output towards the databases of agents that are pretty quick. \n\n2. Do the authors have thoughts on the second weakness I raised above? In particular, it is an open question to obtain algorithms with optimal round and bit communication complexity, and understanding the regime with non-trivial bit complexity can help make progress towards that setting.\n\nLIMITATIONS:\nNothing big, but please see my comments above in the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel method for non-convex stochastic optimization in an asynchronous centralized distributed setup, focusing on improving time complexity in heterogeneous environments. Additionally, the authors demonstrate that the proposed method achieves theoretically optimal time complexity under compression.\n\nSTRENGTHS:\n1. **Novel Method**: The introduction of Shadowheart SGD guarantees finding an $\\varepsilon$-stationary point with optimal time complexity in heterogeneous environments.\n2. **Theoretical Contributions**: The paper provides rigorous proofs showing that Shadowheart SGD has better or equal time complexity compared to existing centralized methods.\n3. **Optimal Time Complexity**: The time complexity analysis shows that Shadowheart SGD can be significantly better than previous methods in many regimes.\n\nWEAKNESSES:\n1. The definition and calculation of equilibrium time are complex and not very intuitive. The implicit nature of this definition might make practical implementation and understanding challenging.\n2. While the proposed method improves time complexity, the paper does not fully discuss the impact on final convergence and loss in real-world scenarios with limited training datasets. Additionally, the experiments only compare the loss of different SGD methods under the same wall time, but it would be insightful to know if Shadowheart SGD maintains lower loss when comparing the number of training samples.\n\nQUESTIONS:\n1. Can the process of calculating equilibrium time be simplified or made more intuitive? Are there any practical algorithms or tools recommended for this calculation?\n2. Similar to Weakness 2. Regarding the impact on convergence and loss in scenarios with limited training datasets, how does Shadowheart SGD perform when comparing the loss against the number of training samples?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-22"}