Title: 2Direction: Theoretically Faster Distributed Training with Bidirectional Communication Compression
Venue: NeurIPS 2023
Accepted: yes
Keywords: convex optimization

accelerated method

communication compression

bidirectional compression

distributed optimization
Abstract: We consider distributed convex optimization problems in the regime when the communication between the server and the workers is expensive in both uplink and downlink directions. We develop a new and provably accelerated method, which we call 2Direction, based on fast bidirectional compressed communication and a new bespoke error-feedback mechanism which may be of independent interest. Indeed, we find that the EF and EF21-P mechanisms (Seide et al., 2014; Gruntkowska et al., 2023) that have considerable success in the design of efficient non-accelerated methods are not appropriate for accelerated methods. In particular, we prove that 2Direction improves the previous state-of-the-art communication complexity $\widetilde{\Theta}\left(K \times \left(\frac{L}{\alpha \mu} + \frac{L_{\max} \omega}{n \mu} + \omega\right)\right)$ (Gruntkowska et al., 2023) to $\widetilde{\Theta}(K \times (\sqrt{\frac{L (\omega + 1)}{\alpha \mu}} + \sqrt{\frac{L_{\max} \omega^2}{n \mu}} + \frac{1}{\alpha} + \omega))$ in the $\mu$--strongly-convex setting, where $L$ and $L_{\max}$ are smoothness constants, $n$ is \# of workers, $\omega$ and $\alpha$ are compression errors of the Rand$K$ and Top$K$ sparsifiers (as examples), $K$ is \# of coordinates/bits that the server and workers send to each other. Moreover, our method is the first that improves upon the communication complexity of the vanilla accelerated gradient descent method (AGD). We obtain similar improvements in the general convex regime as well. Finally, our theoretical findings are corroborated by experimental evidence.

2Direction: Theoretically Faster Distributed Training with Bidirectional Communication Compression

Abstract

We consider distributed convex optimization problems in the regime when the communication between the server and the workers is expensive in both uplink and downlink directions. We develop a new and provably accelerated method, which we call 2Direction , based on fast bidirectional compressed communication and a new bespoke error-feedback mechanism which may be of independent interest. Indeed, we find that the EF and EF21-P mechanisms (Seide et al., 2014; Gruntkowska et al., 2023) that have considerable success in the design of efficient non-accelerated methods are not appropriate for accelerated methods. In particular, we prove that 2Direction improves the previous state-of-the-art communication complexity ˜ Θ( K × ( L / αµ + L max ω / nµ + ω )) (Gruntkowska et al., 2023) to ˜ Θ( K × ( √ L ( ω +1) / αµ + √ L max ω 2 / nµ + 1 / α + ω )) in the µ -strongly-convex setting, where L and L max are smoothness constants, n is # of workers, ω and α are compression errors of the Rand K and Top K sparsifiers (as examples), K is # of coordinates/bits that the server and workers send to each other. Moreover, our method is the first that improves upon the communication complexity of the vanilla accelerated gradient descent ( AGD ) method (Nesterov, 2018). We obtain similar improvements in the general convex regime as well. Finally, our theoretical findings are corroborated by experimental evidence.

1 Introduction

We consider convex optimization problems in the centralized distributed setting. These types of problems appear in federated learning (Koneˇ cn´ y et al., 2016; McMahan et al., 2017) and distributed optimization (Ramesh et al., 2021). In this setting, one of the main problems is the communication bottleneck: the connection link between the server and the workers can be very slow. We focus our attention on methods that aim to address this issue by applying lossy compression to the communicated messages (Alistarh et al., 2017; Mishchenko et al., 2019; Gruntkowska et al., 2023).

1.1 The problem

Formally, we consider the optimization problem

where n is the number of workers and f i : R d → R are smooth convex functions for all i ∈ [ n ] := { 1 , . . . , n } . We consider the centralized distributed optimization setting in which each i th worker contains the function f i , and all workers are directly connected to a server (Kairouz et al., 2021). In general, we want to find a (possibly random) point ̂ x such that E [ f ( ̂ x )] -f ( x ∗ ) ≤ ε, where x ∗ is

an optimal point. In the strongly convex setup, we also want to guarantee that E [ ∥ ˜ x -x ∗ ∥ 2 ] ≤ ε for some point ˜ x.

Virtually all other theoretical works in this genre assume that, compared to the worker-to-server (w2s) communication cost, the server-to-workers (s2w) broadcast is so fast that it can be ignored. We lift this limitation and instead associate a relative cost r ∈ [0 , 1] with the two directions of communication. If r = 0 , then s2w communication is free, if r = 1 , then w2s communication is free, and if r = 1 / 2 , then the s2w and w2s costs are equal. All our theoretical results hold for any r ∈ [0 , 1] . We formalize and elaborate upon this setup in Section 2.

1.2 Assumptions

Throughout the paper we rely on several standard assumptions on the functions f i and f.

Assumption 1.1. Functions f i are L i -smooth, i.e., ∥∇ f i ( x ) -∇ f i ( y ) ∥ ≤ L i ∥ x -y ∥ for all x, y ∈ R d , for all i ∈ [ n ] . We let L max := max i ∈ [ n ] L i . Further, let ̂ L > 0 be a constant such that 1 n ∑ n i =1 ∥∇ f i ( x ) -∇ f i ( y ) ∥ 2 ≤ ̂ L 2 ∥ x -y ∥ 2 for all x, y ∈ R d .

Assumption 1.2. Function f is L -smooth, i.e., ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ for all x, y ∈ R d . Assumption 1.3. Functions f i are convex for all i ∈ [ n ] , and f is µ -strongly convex with µ ≥ 0 , attaining a minimum at some point x ∗ ∈ R d .

Note that if the functions f i are L i -smooth for all i ∈ [ n ] , then ̂ L ≤ L max .

It is known that the above smoothness constants are related in the following way.

Lemma 1.4 (Gruntkowska et al. (2023)) . If Assumptions 1.2, 1.1 and 1.3 hold, then ̂ L ≤ L max ≤ nL and L ≤ ̂ L ≤ √ L max L.

2 Motivation: From Unidirectional to Bidirectional Compression

In this work, we distinguish between worker-to-server (w2s=uplink) and server-to-worker (s2w=downlink) communication cost, and define w2s and s2w communication complexities of methods in the following natural way.

Definition 2.1. For a centralized distributed method M aiming to solve problem (1), the communication complexity m w2s M is the expected number of coordinates/floats 1 that each worker sends to the server to solve problem (1). The quantity m s2w M is the expected number of floats/coordinates the server broadcasts to the workers to solve problem (1). If m w2s M = m s2w M , then we use the simplified notation m M := m s2w M = m w2s M .

Let us illustrate the above concepts on the simplest baseline: vanilla gradient descent ( GD ). It is well known (Nesterov, 2018) that for L -smooth, µ -strongly convex problems, GD returns an ε -solution after O ( L / µ log 1 / ε ) iterations. In each iteration, the workers and the server communicate all Θ( d ) coordinates to each other (since no compression is applied). Therefore, the communication complexity of GD is m GD = Θ( dL / µ log 1 / ε ) . The same reasoning applies to the accelerated gradient method ( AGD ) (Nesterov, 2018), whose communication complexity is m AGD = Θ( d √ L / µ log 1 / ε ) .

2.1 Compression mappings

In the literature, researchers often use the following two families of compressors:

Definition 2.2. A (possibly) stochastic mapping C : R d → R d is a biased compressor if there exists α ∈ (0 , 1] such that

Definition 2.3. Astochastic mapping C : R d → R d is an unbiased compressor if there exists ω ≥ 0 such that

Table 1: Communication Rounds in the Strongly Convex Case. The number of communication rounds and rounds costs to get an ε -solution ( E [ ∥ ̂ x -x ∗ ∥ 2 ] ≤ ε ) up to logarithmic factors. The table shows the most relevant bidirectional compressed methods that are ordered by the total communication complexity # Communication Rounds × Round Cost (see (4) for details).

The parameter r weights the importance/speed of uplink and downlink connections. When r = 1 / 2 , it means that the uplink and downlink speeds are equal.

The parameters K ω and K α are the expected densities Definition 2.5 of compressors C D ∈ U ( ω ) and C P ∈ B ( α ) (a) , that operate in the workers and the server accordingly. Less formally, K ω and K α are the number of coordinates/bits that the workers and the server send to each other in each communication round.

(a) The Dore , Artemis , MURANA , and MCM methods do not support biased compressors for server-to-worker compression. In these methods, the error α equals 1 / ( ω s +1) , where the error ω s is a parameter of an unbiased compressor that is used in server-to-worker compression. For these methods, we define 1 / ( ω s +1) as α to make comparison easy with EF21-P + DIANA and 2Direction .

We will make use of the following assumption.

Assumption 2.4. The randomness in all compressors used in our method is drawn independently.

Let us denote the set of mappings satisfying Definition 2.2 and 2.3 by B ( α ) and U ( ω ) , respectively. The family of biased compressors B is wider. Indeed, it is well known if C ∈ U ( ω ) , then 1 / ( ω +1) ·C ∈ B ( 1 / ( ω +1) ) . The canonical sparsification operators belonging to these classes are Top K ∈ B ( K / d ) and Rand K ∈ U ( d / K -1) . The former outputs the K largest values (in magnitude) of the input vector, while the latter outputs K random values of the input vector, scaled by d / K (Beznosikov et al., 2020). Following (Gorbunov et al., 2021; Tyurin and Richt´ arik, 2023), we now define the expected density of a sparsifier as a way to formalize its compression performance.

Definition 2.5. The expected density of a sparsifier C : R d → R d is the quantity K C := sup x ∈ R d E [ ∥C ( x ) ∥ 0 ] , where ∥ y ∥ 0 is the number of of non-zero components of y ∈ R d .

Trivially, for the Rand K and Top K sparsifiers we have K C = K.

2.2 Unidirectional (i.e., w2s) compression

As mentioned in the introduction, virtually all theoretical works in the area of compressed communication ignore s2w communication cost and instead aim to minimize m w2s M . Algorithmic work related

to methods that only perform w2s compression has a long history, and this area is relatively well understood (Alistarh et al., 2017; Mishchenko et al., 2019; Richt´ arik et al., 2021).

We refer to the work of Gruntkowska et al. (2023) for a detailed discussion of the communication complexities of non-accelerated methods in the convex and non-convex settings. For instance, using Rand K , the DIANA method of Mishchenko et al. (2019) provably improves 2 the communication complexity of GD to m w2s DIANA = ˜ Θ( d + KL / µ + dL max / nµ ) . Accelerated methods focusing on w2s compression are also well investigated. For example, Li et al. (2020) and Li and Richt´ arik (2021) developed accelerated methods, which are based on (Mishchenko et al., 2019; Kovalev et al., 2020), and provably improve the w2s complexity of DIANA . Moreover, using Rand K with K ≤ d / n , ADIANA improves the communication complexity of AGD to m w2s ADIANA = ˜ Θ( d + d √ L max / nµ ) .

2.3 Bidirectional (i.e., w2s and s2w) compression

The methods mentioned in Section 2.2 do not perform server-to-workers (s2w) compression, and one can show that the server-to-workers (s2w) communication complexities of these methods are worse than m AGD = ˜ Θ( d √ L / µ ) . For example, using the Rand K , the s2w communication complexity of ADIANA is at least m s2w ADIANA = ˜ Ω( d × ω ) = ˜ Ω( d 2 / K ) , which can be d / K times larger than in GD or AGD . Instead of m w2s M , methods performing bidirectional compression attempt to minimize the total communication complexity , which we define as a convex combination of the w2s and s2w communication complexities:

The parameter r ∈ [0 , 1] weights the importance of uplink (w2s) and downlink (s2w) connections 3 . Methods from Section 2.2 assume that r = 0 , thus ignoring the s2w communication cost. On the other hand, when r = 1 / 2 , the uplink and downlink communication speeds are equal. By considering any r ∈ [0 , 1] , our methods and findings are applicable to more situations arising in practice. Obviously, m r GD = m GD and m r AGD = m AGD for all r ∈ [0 , 1] . Recently, Gruntkowska et al. (2023) proposed the EF21-P + DIANA method. This is the first method supporting bidirectional compression that provably improves both the w2s and s2w complexities of GD : m r EF21-P + DIANA ≤ m GD for all r ∈ [0 , 1] . Bidirectional methods designed before EF21-P + DIANA , including (Tang et al., 2020; Liu et al., 2020; Philippenko and Dieuleveut, 2021), do not guarantee the total communication complexities better than that of GD . The EF21-P + DIANA method is not an accelerated method and, in the worst case, can have communication complexities worse than AGD when the condition number L / µ is large.

3 Contributions

Motivated by the above discussion, in this work we aim to address the following

Main Problem :

Is it possible to develop a method supporting bidirectional communication compression that improves the current best theoretical total communication complexity of EF21-P + DIANA , and guarantees the total communication complexity to be no worse than the communication complexity m AGD = ˜ Θ( d √ L / µ ) of AGD , while improving on AGD in at least some regimes?

A) We develop a new fast method ( 2Direction ; see Algorithm 1) supporting bidirectional communication compression. Our analysis leads to new state-of-the-art complexity rates in the centralized distributed setting (see Table 1), and as a byproduct, we answer Main Problem in the affirmative.

B) Gruntkowska et al. (2023) proposed to use the EF21-P error-feedback mechanism (8) to improve the convergence rates of non-accelerated methods supporting bidirectional communication compression. EF21-P is a reparameterization of the celebrated EF mechanism (Seide et al., 2014). We tried to use EF21-P in our method as well, but failed. Our failures indicated that a fundamentally

new approach is needed, and this eventually led us to design a new error-feedback mechanism (9) that is more appropriate for accelerated methods. We believe that this is a contribution of independent interest that might motivate future growth in the area.

C) Unlike previous theoretical works (Li et al., 2020; Li and Richt´ arik, 2021) on accelerated methods, we present a unified analysis in both the µ -strongly-convex and general convex cases. Moreover, in the general convex setting and low accuracy regimes, our analysis improves the rate O ( 1 / ε 1 / 3 ) of Li and Richt´ arik (2021) to O (log 1 / ε ) (see details in Section R).

D) Even though our central goal was to obtain new SOTA theoretical communication complexities for centralized distributed optimization, we show that the newly developed algorithm enjoys faster communication complexities in practice as well (see details in Section Q).

Algorithm 1 2Direction : A Fast Gradient Method Supporting Bidirectional Compression

4 New Method: 2Direction

In order to provide an answer to Main Problem, at the beginning of our research journey we hoped that a rather straightforward approach might bear fruit. In particular, we considered the current statethe-art methods ADIANA (Algorithm 3) (Li et al., 2020), CANITA (Li and Richt´ arik, 2021) and EF21P + DIANA (Algorithm 4) (Gruntkowska et al., 2023), and tried to combine the EF21-P compression

Algorithm 2 CalculateLearningRates

1: Parameters: element Γ t ; parameter ¯ L > 0 ; strong-convexity parameter µ ≥ 0 , probability p, contraction parameter α , momentum τ , parameter β

2: Find the largest root ¯ θ t +1 of the quadratic equation

mechanism on the server side with the ADIANA (accelerated DIANA ) compression mechanism on the workers' side. In short, we were aiming to develop a ' EF21-P + ADIANA ' method. Note that while EF21-P + DIANA provides the current SOTA communication complexity among all methods supporting bidirectional compression, the method is not 'accelerated'. On the other hand, while ADIANA (in the strongly convex regime) and CANITA (in the convex regime) are 'accelerated', they support unidirectional (uplink) compression only.

In Sections B and C we list the ADIANA and EF21-P + DIANA methods, respectively. One can see that in order to calculate x t +1 , y t +1 , and z t +1 in ADIANA (Algorithm 3), it is sufficient for the server to broadcast the point u t +1 . At first sight, it seems that we might be able to develop a ' EF21-P + ADIANA ' method by replacing Line 15 in Algorithm 3 with Lines 12, 13, 14, and 16 from Algorithm 4. With these changes, we can try to calculate x t +1 and y t +1 using the formulas

instead of Lines 7 and 17 in Algorithm 3. Unfortunately, all our attempts of making this work failed, and we now believe that this 'naive' approach will not lead to a resolution of Main Problem. Let us briefly explain why we think so, and how we ultimately managed to resolve Main Problem.

♦ The first issue arises from the fact that the point x t +1 , and, consequently, the point z t +1 , depend on w t +1 instead of u t +1 , and thus the error from the primal (i.e., server) compressor C P affects them. In our proofs, we do not know how to prove a good convergence rate with (6). Therefore, we decided to use the original update (Line 17 from Algorithm 3) instead. We can do this almost for free because in Algorithm 3 the point x t +1 is only used in Line 18 (Algorithm 3) with small probability p . In the final version of our algorithm 2Direction (see Algorithm 1), we broadcast a non-compressed messages x t +1 with probability p. In Section 6, we show that p is so small that these non-compressed rare messages do not affect the total communication complexity of Algorithm 3.

♦ The second issue comes from the observation that we can not perform the same trick for point y t +1 since it is required in each iteration of Algorithm 3. We tried to use (5) and (7), but this still does not work. Deeper understanding of this can only be gained by a detailed examination our proof and the proofs of (Li et al., 2020; Gruntkowska et al., 2023). One way to explain the difficulty is to observe that in non-accelerated methods (Gorbunov et al., 2020; Gruntkowska et al., 2023), the variance-reducing shifts h t converge to the fixed vector ∇ f ( x ∗ ) , while in the accelerated methods (Li et al., 2020; Li and Richt´ arik, 2021), these shifts h t converge to the non-fixed vectors ∇ f ( z t ) in the corresponding Lyapunov functions. Assume that µ = 0 . Then, instead of the EF21-P mechanism

we propose to perform the step

Unlike (8), step (9) resolves all our previous problems, and we were able to obtain new SOTA rates.

♦ However, step (9) is not implementable since the server and the nodes need to know the vector ∇ f ( z t ) . The last crucial observation is the same as with the points x t and z t : the vector ∇ f ( z t ) changes with probability p since the point z t changes with probability p. Intuitively, this means that it easier to communicate ∇ f ( z t ) between the server and the workers. We do this using two auxiliary control vectors, v t and k t . The former 'learns' the value of ∇ f ( z t ) in Line 30 (Algorithm 1), and the latter is used in Line 14 (Algorithm 1) instead of ∇ f ( z t ) . Then, when the algorithm updates z t +1 , it also updates k t in Line 18 (Algorithm 1) and the updated non-compressed vector k t is broadcast to the workers.

The described changes are highlighted in Lines 6, 12, 13, 14, 18, 20, 23, 24 and 30 of our new Algorithm 1 in green color. Other steps of the algorithm correspond to the original ADIANA method (Algorithm 3). Remarkably, all these new steps are only required to substitute a single Line 15 of Algorithm 3!

5 Theoretical Communication Complexity of 2Direction

Having outlined our thought process when developing 2Direction (Algorithm 1), we are now ready to present our theoretical iteration and communication complexity results. Note that 2Direction depends on two hyper-parameters, probability p (used in Lines 4 and 17) and momentum τ (used in Lines 4 and 30). Further, while Li et al. (2020); Li and Richt´ arik (2021) assume a strong relationship between L and L max ( L = L max ), Gruntkowska et al. (2023) differentiate between L and L max , and thus perform a more general and analysis of their method. In order to perform a fair comparison to the above results, we have decided to minimize the total communication complexity m r M as a function of the hyper-parameters p and τ , depending on whether the ratio L max / L is known or not.

Defining R 2 := ∥ ∥ x 0 -x ∗ ∥ ∥ 2 , Theorems E.12 and E.14 state that 2Direction converges after

iterations, in the µ -strongly convex and general convex regimes, respectively. These complexities depend on two hyper-parameters: p ∈ (0 , 1] and τ ∈ (0 , 1] . For simplicity, in what follows we consider the µ -strongly convex case only 4 . While the iteration complexities (10) and (11) are clearly important, in the context of our paper, optimizing communication complexity (4) is more important. In the following simple theorem, we give expressions for the communication complexities of 2Direction , taking into account both workers-to-server (w2s) and server-to-workers (s2w) communication.

Theorem 5.1. Assume that C D, · i and C P have expected densities equal to K ω and K α , respectively (see Definition 2.5). In view of Theorem E.12, in expectation, the w2s and s2w communication complexities are equal to

Proof. The first complexity in (12) follows because w2s communication involves C D,y i ( · ) and C D,z i ( · ) only. The second complexity in (12) follows because s2w communication involves C P i ( · ) , plus two non-compressed vectors x t +1 and k t +1 with the probability p. The term d comes from the fact that non-compressed vectors are communicated in the initialization phase.

5.1 The ratio L max / L is not known

In the following theorem, we consider the regime when the exact value of L max / L is not known. Hence, we seek to find p and τ that minimize the worst case m r new (see (4)) w.r.t. L max ∈ [ L, nL ] .

Theorem 5.2. Choose r ∈ [0 , 1] and let µ r ω,α := rd (1 -r ) K ω + rK α . In view of Theorem 5.1, the values p = min { 1 ω +1 , 1 µ r ω,α } and τ = p 1 / 3 ( ω +1) 2 / 3 minimize max L max ∈ [ L,nL ] m r new . This choice leads to the following number of communication rounds:

The total communication complexity thus equals m r realistic = ˜ Θ(((1 -r ) K ω + rK α ) T realistic + d ) .

Remark 5.3 . To simplify the rate (13) and understand the quantity µ r ω,α , let C D, · i be the Rand K sparsifier 5 and consider the case when the s2w communication is not slower than the w2s communication, i.e., r ≤ 1 / 2 . Then T realistic = ˜ Θ ( max {√ L ( ω +1) αµ , √ L max ω ( ω +1) npµ , 1 α , ( ω +1) }) and µ r ω,α ≤ ω + 1 . Indeed, this follows from r ≤ 1 / 2 and the fact that ω + 1 = d / K ω for the Rand K compressor: µ r ω,α := rd (1 -r ) K ω + rK α ≤ r 1 -r × d K ω ≤ ω +1 .

5.2 The ratio L max / L is known

We now consider the case when we have information about the ratio L max / L .

Theorem 5.4. Choose r ∈ [0 , 1] , and let µ r ω,α := rd (1 -r ) K ω + rK α . In view of Theorem 5.1, the values p and τ given by (63) and (58) , respectively, minimize m r new from (10) . This choice leads to the following number of communication rounds:

The total communication complexity thus equals m r optimistic = ˜ Θ(((1 -r ) K ω + rK α ) T optimistic + d ) .

Note that information about L max / L leads to a better rate that in Theorem 5.2.

Remark 5.5 . To simplify the rate (14), let C D, · i be the Rand K sparsifier 6 and consider the case when the s2w communication is not slower than the w2s communication, i.e., r ≤ 1 / 2 . Then T optimistic = ˜ Θ ( max { √ L max { 1 ,r ( ω +1) } αµ , √ L 2 / 3 L 1 / 3 max ( ω +1) αn 1 / 3 µ , √ L 1 / 2 L 1 / 2 max ( ω +1) 3 / 2 √ αnµ , √ L max ω ( ω +1) nµ , 1 α , ( ω +1) }) . Indeed, this follows from r ≤ 1 / 2 and the fact that ω + 1 = d / K ω for the Rand K compressor: µ r ω,α := rd (1 -r ) K ω + rK α ≤ r 1 -r × d K ω ≤ 2 r ( ω +1) .

6 Theoretical Comparison with Previous State of the Art

We now show that the communication complexity of 2Direction is always no worse than that of EF21 + DIANA and AGD . Crucially, in some regimes, it can be substantially better. Furthermore, we show that if the s2w communication cost is zero (i.e., if r = 0 ), the 2Direction obtains the same communication complexity as ADIANA (Li et al., 2020) (see Section S).

Comparison with EF21 + DIANA . The EF21-P + DIANA method has the communication complexities that equal

where K ω and K α are the expected densities of C D i and C P in Algorithm 4. The last term d comes from the fact that EF21-P + DIANA sends non-compressed vectors in the initialization phase. Let us define K r ω,α := (1 -r ) K ω + rK α . Therefore, the total communication complexity equals

Theorem 5.2 ensures that the total communication complexity of 2Direction is

One can see that (16) is an accelerated rate; it has much better dependence on the condition numbers L / µ and L max / µ . In Section E.8, we prove the following simple theorem, which means that 2Direction is not worse than EF21-P + DIANA .

Theorem 6.1. For all r ∈ [0 , 1] , m r realistic = ˜ O ( m r EF21-P + DIANA ) .

Comparison with AGD . To compare the abstract complexity (16) with the non-abstract complexity ˜ Θ( d √ L / µ ) , we take the Rand K and Top K compressors in Algorithm 1.

Theorem 6.2. For all r ∈ [0 , 1] and for all K ∈ [ d ] , let us take the Rand K and Top K compressors with the parameters (expected densities) i) K ω = K and K α = min {⌈ 1 -r / r K ⌉ , d } for r ∈ [0 , 1 / 2 ] , ii) K ω = min {⌈ r / 1 -r K ⌉ , d } and K α = K for r ∈ ( 1 / 2 , 1] . Then we have m r realistic = ˜ O ( m AGD ) .

Theorem 6.2 states that the total communication complexity of 2Direction is not worse than that of AGD . It can be strictly better in the regimes when α > K / d (for Top K ), L max < nL, and n > 1 .

7 Proof Sketch

After we settle on the final version of Algorithm 1, the proof technique is pretty standard at the beginning (Li et al., 2020; Tyurin and Richt´ arik, 2023; Gruntkowska et al., 2023). We proved a descent lemma (Lemma E.4) and lemmas that control the convergences of auxiliary sequences (Lemmas E.5, E.6, E.7, E.8). Using these lemmas, we construct the Lyapunov function (45) with the coefficients κ, ρ, λ and ν t ≥ 0 for all t ≥ 0 .

One of the main problems was to find appropriate ¯ L, κ, ρ, λ and ν t ≥ 0 such that we get a convergence. In more details, to get a converge it is sufficient to find ¯ L, κ, ρ, λ and ν t ≥ 0 such that (46), (47) and (48) hold. Using the symbolic computation SymPy library (Meurer et al., 2017), we found appropriate κ, ρ, λ and ν t ≥ 0 ((90), (88), (82), (83)) such that the inequalities hold. But that is not all. To get a convergence, we also found the bounds on the parameter ¯ L , which essentially describe the speed of the convergence. In raw form, using symbolic computations, we obtained a huge number of bounds on ¯ L (see Sections I, J, L, N). It was clear that most of them are redundant and it was required to find the essential ones. After a close look at the bounds on ¯ L , we found out that it is sufficient to require (44) to ensure that all inequalities from Sections I, J, L and N hold (see Section O for details).

(Lyapunov Function

Convergence)

(

µ

-strongly convex)

Theorem E.14

(general convex)

Theorems 5.1, 5.2

(Find comm. compl.

and optimal

p

and

τ

)

Theorem 6.1

(No worse than

EF21-P + DIANA

)

Theorem 6.2

(No worse than

AGD

)

8 Limitations and Future Work

In contrast to Algorithm 4 ( EF21+DIANA ), in which the server always broadcasts compressed vectors, in our Algorithm 1 ( 2Direction ) the server needs to broadcast non-compressed vectors with

probability p (see Line 20). While in Section 6 we explain that this does not have an adverse effect on the theoretical communication complexity since p is small, one may wonder whether it might be possible to achieve the same (or better) bounds as ours without having to resort to intermittent non-compressed broadcasts. This remains an open problem; possibly a challenging one. Another limitation comes from the fact that 2Direction requires more iterations than AGD in general (this is the case of all methods that reduce communication complexity). While, indeed, (10) can be higher than ˜ Θ( √ L / µ ) , the total communication complexity of 2Direction is not worse than that of AGD .

References

Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. (2017). QSGD: Communicationefficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems (NIPS) , pages 1709-1720.

Beznosikov, A., Horv´ ath, S., Richt´ arik, P., and Safaryan, M. (2020). On biased compression for distributed learning. arXiv preprint arXiv:2002.12410 .

Chang, C.-C. and Lin, C.-J. (2011). LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) , 2(3):1-27.

Condat, L. and Richt´ arik, P. (2022). Murana: A generic framework for stochastic variance-reduced optimization. In Mathematical and Scientific Machine Learning , pages 81-96. PMLR.

Gorbunov, E., Burlachenko, K., Li, Z., and Richt´ arik, P. (2021). MARINA: Faster non-convex distributed learning with compression. In 38th International Conference on Machine Learning .

Gorbunov, E., Hanzely, F., and Richt´ arik, P. (2020). A unified theory of SGD: Variance reduction, sampling, quantization and coordinate descent. In International Conference on Artificial Intelligence and Statistics , pages 680-690. PMLR.

Gruntkowska, K., Tyurin, A., and Richt´ arik, P. (2023). EF21-P and friends: Improved theoretical communication complexity for distributed optimization with bidirectional compression. In International Conference on Machine Learning .

Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. (2021). Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210.

Koneˇ cn´ y, J., McMahan, H. B., Yu, F. X., Richt´ arik, P., Suresh, A. T., and Bacon, D. (2016). Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 .

Kovalev, D., Horv´ ath, S., and Richt´ arik, P. (2020). Don't jump through hoops and remove those loops: Svrg and katyusha are better without the outer loop. In Algorithmic Learning Theory , pages 451-467. PMLR.

Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto.

Lan, G. (2020). First-order and stochastic optimization methods for machine learning . Springer.

Lan, G., Li, Z., and Zhou, Y. (2019). A unified variance-reduced accelerated gradient method for convex optimization. Advances in Neural Information Processing Systems , 32.

Li, Z., Kovalev, D., Qian, X., and Richt´ arik, P. (2020). Acceleration for compressed gradient descent in distributed and federated optimization. In International Conference on Machine Learning .

Li, Z. and Richt´ arik, P. (2021). CANITA: Faster rates for distributed convex optimization with communication compression. Advances in Neural Information Processing Systems , 34:1377013781.

Liu, X., Li, Y., Tang, J., and Yan, M. (2020). A double residual compression algorithm for efficient distributed learning. In International Conference on Artificial Intelligence and Statistics , pages 133-143. PMLR.

McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communicationefficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR.

Meurer, A., Smith, C. P., Paprocki, M., ˇ Cert´ ık, O., Kirpichev, S. B., Rocklin, M., Kumar, A., Ivanov, S., Moore, J. K., Singh, S., Rathnayake, T., Vig, S., Granger, B. E., Muller, R. P., Bonazzi, F., Gupta, H., Vats, S., Johansson, F., Pedregosa, F., Curry, M. J., Terrel, A. R., Rouˇ cka, v., Saboo, A., Fernando, I., Kulal, S., Cimrman, R., and Scopatz, A. (2017). Sympy: symbolic computing in python. PeerJ Computer Science , 3:e103.

Mishchenko, K., Gorbunov, E., Tak´ aˇ c, M., and Richt´ arik, P. (2019). Distributed learning with compressed gradient differences. arXiv preprint arXiv:1901.09269 .

Nesterov, Y. (2018). Lectures on convex optimization , volume 137. Springer.

Philippenko, C. and Dieuleveut, A. (2020). Artemis: tight convergence guarantees for bidirectional compression in federated learning. arXiv preprint arXiv:2006.14591 .

Philippenko, C. and Dieuleveut, A. (2021). Preserved central model for faster bidirectional compression in distributed settings. Advances in Neural Information Processing Systems , 34:2387-2399.

Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021). Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092 .

Richt´ arik, P., Sokolov, I., and Fatkhullin, I. (2021). EF21: A new, simpler, theoretically better, and practically faster error feedback. In Neural Information Processing Systems, 2021.

Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. (2014). 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In Fifteenth Annual Conference of the International Speech Communication Association .

Stonyakin, F., Tyurin, A., Gasnikov, A., Dvurechensky, P., Agafonov, A., Dvinskikh, D., Alkousa, M., Pasechnyuk, D., Artamonov, S., and Piskunova, V. (2021). Inexact model: a framework for optimization and variational inequalities. Optimization Methods and Software , 36(6):1155-1201.

Tang, H., Lian, X., Yu, C., Zhang, T., and Liu, J. (2020). DoubleSqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression. In Proceedings of the 36th International Conference on Machine Learning (ICML) .

Tyurin, A. and Richt´ arik, P. (2023). DASHA: Distributed nonconvex optimization with communication compression and optimal oracle complexity. International Conference on Learning Representations (ICLR) .

Appendix

Contents

A Table of Notations

B The Original ADIANA Algorithm

In this section, we present the ADIANA algorithm from (Li et al., 2020). In the following method, the notations, parameterization, and order of steps can be slightly different, but the general idea is the same.

Algorithm 3 Accelerated DIANA ( ADIANA ) by Li et al. (2020)

{

}

C The Original EF21-P + DIANA Algorithm

In this section, we present the EF21-P + DIANA algorithm from (Gruntkowska et al., 2023). In the following method, the notations, parameterization, and order of steps can be slightly different, but the general idea is the same.

Algorithm 4 EF21-P + DIANA by Gruntkowska et al. (2023)

D Useful Identities and Inequalities

For all x, y, x 1 , . . . , x n ∈ R d , s > 0 and α ∈ (0 , 1] , we have:

Variance decomposition: For any random vector X ∈ R d and any non-random c ∈ R d , we have

Lemma D.1 (Nesterov (2018)) . Let f : R d → R be a function for which Assumptions 1.2 and 1.3 are satisfied. Then for all x, y ∈ R d we have:

E Proofs of Theorems

E.1 Analysis of learning rates

In this section, we establish inequalities for the sequences from Algorithm 2.

Lemma E.1. Suppose that parameter ¯ L > 0 , strong-convexity parameter µ ≥ 0 , probability p ∈ (0 , 1] , ¯ L ≥ µ, and Γ 0 ≥ 1 . Then the sequences generated by Algorithm 2 have the following properties:

The quantities θ t +1 , γ t +1 , and Γ t +1 are well-defined and θ t +1 , γ t +1 ≥ 0 for all t ≥ 0 .

γ t +1 = pθ t +1 Γ t +1 for all t ≥ 0 .

¯ Lθ t +1 γ t +1 ≤ ( ¯ L +Γ t µ ) for all t ≥ 0 .

4.

5.

for all t ≥ 0 .

which is equivalent to

6. { θ t +1 } ∞ t =0 is a non-increasing sequence.

Proof.

Note that ¯ θ t +1 is the largest root of

We fix t ≥ 0 . Assume that Γ t > 0 .

Then

Therefore, the largest root ¯ θ t +1 is well-defined and ¯ θ t +1 ≥ 0 , and θ t +1 = min { ¯ θ t +1 , θ min } ≥ 0 . Next, γ t +1 is well-defined and

since pθ t +1 ∈ [0 , 1 / 4 ] . Finally, Γ t +1 = Γ t + γ t +1 > 0 . We showed that, for all t ≥ 0 , if Γ t > 0 , then θ t +1 , γ t +1 ≥ 0 and Γ t +1 > 0 . Note that Γ 0 > 0 , thus θ t +1 , γ t +1 ≥ 0 and Γ t +1 > 0 for all t ≥ 0 .

From the definition of γ t +1 and Γ t +1 , we have

Recall again that ¯ θ t +1 ≥ 0 is the largest root of

If ¯ θ t +1 ≤ θ min , then

Otherwise, if ¯ θ t +1 > θ min , since

and θ t +1 = θ min < ¯ θ t +1 , then

In all cases, the inequality (26) holds. From this inequality, we can get

and

Using the definition of γ t +1 , we obtain for all t ≥ 0 .

Let us find the largest root of the quadratic equation (25):

Let us define a := p 2 ( ¯ L +Γ t µ ) 2 and b := 4 ¯ Lp ( ¯ L +Γ t µ ) Γ t , then ¯ θ t +1 = - √ a + √ a + b 2 ¯ Lp Γ t

Since Γ t ≥ 1 for all t ≥ 0 , and ¯ L ≥ µ, we have

Using √ x + y ≥ ( √ x + √ y ) / √ 2 for all x, y ≥ 0 , and √ b ≥ √ 2 √ a we have

Therefore

and

Next, since pθ t +1 ∈ [0 , 1 / 4 ] , we have and

Using (28) and (29), we obtain

where we use that 1 + x ≥ e x / 2 for all x ∈ [0 , 1] .

Using (28) and (29), we have

The sequence Γ t +1 is strongly increasing. Thus, exists the minimal ̂ t ≥ 0 such that Γ ̂ t ≥ (4 pθ 2 min ) -1 . For all 0 ≤ t < ̂ t, it holds that Γ t < (4 pθ 2 min ) -1 , √ 1 4 p Γ t > θ min , and

Therefore, if ̂ t > 0 , then

Thus, we have the following bound for ̂ t :

For all t ≥ ̂ t, we have √ 1 4 p Γ t ≤ θ min and

Using mathematical induction, let us show that

for all t ≥ ̂ t. For t = ̂ t it is true since Γ ̂ t ≥ ( 4 pθ 2 min ) -1 by the definition of ̂ t. Next, for some t ≥ ̂ t, assume that (32) holds, then

We proved the inequality using mathematical induction. Combining (31) and (32), we obtain the unified inequality for Γ t :

for all t ≥ 0 . Also, if t < ¯ t, then the first term in the minimum is less or equal than the second one. Therefore,

Let us bound the second term. Recall that ¯ t ≥ ̂ t, thus, if t ≥ ¯ t, then ( t -̂ t ) 2 ≥ ( t -¯ t ) 2 . In (33), we can change ̂ t to ¯ t and get

Using the Taylor expansion of the exponent at the point ¯ t , we get

6. Let us rewrite (27):

Let us temporarily denote a t := ( 1 Γ t + µ ¯ L ) for all t ≥ 0 . Note that a t is a non-increasing sequence, since Γ t +1 ≥ Γ t for all t ≥ 0 . Therefore,

Let us take the derivative of the last term w.r.t. a t and compare it to zero:

Thus, ¯ θ t +1 is a non-decreasing sequence w.r.t. a t . But the sequence a t is non-increasing, therefore ¯ θ t +1 is a non-increasing sequence w.r.t. t. It is left to use that θ t +1 is the minimum of ¯ θ t +1 and the constant quantity.

E.2 Generic lemmas

First, we prove a well-known lemma from the theory of accelerated methods (Lan, 2020; Stonyakin et al., 2021).

Lemma E.2. Let us take vectors a, b, g ∈ R d , numerical quantities α, β ≥ 0 , and

Then

for all x ∈ R d .

Proof. A function

is strongly-convex with the parameter α + β. From the strong-convexity and the optimality of u , we obtain

for all x ∈ R . This inequality is equivalent to (34).

We assume that the conditional expectation E t [ · ] is conditioned on the randomness from the first t iterations. Also, let us define D f ( x, y ) := f ( x ) -f ( y ) -⟨∇ f ( y ) , x -y ⟩ .

Lemma E.3. Suppose that Assumptions 1.2, 1.1, 1.3 and 2.4 hold. For Algorithm 1, the following inequality holds:

Proof. Using the definition of g t +1 , we have

where we use the independence and the unbiasedness of the compressors (see Assumption 2.4). Using Definition 2.3 and Assumption 1.1, we have

Using that L max = max i ∈ [ n ] L i , we obtain (35).

Lemma E.4. Suppose that Assumptions 1.2, 1.1, 1.3 and 2.4 hold. For Algorithm 1, the following inequality holds:

Proof. Using Assumption 1.2, we have

Using the definition of x t +1 , we obtain

in the last inequality we add and subtract g t +1 . Using the definition of u t +1 and Lemma E.2 with x = x ∗ , we have

We use the fact that Γ t +1 = Γ t + γ t +1 . Since ∥ ∥ u t -y t +1 ∥ ∥ 2 ≥ 0 , we have

By substituting this inequality to (36), we get

Using µ -strong convexity, we have

and

Let us take the conditional expectation E t [ · ] conditioned on the randomness from the first t iterations:

where use that E t [ g t +1 ] = ∇ f ( y t +1 ) . We can find u t +1 analytically and obtain that

Therefore, using that E t [ g t +1 ] = ∇ f ( y t +1 ) and u t and y t +1 are conditionally nonrandom, we obtain

Combining (35) from Lemma E.3 with (37) and (38), one can get

In the last equality, we grouped the terms with f ( z t ) -f ( x ∗ ) . In Algorithm 2, we choose the learning rates so that (see Lemma E.1)

Since Γ t +1 ≥ Γ t for all t ∈ N 0 , thus

and

E.3 Construction of the Lyapunov function

In this section, we provide lemmas that will help us to construct a Lyapunov function.

Lemma E.5. Suppose that Assumptions 1.2, 1.1, 1.3 and 2.4 hold. The parameter β ≤ 1 / ω +1 . Then, for Algorithm 1, the following inequality holds:

Proof. Using the definition of h t +1 i , we have

where E C [ · ] is a conditional expectation that is conditioned on z t +1 and h t i . Therefore,

Since β ≤ 1 / ω +1 , we have

Next, we use the definition of z t +1 and obtain

Using 1 -β ≤ 1 , (18) and (20), we get

From Assumptions 1.1 and 1.3 and Lemma D.1, we obtain

Lemma E.6. Suppose that Assumptions 1.2, 1.1, 1.3 and 2.4 hold. Then, for Algorithm 1, the following inequality holds:

Proof. Using the definition of w t +1 and Definition 2.2, we get the following inequality

We can find the analytical formulas for u t +1 and q t +1 , and obtain that

and

Therefore,

One can substitute (35) from Lemma E.3 to the last inequality and get

Using Lemma D.1 and 1 -α ≤ 1 , we obtain

Lemma E.7. Suppose that Assumptions 1.2 and 1.3 hold. Then, for Algorithm 1, the following inequality holds:

where we use Assumptions 1.2 and Lemma D.1.

Lemma E.8. Suppose that Assumptions 1.2, 1.1 and 1.3 hold. The momentum τ ∈ (0 , 1] and the probability p ∈ (0 , 1] . Then, for Algorithm 1, the following inequality holds:

Proof. Using the definition of v t +1 , we get

Assumption 2.4, including the independence and the unbiasedness of the compressors, insures that

Using the definition of z t +1 , we have

Using 0 ≤ 1 -τ ≤ 1 , p ∈ (0 , 1] and (20), we get

It is left to use Assumptions 1.2 and 1.1 with Lemma D.1 to obtain

E.4 Main theorem

Theorem E.9. Suppose that Assumptions 1.2, 1.1, 1.3, 2.4 hold. Let

{ }

for some κ, ρ, λ, ν t ≥ 0 .

Proof. We fix some constants κ, ρ, λ, ν t ≥ 0 for all t ≥ 0 that we define later. By combining Lemma E.4 with κ × (39) from Lemma E.5, ν t × (41) from Lemma E.6, ρ × (42) from Lemma E.7, and λ × (43) from Lemma E.8, we get the following inequality:

We regroup the terms and obtain

Using x t +1 -y t +1 = θ t +1 ( u t +1 -w t ) , we get (we mark the changes with color)

The inequality (17) implies ∥ ∥ u t +1 -w t ∥ ∥ 2 ≤ 2 ∥ ∥ u t +1 -u t ∥ ∥ 2 +2 ∥ u t -w t ∥ 2 and

We analyze inequalities (46) in the following lemma:

Lemma E.10 (First Symbolically Computed) . Assume that for the parameter ¯ L, the inequalities from Sections I and J hold. Then, for all t ≥ 0 , exists ρ in (90) , κ in (88) , λ in (82) , and ν t in (83) such that (46) holds.

We proof lemma separately in Section G. Using the lemma, we have

Let us separately analyze the terms w.r.t. D f ( z t , y t +1 ) and E t [ ∥ ∥ u t +1 -u t ∥ ∥ 2 ] :

Lemma E.11 (Second Symbolically Computed) . Consider the parameters ρ, κ, λ, and ν t from Lemma E.10. Assume that for the parameter ¯ L, the inequalities from Sections L and N hold, and the step size θ t +1 ≤ 1 / 4 for all t ≥ 0 . Then, for all t ≥ 0 , the following inequalities are satisfied:

and

We prove Lemma E.11 in Section H. Using the lemma, we get

Note that 0 ≤ ν t +1 ≤ ν t for all t ≥ 0 , since θ t +1 is a non-increasing sequence (see Lemma E.1 and the definition of ν t in (83)). Using

for all t ≥ 0 , we obtain

Let us multiply the inequality by γ t +1 pθ t +1 :

It is left to use that Γ t +1 := Γ t + γ t +1 and γ t +1 = pθ t +1 Γ t +1 (see Lemma E.1) and take the full expectation to obtain (45).

In the proof we require that, for the parameter ¯ L, the inequalities from Sections I, J, L and N hold. In Section O, we show that these inequalities follow from (44).

E.5 Strongly-convex case

Theorem E.12. Suppose that Assumptions 1.2, 1.1, 1.3, 2.4 hold. Let

β = 1 ω +1 , θ min = 1 4 min { 1 , α p , τ p , β p } , h 0 i = ∇ f i ( z 0 ) for all i ∈ [ n ] , w 0 = u 0 , k 0 = ∇ f ( z 0 ) , v 0 = ∇ f ( z 0 ) , and Γ 0 ≥ 1 . Then Algorithm 1 guarantees that

where

Remark E.13 . Up to a constant factor of 2 , one can see that the optimal Γ 0 in (50) from Theorem E.12 equals Γ 0 = ¯ L / µ . But the dependence on Γ 0 is under the logarithm, so if the dependence on the logarithm is not critical, one can take any Γ 0 ≥ 1 .

Proof. All conditions from Theorem E.9 are satisfied. Let us sum the inequality (45) for t = 0 to T -1 :

Using the initial conditions and the non-negativity of the terms, we get

It is left to use the definitions of L and θ min .

E.6 General convex case

Let us use Theorem E.9 to analyze the general convex case ( µ can possibly be equal to zero):

Theorem E.14. Suppose that Assumptions 1.2, 1.1, 1.3, 2.4 hold. Let

β = 1 ω +1 , θ min = 1 4 min { 1 , α p , τ p , β p } , h 0 i = ∇ f i ( z 0 ) for all i ∈ [ n ] , w 0 = u 0 , k 0 = ∇ f ( z 0 ) , v 0 = ∇ f ( z 0 ) , and Γ 0 ∈ [1 , ¯ L / L ] . Then Algorithm 1 returns ε -solution, i.e., E [ f ( z T ) ] -f ( x ∗ ) ≤ ε, after

iterations, where

Remark E.15 . One can see that the optimal Γ 0 in (52) from Theorem E.14 equals Γ 0 = ¯ L / L . But the dependence on Γ 0 is under the logarithm, so if the dependence on the logarithm is not critical, one can take any Γ 0 ∈ [1 , ¯ L / L ] .

Proof. All conditions from Theorem E.9 are satisfied. Let us sum the inequality (45) for t = 0 to T -1 :

Using the initial conditions, the non-negativity of the terms, and Γ 0 ≤ ¯ L / L , we get

Using the L -smoothness and µ ≤ L ≤ ¯ L, we have

where ¯ t := max {⌈ 1 pθ min log 1 2Γ 0 pθ 2 min ⌉ , 0 } . The last inequalities guarantees that Algorithm 1 returns ε -solution after (52) iterations.

E.7 Choosing optimal parameters

Theorem 5.2. Choose r ∈ [0 , 1] and let µ r ω,α := rd (1 -r ) K ω + rK α . In view of Theorem 5.1, the values p = min { 1 ω +1 , 1 µ r ω,α } and τ = p 1 / 3 ( ω +1) 2 / 3 minimize max L max ∈ [ L,nL ] m r new . This choice leads to the following number of communication rounds:

The total communication complexity thus equals m r realistic = ˜ Θ(((1 -r ) K ω + rK α ) T realistic + d ) .

Proof. We implicitly assume that p ∈ (0 , 1] and τ ∈ (0 , 1] . Using (12) and (4), we have

Note that only T depends on L max and τ. We have min τ max T

The last term attains the minimum when

Therefore, we get

where we use that

It is left to find

where A := (1 -r ) K ω + rK α ≥ 0 and B := rd ≥ 0 . Note that A and B do not depend on p. If p ≥ A / B , then ˜ Θ( AT ′ + BpT ′ ) = ˜ Θ( BpT ′ ) . The term pT ′ is non-decreasing function w.r.t. p. For p ≥ A / B , it means that an optimal point p = A / B . Thus the argmin (55) is equivalent to

where Q 0 := { p ∣ ∣ p ≤ A B } . Thus, we have

The next observation is that this argmin is non-decreasing when p ≥ 1 / ω +1 . It means that the minimum attains at some point p ∈ Q 1 := { p | p ≤ 1 / ω +1 , p ∈ Q 0 } . Using this information, we can eliminate the redundant terms (for instance, A √ L ( ω +1) αµ ≤ A √ L αpµ for p ∈ Q 1 ) and get an equivalent argmin

The last term attains the minimum at

Using (56) and (53), an optimal τ is

We substitute (56) and (57) to (10) and obtain (13). We use Lemma 1.4 to eliminate the redundant terms. Note that µ r ω,α := B / A .

Using (12) and (56), we have

Note that

Theorem 5.4. Choose r ∈ [0 , 1] , and let µ r ω,α := rd (1 -r ) K ω + rK α . In view of Theorem 5.1, the values p and τ given by (63) and (58) , respectively, minimize m r new from (10) . This choice leads to the following number of communication rounds:

The total communication complexity thus equals m r optimistic = ˜ Θ(((1 -r ) K ω + rK α ) T optimistic + d ) .

Proof. We implicitly assume that p ∈ (0 , 1] and τ ∈ (0 , 1] . We start the proof as in Theorem 5.2. Using (12) and (4), we have

Unlike Theorem 5.2, we know the ratio L max / L , thus min τ T

The last term attains the minimum when

Therefore, we get

T ′ := min τ T

It is left to find

where A := (1 -r ) K ω + rK α ≥ 0 and B := rd ≥ 0 . Note that A and B do not depend on p. If p ≥ A / B , then ˜ Θ( AT ′ + BpT ′ ) = ˜ Θ( BpT ′ ) . The term pT ′ is non-decreasing function w.r.t. p. For p ≥ A / B , it means that an optimal point p = A / B . Thus the argmin (60) is equivalent to

It means that for p ≥ ( Ln L max ) 1 / 3 1 ω +1 , the argmin (61) is equivalent to

Since all terms are non-increasing functions of p, the minimum is attained at a point p = ( Ln L max ) 1 / 3 1 ω +1 for all p ≥ ( Ln L max ) 1 / 3 1 ω +1 . Let us define

The last observation means that the argmin (61) is equivalent to

where we eliminate the redundant terms using the additional information p ≤ ( Ln L max ) 1 / 3 1 ω +1 . In particular,

for all p ≤ ( Ln L max ) 1 / 3 1 ω +1 . Without the condition p ∈ Q 1 , the argmin (62) attains the minimum at a point

Considering the condition p ∈ Q 1 and µ r ω,α := B / A , we have

It is left carefully to substitute (63) to

and obtain (14). The proof of m r realistic = ˜ Θ(((1 -r ) K ω + rK α ) T realistic ) is the same as in Theorem 5.2.

E.8 Comparison with EF21 + DIANA

Theorem 6.1. For all r ∈ [0 , 1] , m r realistic = ˜ O ( m r EF21-P + DIANA ) .

Proof. Using the inequality of arithmetic and geometric means, i.e., √ xy ≤ x + y 2 for all x, y ≥ 0 , and L ≥ µ, we have

From the definition of µ r ω,α := rd / K r ω,α , we get

Using the inequality of arithmetic and geometric means again and r ≤ 1 , we obtain

The last equality means that m r realistic = ˜ ( m r EF21-P + DIANA ) for all r [0 , 1]

E.9 Comparison with AGD

Theorem 6.2. For all r ∈ [0 , 1] and for all K ∈ [ d ] , let us take the Rand K and Top K compressors with the parameters (expected densities) i) K ω = K and K α = min {⌈ 1 -r / r K ⌉ , d } for r ∈ [0 , 1 / 2 ] , ii) K ω = min {⌈ r / 1 -r K ⌉ , d } and K α = K for r ∈ ( 1 / 2 , 1] . Then we have m r realistic = ˜ O ( m AGD ) .

Proof. Consider that r ∈ [0 , 1 / 2] , then K ω = K, K α = min {⌈ 1 -r / r K ⌉ , d } . Therefore, we have

and

O ∈ .

Using this observation, we obtain µ r ω,α := rd K r ω,α ≤ rd (1 -r ) K ≤ d K . Note that α ≥ K α / d and ω ≤ d / K ω -1 for Top K and Rand K . Thus α ≥ min { (1 -r ) K rd , 1 } and ω ≤ d K -1 . We substitute the bounds to (16) and obtain

Since r ∈ [0 , 1 / 2] and K ≤ d, one can easily show that

It is left to use Lemma 1.4, to get m r realistic = ˜ O ( d √ L µ ) = ˜ O ( m AGD ) for all r ∈ [0 , 1 / 2] .

Assume that r ∈ (1 / 2 , 1] , then K ω = min {⌈ r / 1 -r K ⌉ , d } and K α = K. Using the same reasoning, we have

By substituting these inequalities to (16), we obtain

Using Lemma 1.4, one can easily show that m r realistic = ˜ O ( d √ L µ ) = ˜ O ( m AGD ) for all r ∈ (1 / 2 , 1] .

F Auxillary Inequalities For ¯ L

We now prove useful bounds for ¯ L.

Lemma F.1 (Auxillary Inequalities) . Assume that the constraint (44) hold, and a constant c = 660508 . Then

Proof. The inequalities (64) and (66) follow from (44). The inequality (65) follows from (64) and (66):

The inequalities (67) and (68) follow from (44). The inequality (69) follows from (68) and β ∈ (0 , 1] :

Using Lemma 1.4, (67), (68), and (69), the inequalities (70), (71), and (72) follow from

Next, using Lemma 1.4, and x + y 2 ≥ √ xy for all x, y ≥ 0 , the inequality (73) follows from

The inequality (74) follows from

The inequalities (75) and (76) follow from (73), (74), and α ∈ (0 , 1] :

The inequalities (77) and (78) follow from (44), and (79) follows from (77) and α ∈ (0 , 1] . Using Lemma 1.4, and x + y + z 3 ≥ ( xyz ) 1 / 3 for all x, y, z ≥ 0 , the inequalities (80) and (81) follows from

G Proof of Lemma E.10 (First Symbolically Computed)

We use the notations from the proof of Theorem E.9.

Lemma E.10 (First Symbolically Computed) . Assume that for the parameter ¯ L, the inequalities from Sections I and J hold. Then, for all t ≥ 0 , exists ρ in (90) , κ in (88) , λ in (82) , and ν t in (83) such that (46) holds.

Proof. The inequalities (46) are equivalent to

Let us take

to ensure that the third inequality holds. It left to find the parameters such that

Let us take where we additionally define

to ensure that the first inequality holds. It left to find the parameters κ and ρ such that

Using Lemma E.1, we have γ t +1 θ t +1 ¯ L +Γ t +1 µ ≤ γ t +1 θ t +1 ¯ L +Γ t µ ≤ 1 ¯ L , so it is sufficient to show that stronger inequalities hold:

From this point all formulas in this lemma are generated by the script from Section P (see Section 4 in Section P). We use the SymPy library (Meurer et al., 2017).

Using the definition of ̂ ν, the left hand side of (86) equals

where we grouped the terms w.r.t. κ. Let us take ¯ L such that the bracket is less or equal to 1 / 2 . We define the constraints in Section I. Therefore, (86) holds if

Using the definition of ̂ ν and κ, the left hand side of (85) equals

where we grouped the terms w.r.t. ρ. Let us take ¯ L such that the bracket is less or equal to 1 / 2 . We define the constraints in Section J. Therefore, (85) holds if

Finally, under the constraints from Sections I and J on ¯ L , the choices of parameters (90), (88), (83) and (82) insure that (46) holds.

H Proof of Lemma E.11 (Second Symbolically Computed)

We use the notations from the proof of Theorem E.9.

Lemma E.11 (Second Symbolically Computed) . Consider the parameters ρ, κ, λ, and ν t from Lemma E.10. Assume that for the parameter ¯ L, the inequalities from Sections L and N hold, and the step size θ t +1 ≤ 1 / 4 for all t ≥ 0 . Then, for all t ≥ 0 , the following inequalities are satisfied:

Proof. Since p ≥ 0 and D f ( z t , y t +1 ) ≥ 0 for all t ≥ 0 , the inequality (47) is satisfied if

Note that θ t +1 ≤ 1 4 for all t ≥ 0 . Therefore, it is sufficient to show that

In the view of (83), we have to show that

Using Lemma E.1, we have γ t +1 θ t +1 ¯ L +Γ t +1 µ ≤ γ t +1 θ t +1 ¯ L +Γ t µ ≤ 1 ¯ L , so it is sufficient to show that

From this point all formulas in this lemma are generated by the script from Section P (see Section 5 in Section P).

Let us substitute (90), (88), (82), and (84) to the last inequality and obtain the inequality from Section K. The conditions from Section L insure that the inequality from Section K holds. It left to prove (48). Since p ≥ 0 , E t [ ∥ ∥ u t +1 -u t ∥ ∥ 2 ] ≥ 0 and θ 2 t +1 ≥ 0 for all t ≥ 0 , the inequality (48) holds if

Let us substitute (90), (88) and (82) to the last inequality and obtain the inequality from Section M. The inequality from Section M holds if ¯ L satisfy the inequalities from Section N.

I Symbolically Computed Constraints for ¯ L Such That The Term w.r.t. κ is less or equal 1 / 2 in (87)

J Symbolically Computed Constraints for ¯ L Such That The Term w.r.t. ρ is less or equal 1 / 2 in (89)

K Symbolically Computed Expression (91)

4294967296 L 4 ˆ L 4 ω 2 p 2 ¯ L 8 α 6 β 2 n 2 + 8589934592 L 4 ˆ L 4 ω 2 p 3 ¯ L 8 α 6 β 3 n 2 + 8192 LL max ˆ L 2 ω 2 p ¯ L 4 α 2 βn 2

8192 LL max ˆ L 2 ω 2 p 2 ¯ L 4 α 2 β 2 n 2 + 65536 LL max ωpτ ¯ L 2 α 2 βn + 65536 LL max ωp 2 τ ¯ L 2 α 2 β 2 n

65536 LL max ˆ L 2 ω 2 pτ ¯ L 4 α 3 n 2 + 262144 L ˆ L 2 ωp 2 ¯ L 3 α 2 βnτ + 262144 L ˆ L 2 ωp 3 ¯ L 3 α 2 β 2 nτ

524288 L ˆ L 2 ωp 3 ¯ L 3 α 2 βnτ 2 + 524288 L ˆ L 2 ωp 4 ¯ L 3 α 2 β 2 nτ 2 + 524288 L max ˆ L 2 ω 2 p 2 τ ¯ L 3 α 2 βn 2

524288 L max ˆ L 2 ω 2 p 3 τ ¯ L 3 α 2 β 2 n 2 + 1048576 L 3 L max ωp 2 ¯ L 4 α 3 βnτ + 1048576 L 3 L max ωp 3 ¯ L 4 α 3 β 2 nτ

1048576 L 2 ˆ L 2 ωp 2 ¯ L 4 α 3 βnτ + 1048576 L 2 ˆ L 2 ωp 3 ¯ L 4 α 3 β 2 nτ + 1048576 L 2 L max ˆ L 2 ω 2 p ¯ L 5 α 3 βn 2

2097152 L 3 L max ωp 3 ¯ L 4 α 3 βnτ 2 + 2097152 L 3 L max ωp 4 ¯ L 4 α 3 β 2 nτ 2 + 2097152 L 2 ˆ L 2 ωpτ ¯ L 4 α 4 βn

2097152 L 2 ˆ L 2 ωp 2 τ ¯ L 4 α 4 β 2 n + 2097152 L 2 ˆ L 2 ωp 3 ¯ L 4 α 3 βnτ 2 + 2097152 L 2 ˆ L 2 ωp 4 ¯ L 4 α 3 β 2 nτ 2

8388608 L 3 ˆ L 2 ωp 2 ¯ L 5 α 4 βnτ + 8388608 L 3 ˆ L 2 ωp 3 ¯ L 5 α 4 β 2 nτ + 8388608 L 3 L max ˆ L 2 ω 2 p ¯ L 6 α 4 βn 2

8388608 L max ˆ L 2 ω 2 p 4 τ ¯ L 3 α 2 β 4 n 2 + 8388608 L max ˆ L 2 ω 2 p 2 τ ¯ L 3 α 2 β 2 n 2 + 8388608 L max ˆ L 4 ω 3 p 2 τ ¯ L 5 α 3 βn 3

16777216 L 3 ˆ L 2 ωp 3 ¯ L 5 α 4 βnτ 2 + 16777216 L 3 ˆ L 2 ωp 4 ¯ L 5 α 4 β 2 nτ 2 + 16777216 L ˆ L 4 ω 2 p 2 τ ¯ L 5 α 4 βn 2

16777216 L ˆ L 4 ω 2 p 3 τ ¯ L 5 α 4 β 2 n 2 + 16777216 L max ˆ L 2 ω 2 p 3 τ ¯ L 3 α 2 β 3 n 2 + 33554432 L 2 L max ˆ L 2 ω 2 p 4 ¯ L 5 α 3 β 4 n 2

34603008 L 2 L max ˆ L 2 ω 2 p 2 ¯ L 5 α 3 β 2 n 2 + 67108864 L 4 ˆ L 2 ωp 2 ¯ L 6 α 5 βnτ + 67108864 L 4 ˆ L 2 ωp 3 ¯ L 6 α 5 β 2 nτ

67108864 L 2 L max ˆ L 2 ω 2 p 3 ¯ L 5 α 3 β 3 n 2 + 134217728 L 4 ˆ L 2 ωp 3 ¯ L 6 α 5 βnτ 2 + 134217728 L 4 ˆ L 2 ωp 4 ¯ L 6 α 5 β 2 nτ 2

134217728 L max ˆ L 4 ω 3 p 4 τ ¯ L 5 α 3 β 4 n 3 + 134217728 L max ˆ L 4 ω 3 p 2 τ ¯ L 5 α 3 β 2 n 3 + 134217728 L 3 L max ˆ L 2 ω 2 p 4 ¯ L 6 α 4 β 4 n 2

134217728 L 2 ˆ L 4 ω 2 p 2 τ ¯ L 6 α 5 βn 2 + 134217728 L 2 ˆ L 4 ω 2 p 3 τ ¯ L 6 α 5 β 2 n 2 + 142606336 L 3 L max ˆ L 2 ω 2 p 2 ¯ L 6 α 4 β 2 n 2

268435456 L ˆ L 4 ω 2 p 4 τ ¯ L 5 α 4 β 4 n 2 + 268435456 L ˆ L 4 ω 2 p 2 τ ¯ L 5 α 4 β 2 n 2 + 268435456 L max ˆ L 4 ω 3 p 5 τ ¯ L 5 α 3 β 4 n 3

268435456 L max ˆ L 4 ω 3 p 3 τ ¯ L 5 α 3 β 3 n 3 + 268435456 L 3 L max ˆ L 2 ω 2 p 3 ¯ L 6 α 4 β 3 n 2 + 276824064 L max ˆ L 4 ω 3 p 3 τ ¯ L 5 α 3 β 2 n 3

536870912 L ˆ L 4 ω 2 p 3 τ ¯ L 5 α 4 β 3 n 2 + 536870912 L max ˆ L 4 ω 3 p 4 τ ¯ L 5 α 3 β 3 n 3 + 536870912 L 2 L max ˆ L 4 ω 3 p 4 ¯ L 7 α 4 β 4 n 3

536870912 L 2 L max ˆ L 4 ω 3 p 2 ¯ L 7 α 4 β 2 n 3 + 1073741824 L 2 L max ˆ L 4 ω 3 p 3 ¯ L 7 α 4 β 3 n 3 + 1073741824 L 2 ˆ L 4 ω 2 p 4 τ ¯ L 6 α 5 β 4 n 2

1073741824 L 2 ˆ L 4 ω 2 p 2 τ ¯ L 6 α 5 β 2 n 2 + 2147483648 L 3 L max ˆ L 4 ω 3 p 4 ¯ L 8 α 5 β 4 n 3 + 2147483648 L 3 L max ˆ L 4 ω 3 p 2 ¯ L 8 α 5 β 2 n 3

2147483648 L 2 ˆ L 4 ω 2 p 3 τ ¯ L 6 α 5 β 3 n 2 + 4294967296 L max ˆ L 6 ω 4 p 5 τ ¯ L 7 α 4 β 4 n 4 + 4294967296 L max ˆ L 6 ω 4 p 3 τ ¯ L 7 α 4 β 2 n 4

4294967296 L 3 L max ˆ L 4 ω 3 p 3 ¯ L 8 α 5 β 3 n 3 + 4294967296 L 3 ˆ L 4 ω 2 p 5 ¯ L 7 α 5 β 4 n 2 τ + 4294967296 L 3 ˆ L 4 ω 2 p 3 ¯ L 7 α 5 β 2 n 2 τ

8589934592 L ˆ L 6 ω 3 p 5 τ ¯ L 7 α 5 β 4 n 3 + 8589934592 L ˆ L 6 ω 3 p 3 τ ¯ L 7 α 5 β 2 n 3 + 8589934592 L max ˆ L 6 ω 4 p 4 τ ¯ L 7 α 4 β 3 n 4

+ 8589934592

L

α

L

β

3

n

L

ˆ

2

4

τ

¯

7

5

3

+ 8589934592

+ 8589934592

β

L

4

3

n

α

L

ˆ

2

τ

5

L

7

¯

ω

4

2

L

¯

7

α

5

β

L

2

3

L

ˆ

2

n

τ

4

ω

2

17179869184 L ˆ L 6 ω 3 p 4 τ ¯ L 7 α 5 β 3 n 3 + 17179869184 L 3 ˆ L 4 ω 2 p 5 ¯ L 7 α 5 β 3 n 2 τ 2 + 17179869184 L 4 ˆ L 4 ω 2 p 5 ¯ L 8 α 6 β 4 n 2 τ

17179869184 L 4 ˆ L 4 ω 2 p 3 ¯ L 8 α 6 β 2 n 2 τ + 34359738368 L 2 ˆ L 6 ω 3 p 5 τ ¯ L 8 α 6 β 4 n 3 + 34359738368 L 2 ˆ L 6 ω 3 p 3 τ ¯ L 8 α 6 β 2 n 3

+ 34359738368

L

α

β

8

6

4

¯

+ 68719476736

L

4

n

2

L

ˆ

4

τ

2

L

¯

L

α

8

6

+ 4194304

2

ˆ

L

β

6

n

3

LL

¯

α

4

L

+ 8388608

3

ˆ

L

3

βn

2

L

¯

2

2

L

α

5

L

+ 34603008

3

βn

ˆ

L

2

τ

LL

¯

4

L

+ 67108864

L

L

¯

4

3

n

L

α

β

8

6

+ 34359738368

τ

¯

L

α

β

8

6

+ 68719476736

+ 4194304

L

ˆ

2

τ

4

L

3

n

LL

L

ˆ

2

¯

L

α

4

+ 33554432

τ

ω

4

4

p

2

ω

2

ˆ

L

3

β

n

2

LL

¯

ω

2

2

4

α

L

τ

L

ˆ

β

3

+ 67108864

n

4

2

2

2

LL

¯

4

α

L

3

+ 134217728

L

ˆ

β

ω

4

2

p

p

3

2

p

2

ω

n

3

2

L

α

2

L

β

n

L

τ

¯

5

3

4

2

L

+ 34359738368

L

α

¯

8

6

β

2

L

4

L

ˆ

n

τ

2

5

4

2

LL

¯

L

α

4

+ 1048576

+ 4194304

L

2

¯

α

L

L

ˆ

3

βn

2

L

3

L

ω

2

2

ω

p

2

4

pτ

2

ω

ˆ

2

βn

5

+ 33554432

τ

L

3

¯

L

6

α

L

τ

βn

4

+ 67108864

L

ˆ

2

τ

LL

¯

L

α

6

4

+ 138412032

L

p

2

2

ω

4

2

2

p

ω

ˆ

3

βn

2

L

α

L

β

n

L

ˆ

τ

¯

5

3

2

2

¯

L

3

α

2

ˆ

L

β

3

L

6

α

2

2

2

n

L

4

2

βn

L

τ

ˆ

2

268435456 L 2 L max ˆ L 2 ω 2 p 6 ¯ L 5 α 3 β 4 n 2 τ 2 + 268435456 L 2 L max ˆ L 2 ω 2 p 4 ¯ L 5 α 3 β 3 n 2 τ + 276824064 L 2 L max ˆ L 2 ω 2 p 4 ¯ L 5 α 3 β 2 n 2 τ 2

536870912 LL max ˆ L 4 ω 3 p 4 τ ¯ L 6 α 4 β 4 n 3 + 536870912 LL max ˆ L 4 ω 3 p 2 τ ¯ L 6 α 4 β 2 n 3 + 536870912 L 2 L max ˆ L 2 ω 2 p 5 ¯ L 5 α 3 β 3 n 2 τ 2

536870912 L 3 L max ˆ L 2 ω 2 p 5 ¯ L 6 α 4 β 4 n 2 τ + 570425344 L 3 L max ˆ L 2 ω 2 p 3 ¯ L 6 α 4 β 2 n 2 τ + 1073741824 LL max ˆ L 4 ω 3 p 5 τ ¯ L 6 α 4 β 4 n 3

1073741824 LL max ˆ L 4 ω 3 p 3 τ ¯ L 6 α 4 β 3 n 3 + 1073741824 L 3 L max ˆ L 2 ω 2 p 6 ¯ L 6 α 4 β 4 n 2 τ 2 + 1073741824 L 3 L max ˆ L 2 ω 2 p 4 ¯ L 6 α 4 β 3 n 2 τ

1140850688 LL max ˆ L 4 ω 3 p 3 τ ¯ L 6 α 4 β 2 n 3 + 1140850688 L 3 L max ˆ L 2 ω 2 p 4 ¯ L 6 α 4 β 2 n 2 τ 2 + 2147483648 LL max ˆ L 4 ω 3 p 4 τ ¯ L 6 α 4 β 3 n 3

+ 17179869184

L

L

L

ω

p

ˆ

3

4

3

4

α

β

n

τ

L

¯

5

3

3

8

+ 17179869184

3

L

5

β

2

L

n

3

τ

2

ˆ

4

L

ω

3

p

4

+ 34359738368

LL

α

5

β

3

n

4

ˆ

6

L

ω

4

p

4

τ

3

2

p

2

2

ω

τ

2

p

τ

4

2

ˆ

2

τ

p

3

ω

2

p

ω

ω

ω

2

3

2

2

p

p

p

ω

6

4

2

2

p

ω

2

2

τ

3

p

ω

2

2

p

2

p

6

ω

2

p

4

2

p

4

3

¯

8

L

α

5

¯

8

L

3

L Symbolically Computed Constraints for ¯ L Such That The Inequality from Section K Holds

278528

¯

L

4

L

α

3

2

ˆ

L

β

2

2

ωp

n

1

326

(133)

2097152

¯

L

5

L

α

4

3

ˆ

L

βn

1

326

(134)

2097152

¯

L

5

L

α

4

3

β

ˆ

L

2

2

n

1

326

(135)

2

2

ωp

2

ωp

M Symbolically Computed Expression (92)

N Symbolically Computed Constraints for ¯ L Such That The Inequality from Section M Holds

O Symbolical Check That The Constraints from Sections I, J, L and N Follow From The Constraint (44)

Note that the inequalities from Lemma F.1 follow from (44). Therefore, the inequalities from Sections I, J, L and N follow from (44), if they follow from the inequalities from Lemma F.1. We now present it 7 . These results are checked and generated using the script in Section P (see Section 6 in Section P)

Jupyter Notebook for Symbolic Computations

1 Import Necessary Libraries

2 Initialize a File for Results

3 Initialize Symbols From the Paper

4 Assistant for 'First Symbolically Computed' Lemma

4.1 Calculate Expressions

4.2 Display Them

4.3 Symbolically Calculate The Steps From The Proof

5 Assistant for 'Second Symbolically Computed' Lemma

5.1 First Inequality

5.2 Second Inequality

6 Check That The Constraints Follow From The Inequalities from The 'Auxillary Inequalities' Lemma.

6.1 The Inequalities from The 'Auxillary Inequalities' Lemma:

6.2 Search The Right Inequalities for Each Constraint

P.1 File utils.py

Q Experiments

Q.1 Setup

Wenowconduct experiments on the practical logistic regression task with LIBSVM datasets (Chang and Lin, 2011) (under the 3-clause BSD license). The experiments were implemented in Python 3.7.9. The distributed environment was emulated on machines with Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz. In each plot we show the relations between the total number of coordinates transmitted from and to the server and function values. The parameters of the algorithms are taken as suggested by the corresponding theory, except for the stepsizes that we fine-tune from a set { 2 i | i ∈ [ -20 , 20] } . For 2Direction , we use parameters from Theorem 5.2 and finetune the step size ¯ L.

We solve the logistic regression problem:

where x 1 , . . . , x c ∈ R d , c is the number of unique labels, a ij ∈ R d is a feature of a sample on the i th worker, y ij is a corresponding label and m is the number of samples located on the i th worker. The Rand K compressor is used to compress information from the workers to the server, the Top K compressor is used to compress information from the server to the workers. The performance of algorithms is compared on CIFAR10 (Krizhevsky et al., 2009) (# of features = 3072 , # of samples equals 50 , 000 ), and real-sim (# of features = 20958 , # of samples equals 72 , 309 ) datasets.

Q.2 Results

In Figure 2, 3 and 4 we provide empirical communication complexities of our experiments. For each algorithm, we show the three best experiments. The experiments are collaborative with our theory. 2Direction enjoys faster convergence rates than EF21-P + DIANA and AGD .

0

1

2

3

4

5

#bits / n (workers-to-server)

1e8

10

2

10

1

10 0

f

(

x t

)

AGD (No Compression): Step size: 128.0

AGD (No Compression): Step size: 64.0

AGD (No Compression): Step size: 256.0

EF21-P + DIANA: Step size: 256.0

EF21-P + DIANA: Step size: 128.0

EF21-P + DIANA: Step size: 64.0

2Direction: Step size: 512.0

2Direction: Step size: 256.0

2Direction: Step size: 64.0

0

1

2

3

4

5

#bits / n (server-to-workers)

1e8

10

2

10

1

10 0

f

(

x t

)

AGD (No Compression): Step size: 128.0

AGD (No Compression): Step size: 64.0

AGD (No Compression): Step size: 256.0

EF21-P + DIANA: Step size: 256.0

EF21-P + DIANA: Step size: 128.0

EF21-P + DIANA: Step size: 64.0

2Direction: Step size: 512.0

2Direction: Step size: 256.0

2Direction: Step size: 64.0

0.0

0.5

1.0

1.5

2.0

#bits / n (workers-to-server)

1e10

1.35 × 10 0

1.4 × 10 0

1.45 × 10 0

1.5 × 10 0

1.55 × 10 0

1.6 × 10 0

1.65 × 10 0

1.7 × 10 0

f

(

x t

)

EF21-P + DIANA: Step size: 0.03125

EF21-P + DIANA: Step size: 0.015625

EF21-P + DIANA: Step size: 0.0078125

2Direction: Step size: 0.0009765625

2Direction: Step size: 0.000244140625

2Direction: Step size: 0.0001220703125

0.0

0.5

1.0

1.5

2.0

#bits / n (server-to-workers)

1e10

1.35 × 10 0

1.4 × 10 0

1.45 × 10 0

1.5 × 10 0

1.55 × 10 0

1.6 × 10 0

1.65 × 10 0

1.7 × 10 0

f

(

x t

)

EF21-P + DIANA: Step size: 0.03125

EF21-P + DIANA: Step size: 0.015625

EF21-P + DIANA: Step size: 0.0078125

2Direction: Step size: 0.0009765625

2Direction: Step size: 0.000244140625

2Direction: Step size: 0.0001220703125

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

#bits / n (workers-to-server)

1e10

1.35 × 10 0

1.4 × 10 0

1.45 × 10 0

1.5 × 10 0

1.55 × 10 0

1.6 × 10 0

1.65 × 10 0

1.7 × 10 0

f

(

x t

)

EF21-P + DIANA: Step size: 0.0625

EF21-P + DIANA: Step size: 0.03125

EF21-P + DIANA: Step size: 0.015625

2Direction: Step size: 0.0009765625

2Direction: Step size: 0.00048828125

2Direction: Step size: 0.000244140625

0.0

0.5

1.0

1.5

2.0

2.5

#bits / n (server-to-workers)

1e10

1.35 × 10 0

1.4 × 10 0

1.45 × 10 0

1.5 × 10 0

1.55 × 10 0

1.6 × 10 0

1.65 × 10 0

1.7 × 10 0

f

(

x t

)

EF21-P + DIANA: Step size: 0.0625

EF21-P + DIANA: Step size: 0.03125

EF21-P + DIANA: Step size: 0.015625

2Direction: Step size: 0.0009765625

2Direction: Step size: 0.00048828125

2Direction: Step size: 0.000244140625

R Convergence Rate of CANITA obtained by Li and Richt´ arik (2021)

In their Equation (54), Li and Richt´ arik (2021) derive the following bound for their CANITA method:

In the regime when ω ≥ n, choosing b = ω and β = Θ ( ω n ) in their Equation (10) gives

This means that the correct convergence rate of the CANITA method (Li and Richt´ arik, 2021) is

Comparing this result with our Theorem E.14 describing the convergence of our method 2Direction , one can see that in the low accuracy regimes (in particular, when ω ε 1 / 3 dominates in (329)), our result improves Θ ( 1 ε 1 / 3 ) to at least Θ ( log 1 ε ) . However, the dependence Θ ( log 1 ε ) should not be overly surprising as it was observed by Lan et al. (2019) already, albeit in a somewhat different context.

S Comparison with ADIANA

We now want to check that our rate (14) restores the rate from (Li et al., 2020). Since ADIANA only compresses from the workers to the server, let us take r = 0 , the identity compressor operator C P ( x ) = x for all x ∈ R d , which does not perform compression, and, as in (Li et al., 2020), consider the optimistic case, when L max = L . For this compressor, we have α = 1 in (2). Note that µ r ω,α = 0 . Thus the iteration complexity (14) equals

where we use Young's inequality: √ L µ √ ( ω +1) n 1 / 3 ≤ √ L µ √ 1 3 × 1 3 + 2 3 ( ω +1) 3 / 2 √ n . Without the serverto-worker compression, Algorithm 1 has the same iteration (330) and communication complexity as (Li et al., 2020).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes 2Direction, a new GD-based distributed training algorithm for convex optimization problems. The improvement of 2Direction over previous works that target communication complexity in the convex case is that 2Direction is both accelerated and allows for bi-directional compression. 2Direction is proven to offer better communication complexity than previous works in certain regimes and not being worse otherwise.\n\nSTRENGTHS:\n1.\tThe scope of the paper and the main problem are well-defined and clear. \n2.    The paper considers bi-directional compression. SO far, bi-directional compression received less attention in the literature that mostly considers only compressing uplink communication. \n2.\tThe paper’s theoretical analysis and results are mostly clear and solid. The paper theoretically compares 2Direction to several SOTA alternatives.\n3.\tThe design choices of 2Direction are well articulated.\n\nWEAKNESSES:\n1.\tThe communication complexity analysis and its optimization are not fully clear. Specifically, there is dependency between the round cost and the number of communication rounds (i.e., via K and w). \n2.\t2Direction (Algorithm 1) requires more memory and compute resources in comparison to some previous methods such as DG, AGD and EF21-P + DIANA  (e.g., 2Direction requires arrays for storing h, v and solving (11)-(12)-(23)). This overhead and its consequences, especially in federated learning contexts, are not discussed.\n3.\tThe experimental part is somewhat weak (appears in the supplementary material and contains a single logistic regression problem with two datasets).\n4. The presentation can be improved. In particular, the paper is not self contained and relies on the supplementary material to convey main ideas. Notably: (i) Section 4 points to Sections B and C to articulate design choices; (ii) Section 5.2 points to equations 58 and 63\n\nQUESTIONS:\n1.\tCan the authors comment on weaknesses (1)(2)(3)?\n2.\tCan the authors shed more light on why with small probability exact values must be sent? Clearly the proof requires this, but more intuition on whether this is inherent or just an artifact of the proof technique would be helpful. \n3.\tThe comparison/analysis is done with TopK (downlink) and RandK (uplink) as example compressors. However, these compressors have weaker worst-case MSE guarantees in comparison to SOTA compression techniques that provide asymptotically better worst-case tradeoffs between communication and MSE. Can the authors shed light on that? In particular, if compressors with stronger guarantees are used, can that affect the communication complexity analysis? E.g., since line 10 in algorithm 1 can be viewed as an instance of the distributed mean estimation problem, can the usage of techniques such as [1]-[3] lead to better communication complexity or alter the conclusions? (this also relates to weakness (1))\n4.\tAssumption 2.4 – while the independence between the workers in the uplink direction is indeed desired, why independence with the downlink direction is important?  \n\n[1] Suresh, Ananda Theertha, et al. \"Distributed mean estimation with limited communication.\" International conference on machine learning. PMLR, 2017.\n\n[2] Davies, Peter, et al. \"New Bounds For Distributed Mean Estimation and Variance Reduction.\" International Conference on Learning Representations, 2021.\n\n[3] Vargaftik, Shay, et al. \"Eden: Communication-efficient and robust distributed mean estimation for federated learning.\" International Conference on Machine Learning. PMLR, 2022.\n\nLIMITATIONS:\nThe authors clearly stated a limitation of their work in Section 8. I do have few additional suggestions about the memory/computational efficiency aspect. Moreover, since federated learning (FL) is mentioned as motivation, it worth discussing whether 2Direction can be potentially extended to common FL scenarios with partial participation and stochastic gradients.\n\nI did not recognized potential negative broader impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

comments: SUMMARY:
This paper proposes a new distributed convex optimization problems with bidirectional compression. This algorithm, named as 2Direction, is claimed to be the first that improves upon the communication complexity of the vanilla accelerated gradient descent.

STRENGTHS:
1. The 2Direction method seems novel
2. The studied problem is well-motivated

WEAKNESSES:
1. The proof of the algorithm is rather complicated. I cannot verify the solidness of the established theorem given the very tight review timeline. The appendix I, J, K, L and O are very scary given the symbolically computed expressions. It is basically a computer-aided proof. I will discuss with other reviewers and chairs on how to check this proof.

2. In table 2, some terms are associated with $\Omega$ while the others are not. Please clarify the reason.

3. 2Direction uses a unbiased compressor and a contractive compressor in algorithm design. Can 2Direction use unbiaed compressors for both w2s and s2w compression? Can 2Direction use contractive compressors for both w2s and s2w compression?

4. Does 2Direction have the smallest communication complexity when r = 0 compared with existing uni-directional compression algorithms?

Since I cannot verify the correctness of the established theorem, I would recommend borderline reject at this stage.

QUESTIONS:
See above

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies 2Direction, a new method for compressed communications for the centralized distributed convex optimization problem, in the case where both worker-to-server and server-to-worker communications are compressed. Compared to previous work EF21-P+DIANA, it  guarantees a total communication complexity no worse than Nesterov's accelerated gradient methods.\n\nSTRENGTHS:\n* The paper introduces a new objective of minimizing the *total communication complexity*, which is well defined through equation 4 and clearly explained with the problematic lines 118-122.\n* To alleviate some of the burden of understanding the seemingly cumbersome algorithm and proof, efforts were made to describe some of the steps of the research/thought process that led to the final method in Section 4, and to provide a sketch of the proof in Section 7.\n* For reproducibility purposes, I appreciated the presence of the SymPy code used for the symbolic computation part of the proof.\n* The final result is original, seems to have required more than a *\"small tinkering\"* of previous methods (Section 4), and displays both theoretical and practical improvements over previous state of the art algorithms.\n\nWEAKNESSES:\n* While efforts were made to ease the understanding of the general ideas behind the algorithm and proof, they seem so cluttered that it deters from delving into the technical details.\n\nQUESTIONS:\n* line 130 claims a *\"new error feedback mechanism\"* but only compares it to the EF21-P method. What is the difference between equation 9 and other mechanisms such as MEM-SGD [1] ?\n* In the experiments part (appendix Q), why use a single value of $K$ ? Would it be possible to display the evolution of the convergence rate at different degrees of compression ?\n* line 1033, the stepsize $\\bar L$ is said to be finetuned. However, if we assume $L = L_{\\max}$ (which we can find for logistic regression), wouldn't equation 44 give a closed form expression of $\\bar L$ ?\n\n**Typos :**\n* The order of Equations 6 and 7 seems to have been reversed.\n* line 195: a *\"and\"* seems to have been left between *\"general analysis\"*.\n\n\n**References :**\n\n[1] Sebastian U Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. *Sparsified SGD with memory*. In NeurIPS, 2018.\n\nLIMITATIONS:\n*", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors considered a distributed convex optimization setting where both uplink and downlink communications should be considered. In this setting, the authors proposed an accelerated method called 2Direction which generalizes AGD into the bidirectional compressed communication setting and prove its convergence rate and communication complexity. Finally, empirical results are provided.

STRENGTHS:
For this new setting, the authors propose new strategies to make the adaption of AGD possible, which is nontrivial. Meanwhile, the bounds seem OK to me and can be applicable in the smooth transition from uplink to downlink.

WEAKNESSES:
Such a generalization for the bidirectional case fills an important gap, but it lacks enough motivational examples for why such settings deserve study. 

The algorithm name 2Direction is confusing as we don't really know what it means. It is better to follow the tradition and use the abbreviation of the full algorithm name. 

The authors show the motivation of the new strategy is the existing approaches failed in the bidirectional case but don't show the concrete reason why failed. Is the failure superficial or essential? 

The experiments should be included in the main body.

QUESTIONS:
See Weaknesses

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-29