Title: Improving the Worst-Case Bidirectional Communication Complexity for Nonconvex Distributed Optimization under Function Similarity
Venue: NeurIPS 2024
Accepted: yes
Keywords: Nonconvex Optimization

Distributed Optimization

Federated Learning

Compressed Communication
Abstract: Effective communication between the server and workers plays a key role in distributed optimization. In this paper, we focus on optimizing communication, uncovering inefficiencies in prevalent downlink compression approaches. Considering first the pure setup where the uplink communication costs are negligible, we introduce MARINA-P, a novel method for downlink compression, employing a collection of correlated compressors. Theoretical analysis demonstrates that MARINA-P with permutation compressors can achieve a server-to-worker communication complexity improving with the number of workers, thus being provably superior to existing algorithms. We further show that MARINA-P can serve as a starting point for extensions such as methods supporting bidirectional compression: we introduce M3, a method combining MARINA-P with uplink compression and a momentum step, achieving bidirectional compression with provable improvements in total communication complexity as the number of workers increases. Theoretical findings align closely with empirical experiments, underscoring the efficiency of the proposed algorithms.

Improving the Worst-Case Bidirectional Communication Complexity for Nonconvex Distributed Optimization under Function Similarity

Abstract

Effective communication between the server and workers plays a key role in distributed optimization. In this paper, we focus on optimizing communication, uncovering inefficiencies in prevalent downlink compression approaches. Considering first the pure setup where the uplink communication costs are negligible, we introduce MARINA-P , a novel method for downlink compression, employing a collection of correlated compressors. Theoretical analysis demonstrates that MARINA-P with permutation compressors can achieve a server-to-worker communication complexity improving with the number of workers, thus being provably superior to existing algorithms. We further show that MARINA-P can serve as a starting point for extensions such as methods supporting bidirectional compression: we introduce M3 , a method combining MARINA-P with uplink compression and a momentum step, achieving bidirectional compression with provable improvements in total communication complexity as the number of workers increases. Theoretical findings align closely with empirical experiments, underscoring the efficiency of the proposed algorithms.

1 Introduction

In federated learning (McMahan et al., 2017; Koneˇ cný et al., 2016) and large-scale machine learning (Ramesh et al., 2021; OpenAI, 2023), a typical environment consists of multiple devices working together to train a model. Facilitating this collaborative process requires the transmission of substantial information (e.g., gradients, current model) between these devices. In the centralized framework, communication takes place via a server. As a result, practical challenges arise due to the large size of machine learning models and network speed limitations, potentially creating a communication bottleneck (Kairouz et al., 2021; Wang et al., 2023a). One possible strategy to reduce this communication burden is to use lossy compression (Seide et al., 2014; Alistarh et al., 2017). Our paper focuses on this research direction.

We consider the following nonconvex distributed optimization task:

where x ∈ R d is the vector of parameters of the model, n is the number of workers and f i : R d → R , i ∈ [ n ] := { 1 , . . . , n } are smooth nonconvex functions. We investigate the scenario where the functions f i are stored on n distinct workers, each directly connected to the server via some

communication port (Kairouz et al., 2021). At present, we operate under the following generic assumptions:

Assumption 1.1. The function f is L -smooth, i.e., ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ ∀ x, y ∈ R d . Assumption 1.2. There exists f ∗ ∈ R such that f ( x ) ≥ f ∗ ∀ x ∈ R d .

In the nonconvex world, our goal is to find a (possibly) random point ¯ x such that E [ ∥∇ f (¯ x ) ∥ 2 ] ≤ ε. We refer to such a point an ε -stationary point.

1.1 Related Work

Before we discuss more advanced optimization methods, let us consider the simplest baseline: the gradient descent ( GD ) (Lan, 2020), which iteratively performs updates x t +1 = x t -γ ∇ f ( x t ) = x t -γ / n ∑ n i =1 ∇ f i ( x t ) . In the distributed setting, the method can be implemented as follows: each worker calculates ∇ f i ( x t ) and sends it to the server where the gradients are aggregated, after which the server takes the step and broadcasts x t +1 back to the workers. With step size γ = 1 / L , GD finds an ε -stationary point after O ( δ 0 L / ε ) steps, where δ 0 := f ( x 0 ) -f ∗ for a starting point x 0 . Since at each step the workers and the server send Θ( d ) coordinates/bits, the worker-to-server (w2s, uplink) and server-to-worker (s2w, downlink) communication costs are

Definition 1.3. The worker-to-server (w2s) and server-to-worker (s2w) communication complexities of a method are the expected number of coordinates/floats that a worker sends to the server and that the server sends to a worker, respectively, to find an ε -solution. The total communication complexity is the sum of these complexities.

Unbiased compressors. In this work, to perform lossy compression, we employ mappings from the following family:

Definition 1.4. A stochastic mapping C : R d → R d is an unbiased compressor if there exists ω ≥ 0 such that

We denote the family of such mappings by U ( ω ) . A canonical example is the Rand K ∈ U ( d / K -1) sparsifier, which preserves K random coordinates of a vector scaled by d / K (Beznosikov et al., 2020). More examples can be found in Wangni et al. (2018); Beznosikov et al. (2020); Szlendak et al. (2021); Horváth et al. (2022). A larger family of compressors, called biased compressors , also exists (see Section B). In this paper, we implicitly assume that compressors are mutually independent across iterations of algorithms.

Worker-to-server compression scales with n . Many previous works ignore the s2w communication costs and focus solely on w2s compression, assuming that broadcasting is free . For nonconvex objective functions, the current state-of-the-art w2s communication complexities are achieved by the MARINA and DASHA methods (Gorbunov et al., 2021; Szlendak et al., 2021; Tyurin and Richtárik, 2023b). Here, two additional assumptions are needed:

Assumption 1.5. The function f i is L i -smooth. We define ̂ L 2 := 1 n ∑ n i =1 L 2 i and L max := max i ∈ [ n ] L i .

Assumption 1.6. For all C ∈ U ( ω ) , all calls of C are mutually independent. 4

Under Assumptions 1.1, 1.2, 1.5, 1.6, and considering the Rand K compressor with K ≤ d / √ n as an example, the w2s communication complexity of both methods is

where we use the facts that L ≤ ̂ L and ω = d / K -1 for Rand K . The key observation is that when comparing (2) and (4), one sees that (4) can be √ n times smaller if ̂ L ≈ L. Consequently, the communication complexity of MARINA / DASHA scales with the number of workers n, and can provably improve the worker-to-server communication complexity O ( dδ 0 L / ε ) achieved by GD .

Server-to-worker compression does not scale with n . In certain applications, the significance of s2w communication cannot be ignored. In 4G LTE and 5G networks, w2s and s2w communication speeds can be almost the same (Huang et al., 2012) or differ by at most a factor of 10 (Narayanan et al., 2021). Although important, this issue is often overlooked and that is why it is the s2w communication that this work places a central emphasis on.

There exist many papers using communication compression techniques to reduce the s2w communication (Zheng et al., 2019; Liu et al., 2020; Philippenko and Dieuleveut, 2021; Fatkhullin et al., 2021; Gruntkowska et al., 2023; Tyurin and Richtárik, 2023a). However, to the best of our knowledge, under Assumptions 1.1, 1.2, 1.5, and 1.6, in the worst case, all previous theoretical s2w communication guarantees are greater or equal to (2). As an example, let us consider the result from Gruntkowska et al. (2023)[Theorem E.3]. If the server employs operators from U ( ω ) and we ignore w2s compression, the method from Gruntkowska et al. (2023) converges in O ( ( ω +1) δ 0 L / ε ) iterations. Thus, with Rand K , the s2w communication complexity is O ( K × ( ω +1) δ 0 L / ε ) = O ( dδ 0 L / ε ) . Another method, called CORE , proposed by Yue et al. (2023), achieves s2w and w2s communication complexities equal to O ( r 1 ( f ) δ 0 L / ε ) , where r 1 ( f ) is a uniform upper bound of the trace of the Hessian. When r 1 ( f ) ≤ dL, CORE can improve on GD . However, this complexity does not scale with n and requires an additional assumption about the Hessian of f .

2 Contributions

In our work, we aim to investigate whether the server-to-worker and total communication complexities (2) of the vanilla GD method can be improved. We make the following contributions:

We start by proving the impossibility of devising a method where the server communicates with the workers using unbiased compressors U ( ω ) (or biased compressors from Section B) and achieves an iteration rate faster than Ω ( ( ω +1) Lδ 0 / ε ) (Theorem 3.1) under Assumptions 1.1, 1.2 and 1.6. This result provides a lower bound for any method that applies such compressors to vectors sent from the server to the workers in every iteration. Moreover, we prove a more general iteration lower bound of Ω ( ( ω +1) Lδ 0 / ε ) for all methods where the server zeroes out a coordinate with probability 1 / ( ω +1) (see Remark 3.2).

In view of this result, it is clear that an extra assumption is needed to break the lower bound Ω ( ( ω +1) Lδ 0 / ε ) . In response, we introduce a novel assumption termed 'Functional ( L A , L B ) Inequality' (see Assumption 4.2). We prove that this assumption is relatively weak and holds, for instance, under the local smoothness of the functions f i (see Assumption 1.5).

We develop a new method for downlink compression, MARINA-P , and show that under our new assumption, along with Assumptions 1.1 1.2, and 1.6, it can achieve the iteration rate of

(see Theorem D.1 with p = 1 / ( ω +1) + Lemma A.5). Notably, when L A is small and n ≫ 1 , this complexity is provably superior to Θ( ( δ 0 L ( ω +1)) / ε ) and the complexities of the previous compressed methods. In this context, L A serves as a measure of the similarity between the functions f i , and can be bounded by the 'variance' of the Hessians of the functions f i (see Theorem 4.8). Thus, MARINA-P is the first method whose iteration complexity can provably improve with the number of workers n .

Moreover, MARINA-P can achieve the s2w communication complexity of

When L A is small and n ≫ 1 , this communication complexity is provably superior to (2) and the communication complexities of the previous compressed methods.

5. Our theoretical improvements can be combined with techniques enhancing the w2s communication complexities . In particular, by combining MARINA-P with MARINA (Gorbunov et al., 2021) and adding the crucial momentum step, we develop a new method, M3 , that guarantees a total communication complexity (s2w + w2s) of

When n ≫ 1 and in the close-to-homogeneous regime, i.e., when L A is small, this complexity is better than (2) and the complexities of the previous bidirectionally compressed methods.

6. Our theoretical results are supported by numerical experiments (see Section F).

3 Lower Bound under Smoothness

Let us first investigate the possibility of improving the iteration complexity O ( ( ω +1) δ 0 L / ε ) under Assumptions 1.1,1.2 and 1.6. In Section G, we consider a family of methods that include those proposed in Zheng et al. (2019); Liu et al. (2020); Philippenko and Dieuleveut (2021); Fatkhullin et al. (2021); Gruntkowska et al. (2023), where the server communicates with workers using unbiased/biased compressors, and establish that

Theorem 3.1 (Slightly Less Formal Reformulation of Theorem G.5) . Under Assumptions 1.1, 1.2 and 1.6, all methods in which the server communicates with clients using different and independent unbiased compressors from U ( ω ) and sends one compressed vector to each worker cannot converge before Ω ( ( ω +1) Lδ 0 / ε ) iterations.

Remark 3.2 . The theorem remains applicable to biased compressors B ( α ) (see Section B) with a lower bound of Θ ( Lδ 0 / αε ) . This is because if C ∈ U ( ω ) , then ( ω +1) -1 C ∈ B ( ( ω +1) -1 ) . We also establish a more general result (Theorem G.4): 'all methods in which the server zeroes out a coordinate with probability ≤ p independently across iterations cannot converge before Ω ( Lδ 0 / pε ) iterations.'

This lower bound is tight up to a constant factor. For instance, under exactly the same assumptions, the EF21-P mechanism from Gruntkowska et al. (2023) converges after Θ ( ( ω +1) Lδ 0 / ε ) iterations. Unlike (4), this convergence rate does not scale with n, and Theorem 3.1 leaves no room for improvement. Consequently, breaking the lower bound requires an additional assumption about the structure of the problem. Before presenting our candidate assumption, we first introduce the ingredients needed to leverage it to the fullest extent: our novel downlink compression method and the type of compressors we shall employ.

4 The MARINA-P Method

Let us first recall the MARINA method (Gorbunov et al., 2021; Szlendak et al., 2021):

where g 0 = ∇ f ( x 0 ) . Motivated by MARINA , we design its primal counterpart, MARINA-P (Algorithm 1), operating in the primal space of the model parameters, as outlined in (6).

At each iteration of MARINA-P , the workers calculate ∇ f i ( w t i ) and transmit it to the server. The server then averages the gradients and updates the global model x t . Subsequently, with some (typically small) probability p , the master sends the non-compressed vector x t +1 to all workers. Otherwise, the i th worker receives a compressed vector C t i ( x t +1 -x t ) . Each worker then uses the received message to compute w t +1 i locally. Importantly, C t 1 ( x t +1 -x t ) , . . . , C t n ( x t +1 -x t ) can differ , and this distinction will form the basis of our forthcoming advancements.

The MARINA-P Method:

Initialize vectors x 0 , w 0 1 , . . . , w 0 n ∈ R d , step size γ > 0 , probability 0 < p ≤ 1 and compressors C t 1 , . . . , C t n ∈ U ( ω P ) for all t ≥ 0 . The method iterates

We denote w t := 1 / n ∑ n i =1 w t i . See the implementation in Algorithm 1.

Comparing (5) and (6), MARINA-P and MARINA are dual methods: both learn control variables ( w t i and g t i ), compress the differences ( x t +1 -x t and ∇ f i ( x t +1 ) -∇ f i ( x t ) ), and with some probability p send non-compressed vectors ( x t +1 and ∇ f i ( x t +1 ) ). However, unlike MARINA , which compresses vectors sent from workers to server and operates in the dual space of gradients, MARINA-P compresses messages sent from server to workers and operates in the primal space of arguments.

Let us take Rand K ∈ U ( d / K -1) as an example. If we set p = ( ω +1) -1 = K / d to balance heavy communications of x t +1 and light communications of C t i in (6), MARINA-P averages sending pd +(1 -p ) K ≤ 2 K coordinates per iteration. Then, the lower bound from Theorem G.4 implies that at least Ω ( ( ω +1) δ 0 L / ε ) iterations of the algorithm are needed.

At first glance, it seems that MARINA-P does not offer any extra benefits compared to previous methods, and that is true - we could not expect to break the lower bound. However, as we shall soon see, under an extra assumption, MARINA-P can achieve a communication complexity that improves with n .

4.1 Three ways to compress

Existing algorithms performing s2w compression share a common characteristic: at each iteration, the server broadcasts the same message to all workers (Zheng et al., 2019; Liu et al., 2020; Fatkhullin et al., 2021; Gruntkowska et al., 2023; Tyurin and Richtárik, 2023a). 5 In contrast, in w2s compression methods, each worker sends to the server a different message, specific to the data stored on that particular device. An analogous approach can be taken in the s2w communication: intuitively, sending n distinct messages would convey more information, potentially leading to theoretical improvements. This indeed proves to be the case. While the usual approach of the server broadcasting the same vector to all clients does not lead to an improvement over (2), allowing these vectors to differ enables a well-crafted method to achieve communication complexity that improves with n (see Corollary D.4).

In Appendix A we provide a detailed discussion of the topic and compare the theoretical complexities of MARINA-P when the server employs three different compression techniques: a) uses one compressor and sends the same vector to all clients, b) uses a collection of independent compressors , or c) uses a collection of correlated compressors . We now turn to presenting the technique that gives the best theoretical s2w communication complexity out of these, namely the use of a set of correlated compressors .

4.2 Recap: permutation compressors Perm K

Szlendak et al. (2021) propose compressors that will play a key role in our new theory. For clarity of presentation, we shall assume that d ≥ n and n | d . 6

Definition 4.1 (Perm K (for d ≥ n and n | d )) . Assume that d ≥ n and d = qn , where q ∈ N > 0 . Let π = ( π 1 , . . . , π d ) be a random permutation of { 1 , . . . , d } . For all x ∈ R d and each i ∈ { 1 , 2 , . . . , n } , we define

Unpacking this definition: when the server compresses a vector using a Perm K compressor, it randomly partitions its coordinates across the workers, so that each client receives a sparse vector containing a random subset of entries of the input vector. Like Rand K , Perm K is also a sparsifier. However, unlike Rand K , it does not allow flexibility in choosing K , as it is fixed to d / n . Furthermore, it can be shown (Lemma A.6) that C i ∈ U ( n -1) for all i ∈ [ n ] .

An appealing property of Perm K is the fact that

for all x ∈ R d deterministically. Here, it is important to note that by design, compressors C i from Definition 4.1 are correlated , and do not satisfy Assumption 1.6. This correlation proves advantageous - Szlendak et al. (2021) show that MARINA with Perm K compressors performs provably better than with i.i.d. Rand K compressors.

4.3 Warmup: homogeneous quadratics

We are finally ready to present our first result showing that the s2w communication complexity can scale with the number of workers n . To explain the intuition behind our approach, let us consider the simplest (and somewhat impractical) choice of functions f i - the homogeneous quadratics:

where A ∈ R d × d is a symmetric but not necessarily positive semidefinite matrix, b ∈ R d and c ∈ R . We now investigate the operation of MARINA-P with Perm K compressors. With probability p , we have w t +1 = x t +1 . Otherwise w t +1 = w t + 1 n ∑ n i =1 C t i ( x t +1 -x t ) (7) = x t +1 +( w t -x t ) . Hence, if we initialize w 0 i = x 0 for all i ∈ [ n ] , an inductive argument shows that w t = x t deterministically for all t ≥ 0 . Then, substituting the gradients of f i to (6), one gets

for all t ≥ 0 . Therefore, MARINA-P with Perm K compressor in this setting is essentially a smart implementation of vanilla GD ! Indeed, for p ≤ 1 / n , MARINA-P with Perm K sends on average ≤ 2 d / n coordinates to each worker, so the s2w communication complexity is

which is n times smaller than in (2)!

4.4 Functional ( L A , L B ) Inequality

From the discussion in Section 3, we know that to improve (2), an extra assumption about the structure of the problem is needed. Building on the example from Section 4.3, we introduce the Functional ( L A , L B ) Inequality .

̸

Server-to-Workers Communication Complexities (s2w)

Method

Complexity

and other compressed methods

(a)

dδ

0

∥

A

∥

ε

CORE

δ

0

tr

A

ε

MARINA-P

with independent Rand

K

(b)

dδ

0 1

n

∑

n

i

=1

∥

A

i

∥

√

nε

+

dδ

0

i

∈

[

n

]

∥

A

i

-

A

∥

ε

MARINA-P

K

(b)

(Corollary 4.7)

dδ

0

∥

A

∥

nε

+

dδ

0

i

∈

[

n

]

∥

A

i

-

A

∥

ε

Total Communication Complexities (s2w + w2s)

Method

Complexity

and other compressed

methods

(a)

dδ

0

∥

A

∥

ε

CORE

δ

0

tr

A

ε

M3

K

and Rand

K

(b)

(Theorem 5.1)

dδ

0

i

∈

[

n

]

∥

A

i

∥

n

1

/

3

ε

+

dδ

0

i

∈

[

n

]

∥

A

i

-

A

∥

ε

Assumption 4.2 (Functional ( L A , L B ) Inequality) . There exist constants L A , L B ≥ 0 such that

Remark 4.3 . A similar assumption, termed 'Heterogeneity-driven Lipschitz Condition on Averaged Gradients', is proposed in Wang et al. (2023b). Our assumption aligns with theirs when L B = 0 . However, our formulation proves to be more powerful. The possibility that L B > 0 becomes instrumental in driving the enhancements we introduce.

Assumption 4.2 is defined for all functions together, and intuitively, it tries to capture the similarities between the functions f i . For n = 1 , inequality (9) reduces to

equivalent to standard L -smoothness (Assumption 1.1) with L 2 = L 2 A + L 2 B . The Functional ( L A , L B ) Inequality is reasonably weak also for n > 1 , as the next theorem shows.

Theorem 4.4. For all i ∈ [ n ] , assume that the functions f i are L i -smooth (Assumption 1.5). Then, Assumption 4.2 holds with L A = L max and L B = 0 .

Therefore, Assumption 4.2 holds whenever the functions f i are smooth, which is a standard assumption in the literature. Now, returning to the example from Section 4.3,

Theorem 4.5. For all i ∈ [ n ] , assume that the functions f i are homogeneous quadratics defined in (8) . Then, Assumption 4.2 holds with L A = 0 and L B = ∥ A ∥ .

Under Assumption 1.5, no information about the similarity of the functions f i is available, yielding L B = 0 and L A > 0 in Theorem 4.4. However, once we have some information limiting heterogeneity, L A can decrease. Notably, L A = 0 for homogeneous quadratics. As we shall see in Section 4.5, the values L A and L B significantly influence the s2w communication complexity of MARINA-P , with lower L A values leading to greatly improved performance.

4.5 The Convergence Theory of MARINA-P with Perm K

We are ready to present our main convergence result, focusing on the Perm K compressor from Section 4.2. This choice simplifies the presentation, but our approach generalizes to a much larger class of compression operators. The full theoretical framework, covering all unbiased compressors, is detailed in Appendix D.

Theorem 4.6. Let Assumptions 1.1, 1.2 and 4.2 be satisfied. Set w 0 i = x 0 for all i ∈ [ n ] . Take Perm K as C t i and γ = ( L + L A √ ω P ( 1 / p -1) ) -1 , where ω P = n -1 (Lemma A.6). Then, MARINA-P finds an ε -stationary point after

iterations.

Corollary 4.7. Let p = K / d ≡ 1 / n . Then, in the view of Theorem 4.6, the average s2w communication complexity of MARINA-P with Perm K compressor is

The key observation is that (10) is independent of L B , and only depends on L A . This particular property is specific to correlated compressors with parameter θ = 0 (defined in Appendix A), such as Perm K . A similar result holds for independent Rand K compressors (see Corollary D.4), but the convergence rate is worse and depends on L B . Nevertheless, this dependence improves with n .

When L A = 0 , which is the case for homogeneous quadratics, the step size bound from Theorem 4.6 simplifies to γ ≤ 1 / L , the standard GD stepsize (recall that in this case our method reduces to GD ). Most importantly, (10) scales with the number of workers n ! Even when L A > 0 , for sufficiently big n, (10) can improve (2) to O ( dδ 0 L A / ε ) .

Let us now investigate how the constants L A and L B change in the general case.

4.6 Estimating L A and L B in the General Case

It is clear from Corollary 4.7 that MARINA-P with Perm K shines when L A is small. To gain further insights into what values L A may take, we now provide an analysis based on the Hessians of the functions f i .

Theorem 4.8. Assume that the functions f i are twice continuously differentiable, L i -smooth (Assumption 1.5), and that there exist D i ≥ 0 such that

for all i ∈ [ n ] . Then, Assumption 4.2 holds with L A = √ 2 max i ∈ [ n ] D i ≤ 2 √ 2 max i ∈ [ n ] L i and L B = √ 2 ( 1 n ∑ n i =1 L i ) .

Intuitively, (11) measures the similarity between the functions f i . The above theorem yields a more refined result than Theorem 4.4: it is always true that max i ∈ [ n ] D i ≤ 2 max i ∈ [ n ] L i , and, in fact, max i ∈ [ n ] D i can be much smaller, as the next result shows.

Theorem 4.9. Assume that f i ( x ) = 1 2 x ⊤ A i x + b ⊤ i x + c i , where A i ∈ R d × d is symmetric but not necessarily positive semidefinite, b i ∈ R d and c i ∈ R for i ∈ [ n ] . Define A = 1 n ∑ n i =1 A i . Then, Assumption 4.2 holds with L A = √ 2 max i ∈ [ n ] ∥ A i -A ∥ and L B = √ 2 ( 1 n ∑ n i =1 ∥ A i ∥ ) .

Thus, L A is less than or equal to √ 2 max i ∈ [ n ] ∥ A i -A ∥ , which serves as a measure of similarity between the matrices. The smaller the values of ∥ A i -A ∥ (indicating greater similarity among the functions f i ), the smaller the L A value.

In the view of this theorem, the s2w communication complexity of MARINA-P with Perm K on non-homogeneous quadratics is

Since the corresponding complexity of GD is

in the close-to-homogeneous regimes (i.e., when max i ∈ [ n ] ∥ A i -A ∥ is small), the complexity (12) can be provably much smaller than (13). The same reasoning applies to the general case when the

The M3 Method

( M3 = MARINA-P + Momentum + MARINA ):

Initialize vectors x 0 , w 0 i , g 0 i , z 0 i ∈ R d for all i ∈ [ n ] , step size γ > 0 , probabilities 0 < p P , p D ≤ 1 and compressors C t 1 , . . . , C t n ∈ U ( ω P ) ∩ P ( θ ) a , Q t 1 , . . . , Q t n ∈ U ( ω D ) for all t ≥ 0 . The method iterates

where the probabilistic decisions are the same for all i ∈ [ n ] , i.e., one coin is tossed for all workers (as in (5) and (6)), and the coins for the first and second probabilistic decisions with p P and p D are independent. We denote w t := 1 / n ∑ n i =1 w t i , g t := 1 / n ∑ n i =1 g t i , z t := 1 / n ∑ n i =1 z t i . See the implementation in Algorithm 2.

functions f i are not quadratics: MARINA-P improves with the number of workers n in the regimes when D i are small (see Theorem 4.8).

Let us note that there is another method, CORE , by Yue et al. (2023), that can also provably outperform GD , achieving the s2w communication complexity of Ω ( δ 0 tr A / ε ) on non-homogeneous quadratics. Neither their method nor ours universally provides the best possible communication guarantees. Our method excels in the close-to-homogeneous regimes: for example, if we take A i = L i I for all i ∈ [ n ] , and define L = 1 / n ∑ n i =1 L i , then the complexity of CORE is Ω ( dδ 0 L / ε ) , while ours is O ( dδ 0 L nε + dδ 0 max i ∈ [ n ] | L i -L | ε ) . Hence, our guarantees are superior in regimes where max i ∈ [ n ] | L i -L |≪ L . One interesting research direction is to develop a universally better method combining the benefits of both approaches.

5 M3 : A New Bidirectional Method

In the previous sections, we introduce a new method that provably improves the server-to-worker communication, but ignores the worker-to-server communication overhead. Our aim now is to treat MARINA-P as a starting point for developing methods applicable to more practical scenarios, by combining it with techniques that compress in the opposite direction. Since the theoretical state-ofthe-art w2s communication complexity is obtained by MARINA (see Section 1.1), our next research step was to combine the two and analyze ' MARINA + MARINA-P ', but this naive approach did not yield communication complexity guarantees surpassing (2) in any regime. It became apparent that some 'buffer' step between these two techniques is needed, and this step turned out to be the momentum. Our new method, M3 (Algorithm 2), is described in (14).

M3 combines (5), (6), and the momentum step z t +1 i = βw t +1 i + (1 -β ) z t i , which is the key to our improvements. A similar technique is used to reduce the variance in Fatkhullin et al. (2023). Let us explain how M3 works in practice. First, the server calculates x t +1 . Depending on the first probabilistic decision, it sends either x t +1 or C t i ( x t +1 -x t ) to the workers, who then calculate w t +1 i locally. Next, the workers compute z t +1 i , and depending on the second probabilistic decision, they send either ∇ f i ( z t +1 i ) or Q t i ( ∇ f i ( z t +1 i ) -∇ f i ( z t i )) back to the server. The server aggregates the received vectors and calculates g t +1 . As in MARINA , p P and p D are chosen in such a way that the

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4

# bits / n (s-to-w) 1e5

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

/

f

(

x

0

)

2

n=10

EF21-P, Top30,

MARINA-P, SameRand30,

MARINA-P, Rand30,

MARINA-P, Perm30,

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5

# bits / n (s-to-w) 1e4

10

6

10

5

10

4

10

3

10

2

10

1

10 0

n=100

EF21-P, Top3,

MARINA-P, SameRand3,

MARINA-P, Rand3,

MARINA-P, Perm3,

0.0

0.5

1.0

1.5

# bits / n (s-to-w) 1e4

10

6

10

5

10

4

10

3

10

2

10

1

10 0

n=1000

EF21-P, Top1,

MARINA-P, SameRand1,

MARINA-P, Perm1,

non-compressed communication does not negatively affect the communication complexity. Therefore, the method predominantly transmits compressed information, with only a marginal probability of sending uncompressed vectors.

5.1 The Convergence Theory of M3

For simplicity, we consider Perm K in the role of C t i and Rand K in the role of Q t i . The general theory for all unbiased compressors is presented in Section E.

Theorem 5.1. Let Assumptions 1.1, 1.2, 1.5 and 4.2 be satisfied. Take γ = ( L + 34 ( nL A + n 2 / 3 L B + n 2 / 3 L max ) ) -1 , p D = p P = 1 / n , β = n -2 / 3 , w 0 i = z 0 i = x 0 and g 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] . Then MARINA-P with C t i = Perm K and Q t i = Rand K with K = d / n finds an ε -stationary point after O ( δ 0 ε ( n 2 / 3 L max + nL A ) ) iterations. The total communication complexity is

Once again, we observe improvement with the number of workers n , and the obtained complexity (15) can be provably smaller than (2). Indeed, in scenarios like federated learning, where the number of workers (e.g., mobile phones) is typically large (Kairouz et al., 2021; Chowdhery et al., 2023), the first term can be significantly smaller than dδ 0 L / ε . The second term can also be small in close-tohomogeneous regimes (see Section 4).

6 Experimental Highlights

This section presents insights from the experiments, with further details and additional results in Appendix F. The experiment aims to empirically test the theoretical results from Section 4. We consider a quadratic optimization problem, where the functions f i are as defined in Theorem 4.9 and A i ∈ R 300 × 300 . We compare GD , MARINA-P sending the same message compressed using a single Rand K compressor to all workers ('SameRand K ' from Appendix A), MARINA-P with independent Rand K compressors, MARINA-P with Perm K compressors, and EF21-P with Top K compressor. We consider n ∈ { 10 , 100 , 1000 } and fine-tune the step size for each algorithm. The results, presented in Figure 1, align closely with the theory, with MARINA-P using Perm K compressors consistently performing best. Moreover, the convergence rate of MARINA-P with Perm K and independent Rand K compressors improves with n . Since this is not the case for EF21-P , even though it outperforms MARINA-P with independent Rand K compressors for n = 10 , it falls behind for n ∈ { 100 , 1000 } .

Acknowledgments and Disclosure of Funding

The research reported in this publication was supported by funding from King Abdullah University of Science and Technology (KAUST): i) KAUST Baseline Research Scheme, ii) Center of Excellence for Generative AI, under award number 5940, iii) SDAIA-KAUST Center of Excellence in Artificial Intelligence and Data Science. The work of A.T. was partially supported by the Analytical center under the RF Government (subsidy agreement 000000D730321P5Q0002, Grant No. 70-2021-00145 02.11.2021).

References

Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. (2017). QSGD: Communicationefficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems (NIPS) , pages 1709-1720.

Arjevani, Y., Carmon, Y., Duchi, J. C., Foster, D. J., Srebro, N., and Woodworth, B. (2022). Lower bounds for non-convex stochastic optimization. Mathematical Programming , pages 1-50.

Beznosikov, A., Horváth, S., Richtárik, P., and Safaryan, M. (2020). On biased compression for distributed learning. arXiv preprint arXiv:2002.12410 .

Carmon, Y., Duchi, J. C., Hinder, O., and Sidford, A. (2020). Lower bounds for finding stationary points I. Mathematical Programming , 184(1):71-120.

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2023). PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1-113.

Fang, C., Li, C. J., Lin, Z., and Zhang, T. (2018). SPIDER: Near-optimal non-convex optimization via stochastic path integrated differential estimator. In NeurIPS Information Processing Systems .

Fatkhullin, I., Sokolov, I., Gorbunov, E., Li, Z., and Richtárik, P. (2021). EF21 with bells & whistles: Practical algorithmic extensions of modern error feedback. arXiv preprint arXiv:2110.03294 .

Fatkhullin, I., Tyurin, A., and Richtárik, P. (2023). Momentum provably improves error feedback! Advances in Neural Information Processing Systems .

Gorbunov, E., Burlachenko, K., Li, Z., and Richtárik, P. (2021). MARINA: Faster non-convex distributed learning with compression. In 38th International Conference on Machine Learning .

Gruntkowska, K., Tyurin, A., and Richtárik, P. (2023). EF21-P and friends: Improved theoretical communication complexity for distributed optimization with bidirectional compression. In International Conference on Machine Learning , pages 11761-11807. PMLR.

Horváth, S., Ho, C.-Y., Horvath, L., Sahu, A. N., Canini, M., and Richtárik, P. (2022). Natural compression for distributed deep learning. In Mathematical and Scientific Machine Learning , pages 129-141. PMLR.

Huang, J., Qian, F., Gerber, A., Mao, Z. M., Sen, S., and Spatscheck, O. (2012). A close examination of performance and power characteristics of 4G LTE networks. In Proceedings of the 10th international conference on Mobile systems, applications, and services , pages 225-238.

Huang, X., Chen, Y., Yin, W., and Yuan, K. (2022). Lower bounds and nearly optimal algorithms in distributed learning with communication compression. arXiv preprint arXiv:2206.03665 .

Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. (2021). Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210.

Koneˇ cný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., and Bacon, D. (2016). Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 .

Lan, G. (2020). First-order and stochastic optimization methods for machine learning . Springer.

LeCun, Y., Cortes, C., and Burges, C. (2010). MNIST handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2.

Li, Z., Bao, H., Zhang, X., and Richtárik, P. (2021). PAGE: A simple and optimal probabilistic gradient estimator for nonconvex optimization. In International Conference on Machine Learning , pages 6286-6295. PMLR.

Liu, X., Li, Y., Tang, J., and Yan, M. (2020). A double residual compression algorithm for efficient distributed learning. In International Conference on Artificial Intelligence and Statistics , pages 133-143. PMLR.

Lu, Y. and De Sa, C. (2021). Optimal complexity in decentralized training. In International Conference on Machine Learning , pages 7111-7123. PMLR.

McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communicationefficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR.

Narayanan, A., Zhang, X., Zhu, R., Hassan, A., Jin, S., Zhu, X., Zhang, X., Rybkin, D., Yang, Z., Mao, Z. M., et al. (2021). A variegated look at 5G in the wild: performance, power, and QoE implications. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference , pages 610-625.

OpenAI (2023). GPT-4 technical report. ArXiv , abs/2303.08774.

Philippenko, C. and Dieuleveut, A. (2021). Preserved central model for faster bidirectional compression in distributed settings. Advances in Neural Information Processing Systems , 34:2387-2399.

Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021). Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR.

Richtárik, P., Sokolov, I., and Fatkhullin, I. (2021). EF21: A new, simpler, theoretically better, and practically faster error feedback. In Neural Information Processing Systems, 2021.

Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. (2014). 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In Fifteenth Annual Conference of the International Speech Communication Association .

Szlendak, R., Tyurin, A., and Richtárik, P. (2021). Permutation compressors for provably faster distributed nonconvex optimization. In International Conference on Learning Representations .

Tyurin, A. and Richtárik, P. (2023a). 2Direction: Theoretically faster distributed training with bidirectional communication compression. Advances in Neural Information Processing Systems .

Tyurin, A. and Richtárik, P. (2023b). DASHA: Distributed nonconvex optimization with communication compression, optimal oracle complexity, and no client synchronization. 11th International Conference on Learning Representations (ICLR) .

Tyurin, A. and Richtárik, P. (2023c). Optimal time complexities of parallel stochastic optimization methods under a fixed computation model. Advances in Neural Information Processing Systems .

Wang, J., Lu, Y., Yuan, B., Chen, B., Liang, P., De Sa, C., Re, C., and Zhang, C. (2023a). CocktailSGD: Fine-tuning foundation models over 500Mbps networks. In International Conference on Machine Learning , pages 36058-36076. PMLR.

Wang, J., Wang, S., Chen, R.-R., and Ji, M. (2023b). A new theoretical perspective on data heterogeneity in federated optimization. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities .

Wangni, J., Wang, J., Liu, J., and Zhang, T. (2018). Gradient sparsification for communicationefficient distributed optimization. Advances in Neural Information Processing Systems , 31.

Yue, P., Zhao, H., Fang, C., He, D., Wang, L., Lin, Z., and Zhu, S.-c. (2023). CORE: Common random reconstruction for distributed optimization with provable low communication complexity. arXiv preprint arXiv:2309.13307 .

Zheng, S., Huang, Z., and Kwok, J. (2019). Communication-efficient distributed blockwise momentum SGD with error-feedback. Advances in Neural Information Processing Systems , 32.

Contents

A Three unbiased ways to compress

The main focus of this paper is handling the server-to-worker communication costs. To explain better where the improvements outlined in the main part of this paper come from, let us first consider the scenario where uplink communication cost is negligible but downilnk communication cost is not. While we do no necessarily say that this is a realistic setup, examining it first enables us to understand how downlink compression should be performed, capturing all the intricacies.

Existing algorithms with lossy s2w and w2s communication have a certain common feature. The compression mechanism employed on the clients is very different from the one used on the server: while each client transmits to the server a different message, specific to the data stored on each device, the server broadcasts the same update to all clients. We want to question this algorithmic step and suggest to the reader that if compression is applied multiple times and each worker receives its individual update, then intuitively more information can be transmitted. A well-designed algorithm should be able to take advantage of this.

One can depart from the usual approach of sending the same update to all workers in two ways: a) compress the update n times independently , or b) produce n such updates in a correlated way. Either way, the server broadcasts n different compressed messages rather than one, and sends a different update to each worker. The key discovery here is that both a) and b) are mathematically provably better than the prevalent approach of sending the same update to all clients.

This is a crucial improvement in a system where the above setup is a good approximation of reality. And even if it is not, and the current model is not perfectly capturing the reality, we can accept it for now, as it allows us to focus on the novel aspects of the approach. With that said, these considerations can serve as a starting point for thinking about bidirectional compression : having focused on the simplified setup and equipped with knowledge on how the compression on the master should be performed, we employ this mechanism in more complex scenarios (see Section 5).

Let us now describe the three possible ways to perform compression on the server.

'Same' compressors. The prevalent approach in downlink compression is to transmit the same update to all workers. To illustrate this, let us call a collection C 1 , . . . , C n of compressors 'SameRand K ' if for all i ∈ [ n ] we have C i = C for some Rand K compressor C . Now, consider one iteration t of MARINA-P with SameRand K compressor. The server calculates C t i ( x t +1 -x t ) for i ∈ [ n ] , but in this case, C t 1 ( x t +1 -x t ) = . . . = C t n ( x t +1 -x t ) = C t ( x t +1 -x t ) . Thus, applying a collection of SameRand K compressors to some vector x ∈ R d is equivalent to using a single Rand K compression operator and transmitting the same message C ( x ) to all workers.

Independent Compressors. Rather than setting C i ( x ) = C ( x ) for all i ∈ [ n ] , one can break the dependency between the messages and allow the compressors to differ. For illustrational purposes, suppose that C i , i ∈ [ n ] are independent Rand K compressors (Assumption 1.6). Then, applying such a collection of mappings to the vector of interest x ∈ R d , one obtains n distinct and independent sparse vectors C 1 ( x ) , . . . , C n ( x ) .

Remark A.1 . We are aware of only one method that uses n distinct compressors in downlink compression, Rand-MCM by Philippenko and Dieuleveut (2021). Given the absence of results in the non-convex case, let us compare the communication complexities of Rand-MCM and M3 under the Polyak-Łojasiewicz condition (Assumption D.9), which holds under strong convexity. In the strongly convex case, the proved iteration complexity of Rand-MCM is

Assuming for simplicity that the server and the workers use Rand K compressors with K = d / n , this gives the total communication complexity of

which is getting worse as the number of workers n increases. Meanwhile, by Corollary E.10, the total communication complexity of M3 (where C t i are the Perm K compressors and Q t i are independent

Rand K compressors, both with K = d/n ) under the Polyak-Łojasiewicz condition is

Since n is typically large, the total communication complexity of M3 can be much better than that of Rand-MCM .

Correlated Compressors. In their work, Szlendak et al. (2021) introduce an alternative class of compressors, which satisfy the following condition:

Definition A.2 (AB-inequality (Szlendak et al., 2021)) . There exist constants A,B ≥ 0 such that the random operators C 1 , . . . C n satisfy

for all x, x 1 , . . . , x n ∈ R d . If these conditions hold, we write {C i } n i =1 ∈ U ( A,B ) .

Following on this idea, we introduce the concept of a collection of correlated compressors .

Definition A.3 (Collection of Correlated Compressors) . There exists a constant θ ≥ 0 such that the random operators C 1 , . . . C n satisfy:

for all x ∈ R d . If these conditions hold, we write {C i } n i =1 ∈ P ( θ ) .

Definition A.3 will play a key role in our upcoming advancements. But what makes this assumption reasonable?

First, it is easy to note that condition (17) is weaker than (16). Indeed, if {C i } n i =1 ∈ U ( A,B ) , then inequality (17) holds with θ := A -B . It turns out that it is in fact strictly weaker, as the following example shows.

Example A.4 . Let n = 2 , d = 1 . Let { ζ x : x ∈ R } be a collection of independent Cauchy variables indexed by real numbers. Define C 1 ( u ) = u + ζ u , and C 2 ( u ) = u -ζ u . Then

so C 1 ( u ) and C 2 ( u ) satisfy Definition A.3 with θ = 0 . However, for u 1 = u 2 , by the properties of Cauchy distribution we have

̸

Thus, C 1 ( u ) and C 2 ( u ) do not satisfy Definition A.2.

In fact, the condition specified in Definition A.3 does not impose any restrictions on the compressor class when working with unbiased compressors. This is because, for any set of compressors C 1 , . . . , C n ∈ U ( ω ) , there exists θ ≥ 0 such that {C i } n i =1 ∈ P ( θ ) , as shown in the following lemma.

Lemma A.5.

Let C 1 , . . . , C n be a collection of compressors such that C i ∈ U ( ω ) for all i ∈ [ n ] . Then {C i } n i =1 ∈ P ( ω ) .

Let us further assume that C 1 , . . . , C n are independent (Assumption 1.6). Then {C i } n i =1 ∈ P ( ω / n ) .

Proof. 1. Jensen's inequality gives

Using independence of compressors, we have

Thus θ = ω / n .

However, the true advantages of employing correlated compressors become apparent when the definition holds with θ = 0 , as in the case of Perm K compressors.

Lemma A.6. Let C 1 , . . . , C n be a collection of a) SameRand K , b) independent Rand K , c) Perm K compressors. Then

{C i } n ∈ P ( ω ) and C i ∈ U ( ω ) where ω = d / K -1 for all i ∈ [ n ]

{C i } i =1 ∈ P (0) and C i ∈ U ( ω ) where ω = n -1 for all i ∈ [ n ]

Proof. Unbiasedness follows easily from definitions of compressors (the proof for Perm K compressors can be found in Szlendak et al. (2021)). That Rand K ∈ U ( d / K -1) (and hence trivially SameRand K ∈ U ( d / K -1) ) is a well-known fact. Next, the fact that a) {C i } n i =1 ∈ P ( ω ) for SameRand K compressors and b) {C i } n i =1 ∈ P ( ω / n ) for independent Rand K compressors follows directly from Lemma A.5.

To compute ω for Perm K compressor, first assume that d ≥ n . Then E [ ∥C i ( x ) ∥ 2 ] = n ∥ x ∥ 2 (Szlendak et al., 2021), so

Similarly, suppose that d ≤ n , and write n as n = qd + r , where q ∈ N > 0 and 0 ≤ r < d . Then E [ ∥C i ( x ) ∥ 2 ] = n / q ∥ x ∥ 2 (Szlendak et al., 2021), and hence

In both cases ω = n -1 .

Finally, by construction of Perm K , we have 1 n ∑ n i =1 C i ( x ) = x , implying θ = 0 .

In what follows, when considering the Perm K compressor, we shall assume for simplicity that d ≥ n . The results for d < n are analogous.

B Biased Compressors

In addition to unbiased compressors (Definition 1.4), the literature of compressed methods distinguishes another class of mappings:

Algorithm 1 MARINA-P

1: Input: initial model x 0 ∈ R d (stored on the server), initial model shifts w 0 1 = . . . = w 0 n = x 0 (stored on the workers), step size γ > 0 , probability 0 < p ≤ 1 , compressors C t , . . . , C t ∈ U ( ω P )

Definition B.1. A stochastic mapping C : R d → R d is a biased compressor if there exists α ∈ (0 , 1] such that

The family of such compressors is denoted by B ( α ) . It is well-known that if C ∈ U ( ω ) , then ( ω +1) -1 C ∈ B ( ( ω +1) -1 ) , meaning that the family of biased compressors is broader. A canonical example is the Top K ∈ B ( K / d ) compressor, which preserves the K largest in magnitude coordinates of the input vector (Beznosikov et al., 2020).

C Properties of L A and L B

We first prove the results from Section 4, starting with calculating the constants L A and L B from Assumption 4.2 in some special cases.

Theorem 4.4. For all i ∈ [ n ] , assume that the functions f i are L i -smooth (Assumption 1.5). Then, Assumption 4.2 holds with L A = L max and L B = 0 .

Proof. From Assumption 1.5 it follows that

.

so Assumption 4.2 holds with L A = L max and L B = 0

Theorem 4.5. For all i ∈ [ n ] , assume that the functions f i are homogeneous quadratics defined in (8) . Then, Assumption 4.2 holds with L A = 0 and L B = ∥ A ∥ .

Proof. It is easy to verify that

meaning that Assumption 4.2 holds with L A = 0 and L B = ∥ A ∥ .

Lemma C.1. Let Assumption 1.5 hold. Then, there exist constants L A , L B ≥ 0 such that Assumption 4.2 holds and L 2 A + L 2 B ≤ L 2 max .

Proof. Assumption 1.5 gives

and hence Assumption 4.2 holds with L 2 A = L 2 max and L 2 B = 0 .

Remark C.2 . Under Assumption 4.2 we have

so, in principle, one could always set L 2 B = 0 . However, the bound could be tightened by decreasing L A and increasing L B . The smaller L A , the better the performance of our algorithms (see Corollaries D.4 and E.3).

Now, we proceed to prove the result that relates the values of L A and L B to the Hessians of the functions f i .

Theorem 4.8. Assume that the functions f i are twice continuously differentiable, L i -smooth (Assumption 1.5), and that there exist D i ≥ 0 such that

for all i ∈ [ n ] . Then, Assumption 4.2 holds with L A = √ 2 max i ∈ [ n ] D i ≤ 2 √ 2 max i ∈ [ n ] L i and L B = √ 2 ( 1 n ∑ n i =1 L i ) .

Proof. By the fundamental theorem of calculus,

Further, and

By combining the above, we get

which means that Assumption 4.2 holds with L A = 2(max i D i ) and L B = 2 n i =1 L i .

Remark C.3 . Clearly, if Assumption 1.5 holds, i.e., if there exists L i ≥ 0 such that sup z i ∈ R d ∥ ∥ ∇ 2 f i ( z i ) ∥ ∥ ≤ L i for all i ∈ [ n ] , then there exists D i such that sup z 1 ,...,z n ∈ R d ∥ ∥ ∥ ∇ 2 f i ( z i ) -1 n ∑ n j =1 ∇ 2 f j ( z j ) ∥ ∥ ∥ ≤ D i , which means that this latter condition is not restrictive. Indeed,

However, D i can be small even if the constants { L i } are large, as the next theorem shows.

Theorem 4.9. Assume that f i ( x ) = 1 2 x ⊤ A i x + b ⊤ i x + c i , where A i ∈ R d × d is symmetric but not necessarily positive semidefinite, b i ∈ R d and c i ∈ R for i ∈ [ n ] . Define A = 1 n ∑ n i =1 A i . Then, Assumption 4.2 holds with L A = √ 2 max i ∈ [ n ] ∥ A i -A ∥ and L B = √ 2 ( 1 n ∑ n i =1 ∥ A i ∥ ) .

Proof. In this case ∇ 2 f i ( z i ) ≡ A i , and the result easily follows from Theorem 4.8.

D Convergence of MARINA-P in the General Case

D.1 Main Results

As promised, we now present a result generalizing Theorem 4.6 to all unbiased compressors.

Theorem D.1. Let Assumptions 1.1, 1.2 and 4.2 be satisfied and suppose that {C t i } n i =1 ∈ P ( θ ) (Def. A.3) and C t i ∈ U ( ω P ) (Def. 1.4) for all i ∈ [ n ] . Let

Letting

for each T ≥ 1 we have

Let us provide some important examples:

Theorem D.2. Let Assumptions 1.1, 1.2 and 4.2 be satisfied. Choose

and set w 0 i = x 0 for all i ∈ [ n ] . Then MARINA-P finds an ε -stationary point after

iterations.

The result for Perm K compressors proves Theorem 4.6.

The above theorem demonstrates the complexities for a) SameRand K , b) independent Rand K and c) Perm K compressors. However, the result applies to any families of compressors such that for all t ≥ 0 we have a) C t 1 = . . . = C t n = C t ∈ U ( ω P ) , b) C t 1 , . . . , C t n ∈ U ( ω P ) are independent, and c) C t 1 , . . . , C t n ∈ U ( ω P ) ∩ P ( θ ) , respectively.

We now derive the communication complexities:

Corollary D.4. Let us take p = 1 / n and set K = d / n (corresponding to the sparsification level of a Perm K compressor). Then, in the view of Theorem D.2, the average s2w communication complexity of MARINA-P is

Remark D.5 .

The result for Perm K compressors proves Corollary 4.7.

The key observation from (19) is the dependence on L A and L B . In particular, if L A ≈ 0 (which is the case, e.g., for homogeneous quadratics), the above communication complexities are

Hence, only by sending different messages to different clients, one obtains complexities improving with n . In particular, for Perm K , the complexity scales linearly with the number of workers.

D.2 Proofs

To prove the results from the previous section, we first establish several identities and inequalities satisfied by the sequences { w t 1 , . . . , w t n } t ≥ 0 . We start by studying the evolution of the quantity ∥ w t i -x t ∥ 2 . In what follows, E t [ · ] denotes the expectation conditioned on the first t iterations.

Lemma D.6. Let C t i ∈ U ( ω P ) for all i ∈ [ n ] . Then

Proof. In the view of definition of w t +1 , we get

Averaging, taking expectation and using the tower property, we get the result.

This lemma is less powerful: it is not an identity, and hence some information is lost. Moreover, it focuses on a single client i , and is therefore not able to take advantage of the correlation among the compressors. On the other hand, it can be used in the convergence analysis without any need to restrict the function class.

Next, we study the evolution of the quantity ∥ w t -x t ∥ 2 .

Lemma D.7. Let {C t } n ∈ P ( θ ) . Then

Proof. In the view of definition of w t +1 , we get

Taking expectation and using the tower property, we get the result.

This lemma is more powerful since it is able to take advantage of the correlation among the compressors. Indeed, if θ = 0 (as in the case of Perm K compressors), then it becomes an identity:

We now prove convergence of MARINA-P in the general case.

Theorem D.1. Let Assumptions 1.1, 1.2 and 4.2 be satisfied and suppose that {C t i } n i =1 ∈ P ( θ ) (Def. A.3) and C t i ∈ U ( ω P ) (Def. 1.4) for all i ∈ [ n ] . Let

Letting

for each T ≥ 1 we have

Proof. First, combining the inequalities in Lemmas D.6 and D.7, we get

Next, using Assumption 4.2, we have

Combining the above inequality with Lemma H.1 gives

By adding inequalities (20) and (21), we get

iterations.

Proof. In view of Theorem D.1, the step size satisfies the inequality

Since by Lemma A.5, when the compressors are independent we have θ = ω P / n , the algorithm converges in

iterations.

Theorem D.2. Let Assumptions 1.1, 1.2 and 4.2 be satisfied. Choose

where in the last line we use the assumption on the step size and Lemma H.2. Summing up the above inequality for t = 0 , 1 , . . . , T -1 and rearranging the terms, we get

With the above result, we can establish the iteration and communication complexities of MARINA-P for three different compression schemes described in Appendix A. First, let us prove a result when independent compressors are used.

Theorem D.8. Let Assumptions 1.1, 1.2 and 4.2 be satisfied and suppose that C t 1 , . . . , C t n is a collection of independent compressors (Assumption 1.6) such that C t i ∈ U ( ω ) for all i ∈ [ n ] , t ∈ N . Choose

and set w 0 i = x 0 for all i ∈ [ n ] . Then MARINA-P finds an ε -stationary point after

and set w 0 i = x 0 for all i ∈ [ n ] . Then MARINA-P finds an ε -stationary point after

iterations.

Proof. In view of Theorem D.1, the step size is such that

We now use Lemma A.6 and substitute the vales of θ specific to each compression type.

For SameRand K , we have θ = ω P , so the algorithm converges after

iterations. Following the same reasoning as in the proof of Theorem D.8, for Rand K we have

Finally, for Perm K we have θ = 0 , so

The result follows from the fact that w 0 i = x 0 for all i ∈ [ n ] .

Corollary D.4. Let us take p = 1 / n and set K = d / n (corresponding to the sparsification level of a Perm K compressor). Then, in the view of Theorem D.2, the average s2w communication complexity of MARINA-P is

Proof. The expected number of floats a server is relaying to each client at each iteration of MARINA-P is

Next, using the results from Lemma A.6, our choice of compressors and parameters gives ω P = d / K -1 = n -1 in each of the three cases. Hence, substituting p = 1 / n in (23), (24) and (25), we obtain the following server-to-worker communication complexities:

for SameRand K compressors:

for Rand K compressors:

for Perm K compressors:

D.3 Polyak-Łojasiewicz condition

D.3.1 Main Results

To complete the theory, we now establish a convergence result for MARINA-P under the PolyakŁojasiewicz assumption.

Assumption D.9 (Polyak-Łojasiewicz condition) . The function f satisfies Polyak-Łojasiewicz (PŁ) condition with parameter µ , i.e., for all x ∈ R d there exists x ∗ ∈ arg min x ∈ R d f ( x ) such that

Theorem D.10. Let Assumptions 1.1, 1.2, 4.2 and D.9 be satisfied and suppose that {C t i } n i =1 ∈ P ( θ ) and C t i ∈ U ( ω P ) for all i ∈ [ n ] . Take

Letting

for each T ≥ 1 we have iterations.

Corollary D.11. Let C t i ∈ P (0) for all i ∈ [ n ] (e.g. Perm K ), choose p = 1 / ( ω P +1) . Then, in the view of Theorem D.10, Algorithm 1 ensures that E [ f ( x T ) -f ∗ ] ≤ ε after

Corollary D.12. Let C t i be the Perm K compressors ( K = d / n ). Then, in the view of Corollary D.11, the s2w communication complexity of MARINA-P with Perm K is

D.3.2 Proofs

Theorem D.10. Let Assumptions 1.1, 1.2, 4.2 and D.9 be satisfied and suppose that {C t i } n i =1 ∈ P ( θ ) and C t i ∈ U ( ω P ) for all i ∈ [ n ] . Take

Letting

for each T ≥ 1 we have

Proof. We proceed similarly as in the proof of Theorem D.1. Combining the inequalities in Lemmas D.6 and D.7 gives

By adding inequalities (21) and (29), we get

iterations.

Proof. In view of Theorem D.10, the step size satisfies

Therefore, since θ = 0 and p = 1 / ( ω P +1) , the algorithm converges after

iterations.

Corollary D.12. Let C t i be the Perm K compressors ( K = d / n ). Then, in the view of Corollary D.11, the s2w communication complexity of MARINA-P with Perm K is

Proof. For Perm K , ω P = n -1 . Therefore, the iteration complexity is

Since the expected number of floats the server is relaying to each client is

the server-to-worker communication complexity is

where the last inequality follows from the Polyak-Łojasiewicz condition, Lemma H.2 and our choice of γ . Applying the above inequality iteratively, we finish the proof.

Corollary D.11. Let C t i ∈ P (0) for all i ∈ [ n ] (e.g. Perm K ), choose p = 1 / ( ω P +1) . Then, in the view of Theorem D.10, Algorithm 1 ensures that E [ f ( x T ) -f ∗ ] ≤ ε after

E Convergence of M3 in the General Case

We now move on to the bidirectionally compressed method. Below is a generalization of Theorem 5.1 to all unbiased compressors.

E.1 Main Results

Theorem E.1. Let Assumptions 1.1, 1.2, 1.5 and 4.2 hold and suppose that the compressors Q t i ∈ U ( ω D ) satisfy Assumption 1.6, {C t i } n i =1 ∈ P ( θ ) and C t i ∈ U ( ω P ) for all i ∈ [ n ] . Let γ > 0 be such that

Letting

where κ = γ p D , η = 4 γL 2 B β , ν = 4 γL 2 A β + 6 γω D βL 2 max np D , ρ = 32 γL 2 B ( 1 p P + p P β 2 ) and µ = 32 γL 2 A ( 1 p P + p P β 2 ) + 48 γω D L 2 max np D ( β + p P ) , M3 ensures that

We now simplify the above result by considering θ = 0 .

Corollary E.2. Let C t i ∈ P (0) for all i ∈ [ n ] (e.g. Perm K ), choose p P = 1 / ( ω P + 1) , p D = 1 / ( ω D +1) and

Then, in the view of Theorem E.1, the iteration complexity is

We now give the bound for the total communication complexity of M3 .

Corollary E.3. Let C t i be the Perm K compressors and Q t i be the independent (Assumption 1.6) Rand K compressors, both with K = d / n . Then, in the view of Corollary E.2, the iteration complexity is

and the total communication complexity is

Remark E.4 . The above result proves the complexities from Theorem 5.1.

Algorithm 2 M3

1: Input: initial model x 0 ∈ R d (stored on the server), initial model shifts w 0 i = z 0 i = x 0 , i ∈ [ n ] (stored on the workers), initial gradient estimators g 0 = ∇ f ( x 0 ) (stored on the sever), step size γ > 0 , probabilities 0 < p P , p D ≤ 1 , compressors C t 1 , . . . , C t n ∈ U ( ω P ) ∩ P ( θ ) , Q t 1 , . . . , Q t n ∈ U ( ω D ) for all t ≥ 0 .

2: for t = 0 , . . . , T do

3:

x t +1 = x t -γg t

Server takes a gradient-type step to update the global model

4: Sample c t P ∼ Bernoulli ( p P ) , c t D ∼ Bernoulli ( p D )

6: for i = 1 , . . . , n in parallel do

5: For i ∈ [ n ] , send C t i ( x t +1 -x t ) to worker i if c t P = 0 and x t +1 otherwise

9:

Send

Q

t

i

(

∇

f

i

(

z

)

-∇

10: end for

t

13: else

t

i

15: end if

16: end for

(maintaining only the sequence g t in the implementation is sufficient; the sequences g t i from (14) are virtual )

E.2 Proofs

Similar to our approach from the previous section, we start by establishing several inequalities satisfied by the sequences { w t 1 , . . . , w t n } t ≥ 0 , { z t 1 , . . . , z t n } t ≥ 0 and { g t 1 , . . . , g t n } t ≥ 0 .

Lemma E.5. Let C t i ∈ U ( ω P ) for all i ∈ [ n ] and {C t i } n i =1 ∈ P ( θ ) . Then

for all i ∈ [ n ] , and

Proof. Using the definition of w t +1 i , we have

f

i

(

z

t

i

))

to the server if

+1

Worker

i

updates its local model shift

Worker

f

i

i

takes the momentum step

(

z

t

+1

i

)

otherwise and hence

Using the definition of w t +1 i again, we get

t

D

c

= 0

and

∇

Using the same reasoning, we now prove the second inequality:

Lemma E.6. Let Assumption 1.5 hold. Furthermore, suppose that the compressors Q t i ∈ U ( ω D ) satisfy Assumption 1.6 and that C t i ∈ U ( ω P ) for i ∈ [ n ] . Then

Proof. First, from the definition of g t +1 i , we get

and hence

Let us consider the first term separately:

Using the result from Lemma E.5, we have

where in the last line we use the fact that p P ≤ 1 . It remains to substitute the above inequality in (30).

Lemma E.7. Let C t i ∈ U ( ω P ) for all i ∈ [ n ] and {C t i } n i =1 ∈ P ( θ ) . Then

for all i ∈ [ n ] , and

Proof. From the definition of z t +1 i , we get

Then, Lemma E.5 gives

The second inequality is proved almost in the same way. First,

and using Lemma E.5, we obtain

Theorem E.1. Let Assumptions 1.1, 1.2, 1.5 and 4.2 hold and suppose that the compressors Q t i ∈ U ( ω D ) satisfy Assumption 1.6, {C t i } n i =1 ∈ P ( θ ) and C t i ∈ U ( ω P ) for all i ∈ [ n ] . Let γ > 0 be such that

Letting

where κ = γ p D , η = 4 γL 2 B β , ν = 4 γL 2 A β + 6 γω D βL 2 max np D , ρ = 32 γL 2 B ( 1 p P + p P β 2 ) and µ = 32 γL 2 A ( 1 p P + p P β 2 ) + 48 γω D L 2 max np D ( β + p P ) , M3 ensures that

Proof. Lemma H.1 gives

Let κ, η, ν, ρ, µ ≥ 0 be some non-negative numbers that we define later. Using Lemmas D.6, D.7, E.6 and E.7, we get

We rearrange the terms to obtain

We now consider the coefficient of the term E [ ∥ w t -x t ∥ 2 ] . Using the inequality xy ≤ x + y 2 for all x, y ≥ 0 , we get

Next, taking ν = 4 γL 2 A β + 6 γω D βL 2 max np D gives

Let us consider the last bracket:

Let us simplify the last bracket.

We next consider the coefficients of L 2 B , L 2 A and L 2 max . First, for L 2 B , we have

Next, the coefficient of L 2 A can be bounded as

and for L 2 max we obtain

since ( ω P +1) β 2 np D ≤ 1 np D + ω P β np D . Substituting these inequalities to (31) and (32), we get

By collecting all the terms w.r.t. E [ ∥ ∥ x t +1 -x t ∥ ∥ 2 ] , using the step size γ from the theorem and Lemma H.2, we obtain

It remains to rearrange and sum the last inequality for t = 0 , . . . , T -1 .

Corollary E.2. Let C t i ∈ P (0) for all i ∈ [ n ] (e.g. Perm K ), choose p P = 1 / ( ω P + 1) , p D = 1 / ( ω D +1) and

Then, in the view of Theorem E.1, the iteration complexity is

Proof. By Theorem E.1, up to a constant factor, the algorithm converges after

iterations, where we use the choice of p P and p D . Using Lemma C.1, we have L 2 A + L 2 B ≤ L 2 max and hence

where we substitute our choice of β.

Corollary E.3. Let C t i be the Perm K compressors and Q t i be the independent (Assumption 1.6) Rand K compressors, both with K = d / n . Then, in the view of Corollary E.2, the iteration complexity is

and the total communication complexity is

Proof. The choice of compressors and parameters ensures that ω P = ω D = n -1 (Lemma A.6). Thus, the iteration complexity is

Since p P = p D = 1 /n and K = d/n, on average, the algorithm sends ≤ 2 d n coordinates in both directions. Therefore, the total communication complexity is

E.3 Polyak-Łojasiewicz condition

E.3.1 Main Results

As with MARINA-P , we provide the analysis of M3 under the Polyak-Łojasiewicz condition.

Theorem E.8. Let Assumptions 1.1, 1.2, 1.5, 4.2 and D.9 be satisfied and suppose that the compressors Q t i ∈ U ( ω D ) satisfy Assumption 1.6, {C t i } n i =1 ∈ P ( θ ) and C t i ∈ U ( ω P ) for all i ∈ [ n ] . Let γ > 0 be such that

Letting

where κ = 2 γ p D , η = 8 γL 2 B β , ν = 8 γL 2 A β + 24 γω D βL 2 max np D , ρ = 128 γL 2 B ( 1 p P + p P β 2 ) and τ = 128 γL 2 A ( 1 p P + p P β 2 ) + 384 γω D L 2 max np D ( β + p P ) , M3 ensures that for each T ≥ 1

Corollary E.9. Let C t i ∈ P (0) for all i ∈ [ n ] (e.g. Perm K ), choose p P = 1 / ( ω P + 1) , p D = 1 / ( ω D +1) and

Then, in the view of Theorem E.8, Algorithm 2 ensures that E [ f ( x T ) -f ∗ ] ≤ ε after

iterations.

Corollary E.10. Let C t i be the Perm K compressors and Q t i be the independent (Assumption 1.6) Rand K compressors, both with K = d / n . Then, in the view of Corollary E.9, the total communication complexity is

E.3.2 Proofs

Theorem E.8. Let Assumptions 1.1, 1.2, 1.5, 4.2 and D.9 be satisfied and suppose that the compressors Q t i ∈ U ( ω D ) satisfy Assumption 1.6, {C t i } n i =1 ∈ P ( θ ) and C t i ∈ U ( ω P ) for all i ∈ [ n ] . Let γ > 0 be such that

Letting

where κ = 2 γ p D , η = 8 γL 2 B β , ν = 8 γL 2 A β + 24 γω D βL 2 max np D , ρ = 128 γL 2 B ( 1 p P + p P β 2 ) and τ = 128 γL 2 A ( 1 p P + p P β 2 ) + 384 γω D L 2 max np D ( β + p P ) , M3 ensures that for each T ≥ 1

Proof. Starting as in the proof of Theorem E.1, we have

for some κ, η, ν, ρ, τ ≥ 0 . This time, we let κ = 2 γ p D and η = 8 γL 2 B β , which gives γ + κ (1 -p D ) = κ ( 1 -p D 2 ) and 2 γL 2 B + η (1 -β / 2 ) = η ( 1 -β 4 ) . Hence

Rearranging the terms

Considering the coefficient of E [ ∥ w t -x t ∥ 2 ] and using the inequality xy ≤ x 2 + y 2 2 for all x, y ≥ 0 , we get

where we define ρ = 128 γL 2 B ( 1 p P + p P β 2 ) . Substituting this choice of ρ , we obtain

Considering the last bracket, we have

where the last bracket can be bounded as

We next consider the coefficients of L 2 B , L 2 A and L 2 max . First, for L 2 B , we have

Next, the coefficient of L 2 A can be bounded as

and for L 2 max we obtain

since ( ω P +1) β 2 np D ≤ 1 np D + ω P β np D . Substituting these inequalities to (34) and (35), we get

By collecting all the terms w.r.t. E [ ∥ ∥ x t +1 -x t ∥ ∥ 2 ] , using the step size γ from the theorem and Lemma H.2, we obtain

Lastly, Assumption D.9 gives

It remains to apply the last inequality iteratively to finish the proof.

Corollary E.9. Let C t i ∈ P (0) for all i ∈ [ n ] (e.g. Perm K ), choose p P = 1 / ( ω P + 1) , p D = 1 / ( ω D +1) and

Then, in the view of Theorem E.8, Algorithm 2 ensures that E [ f ( x T ) -f ∗ ] ≤ ε after

iterations.

Proof. Note that Ψ T ≥ f ( x T ) -f ∗ . In view of condition (33) from Theorem E.8, the step size satisfies

Therefore, since θ = 0 , the algorithm converges after

iterations. Using the choice of p P and p D , we have

Using the choice of β, we obtain the result of the theorem.

Corollary E.10. Let C t i be the Perm K compressors and Q t i be the independent (Assumption 1.6) Rand K compressors, both with K = d / n . Then, in the view of Corollary E.9, the total communication complexity is

Proof. The choice of compressors and parameters ensures that ω P = ω D = n -1 (Lemma A.6). Thus, the iteration complexity is

Since p P = p D = 1 / n and K = d / n , on average, the algorithm sends ≤ 2 d / n coordinates in both directions. Therefore, the total communication complexity is

0

100000

200000

300000

400000

500000

#bits / n (s-to-w)

10

1

10 0

10 1

||

f

(

x t

)||

M3 (# of workers = 10)

M3 (# of workers = 100)

M3 (# of workers = 1000)

CORE (# of workers = 10)

CORE (# of workers = 100)

0

100000

200000

300000

400000

500000

#bits / n (w-to-s)

10

1

10 0

10 1

||

f

(

x t

)||

M3 (# of workers = 10)

M3 (# of workers = 100)

M3 (# of workers = 1000)

CORE (# of workers = 10)

CORE (# of workers = 100)

0.0

0.5

1.0

1.5

2.0

#bits / n (s-to-w)

1e7

10

3

10

2

||

f

(

x t

)||

MARINA-P

EF21-P + DCGD

M3

CORE

0.0

0.5

1.0

1.5

2.0

#bits / n (w-to-s)

1e7

10

3

10

2

||

f

(

x t

)||

MARINA-P

EF21-P + DCGD

M3

CORE

F Experiments

The experiments were prepared in Python. The distributed environment was emulated on a machine with Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz and 64 cores.

F.1 Experiments with M3 on quadratic optimization tasks

We consider close-to-homogeneous quadratic optimization problem with A i = (1 + ξ i ) I d , where ξ i ∼ N (0 , 0 . 01) for all i ∈ [ n ] , and d = 1000 . We run two algorithms from Table 1, M3 and CORE , and check whether theory matches practice. In M3 , we use Perm K followed by the natural compressor C nat (Horváth et al., 2022) (composition of two unbiased compressors) on the server's side, and Rand K followed by C nat on the workers' side. We use K = ⌊ d/n ⌋ ∈ { 1 , 10 , 100 } for n ∈ { 1000 , 100 , 10 } . In CORE , the number of communicated coordinates is set to 10 . We run each experiment 5 times with different seeds and plot the average to reduce the noise factor. Only the step size is fine-tuned for each algorithm.

The results are presented in Figure 2. As expected, CORE does not change its behavior as the number of workers increases from 10 to 100; this is expected since CORE does not depend on n. At the same time, M3 does improve with n , which supports our findings from Theorem 5.1.

F.2 Experiments with an autoencoder and MNIST

We now compare MARINA-P , M3 , CORE , EF21-P + DCGD , and GD on a non-convex autoencoder problem. We train it on the MNIST dataset (LeCun et al., 2010) with objective function f ( D , E ) := 1 m ∑ m i =1 ∥ DE b i -b i ∥ 2 + λ 2 ∥ DE -I ∥ 2 F , where D ∈ R d 1 × d 2 , E ∈ R d 2 × d 1 , b i ∈ R d 1 are samples,

d 1 = 784 is the number of features, d 2 = 16 is the size of the encoding space, λ = 0 . 001 is a regularizer, and m = 60000 is the number of samples. The dimension of the problem is d = 25088 . We randomly split the dataset among n = 100 workers. For MARINA-P and M3 , we take Perm K followed by the natural compressor C nat on the server's side. On the workers' side, M3 uses Rand K and C nat. For EF21-P + DCGD , we take Rand K with C nat on both the workers' and server's sides. In each case, K = ⌊ d/n ⌋ = 250 . For CORE , we set the number of communicated coordinates to 100 . As in previous experiments, we only fine-tune the step size, repeat each experiment 5 times, and plot the average results.

The results are presented in Figure 3. All methods with bidirectional compression: M3 , CORE , and EF21-P + DCGD , converge much faster than GD . MARINA-P converges fastest only in the first plot. This is expected since it compresses only from the server to the workers. M3 , CORE , and EF21-P + DCGD have similar convergence rates in both metrics, with M3 performing better in the low accuracy regime.

F.3 Extra experiments with quadratic optimization tasks

The aim of this set of experiments is to empirically test our results under Assumption 4.2. We consider the problem of quadratic minimization with varying level of heterogeneity between the n functions stored on the workers. The goal is to minimize the squared norm of the gradient of ∑ n i =1 f i , where the functions f i are of form

Here, A i are d × d matrices generated following the procedure in Algorithm 3, and b i denotes a standard normal vector in R d . The constants L A and L B from Assumption 4.2 (in this case, by Theorem 4.8 L A = √ 2 max i ∈ [ n ] ∥ A i -A ∥ and L B = √ 2 ( 1 n ∑ n i =1 ∥ A i ∥ ) ) are controlled by parameters v i and σ 2 i . In particular, for σ 2 i = 0 , all workers hold the same matrix A i , and hence in this case L A = 0 .

We compare the following algorithms:

MARINA-P with Perm K compressors,

MARINA-P with Rand K compressors,

MARINA-P with SameRand K compressor,

EF21-P with Top K compressor,

GD .

In all compressed methods, we set K = d / n and use p = k / d in MARINA-P .

The step sizes are tuned from 2 i , i ∈ Z multiples of the values predicted by the theory (indicated by × 1 , × 2 , . . . in the plots). We fix d = 300 and generate optimization tasks with n ∈ { 10 , 100 , 900 } . The results are presented in Figures 4, 5, 6.

The empirical results align well with the theory. Among the algorithms tested, MARINA-P with Perm K compressor exhibits the best performance, while MARINA-P with SameRand K converges the slowest and comparable to GD . MARINA-P with Rand K compressor and EF21-P achieve performance levels somewhere in between. Notably, the differences between the runs of MARINA-P with different compressors become more pronounced as the value of n increases. As anticipated, the performance of MARINA-P with Rand K and Perm K compressors improves with an increase in the number of workers, while the performance of EF21-P does not follow the same behaviour. Specifically, for n = 10 , EF21-P outperforms MARINA-P with Rand K compressor, but this pattern reverses for both n = 100 and n = 1000 .

Algorithm 3 Heterogeneous quadratic problem generation

1: Parameters: v 0 , . . . , v 4 ∈ R + , σ 0 , . . . , σ 4 ∈ R ≥ 0 .

2: Let

3: for k = 0 , . . . , 4 do

10:

4: Generate ξ i ∼ N (0 , σ 2 k ) ∩ [ -v 0 , v 0 ] for i ∈ [ n ] 5: for l = 0 , . . . , 4 do 6: Set A k,l i = ( v l + ξ i ) X for i ∈ [ n ] 7: Sample b k,l i ∼ N (0 , I d ) for i ∈ [ n ] 8: end for 9: Output: matrices A k,l i , vectors b k,l i , i ∈ [ n ] , k, l ∈ [4] . end for

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=0,

L

2

B

=100, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×0.5

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=0,

L

2

B

=1000, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×0.5

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=0,

L

2

B

=10000, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×0.5

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=0,

L

2

B

=100000, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×0.5

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=1,

L

2

B

=100, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×1

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=1,

L

2

B

=1000, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×1

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=1,

L

2

B

=10000, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×1

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=1,

L

2

B

=100000, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×1

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=10,

L

2

B

=100, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×1

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=10,

L

2

B

=1000, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×1

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=10,

L

2

B

=10000, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×1

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=10,

L

2

B

=100000, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×1

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=100,

L

2

B

=100, n=10, d=300

EF21-P, Top30, ×16

MARINA-P, SameRand30, ×1

MARINA-P, Rand30, ×2

MARINA-P, Perm30, ×4

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=100,

L

2

B

=1000, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×1

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×4

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=100,

L

2

B

=10000, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×1

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

100000

200000

300000

400000

500000

600000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=100,

L

2

B

=100000, n=10, d=300

EF21-P, Top30, ×8

MARINA-P, SameRand30, ×1

MARINA-P, Rand30, ×1

MARINA-P, Perm30, ×2

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=0,

L

2

B

=100, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×2

MARINA-P, Perm3, ×2

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=0,

L

2

B

=1000, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×2

MARINA-P, Perm3, ×2

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=0,

L

2

B

=10000, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×1

MARINA-P, Perm3, ×2

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=0,

L

2

B

=100000, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×2

MARINA-P, Perm3, ×2

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=1,

L

2

B

=100, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×2

MARINA-P, Perm3, ×8

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=1,

L

2

B

=1000, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×2

MARINA-P, Perm3, ×4

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=1,

L

2

B

=10000, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×1

MARINA-P, Perm3, ×2

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=1,

L

2

B

=100000, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×2

MARINA-P, Perm3, ×2

DASHA-P, Rand3, ×4

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=10,

L

2

B

=100, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×4

MARINA-P, Perm3, ×16

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=10,

L

2

B

=1000, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×2

MARINA-P, Perm3, ×8

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=10,

L

2

B

=10000, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×1

MARINA-P, Perm3, ×4

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=10,

L

2

B

=100000, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×1

MARINA-P, Perm3, ×2

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=100,

L

2

B

=100, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×16

MARINA-P, Perm3, ×32

0

10000

20000

30000

40000

50000

60000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=100,

L

2

B

=1000, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×2

MARINA-P, Perm3, ×16

0

10000

20000

30000

40000

50000

60000

70000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=100,

L

2

B

=10000, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×2

MARINA-P, Perm3, ×8

0

10000

20000

30000

40000

50000

60000

70000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=100,

L

2

B

=100000, n=100, d=300

EF21-P, Top3, ×8

MARINA-P, SameRand3, ×0.5

MARINA-P, Rand3, ×1

MARINA-P, Perm3, ×4

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=0,

L

2

B

=100, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×4

MARINA-P, Perm1, ×2

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=0,

L

2

B

=1000, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×4

MARINA-P, Perm1, ×2

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=0,

L

2

B

=10000, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×4

MARINA-P, Perm1, ×2

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=0,

L

2

B

=100000, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×4

MARINA-P, Perm1, ×2

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=1,

L

2

B

=100, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×2

MARINA-P, Perm1, ×32

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=1,

L

2

B

=1000, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×2

MARINA-P, Perm1, ×8

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=1,

L

2

B

=10000, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×2

MARINA-P, Perm1, ×4

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=1,

L

2

B

=100000, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×2

MARINA-P, Perm1, ×2

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=10,

L

2

B

=100, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×4

MARINA-P, Perm1, ×64

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=10,

L

2

B

=1000, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×4

MARINA-P, Perm1, ×16

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=10,

L

2

B

=10000, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×4

MARINA-P, Perm1, ×8

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=10,

L

2

B

=100000, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×4

MARINA-P, Perm1, ×4

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=100,

L

2

B

=100, n=900, d=300

EF21-P, Top1, ×32

MARINA-P, SameRand1, ×1

MARINA-P, Rand1, ×64

MARINA-P, Perm1, ×64

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=100,

L

2

B

=1000, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×16

MARINA-P, Perm1, ×32

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=100,

L

2

B

=10000, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×4

MARINA-P, Perm1, ×16

0

5000

10000

15000

20000

Number of bits sent (s-to-w)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

f

(

x t

)

2

f

(

x

0

)

2

L

2

A

=100,

L

2

B

=100000, n=900, d=300

EF21-P, Top1, ×16

MARINA-P, SameRand1, ×0.5

MARINA-P, Rand1, ×4

MARINA-P, Perm1, ×8

G Proof of the Lower Bounds

G.1 The 'difficult' function from the nonconvex world

In our lower bound, we use the function from Carmon et al. (2020); Arjevani et al. (2022). For any T ∈ N , let

where

Carmon et al. (2020); Arjevani et al. (2022) also proved the following properties of the function:

Lemma G.1 (Carmon et al. (2020); Arjevani et al. (2022)) . The function F

F T (0) -inf x ∈ R T F T ( x ) ≤ ∆ 0 T, where ∆ 0 = 12 .

The function F T is l 1 -smooth, where l 1 = 152 .

For all x ∈ R T , ∥∇ F T ( x ) ∥ ∞ ≤ γ ∞ , where γ ∞ = 23 .

For all x ∈ R T , prog ( ∇ F T ( x )) ≤ prog ( x ) + 1 .

For all x ∈ R T , if prog ( x ) < T, then ∥∇ F T ( x ) ∥ > 1 ,

̸

where prog ( x ) := max { i ≥ 0 | x i = 0 } ( x 0 ≡ 1) .

The function is a standard function that is used to establish lower bounds in the nonconvex world (Carmon et al., 2020; Arjevani et al., 2022; Lu and De Sa, 2021; Tyurin and Richtárik, 2023c).

G.2 Theorems

Our lower bound applies to the family of methods with the following structure:

Protocol 4 Protocol

1: Input: functions f 1 , . . . , f n ∈ F , algorithm A, probability p

2: for k = 0 , . . . , ∞ do

3: Server calculates a new point: x k = B k 1 ( g 1 1 , . . . , g k 1 , . . . , g 1 n , . . . , g k n )

4: Server aggregates all available information: s k i = B k 2 ,i ( g 1 1 , . . . , g k 1 , . . . , g 1 n , . . . , g k n )

5: Server sends sparsified vectors ¯ s k i to the workers, where

̸

and η k i,j is a random variable such that P ( η k i,j = 0 ) ≤ p for all j ∈ [ d ′ ] and for all i ∈ [ n ] . We define d ′ := dim(dom( f 1 )) , and [ · ] j means the j th coordinate.

6: Workers aggregate all available local information and calculate gradients: g k +1 i = L k i (¯ s 0 i , . . . , ¯ s k i )

7: Workers send g k +1 i to the server

( L k i has access to the gradient oracle of f i and can call it as many times as it wants according to the rules (37) and (38))

8: end for

We consider the following standard classes of functions and algorithms:

Definition G.2. Let the function f : R d → R be differentiable, L -smooth (i.e., ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ for all x, y ∈ R d ), and f (0) -inf x ∈ R d f ( x ) ≤ δ 0 . We denote the family of functions that satisfy these properties by F δ 0 ,L .

T satisfies:

Definition G.3. Consider Protocol 4. A sequence of tuples of mappings A = { ( B k 1 , B k 2 , 1 , . . . , B k 2 ,n , L k 1 , . . . , L k n ) } ∞ k =0 is a zero-respecting algorithm, if,

we have

̸

for all k ∈ N 0 and for all i ∈ [ n ] , where supp ( x ) := { i ∈ [ d ] | x i = 0 } .

We denote the set of all algorithms that satisfy these properties by A zr .

The first three properties define the domains of the mapping. The last property is a standard assumption for a zero-respecting algorithm. Assumption (38) allows the mappings L k i to calculate gradients.

Theorem G.4. Consider Protocol 4. Assume that the sets { η 0 i,j } i ∈ [ n ] ,j ∈ [ d ′ ] , { η 1 i,j } i ∈ [ n ] ,j ∈ [ d ′ ] , . . . , { η k i,j } i ∈ [ n ] ,j ∈ [ d ′ ] , . . . are mutually independent (the variables within one set can be dependent). Let p > 0 , L, δ 0 , ε > 0 , n ≥ 2 be any numbers such that ¯ cε < Lδ 0 . Then, for any algorithm A ∈ A zr , there exists a function f ∈ F δ 0 ,L and functions f 1 , . . . , f n such that f = 1 n ∑ n i =1 f i and E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] > ε for all

The quantities ¯ c and ˆ c are universal constants.

Proof. The proof is conceptually the same as in Arjevani et al. (2022); Lu and De Sa (2021); Huang et al. (2022); Fang et al. (2018); Carmon et al. (2020); Tyurin and Richtárik (2023c). We fix λ > 0 , and consider the following function f : R T → R :

One can show (Arjevani et al., 2022)[Theorem 1] that f ∈ F δ 0 ,L if

Next, we define

and

The functions f i are zero-chain (Arjevani et al., 2022): for all i ∈ [ n ] , if prog ( x ) = j and ( j mod n )+ 1 = i, then prog ( ∇ f i ( x )) ≤ j + 1 , and for all i ∈ [ n ] , if prog ( x ) = j and ( j mod n ) + 1 = i, then prog ( ∇ f i ( x )) ≤ j . Using the zero-chain property and the fact that we consider the family of zero-respecting algorithms:

̸

The first non-zero coordinate can be discovered only by the first worker.

Assume that max k j =1 max n i =1 prog ( g j i ) = j ≥ 1 . An algorithm can discover one new non-zero coordinate in the ( j + 1) th position only if the ( j mod n + 1) th worker gets a non-zero j th coordinate from the server . This is by the construction of the functions f i . Note that for n ≥ 2 , one worker cannot discover two consecutive coordinates.

Let us define

ξ j = I [ In the j th iteration, the coordinate with index ¯ p ≡ k max j =1 n max i =1 prog ( g j i ) is not zeroed out in Line 5 of Protocol 4 to the worker with index (¯ p mod n +1) AND T -1 ≥ ¯ p ≥ 1] (¯ p = 0 if k = 0) . Then, we have

Assume that G j is the σ -algebra generated by all randomness up to the j th iteration (inclusive). Then, ξ j is G j -measurable, and, by the construction of Line 5 of Protocol 4, P ( ξ j +1 = 1 ∣ ∣ G j ) ≤ p, where we also use the assumption of the theorem that the sets of random variables are mutually independent. Using the standard approach with the Chernoff method (Arjevani et al., 2022; Lu and De Sa, 2021; Huang et al., 2022), one can show that

for all and ρ ∈ (0 , 1] . Therefore, we get

The idea is that we take the first block from (36) to the first worker, the second block to the second worker, . . . , ( n +1) th block to the first worker, and so on. Then, one can show that

for all

Using (40), we have

Let us take λ = √ 2 εl 1 L . Then

From (41) with ρ = 1 2 , we get

By the theorem's assumption, Lδ 0 ≥ ¯ cε. One can choose a universal constant ¯ c such that (42) holds for

where Θ hides only a universal constant.

G.3 Compressed communication with independent compressors

Protocol 5 is exactly the same as Protocol 4 except for Line 5 and describes the family of methods that send compressed vectors from the server to the workers.

Protocol 5 Protocol with Compressors

1: Input: functions f 1 , . . . , f n ∈ F , algorithm A, compressors C 1 , . . . , C n

2: for k = 0 , . . . , ∞ do

3: Server calculates a new point: x k = B k 1 ( g 1 1 , . . . , g k 1 , . . . , g 1 n , . . . , g k n )

4: Server aggregates all available information: s k i = B k 2 ,i ( g 1 1 , . . . , g k 1 , . . . , g 1 n , . . . , g k n )

5: Server sends compressed vectors ¯ s k i = C i ( s k i ) to the workers

( L k i has access to the gradient oracle of f i and can call it as many times as it wants according to the rules (37) and (38))

6: Workers aggregate all available local information and calculate gradients: g k +1 i = L k i (¯ s 0 i , . . . , ¯ s k i )

7: Workers send g k +1 i to the server

8: end for

Theorem G.5. Consider Protocol 5. Let ω ≥ 0 , L, δ 0 , ε > 0 , n ≥ 2 be any numbers such that ¯ cε < Lδ 0 . Then for any algorithm A ∈ A zr , there exists a function f ∈ F δ 0 ,L , functions f 1 , . . . , f n such that f = 1 n ∑ n i =1 f i , and i.i.d. compressors C 1 , . . . , C n ∈ U ( ω ) such that E [ ∥ ∥ ∇ f ( x k ) ∥ ∥ 2 ] > ε for all

The quantities ¯ c and ˆ c are universal constants.

for all

From (39), one can conclude that

Proof. We can use the result of Theorem G.4. It is sufficient to construct an appropriate compressor. Let us define p := 1 / ω +1 . We define the following compressor:

where S is a random subset of [ T ] and each element from [ T ] appears with probability p independently. Then, C is unbiased:

and

Therefore, we get C ∈ U ( ω ) . Let C i be i.i.d. instantiations of C for all i ∈ [ n ] . Since C is a sparsifier as in Line 5 of Protocol 4, we can use Theorem G.4 with p = 1 / ω +1 to finish the proof.

H Useful Identities and Inequalities

For all x, y, x 1 , . . . , x m ∈ R d , s > 0 and α ∈ (0 , 1] , we have:

Variance decomposition: For any random vector X ∈ R d and any non-random vector c ∈ R d , we have

Tower property: For any random variables X and Y , we have

Jensen's inequality: If f is a convex function and X is a random variable, then

Lemma H.1 (Lemma 2 of Li et al. (2021)) . Suppose that function f is L -smooth and let x t +1 = x t -γg t . Then for any g t ∈ R d and γ > 0 , we have

LemmaH.2 (Lemma 5 of Richtárik et al. (2021)) . Let a, b > 0 . If 0 ≤ γ ≤ 1 √ a + b , then aγ 2 + bγ ≤ 1 . Moreover, the bound is tight up to the factor of 2 since 1 √ a + b ≤ min { 1 √ a , 1 b } ≤ 2 √ a + b .

I Notation

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Sections 3, 4, and 5

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Sections 4.6 and 5.1

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions and the proofs are in Section 1.1 in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section F

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In the supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section F

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We run experiments with different seeds and plot averages to reduce noise factors (see the description in Sections F.1 and F.2).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section F

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the code of ethics, and we are confident that our paper is in compliance with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work considers a mathematical problem from machine learning.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Section F

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: In the supplementary materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDifferently from other papers in the field of federated learning, the authors consider the problem of uplink compression - from the server to the workers. The main contributions of the paper are MARINA-P and its variants. These optimization schemes leverage PermK sparsification to reduce the communication complexity in the uplink. When combined with momentum, MARINA-P achieves a total communication complexity that, in the case of $n>>1$ and for close-to-homogeneous clients, is smaller than competing state-of-the-art alternatives.\n\nSTRENGTHS:\nThe paper is original as it focuses on uplink compression, an aspect which is often overlooked in the federated learning community. The results are very interesting, as the authors not only show that under suitable conditions, MARINA-P has very low communication complexity, but they also demonstrate that without additional assumptions, there is an insurpassable lower bound to the communication complexity. The paper is very well-written; in fact, despite being very dense, it still introduces the problem and the proposed solution in a very clear manner.\n\nWEAKNESSES:\n- The main weakness of the paper is the assumption of data homogeneity. While I understand that this assumption is necessary, it would be interesting to include experimental evidence in the main text showing how MARINA-P performs with heterogeneous data.\n- The following might be erroneous, but to me seems like that the overall number of coordinates sent in the uplink is $d$. From the discussion in Section 4.5 it seems like that the parameter vector is permuted and divided into non-overlapping chunks, and each chunk is sent to a different client. As such, it looks like that the \\emph{overall} number of coordinates sent to the server is $d$.\n\nQUESTIONS:\n- Related to my previous point, how many coordinates are sent into the uplink at each round from the server to all users? i.e. what is the support of $\\sum^n_{i=1} \\mathcal{C}_i(x)$?\n- Is MARINA-P compatible with client selection? Meaning, is it possible to apply MARINA-P if, at each round, the number of users varies or a different subset is chosen?\n- Would it be possible to compare MARINA-P to other state-of-the-art schemes that use masking to reduce communication (e.g., SPARSE RANDOM NETWORKS FOR COMMUNICATION-EFFICIENT FEDERATED LEARNING)?\"\n\nLIMITATIONS:\nLimitations are adequately discussed in the text.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents methods to improve communication efficiency in distributed optimization, particularly focusing on the server-to-worker (s2w) communication costs, which are often overlooked. The authors first present a lower bound on the required round of communication.  They used it to state that under no assumption between the individual functions stored at the distributed workers, the s2w communication loads can not be improved compared to the naive design. Then, the authors provide an algorithm and analysis showing that when the hessian of the individual functions are similar or aligned, improvements in communication complexity can be achieved.\n\nSTRENGTHS:\nThe paper introduces a formal quantification, called functional $(L_A, L_B)$ inequality, to measure the similarity in the smoothness of the individual functions available to the workers. Novel algorithms and analysis are provided to show that the similarity can be used to improve the s2w communication load. \n\nThe paper also includes an interesting theoretical analysis, providing lower bounds on the required rounds of communication in the worst case.\n\nWEAKNESSES:\n1. The authors use the lower bound on the rounds communication to claim a lower bound on the communication load. However, there is a gap in this argument: The authors state that since the rounds of communication are lower bounded by a certain function, the overall communication load is lower bounded by d times that function, given that the full gradient contains d entries. But if a compressor is used as assumed by the authors, where the server can deliver an unbiased function that has fewer entries, it is totally possible that the communication load is smaller than d times the rounds of communication, violating this claimed bound. \n\nTherefore, I have to suggest a weak rejection. I'm happy to raise my score if this issue can be appropriately fixed.\n\n2. The concept of functional $(L_A, L_B)$ inequality can be simplified since only L_A is used in the main results. It can be stated such as the functions satisfy an $L_A$ condition, if we can find $L_B$ such that ineq. (9) is satisfied.\n\nQUESTIONS:\nIs there any connection between $L_A$ and the Hessians of the individual functions, similar to the smoothness parameter $L$, which can be interpreted as the largest eigenvalue of the Hessian of f?\n\nLIMITATIONS:\nSee weekness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses optimizing server-to-worker communication in distributed optimization by introducing MARINA-P, a novel method using correlated compressors for downlink compression. The study identifies inefficiencies in current downlink compression approaches and shows that MARINA-P can reduce communication complexity as the number of workers increases. Additionally, MARINA-P serves as a foundation for bidirectional compression methods. The paper also presents M3, which combines MARINA-P with uplink compression and a momentum step, further improving total communication complexity. Both theoretical analyses and empirical experiments validate the efficiency of the proposed algorithms.\n\nSTRENGTHS:\nThe paper is exceptionally well written, with technical results rigorously proved. One of the main novelties is the use of correlated compressors to achieve a server-to-worker communication complexity that improves with the number of workers.\n\nWEAKNESSES:\nThe compressor model (Definition 1.4), while standard in the federated learning literature, is somewhat contrived. It is primarily defined in its current form to facilitate analysis. It remains unclear whether the main conclusions derived under this model can be directly translated to the information-theoretic setting (quantization plus entropy coding), where compression efficiency is measured in terms of bits per sample.\n\nQUESTIONS:\nThe role of L_B is not completely clear, as it does not prominently affect communication complexities compared to L_A. Is it that by isolating L_B from L_A, one can effectively reduce L_A  and, consequently, the communication complexities as well? Moreover, the class of functions for which L_B is known to be positive is quite restricted (Theorem 4.5). It would be beneficial to find more general examples.\n\nLIMITATIONS:\nThe paper does not explicitly address its limitations. The only related point is that the proposed method does not consistently outperform the CORE method by Yue et al. (2023) in terms of performance (p. 8).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with the downlink (server-to-client) communication cost in federated learning (FL). The main motivation of the paper is to provide a theoretical analysis of the total communication cost with downlink compression. This is achieved considering unbiased or a certain class of biased lossy compression algorithms at the server. First, the authors present an impossibility result (i.e., a converse theorem), showing that faster than certain rate is not possible if the server employs an unbiased compressor (or biased compressor as specified in the paper). Then, they proposed a downlink compression method called MARINA-P, which achieves better iteration than those in the literature. This method, basically, transmits an uncompressed model with a small probability p to all the clients, while the compressed model update is transmitted otherwise. The main twist in the proposed scheme is to transmit different compressed versions to the clients, rather than sending the same compression. In particular, an orthogonal sparse subset of the parameters are sent to each client, so that they together can recover all the parameters. This way, each parameter is considered at least by one of the workers.  Here, the compressed versions transmitted to the users become correlated. The authors show that this correlated compression approach achieves the best convergence rate. Finally, they extend this framework by combining downlink compression with uplink compression using MARINA together with a momentum step.\n\nSTRENGTHS:\nThe paper looks at an important yet under explored aspect of communication in FL. The authors propose a novel server-to-client compression method, and provide upper and lower bounds on the convergence rate under various assumptions. Overall, the assumptions seem reasonable, and are shown to be satisfied by common compressors such as randK (and the introduced permK). \n\nThe proposed compressor is rather simple, but the observation that the best convergence rate is achieved by this particular correlated compression method is interesting.\n\nWEAKNESSES:\nI would say that the lack of any empirical results is the main weakness of the paper. While it is interesting to obtain the theoretical convergence guarantees, one also wonders how these three different compression schemes would perform in practice (for scenarios that satisfy the assumptions, as well as those that do not). \n\nAlso, only sparsification type compression methods are considered. What about quantization? Can the arguments be easily extended to the case with quantization?\n\nQUESTIONS:\n- While the authors have covered most of the relevant literature, as far as I'm aware, one missing paper that I am aware of is the following:\n\nM. Mohammadi Amiri, D. Gündüz, S. R. Kulkarni, and H. V. Poor, \"Federated Learning With Quantized Global Model Updates\" arXiv preprint arXiv:2006.10672, Jun 2020. \n\nHere, not only the authors consider downlink communication, but they actually transmit different updates to different clients. Instead of compressing the difference with rest to the previous global model, here the authors compress the difference with respect to the model updated by the client. This way the gap to the new global model is potentially lower. I can't say how the convergence rates compare, but I would like the authors to comment on the pros/cons of this approach compared to their own. \n\n- How limiting is the assumption that the calls to the compressor are independent? There are some works in the literature that consider time-correlated sparsification, and show improved results (which also makes sense as the sparsity pattern should not change drastically from one iteration to the next). Please provide some comments in the paper if this may be a significant limitation on the practical performance of the considered class of compressors.\n\nLIMITATIONS:\nAs mentioned above: \n- lack of numerical comparisons,\n- one missing reference", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-30"}