Title: Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?
Venue: NeurIPS 2023
Accepted: yes
Keywords: Communication Compression

Distributed Optimization

Unbiased Compression

Optimal Complexity
Abstract: Communication compression is a common technique in distributed optimization
that can alleviate communication overhead by transmitting compressed gradients
and model parameters. However, compression can introduce information distortion,
which slows down convergence and incurs more communication rounds to achieve
desired solutions. Given the trade-off between lower per-round communication
costs and additional rounds of communication, it is unclear whether communication
compression reduces the total communication cost.

This paper explores the conditions under which unbiased compression, a widely
used form of compression, can reduce the total communication cost, as well as the
extent to which it can do so. To this end, we present the first theoretical formulation
for characterizing the total communication cost in distributed optimization with
unbiased compressors. We demonstrate that unbiased compression alone does not
necessarily save the total communication cost, but this outcome can be achieved
if the compressors used by all workers are further assumed independent. We
establish lower bounds on the communication rounds required by algorithms using
independent unbiased compressors to minimize smooth convex functions and
show that these lower bounds are tight by refining the analysis for ADIANA.
Our results reveal that using independent unbiased compression can reduce the
total communication cost by a factor of up to $\Theta(\sqrt{\min\\{n,\kappa\\}})$ when all local
smoothness constants are constrained by a common upper bound, where $n$ is the
number of workers and $\kappa$ is the condition number of the functions being minimized.
These theoretical findings are supported by experimental results.

Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?

Abstract

Communication compression is a common technique in distributed optimization that can alleviate communication overhead by transmitting compressed gradients and model parameters. However, compression can introduce information distortion, which slows down convergence and incurs more communication rounds to achieve desired solutions. Given the trade-off between lower per-round communication costs and additional rounds of communication, it is unclear whether communication compression reduces the total communication cost.

This paper explores the conditions under which unbiased compression, a widely used form of compression, can reduce the total communication cost, as well as the extent to which it can do so. To this end, we present the first theoretical formulation for characterizing the total communication cost in distributed optimization with unbiased compressors. We demonstrate that unbiased compression alone does not necessarily save the total communication cost, but this outcome can be achieved if the compressors used by all workers are further assumed independent. We establish lower bounds on the communication rounds required by algorithms using independent unbiased compressors to minimize smooth convex functions and show that these lower bounds are tight by refining the analysis for ADIANA. Our results reveal that using independent unbiased compression can reduce the total communication cost by a factor of up to Θ( √ min { n, κ } ) when all local smoothness constants are constrained by a common upper bound, where n is the number of workers and κ is the condition number of the functions being minimized. These theoretical findings are supported by experimental results.

1 Introduction

Distributed optimization is a widely used technique in large-scale machine learning, where data is distributed across multiple workers and training is carried out through worker communication. However, dealing with a vast number of data samples and model parameters across workers poses a significant challenge in terms of communication overhead, which ultimately limits the scalability of distributed machine learning systems. To tackle this issue, communication compression strategies [3, 8, 52, 55, 49] have emerged, aiming to reduce overhead by enabling efficient yet imprecise message transmission. Instead of transmitting full-size gradients or models, these strategies exchange compressed gradients or model vectors of much smaller sizes in communication.

There are two common approaches to compression: quantization and sparsification. Quantization [3, 20, 39, 52] maps input vectors from a large, potentially infinite, set to a smaller set of discrete values. In contrast, sparsification [57, 55, 50] drops a certain amount of entries to obtain a sparse vector for communication. In literature [3, 27, 21], these compression techniques are often modeled

as a random operator C , which satisfies the properties of unbiasedness E [ C ( x )] = x and ω -bounded variance E ∥ C ( x ) -x ∥ 2 ≤ ω ∥ x ∥ 2 . Here, x represents the input vector to be compressed, and ω is a fixed parameter that characterizes the degree of information distortion. Besides, part of the compressors can also be modeled as biased yet contractive operators [19, 48, 49].

While communication compression efficiently reduces the volume of vectors sent by workers, it suffers substantial information distortion. As a result, algorithms utilizing communication compression require additional rounds of communication to converge satisfactorily compared to algorithms without compression. This adverse effect of communication compression has been extensively observed both empirically [57, 25, 8] and theoretically [21, 49]. Since the extra rounds of communication needed to compensate for the information loss may outweigh the saving in the per-round communication cost from compression, this naturally motivates the following fundamental question:

Q1. Can unbiased compression alone reduce the total communication cost?

By 'unbiased compression alone', we refer to the compression that solely satisfies the assumptions of unbiasedness and ω -bounded variance without any additional advanced properties. To address this open question, we formulate the total communication cost as the product of the per-round communication cost and the number of rounds needed to reach an ϵ -accurate solution to distributed optimization problems. Using this formulation, we demonstrate the decrease in the per-round communication cost from unbiased compression is completely offset by additional rounds of communication. Therefore, we answer Q1 by showing unbiased compression alone cannot ensure a lower total communication cost, even with an optimal algorithmic design, see Sec. 3 for more details. This negative conclusion drives us to explore the next fundamental open question:

Q2. Under what additional conditions and how much can unbiased compression provably save the total communication cost?

Fortunately, some pioneering works [40, 32, 33] have shed light on this question. They impose independence on unbiased compressors, i .e., the compressed vectors { C i ( x i ) } n i =1 sent by workers are mutually independent regardless of the inputs { x i } n i =1 . This independence assumption enables an "error cancellation" effect, producing a more accurate compressed vector n -1 ∑ n i =1 C i ( x i ) and hence incurring fewer additional rounds of communication compared to dependent compressors. Consequently, the decrease in the per-round communication cost outweighs the extra communication rounds, reducing the total communication cost.

However, it remains unclear how much the total communication cost can be reduced at most by independent unbiased compression and whether we can develop algorithms to achieve this optimal reduction. Addressing this question poses significant challenges as it necessitates a study of the optimal convergence rate for algorithms using independent unbiased compression.

This paper provides the first affirmative answer to this question for convex problems by: (i) establishing lower bounds on convergence rates of distributed algorithms employing independent unbiased compression, and (ii) demonstrating the tightness of these lower bounds by revisiting ADIANA [32] and presenting novel and refined convergence rates nearly attaining the lower bounds. Our results reveal that compared to non-compression algorithms, independent unbiased compression can save the total communication cost by up to Θ( √ min { n, κ } ) -fold, where n is the number of workers and κ ∈ [1 , + ∞ ] is the function condition number. Figure 1 provides a simple empirical justification. It shows independent compression (ADIANA i.d.) reduces communication costs compared to no compression (Nesterov's Accelerated algorithm), while dependent compression (ADIANA s.d.) does not, which validates our theory.

1.1 Contributions

Specifically, our contributions are as follows:

0.0

1.0

2.0

3.0

4.0

5.0

6.0

communication bits (×1e5)

10 -12

10 -10

10 -8

10 -6

10 -4

10 -2

10 0

10 2

f(x) - f*

Nesterov

ADIANA i.d.Rand-1

ADIANA i.d.Rand-2

ADIANA i.d.Rand-4

ADIANA s.d.Rand-1

ADIANA s.d.Rand-2

ADIANA s.d.Rand-4

♢ Results obtained in the single-worker setting and cannot be extended to the distributed setting.

‡ The rate is obtained by correcting mistakes in the derivations of [33]. See details in Appendix E.

Wepresent a theoretical formalization of the total communication cost in distributed optimization with unbiased compression. With this formulation, we demonstrate that unbiased compression alone is insufficient to save the total communication cost, even with an optimal algorithmic design. This is because any reduction in the per-round communication cost is fully offset by the additional rounds of communication required due to the presence of compression errors.

We prove lower bounds on the convergence complexity of distributed algorithms using independent unbiased compression to minimize smooth convex functions. Compared to lower bounds when using unbiased compression without independence [19], our lower bounds demonstrate significant improvements when n and κ are large, see the first two lines in Table 1. This improvement highlights the importance of independence in unbiased compression.

We revisit ADIANA [32] by deriving an improved rate for strongly-convex functions and proving a novel convergence result for generally-convex functions. Our rates nearly match the lower bounds, suggesting their tightness and optimality. Our optimal complexities reveal that, compared to non-compression algorithms, independent unbiased compression can decrease total communication costs by up to O ( √ min { n, κ } ) -fold when all local smoothness constants are constrained by a common upper bound.

We support our theoretical findings with experiments on both synthetic data and real datasets.

Wepresent the lower bounds, upper bounds, and complexities of state-of-the-art distributed algorithms using independent unbiased compressors in Table 1. With our new and refined analysis, ADIANA nearly matches the lower bounds for both strongly-convex and generally-convex functions.

1.2 Related work

Communication compression. Two main approaches to compression are extensively explored in literature: quantization and sparsification. Quantization coarsely encodes input vectors into fewer discrete values, e .g., from 32-bit to 8-bit integers [37, 18]. Schemes like Sign-SGD [52, 8] use 1 bit per entry, introducing unbiased random information distortion. Other variants such as Q-SGD [3], TurnGrad [58], and natural compression [20] quantize each entry with more effective bits. In contrast, sparsification either randomly zeros out entries to yield sparse vectors [57], or transmits only the largest model/gradient entries [55].

Error compensation. Recent works [52, 59, 55, 4, 49] propose error compensation or feedback to relieve the effects of compression errors. These techniques propagate information loss backward during compression, thus preserving more useful information. Reference [52] uses error compensation for 1-bit quantization, while the work [59] proposes error-compensated quantization for quadratic problems. Error compensation also reduces sparsification-induced errors [55] and is studied for convergence in non-convex scenarios [4]. Recently, the work [49] introduces EF21, an error feedback scheme that compresses only local gradient increments with improved theoretical guarantees.

Lower bounds. Lower bounds in optimization set a limit for the performance of a single or a class of algorithms. Prior works have established numerous lower bounds for optimization algorithms [1, 15, 6, 43, 7, 21, 64, 17, 44]. In the field of distributed optimization with communication compression, reference [46] provides an algorithm-specific lower bound for strongly-convex functions, while the work [53] establishes the bit-wise complexities for PL-type problems, which reflect the influence of the number of agents n and dimension d , but not the condition number κ and the compression ω . In particular, [21] characterizes the optimal convergence rate for all first-order and linear-spanning algorithms, in the stochastic non-convex case, which is later extended by [19] to convex cases.

Accelerated algorithms with communication compression. There is a scarcity of academic research on compression algorithms incorporating acceleration, as evident in a limited number of studies [32, 33, 48]. References [32, 33] develop accelerated algorithms with compression in the strongly-convex and generally-convex cases, respectively. For distributed finite-sum problems, accelerated algorithms with compression can further leverage variance-reduction techniques to expedite convergence [48].

Other communication-efficient strategies. Other than communication compression studied in this paper, there are a few different techniques to mitigate the communication overhead in distributed systems, including decentralized communication and lazy communication. Notable examples of decentralized algorithms encompass decentralized SGD [12, 34, 30, 64], D2/Exact-Diffusion [56, 62, 61], gradient tracking [47, 60, 29, 2], and their momentum variants [35, 63]. Lazy communication allows each worker to either perform multiple local updates as opposed to a single communication round [38, 54, 41, 24, 14, 22], or by adaptively skipping communication [13, 36].

2 Problem setup

This section introduces the problem formulation and assumptions used throughout the paper. We consider the following distributed stochastic optimization problem

where the global objective function f ( x ) is decomposed into n local objective functions { f i ( x ) } n i =1 , and each local f i ( x ) is maintained by node i . Next, we introduce the setup and assumptions.

2.1 Function class

We let F ∆ L,µ ( 0 ≤ µ ≤ L ) denote the class of convex and smooth functions satisfying Assumption 1. We define κ ≜ L/µ ∈ [1 , + ∞ ] as the condition number of the functions to be optimized. When µ > 0 , F ∆ L,µ represents strongly-convex functions. Conversely, when µ = 0 , F ∆ L,µ represents generally-convex functions with κ = ∞ .

Assumption 1 (CONVEX AND SMOOTH FUNCTION) . We assume each f i ( x ) is L -smooth and µ -strongly convex, i .e., there exists constants L ≥ µ ≥ 0 such that

for any x, y ∈ R d and 1 ≤ i ≤ n . We further assume ∥ x 0 -x ⋆ ∥ 2 ≤ ∆ where x ⋆ is one of the global minimizers of f ( x ) = 1 n ∑ n i =1 f i ( x ) .

2.2 Compressor class

Each worker i ∈ { 1 , · · · , n } is equipped with a potentially random compressor C i : R d → R d . We let U ω denote the set of all ω -unbiased compressors satisfying Assumption 2, and U ind ω denote the set of all independent ω -unbiased compressors satisfying both Assumption 2 and Assumption 3.

Assumption 2 (UNBIASED COMPRESSOR) . We assume all compressors { C i } n i =1 satisfy

for constant ω ≥ 0 and any input x ∈ R d , where the expectation is taken over the randomness of the compression operator C i . 2

Assumption 3 (INDEPENDENT COMPRESSOR) . We assume all compressors { C i } n i =1 are mutually independent, i .e., outputs { C i ( x i ) } n i =1 are mutually independent random variables for any { x i } n i =1 .

2.3 Algorithm class

Similar to [19], we consider centralized and synchronous algorithms in which first, every worker is allowed to communicate only directly with a central server but not between one another; second, all iterations/communications are synchronized, meaning that all workers start each of their iterations simultaneously. We further require algorithms to satisfy the so-called 'linear-spanning' property, which appears in [9, 10, 21, 19] (see formal definition in Appendix C). Intuitively, this property requires each local model x k i to lie in the linear manifold spanned by the local gradients and the received messages at worker i . The linear-spanning property is satisfied by all algorithms in Table 1 as well as most first-order methods [42, 28, 23, 65].

Formally, this paper considers a class of algorithms specified by Definition 1.

Definition 1 (ALGORITHM CLASS) . Given compressors { C i } n i =1 , we let A { C i } n i =1 denote the set of all centralized, synchronous, linear-spanning algorithms admitting compression in which compressor C i , ∀ 1 ≤ i ≤ n , is applied for the messages sent by worker i to the server.

For any algorithm A ∈ A { C i } n i =1 , we define ˆ x k and x k i as the output of the server and worker i respectively, after k communication rounds.

2.4 Convergence complexity

With all the interested classes introduced above, we are ready to define our complexity metric for convergence analysis. Given a set of local functions { f i } n i =1 ∈ F ∆ L,µ , a set of compressors { C i } n i =1 ∈ C ( C = U ind ω or U ω ), and an algorithm A ∈ A { C i } n i =1 , we let ˆ x t A denote the output of algorithm A after t communication rounds. The convergence complexity of A solving f ( x ) = 1 n ∑ n i =1 f i ( x ) under { ( f i , C i ) } n i =1 is defined as

This measure corresponds to the number of communication rounds required by algorithm A to achieve an ϵ -accurate optimum of f ( x ) in expectation.

Remark 1. The measure in (3) is commonly referred to as the communication complexity in literature [51, 21, 31, 32]. However, we refer to it as the convergence complexity here to avoid potential confusion with the notion of 'communication complexity' and 'total communication cost'. This complexity metric has been traditionally used to compare communication rounds used by distributed algorithms [32, 33]. However, it cannot capture the total communication costs of multiple algorithms with different per-round communication costs, e .g., algorithms with or without communication compression. Therefore, it is unable to address the motivating questions Q1 and Q2.

Remark 2. The definition of T ϵ can be independent of the per-round communication cost, which is specified only through the degree of compression ω ( i .e., choice of compressor class). However, to be precise, we may further assume these compressors are non-adaptive with the same fixed per-round communication cost. Namely, the compressors output compressed vectors that can be represented by a fixed and common number of bits. Notably, such hypothesis of non-adaptive cost is widely adopted for practical comparison of communication costs and is valid when input x is bounded or can be encoded with finite bits [3, 59, 20, 22].

3 Total communication cost

3.1 Fomulation of total communication cost

This section introduces the concept of Total Communication Cost (TCC). TCC can be calculated at both the level of an individual worker and of the overall distributed machine learning system comprising all n workers. In a centralized and synchronized algorithm where each worker communicates compressed vectors of the same dimension, the TCC of the entire system is directly proportional to the TCC of a single worker. Therefore, it is sufficient to use the TCC of a single worker as the metric for comparing different algorithms. In this paper, we let TCC denote the total communication cost incurred by each worker in achieving a desired solution when no ambiguity is present.

Let each worker to be equipped with a non-adaptive compressor with the same fixed per-round communication cost, i .e., the compressor outputs compressed vectors of the same length (size), the TCC of an algorithm A to solve problem (1) using a set of ω -unbiased compressors { C i } n i =1 in achieving an ϵ -accurate optimum can be characterized as

3.2 A tight lower bound for per-round cost

The per-round communication cost incurred by { C i } n i =1 in (4) will vary with different ω values. Typically, compressors that induce less information distortion, i .e., associated with a smaller ω , incur higher per-round costs. To illustrate this, we consider randoms sparsification compressors, whose per-round cost corresponds to the transmission of s entries, which depends on parameter ω through s = d/ (1+ ω ) (see Example 1 in Appendix A). Specifically, if each entry of the input x is numerically represented with r bits, then the randoms sparsification incurs a per-round cost of rd/ (1 + ω ) bits up to a logarithm factor.

The following proposition, motivated by the inspiring work [50], establishes a lower bound of TCC when using any compressor satisfying Assumption 2.

Proposition 1. Let x ∈ R d be the input to a compressor C and b be the number of bits needed to compress x . Suppose each entry of input x is numerically represented with r bits, i .e., errors smaller than 2 -r are ignored. Then for any compressor C satisfying Assumption 2, the per-round communciation cost of C ( x ) is lower bounded by b = Ω r ( d/ (1 + ω )) where r is viewed as an absolute number in Ω r ( · ) (See the proof in Appendix B).

Proposition 1 presents a lower bound on the per-round cost of an arbitrary compressor satisfying Assumption 2. This lower bound is tight since the randoms compressor discussed above can achieve this lower bound up to a logarithm factor. Since d only relates to the problem instance itself and r is often a constant absolute number in practice, e .g., r = 32 or 64 , both of which are independent of the choices of compressors and algorithm designs, they can be omitted from the lower bound order. As a result, the TCC in (4) can be lower bounded by

Notably, when no compression is employed ( i .e., ω = 0 ), TCC ϵ = Ω(1) × T ϵ ( A, { ( f i , C i ) } n i =1 ) is consistent with the convergence complexity.

4 Unbiased compressor alone cannot save total communication cost

With formulation (5), given the number of communication rounds T ϵ , the total communication cost can be readily characterized. A recent pioneer work [19] characterizes a tight lower bound for T ϵ ( A, { ( f i , C i ) } n i =1 ) when each C i satisfies Assumption 2.

Lemma 1 ([19], Theorem 1, Informal) . Relying on unbiased compressibility alone, i .e., { C i } n i =1 ∈ U ω , without leveraging additional property of compressors such as mutual independence, the fewest rounds of communication needed by algorithms with compressed communication to achieve an ϵ -accurate solution to distributed strongly-convex and generally-convex optimization problems are lower bounded by T ϵ = Ω((1 + ω ) √ κ ln ( µ ∆ /ϵ )) and T ϵ = Ω((1 + ω ) √ L ∆ /ϵ ) , respectively.

Substituting Lemma 1 into our TCC lower bound in (5), we obtain TCC ϵ = ˜ Ω( √ κ ln(1 /ϵ )) or Ω( √ L ∆ /ϵ ) in the strongly-convex or generally-convex case, respectively, by relying solely on

unbiased compression. These results do not depend on the compression parameter ω , indicating that the lower per-round cost is fully compensated by the additional rounds of communication incurred by compressor errors . Notably, these lower bounds are of the same order as optimal algorithms without compression such as Nesterov's accelerated gradient descent [43, 44], leading to the conclusion:

Theorem 1. When solving convex optimization problems following Assumption 1, any algorithm A ∈ A { C i } n i =1 that relies solely on unbiased compression satisfying Assumption 2 cannot reduce the total communication cost compared to not using compression. The best achievable total communication cost with unbiased compression alone is of the same order as without compression.

Theorem 1 presents a negative finding that unbiased compression alone is insufficient to reduce the total communication cost, even with an optimal algorithmic design. 3 Meanwhile, it also implies that to develop algorithms that provably reduce the total communication cost, one must leverage compressor properties beyond ω -unbiasedness as defined in (2). Fortunately, mutual independence is one such property which we discuss in depth in later sections.

5 Independent unbiased compressor provably saves communication

5.1 An intuition on why independence can help

A series of works [40, 32, 33] have shown theoretical improvements in the total communication cost by imposing independence across compressors, i .e., { C i } n i =1 ∈ U ind ω . The intuition behind the role of independence among worker compressors can be illustrated by a simple example where workers intend to transmit the same vector x to the server. Each worker i sends a compressed message C i ( x ) that adheres to Assumption 2. Consequently, the aggregated vector n -1 ∑ n i =1 C i ( x ) is an unbiased estimate of x with variance

̸

If the compressed vectors { C i ( x ) } n i =1 are further assumed to be independent, i .e., { C i } n i =1 ∈ U ind ω , then the cancellation of cross error terms leads to the following equation:

We observe that the mutual independence among unbiased compressors leads to a decreased variance, which corresponds to the information distortion, of the aggregated message. Remarkably, this reduction is achieved by a factor of n compared to the transmission of a single compressor. Therefore, the independence among the compressors plays a pivotal role in enhancing the accuracy of the aggregated vector, consequently reducing the number of required communication rounds.

On the contrary, in cases where independence is not assumed and no other properties of compressors can be leveraged, the use of Cauchy's inequality only allows us to bound variance (6) as follows:

It is important to note that the upper bound ω ∥ x ∥ 2 can only be attained when the compressors { C i } n i =1 are identical, indicating that this bound cannot be generally improved further. By comparing (7) and (8), we can observe that the variance of the aggregated vector achieved through unbiased compression with independence can be n times smaller than the variance achieved without independence.

5.2 Convergence lower bounds with independent unbiased compressors

While mutual independence can boost the unbiased worker compressors, it remains unclear how much the total communication cost can be reduced at most by independent unbiased compression

and how to develop algorithms to achieve this optimal reduction. The following subsections aim to address these open questions.

Following the formulation in (5), to establish the best achievable total communication cost using independent unbiased compression , we shall study tight lower bounds on the number of communication rounds T ϵ to achieve an ϵ -accurate solution, which is characterized by the following theorem.

Theorem 2. For any L ≥ µ ≥ 0 , n ≥ 2 , the following results hold. See the proof in Appendix C.

Strongly-convex: For any ∆ > 0 , there exists a constant c κ only depends on κ ≜ L/µ , a set of local loss functions { f i } n i =1 ∈ F ∆ L,µ> 0 , independent unbiased compressors { C i } n i =1 ∈ U ind ω , such that the output ˆ x of any A ∈ A { C i } n i =1 starting from x 0 requires

rounds of communication to reach E [ f (ˆ x )] -min x f ( x ) ≤ ϵ for any 0 < ϵ ≤ c κ µ ∆ .

Generally-convex: For any ∆ > 0 , there exists a constant c = Θ(1) , a set of local loss functions { f i } n i =1 ∈ F ∆ L, 0 , independent unbiased compressors { C i } n i =1 ∈ U ind ω , such that the output ˆ x of any A ∈ A { C i } n i =1 starting from x 0 requires at least

rounds of communication to reach E [ f (ˆ x )] -min x f ( x ) ≤ ϵ for any 0 < ϵ ≤ cL ∆ .

Consistency with prior works. The lower bounds established in Theorem 2 are consistent with the best-known lower bounds in previous literature. When ω = 0 , our result reduces to the lower bound for distributed first-order algorithms established by Y. Nesterov in [43]. When n = 1 , our result reduces to the lower bound established in [19] for the single-node case.

Independence improves lower bounds. A recent work [19] establishes lower bounds for unbiased compression without the independence assumption, listed in the second row of Table 1. Compared to these results, our lower bound in Theorem 2 replaces ω with ω/ √ n , showing a reduction in order. This reduction highlights the role of independence in unbiased compression. To better illustrate the reduction, we take the strongly-convex case as an example. The ratio of the number of communication rounds T ϵ under unbiased compression with independence to the one without independence is:

Clearly, using independent unbiased compression can allow algorithms to converge faster, by up to a factor of Θ( √ min { n, κ } ) (attained at ω ≳ √ min { n, κ } ), in terms of the number of communication rounds, compared to the best algorithms with unbiased compressors but without independence.

Total communication cost. Substituting Theorem 2 into the TCC formulation in (5), we can obtain the TCC of algorithms using independent unbiased compression. Comparing this with algorithms without compression, such as Nesterov's accelerated algorithm, and using the relations in (9), we can demonstrate that independent unbiased compression can reduce the total communication cost. Such reduction can be up to Θ( √ min { n, κ } ) by using compressors with ω ≳ √ min { n, κ } , e .g., randoms sparsification with s ≲ d/ √ min { n, κ } .

5.3 ADIANA: a unified optimal algorithm

By comparing existing algorithms using independent unbiased compression, such as DIANA, ADIANA, and CANITA, to our established lower bounds in Table 1, it becomes clear that there is a noticeable gap between their convergence complexities and our established lower bounds. This gap could indicate that these algorithms are suboptimal, but it could also mean that our lower bounds are loose. As a result, our claim that using independent unbiased compression reduces the total communication cost by up to Θ( √ min { n, κ } ) times is not well-grounded yet. In this section, we address this issue by revisiting ADIANA [32] (Algorithm 1) and providing novel and refined convergence results in both strongly- and generally-convex cases.

Algorithm 1: ADIANA

Input:

1

,k

Scalars

θ

{

}

Initialize

w

T

0

=

for

x

k

= 0

,

· · ·

0

, T

On server:

Update x : x k = θ 1 ,k z k + θ 2 w k +(1 -θ 1 ,k -θ 2 ) y k and broadcast to all workers;

On all workers in parallel:

Compress the increment of local gradient m k i = C i ( ∇ f i ( x k ) -h k i ) and send to the server; Compress the increment of local gradient c k i = C i ( ∇ f i ( w k ) -h k i ) and send to the server; Update local shift h k +1 i = h k i + αc k i ;

On server:

Aggregate received compressed message

Update shift

=

h

h

k

Apply gradient descent

;

+1

k

1

∑

n

=

k

i

c

η

k

g

g

k

k

;

+

y

k

α

+1

x

n

i

=1

k

-

In the strongly-convex case, we refine the analysis of [32] by: (i) adopting new parameter choices where the initial scalar θ 2 is delicately chosen instead of being fixed as θ 2 = 1 / 2 in [32], (ii) balancing different terms in the construction of the Lyapunov function. While we do not modify the algorithm design, our technical ingredients are necessary to obtain an improved convergence rate. In the generally-convex case, we provide the first convergence result for ADIANA, which is missing in literature to our knowledge. In both strongly- and generally-convex cases, our convergence results (nearly) match the lower bounds in Theorem 2. This verifies the tightness of our lower bounds for both the convergence complexity and the total communication cost. In particular, our results are:

Theorem 3. For any L ≥ µ ≥ 0 , ∆ ≥ 0 , n ≥ 1 , and precision ϵ > 0 , the following results hold. See the proof in Appendix D.

Strongly-convex: If µ > 0 , by setting parameters η k ≡ η = nθ 2 / (120 ωL ) , θ 1 ,k ≡ θ 1 = 1 / (3 √ κ ) , α = p = 1 / (1 + ω ) , γ k ≡ γ = η/ (2 θ 1 + ηµ ) , β = 2 θ 1 / (2 θ 1 + ηµ ) , and θ 2 = 1 / (3 √ n +3 n/ω ) , ADIANA requires

rounds of communication to reach E [ f (ˆ x )] -min x f ( x ) ≤ ϵ .

Generally-convex: If µ = 0 , by setting parameters α = 1 / (1 + ω ) , β = 1 , p = θ 2 = 1 / (3(1 + ω )) , θ 1 ,k = 9 / ( k +27(1 + ω )) , γ k = η k / (2 θ 1 ,k ) , and

ADIANA requires

=

h

k

+

1

n

∑

n

i

=1

m

k

i

;

=

-

1

k

-

=0

y

1

,

0

=

do

θ

2

z

,

0

α

,

=

β

,

h

0

{

γ

=

k

}

h

T

1

-

k

0

=0

i

,

,

∀

1

{

η

k

i

}

T

1

-

k

=0

n

.

,

p

.

rounds of communication to reach E [ f (ˆ x )] -min x f ( x ) ≤ ϵ .

Tightness of our lower bounds. Comparing the upper bounds in Theorem 3 with the lower bounds in Theorem 2, ADIANA attains the lower bound in the strongly-convex case up to a ln( κ ) factor, implying the tightness of our lower bound and ADIANA's optimality. In the generally-convex case, the upper bound matches the lower bound's dominating term (1 + ω/ √ n ) √ L ∆ /ϵ but mismatches the smaller term. This shows the tightness of our lower bound and ADIANA's optimality in the

0.0

0.2

0.4

0.6

0.8

1.0

1.2

communication bits (×1e6)

10 -8

10 -7

10 -6

10 -5

10 -4

10 -3

10 -2

10 -1

f(x) - f*

Nesterov

ADIANA

DIANA

EF21

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

communication bits (×1e6)

10 -5

10 -4

10 -3

10 -2

10 -1

f(x) - f*

Nesterov

ADIANA

CANITA

DIANA

EF21

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

communication bits (×1e6)

10 -5

10 -4

10 -3

10 -2

10 -1

f(x) - f*

Nesterov

ADIANA

CANITA

DIANA

EF21

high-precision regime ϵ < L ∆ ( 1+ ω/ √ n 1+ ω ) 6 . Our refined rates for ADIANA are state-of-the-art among existing algorithms using independent unbiased compression.

6 Experiments

In this section, we empirically compare ADIANA with DIANA [32], EF21 [49], and CANITA [33] using unbiased compression, as well as Nesterov's accelerated algorithm [43] which is an optimal algorithm when no compression is employed. We conduct experiments on least-square problems (strongly-convex) with synthetic datasets as well as logistic regression problems (generally-convex) with real datasets. In all experiments, we measure the total communicated bits sent by a single worker, which is calculated through communication rounds to acheive an ϵ -accurate solutions × per-round communicated bits . All curves are averaged over 20 trials with the region of standard deviations depicted. Due to the space limit, we only provide results with randoms compressors here. More experimental results can be found in Appendix F.2.

Least squares. Consider a distributed least-square problem (1) with f i ( x ) := 1 2 ∥ A i x -b i ∥ 2 , where A i ∈ R M × d and b i ∈ R M are randomly generated. We set d = 20 , n = 400 , and M = 25 , and generate A i 's by randomly generating a Gaussian matrix in R nM × d , then modify its condition number to 10 4 through the SVD decomposition, and finally distribute its rows to all A i . We use independent random1 compressors for communication compression. The results are depicted in Fig. 2 (left) where we observe ADIANA beats all baselines in terms of the total communication cost. We do not compare with CANITA since it does not have theoretical guarantees for strongly-convex problems.

Logistic regression. Consider a distributed logistic regression problem (1) with f i ( x ) := 1 M ∑ M m =1 ln(1 + exp( -b i,m a ⊤ i,m x )) , where { ( a i,m , b i,m ) } 1 ≤ i ≤ n, 1 ≤ m ≤ M are datapoints in a9a and w8a datasets from LIBSVM [11]. We set n = 400 and choose independent random-⌊ d/ 20 ⌋ compressors for algorithms with compressed communication. The results are as shown in Fig. 2 (middle and right). Again, we observe that ADIANA outperforms all baselines.

Influence of independence in unbiased compression. Wealso construct a delicate quadratic problem to validate the role of independence in unbiased compression to save communication, see Fig. 1. Experimental details are in Appendix F.1. We observe that ADIANA with independent randoms compressors saves more bits than Nesterov's accelerated algorithm while randoms compressors of shared randomness do not. Furthermore, more aggresive compression, i .e., a larger ω , saves more communication costs in total. These observations are consistent with our theories implied in (9).

7 Conclusion

This paper clarifies that unbiased compression alone cannot save communication, but this goal can be achieved by further assuming mutual independence between compressors. We also demonstrate the saving can be up to Θ( √ min { n, κ } ) . Future research can explore when and how much biased compressors can save communication in non-convex and stochastic scenarios.

8 Acknowledgment

This work is supported by NSFC Grant 12301392, 92370121, and 12288101.

References

A. Agarwal and L. Bottou. A lower bound for the optimization of finite sums. In International Conference on Machine Learning , 2015.

S. A. Alghunaim and K. Yuan. A unified and refined convergence analysis for non-convex decentralized learning. arXiv preprint arXiv:2110.09993 , 2021.

D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. QSGD: Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems , 2017.

D. Alistarh, T. Hoefler, M. Johansson, S. Khirirat, N. Konstantinov, and C. Renggli. The convergence of sparsified gradient methods. In Advances in Neural Information Processing Systems , 2018.

Y. Arjevani, Y. Carmon, J. C. Duchi, D. J. Foster, N. Srebro, and B. E. Woodworth. Lower bounds for non-convex stochastic optimization. ArXiv , abs/1912.02365, 2019.

Y. Arjevani and O. Shamir. Communication complexity of distributed convex learning and optimization. In Advances in Neural Information Processing Systems , 2015.

E. Balkanski and Y. Singer. Parallelization does not accelerate convex optimization: Adaptivity lower bounds for non-smooth convex minimization. ArXiv , abs/1808.03880, 2018.

J. Bernstein, Y .-X. Wang, K. Azizzadenesheli, and A. Anandkumar. Signsgd: compressed optimisation for non-convex problems. In International Conference on Machine Learning , 2018.

Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford. Lower bounds for finding stationary points i. Mathematical Programming , 184(1):71-120, 2020.

Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford. Lower bounds for finding stationary points ii: first-order methods. Mathematical Programming , 185:315-355, 2021.

C.-C. Chang and C.-J. Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) , 2(3):1-27, 2011.

J. Chen and A. H. Sayed. Diffusion adaptation strategies for distributed optimization and learning over networks. IEEE Transactions on Signal Processing , 60(8):4289-4305, 2012.

T. Chen, G. Giannakis, T. Sun, and W. Yin. LAG: Lazily aggregated gradient for communication-efficient distributed learning. In Advances in Neural Information Processing Systems , pages 5050-5060, 2018.

Z. Cheng, X. Huang, and K. Yuan. Momentum benefits non-iid federated learning simply and provably. arXiv preprint arXiv:2306.16504 , 2023.

J. Diakonikolas and C. Guzmán. Lower bounds for parallel and randomized convex optimization. In Conference on Learning Theory , 2019.

P. Elias. Universal codeword sets and representations of the integers. IEEE transactions on information theory , 21(2):194-203, 1975.

D. J. Foster, A. Sekhari, O. Shamir, N. Srebro, K. Sridharan, and B. E. Woodworth. The complexity of making the gradient small in stochastic convex optimization. In Conference on Learning Theory , 2019.

V. Gandikota, D. Kane, R. K. Maity, and A. Mazumdar. vqsgd: Vector quantized stochastic gradient descent. In International Conference on Artificial Intelligence and Statistics , pages 2197-2205. PMLR, 2021.

Y. He, X. Huang, Y. Chen, W. Yin, and K. Yuan. Lower bounds and accelerated algorithms in distributed stochastic optimization with communication compression. arXiv preprint arXiv:2305.07612 , 2023.

S. Horvath, C.-Y. Ho, L. Horvath, A. N. Sahu, M. Canini, and P. Richtárik. Natural compression for distributed deep learning. ArXiv , abs/1905.10988, 2019.

X. Huang, Y. Chen, W. Yin, and K. Yuan. Lower bounds and nearly optimal algorithms in distributed learning with communication compression. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

X. Huang, P. Li, and X. Li. Stochastic controlled averaging for federated learning with communication compression. arXiv preprint arXiv:2308.08165 , 2023.

X. Huang, K. Yuan, X. Mao, and W. Yin. Improved analysis and rates for variance reduction under without-replacement sampling orders. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , 2020.

S. P. Karimireddy, Q. Rebjock, S. U. Stich, and M. Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning , 2019.

A. Khaled, O. Sebbouh, N. Loizou, R. M. Gower, and P. Richtárik. Unified analysis of stochastic gradient methods for composite convex and smooth optimization. arXiv preprint arXiv:2006.11573 , 2020.

S. Khirirat, H. R. Feyzmahdavian, and M. Johansson. Distributed learning with compressed gradients. arXiv preprint arXiv:1806.06573 , 2018.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2015.

A. Koloskova, T. Lin, and S. U. Stich. An improved analysis of gradient tracking for decentralized machine learning. In Advances in Neural Information Processing Systems , 2021.

A. Koloskova, N. Loizou, S. Boreiri, M. Jaggi, and S. Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning , pages 5381-5393. PMLR, 2020.

D. Kovalev, E. Shulgin, P. Richtarik, A. V. Rogozin, and A. Gasnikov. Adom: Accelerated decentralized optimization method for time-varying networks. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 5784-5793. PMLR, 18-24 Jul 2021.

Z. Li, D. Kovalev, X. Qian, and P. Richtárik. Acceleration for compressed gradient descent in distributed and federated optimization. arXiv preprint arXiv:2002.11364 , 2020.

Z. Li and P. Richtárik. Canita: Faster rates for distributed convex optimization with communication compression. Advances in Neural Information Processing Systems , 34:13770-13781, 2021.

X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang, and J. Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. In Advances in Neural Information Processing Systems , 2017.

T. Lin, S. P. Karimireddy, S. U. Stich, and M. Jaggi. Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data. In International Conference on Machine Learning , 2021.

Y. Liu, W. Xu, G. Wu, Z. Tian, and Q. Ling. Communication-censored admm for decentralized consensus optimization. IEEE Transactions on Signal Processing , 67(10):2565-2579, 2019.

P. Mayekar and H. Tyagi. Ratq: A universal fixed-length quantizer for stochastic optimization. In International Conference on Artificial Intelligence and Statistics , pages 1399-1409. PMLR, 2020.

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics , 2017.

P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations , 2018.

K. Mishchenko, E. A. Gorbunov, M. Takác, and P. Richtárik. Distributed learning with compressed gradient differences. ArXiv , abs/1901.09269, 2019.

K. Mishchenko, G. Malinovsky, S. Stich, and P. Richtárik. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning , pages 15750-15769. PMLR, 2022.

Y. Nesterov. A method for unconstrained convex minimization problem with the rate of convergence o(1/k2). In Doklady an ussr , volume 29, 1983.

Y. Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2003.

Y. E. Nesterov. A method of solving a convex programming problem with convergence rate o (1 /k 2 ) . In Doklady Akademii Nauk , volume 269, pages 543-547. Russian Academy of Sciences, 1983.

F. Nogueira et al. Bayesian optimization: Open source constrained global optimization tool for python. 2014.

C. Philippenko and A. Dieuleveut. Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees. ArXiv , 2022.

S. Pu and A. Nedi´ c. Distributed stochastic gradient tracking methods. Mathematical Programming , 187:409-457, 2021.

X. Qian, P. Richtárik, and T. Zhang. Error compensated distributed sgd can be accelerated. Advances in Neural Information Processing Systems , 34:30401-30413, 2021.

P. Richtárik, I. Sokolov, and I. Fatkhullin. Ef21: A new, simpler, theoretically better, and practically faster error feedback. ArXiv , abs/2106.05203, 2021.

M. H. Safaryan, E. Shulgin, and P. Richtárik. Uncertainty principle for communication compression in distributed and federated learning and the search for an optimal compressor. Information and Inference: A Journal of the IMA , 2021.

K. Scaman, F. R. Bach, S. Bubeck, Y. T. Lee, and L. Massoulié. Optimal algorithms for smooth and strongly convex distributed optimization in networks. In International Conference on Machine Learning , 2017.

F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In INTERSPEECH , 2014.

B. Song, I. Tsaknakis, C.-Y. Yau, H.-T. Wai, and M. Hong. Distributed optimization for overparameterized problems: Achieving optimal dimension independent communication complexity. In Advances in Neural Information Processing Systems , volume 35, pages 6147-6160, 2022.

S. U. Stich. Local sgd converges fast and communicates little. In International Conference on Learning Representations , 2019.

S. U. Stich, J.-B. Cordonnier, and M. Jaggi. Sparsified sgd with memory. In Advances in Neural Information Processing Systems , 2018.

H. Tang, X. Lian, M. Yan, C. Zhang, and J. Liu. d 2 : Decentralized training over decentralized data. In International Conference on Machine Learning , pages 4848-4856, 2018.

J. Wangni, J. Wang, J. Liu, and T. Zhang. Gradient sparsification for communication-efficient distributed optimization. In Advances in Neural Information Processing Systems , 2018.

W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. H. Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems , 2017.

J. Wu, W. Huang, J. Huang, and T. Zhang. Error compensated quantized sgd and its applications to large-scale distributed optimization. In International Conference on Machine Learning , 2018.

R. Xin, U. A. Khan, and S. Kar. An improved convergence analysis for decentralized online stochastic non-convex optimization. IEEE Transactions on Signal Processing , 2020.

K. Yuan, S. A. Alghunaim, and X. Huang. Removing data heterogeneity influence enhances network topology dependence of decentralized sgd. Journal of Machine Learning Research , 24(280):1-53, 2023.

K. Yuan, S. A. Alghunaim, B. Ying, and A. H. Sayed. On the influence of bias-correction on distributed stochastic optimization. IEEE Transactions on Signal Processing , 2020.

K. Yuan, Y. Chen, X. Huang, Y. Zhang, P. Pan, Y. Xu, and W. Yin. Decentlam: Decentralized momentum sgd for large-batch deep training. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 3009-3019, 2021.

K. Yuan, X. Huang, Y. Chen, X. Zhang, Y. Zhang, and P. Pan. Revisiting optimal convergence rate for smooth and non-convex stochastic decentralized optimization. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

M. D. Zeiler. Adadelta: An adaptive learning rate method. ArXiv , abs/1212.5701, 2012.

A Random sparsification

We illustrate the randoms sparsification here. More examples of unbiased compressors can be found in literature [50].

Example 1 (RANDOMs SPARSIFICATION) . For any x ∈ R d , the randoms sparsification is defined by C ( x ) := d s ( ξ ⊙ x ) where ⊙ denotes the entry-wise product and ξ ∈ { 0 , 1 } d is a uniformly random binary vector with s non-zero entries. This randoms sparsification operator C satisfies Assumption 2 with ω = d/s -1 . When each entry of the input x is represented with r bits, randoms sparsification compressor takes rs bits to transmit s entries and log 2 ( d s ) bits to transmit the indices of s transmitted entries, resulting in a total rd 1+ ω +log 2 ( d s ) bits in each communication round, see [50, Table 1].

B Proof of Proposition 1

We first recall a result proved by [50].

Lemma 2 ([50], Theorem 2) . Let C : R d → R d be any unbiased compressors satisfying 2 and b be the total number of bits needed to encode the compressed vector C ( x ) for any x ∈ R d . If each entry of the input x is represented with r bits, it holds that max { ω 1+ ω , 4 -r } 4 b/d ≥ 1 .

Using Lemma 2, when ω/ (1 + ω ) ≤ 4 -r , i .e., ω ≤ (4 r -1) -1 ≤ 1 / 3 , we have (1 + ω ) = Θ(1) and b ≥ rd = Ω r ( d/ (1 + ω )) , where r is regarded as a constant in Ω r ( · ) . When ω/ (1 + ω ) ≥ 4 -r , we have

where we use the inequality ln(1 + t ) ≥ t/ (1 + t ) with t = ω -1 ≥ 0 .

C Proof of Theorem 2

Following [5, 9] , we denote the k -th coordinate of a vector x ∈ R d by [ x ] k for k = 1 , . . . , d , and let prog( x ) be

̸

Similarly, for a set of multiple points X = { x 1 , x 2 , . . . } , we define prog( X ) := max x ∈X prog( x ) . We call a function f zero-chain if it satisfies

which implies that starting from x 0 = 0 , a single gradient evaluation can only earn at most one more non-zero coordinate for the model parameters.

Let us now illustrate the setup of distributed optimization with communication compression. For any t ≥ 1 , we consider the t -th communication round, which begins with the server broadcasting a vector denoted as u t to all workers. We initialize u 1 as x 0 . Upon receiving the vector u t from the server, each worker performs necessary algorithmic operations, and the round concludes with each worker sending a compressed message back to the server.

We denote v t i as the vector that worker i aims to send in the t -th communication round before compression, and ˆ v t i as the compressed vector that will be received by the server, i .e., ˆ v t i = C i ( v t i ) . While we require communication to be synchronous among workers, we do not impose restrictions on the number of gradient queries made by each worker within a communication round. We use Y t i to represent the set of vectors at which worker i makes gradient queries in the t -th communication round, after receiving u t but before sending ˆ v t i .

Following the above description, we now formally state the linear spanning property in the setting of centralized distributed optimization with communication compression.

Definition 2 (LINEAR-SPANNING ALGORITHMS) . We say a distributed algorithm A is linearspanning if, for any t ≥ 1 , the following conditions hold:

The server can only send a vector in the linear manifold spanned by all the past received messages, sent messages, i .e., u t ∈ span ( { u r } t -1 r =1 ∪ { ˆ v r i : 1 ≤ i ≤ n } t -1 r =1 ) .

Worker i can only query at vectors in the linear manifold spanned by its past received messages, compressed messages, and gradient queries, i .e., Y t i ⊆ span ( { u r } t r =1 ∪ {∇ f i ( y ) : y ∈ Y r i } t -1 r =1 ∪ { ˆ v r i } t -1 r =1 ) .

Worker i can only send a vector in the linear manifold spanned by its past received messages, compressed messages, and local gradient queries, i .e., v t i ∈ span ( { u r } t r =1 ∪ {∇ f i ( y ) : y ∈ Y r i } t r =1 ∪ { ˆ v r i } t -1 r =1 ) .

After t communication rounds, the server can only output a model in the linear manifold spanned by all the past received messages, sent messages, i .e., ˆ x t ∈ span ( { u r } t r =1 ∪ { ˆ v r i : 1 ≤ i ≤ n } t r =1 ) .

In essence, when starting from x 0 = 0 , the above linear-spanning property requires that any expansion of non-zero coordinates in vectors held by worker i ( e .g., Y t i , v t i ) are attributed to its past local gradient updates, local compression, or synchronization with the server. Meanwhile, it also requires that any expansion of non-zero coordinate in vectors held, including the final algorithmic output, in the server is due to the received compressed messages from workers.

Without loss of generality, we assume algorithms to start from x 0 = 0 throughout the proofs. When { f i } n i =1 are further assumed to be zero-chain, following Definition 2, one can easily establish by induction that for any t ≥ 1 ,

Next, we outline the proofs for the lower bounds presented in Theorem 2. For each case, we provide separate proofs for terms in the lower bound by constructing different hard-to-optimize examples, respectively. The construction of these proofs follows four steps:

Constructing a set of zero-chain local functions { f i } n i =1 .

Constructing a set of independent unbiased compressors { C i } n i =1 ⊆ U ind ω . These compressors are delicately designed to impede algorithms from expanding the non-zero coordinates of model parameters.

Establishing a limitation on zero-respecting algorithms that utilize the predefined compressor with t rounds of compressed communication on each worker. This limitation is based on the non-zero coordinates of model parameters.

Translating the above limitation into the lower bound of the complexity measure defined in equation (3).

While the overall proof structure is similar to that of [19], our novel construction of functions and compressors enable us to derive lower bounds for independent compressors. These lower bounds clarify the unique properties and benefits of independent compressors.

We will use the following lemma in the analysis of the third step.

Lemma 3 ([19], Lemma 3) . Given a constant p ∈ [0 , 1] and random variables { B t } ∞ t =0 such that B t ≤ B ( t -1) +1 and P ( B t ≤ B t -1 | { B r } t -1 r =0 ) ≥ 1 -p for any t ≥ 1 , it holds for t ≥ 1 /p , with probability at least 1 -e -1 , that B t ≤ B 0 + ept .

C.1 Strongly-convex case

Below, we present two examples, each of which corresponding to a lower bound LB m for T ϵ . We integrate the two lower bounds together and use the inequality

to accomplish the lower bound for strongly-convex problems in Theorem 2.

Example 1. In this example, we prove the lower bound Ω((1 + ω )(1 + √ κ/n ) ln ( µ ∆ /ϵ )) .

(Step 1.) We assume the variable x ∈ ℓ 2 ≜ { ([ x ] 1 , [ x ] 2 , . . . , ) : ∑ ∞ r =1 [ x ] 2 r < ∞} to be infinitely dimensional and square-summable for simplicity. It is easy to adapt the argument for finitely dimensional variables as long as the dimension is proportionally larger than t . Let M be

then it is easy to see 0 ⪯ M ⪯ 4 I . Let { f i } n i =1 be as follows

where λ ∈ R \{ 0 } is to be specified. It is easy to see that ∑ r ≥ 0 ([ x ] nr + i -[ x ] nr + i +1 ) 2 and [ x ] 2 1 + ∑ r ≥ 0 ([ x ] nr -[ x ] nr +1 ) 2 -2 λ [ x ] 1 are convex and 4 -smooth. Consequently, all f i s are L -smooth and µ -strongly convex. More importantly, it is easy to verify that all f i s defined above are zero-chain functions and satisfy

We further have f ( x ) = 1 n ∑ n i =1 f i ( x ) = µ 2 ∥ x ∥ 2 + L -µ 4 n ( x ⊤ Mx -2 λ [ x ] 1 ) . For the functions defined above, we also establish that

Lemma 4. Let κ ≜ L/µ ≥ 1 , it holds for any x that,

Proof. The minimum x ⋆ of function f satisfies ( L -µ 2 n M + µ ) x ⋆ -λ L -µ 2 e 1 = 0 , which is equivalent to

Note that

is the only root of the equation q 2 -2 κ +2 n -2 κ -1 q +1 = 0 that is smaller than 1 . Then it is straight forward to check x ⋆ = ( [ x ⋆ ] j = λq j ) j ≥ 1 satisfies (12). By the strong convexity of f , x ⋆ is the unique solution. Therefore, we have that

Finally, using the strong convexity of f leads to the conclusion.

Following the proof of Lemma 4, we have

Therefore, for any given ∆ > 0 , letting λ = √ ((1 -q 2 )∆) /q 2 results in ∥ x 0 -x ⋆ ∥ 2 = ∆ . Consequently, our construction ensures { f i } n i =1 ∈ F ∆ L,µ .

(Step 2.) For the construction of ω -unbiased compressors, we consider { C i } n i =1 to be independent random sparsification compressors. Building upon Example 1, we make a slight modification: during a round of communication on any worker, each coordinate is independetly chosen with a probability of (1 + ω ) -1 to be transmitted, and if selected, its value is scaled by (1 + ω ) and then the scaled value is transmitted. Notably, the indices of chosen coordinates are not identical across all workers due to the independence of compressors. It can be easily verified that this construction ensures that { C i } n i =1 ⊆ U ind ω .

(Step 3.) Since the algorithmic output ˆ x t calculated by the server lies in the linear manifold spanned by received messages, we can use (10) to obtain the following expression:

We next bound B t with B 0 := 0 by showing that { B t } ∞ t =0 satisfies Lemma 3 with p = (1 + ω ) -1 .

For any linear-spanning algorithm A , according to (11), the worker i can only attain one additional non-zero coordinate through local gradient-based updates when prog( Y t i ) ≡ i mod n . In other words, upon receiving messages { u r i } t r =1 from the server, we have

Consequently, we have

It then follows from the definition of the constructed C i in Step 2 that max 1 ≤ i ≤ n prog(ˆ v t i ) ≤ max 1 ≤ i ≤ n prog( v t i ) , and therefore we have:

Next, we aim to prove that B t ≤ B t -1 +1 with a probability of at least ω/ (1 + ω ) . For any t ≥ 1 , let i ∈ { 1 , . . . , n } be such that B t -1 ≡ i mod n . Due to the property in equation (11), during the t -th communication round, if prog( Y t i ) = B t -1 , worker i can push the number of non-zero entries forward by 1 , resulting in prog( v t i ) = B t -1 +1 , using local gradient updates. Note that any other worker j cannot achieve this even if prog( Y t j ) = B t -1 due to equation (11).

Therefore, to achieve B t = B t -1 +1 , it is necessary for worker i to transmit a non-zero value at the ( B t -1 +1) -th entry to the server. Otherwise, we have B t ≤ B t -1 . However, since the compressor C i associated with worker i has a probability ω/ (1 + ω ) to zero out the ( B t -1 +1) -th entry in the t -th communication round, we have

In summary, we have shown that B t ≤ B t -1 +1 and P ( B t ≤ B t -1 | { B r } t -1 r =0 ) ≥ ω/ (1 + ω ) .

By applying Lemma 3, we can conclude that for any t ≥ (1 + ω ) -1 , with a probability of at least 1 -e -1 , it holds that B t ≤ et/ (1 + ω ) and hence prog(ˆ x t ) ≤ et/ (1 + ω ) due to (13).

(Step 4.) Using Lemma 4 and that prog(ˆ x t ) ≤ et/ (1 + ω ) with probability at least 1 -e -1 , we obtain

Therefore, to ensure E [ f (ˆ x t )] -min x f ( x ) ≤ ϵ , relation (14) implies the lower bound T ϵ = Ω((1 + ω )(1 + √ κ/n ) ln( µ ∆ /ϵ )) .

Example 2. Considering f 1 = f to be homogeneous and C i = I to be a loss-less compressor for all 1 ≤ i ≤ n , the problem reduces to single-node convex optimization. In this case, the lower bound of Ω( √ κ ln ( µ ∆ /ϵ )) is well-known in the literature, as shown in [44, 43].

With the two lower bounds achieved in Examples 1 and 2, we have

which is the result for the strongly-convex case in Theorem 2.

C.2 Generally-convex case

Below, we present three examples, each of which corresponding to a lower bound LB m for T ϵ . We integrate the three lower bounds together and use the inequality

to accomplish the lower bound for the generally-convex case in Theorem 2.

Example 1. In this example, we prove the lower bound Ω((1 + ω )( L ∆ /ϵ ) 1 / 2 ) .

(Step 1.) We assume variable x ∈ R d , where d can be sufficiently large and will be determined later. Let M denote

it is easy to verify 0 ⪯ M ⪯ 4 I . Similar to example 1 of the strongly-convex case, we consider

where λ ∈ R \{ 0 } is to be specified. It is easy to see that all f i s are L -smooth. We further have f ( x ) = 1 n ∑ n i =1 f i ( x ) = L 4 n ( x ⊤ Mx -2 λ [ x ] 1 ) . The f i functions defined above are also zero-chain functions satisfying (11).

Following [43], it is easy to verify that the optimum of f satisfies

More generally, it holds for any 0 ≤ k ≤ d that

Since ∥ x 0 -x ⋆ ∥ 2 = λ 2 ( d +1) 2 ∑ d k =1 k 2 = λ 2 d (2 d +1) 6( d +1) ≤ λ 2 d 3 , letting λ = √ 3∆ /d , we have { f i } n i =1 ∈ F ∆ L, 0 .

(Step 2.) Same as Step 2 of Example 1 of the strongly-convex case, we consider { C i } n i =1 to be independent random sparsification operators.

(Step 3.) Following the same argument as step 3 of example 1 of the strongly-convex case, we have that for any t ≥ (1 + ω ) -1 , it holds with probability at least 1 -e -1 that prog(ˆ x t ) ≤ et/ (1 + ω ) .

(Step 4.) Thus, combining (15), we have

Letting d = 1 + et/ (1 + ω ) , we further have

Therefore, to ensure E [ f (ˆ x t )] -min x f ( x ) ≤ ϵ , the above inequality implies the lower bound to be T = Ω((1 + ω )( L ∆ / ( nϵ )) 1 2 ) .

Example 2. Considering f 1 = f to be homogeneous and C i = I to be a loss-less compressor for all 1 ≤ i ≤ n . The problem reduces to the single-node convex optimization. The lower bound Ω( √ L ∆ /ϵ ) is well-known in literature, see, e .g., [44, 43].

Example 3. In this example, we prove the lower bound Ω( ω ln ( L ∆ /ϵ )) .

(Step 1.) We consider f 1 = · · · = f n -1 = L ∥ x ∥ 2 / 2 and f n = L ∥ x ∥ 2 / 2 + nλ ⟨ 1 d , x ⟩ where 1 d ∈ R d is the vector with all enries being 1 and λ ∈ R is to be determined. By definition, { f i } n i =1 are µ -strongly-convex and L -smooth and the solution x ⋆ = -λ L 1 d . Letting λ = L √ ∆ / √ n , we have ∥ x ⋆ -x 0 ∥ 2 = ∆ . Thus, the construction ensures { f i } n i =1 ∈ F ∆ L,µ .

(Step 2.) Same as in Example 1, we consider { C i } n i =1 to be independent random sparsification operators.

(Step 3.) By the construction of { f i } n i =1 , we observe that the optimization process relies solely on transmitting the information of 1 d from worker n to the server. Let E t denote the set of entries at which the server has received a non-zero value from worker n in the first t communication rounds. Note that for each entry, due to the construction of { C i } n i =1 , the server has a probability of at least ( ω/ (1 + ω )) t of not receiving a non-zero value at that entry from worker n . Consequently, | ( E t ) c | is lower bounded by the sum of n independent Bernoulli( ω t / (1 + ω ) t ) random variables. Therefore, we have E [ | ( E t ) c | ] ≥ dω t / (1 + ω ) t .

(Step 4.) Given | E t | , due to the linear-spanning property, we have ˆ x t ∈ span { e j : j ∈ E t } where e j is the j -th canonical vector. As a result, we have

Therefore, to ensure E [ f (ˆ x t )] -min x f ( x ) ≤ ϵ , (16) implies the lower bound T ϵ = Ω( ω ln( L ∆ /ϵ )) .

With the three lower bounds achieved in Examples 1, 2, and 3, we have

which is the result for the generally-convex case in Theorem 2.

D Proof of Theorem 3

D.1 Strongly-convex case

We first present several important lemmas, followed by the definition of a Lyapunov function with delicately chosen coefficients for each term. Finally, we prove Theorem 3 by utilizing these lemmas. Throughout the convergence analysis, we use the following notations:

We use E k or E indicate the expectation with respect to the randomness in the k -th iteration or all histortical randomness, respectively.

Lemma 5. If 0 ≤ β ≤ 1 , it holds for ∀ k ≥ 0 that,

Proof. Following the update rules in Algorithm 1, we have

Since x k = θ 1 ,k z k + θ 2 w k +(1 -θ 1 ,k -θ 2 ) y k , we have

Plugging (19) into (18), using

we obtain (17).

Lemma 6. Under Assumption 1, if parameters satisfy θ 1 ,k , θ 2 , 1 -θ 1 ,k -θ 2 ∈ (0 , 1) , η k ∈ ( 0 , 1 2 L ] , γ k = η k 2 θ 1 ,k + η k µ and β = 1 -γµ = 2 θ 1 ,k 2 θ 1 ,k + η k µ , then we have for any iteration k ≥ 0 that

Proof. By Assumption 1 and update rules in Algorithm 1, we have

By L -smoothness and µ -strongly convexity, we have for ∀ u ∈ R d that

and that

thus we obtain for ∀ u ∈ R d ,

Applying Young's inequality to (21) and using η k ≤ 1 / (2 L ) , we reach

Adding (17) in Lemma 5 to ( 2 γ k β θ 1 ,k +2 γ k (1 -β ) ) × (23) + 2 γ k × (22) (where u = x ⋆ ) + 2 γ k βθ 2 θ 1 ,k × (22) (where u = w k ) + 2 γ k β (1 -θ 1 ,k -θ 2 ) θ 1 ,k × (22) (where u = y k ) and using the unbiasedness of g k , we obtain

On top of that, by applying our choice of the parameters, it can be easily verified that 1 -β -µγ k = 0 , γ 2 k -η k γ k β 2 θ 1 ,k = 0 , 1 -β ≤ β 4 θ 1 ,k , which leads to (20).

Lemma 7 ([32], Lemma 3, 4, 5) . Under Assumptions 1, 2, and 3, the iterates of Algorithm 1 satisfy the following inequalities:

Now we define a Lyapunov function Ψ k for k ≥ 1 as

where λ k = γ k β pθ 1 ,k ( θ 1 ,k + θ 2 -p + √ ( p -θ 1 ,k -θ 2 ) 2 +4 pθ 2 ) . Furthermore, it is straightforward to verify that

Now we restate the convergence result in the strongly-convex case in Theorem 3 and prove it using Lemma 6, 7 and the Lyapunov function.

Theorem 4. If µ > 0 and parameters satisfy η k ≡ η = nθ 2 / (120 ωL ) , θ 1 ,k ≡ θ 1 = 1 / (3 √ κ ) , α = p = 1 / (1 + ω ) , γ k ≡ γ = η/ (2 θ 1 + ηµ ) , β = 2 θ 1 / (2 θ 1 + ηµ ) , and θ 2 = 1 / (3 √ n +3 n/ω ) , then the number of communication rounds performed by ADIANA to find an ϵ -accurate solution such that E [ f (ˆ x )] -min x f ( x ) ≤ ϵ is at most O (( ω +(1 + ω/ √ n ) √ κ ) ln( L ∆ /ϵ )) .

Proof. In the strongly-convex case, parameters { γ k } k ≥ 1 and { θ 1 ,k } k ≥ 1 are constants, then so is λ k . Thus, we simply write γ ≜ γ k , θ 1 ≜ θ 1 ,k , and λ ≜ λ k for all k ≥ 1 . Considering (20)+ λ (24)+ 5 γβη 4 θ 1 (25)+ 10 ηω (1+ ω ) γβ nθ 1 (26), we have

By the definition of λ , we have

and

From the choice of η , it is easy to verify that

and further noting 1 -θ 1 -θ 2 ≥ θ 2 ,

Plugging (29), (30), (31), and (32) into (28), we obtain

where Ψ 0 := λ W 0 + 2 γβ θ 1 Y 0 + Z 0 + 10 ηω (1+ ω ) γβ θ 1 n H 0 . Note that since we use initialization y 0 = z 0 = w 0 = h 0 i = h 0 , ∀ 1 ≤ i ≤ n , we have W 0 = Y 0 ≤ ( L ∆) / 2 , Z 0 ≤ ∆ , H 0 ≤ L 2 ∆ , which indicates that

where λ W = λ ≥ 2 γβθ 2 θ 1 p , λ Y = 2 γβ θ 1 , λ Z = 2 L , λ H = 20 ηω (1+ ω ) γβL θ 1 n . These coefficients have the following inequalities:

and

Consequently, the initial value of the Lyapunov function can be bounded as

which together with (33) further implies that

Thus, O (( ω + ( 1 + ω √ n ) √ κ ) ln ( L ∆ ϵ ) ) iterations are sufficient to guarantee an ϵ -solution.

D.2 Generally-convex case

In this subsection, we restate the convergence result in the generally-convex case as in Theorem 3 and prove it using Lemma 6, 7 and the Lyapunov function defined in (27).

Theorem 5. If µ = 0 and parameters satisfy α = 1 / (1 + ω ) , β = 1 , p = θ 2 = 1 / (3(1 + ω )) , θ 1 ,k = 9 / ( k +27(1 + ω )) , γ k = η k / (2 θ 1 ,k ) , and

then the number of communication rounds performed by ADIANA to find an ϵ -accurate solution such that E [ f (ˆ x )] -min x f ( x ) ≤ ϵ is provided by O ((1 + ω/ √ n ) √ L ∆ /ϵ +(1 + ω ) 3 √ L ∆ /ϵ ) .

Proof. Considering (20) + λ k (24) + 5 γ k βη k 4 θ 1 ,k (25) + 10 η k ω (1+ ω ) γ k β nθ 1 ,k (26) and applying the choice of θ 2 , p and α , we have

Similar to the proof of Theorem 4, we can simplify (34) by validating

and then obtain

For ∀ k ≥ 1 , we have θ 1 ,k ≤ θ 1 ,k -1 and thus

Further noting η k η k -1 ≤ 1 + 1 k +27(1+ ω ) , we obtain

Similarly,

and

Combining (35),(36),(37), and (38), we have for ∀ k ≥ 1 that

By applying (39) with k = T -1 , T -2 , · · · , 1 and (35) with k = 0 , we obtain

Note that

thus

thus it suffices to achieve an ϵ -solution with O (( 1 + ω √ n )√ L ∆ ϵ +(1 + ω ) 3 √ L ∆ ϵ ) iterations.

E Correction on CANITA [33]

We observe that when ω ≫ n , the original convergence rate of CANITA [33] contradicts the lower bounds presented in our Theorem 2. This discrepancy may stem from errors in the derivation of equations (35) and (36) in [33], or from the omission of certain conditions such as ω = Ω( n ) . To address this issue, we provide a corrected proof and the corresponding convergence rate. Here we modify the choice of β 0 in ([33], Theorem 2) to 9(1 + b + ω ) 2 / (2(1 + b )) , while keeping all other choices consistent with the original proof, i .e., b = min { ω, √ ω (1 + ω ) 2 /n } , p t ≡ 1 / (1 + b ) , α t ≡ 1 / (1 + ω ) , θ t = 3(1 + b ) / ( t +9(1+ b + ω )) , β = 48 ω (1 + ω )(1 + b +2(1+ ω )) / ( n (1 + b ) 2 ) and

By definition we have

Plugging (40) and ([33],34) into ([33],33), we obtain

Using b = min { ω, √ ω (1 + ω ) 2 /n } , we have

and

thus (41) can be simplified as

Consequently, for ϵ < L ∆ / 2 ( i .e., a precision that the initial point does not satisfy), the communication rounds to achieve precision ϵ is given by O ( ω 3 √ L ∆ 3 √ ϵ + ( 1 + ω 3 / 4 n 1 / 4 + ω √ n ) √ L ∆ √ ϵ ) .

F Experimental details and additional results

This section provides more details of the experiments listed in Sec. 6, as well as a few new experiments to validate our theories.

F.1 Experimental details

This section offers a comprehensive and detailed description of the experiments listed in Sec. 6, including problem formulation, data generation, cost calculation, and algorithm implementation.

Least squares. The local objective function of node i is defined as f i ( x ) := 1 2 ∥ A i x -b i ∥ 2 , where A i ∈ R M × d , b i ∈ R M . Weset d = 20 , M = 25 , and the number of nodes n = 400 . To generate A i 's, we first randomly generate a Gaussian matrix G ∈ R nM × d ; we then apply the SVD decomposition G = U Σ V ⊤ and replace the singular values in Σ by an arithmetic sequence starting from 1 and ending at 100 to get ˜ Σ and the resulted data matrix ˜ G = U ˜ Σ V ⊤ ; we finally allocate the submatrix of ˜ G composed of the (( i -1) M +1) -th row to the ( iM ) -th row to be A i for all 1 ≤ i ≤ n .

Logistic regression. The local objective function of node i is defined as f i ( x ) := 1 M ∑ M m =1 ln(1 + exp( -b i,m a ⊤ i,m x ) , where number of nodes n = 400 , a i,m stands for the feature of the m -th datapoint in the node i 's dataset, and b i,m stands for the corresponding label. In a9a dataset, node i owns the (81( i -1) + 1) -th to the (81 i ) -th datapoint with feature dimension d = 123 . In w8a dataset, node i owns the (120( i -1) + 1) -th to the (120 i ) -th datapoint with feature dimension d = 300 .

Constructed problem. The local objective function of node i is defined as

where [ x ] l denotes the l -th entry of vector x ∈ R d . We set µ = 1 , L = 10 4 , d = 20 and number of nodes n = 400 .

Compressors. We apply various compressors to the algorithms with communication compression through our experiments. In the constructed quadratic problem, we consider ADIANA algorithm with randoms compressors (see Example 1 in Appendix A) in six different settings, i .e., three choices of s ( s = 1 , 2 , 4 ), with two different (shared or independent) randomness settings. In the least squares and logistic regression problems, we apply the independent random-⌊ d/ 20 ⌋ compressor to ADIANA, CANITA and DIANA algorithm. In particular, we use the unscaled version of the independent random-⌊ d/ 20 ⌋ compressor for EF21 to guarantee convergence, where the values of selected entries are transmitted directly to the server without being scaled by d/s times. In Appendix F.2, we further apply independent natural compression [20] and random quantization [3] with s = ⌈ √ d ⌉ in the above algorithms.

0.0

0.2

0.4

0.6

0.8

1.0

1.2

communication bits (×1e6)

10 -8

10 -7

10 -6

10 -5

10 -4

10 -3

10 -2

10 -1

f(x) - f*

Nesterov

ADIANA

DIANA

EF21

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

communication bits (×1e6)

10 -5

10 -4

10 -3

10 -2

10 -1

10 0

f(x) - f*

Nesterov

ADIANA

CANITA

DIANA

EF21

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

communication bits (×1e6)

10 -4

10 -3

10 -2

10 -1

f(x) - f*

Nesterov

ADIANA

CANITA

DIANA

EF21

Total communicated bits. For non-compression algorithms and algorithms with a fixed-length compressor, such as randoms and natural compression, the total communication bits can be calculated using the following formula: total communication bits = number of iterations × communication rounds per iteration × communicated bits per round . Among the algorithms we compare, ADIANA and CANITA communicate twice per iteration, while the other algorithms communicate only once. The communicated bits per round for non-compression algorithms amount to 64 d for d float64 entries. In the case of the randoms compressor, the communicated bits per communication are calculated as 64 s + ⌈ log 2 ( d s ) ⌉ . Similarly, for natural compression, the communicated bits per round are fixed at 12 d , with 1 sign bit and 11 exponential bits allocated for each entry. In the case of adaptive-length random quantization, the communication cost is evaluated using Elias integer encoding [16]. This cost is then averaged among n nodes, providing a more representative estimate.

Algorithm implementation. We implement ADIANA, CANITA, DIANA, EF21 algorithms following the formulation in Algorithm 1, [33], [26], and [49], respectively. We implement Nesterov's accelerated algorithm with the following recursions:

The value of α in ADIANA, CANITA and DIANA are all set to 1 / (1 + ω ) , and we set γ k , β of ADIANA as in Theorem 3. Other parameters are all selected through running Bayesian Optimization [45] for the first 20% iterations with 5 initial points and 20 trials. The exact value of the selected parameters are listed in Appendix F.3. Each curve (except for Nesterov's accelerated algorithm which does not involve randomness) is averaged through 20 trials, with the range of standard deviation depicted.

Computational resource. All experiments are run on an NVIDIA A100 server. Each trial consumes up to 10 minutes of running time.

F.2 Additional experiments

Addtional compressors. In addition to the experiments in Sec. 6, we consider applying different compressors in the algorithms with communication compression. Fig. 3 and Fig. 4 show results of using natural compression and random quantization, respectively. These results are consistent with the results in Sec. 6.

CIFAR-10 dataset. We also consider binominal logistic regression with CIFAR-10 dataset, where labels of each datum are categorized by whether they equal to 3, i .e., the corresponding figures belong to the cat category. The full training set with 50000 images, are devided equally to n = 250 nodes. The compressor choices follow the same strategies as in Appendix F.1, where dimension d = 3072 . Fig. 5 compares convergence results between Nesterov method and ADIANA with different compressors. It can be observed that ADIANA equipped with more aggressive compressors, i .e., those with bigger ω , benefits more from the compression, which is consistent with our theoretical results.

0.0

0.2

0.4

0.6

0.8

1.0

1.2

communication bits (×1e6)

10 -8

10 -7

10 -6

10 -5

10 -4

10 -3

10 -2

10 -1

f(x) - f*

Nesterov

ADIANA

DIANA

EF21

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

communication bits (×1e6)

10 -6

10 -5

10 -4

10 -3

10 -2

10 -1

10 0

f(x) - f*

Nesterov

ADIANA

CANITA

DIANA

EF21

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

communication bits (×1e6)

10 -7

10 -6

10 -5

10 -4

10 -3

10 -2

10 -1

10 0

f(x) - f*

Nesterov

ADIANA

CANITA

DIANA

EF21

0.0

2.0

4.0

6.0

8.0

communication bits (×1e6)

10 -2

10 -1

f(x) - f*

Nesterov

ADIANA w. N.C.

F.3 Parameter values

In this subsection, we list all the parameter values that are selected by applying Bayesian Optimization. Table 2, 3, 4, 5, 6 list the parameters chosen in the least squares problem, logistic regression using a9a dataset, logistic regression using w8a dataset, the constructed problem, and logistic regression using CIFAR-10 dataset, respectively.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the distributed convex optimization problem in the centralized setting. In this setting, the authors provide the new lower bounds on the total communication cost under the assumption that nodes send unbiased independent compressed vectors to a master. They also provide improved analysis of the current state-of-the-art ADIANA method in the general convex and strongly convex regimes.\n\nSTRENGTHS:\nThe paper provides two important things to the distributed optimization community:\n1. While the upper bounds obtained by the ADIANA and CANITA methods are well known, finally, the authors provide lower bounds in the described setting.\n2. The improved analysis of ADIANA is very surprising. I checked the proof. I'm not 100% sure, but the improved analysis seems to be correct. I believe that it is an interesting contribution on its own.\nAll in all, I think that this paper deserves to be published in NeurIPS.\n\nWEAKNESSES:\nHowever, I would like to point out some very important weakness that I want the authors to fix.\n\nThe authors compare the total communication cost of compressed methods with the accelerated (Nesterov's) method. They claim that it is possible to improve the complexity by $\\min [n, \\kappa].$ I agree with this fact but only under the assumption that **the local smoothness constants are equal to the global smoothness constant.** The author ignore the fact (or I missed it in the text, then sorry me for that) that the local smoothness constants can be $n$ times larger than the global one. The authors define the local $L$--smoothness constant as \n$$f_i(y) \\leq f_i(x) + <\\nabla f_i(x), y - x> + \\frac{L}{2} ||x - y||^2 \\quad \\forall i$$ While the non-compressed methods define it as \n$$f(y) \\leq f(x) + <\\nabla f(x), y - x> + \\frac{L}{2} ||x - y||^2.$$ The $L$--constant of $f_i$  can be $n$ times larger than $L$--constant of $f$! I want to ask the authors to clarify it clearly in the paper. Ideally, in the abstract and in the contributions.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the extent to which unbiased compression can reduce the total communication cost of optimizing smooth convex functions. Additionally, they refine the analysis of ADIANA ([25]) to show that the bounds are near-optimal.\n\nSTRENGTHS:\nQuantifying how much unbiased compression helps is an interesting topic. While I believe that nearly anyone (that uses compression) works with unbiased compressors in the distributed case, it is interesting to learn by how much it helps compared with biased compressors.\n\nWEAKNESSES:\nIt is not very surprising to me what without independence, unbiasedness cannot help in general. \nFurther, I do not recall seeing works that do not use independence to get the error-cancellation, *other than works that use correlated compression to drive the error lower than that of i.i.d. compressors* (e.g.,  Correlated quantization for distributed mean estimation and optimization, ICML 2022). \n\nAlso, the writing is hard to follow and seems to be contradictory in places.\nFor example, You write, \"Compared to lower bounds when using unbiased compression without independence [13], our lower bounds demonstrate significant improvements when n and κ are large\". \nHowever, the upper and lower bounds of [13] seem to match in your Table 1, while your lower bound seems *weaker* than that of [13].\n \nIn general, there cannot be two matching sets of upper and lower bounds.\n\nSome of the claims also seem inflated; I don't view $O(\\omega \\epsilon^{-O(1)})$ as being near optimal for a lower bound of $\\Omega(\\omega\\log\\epsilon^{-1})$ -- the dependency on $\\epsilon$ is exponentially worse.\n\nThe improvement in the analysis of ADIANA is only applicable to some parameter range as it shaves an additive factor.\n\n(As a side note, it took me a minute to understand that by **ADIANA (Ours)** you don't actually mean that it's your *algorithm*, but just a refined analysis.)\n\nQUESTIONS:\nCan you please clarify the differences in the asymptotics with [13]? (Also, what is specified in the table seems to be different than in their paper, where they have an additional additive term.)\n\nLIMITATIONS:\nSeems fine", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the communication saving of unbiased compression on distributed optimization. The main contributions are: \n\ni) a communication lower bound of distributed optimization algorithms with (not necessarily independent) unbiased compression;\n\nii) a communication lower bound of distributed optimization algorithms with (independent) unbiased compression;\n\niii) refined analysis of ADIANA to show tightness of these bounds;\n\niv) discussions on the importance of independence to the communication saving.\n\nSTRENGTHS:\ni) Overall, this paper is well written and easy to follow.\n\nii) The investigated problem is important to distributed optimization, and the results are convincing.\n\nWEAKNESSES:\ni) Section 1 claims that quantization and sparsification are modeled as unbiased compressors in some papers. But they can be biased too. Please clarify.\n\nii) The error cancellation explanation of independence to unbiased compression makes sense. But, if $x_i$ at different nodes are quite different, do we still need independence to achieve the communication saving?\n\niii) Table I considers both convex and strongly convex cases. Are there any difficulties in analyzing the non-convex case?\n\niv) Some bounds in Table 1 are summations of two terms, for example, $\\ln(1/\\epsilon)$ and $\\epsilon^{-1/2}$, $\\epsilon^{-1/3}$ and $\\epsilon^{-1/2}$, etc. Is it possible to reduce to one term?\n\nv) Which compressors satisfy Assumption 2 (unbiased compressor) and Assumption 3 (independent compressor)? Please comment below the assumptions.\n\nvi) The communication saving is discussed via comparing the lower bounds. However, in some datasets some algorithms might perform much better than the lower bounds. In this case, the comparisons would yield different conclusions. Please comment on this.\n\nvii) I give a score of 6 at this stage, and would like to change it after the rebuttal period.\n\nQUESTIONS:\ni) Section 1 claims that quantization and sparsification are modeled as unbiased compressors in some papers. But they can be biased too. Please clarify.\n\nii) The error cancellation explanation of independence to unbiased compression makes sense. But, if $x_i$ at different nodes are quite different, do we still need independence to achieve the communication saving?\n\niii) Table I considers both convex and strongly convex cases. Are there any difficulties in analyzing the non-convex case?\n\niv) Some bounds in Table 1 are summations of two terms, for example, $\\ln(1/\\epsilon)$ and $\\epsilon^{-1/2}$, $\\epsilon^{-1/3}$ and $\\epsilon^{-1/2}$, etc. Is it possible to reduce to one term?\n\nv) Which compressors satisfy Assumption 2 (unbiased compressor) and Assumption 3 (independent compressor)? Please comment below the assumptions.\n\nvi) The communication saving is discussed via comparing the lower bounds. However, in some datasets some algorithms might perform much better than the lower bounds. In this case, the comparisons would yield different conclusions. Please comment on this.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the communication cost of unbiased compression algorithms for distributed optimization and proves that unbiased compression alone cannot reduce communication cost since whatever is saved via compression is lost due to an increase in the number of communication rounds needed for convergence. The authors then show that if in addition to being unbiased if the compressors are also independent then the lower bound on the number of communication rounds is reduced by a factor of $\\Theta(\\sqrt(\\min(n, \\kappa)))$ where $n$ is the number of nodes and $\\kappa$ is the condition number of the function being minimized. They then improve the convergence analysis of ADIANA, an existing algorithm for communication compresssion in distributed optimization, and show that the lower bounds can be matched upto a log factor. Simulations on distributed least squares and distributed logistic regression corroborate their analysis and validate their claims.\n\nSTRENGTHS:\n1. The authors identify an important gap in the existing literature on communication compression where emphasis is primarily laid on the unbiasedness of compressors, without also considering the effect of independence. They systematically build and explain their analysis providing theoretical proofs for all their claims. The theoretical analysis coupled with the clear explanation can make this an important and impactful addition to the literature.\n\n2. Considering the total communication cost (per-round cost x number of rounds) makes the analysis more realistic and complete than previous works which just show savings in either per-round cost or on the number of rounds. Moreover, by proposing a general lower bound for all distributed optimization algorithms with unbiased and independent communication compression this work clarifies the target communication cost for all such communication compression algorithms which can make them easier to evaluate and analyze theoretically.\n\nWEAKNESSES:\n1. The writing is a bit heavy on notation and the analysis may be hard to follow for readers unfamiliar with the literature. Especially since their is not clear example or intuition provided in the main paper to indicate why independence may help.\n\n2. It is a bit unclear why ADIANA is chosen for analysis. The transition from deriving a general lower bound to analyzing a specific algorithm feels a bit abrupt. If possible, it would be good to provide some intuition for the kind of properties that would make a communication compression algorithm likely to be close to the lower bound thereby justifying the choice of ADIANA for analysis. This is a minor point however, as I understand that such an intuition may not be available at this time so it is okay if that is the case.\n\nQUESTIONS:\n1. To address the first point under weaknesses above, I would recommend moving the content of Appendix A which provides intuition on why independence helps to the main pape. Algorithm 1 can be moved to the Appendix to make space, since it appears to be just a reproduction of the ADIANA algorithm from [25]. \n\n2. Can you also provide an example of a compression scheme where the compressors are unbiased but not mutually independent and provide some intution for its sub-optimality or show it through any of the experiments in Section 6?\n\n3. Were any changes made to the ADIANA algorithm at all in your analysis or experiments or was the algorithm the same as that in [25] everywhere?\n\nLIMITATIONS:\nThe authors have identified the major limitation of the analysis being limited to convex functions only and have marked that as an issue to be addressed in future work which is fine with me. I have identified a couple of more minor limitations under Weaknesses above and look forward to the authors' responses to those.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}

comments: SUMMARY:
Several communication compression strategies have been proposed in the last few years to improve distributed optimization. The authors consider the tradeoffs between lowering the communication costs per round vs the number of communication rounds and understand how they affect total communication costs. They theoretically formulate this and prove that under some assumptions, unbiased compression might not save total communication. However,  if the compressors used by all workers are further assumed independent, they prove that total communication cost can be decreased. They also prove lower bounds on certain classes of algorithms by refining the analysis for ADIANA.

STRENGTHS:
The results are provably improved. The presentation is clear and easy to follow.

WEAKNESSES:
The lower bounds are proved only for certain classes of algorithms.

QUESTIONS:
It would be better if some experiments from the appendix were moved to main body.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-26

comments: SUMMARY:
The paper explores the conditions of unbiased compression to reduce the communication cost in distributed optimization. Specifically, the paper first presents a theoretical formalization of the total communication cost (TCC) in distributed optimization. With this formulation, the paper proves that unbiased compressor alone cannot necessarily save TCC. Then the paper proves lower bounds on the convergence complexity and shows that independent unbiased compressor provably saves TCC. The paper also improves ADIANA and provides the upper bound of the TCC that can be reduced. Experiments are also provided to support the theoretical findings.

STRENGTHS:
Originality:

The paper focuses on how to save TCC, which is a more practical concern in distribution optimization. Though the paper relies on the findings (independence on unbiased compressors) of previous works, it provides new findings (on the upper bound of the TTC that can be reduced) and applicable modifications of existing methods to obtain an match the bound. Overall the paper seems novel to me. 

Quality:

The claims in the paper are well supported by theoretical analysis.

Clarity:

The paper is well organized and easy to read. The authors give good introduction and clearly list the questions (Q1, Q2) they need to answer. They also provide detailed intuition on why independent unbiased compressor saves communications.

WEAKNESSES:
Overall the paper looks good to me.  One concern is that the current dataset is relatively small. It would be better if the author can provide experiments on a larger real dataset to better prove the effectiveness of the theoretical findings.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27

comments: SUMMARY:
The paper investigated the overall communication costs for federated learning algorithms. A lower bound on the per-round communication cost is presented, then an analysis of an existing algorithm, ADIANA, is provided to obtain an improved upper bound.

STRENGTHS:
The general ideas for the settings and the results are clearly presented. Besides, the presented bounds are clean compared to related works in this field.

WEAKNESSES:
There is an absence of several crucial details, or perhaps typos, which makes it difficult to concretely understand the results. For example, one central concept required in this work is the unbiased compressor, which based on the current manuscript, applies to all real input values. However, if we require it to output a discrete variable with a bounded number of bins, such a compressor may not exist as it is not clear how the unbiased condition could hold for all $x\in\mathbb{R}$.

QUESTIONS:
There are possible places of improvement in terms of clarity. For example:

1. Generally, it would be better if any concept is defined before it is formally used. For example, in the current manuscript, the definition of “linear spanning” is lost in the texts.  

2. Similarly, the assumption of fixed communication load per round could be introduced before the definition of $T_{\epsilon}$ in Sec. 2.4, otherwise, the meaning of minimization in its definition is not clear. 
 
3. For strongly convex results, it might be cleaner to replace $L/\mu$ with $\kappa$. 

4. The notation of $U^{int}_{\omega}$ (or $U_\omega$) can be removed (or moved to appendices) if not used.

LIMITATIONS:
The limitation is appropriately discussed in Section 7.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-29