SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yutong He*
Peking University yutonghe@pku.edu.cn
Xinmeng Huang*
University of Pennsylvania  
xinmengh@sas.upenn.edu
Kun Yuan
Peking University kunyuan@pku.edu.cn
Communication compression is a common technique in distributed optimization that can alleviate communication overhead by transmitting compressed gradients and model parameters. However, compression can introduce information distortion, which slows down convergence and incurs more communication rounds to achieve desired solutions. Given the trade-off between lower per-round communication costs and additional rounds of communication, it is unclear whether communication compression reduces the total communication cost.
This paper explores the conditions under which unbiased compression, a widely used form of compression, can reduce the total communication cost, as well as the extent to which it can do so. To this end, we present the first theoretical formulation for characterizing the total communication cost in distributed optimization with unbiased compressors. We demonstrate that unbiased compression alone does not necessarily save the total communication cost, but this outcome can be achieved if the compressors used by all workers are further assumed independent. We establish lower bounds on the communication rounds required by algorithms using independent unbiased compressors to minimize smooth convex functions and show that these lower bounds are tight by refining the analysis for ADIANA. Our results reveal that using independent unbiased compression can reduce the total communication cost by a factor of up to  $\Theta (\sqrt{\min\{n,\kappa\}})$  when all local smoothness constants are constrained by a common upper bound, where  $n$  is the number of workers and  $\kappa$  is the condition number of the functions being minimized. These theoretical findings are supported by experimental results.
Distributed optimization is a widely used technique in large-scale machine learning, where data is distributed across multiple workers and training is carried out through worker communication. However, dealing with a vast number of data samples and model parameters across workers poses a significant challenge in terms of communication overhead, which ultimately limits the scalability of distributed machine learning systems. To tackle this issue, communication compression strategies [3, 8, 52, 55, 49] have emerged, aiming to reduce overhead by enabling efficient yet imprecise message transmission. Instead of transmitting full-size gradients or models, these strategies exchange compressed gradients or model vectors of much smaller sizes in communication.
There are two common approaches to compression: quantization and sparsification. Quantization [3, 20, 39, 52] maps input vectors from a large, potentially infinite, set to a smaller set of discrete values. In contrast, sparsification [57, 55, 50] drops a certain amount of entries to obtain a sparse vector for communication. In literature [3, 27, 21], these compression techniques are often modeled
*Equal Contribution. Corresponding Author: Kun Yuan. Kun Yuan is also affiliated with National Engineering Laboratory for Big Data Analytics and Applications, and AI for Science Institute, Beijing, China.
as a random operator  $C$ , which satisfies the properties of unbiasedness  $\mathbb{E}[C(x)] = x$  and  $\omega$ -bounded variance  $\mathbb{E}\|C(x) - x\|^2 \leq \omega \|x\|^2$ . Here,  $x$  represents the input vector to be compressed, and  $\omega$  is a fixed parameter that characterizes the degree of information distortion. Besides, part of the compressors can also be modeled as biased yet contractive operators [19, 48, 49].
While communication compression efficiently reduces the volume of vectors sent by workers, it suffers substantial information distortion. As a result, algorithms utilizing communication compression require additional rounds of communication to converge satisfactorily compared to algorithms without compression. This adverse effect of communication compression has been extensively observed both empirically [57, 25, 8] and theoretically [21, 49]. Since the extra rounds of communication needed to compensate for the information loss may outweigh the saving in the per-round communication cost from compression, this naturally motivates the following fundamental question:
Q1. Can unbiased compression alone reduce the total communication cost?
By "unbiased compression alone", we refer to the compression that solely satisfies the assumptions of unbiasedness and  $\omega$ -bounded variance without any additional advanced properties. To address this open question, we formulate the total communication cost as the product of the per-round communication cost and the number of rounds needed to reach an  $\epsilon$ -accurate solution to distributed optimization problems. Using this formulation, we demonstrate the decrease in the per-round communication cost from unbiased compression is completely offset by additional rounds of communication. Therefore, we answer Q1 by showing unbiased compression alone cannot ensure a lower total communication cost, even with an optimal algorithmic design, see Sec. 3 for more details. This negative conclusion drives us to explore the next fundamental open question:
Q2. Under what additional conditions and how much can unbiased compression provably save the total communication cost?
Fortunately, some pioneering works [40, 32, 33] have shed light on this question. They impose independence on unbiased compressors, i.e., the compressed vectors  $\{C_i(x_i)\}_{i=1}^n$  sent by workers are mutually independent regardless of the inputs  $\{x_i\}_{i=1}^n$ . This independence assumption enables an "error cancellation" effect, producing a more accurate compressed vector  $n^{-1}\sum_{i=1}^n C_i(x_i)$  and hence incurring fewer additional rounds of communication compared to dependent compressors. Consequently, the decrease in the per-round communication cost outweighs the extra communication rounds, reducing the total communication cost.
However, it remains unclear how much the total communication cost can be reduced at most by independent unbiased compression and whether we can develop algorithms to achieve this optimal reduction. Addressing this question poses significant challenges as it necessitates a study of the optimal convergence rate for algorithms using independent unbiased compression.
This paper provides the first affirmative answer to this question for convex problems by: (i) establishing lower bounds on convergence rates of distributed algorithms employing independent unbiased compression, and (ii) demonstrating the tightness of these lower bounds by revisiting ADIANA [32] and presenting novel and refined convergence rates nearly attaining the lower bounds. Our results reveal that compared to non-compression algorithms, independent unbiased compression can save the total communication cost by up to  $\Theta (\sqrt{\min\{n,\kappa\}})$ -fold, where  $n$  is the number of workers and  $\kappa \in [1, + \infty ]$  is the function condition number. Figure 1 provides a simple empirical justification. It shows independent compression (ADIANA i.d.) reduces communication costs compared to no compression (Nesterov's Accelerated algorithm), while dependent compression (ADIANA s.d.) does not, which validates our theory.
Figure 1: Performance of ADIANA using random-s sparsification compressors with shared (s.d.) or independent (i.d.) randomness against distributed Nesterov's accelerated algorithm with no compression in communication. Experimental descriptions are in Appendix F.1
Specifically, our contributions are as follows:
Table 1: Lower and upper bounds on the number of communication rounds for distributed algorithms using unbiased compression to achieve an  $\epsilon$ -accurate solution. Notations  $\Delta, n, L, \mu$  ( $\kappa \triangleq L / \mu \geq 1$ ) are defined in Section 2.  $\omega$  is a parameter for unbiased compressors (Assumption 2).  $\tilde{\mathcal{O}}$  and  $\tilde{\Omega}$  hides logarithmic factors independent of  $\epsilon$ . GC and SC denote generally-convex and strongly-convex functions respectively.
Method	GC	SC
Lower Bound	Ω(ωln(1/ε)+(1+ω/√n)√LΔ/√ε)	Ω((ω+(1+ω/√n)√κ) ln(1/ε))
Lower Bound [19]‡	Ω((1+ω)√LΔ/√ε)	Ω((1+ω)√κ ln(1/ε))
CGD [27]◇	O((1+ω)LΔ/ε)	O((1+ω)κ ln(1/ε))
ACGD [32]◇	O((1+ω)√LΔ/√ε)	O((1+ω)√κ ln(1/ε))
DIANA [40]	O((1+ω2+ω/n+ω)LΔ/ε)	O((ω+(1+ω/n)κ) ln(1/ε))
EF21 [49]‡	O((1+ω)LΔ/ε)	O((1+ω)κ ln(1/ε))
ADIANA [32]	-	O((ω+(1+ω3/4/n1/4+ω/√n)√κ) ln(1/ε))
CANITA [33]‡	O(ω3√LΔ/√3/ε+(1+ω3/4/n1/4+ω/√n)√LΔ/√ε)	-
NEOLITHIC [19]‡	O((1+ω)√LΔ/√ε)	O((1+ω)√κ ln(1/ε))
ADIANA (Thm. 3)	O(ω3√LΔ/√3/ε+(1+ω/√n)√LΔ/√ε)	O((ω+(1+ω/√n)√κ) ln(1/ε))
Results obtained in the single-worker setting and cannot be extended to the distributed setting.
$\ddagger$  The rate is obtained by correcting mistakes in the derivations of [33]. See details in Appendix E.
Results hold without assuming independence across compressors.
- We present a theoretical formalization of the total communication cost in distributed optimization with unbiased compression. With this formulation, we demonstrate that unbiased compression alone is insufficient to save the total communication cost, even with an optimal algorithmic design. This is because any reduction in the per-round communication cost is fully offset by the additional rounds of communication required due to the presence of compression errors.
- We prove lower bounds on the convergence complexity of distributed algorithms using independent unbiased compression to minimize smooth convex functions. Compared to lower bounds when using unbiased compression without independence [19], our lower bounds demonstrate significant improvements when  $n$  and  $\kappa$  are large, see the first two lines in Table 1. This improvement highlights the importance of independence in unbiased compression.
- We revisit ADIANA [32] by deriving an improved rate for strongly-convex functions and proving a novel convergence result for generally-convex functions. Our rates nearly match the lower bounds, suggesting their tightness and optimality. Our optimal complexities reveal that, compared to non-compression algorithms, independent unbiased compression can decrease total communication costs by up to  $\mathcal{O}(\sqrt{\min\{n,\kappa\}})$ -fold when all local smoothness constants are constrained by a common upper bound.
- We support our theoretical findings with experiments on both synthetic data and real datasets.
We present the lower bounds, upper bounds, and complexities of state-of-the-art distributed algorithms using independent unbiased compressors in Table 1. With our new and refined analysis, ADIANA nearly matches the lower bounds for both strongly-convex and generally-convex functions.
Communication compression. Two main approaches to compression are extensively explored in literature: quantization and sparsification. Quantization coarsely encodes input vectors into fewer discrete values, e.g., from 32-bit to 8-bit integers [37, 18]. Schemes like Sign-SGD [52, 8] use 1 bit per entry, introducing unbiased random information distortion. Other variants such as Q-SGD [3], TurnGrad [58], and natural compression [20] quantize each entry with more effective bits. In contrast, sparsification either randomly zeros out entries to yield sparse vectors [57], or transmits only the largest model/gradient entries [55].
Error compensation. Recent works [52, 59, 55, 4, 49] propose error compensation or feedback to relieve the effects of compression errors. These techniques propagate information loss backward during compression, thus preserving more useful information. Reference [52] uses error compensation for 1-bit quantization, while the work [59] proposes error-compensated quantization for quadratic problems. Error compensation also reduces sparsification-induced errors [55] and is studied for convergence in non-convex scenarios [4]. Recently, the work [49] introduces EF21, an error feedback scheme that compresses only local gradient increments with improved theoretical guarantees.
Lower bounds. Lower bounds in optimization set a limit for the performance of a single or a class of algorithms. Prior works have established numerous lower bounds for optimization algorithms [1, 15, 6, 43, 7, 21, 64, 17, 44]. In the field of distributed optimization with communication compression, reference [46] provides an algorithm-specific lower bound for strongly-convex functions, while the work [53] establishes the bit-wise complexities for PL-type problems, which reflect the influence of the number of agents  $n$  and dimension  $d$ , but not the condition number  $\kappa$  and the compression  $\omega$ . In particular, [21] characterizes the optimal convergence rate for all first-order and linear-spanning algorithms, in the stochastic non-convex case, which is later extended by [19] to convex cases.
Accelerated algorithms with communication compression. There is a scarcity of academic research on compression algorithms incorporating acceleration, as evident in a limited number of studies [32, 33, 48]. References [32, 33] develop accelerated algorithms with compression in the strongly-convex and generally-convex cases, respectively. For distributed finite-sum problems, accelerated algorithms with compression can further leverage variance-reduction techniques to expedite convergence [48].
Other communication-efficient strategies. Other than communication compression studied in this paper, there are a few different techniques to mitigate the communication overhead in distributed systems, including decentralized communication and lazy communication. Notable examples of decentralized algorithms encompass decentralized SGD [12, 34, 30, 64], D2/Exact-Diffusion [56, 62, 61], gradient tracking [47, 60, 29, 2], and their momentum variants [35, 63]. Lazy communication allows each worker to either perform multiple local updates as opposed to a single communication round [38, 54, 41, 24, 14, 22], or by adaptively skipping communication [13, 36].
This section introduces the problem formulation and assumptions used throughout the paper. We consider the following distributed stochastic optimization problem
where the global objective function  $f(x)$  is decomposed into  $n$  local objective functions  $\{f_i(x)\}_{i=1}^n$ , and each local  $f_i(x)$  is maintained by node  $i$ . Next, we introduce the setup and assumptions.
2.1 Function class
We let  $\mathcal{F}_{L,\mu}^{\Delta}$ $(0\leq \mu \leq L)$  denote the class of convex and smooth functions satisfying Assumption 1. We define  $\kappa \triangleq L / \mu \in [1, + \infty ]$  as the condition number of the functions to be optimized. When  $\mu >0$ $\mathcal{F}_{L,\mu}^{\Delta}$  represents strongly-convex functions. Conversely, when  $\mu = 0$ $\mathcal{F}_{L,\mu}^{\Delta}$  represents generally-convex functions with  $\kappa = \infty$
Assumption 1 (CONVEX AND SMOOTH FUNCTION). We assume each  $f_{i}(x)$  is  $L$ -smooth and  $\mu$ -strongly convex, i.e., there exists constants  $L \geq \mu \geq 0$  such that
$$
\frac {\mu}{2} \| y - x \| ^ {2} \leq f _ {i} (y) - f _ {i} (x) - \langle \nabla f _ {i} (x), y - x \rangle \leq \frac {L}{2} \| y - x \| ^ {2}
$$
for any  $x, y \in \mathbb{R}^d$  and  $1 \leq i \leq n$ . We further assume  $\| x^0 - x^\star \|^2 \leq \Delta$  where  $x^\star$  is one of the global minimizers of  $f(x) = \frac{1}{n} \sum_{i=1}^{n} f_i(x)$ .
2.2 Compressor class
Each worker  $i \in \{1, \dots, n\}$  is equipped with a potentially random compressor  $C_i: \mathbb{R}^d \to \mathbb{R}^d$ . We let  $\mathcal{U}_{\omega}$  denote the set of all  $\omega$ -unbiased compressors satisfying Assumption 2, and  $\mathcal{U}_{\omega}^{\mathrm{ind}}$  denote the set of all independent  $\omega$ -unbiased compressors satisfying both Assumption 2 and Assumption 3.
Assumption 2 (UNBIASED COMPRESSOR). We assume all compressors  $\{C_i\}_{i=1}^n$  satisfy
$$
\mathbb {E} \left[ C _ {i} (x) \right] = x, \quad \mathbb {E} \left[ \| C _ {i} (x) - x \| ^ {2} \right] \leq \omega \| x \| ^ {2}, \quad \forall x \in \mathbb {R} ^ {d} \tag {2}
$$
for constant  $\omega \geq 0$  and any input  $x\in \mathbb{R}^d$ , where the expectation is taken over the randomness of the compression operator  $C_i$ .
Assumption 3 (INDEPENDENT COMPRESSOR). We assume all compressors  $\{C_i\}_{i=1}^n$  are mutually independent, i.e., outputs  $\{C_i(x_i)\}_{i=1}^n$  are mutually independent random variables for any  $\{x_i\}_{i=1}^n$ .
2.3 Algorithm class
Similar to [19], we consider centralized and synchronous algorithms in which first, every worker is allowed to communicate only directly with a central server but not between one another; second, all iterations/communications are synchronized, meaning that all workers start each of their iterations simultaneously. We further require algorithms to satisfy the so-called "linear-spanning" property, which appears in [9, 10, 21, 19] (see formal definition in Appendix C). Intuitively, this property requires each local model  $x_{i}^{k}$  to lie in the linear manifold spanned by the local gradients and the received messages at worker  $i$ . The linear-spanning property is satisfied by all algorithms in Table 1 as well as most first-order methods [42, 28, 23, 65].
Formally, this paper considers a class of algorithms specified by Definition 1.
Definition 1 (ALGORITHM CLASS). Given compressors  $\{C_i\}_{i=1}^n$ , we let  $\mathcal{A}_{\{C_i\}_{i=1}^n}$  denote the set of all centralized, synchronous, linear-spanning algorithms admitting compression in which compressor  $C_i$ ,  $\forall 1 \leq i \leq n$ , is applied for the messages sent by worker  $i$  to the server.
For any algorithm  $A \in \mathcal{A}_{\{C_i\}_{i=1}^n}$ , we define  $\hat{x}^k$  and  $x_i^k$  as the output of the server and worker  $i$  respectively, after  $k$  communication rounds.
2.4 Convergence complexity
With all the interested classes introduced above, we are ready to define our complexity metric for convergence analysis. Given a set of local functions  $\{f_i\}_{i=1}^n \in \mathcal{F}_{L,\mu}^\Delta$ , a set of compressors  $\{C_i\}_{i=1}^n \in \mathcal{C}$  ( $\mathcal{C} = \mathcal{U}_{\omega}^{\mathrm{ind}}$  or  $\mathcal{U}_{\omega}$ ), and an algorithm  $A \in \mathcal{A}_{\{C_i\}_{i=1}^n}$ , we let  $\hat{x}_A^t$  denote the output of algorithm  $A$  after  $t$  communication rounds. The convergence complexity of  $A$  solving  $f(x) = \frac{1}{n} \sum_{i=1}^n f_i(x)$  under  $\{(f_i, C_i)\}_{i=1}^n$  is defined as
$$
T _ {\epsilon} \left(A, \left\{\left(f _ {i}, C _ {i}\right) \right\} _ {i = 1} ^ {n}\right) = \min  \left\{t \in \mathbb {N}: \mathbb {E} \left[ f \left(\hat {x} _ {A} ^ {t}\right) \right] - \min  _ {x} f (x) \leq \epsilon \right\}. \tag {3}
$$
This measure corresponds to the number of communication rounds required by algorithm  $A$  to achieve an  $\epsilon$ -accurate optimum of  $f(x)$  in expectation.
Remark 1. The measure in (3) is commonly referred to as the communication complexity in literature [51, 21, 31, 32]. However, we refer to it as the convergence complexity here to avoid potential confusion with the notion of "communication complexity" and "total communication cost". This complexity metric has been traditionally used to compare communication rounds used by distributed algorithms [32, 33]. However, it cannot capture the total communication costs of multiple algorithms with different per-round communication costs, e.g., algorithms with or without communication compression. Therefore, it is unable to address the motivating questions Q1 and Q2.
Remark 2. The definition of  $T_{\epsilon}$  can be independent of the per-round communication cost, which is specified only through the degree of compression  $\omega$  (i.e., choice of compressor class). However, to be precise, we may further assume these compressors are non-adaptive with the same fixed per-round communication cost. Namely, the compressors output compressed vectors that can be represented by a fixed and common number of bits. Notably, such hypothesis of non-adaptive cost is widely adopted for practical comparison of communication costs and is valid when input  $x$  is bounded or can be encoded with finite bits [3, 59, 20, 22].
${}^{2}$  Compression is typically employed for input  $x$  that is bounded [3, 59] or encoded with finite bits (e.g., float64 numbers) [20]. In these practical scenarios,  $\omega$  -unbiased compression can be employed with finite bits. For instance, random-  $s$  sparsification for  $r$  -bit  $d$  -dimensional vectors costs (nearly)  ${rs} = {rd}/\left( {1 + \omega }\right)$  bits per communication where  $\omega  = d/s - 1$  .
3 Total communication cost
3.1 Formulation of total communication cost
This section introduces the concept of Total Communication Cost (TCC). TCC can be calculated at both the level of an individual worker and of the overall distributed machine learning system comprising all  $n$  workers. In a centralized and synchronized algorithm where each worker communicates compressed vectors of the same dimension, the TCC of the entire system is directly proportional to the TCC of a single worker. Therefore, it is sufficient to use the TCC of a single worker as the metric for comparing different algorithms. In this paper, we let TCC denote the total communication cost incurred by each worker in achieving a desired solution when no ambiguity is present.
Let each worker to be equipped with a non-adaptive compressor with the same fixed per-round communication cost, i.e., the compressor outputs compressed vectors of the same length (size), the TCC of an algorithm  $A$  to solve problem (1) using a set of  $\omega$ -unbiased compressors  $\{C_i\}_{i=1}^n$  in achieving an  $\epsilon$ -accurate optimum can be characterized as
$$
\operatorname {T C C} _ {\epsilon} \left(A, \left\{\left(f _ {i}, C _ {i}\right) \right\} _ {i = 1} ^ {n}\right) := \text {p e r - r o u n d} \cos \left(\left\{C _ {i} \right\} _ {i = 1} ^ {n}\right) \times T _ {\epsilon} \left(A, \left\{\left(f _ {i}, C _ {i}\right) \right\} _ {i = 1} ^ {n}\right). \tag {4}
$$
3.2 A tight lower bound for per-round cost
The per-round communication cost incurred by  $\{C_i\}_{i=1}^n$  in (4) will vary with different  $\omega$  values. Typically, compressors that induce less information distortion, i.e., associated with a smaller  $\omega$ , incur higher per-round costs. To illustrate this, we consider random- $s$  sparsification compressors, whose per-round cost corresponds to the transmission of  $s$  entries, which depends on parameter  $\omega$  through  $s = d / (1 + \omega)$  (see Example 1 in Appendix A). Specifically, if each entry of the input  $x$  is numerically represented with  $r$  bits, then the random- $s$  sparsification incurs a per-round cost of  $rd / (1 + \omega)$  bits up to a logarithm factor.
The following proposition, motivated by the inspiring work [50], establishes a lower bound of TCC when using any compressor satisfying Assumption 2.
Proposition 1. Let  $x \in \mathbb{R}^d$  be the input to a compressor  $C$  and  $b$  be the number of bits needed to compress  $x$ . Suppose each entry of input  $x$  is numerically represented with  $r$  bits, i.e., errors smaller than  $2^{-r}$  are ignored. Then for any compressor  $C$  satisfying Assumption 2, the per-round communication cost of  $C(x)$  is lower bounded by  $b = \Omega_r(d / (1 + \omega))$  where  $r$  is viewed as an absolute number in  $\Omega_r(\cdot)$  (See the proof in Appendix B).
Proposition 1 presents a lower bound on the per-round cost of an arbitrary compressor satisfying Assumption 2. This lower bound is tight since the random- $s$  compressor discussed above can achieve this lower bound up to a logarithm factor. Since  $d$  only relates to the problem instance itself and  $r$  is often a constant absolute number in practice, e.g.,  $r = 32$  or 64, both of which are independent of the choices of compressors and algorithm designs, they can be omitted from the lower bound order. As a result, the TCC in (4) can be lower bounded by
$$
\mathrm {T C C} _ {\epsilon} = \Omega \left(\left(1 + \omega\right) ^ {- 1}\right) \times T _ {\epsilon} (A, \left\{\left(f _ {i}, C _ {i}\right) \right\} _ {i = 1} ^ {n}). \tag {5}
$$