SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Notably, when no compression is employed (i.e.,  $\omega = 0$ ),  $\mathrm{TCC}_{\epsilon} = \Omega(1) \times T_{\epsilon}(A, \{(f_i, C_i)\}_{i=1}^n)$  is consistent with the convergence complexity.
4 Unbiased compressor alone cannot save total communication cost
With formulation (5), given the number of communication rounds  $T_{\epsilon}$ , the total communication cost can be readily characterized. A recent pioneer work [19] characterizes a tight lower bound for  $T_{\epsilon}(A, \{(f_i, C_i)\}_{i=1}^n)$  when each  $C_i$  satisfies Assumption 2.
Lemma 1 ([19], Theorem 1, Informal). Relying on unbiased compressibility alone, i.e.,  $\{C_i\}_{i=1}^n \in \mathcal{U}_\omega$ , without leveraging additional property of compressors such as mutual independence, the fewest rounds of communication needed by algorithms with compressed communication to achieve an  $\epsilon$ -accurate solution to distributed strongly-convex and generally-convex optimization problems are lower bounded by  $T_\epsilon = \Omega((1 + \omega)\sqrt{\kappa}\ln(\mu\Delta/\epsilon))$  and  $T_\epsilon = \Omega((1 + \omega)\sqrt{L\Delta/\epsilon})$ , respectively.
Substituting Lemma 1 into our TCC lower bound in (5), we obtain  $\mathrm{TCC}_{\epsilon} = \tilde{\Omega} (\sqrt{\kappa}\ln (1 / \epsilon))$  or  $\Omega (\sqrt{L\Delta / \epsilon})$  in the strongly-convex or generally-convex case, respectively, by relying solely on
unbiased compression. These results do not depend on the compression parameter  $\omega$ , indicating that the lower per-round cost is fully compensated by the additional rounds of communication incurred by compressor errors. Notably, these lower bounds are of the same order as optimal algorithms without compression such as Nesterov's accelerated gradient descent [43, 44], leading to the conclusion:
Theorem 1. When solving convex optimization problems following Assumption 1, any algorithm  $A \in \mathcal{A}_{\{C_i\}_{i=1}^n}$  that relies solely on unbiased compression satisfying Assumption 2 cannot reduce the total communication cost compared to not using compression. The best achievable total communication cost with unbiased compression alone is of the same order as without compression.
Theorem 1 presents a negative finding that unbiased compression alone is insufficient to reduce the total communication cost, even with an optimal algorithmic design. Meanwhile, it also implies that to develop algorithms that provably reduce the total communication cost, one must leverage compressor properties beyond  $\omega$ -unbiasedness as defined in (2). Fortunately, mutual independence is one such property which we discuss in depth in later sections.
5 Independent unbiased compressor provably saves communication
5.1 An intuition on why independence can help
A series of works [40, 32, 33] have shown theoretical improvements in the total communication cost by imposing independence across compressors, i.e.,  $\{C_i\}_{i=1}^n \in \mathcal{U}_\omega^{\mathrm{ind}}$ . The intuition behind the role of independence among worker compressors can be illustrated by a simple example where workers intend to transmit the same vector  $x$  to the server. Each worker  $i$  sends a compressed message  $C_i(x)$  that adheres to Assumption 2. Consequently, the aggregated vector  $n^{-1} \sum_{i=1}^n C_i(x)$  is an unbiased estimate of  $x$  with variance
$$
\mathbb {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} C _ {i} (x) - x \right\| ^ {2} \right] = \frac {1}{n ^ {2}} \left(\sum_ {i = 1} ^ {n} \mathbb {E} [ \| C _ {i} (x) - x \| ^ {2} ] + \sum_ {i \neq j} \mathbb {E} [ \langle C _ {i} (x) - x, C _ {j} (x) - x \rangle ]\right) \tag {6}
$$
If the compressed vectors  $\{C_i(x)\}_{i = 1}^n$  are further assumed to be independent, i.e.,  $\{C_i\}_{i = 1}^n\in \mathcal{U}_\omega^{\mathrm{ind}}$ , then the cancellation of cross error terms leads to the following equation:
$$
\mathbb {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} C _ {i} \left(x _ {i}\right) - x \right\| ^ {2} \right] = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} [ \| C _ {i} (x) - x \| ^ {2} ] \leq \frac {\omega}{n} \| x \| ^ {2}. \tag {7}
$$
We observe that the mutual independence among unbiased compressors leads to a decreased variance, which corresponds to the information distortion, of the aggregated message. Remarkably, this reduction is achieved by a factor of  $n$  compared to the transmission of a single compressor. Therefore, the independence among the compressors plays a pivotal role in enhancing the accuracy of the aggregated vector, consequently reducing the number of required communication rounds.
On the contrary, in cases where independence is not assumed and no other properties of compressors can be leveraged, the use of Cauchy's inequality only allows us to bound variance (6) as follows:
$$
\mathbb {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} C _ {i} (x) - x \right\| ^ {2} \right] \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} [ \| C _ {i} (x) - x \| ^ {2} ] \leq \omega \| x \| ^ {2}. \tag {8}
$$
It is important to note that the upper bound  $\omega \| x\| ^2$  can only be attained when the compressors  $\{C_i\}_{i = 1}^n$  are identical, indicating that this bound cannot be generally improved further. By comparing (7) and (8), we can observe that the variance of the aggregated vector achieved through unbiased compression with independence can be  $n$  times smaller than the variance achieved without independence.
5.2 Convergence lower bounds with independent unbiased compressors
While mutual independence can boost the unbiased worker compressors, it remains unclear how much the total communication cost can be reduced at most by independent unbiased compression
3The theoretical results can vary from practical observations due to the particularities of real datasets with which compressed algorithms can enjoy faster convergence, compared to the minimax optimal rates (e.g., ours and [19]) justified without resorting any additional condition.
and how to develop algorithms to achieve this optimal reduction. The following subsections aim to address these open questions.
Following the formulation in (5), to establish the best achievable total communication cost using independent unbiased compression, we shall study tight lower bounds on the number of communication rounds  $T_{\epsilon}$  to achieve an  $\epsilon$ -accurate solution, which is characterized by the following theorem.
Theorem 2. For any  $L \geq \mu \geq 0$ ,  $n \geq 2$ , the following results hold. See the proof in Appendix C.
- Strongly-convex: For any  $\Delta > 0$ , there exists a constant  $c_{\kappa}$  only depends on  $\kappa \triangleq L / \mu$ , a set of local loss functions  $\{f_i\}_{i=1}^n \in \mathcal{F}_{L, \mu > 0}^\Delta$ , independent unbiased compressors  $\{C_i\}_{i=1}^n \in \mathcal{U}_\omega^{\mathrm{ind}}$ , such that the output  $\hat{x}$  of any  $A \in \mathcal{A}_{\{C_i\}_{i=1}^n}$  starting from  $x^0$  requires
$$
T _ {\epsilon} (A, \{(f _ {i}, C _ {i}) \} _ {i = 1} ^ {n}) = \Omega \left(\left(\omega + \left(1 + \frac {\omega}{\sqrt {n}}\right) \sqrt {\kappa}\right) \ln \left(\frac {\mu \Delta}{\epsilon}\right)\right)
$$
rounds of communication to reach  $\mathbb{E}[f(\hat{x})] - \min_x f(x) \leq \epsilon$  for any  $0 < \epsilon \leq c_\kappa \mu \Delta$ .
- Generally-convex: For any  $\Delta > 0$ , there exists a constant  $c = \Theta(1)$ , a set of local loss functions  $\{f_i\}_{i=1}^n \in \mathcal{F}_{L,0}^\Delta$ , independent unbiased compressors  $\{C_i\}_{i=1}^n \in \mathcal{U}_\omega^{\mathrm{ind}}$ , such that the output  $\hat{x}$  of any  $A \in \mathcal{A}_{\{C_i\}_{i=1}^n}$  starting from  $x^0$  requires at least
$$
T _ {\epsilon} (A, \{(f _ {i}, C _ {i}) \} _ {i = 1} ^ {n}) = \Omega \left(\omega \ln \left(\frac {L \Delta}{\epsilon}\right) + \left(1 + \frac {\omega}{\sqrt {n}}\right) \left(\frac {L \Delta}{\epsilon}\right) ^ {\frac {1}{2}}\right)
$$
rounds of communication to reach  $\mathbb{E}[f(\hat{x})] - \min_x f(x) \leq \epsilon$  for any  $0 < \epsilon \leq cL\Delta$ .
Consistency with prior works. The lower bounds established in Theorem 2 are consistent with the best-known lower bounds in previous literature. When  $\omega = 0$ , our result reduces to the lower bound for distributed first-order algorithms established by Y. Nesterov in [43]. When  $n = 1$ , our result reduces to the lower bound established in [19] for the single-node case.
Independence improves lower bounds. A recent work [19] establishes lower bounds for unbiased compression without the independence assumption, listed in the second row of Table 1. Compared to these results, our lower bound in Theorem 2 replaces  $\omega$  with  $\omega / \sqrt{n}$ , showing a reduction in order. This reduction highlights the role of independence in unbiased compression. To better illustrate the reduction, we take the strongly-convex case as an example. The ratio of the number of communication rounds  $T_{\epsilon}$  under unbiased compression with independence to the one without independence is:
$$
\frac {\omega + (1 + \omega / \sqrt {n}) \sqrt {\kappa}}{(1 + \omega) \sqrt {\kappa}} = \frac {1}{1 + \omega} + \frac {\omega}{1 + \omega} \left(\frac {1}{\sqrt {n}} + \frac {1}{\sqrt {\kappa}}\right) = \Theta \left(\frac {1}{\min \{1 + \omega , \sqrt {n} , \sqrt {\kappa} \}}\right). \quad (9)
$$
Clearly, using independent unbiased compression can allow algorithms to converge faster, by up to a factor of  $\Theta (\sqrt{\min\{n,\kappa\}})$  (attained at  $\omega \gtrsim \sqrt{\min\{n,\kappa\}}$ ), in terms of the number of communication rounds, compared to the best algorithms with unbiased compressors but without independence.
Total communication cost. Substituting Theorem 2 into the TCC formulation in (5), we can obtain the TCC of algorithms using independent unbiased compression. Comparing this with algorithms without compression, such as Nesterov's accelerated algorithm, and using the relations in (9), we can demonstrate that independent unbiased compression can reduce the total communication cost. Such reduction can be up to  $\Theta(\sqrt{\min\{n,\kappa\}})$  by using compressors with  $\omega \gtrsim \sqrt{\min\{n,\kappa\}}$ , e.g., random- $s$  sparsification with  $s \lesssim d / \sqrt{\min\{n,\kappa\}}$ .
5.3 ADIANA: a unified optimal algorithm
By comparing existing algorithms using independent unbiased compression, such as DIANA, ADIANA, and CANITA, to our established lower bounds in Table 1, it becomes clear that there is a noticeable gap between their convergence complexities and our established lower bounds. This gap could indicate that these algorithms are suboptimal, but it could also mean that our lower bounds are loose. As a result, our claim that using independent unbiased compression reduces the total communication cost by up to  $\Theta (\sqrt{\min\{n,\kappa\}})$  times is not well-grounded yet. In this section, we address this issue by revisiting ADIANA [32] (Algorithm 1) and providing novel and refined convergence results in both strongly- and generally-convex cases.
Algorithm 1: ADIANA
Input: Scalars  $\{\theta_{1,k}\}_{k = 0}^{T - 1},\theta_2,\alpha ,\beta ,\{\gamma_k\}_{k = 0}^{T - 1},\{\eta_k\}_{k = 0}^{T - 1},p$
Initialize  $w^0 = x^0 = \tilde{y}^0 = z^0 = h^0 = h_i^0, \forall 1 \leq i \leq n$ .
for  $k = 0,\dots ,T - 1$  do
On server:
Update  $x\colon x^{k} = \theta_{1,k}z^{k} + \theta_{2}w^{k} + (1 - \theta_{1,k} - \theta_{2})y^{k}$  and broadcast to all workers;
On all workers in parallel:
- Compress the increment of local gradient  $m_i^k = C_i(\nabla f_i(x^k) - h_i^k)$  and send to the server;
- Compress the increment of local gradient  $c_i^k = C_i(\nabla f_i(w^k) - h_i^k)$  and send to the server;
- Update local shift  $h_i^{k+1} = h_i^k + \alpha c_i^k$ ;
Aggregate received compressed message  $g^{k} = h^{k} + \frac{1}{n}\sum_{i = 1}^{n}m_{i}^{k};$
Update shift  $h^{k + 1} = h^k + \alpha \frac{1}{n} \sum_{i=1}^{n} c_i^k$ ;
Apply gradient descent  $y^{k + 1} = x^k - \eta_k g^k$ ;
Update  $z\colon z^{k + 1} = \beta z^k +(1 - \beta)x^k +\frac{\gamma_k}{\eta_k} (y^{k + 1} - x^k);$
Update  $w$ :  $w^{k + 1} = \left\{ \begin{array}{ll} y^k, & \text{with probability } p, \\ w^k, & \text{with probability } 1 - p; \end{array} \right.$
Output:  $\hat{x} = w^{T}$  if  $f(w^{T})\leq f(y^{T})$  else  $\hat{x} = y^T$
In the strongly-convex case, we refine the analysis of [32] by: (i) adopting new parameter choices where the initial scalar  $\theta_{2}$  is delicately chosen instead of being fixed as  $\theta_{2} = 1 / 2$  in [32], (ii) balancing different terms in the construction of the Lyapunov function. While we do not modify the algorithm design, our technical ingredients are necessary to obtain an improved convergence rate. In the generally-convex case, we provide the first convergence result for ADIANA, which is missing in literature to our knowledge. In both strongly- and generally-convex cases, our convergence results (nearly) match the lower bounds in Theorem 2. This verifies the tightness of our lower bounds for both the convergence complexity and the total communication cost. In particular, our results are:
Theorem 3. For any  $L \geq \mu \geq 0$ ,  $\Delta \geq 0$ ,  $n \geq 1$ , and precision  $\epsilon > 0$ , the following results hold. See the proof in Appendix D.
- Strongly-convex: If  $\mu > 0$ , by setting parameters  $\eta_k \equiv \eta = n\theta_2 / (120\omega L)$ ,  $\theta_{1,k} \equiv \theta_1 = 1 / (3\sqrt{\kappa})$ ,  $\alpha = p = 1 / (1 + \omega)$ ,  $\gamma_k \equiv \gamma = \eta / (2\theta_1 + \eta \mu)$ ,  $\beta = 2\theta_1 / (2\theta_1 + \eta \mu)$ , and  $\theta_2 = 1 / (3\sqrt{n} + 3n / \omega)$ , ADIANA requires
$$
\mathcal {O} \left(\left(\omega + \left(1 + \frac {\omega}{\sqrt {n}}\right) \sqrt {\kappa}\right) \ln \left(\frac {L \Delta}{\epsilon}\right)\right)
$$
rounds of communication to reach  $\mathbb{E}[f(\hat{x})] - \min_x f(x) \leq \epsilon$ .
- Generally-convex: If  $\mu = 0$ , by setting parameters  $\alpha = 1/(1 + \omega)$ ,  $\beta = 1$ ,  $p = \theta_2 = 1/(3(1 + \omega))$ ,  $\theta_{1,k} = 9/(k + 27(1 + \omega))$ ,  $\gamma_k = \eta_k/(2\theta_{1,k})$ , and
$$
\eta_ {k} = \min  \left\{\frac {k + 1 + 2 7 (1 + \omega)}{9 (1 + \omega) ^ {2} (1 + 2 7 (1 + \omega)) L}, \frac {3 n}{2 0 0 \omega (1 + \omega) L}, \frac {1}{2 L} \right\},
$$
ADIANA requires
$$
\mathcal {O} \left((1 + \omega) \sqrt [ 3 ]{\frac {L \Delta}{\epsilon}} + \left(1 + \frac {\omega}{\sqrt {n}}\right) \sqrt {\frac {L \Delta}{\epsilon}}\right)
$$
Tightness of our lower bounds. Comparing the upper bounds in Theorem 3 with the lower bounds in Theorem 2, ADIANA attains the lower bound in the strongly-convex case up to a  $\ln (\kappa)$  factor, implying the tightness of our lower bound and ADIANA's optimality. In the generally-convex case, the upper bound matches the lower bound's dominating term  $(1 + \omega /\sqrt{n})\sqrt{L\Delta / \epsilon}$  but mismatches the smaller term. This shows the tightness of our lower bound and ADIANA's optimality in the
Figure 2: Convergence results of various distributed algorithms on a synthetic least squares problem (left), logistic regression problems with dataset a9a (middle) and w8a (right). The  $y$ -axis represents  $f(\hat{x}) - f^{\star}$  and the  $x$ -axis indicates the total communicated bits sent by per worker.
high-precision regime  $\epsilon < L\Delta \left(\frac{1 + \omega / \sqrt{n}}{1 + \omega}\right)^6$ . Our refined rates for ADIANA are state-of-the-art among existing algorithms using independent unbiased compression.
In this section, we empirically compare ADIANA with DIANA [32], EF21 [49], and CANITA [33] using unbiased compression, as well as Nesterov's accelerated algorithm [43] which is an optimal algorithm when no compression is employed. We conduct experiments on least-square problems (strongly-convex) with synthetic datasets as well as logistic regression problems (generally-convex) with real datasets. In all experiments, we measure the total communicated bits sent by a single worker, which is calculated through communication rounds to achieve an  $\epsilon$ -accurate solutions  $\times$  per-round communicated bits. All curves are averaged over 20 trials with the region of standard deviations depicted. Due to the space limit, we only provide results with random- $s$  compressors here. More experimental results can be found in Appendix F.2.
Least squares. Consider a distributed least-square problem (1) with  $f_{i}(x) \coloneqq \frac{1}{2} \| A_{i}x - b_{i}\|^{2}$ , where  $A_{i} \in \mathbb{R}^{M \times d}$  and  $b_{i} \in \mathbb{R}^{M}$  are randomly generated. We set  $d = 20$ ,  $n = 400$ , and  $M = 25$ , and generate  $A_{i}$ 's by randomly generating a Gaussian matrix in  $\mathbb{R}^{nM \times d}$ , then modify its condition number to  $10^{4}$  through the SVD decomposition, and finally distribute its rows to all  $A_{i}$ . We use independent random-1 compressors for communication compression. The results are depicted in Fig. 2 (left) where we observe ADIANA beats all baselines in terms of the total communication cost. We do not compare with CANITA since it does not have theoretical guarantees for strongly-convex problems.
Logistic regression. Consider a distributed logistic regression problem (1) with  $f_{i}(x) \coloneqq \frac{1}{M} \sum_{m=1}^{M} \ln(1 + \exp(-b_{i,m}a_{i,m}^{\top}x))$ , where  $\{(a_{i,m}, b_{i,m})\}_{1 \leq i \leq n, 1 \leq m \leq M}$  are datapoints in a9a and w8a datasets from LIBSVM [11]. We set  $n = 400$  and choose independent random  $\lfloor d/20 \rfloor$  compressors for algorithms with compressed communication. The results are as shown in Fig. 2 (middle and right). Again, we observe that ADIANA outperforms all baselines.
Influence of independence in unbiased compression. We also construct a delicate quadratic problem to validate the role of independence in unbiased compression to save communication, see Fig. 1. Experimental details are in Appendix F.1. We observe that ADIANA with independent random- $s$  compressors saves more bits than Nesterov's accelerated algorithm while random- $s$  compressors of shared randomness do not. Furthermore, more aggressive compression, i.e., a larger  $\omega$ , saves more communication costs in total. These observations are consistent with our theories implied in (9).
This paper clarifies that unbiased compression alone cannot save communication, but this goal can be achieved by further assuming mutual independence between compressors. We also demonstrate the saving can be up to  $\Theta (\sqrt{\min\{n,\kappa\}})$ . Future research can explore when and how much biased compressors can save communication in non-convex and stochastic scenarios.
This work is supported by NSFC Grant 12301392, 92370121, and 12288101.
A Random sparsification
We illustrate the random- $s$  sparsification here. More examples of unbiased compressors can be found in literature [50].
Example 1 (RANDOM- $s$  SPARSIFICATION). For any  $x \in \mathbb{R}^d$ , the random- $s$  sparsification is defined by  $C(x) := \frac{d}{s} (\xi \odot x)$  where  $\odot$  denotes the entry-wise product and  $\xi \in \{0,1\}^d$  is a uniformly random binary vector with  $s$  non-zero entries. This random- $s$  sparsification operator  $C$  satisfies Assumption 2 with  $\omega = d / s - 1$ . When each entry of the input  $x$  is represented with  $r$  bits, random- $s$  sparsification compressor takes  $rs$  bits to transmit  $s$  entries and  $\log_2\binom{d}{s}$  bits to transmit the indices of  $s$  transmitted entries, resulting in a total  $\frac{rd}{1 + \omega} + \log_2\binom{d}{s}$  bits in each communication round, see [50, Table 1].
We first recall a result proved by [50].
Lemma 2 ([50], Theorem 2). Let  $C: \mathbb{R}^d \to \mathbb{R}^d$  be any unbiased compressors satisfying 2 and  $b$  be the total number of bits needed to encode the compressed vector  $C(x)$  for any  $x \in \mathbb{R}^d$ . If each entry of the input  $x$  is represented with  $r$  bits, it holds that  $\max \left\{\frac{\omega}{1 + \omega}, 4^{-r}\right\} 4^{b / d} \geq 1$ .
Using Lemma 2, when  $\omega/(1+\omega) \leq 4^{-r}$ , i.e.,  $\omega \leq (4^r - 1)^{-1} \leq 1/3$ , we have  $(1+\omega) = \Theta(1)$  and  $b \geq rd = \Omega_r(d/(1+\omega))$ , where  $r$  is regarded as a constant in  $\Omega_r(\cdot)$ . When  $\omega/(1+\omega) \geq 4^{-r}$ , we have
$$
b \geq d \log_ {4} (1 + \omega^ {- 1}) = d \ln (1 + \omega^ {- 1}) / \ln (4) \geq d \frac {\omega^ {- 1}}{\ln (4) (1 + \omega^ {- 1})} = \Omega_ {r} \left(\frac {d}{1 + \omega}\right),
$$
where we use the inequality  $\ln (1 + t)\geq t / (1 + t)$  with  $t = \omega^{-1}\geq 0$
Following [5, 9], we denote the  $k$ -th coordinate of a vector  $x \in \mathbb{R}^d$  by  $[x]_k$  for  $k = 1, \ldots, d$ , and let  $\operatorname{prog}(x)$  be
$$
\operatorname {p r o g} (x) := \left\{ \begin{array}{l l} 0, & \text {i f} x = 0, \\ \max  _ {1 \leq k \leq d} \{k: [ x ] _ {k} \neq 0 \}, & \text {o t h e r w i s e}. \end{array} \right.
$$
Similarly, for a set of multiple points  $\mathcal{X} = \{x_1,x_2,\ldots \}$ , we define  $\mathrm{prog}(\mathcal{X})\coloneqq \max_{x\in \mathcal{X}}\mathrm{prog}(x)$ . We call a function  $f$  zero-chain if it satisfies
$$
\operatorname {p r o g} (\nabla f (x)) \leq \operatorname {p r o g} (x) + 1, \quad \forall x \in \mathbb {R} ^ {d},
$$
which implies that starting from  $x^0 = 0$ , a single gradient evaluation can only earn at most one more non-zero coordinate for the model parameters.
Let us now illustrate the setup of distributed optimization with communication compression. For any  $t \geq 1$ , we consider the  $t$ -th communication round, which begins with the server broadcasting a vector denoted as  $u^t$  to all workers. We initialize  $u^1$  as  $x^0$ . Upon receiving the vector  $u^t$  from the server, each worker performs necessary algorithmic operations, and the round concludes with each worker sending a compressed message back to the server.
We denote  $v_{i}^{t}$  as the vector that worker  $i$  aims to send in the  $t$ -th communication round before compression, and  $\hat{v}_{i}^{t}$  as the compressed vector that will be received by the server, i.e.,  $\hat{v}_{i}^{t} = C_{i}(v_{i}^{t})$ . While we require communication to be synchronous among workers, we do not impose restrictions on the number of gradient queries made by each worker within a communication round. We use  $\mathcal{V}_{i}^{t}$  to represent the set of vectors at which worker  $i$  makes gradient queries in the  $t$ -th communication round, after receiving  $u^{t}$  but before sending  $\hat{v}_{i}^{t}$ .
Following the above description, we now formally state the linear spanning property in the setting of centralized distributed optimization with communication compression.
Definition 2 (LINEAR-SPANNING ALGORITHMS). We say a distributed algorithm  $A$  is linear-spanning if, for any  $t \geq 1$ , the following conditions hold:
1. The server can only send a vector in the linear manifold spanned by all the past received messages, sent messages, i.e.,  $u^t \in \operatorname{span}\left(\{u^r\}_{r=1}^{t-1} \cup \{\hat{v}_i^r : 1 \leq i \leq n\}_{r=1}^{t-1}\right)$ .
2. Worker  $i$  can only query at vectors in the linear manifold spanned by its past received messages, compressed messages, and gradient queries, i.e.,  $\mathcal{V}_i^t \subseteq \operatorname{span}\left(\{u^r\}_{r=1}^t \cup \{\nabla f_i(y): y \in \mathcal{V}_i^r\}_{r=1}^{t-1} \cup \{\hat{v}_i^r\}_{r=1}^{t-1}\right)$ .
3. Worker  $i$  can only send a vector in the linear manifold spanned by its past received messages, compressed messages, and local gradient queries, i.e.,  $\hat{v}_i^t \in \operatorname{span}\left(\{u^r\}_{r=1}^t \cup \{\nabla f_i(y) : y \in \mathcal{Y}_i^r\}_{r=1}^t \cup \{\hat{v}_i^r\}_{r=1}^{t-1}\right)$ .
4. After  $t$  communication rounds, the server can only output a model in the linear manifold spanned by all the past received messages, sent messages, i.e.,  $\hat{x}^t \in \mathrm{span}\left(\{u^r\}_{r=1}^t \cup \{\hat{v}_i^r : 1 \leq i \leq n\}_{r=1}^t\right)$ .