SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alexander Tyurin
KAUST
alexandertiurin@gmail.com
Peter Richtárik
richtarik@gmail.com
We consider distributed convex optimization problems in the regime when the communication between the server and the workers is expensive in both uplink and downlink directions. We develop a new and provably accelerated method, which we call 2Direction, based on fast bidirectional compressed communication and a new bespoke error-feedback mechanism which may be of independent interest. Indeed, we find that the EF and EF21-P mechanisms (Seide et al., 2014; Gruntkowska et al., 2023) that have considerable success in the design of efficient non-accelerated methods are not appropriate for accelerated methods. In particular, we prove that 2Direction improves the previous state-of-the-art communication complexity  $\widetilde{\Theta}\left(K\times \left(L / \alpha \mu +L_{\mathrm{max}}\omega /n\mu +\omega\right)\right)$  (Gruntkowska et al., 2023) to  $\widetilde{\Theta} (K\times (\sqrt{L(\omega + 1) / \alpha\mu} +\sqrt{L_{\mathrm{max}}\omega^2 / n\mu} +1 / \alpha +\omega))$  in the  $\mu$  -strongly-convex setting, where  $L$  and  $L_{\mathrm{max}}$  are smoothness constants,  $n$  is # of workers,  $\omega$  and  $\alpha$  are compression errors of the Rand  $K$  and Top  $K$  sparsifiers (as examples),  $K$  is # of coordinates/bits that the server and workers send to each other. Moreover, our method is the first that improves upon the communication complexity of the vanilla accelerated gradient descent (AGD) method (Nesterov, 2018). We obtain similar improvements in the general convex regime as well. Finally, our theoretical findings are corroborated by experimental evidence.
We consider convex optimization problems in the centralized distributed setting. These types of problems appear in federated learning (Konečný et al., 2016; McMahan et al., 2017) and distributed optimization (Ramesh et al., 2021). In this setting, one of the main problems is the communication bottleneck: the connection link between the server and the workers can be very slow. We focus our attention on methods that aim to address this issue by applying lossy compression to the communicated messages (Alistarh et al., 2017; Mishchenko et al., 2019; Gruntkowska et al., 2023).
1.1 The problem
Formally, we consider the optimization problem
$$
\min  _ {x \in \mathbb {R} ^ {d}} \left\{f (x) := \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} (x) \right\}, \tag {1}
$$
where  $n$  is the number of workers and  $f_{i}:\mathbb{R}^{d}\to \mathbb{R}$  are smooth convex functions for all  $i\in [n]\coloneqq \{1,\ldots ,n\}$ . We consider the centralized distributed optimization setting in which each  $i^{\mathrm{th}}$  worker contains the function  $f_{i}$ , and all workers are directly connected to a server (Kairouz et al., 2021). In general, we want to find a (possibly random) point  $\widehat{x}$  such that  $\mathbb{E}[f(\widehat{x})] - f(x^{*})\leq \varepsilon$ , where  $x^{*}$  is
37th Conference on Neural Information Processing Systems (NeurIPS 2023).
an optimal point. In the strongly convex setup, we also want to guarantee that  $\mathbb{E}[\| \widetilde{x} - x^{*}\|^{2}] \leq \varepsilon$  for some point  $\widetilde{x}$ .
Virtually all other theoretical works in this genre assume that, compared to the worker-to-server (w2s) communication cost, the server-to-workers (s2w) broadcast is so fast that it can be ignored. We lift this limitation and instead associate a relative cost  $r \in [0,1]$  with the two directions of communication. If  $r = 0$ , then s2w communication is free, if  $r = 1$ , then w2s communication is free, and if  $r = 1/2$ , then the s2w and w2s costs are equal. All our theoretical results hold for any  $r \in [0,1]$ . We formalize and elaborate upon this setup in Section 2.
1.2 Assumptions
Throughout the paper we rely on several standard assumptions on the functions  $f_{i}$  and  $f$ .
Assumption 1.1. Functions  $f_{i}$  are  $L_{i}$ -smooth, i.e.,  $\| \nabla f_{i}(x) - \nabla f_{i}(y)\| \leq L_{i}\| x - y\|$  for all  $x,y \in \mathbb{R}^d$ , for all  $i \in [n]$ . We let  $L_{\max} \coloneqq \max_{i\in [n]}L_{i}$ . Further, let  $\widehat{L} > 0$  be a constant such that  $\frac{1}{n}\sum_{i = 1}^{n}\| \nabla f_i(x) - \nabla f_i(y)\| ^2\leq \widehat{L}^2\| x - y\| ^2$  for all  $x,y \in \mathbb{R}^d$ .
Note that if the functions  $f_{i}$  are  $L_{i}$ -smooth for all  $i \in [n]$ , then  $\widehat{L} \leq L_{\max}$ .
Assumption 1.2. Function  $f$  is  $L$ -smooth, i.e.,  $\| \nabla f(x) - \nabla f(y) \| \leq L \| x - y \|$  for all  $x, y \in \mathbb{R}^d$ . Assumption 1.3. Functions  $f_i$  are convex for all  $i \in [n]$ , and  $f$  is  $\mu$ -strongly convex with  $\mu \geq 0$ , attaining a minimum at some point  $x^* \in \mathbb{R}^d$ .
It is known that the above smoothness constants are related in the following way.
Lemma 1.4 (Gruntkowska et al. (2023)). If Assumptions 1.2, 1.1 and 1.3 hold, then  $\widehat{L} \leq L_{\max} \leq nL$  and  $L \leq \widehat{L} \leq \sqrt{L_{\max} L}$ .
2 Motivation: From Unidirectional to Bidirectional Compression
In this work, we distinguish between worker-to-server (w2s=uplink) and server-to-worker (s2w=downlink) communication cost, and define w2s and s2w communication complexities of methods in the following natural way.
Definition 2.1. For a centralized distributed method  $\mathcal{M}$  aiming to solve problem (1), the communication complexity  $\mathfrak{m}_{\mathcal{M}}^{\mathrm{w2s}}$  is the expected number of coordinates/floats<sup>1</sup> that each worker sends to the server to solve problem (1). The quantity  $\mathfrak{m}_{\mathcal{M}}^{\mathrm{s2w}}$  is the expected number of floats/coordinates the server broadcasts to the workers to solve problem (1). If  $\mathfrak{m}_{\mathcal{M}}^{\mathrm{w2s}} = \mathfrak{m}_{\mathcal{M}}^{\mathrm{s2w}}$ , then we use the simplified notation  $\mathfrak{m}_{\mathcal{M}} := \mathfrak{m}_{\mathcal{M}}^{\mathrm{s2w}} = \mathfrak{m}_{\mathcal{M}}^{\mathrm{w2s}}$ .
Let us illustrate the above concepts on the simplest baseline: vanilla gradient descent (GD). It is well known (Nesterov, 2018) that for  $L$ -smooth,  $\mu$ -strongly convex problems, GD returns an  $\varepsilon$ -solution after  $\mathcal{O}\left(L / \mu \log^{1 / \varepsilon}\right)$  iterations. In each iteration, the workers and the server communicate all  $\Theta(d)$  coordinates to each other (since no compression is applied). Therefore, the communication complexity of GD is  $\mathfrak{m}_{\mathrm{GD}} = \Theta\left(dL / \mu \log^{1 / \varepsilon}\right)$ . The same reasoning applies to the accelerated gradient method (AGD) (Nesterov, 2018), whose communication complexity is  $\mathfrak{m}_{\mathrm{AGD}} = \Theta\left(d\sqrt{L / \mu} \log^{1 / \varepsilon}\right)$ .
2.1 Compression mappings
In the literature, researchers often use the following two families of compressors:
Definition 2.2. A (possibly) stochastic mapping  $\mathcal{C}:\mathbb{R}^d\to \mathbb{R}^d$  is a biased compressor if there exists  $\alpha \in (0,1]$  such that
$$
\mathbb {E} \left[ \| \mathcal {C} (x) - x \| ^ {2} \right] \leq (1 - \alpha) \| x \| ^ {2}, \quad \forall x \in \mathbb {R} ^ {d}. \tag {2}
$$
Definition 2.3. A stochastic mapping  $\mathcal{C}:\mathbb{R}^d\to \mathbb{R}^d$  is an unbiased compressor if there exists  $\omega \geq 0$  such that
$$
\underline {{\mathbb {E}}} [ \mathcal {C} (x) ] = x, \quad \mathbb {E} \left[ \| \mathcal {C} (x) - x \| ^ {2} \right] \leq \omega \| x \| ^ {2}, \quad \forall x \in \mathbb {R} ^ {d}. \tag {3}
$$
<sup>1</sup>Some works measure bits instead of coordinates. For computer systems, where coordinates are represented by 32 or 64 bits, these measures are equivalent up to the constant factors 32 or 64.
Table 1: Communication Rounds in the Strongly Convex Case. The number of communication rounds and rounds costs to get an  $\varepsilon$ -solution  $(\mathbb{E}\left[||\widehat{x} -x^{*}||^{2}\right]\leq \varepsilon)$  up to logarithmic factors. The table shows the most relevant bidirectional compressed methods that are ordered by the total communication complexity # Communication Rounds  $\times$  Round Cost (see (4) for details).
i. The parameter  $r$  weights the importance/speed of uplink and downlink connections. When  $r = 1/2$ , it means that the uplink and downlink speeds are equal.
ii. The parameters  $K_{\omega}$  and  $K_{\alpha}$  are the expected densities Definition 2.5 of compressors  $\mathcal{C}^D \in \mathbb{U}(\omega)$  and  $\mathcal{C}^P \in \mathbb{B}(\alpha)^{\mathrm{(a)}}$ , that operate in the workers and the server accordingly. Less formally,  $K_{\omega}$  and  $K_{\alpha}$  are the number of coordinates/bits that the workers and the server send to each other in each communication round.
Method	# Communication Rounds	Round Cost(c)
Dore, Artemis, MURANA(a)(Liu et al., 2020)(Philippenko and Dieulevut, 2020)(Condat and Richtárik, 2022)	\(\widetilde{\Omega}\left(\frac{\omega}{\alpha n}\frac{L_{\max}}{\mu}\right)^{(f)}\)	(1-r)Kω+rKα
MCM(a)(Philippenko and Dieulevut, 2021)	\(\widetilde{\Omega}\left(\left(\frac{1}{\alpha^{3/2}}+\frac{\omega^{1/2}}{\alpha\sqrt{n}}+\frac{\omega}{n}\right)\frac{L_{\max}}{\mu}\right)^{(f)}\)	(1-r)Kω+rKα
GD(Nesterov, 2018)	\(\frac{L}{\mu}\)	d
EF21-P + DIANA(Gruntkowska et al., 2023)	\(\frac{L}{\alpha\mu}+\frac{L_{\max}\omega}{n\mu}+\omega\)	(1-r)Kω+rKα
AGD(Nesterov, 2018)	\(\sqrt{\frac{L}{\mu}}\)	d
2Direction(Remark 5.3)(b), (Theorem 5.2)	\(\sqrt{\frac{L(\omega+1)}{\alpha\mu}}+\sqrt{\frac{L_{\max}\omega^2}{n\mu}}+\frac{1}{\alpha}+\omega\)	(1-r)Kω+rKα
2Direction(Remark 5.5)(b), (Theorem 5.4)(requires Lmax/L)(d)	\(\sqrt{\frac{L_{\max}\{1,r(\omega+1)\}}{\alpha\mu}}+\sqrt{\frac{L^2/3L_{\max}^{1/3}(\omega+1)}{\alpha n^{1/3}\mu}}+\sqrt{\frac{L^1/2L_{\max}^{1/2}(\omega+1)^{3/2}}{\sqrt{\alpha n\mu}}}+\sqrt{\frac{L_{\max}\omega^2}{n\mu}}+\frac{1}{\alpha}+\omega\)	(1-r)Kω+rKα
(a) The Dore, Artemis, MURANA, and MCM methods do not support biased compressors for server-to-worker compression. In these methods, the error  $\alpha$  equals  $1 / (\omega_{\mathrm{s}} + 1)$ , where the error  $\omega_{s}$  is a parameter of an unbiased compressor that is used in server-to-worker compression. For these methods, we define  $1 / (\omega_{\mathrm{s}} + 1)$  as  $\alpha$  to make comparison easy with EF21-P + DIANA and 2Direction.
(b) In this table, we present the simplified iteration complexity of 2Direction assuming that  $r \leq 1/2$  and  $\omega + 1 = \Theta\left(d / K_{\omega}\right)$ . The full complexities are in (13) and (14). In Section 6, we show that 2Direction has no worse total communication complexity than EF21-P + DIANA for all  $r \in [0,1]$  and for any choice of compressors.
(c) We define Round Cost of a method  $\mathcal{M}$  as a constant such that  $\mathfrak{m}_{\mathcal{M}}^r = \# \text{Communication Rounds} \times \text{Round Cost}$ , where  $\mathfrak{m}_{\mathcal{M}}^r$  is the total communication complexity (4).
(d) 2Direction can have even better total communication complexity if the algorithm can use the ratio  $L_{\max} / L$  when selecting the parameters  $\tau$  and  $p$  in Algorithm 1. For instance, this is the case if we assume that  $L_{\max} = L$ , which was done by Li et al. (2020); Li and Richtárik (2021), for example.
(1) The notation  $\widetilde{\Omega} (\cdot)$  means "at least up to logarithmic factors."
We will make use of the following assumption.
Assumption 2.4. The randomness in all compressors used in our method is drawn independently.
Let us denote the set of mappings satisfying Definition 2.2 and 2.3 by  $\mathbb{B}(\alpha)$  and  $\mathbb{U}(\omega)$ , respectively. The family of biased compressors  $\mathbb{B}$  is wider. Indeed, it is well known if  $\mathcal{C} \in \mathbb{U}(\omega)$ , then  $^{1 / (\omega + 1)}\cdot \mathcal{C} \in \mathbb{B}(1 / (\omega + 1))$ . The canonical sparsification operators belonging to these classes are  $\operatorname{Top}K \in \mathbb{B}(K / d)$  and  $\operatorname{Rand}K \in \mathbb{U}(d / K - 1)$ . The former outputs the  $K$  largest values (in magnitude) of the input vector, while the latter outputs  $K$  random values of the input vector, scaled by  $d / K$  (Beznosikov et al., 2020). Following (Gorbunov et al., 2021; Tyurin and Richtárik, 2023), we now define the expected density of a sparsifier as a way to formalize its compression performance.
Definition 2.5. The expected density of a sparsifier  $\mathcal{C}:\mathbb{R}^d\to \mathbb{R}^d$  is the quantity  $K_{\mathcal{C}}\coloneqq \sup_{x\in \mathbb{R}^{d}}\mathbb{E}\left[\| \mathcal{C}(x)\|_{0}\right]$ , where  $\| y\| _0$  is the number of of non-zero components of  $y\in \mathbb{R}^d$ .
Trivially, for the Rand  $K$  and Top  $K$  sparsifiers we have  $K_{\mathcal{C}} = K$ .
2.2 Unidirectional (i.e., w2s) compression
As mentioned in the introduction, virtually all theoretical works in the area of compressed communication ignore s2w communication cost and instead aim to minimize  $\mathfrak{m}_{\mathcal{M}}^{w_2}$ . Algorithmic work related
to methods that only perform w2s compression has a long history, and this area is relatively well understood (Alistarh et al., 2017; Mishchenko et al., 2019; Richtárik et al., 2021).
We refer to the work of Gruntkowska et al. (2023) for a detailed discussion of the communication complexities of non-accelerated methods in the convex and non-convex settings. For instance, using Rand  $K$ , the DIANA method of Mishchenko et al. (2019) provably improves<sup>2</sup> the communication complexity of GD to  $\mathfrak{m}_{\mathrm{DIANA}}^{\mathrm{w2s}} = \widetilde{\Theta}\left(d + KL / \mu + dL_{\max} / n\mu\right)$ . Accelerated methods focusing on w2s compression are also well investigated. For example, Li et al. (2020) and Li and Richtárik (2021) developed accelerated methods, which are based on (Mishchenko et al., 2019; Kovalev et al., 2020), and provably improve the w2s complexity of DIANA. Moreover, using Rand  $K$  with  $K \leq d / n$ , ADIANA improves the communication complexity of AGD to  $\mathfrak{m}_{\mathrm{ADIANA}}^{\mathrm{w2s}} = \widetilde{\Theta}\left(d + d\sqrt{L_{\max} / n\mu}\right)$ .
2.3 Bidirectional (i.e., w2s and s2w) compression
The methods mentioned in Section 2.2 do not perform server-to-workers (s2w) compression, and one can show that the server-to-workers (s2w) communication complexities of these methods are worse than  $\mathfrak{m}_{\mathrm{AGD}} = \widetilde{\Theta}(d\sqrt{L / \mu})$ . For example, using the RandK, the s2w communication complexity of ADIANA is at least  $\mathfrak{m}_{\mathrm{ADIANA}}^{s2w} = \widetilde{\Omega}(d \times \omega) = \widetilde{\Omega}(d^2 / K)$ , which can be  $d / K$  times larger than in GD or AGD. Instead of  $\mathfrak{m}_{\mathcal{M}}^{w2s}$ , methods performing bidirectional compression attempt to minimize the total communication complexity, which we define as a convex combination of the w2s and s2w communication complexities:
$$
\mathfrak {m} _ {\mathcal {M}} ^ {r} := (1 - r) \mathfrak {m} _ {\mathcal {M}} ^ {\mathrm {w} 2 \mathrm {s}} + r \mathfrak {m} _ {\mathcal {M}} ^ {\mathrm {s} 2 \mathrm {w}}. \tag {4}
$$
The parameter  $r \in [0,1]$  weights the importance of uplink (w2s) and downlink (s2w) connections. Methods from Section 2.2 assume that  $r = 0$ , thus ignoring the s2w communication cost. On the other hand, when  $r = 1/2$ , the uplink and downlink communication speeds are equal. By considering any  $r \in [0,1]$ , our methods and findings are applicable to more situations arising in practice. Obviously,  $\mathfrak{m}_{\mathrm{GD}}^r = \mathfrak{m}_{\mathrm{GD}}$  and  $\mathfrak{m}_{\mathrm{AGD}}^r = \mathfrak{m}_{\mathrm{AGD}}$  for all  $r \in [0,1]$ . Recently, Gruntkowska et al. (2023) proposed the EF21-P + DIANA method. This is the first method supporting bidirectional compression that provably improves both the w2s and s2w complexities of GD:  $\mathfrak{m}_{\mathrm{EF21 - P + DIANA}}^r \leq \mathfrak{m}_{\mathrm{GD}}$  for all  $r \in [0,1]$ . Bidirectional methods designed before EF21-P + DIANA, including (Tang et al., 2020; Liu et al., 2020; Philippenko and Dieuleveut, 2021), do not guarantee the total communication complexities better than that of GD. The EF21-P + DIANA method is not an accelerated method and, in the worst case, can have communication complexities worse than AGD when the condition number  $L / \mu$  is large.
3 Contributions
Motivated by the above discussion, in this work we aim to address the following
Main Problem:
Is it possible to develop a method supporting bidirectional communication compression that improves the current best theoretical total communication complexity of EF21-P + DIANA, and guarantees the total communication complexity to be no worse than the communication complexity  $\mathfrak{m}_{\mathrm{AGD}} = \widetilde{\Theta}(d\sqrt{L / \mu})$  of AGD, while improving on AGD in at least some regimes?
A) We develop a new fast method (2Direction; see Algorithm 1) supporting bidirectional communication compression. Our analysis leads to new state-of-the-art complexity rates in the centralized distributed setting (see Table 1), and as a byproduct, we answer Main Problem in the affirmative.
B) Gruntkowska et al. (2023) proposed to use the EF21-P error-feedback mechanism (8) to improve the convergence rates of non-accelerated methods supporting bidirectional communication compression. EF21-P is a reparameterization of the celebrated EF mechanism (Seide et al., 2014). We tried to use EF21-P in our method as well, but failed. Our failures indicated that a fundamentally
$^{2}$ Indeed, using Lemma 1.4,  $K \leq d$ , and  $L \geq \mu$ , one can easily show that  $d + KL / \mu + dL_{\max} / n\mu = \mathcal{O}(dL / \mu)$ .  
 $^{3}\mathfrak{m}_{\mathcal{M}}^{r} \propto s^{\mathrm{w2s}}\mathfrak{m}_{\mathcal{M}}^{\mathrm{w2s}} + s^{\mathrm{s2w}}\mathfrak{m}_{\mathcal{M}}^{\mathrm{s2w}}$ , where  $s^{\mathrm{w2s}}$  and  $s^{\mathrm{s2w}}$  are connection speeds, and  $r = s^{\mathrm{s2w}} / s^{\mathrm{w2s}} + s^{\mathrm{s2w}}$ .
new approach is needed, and this eventually led us to design a new error-feedback mechanism (9) that is more appropriate for accelerated methods. We believe that this is a contribution of independent interest that might motivate future growth in the area.
C) Unlike previous theoretical works (Li et al., 2020; Li and Richtárik, 2021) on accelerated methods, we present a unified analysis in both the  $\mu$ -strongly-convex and general convex cases. Moreover, in the general convex setting and low accuracy regimes, our analysis improves the rate  $\mathcal{O}\left(1 / \varepsilon^{1 / 3}\right)$  of Li and Richtárik (2021) to  $\mathcal{O}\left(\log^{1 / \varepsilon}\right)$  (see details in Section R).
D) Even though our central goal was to obtain new SOTA theoretical communication complexities for centralized distributed optimization, we show that the newly developed algorithm enjoys faster communication complexities in practice as well (see details in Section Q).
Algorithm 1 2Direction: A Fast Gradient Method Supporting Bidirectional Compression
1: Parameters: Lipschitz-like parameter  $\bar{L} >0$  strong-convexity parameter  $\mu \geq 0$  probability  $p\in (0,1]$    
parameter  $\Gamma_0\geq 1$  momentum  $\tau \in (0,1]$  , contraction parameter  $\alpha \in (0,1]$  from (2), initial point  $x^0\in \mathbb{R}^d$    
initial gradient shifts  $h_1^0,\ldots ,h_n^0\in \mathbb{R}^d$  gradient shifts  $k^0\in \mathbb{R}^d$  and  $v^{0}\in \mathbb{R}^{d}$    
2: Initialize  $\beta = 1 / (\omega +1)$ $w^{0} = z^{0} = u^{0} = x^{0}$  , and  $h^0 = \frac{1}{n}\sum_{i = 1}^{n}h_i^0$    
3: for  $t = 0,1,\dots ,T - 1$  do   
4:  $\Gamma_{t + 1},\gamma_{t + 1},\theta_{t + 1} =$  CalculateLearningRates(  $\Gamma_t,\bar{L},\mu ,p,\alpha ,\tau ,\beta)$  Get learning rates using Algorithm 2   
5: for  $i = 1,\dots ,n$  in parallel do   
6:  $y^{t + 1} = \theta_{t + 1}w^{t} + (1 - \theta_{t + 1})z^{t}$    
7:  $m_i^{t,y} = C_i^{D,y}(\nabla f_i(y^{t + 1}) - h_i^t)$  Worker i compresses the shifted gradient via the compressor  $C_i^{D,y}\in \mathbb{U}(\omega)$    
8: Send compressed message  $m_i^{t,y}$  to the server   
9: end for   
10:  $g^{t + 1} = h^t +\frac{1}{n}\sum_{i = 1}^n m_i^{t,y}$    
11:  $u^{t + 1} = \arg \min_{x\in \mathbb{R}^d}\langle g^{t + 1},x\rangle +\frac{\bar{L} + \Gamma_t\mu}{2\gamma_t + 1}\| x - u^t\|^2 +\frac{\mu}{2}\| x - y^{t + 1}\|^2$  A gradient-like descent step   
12:  $q^{t + 1} = \arg \min_{x\in \mathbb{R}^d}\langle k^t,x\rangle +\frac{\bar{L} + \Gamma_t\mu}{2\gamma_t + 1}\| x - w^t\|^2 +\frac{\mu}{2}\| x - y^{t + 1}\|^2$    
13:  $p^{t + 1} = \mathcal{C}^P (u^{t + 1} - q^{t + 1})$  Server compresses the shifted model via the compressor  $\mathcal{C}^P\in \mathbb{B}(\alpha)$    
14:  $w^{t + 1} = q^{t + 1} + p^{t + 1}$    
15:  $x^{t + 1} = \theta_{t + 1}u^{t + 1} + (1 - \theta_{t + 1})z^t$    
16: Send compressed message  $p^{t + 1}$  to all n workers   
17: Flip a coin  $c^t\sim$  Bernoulli(p)   
18:  $k^{t + 1} = \left\{ \begin{array}{ll}v^t, & c^t = 1\\ k^t, & c^t = 0 \end{array} \right.$  and  $z^{t + 1} = \left\{ \begin{array}{ll}x^{t + 1}, & c^t = 1\\ z^t, & c^t = 0 \end{array} \right.$    
19: if  $c^t = 1$  then   
20: Broadcast non-compressed messages  $x^{t + 1}$  and  $k^{t + 1}$  to all n workers With small probability  $p!$    
21: end if   
22: for  $i = 1,\dots ,n$  in parallel do   
23:  $q^{t + 1} = \arg \min_{x\in \mathbb{R}^d}\langle k^t,x\rangle +\frac{\bar{L} + \Gamma_t\mu}{2\gamma_t + 1}\| x - w^t\|^2 +\frac{\mu}{2}\| x - y^{t + 1}\|^2$    
24:  $w^{t + 1} = q^{t + 1} + p^{t + 1}$    
25:  $z^{t + 1} = \left\{ \begin{array}{ll}x^{t + 1}, & c^t = 1\\ z^t, & c^t = 0 \end{array} \right.$    
26:  $m_i^{t,z} = C_i^{D,z}(\nabla f_i(z^{t + 1}) - h_i^t)$  Worker i compresses the shifted gradient via the compressor  $C_i^{D,z}\in \mathbb{U}(\omega)$    
27:  $h_i^{t + 1} = h_i^t +\beta m_i^{t,z}$    
28: Send compressed message  $m_i^{t,z}$  to the server   
29: end for   
30:  $v^{t + 1} = (1 - \tau)v^t +\tau (h^t +\frac{1}{n}\sum_{i = 1}^n m_i^{t,z})$    
31:  $h^{t + 1} = h^t +\beta \frac{1}{n}\sum_{i = 1}^n m_i^{t,z}$    
32: end for
4 New Method: 2Direction
In order to provide an answer to Main Problem, at the beginning of our research journey we hoped that a rather straightforward approach might bear fruit. In particular, we considered the current state-the-art methods ADIANA (Algorithm 3) (Li et al., 2020), CANITA (Li and Richtárik, 2021) and EF21-P + DIANA (Algorithm 4) (Gruntkowska et al., 2023), and tried to combine the EF21-P compression
Algorithm 2 CalculateLearningRates
1: Parameters: element  $\Gamma_{t}$ ; parameter  $\bar{L} > 0$ ; strong-convexity parameter  $\mu \geq 0$ , probability  $p$ , contraction parameter  $\alpha$ , momentum  $\tau$ , parameter  $\beta$
2: Find the largest root  $\bar{\theta}_{t + 1}$  of the quadratic equation
$$
p \bar {L} \Gamma_ {t} \bar {\theta} _ {t + 1} ^ {2} + p (\bar {L} + \Gamma_ {t} \mu) \bar {\theta} _ {t + 1} - (\bar {L} + \Gamma_ {t} \mu) = 0
$$
$$
3: \theta_ {\min } = \frac {1}{4} \min  \left\{1, \frac {\alpha}{p}, \frac {\tau}{p}, \frac {\beta}{p} \right\}; \quad \theta_ {t + 1} = \min  \{\bar {\theta} _ {t + 1}, \theta_ {\min } \}; \quad \gamma_ {t + 1} = \frac {p \theta_ {t + 1} \Gamma_ {t}}{1 - p \theta_ {t + 1}}; \quad \Gamma_ {t + 1} = \Gamma_ {t} + \gamma_ {t + 1}
$$
mechanism on the server side with the ADIANA (accelerated DIANA) compression mechanism on the workers' side. In short, we were aiming to develop a "EF21-P + ADIANA" method. Note that while EF21-P + DIANA provides the current SOTA communication complexity among all methods supporting bidirectional compression, the method is not "accelerated". On the other hand, while ADIANA (in the strongly convex regime) and CANITA (in the convex regime) are "accelerated", they support unidirectional (uplink) compression only.