SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Sections B and C we list the ADIANA and EF21-P + DIANA methods, respectively. One can see that in order to calculate  $x^{t+1}$ ,  $y^{t+1}$ , and  $z^{t+1}$  in ADIANA (Algorithm 3), it is sufficient for the server to broadcast the point  $u^{t+1}$ . At first sight, it seems that we might be able to develop a "EF21-P + ADIANA" method by replacing Line 15 in Algorithm 3 with Lines 12, 13, 14, and 16 from Algorithm 4. With these changes, we can try to calculate  $x^{t+1}$  and  $y^{t+1}$  using the formulas
$$
y ^ {t + 1} = \theta_ {t + 1} w ^ {t} + (1 - \theta_ {t + 1}) z ^ {t}, \tag {5}
$$
$$
x ^ {t + 1} = \theta_ {t + 1} w ^ {t + 1} + (1 - \theta_ {t + 1}) z ^ {t} \tag {6}
$$
$$
w ^ {t + 1} = w ^ {t} + \mathcal {C} ^ {P} \left(u ^ {t + 1} - w ^ {t}\right), \tag {7}
$$
instead of Lines 7 and 17 in Algorithm 3. Unfortunately, all our attempts of making this work failed, and we now believe that this "naive" approach will not lead to a resolution of Main Problem. Let us briefly explain why we think so, and how we ultimately managed to resolve Main Problem.
$\spadesuit$  The first issue arises from the fact that the point  $x^{t + 1}$ , and, consequently, the point  $z^{t + 1}$ , depend on  $w^{t + 1}$  instead of  $u^{t + 1}$ , and thus the error from the primal (i.e., server) compressor  $\mathcal{C}^P$  affects them. In our proofs, we do not know how to prove a good convergence rate with (6). Therefore, we decided to use the original update (Line 17 from Algorithm 3) instead. We can do this almost for free because in Algorithm 3 the point  $x^{t + 1}$  is only used in Line 18 (Algorithm 3) with small probability  $p$ . In the final version of our algorithm 2Direction (see Algorithm 1), we broadcast a non-compressed messages  $x^{t + 1}$  with probability  $p$ . In Section 6, we show that  $p$  is so small that these non-compressed rare messages do not affect the total communication complexity of Algorithm 3.
$\spadesuit$  The second issue comes from the observation that we can not perform the same trick for point  $y^{t + 1}$  since it is required in each iteration of Algorithm 3. We tried to use (5) and (7), but this still does not work. Deeper understanding of this can only be gained by a detailed examination our proof and the proofs of (Li et al., 2020; Gruntkowska et al., 2023). One way to explain the difficulty is to observe that in non-accelerated methods (Gorbunov et al., 2020; Gruntkowska et al., 2023), the variance-reducing shifts  $h^t$  converge to the fixed vector  $\nabla f(x^{*})$ , while in the accelerated methods (Li et al., 2020; Li and Richtárik, 2021), these shifts  $h^t$  converge to the non-fixed vectors  $\nabla f(z^t)$  in the corresponding Lyapunov functions. Assume that  $\mu = 0$ . Then, instead of the EF21-P mechanism
$$
w ^ {t + 1} = w ^ {t} + \mathcal {C} ^ {P} \left(u ^ {t + 1} - w ^ {t}\right) \stackrel {\text {L i n e} 1 3 \text {i n A l g .} 3} {=} w ^ {t} + \mathcal {C} ^ {P} \left(u ^ {t} - \frac {\gamma_ {t + 1}}{L} g ^ {t + 1} - w ^ {t}\right) \tag {8}
$$
$$
\overset {\nabla f (x ^ {*}) = 0} {=} w ^ {t} - \frac {\gamma_ {t + 1}}{L} \nabla f (x ^ {*}) + \mathcal {C} ^ {P} \left(u ^ {t} - \frac {\gamma_ {t + 1}}{L} (g ^ {t + 1} - \nabla f (x ^ {*})) - w ^ {t}\right),
$$
we propose to perform the step
$$
\begin{array}{l} w ^ {t + 1} = w ^ {t} - \frac {\gamma_ {t + 1}}{\bar {L}} \nabla f (z ^ {t}) + \mathcal {C} ^ {P} \left(u ^ {t} - \frac {\gamma_ {t + 1}}{\bar {L}} \left(g ^ {t + 1} - \nabla f (z ^ {t})\right) - w ^ {t}\right) \tag {9} \\ = q ^ {t + 1} + \mathcal {C} ^ {P} (u ^ {t + 1} - q ^ {t + 1}), \text {w h e r e} \\ \end{array}
$$
$$
u ^ {t + 1} = \underset {x \in \mathbb {R} ^ {d}} {\arg \min } \left\langle g ^ {t + 1}, x \right\rangle + \frac {\bar {L}}{2 \gamma_ {t + 1}} \left\| x - u ^ {t} \right\| ^ {2}, q ^ {t + 1} = \underset {x \in \mathbb {R} ^ {d}} {\arg \min } \left\langle \nabla f (z ^ {t}), x \right\rangle + \frac {\bar {L}}{2 \gamma_ {t + 1}} \left\| x - u ^ {t} \right\| ^ {2}.
$$
Unlike (8), step (9) resolves all our previous problems, and we were able to obtain new SOTA rates.
$\spadesuit$  However, step (9) is not implementable since the server and the nodes need to know the vector  $\nabla f(z^t)$ . The last crucial observation is the same as with the points  $x^t$  and  $z^t$ : the vector  $\nabla f(z^t)$  changes with probability  $p$  since the point  $z^t$  changes with probability  $p$ . Intuitively, this means that it is easier to communicate  $\nabla f(z^t)$  between the server and the workers. We do this using two auxiliary control vectors,  $v^t$  and  $k^t$ . The former "learns" the value of  $\nabla f(z^t)$  in Line 30 (Algorithm 1), and the latter is used in Line 14 (Algorithm 1) instead of  $\nabla f(z^t)$ . Then, when the algorithm updates  $z^{t+1}$ , it also updates  $k^t$  in Line 18 (Algorithm 1) and the updated non-compressed vector  $k^t$  is broadcast to the workers.
The described changes are highlighted in Lines 6, 12, 13, 14, 18, 20, 23, 24 and 30 of our new Algorithm 1 in green color. Other steps of the algorithm correspond to the original ADIANA method (Algorithm 3). Remarkably, all these new steps are only required to substitute a single Line 15 of Algorithm 3!
5 Theoretical Communication Complexity of 2Direction
Having outlined our thought process when developing 2Direction (Algorithm 1), we are now ready to present our theoretical iteration and communication complexity results. Note that 2Direction depends on two hyper-parameters, probability  $p$  (used in Lines 4 and 17) and momentum  $\tau$  (used in Lines 4 and 30). Further, while Li et al. (2020); Li and Richtárik (2021) assume a strong relationship between  $L$  and  $L_{\mathrm{max}}$  ( $L = L_{\mathrm{max}}$ ), Gruntkowska et al. (2023) differentiate between  $L$  and  $L_{\mathrm{max}}$ , and thus perform a more general and analysis of their method. In order to perform a fair comparison to the above results, we have decided to minimize the total communication complexity  $\mathfrak{m}_{\mathcal{M}}^r$  as a function of the hyper-parameters  $p$  and  $\tau$ , depending on whether the ratio  $L_{\mathrm{max}} / L$  is known or not.
Defining  $R^2 \coloneqq \left\| x^0 - x^* \right\|^2$ , Theorems E.12 and E.14 state that 2Direction converges after
$$
T := \widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L}{\alpha \tau \mu}}, \sqrt {\frac {\sqrt {L L _ {\operatorname* {m a x}}} (\omega + 1) \sqrt {\omega \tau}}{\alpha \sqrt {n} \mu}}, \sqrt {\frac {\sqrt {L L _ {\operatorname* {m a x}}} \sqrt {\omega + 1} \sqrt {\omega \tau}}{\alpha \sqrt {p} \sqrt {n} \mu}}, \right. \right. \tag {10}
$$
$$
\left. \left. \sqrt {\frac {L _ {\operatorname* {m a x}} \omega (\omega + 1) ^ {2} p}{n \mu}}, \sqrt {\frac {L _ {\operatorname* {m a x}} \omega}{n p \mu}}, \frac {1}{\alpha}, \frac {1}{\tau}, (\omega + 1), \frac {1}{p} \right\}\right), \mathrm {o r}
$$
$$
\begin{array}{l} T _ {\mathrm {g c}} := \Theta \left(\max  \left\{\sqrt {\frac {L R ^ {2}}{\alpha p \varepsilon}}, \sqrt {\frac {L R ^ {2}}{\alpha \tau \varepsilon}}, \sqrt {\frac {\sqrt {L L _ {\operatorname* {m a x}}} (\omega + 1) \sqrt {\omega \tau} R ^ {2}}{\alpha \sqrt {n} \varepsilon}}, \sqrt {\frac {\sqrt {L L _ {\operatorname* {m a x}}} \sqrt {\omega + 1} \sqrt {\omega \tau} R ^ {2}}{\alpha \sqrt {p} \sqrt {n} \varepsilon}}, \right. \right. \tag {11} \\ \left. \sqrt {\frac {L _ {\max} \omega (\omega + 1) ^ {2} p R ^ {2}}{n \varepsilon}}, \sqrt {\frac {L _ {\max} \omega R ^ {2}}{n p \varepsilon}} \right\} + \widetilde {\Theta} \left(\max  \left\{\frac {1}{\alpha}, \frac {1}{\tau}, (\omega + 1), \frac {1}{p} \right\}\right) \\ \end{array}
$$
iterations, in the  $\mu$ -strongly convex and general convex regimes, respectively. These complexities depend on two hyper-parameters:  $p \in (0,1]$  and  $\tau \in (0,1]$ . For simplicity, in what follows we consider the  $\mu$ -strongly convex case only<sup>4</sup>. While the iteration complexities (10) and (11) are clearly important, in the context of our paper, optimizing communication complexity (4) is more important. In the following simple theorem, we give expressions for the communication complexities of 2Direction, taking into account both workers-to-server (w2s) and server-to-workers (s2w) communication.
Theorem 5.1. Assume that  $\mathcal{C}_i^{D,\cdot}$  and  $\mathcal{C}^P$  have expected densities equal to  $K_{\omega}$  and  $K_{\alpha}$ , respectively (see Definition 2.5). In view of Theorem E.12, in expectation, the w2s and s2w communication complexities are equal to
$$
\mathfrak {m} _ {\text {n e w}} ^ {\mathrm {w 2 s}} = \widetilde {\Theta} \left(K _ {\omega} \times T + d\right) \quad \text {a n d} \quad \mathfrak {m} _ {\text {n e w}} ^ {\mathrm {s 2 w}} = \widetilde {\Theta} \left(\left(K _ {\alpha} + p d\right) \times T + d\right). \tag {12}
$$
Proof. The first complexity in (12) follows because w2s communication involves  $\mathcal{C}_i^{D,y}(\cdot)$  and  $\mathcal{C}_i^{D,z}(\cdot)$  only. The second complexity in (12) follows because s2w communication involves  $\mathcal{C}_i^P (\cdot)$ , plus two non-compressed vectors  $x^{t + 1}$  and  $k^{t + 1}$  with the probability  $p$ . The term  $d$  comes from the fact that non-compressed vectors are communicated in the initialization phase.
One can always take  $\mu = \varepsilon / R^2$  to understand the dependencies in the general convex case.
5.1 The ratio  $L_{\mathrm{max}} / L$  is not known
In the following theorem, we consider the regime when the exact value of  $L_{\max} / L$  is not known. Hence, we seek to find  $p$  and  $\tau$  that minimize the worst case  $\mathfrak{m}_{\mathrm{new}}^r$  (see (4)) w.r.t.  $L_{\max} \in [L, nL]$ .
Theorem 5.2. Choose  $r \in [0,1]$  and let  $\mu_{\omega,\alpha}^r \coloneqq \frac{rd}{(1-r)K_\omega + rK_\alpha}$ . In view of Theorem 5.1, the values  $p = \min \left\{\frac{1}{\omega+1}, \frac{1}{\mu_{\omega,\alpha}^r}\right\}$  and  $\tau = \frac{p^{1/3}}{(\omega+1)^{2/3}}$  minimize  $\max_{L_{\mathrm{max}} \in [L,nL]} \mathfrak{m}_{\mathrm{new}}^r$ . This choice leads to the following number of communication rounds:
$$
T ^ {\text {r e a l i s t i c}} := \widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L \operatorname* {m a x} \left\{\omega + 1 , \mu_ {\omega , \alpha} ^ {r} \right\}}{\alpha \mu}}, \sqrt {\frac {L _ {\operatorname* {m a x}} \omega \operatorname* {m a x} \left\{\omega + 1 , \mu_ {\omega , \alpha} ^ {r} \right\}}{n \mu}}, \frac {1}{\alpha}, (\omega + 1), \mu_ {\omega , \alpha} ^ {r} \right\}\right). \tag {13}
$$
The total communication complexity thus equals  $\mathfrak{m}_{\mathrm{realistic}}^r = \widetilde{\Theta}\left((1 - r)K_\omega +rK_\alpha\right)T_{\mathrm{realistic}} + d$ .
Remark 5.3. To simplify the rate (13) and understand the quantity  $\mu_{\omega, \alpha}^{r}$ , let  $\mathcal{C}_i^{D,\cdot}$  be the RandK sparsifier<sup>5</sup> and consider the case when the s2w communication is not slower than the w2s communication, i.e.,  $r \leq 1/2$ . Then  $T^{\mathrm{realistic}} = \widetilde{\Theta}\left(\max \left\{\sqrt{\frac{L(\omega + 1)}{\alpha\mu}}, \sqrt{\frac{L_{\max}\omega(\omega + 1)}{np\mu}}, \frac{1}{\alpha}, (\omega + 1)\right\}\right)$  and  $\mu_{\omega, \alpha}^{r} \leq \omega + 1$ . Indeed, this follows from  $r \leq 1/2$  and the fact that  $\omega + 1 = d / K_{\omega}$  for the RandK compressor:  $\mu_{\omega, \alpha}^{r} := \frac{rd}{(1 - r)K_{\omega} + rK_{\alpha}} \leq \frac{r}{1 - r} \times \frac{d}{K_{\omega}} \leq \omega + 1$ .
5.2 The ratio  $L_{\max} / L$  is known
We now consider the case when we have information about the ratio  $L_{\mathrm{max}} / L$ .
Theorem 5.4. Choose  $r \in [0,1]$ , and let  $\mu_{\omega, \alpha}^r \coloneqq \frac{rd}{(1 - r)K_\omega + rK_\alpha}$ . In view of Theorem 5.1, the values  $p$  and  $\tau$  given by (63) and (58), respectively, minimize  $\mathfrak{m}_{\mathrm{new}}^r$  from (10). This choice leads to the following number of communication rounds:
$$
\begin{array}{l} T ^ {\text {o p t i m i s t i c}} = \widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L \max  \{1 , \mu_ {\omega , \alpha} ^ {r} \}}{\alpha \mu}}, \sqrt {\frac {L ^ {2 / 3} L _ {\max } ^ {1 / 3} (\omega + 1)}{\alpha n ^ {1 / 3} \mu}} \right., \sqrt {\frac {L ^ {1 / 2} L _ {\max } ^ {1 / 2} (\omega + 1) ^ {3 / 2}}{\sqrt {\alpha n} \mu}}, \right. \tag {14} \\ \left. \sqrt {\frac {L _ {\operatorname* {m a x}} \omega \operatorname* {m a x} \{\omega + 1 , \mu_ {\omega , \alpha} ^ {r} \}}{n \mu}}, \frac {1}{\alpha}, (\omega + 1), \mu_ {\omega , \alpha} ^ {r} \right\}\Bigg). \\ \end{array}
$$
The total communication complexity thus equals  $\mathfrak{m}_{\mathrm{optimistic}}^r = \widetilde{\Theta}\left((1 - r)K_\omega +rK_\alpha\right)T_{\mathrm{optimistic}} + d$ .
Note that information about  $L_{\mathrm{max}} / L$  leads to a better rate that in Theorem 5.2.
Remark 5.5. To simplify the rate (14), let  $\mathcal{C}_i^{D,\cdot}$  be the Rand  $K$  sparsifier and consider the case when the s2w communication is not slower than the w2s communication, i.e.,  $r\leq 1 / 2$ . Then  $T^{\mathrm{optimistic}} = \widetilde{\Theta}\left(\max \left\{\sqrt{\frac{L\max\{1,r(\omega + 1)\}}{\alpha\mu}},\sqrt{\frac{L^{2 / 3}L_{\max}^{1 / 3}(\omega + 1)}{\alpha n^{1 / 3}\mu}},\sqrt{\frac{L^{1 / 2}L_{\max}^{1 / 2}(\omega + 1)^{3 / 2}}{\sqrt{\alpha n}\mu}},\sqrt{\frac{L_{\max}\omega(\omega + 1)}{n\mu}},\frac{1}{\alpha},(\omega +1)\right\}\right)$ . Indeed, this follows from  $r\leq 1 / 2$  and the fact that  $\omega +1 = d / K_{\omega}$  for the Rand  $K$  compressor:  $\mu_{\omega ,\alpha}^{r}\coloneqq \frac{rd}{(1 - r)K_{\omega} + rK_{\alpha}}\leq \frac{r}{1 - r}\times \frac{d}{K_{\omega}}\leq 2r~(\omega +1)$ .
6 Theoretical Comparison with Previous State of the Art
We now show that the communication complexity of 2Direction is always no worse than that of EF21 + DIANA and AGD. Crucially, in some regimes, it can be substantially better. Furthermore, we show that if the s2w communication cost is zero (i.e., if  $r = 0$ ), the 2Direction obtains the same communication complexity as ADIANA (Li et al., 2020) (see Section S).
Comparison with EF21 + DIANA. The EF21-P + DIANA method has the communication complexities that equal
$$
\mathfrak {m} _ {\mathrm {E F 2 1 - P + D I A N A}} ^ {\mathrm {w 2 s}} = \widetilde {\Theta} \left(K _ {\omega} \times T ^ {\mathrm {E F 2 1 - P + D I A N A}} + d\right) \quad \text {a n d} \quad \mathfrak {m} _ {\mathrm {E F 2 1 - P + D I A N A}} ^ {\mathrm {s 2 w}} = \widetilde {\Theta} \left(K _ {\alpha} \times T ^ {\mathrm {E F 2 1 - P + D I A N A}} + d\right),
$$
It is sufficient to assume that  $\omega + 1 = \Theta\left(d / K_{\omega}\right)$ .
It is sufficient to assume that  $\omega + 1 = \Theta \left( \frac{d}{K_{\omega}} \right)$ .
where  $K_{\omega}$  and  $K_{\alpha}$  are the expected densities of  $\mathcal{C}_i^D$  and  $\mathcal{C}^P$  in Algorithm 4. The last term  $d$  comes from the fact that EF21-P + DIANA sends non-compressed vectors in the initialization phase. Let us define  $K_{\omega, \alpha}^{r} := (1 - r)K_{\omega} + rK_{\alpha}$ . Therefore, the total communication complexity equals
$$
\mathfrak {m} _ {\mathrm {E F 2 1 - P + D I A N A}} ^ {r} = \widetilde {\Theta} \left(K _ {\omega , \alpha} ^ {r} \left(\frac {L}{\alpha \mu} + \frac {\omega L _ {\max }}{n \mu} + \omega\right) + d\right). \tag {15}
$$
Theorem 5.2 ensures that the total communication complexity of 2Direction is
$$
\mathfrak {m} _ {\text {r e a l i s t i c}} ^ {r} = \widetilde {\Theta} \left(K _ {\omega , \alpha} ^ {r} \left(\sqrt {\frac {L \operatorname* {m a x} \left\{\omega + 1 , \mu_ {\omega , \alpha} ^ {r} \right\}}{\alpha \mu}} + \sqrt {\frac {L _ {\operatorname* {m a x}} \omega \operatorname* {m a x} \left\{\omega + 1 , \mu_ {\omega , \alpha} ^ {r} \right\}}{n \mu}} + \frac {1}{\alpha} + \omega + \mu_ {\omega , \alpha} ^ {r}\right) + d\right). \tag {16}
$$
One can see that (16) is an accelerated rate; it has much better dependence on the condition numbers  $L / \mu$  and  $L_{\mathrm{max}} / \mu$ . In Section E.8, we prove the following simple theorem, which means that 2Direction is not worse than EF21-P + DIANA.
Theorem 6.1. For all  $r \in [0,1]$ ,  $\mathfrak{m}_{\mathrm{realistic}}^r = \tilde{\mathcal{O}}\left(\mathfrak{m}_{\mathrm{EF21 - P + DIANA}}^r\right)$ .
Comparison with AGD. To compare the abstract complexity (16) with the non-abstract complexity  $\widetilde{\Theta}(d\sqrt{L / \mu})$ , we take the Rand  $K$  and Top  $K$  compressors in Algorithm 1.
Theorem 6.2. For all  $r \in [0,1]$  and for all  $K \in [d]$ , let us take the RandK and TopK compressors with the parameters (expected densities)  $i) K_{\omega} = K$  and  $K_{\alpha} = \min \{\lceil 1 - r / rK\rceil ,d\}$  for  $r \in [0,1 / 2]$ ,  $ii) K_{\omega} = \min \{\lceil r / 1 - rK\rceil ,d\}$  and  $K_{\alpha} = K$  for  $r \in (1 / 2,1]$ . Then we have  $\mathfrak{m}_{\mathrm{realistic}}^r = \widetilde{\mathcal{O}} (\mathfrak{m}_{\mathrm{AGD}})$ .
Theorem 6.2 states that the total communication complexity of 2Direction is not worse than that of AGD. It can be strictly better in the regimes when  $\alpha > K / d$  (for  $\mathrm{Top}K$ ),  $L_{\max} < nL$ , and  $n > 1$ .
7 Proof Sketch
After we settle on the final version of Algorithm 1, the proof technique is pretty standard at the beginning (Li et al., 2020; Tyurin and Richtárik, 2023; Gruntkowska et al., 2023). We proved a descent lemma (Lemma E.4) and lemmas that control the convergences of auxiliary sequences (Lemmas E.5, E.6, E.7, E.8). Using these lemmas, we construct the Lyapunov function (45) with the coefficients  $\kappa$ ,  $\rho$ ,  $\lambda$  and  $\nu_{t} \geq 0$  for all  $t \geq 0$ .
One of the main problems was to find appropriate  $\bar{L}$ ,  $\kappa$ ,  $\rho$ ,  $\lambda$  and  $\nu_{t} \geq 0$  such that we get a convergence. In more details, to get a converge it is sufficient to find  $\bar{L}$ ,  $\kappa$ ,  $\rho$ ,  $\lambda$  and  $\nu_{t} \geq 0$  such that (46), (47) and (48) hold. Using the symbolic computation SymPy library (Meurer et al., 2017), we found appropriate  $\kappa$ ,  $\rho$ ,  $\lambda$  and  $\nu_{t} \geq 0$  ((90), (88), (82), (83)) such that the inequalities hold. But that is not all. To get a convergence, we also found the bounds on the parameter  $\bar{L}$ , which essentially describe the speed of the convergence. In raw form, using symbolic computations, we obtained a huge number of bounds on  $\bar{L}$  (see Sections I, J, L, N). It was clear that most of them are redundant and it was required to find the essential ones. After a close look at the bounds on  $\bar{L}$ , we found out that it is sufficient to require (44) to ensure that all inequalities from Sections I, J, L and N hold (see Section O for details).
Figure 1: Roadmap to our resolution of Main Problem.
8 Limitations and Future Work
In contrast to Algorithm 4 (EF21+DIANA), in which the server always broadcasts compressed vectors, in our Algorithm 1 (2Direction) the server needs to broadcast non-compressed vectors with
probability  $p$  (see Line 20). While in Section 6 we explain that this does not have an adverse effect on the theoretical communication complexity since  $p$  is small, one may wonder whether it might be possible to achieve the same (or better) bounds as ours without having to resort to intermittent non-compressed broadcasts. This remains an open problem; possibly a challenging one. Another limitation comes from the fact that 2Direction requires more iterations than AGD in general (this is the case of all methods that reduce communication complexity). While, indeed, (10) can be higher than  $\widetilde{\Theta}(\sqrt{L / \mu})$ , the total communication complexity of 2Direction is not worse than that of AGD.
This work of P. Richtárik and A. Tyurin was supported by the KAUST Baseline Research Scheme (KAUST BRF) and the KAUST Extreme Computing Research Center (KAUST ECRC), and the work of P. Richtárik was supported by the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI).
Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. (2017). QSGD: Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems (NIPS), pages 1709-1720.
Beznosikov, A., Horvath, S., Richtárik, P., and Safaryan, M. (2020). On biased compression for distributed learning. arXiv preprint arXiv:2002.12410.
Chang, C.-C. and Lin, C.-J. (2011). LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):1-27.
Condat, L. and Richtárik, P. (2022). Murana: A generic framework for stochastic variance-reduced optimization. In Mathematical and Scientific Machine Learning, pages 81-96. PMLR.
Gorbunov, E., Burlachenko, K., Li, Z., and Richtárik, P. (2021). MARINA: Faster non-convex distributed learning with compression. In 38th International Conference on Machine Learning.
Gorbunov, E., Hanzely, F., and Richtárik, P. (2020). A unified theory of SGD: Variance reduction, sampling, quantization and coordinate descent. In International Conference on Artificial Intelligence and Statistics, pages 680-690. PMLR.
Gruntkowska, K., Tyurin, A., and Richtárik, P. (2023). EF21-P and friends: Improved theoretical communication complexity for distributed optimization with bidirectional compression. In International Conference on Machine Learning.
Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Dennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. (2021). Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1-2):1-210.
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., and Bacon, D. (2016). Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492.
Kovalev, D., Horváth, S., and Richtárik, P. (2020). Don't jump through hoops and remove those loops: Svrg and katyusha are better without the outer loop. In Algorithmic Learning Theory, pages 451-467. PMLR.
Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto.
Lan, G. (2020). First-order and stochastic optimization methods for machine learning. Springer.
Lan, G., Li, Z., and Zhou, Y. (2019). A unified variance-reduced accelerated gradient method for convex optimization. Advances in Neural Information Processing Systems, 32.
Li, Z., Kovalev, D., Qian, X., and Richtárik, P. (2020). Acceleration for compressed gradient descent in distributed and federated optimization. In International Conference on Machine Learning.