SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and taking  $\gamma \leq \frac{1}{48(M + 1)(P + 1)(1 + \chi_1)}$  works. We verify that with  $\gamma$  defined as such, (65) is also negative. Finally, we upper bound (66) with  $3L\gamma^2\left(\sigma^2 + 3(M + 1 + 4\chi_1)\zeta^2\right)$ . As we have:
$$
\mathbb {E} \left[ \phi_ {3} (T, x _ {T}) - \phi_ {3} \left(0, x _ {0}\right) \right] = \int_ {0} ^ {T} \left(A _ {t} ^ {\prime} d _ {f} \left(\bar {x} _ {t}, x ^ {*}\right) + B _ {t} ^ {\prime} \| \pi x _ {t} \| ^ {2} + A _ {t} \mathbb {E} [ \Delta f ] + B _ {t} \mathbb {E} [ \Delta_ {\pi} ]\right) d t \tag {68}
$$
we note that if  $\gamma \leq \frac{c}{L(\chi_1 + 1)}$  for some constant  $c > 0$  which depends on  $M, P$ , we will get:
$$
\frac {\gamma}{4} \int_ {0} ^ {T} \mathbb {E} [ \| \nabla f (\bar {x} _ {t}) \| ^ {2} ] d t \leq \frac {L}{n} \| \pi x _ {0} \| ^ {2} + d _ {f} \left(x _ {0}, x ^ {*}\right) + \mathcal {O} \left(L T \gamma^ {2} \left(\sigma^ {2} + (1 + \chi_ {1}) \zeta^ {2}\right)\right) \tag {69}
$$
which also writes:
$$
\frac {1}{T} \int_ {0} ^ {T} \mathbb {E} [ \| \nabla f (\bar {x} _ {t}) \| ^ {2} ] d t \leq \frac {4}{\gamma T} \left(f \left(x _ {0}\right) - f \left(x ^ {*}\right)\right) + \mathcal {O} \left(L \gamma \left(\sigma^ {2} + (1 + \chi_ {1}) \zeta^ {2}\right)\right) \tag {70}
$$
Non-convex case, with  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  .We have:
$$
\phi_ {4} (t, x) = A _ {t} d _ {f} (\bar {x}, x ^ {*}) + B _ {t} \| \pi x \| ^ {2} + \tilde {B} _ {t} \| \tilde {x} \| _ {\Lambda^ {+}} ^ {2}
$$
Here, we pick  $\alpha = \frac{1}{2}$ ,  $\eta = \frac{1}{2\sqrt{\chi_1\chi_2}}$ ,  $A_t = 1$ ,  $B_t = \frac{L}{n} A_t$ ,  $B_t = \chi_1\tilde{B}_t$  and an identical reasoning to the convex setting allows to say we can find a constant  $c > 0$  such that if  $\gamma \leq \frac{c}{L(1 + \sqrt{\chi_1\chi_2})}$ , then:
$$
\frac {1}{T} \int_ {0} ^ {T} \mathbb {E} [ \| \nabla f (\bar {x} _ {t}) \| ^ {2} ] d t = \mathcal {O} \left(\frac {1}{\gamma T} \left(f \left(x _ {0}\right) - f \left(x ^ {*}\right)\right) + L \gamma \left(\sigma^ {2} + \left(1 + \sqrt {\chi_ {1} \chi_ {2}}\right) \zeta^ {2}\right)\right) \tag {71}
$$
C.3 Optimizing the step-size
In this part, we follow [40, 21] and optimize the step-size a posteriori. We set  $\chi = \chi_{1}$  for the non-accelerated setting and  $\chi = \sqrt{\chi_1\chi_2}$  with  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ .
Strongly-convex cases: From (51) and (63), we can write that, for  $\gamma \leq \frac{1}{16L(1 + \chi)}$  and initializing  $x_0$  such that  $\pi x_0 = 0$ , we have:
$$
\mathbb {E} \| \bar {x} _ {t} - x ^ {*} \| ^ {2} = \mathcal {O} \left(\| \bar {x} _ {0} - x ^ {*} \| ^ {2} e ^ {- \gamma \mu t} + \frac {\gamma}{\mu} \left(\sigma^ {2} + \zeta^ {2} (1 + \chi)\right)\right) \tag {72}
$$
Then, taking the proof of [40] and adapting the threshold, we consider two cases (with  $r_0 \triangleq \| \bar{x}_0 - x^* \|^2$ ):
- if  $\frac{1}{16L(1 + \chi)} \geq \frac{\log(\max\{2, \mu^2 r_0 T / \sigma^2\})}{\mu T}$ , then we set  $\gamma = \frac{\log(\max\{2, \mu^2 r_0 T / \sigma^2\})}{\mu T}$ .
In this case, (72) gives:
$$
\mathbb {E} \| \bar {x} _ {T} - x ^ {*} \| ^ {2} = \tilde {\mathcal {O}} \left(\frac {1}{\mu^ {2} T} \left(\sigma^ {2} + \zeta^ {2} (1 + \chi)\right)\right) \tag {73}
$$
- if  $\frac{1}{16L(1 + \chi)} < \frac{\log(\max\{2, \mu^2 r_0 T / \sigma^2\})}{\mu T}$ , then we set  $\gamma = \frac{1}{16L(1 + \chi)}$ .
Then, (72) gives:
$$
\begin{array}{l} \mathbb {E} \| \bar {x} _ {T} - x ^ {*} \| ^ {2} = \mathcal {O} \left(r _ {0} e ^ {- \frac {\mu T}{1 6 L (1 + \chi)}} + \frac {1}{\mu} \frac {1}{1 6 L (1 + \chi)} \left(\sigma^ {2} + \zeta^ {2} (1 + \chi)\right)\right) (74) \\ = \tilde {\mathcal {O}} \left(r _ {0} e ^ {- \frac {\mu T}{1 6 L (1 + \chi)}} + \frac {1}{\mu^ {2} T} \left(\sigma^ {2} + \zeta^ {2} (1 + \chi)\right)\right) (75) \\ \end{array}
$$
Non-convex cases: From (70) and (71), we can write that, for some constant  $c > 0$  depending on  $M, P$  such that  $\gamma \leq \frac{c}{L(1 + \chi)}$ , we have:
$$
\frac {1}{T} \int_ {0} ^ {T} \mathbb {E} [ \| \nabla f (\bar {x} _ {t}) \| ^ {2} ] d t = \mathcal {O} \left(\frac {1}{\gamma T} \left(f \left(x _ {0}\right) - f \left(x ^ {*}\right)\right) + L \gamma \left(\sigma^ {2} + (1 + \chi) \zeta^ {2}\right)\right) \tag {76}
$$
Then, taking the proof of Lemma 17 in [21] and adapting the threshold, we consider two cases (with  $f_0 \triangleq f(x_0) - f(x^*)$ ):
- if  $\frac{c}{L(1 + \chi)} < \left(\frac{f_0}{TL(\sigma^2 + (1 + \chi)\zeta^2)}\right)^{1/2}$ , then we take  $\gamma = \frac{c}{L(1 + \chi)}$ , giving:
$$
\begin{array}{l} \frac {1}{T} \int_ {0} ^ {T} \mathbb {E} [ \| \nabla f (\bar {x} _ {t}) \| ^ {2} ] d t = \mathcal {O} \left(\frac {L (1 + \chi)}{T} f _ {0} + L \left(\frac {f _ {0}}{T L \left(\sigma^ {2} + (1 + \chi) \zeta^ {2}\right)}\right) ^ {1 / 2} \left(\sigma^ {2} + (1 + \chi) \zeta^ {2}\right)\right) (77) \\ = \mathcal {O} \left(\frac {L (1 + \chi)}{T} f _ {0} + \sqrt {\frac {L f _ {0}}{T} \left(\sigma^ {2} + (1 + \chi) \zeta^ {2}\right)}\right) (78) \\ \end{array}
$$
- if  $\frac{c}{L(1 + \chi)} \geq \left(\frac{f_0}{TL(\sigma^2 + (1 + \chi)\zeta^2)}\right)^{1/2}$ , then we take  $\gamma = \left(\frac{f_0}{TL(\sigma^2 + (1 + \chi)\zeta^2)}\right)^{1/2}$ , giving:
$$
\begin{array}{l} \frac {1}{T} \int_ {0} ^ {T} \mathbb {E} [ \| \nabla f (\bar {x} _ {t}) \| ^ {2} ] d t = \mathcal {O} \left(\frac {f _ {0}}{T} \left(\frac {T L \left(\sigma^ {2} + (1 + \chi) \zeta^ {2}\right)}{f _ {0}}\right) ^ {1 / 2} + \sqrt {\frac {L f _ {0}}{T} \left(\sigma^ {2} + (1 + \chi) \zeta^ {2}\right)}\right) (79) \\ = \mathcal {O} \left(\sqrt {\frac {L f _ {0}}{T} \left(\sigma^ {2} + (1 + \chi) \zeta^ {2}\right)}\right) (80) \\ \end{array}
$$
D Comparison with accelerated synchronous methods
By definition of  $\Lambda$  (3.1), our communication complexity (the expected number of communications) is simply given by  $\frac{\mathrm{Tr}(\Lambda)}{2}$  per time unit. As discussed in Sec. 3.5, our goal is to replicate the behaviour of accelerated synchronous methods such as DeTAG [31], MSDA [37] and OPAPC [23] by communicating sufficiently so that the graph connectivity does not impact the time to converge, leading to the condition  $\sqrt{\chi_1[\Lambda]\chi_2[\Lambda]} = \mathcal{O}(1)$ .
Now, let us consider a gossip matrix  $W$  as in [31, 37, 23] (i.e.,  $W$  is symmetric doubly stochastic) and its Laplacian  $\mathcal{L} = I_n - W$ . Then, using  $\Lambda = \sqrt{\chi_1[\mathcal{L}]\chi_2[\mathcal{L}]}\mathcal{L}$  is sufficient for having  $\sqrt{\chi_1[\Lambda]\chi_2[\Lambda]} = \mathcal{O}(1)$ .
- Synchronous methods: between two rounds of computations ("steps"), the number of communication edges used is  $\frac{|\mathcal{E}|}{\sqrt{1 - \theta}}$  with  $\theta = \max \{| \lambda_2 |, |\lambda_n | \}$  the eigenvalues of  $W$ .
- Ours: the number of communication edges used per time unit for our method is  $\frac{\operatorname{Tr}(\Lambda)}{2} = \frac{1}{2}\sqrt{\chi_1[\mathcal{L}]\chi_2[\mathcal{L}]}\operatorname{Tr}(\mathcal{L})$ .
As, in [31, 37, 23], each communication edge is used at the same rate, we can apply Lemma 3.3 of [34] stating:  $\sqrt{\chi_1[\mathcal{L}]\chi_2[\mathcal{L}]}\mathrm{Tr}(\mathcal{L})\leq \sqrt{\|\mathcal{L}\|_{\chi_1(n - 1)}|\mathcal{E}|}$ . We have:
-  $W$  is stochastic:  $\|\mathcal{L}\| \leq 2$ .
- the graph is connected:  $n - 1 \leq |\mathcal{E}|$ .
definition of  $\chi_{1}$  and  $\theta: 1 - \theta \leq \frac{1}{\chi_1[\mathcal{L}]}$
Thus,  $\sqrt{\chi_1[\mathcal{L}]\chi_2[\mathcal{L}]}\mathrm{Tr}(\mathcal{L})\leq \frac{\sqrt{2}|\mathcal{E}|}{\sqrt{1 - \theta}}$ , which proves that our communication complexity per time unit is at least as good as any accelerated synchronous method.
E.1 Graph topologies
Figure 6: The three types of graph topology implemented. From left to right: complete, exponential, cycle, all with 16 nodes. From left to right, the approximate values of  $(\chi_1,\chi_2)$  with a communication rate of "1 p2p comm./ $\nabla$  comp." for each worker are:  $(1,1)$ ,  $(2,1)$ ,  $(13,1)$ .
Fig. 6 displays an example of each of the three graph topologies implemented. The exponential graph follows the architecture described in [28, 2]. Note the discrepancy between the values of  $\chi_{1}$  and  $\chi_{2}$  for the cycle graph, highlighting the advantage of using  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  in the asynchronous setting (to lower the complexity from  $\chi_{1}$  to  $\sqrt{\chi_1\chi_2}$ ).
E.2 Uniform neighbor selection check
Figure 7: Heat-map of the communication history (showed through a weighted adjacency matrix) during the asynchronous training on CIFAR10 with 32 workers. We display the results for the complete graph (left), exponential (centre) and ring (right) graph.
Our asynchronous algorithm acts as follows: to reduce latency, the first two workers (i.e., GPUs) in the whole pool that declare they are ready to communicate (i.e., finished their previous communication and have to perform more before the next gradient step) are paired together for a p2p communication if they are neighbors in the connectivity network. During training, we registered the history of the pairwise communications that happened. Fig. 7 displays the heat-map of the adjacency matrix, confirming that our assumption of "uniform pairing among neighbors" (used to compute the values of  $\chi_{1},\chi_{2}$ ) seems to be sound.