SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(their Accelerated Gossip procedure) between rounds of gradient computations by using an inner loop inside their main loop (the one counting "steps"), so that the graph connectivity do not impact the total number of "steps" necessary to reach  $\epsilon$ -precision. As Prop. 3.6 shows, the quantity  $1 + \sqrt{\chi_1[\Lambda]\chi_2[\Lambda]}$  is a factor in our convergence rate.  $\Lambda$  containing the information of both the topology  $\mathcal{E}$  and the edge communication rates  $\lambda_{ij}$ , this is analogous to saying  $\sqrt{\chi_1[\Lambda]\chi_2[\Lambda]} = \mathcal{O}(1)$  for our method (i.e., the graph connectivity does not impact the time to converge), which, given the graph's topology, dictates the communication rate, see Appendix D for more details. Tab. 2 compares the subsequent communication rates with synchronous methods.
Now, we experimentally compare  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  to a synchronous baseline All-Reduce SGD (AR-SGD, see [26]) and an asynchronous baseline using randomized pairwise communications (a variant of AD-PSGD [28], traditionally used in state-of-the-art decentralized asynchronous training of DNNs). In our case, the asynchronous baseline corresponds to the dynamic Eq. (6). Our approach is standard: we empirically study the decentralized training behavior of our asynchronous algorithm by training ResNets [17] for image recognition. Following [2], we pick a ResNet18 for CIFAR-10 [24] and ResNet50 for ImageNet [11]. To investigate how our method scales with the number of workers, we run multiple experiments using up to 64 NVIDIA A100 GPUs in a cluster with 8 A100 GPUs per node using an Omni-PAth interconnection network at  $100\mathrm{Gb / s}$ , and set one worker per GPU.
Hyper-parameters. Training a DNN using multiple workers on a cluster requires several adaptations compared to the standard setting. As the effective batch-size grows linearly with the
Table 3: Training times on CIFAR10 (± 6s).
n	4	8	16	32	64
Ours	t (min)	20.9	10.5	5.2	2.7	1.5
AR	t (min)	21.9	11.1	6.6	3.2	1.8
number of workers  $n$ , we use the learning-rate schedule for large batch training of [16] in all our experiments. Following [30], we fixed the local batch size to 128 on both CIFAR-10 and ImageNet.
(b)
Figure 3: (a) Training loss for CIFAR10 with minibatch size 128 on the complete graph, w/o  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ . As the number of worker increases, the loss degrades, especially for  $n = 64$ . (b) Focus on the training loss for the complete graph of size  $n = 64$ , w/o  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ . As the rate of communication increases, the gap with All-Reduce decreases. With 2 com/ $\nabla$ , a test accuracy of  $94.6 \pm 0.04$  is reached.
Our goal being to divide the compute load between the  $n$  workers, all methods access the same total amount of data samples, regardless of the number of local steps. On CIFAR-10 and ImageNet, this
Algorithm 1: This algorithm block describes our implementation of our Asynchronous algorithm with  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  on each local machine. p2p comm. and  $\nabla$  comp. are run independently in parallel.  
Input: On each machine  $i\in \{1,\dots,n\}$ , gradient oracle  $\nabla f_{i}$ , parameters  $\eta, \alpha, \tilde{\alpha}, \gamma, T$ .  
Initialize on each machine  $i\in \{1,\dots,n\}$ :  
Initialize  $x^{i}, \tilde{x}^{i}\gets x^{i}, t^{i}\gets 0$  and put  $x^{i}, \tilde{x}^{i}, t^{i}$  in shared memory;  
Synchronize the clocks of all machines;  
In parallel on workers  $i\in \{1,\dots,n\}$ , while  $t < T$ , continuously do:  
In one thread on worker  $i$  continuously do:  
 $t\gets \text{clock()}$ ;  
Sample a batch of data via  $\xi_{i}\sim \Xi$ ;  
 $g_{i}\gets \nabla F_{i}(x_{i},\xi_{i})$ ; // Compute gradients  
 $\begin{pmatrix} x^{i}\\ \tilde{x}^{i} \end{pmatrix} \gets \exp \left((t - t^{i})\begin{pmatrix} -\eta & \eta \\ \eta & -\eta \end{pmatrix}\begin{pmatrix} x^{i}\\ \tilde{x}^{i} \end{pmatrix}; \end{pmatrix}$ $x^{i}\gets x^{i} - \gamma g_{i}$ ; // Apply A²CiD²  
 $\tilde{x}^{i}\gets \tilde{x}^{i} - \gamma g_{i}$ ; // Take the grad step  
 $t^{i}\gets t$ ;  
In one thread on worker  $i$  continuously do:  
 $t\gets \text{clock()}$ ;  
Find available worker  $j$ ; // Synchronize workers  $i$  and  $j$ $m_{ij}\gets (x^{i} - x^{j})$ ; // Send  $x^{i}$  to  $j$  and receive  $x^{j}$  from  $j$ $\begin{pmatrix} x^{i}\\ \tilde{x}^{i} \end{pmatrix} \gets \exp \left((t - t^{i})\begin{pmatrix} -\eta & \eta \\ \eta & -\eta \end{pmatrix}\begin{pmatrix} x^{i}\\ \tilde{x}^{i} \end{pmatrix}; \end{pmatrix}$  // Apply A²CiD²  
 $x^{i}\gets x^{i} - \alpha m_{ij}$ ; // p2p averaging  
 $\tilde{x}^{i}\gets \tilde{x}^{i} - \tilde{\alpha} m_{ij}$ ;  
 $t^{i}\gets t$ ;  
return  $(x_T^i)_{1\leq i\leq n}$ .
number is set to 300 and 90 epochs respectively, following standard practice [22]. To circumvent the fuzziness of the notion of epoch in the asynchronous decentralized setting, we do not "split the dataset and re-shuffle it among workers at each epoch" as done with our standard All-Reduce baseline. Rather, we give access to the whole dataset to all workers, each one shuffling it with a different random seed. We use SGD with a base learning rate of 0.1, a momentum value set at 0.9 and  $5 \times 10^{-4}$  for weight decay. As advocated in [16], we do not apply weight decay on the learnable batch-norm coefficients. For ImageNet training with the SGD baseline, we decay the learning-rate by a factor of 10 at epochs 30, 60, 80 (epochs 50, 75 for CIFAR-10), and apply an analogous decay schedule with our asynchronous decentralized methods. All of our neural network parameters are initialized with the default Pytorch settings, and one All-Reduce averaging is performed before and after the training to ensure consensus at initialization and before testing. For our continuous momentum, we also need to set the parameters  $\eta$ ,  $\tilde{\alpha}$ . For all our experiments, we use the values given by Prop. 3.6. As advocated, the asynchronous baseline corresponds to the setting without acceleration, i.e. with  $\eta = 0$  and  $\alpha = \tilde{\alpha} = \frac{1}{2}$ , whereas using  $\mathbf{A}^2\mathbf{C}\mathbf{D}^2$  leads to consider  $\eta = \frac{1}{2\sqrt{\chi_1\chi_2}}$ ,  $\alpha = \frac{1}{2}$ ,  $\tilde{\alpha} = \frac{1}{2}\sqrt{\frac{\chi_1}{\chi_2}}$ , where  $\chi_1$ ,  $\chi_2$  are set to their theoretical value given by (2), (3) depending on the communication rate and graph's topology, assuming that each worker chose their peers uniformly among their neighbors (we verify empirically that it is the case in practice, see Appendix E.2).
Practical implementation of the dynamic. The dynamic studied in Eq. (4) is a model displaying many of the properties sought after in practice. In our implementation, described in Algo. 1, each worker  $i$  has indeed two independent processes and the DNN parameters and momentum variable  $\{x^i,\tilde{x}^i\}$  are locally stored such that both processes can update them at any time. One process continuously performs gradient steps, while the other updates  $\{x^i,\tilde{x}^i\}$  via peer-to-peer averaging. The gradient process maximizes its throughput by computing forward and backward passes back to back. Contrary to All-Reduce based methods that require an increasing number of communications with the growing number of workers, inevitably leading to an increasing time between two rounds of computations,
we study the case where each worker has a fixed communication rate, given as hyperparameter in our implementation. We implement 3 different graph topologies: complete, ring, and exponential [28, 2], see Appendix E.1 for details. To emulate the P.P.Ps for the communications, each worker samples a random number of p2p averaging to perform between each gradient computation, following a Poisson law using the communication rate as mean. To minimize idle time of the communication process, workers are paired with one of their neighbors in a "First In First Out" manner in an availability queue (a worker is available when it finished its previous averaging and still has some to do before the next gradient step). To implement this, we use a central coordinator to store the availability queues and the graph topology (this
is lightweight in a cluster: the coordinator only exchanges integers with the workers), but it could be done in different ways, e.g. by pinging each other at high frequency. As we assumed a unit time for the gradient process in our analysis, and that real time is used in our algorithm to apply our  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  momentum (see Algo. 1), we maintain a running average measure of the duration of the previous gradient steps to normalize time.
Figure 4: Training loss for CIFAR10 using a minibatch size of 128. We display the training loss with up to 64 workers, w/ and w/o  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ , on the challenging ring graph.
4.2 Evaluation on large scale datasets
CIFAR10. This simple benchmark allows to understand the benefits of our method in a well-controlled environment. Tab. 4 reports our numerical accuracy on the test set of CIFAR10, with a standard deviation calculated over 3 runs. Three scenarios are considered: a complete, an exponential and a ring graph. In Fig. 3 (a), we observe that with the asynchronous baseline on the complete graph, the more workers, the more the training loss degrades. Fig. 3 (b) hints that it is in part due to an insufficient communication rate, as increasing it allows to lower the loss and close the gap with the All-Reduce baseline. However, this is not the only causative factor as Tab. 4 indicates that accuracy generally degrades as the number of
workers increases even for AR-SGD, which is expected for large batch sizes. Surprisingly, even with a worse training loss for  $n = 64$ , the asynchronous baseline still leads to better generalization than
Table 5: Accuracy on ImageNet for a batch-size of 128. We compared a vanilla asynchronous pairwise gossip approach with and without  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  , demonstrating the improvement of our method. We also varied the communication rate.
#Workers	#com/#grad	16	32	64
AR-SGD baseline	-	75.5	75.2	74.5
Complete graph
Async. baseline	1	74.6	73.8	71.3
Ring graph
Async. baseline	1	74.8	71.6	64.1
A²CiD²	1	74.7	73.4	68.0
Async. baseline	2	74.8	73.7	68.2
A²CiD²	2	75.3	74.4	71.4
Table 4: Accuracy of our method on CIFAR10 for a 128 batchsize with an equal number of pairwise communications and gradient computations per worker. We compared a vanilla asynchronous pairwise gossip approach with and without  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ , demonstrating the improvement of our method.
#Workers	4	8	16	32	64
AR-SGD baseline	94.5±0.1	94.4±0.1	94.5±0.2	93.7±0.3	92.8±0.2
Complete graph
Async. baseline	94.93±0.11	94.91±0.07	94.86±0.01	94.55±0.01	93.38±0.21
Exponential graph
Async. baseline	95.07±0.01	94.89±0.01	94.82±0.06	94.44±0.02	93.41±0.02
A²CiD²	95.17±0.04	95.04±0.01	94.87±0.02	94.56±0.01	93.47±0.01
Ring graph
Async. baseline	95.02±0.06	95.01±0.01	95.00±0.01	93.95±0.11	91.90±0.10
A²CiD²	94.95±0.02	95.01±0.10	95.03±0.01	94.61±0.02	93.08±0.20
AR-SGD, and consistently improves the test accuracy across all tested values of  $n$ . The communication rate being identified as a critical factor at large scale, we tested our continuous momentum on the ring graph, each worker performing one p2p averaging for each gradient step. Fig. 4 shows that incorporating  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  leads to a significantly better training dynamic for a large number of workers, which translates into better performances at test time as shown in Tab. 4.
(b)
Figure 5: (a) Training loss for ImageNet using 128 batch size, with an equal number of communications and computations per worker. We display the training loss for various numbers of workers (up to 64), using  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ , for the ring graph. (b) Comparison of consensus distances when  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  is applied versus doubling the rate of communications on the rign graph with 64 workers: applying  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  has the same effect as doubling communications.
ImageNet. For validating our method in a real-life environment, we consider the large-scale ImageNet dataset. Tab. 6 confirms the advantage of asynchronous methods by allocating less compute to the slowest workers, leading to faster training times. Tab. 5 reports our accuracy for
the complete and ring graphs. As  $\chi_{1} = \chi_{2}$  for the complete graph, we simply run our baseline asynchronous method for reference. The case of the ring graph is much more challenging: for  $n = 64$  workers, the accuracy drops by  $10\%$  compared to the synchronous baseline given by AR-SGD. Systematically, with  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ , the final accuracy increases: up to  $4\%$  absolute percent in the difficult  $n = 64$  setting. This is corroborated by Fig. 5, which indicates that incorporating  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  significantly improves the training dynamic on ImageNet. However, for reducing the gap with the AR-SGD baseline, it will be necessary to increase the communication rate as discussed next.
Table 6: Statistics of runs on Imagenet with 64 workers (for ours, on the exponential graph).
Method	t (min)	# ∇ slowest worker	# ∇ fastest worker
AR-SGD	1.7 102	14k	14k
Baseline (ours)	1.5 102	13k	14k
A²CiD²(ours)	1.5 102	13k	14k
Consensus improvement. The bottom of Tab. 5, as well as Fig. 5 (b) study the virtual acceleration thanks to  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ . Not only increasing communications combined with  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  allows to obtain competitive performance, but Fig. 1 shows that doubling the rate of communication has an identical effect on the training loss than adding  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ . This is verified in Fig. 5 (b) by tracking the consensus distance between workers:  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  significantly reduces it, which validates the results of Sec. 3.4.
In this work, we confirmed that the communication rate is a key performance factor to successfully train DNNs at large scale with decentralized asynchronous methods. We introduced  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ , a continuous momentum which only adds a minor local memory overhead while allowing to mitigate this need. We demonstrated, both theoretically and empirically, that  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  substantially improves performances, especially on challenging network topologies. As we only focused on data parallel methods for training Deep Neural Networks in a cluster environment, in a future work, we would like to extend our empirical study to more heterogeneous compute and data sources, as our theory could encompass local SGD methods [39] and data heterogeneity inherent in Federated Learning [33].
EO, AN, and EB's work was supported by Project ANR-21-CE23-0030 ADONIS and EMERG-ADONIS from Alliance SU. This work was granted access to the HPC/AI resources of IDRIS under the allocation AD011013743 made by GENCI. EB and AN acknowledge funding and support from NSERC Discovery Grant RGPIN- 2021-04104, FRQNT New Scholar, and resources from Compute Canada and Calcul Quebec. In addition, the authors would like to thank Olexa Bilaniuk and Louis Fournier for their helpful insights regarding our code implementation.
[37] K. Scaman, F. Bach, S. Bubeck, Y. T. Lee, and L. Massoulie. Optimal algorithms for smooth and strongly convex distributed optimization in networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3027-3036. PMLR, 06-11 Aug 2017.
[38] A. Sergeev and M. D. Balso. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799, 2018.
[39] S. U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations, 2019.
[40] S. U. Stich. Unified optimal analysis of the (stochastic) gradient method, 2019.
[41] T. Vogels, L. He, A. Koloskova, S. P. Karimireddy, T. Lin, S. U. Stich, and M. Jaggi. Relaysum for decentralized deep learning on heterogeneous data. Advances in Neural Information Processing Systems, 34:28004-28015, 2021.
[42] Y. Yakimenka, C.-W. Weng, H.-Y. Lin, E. Rosnes, and J. Kliewer. Straggler-resilient differentially-private decentralized learning. In 2022 IEEE Information Theory Workshop (ITW), pages 708-713. IEEE, 2022.
[43] B. Ying, K. Yuan, Y. Chen, H. Hu, P. Pan, and W. Yin. Exponential graph is provably efficient for decentralized deep training. Advances in Neural Information Processing Systems, 34:13975-13987, 2021.
[44] B. Ying, K. Yuan, H. Hu, Y. Chen, and W. Yin. Bluefog: Make decentralized algorithms practical for optimization and deep learning. arXiv preprint arXiv:2111.04287, 2021.
[45] B. Ying, K. Yuan, H. Hu, Y. Chen, and W. Yin. Bluefog: Make decentralized algorithms practical for optimization and deep learning. 2021.
[46] Y. You, I. Gitman, and B. Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017.
[47] B. Yuan, Y. He, J. Davis, T. Zhang, T. Dao, B. Chen, P. S. Liang, C. Re, and C. Zhang. Decentralized training of foundation models in heterogeneous environments. Advances in Neural Information Processing Systems, 35:25464-25477, 2022.
[48] S. Zhang, A. E. Choromanska, and Y. LeCun. Deep learning with elastic averaging sgd. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
A Notations 14
B Technical Preliminaries 14
C Proof of the main result of this paper 16
C.1 Some useful upper-bounds 17
C.2 Resolution: putting everything together 20
C.3 Optimizing the step-size 22
D Comparison with accelerated synchronous methods 23
E Experimental details 24
E.1 Graph topologies 24
E.2 Uniform neighbor selection check 24
For  $n \in \mathbb{N}^*$  the number of workers and  $d \in \mathbb{N}^*$  an ambient dimension, for all  $t > 0$ , the variable  $x_t \in \mathbb{R}^{n \times d}$  is a matrix such that  $x_t = [x_t^1, \dots, x_t^n]^\top$ , with  $x_t^i \in \mathbb{R}^d$  for all  $i \in \{1, \dots, n\}$ . We remind that  $\mathbf{1}$  is the vector of  $n$  ones such that  $\bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i = \frac{1}{n} x^\top \mathbf{1} \in \mathbb{R}^d$ . With  $\mathbf{I}$  the identity and  $\|. \|_F$  the matrix Frobenius norm, we write  $\pi = \mathbf{I} - \frac{1}{n} \mathbf{1} \mathbf{1}^\top$  the projection so that  $\| \pi x \|_F^2 = \sum_{i=1}^{n} \| x_i - \bar{x} \|^2$ .
For a continuously differentiable function  $f: \mathbb{R}^d \to \mathbb{R}$  and  $a, b \in \mathbb{R}^d$ , the Bregman divergence is defined with  $d_f(a, b) = f(a) - f(b) - \langle \nabla f(b), a - b \rangle$ . For  $\Lambda \in \mathbb{R}^{n \times n}$ , we denote by  $\Lambda^+$  its pseudo inverse. For a positive semi-definite  $\Lambda \in \mathbb{R}^{n \times n}$ , and  $x \in \mathbb{R}^{n \times d}$ , we introduce  $\| x \|_{\Lambda}^2 \triangleq \mathrm{Tr}(x^\top \Lambda x)$  and  $\Lambda^{1/2}$  its square-root. We recall that the connectivity between workers is given by a set of edges  $\mathcal{E}$ , and denote by  $e_i$  the  $i^{th}$  basis vector of  $\mathbb{R}^n$ .
For  $x\in \mathbb{R}^{n\times d}$ , we introduce:
$$
\nabla F (x) \triangleq [ \nabla f _ {1} (x _ {1}), \dots ., \nabla f _ {n} (x _ {n}) ] ^ {\mathsf {T}} \in \mathbb {R} ^ {n \times d} \mathrm {a n d} \nabla \tilde {F} (x, \xi) \triangleq [ \nabla F _ {1} (x _ {1}, \xi_ {1}), \dots , \nabla F _ {n} (x _ {n}, \xi_ {n}) ] ^ {\mathsf {T}} \in \mathbb {R} ^ {n \times d}.
$$
Finally, to study the gradient steps taken on each individual worker independently, we introduce:
$$
\nabla \tilde {F} _ {i} (x, \xi) \triangleq [ 0, \dots 0, \nabla F _ {i} (x _ {i}, \xi_ {i}), 0, \dots , 0 ] ^ {\mathsf {T}} \in \mathbb {R} ^ {n \times d}.
$$
We recall some basic properties that we will use throughout our proofs.
Lemma B.1 (Implications of Assumption 3.4). If each  $f_{i}$  is  $\mu$ -strongly convex and  $L$ -smooth, we have, for any  $a, b \in \mathbb{R}^d$ :
$$
\frac {1}{2 L} \| \nabla f _ {i} (a) - \nabla f _ {i} (b) \| ^ {2} \leq d _ {f _ {i}} (a, b) \leq \frac {L}{2} \| a - b \| ^ {2},
$$
$$
\frac {\mu}{2} \| a - b \| ^ {2} \leq d _ {f _ {i}} (a, b) \leq \frac {1}{2 \mu} \| \nabla f _ {i} (a) - \nabla f _ {i} (b) \| ^ {2}.
$$
Lemma B.2 (Generalized triangle inequality). For any  $a, b, c \in \mathbb{R}^d$  and continuously differentiable function  $f: \mathbb{R}^d \to \mathbb{R}$ , by definition of the Bregman divergence, we have:
$$
d _ {f} (a, b) + d _ {f} (b, c) = d _ {f} (a, c) + \langle a - b, \nabla f (c) - \nabla f (b) \rangle .
$$
Lemma B.3 (Variance decomposition). For a random vector  $a \in \mathbb{R}^d$  and any  $b \in \mathbb{R}^d$ , the variance of  $a$  can be decomposed as:
$$
\mathbb {E} \left[ \| a - \mathbb {E} [ a ] \| ^ {2} \right] = \mathbb {E} \left[ \| a - b \| ^ {2} \right] - \mathbb {E} \left[ \| \mathbb {E} [ a ] - b \| ^ {2} \right].
$$
Lemma B.4 (Jensen's inequality). For any vectors  $a_1, \ldots, a_n \in \mathbb{R}^d$ , we have:
$$
\left\| \sum_ {i = 1} ^ {n} a _ {i} \right\| ^ {2} \leq n \sum_ {i = 1} ^ {n} \| a _ {i} \| ^ {2}.
$$
Lemma B.5. For any vectors  $a, b \in \mathbb{R}^d$  and  $\alpha > 0$ :
$$
2 \langle a, b \rangle \leq \alpha \| a \| ^ {2} + \alpha^ {- 1} \| b \| ^ {2}.
$$
Lemma B.6. For any vectors  $a, b \in \mathbb{R}^d$  and  $\alpha > 0$ :
$$
\left\| a - b \right\| ^ {2} \leq (1 + \alpha) \left\| a \right\| ^ {2} + (1 + \alpha^ {- 1}) \left\| b \right\| ^ {2}.
$$
Lemma B.7. For any  $A \in \mathbb{R}^{n \times d}$  and  $B \in \mathbb{R}^{n \times n}$ , we have:
$$
\| B A \| _ {F} \leq \| A \| _ {F} \| B \| _ {2}.
$$
Lemma B.8 (Effective resistance contraction). For  $(i,j) \in \mathcal{E}$  and any  $x \in \mathbb{R}^{n \times d}$ , we have:
$$
\left\| \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\mathsf {T}} x \right\| _ {\Lambda^ {+}} ^ {2} \leq \chi_ {2} \left\| \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\mathsf {T}} x \right\| _ {F} ^ {2}.
$$
Proof. Indeed, we note that, by definition of  $\chi_{2}$  (3):
$$
\begin{array}{l} \left\| \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\mathsf {T}} x \right\| _ {\Lambda^ {+}} ^ {2} = \operatorname {T r} \left(x ^ {\mathsf {T}} \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\mathsf {T}} \Lambda^ {+} \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\mathsf {T}} x\right) (7) \\ \leq 2 \chi_ {2} \operatorname {T r} \left(x ^ {\mathsf {T}} \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\mathsf {T}} x\right) (8) \\ = \chi_ {2} \| \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\top} x \| _ {F} ^ {2} (9) \\ \end{array}
$$
Lemma B.9. For any  $x \in \mathbb{R}^{n \times d}$ , and  $\Lambda$  the Laplacian of a connected graph, we have:
$$
\sum_ {(i, j) \in \mathcal {E}} \lambda^ {i j} \| (e _ {i} - e _ {j}) (e _ {i} - e _ {j}) ^ {\intercal} x \| _ {F} ^ {2} = 2 \| x \| _ {\Lambda} ^ {2}.
$$
Proof. Indeed, by definition of the Laplacian  $\Lambda$  (3.1), we have:
$$
\begin{array}{l} \sum_ {(i, j) \in \mathcal {E}} \lambda^ {i j} \| \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\top} x \| _ {F} ^ {2} = \sum_ {(i, j) \in \mathcal {E}} \lambda^ {i j} \operatorname {T r} \left(x ^ {\top} \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\top} \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\top} x\right) (10) \\ = 2 \sum_ {(i, j) \in \mathcal {E}} \lambda^ {i j} \operatorname {T r} \left(x ^ {\mathsf {T}} \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\mathsf {T}} x\right) (11) \\ = 2 \operatorname {T r} \left(x ^ {\mathsf {T}} \sum_ {(i, j) \in \mathcal {E}} \lambda^ {i j} \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\mathsf {T}} x\right) (12) \\ \end{array}
$$