SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\mathbf{A}^{2}\mathbf{C}\mathbf{i}\mathbf{D}^{2}$ : Accelerating Asynchronous Communication in Decentralized Deep Learning
Adel Nabli
adel.nabli@sorbonne-universite.fr
Distributed training of Deep Learning models has been critical to many recent successes in the field. Current standard methods primarily rely on synchronous centralized algorithms which induce major communication bottlenecks and synchronization locks at scale. Decentralized asynchronous algorithms are emerging as a potential alternative but their practical applicability still lags. In order to mitigate the increase in communication cost that naturally comes with scaling the number of workers, we introduce a principled asynchronous, randomized, gossip-based optimization algorithm which works thanks to a continuous local momentum named  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ . Our method allows each worker to continuously process mini-batches without stopping, and run a peer-to-peer averaging routine in parallel, reducing idle time. In addition to inducing a significant communication acceleration at no cost other than adding a local momentum variable, minimal adaptation is required to incorporate  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  to standard asynchronous approaches. Our theoretical analysis proves accelerated rates compared to previous asynchronous decentralized baselines and we empirically show that using our  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  momentum significantly decrease communication costs in poorly connected networks. In particular, we show consistent improvement on the ImageNet dataset using up to 64 asynchronous workers (A100 GPUs) and various communication network topologies.
As Deep Neural Networks (DNNs) and their training datasets become larger and more complex, the computational demands and the need for efficient training schemes continues to escalate. Distributed training methods offer a solution by enabling the parallel optimization of model parameters across multiple workers. Yet, many of the current distributed methods in use are synchronous, and have significantly influenced the design of cluster computing environments. Thus, both the environments and algorithms rely heavily on high synchronicity in machine computations and near-instantaneous communication in high-bandwidth networks, favoring the adoption of centralized algorithms [7].
However, several studies [27, 44, 2, 28] are challenging this paradigm, proposing decentralized asynchronous algorithms that leverage minor time-delays fluctuations between workers to enhance the parallelization of computations and communications. Unlike centralized algorithms, decentralized approaches allow each node to contribute proportionally to its available resources, eliminating the necessity for a global central worker to aggregate results. Combined with asynchronous peer-to-peer (p2p) communications, these methods can streamline the overall training
process, mitigating common bottlenecks. This includes the Straggler Problem [42], the synchronization between computations and communications [9], or bandwidth limitations [47], potentially due to particular network topologies like a ring graph [43]. However, due to the large number of parameters which are optimized, training DNNs with these methods still critically requires a considerable amount of communication [22], presenting an additional challenge [32].
This work aims to address these challenges by introducing a principled acceleration method for pair-wise communications in peer-to-peer training of DNNs, in particular for cluster computing. While conventional synchronous settings accelerate communications by integrating a Chebychev acceleration followed by Gradient Descent steps [37], the potential of accelerated asynchronous pair-wise gossip for Deep Learning (DL) remains largely unexplored. Notably, the sophisticated theory of Stochastic Differential Equations (SDEs) offers an analytical framework for the design and study of the convergence of these algorithms [12]. We introduce a novel algorithm  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  (standing for Accelerating Asynchronous Communication in
Decentralized Deep Learning) that requires minimal overhead and effectively decouples communications and computations, accelerating pair-wise communications via a provable, accelerated, randomized gossip procedure based on continuous momentum (i.e., a mixing ODE) and time [12, 34]. We emphasize that beyond the aforementioned hardware superiority, stochastic algorithms also allows us to theoretically reach sublinear rates in convex settings [10], which opens the possibility to further principled accelerations. In practice, our method enables a virtual doubling of the communication rate in challenging network topologies without any additional cost, simply by maintaining a local momentum variable in each worker (see Fig. 1).
Our key contributions are as follows: (1) We extend the continued framework [12] to the nonconvex setting, in order to obtain a neat framework to describe asynchronous decentralized DL training. (2) This framework allows us to refine the analysis of a baseline asynchronous decentralized optimization algorithm. (3) We propose a novel and simple continued momentum which allows to significantly improve communication efficiency in challenging settings, which we name  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ . (4) We demonstrate that our method effectively minimizes the gap between centralized settings in environments hosting up to 64 asynchronous GPUs. (5) Our code is implemented in Pytorch [35], remove locks put on previous asynchronous implementations by circumventing their deadlines, and can be found in an open-source repository: https://github.com/AdelNabli/ACiD.
This paper is structured as follows: Sec. 3.1 outlines our model for asynchronous decentralized learning, while Sec. 3.2 discusses the training dynamic used to optimize our Deep models. Sec. 3.4 offers a comprehensive theoretical analysis of our method, which is validated empirically in Sec. 4.
Large-scale distributed DL. Two paradigms allow to maintain high-parallelization. On one side, model-parallelism [9, 25], which splits a neural network on independent machines, allowing to use local learning methods [4, 3]. On the other hand data-parallelism, which accelerates learning by making use of larger mini-batch splitted across multiple nodes [38] to maximally use GPU capacities. This parallelization entailing the use of larger batch-sizes, it requires an important process of adapting hyper-parameters [16], and in particular the learning rate scheduler. Developed for this setting, methods such as [16, 46] allow to stabilize training while maintaining good generalization performances. However, they have been introduced in the context of centralized synchronous training using All-Reduce schemes for communication, which still is the default setting of many approaches to data parallelism.
Decentralized DL. The pioneer work [27] is one of the first studies to suggest the potential superiority of synchronous decentralized training strategies in practice. In terms of implementation in the cluster
Figure 1: Adding  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  has the same effect as doubling the communication rates on ImageNet on the ring graph with 64 workers. See Sec. 4.
setting, decentralized frameworks have been shown to achieve higher throughput than optimized All-Reduce strategies [45, 38]. From the theoretical side, [21] propose a framework covering many settings of synchronous decentralized learning. However, as it consistently relies on using a global discrete iterations count, the notion of time is more difficult to exploit, which reveals crucial in our setting. Furthermore, no communication acceleration is incorporated in these algorithms. [22] provides a comprehensive methodology to relate the consensus distance, i.e. the average distance of the local parameters to the global average, to a necessary communication rate to avoid degrading performance and could be easily applied to our method. [29] is a method focusing on improving performance via a discrete momentum modification, which indicates momentum variables are key to decentralized DL.
Asynchronous Decentralized DL. There exist many attempts to incorporate asynchrony in decentralized training [48, 5, 8, 28, 2], which typically aim at removing lock barriers of synchronous decentralized algorithms. To the best of our knowledge, none of them introduce communication acceleration, yet they could be simply combined with our approach. Although recent approaches such as [2, 28] perform peer-to-peer averaging of parameters instead of gradients, thus allowing to communicate in parallel of computing (as there is no need to wait for the gradients before communicating), they are still coupled: parameter updates resulting from computations and communications are scheduled in a specific order, limiting their speed. Furthermore, in practice, both those works only implement a periodic averaging on the exponential graph (more favorable, see [43]) instead of investigating the influence of the graph's topology on the convergence of a randomized gossip method, as we do. In fact, AD-PSGD [28], the baseline algorithm in asynchronous decentralized DL, comes with a major caveat to avoid deadlocks in practice: they require a bipartite graph and schedule p2p communications in a pseudo-random manner instead of basing the decision on worker's current availability, hindering the advantage given by asynchronous methods in the mitigation of stragglers. Contrary to them, our implementation allows to pair workers in real time based on their availability, minimizing idle time for communications.
Communication reduction. Reducing communication overhead is an important topic for scalability [36]. For instance, [19, 20] allow to use of compression factor in limited bandwidth setting, and the local SGD communication schedule of [30] is shown to be beneficial. Those methods could be independently and simply combined with ours to potentially benefit from an additional communication acceleration. By leveraging key properties of the resistance of the communication network [14], [12] showed that standard asynchronous gossip [6] can be accelerated, even to give efficient primal algorithms in the convex setting [34]. However, this acceleration has never been deployed in the DL context, until now. RelaySum [41] is an approach which allows to average exactly parameters produced by different time steps and thus potentially delayed. However, it requires either to use a tree graph topology, either to build ad-hoc spanning trees and has inherent synchronous locks as it averages neighbor messages in a specific order.
Notations: Let  $n \in \mathbb{N}^*$  and  $d \in \mathbb{N}^*$  an ambient dimension, for  $x = (x^1, \ldots, x^n) \in \bigotimes_{i=1}^{n} \mathbb{R}^d$ , we write  $\bar{x} = \frac{1}{n} \sum_{i=1}^{n} x^i$  and  $\mathbf{1}$  the tensor of ones such that  $\bar{x} = \frac{1}{n} \mathbf{1}^\top x$ .  $\Xi$  is a probability space with measure  $\mathcal{P}$ .  $f(t) = \mathcal{O}(1)$  means there is a  $C > 0$  such that for  $t$  large enough,  $|f(t)| \leq C$ , whereas  $\tilde{\mathcal{O}}$ -notation hides constants and polylogarithmic factors.
3.1 Model for a decentralized environment
We consider a network of  $n$  workers whose connectivity is given by edges  $\mathcal{E}$ . Local computations are modeled as (stochastic) point-wise processes  $N_{t}^{i}$ , and communications between nodes  $(i,j) \in \mathcal{E}$  as  $M_{t}^{ij}$ . We assume that the communications are symmetric, meaning that if a message is sent from node  $i$  to  $j$ , then the reverse is true. In practice, such processes are potentially highly correlated and could follow any specific law, and could involve delays. For the sake of simplicity, we do not model lags, though it is possible to obtain guarantees via dedicated Lyapunov functions [13]. In our setting, we assume that all nodes have similar buffer variables which correspond to a copy of a common model (e.g., a DNN). For a parameter  $x$ , we write  $x_{t}^{i}$  the model's parameters at node  $i$  and time  $t$  and  $x_{t} = (x_{t}^{1},\dots,x_{t}^{n})$  their concatenation. In the following, we assume that each worker computes
Table 1: Comparison of convergence rates for strongly convex and non-convex objectives against concurrent works in the fixed topology setting. We neglect logarithmic terms. Observe that thanks to the maximal resistance  $\chi_{2} \leq \chi_{1}$ , our method obtains substantial acceleration for the bias term. Moreover, while our baseline is strongly related to AD-PSGD [28], our analysis refines its complexity when workers sample data from the same distribution.
Method	Strongly Convex	Non-Convex
Koloskova et al. [21]	σ2/nμ2ε + √L χ1ξ+√χ1σ/μ3/2√ε + L/μχ1	L/σ2/nε2 + L/χ1ξ+√χ1σ/ε3/2 + L/χ1/ε
AD-PSGD [28]	-	L/σ2+ξ2/ε2 + n2/Lχ1/ε
Baseline (Ours)	σ2+χ1ξ2/μ2ε + L/μχ1	L/σ2+χ1ξ2/ε2 + L/χ1/ε
A2CiD2(Ours)	σ2+√χ1χ2/μ2ε + L/μ√χ1χ2	L/σ2+√χ1χ2/ε2 + L/√χ1χ2/ε
about 1 mini-batch of gradient per unit of time (not necessarily simultaneously), which is a standard homogeneity assumption [18], and we denote by  $\lambda^{ij}$  the instantaneous expected frequency of edge  $(i,j)$ , which we assume time homogeneous.
Definition 3.1 (Instantaneous expected Laplacian). We define the Laplacian  $\Lambda$  as:
$$
\Lambda \triangleq \sum_ {(i, j) \in \mathcal {E}} \lambda^ {i j} \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\top}. \tag {1}
$$
In this context, a natural quantity is the algebraic connectivity [6] given by:
$$
\chi_ {1} \triangleq \sup  _ {\| x \| = 1, x \perp \mathbf {1}} \frac {1}{x ^ {\top} \Lambda x}. \tag {2}
$$
For a connected graph (i.e.,  $\chi_{1} < +\infty$ ), we will also use the maximal resistance of the network:
$$
\chi_ {2} \triangleq \frac {1}{2} \sup  _ {(i, j) \in \mathcal {E}} (e _ {i} - e _ {j}) ^ {\mathsf {T}} \Lambda^ {+} (e _ {i} - e _ {j}) \leq \chi_ {1}. \tag {3}
$$
The next sections will show that it is possible to accelerate the asynchronous gossip algorithms from  $\chi_{1}$  to  $\sqrt{\chi_1\chi_2} \leq \chi_1$ , while [12] or [34] emphasize the superiority of accelerated asynchronous gossips over accelerated synchronous ones.
3.2 Training dynamic
The goal of a typical decentralized algorithm is to minimize the following quantity:
$$
\inf  _ {x \in \mathbb {R} ^ {d}} f (x) \triangleq \inf  _ {x \in \mathbb {R} ^ {d}} \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} (x) = \inf  _ {x _ {i} = x _ {1}} \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} (x _ {i}).
$$
For this, we follow a first order optimization strategy consisting in using estimates of the gradient  $\nabla f_{i}(x_{i})$  via i.i.d unbiased Stochastic Gradient (SG) oracles given by  $\nabla F_{i}(x_{i},\xi_{i})$  s.t.  $\mathbb{E}_{\xi_i}[\nabla F_i(x_i,\xi_i)] = \nabla f_i(x_i)$ . The dynamic of updates of our model evolves as the following SDE, for  $\eta ,\gamma ,\alpha ,\tilde{\alpha}$  some time-independent scalar hyper-parameters, whose values are found in our theoretical analysis and used in our implementation, and  $dN_{t}^{i}(\xi_{i})$  some point processes on  $\mathbb{R}_+ \times \Xi$  with intensity  $dt \otimes d\mathcal{P}$ :
$$
d x _ {t} ^ {i} = \eta \left(\tilde {x} _ {t} ^ {i} - x _ {t} ^ {i}\right) d t - \gamma \int_ {\Xi} \nabla F _ {i} \left(x _ {t} ^ {i}, \xi_ {i}\right) d N _ {t} ^ {i} \left(\xi_ {i}\right) - \alpha \sum_ {j, (i, j) \in \mathcal {E}} \left(x _ {t} ^ {i} - x _ {t} ^ {j}\right) d M _ {t} ^ {i j}, \tag {4}
$$
$$
d \tilde {x} _ {t} ^ {i} = \eta (x _ {t} ^ {i} - \tilde {x} _ {t} ^ {i}) d t - \gamma \int_ {\Xi} \nabla F _ {i} (x _ {t} ^ {i}, \xi_ {i}) d N _ {t} ^ {i} (\xi_ {i}) - \tilde {\alpha} \sum_ {j, (i, j) \in \mathcal {E}} (x _ {t} ^ {i} - x _ {t} ^ {j}) d M _ {t} ^ {i j}.
$$
We emphasize that while the dynamic Eq. 4 is formulated using SDEs [1], which brings the power of the continuous-time analysis toolbox, it is still event-based and thus discrete in nature. Hence, it can efficiently model practically implementable algorithms, as shown by Algo. 1. The coupling  $\{x_{t},\tilde{x}_{t}\}$  corresponds to a momentum term which will be useful to obtain communication acceleration as explained in the next section. Again,  $\int_{\Xi}\nabla F_i(x_t^i,\xi_i)dN_t^i (\xi_i)$  will be estimated via i.i.d SGs sampled as  $N_{t}^{i}$  spikes. Furthermore, if  $\bar{x}_0 = \overline{\bar{x}}_0$  , then,  $\bar{x}_t = \overline{\bar{x}_t}$  and we obtain a tracker of the average across workers which is similar to what is achieved through Gradient Tracking methods [19]. This is a key advantage of our method to obtain convergence guarantees, which writes as:
$$
d \bar {x} _ {t} = - \gamma \frac {1}{n} \sum_ {i = 1} ^ {n} \int_ {\Xi} \nabla F _ {i} \left(x _ {t} ^ {i}, \xi_ {i}\right) d N _ {t} ^ {i} \left(\xi_ {i}\right). \tag {5}
$$
3.3 Informal explanation of the dynamic through the Baseline case
To give some practical intuition on our method, we consider a baseline asynchronous decentralized dynamic, close to AD-PSGD [28]. By considering  $\eta = 0$ ,  $\alpha = \tilde{\alpha} = \frac{1}{2}$ , the dynamic (4) simplifies to:
$$
d x _ {t} ^ {i} = - \gamma \int_ {\Xi} \nabla F _ {i} \left(x _ {t} ^ {i}, \xi_ {i}\right) d N _ {t} ^ {i} \left(\xi_ {i}\right) - \frac {1}{2} \sum_ {j, (i, j) \in \mathcal {E}} \left(x _ {t} ^ {i} - x _ {t} ^ {j}\right) d M _ {t} ^ {i j}. \tag {6}
$$
In a DL setting,  $x^{i}$  contains the parameters of the DNN hosted on worker  $i$ . Thus, (6) simply says that the parameters of the DNN are updated either by taking local SGD steps, or by pairwise averaging with peers  $j$ ,  $(i,j) \in \mathcal{E}$ . These updates happen independently, at random times: although we assume that all workers compute gradients at the same speed on average (and re-normalized time accordingly), the use of Poisson Processes model the inherent variability in the time between these updates. However, the p2p averaging depends on the capabilities of the network, and we allow each link  $(i,j)$  to have a different bandwidth, albeit constant through time, modeled through the frequency  $\lambda^{ij}$ . The gradient and communication processes are decoupled: there is no need for one to wait for the other, allowing to compute stochastic gradients uninterrupted and run the p2p averaging in parallel, as illustrated by Fig.2. Finally, (4) adds a momentum step mixing the local parameters  $x^{i}$  and momentum buffer  $\tilde{x}^{i}$  before each type of update, allowing for significant savings in communication costs, as we show next.
Figure 2: Example of worker updates in synchronous (left) and asynchronous (right) optimization methods. We remark that our asynchronous algorithm reduces idle time, and allows to communicate in parallel of computing gradient, only synchronizing two workers at a time for averaging parameters. Here, one p2p communication is performed per computation in expectation.
3.4 Theoretical analysis of  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$
We now provide an analysis of our decentralized, asynchronous algorithm. For the sake of simplicity, we will consider that communications and gradients spike as Poisson processes:
Assumption 3.2 (Poisson Processes).  $N_{t}^{i}$ ,  $M_{t}^{ij}$  are independent, Point-wise Poisson Processes. The  $\{N_{t}^{i}\}_{i = 1}^{n}$  have a rate of 1, and for  $(i,j)\in \mathcal{E}$ ,  $M_t^{ij}$  have a rate  $\lambda^{ij}$ .
We also assume that the communication network is connected during the training:
Assumption 3.3 (Strong connectivity). We assume that  $\chi_{1} < \infty$ .
We will now consider two generic assumptions obtained from [21], which allow us to specify our lemma to convex and non-convex settings. Note that the non-convex Assumption 3.5 generalizes the assumptions of [28], by taking  $M = P = 0$ .
Assumption 3.4 (Strongly convex setting). Each  $f_{i}$  is  $\mu$ -strongly convex and  $L$ -smooth, and:
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\xi_ {i}} \left[ \| \nabla F _ {i} (x, \xi_ {i}) - \nabla f _ {i} (x) \| ^ {2} \right] \leq \sigma^ {2} \text {a n d} \frac {1}{n} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x ^ {*}) - \nabla f (x ^ {*}) \| ^ {2} \leq \zeta^ {2}.
$$
Assumption 3.5 (Non-convex setting). Each  $f_{i}$  is  $L$ -smooth, and there exists  $P, M > 0$  such that:
$$
\forall x \in \mathbb {R} ^ {d}, \frac {1}{n} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x) - \nabla f (x) \| ^ {2} \leq \zeta^ {2} + P \| \nabla f (x) \| ^ {2},
$$
$$
\forall x _ {1}, \dots , x _ {n} \in \mathbb {R} ^ {d}, \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\xi_ {i}} \| \nabla F _ {i} (x _ {i}, \xi_ {i}) - \nabla f _ {i} (x _ {i}) \| ^ {2} \leq \sigma^ {2} + \frac {M}{n} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x _ {i}) \| ^ {2}.
$$
We can now state our convergence guarantees. An informal way to understand our proposition, is that while gradient updates are non-convex, the communication updates are linear and thus benefit from local convexity; its proof is delayed to Appendix C.
Proposition 3.6 (Convergence guarantees.). Assume that  $\{x_{t},\tilde{x}_{t}\}$  follow the dynamic Eq. 4 and that Assumption 3.2-3.3 are satisfied. Assume that  $\mathbf{1}\bar{x}_0 = x_0 = \tilde{x}_0$  and let  $T$  the total running time. Then:
- Non-accelerated setting, we pick  $\eta = 0$ ,  $\alpha = \tilde{\alpha} = \frac{1}{2}$  and set  $\chi = \chi_{1}$
- Acceleration  $(A^2 C_i D^2)$ , we set  $\eta = \frac{1}{2\sqrt{\chi_1\chi_2}}, \alpha = \frac{1}{2}, \tilde{\alpha} = \frac{1}{2}\sqrt{\frac{\chi_1}{\chi_2}}$ , and  $\chi = \sqrt{\chi_1\chi_2} \leq \chi_1$ .
Then, there exists a constant step size  $\gamma > 0$  such that if:
- (strong-convexity) the Assumption 3.4 is satisfied, then  $\gamma \leq \frac{1}{16L(1 + \chi)}$  and:
$$
\mathbb {E} \left[ \| \bar {x} _ {T} - x ^ {*} \| ^ {2} \right] = \tilde {\mathcal {O}} \left(\| \bar {x} _ {0} - x ^ {*} \| ^ {2} e ^ {- \frac {\mu T}{1 6 L (1 + \chi)}} + \frac {\sigma^ {2} + \zeta^ {2} (1 + \chi)}{\mu^ {2} T}\right),
$$
- (non-convexity) the Assumption 3.5 is satisfied, then there is  $c > 0$  which depends only on  $P, M$  from the assumptions such that  $\gamma \leq \frac{c}{L(\chi + 1)}$  and:
$$
\frac {1}{T} \int_ {0} ^ {T} \mathbb {E} \left[ \| \nabla f (\bar {x} _ {t}) \| ^ {2} \right] d t = \mathcal {O} \left(\frac {L (1 + \chi)}{T} (f (x _ {0}) - f (x ^ {*})) + \sqrt {\frac {L (f (x _ {0}) - f (x ^ {*}))}{T} (\sigma^ {2} + (1 + \chi) \xi^ {2})}\right).
$$
Also, the expected number of gradient steps is  $nT$  and the number of communications is  $\frac{\operatorname{Tr}(\Lambda)}{2} T$ .
Tab. 1 compares our convergence rates with concurrent works. Compared to every concurrent work, the bias term of  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  is smaller by a factor  $\sqrt{\frac{\chi_1}{\chi_2}} \geq 1$  at least. Yet, as expected, in the non-accelerated setting, we would recover similar rates to those. Compared to [20], the variance terms held no variance reduction with the number of workers; however, this should not be an issue in a DL setting, where it is well-known that variance reduction techniques degrade generalization during training [15]. Comparing directly the results of [2] is difficult as they only consider the asymptotic rate, even if the proof framework is similar to [28] and should thus lead to similar rates of convergence.
3.5 Informal interpretation and comparison with decentralized synchronous methods
Here, we informally discuss results from Prop. 3.6 and compare our communication rate with state-of-the-art decentralized synchronous methods such as DeTAG [31], MSDA [37] and OPAPC [23].
As we normalize time so that each node takes one gradient step per time unit in expectation, one time unit for us is analogous to one round of computation (one "step") for synchronous methods. Synchronous methods such as [31, 37, 23] perform multiple rounds of communications
Table 2: # of communications per "step"/time unit on several graphs.
Method	Star	Ring	Complete
Accelerated Synchronous (e.g., [31, 37, 23])	n3/2	n2	n2
A2CiD2	n	n2	n