Title: $\textbf{A}^2\textbf{CiD}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Decentralized Optimization for Deep Learning

Asynchronous Optimization

Distributed Training

Data-Parallel
Abstract: Distributed training of Deep Learning models has been critical to many recent successes in the field. Current standard methods primarily rely on synchronous centralized algorithms which induce major communication bottlenecks and synchronization locks at scale. Decentralized asynchronous algorithms are emerging as a potential alternative but their practical applicability still lags. In order to mitigate the increase in communication cost that naturally comes with scaling the number of workers, we introduce a principled asynchronous, randomized, gossip-based optimization algorithm which works thanks to a continuous local momentum named $\textbf{A}^2\textbf{CiD}^2$. Our method allows each worker to continuously process mini-batches without stopping, and run a peer-to-peer averaging routine in parallel, reducing idle time. In addition to inducing a significant communication acceleration at no cost other than adding a local momentum variable, minimal adaptation is required to incorporate $\textbf{A}^2\textbf{CiD}^2$ to standard asynchronous approaches. Our theoretical analysis proves accelerated rates compared to previous asynchronous decentralized baselines and we empirically show that using our $\textbf{A}^2\textbf{CiD}^2$ momentum significantly decrease communication costs in poorly connected networks. In particular, we show consistent improvement on the ImageNet dataset using up to 64 asynchronous workers (A100 GPUs) and various communication network topologies.

A 2 CiD 2 : Accelerating Asynchronous Communication in Decentralized Deep Learning

Abstract

Distributed training of Deep Learning models has been critical to many recent successes in the field. Current standard methods primarily rely on synchronous centralized algorithms which induce major communication bottlenecks and synchronization locks at scale. Decentralized asynchronous algorithms are emerging as a potential alternative but their practical applicability still lags. In order to mitigate the increase in communication cost that naturally comes with scaling the number of workers, we introduce a principled asynchronous, randomized, gossip-based optimization algorithm which works thanks to a continuous local momentum named A 2 CiD 2 . Our method allows each worker to continuously process mini-batches without stopping, and run a peer-to-peer averaging routine in parallel, reducing idle time. In addition to inducing a significant communication acceleration at no cost other than adding a local momentum variable, minimal adaptation is required to incorporate A 2 CiD 2 to standard asynchronous approaches. Our theoretical analysis proves accelerated rates compared to previous asynchronous decentralized baselines and we empirically show that using our A 2 CiD 2 momentum significantly decrease communication costs in poorly connected networks. In particular, we show consistent improvement on the ImageNet dataset using up to 64 asynchronous workers (A100 GPUs) and various communication network topologies.

1 Introduction

As Deep Neural Networks (DNNs) and their training datasets become larger and more complex, the computational demands and the need for efficient training schemes continues to escalate. Distributed training methods offer a solution by enabling the parallel optimization of model parameters across multiple workers. Yet, many of the current distributed methods in use are synchronous, and have significantly influenced the design of cluster computing environments. Thus, both the environments and algorithms rely heavily on high synchronicity in machine computations and near-instantaneous communication in high-bandwidth networks, favoring the adoption of centralized algorithms [7].

However, several studies [27, 44, 2, 28] are challenging this paradigm, proposing decentralized asynchronous algorithms that leverage minor time-delays fluctuations between workers to enhance the parallelization of computations and communications. Unlike centralized algorithms, decentralized approaches allow each node to contribute proportionally to its available resources, eliminating the necessity for a global central worker to aggregate results. Combined with asynchronous peer-to-peer (p2p) communications, these methods can streamline the overall training

process, mitigating common bottlenecks. This includes the Straggler Problem [42], the synchronization between computations and communications [9], or bandwidth limitations [47], potentially due to particular network topologies like a ring graph [43]. However, due to the large number of parameters which are optimized, training DNNs with these methods still critically requires a considerable amount of communication [22], presenting an additional challenge [32].

This work aims to address these challenges by introducing a principled acceleration method for pair-wise communications in peer-to-peer training of DNNs, in particular for cluster computing. While conventional synchronous settings accelerate communications by integrating a Chebychev acceleration followed by Gradient Descent steps [37], the potential of accelerated asynchronous pair-wise gossip for Deep Learning (DL) remains largely unexplored. Notably, the sophisticated theory of Stochastic Differential Equations (SDEs) offers an analytical framework for the design and study of the convergence of these algorithms [12]. We introduce a novel algorithm A 2 CiD 2 (standing for A ccelerating A synchronous C ommunication i n

0

2000

4000

6000

8000

10000

12000

14000

# grad steps per worker

2 × 10

0

3 × 10

0

4 × 10

0

6 × 10

0

Training loss

1 com/

2 com/

w/

A

2

CiD

2

w/o

A

2

CiD

2

D ecentralized D eep Learning) that requires minimal overhead and effectively decouples communications and computations, accelerating pair-wise communications via a provable, accelerated, randomized gossip procedure based on continuous momentum (i.e., a mixing ODE) and time [12, 34]. We emphasize that beyond the aforementioned hardware superiority, stochastic algorithms also allows us to theoretically reach sublinear rates in convex settings [10], which opens the possibility to further principled accelerations. In practice, our method enables a virtual doubling of the communication rate in challenging network topologies without any additional cost, simply by maintaining a local momentum variable in each worker (see Fig. 1).

Our key contributions are as follows: (1) We extend the continuized framework [12] to the nonconvex setting, in order to obtain a neat framework to describe asynchronous decentralized DL training. (2) This framework allows us to refine the analysis of a baseline asynchronous decentralized optimization algorithm. (3) We propose a novel and simple continuized momentum which allows to significantly improve communication efficiency in challenging settings, which we name A 2 CiD 2 . (4) We demonstrate that our method effectively minimizes the gap between centralized settings in environments hosting up to 64 asynchronous GPUs. (5) Our code is implemented in Pytorch [35], remove locks put on previous asynchronous implementations by circumventing their deadlocks, and can be found in an open-source repository: https://github.com/AdelNabli/ACiD.

This paper is structured as follows: Sec. 3.1 outlines our model for asynchronous decentralized learning, while Sec. 3.2 discusses the training dynamic used to optimize our Deep models. Sec. 3.4 offers a comprehensive theoretical analysis of our method, which is validated empirically in Sec. 4.

2 Related Work

Large-scale distributed DL. Two paradigms allow to maintain high-parallelization. On one side, model-parallelism [9, 25], which splits a neural network on independent machines, allowing to use local learning methods [4, 3]. On the other hand data-parallelism, which accelerates learning by making use of larger mini-batch splitted across multiple nodes [38] to maximally use GPU capacities. This parallelization entailing the use of larger batch-sizes, it requires an important process of adapting hyper-parameters [16], and in particular the learning rate scheduler. Developed for this setting, methods such as [16, 46] allow to stabilize training while maintaining good generalization performances. However, they have been introduced in the context of centralized synchronous training using All-Reduce schemes for communication, which still is the default setting of many approaches to data parallelism.

Decentralized DL. The pioneer work [27] is one of the first study to suggest the potential superiority of synchronous decentralized training strategies in practice. In terms of implementation in the cluster

setting, decentralized frameworks have been shown to achieve higher throughput than optimized All-Reduce strategies [45, 38]. From the theoretical side, [21] propose a framework covering many settings of synchronous decentralized learning. However, as it consistently relies on using a global discrete iterations count, the notion of time is more difficult to exploit, which reveals crucial in our setting. Furthermore, no communication acceleration is incorporated in these algorithms. [22] provides a comprehensive methodology to relate the consensus distance, i.e. the average distance of the local parameters to the global average, to a necessary communication rate to avoid degrading performance and could be easily applied to our method. [29] is a method focusing on improving performance via a discrete momentum modification, which indicates momentum variables are key to decentralized DL.

Asynchronous Decentralized DL. There exist many attempts to incorporate asynchrony in decentralized training [48, 5, 8, 28, 2], which typically aim at removing lock barriers of synchronous decentralized algorithms. To the best of our knowledge, none of them introduce communication acceleration, yet they could be simply combined with our approach. Although recent approaches such as [2, 28] perform peer-to-peer averaging of parameters instead of gradients, thus allowing to communicate in parallel of computing (as there is no need to wait for the gradients before communicating), they are still coupled: parameter updates resulting from computations and communications are scheduled in a specific order, limiting their speed. Furthermore, in practice, both those works only implement a periodic averaging on the exponential graph (more favorable, see [43]) instead of investigating the influence of the graph's topology on the convergence of a randomized gossip method, as we do. In fact, AD-PSGD [28], the baseline algorithm in asynchronous decentralized DL, comes with a major caveat to avoid deadlocks in practice: they require a bipartite graph and schedule p2p communications in a pseudo-random manner instead of basing the decision on worker's current availability, hindering the advantage given by asynchronous methods in the mitigation of stragglers. Contrary to them, our implementation allows to pair workers in real time based on their availability, minimizing idle time for communications.

Communication reduction. Reducing communication overhead is an important topic for scalability [36]. For instance, [19, 20] allow to use of compression factor in limited bandwidth setting, and the local SGD communication schedule of [30] is shown to be beneficial. Those methods could be independently and simply combined with ours to potentially benefit from an additional communication acceleration. By leveraging key properties of the resistance of the communication network [14], [12] showed that standard asynchronous gossip [6] can be accelerated, even to give efficient primal algorithms in the convex setting [34]. However, this acceleration has never been deployed in the DL context, until now. RelaySum [41] is an approach which allow to average exactly parameters produced by different time steps and thus potentially delayed. However, it requires either to use a tree graph topology, either to build ad-hoc spanning trees and has inherent synchronous locks as it averages neighbor messages in a specific order.

Notations: Let n ∈ N ∗ and d ∈ N ∗ an ambient dimension, for x = ( x 1 , ..., x n ) ∈ ⊗ n i =1 R d , we write ¯ x = 1 n ∑ n i =1 x i and 1 the tensor of ones such that ¯ x = 1 n 1 T x . Ξ is a probability space with measure P . f ( t ) = O (1) means there is a C > 0 such that for t large enough, | f ( t ) | ≤ C , whereas ˜ O -notation hides constants and polylogarithmic factors..

3 Method

3.1 Model for a decentralized environment

We consider a network of n workers whose connectivity is given by edges E . Local computations are modeled as (stochastic) point-wise processes N i t , and communications between nodes ( i, j ) ∈ E as M ij t . We assume that the communications are symmetric, meaning that if a message is sent from node i to j , then the reverse is true. In practice, such processes are potentially highly correlated and could follow any specific law, and could involve delays. For the sake of simplicity, we do not model lags, though it is possible to obtain guarantees via dedicated Lyapunov functions [13]. In our setting, we assume that all nodes have similar buffer variables which correspond to a copy of a common model (e.g., a DNN). For a parameter x , we write x i t the model's parameters at node i and time t and x t = ( x 1 t , ..., x n t ) their concatenation. In the following, we assume that each worker computes

about 1 mini-batch of gradient per unit of time (not necessarily simultaneously), which is a standard homogeneity assumption [18], and we denote by λ ij the instantaneous expected frequency of edge ( i, j ) , which we assume time homogeneous.

Definition 3.1 (Instantaneous expected Laplacian) . We define the Laplacian Λ as:

In this context, a natural quantity is the algebraic connectivity [6] given by:

For a connected graph ( i.e. , χ 1 < + ∞ ), we will also use the maximal resistance of the network:

The next sections will show that it is possible to accelerate the asynchronous gossip algorithms from χ 1 to √ χ 1 χ 2 ≤ χ 1 , while [12] or [34] emphasize the superiority of accelerated asynchronous gossips over accelerated synchronous ones.

3.2 Training dynamic

The goal of a typical decentralized algorithm is to minimize the following quantity:

For this, we follow a first order optimization strategy consisting in using estimates of the gradient ∇ f i ( x i ) via i.i.d unbiased Stochastic Gradient (SG) oracles given by ∇ F i ( x i , ξ i ) s.t. E ξ i [ ∇ F i ( x i , ξ i )] = ∇ f i ( x i ) . The dynamic of updates of our model evolves as the following SDE, for η, γ, α, ˜ α some time-independent scalar hyper-parameters, whose values are found in our theoretical analysis and used in our implementation, and dN i t ( ξ i ) some point processes on R + × Ξ with intensity dt ⊗ d P :

We emphasize that while the dynamic Eq. 4 is formulated using SDEs [1], which brings the power of the continuous-time analysis toolbox, it is still event-based and thus discrete in nature. Hence, it can efficiently model practically implementable algorithms, as shown by Algo. 1. The coupling { x t , ˜ x t } corresponds to a momentum term which will be useful to obtain communication acceleration as explained in the next section. Again, ∫ Ξ ∇ F i ( x i t , ξ i ) dN i t ( ξ i ) will be estimated via i.i.d SGs sampled as N i t spikes. Furthermore, if ¯ x 0 = ˜ x 0 , then, ¯ x t = ˜ x t and we obtain a tracker of the average across workers which is similar to what is achieved through Gradient Tracking methods [19]. This is a key advantage of our method to obtain convergence guarantees, which writes as:

3.3 Informal explanation of the dynamic through the Baseline case

To give some practical intuition on our method, we consider a baseline asynchronous decentralized dynamic, close to AD-PSGD [28]. By considering η = 0 , α = ˜ α = 1 2 , the dynamic (4) simplifies to:

In a DL setting, x i contains the parameters of the DNN hosted on worker i . Thus, (6) simply says that the parameters of the DNN are updated either by taking local SGD steps, or by pairwise averaging with peers j, ( i, j ) ∈ E . These updates happen independently, at random times: although we assume that all workers compute gradients at the same speed on average (and re-normalized time accordingly), the use of Poisson Processes model the inherent variability in the time between these updates. However, the p2p averaging depends on the capabilities of the network, and we allow each link ( i, j ) to have a different bandwidth, albeit constant through time, modeled through the frequency λ ij . The gradient and communication processes are decoupled: there is no need for one to wait for the other, allowing to compute stochastic gradients uninterruptedly and run the p2p averaging in parallel, as illustrated by Fig.2. Finally, (4) adds a momentum step mixing the local parameters x i and momentum buffer ˜ x i before each type of update, allowing for significant savings in communication costs, as we show next.

Time

Worker 1

Worker 2

Worker 3

Comm. delay

Comp. delay

Idling

Time

Worker 1

Worker 2

Worker 3

comm. synchro.

Comm. delay

Comp. delay

Idling

comm. synchro.

3.4 Theoretical analysis of A 2 CiD 2

We now provide an analysis of our decentralized, asynchronous algorithm. For the sake of simplicity, we will consider that communications and gradients spike as Poisson processes:

Assumption 3.2 (Poisson Processes) . N i t , M ij t are independent, Point-wise Poisson Processes. The { N i t } n i =1 have a rate of 1 , and for ( i, j ) ∈ E , M ij t have a rate λ ij .

We also assume that the communication network is connected during the training:

Assumption 3.3 (Strong connectivity) . We assume that χ 1 < ∞ .

We will now consider two generic assumptions obtained from [21], which allow us to specify our lemma to convex and non-convex settings. Note that the non-convex Assumption 3.5 generalizes the assumptions of [28], by taking M = P = 0 .

Assumption 3.4 (Strongly convex setting) . Each f i is µ -strongly convex and L -smooth, and:

Assumption 3.5 (Non-convex setting) . Each f i is L -smooth, and there exists P, M > 0 such that:

and,

We can now state our convergence guarantees. An informal way to understand our proposition, is that while gradient updates are non-convex, the communication updates are linear and thus benefit from local convexity; its proof is delayed to Appendix C.

Proposition 3.6 (Convergence guarantees.) . Assume that { x t , ˜ x t } follow the dynamic Eq. 4 and that Assumption 3.2-3.3 are satisfied. Assume that 1 ¯ x 0 = x 0 = ˜ x 0 and let T the total running time. Then:

Non-accelerated setting , we pick η = 0 , α = ˜ α = 1 2 and set χ = χ 1 ,

Acceleration (A 2 CiD 2 ) , we set η = 1 2 √ χ 1 χ 2 , α = 1 2 , ˜ α = 1 2 √ χ 1 χ 2 , and χ = √ χ 1 χ 2 ≤ χ 1 .

Then, there exists a constant step size γ > 0 such that if:

( strong-convexity ) the Assumption 3.4 is satisfied, then γ ≤ 1 16 L (1+ χ ) and:

( non-convexity ) the Assumption 3.5 is satisfied, then there is c > 0 which depends only on P, M from the assumptions such that γ ≤ c L ( χ +1) and:

Also, the expected number of gradient steps is nT and the number of communications is Tr(Λ) 2 T .

Tab. 1 compares our convergence rates with concurrent works. Compared to every concurrent work, the bias term of A 2 CiD 2 is smaller by a factor √ χ 1 χ 2 ≥ 1 at least. Yet, as expected, in the non-accelerated setting, we would recover similar rates to those. Compared to [20], the variance terms held no variance reduction with the number of workers; however, this should not be an issue in a DL setting, where it is well-known that variance reduction techniques degrade generalization during training [15]. Comparing directly the results of [2] is difficult as they only consider the asymptotic rate, even if the proof framework is similar to [28] and should thus lead to similar rates of convergence.

3.5 Informal interpretation and comparison with decentralized synchronous methods

Here, we informally discuss results from Prop. 3.6 and compare our communication rate with stateof-the-art decentralized synchronous methods such as DeTAG [31], MSDA [37] and OPAPC [23].

As we normalize time so that each node takes one gradient step per time unit in expectation, one time unit for us is analogous to one round of computation (one "step") for synchronous methods. Synchronous methods such as [31, 37, 23] perform multiple rounds of communications

(their Accelerated Gossip procedure) between rounds of gradient computations by using an inner loop inside their main loop (the one counting "steps"), so that the graph connectivity do not impact the total number of "steps" necessary to reach ϵ -precision. As Prop. 3.6 shows, the quantity 1+ √ χ 1 [Λ] χ 2 [Λ] is a factor in our convergence rate. Λ containing the information of both the topology E and the edge communication rates λ ij , this is analogous to saying √ χ 1 [Λ] χ 2 [Λ] = O (1) for our method ( i.e. , the graph connectivity does not impact the time to converge), which, given the graph's topology, dictates the communication rate, see Appendix D for more details. Tab. 2 compares the subsequent communication rates with synchronous methods.

4 Numerical Experiments

Now, we experimentally compare A 2 CiD 2 to a synchronous baseline All-Reduce SGD (AR-SGD, see [26]) and an asynchronous baseline using randomized pairwise communications (a variant of AD-PSGD [28], traditionally used in state-of-the-art decentralized asynchronous training of DNNs). In our case, the asynchronous baseline corresponds to the dynamic Eq. (6). Our approach is standard: we empirically study the decentralized training behavior of our asynchronous algorithm by training ResNets [17] for image recognition. Following [2], we pick a ResNet18 for CIFAR-10 [24] and ResNet50 for ImageNet [11]. To investigate how our method scales with the number of workers, we run multiple experiments using up to 64 NVIDIA A100 GPUs in a cluster with 8 A100 GPUs per node using an Omni-PAth interconnection network at 100 Gb/s, and set one worker per GPU.

4.1 Experimental Setup

Hyper-parameters. Training a DNN using multiple workers on a cluster requires several adaptations compared to the standard setting. As the effective batch-size grows linearly with the

number of workers n , we use the learning-rate schedule for large batch training of [16] in all our experiments. Following [30], we fixed the local batch size to 128 on both CIFAR-10 and ImageNet.

0

5000

10000

15000

20000

25000

# grad steps per worker

10

2

10 0

Training loss

Workers

4

8

16

32

64

(a)

0

250

500

750

1000

1250

1500

1750

# grad steps per worker

10

3

10

2

10

1

10 0

10 1

l

2

distance

1 com/

2 com/

4 com/

All-Reduce

(b)

Our goal being to divide the compute load between the n workers, all methods access the same total amount of data samples, regardless of the number of local steps. On CIFAR-10 and ImageNet, this

Algorithm 1: This algorithm block describes our implementation of our Asynchronous algorithm with A 2 CiD 2 on each local machine. p2p comm. and ∇ comp. are run independently in parallel.

number is set to 300 and 90 epochs respectively, following standard practice [22]. To circumvent the fuzziness of the notion of epoch in the asynchronous decentralized setting, we do not "split the dataset and re-shuffle it among workers at each epoch" as done with our standard All-Reduce baseline. Rather, we give access to the whole dataset to all workers, each one shuffling it with a different random seed. We use SGD with a base learning rate of 0 . 1 , a momentum value set at 0 . 9 and 5 × 10 -4 for weight decay. As advocated in [16], we do not apply weight decay on the learnable batch-norm coefficients. For ImageNet training with the SGD baseline, we decay the learning-rate by a factor of 10 at epochs 30, 60, 80 (epochs 50, 75 for CIFAR-10), and apply an analogous decay schedule with our asynchronous decentralized methods. All of our neural network parameters are initialized with the default Pytorch settings, and one All-Reduce averaging is performed before and after the training to ensure consensus at initialization and before testing. For our continuous momentum, we also need to set the parameters η, ˜ α . For all our experiments, we use the values given by Prop. 3.6. As advocated, the asynchronous baseline correspond to the setting without acceleration, i.e. with η = 0 and α = ˜ α = 1 2 , whereas using A 2 CiD 2 leads to consider η = 1 2 √ χ 1 χ 2 , α = 1 2 , ˜ α = 1 2 √ χ 1 χ 2 , where χ 1 , χ 2 are set to their theoretical value given by (2), (3) depending on the communication rate and graph's topology, assuming that each worker chose their peers uniformly among their neighbors (we verify empirically that it is the case in practice, see Appendix E.2).

Practical implementation of the dynamic. The dynamic studied in Eq. (4) is a model displaying many of the properties sought after in practice. In our implementation, described in Algo. 1, each worker i has indeed two independent processes and the DNN parameters and momentum variable { x i , ˜ x i } are locally stored such that both processes can update them at any time. One process continuously performs gradient steps, while the other updates { x i , ˜ x i } via peer-to-peer averaging. The gradient process maximizes its throughput by computing forward and backward passes back to back. Contrary to All-Reduce based methods that require an increasing number of communications with the growing number of workers, inevitably leading to an increasing time between two rounds of computations,

we study the case where each worker has a fixed communication rate, given as hyperparameter in our implementation. We implement 3 different graph topologies: complete, ring, and exponential [28, 2], see Appendix E.1 for details. To emulate the P.P.Ps for the communications, each worker samples a random number of p2p averaging to perform between each gradient computation, following a Poisson law using the communication rate as mean. To minimize idle time of the communication process, workers are paired with one of their neighbors in a "First In First Out" manner in an availability queue (a worker is available when it finished its previous averaging and still has some to do before the next gradient step). To implement this, we use a central coordinator to store the availability queues and the graph topology (this

0

2000

4000

6000

# grad steps per worker

10

3

10

2

10

1

10 0

Training loss

n=16

n=32

n=64

w/

A

2

CiD

2

w/o

A

2

CiD

2

is lightweight in a cluster: the coordinator only exchanges integers with the workers), but it could be done in different ways, e.g. by pinging each other at high frequency. As we assumed a unit time for the gradient process in our analysis, and that real time is used in our algorithm to apply our A 2 CiD 2 momentum (see Algo. 1), we maintain a running average measure of the duration of the previous gradient steps to normalize time.

4.2 Evaluation on large scale datasets

CIFAR10. This simple benchmark allows to understand the benefits of our method in a well-controlled environment. Tab. 4 reports our numerical accuracy on the test set of CIFAR10, with a standard deviation calculated over 3 runs. Three scenarios are considered: a complete, an exponential and a ring graph. In Fig. 3 (a), we observe that with the asynchronous baseline on the complete graph, the more workers, the more the training loss degrades. Fig. 3 (b) hints that it is in part due to an insufficient communication rate, as increasing it allows to lower the loss and close the gap with the All-Reduce baseline. However, this is not the only causative factor as Tab. 4 indicates that accuracy generally degrades as the number of

Table 5: Accuracy on ImageNet for a batchsize of 128. We compared a vanilla asynchronous pairwise gossip approach with and without A 2 CiD 2 , demonstrating the improvement of our method. We also varied the communication rate.

workers increases even for AR-SGD, which is expected for large batch sizes. Surprisingly, even with a worse training loss for n = 64 , the asynchronous baseline still leads to better generalization than

AR-SGD, and consistently improves the test accuracy across all tested values of n . The communication rate being identified as a critical factor at large scale, we tested our continuous momentum on the ring graph, each worker performing one p2p averaging for each gradient step. Fig. 4 shows that incorporating A 2 CiD 2 leads to a significantly better training dynamic for a large number of workers, which translates into better performances at test time as shown in Tab. 4.

0

10000

20000

30000

40000

50000

# grad steps per worker

10 0

2 × 10

0

3 × 10

0

4 × 10

0

6 × 10

0

Training loss

n=16

n=32

n=64

w/

A

2

CiD

2

w/o

A

2

CiD

2

(a)

0

5000

10000

15000

20000

25000

# grad steps per worker

10

1

10 0

10 1

10 2

l

2

distance

1 com/

2 com/

w/

A

2

CiD

2

w/o

A

2

CiD

2

(b)

ImageNet. For validating our method in a reallife environment, we consider the large-scale ImageNet dataset. Tab. 6 confirms the advantage of asynchronous methods by allocating less compute to the slowest workers, leading to faster training times. Tab. 5 reports our accuracy for

the complete and ring graphs. As χ 1 = χ 2 for the complete graph, we simply run our baseline asynchronous method for reference. The case of the ring graph is much more challenging: for n = 64 workers, the accuracy drops by 10% compared to the synchronous baseline given by AR-SGD. Systematically, with A 2 CiD 2 , the final accuracy increases: up to 4% absolute percent in the difficult n = 64 setting. This is corroborated by Fig. 5, which indicates that incorporating A 2 CiD 2 significantly improves the training dynamic on ImageNet. However, for reducing the gap with the AR-SGD baseline, it will be necessary to increase the communication rate as discussed next.

Consensus improvement. The bottom of Tab. 5, as well as Fig. 5 (b) study the virtual acceleration thanks to A 2 CiD 2 . Not only increasing communications combined with A 2 CiD 2 allows to obtain competitive performance, but Fig. 1 shows that doubling the rate of communication has an identical effect on the training loss than adding A 2 CiD 2 . This is verified in Fig. 5 (b) by tracking the consensus distance between workers: A 2 CiD 2 significantly reduces it, which validates the results of Sec. 3.4.

5 Conclusion

In this work, we confirmed that the communication rate is a key performance factor to successfully train DNNs at large scale with decentralized asynchronous methods. We introduced A 2 CiD 2 , a continuous momentum which only adds a minor local memory overhead while allowing to mitigate this need. We demonstrated, both theoretically and empirically, that A 2 CiD 2 substantially improves performances, especially on challenging network topologies. As we only focused on data parallel methods for training Deep Neural Networks in a cluster environment, in a future work, we would like to extend our empirical study to more heterogeneous compute and data sources, as our theory could encompass local SGD methods [39] and data heterogeneity inherent in Federated Learning [33].

References

L. Arnold. Stochastic differential equations. New York , 1974.

M. Assran, N. Loizou, N. Ballas, and M. Rabbat. Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning , pages 344-353. PMLR, 2019.

E. Belilovsky, M. Eickenberg, and E. Oyallon. Decoupled greedy learning of cnns. In International Conference on Machine Learning , pages 736-745. PMLR, 2020.

E. Belilovsky, L. Leconte, L. Caccia, M. Eickenberg, and E. Oyallon. Decoupled greedy learning of cnns for synchronous and asynchronous distributed learning. arXiv preprint arXiv:2106.06401 , 2021.

M. Blot, D. Picard, M. Cord, and N. Thome. Gossip training for deep learning. In Advances in Neural Information Processing Systems , volume 30, 2016.

S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah. Randomized gossip algorithms. IEEE Transactions on Information Theory , 52(6):2508-2530, 2006.

J. Chen, X. Pan, R. Monga, S. Bengio, and R. Jozefowicz. Revisiting distributed synchronous sgd. arXiv preprint arXiv:1604.00981 , 2016.

J. Daily, A. Vishnu, C. Siegel, T. Warfel, and V . Amatya. Gossipgrad: Scalable deep learning using gossip communication based asynchronous gradient descent, 2018.

J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems , 25, 2012.

A. Defazio, F. Bach, and S. Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems , 27, 2014.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

M. Even, R. Berthier, F. Bach, N. Flammarion, H. Hendrikx, P. Gaillard, L. Massoulié, and A. Taylor. A continuized view on nesterov acceleration for stochastic gradient descent and randomized gossip. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

M. Even, H. Hendrikx, and L. Massoulie. Decentralized optimization with heterogeneous delays: a continuous-time approach. arXiv preprint arXiv:2106.03585 , 2021.

A. Ghosh, S. Boyd, and A. Saberi. Minimizing effective resistance of a graph. SIAM Review , 50(1):37-66, 2008.

R. M. Gower, M. Schmidt, F. Bach, and P. Richtárik. Variance-reduced methods for machine learning. Proceedings of the IEEE , 108(11):1968-1983, 2020.

P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2015.

H. Hendrikx, F. Bach, and L. Massoulié. An accelerated decentralized stochastic proximal algorithm for finite sums. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

A. Koloskova, T. Lin, and S. U. Stich. An improved analysis of gradient tracking for decentralized machine learning. Advances in Neural Information Processing Systems , 34:11422-11435, 2021.

A. Koloskova, T. Lin, S. U. Stich, and M. Jaggi. Decentralized deep learning with arbitrary communication compression. arXiv preprint arXiv:1907.09356 , 2019.

A. Koloskova, N. Loizou, S. Boreiri, M. Jaggi, and S. Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning , pages 5381-5393. PMLR, 2020.

L. Kong, T. Lin, A. Koloskova, M. Jaggi, and S. Stich. Consensus control for decentralized deep learning. In International Conference on Machine Learning , pages 5686-5696. PMLR, 2021.

D. Kovalev, A. Salim, and P. Richtarik. Optimal and practical algorithms for smooth and strongly convex decentralized optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 18342-18352. Curran Associates, Inc., 2020.

A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Commun. ACM , 60(6):84-90, may 2017.

S. Li and T. Hoefler. Near-optimal sparse allreduce for distributed deep learning. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming , pages 135-149, 2022.

X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang, and J. Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems , 30, 2017.

X. Lian, W. Zhang, C. Zhang, and J. Liu. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning , pages 3043-3052. PMLR, 2018.

T. Lin, S. P. Karimireddy, S. U. Stich, and M. Jaggi. Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data. arXiv preprint arXiv:2102.04761 , 2021.

T. Lin, S. U. Stich, K. K. Patel, and M. Jaggi. Don't use large mini-batches, use local sgd. In International Conference on Learning Representations , 2020.

Y. Lu and C. De Sa. Optimal complexity in decentralized training. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 7111-7123. PMLR, 18-24 Jul 2021.

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics , 2016.

A. Nabli and E. Oyallon. DADAO: Decoupled accelerated decentralized asynchronous optimization. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 25604-25626. PMLR, 23-29 Jul 2023.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

M. Ryabinin, E. Gorbunov, V. Plokhotnyuk, and G. Pekhimenko. Moshpit sgd: Communication-efficient decentralized training on heterogeneous unreliable devices. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 18195-18211. Curran Associates, Inc., 2021.

K. Scaman, F. Bach, S. Bubeck, Y. T. Lee, and L. Massoulié. Optimal algorithms for smooth and strongly convex distributed optimization in networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 3027-3036. PMLR, 06-11 Aug 2017.

A. Sergeev and M. D. Balso. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799 , 2018.

S. U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations , 2019.

S. U. Stich. Unified optimal analysis of the (stochastic) gradient method, 2019.

T. Vogels, L. He, A. Koloskova, S. P. Karimireddy, T. Lin, S. U. Stich, and M. Jaggi. Relaysum for decentralized deep learning on heterogeneous data. Advances in Neural Information Processing Systems , 34:28004-28015, 2021.

Y. Yakimenka, C.-W. Weng, H.-Y. Lin, E. Rosnes, and J. Kliewer. Straggler-resilient differentially-private decentralized learning. In 2022 IEEE Information Theory Workshop (ITW) , pages 708-713. IEEE, 2022.

B. Ying, K. Yuan, Y. Chen, H. Hu, P. Pan, and W. Yin. Exponential graph is provably efficient for decentralized deep training. Advances in Neural Information Processing Systems , 34:13975-13987, 2021.

B. Ying, K. Yuan, H. Hu, Y. Chen, and W. Yin. Bluefog: Make decentralized algorithms practical for optimization and deep learning. arXiv preprint arXiv:2111.04287 , 2021.

B. Ying, K. Yuan, H. Hu, Y. Chen, and W. Yin. Bluefog: Make decentralized algorithms practical for optimization and deep learning. 2021.

Y. You, I. Gitman, and B. Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 , 2017.

B. Yuan, Y. He, J. Davis, T. Zhang, T. Dao, B. Chen, P. S. Liang, C. Re, and C. Zhang. Decentralized training of foundation models in heterogeneous environments. Advances in Neural Information Processing Systems , 35:25464-25477, 2022.

S. Zhang, A. E. Choromanska, and Y. LeCun. Deep learning with elastic averaging sgd. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Appendix

Table of Contents

A Notations

For n ∈ N ∗ the number of workers and d ∈ N ∗ an ambient dimension, for all t > 0 , the variable x t ∈ R n × d is a matrix such that x t = [ x 1 t , ..., x n t ] T , with x i t ∈ R d for all i ∈ { 1 , ..., n } . We remind that 1 is the vector of n ones such that ¯ x = 1 n ∑ n i =1 x i = 1 n x T 1 ∈ R d . With I the identity and ∥ . ∥ F the matrix Frobenius norm, we write π = I -1 n 11 T the projection so that ∥ πx ∥ 2 F = ∑ n i =1 ∥ x i -¯ x ∥ 2 .

For a continuously differentiable function f : R d → R and a, b ∈ R d , the Bregman divergence is defined with d f ( a, b ) = f ( a ) -f ( b ) -⟨∇ f ( b ) , a -b ⟩ . For Λ ∈ R n × n , we denote by Λ + its pseudo inverse. For a positive semi-definite Λ ∈ R n × n , and x ∈ R n × d , we introduce ∥ x ∥ 2 Λ ≜ Tr( x T Λ x ) and Λ 1 / 2 its square-root. We recall that the connectivity between workers is given by a set of edges E , and denote by e i the i th basis vector of R n .

For x ∈ R n × d , we introduce:

Finally, to study the gradient steps taken on each individual worker independently, we introduce:

B Technical Preliminaries

We recall some basic properties that we will use throughout our proofs.

Lemma B.1 (Implications of Assumption 3.4) . If each f i is µ -strongly convex and L -smooth, we have, for any a, b ∈ R d :

and

Lemma B.2 (Generalized triangle inequality) . For any a, b, c ∈ R d and continuously differentiable function f : R d → R , by definition of the Bregman divergence, we have:

Lemma B.3 (Variance decomposition) . For a random vector a ∈ R d and any b ∈ R d , the variance of a can be decomposed as:

Lemma B.4 (Jensen's inequality) . For any vectors a 1 , ..., a n ∈ R d , we have:

Lemma B.5. For any vectors a, b ∈ R d and α > 0 :

Lemma B.6. For any vectors a, b ∈ R d and α > 0 :

Lemma B.7. For any A ∈ R n × d and B ∈ R n × n , we have:

Lemma B.8 (Effective resistance contraction) . For ( i, j ) ∈ E and any x ∈ R n × d , we have:

Proof. Indeed, we note that, by definition of χ 2 (3):

Lemma B.9. For any x ∈ R n × d , and Λ the Laplacian of a connected graph, we have:

Proof. Indeed, by definition of the Laplacian Λ (3.1), we have:

Lemma B.10. For any x, y ∈ R n × d , and Λ the Laplacian of a connected graph, we have:

Proof. By property of the Laplacian of a connected graph, we have that π = (Λ + ) 1 / 2 Λ 1 / 2 . Thus:

C Proof of the main result of this paper

We remind that we study the following dynamic:

which simplifies to:

if A 2 CiD 2 is not applied. We also recall the main proposition, which we prove next:

Proposition C.1 (Convergence guarantees.) . Assume that { x t , ˜ x t } follow the dynamic Eq. 4 and that Assumption 3.2-3.3 are satisfied. Assume that 1 ¯ x 0 = x 0 = ˜ x 0 and let T the total running time. Then:

Non-accelerated setting , we pick η = 0 , α = ˜ α = 1 2 and set χ = χ 1 ,

Acceleration (A 2 CiD 2 ) , we set η = 1 2 √ χ 1 χ 2 , α = 1 2 , ˜ α = 1 2 √ χ 1 χ 2 , and χ = √ χ 1 χ 2 ≤ χ 1 .

Then, there exists a constant step size γ > 0 such that if:

( strong-convexity ) the Assumption 3.4 is satisfied, then γ ≤ 1 16 L (1+ χ ) and 1 :

( non-convexity ) the Assumption 3.5 is satisfied, then there is c > 0 which depends only on P, M from the assumptions such that γ ≤ c L ( χ +1) and:

Also, the expected number of gradient steps is nT and the number of communications is Tr(Λ) 2 T .

Proof. The core of the proof is to introduce the appropriate Lyapunov potentials ϕ k ( t, X ) , where X could be ( x, ˜ x ) or x depending on whether we apply A 2 CiD 2 or not. If we apply A 2 CiD 2 , we introduce the momentum matrix A = ( -η η η -η ) . Then, by Ito's lemma, given that all the functions are smooth, and remembering that all Point-wise Poisson Processes N i t have unit intensity, that the M ij t have intensity λ ij and that they are all independent, we obtain in a similar fashion to [12, 34]:

where M T is a martingale. In the case where A 2 CiD 2 is not applied, we set A = 0 to remove the momentum term, and all updates are done only along x as there is no ˜ x . We remind that:

We now present our choice of potential for each cases:

For the convex case in the non-accelerated setting, we introduce:

For the convex case with A 2 CiD 2 , we introduce:

For the non-convex case in the non-accelerated setting, we introduce:

For the non-convex case with A 2 CiD 2 , we introduce:

C.1 Some useful upper-bounds

As the same terms appear in several potentials, we now prepare some intermediary results which will be helpful for the proofs.

First, we study the variations in the ∥ ¯ x -x ∗ ∥ 2 term appearing in ϕ 1 and ϕ 2 . As the updates due to the communication are in the orthogonal of 1 , it is only necessary to study the variations induced by the gradient steps. Thus, we define:

We note that ∇ ˜ F i ( x, ξ ) = 1 n ∇ F i ( x i , ξ i ) , which, using ∑ i ∇ f i ( x ∗ ) = 0 , leads to:

Study of the d f (¯ x, x ∗ ) terms:

Next, using the same reasoning as for ∆ x , we also need to only study the gradient updates in the non-convex setting for the first part of ϕ 3 , ϕ 4 . Thus, we set:

First, it is useful to note that under Assumption 3.5, using (B.3), we have:

Then, using ∑ i ∇ f i ( x ∗ ) = 0 and the L -smoothness of f = 1 n ∑ i f i , we get:

As we also have:

We get in the end:

Remark C.2 . As observed in (5), note that for both the terms ∥ ¯ x -x ∗ ∥ 2 and d f (¯ x, x ∗ ) , as we are considering ¯ x and 1 n 11 T π = 0 , Poisson updates from the communication process amount to zero. Moreover, as 1 n 11 T ( x -˜ x ) = 0 , the update from the momentum is also null for these terms.

Study of the ∥ πx ∥ 2 F terms:

We get from the Poisson updates for the gradient processes:

and, using the definition of the Laplacian Λ (3.1), we get from the communication processes:

Putting together both types of Poisson updates, we define:

For α = 1 2 , we get:

For A 2 CiD 2 , we add the momentum term ⟨ η (˜ x -x ) , 2 πx ⟩ to define:

Study of the ∥ ˜ x ∥ 2 Λ + terms:

These terms only appear in the Lyapunov potentials used when applying A 2 CiD 2 . From the Poisson updates and momentum, we get:

Taking the expectation and using (B.9), (B.8), (B.7), (B.5) leads to:

C.2 Resolution: putting everything together

In this part, we combine the terms for each potential. We remind that with Assumption 3.4:

Convex case, non-accelerated. We remind that

Then, using (23), (35) and defining Ψ ≜ ∂ ϕ ( t, X ) + E [ A ∆ x + B ∆ ]

We pick α = 1 2 , B t = 1 n A t , with A t = e -rt (we denote by r the rate of the exponentials A t , B t ). Then (43), (44) imply:

As we want (45) to be negative, we have:

and taking γ ≤ 1 2 1 2 L (3+4 χ 1 +1) = 1 16 L (1+ χ 1 ) works. Now, as 1 4 χ 1 -Lγ ≥ 3 16 χ 1 and µγ ≤ 1 16 χ 1 , we pick r = µγ . As we have:

using (46) and (16) leads to:

which leads to:

Convex case with A 2 CiD 2 . We remind that

Then, using (23), (37), (39) and defining Ψ 2 ≜ ∂ t ϕ 2 ( t, X t ) + E [ A t ∆ x + B t ∆ 1 Λ + ˜ B t ∆ 2 Λ ] , we have:

Then, we assume α = 1 2 , ˜ α = 1 2 √ χ 1 χ 2 , η = 1 2 √ χ 1 χ 2 , B t = 1 n A t , ˜ B t = 1 χ 1 B t , A t = e -rt (we denote by r the rate of the exponentials A t , B t , ˜ B t ), which satisfies (55) and (56). Then (52), (53), (54) imply:

As we want (57) to be negative, we have:

and taking γ ≤ 1 2 L ( 6+ 4 η +2 ) = 1 16 L ( 1+ √ χ 1 χ 2 ) works. Now, we have:

As µγ ≤ µ 16 L ( 1+ √ χ 1 χ 2 ) ≤ 1 16 √ χ 1 χ 2 , taking r = µγ works. Finally, using (58) and (16) leads to:

Non-convex case, non-accelerated. We remind that:

Here, we pick α = 1 2 , A t = 1 , B t = L n A t . Thus, A ′ t = B ′ t = 0 . Then, using (32), (35), (31), (25)

which leads to:

Our goal is to use half of the negative term of (64) to cancel the positive ones, so that there remains at least -γ 4 A t ∥∇ f (¯ x ) ∥ 2 in the end. Thus, we want:

and taking γ ≤ 1 48( M +1)( P +1)(1+ χ 1 ) works. We verify that with γ defined as such, (65) is also negative. Finally, we upper bound (66) with 3 Lγ 2 ( σ 2 +3( M +1+4 χ 1 ) ζ 2 ) . As we have:

we note that if γ ≤ c L ( χ 1 +1) for some constant c > 0 which depends on M,P , we will get:

which also writes:

Non-convex case, with A 2 CiD 2 . We have:

Here, we pick α = 1 2 , η = 1 2 √ χ 1 χ 2 , A t = 1 , B t = L n A t , B t = χ 1 ˜ B t and an identical reasoning to the convex setting allows to say we can find a constant c > 0 such that if γ ≤ c L (1+ √ χ 1 χ 2 ) , then:

C.3 Optimizing the step-size

In this part, we follow [40, 21] and optimize the step-size a posteriori. We set χ = χ 1 for the non-accelerated setting and χ = √ χ 1 χ 2 with A 2 CiD 2 .

Strongly-convex cases: From (51) and (63), we can write that, for γ ≤ 1 16 L (1+ χ ) and initializing x 0 such that πx 0 = 0 , we have:

Then, taking the proof of [40] and adapting the threshold, we consider two cases (with r 0 ≜ ∥ ¯ x 0 -x ∗ ∥ 2 ):

if 1 16 L (1+ χ ) < log(max { 2 ,µ 2 r 0 T/σ 2 } ) µT , then we set γ = 1 16 L (1+ χ ) .

Then, (72) gives:

Non-convex cases: From (70) and (71), we can write that, for some constant c > 0 depending on M,P such that γ ≤ c L (1+ χ ) , we have:

Then, taking the proof of Lemma 17 in [21] and adapting the threshold, we consider two cases (with f 0 ≜ f ( x 0 ) -f ( x ∗ ) ):

D Comparison with accelerated synchronous methods

By definition of Λ (3.1), our communication complexity (the expected number of communications) is simply given by Tr(Λ) 2 per time unit. As discussed in Sec. 3.5, our goal is to replicate the behaviour of accelerated synchronous methods such as DeTAG [31], MSDA [37] and OPAPC [23] by communicating sufficiently so that the graph connectivity does not impact the time to converge, leading to the condition √ χ 1 [Λ] χ 2 [Λ] = O (1) .

Now, let us consider a gossip matrix W as in [31, 37, 23] ( i.e. , W is symmetric doubly stochastic) and its Laplacian L = I n -W . Then, using Λ = √ χ 1 [ L ] χ 2 [ L ] L is sufficient for having √ χ 1 [Λ] χ 2 [Λ] = O (1) .

Synchronous methods: between two rounds of computations ("steps"), the number of communication edges used is |E| √ 1 -θ with θ = max {| λ 2 | , | λ n |} the eigenvalues of W .

Ours: the number of communication edges used per time unit for our method is Tr(Λ) 2 = 1 2 √ χ 1 [ L ] χ 2 [ L ]Tr( L ) .

As, in [31, 37, 23], each communication edge is used at the same rate, we can apply Lemma 3.3 of [34] stating: √ χ 1 [ L ] χ 2 [ L ]Tr( L ) ≤ √ ∥L∥ χ 1 ( n -1) |E| . We have:

W is stochastic: ∥L∥ ≤ 2 .

the graph is connected: n -1 ≤ |E| .

definition of χ 1 and θ : 1 -θ ≤ 1 χ 1 [ L ]

Thus, √ χ 1 [ L ] χ 2 [ L ]Tr( L ) ≤ √ 2 |E| √ 1 -θ , which proves that our communication complexity per time unit is at least as good as any accelerated synchronous method.

E Experimental details

E.1 Graph topologies

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

0

15

14

12

8

1

2

4

3

5

9

13

10

6

7

11

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Fig. 6 displays an example of each of the three graph topologies implemented. The exponential graph follows the architecture described in [28, 2]. Note the discrepancy between the values of χ 1 and χ 2 for the cycle graph, highlighting the advantage of using A 2 CiD 2 in the asynchronous setting (to lower the complexity from χ 1 to √ χ 1 χ 2 ).

E.2 Uniform neighbor selection check

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

0

25

50

75

100

125

150

175

200

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

0

100

200

300

400

500

600

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

0

250

500

750

1000

1250

1500

1750

Our asynchronous algorithm acts as follows: to reduce latency, the first two workers ( i.e. , GPUs) in the whole pool that declare they are ready to communicate ( i.e. , finished their previous communication and have to perform more before the next gradient step) are paired together for a p2p communication if they are neighbors in the connectivity network. During training, we registered the history of the pairwise communications that happened. Fig. 7 displays the heat-map of the adjacency matrix, confirming that our assumption of "uniform pairing among neighbors" (used to compute the values of χ 1 , χ 2 ) seems to be sound.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work discusses the challenges and potential solutions related to training complex Deep Neural Networks (DNNs), particularly regarding the computational and communication demands. The traditional synchronous, centralized approaches to DNN training, while widely used, face limitations in terms of efficiency and scalability, which can be tackled by distributed training methods.\n\nAsynchronous and decentralized methods are suggested as they allow for more efficient parallelization of computations and communications, using time-delay fluctuations between workers. Such methods eliminate the need for a central worker to aggregate results, allow nodes to contribute in proportion to their available resources, and use peer-to-peer communication to streamline training. However, the complexity of these methods and the large number of parameters in DNNs still pose considerable communication challenges.\n\nThe authors address these issues by introducing a novel acceleration method, A2CiD2 (Accelerating Asynchronous Communication in Decentralized Deep Learning), specifically for peer-to-peer DNN training. This method uses pair-wise gossip acceleration, which is largely unexplored for deep learning, and is supported by the analytical framework of Stochastic Differential Equations (SDEs). It decouples computations and communications, requiring minimal overhead and enhancing communication rates.\n\nKey contributions of this work include extending the asynchronous decentralized deep learning training framework to non-convex settings, proposing the A2CiD2 mechanism for improving communication efficiency, and minimizing the gap between centralized and decentralized settings in environments with up to 64 asynchronous GPUs. The method has been implemented in PyTorch and will be released as open-source upon publication.\n\nSTRENGTHS:\nThe strengths of this paper are as follows:\n\n1: Within the realm of deep learning, the widespread use of High-Performance Computing (HPC) technology has made it possible to achieve exceptional performance in a synchronous and centralized environment. However, not everyone has the means to afford such costly setups, making the development and evaluation of distributed asynchronous algorithms crucial. The authors propose a cost-effective method that significantly enhances communication speed.\n\n2: The A2CiD2 algorithm effectively minimizes the discrepancy between centralized and distributed setups. It functions efficiently in environments hosting up to 64 asynchronous GPUs. This characteristic enhances the algorithm's scalability, rendering it suitable for large-scale machine learning tasks.\n\n3: This study broadens the analytical framework for investigating the design and convergence of these algorithms in non-convex settings. It provides new insights into asynchronous distributed deep learning training.\n\n4: The method proposed substantially improves communication efficiency in distributed learning environments. This could assist in addressing common challenges, such as the straggler problem, synchronization between computation and communication, and bandwidth limitations.\n\nWEAKNESSES:\nThe weaknesses of this paper can be outlined as follows:\n\n1: While the theoretical analysis provided in Section 3.3 is well substantiated, its tightness with respect to the upper bound remains ambiguous. Further explanation is needed on how the asymptotic convergence corresponds with the experimental results for relatively smaller sizes, like those discussed in Section 4.\n\n2: The study mentions that the proposed A2CiD2 algorithm has been tested with up to 64 asynchronous GPUs. While this is quite remarkable, it may not comprehensively represent the broad array of hardware configurations and scales found in real-world applications. A broader evaluation of the method across different scales and architectures would be advantageous.\n\n3: The paper lacks a clear comparison with other existing distributed training methods, be they synchronous or asynchronous, in terms of computational cost, communication overhead, and performance. Such a comparison could help determine whether the proposed method truly surpasses other methods and under which specific conditions.\n\n4: The paper asserts that training Deep Neural Networks (DNNs) with decentralized methods necessitates substantial communications due to the large quantity of optimized parameters. It remains unclear whether the proposed acceleration method sufficiently addresses this issue and how it compares with other techniques that aim to reduce communication overhead.\n\nQUESTIONS:\nIn addition to the issues delineated as weaknesses above, I would be grateful if you could respond to the following queries:\n\n1: In Section 3.1, \"Model for a Decentralized Environment,\" you presuppose the symmetry of communication. Could you elaborate on this assumption and explicate its implications for your model?\n\n2: In Section 3.3, \"Theoretical Analysis of A2CiD2,\" you propose a Poisson process. Nevertheless, the validity of this assumption remains nebulous. Could you furnish some justification or explanation for this assumption?\n\n3: In Section 4, \"Numerical Experiments,\" you cite the utilization of 64 A100 GPUs. Could you provide more details about the network configuration they are connected to, such as Infiniband? The performance of the network itself is likely to affect the results of this experiment, and this information would help elucidate the experimental setup.\n\nLIMITATIONS:\nN/A : No pertinent content is found in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

comments: SUMMARY:
This work, A2CiD2, proposes a novel decentralized asynchronous training method that incurs only minimal overhead but effectively decouples communications and computations, accelerating pair-wise communications via a provable, accelerated, randomized gossip procedure based on continuous momentum and time.  A2CiD2 is also compatible with other asynchronous approaches. Across ResNet benchmarks on image classification, A2CiD2 is able to outperform AllReduce-SGD and AD-PSGD on a cluster of 64 asynchronous workers
with A100 GPUs, using various communication network topologies.

STRENGTHS:
+. Proposed a decentralized asynchronous training algorithm that outperforms the SOTA approaches

+. Demonstrated the advantage of the proposed approach both theoretically and empirically

+. Provided code for reproducibility

WEAKNESSES:
-. Writing needs improvement. Hard to follow.

-. Missing benchmark: 
1. only ResNet is evaluated (e.g., ResNet18 and ResNet50), how about more models like RNNs and larger models like GPT2?

2. only image classification tasks is evaluated, how about more tasks like language modeling?

-. Missing details in experiments: 
1. what is the network type and bandwidth?
2. how many GPUs per machine? 

-. Missing evaluation: 
1. what is the time-to-accuracy or time-to-loss speedup for AC2CiD2 over AllReduce-SGD and ADPSGD?

2. what is performance of AC2CiD2 with random stranger in the cluster?

QUESTIONS:
See Weakeness

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-04

comments: SUMMARY:
This paper proposes an asynchronous gossip-based algorithm for decentralized deep learning by using a continuous momentum. Experiments on real datasets are used for evaluation.

STRENGTHS:
1. The studied problem about accelerating communication in decentralized deep learning is interesting.

2. The proposed algorithm accelerates the asynchronous baseline both theoretically and empirically.

WEAKNESSES:
1. The writing can be improved. There are many typos and grammatical errors. Furthermore, there are many informal representations, such as “We demonstrate its efficiency theoretically and numerically; empirically on the ring graph…” in abstract.

2. The proposed method needs to store one more copy of the model, compared with baselines. Hence, it may not be suitable for large models.

3. The experimental results are not convincing. The main baseline for comparison is All-Reduce SGD which has very high communication cost. There have existed many sophisticated decentralized methods, which should be adopted for comparison. 

4. This work only considers undirected network topology, which means that the communications are symmetric. But in recent years, directed topology has attracted more and more attention.

QUESTIONS:
Is the formula between Line 139 and Line 140 correct?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a new method for decentralized optimization that leverages the notion of continuous momentum to speed up its convergence. The method is justified with theoretical analysis and large-scale experiments on the ImageNet dataset.\n\nSTRENGTHS:\n* The work studies an important problem of distributed training over communication-constrained networks\n* The results of authors have both theoretical justification and empirical validation in a large-scale setting\n* Overall, the approach is clearly explained and the contributions are easy to understand\n\nWEAKNESSES:\n* The primary disadvantage of the proposed method is its memory overhead. Having an additional copy of model parameters on each worker is quite expensive for models where the memory footprint of the parameters dominates that of the activations (in particular, for Transformer models). As a result, it might be quite difficult to apply A$^2$CiD$^2$ to models beside convolutional networks (for example, to train modern language models), which are a highly popular area of application for distributed training\n* I feel that the statement in L241 needs to be clarified. All-Reduce methods indeed require more *connections* with the growth in the number of workers, but the total amount of bandwidth is asymptotically independent of the network size for methods like Ring All-Reduce\n* From my understanding, momentum acceleration for decentralized DL has already been studied in the past (e.g., [1], which has been cited in the submission), although the underlying frameworks are clearly different. I think this work needs to more explicitly distinguish their approach from the QG-DSGDm and other results (e.g., [2]), and ideally include those methods as an additional baseline on top of AD-PSGD\n\n[1] Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data. Tao Lin, Sai Praneeth Karimireddy, Sebastian U. Stich, Martin Jaggi. ICML 2021\n\n[2] SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum. Jianyu Wang, Vinayak Tantia, Nicolas Ballas, Michael Rabbat. ICLR 2020\n\nQUESTIONS:\n* In L197, what is the connection speed between the nodes?\n* What was your motivation for choosing the ring graph topology in the experiments? Is this topology frequently used in practice for peer-to-peer networks?\n* Is it correct that you consider standard All-Reduce to be a centralized method? (e.g., L80) From my understanding, there is no central worker involved when aggregating updates with this family of algorithms\n\nLIMITATIONS:\nI did not see any explicit discussion of limitations of the work: I would be happy if they mentioned the applicability of their method to models with a larger parameter count (for example, Transformers with hundreds of millions or billions of parameters).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-28"}