From Promise to Practice: Realizing High-performance Decentralized Training
Authors: Zesen Wang
Jiaojiao Zhang
Xuyang Wu
Mikael Johansson
Venue: ICLR 2025 Poster (2025)
Keywords: distributed training
data parallelism
decentralized optimization
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: A system-level design for practical decentralized training in multi-GPU clusters

Abstract: Decentralized training of deep neural networks has attracted significant attention for its theoretically superior scalability compared to synchronous data-parallel methods like All-Reduce. However, realizing this potential in multi-node training is challenging due to the complex design space that involves communication topologies, computation patterns, and optimization algorithms. This paper identifies three key factors that can lead to speedups over All-Reduce training and constructs a runtime model to determine when and how decentralization can shorten the per-iteration runtimes. To support the decentralized training of transformer-based models, we introduce a decentralized Adam algorithm that overlaps communications with computations, prove its convergence, and propose an accumulation technique to mitigate the high variance caused by small local batch sizes. We deploy our solution in clusters with up to 64 GPUs, demonstrating its practical advantages in both runtime and generalization performance under a fixed iteration budget.
The experiment code is open-source at [https://github.com/WangZesen/Decentralized-Training-Exp](https://github.com/WangZesen/Decentralized-Training-Exp), and the extension code is open-source at [https://github.com/WangZesen/Decent-DP](https://github.com/WangZesen/Decent-DP).

Published as a conference paper at ICLR 2025
FROM PROMISE TO PRACTICE: REALIZING HIGH-PERFORMANCE DECENTRALIZED TRAINING
Zesen W Jiaojiaoo Zhang', Xuyang Wu', Mikael Johansson" KTHH Royal Institute of Technology, 'Southern University of Science and Technology (zesen, jiaoz] @kthh se, wuxy6esustech. edu. cn, mikaelj@kth.
ABSTRACT
Decentralized training of deep neural networks has attracted significant attention for its theoretically superior scalability compared to synchronous datalparallll methods like All-Reduce. However, realizing this potential multi-node training is chal- lenging due to the complex design space that involves communication topologies, computation patterns, and optimization algorithms. This paper identifies three key factors that can lead to speedups over All-Reduce training and constructs runtime model to determine when and how decentralization can shorten the per-iteration runtimes. To support the decentralized training of transformer- based models, we introduce decentralized Adam algorithm that overlaps communications with computations, prove its convergence, and propose an accumulation technique to mitigate the high variance caused by small local batch sizes. We deploy our solu- tion in clusters with up to 64 GPUs, demonstrating its practical advantages in both runtime and generalization performance under fixed iteration budge
INTRODUCTION
With the rapid advancement of deep neural networks (DNNs), distributed training has become the mainstream approach for efficiently scaling up models. Among the various parallelization strategics, data parallelism has emerged as the most straightforward and commonly used approach for distributing training across multiple devices. One of the most popular algorithms used in data nata-parallel training is All-Reduce (Li al. 2020), known for its simplicity and its ability to maintain consistency with angle-machine training.
However, All-Reduce training relies on high-speed network connections and homogeneous computa- tional devices to ensure its efficiency (Zhang et al. 2020 Tandon et al. (017). Large companies with abundant resources address these challenges by investing in specialized high-performance computing (HPC) clusters optimized for large-scale distributed training (Shoeybi al. 2019). These clusters are equipped with cutting-edge uniform hardware and high-speed interconnects, making them ideal for All-Reduce-based training. Unfortunately, such resources are often inaccessible the broader research community. Most researchers rely on self-built servers or shared clusters by universities or cloud service providers, which may lack the high-speed networking infrastructure necessary fully leverage All-Reduce, thereby limiting the potential of the available computational resources.
Decentralized algorithms, which originally gained attention in the fields of consensus algorithms (Johansson et al. 2007] [Shi et al., 2015) and privacy-preservee have recently been explored as alternatives to All-Reduce in distributed training, especially in environments with suboptimal network conditions. By replacing the costly All-Reduce operations with decentralized communication patterns, these algorithms can significantly reduce communication overhead. Over the past few years, decentralized algorithms have demonstrated promising practical speedup in setups with poor network performance (Lian et al.) 2017:: Assran et al.) 2019 Yuan et al. 2022). Research in this area has focused on the convergence properties of gossiping different communication topologies (Ying et al.
techniques (Yan et al.) (012)).
2021a; Takezawa et al. 2024), decentralized optimization 2021). and alternative communication patterns (Cattivelli &
algorithms (Lin al. 2021 Yuan al.
'The experiment code is open-source at http://hhhbbbb////////////////ee Trainiigg-d-aiin---E----- and the extension ende open-source at http://gttaaaaam////////////////

---

Published as a conference paper at ICLR 2025
Sayed 2009]. Despite this growing interest, decentralized approaches have yet to achieve the same level of popularity as All-Reduce-base methods in distributed training.
We believe there several reasons for this: (1) Simply combining the best of each line of work does not necessarily lead to an effective overall system. The potential speedup of decentralized algorithms in real-world environments remains underexplored and poorly understood (Assran et al., 2019). (2) The generalization performance of models trained with decentralized methods often falls short compared those trained with All-Reduce under the same iteration budget, raising concerns about the effectiveness of decentralized training (Lian et al., 2017 Assran et al. (019). (3) Despite the growing interest in large language models (LLMs), there is still no well-suited decentralized version of the Adam optimizer, which is crucial for efficiently training cansformer-basee models in a decentralized setting.
This paper aims to develop a deeper understanding of decentralized training in practical setups and includes the following key contributions:
We propose a simple yet accurate runtime model that quantifies the impact of key environ- mental parameters and estimates potential speedups. This model enables the optimization of decentralized training configurations to achieve the best performance in different settings.
2. We design and analyze a decentralized variant of the Adam optimizer. Our algorithm leverages a collective gossip communication mechanism to enable parallelism between computation and communication. In addition, it introduces a novel accumulation technique that enhances performance and delivers promising experimental results.
3. We implement PyTorch extension for decentralized training across multiple GPUs and nodes. Our implementation demonstrates strong performance on typical machine learning workloads, including image classification, machine translation, and GPT-2 pre-training. Extensive experiments validate the feasibility and practical benefits of decentralized training.
1.1 RELATED WORK
Decentralized training The combination of SGD updates with gossip communication (Lin et al.
2021; Yuan et al., 2021) has gained significant attention for its theoretical benefits: it retains the convergence rate as centralizee//yychrroouu mini-batch SGD at a reduced communication cost.
One line of work (Neglia et al. 2020 Koloskova et al., 2020 Ying et al. 2021a] Takezawa et al. 2024) focuses on the design of the communication topology to improve the guaranteed convergence rate in terms of the number of iterations. However, these papers demonstrate improvements in the number of iterations but do not show if these results translate to actual speedups in terms of total training time. A handful of more practical studies, e.g., (Lian al., 2017 Assran et al. [019) report actual speedups of decentralized training with multiple workers, but they do not isolate system-level bottlenecks or bound the potential speedups. Moreover, Lian et al. (2017) focuses on low-bandwidth scenarios, and the implementation in Assran et al. (2019) groups GPUs in node as a single unit for decentralization, which makes sensitive Intra-node stragglers. Another line of work that is orthogonal to decentralized data parallelism is the slow momentum (Wang et al.) 2219)) Lin et al. 202T), which maintains an outer-loop momentum to improve the training loss. However, the method introduces additional communication and new hyperparameters.
Variance in computation time In the training of language models, one way to preprocess the input data is to group samples with similar lengths (Vaswani al. 2017). but this may introduce the straggler effect because of non-uniform batches. Increasing the number of local mini-batches (Ott al. 2019) or dropping samples on stragglers (Giladi al., 2024) mitigate the problem but may worsen the generalization performance by altering the effective batch size (Keskar et al., (016)). In pre-training L.Mss it is common to pack the sequences from different documents (Raffel et al.. 2020) to create uniform batches. However, the sequence pack technique has a negative influence on the training quality (Ding et al., 2024). Another possibility is to pad sequences to a fixed length (Smith al. 2022). but this leads to lower resource utilization. Moreover, the methods are not compatible with tasks like video processing and seq-to-seq tasks. As for the performance of the decentralized training
imbalanced workloads, in Assran et al. (2019), the authors argued that decentralized training is more resilient to computation time variance by showing a smaller runtime variance. However, the speedup advantage brought by decentralized training remains unexplored.
2

---

Published as a conference paper at ICLR 2025
U1
U
0/
Computation
Bs
B2
B2
Communication
Ci
Figure I: Timelines and dependency relations of decentralized training. F: forward pass, B: backward pass, C:: decentralized communicaiion/agrraaaao of model parameters with other workers, U: update model parameters. An arrow from task X to Y means that task Y can start only after X finishes.
Orthogonal scalability techniques There are many other ways to improve scalability that are orthogonal to this work. This includes compression of gradients (Vogels et al. 2019) and models (Tang et al.) 2018) to reduce the communication time, as well as model parallelism (Xu & You 2023)
and pipeline parallelism (Li & Hoefler 2021) scale up distributed training. These approaches could potentially be combined with our work to improve its scalability further.
2 IMPORTANT ASPECTS OF MULTI-NODE DECENTRALIZEI TRAINING
The basic idea behind many decentralized optimization algorithms is to replace the global model parameter x R by local copies 2; R each worker 1,2, ...... NN and require that the local models reach consensus. In this spirit, a decentralized learning problem can be written as
Σ F((2.;; subject to x; = Ø3 Vi,J.
minimize {
where (x;) is the expected training loss of the local model x, on the data held by worker i. In our theoretical analysis, let F((x:) [E where from the distribution D, and {(.) is a non-convex but smooth function. The assumption that each worker can draw samples from the same distribution D is reasonable if not for the privacy-preserving
local data point of worker i drawn
purpose Yiing et al. 2021a Lian et al. (017)).
Decentralized algorithms for solving ① allow the local model z; disagree during the training process but drive them towards the same stationary point of the global training loss. We focus on adaptive momentum versions of decentralized gradient descent,
Σ
(0) w
((((1)) Here, (t) is the learning rate, - d is update direction that reduces the value of F; at a and the final term accounts for the averaged model parameters of neighboring workers and drives the local parameters towards the same first-order stationary point. For decentralized SGD, d/d
vi(x!' "; (() while the expression is more complex for the adaptive momentum variations that develop in $ $ The weights w/f satisfy wif > 0 if and j communicate iteration / or i j. and 0 otherwise. We define NT as the set of workers j (including itself) such that w The weights
are selected so that if dea) = 0 for all i and all / 2 the local models will converge to the same value; see 4 for details.
Next, we will describe three factors that are essential to consider in order to implement efficiently.
2.1 EXPLOITING OVERLAPPING COMMUNICATION AND COMPUTATION
Modern accelerators like Nvidia GPUs are capable of interleaving communications with computations. After deciding which training scheme to use, hiding communication by computation is one of the key optimizations in the design an efficient distributed training system. For the decentralized implementation of (2). general idea of achieving is to restrict the search direction d, only depend on the local model. While for DNN training, we can take this idea further.
Inspired by the gradient bucketing technique used in data fata-parallel training in Py Torch (Li al. 2020), we divide both gradients and models into buckets corresponding to specific layers or groups of layers. Due the sequential nature of backpropagation,
begin communicating the gradients
3

---

Published as a conference paper at ICLR 2025
Setups \ 'Topology
Complete
One-peer Exp.
One-peer Ring
Alternating Exp Ring
4x4xA100, 100Gbps
55.31 s
55.44 s
32.97
29.55
4x4xA40, 25Gbps
395.99 s
429.47
217.90
185.93
Table 1: Communication time of performing averaging operations for 8196 iterations over three
25MB FP32 tensors (selection of 25MB is based on the default bucket size used in PyTorch DDP (Li ett al. 2020)). Setup I: 16 A A100 GPUs on 4 nodes inter-connected by 100Gbps Infiniband. Setup 2: 16 A40 GPUs on 4 nodes inter-connected by 25Gbps Ethernet. Here, Complete means global averaging. All other topologies are illustrated in Appendix A.4
of earlier finished layers while the gradients of later layers are still being computed. As illustrated in [ we split the execution of the backward pass into buckets and interleave the corresponding local model updates. After updating bucket, the corresponding communication can be initiated, and its results are not needed by the worker and its neighbors until the update of the same bucket in the next iteration. This significant overlap is possible because we have made the decentralized update independent of neighbor information within the same iteration. For All-Reduce training which aggregates gradients instead of models, communication cannot overlap with the forward pass and the backward pass of the last bucket. This is because the forward pass needs the complete aggregation of gradients from the last iteration, and no gradient is available before the backward pass of the last bucket. This indicates that decentralized training, using the scheme described in equation 2 can utilize simultaneous computational resources more efficiently.
Making the update direction calculation independent of the local models at other workers in the same iteration essential for overlapping computations and communications. Updates on the form a!!" ΣjENço w/y (xi) (x-(-") dd ()). as used in (Lin et al. 2021 Gao & Huang 2020),, cannot
easily achieve this; see Appendix B.2
2.2 RESPECTING HETEROGENEOUS COMMUNICATION COSTS
A key advantage of well-designed decentralized training over All-Reduce training is the reduced com- munication cost due to the sparsity of gossip communication. Unlike All-Reduce, where all workers must participate in each iteration, decentralized training allows workers communicate only with their immediate neighbors. A drawback with gossiping is that it only provides an approximate average that converges asymptotically the true global average. Several studies have characterized how the convergence rate of gossiping, affecting both iteration complexity and generalization performance in decentralized training, is influenced by the communication topology (Xiao & Boyd [2044] (Shi et al.) 2015, Nedić et al. 2018: Ying et al., 2021a Takezawa et al. 2024). In practical cluster environments, latency and bandwidth can vary significantly between nodes. While communication between GPUs within single compute node is very fast, communication between nodes is slower and subject to congestion at switches and other network interfaces. Thus, when designing a communication topology for decentralized training in such environments, it is crucial to consider both the degree of connectivity, which affects the speed of information propagation, and the heterogeneity of networking connections, which impacts the execution time of each communication round.
To demonstrate the practical impact of topology and heterogeneous connections on communication time, we set up experiment with two scenarios: four nodes, each equipped with four Nvidia GPUs, are interconnected by either 25Gbps Ethernet or 100Gbps Infiniband, Table I shows the communication time of performing averaging operations with different communication topologies. Even though the One-peer Exponential topology enjoys rapid convergence of consensus errors in terms of iterations, it requires more time than the Complete topology in this setup due frequent node communication. The One-peer Ring topology nearly halves the per-iteration communi- cation time compared the Complete topology. However, it exhibits a slower convergence rate of consensus errors than both the Complete and the One-peer Exponential topology in terms of communication rounds, which could potentially influence the theoretical generalization guarantees. However, experiments in (Kong et al. (021) have shown that long the consensus error is below
We provided the pseudocode of the distributed algorithm using All-Reduce in Appendix A.1 and its timeline in Appendix A.3 All-Reduce also considered in our experiments for comparison.
4

---

Published as a conference paper at ICLR 2025
certain value, it does not negatively affect performance. We have observed the same phenomenon in our practical deployments.
Unlike other implementations of decentralized training. such as those described in Assran et al. (2019); y; Lin et al. (202T): Ying al. (2021a) which perform decentralized communication on the node level, we do not synchronize gradients within nodes. This implementation choice can provide significant resilience to variations in computation
Consensus Error (1ogg
2.5
-5.0
Topology peer ring alternating exp
time, which will be discussed in 22.3
-7.5
To adapt to the communication imbalance in the two clus- ters used in our experiments, we designed an Alternating Exponential Ring (AER) topology, see Figure 12 in Ap- X The last column in Tand the comparison in Figure demonstrate that the proposed topology offers advantages both communication time and convergence rate. Additionally, our numerical experiments in Section ddemonstrate that it is able to strike a favourable balance between the connectivity of communication topologies and per-iteration runtime while maintaining comparable test accuracy.
-10.0
Communication Rounds
Figure 2: Comparison of consensus er- rors by communication rounds with four 4-GPU nodes.
It is important note that the effectiveness of a specific topology may vary depending the opera- tional environment. By quantifying the communication costs associated with different connections and the congestion caused by concurrent communications, one can formulate an optimization problem that targets both the convergence rate and average runtime as objectives. We leave this as future work.
2.3 REDUCING THE SENSITIVITY TO VARYING COMPUTATION TIMES
The timeline in Figure D shows single worker and assumes that all other workers run their cor- responding tasks in perfect synchrony. This is certainly not realistic. Even in homogeneous HPC environments, variable computation times appear due to system noise (Hoefler et al.) 2010) and workload imbalance (Ott et al. 2019). Figure shows shows the distribution of actual gradient computation times for two different tasks. For image classífication with ResNet-50 (He et al. 2016), even though all mini-batches of identical size, the variance still exceeds 0.001. For neural machine translation, the variance is much higher since bati-b vary in size even if we batch together sequences with similar lengths (Vaswani et al. 2017 Ott et al. 2019).
In All-Reduce training, workers have to wait for the slowest one, which makes the approach sensitive to workload variability. If the computation times of workers would be i.i.d. and follow normal distribution, the expected value of the maximum of the N computation times is asymptotically @ N) (Giladi et al. 2024). Hence, we can expect decentralized training, as shown in Appendix A.5) it is still possible to hide some (and in some cases all) workload variability. Simulations of the runtimes of All-Reduce and decentralized training. shown in Figure 4 confirm these predictions. The runtime for All-Reduce increases with N, while the runtime for decentralized training remains essentially constant. This suggests that decentralized training could achieve better strong scaling efficiency (fixed global batch size).
be less efficient N increases. For
In other decentralized frameworks like SGP (Assran al. 2019 intra-node gradient synchronization is typically performed before the local update, and the parameter communication occurs at node level. However, this design becomes suboptimal in terms of runtime when the variance in computation time is non-negligible, since gradient synchronization is susceptible to intra-node stragglers. To better leverage the resilience of decentralized training to variations in computation time, we decouple the dependency of the local update the gradient information of its neighbors within the same iteration. To demonstrate the benefits of the new design, we perform transformer experiments under heavy computation workloads (2x4x A 100 and 100K tokens global batch size) and fast network connections (NVLink within the node and 100Gbps Infiniband inter-connecting nodes) to make sure the network is not the bottleneck. The results show that removing the dependency decreases the per-iteration runtime by 10.95% (from 225.09 ms to 200.43 ms). For more comparisons, see $ 5.2
http://ghthhcammmccccchhoersssee gradient push
5

---

Published as a conference paper at ICLR 2025
0.25
Decentralized
0.20
Image Classification Neural Machine Translation
All Al-Reedce
Probability
Runtims (me)
Decent, Data
0.15
AllReduce
0.10
300
0.05
0.00
0.6
0.7
0.9
1.0
1.1
1.2
1.3
1.4
Normalized Time
Figure 3: Distributions of normalized computation times of ResNet-50 training for an image classifi- cation task and of transformer training machine tramslation. The variances of normalized computation time for the image task and the transla- tion task are 0.0017 and 0.0134, respectively.
Figure 4: Comparison of the predicted runtime by runtime model and the actual runtime for transla- tion task based on transformer model. The lines predicted runtime based on the runtime model with scaling to the real runtime in milliscconds. The scat- ters are the measured runtime.
neural
RUNTIME MODEL AND DESIGN INSIGHT
In this section, we propose a runtime model that characterizes how the per-iteration runtime depends on critical system parameters and workload characteristics. The model is simple yet surprisingly accurate when validated against experiments real systems (see Appendix A..5,, and helps to identify when and with what means we can obtain significant reductions the per-iteration runtime.
The model considers neural network training in environment with N workers and assumes that the model parameters are organized into buckets. uses the time that a single worker needs to process the forward pass with the global batch size unit time. In this way, the forward pass of one worker takes 1/N time units while the backward pass is assumed to take 2/N time units. The time to update bucket is 0 time units. To characterize the system and the training workload, we introduce three additional parameters: (1) E (0, oc) is the time taken for global All-Reduce on one bucket; this parameter large in systems with slow communication or many workers, (2) w (0, 1] is the ratio of the time taken by decentralized communication round and a global All-Reduce. A small means a sparse decentralized communication topology, and (3) d² is the variance of truncated normal distribution that models the varying computation time. A large o² means the computation time taken by the forward pass and backward pass may vary significantly. Based on the workload measurements in Figure we restrict the computation time between 0.5x and 1.5x of its average value.
The model detailed Appendix A.5 In brief, simulates randomly varying forward and backward pass calculation times and respects the dependency relationships between communication and update tasks described in Figure T and Figure T3
Simulated Runtime vs. Ny (L
Relative Specdup (5) 2 (o
Relative Specdup (5)
All Reduce
- 1.000
7 1.0 u
Decontraliad
40
7- 4.00%
4.00 1.00//4
20
5.00/M
0.000
0.005
0.010
0.015
0.000
0.005
Figure 5: Simulation by the runtime model. In the simulation, N = 8 workers), b = 4 (4 buckets, from ResNet-50 experiments), 0 0.2 (time taken by updating bucket 0.2 unit time), and the communication topology decentralized training complete topology
Figure 5 shows some representative results generated by the runtime model. The leftmost figure illustrates how the per-iteration runtime of All-Reduce and decentralized training depends on 7 when and ² 0. For decentralized training, the communication time can be completely hidden by the corresponding forward, backward, and bucket update times, as long as the network is fast enough (> (3 N)/N.. After this point, the per-iteration time increases as (by)). In All-Reduce training,

---

Published as a conference paper at ICLR 2025
on the other hand, the last bucket can never be hidden and the per- per-iteration time increases also for small values of y. More importantly, since the All-Reduce time of buckets can only be hidden by the corresponding backward pass, the more rapid increase in runtime starts already at y 2/N.
Figure 5 (middle) shows the relative speedup of decentralized training over All-Reduce as a function w for different values of nN. For w and fixed value yN,, the speedup is just the ratio of the two curves shown in the leftmost figure. Decreasing w reduces the time for a bucket communication in the decentralized setting, which effectively moves to the left along the decentralized runtime graph. This leads to increasing speed-ups until wy = (3 + 0)/N, after which no runtime improvements are obtained by decreasing w further. This explains why no effect of seen when y S (3 + 9), and the speed-ups saturate below = (3 + 0)/"NN The model can also be used to estimate the best possible speedup. Specifically, when x² = 0, we find
1
NO 3I0N 242/b
E (0,2/N),
best possible speedup(1)
7 o0).
3+ON
As N grows large, the second case suggests that the speedup tends towards 1 +4/60. Since y increases with N, the model suggests that the speedups will increase in larger networks.
Figure 5 (right) shows the resilience of the computation time variance of decentralized training. The three combinations o and w represent the scenarios that (a) computation- botiiiooo-ouudddd (y = 1/N, w = 1) (b) communication--oune bound (? 4/N,
1) (c) communication--ount and using decentralized communication topology (? = 4/N, w = 0.6). In the three scenarios, increasing の² brings moderate improvement the speedup of decentralized training.
As for the influence of communication topologies on runtime, they do influence runtime because of the changes in the dependencies of the operations, but the influences caused by adapting decentralized training and reduced communication cost by sparser topologies (see $ A.5.2 in appendix).
In Figure 4 we validate the accuracy of the runtime model by comparing the predictions with the actual runtime as increases. The parameters b 7, N = 32, 0 0.364, o² = 6.95 x 10-3 measured from the profiling results are used for the predictions, and we manipulate the global batch size to mimic different values of y. Given its simplicity, the model is surprisingly accurate.
4 DECENTRALIZED ADAM AND CONVERGENCE
The per-iteration runtime that we have just studied is only part of the story. Equally important is the ability of the decentralized optimization algorithm converge quickly and reliably toward solution with good generalization properties. This section introduces decentralized Adam algorithm (DAdam, described in Algorithm [ that enables overlapping communications and computations and proves its convergence. The notations are defined in Appendix B.I.T.
Algorithm 1 Decentralized Adam on worker
(0) (0) <- 0; x +- (0)
(0)
1: m;
2: for = 1,2 2.......
.,. T do
(t) - V((m!" (t 1), ()))
3:
g.
4:
m
(1) t 3.m, (t-1)
+ + (t-1)
(1 )0
(1)
+ Bzn (20(111 + +11 g 1012
((-1)
5:
U:
(8) t
!1) 11- + EjEN, Wayz) (t-1)
6:
V vf /11 pe)+
7: end for
Algorithm Cean be cast into the form of equation with a particular search direction. For simplicity, we use a static topology with mixing matrix W. i.e., W(() = W for all t. In every iteration, each worker i first draws mini-bate from its local data and computes local stochastic gradient 9i- Then, proceeds similarly to the standard Adam scheme to update the local first- and second-order momentum my and v; respectively. Subsequently, worker implements the mixing of neighbor
7

---

Published as a conference paper at ICLR 2025
models and uses the result to update the local model where is step-size and B2 are parameters. Note that m,, V and the resulting search direction do not depend on models of neighboring workers. Therefore, the algorithm supports overlapping communication and computations as discussed in Section 2.T The comparison of Algorithm with other decentralized Adam-based methods is in Appendix [.2
Next, we establish convergence of Algorithm | based on the following Assumptions.
Assumption 1. For each worker i, the gradient F.; I--Lipschitz continuous:
(z) (v)lll) L|| vllz. Vx, Y ER.
Assumption 2. There exists a constant R 2 V such that [V S R- R- almost surely.
Assumption 3. The mixing mutrix W = [ur,p] symmetric, non-negative, and doubly stochastic. The eigenvalues of W in order satisfies = λ X2 ...
2...2.N AN -1.
Assumptions and 2 are used in Défossez et al. 2020)) to provide a simple analysis for centralized Adam. Assumption is standard in decentralized learning and holds when the underlying communi- cation topology is connected. Under Assumption we denote À = max [221, [Awl] E [0, 1). Fix
the number of iterations as T. We define 7r as random index from (O...., T 1} with distribution
v P[T=
3T-
and denote F(Ø) = N Σ (z where F. 1 Σi=1 Ft. Then, we have the following theorem.
N
Theorem 4.1. Under Assumptions
if0 < 82 < 1, for Algorithm we have
E V
[1]]
4R S
(( + E
In 1
R
T
In(33)
(5)
af
c(1
(2)
T
where a(() = N Ee ai is defined in equation
4
T = T
8:
F. is the optimal
I-31
value of equation 1 and E VT- 10:75118111111 11/21181//2177 Sa2(2+2))RLL (--X))1-B811--BBBB1
24D -
2aDLR(1-8.) (1B21(1-1171a)
1,2
(-3,//2)(1--a99)
We refer the readers to Appendix B.4 for the proof. Compared to Défossez al, 2020) Theorem 4) for the single-machine Adam, the error bound in Theorem 4.1 only has one additional term the last term in E which proportional to a and can be ignored for sufficiently small values of a.
NUMERICAL EXPERIMENTS
In this section, we compare decentralized training with All-Reduce training and assess the generaliza- tion performance and practical training- timinng-time speedups in cluster environments. We consider three large-scale tasks: neural machine translation, image classification, and GPT-2 pretraining. For neural machine translation, following Vaswani et al. (2017), we trained the transformer (~~5M parameters for base variant and ~2113M parameters for big variant) on the English-German Gerssa and English-Freoe WMT14 dataset (Bojar et al. 2014). In each run, evaluate the trained model by BLEU (Papineni et al. 2002) and METHOR (Banerjee & Lavie 2005) on the test set. In the image classification task, trained ResNet-50 (He al.] 2016) ImageNet-1 K (Deng et al.) 2009). In each run, evaluate the trained model by Top-1 and Top-5 accuracies on the validation set. For the GPT-2 pretraining task, we trained GPT-2 (small) (Radford et al. 2019) Open WebText Vebkeex(Gokkssan Gokaslan al. (019). and we report the training and validation losses of the trained models. The baseline, All-Reduce training, uses Adam based optimizers for all experiments. Detailed descriptions of all tasks and experimental setups can be found in Appendix A.6
5.1 GENERALIZATION PERFORMANCE
To make decentralized training useful, it should not compromise the generalization performance of the trained model. As shown in Tables and 3 decentralized training can obtain generalization per- formance that is comparable with All-Reduce training. In all experiments, Accum outperforms the others, while DAdam and All-Reduce perform similarly.
8

---

Published as a conference paper at ICLR 2025
BLEU 1/ METEOR
Method
Topology
En-De Transformer (base)
En-Fr / Transformer (big)
AllReduce
Complete
27.79 = 0.16 0.54//..55555// + 0.0018
45.70 0.30/ 0.667 = 0.0017
AER
27.58 = 0.06 0.5440 + 0.0013
45.60 = 0.12 0.6664 = 0.0014
DAdam
Complete
27.60 + 0.18 0..540.545555 0.0021
45.65 + 2.2.0...22060600 0.6670 0.0017
AER
27.71 = 0.09 0.09/0.5441 0.0027
46.05 = 0.037 0.0029
AccumAdam
Complete
27.80 = 0.34 0.5474 + 0.0027
46.18 = 0.20 0.6704 0.0017
Table 2: Neural Machine Translation on WMT14 (Bojar et al., 2014): The results show the generalization performance of training transformer (base) on (big) on English-to-Frendd The last checkpoint of each method is evaluated by BLEU score (Papineni) (tt al., 2002) and METEOR (Banerjee & Lavie 2005) the corresponding newtest2014 testset. The number of workers
and transformer
and the number of workers per node is 4 for all experiments. The error bands with #2a based on 3 runs are reported.
Top-1 Acc. t/ Top-5 Acc. 1 Train Loss Complete
Methods
Alternatin Exp Ring
AllReduce
76.01 / 93.05 / 1.864
DAdam
75.31 92.54 / 1.998
75.27 92.60 1.993
AccumAdam
76.54 93.18 1.836 76.34/ / 93.15 1.835
Table 3: Image Classification on ImageNet-1K (Deng et al., 2009): The results are show the generalization of training ResNet-50 ImageNet-IK K for image classification. Top-1 and Top-5 accuracies and averaged training loss in the last epoch are reported, and the accuracies are evaluated on the ImageNet validation set. The number of workers is 32 and the number of workers per node is for all experiments.
5.2 PER-ITERATION RUNTIME AND SCALABILITY
We will now demonstrate the practicality of decentralized training by showing its superior runtime compared All-Reduce training various scenarios, including communication-bouut cases. For fair comparison, report strong scaling performance in Figure 6 Other works, such as Assran et al. (019)); Wang et al. (2019). present weak scaling performance which keeps the worker batch size fixed. However, this results in fewer iterations and larger effective global batch size for the same number of epochs, which alters the training dynamics.
straggler-
bound, and computation-bouet
As shown in Figure 6 the results align with the proposed runtime model: (1) sparse communica- tion topology enables more significant speedups in communication-liiite. scenarios compared to computation-limited scenarios; (2) decentralized training has better resilience computation time variance; (3) decentralized training is more efficient than All-Reduce training in all tested scenarios. Additionally, we re-implement 1-OSGP (Assran et al. 2019) (the best asynchronous variant of SGP) with the one-peer exponential graph for comparison. Due to the extra intra-node broadcast required to synchronize the parameters, 1-OSGP may experience longer per-iteration runtimes when the mntr connection is not significantly faster than the inter-node connection (nodes with 8 T4 GPU with 100Gbps InfiniBand connection, for example). Consistent with the analysis in $ $2.3) our implementations outperform 1-OSGP in all setups, especially in computation-bouut bound scenarios.
To demonstrate the relationship between generalization performance and runtime, we present training curves for an En-De translation task in Figure 7 showing 60% speedup at the same validation loss.
We also conducted experiments on the pre-training of GPT-2 (small) (Radford et al. 2019 Compared to All-Reduce training, decentralized training with AccumAdam and the Alternating Exponential Ring graph achieves better training loss (2.8419 2.8468), validation loss (2.8561 1 2.8626), and runtime (30.41 hours vs. 32.461 hours) after 600k training steps on Open WebText (Gokaslan al. 2019) using 4x4x A40 GPUs. Even though the workloads balanced among the workers,

---

Published as a conference paper at ICLR 2025
still achieves relative speedup of 6.31%. More speedup be expected by replacing the sequence packing with other batching strategies that may introduce imbalance workloads.
Due to space limitations, defer the presentation of other speedup results A.8 in the appendix.
Alduue
SGP (Exp)
Dacent (Exp)
Decent (AER)
8x Per Node 100Gbps Infiniband
4x,1100 Per Node 100Gbps Infiniband
4x Per Node 25Gbas Ethernet
Iteration Time (ms)
200
Num GPUs
Number of GPUs
Num GPUs
Figure 6: The results show the strong scaling performance (fixed global batch size, 25K tokens) of All-Reduce, SGP (Assran et al. 2019) and our implementations by reporting the per- per-iteration runtime with various numbers of workers and network settings. Here, Exp and AER stand for one-peer exponential graph and alternating exponential ring, respectively.
28
Loss (Cross Entropy)
label
0.54
3.6
AllReduce
26
3.5
AccumAdam complete
BLEU Score
0.52
3.4
24
0.50
3.3
label
label
Validateo
3.2
AccumAdam AER
AccumAdam AER
3.1
AccumAdam- complete
Accum/dam complete
Speedup
3.0
10
15 20
25
0
10
15
20
25
10
15
20
25
Time (1000
Time (1000 s)
Time (1000 s)
Figure 7: Training a transformer (base) model on the English-to-Genne Germann translation task on four 4xA40 nodes with 25Gbps Ethernet connections. The error bands of #20 based three The red dashed line on the left the baseline (27.3 BLEU score) from Vaswani et al. (2017).
CONCLUSIONS
This paper has aimed to address practical and sometimes overlooked aspects of decentralized training, make superior to All-Reduce in terms of total training time and generalization performance in practical settings. We identified key sources of speed-up in decentralized training of deep neural networks and developed a simple model to optimize per-iteration runtimes. We introduced a decen- tralized Adam algorithm that supports overlapping communication and computation and proposed an accumulation mechanism to mitigate the high variance that occurs in small local batch sizes. Numer- deployments of methods in practical environments validated their effectiveness. achieving faster training speed than standard All-Reduce training without compromising generalization.
7
ACKNOWLEDGMENT
The computation and storage resources were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725. This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Mike Rabbat. Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning, pp. 344-353. PMLR, 2019.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Ondřej Bojar, Christian Buck. Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. Findings of the 2014 workshop statistical machine translation. In Proceedings of the ninth workshop statistical machine translation, pp. 12-58, 2014.
Federico S Cattivelli and Ali H Sayed. Diffusion LMS strategies for distributed estimation. IEEE transactions on signal processing, 58(3): 1035-108,, 2009.
Xiangyi Chen, Belhal Karimi, Weijie Zhao, and Ping Li. On the convergence decentralized adaptive gradient methods. In Asian Conference on Machine Learning, pp. 217-232. PMLR, 2023.
Alexandre Défossez, Léon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of adam and adagrad. arXiv preprint (rXiiv20033023300 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-2482255 255. Ieee, 2009.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04888 2018.
Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, and Stefano Soatto. Fewer truncations improve language modeling. arXiv preprint 10Xvv2244.108300 2024.
Hongchang Gao and Heng Huang. Periodic stochastic gradient descent with momentum for decen- tralized training. arXiv preprint arXiv:2008. 10435, 2020.
Niv Giladi, Shahar Gottlieb, Asaf Karnieli, Ron Banner, Elad Hoffer, Kfir Y Levy, Daniel Soudry, et al. Dropcompute: simple and more robust distributed synchronous training via compute variance reduction. Advances in Neural Information Processing Systems, 36, 2024.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http: 2019.
//skylion007, .11.00...ititii qithub. io
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Torsten Hoefler, Timo Schneider, and Andrew I Lumsdaine. Characterizing the influence of system noise on scage applications by simulation. In 10: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-11. IEEE, 2010.
Bjorn Johansson, Maben Rabi, and Mikael Johansson. simple peer-to-peer algorithm for distributed optimization in sensor networks. In 2007 46th IEEE Conference on Decision and Control, pp. 4705-4710. IEEE, 2007.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08360 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-bate batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 1609. 04836, 2016.
11

---

Published as a conference paper at ICLR 2025
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference Learning Representations. 2017. URL /tpptttp////oeeeeeeeer
net/ net/forunm d Ygg
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates, In International conference on machine learning. pp. 5381-5393 PMLR, 2020.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 2016.
Lingjing Kong, Tao Lin, Anastasia Koloskova, Martin Jaggi, and Sebastian Stich. Consensus control for decentralized deep learning, In International Conference on Machine Learning, pp. 5686-5696. PMLR, 2021.
Guillaume Leclere, Andrew Iyas, Logan Engstrom. Sung Min Park, Hadi Salman, and Aleksander Madry. Ffev: Accelerating training by removing data bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1201 -20120,, 2023.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, al. Pytorch distributed: experiences on accelerating data parallel training. Proceedings of the VLDB Endowment, 13(12):3005-3018, 2020.
Shigang Li and Torsten Hoefler. Chimera: efficiently training large-scale neural networks with bidirectional pipelines. In Proceedings of the International Conference for High Performance Computing, Networking. Storage and Analysis, pp. 1-14, 2021.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017.
Tao Lin, Sai Praneeth Karimireddy, Sebastian Stich, and Martin Jaggi. Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data. International Conference on Machine Learning, pp. 6654 6665. PMLR, 2021.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint
2017.
Dmitry Metelev, Alexander Rogozin, Dmitry Kovalev, and Alexander Gasnikov. Is consensus acceleration possible in decentralized optimization over slowly time-varying networks? In International Conference Machine Learning, pp. 24532-24554. PMLR, 2023.
Parvin Nazari, Davoud Ataee Tarzanagh, and George Michailidis. Dadam: A consensus-based distributed adaptive gradient method for online optimization. IEEE Transactions on Signal Processing, 70:6065 6079, 2022.
Angelia Nedic, Alex Olshevsky, and Wei Shi. Achieving geometric convergence for distributed optimization over time-varying graphs. SIAM Journal on Optimization, 27(4) 2633, 2017.
Angelia Nedić, Alex Olshevsky, and Michael G Rabbat. Network topology and communication- computation tradeoffs in decentralized optimization. Proceedings of the IEEE, 106(5):953-978 2018.
Giovanni Neglia, Chuan Xu, Don Towsley, and Gianmarco Calbi. Decentralized gradient methods: does topology matter? In International Conference on Artificial Intelligence and Statistics, pp. 2348-2358. PMLR, 2020.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli, fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of NAACI-HLT 2019: Demonstrations, 2019.
12

---

Published as a conference paper at ICLR 2025
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th ammual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020
Fakhteh Saadatniaki, Ran Xin, and Usman A Khan. Decentralized optimization over time-varying directed graphs with row and 65(11))4769-477 47800 2020.
matrices. IEEE Transactions on Automatic Control,
Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:
2018.
Guodong Shi, Bo Li, Mikael Johansson, and Karl Henrik Johansson. Finite-time convergent gossiping. IEEE/ACM Transactions on Networking, 24(5):7822-2794, 2015.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im: Training multi-biliin parameter language models using model parallelism. arXiv preprint arXiv:1909.0805: 2019.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron train megatron-turine turing nlg 530b, large-scale generative language model. arXiv preprint ar Xrv::22001. 19900 2022.
Yuki Takezawa, Ryoma Sato, Han Bao, Kenta Niwa, and Makoto Yamada. Beyond exponential graph: Communication-efice topologies for decentralized learning via finite-time convergence. Advances in Neural Information Processing Systems, 36, 2024.
Rashish Tandon, Qi Lei, Alexandros G Dimakis, and Nikos Karampatziakis. Gradient coding: Avoiding stragglers in distributed learning. In International Conference on Machine Learning, pp. 3368-3376. PMLR, 2017.
Hanlin Tang, Shaoduo Gan, Ce Zhang, Tong Zhang, and Liu. Communication compression for decentralized training. Advances in Neural Information Processing Systems, 31, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32, 2019.
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving communication--ffian distributed SGD with slow momentum. arXiv preprint 13X1v:1910.00643. 2019.
Lin Xiao and Stephen Boyd. Fast linear iterations for distributed averaging. Systems & Control Letters, 53(1):65-7,, 2004.
Qifan Xu and Yang You. An efficient 2d method for training super-large deep learning models. In 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 222-232. IEEE, 2023.
13

---

Published as a conference paper at ICLR 2025
Feng Yan, Shreyas Sundaram, SVN Vishwanathan, and Yuan Qi. Distributed autonomous online learning: Regrets and intrinsic privacy preserving properties. IEEE Transactions on Knowledge and Data Engineering. 25(1 1):2443-24493 2012.
Bicheng Ying, Kun Yuan, Yiming Chen, Hanbin Hu, Pan Pan, and Wotao Yin. Exponential graph is provably efficient for decentralized deep training, Advances in Neural Information Processing Systems, 34:13975-13987 2021a.
Bicheng Ying. Kun Yuan, Yiming Chen, Hanbin Hu, Yingya Zhang, Pan Pan, and Wotao Yin. Communicate then adapt: An effective decentralized adaptive method for deep training. 2021b.
Binhang Yuan, Yongjun He, Jared Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy S Liang, Christopher Re, and Ce Zhang. Decentralized training of foundation models in heterogeneous environments. Advances in Neural Information Processing Systems, 35:25764-25477, 2022.
Kun Yuan, Qing Ling, and Wotao Yin. On the convergence of decentralized gradient descent. SIAM Journal on Optimization, 26(3):1835-1854, 2016.
Kun Yuan, Yiming Chen, Xinmeng Huang, Yingya Zhang, Pan Pan, Yinghui Xu, and Wotao Yin. Decentlam: Decentralized momentum sgd for large-batch deep training. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3029-3039, 2021.
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. Advances in neural information processing systems, 31, 2018.
Jinshan Zeng and Wotao Yin. On nonconvex decentralized gradient descent. IEEE Transactions on signal processing. 66(1 2844, 2018.
Zhen Zhang, Chaokun Chang, Haibin Lin, Yida Wang, Raman Arora, and Xin Jin. Is network the bottleneck of distributed training? In Proceedings of the Workshop Network Meets Al & ML, pp. 8-13, 2020.
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX / SUPPLEMENTARY MATERIAL
A.1 ALL-REDUCE ADAM IN THE COMPARISON OF EXPERIMENTS
The All-Reduce Adam is shown as Algorithm 2
contrast to the decentralized algorithms in Sec- it averages gradients instead of models and uses All-Reduce instead of gossip communication.
tion 4
Algorithm All-Reduce Adam on worker
1: m
0 (0)
t 0; a
(0)
2: for
t 1,2, T do
< Ve 1 (1-1), (())
3:
9i
4:
g0)) + # + 9i
(t)
+ (All-Reduce)
5:
m
(1) BLm,
(t-1)
(1)
(1) 3zn (t-1)
+ (1- B2) [g(
6:
(1+1)
m (11-))
7:
-
+ 22
Vol (11-8/)-
8: end for
A.2 HETEROGENEOUS COMMUNICATION IN HPC CLUSTERS
Although there no single HPC architecture, typical HPC cluster with many GPU nodes includes high-performance CPUs and GPUs, high-speed interconnects, scalable storage solutions, and manage- ment and software stack to handle various computational tasks. Figure shows the typical topology that have encountered in experiments different clusters. Here, the GPUs within node communicate with high bandwidth and low latency, while communication through the switch much more limited. While Figure shows one example of cluster topology, there are many other possible topologies such as (1) fast NVLinks inter-connecting intra-node GPUs (2) star, ring, tree, mesh, or hybrid connection topologies that link nodes together as cluster. Different topologies impose different limitations on the communication between GPUs in the same and in different nodes.
Switch
Node
Node
NIC
NIC
PCIe Switches
PCIe Switches
GPUI
GPU3
GPUI
GPU3
GPU2
GPU4
GPU2
GPU4
Figure 8: In this HPC environment, each of the two nodes comprises four workers interconnected via PCIe (Peripheral Component Interconnect Express) switches and Network Interface Cards (NICs). The nodes linked to each other through switch. We use line thickness to indicate the bandwidth of these connections: wider lines represent higher bandwidth. facilitating faster communication within nodes, while thinner lines depict the narrower bandwidth of the switch, resulting in more time-consuming node communication.
A.3 TIMELINES AND DEPENDENCY GRAPHS OF DIFFERENT DATA PARALLEL SCHEMES
To better illustrate the capability of our scheme in overlapping communication and computation, consider two comparison schemes, including the vanilla scheme and a well-optimized scheme that is also used in modern distributed training frameworks like PyTorch's DataDistributedPaarlle (Paszke)
15

---

Published as a conference paper at ICLR 2025
Forward
Backward
Comm.
Update
Dependency
Comp.
Vannila
Comp.
Bg
PyToc DDP Horovod
Comm.
U/
0/
U/
Comp.
B2
B1
B1
81
B: B1 F
Decestralized Scheme
Comm.
Ca
C1
Figure 9: Timelines and dependency relations of decentralized training. F: forward pass, B: backward pass, C:: catmmmmniiaaiooagggggatiiio of gradients or model parameters, U: update model parameters. Arrow from X means that Y can start only after X finishes.
ct al. 2017) and Horovod (Sergeev Balso 2018). We visualize the communiaaionnkcomppaaiin of all three schemes in Figure In the vanilla scheme, communication operation only begins to aggregate the gradients among all the workers after completing both forward pass and backward pass. The update of the model only happens after the aggregation ends, and the computation does not overlap with the communication. Similar to our scheme, in the Py Torch DDP & Horovod scheme, the model parameters Figure 19. The communication of the bucket can start immediately after its corresponding backward pass finishes. In this scheme, some time of the communication interleaves with the computation, which greatly improves the utilization. However, the communication still can not be interleaved with the computation the forward pass and the last bucket of the backward pass. contrast to the two comparison schemes, our decentralized scheme hides all communications by computations.
divided into buckets (the parameters
divided into 3 in the example in
A.4 DECENTRALIZED COMMUNICATION TOPOLOGIES
We display the One-peer Ring (Figure 10 the One-peer Exponential (Figure I1]. and the Alternating Exponential Ring (Figure 12 topologies this subsection.
Figure 10: One-peer Ring for 16-worker example where workers are distributed in four nodes. Dashed lines are the boundaries of the compute nodes.
16

---

Published as a conference paper at ICLR 2025
Figure 11: One-peer Exponential graph for 16-worker example where workers distributed in four nodes. Dashed lines are the boundarics of the compute nodes.
Alternating Exponential Ring topology (ours) for a 16-worker example where workers are distributed in four nodes. Dashed lines are the boundaries of nodes. The loops All-Reduce workers in the loop.
Figure 12:
A.5 RUNTIME MODEL
Our runtime model is constructed under reasonable settings and experimental results. It assumes that there are N workers and that the neural network model parameters are divided into buckets so that the average computation and communication time of each bucket are all the same. The global batch size is fixed and the time needed to process the forward pass on a single worker takes unit time. According to Kaplan et al. (2020) and our profiling results, the time taken by the backward pass is roughly twice as the forward pass. With the workload evenly distributed on N workers, the time taken by the forward pass is then 1/N unit time and the time taken by the backward pass 2/N unit time. We also assume the time that it takes to update one bucket is 0 time units where 0 is the same for All-Reduce training and decentralized training and independent of the number of workers. Three additional factors are introduced to characterize the system setup and workload:
1. oz) is the time of an All-Reduce communication of one bucket. Note that y is the actual communication time, while the actual time of the All-Reduce operation could be longer due to synchronization across workers. The parameter is influenced by several factors:
(a) Network technology. A poor network connection leads to slower All-Reduce and larger y The number of workers. More workers lead a slower All-Reduce and higher value of 7 2. E (0, is the ratio of the time of decentralized communication round and an All-Reduce operation. Note that could exceed one, but in that case it would always be better to use All-Reduce (as discussed in Section 2.2 Using a more sparse communication topology could potentially decrease w.
3. o² E [0, oo) is the variance in the truncated normal distributions used to model the variance in
the computation time. For worker iteration random number (2) is sampled from the truncated normal distribution with mean 1, variance o², and support [0.5, 1.5] (chosen to match our
experimental results in Figure 3 The random number is used multiplier for the computation time of worker i at iteration f, i.e. the time for both forward and backward passes are multiplied by the same p(t1))
With these quantities, we further take the dependency relationship into consideration. We denote the completion time of task X at worker i in iteration by T (i.t)) Then, based on the dependency relation
17

---

Published as a conference paper at ICLR 2025
Forward
Backward
B
Comm
Update
Dependency
Comp.
Fg
Ba
B1
Bg
B2
B1
F
Worker
Comm.
C2
C1
Worker 2
Comp.
F
Fs
13
B1
F
Bs
12
B1
F
U1
U3
Ug
Figure 13: Timelines of two workers in decentralized training. In the example, worker I has uniform computation time across two iterations while worker has faster iteration and then slower one. first iteration, C' of worker 2 is prolonged because it needs to synchronize with worker to start the communication.
of All-Reduce training, we have
T(e
-T(i
(11,1-1)
p((t) b
N + +T
(1,0)
pli,))
N
-(.00)
if =
pla.))
2
+ By-1
i E (l,...,
N
(6)
(5,0)
+ [ (T(:0)
k = if E ( b-1
C2
+ maxjE[N] (TH T6.00
(6.0
T + 0b
where all out-of-bound finish times like T (2.0) are defined to be zero
To state similar dependency relations for decentralized training, we denote the set of neighbors of worker i in iteration as A( Then, we have
(())
(...1) = - (1.1-1) + +p(in (E.) 1
N
(6.11)
{ p(i.t)
NV
+ ((1))
if
Bz 11(.1)
U
(i.t)
if (1,...,b- 1}
(7)
(r(.1) Us
max (T
(1.-11)
+0
Ci.
(11))) 1(1a-1)
(1.0)
wh + max
JEN Us
C,
if = if E bl,...bb-1)
C.
wy max JENiO (T(5,) Tonn
where all out-of-bound finish times like T6, and T( are defined as zeros.
When o² O. we need to simulate the model and perform Monte Carlo simulations to draw conclusions. When the workloads are deterministic, on the other hand, the model can be used to derive closed-form expressions guide the design; see Section 3 in the main document.
A.5.1 SEQUENTIAL TIMELINE ON THE MAIN THREAD
Algorithm 3 describes the main thread of one worker and relates the different steps to the sequential timeline in Section A.5 of the appendix. In Algorithm below, F(.()) stands for the forward pass
of worker at iteration I, (f.1) stands for the backward pass bucket k of worker at iteration I,
18

---

Published as a conference paper at ICLR 2025
Ul(i)) stands for the optimizer update bucket k of worker iteration and C stands for the decentralized communication of bucket & of worker at iteration L.
Algorithm 3 Decentralized Training on worker
1: for t 1, ' do
2: Perform forward pass: F(i.t)
b. Time taken: pli.t b
N
3:
for < b, 1, do
4:
Perform backward for k-th bucket: (((.))
Time taken: pl
NV
5:
Wait for the comm. op. (C (C(1,t-1) in the last iteration.
Time taken: U.
T
Update the parameters in k-th bucket: U((
6:
Time taken:
Launch comm. op. for k-th bucket asynchronously: (((.t)
7:
Time taken: 0
(non-blocking)
8: end for
9:
end for
A.5.2 INFLUENCE OF COMMUNICATION TOPOLOGY RUNTIME
To demonstrate the influence of the communication topology on the runtime, we conduct simulations under various topologies but keep 1, which can exclude the impact of communication topology on reduced communication cost. The simulation result is in Figure 14 Compared with the speedup by adapting decentralized training and reduced the simulation result shows that the changes in the event dependencies caused by the communication topology do not have a significant influence. The difference becomes more insignificant when the communication cost is the bottleneck (Na > 3.3).
Sinnlated Runcime vs Ny (N 32, 0.0034)
AllRaduue
Decentralzed Compltte
3.25
Decenraaioos AER
Deontralizol Ring
Samaletd Punticae
277
2.25
2.0
1.75
Figure 14: Simulated runtime under various communication topologies. The parameters are = 4.
0.02, and the number of workers one node is 4.
19

---

Published as a conference paper at ICLR 2025
A.6 TASKS AND DETAILED EXPERIMENT SETUPS
All of our experiment implementation is based on PyTorch (Li et al., 2020). As for the implemen- tation of All-Reduce training baselines, we use DistribuledDattaarulee (DDP) module. In all our experiments, we use full-precision training and activate tensor cores for devices that support the feature.
Neural machine translation. Following (Vaswani et al., 2017). we trained the base transformer (~65M parameters) on the WMT14 English-German dataset (Bojar et al. 2014) using the global batch size which is roughly 25K source and target tokens, and the models were trained for ~ 130,000 steps (or 24 epochs) in total. We trained the big transformer (~213M parameters) on the WMT14 English-French dataset using the same batch size, and the models were trained for ~300,000 steps (or 5 epochs) in total. The All-Reduce training baselines use Adam optimizer (Kingma & Ba 2014). In each run, the last checkpoint is used to be evaluated by BLEU score (Papineni al. 2002 on the corresponding test sets.
Image classification. As for the image classification task, we trained ResNet-50 (He al. 2016) on ImageNet-IK (~ 1.2M images and IK classes) (Deng et al., 2009). Similar to the translation task, All-Reduce training baselines use Adam optimizer. In all experiments. the models were trained for 90 epochs with 1024 as the global batch size. In each run, the last checkpoint is used to be evaluated by top-1 and t accuracies.
Note that, for all experiments, we fix the global batch sizes of tasks. Unlike the experiments in (Lian et al., 2017) and (Assran et al. 2019) which keep the local batch sizes fixed, we argue that using tixed global batch sizes yields more fair comparison with All-Reduce training because (1) increasing the global batch size as scaling up the number of workers may impose negative effects on the generalization performance of All-Reduce training (Keskar et al. 2017), (2) increasing the
global batch size but keeping the total number of epochs fixed leads to speedup in runtime because of the fewer number of updates, and (3) it's important to have same number of iterations to achieve comparable performance under fixed epoch budget.
A. NEURAL MACHINE TRANSLATION
As for the model architecture, we use the transformer implementation provided by PyTorch. We adopt the scripts open-sourced by Fairsed to prepare and clean the raw data.
(bans))rmer(basee) on WMT14 En-De Following the experiments in (Vaswani et al. 2017). the sentence pairs are encoded using the byte-pair encoder tokenizer, which has a shared vocabulary with size 37056. The global batch size is fixed to around 25k tokens for source tokens and target tokens, respectively. The sentence pairs with similar lengths are batched together in the training. All experiments use 0.1 as the dropout rate and 0.1 as the label smoothing. The learning rate schedule is the inversed square root with epoch of linear warmup. In the experiments of All-Reduce baselines, the learning rate is 0.0007, and the betas for the Adam optimizer are (0.9, 0.98), which are taken from Vaswani et al. 2017). In the experiments of decentralized Adam, the betas for the Adam optimizer set to (0.974, 0.999) where the choice of 31 is inspired by AccumAdam when setting s 4 (defined in Algorithm 4) and the choice of 33 is from Devlin et al. (2018). The choice of betas empirically shows much better performance than (0.9, 0.98), so we use the setting for all experiments of decentralized Adam. The base learning rates are 0.0020, 0.0028, and 0.0038 for 8, 16, and 32 workers, respectively. In the experiments of AccumAdam, the betas for the AccumAdam optimizer are (0.9, 0.999), and the number of accumulation iterations s is N/4. The base learning rate is 0,0008 for 8 workers and 0.0012 for 16 and 32 workers.
Transformer(big) on WMT14 En-Fr Following the experiments in Vaswani et al. (017). the sentence pairs are encoded using the word-piece encoder tokenizer, which has shared vocabulary with size 32000. The global batch size is fixed to around 25k tokens for source tokens and target tokens, respectively. The sentence pairs with similar lengths are batched together in the training. All experiments use 0.1 as the dropout rate and 0.1 as the label smoothing. The learning rate schedule is the inversed square root with I epoch of linear warmup. In the experiments of All-Reduce baselines,
20

---

module in Published as a conference paper at ICLR 2025
the learning rate is 0.0007, and the betas for the Adam optimizer are (0.9, 0.98) Vaswani al. 2017). In the experiments of decentralized Adam, the betas for the Adam optimizer are set to (0.974, 0. 0.999). The base learning rate is 0.0012 for 8 workers. In the experiments of AccumAdam, the betas for the AdaumAdam optimizer are (0.9, 0.999), and the number of accumulation iterations s is N/4. The base learning rate is 0.0008 for 8 workers.
A.6.2 IMAGE CLASSIFICATION
ResNet50 on ImageNet For the experiment setup of the image classification task, we follow the training recipe used to train the weights by torchvision To avoid the bottleneck caused by the data loading and preprocessing on CPU, we use DAL for the pipeline of preprocessing. In the preprocessing and the data augmentation, the images are: (1) randomly cropped (aspect ratio is from 3/4 to 4/3, random area is from 0.08 to 1.0) (2) resized to horizontally with 0.5 probability (4) normalized by channel means and variances. In all experiments, the global batch size is fixed to 1024, the learning rate schedule is cosine annealing with 5 epochs of linear warmup, the weight decay is x 10 on non label smoothing is 0.1. In the experiments of All-Reduce baselines, the optimizer is Adam, the base learning rate is 0.001, and the betas are (0.9. 0.999). In the experiments of decentralized Adam, the base learning rate is 0.0024, 0.0048, and 0.0070 for 8, 16, and 32 workers, respectively. The betas (0.974, 0.999). In the experiments of AccumAdam, the base learning rate is 0.002 for all numbers of workers, the betas are (0.9, 0.999), and s set to N/4.
(3) randomly flipped
parameters, and the
A.6.3 GPT-2 PRETRAINING
GPT-2 (small) on Open WebText For the experiment setup of the LLM pretraining, we follow the training recipe from nanoGP1 The batching strategy used the repository is sequence packing, and the global batch size is around 0.5 million tokens. The hyperparameters of the baseline are the same as in the repository. The baseline uses AdamW optimizer. For the AccumAdam experiments with 16 A100 GPUs, we only change 32 and to 0.999 and 8, respectively, and keep the other hyperparameters the same as the baseline.
A.7 PYTORCH EXTENSION FOR DECENTRALIZED TRAINING
The implementation of the PyTorch extension facilitates decentralized training (with the same scheme as in Eq. 2 The design is inspired by the aatrrratttedaaaaaaaaalelleeeeeee
PyTorch (Li et al. 2020).
The main idea of the implementation is as follows. The implementation is a wrapper of the Py- Torch models (torch. nn. Module). By using the hook mechanism in PyTorch, register the backward hook for all weights before the first forward and backward passes. Then, in the first backward pass, the wrapper collects the computation order of weights, groups the weights into buckets by their order, removes the hooks, and registers new hooks on the last weight in the buckets using
(available from PyTorch 2.1). The post accumulate grad hook indicates the event that the corresponding gradient of one weight computed and accumulated, which means the weight will no longer used this iteration and it's ready for update. In the hook, the wrapper checks if the decentralized communication for the bucket in the last iteration finished, updates the weights in the bucket, and launches the decentralized communication for the next iteration. Another module is implemented for scheduling the neighbors of workers in each iteration, and it makes the extension flexible for various topologies.
Subject to the complexity of implementation and limited time, the implemented wrapper only supports one GPU per worker, and it does not support automatic mixed precision and sparse gradients like in DDP, but the improvements are planned for wider applications in the future.
The implementation is based on the Python API provided by PyTorch. We believe that further practical speedups could be achieved by modifying the low-level implementation of the communication to
21

---

Published as a conference paper at ICLR 2025
optimize it for gossip communication. However, for a fair comparison with DDP in PyTorch, implementation at the appropriate level of optimization.
The code bases for the PyTorch extension and the experiments open-souree at https:/github.com// WanggZeee//Deeeent Training-Exp, respectively.
and https//gituubbboommmaazzzeeeeee
A.7.1 COMPARISON WITH IMPLEMENTATION IN STOCHASTIC GRADIENT PUSH
In the implementation of stochastic gradient push (SGP) (Assran et al. 2019). utilize the fast connections (NVLinks between GPUs in one node), it groups the GPUs in one node, In each iteration, the workers one node will aggregate the gradients and update the model shared in the node combining with the received communication messages. Therefore, the communication topology discussed in Assran al. (209)) is on the node level. One of the advantages that the design of topology does not involve the significant heterogeneity caused by intra-node and inter-node connections. However, heterogeneity in connections still exists because the characteristics of inter- node connections may differ depending on the cluster topology and congestion caused by concurrent communications.
Compared with our design, it's different in that workers always exchange the model parameters instead of gradients, and our communication topology is defined on the worker level. As for the topology. could utilize the connections in the same way SGP's implementation as the worker- level topology is superset of node-level topology. Moreover, we argue that our implementation is more advantageous in the resilience the variance because aggregating the gradients will cause the straggler problem within the node.
We demonstrate the difference by making one change in the dependency relationship of the update task. In SGP's implementation, the update on one bucket does not only depend on the corresponding backward and the communication from the last iteration of the work itself but also its local neighbors' backward pass and communication. Therefore, compared with the runtime model of decentralized training (Eq. 77). the finish time of Ug changed to
T (6.)) max Ba (15Mt11) + UR JEC(()
(8)
where C(i) is the workers that are in the same node as worker i.
Note that in this simplified model, ignore the time taken by the head worker to broadcast gossip updates to other workers the node. In most cases, the operation does not incur significant cost to the per-iteration runtime, but only when the intra-node connection comparable with the inter-node connection (the T4 GPU setup in 5.2 for example).
Relative Speedup (%) o²
40
= 1.0/n, 1.0 (our)
1.0/, w 1.0 (SGP)
= 4.0/n, 1.0 (our)
10
7 4.0/n, (SCP)
4.0/n, 0.6 (our)
4.0/n, 0.6 (SCP)
0.000
0. 0.005
0.010
0.015 6²
0.020
0.025
0.030
Figure 15: Comparison with Stochastic Gradient Push
22

---

Published as a conference paper at ICLR 2025
The results of the simulation by the runtime models shown in Figure 15 shows that, with in- creasing variance in computation time, our implementation has better speedup (hence better resilience to computation time variance) than SGP when it's cnpputation-boune In the communication- case, since the communication time dominates the runtime, both implementations have the same speedup.
Since the dependency libraries of the open-source code of SGI have been deprecated due to the fast updates of dependencies, we can reproduce the runtime results our hardware. Instead, we re-implemented 1-OSGP (the best asynchronous version of SGP) in our numerical experiments. The relative comparisons between 1-OSGP and All-Reduce training are as similar as in Asssan et al.] (2019). The comparisons of runtime performance in $ 5.2 justify the advantages of our implementation.
httt://gtthccccmmiccccoorrssssss gradient push
23

---

Published as a conference paper at ICLR 2025
A.8
ADDITIONAL EXPERIMENTS
The section presents the experiment results that not shown in the main text due to space limitation.
A.8.1 GENERALIZATION PERFORMANCE
Table shows the generalization performance of the training transformer (big) WMT14 English- to-French, and it shows that both DAdam and AccumAdam outperform All-Reduce training baseline, and AccumAdam outperforms DAdam in most of the metrics.
Table 4: Neural Machine Translation on WMT14 (Bojar et al., 2014): The results show the generalization performance of training transformer (base) on Anglish-to-German and transformer (big) on English-to-Freee
The last checkpoint of each method is evaluated by BLEU score (Papineni al. 2002) and METEOR (Banerjee & Lavie 2005) the corresponding newtest2014 testset. The number of workers is 8 and the number of workers per node is 4 for all experiments. The error bands with =20 based runs are reported. Some entries are empty because of limited resources.
Method
Topology
En-De Transformrr (base)
En-Fr Transformer (big) 45.69 # 0.30 0.6676 = 0.0017
AllReduce
Complete
27.72 0.37 0.0036
Ring
27.59 = 0.61 0..5555 0.0054
DAdam
AER
27.83 0.11 0.0018
Complete
2..84
45.90 0.6693 + 0.0023
Ring
2760:40.00003447400.0
AccumAdam
AER
0.0000
0.0020
Complete
27.82 0.13 0.0015
0.67 18166718888 0.0009
A.8.2 SPEEDUP
In Tables present the per-iteration runtime of both All-Reduce training and decentralized training (both with Adam optimizer) and the relative speedup of decentralized training over All- Reduce training on the image classification task. Since we conducted the experiments with fixed iteration budget, the relative speedup of per-iteration runtime also reflects the relative speedup of total runtime.
Combined with the results of generalization performance, the experiments demonstrate the practicality of decentralized training.
Table 5: Results of speedup of training ResNet-50 on ImageNet dataset for the image classification task.
Node Setup/ / Inter-Connection
Comm.
Per-iteration Runtime (ms)
Workers
Methods
Speedup
Topology
AllReduce
complete
344.44
8
4x A40 Node/ 25Gbps Ether.
Decentralized
complete
333.07
3.30%
AllReduce
complete
176.72
16
Decentralized
complete
163.68
7.38%
AllReduce
complete
645.59
16
8x Node 100Gbps Infini.
Decentralized
complete
617.75
4.31%
AllReduce
complete
331.51
32
Decentralized
complete
313.56
5.41%
Scalability Experiments with FFCV Additional experiments were performed by integrating our implementation of decentralized training with FFCV Leclerc et al. (2023). The code is adapted from the official repository of FFCV for fast training on ImageNe The following are some important details about the environment used for these tests:
24

---

Published as a conference paper at ICLR 2025
Key software: PyTorch 2.5.1, CUDA 12.1, ffev 1.0.2.
Hardware configuration: 16 CPU cores and 64 GB RAM per GPU. For nodes with 4xA100 each, the nodes are interconnected with 100Gbps InfiniBand. For nodes with 4x each, the nodes are interconnected with 25Gbps Ethernet.
Preprocessing: All images were converted to bento format by FFCV and compressed with 384 pixels as the maximal size. The size of the processed training set is 440GB (same as the statistics from ffev(i) which can be fully loaded into memory for best data loading performance.
Data loader: 12 workers for each GPU, which is the default setting.
Network architecture: ResNet-50.
The input images resized to the size of 160x 160 pixels (default input size at the beginning of the training). We measured the per-iteration and per-epoch runtime for 1, 2, and nodes with the same global batch size of 2048 using two types of nodes, respectively.
Iteration Time (ms) Epoch Time (s) Relative Speedup over AllReduce
Number of Nodes
2
4
AllReduce
123.61 77.26 0.00%
70.23 / 43.89 / 0.00%
41.88 / 26.17 / 0.00%
DT - -Complee
123.00 76.88 0.49%
67.98 / 42.49 / 3.20%
40.57 / 25.36 / 3.13%
-T-AER
N/A
67.07 / 41.92 / 4.49%
38.97 / 24.36 / 6.94%
Table 6: The table shows the runtime performance nodes with 4 100 each. The nodes interconnected with 100Gbps InfiniBand. The statistics are generated by the mean of runtime of epochs. The architecture of the neural network is ResNet-50,, the global batch size is fixed 2048, and the input image size is 160x 160 pixels. The AER topology needs at least two nodes.
Iteration Time (ms) / Epoch Time (s) Relative Speedup over AllReduce
of Nodes
2
AllReduce
274.73/ 171.70 / 0.00%
151.06 94.41 0.00%
121.36 / 75.85 0.00%
DT Complete
273.97 / 171.23 0.27%
149.48 93.42 1.05%
120.19 75.12 0.96%
DT - AER
N/A
145.14 / 90.71 / 3.91%
85.76 53.60 / 29.33%
Table 7: The table shows the runtime performance nodes with 4x A40 each. The nodes interconnected with 25Gbps Ethernet. The statistics are generated by the mean of runtime of 3 epochs. The architecture of the neural network is ResNet-50. the global batch size is fixed 2048, and the input image size is 160x 160 pixels. The AER topology needs at least two nodes.
The results show that inereasing the number of nodes can still enable sigmificant speedups: for AllReduce with A100 nodes, the per-epoch runtime reduces from 76,876s with node to 25.355s with 4 nodes. This demonstrates that the advantages of scaling up beyond node remain even when we incorporate FFCV. Moreover, the decentralized training outperforms AllReduce training experiments even with comple leteleteen topology. With Alternating Exponential Ring (AER), the decentralized training outperforms AllReduce training by clear margin in runtime (up to 29.33% on four 4x A40 nodes).
1I [ntel(() Xeon(R) Gold 6338 CPU @ 2GHz "NVIIIIA Tesla A100 HGX GPU with 40GB vRAM
13 "VVIDIIA Tesla A40 http::///does.fccc.ooeeeeeeemee
25

---

Published as a conference paper at ICLR 2025
A.8.3 TRAINING CURVES
Training Curves for GPT-2 Pre-training This section provides additional details about the experiments of pre-training GPT-2 (small) (Radlord et al. 2019) on Open WebText dataset (Gokaslan
al. 2019).
The curves in Fig. 16 and Fig. 17 demonstrate the cross-entropy loss for next-word prediction where the x-axes are the number of iterations and training time, respectively. The figures illustrate that decentralized training attains comparable generalization performance to AllReduce, but at speedup of ~6.31%
The training details including datasets, source code, number of iterations, and training time were not revealed in the original paper (Radford et al. 2019). The code for the experiments was adapted which, to the best of our knowledge, is the most comprehensive pre-training repository available. The hyperparameters of the AllReduce training baseline use the default settings in the repository (the batch size is ~0.5 million tokens). In the repository, the training was conducted for 400k steps, which we extended to 600k to better validate the performance of decentralized training.
from the open-source implementation of GPT-2 whichh,
The experiments were conducted on 4 nodes with 4x A100 each. Since the workloads are balanced (sequence packing is used the batching strategy) and the network is relatively fast (nodes interconnected by 100Gbps InfiniBand), using sparser topologies will not significantly improve runtime for this case. Therefore, used the complete topology in the experiment of decentralized training.
(Train]
AR Loss (Valld)
(Trein)
Decent Loss (Velid)
LO55
3.2
8 3.1
3.0
2.9
2.8
Iteration
Figure 16: The curves of train and validation losses where the x-axis is the number of iterations. AR: AllReduce training. Decent: Decentralized training with AccumAdam.
AR Loss (Train]
AR Loss (Valld) Dacant (Train) Decant Loss (Valid)
8 3.1
3.0
2.9
2.8
15
20
25
30
Training Time (hour)
Figure 17: The curves of train and validation losses where the x-axis is training time. AR; AllReduce training. Decent: Decentralized training with Accum Adam.
26

---

Published as a conference paper at ICLR 2025
B CONVERGENCE ANALYSIS
B.1 PRELIMINARIES
B.1.1 NOTATION
Our convergence analysis uses the following, relatively standard, notation.
|| |l2 (ll llee ): the lz- (foo-) norm.
N: the number of workers.
D: the number of dimensions of model parameters.
my Ui E P: the local model and the local momentums of worker at iteration é. (t)
(c) (t)
i.d iias iid E R: the d-th entry of the local model and the local momentums worker at iteration
(1))
F, F;: the global objective function and the local objective function worker V((=, (t-1), =.(") IR P: the stochastic gradient evaluated by the
93 (4)
ff(x
local model x; (l-1) and the local sample c! on the loss function &
(t)
(t)
9: the entry-wise square of vector 9;
Vafe: : the d-th entry of ft.
W [waa]] E RNNN the mixing matrix where wij is the ij-th entry.
W E RNNN; a matrix with all entries equal to I/N.
1y E NNN; an x matrix with all entries equal 1.
Ns the set of neighbours of worker (wiy Wys > 0).
[N] : 21,
E,-1 |l: the conditional expectation given all stochastic gradients before the t-th iteration.
By assumption, it holds that
E
B.2 COMPARION WITH RELATED DECENTRALIZED METHODS
Ability of overlap To achieve the overlap of communication and computation, we use combine- then-adapt (CTA) type of decentralized gradient descent (DGD) in Algorithm I In contrast, another type, selap-tteenccoming (ATC) DGD in the form of xr EjeN(tn w/y (a) (1) (() cannot
straightforwardly achieve this overlap, even though it has been shown faster convergence in terms of communication rounds Cattivelli & Sayed 2009).. Specifically, for worker compute
,,(") assume
that a neighbor j E 2 takes a very long time to calculate its direction dfa) and then take a long (t-1) ( (9 to worker i. Worker must then wait for receiving z; time to send 2;
(t-1) adaf
from the straggler j before it can compute x;;" In contrast, with the CTA type algorithm, worker i only needs the previous iterate x (t-1) from the straggler which is already on the way, not the de0) which being calculated with 2 (t-1), This allows for the overlap of computation of de and the communication of 2; (t-1) In other words, the communication task of the straggler worker j. z; (t-1) is hidden within the computation of d(d)
Comparison with state-of-art decentralized Adam-based methods The authors in Nazari et al. (2022) proposed DADAM for online optimization based on the principle of CTA. Following CTA, the authors Chen et al. (2023) introduced unifying decentralized adaptive gradient framework, which involves communication not only of the
decentralized network, with the algorithm
27

---

Published as a conference paper at ICLR 2025
local models but also of the second-order momentums 2/. While this enhances the convergence accuracy of the algorithm, it also increases the communication cost. The authors in Ying et al. (2021b) proposed DAG-Adam, which incorporates consensus constraints penalty in the objective function. Based on the gradient of this penalized augmented objective function, the DAG-Adam algorithm is proposed, improving accuracy in non-iid data scenarios for decentralized Adam. Although DAG- Adam theoretically explains that using different oy across workers can lead to inexact convergence when data are non-iid, our algorithm is applied under the iid assumption, which holds for our HPC environments. Additionally, the paper Lin et al. (2021) introduces QG-ADAM which uses momentum technique to handle non-iid data. Since QG-ADAM uses the ATC approach, each worker must wait to receive all neighboring workers' local messages before updating its local model, thus limiting the ability to effectively overlap computation and communication.
B.2.1 USEFUL INEQUALITIES
3 S
1
Va, b E Rd and 43 > 0, (a,
||al2
+
|611
(9)
2
2
2
Vz, E Rd and n EN
Σ
x 11.
(10)
AA R and x E R", where Amax is largest singular value of A.
||x
X2
|
(11)
max
We first establish the theorem for 31 O. Theorem B.1 (Without Heavy-ball Momentum). Under Assumptions
7
3 for Algorithm we have
Σ
T
v Va
2 1:-
4R aT
F
(
F.
+
E
1
In
1
R²
- II((32)
T
c(1
B2)
where 2(()
1 N
Ei
2
and
E 8DR2
2aDLR
8a (1 22
32 (1-
(1- 32)3/2
Compared with (Défossez et al. 2020 Theorem 2) for the single-machine Adam, Theorem B. shows that our decentralized Adam achieves the same results as Détossez. et al. (2020) (ignoring scaled constants), except that there is an additional third term in F, which is related to the second largest absolute value eigenvalue of the mixing matrix X. Since a close to I indicates poor connectivity of the topology, it increases the third term in E. Notably, this term is proportional to the square of the step size, a. By selecting a smaller a, we can effectively mitigate this term.
The proof can be found in Appendix B.3
B.3 PROOF OF THOREM B.1 DECENTRALIZED ADAM WITHOUT HEAVY-BALL MOMENTUM
B.3.1 ALGORITHM UPDATE UNDER ANALYSIS
We set 31 = 0. which means that there is no heavy-ball momentum. We have the algorithm update under analysis is
9; (0)
= ve(x (t-1) ((- (())
vidd = 320
(t)
(t-1)
gi.d
[
1 - 35 (t)
Didd
(t)
(f-1) Wiyz j.d
(12)
a: V
1 32"
+
JEN.
Taking the average across workers and using the Assumption globally averaged model is
3
we get that the (virtual) update of the
(1)
C2
Ft
(2-1)
(13)
N
(t)
28

---

Published as a conference paper at ICLR 2025
We denote 5/!0 : B20 + E1 (t) Via with its conditional expectation. To simplify the notation, we denote
(
(0!2)²
27
which differs from via by replacing the last term of
N
= VaF (zlt-1)
V Vali (z((-1))
and G = = VaBi(a
(14)
B.3.2 TECHNICAL LEMMAS
Lemma 1 (adaptive update approximately follow a descent direction) we have
Under Assumption 2 for
Algorithm D
N
VaF (z)1-1)))
N VaPi (x. (x(1-1))
VaF VaF (x)
)
N
(Vaiias 27
P
2R
B
Ve+ iia
2=1
(t) pidd
c Uid
(1)
(15)
where
N Vafi xi (1-1)
E1-1 VaF (z((-1)
K
v c +
(t)
vi
Proof. According to the definition of K. we have
N
(2)
K = G
E111
9
(1)
+
s.d.
= 6 E
(t)
Did
(1)
+
N
(1)
N
(16)
Di,d
+ EE-1
(1t1 Σ
Pid
Fi.d
(1)
(8)
(8)
+ iid
i=1
it
(( i,d
(2)
(8)
(1)
Pi.d
Di.d
Gd
+E,-1
(8)
(e)
+
i.d
i=1
c +
i.d
+
Firstly, for the term A, we have
N
(())
(() Gd
N (0)
N
A = G
-
Σ
Gid
d
(17)
i.d
+
in
By applying the Young's inequality and letting
9 for N times on the first term on the right hand of equation 17
G(O)
(((+)
1 b
7
2v
+
(0) Li
(1)
Lda
we have
N
( 2
N (1)
2 2
G
(xt 2
A> >
4 1=1
(1)
Ve+ + ((())
((00 i,d
c +
i.
6 +
(18)
(1)
((())
1. - [00
3
+ 1id
(t))
V e 1
(t)
29

---

Published as a conference paper at ICLR 2025
For the term B in (66., we have
N
(0)
(0)
B = G
Pi.d
Di.d
d
(
10
t=1
+
0 3,d
+
Di,d
d
(1)
(ee Fid + V
(19)
+
1. V
E
i.d
N
gi,d
(4)
Bx-1
( (idd) (0()) 21
(i.d)
(1) 27
Gd
(2)
+ (1)
Ve+ +e
vid + V / (:(0)
+
vi,d
Vid
Given the fact that V E + 0,
(t)
> and Ve [((]] > O, we have
+ e
(t) + Vid
and Ve+ uld + Vee
(1)) oid
> Ve + ((0)
(20)
By taking the absolute value of B, have
Gl0 (e)
Pi
E
9i,d
(1)
(2) 27
(1) (e)
d Rid
Fidd
(2)
|B|
(21)
(t)
((t)
(t)
(t) iidd
Vid
+
i.d)
+
H
To bound C, by applying the Young's inequality equation for N times and let
6
Di.d
(1)
E--1 1
Fi
(2) 27
a
2
(e) Ve
c + (t))
V c
+
+
we have
N
( 2
N
Fidd
(t)
2
F--11
Didd
(t)
27
C
<
+
(22)
3/2
1=1
+ V
i=l
+
idd
(1)
c +
iid
(1)
(1000) 2
Given the fact that V +
(1)
lidd
2
using Assumption and taking the conditional
expectation on C, we have
27
(1)
N1
N
(C) Fid
E
gi.d
1
E--1C|5
E--1
4
((()
c + (/to) 3/2
c +
i,d
E 1,a
70) 2
(1) 2 9i.d
1
N
d
N
E (0!2)) 27
(23)
E
4
i=1
+ i,d
1=1
c 2id
(1)
(00)
(0)
27
1
N
gid
<
+ RRY
E
4
(t)
11
c +
i,d
1=1
E j.d
30

---

Published as a conference paper at ICLR 2025
Now we focus H in 21). By applying the Young's inequality equation 9 for N times and let
Ve+ +
(1)
((t) Di.d
(t)
2
Di.d
7
-1
(ol:2)
(1)
2
a
c + (:(t)
b
+
2
(1)
(24)
we have
(1(1) 27
(1)))2
( 2
(t) 2
4
N
9i.d
N E
gi,d
Fi,d
H S
(25)
(1))
(1)
V + (0())
(0)
2
i,
E1-1
(
+
U/dd
Given the fact that +
9 and V Ve+ ((()
V Ec-1
(0/.2)
(1) 27
using Assumption 2
and taking the conditional expectation, we have
N
(1)
E,
g
(1)
2
N
E
9idd
(1)
gidd
E
[M]
5
EE-1
1,
E1
(olid)
2
+ w
(t)
c + vidd
(0)
2
(26)
N
2
N
Fidd
(0)
27
+
R
E
4
E
i,
By plugging (33 and (2) back to
21
we have
E-
E
[B]
>
2
Ex-1[1H)
[G()]
2
(8)
(27)
N
d
Fidd
-2R
E
c
(t)
c
(1)
By plugging
18 and 27
back to 16., we have
A E,
[B]
2 2
(t)
i,d
2
N
gi.d
(1) 2
-2
Ex-1
=(() i,d
Ve+ (0
E 1
(t)
F (zt(C-1)))²
N
V VaF, [
(1-1) - v (z((-11)
(28)
(1) Ve+
/ cct (.())
Vafila!!!1)
2
N
2R
E
c u
Uid
which completes the proof of Lemma T
Lemma 2 (sum of ratios with the denominator being the sum of past numerators) Given B2 < 1 and a non-negative sequence (ot) define b, - 85 aa for all t N+, then we have
a S in 1 br) ) T In(
Σ
(29)
E + bj
31

---

Published as a conference paper at ICLR 2025
Proof. Refer to Lemma 5.2 in Défossez et al. (2020).
Lemma 3 (averaged consensus error is bounded if sampling from i.i.d data distribution)
Under
Assumptions H for Algorithm
[
we have
VaFi(al -1p3)-VVaa [=F(E!(-))]
d=1
(30)
2(1 + NDD (1- (1- - 32)
R²2
[ In (1+
c(1 32) 31))11(33)) -In(32)
Proof. Denote the concatenation of the d-th dimension of all local models at t-th iteration
R.V.
==
J 1,d
2,
N.d
Denote the concatenation of N d-th dimension of the (virtual) averaged model
(t)
(t) E
ERN.
Denote the concatenation of the d-th dimension of updates of all local models at I-th iteration as
Vafi
(1-1)
((-1)) VAIn
(1) -04
N
ERV.
(1)
Ve+ ",d
V UN
N,
Denote the concatenation of N d-th dimension of updates of the (virtual) averaged model at L-th iteration as
N
Vafi
x
(t-1)
N Vafi x )
6/0
Cr N
E ERR
3,d
+ Vid
+
From the update function
12
it can be inferred that
d
(1)
X
(1-1)
1
1Nx
(1-1)
+ N
1 (1)
10x
(1-1)
+
2 (0)
ed
(31)
N
The update of x
can be rewritten as
(8) Wx(1-1) + c
(t-1) (t)
(32)
Then we have
x(1) 2
(t-1)
+ 1
x
(1-1) 8((0) [
1
d
d
W
IIN
1 IN
x
(t-1)
1 IN (0))
W
1
IN
(t-1) 1)
W
N IN
1
x
I
1
IN
(0)
N
N
(33)
W N IN
x
(1-1)
x((11) +
- - N
1
IN
W
N IN
1
(x ((-1)) x 1)
|
+
[1 -
1 1N
[1-- + N
(x - xx )
+ 2
W N 1N
32

---

Published as a conference paper at ICLR 2025
Recall the definition of À in Assumption
3
By applying the Young's 's inequality equation
9
and letting
7
1- A²
0,
a
W
1 N IN) )
(x (t-1)
- ") b
(1 I
1
1N
ed
2X²
N
we have
(1))
12
S
12
1 X2
2X
W
N
IN) ) (xx - x) |
+ 1 x2
1+ 12
( (
N
1
1
oo
(+)
(34)
By using
[II, we have
(0)
10)
2
1+ X²
(20-1)
x
2
1
(35)
2
x
Given that all local models start from the same point (0)), by summing up for all E [T'] and using telescoping, we have
1
2
X2
T
x
X
Σ
T
0/
12
(36)
Given that a₁ S
and using Lemma
we have
V1-62
1 1- X2
T
11-1)
x
(1-11)
2
N
T
Rid
(1)
X
2
2
X
1=1
id
2
(1 +1+X8 22
gidd
(37)
(1- -32)
(t)
+ Vidd
1=1
(1)
Vid
< (1 - X1 - B2)
In
1 +
TIn(B2)
By using Assumption
2 on the above inequality, we have
Σ
T
x
| - 2 - B2) [ In
2(1 X²>+)
1+
<(1 - 32) ) -1n(3))
R²2
(38)
We now focus on the left side of 30 By using iid assumption and Assumption [ we have
T N
D
T N
D
ΣΣΣ
V.F(a!
V V2F(a!(-1))
T
T
T
N
- ||
VF(a!
t=1
12
(8-1)
(t-1)
2
2
Ia
ΣΣ
T
(t (t-1) 1)
x
(1-1)
|
T
2
(39)
Plugging 38
to
33).
we have
T
N
D
21+*X)o2NL'D
1
Ri2
V
<
In
1
-
-In(32)
T
(1- 32)
(1
T
c(1
B2)
d=1
(40)
which completes the proof of Lemma 3
33

---

Published as a conference paper at ICLR 2025
B.3.3
MAIN PROOF OF THEOREM
From Assumption we have
F F(((1) (z())
(z())s(
SF F
T (z ( (z + VF (z(t-1))
L -((-)) +
=11-1)
x(t) x(1-1) |
(41)
2
By substituting equation 13 into equation 41 and taking the conditional expectation on both sides, have
Vali
(1-1))
B
| (a(t))
S F (z((-1))
at N
Σ
B B
V
(x((-1))
(1)
Vid
(42)
27
La
D
N
V Jafil [x]
11-1)
+
E
2
N
(8)
d=1
V c + v
By applying (T)
42
and letting
Vafi(a! (1-1)
N NV++ Uid
have
E,11 F (2(0)
S F (x(111))
at
D
VaF (x((-1))
T N Vdfi
x
(t-1) 1)
E--1
N
c n (t)
d=1
iid
(43)
Vafi(z!! (1-1))
La
B--1
2N
(1) +
E
uidd
By applying Lemma to (43)
we have
D N
V F (z(1-1))1²
E, | (2(0) ]
< F (z(t-1))
0t
4N
V Ve+ P.d
d=l i=l
VaFi (a!t
VeF (2)1-1)) 2
(44)
d=1
Tafi(a(t11)) 27
200R
La
D
N
+
+
E
N
2N
d=1
+ Vidd
Now focus the term A in 44).
From Assumption 2 have
V VE+ 70(0 S R R 2
1
and
recall that dr a
35
Then we have
D
$ 4R
Σ
=
4R
a
V (zF(
(45)
d=1
34

---

Published as a conference paper at ICLR 2025
Given the fact that dx
S a VI-B.
by plugging 45 into 44
we have
B F
x((1)
S F (z((--1)
a
V
Va
4R
VaFi (oq 11111)
VaF (I(1-11) ]²
N
B d=1i=1
Ve+
(()
(46)
idd
Vaf((ap-1)
27
2a
La
D
N
+
E
E--1
NVI- WVI--2 32
2N(1
B2)
(1)
+
Fidd
By taking the total expectation and summing up
46 over t E (T), we have
F
s F
(1100)
(()()11) ?
4R
T
VaF;
(2((-1)
v&F (z)
a
N
32
Ve+
[() indd
(47)
B
Vafi(a!(-1)) 2
2
La²
T
D
N
+
E
NV1
32
2N
B2)
1=1
d=1
c Vidd
By applying Lemma 2 for each d E [D] on C, we have
(
05
2aR
Laz
In
1
U.
idd
-Tin(B2)
N
B2
2N(1 - B2)
d=1
2aR
+
La
D
N
In
1
c(1 10--.0 32) )
R²
- -3ln(32)
(48)
NVT
332
2N $2)
d=l i=1
2n
DLa²
R²
+
In
1 1
- T (32)
VI -
2(1 32)
c(1
- B2)
Given that VC+ il0 > Ve, by applying Lemma
on B in 47 we have
2(1 + LD
R²
B
In 1+ c(1 ((1)))) -b)) -TI((2))
(49)
(1 - 22) - Bz 3/22 Ve
By plugging 49 and 48
into 47
have
T VoF
VoF(
4R
F
(00)
F.
1
In 1+ e(1 - B2)
R² )
TTl((2)) (50)
of
T
where
8DR2
2DLRa
x3
E
(51)
VI-
32)
(1-
32)
which completes the proof of Theorem
B.I
35

---

Published as a conference paper at ICLR 2025
B.4 PROOF OF THEOREM 4.1 DECENTRALIZED ADAM WITH HEAVY-BALL MOMENTUM
B.4.1 ALGORITHM UPDATE UNDER ANALYSIS
With heavy-ball momentum, we set 0 < B1 < B2
<
1. We have the algorithm update under analysis
(1) Vi(a (t-1), 1), 91
midd
(t)
Bimid
(t-1)
+
Di,d"
(1)
Lid
(1)
B2
(t-1)
+
gi.d
(t)
0t = a(1 - -Bi)
32 (2)
B2 Fidd
= 04
midd
(t)
(t-1) Wiy
j.d
(52)
(0)
U.d
JEN.
should be noted that there a modification on ag which should originally ar = 1- However, 1 3% could potentially make the learning rate non-monotonic. Since replacing it by has little practical influence, to simplify the proof, we replace it by 1 as in Détossez et al. (2020).
1- 8
T-
The (virtual) update of the globally averaged weights is
(8)
m
(1-1) d
(53)
N
We denote a!!h) E R as o(t)) Bzul" (1-1) 1) last term with its conditional expectation. In addition, we denote
(9.2)
which differs from oid by replacing the
T = F (z(1-1))
1
V
F.
(1((-1))
and G VVF:a((aa(-)))
(54)
N
We denote the d-th dimension of the local update of the i-th local model at f-th iteration without the heavy-ball momentum
(4)
U(C)
Did
V
We denote the d-th dimension of the local update of the i-th local model at f-th iteration with the heavy-ball momentum as
m c+
3 (t-R)
(8) v
V v
i.d k=
By replacing the last & terms in ei with their expectation, we denote ((.Nk)
as
(f.R) 1.d = 35011 (t-b) + EA-
Σ B5
(0122)
(55)
j=t-
B.4.2 TECHNICAL LEMMAS
Lemma 4 (adaptive update with momentum approximately follow descent direction)
1
G
()))))
D
Gid
())
Gll
())
ΣΣΣ
E
ΣΣ
3
E
(t,k) V +
(2,
k=
d=1 k=
c +
Did
3R
1-1
31
k
vk+
|
[1(-- 27
0 122 /1
ΣΣ
N t-1
Σ
3 Vi
V1 - 31
B2
4R
k=0
(56)
36

---

Published as a conference paper at ICLR 2025
where
(1) mi,d
K = E
(57)
Proof. According the definition of ki, have
D
1-1
N
(t-k)
(t)
Fi.d
K
Ve
d=1
k=
==1
1-1
((-k)
f-11 N
(1-b)
(58)
ΣΣ
G
Di.d
ΣΣΣ 8 (0 k=
Pid
(I)
(0)
d=1k=0
Now focus on B. By applying the Young's s inequality equation 9 and letting
(C-R) R3
VI- B1 =
9idd
= G (((-k)) /// b
2RVK +
VE+U iid
we have
B S
ΣΣΣ
D-1N
VI-Bi
(G(() )
Ryk +1
Di,d
(t-k)
(59)
ARV
-I E + vidd
+1==0=1
Given the fact that vid
+ B
(l-k)
widd b)), we have
(t-k)
1
||
(60)
(2)
3
d=1
+
By using Assumption I
and 10). we have
Σ ((()
0)²
|000 - -c(t
VF
(z(1)
VF
(1(- k)
2
(61)
T.2
1
(1-1)
02-1114
12
N
L2
N
<
N
1=1
By plugging
(61
and 660
back to
59
and given that /t/ V/ +
S VIc for 2 0, we have
|B|
1-1
a?
ok
B
k
N
RVk+1
31
4:
Σ
N
(t))
122
4R
vI
31
3r
k=0
1=1
1, -
Bi
8K
+
R
V
(1
|
2
4R
VI
3
B2
k=0
1=1
0212
Σ
Σ
1-1
R
N 1-1
Vk+
131 k
|
2
4R
V1-
3
B2
1=1
k=0
(62)
37

---

Published as a conference paper at ICLR 2025
Now focus on A in (58). Temporarily introduce the simplified notation
Σ
$2 3
(1) (1)))2
,²= = Et --1 [oa]
(idd)
j=1-
G G k)
i,d
G = (((-k)
g Didd
((-k)
== 1/dd
(0)
8 = i
(t.k)
Then we have 8 p2 82.
Now focus on A in 158 with the new notation, we have
(t-k)
E
()--)
Di.d
E
Gg
+
Ve+o)
1
Ve+
Ve+u
Vid
Gg
2 82
=R
E
B
+ GGg
Vc+2
/etoyctol (( vc+
(63)
G + G)
72 8²
E
Gg
Ve Ve+ū
VE+UVE o(ve
11
Now focus on in 63
By applying the Young's ' inequality equation 9 and letting
1
G - G
a= G. b Ve+i*
2VE+i"
then we have
3
G2
GG (G- - G)
E [
2
(64)
4
Ve+ē
Ve+u ė
Now focus on H in 63
Given the fact that ve+ 0 and vc > 0, we have
VE+u +
VE VE+U> > Ve+o and Ve+U+VE+ō > V
(65)
and given that 1r2 - 521 S 2 + 8², have
8²
H < Gg
+ GGg
(66)
VE + v(e (() 8)
Vc+ ((c+
By applying the Young's inequality equation 9 to 66 and letting
VI Bzve+i
G
1g|1
2
VE+U
Ve+ 8 +
we have
22
VI 02
g2
|Gg
Ve+ +
4VE+U
VI Bzlc + 0)332 +
(67)
G
g2r
4
V1- B2(c + 8)332 ( u)
Given that é + 8 2 22 and taking the conditional expectation, we have
2 4Ve+ + VI- UI-3VE+0
g²
E, E k-1
Ggl
S
(68)
Vc + vic 6)
E
+
Now focus on the second term in
66
By applying the Young's 's inequality equation 9 and letting
Vi +
Go
b 1g6|
a
2r
Ve+o c
38

---

Published as a conference paper at ICLR 2025
have
G CT-VI- 32
g2
Ggg
VE + E +
V
VI- 32V o(
(69)
g202p2
v1 +
Given that + 2 82 and E
6a
after taking the conditional expectation, we have
82
1
g²
E, k-1
Ggl
S
E,
(70)
vc d(c + v)
4VC+0
VT
32ve
6 c
By plugging (70) and (68) back to 66 we have
2,
g²
- [H]
E, k-1
(71)
2/c+0
VI-
B2ve+
c+U
Given rhat S Ve ū, and r Vk + 1R by Assumption
we have
G 2Rvk +
g²
E E [ S
2
+
VI
B2
E,
k-1
C+U
(72)
By recovering the temporarily introduced notation, we have
(((t-b))
((-x) 2
d
2R k +
Fed
B [1-l]] <
EC- --
(73)
2 c ii
(t.k)
B3
+
(2)
Given that + c1110] 2 + B/un1t 2 3/ (c + vid
(t-k)
((-x)), and by taking the complete expectation, we
have
b())))
(1-A)
1
2R K +
Fi.d
B[HH|] - =
(74)
Ve+ + (t.kk)
35 I B2
E + vitta (t-b)
By plugging 74 and (64) back to
63
we have
(((--) 2
(t-k) 27
(t-b)
())-k)
Di.d
2R 1 +
Dodd
E
>
E
E
Ve+rle
(t.k) + Ei
B /1 B3
c iid
((-k)
(75)
((- ((( 27
E
(1,R) c + i,d
39

---

Published as a conference paper at ICLR 2025
By plugging 75
back A in (58) we have
G
k()-R)
(t-b)
N
2R
1-1
+1
Filt
E[A
E
Σ
((
32
32
+ Bid
(t-k)
D
((t-k)
G(- 3,d
Vc+ g((,k)
(((-))
ΣΣ
D
2R
31
N Σ
||100-5016
IE
vk+
32
32
k=
[V Ve+ iid
(G(t=k)
s G G((-1))
ΣΣ
E
V (((,k)
By plugging 766 and 62 back to
58
we have
D
(t-
R)
t-1
Gid
(t--k)
G
G(t--k)
ΣV
K 2
E
(t.k)
(t,bb)
k=
+
d=1
k=
i=1
V + Uid
3R
1
B1
&
k
+
N
|
(t)()))) 27
07 1.2
31
N 1-1
27
8iVk
vI 32
B2
4R
0=
which completes the proof of Lemma
Lemma 5 (sum of ratios of the square of a decayed sum and a decayed sum of square) Given 0 < 31 < f S 1 and a sequence of real number (an)neNt. We define b, = Σド 82 -
ag and
cn = Σズ= 8 3n aa. a.. Then we have
In 1+ On ) In(
Σ E bj
(78)
B1)(1 31/32)
Proof. Refer to A.2 in Défossez et al. (2020).
Lemma (sum of geometric term times a square root)
Given < and Q E N, we have
Σ a c
1 1+
V 2 In(a)
5
2
(79)
l-a
-1-0)3/2 0)3/2'
4=0
Proof. Refer to A.3 in Défossez et (2020).
Lemma (sum of geometric term times a square root)
Given and Q E N, we have
4a
a" Vola+ 1)
(80)
(1 a) 5/22
Proof. Refer A.4 in Défossez et al. (2020)
40

---

Published as a conference paper at ICLR 2025
Lemma 8 (with heavy-bal ball momentum, averaged consensus error is bounded if sampling from i.i.d.
data distribution)
Under Assumptions
3
if 0 < 31 < B2 < 1, we have
D
G(U)
-1
In (1
(1 - 32)c ( 1n((3))
R²a
(1 - 12) 31/
(81)
Proof. Proof of Lemma Bis similar to Lemma 3
(0) Since the heavy-ball momentum is introduced, the definitions of of and oe are changed to
(C)
(1) m7
m
mNd N.d
E ERN
e
1,
N,d
(t)
(t)
Og
m
1.d
midd
E ER.
N
(0)
(1)
E
U
The proof is same as Lemma until
360.
Now we have
2(1 12)
(0)
ed
(82)
(1 22)2
By applying Lemma and given the fact that ar S a 1-9,
we have
VI-Dz²
T
(2)
Σ
i,d
(
1=1
6
(83)
(1 B1) a
In 1
Tln(32)
82
(1 - 31)(1 B1/32)
By using Assumption
2 it is given that (T) 1id
S RR AR//(11 (2). Then we have
Σ
(0)
2
S (1-3:)a2N
In 1+ (1 B2)
R²2
- -Tln(32))
(84)
2 (1 - 31 B1)
By plugging 84 back to 82 and
39
we have
2(1 + - Ni)
R²2
Lit
In
1+
-m((32)
T
i,
(1 (1 32)
T
(1 - B2)e
(85)
which completes the proof of Lemma g
B.4.3 MAIN PROOF OF THEOREM
Firstly introduce the notation Qt as
Ω
a: (1 (1-31) 3,)
1=0
By using Assumption T we have
(2(0) < (z(t-1)) + VF (z(--1)) T (a(0) ( x((-11) + L (()) - z(t-1)
(86)
41

---

Published as a conference paper at ICLR 2025
By using (10) and taking the complete expectation, we have
E
F
x
S E
F
[
Ctt
V a
(zl--1)
N
i.d
(4)
Loi
E
N
N
+ (0)
2N
d=1
1=1
i,d
(t)
S E
F
[
Ott
E
V F (1(111)
midd
La
E
(1)
N
(1)
2N
2
c + 1idd
(87)
From Assumption
we have Ve %
(((,))
S RR2, and by applying Lemma
we have
E
F
x
s E F
[x((-1)]
02
B
4RS2
N
3Ra
|
V
E
NVI - 11
32
Y=1
L2
B
N t-1 ΣΣ
(88)
E
3 Vk
4RN
1=1
1=1
((t-k) id
- -((-)) 27
Lo
N
3
E
E
N d=
(t.k) i,d
2N
By summing over all iterations E (T), moving terms, and given that or is set to be non-decreasing, we have
T
1-1
27
F [
Laz
N
T
S E
F.
+
4R
2N
a /1 B
N
T
1-1
V/V
3
4RN
R=
3RTT
T
k
N
| 1:2
Vk+
E
N
31
1=1
(t-k) -c -(( 27
D
1-1
N
T
QT
N
(2.b6) Vird
k=
1=1
E +
(89)
Now focus on B in 88
By using Lemma
5 and given that u
(9) < R22(11 (1--,) 32) (from Assumption 2
and ar 5 a- 1-8,
we have
"I-
DLa2 (1 B1) In 1+ (1 - --)3
R²
B
- TI((32)
(90)
2(1 - 32) B1/32)
Now focus in C' in 89 By introducing the index j = 1-1 l, we have
C L2 B1
E
u
(
Σ 8t V/ 1).
(91)
4RN
42

---

Published as a conference paper at ICLR 2025
By applying Lemma 7 and then Lemma 5
(similar to (90),, we have
B10 1?2
C (1 - 32)2RN
Σ
2
j=1
B1 B1 L2D
In 1 (1 B2)e
R²
TlI(32))
(92)
B1 (1 31/32)
112
In 1 + (1 32)c
Tln((B2)
(1 -11
3/22
Now focus H in and then Lemma we have
89 By introducing the index j = t- - k (similar to (91) and applying Lemma 6
3RTt
N
T
27 T Σ
=
VI+1-j j
N 31
B2
=1
[=]
(93)
6Rav 1
B1
R²a
In 1 (1
-TIn(3a)
31/32)3/2 /112//11111 /1/2/11///1111 -1-
B2)c
V c s((ak) 2 Ve, and by introducing index j = t k: we have
Now focus on / in
89
Given that
N
21 T Σ 3t
IS QT
(
E
NVC
f=1
j=1
(94)
N
D
T
0
ΣΣΣ
(G - -
-c() )
E
NVe- - B2
By applying Lemma
g we have
2(1 XP 110
R²
1 -1 12) - 12(1-11/22ve In 1+ (1 - 32)e
TlI((3)
(95)
Now focus on in 188. By introducing the index j = t k. we have
T
1-1
A=
at
B/E
||a
4R
t=1
K=0
a(1
- 31)
T Σ
EE
81-3
4R
(96)
T
a
Σ
(1
81
D)
E
(6) [
4R
j=1
a
Σ
T
(1-81
E
|
[
4R
For any T N+, we define a random index with value {0, TT 1} with distribution
N,jN, Pfr x -1----- -3-
(97)
Then we have
1- अT
B1
E1-
(1-
-T- =T 3
T
(98)
1 31
1- 3
43

---

Published as a conference paper at ICLR 2025
By introducing ==
31
we have
1-4
oT
A
E
VF
(99)
4Rt
By putting together (99).
155.
93
92 and 99
into
89
we have
E
2
S at
4R
(F F.)
E
In
1
R
T
In(32)
(100)
c(1- 32)
2
where
2DLR(1 3:)a
DL2D
24DR2 -1 B1
E
32)(1 B1/B2) + (1 31/ 31/32](1 -B6)3/2
(1 B1/B2) 3/2 /1 32
8(1 12)+ DV- AIa²
(1- X2) - 31) -31/32)ve
which completes the proof of Theorem 4.1
C DECENTRALIZED ACCUMULATED ADAM
Algorithm 4 describes the decentralized accumulated Adam. As is displayed in Section S the
decentralized accumulated Adam demonstrates competitive practical performances for several training tasks compared alternative methods.
Algorithm 4 Decentralized Accumulated Adam on worker
1: # of iterations in accumulation loop (for example, 4); T such that (T mod s) 0
(()) < 0: x + x b, +
2: m
3: for = 1.2, ..... T do
4:
+ [t/s]
(4)
Vi(a (t-
(1: (())))))
5:
0i
(1)
t 3 (1-1)
+ (1 31)9; (1)
6:
m;
(1) + B20 (20((-1) + +11- 9::²)
(t)
7:
(1)
m (11
(1) (t-1)
8:
4
+ EjENTO wij Tjj
v /(1-11111))) -/)=-
9:
b. - b, 9
(t)
10:
ift mod 8 == 0 then
11:
m (1) < fim (i-1)
+ (1 - -11bbb
a1) + 310 (f-f11
1) + (1 31)[6.]
12:
b. 0
14: end if
end for
To understand Algorithm let introduce & to divide the sequence of iteration index into groups of every iterations:
T --1
T s
t
1,...,s,
$ s+ 1......, 2s
8
f
1,
2,
For given iteration index f, it falls in the f = [t/s]-th group, so that m (1) in Algorithm can be equivalently expressed as
= (1 (1-31) (29 (0) + Bfc(f-11 +... 8 (!
(101)
44

---

(t) Published as a conference paper at ICLR 2025
where G:!0 := (s! ( +... 9; ). The update of v/"" is similar.
Algorithm uses accumulated stochastic gradients to construct the first-order momentum m; and second-momentum uith, which reduces the variance of the stochastic gradients while does not reduce the number of model updates, i.e., within 7 stochastic gradient computations at worker i, the model ( is updated for T instead of T/s times. To see these, assume the empirically optimal batch size in single-machine enviromment is B, and we call a batch of size B a global mini-batch. The variance of the global mini-batch gradients for the global gradient F is > Under a decentralized setup, these B batches are evenly distributed which we call a local mini-batch Then, the variance of each worker's 's local batch gradient for the global gradient V F increases to NE², This increased variance exacerbates the consensus errors among workers. To compensate for the increased variance, a straightforward idea is to have each worker accumulate s (proportional to N) local mini-batch random gradients before performing an update, which improves the variance to E² However, this accumulation leads to a reduction in the number of model update iterations, i.c., within T' stochastic gradient computations, the number of model updates is T/s instead of T. With a novel accumulation technique, this issue is avoided in Algorithm 4 As described in Algorithm when a worker computes a local mini-batch g," it assigns it weight of adds it to the exponential averaging of all historical ( G(i-1) to generate m!") and (t) and updates the local model once. In this way, the number of updates is equal to T instead of T/s.
N workers, with each worker receiving B B/N samples,
Algorithm has close connections with existing well-known algorithms. On one hand, it performs exponential moving averaging over each group of stochastic gradients, which yields higher weight
of the oldest information compared to the vanilla Adam (8f-1 versus 84 see equation [01). This technique is similar to SlowMo Wang et al. (2019) that has been shown to be effective for DNN training. On the other hand, Algorithm 4 updates the model upon receiving each gi, which similar to FedAvg Konečný et al. (2016). Is worth noting that FedAvg does not communicate every iteration and may yield a lower communication cost compared to Algorithm 4 Whether this technique can further accelerate Algorithm 4 is worthwhile research direction. We leave it as well as the convergence analysis of Algorithm 4 as future work.
Similar to the changes in AdamW Loshchilov (017) compared with Adam, AccumAdam can also be similarly adapted to AccumAdam W.
D DISCUSSION AND LIMITATION
Our analysis of the runtime model in Section 3 and convergence guarantees in show how the com- munication topology affects both the per-iteration runtime and the convergence. A sparser topology shortens the per-iteration runtime but may result in a topology with slower mixing and worse training results. Consistent with Kong et al. (021)) we have found that for a limited number of workers (32 or less), can obtain significant runtime improvements by sparsifying the communication without affecting the quality of the final model. For a network with a large number of workers, the sparse topology plays more important role and we may need to study the runtime and optimization aspects separately. Although decentralized training may face challenges with a large number of workers (> 64), data parallelism not the only approach for scaling up distributed training. For large language models, tensor parallelism and model parallelism are also crucial techniques, and the superiority of decentralized training in interleaving communication and computation can be a great complement to overall efficiency. In Shhoeyi et al. (2019). for example, they used 64-way data parallel and 8- 8-way model parallel to scale up the training to 512 GPUs. Additionally, in common machine learning workloads, toolsmall local batch size could lead to low utilization for single GPU, and a 64-way data parallel is already sufficient for most tasks.
As for memory consumption, since the decentralized training has asynchronous communications, it is inevitable to allocate additional memory for the communication buffers to hide the time taken by communications, Similar implementations Even though the decentralized training allocates more memory, it is a off between memory and per-iteration runtime if compared with All-Reduce training. Moreover, it should also be noted
be found in Assran et al. (2019); Wang et al. (2019).
45

---

Published as a conference paper at ICLR 2025
that more memory is spent on the backward pass and the optimizer states (like Adam Kingma & Ba (2014)) The additional memory usage can alleviated by orthogonal techniques like compression.
In experiments, used time-varying mixing matrix, but for case of convergence analysis, we used fixed one. Techniques from some works for time-varying topologies Nedic et al. (2017);
Saadatniaki et al. (2020); Metelev et al. (2023) can be applied here. Our accumulated Adam performs well in practice, but it lacks theoretical analysis. It could be finished by using the analysis techniques of the Adam method Zaheer al. (2018) and decentralized algorithms Yuan et al. (2016); Zeng & Yin (2018).
46


Reviews:
[{"review_id": "k3fSRcrSn2", "reviewer": "Reviewer_oHbd", "summary": "The paper investigates decentralized training of DNNs. This used to be a fairly hot topic a few years ago (e.g. Lian et al 2017 and follow-up work), but has to some extent cooled down recently as models and workloads have changed, and model-parallelism has become more standard. \nThe paper focuses on the classic data-parallel setting; its contributions are as follows: \n\n- An analytical model to understand what are the conditions under which decentralized training can bring gains.\n\n- A decentralized version of Adam, which decouples computation from communication and provides convergence guarantees under fairly standard assumptions (for Adam) \n\n- An implementation of the algorithm and its technical evaluation. This is done fairly thoroughly, on systems with up to 64 GPUs, showing the method’s potential.", "questions": "Detailed comments and questions:\n\nQ1. The paper seems to ignore work on the decentralized setting since late 2020 to early 2021. Here are some of the many references that are missed:\n\n- There is a lot of nice work by Anastasia Koloskova and co-authors on analyzing Gossip variants of SGD, many of which are missed:\n\n1. Koloskova, Anastasia, et al. \"A unified theory of decentralized sgd with changing topology and local updates.\" International Conference on Machine Learning. PMLR, 2020.\n\n2. Koloskova, Anastasiia, Tao Lin, and Sebastian U. Stich. \"An improved analysis of gradient tracking for decentralized machine learning.\" Advances in Neural Information Processing Systems 34 (2021): 11422-11435. \n\nSee also:\n\nZhang, Jiaqi, and Keyou You. \"Fully asynchronous distributed optimization with linear convergence in directed networks.\" arXiv preprint arXiv:1901.08215 (2019).\n\n\"SQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimization\" by Navjot Singh et al. (2020)\n\nIn addition: \n\n- Nadiradze, Giorgi, et al. \"Asynchronous decentralized SGD with quantized and local updates.\" Advances in Neural Information Processing Systems 34 (2021): 6829-6842.\n\nThis paper seems to be trying to do something similar to what is done here–decoupling communication from computation–but in some sense goes further since it also supports quantization and completely non-blocking reads. Even the experimental setup is very similar to the one presented in this paper, so I am surprised to see that there is no mention of this work whatsoever. \n\n- Similar work: \n Li, Shigang, et al. \"Breaking (global) barriers in parallel stochastic optimization with wait-avoiding group averaging.\" IEEE Transactions on Parallel and Distributed Systems 32.7 (2020): 1725-1739.\n\nI would therefore recommend that the authors go over related work more thoroughly, and provide more complete positioning of their work relative to prior art. I don’t think the paper can be accepted in the absence of this. \n\nQ2: Are the loss improvements quoted in L488 (2.841 vs 2.846) actually statistically significant?\n\nQ3: Is your setup really faster than just setting up large-batch SGD using e.g. FFCV on a single 4-GPU system?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n\n1. new perspectives on an “old” problem by today’s standards (decentralized training), from the point of view of modeling and adaptive optimization\n\n2. analytical results are a plus\n\n3. the experiments are fairly thorough", "weaknesses": "Weaknesses: \n\n1. unfortunately the paper seems to be completely missed some important related work in the area, which makes it very hard to position the paper properly in terms of its contribution \n\n2. more broadly, the experimental results are in a system parameter range that has been rendered somewhat obsolete by current-day systems, which are able to e.g. train ImageNet in minutes on a single node https://github.com/libffcv/ffcv", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JrFK5m8nTI", "reviewer": "Reviewer_pREt", "summary": "This paper investigates the gap between decentralized training of DNN and data parallel (DP) methods. \nIn particularly, although decentralized training is theoretically superior to DP, certain network topologies make the use of this technique challenging. \nThe present propose to quantify such gap through the use of a \"runtime model\".\nMoreover, the author introduce a decentralized version of Adam particularly suited to the training of transformer based models.\nExtensive numerical experiments illustrate the speedup investigation.", "questions": "The reviewer is curious about the intuition behind the runtime model: were the assumptions behind the model proposed previously by some other work or is this coming solely from the author?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes to study an important, impactful problem (practical gap between DP and decentralized training)\n- the accuracy of the model (ie how it captures real time speed discrepancies) is well illustrated\n- the numerical experiments are extensive and thorough. There is a clear focus on transformer based architectures, making it relevant to challenges posed by modern LLMs.\n- the version of decentralized Adam optimizer is thoroughly compared to other decentralized adam based methods\n- the papers lists 3 factors that could explain the aforementioned discrepancy  1) overlapping communication and computation 2) heterogeneous communications costs 3) sensitivity to varying computation times. For each factor, an extensive investigation is proposed supported by numerical experiments.", "weaknesses": "- the explanation/details of the runtime model in Appendix A.5 could be stated more clearly \n(the reviewers would like to respectfully suggest an explanation similar in spirit to the way Algorithm 1 is stated, although this is relatively minor)\n- there is no clear link to the reviewer between the decentralized adam optimizer and the runtime model (please correct me if missed anything).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y50wCXALKN", "reviewer": "Reviewer_LAtq", "summary": "The paper explores decentralized training of deep neural networks (DNNs) as an alternative to synchronous data-parallel methods like All-Reduce, which are known for their scalability issues. The authors identify three key factors that can lead to speedups in decentralized training: overlapping communication and computation, respecting heterogeneous communication costs, and reducing sensitivity to varying computation times. They propose a runtime model to optimize decentralized training configurations and design a decentralized variant of the Adam optimizer, called DAdam, which supports overlapping communications and computations. Additionally, they introduce an accumulation technique to mitigate the high variance caused by small local batch sizes.", "questions": "Please refer to the weaknesses. \n\nIt would be better to provide training convergence curves of using DAdam and the All-Reduce Adam on OpenWebText. \n\nAlso, it is important to illustrate how the decentralized training is implemented, as the Pytorch and DeepSpeed have conducted many system optimizations on the All-Reduce, there should be some convincing illustrations to show that the implemented decentralized training can outperform the All-Reduce. Specifically, how the communication is launched, using torchrun or MPIRUN or something else? how the synchronization between different nodes is implemented, is there a distributed.barrier()? how the overlapping between communication and computation is implemented? How do you manage the communication topology? The source code with a document may be very helpful.\n\nIf these questions are addressed, I'd like to increase my score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors provide a runtime model that quantifies key environmental parameters and estimates potential speedups, offering valuable insights into the conditions under which decentralized training is advantageous.\n2. There is theoretical analysis to support the proposed algorithm.\n3. The experiment consists of 64 GPUs, which is a suitable scale.\n4. There is detailed hyper-parameter settings for reproducibility.", "weaknesses": "1. Some important related works in MLsys [1,2] are missed, in which the dynamic gossip communication topologies are also discussed, and the communication compression is utilized.\n2. The novelty of the DAdam is limited. There is no improvements on the Adam itself. The proposed method can be seen as a system optimization instead of the optimizer itself. However, the systematic optimization including the overlapping between communication and computation is largely used in many ML systems [3,4].\n3. While the paper presents extensive experiments, it lacks detailed ablation studies to isolate the impact of individual components of the proposed approach.\n4. The tested topologies are limited.\n5. For training GPT-2 on OpenWebText, it is unclear whether the final convergence of DAdam can match the All-Reduce Adam when training with more iterations. The convergence curves of training on OpenWebText are not provided.\n6. There lacks enough details of how the decentralized training is implemented built upon PyTorch.\n\n\n[1] GossipFL: A Decentralized Federated Learning Framework With Sparsified and Adaptive Communication. In TPDS 2022.\n[2] Communication-efficient decentralized learning with sparsification and adaptive peer selection. In ICDCS 2020.\n[3] Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models. In ASPLOS 2022.\n[4] Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communication Partitioning. In ASPLOS 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)