From Promise to Practice: Realizing High-performance Decentralized Training

Authors: Zesen Wang, Jiaojiao Zhang, Xuyang Wu, Mikael Johansson
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: distributed training, data parallelism, decentralized optimization

Abstract:
Decentralized training of deep neural networks has attracted significant attention for its theoretically superior scalability compared to synchronous data-parallel methods like All-Reduce. However, realizing this potential in multi-node training is challenging due to the complex design space that involves communication topologies, computation patterns, and optimization algorithms. This paper identifies three key factors that can lead to speedups over All-Reduce training and constructs a runtime model to determine when and how decentralization can shorten the per-iteration runtimes. To support the decentralized training of transformer-based models, we introduce a decentralized Adam algorithm that overlaps communications with computations, prove its convergence, and propose an accumulation technique to mitigate the high variance caused by small local batch sizes. We deploy our solution in clusters with up to 64 GPUs, demonstrating its practical advantages in both runtime and generalization performance under a fixed iteration budget.
The experiment code is open-source at [https://github.com/WangZesen/Decentralized-Training-Exp](https://github.com/WangZesen/Decentralized-Training-Exp), and the extension code is open-source at [https://github.com/WangZesen/Decent-DP](https://github.com/WangZesen/Decent-DP).

Decision: accept (poster)
Reviewer scores: 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730377834445

thread_type:
official_review

<thread_object>
creator: reviewer_latq

<content>
summary:
The paper explores decentralized training of deep neural networks (DNNs) as an alternative to synchronous data-parallel methods like All-Reduce, which are known for their scalability issues. The authors identify three key factors that can lead to speedups in decentralized training: overlapping communication and computation, respecting heterogeneous communication costs, and reducing sensitivity to varying computation times. They propose a runtime model to optimize decentralized training configurations and design a decentralized variant of the Adam optimizer, called DAdam, which supports overlapping communications and computations. Additionally, they introduce an accumulation technique to mitigate the high variance caused by small local batch sizes.

strengths:
1. The authors provide a runtime model that quantifies key environmental parameters and estimates potential speedups, offering valuable insights into the conditions under which decentralized training is advantageous.
2. There is theoretical analysis to support the proposed algorithm.
3. The experiment consists of 64 GPUs, which is a suitable scale.
4. There is detailed hyper-parameter settings for reproducibility.

weaknesses:
1. Some important related works in MLsys [1,2] are missed, in which the dynamic gossip communication topologies are also discussed, and the communication compression is utilized.
2. The novelty of the DAdam is limited. There is no improvements on the Adam itself. The proposed method can be seen as a system optimization instead of the optimizer itself. However, the systematic optimization including the overlapping between communication and computation is largely used in many ML systems [3,4].
3. While the paper presents extensive experiments, it lacks detailed ablation studies to isolate the impact of individual components of the proposed approach.
4. The tested topologies are limited.
5. For training GPT-2 on OpenWebText, it is unclear whether the final convergence of DAdam can match the All-Reduce Adam when training with more iterations. The convergence curves of training on OpenWebText are not provided.
6. There lacks enough details of how the decentralized training is implemented built upon PyTorch.


[1] GossipFL: A Decentralized Federated Learning Framework With Sparsified and Adaptive Communication. In TPDS 2022.
[2] Communication-efficient decentralized learning with sparsification and adaptive peer selection. In ICDCS 2020.
[3] Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models. In ASPLOS 2022.
[4] Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communication Partitioning. In ASPLOS 2024.

questions:
Please refer to the weaknesses. 

It would be better to provide training convergence curves of using DAdam and the All-Reduce Adam on OpenWebText. 

Also, it is important to illustrate how the decentralized training is implemented, as the Pytorch and DeepSpeed have conducted many system optimizations on the All-Reduce, there should be some convincing illustrations to show that the implemented decentralized training can outperform the All-Reduce. Specifically, how the communication is launched, using torchrun or MPIRUN or something else? how the synchronization between different nodes is implemented, is there a distributed.barrier()? how the overlapping between communication and computation is implemented? How do you manage the communication topology? The source code with a document may be very helpful.

If these questions are addressed, I'd like to increase my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730607606127

thread_type:
official_review

<thread_object>
creator: reviewer_pret

<content>
summary:
This paper investigates the gap between decentralized training of DNN and data parallel (DP) methods. 
In particularly, although decentralized training is theoretically superior to DP, certain network topologies make the use of this technique challenging. 
The present propose to quantify such gap through the use of a "runtime model".
Moreover, the author introduce a decentralized version of Adam particularly suited to the training of transformer based models.
Extensive numerical experiments illustrate the speedup investigation.

strengths:
- This paper proposes to study an important, impactful problem (practical gap between DP and decentralized training)
- the accuracy of the model (ie how it captures real time speed discrepancies) is well illustrated
- the numerical experiments are extensive and thorough. There is a clear focus on transformer based architectures, making it relevant to challenges posed by modern LLMs.
- the version of decentralized Adam optimizer is thoroughly compared to other decentralized adam based methods
- the papers lists 3 factors that could explain the aforementioned discrepancy  1) overlapping communication and computation 2) heterogeneous communications costs 3) sensitivity to varying computation times. For each factor, an extensive investigation is proposed supported by numerical experiments.

weaknesses:
- the explanation/details of the runtime model in Appendix A.5 could be stated more clearly 
(the reviewers would like to respectfully suggest an explanation similar in spirit to the way Algorithm 1 is stated, although this is relatively minor)
- there is no clear link to the reviewer between the decentralized adam optimizer and the runtime model (please correct me if missed anything).

questions:
The reviewer is curious about the intuition behind the runtime model: were the assumptions behind the model proposed previously by some other work or is this coming solely from the author?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628428648

thread_type:
official_review

<thread_object>
creator: reviewer_ohbd

<content>
summary:
The paper investigates decentralized training of DNNs. This used to be a fairly hot topic a few years ago (e.g. Lian et al 2017 and follow-up work), but has to some extent cooled down recently as models and workloads have changed, and model-parallelism has become more standard. 
The paper focuses on the classic data-parallel setting; its contributions are as follows: 

- An analytical model to understand what are the conditions under which decentralized training can bring gains.

- A decentralized version of Adam, which decouples computation from communication and provides convergence guarantees under fairly standard assumptions (for Adam) 

- An implementation of the algorithm and its technical evaluation. This is done fairly thoroughly, on systems with up to 64 GPUs, showing the method’s potential.

strengths:
Strengths:

1. new perspectives on an “old” problem by today’s standards (decentralized training), from the point of view of modeling and adaptive optimization

2. analytical results are a plus

3. the experiments are fairly thorough

weaknesses:
Weaknesses: 

1. unfortunately the paper seems to be completely missed some important related work in the area, which makes it very hard to position the paper properly in terms of its contribution 

2. more broadly, the experimental results are in a system parameter range that has been rendered somewhat obsolete by current-day systems, which are able to e.g. train ImageNet in minutes on a single node https://github.com/libffcv/ffcv

questions:
Detailed comments and questions:

Q1. The paper seems to ignore work on the decentralized setting since late 2020 to early 2021. Here are some of the many references that are missed:

- There is a lot of nice work by Anastasia Koloskova and co-authors on analyzing Gossip variants of SGD, many of which are missed:

1. Koloskova, Anastasia, et al. "A unified theory of decentralized sgd with changing topology and local updates." International Conference on Machine Learning. PMLR, 2020.

2. Koloskova, Anastasiia, Tao Lin, and Sebastian U. Stich. "An improved analysis of gradient tracking for decentralized machine learning." Advances in Neural Information Processing Systems 34 (2021): 11422-11435. 

See also:

Zhang, Jiaqi, and Keyou You. "Fully asynchronous distributed optimization with linear convergence in directed networks." arXiv preprint arXiv:1901.08215 (2019).

"SQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimization" by Navjot Singh et al. (2020)

In addition: 

- Nadiradze, Giorgi, et al. "Asynchronous decentralized SGD with quantized and local updates." Advances in Neural Information Processing Systems 34 (2021): 6829-6842.

This paper seems to be trying to do something similar to what is done here–decoupling communication from computation–but in some sense goes further since it also supports quantization and completely non-blocking reads. Even the experimental setup is very similar to the one presented in this paper, so I am surprised to see that there is no mention of this work whatsoever. 

- Similar work: 
 Li, Shigang, et al. "Breaking (global) barriers in parallel stochastic optimization with wait-avoiding group averaging." IEEE Transactions on Parallel and Distributed Systems 32.7 (2020): 1725-1739.

I would therefore recommend that the authors go over related work more thoroughly, and provide more complete positioning of their work relative to prior art. I don’t think the paper can be accepted in the absence of this. 

Q2: Are the loss improvements quoted in L488 (2.841 vs 2.846) actually statistically significant?

Q3: Is your setup really faster than just setting up large-batch SGD using e.g. FFCV on a single 4-GPU system?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732043426992

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to thank all the reviewers for their kind and constructive reviews of our paper. 

Generally, the reviewers affirmed our contributions on 
1. the runtime model that quantifies key factors in enabling speedup of decentralized training, 
2. thorough numerical experiments with a focus on Transformer-based models, and 
3. design and analysis of decentralized optimizers with second-order adaptive momentum. 

The reviewers also raised questions about 
1. related works in decentralized training and the benefits over other efficient training frameworks, 
2. a few minor issues that were unclear in the initial submission, and 
3. implementation details of the decentralized training extension. 

In the responses, we addressed the questions by:
1. clarifying how the related work suggested by the reviewers relates to our paper, conducting new experiments to highlight differences, and describing how the reviewers' suggestions will be incorporated in the revised version of our paper,
2. providing additional explanations and experiments that clarify our work, and 
3. uploading the source code of our implementation with examples and detailed explanations.

In general, our paper investigates the design space of decentralized training systems that facilitate practical runtime speedups over AllReduce training and maintain a comparable generalization performance, especially on Transformer-based models. Key contributions include a runtime model that exposes tradeoffs and guides the overall design process, a decentralized variant of the Adam optimizer with a novel accumulation mechanism, an overall system design implemented as a PyTorch extension, and extensive numerical experiments that validate both runtime and generalization performance.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper studies decentralized training in the classic data parallel setting. It gives an analytical model to understand under what conditions decentralized training can bring gains. It also gives a relevant decentralized version of Adam, with a novel convergence analysis. Finally empirical results are convincing on realistic  settings with up to 64 GPUs.
Some concerns were raised about missing discussions of related work, which we hope authors will resolve in the final version.
All four reviewers ultimately recommend acceptance.