Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a system for training large language models on hardware connected over the public internet and distributed around the world. This is a difficult problem (EleutherAI writes that “doing volunteer-driven distributed compute well for this use case is an unsolved problem”) and the authors tackle it with a combination of pipelining and data parallelism.

As this setting can’t assume homogeneous network performance, the parallelism assignment problem is much more difficult than it is in datacenters, and the authors apply a novel combination of algorithmic approaches to solve it, including treating pipelining and data parallelism independently and scheduling parallel units onto workers using an evolutionary algorithm.

The authors report benchmark results on a series of increasingly heterogeneous network configurations, showing significant speedup relative to a grid-search-optimized Megatron setup with homogeneous data and pipeline parallelism.

strengths_and_weaknesses: The authors do a good job of motivating the problem and pointing out weaknesses in existing systems (e.g. a focus on data parallelism or mixture-of-experts models).

The paper uses the terminology “macro-batch” but it’s sometimes unclear whether this refers to the portion of the global batch that runs on one data-parallel replica, or to the portion that runs on one stage of one data-parallel replica (more often called a microbatch). I’d recommend being a little clearer here.

The algorithm the authors introduce seems reasonable and well-motivated. If it’s fast enough, and if decomposing the assignment problem into separate data and pipeline levels isn’t a restrictive assumption (both of these appear to be true!) then it also seems like the only algorithm we need for this problem. (It might be possible to optimize in an even greedier way, but if the evolutionary approach is fast enough I’m not sure it matters).

I’m somewhat disappointed that a system for heterogeneous distributed training (justified in part by a seemingly accurate claim that home electricity is 50-100x cheaper than spot GPU instances) was benchmarked only on cloud machines with cloud networking, not on machines with conventional home internet. But it looks like the bandwidth between distant datacenters in the authors’ experiments is low enough that it might be representative of home network connections.

The authors present good benchmarks against grid-search-optimized Megatron configurations, and their ablations, although brief, are convincing about the value of their local search strategy.

questions: How much do you lose by decomposing the assignment problem into separate data and pipeline levels? I think the answer might be “nothing”, but it would be great to be more explicit about that.

The intra-node bandwidth is quoted as 100 Gb/s in your experiments, but IIUC V100 NVlink is 150 GB/s (1200 Gb/s); is there a reason this isn’t the right number to use for intra-node bandwidth? And in general, are the bandwidths you’re assuming the result of benchmarks? If so, maybe include those benchmarks somewhere?

Were the benchmarks run on cloud machines that were really physically located in different regions, or were the bandwidths between machines limited artificially to simulate inter-region connections?

limitations: The authors are clear about most of the limitations of their system, including lack of online failure handling and dynamic replanning.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The work develops communication-efficient schedulers for parallel sub-units for the geo-distributed low-bandwidth high-latency training of GPT-2-style transformer language models. Both data-parallel sub-units (gradient communication) and pipeline-parallel sub-units (activation communication) are considered. The sub-units are optimized using statistics of latency and bandwidth together with a global genetic algorithm with local node-swap optimization (similar to Kernighan-Lin).

strengths_and_weaknesses: Strengths:

- The presentation is excellent. The writing is very clear, figures help with understanding and math is used appropriately.
- The evaluations are robust and extensive. The experiments were conducted in real geo-distributed settings.
- Ablations provide insight into baseline Kernighan-Lin search vs improved swapping algorithm

Weaknesses:
- Failure of nodes/GPUs is not discussed but mentioned as a limitation

This is one of the best papers that I read in this area for a while and I am happy to champion the paper and fight for its acceptance.
questions: Questions:

How does your approach scale vs Megatron as you scale the model?

Comments:
- Typo in line 226: generic = genetic
- Figure 4: It would be better to replace case 1,2... with On Demand, Spot, multiple DC, regional, world-wide. If you can find room, it would be beneficial to enlarge this figure as it is an important results that deserves a little more attention.
- It is better to refer to GPT3-XL by the number of parameters, that is, GPT-3 1.3B

limitations: A few limitations are discussed in the paper, but no separate limitation section exists. Ethical considerations should be explored. Making large model training more accessible to not require a supercomputer-quality network increases accessibility both for small institutions (democratization) but might also favor those that already have the most resources more (companies buying all the cloud GPUs that they can find world-wide, to train large neural networks cheaply)
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Proposes techniques for scheduling training tasks in decentralized heterogenous clusters. 
strengths_and_weaknesses: 
Strengths
---
* achieves good speed up over Megatron in the decentralized setting
* only 2x slower in such decentralized setting even though the bandwidth/latency is several orders slower when going geo distributed
* 2 level scheduling models is intuitive

Weakness
---

* Ignores some of the latest innovation in Tensor model-parallel (TMP) strategies
* Ignores the fact that GPUs may itself be of different types across geos

questions: * why not compare against other state of the art approaches such as

1. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020

2. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.

* It is not clear as to why there is a need for two-level optimization. Many of the cost functions are rather heuristic-driven. Is there a space for employing RL techniques for scheduling as suggested by others "Shyalika, Chathurangi, Thushari Silva, and Asoka Karunananda. "Reinforcement learning in dynamic task scheduling: A review."
limitations: * The related work and comparison could have been better.

* Exploration of newer techniques for optimization rather than using genetic algorithms
review_title: null

Meta-review: metareview: All of the reviewers felt that this is a strong submission.  The paper gives a new novel approach for scheduling decentralized training tasks.  This will be of general interest to the community.
confidence: Certain
Decision: Accept