Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method called “SWARM parallelism” to allow model-parallel training of large models on heterogeneous clusters (e.g., a mix of weak and strong GPUs, unreliable compute nodes or networks with uneven bandwidth). 
strength_and_weaknesses: Strength:

- The problem this paper is trying to address is important.
- Overall the paper is easy to follow
- The proposed technique looks sensible.


Weakness:
- In writing, the paper seems to spend too much effort explaining the background or something obvious, instead of giving an in-depth study, empirical or theoretical, about the proposed methods.

- Some contributions claimed by this paper are questionable. For example, I doubt the so-called “square-cube” law is a contribution of this paper, because It has been studied quite intensively in several large LM system papers, and has been used in many papers published before this paper. This paper seems to just re-brand this common wisdom. Accordingly, I feel the writing in section 3.1 and the experiments in 4.1 are unnecessary and does not reveal any new insight to this area.

- What remains novel in this paper is the SWARM parallelism, which is indeed interesting and trying to address a very important problem. However, I feel the authors’ execution (such as in section 3.2 and in experiments) in explaining the proposed method and proving the method is practical could have been done better. I have some detailed comments provided next.

clarity,_quality,_novelty_and_reproducibility: The “square-cube” law proposed in this paper is nice but quite obvious to me; it has been discussed and revealed in many papers such as scaling language models [1]. Also, pipeline parallelism having lower communication (in this paper’s phrasing – “pipeline parallelism naturally grows more communication-efficient with model size”) had also been revealed in many previous papers such as [2][3][4][5].
Given that the “square-cube” laws seem to be just a summarization/re-branding of several obvious facts (which have been leveraged in many papers), I don’t think the contents in section 3.1 are novel or interesting. I am not sure if this can be counted as a contribution to this paper.

[1] Scaling Laws for Neural Language Models

[2] Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

[3] Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

[4] Piper: Multidimensional Planner for DNN Parallelization

Writing and clarity:

The writing of section 3.2 is rather ad-hoc. Is it possible to formalize a few symbols for each stage of the model and each device of the cluster, and develop an algorithm (and if possible, equations) to illustrate how exactly the stochastic wiring and rebalancing work?


Experiments:

In sec 4.2, the choices of baselines are very tricky. If the goal is to compare the performance in ideal conditions, why don’t you compare it to stronger and more practical baselines? Here are my concerns:
- Gpipe faces a peak memory issue, which limits the largest possible number of micro-batches it can use, which in turn limits its throughput because a smaller number of micro-batch causes bubbles. I think you should choose the 1F1B schedule as a baseline, which addresses this problem.
- For Zero, why don’t you choose Zero-2 or Zero-3 but Zero-offload? If the goal is to compare the performance in ideal condition, I guess Zero-2 and Zero-3 can almost always give better performance than Zero-offload (since zero-offload has offloading which would be a penalty on performance). This is also related to the cluster you choose to perform the experiments in sec 4.2 --- because I do notice that you have 7x nodes with 8 A100 each. I suppose the experiment in 4.2 should be performed on the A100 cluster (which has more GPU memory and make Zero-Offload unnecessary).


- In sec 4.3, could you elaborate on this sentence:  “with 1.01 billion parameters in total: because of layer sharing, it is equivalent to a 13B model from (Brown et al., 2020) in terms of computing requirements.” ?
I am wondering if this means the layer sharing you introduced makes your method more advantageous because it increases the compute-to-communication ratio (because this layer sharding seems to increase the flops needed per parameter). Could you perform experiments on a standard, smaller GPT-3 model without layer sharing and report the results?

- The results in Table 3 look good. But these results seem to be achieved by a combination of many techniques, including the proposed ones in this paper, and some compression techniques which can reduce communication. Is it possible to isolate these factors with an ablation study to show how much improvement your approaches exactly bring?

- Overall, I feel the experiments section should focus on how SWARM can handle heterogeneous devices, handle failures, and handle uneven bandwidth. The results presented in 4.1 is not interesting; the results presented in 4.2 is less relevant and do not contribute to many of the claims you have made earlier in the paper about SWARM. 

summary_of_the_review: The paper has proposed a nice and sensible idea to address the model-parallel training in a very heterogeneous cluster environment, but the paper does not provide enough evidence to prove the method is indeed practical and verify the author's many claims.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces SWARM Parallelism, a new framework for large-scale distributed training of large ML models. The framework is designed based on the generic square-cube law, and includes several designs to further handle nonrobust devices and network communications. Large-scale experiments with GPT-3 and xxlarge in comparison with state-of-the-art baselines show the efficiency advantages of the proposed framework.
strength_and_weaknesses: S1: The problem of large-scale distributed model training is important and relatively less studied.
S2: The proposed framework is based on sound observations. The technical designs are well-motivated and straightforward, with clear reasoning, explanations, and illustrations.
S3: Experimental evaluations are comprehensive and strong.

W1: Analysis and discussions on convergence are lacking.
W2: Ablation evaluations regarding the stochastic wiring and adaptive swarm rebalancing techniques seem to be missing (at least in the main content).

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well written and the problems/techniques are well illustrated.
Quality: The technical designs are well motivated and sound. The experiments are strong and convincing.
Novelty: The formulated square-cube law seems to be novel and so are the design considerations. Distinctions to existing related works are clearly discussed.
Reproducibility: This might be a problem because the experimental environments are relatively complicated and implementations are not provided (would imagine the results to be hard to exactly reproduce even with implementations). But this is also understandable.


summary_of_the_review: Overall an interesting effort and read for the community, but direct applications are still concerning.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of training a huge deep neural network on a cluster with low interconnect bandwidth, unstable network, and preemptible nodes. The paper first shows that the computation-to-communication ratio goes up with larger models, indicating that using low-bandwidth clusters to train a huge model-parallelized model makes practical sense. Then, the paper proposes a new training algorithm named SWARM parallelism. It uses multiple devices to serve a single pipeline stage and schedules with a randomized algorithm for load-balancing and fault-tolerance. The system in addition move devices across pipeline stages to minimize the time spent on the slowest pipeline stage.
strength_and_weaknesses: **Strengths**

1. In general, I think the authors are working on an important and interesting new direction of distributed training. The randomized scheduling and load-balancing algorithm is novel and is insightful for the broader research community.
2. I find most experiments of the paper can show the effectiveness of the proposed SWARM parallelism method.

**Weakness**

1. The square-cube law shows that the total size of tensors to be communicated is grows at slower quadratic rate compare to the total amount of compute (flops), which grows at a faster cubic rate. However, given a fixed per-GPU memory constraint, larger models require more GPUs, which will in turn makes the total communication cost higher (because the communication cost is related to the total number of devices involved). How does the square-cube law apply to this case?
2. It would be great to include an experiment that applies SWARM parallelism to a plain Transformer model without any layer sharing/compression to purely evaluate the performance of SWARM parallelism.
3. For pipeline parallelism, how do you schedule the forward and backward passes? Do you use the GPipe schedule or the 1F1B schedule? Please elaborate.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and easy-to-understand. This is not the first time I’m reviewing this paper, but I found the current version much clearer than the old version. The paper should be able to be reproduced with the open-sourced code by any researchers and engineers with access to a typical GPU hardware. I find this paper novel, and the detailed comment can be found in the discussion in the strength above.

In terms of paper writing, I would suggest move more related works into the appendix and move more experimental results to the main paper.
summary_of_the_review: I found the paper novel and well-written. I vote for accepting the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies efficient large-scale neural network training. The authors observe that the larger the hidden dimension of models the less communication intensive. Based on such observation, the authors propose a heuristic-driven large model training method, namely SWARM.
strength_and_weaknesses: Pros:

1. The problem is well-motivated. Indeed, training large models is important but could also be computationally expensive and time-consuming. Existing model parallelism methods propose to assign each computation device with several layers and forward/backward sequentially (the input of one device is the output of another device).
However, these methods are limited by the device memory issue and could be less efficient when the subset of layers assigned to each device requires different computation costs. Besides, each device could disconnect abruptly due to failure or preemption, which makes model parallelism more challenging.

2. The proposed method is intuitive and easy to implement. I would always encourage more on a simpler method than some complicated algorithms with massive engineering efforts and sensitivity to hyper-parameters.

3. Ablation study on training efficiency in experiment sections and appendix are quite sufficient. 

Cons:

1. Although this paper is more on the empirical side of distributed learning, I am still looking forward to seeing some discussion/results on its convergence properties on the theoretical side. For example, the authors could check existing async distributed learning papers and see how they show such kind of convergence properties.

2. The main argument that the authors made on "increasing the hidden dimension will reduce the computation load per device per unit of time ..." in Section 3.1 is kind of ambiguous. According to my understanding, increasing the hidden dimension should increase both the computation cost and communication cost, we cannot conclude that ``reduce the computation load'' given the communication cost $O(n^2)$ grows slower than the computation cost $O(n^3)$.

3. The experiment sections mainly focused on time efficiency. However, I am more interested in the trade-off between efficiency and model performance. Using async randomized temporary routing for large model training is expected to degenerate the model performance and I am interested in seeing how much it will affect the model performance.

4. The presentation could be improved. For example, an experiment setup introduced could be given at the beginning of Section 4. Besides, Section 2 could be better organized to give readers a better overview of the pros and cons of the existing method. Moreover, the authors could use one paragraph to give a high-level introduction to their method in Section 1 instead of just saying "replacing traditional pipelines with randomized temporary routing between swarms of peers".
clarity,_quality,_novelty_and_reproducibility: The presentation could be improved. 

The method proposed in this paper sounds novel to me, however, I am not an expert in the applied side of async distributed learning.

Code to reproduce the experiments is provided. The ablation study sounds solid to me, however, I would also expect to see a trade-off between accuracy and efficiency. 
summary_of_the_review: This paper studies an important problem. The method is well-motivated and easy to implement/reproduce according to the descriptions in the paper. The ablation studies are efficient but I would also expect to see a trade-off between accuracy and efficiency. Meanwhile, for some theoretical results on convergence (similar to most distributed learning, e.g., FedAvg), the presentation could be improved and some arguments are ambiguous. Please refer to "Strength And Weaknesses" for details.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes SWARM, a system for training large language models (LLM) on alternative environments to HPC clusters that comprise of consumer-grade, preemptible, unreliable, and geographically distributed devices. SWARM employs pipeline parallelism to partition the model layers into pipeline stages on the distributed devices and coordinates the exchange of activations. SWARM uses randomized pipelines and rebalancing to handle node failures. Also, SWARM relies on the compute-bound nature of LLMs and low communication of pipeline parallelism for performance. 
strength_and_weaknesses: The paper addresses an important problem of making AI innovations more affordable and thus more broadly accessible. The approach of pooling together many consumer-grade, heterogeneous, and preemptible resources is a promising one. Also, using pipeline parallelism to support LLM training is a reasonable solution. 

The main issue is that the paper does not provide much new insight beyond prior work. Moreover, I feel the evaluation did not effectively address important functionality and performance questions: 
 1) I suspect that some form of redundancy is required to enable recovery. I did not notice any discussion of such redundancy. 
 2) How does recovery mechanism handle the loss of intermediate training state such as accumulated gradients, activations (including checkpointing state), model checkpoints, data loading iterator, etc.
3)  I observe some issues in the performance results: 
     (1) It is unclear whether Figure 3 is measuring forward or/and backward pass of the layer(s)
     (2) Reporting FLOP/GPU in Figure 3 would be more helpful to understand the baseline strength and claims of peak efficiency 
     (3) It is unclear whether 8-bit quantization is appropriate for GPT3 training, since it is part of performance claims
     (4) The performance comparison in 4.2 is quite confusing because the size of GPT3-style model is unspecified. Also, GPipe needed 
           ZeRO-Offload to fit the model even though it uses pipeline parallelism. Is SWARM more memory efficient than GPipe? It is also 
          unclear whether the V100 is 16GB or 32GB because as that is important to understand why ZeRO-Offload cannot fit 4 GPT3 layers.  


clarity,_quality,_novelty_and_reproducibility: I feel the work could be improved on clarity, novelty, and reproducibility. As expressed earlier, more evaluation details could be provided to help understand the results.  
summary_of_the_review: The central claims that (1) model scaling results in computation scaling faster than communication, and (2) pipeline parallelism communication volume is the lowest of the training parallelism techniques are both previously known. This undercuts the main contribution of the paper. Also, I feel that the functionality and performance questions above concerning whether SWARM could replace HPC clusters for LLM training were unanswered. Although, I think the paper is going in the right direction on an important problem, I feel that it is incomplete.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the efficient training of large language models on heterogeneous and preemptive instances.
It proposes SWARM parallelism and finds that training can be made communication-efficient with SWAM parallelism.
Evaluation of a 13B model shows that the system achieves good training throughput and convergence.

strength_and_weaknesses: Strength:
- This paper studies an important and unsolved problem: training LLM outside HPC clusters with heterogeneous and preemptive instances.
- The paper proposes a novel and interesting solution: SWARM parallelism, which is decentralized and highly adaptive.
- Solid experiments with throughput and convergence analysis

Weaknesses:
- This paper uses too many approximate/lossy/asynchronous methods, which can harm the accuracy.
- Stochastic wiring and adaptive rebalancing are local greedy adjustment approaches. However, I think if we know the computing speed and memory capacity of all devices, an optimal placement/routing can be computed. [1][2]
- Section 4.2 's comparison is a little bit unfair. In a typical HPC cluster, there are typically multiple GPUs on a single node connected by high-speed NVLink. Under this more realistic setting, ZeRO Offload will perform better. If you look at the results in [3], they can achieve linear scaling on HPC clusters. The authors should compare against SOTA performance on the HPC cluster when they talk about "ideal condition".

[1] Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning, OSDI 2022  
[2] Piper: Multidimensional Planner for DNN Parallelization, NeurIPS 2021  
[3] Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

Minor:
Missing a discussion of a recent and highly related paper: Decentralized Training of Foundation Models in Heterogeneous Environments (https://arxiv.org/abs/2206.01288)

clarity,_quality,_novelty_and_reproducibility: The paper is well written with enough background information.
The proposed methods are novel.
I believe most experiments can be reproduced. The code is available.

summary_of_the_review: This paper proposes an interesting solution to an important problem. I am leaning toward acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Reviewers are in agreement about the importance of the problem studied in the paper, training large neural network models with networks of unreliable, poorly-connected devices. Even after in-person discussions, there remained division about the originality and significance of the square cube law (sec 3.1). However, there was agreement even amongst one of the most positive reviewers that the experimental evaluations are a weak point of the paper, and the author responses did not convince the reviewers otherwise, particularly around the responses to reviewer questions about the 1F1B schedule and around assumptions about the assumed impact of async updates on accuracy, and the tradeoffs achievable.
justification_for_why_not_higher_score: Some concerns about the significance and originality of the "square cube" contribution and remaining concerns about experiments.
Decision: Reject