Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes, WPipe, a technique for reducing the memory overheads of weight versioning and improving freshness of weight updates in pipeline parallelism through a novel scheduling of the the pipeline stages. It presents evaluation results which show that WPipe does not harm model quality, and that the memory efficiency can improve the throughput compared to prior pipeline approaches. 
main_review: The paper has following strengths. 
1. It tackles an important problem of how to fit rapidly growing model sizes into slower growing device memory. Pipeline parallelism has been a useful approach for this problem, and this paper provides some improvements in that space. 
2. It proposes an interesting approach to address key issues surrounding memory efficiency and weight update sematics in existing pipeline parallelism approach

The weakness include:
1. The overall appears narrow and incremental in the sense that WPipe is fixing a memory efficiency bug of PipeDream-2BW relative to PipeDream-flush. In other words, WPipe is trying to deliver the best-of-both-worlds for these two existing approaches. As a result, it is not obvious how the results applicable beyond the PipeDream* line of pipeline parallelism.

2. The throughput results (4.2) are difficult to understand because they are based on unfamiliar models (e.g., Bert768 and ResNetXt500) which are not previously defined and different from the familiar models (e.g., Bert_base, ResNetXt50) used for the convergence quality results (4.1). It seems that the models in 4.2 were constructed to show the best case throughput benefits of WPipe, but don't appear to be real world models. 

3.  I feel the writing could be greatly improved as I found multiple portions difficult to follow. Although I am not hands-on with pipeline parallelism, I feel I have a pretty solid understanding at the high-level, but yet this draft was less accessible than expected. Below are some specific examples of confusing portions and questions for the authors:

  a) The idea (stated in 3.2) of splitting the model partitions to double the number of partitions and obtain two groups (G0 & G1) is quite confusing, especially since memory footprint analysis in 3.3 assumes the model is evenly divided so that G0 and G1 are half of the model.  It would be helpful to illustrate the before/after of model partitions with WPipe relative to PipeDream-2BW, independent of mini- or micro-batches. This should be possible since model partitions are simply mappings of model parameters to devices. 

  b)  It seems the cells in Figures 1 & 2 are mini-batches and micro-batches respectively. It would be helpful to clarify this detail since these two Figures are often discussed together. Also, it seems Figure 2(c) is missing the update cell. Is this a typo?

  c) Much of the material in 3.4 is prior work and so could be written more briefly and systematically. 
summary_of_the_review: WPipe is basically a memory efficiency optimization of PipeDream-2BW, but the paper does not report throughput benefits on real world models. So it is not clear how practical the optimizations are.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a novel pipeline parallelism technique for training large DNN models named as WPipe. The method aims to improve upon existing pipeline methods such as Pipedream-2BW by having a better memory efficiency and more fresh weight updates. Key idea is to divide the model partitions into two parts groups and make the forward pass of the second part to reduce the delayed gradients. The method is tested empirically and performs better than proposed current state of the art baselines.
main_review: There is some novelty in the new pipeline scheme, but I feel it is incremental. On the practical side, the method has the potential to be more impactful to large scale DNN transformer models such as BERT, GPT etc. The empirical results are very encouraging to me. The presentation and clarity in the paper could be improved - the authors should re-use terminology used earlier in the pipeline parallelism literature to make the presentation consistent and less confusing.

Pros:
- Simple idea that seems to work very well empirically. Can be readily adapted and used in existing training pipelines.
- Memory Footprint Analysis is provided for the method
- Empirical results are very encouraging

Cons:
- Novelty factor is bit limited
- Presentation could be improved
- Reproducibility can be improved (it would be benefit readers if the authors could provide hyper-parameters and experimental setup in a Table summarized for all of their runs)

Comments:
- In the abstract, authors mention - "It moves the forward pass of the next period ..." -> period is not defined at this point and is too vague a term to use without defining what it exactly refers to. 

- "model partitions" -> do the authors mean pipeline stages? Strongly recommend using standard terminologies from model parallelism literature and explain things more consistently. I feel the paper is written a little hastily.

- ".. raw pipeline parallelism ... suffers from inconsistency issue" -> what does inconsistency mean here? It still is a bit vague term to me.

- How does WPipe react to varying the varying number of pipeline stages? What is the maximum number of pipeline stages the authors tested the method with? It would be useful to understand this scalability.




summary_of_the_review: The novelty factor in the proposed work is incremental and I also feel the paper can benefit from some more polishing of the draft due to the presentation issues I outlined.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a novel pipeline training strategy WPipe. WPipe divides model partitions into two groups and updates each group alternatively, which eliminates half of the delayed gradients and memory redundancy compared to Pipedream-2BW. 

The experimental results show that WPipe can achieve higher throughputs and reduce memory footprint with similar final model accuracy.
main_review: Strengths:
1. The proposed pipeline strategy is novel and effective.
2. Clean analysis of the proposed strategy in terms of memory footprint, communication overhead, bubble time, and update staleness.

Weakness:
There is no strong weakness of this paper. For other minor clarification and writing issues, see the comments below.

Detailed comments:
1. The term "model parallelism" should be used more precisely. There are three types of model parallelism: pipeline parallelism, operator placement (which is referred by you as "conventional model parallelism" or "naive model parallelism"), and operator partition (e.g., the strategy in Megatron-LM v1). In your paper, it seems you totally ignore the third type operator partition. I suggest renaming all "naive model parallelism", "conventional model parallelism" in your paper to "operator placement", and adding some description of the operator partition.
2. The memory footprint analysis with activation recomputation seems wrong. I think all numbers in the last column of Table. 1 are wrong. Take GPipe for example, activation recomputation only reduces the memory footprint of a micro-batch on a stage by a constant factor. It cannot reduce the footprint by a factor of "M", because anyway, each stage has to keep the input activation of the "M" in-flight micro-batches. The same analysis applies to other pipeline schemes.
3. What will happen if we split the weights into more groups instead of just 2 groups?
4. Please provide the exact settings of the models and the cluster. For example, provide the hidden_size, seq_len, num_heads of all BERT models. Provide the network bandwidth of your private cluster.
5. How do you split the models into pipeline stages? For BERT, you can assign an equal number of transformer layers to each stage. How do you split the ResNext?

summary_of_the_review: This paper introduces a novel pipeline training strategy and comprehensively evaluates its performance.
Although the writing can be improved, I recommend accepting this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses the problem of very large scale deep neural network training through model parallelism. A new model parallel pipeline called WPipe is proposed, that builds on previously existing schemes while targeting (1) lower memory redundancy and (2) fresher weight updates. Experimental evaluation shows that WPipe can achieve significant acceleration (1.4x) and lower memory footprint (36% smaller) when compared to state-of-the-art PipeDream-2BW.
main_review: Training very large DNNs models is a hot topic of practical interest. Data parallelism cannot help when the models are so big that they cannot fit in a single accelerator. In this case, model-parallel techniques can be applied. There are several existing pipelines, such as GPipe, PipeDream, PipeDream-2BW that improve various aspects of the naive execution. However, they typically suffer from (1) staleness of the weight updates, caused by inconsistencies in the versions of the weights used in the forward and backward pass, and (2) large number of in-flight activations, caused by the forward passes waiting for their backward passes in the pipeline. 

The paper proposes a new pipeline called WPipe, that builds on the existing ones, but addresses the previously mentioned two issues. A careful construction is proposed, based on two groups of weights. A key feature of the new scheme is the placement of the forward pass of the next step for one of the groups (G0) at the front of the backward pass of G0 of the current step. This elegant construction, allows group G1 to only maintain one weight version and G0 to maintain only two weight versions, thus significantly reducing the memory overhead. 

A theoretical analysis of the memory footprint is provided. WPipe can also benefit from activation recomputation techniques, that can further reduce the in-flight activations at the expense of more computation. Furthermore, the communication overhead is analyzed, and some mitigation strategies are suggested: data parallelism, overlap of computation and communication, heterogeneous network communication. 

A significant part of the paper is dedicated to experimental analysis, including text classification (BERT models) and image classification (ResNeXt models). The experiments show that compared to state-of-the-art PipeDream-2BW, WPipe is more memory efficient (by36%) and has higher throughput (1.4x). Although the final accuracy is similar for the two schemes, WPipe is shown to have weight update semantics closer to data parallelism.
summary_of_the_review: The paper proposes a new model-parallel pipeline called WPipe, for DNN training. A careful construction is proposed, based on two groups of weights. A key feature of the new scheme is the placement of the forward pass of the next step for one of the groups (G0) at the front of the backward pass of G0 of the current step. This elegant construction, allows group G1 to only maintain one weight version and G0 to maintain only two weight versions, thus significantly reducing the memory overhead. Theoretical analysis and an extensive experimental evaluation on text and image data show that WPipe outperforms state-of-the-art schemes, being 1.4 times faster while using 36% less memory.
review_title: null

Meta-review: The paper proposes a new pipeline-parallel training method called WPipe. WPipe works (on a very high level) by replacing the two-buffer structure of PipeDream-2BW with a two-partition-group structure, allowing resources to be shared in a similar way to PipeDream-2BW but with less memory use and less delays in weight update propagation across stages. The 1.4x speedup it achieves over PipeDream-2BW is impressive.

In discussion, the reviewers agreed that the problem WPipe tries to tackle is important and that the approach is novel and interesting. But there was significant disagreement among the reviewers as to score. A reviewer expressed concern about the work being incremental and difficult to follow. And while these were valid concerns, and the authors should take note of them when revising their paper, I do not think they should present a bar to publication, both based on my own read of the work and also in light of the fact that other reviewers with higher confidence scores did not find novelty to be a disqualifying concern. As a result, I plan to follow the majority reviewer opinion and recommend acceptance here.
Decision: Accept (Poster)