Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: One of the challenges with scaling data parallel training of neural networks is that the natural dependencies between the forward propagation, backward propagation, and gradient updates limits the benefits of parallel execution.  A common solution in the state of the art is to use stale gradients to allow the gradient updates to overlap with the data motion between parallel ranks.  However, the use of stale gradients typically leads to a lower quality of solution in the trained network.  This paper presents several methods for addressing the staleness of gradients, using three techniques: partial staleness for “earlier” layers of the network, delay gradient compensation, and weight prediction.  Using these techniques they demonstrate that they can achieve runtime performance similar to the SOTA using stale gradients, but with a quality that nearly matches a standard SGD training approach.  The paper also provides both a convergence analysis and experimental analysis with both visual and language models demonstrating the impact of their proposed techniques.
strengths_and_weaknesses: This paper leverages multiple advancements in the community and integrates them into a common composition and framework, demonstrating that in aggregation they can provide an approach that is performant in terms of both speed and quality of the trained network.  Overall, they clearly indicate where the derived ideas come from, how they were used previously and how they are being integrated into a combined algorithm.  The explanation of the algorithm and visualization via diagram could use some improvement.  Please see the next section for specific comments, however, the reader is left with a general understanding of how the algorithm execute at runtime, but some better figures could significantly improve this.

The evaluation of the algorithm is mixed.  Overall it is good that the authors include both visual and language models, however, using VGG16 on CIFAR-10 for the ablative study is not nearly as interesting as if they had used a more modern and interesting model (e.g. ResNet or GPT-2).  Additionally, the later part of the paper is quite rushed with minimal explanation and analysis of the results.  For example Figure 5 has no analysis in the caption and only a bit in the text.  On a similar note, Figures 3-5 are extremely small and clearly not really meant for a reader to actually read.

One key challenge for this work is that none of the proposed optimization schemes performed the best on all benchmarks, and thus is is not clear a priori as to which technique (Opt1 - 3) should be used when.  This can pose a significant challenge to future application if a researcher would have to conduct an exhaustive test of all optimization methods to determine which may produce the best results for a new application.

questions: Figure 1 is introduced early in the text, prior to any significant discussion of partial stale gradient updates, yet it is illustrating partially stale gradients.  The figure, text, and caption should be refined to indicate that it is not showing the state of the art, but actually part of the proposed method.

The implementation of Algorithm 1 is strange.  Specifically, line 6 is weird for the base case.  For it to say “same as t > 1” should be clarified that there is really only a conditional for the PipeSGD case.

Figure 2 doesn’t really convey what the authors probably intended for explaining the algorithm and does not contribute too much as is.  For subfigure 2a, the fact that the parts of the staleness-aware system are all in the lower box but with no real relationships between them and the runtime parts of the system is not helpful to the reader.  Furthermore for 2b, it would be better to represent the communication pattern with some sort of parallel timeline.  Again, with the cyclic dependencies of iterative training, the interplay of the communication isn’t clearly conveyed by 2b.

In the equation below line 111 it is not clear what u is as it doesn’t appear in the table.

On line 119 a figure that clearly shows how partial staleness / gradient updates works would improve the discussion.

Replace Figure 5 analysis with a more challenging example for the ablation study, i.e. ResNet on ImageNet or one of the transformers, or both.
limitations: The biggest issue is that of the proposed techniques, on the use cases shown, one combination does not always win.  As a result it is not clear if these approaches were to be applied to a new problem area, which set of the staleness compensation techniques should be used.  Without some discussion of this, it is not clear how to leverage this work going forward without actually performing something of this analysis.  Furthermore, the section on sensitivity analysis does not address this.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents SAPipe, a system to support efficient data parallelism, where communication is effectively hidden within computation with adaptive staleness and corresponding compensations. Both theoretical analysis and empirical study are conducted to verify the effectiveness of the proposed solution.  
strengths_and_weaknesses: Strengths:
- The idea of improving PipeSGD with adaptive partial staleness is simple but effective.

- Theoretical analysis is provided to justify the design of the algorithm. 

- The experiment section is solid, and the performance boost is significant. 

Weaknesses:
- Some writing and illustration can be further polished. For example, Figure 1 is a little confusing; for the default pipeline part, it seems that v3 begins before the end of b3 visually, this sees inaccurate without mentioning any potential other optimization, e.g., communicating in a thinner granularity. 

- Some reasonable baseline approaches are missing. For example, PyTorch-DDP should be included because it is a very popular data-parallel implementation and provides efficient system optimizations such as bucketing and communication overlapping.  

questions: N.A. 
limitations: Perhaps some discussion about the scope of models should be considered; for example, can this approach be used for graph neural network training, where the layers in the model are not linearly stacked?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Synchronous data parallelism is widely adopted, but can suffer from poor scaling due to excessive communication overheads. This paper proposes using stale weight updates (weight gradients computed using the not latest weights) to better overlap computation and communication. It discusses various mitigation techniques to get bounded-stale data parallelism to work well in practice.
strengths_and_weaknesses: #### Strengths
- The paper offers an alternative to synchronous data parallelism in situations where the network is not fast enough to hide the cost of communication.
- The paper is well written and clear.
- Distributed training performance is an important problem in Machine Learning Systems, and this paper proposes one more possible solution to the problem. 

#### Weaknesses
- Evaluation is not entirely convincing: for example, Figure 4 shows accuracy vs. time, but I would have also liked to see accuracy vs. iteration to see the impact on convergence speed.
- Baselines seem if-fy: why does ResNet-50 scale worse than GPT-2? I expect the convolutional layers in a ResNet-50 model to be much more amenable to data-parallel-style communication compared to a GPT-2 model with a lot of linear layers in the attention layers. VGG-16 doesn't scale well exactly for these reasons.

questions: - Are the ideas of delay compensation and weight prediction new? Don't these papers propose similar ideas for pipeline parallelism without flushes (which shouldn't make a difference): PipeMare [https://proceedings.mlsys.org/paper/2021/file/6c8349cc7260ae62e3b1396831a8398f-Paper.pdf] and Kosson et al. [https://arxiv.org/pdf/2003.11666.pdf].
- Why does the optimization problem at the top of page 4 have a runtime complexity of O(m^2)?
- What batch size was used? Communication overhead is smaller with larger batch sizes.
- What is the effect of hardware (compute accelerator and network) on performance?
- What is the effect of model on performance?
- How much worse convergence rate (accuracy vs. number of iterations) does SAPipe have compared to synchronous DP (BytePS or Horovod)?
- How much do the proposed mitigation techniques (delay compensation and weight prediction) help with convergence? What if I don't use any mitigation techniques? How badly does this do?
limitations: The authors did not discuss limitations of their work. Some discussion of situations where they expect SAPipe to not be a useful solution (e.g., particular types of models, hardware deployments, optimizers, or other situations where their theoretical analysis breaks down).
review_title: null

Meta-review: metareview: This paper proposes a new algorithm to speed up data-parallel distributed training, focused on mitigating staleness-induced issues that arise when limiting communication between nodes.

All reviewers and myself agree this is a worthwhile contribution, which is backed by both convincing empirical and theoretical results. I consider that the potential novelty concerns that were raised in initial reviews have been addressed by the authors. The main remaining concerns are related to the limitations of the proposed method, that comes with some trade-offs and may not apply to all situations. I believe the authors have adequately answered these concerns by being upfront about these limitations during the discussion period, and I encourage them to make sure this is also clear in the final version of the paper. In spite of these limitations, I believe the novelty and significance of this work meet the bar for acceptance at NeurIPS, since speeding up distributed computations is a very relevant and challenging problem in modern deep learning.
confidence: Less certain
Decision: Accept