Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new pipelined training approach to speedup the training for neural networks. The approach separates forward and backpropagation processes into multiple stages, cache the activation and gradients between stages, processes stages simultaneously, and then uses the stored activations to compute gradients for updating the weights. The approach leads to stale weights and gradients. The authors studied the relation between weight staleness and show that the quality degradation mainly correlates with the percentage of the weights being stale in the pipeline. The quality degradation can also be remedied by turning off the pipelining at the later training steps while overall training speed is still faster than without pipelined training.
Since this work takes the approach of allowing stale weight updates, the author should also compare with existing distributed training approaches that use asynchronous updates, with or without model parallelism, for example, Dean et al., 2012. Without the comparison it’s not clear how much improvement this approach provides compared to existing work that perform stale updates.

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper proposed a new pipelined training strategy to fully utilize the memory and computational power to speed up the training process. In order to overcome the generalization degradation of the proposed method, the authors further introduced the so-called hybrid method to combine their proposed pipelined method and normal training. 

The pipelined method is interesting. For the pipelined process itself, it is similar to model parallelization. For the method proposed by the paper,  it is like the async-SGD method. The paper merged these two ideas together but did not solve the problem from async-SGD, i.e. with a large number of processes, the generalization performance degrades (in the paper, it is so-called "stages"). Even with the hybrid method, the accuracy still drops. 

Also, the sentence, "We demonstrate the implementation and performance of our pipelined backpropagation in PyTorch on 2 GPUs using ResNet, achieving speedups of up to 1.8X over a 1-GPU baseline, with a small drop in inference accuracy.", is confusing. If I use data parallelization, the gain should be also around 2. 

The ResNet on Cifar-10 results are not convincing. The normal accuracy of ResNet20 on Cifar-10 is around 92 but the paper reported 91.1%.

Based on this, I think the paper has some room for improvement.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper investigates the impact of stale weights on the statistical efficiency and performance in a pipelined backpropagation scheme that maximizes accelerator utilization while keeping the memory overhead modest. The paper proposes to combine pipelined and non-pipelined training in a hybrid scheme to address the issue of significant drop in accuracy when pipelining is deeper in the network. The performance of the proposed pipelined backpropagation is demonstrated on 2 GPUs using ResNet with speedups of up to 1.8X over a 1-GPU baseline and a small drop in inference accuracy.

The paper is well written and easy to follow. The proposed idea is interesting and its effectiveness is well demonstrated with a promising speed and a small drop in accuracy. The proposed approach is compared to two existing works:  PipeDream [1] and GPipe [2]. Though promising results have been demonstrated, a drawback of the proposed method is that it introduces more memory overhead compared to GPipe. Although a detailed discussion is provided related to the memory consumption between the proposed method and PipeDream, no detailed discussion is provided with respect to GPipe. Further, no proper convergence analysis of the proposed approach is provided and is desired due to the likely divergence in the optimization. Minor comment: An interesting line of work is that of [3] which could be included in the discussion.

Overall, the proposed approach is interesting and is shown to achieve promising results. However, memory overhead is still an issue compared to existing method.

[1] Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel DNN training, 2018. URL http://arXiv:1806.03377.
[2] Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism, 2018. URL http://arXiv:1811.06965.
[3] Guanhua Wang, Shivaram Venkataraman, Amar Phanishayee, Jorgen Thelin, Nikhil Devanur, Ion Stoica: Blink: Fast and Generic Collectives for Distributed ML. arXiv:1910.04940, 2019.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In the paper, the authors propose a pipelined backpropagation algorithm faster than the traditional backpropagation algorithm. The proposed method allows computing gradients using stale weights such that computations in different layers can be executed in parallel. They also conduct experiments to evaluate the effect of staleness and show that the proposed method is faster than compared methods. I have the following concerns:

1) There are several important works on model-parallelism and convergence guarantee of pipeline-based methods missing in this paper, for example [1][2]. 
2) Does the proposed method store immediate activations or recompute the activations in the backward pass?
3) In the experiments, the accuracy values are too low for me. For example, resnet110 on cifar10 is 91.99% only, it should be around 93%, an example online https://github.com/akamaster/pytorch_resnet_cifar10. 
4) In the experiments, more comparisons with methods in [1] or [2] should be conducted given they are all parallelizing the backpropagation algorithm and achieve speedup in the training.
5) Last but not least, convergence analysis of the proposed method should be provided given that asynchrony may lead to divergence in the optimization.  


[1] Huo, Zhouyuan, et al. "Decoupled parallel backpropagation with convergence guarantee." arXiv preprint arXiv:1804.10574 (2018).
[2] Huo, Zhouyuan, Bin Gu, and Heng Huang. "Training neural networks using features replay." Advances in Neural Information Processing Systems. 2018.

review_title: Official Blind Review #1

Meta-review: The paper proposed a new pipelined training approach to better utilize the memory and computation power to speed up deep convolutional neural network training. The authors experimentally justified that the proposed pipeline training, using stale weights without weights stacking or micro-batching, is simpler and does converge on a few networks. 

The main concern for this paper is the missing of convergence analysis of the proposed method as requested by the reviewers. The authors brought up the concern of the limited space in the paper, which can be addressed by putting convergence analysis into appendix. From a reader perspective, knowing the convergence property of the methods is much more important than knowing it works for a few networks on a particular dataset.    
Decision: Reject