Title: No title found

Abstract: No abstract found

:



Figures and Tables:

Peer review (iclr2020):
Summary: This paper proposes an approach for model parallelism without forward or backward locking (i.e., layers don’t have to wait to perform computation). They maintain forward and backward queues per block of layers, where each block of layers is on a different device. Each block pops activations from the previous block’s forward output queue, computes the output activations and pushes into its forward output queue. Then it pops the gradient queue from the next block, computes gradients of its parameters and the corresponding input, updates its parameters, and pushes the input gradient into the queue. This technique introduces gradient staleness. They evaluate ResNets on CIFAR-10, CIFAR-100, and ImageNet. Comments: * There are a number of prior works that explore the same idea of pipelining forward and backward calculations that are not cited and compared [1, 2, 3]. The novelty of this work is low given these works. * This paper is poorly written. It is overly notation heavy and does not give a clear description of how the method works. The diagrams only add to the confusion instead of making it easier to understand. It takes a long time to understand what ends up being a very simple method. * The baselines in this paper are weak. The ResNet-164 gets 92.8%, but the original ResNet paper [4] has a ResNet-101 that gets 93.6% accuracy. This seems like a nitpicking detail, but the the problem with asynchronous training is not that it can diverge, but that the performance is often lower. Without having a strong baseline, it’s hard to tell whether the stale gradients are affecting performance. The ImageNet baselines are similarly concerning. * No GPipe comparison in the experiments. Furthermore, the GPipe speedup comparison is unfair because these seem like two separate implementations of a training pipeline. It’s hard to tell whether this approach will have speedups over GPipe if the same codebase is used. In conclusion, the lack of comparison with prior work, poor writing, and weak experimental results leads to a suggestion of rejection. [1] “Performance Analysis of a Pipelined Backpropagation Parallel Algorithm”. Alain Petrowski, Gerard Dreyfus, and Claude Girault [2] “AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks”. Alexander L. Gaunt, Matthew A. Johnson, Maik Riechert, Daniel Tarlow, Ryota Tomioka, Dimitrios Vytiniotis, Sam Webster [3] “PipeDream: Fast and Efficient Pipeline Parallel DNN Training”. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, Phil Gibbons [4] “Deep Residual Learning for Image Recognition”. Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun