Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper analyzes the ring-based AllReduce approach for multi-GPU data parallel training of deep net.
Comments
1) The name linear pipeline is somewhat confusing to the readers, as the technique is usually referred as ring based approach in Allreduce literature. The author should use the standard name to make the connection easier. 
2) The cost analysis of ring-based Allreduce is already provided in the existing literature. This paper applied the analysis to the case of multi-GPU deep net training, and concluded that the scaling is invariant of number of GPUs.
3) The ring-based allreduce approach is already supported by NVidia’s NCCL library, although the authors claim that their implementation comes earlier than the NCCL implementation.
4) The overlap of communication of computation is an already applied technique in systems such as TensorFlow and MXNet. The schedule proposed by the authors exploits the overlap partially, doing backprop of t-1 while doing reduce.  Note that the dependency pattern can be further exploited; with the forward of layer t depend on update of parameter of layer t in last iteration. This can be done by a dependency scheduler.	
5) Since this paper is about analysis of Allreduce, it would be nice to include detailed analysis of tree-shape reduction, ring-based approach and all-to-all approach. The discussion of all-to-all approach is missing in the current paper. 
In summary, this is a paper discussed existing Allreduce techniques for data parallel multi-GPU training of deep net, with cost analysis based on existing results. While I personally find the claimed result not surprising as it follows from existing analysis of Allreduce, the analysis might help some other readers. I view this as a baseline paper. The analysis of Allreduce could also been improved (see comment 5).










review_title: review for Efficient Communications in Training Large Scale Neural Networks

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a linear pipeline All-reduce approach for parallel neural networks on multiple GPU. The paper provides both theoretical analysis and experiments. Overall, the results presented in the paper are interesting, but the writing can be improved. 

Comments:

- The authors compare their proposed approach with several alternative approaches and demonstrate strong performance of the proposed approaches. But it is unclear if the improvement is from the proposed approach or from the implementation.  

- The paper is not easy to follow and the writing can be improved in many place (aside from typos and missing references). Specifically, the authors should provide more intuitions of the proposed approach in the introduction and in Section 3. 

- The proposition and the analysis in Section 3.2 do not suggest the communication cost of linear pipeline is approximately 2x and log p faster than BE and MST, respectively, as claimed in many places in the paper. Instead, it suggests LP *cannot* be faster than these methods by 2x and log p  times. More specifically, Eq (2) shows T_broadcase_BE/ T_broadcase_LP < 2. This does not provide an upper-bound of T_broadcase_LP and it can be arbitrary worse when comparing with T_broadcase_BE from this inequality. Therefore, instead of showing T_broadcase_BE/ T_broadcase_LP < 2, the authors should state T_broadcase_BE/ T_broadcase_LP > 1 when n approaches infinity. 

- It would be interesting to emphasize more on the differences between designing parallel algorithms on CPU v.s. on GPU to motivate the paper. 

review_title: null

final_score: rating: null
review_content: The model used in the paper appears to be ring based allreduce strategy, which is typical implemented in the MPI collectives. see, e.g.  http://www.cs.fsu.edu/~xyuan/paper/09jpdc.pdf  http://www.mcs.anl.gov/~thakur/papers/ijhpca-coll.pdf

- The ring base strategy seems to be implemented in the NCCL library, have you compare against that
review_title: The communication is typical ring based allreduce strategy?

Meta-review: The authors propose improvements for the utilization of modern hardware when training using stochastic gradient. However, the reviewers bring up several issues with the paper, including major clarity issues as well as notational issues and some comments about the theory vs. practice.
Decision: Reject