Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a deep learning framework called DeePa that supports multiple dimensions of parallelism in computation to accelerate training of convolutional neural networks.  Whereas the majority of work on parallel or distributed deep learning partitions training over bootstrap samples of training data (called image parallelism in the paper), DeePa is able to additionally partition the operations over image height, width and channel.  This gives more options to parallelize different parts of the neural network.  For example, the best DeePa configurations studied in the paper for AlexNet, VGG-16, and Inception-v3 typically use image parallelism for the initial layers, reduce GPU utilization for the deeper layers to reduce data transfer overhead, and use model parallelism on a smaller number of GPUs for fully connected layers.  The net is that DeePa allows such configurations to be created that provide an increase in training throughput and lower data transfer in practice for training these networks.  These configurations for parellism are not easily programmed in other frameworks like TensorFlow and PyTorch.

The paper can potentially be improved in a few ways.  One is to explore more demanding training workloads that require larger-scale distribution and parallelism.  The ImageNet 22-K would be a good example and would really highlight the benefits of the DeePa in practice.  Beyond that, more complex workloads like 3D CNNs for video modeling would also provide a strong motivation for having multiple dimensions of the data for partitioning operations.
review_title: Review of Exploring the Hidden Dimension in Accelerating Convolutional Neural Networks

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper develops a framework for parallelization of convolutional neural nets. In the framework, parallelism on different dimensions are explored for convolutional layers to accelerate the computation. An algorithm is developed to find the best global configuration.

The presentation needs to be more organized, it is not very easy to follow.

1. Computation throughput is not defined.

2. Although the author mentions DeePa with Tensorflow or Pytorch several times, I think it is not proper to make this comparison. The main idea of this paper is to optimize the parallelization scheme of CNN, which is independent of the framework used. It is more useful if the configuration searching can be developed on tensorflow / pytorch.

3. The per layer comparison is not very informative for practice because the data transfer costs of convolution layers could be completely hidden in data parallelization. In data parallelism, the GPU devices are often fully occupied during the forward pass and backward pass. Gaps are only in between forward and backward, and between iterations. Model parallelism would add gaps everywhere in each layer. This could be more detrimental when the communication is over ethernet. To be more convincing, it is better to show the profile graph of each run to show which gaps are eliminated, rather than just numbers.

4. The batch size is also a crucial factor, difference batch size would favor different methods. More comparisons are necessary.
review_title: Needs more data to support

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes an approach that offers speedup on common convolutional neural networks. It presents the approach well and shows results comparing with other popular frameworks used in the field.

Originality
- The automation of parallelism across the different dimensions in each of the layers appears somewhat new. Although parallelism across each of the individual dimensions has been explored (batch parallel is most common and best supported, height and width is discussed at least in the DistBelief paper), automatically exploring this to find the most efficient approach is new. The splitting across channels seems not to have been covered in a paper before.

Significance
- Paper shows a significant speedup over existing approaches on a single machine (16 GPUs). It is unclear how well this would translate across machines or to more devices, and also on newer devices - the experiments were all done on 16 K80s (3 generations old GPUs). While the approach is interesting, its impact also depends on the speedup on the common hardware used today.

Pros:
- Providing better parallelism opportunities for convolutional neural networks
- Simple approach to finding optimal global configurations that seems to work well
- Positive results with significant speedups across 3 different networks

Cons:
- Unclear if speedups hold on newer devices
- Useful to see how this scales across more than 1 machine
- Claim on overlapping computation with data transfer seems incorrect. I am pretty sure TensorFlow and possibly PyTorch supports this.

Questions:
- How long does finding the optimal global configuration take for each model?

review_title: Potential for accelerating CNNs

Meta-review: While this paper has some very interesting ideas the majority view of the reviewers and their aggregate numerical ratings are just too low to warrant acceptance.
Decision: Reject