Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes to reduce model size and evaluation time of deep CNN models on mobile devices by converting multiple layers into single layer and then retraining the converted model. The paper showed that the computation time can be reduced by 3x to 5x with only 0.4% accuracy loss on a specific model.

Reducing model sizes and speeding up model evaluation are important in many applications. I have several concerns:

1. There are many techniques that can reduce model sizes. For example, it has been shown by several groups that using the teacher-student approach, people can achieve the same and sometimes even better accuracy than the teacher (big model) using a much smaller model. However, this paper does not compare any one of them.
2. The technique proposed in this paper is limited in its applicability since it's designed specifically for the models discussed in the paper. 
3. Replacing several layers with single layer is a relatively standard procedure. For example, the mean variance normalization layer and batch normalization layer can all be absorbed without retraining or losing accuracy.

BTW, the DNN low-rank approximation technique was first proposed in speech recognition. e.g., 

Xue, J., Li, J. and Gong, Y., 2013, August. Restructuring of deep neural network acoustic models with singular value decomposition. In INTERSPEECH (pp. 2365-2369).

review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper looks at the idea of fusing multiple layers (typically a convolution and a LRN or pooling layer) into a single convolution via retraining of just that layer, and shows that simpler, faster models can be constructed that way at minimal loss in accuracy. This idea is fine. Several issues:
- The paper introduces the concept of a 'Deeprebirth layer', and for a while it seems like it's going to be some new architecture. Mid-way, we discover that 1) it's just a convolution 2) it's actually a different kind of convolution depending on whether one fuses serial or parallel pooling layers. I understand the desire to give a name to the technique, but in this case naming the layer itself, when it's actually multiple things, non of which are new architecturally, confuses the argument a lot.
- There are ways to perform this kind of operator fusion without retraining, and some deep learning framework such as Theano and the upcoming TensorFlow XLA implement them. It would have been nice to have a baseline that implements it, especially since most of the additional energy cost from non-fused operators comes from the extra intermediate memory writes that operator fusion removes.
- Batchnorm can be folded into convolution layers without retraining by scaling the weights. Were they folded into the baseline figures reported in Table 7?
- At the time of writing, the authors have not provided the details that would make this research reproducible, in particular how the depth of the fused layers relates to the depth of the original layers in each of the experiments.
- Retraining: how much time (epochs) does the retraining take? Did you consider using any form of distillation?
Interesting set of experiments. This paper needs a lot of improvements to be suitable for publication.
- Open-sourcing: having the implementation be open-source always enhances the usefulness of such paper. Not a requirement obviously.


review_title: Interesting analysis, flawed paper, no answer to reviewer questions

final_score: rating: null
review_content: I would like to submit final review by end of week.
Thanks.
review_title: Please respond to pre-review questions.

final_score: rating: null
review_content: It is not always clear in the write-up how the depth (number of feature maps) of the merged layers relate to the depth of the layers before merging. In the case of branch merging, you suggest that the final depth is the sum total of the merged layers' depth. In streamline merging, it's not clear what happens. Later in the write-up, you say: ' For layer inception 3a/3x3 reduce, we reduce the number of output feature maps from 96 to 48.' It would be helpful to exhaustively list the depths of the layers so that readers can exactly reproduce the experiments.
I also see no link to open-source code reproducing the experiments. Do you intend to provide it?

review_title: Depth of merged layers?

final_score: rating: null
review_content: Replacing pooling layers with strided convolutions doesn't seem like a new idea, yet I don't see mention or comparison to other work studying removing pooling layers?
review_title: Related work

final_score: rating: 4: Ok but not good enough - rejection
review_content: One of the main idea of this paper is to replace pooling layers with convolutions of stride 2 and retraining the model. Authors merge this into a new layer and brand it as a new type of layer. This is very misleading and adding noise to the field. And using strided convolutions rather than pooling is not actually novel (e.g. https://arxiv.org/abs/1605.02346).
While the speed-up obtained are good, the lack of novelty and the rebranding attempt make this paper not a good fit for ICLR.
review_title: Review

Meta-review: The proposed method doesn't have enough novelty to be accepted to ICLR.
Decision: Reject