Title: DeepRebirth: A General Approach for Accelerating Deep Neural Network Execution on Mobile Devices
Authors: Dawei Li
Xiaolong Wang
Deguang Kong
Mooi Choo Chuah
Venue: ICLR 2017

Abstract: Deploying deep neural networks on mobile devices is a challenging task due to computation complexity and memory intensity. Existing works solve this problem by reducing model size using weight approximation methods based on dimension reduction (i.e., SVD, Tucker decomposition and Quantization). However, the execution speed of these compressed models are still far below the real-time processing requirement of mobile services. To address this limitation, we propose a novel acceleration framework: DeepRebirth by exploring the deep learning model parameter sparsity through merging the parameter-free layers with their neighbor convolution layers to a single dense layer. The design of DeepRebirth is motivated by the key observation: some layers (i.e., normalization and pooling) in deep learning models actually consume a large portion of computational time even few learned parameters are involved, and acceleration of these layers has the potential to improve the processing speed significantly. Essentially, the functionality of several merged layers is replaced by the new dense layer – rebirth layer in DeepRebirth. In order to preserve the same functionality, the rebirth layer model parameters are re-trained to be functionality equivalent to the original several merged layers. The extensive experiments performed on ImageNet using several popular mobile devices demonstrate that DeepRebirth is not only providing huge speed-up in model deployment and significant memory saving but also maintaining the model accuracy, i.e., 3x-5x speed-up and energy saving on GoogLeNet with only 0.4% accuracy drop on top-5 categorization in ImageNet. Further, by combining with other model compression techniques, DeepRebirth offers an average of 65ms model forwarding time on each image using Samsung Galaxy S6 with only 2.4% accuracy drop. In addition, 2.5x run-time memory saving is achieved with rebirth layers.


Review discussion:
It is not always clear in the write-up how the depth (number of feature maps) of the merged layers relate to the depth of the layers before merging. In the case of branch merging, you suggest that the final depth is the sum total of the merged layers' depth. In streamline merging, it's not clear what happens.
Later in the write-up, you say: ' For layer inception 3a/3x3 reduce, we reduce the number of output feature maps from 96 to 48.' It would be helpful to exhaustively list the depths of the layers so that readers can exactly reproduce the experiments.
I also see no link to open-source code reproducing the experiments. Do you intend to provide it?
(1) It is not always clear in the write-up how the depth (number of feature maps) of the merged layers relate to the depth of the layers before merging. It would be helpful to exhaustively list the depths of the layers so that readers can exactly reproduce the experiments.
Ans: The depth of feature maps for a rebirth layer is depending on that of the layers before merging. We enforce this to ensure the shape of its top layer remain unchanged and maintain the knowledge (i.e, the weights of its top layer are preserved).  We have listed the details of the merging in our revision and we also plan to release the entire model architectures designed in our paper (in caffe’s prototxt format).
(2) I also see no link to open-source code reproducing the experiments. Do you intend to provide it?
Ans: We plan to release the code after final decision. Please refer to our paper for model details. We have already attached the structure details in the revision. Please have a check. This is our first step for open-source.
Replacing pooling layers with strided convolutions doesn't seem like a new idea, yet I don't see mention or comparison to other work studying removing pooling layers?
We only find one paper “STRIVING FOR SIMPLICITY: THE ALL CONVOLUTIONAL NET” that replaces pooling layer with strided convolution layer. The key differences between deep rebirth and this work are:
1.  We focus on accelerating the overall model execution speed instead of replacing a pooling layer with a convolution layer which may even slow down the execution.
2.  Our approach is a generic method for model acceleration, which is not limited in optimizing pooling layer but also can be applied in other types of non-tensor layers such as LRN, BatchNorm.
We have added the comparison in our revision (in related work section).
I would like to submit final review by end of week.
Thanks.
We are significantly improving the quality of paper.
We update our paper and rebuttals by emphasizing technical contribution,  illustrating technical details, eliminating  ambiguity and providing very strong experimental comparisons against state-of-the-art methods (including recent ICLR'16 and ICLR'17 submitted paper).
This paper looks at the idea of fusing multiple layers (typically a convolution and a LRN or pooling layer) into a single convolution via retraining of just that layer, and shows that simpler, faster models can be constructed that way at minimal loss in accuracy. This idea is fine. Several issues:
- The paper introduces the concept of a 'Deeprebirth layer', and for a while it seems like it's going to be some new architecture. Mid-way, we discover that 1) it's just a convolution 2) it's actually a different kind of convolution depending on whether one fuses serial or parallel pooling layers. I understand the desire to give a name to the technique, but in this case naming the layer itself, when it's actually multiple things, non of which are new architecturally, confuses the argument a lot.
- There are ways to perform this kind of operator fusion without retraining, and some deep learning framework such as Theano and the upcoming TensorFlow XLA implement them. It would have been nice to have a baseline that implements it, especially since most of the additional energy cost from non-fused operators comes from the extra intermediate memory writes that operator fusion removes.
- Batchnorm can be folded into convolution layers without retraining by scaling the weights. Were they folded into the baseline figures reported in Table 7?
- At the time of writing, the authors have not provided the details that would make this research reproducible, in particular how the depth of the fused layers relates to the depth of the original layers in each of the experiments.
- Retraining: how much time (epochs) does the retraining take? Did you consider using any form of distillation?
Interesting set of experiments. This paper needs a lot of improvements to be suitable for publication.
- Open-sourcing: having the implementation be open-source always enhances the usefulness of such paper. Not a requirement obviously.
(1) I understand the desire to give a name to the technique, but in this case naming the layer itself, when it's actually multiple things, non of which are new architecturally, confuses the argument a lot.
Ans: Well, we name our approach as “deep-rebirth” because in our proposed speed optimization pipeline, we have replaced multiple deep network layers which run much slower compared to the newly generated and faster layer (e.g., convolution layer in our experiment) like a rebirth.
(2) There are ways to perform this kind of operator fusion without retraining, and some deep learning framework such as Theano and the upcoming TensorFlow XLA implement them. It would have been nice to have a baseline that implements it, especially since most of the additional energy cost from non-fused operators comes from the extra intermediate memory writes that operator fusion removes.
Ans: As far as we know, the operator fusion without retraining can only be applied to some specific structure, i.e.,  batch normalization or mean normalization after convolution. This kind of operator fusion can’t be applied to other kinds of layers, such as pooling, LRN, etc. Our pipeline is much more general.
The deep learning framework such as Theano and XLA performs the operator fusion to reduce the memory.
However, the execution time of these models may not be accelerated due to the use of same architectures (i.e., performing exactly the same set of floating-point operations).  If we view them as the baseline, their execution time would be much “higher” than deep-rebirth.
In contrast, our method re-trained the rebirth layers and significantly speeds up the execution time.
(3) Batchnorm can be folded into convolution layers without retraining by scaling the weights. Were they folded into the baseline figures reported in Table 7?
Ans: We agree that batchnorm can be folded into convolution layer without retraining by scaling the weights. The results listed in Table 7 also include other layers’  optimization, e.g., pooling, not limited to batchnorm.
(4) At the time of writing, the authors have not provided the details that would make this research reproducible, in particular how the depth of the fused layers relates to the depth of the original layers in each of the experiments. Open-sourcing: having the implementation be open-source always enhances the usefulness of such paper. Not a requirement obviously.
Ans:  We have listed the details in our revision. We plan to release the code after final decision. Our first step for open-source would be the release of the model architectures (in caffe’s prototxt format) used in our paper.
(5) Retraining: how much time (epochs) does the retraining take? Did you consider using any form of distillation?
Ans: Each step of our optimization pipeline takes around 2 epochs (in section 4.1.1). For GoogleNet, it takes 9 steps and 18 epochs. We regard this training time (or rebirth time) as a one-time cost and once trained it can be deployed to any mobile devices.
Our “rebirth” method can be considered as a special form of distillation that transfers the knowledge from the cumbersome substructure of multiple layers to the rebirthed accelerated substructure. Different from traditional distillation method, our approach adopts layer-wise optimization and maintains the knowledge of the rest layers.
Also, our method utilizes a softmax at the end of our CNN to optimize classification accuracy.
(6) Interesting set of experiments. This paper needs a lot of improvements to be suitable for publication.
Ans: Thanks for your appreciation of our paper. We are constantly improving the quality of our paper.
This paper proposes to reduce model size and evaluation time of deep CNN models on mobile devices by converting multiple layers into single layer and then retraining the converted model. The paper showed that the computation time can be reduced by 3x to 5x with only 0.4% accuracy loss on a specific model.
Reducing model sizes and speeding up model evaluation are important in many applications. I have several concerns:
1. There are many techniques that can reduce model sizes.
For example, it has been shown by several groups that using the teacher-student approach, people can achieve the same and sometimes even better accuracy than the teacher (big model) using a much smaller model.
However, this paper does not compare any one of them.
2. The technique proposed in this paper is limited in its applicability since it's designed specifically for the models discussed in the paper.
3. Replacing several layers with single layer is a relatively standard procedure.
For example, the mean variance normalization layer and batch normalization layer can all be absorbed without retraining or losing accuracy.
BTW, the DNN low-rank approximation technique was first proposed in speech recognition. e.g.,
Xue, J., Li, J. and Gong, Y., 2013, August. Restructuring of deep neural network acoustic models with singular value decomposition. In INTERSPEECH (pp. 2365-2369).
(1) There are many techniques that can reduce model sizes.
For example, it has been shown by several groups that using the teacher-student approach, people can achieve the same and sometimes even better accuracy than the teacher (big model) using a much smaller model.
However, this paper does not compare any one of them.
Ans: We agree that there are many different kinds of model compression techniques.
However, these algorithms mainly focus on reducing the model size instead of improving the running speed, especially, the acceleration of the execution time of deep model on mobile devices is not evident. We already compared with several state-of-the-art works in the paper. As listed in Table 5, in terms of running speed, runtime memory, energy, our approach performs significant better than GoogleNet-Tucker (ICLR'16),  Squeezenet (https://openreview.net/forum?id=S1xh5sYgx) and others. The details of comparison are listed as follows:
-----------------------|-----DeepRebirth-----|----SqueezeNet--|--Tucker (ICLR'16)*----|--GoogleNet---|
Acc                      |	      86.5%	  |	      80.3%	 |     	        85.7%          |	        88.9%      |
Execution time    |         65.34 ms      |       75.34 ms        |	     342.5 ms        |       424.7 ms    |
Energy                |	        226 mJ      |          288 mJ        |             902 mJ         |          984 mJ    |
Memory size       |         14.8 MB       |           36.5 MB      |           35.8 MB         |         33.2 MB    |
Parameter size   |       11.99 MB       |         4.72 MB         |            14.38 MB      |        26.72 MB   |
*The number is based on our implementation of the algorithm in ICLR'16 paper.
(2) The technique proposed in this paper is limited in its applicability since it's designed specifically for the models discussed in the paper.
Ans: Our proposed approach can be applied to current general deep neural network structures that consist of different layers, e.g., convolution, pooling, LRN, etc. The reason why we used GoogleNet, Resnet and Alexnet is due to their popularity, as evident in current model compression work, e.g,  Song Han. et al. [ICLR’16],  Yong-Deok Kim et al. [ICLR’16].
(3) Replacing several layers with single layer is a relatively standard procedure.
For example, the mean variance normalization layer and batch normalization layer can all be absorbed without retraining or losing accuracy.
Ans:  As far as we know, only mean variance normalization layer and batch normalization layer can all be absorbed without retraining, however, the operation can be only applied to two-layers and needs specific structure – normalization layer right after convolution.
To address these limitations, we proposed a general approach that can be applied to pooling, LRN, batchnorm and convolution, etc.
(4) BTW, the DNN low-rank approximation technique was first proposed in speech recognition.
Ans: Thanks for the comment.  We have already included this reference in revision.
One of the main idea of this paper is to replace pooling layers with convolutions of stride 2 and retraining the model. Authors merge this into a new layer and brand it as a new type of layer. This is very misleading and adding noise to the field. And using strided convolutions rather than pooling is not actually novel (e.g. https://arxiv.org/abs/1605.02346).
While the speed-up obtained are good, the lack of novelty and the rebranding attempt make this paper not a good fit for ICLR.
The reviewer missed the key points in our paper. There exist clear and significant differences between our work and previous works.
We are not "replacing a pooling layer to a convolution layer of stride 2" which was in fact proposed in another paper "“STRIVING FOR SIMPLICITY: THE ALL CONVOLUTIONAL NET”.
Instead, our method merges various types of non-tensor layers (i.e., pooling, LRN etc.) and their bottom convolution layer to a new convolution layer (i.e., the "rebirth" layer).  This is how we achieve significantly improved efficiency and accelerated model execution.
We view this process as "rebirth" of a new layer  instead of "a new *type* of layer" as an analogy to help readers' catch up the idea and understand the merit of work.
In addition, from the reference (https://arxiv.org/abs/1605.02346), we did not find the similar operation as our algorithm which optimizes multiple layers with a new layer. As indicted in Section 4 (tasks and datasets section in reference), “First, a shallow CNN which consists of six layers each followed by a rectified linear unit [32] and Batch Normalization [33]. The first four layers include max pooling with stride 2, leading to an effective stride of 16,”  the method in reference relied on traditional network structure without applying the optimization schemes as ours.
Regarding the novelty, to the best of our knowledge, this is *first* one that leverages merging of different types of layers (including  *non-tensor* layers) for acceleration of CNN models on mobile devices.  This is also the *best* method (see Table below) reported in accelerating the CNN on mobile devices.  It has technical novelty, and truly works pretty well in reality.
If anyone has questions, we are very happy to do competitions on mobile devices.
Giving all the above considerations, we are very disappointed to see the reviewer viewing our work as "misleading and adding noise to the field. "

Meta-review:
role: PC
data: comment: The proposed method doesn't have enough novelty to be accepted to ICLR.

Decision: Reject