Title: Slimmable Neural Networks
Authors: Jiahui Yu
Linjie Yang
Ning Xu
Jianchao Yang
Thomas Huang
Venue: ICLR 2019
Keywords: Slimmable neural networks
mobile deep learning
accuracy-efficiency trade-offs

Abstract: We present a simple and general method to train a single neural network executable at different widths (number of channels in a layer), permitting instant and adaptive accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve similar (and in many cases better) ImageNet classification accuracy than individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths respectively. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. Lastly we visualize and discuss the learned features of slimmable networks. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks

TL;DR: We present a simple and general method to train a single neural network executable at different widths (number of channels in a layer), permitting instant and adaptive accuracy-efficiency trade-offs at runtime.

Review discussion:
This paper trains a single network executable at different widths. This is implemented by maintaining separate BN parameter and statistics for different width. The problem is well-motivated and the proposed method can be very helpful for deployment of deep models to devices with varying capacity and computational ability.
This paper is well-written and the experiments are performed on various structures.
Still I have several concerns regarding the algorithm.
1. In algo 1, while gradients for convolutional and fully-connected layers are accumulated for all switches before update, how are the parameters for different switches updated?
2. In algo 1, the gradients of all switches are accumulated before the update. This may result in implicit unbalanced gradient information, e.g. the connections in 0.25x model in Figure 1 has gradient flows on all four different switches,  while the right-most 0.25x connections in 1.0x model has only one gradient flow from the 1.0x switch, will this unbalanced gradient information increase optimization difficulty and how is it solved?
3.  In the original ResNet paper, https://arxiv.org/pdf/1512.03385.pdf, the top-1 error of RestNet-50 is <21% in Table 4. The number reported in this paper (Table 3) is 23.9. Where does the difference come from?
Thanks for your review efforts! We have addressed all three questions below:
1. As mentioned in Section 3.3, the only modification is to accumulate all gradients from different switches. It means that the optimizer (SGD for image recognition tasks) is exactly the same as training individual models (same momentum, etc.). The only difference is the value of gradient for each parameter. In Algorithm 1, we follow pytorch-style API and use optimizer.step() to indicate applying gradients. We have not observed any difficulty in optimization of slimmable networks using default optimizer in Algorithm 1.
2. There is no "unbalanced gradient" problem in training slimmable networks (it may seem like so). The parameters of 0.25x seem to have "more gradients", but in the forward view, these parameters of 0.25x are also used four times in Net 0.25x, 0.5x, 0.75x and 1.0x. It means the parameters in 0.25x are more important for the overall performance of slimmable networks.
In fact, back-propagation is strictly based on forward feature propagation. In the forward view, as mentioned in Section 3.3, our primary objective to train a slimmable network is to optimize its accuracy averaged from all switches.
3. Our reported ResNet-50 accuracy is correct (23.9 top-1 error). We evaluate single-crop testing accuracy instead of 10-crop following all our baselines. The ResNet-50 single-crop testing accuracy is publicly reported in ResNeXt paper (Table 3, 1st row) [1], released code [2] and many other publications. Our ResNet-50 has same implementation with PyTorch official pre-trained model zoo [3] where the top-1 error is also 23.9 instead of <21% (in fact ResNet-152 still has > 21% single-crop top-1 error rate).
We sincerely hope the rating can be reconsidered if it was affected by above questions. Thanks for your time and review efforts!
[1] Xie, Saining, et al. "Aggregated residual transformations for deep neural networks." Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on. IEEE, 2017.
[2] https://github.com/facebookresearch/ResNeXt
[3] https://pytorch.org/docs/stable/torchvision/models.html
Thanks for addressing my questions!
This paper presents a straightforward looking approach for creating a neural networks that can run under different resource constraints, e.g. less computation but lower quality solution and expensive high quality solution, while all the networks are having the same filters. The idea is to share the filters of the cheapest network with those of the larger more expensive networksa and train all those networks jointly with weight sharing. One important practical observation is that the batch-normalization parameters should not be shared between those filters in order to get good results.
However, the most interesting surprising observation, that is the main novelty of the work that even the highest quality vision network get substantially better by this training methodology as compared to be training alone without any weight sharing with the smaller networks, when trained for object detection and segmentation purposes (but not for recognition). This is a highly unexpected result and provides a new unanticipated way of training better segmentation models. It is especially nice that the paper does not pretend that this phenomenon is well understood but leaves its proper explanation for future work. I think a lot of interesting work is to be expected along these lines.
Thanks for your positive review and encouragements! We also believe the discovery of slimmable network opens up the possibility to many related fields including model distillation, network compression and better representation learning. We are actively exploring on these topics and hope this submission may contribute to ICLR community.
The idea is really interesting. One only need to train and maintain one single model, but use it in different platforms of different computational power.
And according to the experiment results of COCO detection, the S-version models are much better than original versions (eg. faster-0.25x, from 24.6 to 30.0) . The improvement is huge to me.
However the authors do not explain any deep reasons.
And for classification, there are slightly performance drop instead of a large improvement which is also hard to understand.
For detection, experiments on depth-wise convolution based models (such as mobilenet and shufflenet) are suggested to make this work more solid and meaningful.
Thanks for your review efforts! We have addressed all questions below:
1. We aim to train single neural network executable at different widths. We find slimmable networks achieve better results especially for small models (e.g., 0.25x) on detection tasks. We have mentioned that it is probably due to implicit distillation, richer supervision and better learned representation (since detection results are based on pre-trained ImageNet learned representation). We try to avoid strong claims of any deep reason because none of them is strictly proved by us yet. Explaining deep reasons for improvements are not the motivation or the focus of this paper.
But we are actively exploring on these questions!
2.
In fact, on average the image classification results are also improved (0.5 better top-1 accuracy in total), especially for small models. After submission, we have improved accuracy of S-ShuffleNet due to an additional ReLU layer (our implementation bug) between depthwise convolution and group convolution (Figure 2 of ShuffleNet [3]). Our models will be released.
3. Thanks for the good suggestion! Currently we conduct detection experiments mainly on Detectron [1] and MMDetection [2] framework where ResNet-50 is among the most efficient models. We do value this suggestion and will try to implement mobilenet-based detectors.
Besides, all code (including classification and detection) and pre-trained models will be released soon and we warmly welcome the community to work on together.
Thanks!
[1] https://github.com/facebookresearch/Detectron
[2] https://github.com/open-mmlab/mmdetection
[3] Zhang et al. Shufflenet: An extremely efficientconvolutional neural network for mobile devices.arXiv preprint arXiv:1707.01083, 2017.

Meta-review:
role: AC
data: metareview: This paper proposed a method that creates neural networks that can run under different resource constraints. The reviewers have consensus on accept. The pro is that the paper is novel and provides a practical approach to adjust model for different computation resource, and achieved performance improvement on object detection. One concern from reviewer2 and another public reviewer is the inconsistent performance impact on classification/detection (performance improvement on detection, but performance degradation on classification). Besides, the numbers reported in Table 1 should be confirmed: MobileNet v1 on Google Pixel 1 should have less than 120ms latency [1], not 296 ms. 


[1] Table 4 of https://arxiv.org/pdf/1801.04381.pdf

Decision: Accept (Poster)