Title: SLIMMABLE NEURAL NETWORKS

Abstract: We present a simple and general method to train a single neural network executable at different widths 1 , permitting instant and adaptive accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve similar (and in many cases better) ImageNet classification accuracy than individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths respectively. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. Lastly we visualize and discuss the learned features of slimmable networks. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks.

1. INTRODUCTION:

Recently deep neural networks are prevailing in applications on mobile phones, augmented reality devices and autonomous cars. Many of these applications require a short response time. Towards this goal, manually designed lightweight networks (Howard et al., 2017; Zhang et al., 2017; Sandler et al., 2018) are proposed with low computational complexities and small memory footprints. Automated neural architecture search methods (Tan et al., 2018 ) also integrate on-device latency into search objectives by running models on a specific phone. However, at runtime these networks are not re-configurable to adapt across different devices given a same response time budget. For example, there were over 24,000 unique Android devices in 20152 . These devices have drastically different runtimes for the same neural network (Ignatov et al., 2018) , as shown in Table 1 . In practice, given the same response time constraint, high-end phones can achieve higher accuracy by running larger models, while low-end phones have to sacrifice accuracy to reduce latency. Although a global hyper-parameter, width multiplier, is provided in lightweight networks (Howard et al., 2017; Zhang et al., 2017; Sandler et al., 2018) to trade off between latency and accuracy, it is inflexible and has many constraints. First, models with different width multipliers need to be trained, benchmarked and deployed individually. A big offline table needs to be maintained to document the allocation of different models to different devices, according to time and energy budget. Second, even on a same device, the computational budget varies (for example, excessive consumption of background apps reduces the available computing capacity), and the energy budget varies (for example, a mobile phone may be in low-power or power-saving mode). Third, when switching to a larger or smaller model, the cost of time and data for downloading and offloading models is not negligible. Recently dynamic neural networks are introduced to allow selective inference paths. Liu & Deng (2017) introduce controller modules whose outputs control whether to execute other modules. It has low theoretical computational complexity but is nontrivial to optimize and deploy on mobiles since dynamic conditions prohibit layer fusing and memory optimization. Huang et al. (2017) adapt early-exits into networks and connect them with dense connectivity. Wu et al. (2017) and Wang et al. (2017) propose to selectively choose the blocks in a deep residual network to execute during inference. Nevertheless, in contrast to width (number of channels), reducing depth cannot reduce memory footprint in inference, which is commonly constrained on mobiles.

The question remains: Given budgets of resources, how to instantly, adaptively and efficiently trade off between accuracy and latency for neural networks at runtime? In this work we introduce slimmable neural networks, a new class of networks executable at different widths, as a general solution to trade off between accuracy and latency on the fly. Figure 1 shows an example of a slimmable network that can switch between four model variants with different numbers of active channels. The parameters of all model variants are shared and the active channels in different layers can be adjusted. For brevity, we denote a model variant in a slimmable network as a switch, the number of active channels in a switch as its width. 0.25× represents that the width in all layers are scaled by 0.25 of the full model. In contrast to other solutions above, slimmable networks have several advantages: (1) For different conditions, a single model is trained, benchmarked and deployed.

(2) A near-optimal trade-off can be achieved by running the model on a target device and adjusting active channels accordingly.

(3) The solution is generally applicable to (normal, group, depthwise-separable, dilated) convolutions, fully-connected layers, pooling layers and many other building blocks of neural networks. It is also generally applicable to different tasks including classification, detection, identification, image restoration and more. (4) In practice, it is straightforward to deploy on mobiles with existing runtime libraries. After switching to a new configuration, the slimmable network becomes a normal network to run without additional runtime and memory cost.

However, neural networks naturally run as a whole and usually the number of channels cannot be adjusted dynamically. 2018) encourage structured sparsity by pruning channels, filters and network depth and fine-tuning iteratively with various penalty terms. As another family, model distilling methods (Hinton et al., 2015; Romero et al., 2014; Zhuang et al., 2018) first train a large network or an ensemble of networks, and then transfer the learned knowledge to a small model. Soft-targets and intermediate representations from trained large models are used to train a small model. Conditional Normalization. Many real-world problems require conditional input. Feature-wise transformation (Dumoulin et al., 2018) is a prevalent approach to integrate different sources of information, where conditional scales and biases are applied across the network. It is commonly implemented in the form of conditional normalization layers, such as batch normalization or layer normalization (Ba et al., 2016) . Conditional normalization is widely used in tasks including style transfer (Dumoulin et al., 2016; Li et al., 2017a; Huang & Belongie, 2017; Li et al., 2017b) , image recognition (Li et al., 2016b; Yang et al., 2018) and many others (Perez et al., 2017b; a) .

Adaptive Computation:



3. SLIMMABLE NEURAL NETWORKS:



3.1. NAIVE TRAINING OR INCREMENTAL TRAINING:

To train slimmable neural networks, we begin with a naive approach, where we directly train a shared neural network with different width configurations. The training framework is similar to the one of our final approach, as shown in Algorithm 1. The training is stable, however, the network obtains extremely low top-1 testing accuracy around 0.1% on 1000-class ImageNet classification.

Error curves of the naive approach are shown in Figure 2 . We conjecture the major problem in the naive approach is that: for a single channel in a layer, different numbers of input channels in previous layer result in different means and variances of the aggregated feature, which are then rolling averaged to a shared batch normalization layer. (Ioffe & Szegedy, 2015; Radford et al., 2015) , also it can encode conditional information to feature representations (Perez et al., 2017b; Li et al., 2016b) .

To train slimmable networks, S-BN privatizes all batch normalization layers for each switch in a slimmable network. Compared with the naive training approach, it solves the problem of feature aggregation inconsistency between different switches by independently normalizing the feature mean and variance during testing. The scale and bias in S-BN may be able to encode conditional information of width configuration of current switch (the scale and bias can be merged into variables of moving mean and variance after training, thus by default we also use independent scale and bias as they come for free). Moreover, in contrast to incremental training, with S-BN we can jointly train all switches at different widths, therefore all weights are jointly updated to achieve a better performance. A representative training and validation error curve with S-BN is shown in Figure 2 .

S-BN also has two important advantages. First, the number of extra parameters is negligible. Table 2 enumerates the number and percentage of parameters in batch normalization layers (after training, µ, σ, γ, β are merged into two parameters). In most cases, batch normalization layers only have less than 1% of the model size. Second, the runtime overhead is also negligible for deployment. In practice, batch normalization layers are typically fused into convolution layers for efficient inference. For slimmable networks, the re-fusing of batch normalization can be done on the fly at runtime since its time cost is negligible. After switching to a new configuration, the slimmable network becomes a normal network to run without additional runtime and memory cost. Get next mini-batch of data x and label y.

5::

Clear gradients of weights, optimizer.zero grad().

6::

for width in switchable width list do 7:

Switch the batch normalization parameters of current width on network M .

8::

Execute sub-network at current width, ŷ = M (x).

9::

Compute loss, loss = criterion(ŷ, y).

10:

Compute gradients, loss.backward().

11:

end for 12:

Update weights, optimizer.step(). 13: end for

4. EXPERIMENTS:

In this section, we first evaluate slimmable networks on ImageNet (Deng et al., 2009) classification. Further we demonstrate the performance of a slimmable network with more switches. Finally we apply slimmable networks to a number of different applications.

4.1. IMAGENET CLASSIFICATION:

We experiment with the ImageNet (Deng et al., 2009) classification dataset with 1000 classes. It is comprised of around 1.28M training images and 50K validation images.

We first investigate slimmable neural networks on three state-of-the-art lightweight networks, Mo-bileNet v1 (Howard et al., 2017) , MobileNet v2 (Sandler et al., 2018) , ShuffleNet (Zhang et al., 2017) , and one representative large model ResNet-50 (He et al., 2016) .

To make a fair comparison, we follow the training settings (for example, learning rate scheduling, weight initialization, weight decay, data augmentation, input image resolution, mini-batch size, training iterations, optimizer) in corresponding papers respectively (Howard et al., 2017; Sandler et al., 2018; Zhang et al., 2017; He et al., 2016) . One exception is that for MobileNet v1 and Mo-bileNet v2, we use stochastic gradient descent (SGD) as the optimizer instead of the RMSPropOptimizer (Howard et al., 2017; Sandler et al., 2018) . For ResNet-50 (He et al., 2016) , we train for 100 epochs, and decrease the learning rate by 10× at 30, 60 and 90 epochs. We evaluate the top-1 classication error on the center 224 × 224 crop of images in the validation set. More implementation details are included in Appendix A.

We first show training and validation error curves in Figure 2 . The results of naive training approach are also reported as comparisons. Although both our approach and the naive approach are stable in training, the testing error of naive approach is extremely high. With switchable batch normalization, the error rates of different switches are stable and the rank of error rates is also preserved consistently across all training epochs.

Next we show in Table 3 the top-1 classification error for both individual networks and slimmable networks given same width configurations. We use S-to indicate slimmable models. The error rates for individual models are from corresponding papers except those denoted with † . The runtime FLOPs (number of Multiply-Adds) for each model are also reported as a reference. Table 3 shows that slimmable networks achieve similar performance compared to those that are individually trained. Intuitively compressing different networks into a shared network poses extra optimization constraints to each network, a slimmable network is expected to have lower performance than individually trained ones. However, our experiments show that joint training of different switches indeed improves the performance in many cases, especially for slim switches (for example, MobileNet v1 0.25× is improved by 3.3%). We conjecture that the improvements may come from implicit model distilling (Hinton et al., 2015; Romero et al., 2014) where the large model transfers its knowledge to small model by weight sharing and joint training. Our proposed approach for slimmable neural networks is generally applicable to the above representative network architectures. It is noteworthy that we experiment with both residual and nonresidual networks (MobileNet v1). The training of slimmable models can be applied to convolutions, depthwise-separable convolutions (Chollet, 2016), group convolutions (Xie et al., 2017) , pooling layers, fully-connectted layers, residual connections, feature concatenations and many other building blocks of deep neural networks.

4.2. MORE SWITCHES IN SLIMMABLE NETWORKS:

The more switches available in a slimmable network, the more choices one have for trade-offs between accuracy and latency. We thus investigate how the number of switches potentially impact accuracy. In Table 4 , we train a 8-switch slimmable MobileNet v1 and compare it with 4-switch and individually trained ones. The results show that a slimmable network with more switches have similar performance, demonstrating the scalability of our proposed approach. Finally, we apply slimmable networks on tasks of bounding-box object detection, instance segmentation and keypoints detection based on detection frameworks MMDetection (Chen et al., 2018) and Detectron (Girshick et al., 2018) . Following the settings of R-50-FPN-1× (Lin et al., 2016; Girshick et al., 2018; Chen et al., 2018) , pre-trained ResNet-50 models at different widths are fine-tuned and evaluated. The lateral convolution layers in feature pyramid network (Lin et al., 2016) are same for different pre-trained backbone networks. For individual models, we train ResNet-50 with different width multipliers on ImageNet and fine-tune them on each task individually. For slimmable models, we first train on ImageNet using Algorithm 1. Following Girshick et al. (2018) , the moving averaged means and variances of switchable batch normalization are also fixed after training. Then we fine-tune the slimmable models on each task using Algorithm 1. The detection head and lateral convolution layers in feature pyramid network (Lin et al., 2016) are shared across different switches in a slimmable network. In this way, each switch in a slimmable network is with exactly same network architecture and FLOPs with its individual baseline. More details of implementation are included in Appendix B. We train all models on COCO 2017 train set and report Average Precision (AP) on COCO 2017 validation set in Table 5 . In general, slimmable neural networks perform better than individually trained ones, especially for slim network architectures. The gain of performance is presumably due to implicit model distillation (Hinton et al., 2015; Romero et al., 2014) and richer supervision signals. Visualization of Top-activated Images. Our primary interest lies in understanding the role that the same channel played in different switches in a slimmable network. We employ a simple visualization approach (Girshick et al., 2014) to visualize the images with highest activation values on a specific channel. Figure 3 

6. CONCLUSION:

We introduced slimmable networks that permit instant and adaptive accuracy-efficiency trade-offs at runtime. Switchable batch normalization is proposed to facilitate robust training of slimmable networks. Compared with individually trained models with same width configurations, slimmable networks have similar or better performances on tasks of classification, object detection, instance segmentation and keypoints detection. The proposed slimmable networks and slimmable training could be further applied to unsupervised learning and reinforcement learning, and may help to related fields such as network pruning and model distillation.

end, we mainly conduct COCO experiments based on another detection framework: MMDetection (Chen et al., 2018) , which has hyper-parameter settings with same pytorch-style ResNet-50. With same hyper-parameter settings (i.e., RCNN R50 FPN 1 ×), we fine-tune both individual ResNet-50 models and slimmable ResNet-50 on tasks of object detection and instance segmentation. Our reproduced results on ResNet-50 1.0× is consistent with official models in MMDetection (Chen et al., 2018) . For keypoint detection task, we conduct experiment on Detectron (Girshick et al., 2018) framework by modifying caffe-style ResNet-50 to pytorch-style and training on 4 GPUs without other modification of hyper-parameters. We have released code (training and testing) and pretrained models on both ImageNet classification task and COCO detection tasks.

C ABLATION STUDY OF CONDITIONAL PARAMETERS IN BN:

In our work, private parameters γ, β, µ, σ 2 of BN are introduced in Switchable Batch Normalization for each sub-network to independently normalize feature y = γ y-µ √ σ 2 + + β, where y is input and y is output, γ, β are learnable scale and bias, µ, σ 2 are moving averaged statistics for testing. In switchable batch normalization, the private γ, β come for free because after training, they can be merged as y = γ y + β , γ = γ √ σ 2 + , β = β -γ µ. Nevertheless, we present ablation study on how these conditional parameters affect overall performance. The results are shown in Table 6 . Private γ, β 46.9 35.2 30.5 28.5 Shared γ, β 47.1 (-0.2) 35.9 (-0.7) 30.9 (-0.4) 28.8 (-0.3) 

:

Width refers to number of channels in a layer.

https://opensignal.com/reports/2015/08/android-fragmentation/

Figures and Tables:

Figure 1: Illustration of slimmable neural networks. The same model can run at different widths (number of active channels), permitting instant and adaptive accuracy-efficiency trade-offs.

Graph. To reduce computation of a neural network, some works propose to adaptively construct the computation graph of a neural network.Liu & Deng (2017),Wu et al. (2017),Lin et al. (2017) andWang et al. (2017) introduced additional controller modules or gating functions to determine the computation graph based on the current input.Amthor et al. (2016),Veit & Belongie (2017),Huang et al. (2017),Kuen et al. (2018) andHu et al. (2017) implanted earlyexiting prediction branches to reduce the average execution depth. The computation graph of these methods are conditioned on network input, and lower theoretical computational complexity can be achieved.

Figure 2: Training and validation curves of slimmable networks. Left shows the training error of the largest switch. Right shows testing errors on validation set with different switches. For naive approach, the training is stable (left) but testing error is high (right, zoomed). Slimmable networks trained with S-BN have stable and rank-preserved testing accuracy across all training iterations.

Figure 3: Top-activated images for same channel 3 9 in different switches in S-MobileNet v1. Different rows represent results from different switches. Images with red outlines are mis-classified. Note that the white color in RGB is [255, 255, 255], yellow in RGB is [255, 255, 0].

Figure 4: Values of BN parameters in different switches. We show BN values of both shallow (left, BN 1 1 to 1 8) and deep (right, BN 12 1 to 12 8) layers of S-MobileNet v1.Values of Switchable Batch Normalization. Our proposed S-BN learns different BN transformations for different switches. But how diverse are the learned BN parameters? We show the values of batch normalization weights in both shallow (BN 1 1 to 1 8) and deep (BN 12 1 to 12 8) layers of S-MobileNet v1 in Figure4. The results show that for shallow layers, the mean, variance, scale and bias are very close, while in deep layers they are diverse. The value discrepancy is increased layer by layer in our observation, which also indicates that the learned features of a same channel in different switches have slight variations of semantics.

Runtime of MobileNet v1 for image classification on different devices.

Table content: <table><tr><td/><td colspan="5">OnePlus 6 Google Pixel LG Nexus 5 Samsung Galaxy S3 ASUS ZenFone 2</td></tr><tr><td>Runtime</td><td>24 ms</td><td>116 ms</td><td>332 ms</td><td>553 ms</td><td>1507 ms</td></tr></table>

Empirically training neural networks with multiple switches has an extremely low testing accuracy around 0.1% for 1000-class ImageNet classification. We conjecture it is mainly due to the problem that accumulating different number of channels results in different feature mean and variance. This discrepancy of feature mean and variance across different switches leads to inac-

Table content: <table><tr><td>individually trained networks, we demonstrate similar (and in many cases better) performances of</td></tr><tr><td>slimmable MobileNet v1 [0.25, 0.5, 0.75, 1.0]×, MobileNet v2 [0.35, 0.5, 0.75, 1.0]×, ShuffleNet [0.5, 1.0, 2.0]×</td></tr><tr><td>and ResNet-50 [0.25, 0.5, 0.75, 1.0]× ([ * ]× denotes available switches). We further train a 8-switch</td></tr><tr><td>slimmable MobileNet v1 [0.25, 0.35, 0.45, 0.55, 0.65, 0.75, 0.85, 1.0]× without accuracy drop to demonstrate</td></tr><tr><td>the scalability of our method. Beyond image classification, we also apply slimmable networks to</td></tr><tr><td>various applications including COCO bounding-box object detection, instance segmentation and</td></tr><tr><td>person keypoints detection. Experiments show that slimmable networks achieve better performance</td></tr><tr><td>than individual ones at different widths respectively. The proposed slimmable networks are not only</td></tr><tr><td>flexible and practical by design, but also effective, scalable and widely applicable according to our</td></tr><tr><td>experiments. Lastly we visualize and discuss the learned features of slimmable networks.</td></tr><tr><td>2 RELATED WORK</td></tr><tr><td>Model Pruning and Distilling. Model pruning and distilling have a rich history in the literature of</td></tr><tr><td>deep neural networks. Early methods (Han et al., 2015a;b) sparsify connections in neural networks.</td></tr><tr><td>curate statistics of shared Batch Normalization layers (Ioffe &amp; Szegedy, 2015), an important training</td></tr><tr><td>stabilizer. To this end, we propose a simple and effective approach, switchable batch normalization,</td></tr><tr><td>that privatizes batch normalization for different switches of a slimmable network. The variables</td></tr><tr><td>of moving averaged means and variances can independently accumulate feature statistics of each</td></tr><tr><td>switch. Moreover, Batch Normalization usually comes with two additional learnable scale and bias</td></tr><tr><td>parameter to ensure same representation space (Ioffe &amp; Szegedy, 2015). These two parameters</td></tr><tr><td>may able to act as conditional parameters for different switches, since the computation graph of a</td></tr><tr><td>slimmable network depends on the width configuration. It is noteworthy that the scale and bias can</td></tr><tr><td>be merged into variables of moving mean and variance after training, thus by default we also use</td></tr><tr><td>independent scale and bias as they come for free. Importantly, batch normalization layers usually</td></tr><tr><td>have negligible size (less than 1%) in a model.</td></tr><tr><td>We first conduct comprehensive experiments on ImageNet classification task to show the effective-</td></tr><tr><td>ness of switchable batch normalization for training slimmable neural networks. Compared with</td></tr></table>

We fix it and add extra parameters B to make it an extended model A+B (MobileNet v2 0.5×). The extra parameters are fine-tuned along with the fixed parameters of A on the training data. Although the approach is stable in both training and testing, the top-1 accuracy only increases from 60.3% of A to 61.0% of A+B. In contrast, individually trained MobileNet v2 0.5× achieves 65.4% accuracy on the ImageNet validation set. The major reason for this accuracy degradation is that when expanding base model A to the next level A+B, new connections, not only from B to B, but also from B to A and from A to B, are added in the computation graph. The incremental training prohibits joint adaptation of weights A and B, significantly deteriorating the overall performance.3.2 SWITCHABLE BATCH NORMALIZATIONMotivated by the investigations above, we present a simple and highly effective approach, named Switchable Batch Normalization (S-BN), that employs independent batch normalization(Ioffe & Szegedy, 2015) for different switches in a slimmable network. Batch normalization (BN) was originally proposed to reduce internal covariate shift by normalizing the feature: y = γ y-µ √ σ 2 + + β, where y is the input to be normalized and y is the output, γ, β are learnable scale and bias, µ, σ 2 are mean and variance of current mini-batch during training. During testing, moving averaged statistics of means and variances across all training images are used instead. BN enables faster and stabler training of deep neural networks

Table content: <table/>

Number and percentage of parameters in batch normalization layers. Define switchable width list for slimmable network M , for example, [0.25, 0.5, 0.75, 1.0]×. 1: Initialize shared convolutions and fully-connected layers for slimmable network M . 2: Initialize independent batch normalization parameters for each width in switchable width list.

Table content: <table><tr><td/><td colspan="3">MobileNet v1 1.0× MobileNet v2 1.0× ShuffleNet 2.0×</td><td>ResNet-50 1.0×</td></tr><tr><td colspan="2">Conv and FC 4,210,088 (99.483%)</td><td>3,470,760 (99.027%)</td><td colspan="2">5,401,816 (99.102%) 25,503,912 (99.792%)</td></tr><tr><td>BatchNorm</td><td>21,888 (0.517%)</td><td>34,112 (0.973%)</td><td>48,960 (0.898%)</td><td>53,120 (0.208%)</td></tr><tr><td colspan="3">3.3 TRAINING SLIMMABLE NEURAL NETWORKS</td><td/><td/></tr><tr><td colspan="5">Our primary objective to train a slimmable neural network is to optimize its accuracy averaged from</td></tr><tr><td>all switches.</td><td/><td/><td/><td/></tr></table>

Results of ImageNet classification. We show top-1 error rates of individually trained networks and slimmable networks given same width configurations and FLOPs. We use S-to indicate slimmable models, † to denote our reproduced result.

Table content: <table><tr><td colspan="2">Individual Networks</td><td/><td colspan="3">Slimmable Networks</td><td>FLOPs</td></tr><tr><td>Name</td><td colspan="2">Params Top-1 Err.</td><td>Name</td><td colspan="2">Params Top-1 Err.</td><td/></tr><tr><td>MobileNet v1 1.0×</td><td>4.2M</td><td>29.1</td><td/><td/><td>28.5 (0.6)</td><td>569M</td></tr><tr><td>MobileNet v1 0.75× MobileNet v1 0.5×</td><td>2.6M 1.3M</td><td>31.6 36.7</td><td>S-MobileNet v1 [0.25, 0.5, 0.75, 1.0]×</td><td>4.3M</td><td>30.5 (1.1) 35.2 (1.5)</td><td>317M 150M</td></tr><tr><td>MobileNet v1 0.25×</td><td>0.5M</td><td>50.2</td><td/><td/><td>46.9 (3.3)</td><td>41M</td></tr><tr><td>MobileNet v2 1.0×</td><td>3.5M</td><td>28.2</td><td/><td/><td>29.5 (-1.3)</td><td>301M</td></tr><tr><td>MobileNet v2 0.75× MobileNet v2 0.5×</td><td>2.6M 2.0M</td><td>30.2 34.6</td><td>S-MobileNet v2 [0.35, 0.5, 0.75, 1.0]×</td><td>3.6M</td><td>31.1 (-0.9) 35.6 (-1.0)</td><td>209M 97M</td></tr><tr><td>MobileNet v2 0.35×</td><td>1.7M</td><td>39.7</td><td/><td/><td>40.3 (-0.6)</td><td>59M</td></tr><tr><td>ShuffleNet 2.0× ShuffleNet 1.0× ShuffleNet 0.5×</td><td>5.4M 1.8M 0.7M</td><td>26.3 32.6 43.2</td><td>S-ShuffleNet [0.5, 1.0, 2.0]×</td><td>5.5M</td><td>28.7 (-2.4) 34.5 (-0.9) 42.7 (0.5)</td><td>524M 138M 38M</td></tr><tr><td>ResNet-50 1.0×</td><td>25.5M</td><td>23.9</td><td/><td/><td>24.0 (-0.1)</td><td>4.1G</td></tr><tr><td>ResNet-50 0.75×  † ResNet-50 0.5×  †</td><td>14.7M 6.9M</td><td>25.3 28.0</td><td>S-ResNet-50 [0.25, 0.5, 0.75, 1.0]×</td><td>25.6M</td><td>25.1 (0.2) 27.9 (0.1)</td><td>2.3G 1.1G</td></tr><tr><td>ResNet-50 0.25×  †</td><td>2.0M</td><td>36.2</td><td/><td/><td>35.0 (1.2)</td><td>278M</td></tr></table>



Table content: <table><tr><td colspan="10">: Top-1 error rates on ImageNet classification with individually trained networks, 4-switch</td></tr><tr><td colspan="10">S-MobileNet v1 [0.25, 0.5, 0.75, 1.0]× and 8-switch S-MobileNet v1 [0.25, 0.35, 0.45, 0.55, 0.65, 0.75, 0.85, 1.0]×.</td></tr><tr><td/><td>0.25×</td><td colspan="3">0.35× 0.45× 0.5×</td><td colspan="3">0.55× 0.65× 0.75×</td><td colspan="2">0.85× 1.0×</td></tr><tr><td colspan="2">Individual 50.2</td><td>-</td><td>-</td><td>36.7</td><td>-</td><td>-</td><td>31.6</td><td>-</td><td>29.1</td></tr><tr><td>4-switch</td><td colspan="2">46.9 (3.3) -</td><td>-</td><td colspan="2">35.2 (1.5) -</td><td>-</td><td colspan="2">30.5 (1.1) -</td><td>28.5 (0.6)</td></tr><tr><td>8-switch</td><td colspan="2">47.6 (2.6) 41.1</td><td>36.6</td><td>-</td><td>33.8</td><td>31.4</td><td colspan="2">30.2 (1.4) 29.2</td><td>28.4 (0.7)</td></tr><tr><td colspan="9">4.3 OBJECT DETECTION, INSTANCE SEGMENTATION AND KEYPOINTS DETECTION</td></tr></table>

Average precision (AP) on COCO 2017 validation set with individually trained networks and slimmable networks. ResNet-50 models are used as backbones for Faster-RCNN, Mask-RCNN and Keypoints-RCNN based on detection frameworks(Girshick et al., 2018;Chen et al., 2018). Faster 1.0× indicates Faster-RCNN for object detection with ResNet-50 1.0× as backbone.

Table content: <table><tr><td>Type</td><td colspan="3">Individual Networks</td><td colspan="3">Slimmable Networks</td></tr><tr><td/><td colspan="3">Box AP Mask AP Kps AP</td><td colspan="3">Box AP Mask AP Kps AP</td></tr><tr><td>Faster 1.0×</td><td>36.4</td><td>-</td><td>-</td><td>36.8 (0.4)</td><td>-</td><td>-</td></tr><tr><td>Faster 0.75×</td><td>34.7</td><td>-</td><td>-</td><td>36.1 (1.4)</td><td>-</td><td>-</td></tr><tr><td>Faster 0.5×</td><td>32.7</td><td>-</td><td>-</td><td>34.0 (1.3)</td><td>-</td><td>-</td></tr><tr><td>Faster 0.25×</td><td>27.5</td><td>-</td><td>-</td><td>29.6 (2.1)</td><td>-</td><td>-</td></tr><tr><td>Mask 1.0×</td><td>37.3</td><td>34.2</td><td>-</td><td colspan="2">37.4 (0.1) 34.9 (0.7)</td><td>-</td></tr><tr><td>Mask 0.75×</td><td>35.6</td><td>32.9</td><td>-</td><td colspan="2">36.7 (1.1) 34.3 (1.4)</td><td>-</td></tr><tr><td>Mask 0.5×</td><td>33.4</td><td>30.9</td><td>-</td><td colspan="2">34.7 (1.5) 32.6 (1.7)</td><td>-</td></tr><tr><td>Mask 0.25×</td><td>28.2</td><td>26.6</td><td>-</td><td colspan="2">30.2 (2.0) 28.6 (2.0)</td><td>-</td></tr><tr><td>Kps 1.0×</td><td>50.5</td><td>-</td><td>61.3</td><td>52.8 (2.3)</td><td>-</td><td>63.9 (2.6)</td></tr><tr><td>Kps 0.75×</td><td>49.6</td><td>-</td><td>60.5</td><td>52.7 (3.1)</td><td>-</td><td>63.6 (3.1)</td></tr><tr><td>Kps 0.5×</td><td>48.5</td><td>-</td><td>59.8</td><td>51.6 (3.1)</td><td>-</td><td>62.6 (2.8)</td></tr><tr><td>Kps 0.25×</td><td>45.4</td><td>-</td><td>56.7</td><td>48.2 (2.8)</td><td>-</td><td>59.5 (2.8)</td></tr></table>

Top-1 error rates on ImageNet classification with two S-MobileNet v1 [0.25, 0.5, 0.75, 1.0]× with private scale and bias or shared ones.

Table content: <table><tr><td>0.25×</td><td>0.5×</td><td>0.75×</td><td>1.0×</td></tr></table>

Peer review (iclr2019):
 This paper trains a single network executable at different widths. This is implemented by maintaining separate BN parameter and statistics for different width. The problem is well-motivated and the proposed method can be very helpful for deployment of deep models to devices with varying capacity and computational ability. This paper is well-written and the experiments are performed on various structures. Still I have several concerns regarding the algorithm. 1. In algo 1, while gradients for convolutional and fully-connected layers are accumulated for all switches before update, how are the parameters for different switches updated? 2. In algo 1, the gradients of all switches are accumulated before the update. This may result in implicit unbalanced gradient information, e.g. the connections in 0.25x model in Figure 1 has gradient flows on all four different switches, while the right-most 0.25x connections in 1.0x model has only one gradient flow from the 1.0x switch, will this unbalanced gradient information increase optimization difficulty and how is it solved? 3. In the original ResNet paper, https://arxiv.org/pdf/1512.03385.pdf, the top-1 error of RestNet-50 is <21% in Table 4. The number reported in this paper (Table 3) is 23.9. Where does the difference come from?