Title: A CLOSER LOOK AT NETWORK RESOLUTION FOR EFFI-CIENT NETWORK DESIGN

Abstract: There is growing interest in designing lightweight neural networks for mobile and embedded vision applications. Previous works typically reduce computations from the structure level. For example, group convolution based methods reduce computations by factorizing a vanilla convolution into depth-wise and point-wise convolutions. Pruning based methods prune redundant connections in the network structure. In this paper, we explore the importance of network input for achieving optimal accuracy-efficiency trade-off. Reducing input scale is a simple yet effective way to reduce computational cost. It does not require careful network module design, specific hardware optimization and network retraining after pruning. Moreover, different input scales contain different representations to learn. We propose a framework to mutually learn from different input resolutions and network widths 1 . With the shared knowledge, our framework is able to find better width-resolution balance and capture multi-scale representations. It achieves consistently better ImageNet top-1 accuracy over US-Net (Yu & Huang, 2019) under different computation constraints, and outperforms the best compound scale model of EfficientNet (Tan & Le, 2019) by 1.5%. The superiority of our framework is also validated on COCO object detection and instance segmentation as well as transfer learning.

1. INTRODUCTION:

Deep neural networks have triumphed over various perception tasks such as image classification (He et al., 2016; Huang et al., 2017b; Simonyan & Zisserman, 2014) , object detection (Ren et al., 2015; Redmon et al., 2016) and semantic segmentation (Chen et al., 2017) . However, deep networks usually require large computational resources, making them hard to deploy on mobile devices and embedded systems. This motivates research in reducing the redundancy in deep neural networks or designing light-weight structures. Specifically, MobileNet (Howard et al., 2017) factorizes a standard 3 × 3 convolution into a 3×3 depth-wise convolution and a 1×1 point-wise convolution. ShuffleNet (Zhang et al., 2018a) uses 1 × 1 group convolution to further reduce computations and proposes the shuffle operation to help information flow among different groups. Another kind of approach is to prune redundant connections in the networks. Unstructured pruning methods (Han et al., 2015b; a) delete network connections which are thought unimportant. Structured pruning methods (Li et al., 2016; Anwar et al., 2017; Lemaire et al., 2018) prune the entire filters and feature maps. Recently, dynamic networks have been introduced which adopt a single model to meet varying computing resource constraints. For example, Huang et al. (2017a) proposes a multi-branch structure where early prediction can be made based on the current confidence and resource constraints. Kim et al. (2018) builds a network-in-network structure for multiple resources. Yu et al. (2019) shares weights among different subnetworks and each sub-network has its own batch normalization layer. US-Net (Yu & Huang, 2019) proposes to compute batch normalization statistics after training, and introduces two training techniques to train a network that is executable at any network widths. However, these works approach the problem only from network structure perspective, ignoring the importance of network input.

Reducing network input dimension (e.g., lowering image resolution) is a straightforward way to reduce computational cost. It can be applied to any network structure during testing. Besides, balancing between input resolution and network width can achieve better accuracy-efficiency tradeoffs. For example, to meet a dynamic resource constraint from 13 MFLOPs to 569 MFLOPs on MobileNet v1 backbone, US-Net (Yu & Huang, 2019) needs a network width range of [0.05×, 1.0×] given a 224×224 input resolution. This constraint can also be met via a network width of [0.25×, 1.0×] by adjusting the input resolution from {224, 192, 160, 128} during test time. We denote the latter model as US-Net+. As shown in Figure 1 , simply combining different resolutions with network widths during testing can already achieve a better accuracy-efficiency tradeoff than US-Net without additional efforts. EfficientNet (Tan & Le, 2019 ) also acknowledges the importance of balancing between depth, width and resolution. Moreover, reducing input resolution does not necessarily harm the performance, and may sometimes even be beneficial. Chin et al. (2019) points out that lower image resolution may produce better detection accuracy by reducing focus on redundant details. Chen et al. (2019b) claims that different scaled images contain different information. Sun et al. (2019) fuses multi-scale features with a multi-branch framework to learn robust representations.

Inspired by the observations above, we propose a unified framework to mutually learn from input resolution and network width. Our framework is able to achieve the optimal width-resolution balance under certain resource constraint. Since we share weights among different network widths, each network is able to capture multi-scale representations without any adjustments in the network structure. The whole framework is shown in Figure 2 . We summarize our contributions as follows.

1. We highlight the importance of input resolution for efficient network design. Previous works either ignore it or treat it independently from network structure. In contrast, we propose a unified framework to mutually learn from input resolution and network width.

2. Our framework is simple and general. It is compatible with any network structures (i.e., networkagnostic) and is as simple as training an independent network. Moreover, it does not change network structure, meaning it can benefit from other add-on structural optimizations.

3. We conduct extensive experiments to verify the effectiveness of our framework. Our framework significantly outperforms US-MobileNets (Yu & Huang, 2019) and independently-trained Mo-bileNets on ImageNet classification, as well as COCO object detection and instance segmentation. The experimental results on popular transfer learning datasets also demonstrate the generalization ability of the learned representations of our model.

2. RETHINKING REDUCING COMPUTATIONAL COST:

In this section, we explore the importance of input resolution for efficient network design. The computational cost of a vanilla convolution is

C 1 × C 2 × K × K × H × W (1)

where C 1 and C 2 are the number of input and output channels, K is kernel size, H and W are output feature map size. Most previous works only focus on reducing computational cost from the structure level, that is, reducing the number of channels C 1 × C 2 or reducing kernel size K.

MobileNets (Sandler et al., 2018; Howard et al., 2017) decompose the vanilla convolution into a depth-wise convolution and a 1×1 convolution, reducing the cost to (Zhang et al., 2018a) further divides 1 × 1 convolution into several groups, shrinking the computational cost to

(C 1 × K × K + C 1 × C 2 ) × H × W . ShuffleNet

(C 1 × K × K + C 1 × (C 2 /g)) × H × W

, where g is the number of groups. However, increasing the number of groups leads to high memory access cost (MAC) (Ma et al., 2018) , which makes the network inefficient in practical applications. Pruning based methods aim to prune redundant connections in the network, cutting the computational cost to specialized software and hardware optimization (Han et al., 2016) . Structured pruning methods (Li et al., 2016; Anwar et al., 2017; Lemaire et al., 2018) prune the entire filters and feature maps. The pruned network maintains structured but needs retraining to retain performance after pruning, which makes it hard to meet the dynamic resource constraints in real-world applications. US-Net (Yu & Huang, 2019) addresses the dynamic constraint problem by sharing weights among different subnetworks and optimizing them simultaneously. But it only considers reducing network width and the performance drops dramatically as computational resource goes down.

β 1 × C 1 × β 2 × C 2 × K × K × H × W ,

Instead of only focusing on C 1 × C 2 , we shift our attention to reducing H × W in Eq. 1, i.e., lowering input resolution for the following three reasons. First, reducing H and W can reduce the computational cost without making any adjustments to the network structure. Therefore, it does not require further hardware optimization (Han et al., 2016) or careful structure tuning (Ma et al., 2018) . Second, balancing between input resolution and network width will produce better accuracy-efficiency tradeoffs as shown in Figure 1 . Third, different resolutions contain different information. Lower resolution images may contain more global structures while higher resolution ones may encapsulate more fine-grained patterns. Several works (Chen et al., 2019b; Sun et al., 2019; Ke et al., 2017) have explored such multi-scale representation learning. However, they resort to a multi-path structure, which is unfriendly to parallelization (Ma et al., 2018) .

Motivated by the discussion above, we propose a unified framework to mutually learn from network width and input resolution. Our framework is able to find the optimal width-resolution configuration under certain constraint, and captures multi-scale representations by sharing weights. In addition, it is compatible with any network structures and does not need further optimization and retraining.

3. METHODOLOGY:



3.1. PRELIMINARY:

Our framework leverages the training techniques in US-Net (Yu & Huang, 2019) . Therefore, we first introduce these techniques in this section to make this paper self-contained.

Sandwich Rule. US-Net trains a network that is executable at any resource constraint. The solution is to randomly sample several network widths for training and accumulate their gradients for optimization. However, the performance of all the sub-networks is bounded by the smallest width (e.g., 0.25×) and the largest width (e.g., 1.0×). Thus, the authors introduce the sandwich rule that is always sampling the smallest and largest width plus two random widths for each training iteration.

Inplace Distillation. Knowledge distillation (Hinton et al., 2015) is an effective method to transfer knowledge from a teacher network to a student network. Following the sandwich rule, since the largest network is sampled in each iteration, it is natural to use the largest network as the teacher to guide smaller sub-networks. The largest network is supervised by the ground truth label. This gives a better performance than only training with ground truth labels.

Post-Statistics of Batch Normalization. US-Net proposes that each sub-network needs their own batch normalization (BN) statistics (mean and variance), but it is insufficient to store the statistics of all the sub-networks. As a result, US-Net collects BN statistics for the desired sub-network after training. Experimental results show that 2,000 samples are sufficient to get accurate BN statistics, so this procedure is very efficient.

3.2. MUTUAL LEARNING FROM WIDTH AND RESOLUTION:

As discussed in Section 2, different resolutions contain different information. We want to take full advantage of this attribute to learn robust representations. In US-Net, sandwich rule is proposed to train a network that is executable at any width. However, we view it as a scheme of mutual learning (Zhang et al., 2018b) . Since networks with different widths share weights with each other, larger networks can take advantage of the features captured by smaller networks. Also, smaller networks can benefit from the stronger representation ability of larger networks. In light of this, we propose to mutually learn from network widths and resolutions. (2)

For the other sub-networks, we randomly pick an input resolution from {224, 196, 160, 128} and train it with the output of the full-network. The loss for the i-th sub-network is

EQUATION

where KLDiv is the Kullback-Leibler divergence. The total loss is the summation of the fullnetwork and sub-networks, i.e.,

EQUATION

The reason for training the full-network with the highest resolution is that the highest resolution contains more details. Also, the full-network has the strongest learning ability and captures the detailed dicriminatory information from the image data. We experiment with randomly selecting resolutions for all four widths, but it yields worse performance.

How mutual learning works. In this part, we are going to explain why the proposed framework can mutually learn from different widths and resolutions. For simplicity, we only consider two network widths 0.3× and 0.8×, and two resolutions 128 and 192 in this example. As shown in Figure 3 , sub-network 0.3× selects input resolution 128, sub-network 0.8× selects input resolution 192. Then we can define the gradients for sub-network 0.3× and 0.8× as grad W0:0.3,,I R=128 and grad W0:0.8,,I R=192 , respectively. Since 0.8× shares weights with 0.3×, we can decompose its gradient as grad W0:0.8,,I R=192 = grad W0:0.3,,I R=192 + grad W0.3:0.8,,I R=192 . The gradients of the two networks are accumulated during training, and the total gradients are computed as grad = grad W0:0.3,,I R=128 + grad W0:0.8,,I R=192 = (grad W0:0.3,,I R=128 + grad W0:0.3,,I R=192 ) + grad W0.3:0.8,,I R=192 .

(5)

Therefore, the gradient for sub-network 0.3× is (grad W0:0.3,,I R=128 + grad W0:0.3,,I R=192 ), which enables it to capture multi-scale representations from different input resolutions. Thanks to the random sampling of network width, every sub-network is able to learn multi-scale representations in our framework.

Model Inference. The trained model is executable at various width-resolution configurations. The goal is to find the best configuration under a particular resource constraint. A simple way to achieve this is via query table. Specifically, we sample network width from 0.25× to 1.0× with a stepsize of 0.05×, and sample network resolution from {224, 192, 160, 128}. We test all these widthresolution configurations on a validation set and choose the best one under a given constraint (FLOPs or latency). Since there is no retraining, the whole process is once for all.

4. EXPERIMENTS:

In this section, we first present our results on ImageNet classification (Deng et al., 2009) . We conduct extensive experiments to illustrate the effectiveness of the proposed framework. Next, we fine-tune the pre-trained model on popular transfer learning datasets to demonstrate the robustness and generalization ability of the learned representations. We further apply our framework to COCO object detection and instance segmentation. To the best of our knowledge, we are the first to benchmark arbitrary-constraint dynamic networks on detection and instance segmentation.

4.1. IMAGENET CLASSIFICATION:

We compare our framework with US-Net and independently-trained networks on the ImageNet dataset. We evaluated our framework on two popular light-weight structures, MobileNet v1 (Howard et al., 2017) and MobileNet v2 (Sandler et al., 2018) . These two networks also represent non-residual and residual structures respectively. We follow the training setting in US-Net (Yu & Huang, 2019) .

Please refer to the details in Appendix A.1.

Compare with US-Net. We first compare our framework with US-Net on MobileNet v1 and Mo-bileNet v2 backbones. The Accuracy-FLOPs curve is shown in Figure 4 . We can see that our framework consistently outperforms US-Net on both MobileNet v1 and MobileNet v2 backbones. Specifically, we achieve significant improvement under small computational cost. This is because our framework can find a better balance between network width and resolution. For example, if the resource constraint is 150 MFLOPs, US-Net reduces the width to 0.5× but our framework selects a balanced configuration of 0.7× -160. Moreover, our framework even improves on the full configuration (1. 0× -224) . This demonstrates that our framework is able to learn multi-scale representations by our mutual learning scheme, which further boosts the performance.

Compare with US-Net plus resolution. As evident in Figure 1 , applying different resolutions to US-Net during inference can already achieve improvements over the original US-Net. We denote this method as US-Net+. However, US-Net+ cannot achieve the optimal width-resolution balance due to lack of learning. In Figure 5 , we plot the Accuracy-FLOPs curves of our framework and US-Net+ based on MobileNet v1 backbone, and highlight the selected input resolutions with different colors. We can see that as we decrease the FLOPs (569 → 468 MFLPs), our framework first reduces network width to meet the constraint while keeping the 224×224 resolution (red color line in Figure 5 ). After 468 MFLOPs, our framework selects lower input resolution (192) and then continues reducing the width to meet the constraint. On the other hand, US-Net+ cannot find such balance. It always slims the network width and uses the same (224) resolution as the FLOPs decreasing until it goes to really low. This is because US-Net+ does not incorporate input resolution into the learning framework. Simply applying different resolutions during testing cannot achieve the optimal width-resolution balance. Compare with independently trained networks. To demonstrate that our framework is able to capture multi-scale representations for each sub-network, we compare with different width-resolution configurations. MobileNets (Howard et al., 2017; Sandler et al., 2018) have reported the results under different configurations. However, they consider width and resolution as independent factors, thus cannot take full advantage of the features contained in different resolutions. We compare the performance of our framework and independently-trained MobileNets in Figure 6 . For MobileNet v1, widths are selected from {1.0×, 0.75×, 0.5×, 0.25×}, and resolutions are selected from {224, 192, 160, 128}, leading to 16 configurations in total. Similarly, MobileNet v2 selects configurations from {1.0×, 0.75×, 0.5×, 0.35×} and {224, 192, 160, 128}. Our framework is executable at any FLOPs between the lower bound and upper bound. From Figure 6 , our framework consistently outperforms MobileNets. Even at the same width-resolution configurations, we can achieve much better performance. This demonstrates that our framework not only finds the optimal widthresolution balance but also learns stronger representations by our mutual learning scheme. et al., 2017; Sandler et al., 2018) To further validate the effectiveness of our mutual learning scheme, we compare our results with the state-of-the-art EfficientNet (Tan & Le, 2019) . EfficientNet acknowledges the importance of balancing between network width, depth and resolution. But the authors also treat them as independent factors. They propose to grid search over these three dimensions to find the optimal configuration under certain constraint. Then the network can be scaled up to other constraints following this configuration. We compare with the best configuration they found for MobileNet v1 at 2.3 BFLOPs.

To meet this constraint, we train our framework with a width range of [1.0×, 2.0×], and select resolutions from {224, 256, 288, 320}. This makes our framework executable in the range of [0.57, 4.5] BFLOPs. The results are compared in Table 1 . Our framework finds similar configuration as EfficientNet, except that we arrive at a larger width since depth is not considered. But our framework achieves significantly better performance than EfficientNet. We attribute this to the mutual learning from width and resolution scheme. 

4.2. TRANSFER LEARNING:

To evaluate the representations learned by our framework, we further conduct experiments on three popular transfer learning datasets, Cifar-100 (Krizhevsky, 2009) , Food-101 (Bossard et al., 2014) and MIT-Indoor67 (Quattoni & Torralba, 2009) . Cifar-100 is superordinate-level object classification, Food-101 is fine-grained classification and MIT-Indoor67 is scene classification. Such large variety is suitable to evaluate the robustness of the learned representations. We compare our framework with US-Net and independently-trained MobileNet v1. We fine-tune all the models with a batch size of 256, initial learning rate of 0.1 with cosine decay schedule and a total of 100 epochs. Following the setting in ImageNet training, we use width range [0.25×, 1.0×] and resolutions {224, 192, 160, 128}. The results are shown in Figure 7 . Again, our framework achieves consistently better performance than US-Net and MobileNet. This verifies that our framework is able to learn well-generalized representations.

4.3. OBJECT DETECTION AND INSTANCE SEGMENTATION:

We also apply our framework to COCO object detection and instance segmentation (Lin et al., 2014) . Our experiments are based on Mask-RCNN-FPN (He et al., 2017; Lin et al., 2017) and MMDetection (Chen et al., 2019a) toolbox. We use VGG-16 (Simonyan & Zisserman, 2014) as backbone network to validate our proposed mutual learning scheme.

We first pre-train VGG-16 on ImageNet using US-Net and our framework respectively. 

5. CONCLUSION AND FUTURE WORK:

This paper highlights the importance of simultaneously considering both network width and input resolution for designing efficient network structures. A new framework is defined capable of mutually learning from network width and input resolution. The proposed framework is demonstrated to significantly improve inference performance per FLOP for various datasets and tasks. The simplicity and generality of the framework allows it to translate well to generic problem domains. This also make logical extensions readily available by adding other network dimensions, e.g., network depth, to the framework. The framework can also be extended to video input and 3D neural networks, where we can leverage both spatial and temporal information. 

:

Number of channels in a layer.

Figures and Tables:

Figure 1: Accuracy-FLOPs curve of US-Net+ and US-Net. US-Net+ means simply applying different resolutions to US-Net during testing.

Figure 3: Illustration of the mutual learning from width and resolution scheme.Training framework. Our proposed framework is presented in Figure2. We train a network where its width ranges from 0.25× to 1.0×. We first follow the sandwich rule to sample four sub-networks, i.e., the smallest (0.25×), the largest (1.0×) and two random ones. Then, unlike traditional ImageNet training with 224×224 input, we resize the input image to four resolutions {224, 196, 160, 128} and feed them into different sub-networks. We denote the weights of a sub-network as W 0:w , where w is the width of the sub-network and 0 : w means the sub-network adopts the first w × 100% weights of each layer of the full network. I R=r represents a r × r input image. Then N (W 0:w , I R=r ) represents a sub-network with width w and input resolution r × r. For the largest sub-network (i.e., the full-network in Figure2), we always train it with the highest resolution (224 × 224) and ground truth label y. The loss for the full network is loss f ull = CrossEntropy(N (W 0:1 , I R=224 ), y).(2)

Figure 4: Accuracy-FLOPs curves of our proposed framework and US-Net. (a) is based on Mo-bileNet v1 backbone. (b) is based on MobileNet v2 backbone.

Figure 6: Accuracy-FLOPs curve of our framework and independently-trained MobileNets. (a) is MobileNet v1 backbone. (b) is MobileNet v2 backbone. The results for different MobileNets configuration are taken from the paper(Howard et al., 2017;Sandler et al., 2018)

Figure 7: Accuracy-FLOPs curves on different transfer learning datasets.

Figure 8: Average Precision -FLOPs curves of our framework and US-Net. (a) is bounding box average precision. (b) is mask average precision.

where β 1 and β 2 are pruning ratios. Unstructured pruning methods(Han et al., 2015b;a)  remove the network connections which are thought unimportant. The outcome is a sparsely connected network, which is not efficient on standard libraries and needs

Table content: <table><tr><td>Randomly select a resolution from {224, 192, 160, 128} as input to each sub-224x224 192x192 128x128 160x160 224x224</td></tr></table>

MFLOPs on MobileNet v1 backbone. We follow the sandwich rule(Yu & Huang,  2019)  to sample 4 networks, i.e., upper-bound full width network (1.0×), lower-bound width net-

Table content: <table><tr><td>Figure 2: The training process of our proposed framework. The network width range is [0.25,</td></tr><tr><td>1.0], input resolution is chosen from {224, 192, 160, 128}. This can achieve a computation range</td></tr><tr><td>of [13, 569]</td></tr></table>

ImageNet Top-1 accuracy on MobileNet v1 backbone.

Table content: <table><tr><td>Model</td><td>Best Configuration</td><td colspan="2">FLOPs Top-1 Acc</td></tr><tr><td colspan="2">EfficientNet (Tan &amp; Le, 2019) d = 1.4, w = 1.2, r = 1.3</td><td>2.3B</td><td>75.6%</td></tr><tr><td>Ours</td><td>w = 1.6, r = 1.3</td><td>2.3B</td><td>77.1%</td></tr></table>

The network width range is [0.25×, 1.0×] and resolutions are {224, 192, 160, 128}. Then we fine-tune the pre-trained models on COCO. The FPN neck and detection head are shared among different sub-networks. For simplicity, we don't use inplace distillation. Rather, each sub-network is trained with the ground truth. The other training procedures are the same as training ImageNet classification. Following common settings in object detection, US-Net is trained with image resolution 1000 × 600. Our framework randomly selects resolutions from 1000 × {600, 480, 360, 240}. All models are trained on COCO 2017 training set and tested on COCO 2017 validation set with different image resolutions. The Average Precision (AP) results for object detection and instance segmentation under different computational cost are presented in Figure8. These results reveal that our framework significantly outperforms US-Net under all resource constraints. Specifically, on the full configuration (1.0×-600), US-Net achieves comparable results with independent VGG-16, while our framework performs much better. This again validates the effectiveness of our width-resolution mutual learning scheme. Please refer to Appendix A.4 for visual comparison results.

Table content: <table/>

MobileNet v1 backbone

Table content: <table><tr><td/><td/><td/><td/><td colspan="4">Table 3: MobileNet v2 backbone</td></tr><tr><td colspan="4">Model Config MFLOPs Top-1 Acc</td><td colspan="4">Model Config MFLOPs Top-1 Acc</td></tr><tr><td colspan="2">US-Net 1.0-224</td><td>569</td><td>71.7</td><td colspan="2">US-Net 1.0-224</td><td>300</td><td>71.5</td></tr><tr><td colspan="2">Ours 1.0-224</td><td>569</td><td>72.4</td><td colspan="2">Ours 1.0-224</td><td>300</td><td>72.9</td></tr><tr><td colspan="2">US-Net 0.5-224</td><td>150</td><td>62.9</td><td colspan="2">US-Net 0.55-224</td><td>97</td><td>65.0</td></tr><tr><td colspan="2">Ours 0.7-160</td><td>146</td><td>65.6</td><td colspan="2">Ours 0.75-160</td><td>89</td><td>67.6</td></tr><tr><td colspan="2">US-Net 0.15-224</td><td>21</td><td>33.8</td><td colspan="2">US-Net 0.35-224</td><td>59</td><td>62.2</td></tr><tr><td colspan="2">Ours 0.25-160</td><td>21</td><td>50.1</td><td colspan="2">Ours 0.7-128</td><td>57</td><td>64.2</td></tr><tr><td colspan="4">Table 4: MobileNet v1 backbone</td><td colspan="4">Table 5: MobileNet v2 backbone</td></tr><tr><td>Model</td><td colspan="3">Config MFLOPs Top-1 Acc</td><td>Model</td><td colspan="3">Config MFLOPs Top-1 Acc</td></tr><tr><td colspan="2">MobileNet 1.0-224</td><td>569</td><td>70.6</td><td colspan="2">MobileNet 1.0-224</td><td>300</td><td>71.8</td></tr><tr><td>Ours</td><td>1.0-224</td><td>569</td><td>72.4</td><td>Ours</td><td>1.0-224</td><td>300</td><td>72.9</td></tr><tr><td colspan="2">MobileNet 0.75-192</td><td>239</td><td>67.3</td><td colspan="2">MobileNet 0.75-160</td><td>107</td><td>66.3</td></tr><tr><td>Ours</td><td>0.75-192</td><td>239</td><td>68.4</td><td>Ours</td><td>0.75-160</td><td>107</td><td>68.3</td></tr><tr><td colspan="2">MobileNet 0.25-128</td><td>13</td><td>42.8</td><td colspan="2">MobileNet 0.75-128</td><td>69</td><td>63.2</td></tr><tr><td>Ours</td><td>0.25-128</td><td>13</td><td>45.5</td><td>Ours</td><td>0.75-128</td><td>69</td><td>64.8</td></tr><tr><td colspan="4">Table 6: Object Detection</td><td colspan="4">Table 7: Instance Segmentation</td></tr><tr><td colspan="4">Model Config BFLOPs Box AP</td><td colspan="4">Model Config BFLOPs Mask AP</td></tr><tr><td colspan="2">VGG-16 1.0-600</td><td>110</td><td>32.6</td><td colspan="2">VGG-16 1.0-600</td><td>110</td><td>30.4</td></tr><tr><td colspan="2">US-Net 1.0-600</td><td>110</td><td>32.3</td><td colspan="2">US-Net 1.0-600</td><td>110</td><td>30.5</td></tr><tr><td colspan="2">Ours 1.0-600</td><td>110</td><td>33.2</td><td colspan="2">Ours 1.0-600</td><td>110</td><td>31.4</td></tr><tr><td colspan="3">US-Net 0.45-600 22.7</td><td>29.1</td><td colspan="3">US-Net 0.45-600 22.7</td><td>27.6</td></tr><tr><td colspan="3">Ours 0.75-360 22.2</td><td>30.7</td><td colspan="3">Ours 0.75-360 22.2</td><td>28.8</td></tr><tr><td colspan="2">US-Net 0.25-360</td><td>2.5</td><td>23.6</td><td colspan="2">US-Net 0.25-360</td><td>2.5</td><td>22.5</td></tr><tr><td colspan="2">Ours 0.25-360</td><td>2.5</td><td>25.9</td><td colspan="2">Ours 0.25-360</td><td>2.5</td><td>24.5</td></tr></table>

Peer review (iclr2020):
The authors propose a new training regime for multi-resolution slimmable networks. Their approach is based on US-Net training technique but in addition to sampling from different network widths they also sample from different input resolutions and show that using multi-scale inputs improves the top-1 accuracy on ImageNet comparing to US-Net or MobileNet v1/v2 within the same resource constraints. Pros: + The authors correctly identify input resolution as one of the aspects of lightweight network design that is often overlooked + They propose a practically viable training scheme that can be used to train & select networks given resource constraints + The paper is well written and includes many insightful experimental findings Con: The authors specify the mutual learning from width and resolution as their main contribution. They insist that treating input resolution independently from network structure is what distinguishes previous work from the newly suggested technique. But the paper doesn*t include extensive experimental comparisons with the approaches that treat input resolution independently. Thus its claim that joint width/resolution sampling is beneficial comparing to independent approaches is somewhat unfounded. For example, the authors show that MobileNet with 1.0-224 config (no sampling from widths nor from input resolutions during training) is outperformed by their network with 1.0-224 config (which effectively samples only from input resolutions during training). This is not surprising as one can view sampling from input resolutions as an equivalent to data augmentation. The importance of data augmentation is well known, so to prove the proposed mutual learning is beneficial the authors would need to compare against the networks that were trained using this multi-scale data augmentation. Figure 5 has a similar comparison but the only multi-resolution baseline there is US-Net+ which isn*t using multi-resolution images in training. The paper would greatly benefit from adding such comparisons and proving they are not marginal. On rating: I*d summarize the idea of this paper as A) US-Net + B) multi-scale data augmentation + C) selecting the best network based on both input resolution and width to achieve optimal performance within resource constraints. Although C is practical and novel contribution, it is also quite straightforward. I would like to see authors response on how their approach differs from US-Net + multi-scale data augmentation for training and how/why this works better.