A closer look at network resolution for efficient network design
Authors: Taojiannan Yang
Sijie Zhu
Yan Shen
Mi Zhang
Andrew Willis
Chen Chen
Venue:  (2020)
Keywords: deep learning
computer vision
efficient network design
dynamic neural networks

Abstract: There is growing interest in designing lightweight neural networks for mobile and embedded vision applications. Previous works typically reduce computations from the structure level. For example, group convolution based methods reduce computations by factorizing a vanilla convolution into depth-wise and point-wise convolutions. Pruning based methods prune redundant connections in the network structure. In this paper, we explore the importance of network input for achieving optimal accuracy-efficiency trade-off. Reducing input scale is a simple yet effective way to reduce computational cost. It does not require careful network module design, specific hardware optimization and network retraining after pruning. Moreover, different input scales contain different representations to learn. We propose a framework to mutually learn from different input resolutions and network widths. With the shared knowledge, our framework is able to find better width-resolution balance and capture multi-scale representations. It achieves consistently better ImageNet top-1 accuracy over US-Net under different computation constraints, and outperforms the best compound scale model of EfficientNet by 1.5%. The superiority of our framework is also validated on COCO object detection and instance segmentation as well as transfer learning.

Under review as a conference paper at ICLR 2020
A CLOSER LOOK AT NETWORK RESOLUTION FOR EFFI- CIENT NETWORK DESIGN
Anonymous authors Paper under double-blind review
ABSTRACT
There is growing interest in designing lightweight neural networks for mobile and embedded vision applications. Previous works typically reduce computations from the structure level. For example, group convolution based methods reduce computations by factorizing a vanilla convolution into depth-wise and point-wise convolutions. Pruning based methods prune redundant connections in the network structure. In this paper, we explore the importance of network input for achiev- ing optimal accuracy-efficiency trade-off. Reducing input scale is simple yet effective way to reduce computational cost. It does not require careful network module design, specific hardware optimization and network retraining after prun- ing. Moreover, different input scales contain different representations to learn. We propose a framework to mutually learn from different input resolations and network width:" With the shared knowledge, our framework is able to find bet- ter width-resolution balance and capture multi-scale representations. It achieves consistently better ImageNet top-1 accuracy over US-Net (Yu & Huang 2019) un- der different computation constraints, and outperforms the best compound scale model of EfficientNet (Tan & Le 2019) by 1.5%. The superiority of our frame- work is also validated on COCO object detection and instance segmentation as well transfer learning.
INTRODUCTION
Deep neural networks have triumphed over various per-
ception tasks such image classification (He et al. 2016)]
Huang et al.] 2017b Simonyan & Zisserman) (0014), ob-
Imsgehel Tap-1 Accuracy (%)
ject detection (Ren- al. 2015 Redmon al.
2016)
50
and semantic segmentation (Chen et al.] 2017). How-
ever, deep networks usually require large computational resources, making them hard to deploy on mobile devices and embedded systems. This motivates research in reduc-
ing the redundancy in deep neural networks or designing lightt structures. Specifically, MobileNet (Howard et al. (017)) factorizes standard 3 x 3 convolution into 3x3 depth-wise convolution and a x point-wise convo- lution. ShuffleNet (Zhang et al. 201 8a) uses X 1 group convolution to further reduce computations and proposes the shuffle operation to help information flow among dif- ferent groups. Another kind of approach is to prune re- dundant connections in the networks. Unstructured prun- ing methods (Han et al., 220 a) delete network connec- tions which are thought unimportant. Structured pruning methods (Li al. 2016: Anwar et al. 2017 Lemaire et al., 2018) prune the entire filters and feature maps. Recently, dynamic networks have been introduced which adopt single model to meet varying computing resource constraints. For example, Huang et al. (2017a) proposes a multi-branch structure where early prediction can be made based on the current confidence and resource constraints. Kim et al. (0018) builds a network-
20
800
FLOPs
Figure 1: Accuracy-FL.OPs curve of US-Net+ and US-Net. US-Net+ means simply applying different resolutions to US-Net during testing.
Number of channels in layer.

---

Under review as a conference paper at ICLR 2020
in-network structure for multiple resources. Yu et al. (2019) shares weights among different sub- networks and each sub-network has its own batch normalization layer. US-Net (Yu & Huang 2019) proposes to compute batch normalization statistics after training, and introduces two training tech- niques to train a network that is executable at any network widths. However, these works approach the problem only from network structure perspective, ignoring the importance of network input.
Reducing network input dimension (e.g., lowering image resolution) is a straightforward way to re- duce computational cost. It can be applied to any network structure during testing. Besides, balanc- ing between input resolution and network width can achieve better sccuracy-efficiene tradeoffs. For example, to meet dynamic resource constraint from 13 MFLOPs 569 MFLOPs on MobileNet vI backbone, US-Net (Yu & Huang] 2019) needs network width range of [0.05x, 1.0x] given a x24 input resolution. This constraint can also met via network width of [0.25x, 1.0x] by adjusting the input resolution from {224, 192, 160, 128) during test time. We denote the latter model as US-Net+. As shown in Figure TT simply combining different resolutions with network widths during testing can already achieve better accuracy -eferaccefffcinncyy tradeoff than US-Net without additional efforts. EfficientNet (Tan & Le 2019) also acknowledges the importance of balancing be- tween depth, width and resolution. Moreover, reducing input resolution does not necessarily harm the performance, and may sometimes even be beneficial, CCin et al. (2019) points out that lower im- age resolution may produce better detection accuracy by reducing focus on redundant details. Chen let al.] (2019b) claims that different scaled images contain different information. [Sun et al.) (2019) fuses multi-scale features with a multi- framework to learn robust representations.
Inspired by the observations above, we propose a unified framework to mutually learn from input resolution and network width. Our framework is able achieve the optimal width-resolution bal- ance under certain resource constraint. Since we share weights among different network widths, each network is able to capture multi-scale representations without any adjustments in the network structure. The whole framework is shown in Figure 2
We summarize our contributions as follows.
1. We highlight the importance of input resolution for efficient network design. Previous works either ignore it or treat it independently from network structure. In contrast, we propose a unified framework to mutually learn from input resolution and network width.
2. Our framework is simple and general. It is compatible with any network structures (i.e., network- agnostic) and is as simple as training an independent network. Moreover, it does not change network structure, meaning it can benefit from other add-on structural optimizations.
We conduct extensive experiments to verify the effectiveness of our framework, Our frame- work significantly outperforms US- US-MobillNeess (Yu & Huang 2019) and independently-rraieee Mo- bileNets on ImageNet classification, as well as COCO object detection and instance segmentation. The experimental results on popular transfer learning datasets also demonstrate the generalization ability of the learned representations of our model.
2 RETHINKING REDUCING COMPUTATIONAL COST
In this section, we explore the importance of input resolution for efficient network design. The computational cost of a vanilla convolution is
C XxxXxXXXX W
(1)
where C₁ and C2 are the number of input and output channels, K is kernel size, H and W are output feature map size. Most previous works only focus on reducing computational cost from the structure level, that is, reducing the number of channels C₁ x C2 or reducing kernel size K. MobileNets (Sandler et al., 2018 Howard et al., (017) decompose the vanilla convolution into a depth-wise convolution and a 1x] convolution, reducing the cost to (C1 x K x K + G1 X C2) x convolution into several groups, shrinking the computational cost to (C1 X K x K + C1 X (C2/9)) x H x W, where 9 is the number of groups. However, increasing the number of groups leads to high memory access cost (MAC) (Ma et al. 2018). which makes the network inefficient in practical applications. Pruning based methods aim to prune redundant connections in the network, cutting the computational cost to 81 C1 x 3g x C2 K x K x H x W. where 81 and Bg are pruning ratios. Unstructured pruning methods (Han et al. 2015b a) remove the network connections which are thought unimportant. The outcome is sparsely connected network, which is not efficient standard libraries and needs
H W. ShuffleNet (Zhang et al. 2018a) further divides
2

---

Under review as a conference paper at ICLR 2020
224x224
224x224
192x192
160x160
128x128
Randomly sclect resolution from (224, 192, 160, 128)
Input each sub-network
Full network (1.0x)
Sub-nebworkk (()x))
Sub network 2 (i;
Sub network (0.25*]
Width channels)
the full network)
Ground truth
KL
KL
Figure 2: The training process of our proposed framework. The network width range is [0.25, 1.0], imput resolution is chosen from (224, 192. 160, 128). This can achieve a computation range MobileNet v1 backbone. We follow the sandwich rule (Yu & Huang. 2019) to sample 4 networks, i.e., upper-bound full width network (1.0x), lower-bound width net- work (0.25x), ), and two random width ratios 011, 02 E (0.25, 1). For the full-network, constantly choose 224 2224 resolution. For the other three sub-networks, we randomly select its input resolu- tion. The full-network is optimized with the ground-truth label. Sub-networks are optimized with the prediction of the full-network. Weights are shared among different networks to facilitate mutual learning. CE: Cross Entropy loss. KL: KL Divergence loss.
of [13, 569] MFLOPs
specialized software and hardware optimization (Han al. 2016). Structured pruning methods (Li cct al.] [2166] Anwar et al. 2017 Lemaire et al., 2018) prune the entire filters and feature maps. The pruned network maintains structured but needs retraining retain performance after pruning, which makes hard to meet the dynamic resource constraints in real-world applications. US-Net (Yu & Huang 2019) addresses the dynamic constraint problem by sharing weights among different sub- networks and optimizing them simultancously. But it only considers reducing network width and the performance drops dramatically computational resource goes down.
Instead of only focusing on C x C2. we shift our attention to reducing H x W in Eq. | i.e.. lowering input resolution for the following three reasons. First, reducing H and W can reduce the computational cost without making any adjustments to the network structure. Therefore, it does require further hardware optimization (Han et al., (016] or careful structure tuning (Ma) et al., 2018). Second, balancing between input resolution and network width will produce better accuracy-efficiency tradeoffs as shown Figure [ Third, different resolutions contain different information. Lower resolution images may contain more global structures while higher resolution ones may encapsulate more fine-grained patterns. Several works (Chen et al. 2019b Sun et al.) 2019 Ke et al.) (017) have explored such multi-scale representation learning. However, they resort to multi-path structure, which is unfriendly to parallelization (Ma et al., 2018).
Motivated by the discussion above, propose unified framework mutually learn from network width and input resolution. Our framework is able to find the optimal width-resolution configuration under certain constraint, and captures multi-scale representations by sharing weights. In addition, is compatible with any network structures and does not need further optimization and retraining.
METHODOLOGY
3.1 PRELIMINARY
Our framework leverages the training techniques in US-Net (Yu & Huang 2019). Therefore, first introduce these techniques this section to make this paper self-contained.
Sandwich Rule. US-Net trains network that is executable at any resource constraint. The solu- tion to randomly sample several network widths for training and accumulate their gradients for optimization. However, the performance of all the sub-networks is bounded by the smallest width (e.g., 0.25x) and the largest width (c.g.. 1.0x). Thus, the authors introduce the sandwich rule that is always sampling the smallest and largest width plus two random widths for each training iteration.
3

---

Under review as a conference paper at ICLR 2020
Inplace Distillation. Knowledge distillation (Hinton et al. 2015) is effective method to transfer knowledge from a teacher network to a student network. Following the sandwich rule, since the largest network is sampled in each iteration, it is natural to use the largest network as the teacher to guide smaller sub-networks. The largest network is supervised by the ground truth label. This gives better performance than only training with ground truth labels.
Post-Statistics of Batch Normalization. US-Net proposes that each sub-network needs their own batch normalization (BN) statistics (mean and variance), but it is insufficient to store the statistics of all the sub-networks. As a result, US-Net collects BN statisties for the desired sub-network after training. Experimental results show that 2,000 samples are sufficient to get accurate BN statistics, so this procedure is very efficient.
3.2 MUTUAL LEARNING FROM WIDTH AND RESOLUTION
As discussed in Section 2 different resolutions contain different information. We want to take full advantage of this attribute to learn robust representations. In US-Net, sandwich rule is pro- posed to train a network that is executable at any width. However, we view it as a scheme of mutual learning (hang al.) 2018b). Since networks with different widths share weights with each other, larger networks can take advantage of the features captured by smaller net- works. Also, smaller networks can benefit from the stronger representation ability of larger net- works. In light of this, we propose to mutually learn from network widths and resolutions. Training framework. Our proposed framework is presented in Figure 2 We train a network where its
koy sale
width ranges from 0.25x to 1.0x, We first follow the sandwich rule to sample four sub-networks, i.e., the smallest (0.25x). the largest (1.0x) and two ran- dom ones. Then, unlike traditional ImageNet train- ing with 224 224 input, we resize the input image to four resolutions (224, 196, 160, 128] and feed them into different sub-networks. We denote the weights of uub-network as Wo.w, where is the width of the sub-network and 0 : w means the sub-network adopts the first w 100% weights of each layer of the full network. RR= represents a r X r input im- age. Then N( O:ar: IR=r) represents a sub-network with width w and input resolution r X r. For the largest sub-network (i.e., the full-network in Figure p. we always train it with the highest resolution (224 x 224) and ground truth label y. The loss for the full network is
leamiig (Fg.
Figure 3: Illustration of the mutual learning from width and resolution scheme.
loss CrossEntroppy(N (N (Na.roooyooNWWo........ (Wo1, y).
For the other sub networks, we randomly pick an input resolution from {224, 196, 160, 128) and train it with the output of the nell-network. The loss for the i-th sub-network
loss,ub, = KL.io(N I N(Wo.. I==224)),
(3)
where KLDiv the Kullback-Leible divergence. The total loss is the summation of the full- network and sub-networks, i.e.,
loss loss full + Σ Losssub.
(4)
The reason for training the full-network with the highest resolution is that the highest resolution contains more details. Also, the full-network has the strongest learning ability and captures the detailed dicriminatory information from the image data. We experiment with randomly selecting resolutions for all four widths, but it yields worse performance.
How mutual learning works. In this part, we are going to explain why the proposed framework can mutually learn from different widths and resolutions. For simplicity, we only consider two network widths 0.3x and 0.8x, and two resolutions 128 and 192 in this example. As shown in Figure 3, sub-network 0.3x seleets input resolution 128, sub-network 0.8x selects input resolution 192. Then can define the gradients for sub-network 0.3x and 0.8x as gradwa 22-12 and
4

---

Under review as a conference paper at ICLR 2020
gradWo.o.s..t It. respectively. Since 0.8x shares weights with 0.3x, decompose its gra- dient as gradWo.o.s.o. II== = grodwooa, ne + gradwa.... m The gradients of the two networks are accumulated during training, and the total gradients are computed
grad = gradwoo... 12-12 + gradwwo...
(gradwao.s, J, I +
+ gradw 6 3..ss A
Therefore, the gradient for sub-network 0.3x is a. + gradwoo.s Jn which enables it to capture multi-scale representations from different input resolutions. Thanks to the random sampling of network width, every sub-network is able to learn multi-scale representations in our framework.
Model Inference. The trained model is executable various width-resolution configurations. The goal is to find the best configuration under a particular resource constraint. A simple way to achieve this is via query table. Specifically, we sample network width from 0.25x 1.0x with step- size of 0.05x, and sample network resolution from {224, 192, 160, 128]. We test all these width- resolution configurations on validation set and choose the best one under a given constraint (FLOPs
or latency). Since there is no retraining, the whole process is once for all.
EXPERIMENTS
In this section, we first present our results on ImageNet classification (Deng et al.] 2009). We conduct extensive experiments to illustrate the effectiveness of the proposed framework. Next, we fine-tune the pre-trained model on popular transfer learning datasets to demonstrate the robustness and generalization ability of the learned representations. We further apply our framework to COCO object detection and instance segmentation. To the best of our knowledge, we the first to benchmark arbitrary-constrain dynamic networks on detection and instance segmentation.
4.1 IMAGENET CLASSIFICATION
We compare our framework with US-Net and independently-trained networks the ImageNet dataset. We evaluated our framework on two popular light-weight structures, MobileNet vl (Howard ct al. and MobileNet v2 (Sandler al. 2018). These two networks also represent resiuall and residual structures respectively. We follow the training setting in US-Net Yu & Huang. 2019). Please refer to the details in Appendix A.1
Compare with US-Net. We first compare our framework with US-Net on MobileNet v1 and Mo- bileNet v2 backbones. The Aceuracy-FLOPs curve shown in Figure 4 We can see that our framework consistently outperforms US-Net on both MobileNet v1 and MobileNet v2 backbones. Specifically, we achieve significant improvement under small computational cost. This is because framework can find better balance between network width and resolution. For example, if the resource constraint is 150 MFLOPs, US-Net reduces the width to 0.5x but our framework selects a balanced configuration of 0.7 160. Moreover, our framework even improves the full configura- tion (1.0x- 224). This demonstrates that our framework is able to learn multi-scale representations by our mutual learning scheme, which further boosts the performance.
Compare with US-Net plus resolution. As evident in Figure D applying different resolutions to US-Net during inference can already achieve improvements over the original US-Net. We denote this method as US-Net+. However, US-Net+ cannot achieve the optimal width-resolution balance due to lack of learning. Net+ based on MobileNet vI backbone, and highlight the selected input resolutions with different colors. We can see that as we decrease the FLOPPs (569- - 468 MFLPs), our framework first reduces network width to meet the constraint while keeping the 224 224 resolution (red color line in Figure 5). After 468 MFLOPs, our framework selects lower input resolution (192) and then continues reducing the width to meet the constraint. On the other hand, US-Net+ cannot find such balance. always slims the network width and uses the same (224) resolution as the FLOPs decreasing until it goes to really low. This is because US-Net+ does not incorporate input resolution into the learning framework. Simply applying different resolutions during testing cannot achieve the optimal width-resolution balance.
F we plot the Accuracy-FLoO -LOPs curves of our framework and US-
5

---

Under review as a conference paper at ICLR 2020
74
US Net Oura
US Net
Oura
ImageNat Top-1 Accuracy (%)
70
ImageNet Top-1 Accuracy (%)
72
70
60
68
Model
MPLAAP
40-
US No
Oure
10-221
14-234
S69
569
717
9244
88
18.6
Madd
19:33
Oxeg
Topa Aue
US-Na
US.Nct
Ouze
6.15-234
0.5-251
0.7-160
150
146
21
629
353
65
1336
0.75 160
033-254
976
050
Otes
0.25-100
21
50.1
02-125
20
100
200
300
400
500
800
100
150
200
250
300
350
(Millions)
(Millions)
MobileNet vl backbone
(b) MobileNet v2 backbone
Figure 4: Accuracy-FLOPs curves of our proposed framework and US-Net. (a) is based on Mo-
bileNet vI backbone. (b) is based on MobileNet v2 backbone.
80
ImageNet Top-1 Accuracy (%)
70-
Ours
USiNet
70-
US-Not
40
30-
Rsso9224
Rcso-152
Reso-180
20
Ruso-128
0
100
200
300
400
500
600
FLOPs (Millions)
Figure 5: The Accuracy-FLOPs curve is based on MobileNet vl backbone. We highlight the selected resolution under different FLOPs with different colors. For example, the solid green line indicates when the constraint range is [41, 215] MFLOPs, framework constantly selects input resolution 160 but reduces the width to meet the resource constraint. Best viewed color.
Compare with independently trained networks. To demonstrate that our framework is able to cap- ture multi-scale representations for each netwoe, we compare with different width-resolution configurations. MobileNets (Howard et al. 2017: Sandler et al. 2018) have reported the results under different configurations. However, they consider width and resolution independent factors, thus cannot take full advantage the features contained different resolutions. performance of our framework and independently-trainee MobileNets in Figure [ For MobileNet vl, widths are selected from {1.0x, 0.75x, 0.5x, 0.25 }. and resolutions are selected from (224, 192, 160, 128]. leading to 16 configurations in total. Similarly, MobileNet v2 selects configura- tions from (1.0x, 0.75x, 0.5x, 0.35x) and (224, 192, 160, 128). Our framework executable any FLOPs between the lower bound and upper bound. From Figure 6 our framework consis- tently outperforms MobileNets. Even at the same width-resolution configurations, we can achieve much better performance. This demonstrates that our framework only finds the optimal width- resolution balance but also learns stronger representations by our mutual learning scheme.
compare the
Table 1: ImageNet Top-1 accuracy on MobileNet vl backbone.
Model
Best Configuration
FLOPs
Top- I Acc
EfficientNet Tan & Le 2019)
1.4, T.
1.2,
1.3
2.3B
75.6%
Ours
1.6,1 1.3
2.3B
77.1 %

---

Under review as a conference paper at ICLR 2020
76
Modkshod
ImageNet Accuracy (8)
Imagetett Top- Accuracy (%)
70-
85
66
60-
85
MokilaNa
Madel
Cantg
1-6354
MCL.0%s
500
Top Are
20.5
80
MiNikNNe
Xidd
1..226
Coeng
302
Top-1 ACc
TLS
Dare
1.0-244
599
72.4
Ouzs
100221
300
72.9
50
Mokilnho
Oare
0.75-198
0.75-192
139
219
57.3
68.4
M.biNNe:
0.75-100
0.75-100
107
107
66.3
08.5
Mokuiehox
0:25-128
12.5
86
M.HINN::
0.75-125
68
63.2
45
Oars
0:25-288
45.5
Cuus
0.75-123
64.5
50
100
200
300
400
800
600
50
100
150
200
250
300
350
FLOPs (Millios)
FLOPs (Millions)
(a) MobileNet v1 backbone
(b) MobileNet v2 backhone
Figure 6: Accuracy-FLOPs curve of our framework and independennly--rriieed MobileNets. (a) MobileNet v1 backbone. (b) is MobileNet v2 backbone. The results for different MobileNets configuration are taken from the paper (Howard et al. 2
Sandler et al. 2018
To further validate the effectiveness of our mutual learning scheme. we compare our results with the state-of-the-art EfficientNet (Tan Le 2019). EfficientNet acknowledges the importance of bal- ancing between network width, depth and resolution. But the authors also treat them as independent factors. They propose to grid search over these three dimensions to find the optimal configuration under certain constraint. Then the network can be scaled up to other constraints following this con- figuration. We compare with the best configuration they found for MobileNet vl at 2.3 BFLOPs. To meet this constraint, olutions from (224, 256, 288, 320}. This makes our framework executable in the range [0.57, 4.5] BFLOPs. The results compared in Table I Our framework finds similar configuration as EfficientNet, except that we arrive at a larger width since depth not considered. But our frame- work achieves significantly better performance than EfficientNet. We attribute this the mutual learning from width and resolution scheme.
train our framework with width range of [1.0x, 2.0 and select res-
Cifar 100
Food 101
MIT Indoor
Top-1 Accuracy (%)
65
70
76
60
60
Oura
Durs
Outs
100
400
500
600
200
400
500
600
100
200
300
400
600
FLOPs (Millions)
Figure 7: Accuracy-FLOOPs curves on different transfer learning datasets.
4.2 TRANSFER LEARNING
To evaluate the representations learned by our framework, we further conduct experiments on three popular transfer learning datasets, 100 (Krizhevsky 2009), Food-101 (Bossard al.) 2014) and MIT-Indoor67 (Quattoni & Torralba 2009). Cifar-100 is superordinate-level object classifica- tion, Food-101 is fine-grained classification and MIT-Indoor67 is scene classification. Such large variety is suitable to evaluate the robustness of the learned representations. We compare our frame- work with US-Net and independently-traine batch size of 256, initial learning rate of 0.1 with cosine decay schedule and a total of 100 epochs. Following the setting in ImageNet training, we use width range [0.25x, 1.0x] and resolutions (224, 192, 160, 128). The results shown in Figure 7 Again, framework achieves consistently
MobileNet vl. We finettne all the models with a

---

Under review as a conference paper at ICLR 2020
better performance than US-Net and MobileNet. This verifies that our framework is able to learn well-generalized representations.
4.3 OBJECT DETECTION AND INSTANCE SEGMENTATION
We also apply our framework to COCO object detection and instance segmentation (Lin et al. 2014). Our experiments are based on Mask-RCNN-FPN (He et al., 2017: [in et al., 2017) and
MMDetection (Chen et al., 2019a) toolbox. We use VGG-16 (Simonyan Zisserman, (0144 as backbone network to validate our proposed mutual learning scheme.
We first pre-train VGG-16 on ImageNet using US-Net and our framework respectively. The net- work width range is |0.25x, 1.0x| and resolutions are (224, 192, 160, 128). Then we fine-tune the pre-trained models on COCO. The FPN neck and detection head are shared among different sub-networks. For simplicity, we don't I use inplace distillation. Rather, each sub-network is trained with the ground truth. The other training procedures are the same as training ImageNet classifi- cation. Following common settings in object detection, US-Net is trained with image resolution 1000 x 600. Our framework randomly selects resolutions from 1000 x {600, 480, 360, 240}. All models are trained COCO 2017 training set and tested COCO 2017 validation set with dif- ferent image resolutions. The Average Precision (AP) results for object detection and instance seg- mentation under different computational cost are presented in Figure These results reveal that framework significantly outperforms US-Net under all resource constraints. Specifically, on the full configuration (1.0x-600), US-Net achieves comparable results with independent VGG-16, while our framework performs much better. This again validates the effectiveness of our width-resollttin mutual learning scheme. Please refer Appendix A.4 for visual comparison results.
34
Outs US Not
32
30
US:NN1
Bounding Box Average Precition
Average Precision
VGG 16
Midel
10:600
Conda
XPLOP%
110
Box AP
32.6
24
VOG-16
BELOOS
24
10:400
02
US-Na
Our
1.05000
1.0-00
110
110
31.4
30.5
US-Nat
025-600
20.1
USt
0.45-000
22.7
27.6
22
Oum
075-360
22.2
36.7
Ours
0.75-360
22.2
28.8
UStNNtt
0.25.360
2.5
23.6
Net
0.21-360
2.5
22.5
Cuus
0.25360
2.5
25.9
Ours
0.25 360
2.5
24.5
20
40
60
80
100
20
40
80
100
120
(Billions)
FLOPs (Billions)
(a) Object Detection
(b) Instance Segmentation
Figure 8: Average Precision FLOPs curves of our framework and US-Net. (a) is bounding box average precision. (b) is mask average precision.
5 CONCLUSION AND FUTURE WORK
This paper highlights the importance of simultaneously considering both network width and input resolution for designing efficient network structures. A new framework is defined capable of mutu- ally learning from network width and input resolution. The proposed framework is demonstrated to significantly improve inference performance per FLOP for various datasets and tasks. The simplic- ity and generality of the framework allows it to translate well to generic problem domains. This also make logical extensions readily available by adding other network dimensions, e.g., network depth, to the framework. The framework can also be extended to vidco input and 3D neural networks, where we can leverage both spatial and temporal information.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies Computing Systems (JETC), 13(3):32, 2017.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative com- ponents with random forests. In European Conference Computer Vision, 2014.
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. 2019a.
arXiv preprint
Liang- Chang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834-848, 2017.
Yunpeng Chen, Haoqi Fang, Bing Xu, Zhicheng Yan, Yannis Kalantidis, Marcus Rohrbach, Shuicheng Yan, and Jiashi Feng. Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. arXiv preprint 190p::9940004999 2019b.
Ting- Wu Chin, Ruizhou Ding, and Diana Marculescu. Adascale: Towards real-time video object detection using adaptive scaling, arXiv preprint arXiv:1902.02910 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IFFE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffmman coding, arXiv preprint arXiv:1510.001 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015b.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Eie: efficient inference engine on compressed deep neural network. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pp. 243-254. IEEE, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255 020331, 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprini arXiv: 1704.04611, 2017.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Wein- berger. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXin::703.09844, 2017a
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017b.

---

Under review as a conference paper at ICLR 2020
Tsung- Wei Ke, Michael Maire, and Stella X Yu. Multigrid neural architectures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6665-6673, 2017.
Eunwoo Kim, Chanho Ahn, and Songhwai Oh. Nestednet: Learning nested sparse structures in deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8669-8678, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Carl Lemaire, Andrew Achkar, and Pierre-Marc Jodoin. Structured pruning of neural networks with budget-aware regularization. arXiv preprint arXiv:/811.0933 2018.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1608: 08710, 2016.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puler vision and paltern recognition, pp. 2117-2125, 2017.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer 2018.
Vision (ECCV), pp.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. 2009 IEEE Conference
Computer Vision and Pattern Recognition, pp. 413-420, 2009.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- -hinh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4520-4520, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXivv1409.1 1556, 2014.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5693-5703, 2019.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learn- ing Research, pp. 6105 64, Long Beach, California. USA, 09-15 Jun 2019. PMLR. URL html
http:/ httt:://proceedings mlr. wwwps//97/ta.......
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. arXiv preprint arXiv:1903.065
2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations, 2019. Utt.hhttps:://peeeeeee
net/ 1==
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848-6856, 2018a.
10

---

Under review as a conference paper at ICLR 2020
Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320- 4328, 2018b.
A APPENDIX
A.1 TRAINING DETAILS
pyperparameters. We follow the training settings US-Net (Yu & Huang 2019) when training on ImageNet. The batch size is 1024 on 8 GPUs. Initial learning rate is 0.5 and decreases to 0 with a cosine decay schedule. The momentum set to 0.9 and weight decay is 0.0001. All models are trained for 250 epochs using SGD optimizer.
Network width seales. In Figure 4 for MobileNet v1 backbone, our method uses width scale from [0.25, 1Jx and resolutions from [224, 192, 160, 128) while US-Net uses width scale from [0.05, l|x. For MobileNet v2 backbone, our method uses width scale from [0.7, 1]x and resolutions from (224, 192, 160, 128} while US-Net uses width scale from [0.35, 1]x. In transfer learning experiments, both our method and US-Net use width scale from [0.25, 1]x. Specifically, for US- Net, we adopt the officially released model on width range [0.25, 1]x and finetune it with width range [0.25, 1]x. In object detection and instance segmentation, we also utilize the same width range [0.25 I]x for our method and US-Net.
The reason for using different width scales in Figure is that both our approach and US-Net aim to train adaptive network meet the dynamic resource constraints test time. We compared ap- proach and US-Net using the Accuracy-FLOPs curves under the same dynamic resource constraint. For MobileNet vl backbone (Figure 4 (a)), the dynamic FLOPs constraint is [13, 569] MFLOPs. To meet this constraint, US-Net needs the width scale of [0.05, 1]x. However, our approach can meet this computation constraint by balancing between width scale [0.25, 1]x and resolutions {224, 192, 160, 128]. Similarly, for MobileNet v2 (Tigure(b)) (b)). the dynamic constraint is [59, 300] MFLOPs. US-Net needs width scale of [0.35, l]x, while we can meet this constraint by combining width [0.7. 1Jx and resolutions {224, 192, 160, 128). Therefore, integrating resolution in our framework gives more flcxibility to balance network width. This is one advantage that we illustrated in Figure|
In all the following experiments (US-Nett, transfer learning, object detection and instance segmen- tation), we use the same width scale [0.25, 1]x for both our method and US-Net. This setting is actually in favor of US-Net, because if the original US-Net(o.05.1) is used, which meets the same resource constraints as ours, performs worse.
A.2 CONTRIBUTION OF KL DIVERGENCE
In the experiments, we follow US-Net to train sub-networks using KI. divergence to have a fair comparison. In US-Net, the author claims that training with KI. divergence is better than training with ground truth label. In this section, we give some quantitative results to show the contribution of KL divergence to the overall performance. The training setting is the same training Ima- geNet, except that sub-networks are trained with the ground truth rather than the soft labels from the full-network. As shown in Figure 9 training with KL divergence (i.e., using soft labels) achieves marginal improvement over using the ground truth. The differences tend to be larger when FLOPs goes down. The reason might be that small sub-networks have limited learning capacity and it is easier to learn using soft labels from the teacher rather than using one-hot ground truth labels.
A.3 VGG-16 IMAGENET PRE-TRAINING
We use VGG-16 as the backbone network for object detection and instance segmentation. We first pretrain VGG on ImageNet following US-Net and our framework. The batch size is set to 512. The initial learning rate is 0.01 and decreases to with a cosine decay schedule. Both frameworks trained for 100 epochs. The Accuracy-FLOPs curves shown in Figure I00 We that our framework still constantly outperforms US-Net on such large neural network. This further demonstrates the effectiveness and generalization ability of framework.
11

---

Under review as a conference paper at ICLR 2020
75
Ours w/o KL
Ours w/ KL
ImageNet Top-1 Accuracy (%)
65
Model
Config
MFLOPs
Top-I Acc
55
w/o KL.
1.0-224
569
72.3
w/ KL
1.0-224
569
72.4
KI. 0.8-192
269
69.1
w/ KI.
0,8-192
269
68.5
45
w/o KL.
0.25-160)
21
49.0
KI.
0.25-160
50.1
40
0
100
200
300
400
500
600
FLOPs (Millions)
Figure 9: Accuracy-FLOPs curves of our framework w/ and w/o KL divergence using the MobileNet vI backbone.
75
US-Net
70
Ours
ImageNet Top-1 Accuracy (%)
65
60
Model
Config
BFLOPs
Top-1 Acc
55
US-Net
1.0-224
15.5
71.2
Ours
1.0-224
15.5
72.0
US-Net
0.75-224
8.8
69.5
50
Ours
0,85-192
70.1
US-Net
0.45-224
66.3
Ours
0.65-160
67.5
45
40
10
14
16
FLOPs (Billions)
Figure 10: Accuracy-FLOPs curves of our framework and US-Net VGG-16 backbone.
A.4 OBJECT DETECTION AND INSTANCE SEGMENTATION EXAMPLES
We show some visualization examples in Figure 11 We tested model configurations 1.0x- 600 and 0.5x-480. The upper pairs are 1.0x the lower pairs are 0.5x-480. From Figure [T we can see that our framework is able to detect both small-scale and large-scale objects in both configurations while US-Net fails. This is because our mutual learning scheme enables each sub- network to capture multi-scale representations.
A.5 LARGER TABLES FOR BETTER READABILITY
In this subsection, display larger version of the tables in the paper for better readability. Table and Table show comparison results of our method and US-Net on ImageNet. Table and Table I5 present results of our approach and independently-trriieed MobileNets ImageNet. Table and Table compare the results on COCO object detection and instance segmentation.
12

---

Under review as a conference paper at ICLR 2020
Table 2: MobileNet v1 backbone
Table 3: MobileNet v2 backbone
Model
Config
MFLOPs
Top-1 Acc
Model
Config
MFLOPs
Top-I Acc
US-Net
1.0-224
569
71.7
US-Net
1.0-224
300
71.5
Ours
1.0-224
569
72.4
Ours
1.0-224
300
72.9
US-Net
0,5-224
150
62.9
US-Net
0.55-224
97
65.0
Ours
0.7-160
146
65.6
Ours
0.75-160
89
67.6
US-Net
0.15-224
21
33.8
US-Net
0.35-224
59
62.2
Ours
0.25-160
50.1
Ours
0.7-128
57
64.2
Table 4: MobileNet vl backbone
Table 5: MobileNet v2 backbone
Model
Config
MFLOPs
Top-1 Acc
Model
Config
MFLOPs
Top-1 Acc
MobileNet
1.0-224
569
70.6
MobileNet
1.0-224
300
71.8
Ours
1.0-224
569
72.4
Ours
1.0-224
300
72.9
MobileNet
0.75-192
239
67.3
MobileNet 0.75-160
107
66.3
Ours
0.75-192
239
68.4
Ours
0.75-160
107
68.3
MobileNet
0.25-128
13
42.8
MobileNet 0.75-128
69
63.2
Ours
0.25-128
13
45.5
Ours 0.75-128
69
64,8
Table Object Detection
Table Instance Segmentation
Model
Config
BFLOPs
Box AP
Model
Config
BFLOPs
Mask AP
VGG-16
1.0-600
110
32.6
VGG-16
1.0-600
110
30.4
US-Net
1,0-600
110
32.3
US-Net
1.0-600
110
30.5
Ours
1.0-600
110
33.2
Ours 1.0-600
110
31.4
US-Net
0.45-600
22.7
29.1
US-Net
0.45-600
22.7
27.6
Ours
0.75-360
22.2
30.7
Ours 0.75-360
22.2
28.8
US-Net
0.25-360
2.5
23.6
US-Net
0.25-360
2.5
22.5
Ours
0.25-360
2.5
25.9
Ours
0.25-360
2.5
24.5
13

---

Under review as a conference paper at ICLR 2020
1.0x-600
Ours
US-Net
1.0x-600
Ours 0.5x-480
MIOWEHORN
US-Net O. 0.5X-480
RLON HOR
Figure 11: Object detection and instance segmentation examples.
14


Reviews:
[{"review_id": "ryg1bSmy5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a multi-resolution training scheme for a slimmable network. The proposed method provides a new regime leveraging diverse image resolutions for training sub-networks, which enables efficient network scaling. Throughout the experiments, the authors claim the proposed method shows better performance compared to US-net.\n\nPros)\n(+) The idea of multi-resolution training combining to a slimmable network looks good\n(+) Applying a slimmable network's technique to other tasks including detection and segmentation looks good\n(+) The combination of multi-resolution and the slimmable network seems to be reasonable.\n(+) The paper is well written and looks justified well.\n(+) The authors provided extensive experiments.\n\nCons)\n(-) There is no backups why the proposed method could outperform over US-net. \n(-) The proposed method is incremental and improvements are marginal.\n(-) Looks like there exists missing in details of the experiments.\n(-) The performance report of the compared methods is quite strange.\n\nComments)\n- The proposed method is too straightforward, so the authors should clarify why it works over US-net. Additionally, can the authors provide advantages using a different image-scale need for training a different sub-network? \n- The authors should clarify the training details of US-Net used in this paper. The performance of US-Net in Figure 4 (a) looks the same as the performance of US-Net trained with [0.05, 1]x scaling in the original paper. However, in the original paper, the authors of US-net reported [0.05, 1]x scaling as the worst performance setting in the original paper. Therefore, the authors should compare their method with the best performance setting of US-Net, which is [0.25, 1]x (because the proposed method looks being used [0.25, 1]x training setting, so the comparison should be done in fair).\n- The scaling parameters of US-Net used in the experiments should be specified. All the results of US-Net do not contain where they come from (i.e., the training width bound in US-net).\n- Can the authors report the results for 0.5-224 and 0.15-224 in Figure.4(a)? Why 0.7-160 and 0.25-160 were picked?\n- In Table 1, the performance of EfficientNet is weird. EfficientNet-B2 has 79.8% accuracy with 1.0B FLOPs, but the reported performance in this paper of EfficientNet has 75.6% accuracy with 2.3B FLOPs. Please clarify this.\n- How much does KLdiv contribute to the overall performance?\n- All the tables are not clearly shown. Please reattach all the tables for better readability.\n\nAbout rating)\nI think the idea looks novel, but the method is quite straightforward, and the paper does not incorporate any analysis as a backup for the proposed method. The initial rating is towards reject, but I would like to see the authors' response and the other reviewers' comments. After that, the final rating might be changed.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SygTdtzCKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explore how varying input image resolution can have a powerful effect in reducing computational complexity of Convolutional NN's to the point of outperforming state of the art efficient architectures.\nMore in detail, the paper proposes a method of joint training of multiple resolutions networks, leveraging student/teacher/distillation from scratch. This is based on training a high resolution teacher network and a low resolution student network, as well as a number of intermediate resolutions networks sampled randomly and jointly during training. Thanks to distillation well known regularization effects, the proposed method is achieving competitive results compared to existing state of the art efficient network architectures. The authors claim, and to some extent show, that this is due to the ability of the proposed method to take into account in a optimal way multi-resolution features available in the image. The paper is well written and presented with extensive results, comparing computational complexity/accuracy curves to existing state of the art architectures, as well as results on transfer learning to show that the feature learned do indeed generalize and don't necessarily overfit to imagenet. The idea is rather simple, but the results and the execution is inspiring.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1evBrJ9sr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new training regime for multi-resolution slimmable networks. Their approach is based on US-Net training technique but in addition to sampling from different network widths they also sample from different input resolutions and show that using multi-scale inputs improves the top-1 accuracy on ImageNet comparing to US-Net or MobileNet v1/v2 within the same resource constraints.\n\nPros:\n+ The authors correctly identify input resolution as one of the aspects of lightweight network design that is often overlooked\n+ They propose a practically viable training scheme that can be used to train & select networks given resource constraints\n+ The paper is well written and includes many insightful experimental findings\n\nCon:\n\nThe authors specify the mutual learning from width and resolution as their main contribution. They insist that treating input resolution independently from network structure is what distinguishes previous work from the newly suggested technique. But the paper doesn't include extensive experimental comparisons with the approaches that treat input resolution independently. Thus its claim that joint width/resolution sampling is beneficial comparing to independent approaches is somewhat unfounded. \n\nFor example, the authors show that MobileNet with 1.0-224 config (no sampling from widths nor from input resolutions during training) is outperformed by their network with 1.0-224 config (which effectively samples only from input resolutions during training). This is not surprising as one can view sampling from input resolutions as an equivalent to data augmentation. The importance of data augmentation is well known, so to prove the proposed mutual learning is beneficial the authors would need to compare against the networks that were trained using this multi-scale data augmentation. Figure 5 has a similar comparison but the only multi-resolution baseline there is US-Net+ which isn't using multi-resolution images in training. The paper would greatly benefit from adding such comparisons and proving they are not marginal.\n\nOn rating:\n\nI'd summarize the idea of this paper as A) US-Net + B) multi-scale data augmentation + C) selecting the best network based on both input resolution and width to achieve optimal performance within resource constraints. Although C is practical and novel contribution, it is also quite straightforward. I would like to see authors response on how their approach differs from US-Net + multi-scale data augmentation for training and how/why this works better.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:new training regime for multi-resolution slimmable networks. 

Discussion:
reviewer 4: believes the main contribution of mutual learning from width and resolution is a bit weak
reviewer 1: incremental work, details/baselines missing in experimental section
reviewer 2: (least detailed): well-written with good results
Recommendation: I agree with reviewer 1, 4 that the experimental section could be improved. Leaning to reject. 


Decision: Reject