Resizable Neural Networks
Authors: Yichen Zhu
Xiangyu Zhang
Tong Yang
Jian Sun
Venue:  (2020)

Abstract: 
In this paper, we present a deep convolutional neural network (CNN) which performs arbitrary resize operation on intermediate feature map resolution at stage-level. Motivated by weight sharing mechanism in neural architecture search, where a super-network is trained and sub-networks inherit the weights from the super-network, we present a novel CNN approach. We construct a spatial super-network which consists of multiple sub-networks, where each sub-network is a single scale network that obtain a unique spatial configuration, the convolutional layers are shared across all sub-networks. Such network, named as Resizable Neural Networks, are equivalent to training infinite single scale networks, but has no extra computational cost. Moreover, we present a training algorithm such that all sub-networks achieve better performance than individually trained counterparts. On large-scale ImageNet classification, we demonstrate its effectiveness on various modern network architectures such as MobileNet, ShuffleNet, and ResNet.


To go even further, we present three variants of resizable networks: 1) Resizable as Architecture Search (Resizable-NAS). On ImageNet, Resizable-NAS ResNet-50 attain 0.4% higher on accuracy and 44% smaller than the baseline model. 2) Resizable as Data Augmentation (Resizable-Aug). While we use resizable networks as a data augmentation technique, it obtains superior performance on ImageNet classification, outperform AutoAugment by 1.2% with ResNet-50. 3) Adaptive Resizable Network (Resizable-Adapt). We introduce the adaptive resizable networks as dynamic networks, which further improve the performance with less computational cost via data-dependent inference.

Under review as a conference paper at ICLR 2020
RESIZABLE NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we present deep convolutional neural network (CNN) that performs arbitrary resize operation on intermediate feature map resolution at stage-level. Motivated by weight sharing mechanism in neural architecture search, where a super-network is trained, and sub-networks inherit the weights from the super- network, we introduce novel CNN approach. We present Resizable Neural Net- works (ResizableNet), which resize feature map with arbitrary scaling factor at stage level. The inference cost of ResizableNet is equivalent a single scale because the weights of a convolutional layer are shared across di- versified scales. Additionally, we provide essential techniques build and train our propose ResizableNet. We demonstrate the effectiveness of ResizableNet on large-scale ImageNet classification with various modern network architectures such as MobileNet, ShuffleNet, and ResNet.
network with
To go even further, we present three applications of resizable networks: 1) Resiz- able Augmentation (Resizable-Aug). We present data augmentation technique based on ResizableNet. It obtains superior performance ImageNet classifica- tion, outperform AutoAugment by 1.2% with ResNet-50. 2) Resizable Network for Neural Architecture Search (Resizable-NAS). We introduce ResizableNet as a single network that can deploy on many different hardware to meet various ef- ficiency constraints. 3) Adaptive Resizable Neural Network (Resizable-Adappl We introduce the adaptive resizable networks as dynamic networks, which further improve the performance with less testing cost via data-dependent inference.
INTRODUCTION
Scale fundamental component of the physical world, and it has been an active research topic in the field of computer vision. The latest design of scales for the Convolutional Neural Networks (CNN) fall into two categories: single scale networks and multi-scale networks. In single scale networks, such ResNet (He et al. (2016)), only fixed scale representation is learned from the feature maps. Alternatively, in the multi-scale networks, e.g.. image pyramid (Sermanet et al., 2013; Dalal & Triggs, 2005) /feature pyramid (Ghiasi al., 2019; Chen et al., 2017a) /filter pyramid structure (Szegedy et al., 2015), features from different resolutions are fused in a network; thus the models learn various type of scales from the objects. As a result, these architectures obtain state-of- the-art performance on scale critical tasks, such as object detection (Lin et al. (2017)) and semantic segmentation (Chen al. (2017a)). It can be observed that both single scale networks and multi- scale networks essentially perform representation learning on a predefined feature map scales. This natural and intuitive choice, despite their success on various computer vision tasks, limits the CNNs to constrained predefined scaling mechanism for dealing with scale variation problem (He et al., 2019b).
Our approach is motivated by the recent emergence of weight sharing approach on neural archi- tecture search (NAS) (Pham et al. (2018); Bender et al. (2018); Guo et al. (2019)), where an over-parameterized super-network is trained, and sub-networks inherit the weights from the super- network. Generally, there are large number of architectural configurations within a super-network.
In this work, we propose Resizable Neural Networks (ResizableNet), which can be viewed super-network from a scale perspective. A ResizableNet resizes a feature map with an arbitrary scaling factor stage level. With sequence of scaling factors at different stages, scaling configuration, which can be used to define the feature map resolution of a network. In other words, ResizableNet has multiple scaling configurations. It is generalization of modern
obtain

---

Under review as a conference paper at ICLR 2020
sampled scale configuratins with resized feature map resolution
originll feature mup resolution
resized feature
stage_2
non-intager resize operatione
stago_3
non-nnffmm scaling rato between stages
stage_4
stage_5
a) baseline
b) network with resized input
c) resizable networks
Figure 1: Resolution scaling. (a) is a baseline network example. (b) are conventional scaling on input resolution. (c) is proposed resizable networks. It resizes the feature map resolution by a non-interger scaling ratio at the stage-level. A single network is trained with different scaling configurations with shared convolution. The imaginary line represents the original feature map resolution, and the full line represents the actual feature map resolution after resize operation.
CNN architecture in terms of scale, where only one fixed scaling configuration is adopted. The major advantage ResizableNet is that the weights of a convolution layer with diversified scale factors are shared, thus substantially reduce the computational costs and memory footprints during inference. Figure gives an illustration of the resizable net compared with the baseline network and input resize only network. Note that the network with resized input resolution is a special form of spatial sub-networks.
Training the resizable neural network is non-trivial task because of two characteristics of Resiz- ableNet. First, it requires to optimize network with diversified feature map scales. is challeng- ing since there enumerate all of them, while random sampling cannot guarantee the performance of the network. Secondly, the interference in layer between different scales leads accuracy drops because the weights of layer with various scales are shared in ResizableNets. We conjecture that the accuracy drop is due the discrepancy of feature mean and variance across different scales, which leads to inaccurate batch normalization statistics. To resolve these two problems, we introduce fair sampling and scale-aware batch normalization for training the resizable network. The fair sampling samples the scaling factors multiple times of layer in update step without reusing the same scaling factor, thus maintain the performance of the network with different scaling configuration to some extent. The scale-aware batch normalization introduces to use group of BN layers replace the shared BN layer, and each scaling factor obtains an independent BN layer. Thus solves the inconsistent BN statistics issue.
many scaling configurations, which means it is computationally impractical to
In this paper, we first validate the effectiveness of resizable neural networks compared with stand- alone networks ImageNet. We demonstrate superior performances of Resizable Neural Network on various network architectures, such as ResNet-50, ShuffleNetV2 and MobileNetV1.
Applications of Resizable Neural Network. We further demonstrate three applications of Resiz- ableNet: ResizableNet as architecture search Cdesizable-NAS) ResizableNet for data augmentation (Resizable-Aug) and ResizableNet for data-dependent inference (Resizable-Adap).
2

---

Under review as a conference paper at ICLR 2020
Resizable-Aug. We introduce Resizable Augmentation (Resizable-Aug), data augmentation tech- nique based on ResizableNet. We are able to improve ResNet-50 by 2.4% compare with baseline, which is higher than AutoAugment (Cubuk al. (2018)). Additionally, combine our augmentation technique on a new baseline model, we obtain a model achieve 77.8% with 365M FLOPs. This is significantly better than previous state-of-the-an model EfficientNet-BO (Tan & Le (2019)) which trained with AutoAugment (Cubuk et al. (2018)), 1.5% higher on accuracy with 25M less FLOPs.
Resizable-NAS. We introduce ResizableNet for NAS (Resizable-NAS) that can deploy on many dif- ferent hardware platforms to fulfill the efficiency constraints with one trained model. Given an efficiency constraint, we can search the specialized scaling configuration of ResizableNet for the target platform. Thus it achieves the run-time model compression objective. Additionally, we ob- served that there consistently exists smaller sub-network with similar (even higher) accuracy than the baseline network. For example, Resizable-NAS ResNet-50 is 44% smaller on model size and 0.4% higher on accuracy compare with the baseline.
Resizable-Adapt. We investigate that if there exists an optimal scaling configuration for every input. We introduce a data-dependent inference network, named as Adaptive ResizableNet (Resizable- Adapt). In our experiment, Resizable-Adap ResNet-50 improve the performance of Resizable-NAS ResNet-50 up to 0.5%, depends on the target spatial sub-networks.
2 RELATED WORK
Multi-scale Training. Our work is most related to multi-scale learning, which is important in image recognition (Li et al. (2019); Wang et al. (2019); Chen et al. (2018)), object detection(Lin et al. (2017); Singh al. (2018); Singh & Davis (2018)) and instance segmentation (Zhao et al. (2017); Chen et al. (2017b)). There are several approaches to fusing information of images at different visual resolutions. The naive approach is image pyramid (Sermanet et al. (2013); Dalal & Triggs (2005); Felzenszwalb et al. (2009); Cai et al. (2016)), where an input image is passed through model multiple times at different resolutions. Our work introduce the resizable networks which comprise infinite single scale network, that acted as implicit multi-scale networks.
Dynamic Neural Networks. Our work is related to dynamic neural networks. Dynamic neural networks aim to train a single network to support different architectural configurations, especially adjust networks according to input samples. Slimmable networks (Yu al. (2018a)) train a single model to support multiple width configurations by switching width at runtime. Universal Slimmable (Yu & Huang (2019)) is proposed to expand the number of width that is supported by training a single network. In paper, we explore the dynamic network the spatial level.
Over- preepparamttrized model network with minimal performance drop is appealing on resource-constraing devices. Modern compression methods including channel pruning (Liu et al. (2018); He et al. (2017)), knowledge distillation (Hinton et al. (2015); Furlanello et al. (2018); Romero et al. (2014); Zhang et al. (2018)), hashing (Chen al. (2015)), network bina- rization (Courbariaux et al. (2016); Rastegari et al. (2016)). Our work explore network compression at the runtime on spatial-level, which is complementary to existing compression methods.
Model Compression. Compressing
Neural Architecture Search. Recently the design of efficient neural networks has largely shifted from leveraging human knowledge to automatic methods, which is known as neural architecture search (NAS). Designing search space is of the most important in NAS on various vision tasks such as image recognition (Radosavovic et al. (2019); Cai al. (2018); Howard al. (2019); Guo et al. (2019); Tan et al. (2019)), instance segmentation (Liu et al. (2019)) and object detection( GGhiasi al. (2019)). Resizable networks can be consider as a designed search space, where each spatial sub-networks is optional architecture.
Data Augmentation. Data augmentation is effective for improving the accuracy of modern neural network in various computer vision tasks (Szegedy et al. (2015)). Many new augmentation tech- niques such as Cutout (DeVries & Taylor (2017)) and Mixup (Zhang et al. (2017)) on input image have shown promising results. Recently, new trend is raised that automatically learn the data trans- formation (Ho et al. (2019)). Cubuk et al. (2018) leverage RNN controller to automatically search the best data augmentation combination for training in a large augmentation sub-policy space. We explore the data augmentation technique on feature map spatial-level by letting the network learns different feature map scales.
3

---

Under review as a conference paper at ICLR 2020
Random Sampling
Fair Sampling
stop
step t
stgee
5
S1 1.0
0.8
semple without replecement
5, 0.4
sample
second sample
sample
sample
Figure 2: Fair Sampling. Left: Random Sampling. In each update step, we sample sequence of scaling factors at each stage. Right: Fair Sampling. In each update step, we sample a sequence of scaling factors each stage multiple times. The same scaling factor the stage does not sample twice in the same update step.
3 RESIZABLE NEURAL NETWORKS
Our CNN architecture has N stages, Each stage consists of sequence of layers that have the same resolution of the feature map. The feature map resolution is gradually reduced by resizing (denoted as G) with a scaling factor (denoted as s), at the beginning of stage. From the scale perspective, we denote network by sequence of resize operation with scaling factors, named as scaling configuration (denoted as S), denoted formally as
(G( N, E (0,
where denotes the ith stage that performs resize where i denotes the ith stage that performs resize operation. In the modern CNN architectures (see Figure 1(a)), G is convolution with stride 2 (He et al. (2016); Huang et al. (2017); Xie et al. (2017)) or max/average pooling (He et al. (2019a); Li al. (2019)) reduce the feature map size by half (s 0.5). These CNN architectures learn the input image with a fixed scale. Recent works have indicated the importance of learning the images with various scales. Several works design CNN architecture towards adding parallel branches to deal with different input scales (Li et al. (2019); Wang et al. (2019)). The design of these architectures improve generalization. However, they are inefficient since extra branches require extra parameters and increase FLOOSs
In order to learn image features at different scales without increasing FLOPs, we propose the resiz- able neural networks. The ResizableNets (see Figure Ic) is generalization of the CNN architectures terms of scale. ResizableNets, G is bilinear interpolation. allows the feature map to resize to any resolution. As such, modern CNN architecture can be viewed as special case of ResizableNets. During training. the ResizableNets are trained with different feature map scale dynamically, More- over, the ResizableNets share the weights of convolutional layer across different scaling factors. In this way, we allow the network to learn multiple scales input without extra inference cost.
Training the resizable networks is non-trivial task due to the flexible resize operation and weight sharing mechanism. We now discuss the inherent problems ResizableNets and present solutions to resolve these problems.
Fair Sampling. The resizable neural network focuses on train one network to support various scaling configurations in inference without retraining. From this perspective, the naive approach is
4

---

Under review as a conference paper at ICLR 2020
to randomly sample scaling factor from predefined scaling factor list. With sequence of scaling factors at each stage, we obtain a scaling configuration. We update the weights of ResizableNet use this sampled scaling configuration and repeat this process. However, random sampling brings instability into the ResizableNet training process, because different scaling factor list in a layer are sampled with different number of times by random sampling. As result, some scaling factors in a convolution have more training opportunities than others, lead to imbalance performance between different scaling configurations. To tackle this problem, update step, the fair sampling samples scaling factor from the scaling factor list multiple times. Each time it selects scaling factor in layer if and only if this particular scaling factor in this particular layer has not been selected before in the same update step. In this regard, we ensure that all scaling factors a layer are selected and trained to undergo the same training process. Figure 2 gives illustration of fair sampling. Moreover, we show in Appendix C that training of ResizableNet with fair sampling is faster than random sampling.
introduce the fair sampling. In each
Scale-Aware Batch Normalization. Batch normalization (BN) layer is a standard normalization method to accelerate training and improve the generalization of CNN. It is common to use a "Conv- BM" couple building block. Because the feature map size of convolution is constantly changing during the training of ResizableNets, we say that a BN layer is shared by a convolution layer with multiple sizes. The shared BN layers are not applicable in ResizableNets since different resolutions of the same feature map are aggregated according to Algorithm 1, which results in different mean and variance. It leads inaccurate batch normalization statisties across different scaling factors in shared layers, based on our experiment in Table 9, it impairs the test accuracy.
To tack this issue, we propose use an independent BN layer to respond to a scale factor in layer. In other words, we use a "Comv- (B. .... B N )" as a building block, where & denotes as the number of scaling factors in a layer. As such, each BN layer specifically calculate BN statisties and learns the parameters of corresponding feature map resolution in the stage. The number of extra pa- rameter scale-aware BN layers negligible since the number of parameters in BN layers usually take less than one percent of the total model size. During testing, only the BN layer corresponding to the activated scaling factor is used. Thus its behavior is the same as a regular building block.
Scaling Factor List L. To help to understand the training process of ResizableNets, we describe scaling factor list L, which we used in Algorithm 1, The scaling factor s for a layer is discrete. is chosen from pre-defined scaling factor list L. An typical of scaling factor list we used in the experiments is L = (0.45, 0.65, 0.85, 1.0). The number of scaling configurations of ResizableNets related to the number of stages and the length of T. (number of scaling factors). For example, if network contains four stages, and we use L. = (0.45, 0.65, 0.85, 1.0), then there 4 = 256 scaling configurations. We will discuss the impact of different choice of and number of ResizableNets in Appendix A.
Algorithm Training resizable neural networks RS.
Require: Initialize resizable network. Define scaling factor list L.
for i= I.... n do
Get data x and target y of current mini-batch.
Clear gradients for all parameters, optimizer.zero grad()
for ==0. len.(T))((([T) len(T.) do
Follows the fair sampling, sample scaling configuration Sj.
Select the scale-aware batch normalization for each stage.
Execute current scaling configuration. á Sx)
Compute loss for S,, loss:; criterion (ū,y)
Compute gradients for Sy. backwardd
end
Update weights, optimizer.step().
end
5

---

Under review as a conference paper at ICLR 2020
4 APPLICATIONS
4.1 FOR DATA AUGMENTATION
Previous data augmentation techniques transform the input image directly (Zhang et al. (2017); Cubuk et al. (2018); DeVries Taylor (2017)). In contrast, our method proposes to transform the feature map by scale at stage level, named as Resizable Augmentation (denoted as SesizableAAugl The network learns various scale combinations of feature map by switching scaling configuration during training. thus potentially improve generalization.
The goal of Resizable-Aug is to increase the accuracy of the target network. The challenge is that a convolutional layer shares the weights across all scaling factors. As result, training convolution with a scale could affect the accuracy of the same convolution with other scales. To address this challenge, decouple the model training stage and the model fine-tuning stage. Precisely, in the model training stage, we follow the training Algorithm 1. As such, the model learns the input with various scales. In the model fine-tuning stage, we specifically fine-tune the target network. In other words, we fix the target scaling configuration and optimize the network with target scaling configuration only. We empirically use two-third of total training epochs the model training stage and the rest training epochs at the model fine-tuning stage.
4.2 FOR NEURAL ARCHITECTURE SEARCH
There is a rising demand for designing specialized neural architecture with a short response time on different hardware (e.g., mobile phones). Recently, neural architecture search methods are pro- posed to search for the specialized network architecture on a specific hardware. However, searching network architectures for every hardware is computationally expensive, since different hardwares or even the same hardware with different equipment conditions can have different efficiency constraints (e.g., FLOPs and latency).
We present resizable neural networks for neural architecture search (denoted as Resizable-NAS) The ResizableNets can be intuitively viewed as search space, where scale configurations are can- didates in the search space. Every scale configuration of ResizableNets can be instantiated as sub- network. As such, we can search for the specialized sub-network that fulfiil the hardware efficiency constraints. More importantly, with one trained ResizableNets, it can be deployed to diversified hardware simultaneously, thus cut down the computational cost for retraining networks.
To achieve this, we first train ResizableNet with Algorithm 1. With a trained ResizableNet, we can search for the sub-network for the given efficiency constraints. Specifically, we apply search method on the ResizableNet to find the specialized sub-networks. For simplicity, we present a naive approach. We enumerate all sub-networks and build a lookup table to include information such accuracy, FLOPs, and latency. Therefore, given the efficiency constraints, lookup table to get the specialized sub-network. Additionally, we can increase the accuracy of the specialized sub-network by fineinuning. Empirically, find that fine-tune one epoch is sufficient to increase the accuracy considerably, while fine-tune fifteen epochs can obtain the highest accuracy.
query the
4.3 FOR DYNAMIC INFERENCE
In our work, we observed that the training accuracy on ImageNet classification for ResizableNets with smaller scaling configurations (smaller FLOPs) sometimes perform better than those with larger scaling configurations. We conjecture that categories with complex image patterns bene- fit from predicting with larger scaling configurations, and categories with simpler image patterns benefit from predicting with smaller scaling configurations. (Wang et al. (2019); Li et al. (2019)). Based on this observation, present dynamic inference for the ResizableNets, named as Adaptive Resizable Networks Aeesizable-Adaple Resizable-Adien aims to find the optimal scaling contigura- tion for each sample based on their inherent characteristics (e.g., object size and image complexity) learned by the network.
For the Resizable-Adins Adapt, first randomly sample 50k images from the training data. We test all scaling configurations for each sample on an already trained ResizableNet. We take the scale configuration with the highest accuracy for the sample the label. We then introduce lightweight

---

Under review as a conference paper at ICLR 2020
Table I: This table shows the result of Top-1 accuracy of scaling configurations in resizable networks compare with individually trained counterparts.
Individual Networks
Resizable Networks
Name
FLOPs
Top- I Acc. Name
FLOPs
Top-1 Acc.(gain)
569M
70.6%
569M
72.0%(+1.4%))
Individual-
462M
70.0%
462M
71.3%(+1.3%%)
RS-MobileNerVI
MobileNetV1
391M
69.5%
391M
70.7%(+1.2%))
291M
68.1%
291M
69.5%(+1.4%%)
299M
72.6%
299M
73.3%(+0.7%%)
Individual-
251M
71.5%
251M
72.2%(+0.7%%)
RS-ShuffleNetV2
ShuffleNetV2
199M
70.3%
199M
70.9((+0.6%)
147M
69.2%
147M
69.8%(+0.6%%)
4.1G
76.4%
4.1G
77.1%
Individual- ResNet-50
2.8G
75.4%
2.8G
76.6%(+1.22%)
RS-ResNet-50
2.0G
74.2%
2.0G
75.2%
1.1G
71.6%
1.1G
72.3%)+
scaling configuration classifier that is attached at the last stage of the original resizable network. It composed of three convolutional layers along with a pooling layer and followed by the softmax. It learns the image specific scaling configuration. To be specific, this classifier takes an image as input and predicts an optimal scaling configuration for this input. Then we switch the ResizableNet to this scaling configuration and predict its labels using the input.
Our extensive experiments verify our conjecture that the network can indeed achieve better per- formance if the sample is predicted by suitable scaling configuration. Additionally, the previous resizable network process all images to one scaling configuration, as the network spends equal en- ergy (e.g.. FLOPs) every sample. The adaptive resizable network spend equal or less amount of computational cost during testing since some samples are assigned to amaller-scaling configura- tions. Thus, Resizable-Adar Adapt is more efficient in inference.
EXPERIMENTS
In this section, validate the ResizableNets ImageNet classification from several perspectives. Implementation details can be found in the Appendix A. Section 5. give experiments for resizable networks compared with individual networks. Section 5.2, Section 5.3, and Section 5.4 gives the result of Resizable-NAS, Resizable-Aug and Resizable-Adapt network respectively.
RS- indicate ResizableNets from Section 3, RS-NAS indicates Resizable-NAS from Section 4.2, RS-Aug to represent resizable augmentation from Section 4.1. RS-Adapt to denote adaptive resizable networks from Section 4.3.
Notation. For consistency,
5.1 COMPARISON BETWEEN RESIZABLE NETWORKS WITH SLAND NETWORK.
We show in Table the top-1 classification accuracy for both individually trained networks and resizable networks give the same scaling configurations. Due to the space limit, we only take sev- eral scaling configurations for comparison. We present top-1 test accuracy and FLOPs, which is the primary concerns in inference time, of all networks for reference. Compared to independently trained networks, the spatial sub-networks in resizable networks achieved significantly better top-1 accuracy, across various network architectures and computational complexity. This result confirm hypothesis that the implicit multi-scale training improve the model capacity.

---

Under review as a conference paper at ICLR 2020
(a)
(b)
72-
36-
14
Imagatie:
23-
Redhesdoandd modele
Shuflehber22 Indeede TECSh
67-
(c)
(d)
18-
96-
34-
Restette
13-
ThiketSs
Imapatett
34-
72-
Adapt RustiASS
R5Aemeee0
72-
10-
RS WAS Aushvtts (nw15)
RS WAS Rushet53
2.0
255
3.3
35
40
43
1.0
25
30
3.5
40
45
GRLPPs
GROOPs
Figure (a)-(b): Resizable-NAS ResNet-50 and ShuffleNetV2 compare with the individually trained networks and naive resizable approach. (c): Resizable-NAt NNA ResNet-50 compare with state- of-the-art model compression methods. (d) Resizable-Adap ResNet-50 compare with the Resizable- NAS ResNet-50. The horizontal line mean the FLOPs range of sub-network used in resizable-adapt network for data-dependent inference. Models with larger target network achieve better accuracy.
Table 2: This table summarizes the validation Top-1 accuracy on ImageNet dataset for ResNet-50 architecture. We compare Resizable-Aug with multiple state-of-the-ar augmentation methods and regularization methods. Bold indicates the best results. denotes results reported in the original papers.
Model
Top-1 Agc.(gain)
Top-5 (gain
Baseline ResNet-50 (He et al. (2016))
76.4%
93.1%
ResNet-50 + Cutout (DeVries & Taylor (2017))
76.5%
93.2%
ResNet-50 + MixUp' (Zhang et al. (2017))
76.7%
93.4%
ResNet-50 + AutoAAumeett (Cubuk et al. (2018))
77.6%
93.8%
ResNet-50 + RS-Aug
78.8 (+2.4%)
94.4% (41.3%)
5.2 RESIZABLE-NAS.
Comparison with Naive Resizable Networks. The Figure 3(a) and 3(b) demonstrate the results of ResizableNet and Resizable-NAS network with ResNet-50 and ShuffleNerV2. The only differ- ence between these two networks is that the Resizable-NAr network take 15 epochs to fine-tune target scaling configuration. The Resizable-Nar shows the notable performance improvement over individual network and ResizableNet with the same scaling configuration.
Comparison with State-of-the-art Model Compression Methods. We compare with the following btteeoofttee tiffthheart art model compression methods: NISP (Yu et al. (2018b)), ThiNet (Luo et al. (2017)) ChannelPruning (He et al. (2017)). Sparse Structure Selection ResNet50 (Huang & Wang (2018)) and SlimmableNets (Yu et al. (2018a)), AMC (He et al., 2018) and Netadapt (Yang al., 2018). We
8

---

Under review as a conference paper at ICLR 2020
Table 3: Comparison with state-of-the-art architectures on ImageNet (200M to 400M FLOPs).
denotes reported results employ AutoAugment.
Model
FLOPs (M)
Params (M)
Top-1 Acc.
Top-5 Acc.
ShuffleNetV2 1.5x (Ma et al., 2018)
300
72.6%
MobileNetV2 1. 1.0x (Sandler et al., 2018)
300
3.4
72.0%
91.0%
MobileNetV3 Large (Howard et al., 2019)
291
5.4
75.2%
92.2%
MnasNet-A2 (Tan et al. (2019))
340
4.8
75.6%
92.7%
EfficientNet-BO (Tan & Le (2019))
390
5,3
76.3%
93.2%
Efficient ResizableNet w/o Aug Efficient ResizableNet
365
6.7
77.4%
93.6%
365
6.7
77.8%
93.6%
present the ResNet-50 based results at Figure 3(c), and the detailed results of Resizable-NAS Mo- bileNetV1 I as well as Resizable-NAS ResNet-50 can be found Table B. Resizable-NAS ResNet-50 demonstrate strong performance, especially when the compression ratio is large than 50%. It out- performs listed state-of-the-ard compression methods by a large margin over diverse computational complexity constraints. We emphasize that our Resizable-NAS network only trained once, and fine-tune 15 epochs on target sub-networks, where the other compression methods find the optimal structure and retrain accordingly. Comparing to the other methods, our proposed model is more ef- ficient while performing better. Notice even the version without the fine-tuning process can achieve similar (even better) performance than other model compression methods.
5.3 RESIZABLE AUGMENTATION
On ResNet-50. We compares the Resizable-Auu ResNet-50 with various state-of-the-aat data aug- mentation methods (i.e., Cutout (DeVries & Taylor (2017)), Mixup (Zhang et (2017)) and Au- toAugment (Cubuk al. (2018))). The results are summarized in Table 2. It shows that Resizable- Aug improve the ResNet-50 baseline by 2.4%, outperform AutoAugment by 1.2% using the same the baseline. It is worthy to note that our Resizable- Aug takes the same number of epochs as the baseline to train the network. In our experiment, we use scaling factor list (0.45, 0.65, 0.85, 1.0).
On Efficient Models. We present the resizable augmentation results based on network that search by AutoML. The details of the training strategy and network architecture are presented in the Appendix B.3. Table shows the results. By adopting the resizable augmentation strategy, improve the already high accuracy network from 77.4% to 77.8%, achieve a new state-of-the-an result. The new models is 1.5% better than EfficientNet-BO (Tan & Le (2019)), which is trained AutoAugment (Cubuk et al., 2018) with 25M less FLOPs. In our experiment, we use scaling factor list of (0.45, 0.65, 0.85, 1.0).
5.4 ADAPTIVE RESIZABLE NEURAL NETWORK
Figure 3(d) shows the results of Resizable Adapt ResNet-50. The dynamic inference is performed on multiple computation complexity threshold. During inference, we ensure that only sub-networks that are smaller than the target network is used for inference. The performance gain is range from 0% to 0.5%. It can be observed that the adaptive resizable ResNet-50 consistently obtain better performance than the bascline. This is surprising result since most dynamic network struggle at the trade-off between efficiency and accuracy. Less computational complexity in dynamic network always lowers the accuracy compared with the baseline. Moreover, when the target network is large, sometimes the samples are still better predicted by the small sub-network. This gives the evidence that the explicit way deal with the input scale is indeed beneficial.
CONCLUSION
We introduced a type of convolutional neural network, named Resizable Neural Networks. The resizable networks can do arbitrary resize operation on feature map at stage level. We further pro- vide essential tools build and train proposed network. Moreover, give three applications

---

Under review as a conference paper at ICLR 2020
of ResizableNet. Specifically, present Resizable-NAS, Resizable-Aug, and Resizable-Adage Adapt. Resizable-Aug is data augmentation techniques which do feature map transform by scaling and achieve state-of-the-art results on ImageNet classification. Resizable-NAS can deploy many hardwares with various efficiency constraints by on trained network. Resizable-Adapt performs data-dependent dynamic inference. Overall, believe that the resizable network's approach pro- vides an interesting and practical new perspective on designing a convolution neural network on the spatial dimension.
REFERENCES
Anonymous. [VAENAS]: Sampling matters in neural architecture search. In Submitted to Interna- lional Conference Learning Representations, 2020. URL https: http://pppneeeeeneeeeeeeeee forum? -ww.iiiddsxxxySSSX..X. under review.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understand- ing and simplifying one-shot architecture search. International Conference on Machine Learn- ing, pp. 549-558, 2018.
Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-level network transformation for efficient architecture search. arXiv preprint arXiv:1806.02639, 2018.
Zhaowei Cai, Quanfu Fan, Rogerio S Feris, and Nuno Vasconcelos. A unified multi-scale deep volutional neural network for fast object detection. european conference on computer vision, pp. 354-370. Springer. 2016.
Chun-Fu Chen, Quanfu Fan, Neil Mallinar, Tom Sercu, and Rogerio Feris. Big-little net: An efficient multi-scale feature representation for visual and speech recognition. arXiv preprint arXiv:1807.03848, 2018.
Liang-Chies Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions pattern analysis and machine intelligence, 40(4): 834-848, 2017a.
Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation, arXiv preprint arXiv:1706.05587 2017b.
Wenlin Chen, James Wilson, Stephen Tyree. Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International Conference on Machine Learning, pp. 2285-2294, 2015.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained 2016.
or-1. arXiv preprint
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09555 2018.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. 2005.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 1708. 2017.
Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained bast- models. IEEE transactions on pattern analysis and ma- chine intelligence, 1279)16677 2009.
Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. arXiv preprint arXiv: 1805.0400000770 2018.
10

---

Under review as a conference paper at ICLR 2020
Golnaz Ghiasi, Tsung- Yi Lin, and Quoe V Le. Nas-fpn: Learning scalable feature pyramid architec- ture for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7036-7045, 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint (rXiv:1904.00420) 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 558-567, 2019a.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Conference on Computer Vision, pp. (138-1197) 2017.
Yihui He, Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings the European Conference on Computer Vision (ECCV), pp. 784-800, 2018.
Zewen He, He Huang, Yudong Wu, Guan Huang, and Wensheng Zhang. Consistent scale normal- ization for object recognition. arXiv preprint arXiv: 07323, 2019b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 10Xx::1503.0231,,, 2015.
Daniel Ho, Eric Liang, Ion Stoica, Pieter Abbecl, and Xi Chen. Population based augmentation: Efficient learning of augmentation policy schedules. arXiv preprint arXiv:1905.0539; 2019.
Andrew Howard. Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. arXiv preprint arXiv:/905.0220 02244, 2019.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1rXiv:1704.0486/ 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 304-320, 2018.
Yi Li, Zhanghui Kuang, Yimin Chen, and Wayne Zhang. Data-driven neuron allocation for scale aggregation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11526-11533 11534, 2019.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 82-92, 2019.
Zhuang Liu, Mingjee Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint 1 2018.

---

Under review as a conference paper at ICLR 2020
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruming method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 505555058566 2017.
Ningning Ma, Xiangyu Zhang, Hai- Tao Zheng, and Jian Sun. Shufflenet v2: for efficient architecture design. In Proceedings of the European Conference Computer Vision (ECCV), pp. 116-131, 2018.
Practical guidelines
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: / 802.03268, 2018.
Iiiaa Radosavovic, Justin Johnson, Saining Xie, Wan Yen Lo, and Piotr Dollár. On network design spaces for visual recognition. arXiv preprint arXiv: /905. 132144 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. European Conference on Computer Vision, pp. 525-542. Springer, 2016.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathicu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv: 2013.
Bharat Singh and Larry S Davis. An analysis of scale invariance in object detection snip. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 3578-3587, 2018.
Bharat Singh, Mahyar Najibi, and Larry S Davis. Sniper: Efficient multi-scale training. In Advances in Neural Information Processing Systems, pp. 2310-9320. 2018.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826, 2016.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks, arXiv preprint arXiv: 1946. 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe Le. Mnasnet: Platforn-awware neural architecture search for mobile. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2820-2823 2019.
Huiyu Wang, Aniruddha Kembhavi, Ali Farhadi, Alan Yuille, and Mohammad Rastegari. Elas- tic: Improving enns with dynamic scaling policies. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2258-2267, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
12

---

Under review as a conference paper at ICLR 2020
Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform-awer aware neural network adaptation for mobile applications. In Proceedings of the European Conference on Computer Vision (ECCV). pp. 285-300, 2018.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques.
arXiv preprint urXiv:1933.05144 2019.
Jiahui Yu, Linjie Yang. Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint 18/ 2018a.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9194 9203, 2018b.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 2017.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEKE Conference on Computer Vision and Pattern Recognition, pp. 4320- 4328, 2018.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 2881-2890, 2017.
13

---

Under review as a conference paper at ICLR 2020
A CHOOSING THE PREDEFINED SCALING FACTOR LIST.
We will discuss the choice of predefined scaling factor list to ResizableNet and ResizableAug sepa- rately.
A.1 RESIZABLENET
We first report the scaling configuration list we used in the previous experiments. For Resizable ResNet-50 and Resizable ShuffleNerV2, we use scaling configuration 5 = (0.45, 0.65, 0.85, 1.0)- (0.5, 1.0). We use only three scaling factors
For all Resizable MobileNetV1 we use because the number of stage of MobileNetVI is one larger than ResNet and ShuffleNetV2.
Choice of Scaling Configuration S. The choice of scaling configuration depends on the range of efficiency constraints that need to be covered. There is specific guideline for choosing scaling configuration for ResizableNet.
Number of scaling factors With more scaling factors, the number of scaling configuration increased exponentially, thus more flexible during testing. We investigate how the number of scaling factors impact accuracy Resizable-NAS, In Table 4, train ResNet-50 with two different scaling factor list. We compare a four scaling factors list with six scaling factors list. We display some results. The results of different number of scaling factors are similar, showing the scalability of our proposed method.
Table 4: This table shows the result of Top-1 accuracy of scaling configurations resizable networks compare with individually trained counterparts.
Scaling Factors
FLOPs
Top-1 Ace.
Scaling Factors
FLOPs
Top-1 I Acc.
3.5G
77.8%
3.5G
77.5%
2.8G
77.4%
2.8G
77.1% %
(0.45, 0.65, 0.85,
(0.45, 0.55, 0.65,
2.3G
76.8%
2.3G
77.0%
1.0)
0.75, 0.85, 1.0)
2.0G
76.5%
2.0G
75.8%
1.0G
72.8%
1.0G
72.9%
A.2 RESIZABLEAUG
For all ResizableAug experiment, we use scaling configurations of 5 = {0.45, 0.65, 0.85, 1.0).
Choices of scaling configuration S. We study how choose the scaling factors S such that the best performance via ResizableAug can be achieved. The performance gain in ResizableAug due the implicit multi-scale training. Thus the predefined scaling factor list can be viewed as the choice for multi-scale. We empirically find that, as long as we use the diversified scaling factor, the performance of ResizableAug is robust to the specific choice of scaling factors. The left part of Table 6 shows the results using different combinations of scaling factors. The first three experiments, (0.45, 0.65, 0.85, 1.0). S (0.4, 0.6, 0.8, 1.0) or S (0.25, 0.5, 0.75, 1.0) achieve the same or similar performance, which indicate the specific choice of scaling factor is a critical to increase the accuracy of the model. In the fourth set, we use S (0.7, 0.8, 0.9. 1.0), where every scaling factor is close to each other, and the accuracy falls as expected. By default, in all our experiment we use (0.45, 0.65, 0.85, 1.0).
Number of scaling factors We study how the number of scaling factor candidates affect the performance of ResizableAug. We train three ResizableAug ResNet-50 with number of scaling factors equal to 3, 4 or 5, following the scaling factor diversity rule. Right part of Table 6 shows that the model trained with n = 4 or n 5 achieve the same performance, while n = 3 perform worse than 4 or 5. By default, in all our experiment
=
14

---

Under review as a conference paper at ICLR 2020
Table 5: This table shows the result of Top-1 accuracy of ResizableAug ResNet-50 by various scaling configuration.
Length of Scaling Config.
Choice of Scaling Config.
Scaling Config.
Acc.
Scaling Config.
Acc.
(0.45.0.65.0.85.1.00)
78.8%
(0.45.0.65.0.85.1.0))
78.8%
(0.4.0.6.0.8.1.0))
78.8%
(0.25.0.45.0.5,,088..00
78.8%
(0.25,0.5.0.75.1.00)
78.7%
(0.5,0.75,1.0)
77.6%
(0.7.0.8.0.9.1.00)
77.9%
Table 6: This table shows the result of Top-1 accuracy of ResizableAug ResNet-50 by various scaling configuration.
B IMPLEMENTATION DETAILS
B.1 NETWORK IMPLEMENTATION DETAILS.
Resize operation is place at the first non-stride block at each stage in network. We apply bilinear interpolation resize the feature map to target size. Resize factor uniformly taken from predefined list of possible resize choice, except when it contradicted to fair sampling rule. Our implementation based the Pytorch (Paszke et al. (2017)). The input resolution for all train and test images for ImageNet classification are 224 224.
Other than ResNet-50, which report baseline higher than the original paper, have trained Mobilenet vl. Mobilenet v2 and Shufflenet v2 baseline to match the performance as reported in original paper. And based on the re-implemented settings, we perform both baseline and resizable networks experiments.
For ResNet-50, we train for 100 epochs using weight decay of lc-4. We use batch size of 512 and do learning rate decay by 10 at epochs 30, 60, 80, 90. The other resizable network train for 200 epochs and double the epochs of learning rate decay.
For Mobilenet v1, Mobilenet v2 and Shufflenet v2, we use batch size of 1024 and weight decay of 4e-5. We train both models for 240 epochs using linear learning rate decay where base learning rate is set to 0.5. All models are train across synchronous SGD across 8 GPUs. The resizable networks train for 480 epochs
The resizable augmentation network always use the same training setting as the baseline method. Extending the training epochs is unnecessary.
B.2 DETAIL RESULTS OF RESIZABLE- NAS NETWORKS
The detail results of Resizable-NAS ResNer-50 and Resizable-NAS MobileNetVi be found Table B.
B.3 DETAILS OF EFFICIENT RESIZABLENET
Training Details. Efficient ResizableNet is trained using resizable-aug technique as we discuss in the Section 4.1. It is trained for 480 epochs with batch size 1024 on 8 GPUs. The network parameters are optimized using an SGD optimizer with an initial learning rate 0.5 (decayed linearly after each iteration), momentum of 0.9 and a weight decay of x 10 Additional enhancements including label smoothing (Szegedy et al. (2016)) and Dropout with 0.5 on last FC layer. We follow the common practice use inception augmentation. This architecture is searched with Anonymous (2020) using search space proposed by Guo et al. (2019). The Xception is proposed by Chollet (2017).
Detailed Network Architecture. The network architecture along with feature map resolution and channels number are shown in Figure B.3.
15

---

Under review as a conference paper at ICLR 2020
Model
Top-1 Acc.
Gain
FLOP
Compres. Ratio
ResNet30 (He et al. (2016))
76.4%
+0%
4.1G
0%
RS-NAS-ResNethh (ft=15)
77.8%
+1.4%
3.5G
15%
AS-NAS-ResNethh (ft=1)
77.6%
+1.2%
3.5G
15%
RS-ResNet50
77.1%
+0.7%
3.5G
15%
RS-NAS-ResNeds (ft=15)
77.4%
+1.0%
2.8G
32%
-essee50 (ft=1)
77.1%
+0.7%
2.8G
32%
RS-ResNet50
76.6%
+0.2%
2.8G
32%
SSS-ResNet50 (Huang & Wang (2018))
74.2%
-2.2%
2.8G
32%
ThiNet-70 (Luo et al. (2017))
72.5%
-3.9%
2.9G
29%
NISP-ResNet50-A (Yu et al. (2018b))
76.2%
-0.2%
3.0G
27%
RS-NAS-ResNet500 (ft=15)
76.8%
+0.4%
2.3G
44%
RS-NAS-ResNec550 (ft=1)
76.2%
-0.2%
2.3G
44%
RS-ResNet50
75.4%
-1.0%
2.3G
44%
Slim ResNe150 (Yu et al. (2018a))
75.0%
-1.4%
2.3G
44%
SSS-ResNet50 (Huang & Wang (2018))
75.6%
-0.8%
2.3G
44%
NISP-ResNee50--B (Yu et al. (2018b))
75.5%
-0.9%
2.3G
44%
RS-NAS-ResNetäh (ft=15)
76.5%
+0.1%
2.0G
51%
ReNNeee (ft=1)
75.9%
-0.5%
2.0G
51%
RS- RS-ResNet50
75.2%
-1.2%
2.0G
51%
Cheser0 (He et al. (2017))
73.3%
-3.1%
2.0G
51%
ThiNet-50 (Luo et al. (2017))
72.5%
-3.9%
2.1G
49%
RS-NAS-ResNet50 (ft=15)
72.8%
-3.6%
1.1G
73%
RS-NAS-ResNerts (ft=1)
72.4%
-4.0%
1.1G
73%
RS-ResNet50
72.3%
-4.1%
1.IG
73%
Slim-ResNet50 (Yu et al. (2018a))
72.0%
-4.4%
1.IG
73%
ThiNet-30 (Luo et al. (2017))
69.9%
-6.5%
1.2G
71%
MobileNetV1 (Howard et al. (2017))
70.9%
+0%
569M
0%
RS-NAS-MobileNatVi (ft=15)
71.9%
+1.0%
391M
31
RS-VAS-MobileNatV (ft=15)
70.9%
+0%
291M
49 %
RS-MobileNetV1
69.5%
-1.4%
291M
49%
Slim-MobileNNetI1 (Yu et al. (2018a))
69.5%
-1.4%
317M
44%
NerAdapt-MobileNeet (Yang et al. (2018))
70.1%
-0.9%
285M
50%
AMC-MobileNetVI (He et al. (2018))
70.5%
-0.4%
285M
50%
Table 7: This table compares the Top-1 accuracy of Resizable Network with several state-of-the- art model compression methods given FLOPS. Bold indicates highest top-1 accuracy under certain FLOPS constraint. Blue indicates dynamic networks.
C TRAINING EFFICIENCY ANALYSIS.
We analysis the training time of ResizableNets. We evaluate the training times of ResizableNets with various number of scaling factors (1, 2. and 6) are evaluated, as well as the total training time if the same amount of sub-networks in the resizable net are trained individually. The results are shown in Table 8. On the one hand, the total training time of resizable nets remarkably less than the train the same amount of individual networks separately due to the weight sharing mechanism. the other hand, because we sample multiple scaling configurations one step, we climinate the need to provide mini-batch batch data and update gradients repeatedly. As result, the training time by fair sampling is less than simply extending the total training epochs by the number of predefined scaling factors. Particularly, the number of scaling factors increases, the benefit of fair sampling in terms of training time more significant. For example, compare to random sampling, when len(S) = 4, fair sampling is 10% faster, while len(S) 8, fair sampling is 52% faster.
16

---

Under review as a conference paper at ICLR 2020
HDxf
51-0005
50000
28026
2000000
20000
7C2972
2:01230
33
33
Xeeptton
3x3
00
Koupianys
SiS-SE
5 no//tt
S SisiEE
Koed UNISE
Com
hodrour
Figure 4: Architecture of ResizableNet. We highlight the input and output tensor shape. Conv denotes convolution layer. SF denotes ShuffleNetV2 module. SE denotes Squeeze-and-Exciine Exciitatin module.
Training time (gain.)
Length
# of Networks
Individual
Random
Fair
8 Gds
8 Gds (+0%)
625
4,896 Gds*
32 Gds*
29 Gds (+10%)
6
15625
122,395 Gds
48 Gds*
35 Gds (+26%)
390,625
3,125,000 Gds*
64 Gds*
42 Gds (+52%)
Table 8: Training time comparison among training individual networks, ResizableNet with random sampling and fair sampling, with different number of scaling factors in the predefined scale list. runs by algorithm use ResNet-50 on ImageNet with Pytorch on 8 Nvidia 2080 GPUs with 100 epochs. Gds: GPUs per day. denotes the approximate training time. Number in the bracket represents the percentage of training time gain of fair sampling compare random sampling.
D ABLATION STUDY
D.1 IS NEURAL NETWORK NATURALLY RESIZABLE?
intuitive to ask the question: neural network naturally resizable? The answer is no. We present the result of directly inference neural network with different scaling configurations and compare the result with resizable neural network. The naive network is trained with scale-aware batch normalization. Figure D.I shows that naively trained neural network can not adjust feature map size, which result in a extremely low test accuracy.
70-
Imogohet kccurscy1s
80-
indivioual Shumeelet222
125
150
175
200
225
250
275
300
MFLOPs
Figure 5: Is network naturally resizable? We compare the result of sub-networks in naive Shuf- fleNetV2, RS-ShuffleNerV2 and SS-NAAS-SuffleNett
D.2 SCALE-AWARE BN ANALYSIS.
Scale aware batch normalization especially important for resizable augmentation. The following table compares two networks to adopt resizable augmentation with and without BN calibration. We
17

---

Under review as a conference paper at ICLR 2020
observe that without BN calibration, the resizable augmentation is only slightly better than the baseline model. The improvement negligible. On the other hand, when BN calibrated for scaling configurations, is boost the performance compare with the baseline. The accuracy is increased by 1.1% on ShuffleNetV2 and 2.4% on ResNet-50.
Table 9: This table shows the result of Top-1 accuracy of ResNet-50 and ShuffleNerV2 use resizable augmentation technique with (w/) and without (w/o spatial BN calibration.)
Model
Baseline
w/o BN Calibration
w/ BN Calibration
Resizable-Aug ShufflenetV2
72.6%
72.8%
73.7%
Resizable-Aug ResNet-50
76.4%
76.9%
78.8%
D.3 FAIR SAMPLING ANALYSIS.
We compare the results of random sampling and fair sampling in this section. We the scaling factor list L = (0.5, 0.75, 1.0) on MobileNetVI. The random sampling is trained with three times more epochs than fair sampling. We showed some results in Table 11. The results show that the performance of different scaling configuration is not consistent. By consistency, we mean that some scaling configuration achieves better accuracy, while some scaling configuration achieves lower accuracy, using fair sampling we can get more stable results.
Table 10: This table shows the result of Top-1 accuracy of ResizableAug ResNet-50 by various scaling configuration.
Random Sampling FLOPs
Fair Sampling.
Acc.
FLOPs
Acc.
569M
72.0%
569M
71.9%
(-0.1%)
462M
71.3%
462M
69.3% (-2.0%)
391M
70.7%
391M
71.0% (+0.3%)
343M
70.2%
343M
70.7% (+0.5%)
291M
69.5%
291M
65.7% (-3.7%)
242M
68.9%
242M
68. 1 % (-0.8%)
Table 11: This table shows the result of Top-1 accuracy of ResizableAug ResNet-50 by various scaling configuration.
18


Reviews:
[{"review_id": "Syl6fYR_OH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In a standard multi-stage neural network such as a ResNet, the resize operation between stages typically reduces the spatial resolution by 0.5.  (input ---> stage 1---> 0.5 reduction ---> stage 2 --> 0.5 reduction ---> ....). In this paper the authors apply a variety of reduction factors in between these stages for different training examples (input ---> stage 1---> variable reduction ---> stage 2 --> variable reduction ---> ....). They demonstrate that simply training in this way is a powerful form of augmentation. It also produces a network which may be scaled depending on a FLOP budget.\n\nI think this is a really neat idea, and as far as I'm aware it is novel.  It is similar in spirit to EfficientNet, although more flexible. The experimental results are good. However, the paper is let down by poor writing and a lack of detail.\n\nThe paper needs a rewrite as there are many grammatical errors, which cause a bad impression:\n\n- \"performs arbitrary resize operation\" ->  \"performs arbitrary resize operations\"\n- \"Scale is the fundamental component of the physical world\" ---> *a* fundamental component!\n- \"i.e the acuracy of NISP\" --> e.g.\n- \"compare with baseline\" -> \"compared to the baseline\"\n- There are several instances of a space missed out between a letter and an open bracket \n- \"The Figure 2(a)\" --> \"Figure 2(a)\"\n\nThe comparison to weight-sharing NAS methods is unnecessary. Those entail searching for architectures and sharing weights through the search process, whereas in this work it is having a network that can take different sized inputs at different stages. On that note, it doesn't feel right to me to refer to there being many \"subnetworks\". What there really is, is just a single network that is robust to multi-scaled inputs (which is a good thing!)\n\nFigure 1 is nice!\n\nEfficientNet-B0 is a base-network that can be scaled up with a compound scaling approach found through grid search. What happens if you scale up your Resizeable net to the same FLOPs as e.g. EfficientNet-B7?\n\nI like the old-school vision citations, although referring to object detection makes me wonder why there are no experiments on it. For distillation, I recommend you cite https://arxiv.org/abs/1312.6184, as the Hinton paper is really just an extension of this.\n\nThe fair sampling seems important to the method. Could a detailed explanation be included in the appendix? Do you given the performance as a result of naive sampling? What do you mean by \"Some convolutional layers can go through more training issues than others\"? \n\nA big problem in this paper is that (as far as I can tell) the scaling factors considered aren't given (but we are told that they lie between 0 and 1). It isn't possible to sample these arbitrarily as you indicate that a batch-norm layer is needed for each one, so these must be discrete (because of this, it isn't correct to say that you have infinite networks contained within).  It therefore isn't clear e.g. in Table 1 which permutation of scaling factors were used in upsampling the networks. Are the results given representative of the best-case selection of these scaling? I hope this isn't the case. I am assuming it is uniform scaling in the case of the \"individually trained counterparts\". It would be interesting to know more generally which combinations worked well.\n\nIn Section 3.5 I'm not sure what is meant by \"It is inefficient to process all images to one-subnetwork, as the algorithm spends equal energy at each sample\".  I assume energy use is proportional to the number of FLOPS, which in turn depends on spatial resolution.\n\nThe legend in Figure 2 is close-to unreadable and needs changing.\n\nThe results are impressive, but error bars would be appreciated if possible. As ImageNet is the only dataset considered, this would give some needed clout. \n\nPros\n-------\n\n- Nice, novel method\n- Good experimental results\n\nCons\n--------\n\n- Paper is poorly written\n- Very few details of the scaling factor variations\n- Only one dataset considered\n\nAlthough the paper is written badly and the narrative is muddled, the underlying idea is a nice one, which is executed well experimentally. Because of this I would like to recommend a Weak Accept, subject to the authors (i) doing a rewrite and (ii) including more information regarding the scaling permutations. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HJeQ86Ug9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method that involves multi-scale inputs for each layer that could be used as network architecture search or data agumentation or \n\nPros)\n(+) The idea looks interesting.\n(+) The experimental results look promising.\n\nCons)\n(-) Many typos when denoting figures and tables. See the minor comments below.\n(-) I believe the authors could organize the paper better. Tables and figures that are referred in a page are hard to find quickly. I recommend the authors refine the paper again for better readability.\n(-) Some notations (such as RS-, RS-NAS, and so on) are so vague that hard to follow.  \n(-) I recommend the authors redraw all the figures for clarity. For example, each legend in Figure 2 is hard to take a look at.\n(-) + the comments below.\n\nComments)\n- When doing feature map resize in terms of the resolution, why Bilinear sampling was chosen? Could the authors provide a comparison with other sampling methods?\n- In the related work section for dynamic neural networks, the authors claimed that \"Most dynamic networks methods sacrifice accuracy in exchange of adaption in inference\", but it seems to be quite overclaimed. As shown in the paper [1],  one can find that the author presented they could improve both accuracy and efficiency.\n- How did you find the architecture shown In Figure 3 in the Appendix? What is Xception? Please specify the details.\n-  Designing the pre-defined spatial list of L looks critical, so the authors should describe L in the implementation details. \n- One of the main problem I think is the training budget issue. According to algorithm 1, the inner loop of \"for j=0,..,len(L)\", the overall training time will clearly take L times longer than that of the training setting w/o resizable training. Thus, it does not seem to be fair comparison in terms of the training budget. Namely, it seems that the authors compared with the other data augmentation methods which spend much less training budgets.\n- Hard to grasp Section 3.5 of Adaptive resizable neural network. ResizeLearner looks being attached at the last stage of the original network after the original network is trained, but there is no further information about what ResizeLearner learns and how ResizeLearner selects the optimal sub-network. \n\n[1] Universally Slimmable Networks and Improved Training Techniques, https://arxiv.org/pdf/1903.05134.pdf.\n\nMinor comments)\n- Wrong section and figure references:\n  - 'It also mitigates the co-adaptation issue which we will discuss in Section 3.3'(indeed it is Section 3.4), 'The network architecture along with feature map resolution and channels number are shown in Figure 4' (it should be Figure 3).\n  - - Figure 3(d) referred to in section 4.4  would be Figure 2(d) indeed.\n\nAbout rating)\nThe authors provided a novel technique about the resizable approach and the experimental results look promising. However, the paper needs to be revised and looks like it does not ready to be published now. If the authors could revise the paper and concern my comments well, I would increase my rating. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygNawPTYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Resizable Neural Networks, which trains networks with different resolution scalings at the same time with shared weights. It serves as data-augmentation and improves accuracy over base networks. Additionally, the same technique can perform an architecture search. Experimental results show significant accuracy gains.\n\nThe reported accuracy gains are substantial. The proposed method is potentially useful in many applications. However, several details are missing or hard to understand. Without additional descriptions, it is not straightforward to implement the method. Thus, I suggest for rejection. The score might be raised depending on updates and code release.\n\nMajor comments:\n1) Algorithm 1 has \"predefined spatial list L.\" How to choose it in practice?\n2) Algorithm 1 indicates that the training time is len(L) times longer. Additionally, according to the implementation details, Resizable Networks are trained two times many epochs. It seems hard to justify such a longer training time.\n3) This is similar to (2), but why Resizable-NAS is better than the all len(L) models separately to find better architectures?\n4) In Sec. 3.4, how the \"target model\" is selected after the training of Resizable Networks?\n5) Why is Resizable-Adapt omitted from Table 2?\n6) References are out-dated. For example, there are no \"Figure 5.\"\n\nMinor comments:\n1) Are any ablation studies available for Fair Sampling?\n\n===== Update\n\nThank you for the response and update. The revision made the paper easier to understand. However, I still have concerns about the presentation of the paper, and I keep my score. I think the novelty and experimental results are significant and match the bar of ICLR. If the other two reviewers think that the revised paper is fairly well-written and recommend acceptance, I will not challenge the decision.\n\nMajor comments:\n1) I do not understand how the training times of random-sampling in Appendix C were estimated.\n2) Concerning Table 11, how was the result when we randomly select each scale and train network with the same number of epochs with fair-sampling? If there were no much difference with the result in Table 11, it seems fair sampling does not have clear advantages over random sampling. If so, I suggest to alter fiar-sampling by random-sampling and reduce the complexity of the proposed method.\n\nMinor comments:\n1) Table 10 has two captions.\n2) I did not understand that L is a list of possible scaling factors (scalar) in the initial review. I guess it is partially because I did not understand why the number of sampling should be len(L) for each mini-batch (it is actually due to fair-sampling). I think adding some remarks on it will help to make the pseudo-code easier to understand. (nit: for j= 0, ..., len(L) should be for j=0, ..., len(L) -1 or for j=1,  ..., len(L))", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper offers likely novel schemes for image resizing.  The performance improvement is clear.  Unfortunately two reviewers find substantial clarity issues in the manuscript after revision, and the AC concurs that this is still an issue.  The paper is borderline but given the number of higher ranked papers in the pool is unable to be accepted unfortunately. 
Decision: Reject