Fast Neural Network Adaptation via Parameter Remapping and Architecture Search
Authors: Jiemin Fang*
Yuzhu Sun*
Kangjian Peng*
Qian Zhang
Yuan Li
Wenyu Liu
Xinggang Wang
Venue:  (2020)

Abstract: Deep neural networks achieve remarkable performance in many computer vision tasks. Most state-of-the-art~(SOTA) semantic segmentation and object detection approaches reuse neural network architectures designed for image classification as the backbone, commonly pre-trained on ImageNet. However, performance gains can be achieved by designing network architectures specifically for detection and segmentation, as shown by recent neural architecture search (NAS) research for detection and segmentation. One major challenge though, is that ImageNet pre-training of the search space representation (a.k.a. super network) or the searched networks incurs huge computational cost. In this paper, we propose a Fast Neural Network Adaptation (FNA) method, which can adapt both the architecture and parameters of a seed network (e.g. a high performing manually designed backbone) to become a network with different depth, width, or kernels via a Parameter Remapping technique, making it possible to utilize NAS for detection/segmentation tasks a lot more efficiently. In our experiments, we conduct FNA on MobileNetV2 to obtain new networks for both segmentation and detection that clearly out-perform existing networks designed both manually and by NAS. The total computation cost of FNA is significantly less than SOTA segmentation/detection NAS approaches: 1737$\times$ less than DPC, 6.8$\times$ less than Auto-DeepLab and 7.4$\times$ less than DetNAS. The code is available at https://github.com/JaminFong/FNA .

Published as a conference paper at ICLR 2020
FAST NEURAL NETWORK ADAPTATION VIA PARAME- TER REMAPPING AND ARCHITECTURE SEARCH
Jiemin Fang' -1 Yuzhu Sun
" Kangjian Peng"; Qian Zhang", Yuan Li,
Wenyu Liu', Xinggang Wang
School of EIC, Huazhong University Science and Technology Hoorizon Robotics [jaminfong, yzsun, 1iuwy, xgwang) @hust. edu. kkangjian peng, gian01.2hang yuan. yuan.lijene faa..1i)horizon.dl
ABSTRACT
Deep neural networks achieve remarkable performance in many computer vi- sion tasks. Most state-of-the-art (SOTA) semantic segmentation and object de- tection approaches reuse neural network architectures designed for image clas- sification as the backbone, commonly pre-trained on ImageNet. However, per- formance gains can be achieved by designing network architectures specifically for detection and segmentation, as shown by recent neural architecture search (NAS) research for detection and segmentation. One major challenge though, is that ImageNet pre-training of the search space representation (a.k.a. super net- work) or the searched networks incurs huge computational cost. In this paper, we propose a Fast Neural Network Adaptation (FNA) method, which can adapt both the architecture and parameters a seed network (e.g. a high perform- ing manually designed backbone) to become a network with different depth, width, or kernels via a Parameter Remapping technique, making possible to utilize NAS for detection/seemmentttion tasks lot more efficiently. In periments, we conduct FNA on MobileNetV2 to obtain new networks for both segmentation and detection that clearly out-perform existing networks designed both manually and by NAS. The total computation cost of FNA is significantly less than SOTA segmentation/detectionn NAS approaches: 1737x less than DPC, 6.8x less than Auto-DeepLab and 7.4x x less than DetNAS. The code is available at https::/githin com/ Jami nF womminoong//iioom oon//NN......
INTRODUCTION
Deep convolutional neural networks have achieved great successes in computer vision tasks such as image classification (Krizhevsky et al., 2012; He et al., 2016; Howard et al., 2017), semantic segmentation (Long et al., 2015; Ronneberger al., 2015; Chen al., 2017b) and object detec- tion (Ren et al., 2015; Liu et al., 2016; Lin et al., 2017) etc. Image classification has always served as a fundamental task for neural architecture design. It is common to use networks designed and pre-trained on the classification task as the backbone and fine-tune them for segmentation or detec- tion tasks. However, the backbone plays an important role in the performance on these tasks and the difference between these tasks calls for different design principles of the backbones. For example, segmentation tasks require high-resolution features and object detection tasks need to make both localization and classification predictions from each convolutional feature. Such distinctions make neural architectures designed for classification tasks fall short. Some attempts (Li et al., 2018; Wang al., 2019) have been made tackle this problem.
Handcrafted neural architecture design is inefficient, requires a lot of human expertise, and may not find the besf-performing networks. Recently, neural architecture search (NAS) methods (Zoph et al., 2017; Pham et al., 2018; Liu et al., 2018) see a rise in popularity. Some works (Liu et al.,
'Equal contributions.
'The work is performed during an internship at Horizon Robotics.
1 *Correspnndigg author.

---

Published as a conference paper at ICLR 2020
Seed Network
Super Network
Target Architecture
Target Network
N₂
N sup
Archt
N₁
Network Expansion
Architecture Adaptation
Parameter Adaptation
Q
0
Params
Parameter
Kemapping
Figure 1: The framework of our proposed FNA. Firstly, we select an artificially designed network as the seed network N, and expand N, to super network Naup which is the representation of the search space. Then parameters of Ar are remapped Nsup. We utilize the NAS method start the architecture adaptation with the super network and obtain the target architecture Arch,. Before parameter adaptation, we remap the parameters N,, Arch,. Finally, we adapt the parameters of Arch; to get the target network N,.
2019a; Zhang et al., 2019; Chen et al., 2019b) propose to use NAS to design backbone architectures specifically for segmentation or detection tasks. Nevertheless, pre-training remains inevitable but costly procedure. Though some (He et al., 2018) recently demonstrates that pre-training is not always necessary for accuracy considerations, training from scratch on the target task still takes more iterations than fine-tuning from a pre-trained model. For NAS methods, the pre-training cost non-negligible for evaluating the networks in the search space. One-shot search methods (Brock et al., 2017 7; Bender et al., 2018; 8; Chen et al., 2019b) integrate all possible architectures in one super network but pre-training the super network and the searched network still bears huge computation cost.
As ImageNet (Deng et al., 2009) pre-training has been a standard practice for many computer vi- sion tasks, there are lots of models trained on ImageNet available in the community. To take full advantages of these models, we propose Fast Neural Network Adaptation (FNA) method based on novel parameter remapping paradigm. Our method can adapt both the architecture and parameters of one network to a new task with negligible cost. Fig. shows the whole framework. The adapta- tion is performed on both the architecture- and parameter-lewe We adopt the NAS methods (Zoph et al., 2017; Real et al., 2018; Liu et al., 2019b) to implement the architecture-lew adaptation. We select manually designed network (MobileNetV2 (Sandler et al., 2018) in experiments) the seed net work, which is pre-trained on ImageNet. Then, we expand the seed network to a super network which is the representation of the search space in FNA. New parameters in the super net- work are initialized by mapping those from the seed network using parameter remapping. Thanks to that, the neural architecture search can be performed efficiently on the detection and segmentation tasks. With FNA we obtain new optimal target architecture for the new task. Similarly, we remap the parameters of the seed network the target architecture for initialization and fine-tune tune on the target task with no need of pre-training on a large-scale dataset.
We demonstrate FNA's effectiveness and efficiency via experiments on both segmentation and de- tection tasks. We adapt the manually designed network MobileNetV2 (Sandler et al., 2018) to seg- mentation framework DeepLabv3 (Chen et al., 2017b), detection framework RetinaNet (Lin et al., 2017) and SSDLite (Liu et al., 2016; Sandler et al., 2018). Networks adapted by FNA surpass both manually designed and NAS searched networks in terms of both performance and model MAdds. Compared to NAS methods, FNA costs 1737 x less than DPC (Chen et al., 201 8a), 6.8x less than Auto-DeepLan (Liu et al., 2019a) and 7.4x less than DetNAS (Chen et al., 2019b).
RELATED WORK
Neural Architecture Search With reinforcement learning (RL) and evolutionary algorithm (EA) being applied to NAS methods, many works (Zoph & Le, 6; Zoph al., 2017; Real et al., 2018)
2

---

Published as a conference paper at ICLR 2020
make great progress in promoting the performances of neural networks. Recently, NAS methods based on one-shot model (Brock et al., 2017; Bender et al., 2018) or differentiable representa- tions (Liu al., 2019b; Cai et al., 2019; Fang et al., 2019b) achieve remarkable results with low search cost. We use the differentiable NAS method to implement architecture adaptation, which adjusts the architecture of the backhone network automatically.
Backbone Design As deep neural network designing (Simonyan & Zisserman, 2014; Szegedy et al., 2016; He et al., 2016) develops, the backbones of segmentation or detection networks evolve accordingly. Some works improve the backbone architectures by modifying existing networks. PeleeNet (Wang et al., 2018) proposes a variant of DenseNet (Huang et al., 2017) for more real- mobile devices. DetNet (Li et al., 2018) applies dilated convolution (Yu
time object detection & Koltun, 2016) in the backbone to enlarge the receptive field which helps to detect objects. BiSeNet (Yu et al., 2018) and HRNet (Wang al., 2019) design multiple paths learn both high- and low- resolution representations for better dense prediction.
Parameter Remapping Net2Net (Chen et al., 2015) proposes the function funetion-preserving transfor- deeper wider network. This remapping
mations remap the parameters of one network mechanism accelerates the training of the new larger network and achieves great performances. Following this manner, EAS (Cai et al., 2018) extends the parameter remapping concept to neural architecture search. Moreover, some NAS works (Pham et al., 2018; Fang et al., 2019a; Elsken al., 2019) apply parameters sharing on child models to accelerate the search process. Our pa- rameter remapping paradigm extends the mapping dimension with the kernel level. Parameters can be also mapped shallower or narrower network with our scheme, while Net2Net focuses on mapping parameters to a deeper and wider network. The parameter remapping in our FNA largely decreases the computation cost the network adaptation by taking full advantages of the ImageNet pre-trained parameters.
3
METHOD
As the most commonly used network for designing search spaces in NAS methods (Tan et al., 2018; Cai et al., 2019; Fang et al., 2019b), MobileNetV2 (Sandler et al., 2018) is selected as the seed network to give the details of our method. To adapt the network to segmentation and detection tasks, adjust the architecture elements three levels, i.e., convolution kernel size, depth and width of the network. In this section, we first describe the parameter remapping paradigm. Then we explain the whole procedure of the network adaptation.
3.1 PARAMETER REMAPPING
We define parameter remapping as one paradigm which aims at mapping the parameters of one seed network another one. We denote the seed network as N, and the new network Nn, whose parameters are denoted as W, and W,, respectively. The remapping paradigm is illustrated in the following three aspects. The remapping on the depth-level firstly carried out and then the remapping on the kernel- and width- level is conducted simultaneously. Moreover, we study different remapping strategies in the experiments (Sec. 4.5).
Remapping on Depth-level We introduce more depth settings in our architecture adaptation pro- cess. other word, we adjust the number of inverted residual blocks (MBConvs) (Sandler et al., 2018) in every stage of the network. We assume that one stage in the seed network N, has I layers. The parameters of each layer can be denoted as {W?"), w ! Similarly, we assume that the corresponding stage with m layers in the new network N₂ has parameters (W w
W(")). The remapping process on the depth-level is shown in Fig. 2(a). The parameters of layers in N, which also exit in N,, are just copied from Ag. The parameters of new layers are all copied from the last layer the stage of Ns. It is formulated as
f(i) = min(i,
1(0 W((())
VI visism m.
Remapping on Width-level In the MBConv block of MobileNetV2 (Sandler et al., 2018) net- work, the first point-wise convolution expands the low-dimensional features high dimension.
3

---

Published as a conference paper at ICLR 2020
(a) Depth level
Width level
(c) Kernel level
Figure 2: Our parameters are remapped on three levels. (a) shows the depth-level remapping. The parameters of existing corresponding layers mapped from the original network. The parameters of new layers are mapped from the last layer in the original network. (b) shows the width-level remapping. For each channels-diminished dimension, the parameters are copied from the original existing ones. (c) shows the kernel-level remapping. The original parameters are mapped to the central part of the new larger kernel. The values of the other parameters assigned with 0.
This practice can be utilized for expanding the width and capacity one neural network. We allow smaller expansion ratios for architecture adaptation. We denote the parameters of one convolution in N as W E |Xxxxxxx and that Nr as W E RrxsxxXw where r S and 8 S q. As shown in Fig. 2(b), on the width-level, directly map the first channels of parameters in M, to the narrower one N₁, It can be formulated as
W Kiie
Wi iiga
VI visiiiiijss
(2)
Remapping on Kernel-level The kernel size is commonly set as 3 x 3 in most artificially-desgeee networks. To expand the receptive field and capture abundant features in segmentation detection tasks, we introduce larger kernel size settings in the adaptation process. As Fig. 2(c) shows, to expand the 3 x kernel large kernel with the values of the original x 3 kernel. The values of the other region surrounding the central part are assigned with We denote the parameters of the original and the larger & x kernel W The remapping process on kernel-leve. can be formulated as follows.
larger one, we assign the parameters of the central
region in the
kernel as W
W333 Salhaa
if(k (0--3) 3)/2 h, w 5 (k: 3//
w kith,w
(3)
otherwise
where h, w denote the indices of the spatial dimension. This design principle conforms to the cunction-preserving concept (Chen et al., 2015), which accelerates and stabilizes the optimization of the new network.
3.2 NEURAL NETWORK ADAPTATION
We divide our neural network adaptation into three steps. Fig. demonstrates the whole adaptation procedure. Firstly, we expand the seed network to super network which is the representation of the search space in the latter architecture adaptation process. Secondly, we perform the differen- tiable NAS method to implement network adaptation on the architecture-leell and obtain the target architecture Arch,- Finally, we adapt the parameters of the target architecture and obtain the tar- get network NR. The aforementioned parameter remapping mechanism is deployed before the two stages, i.e. architecture adaptation and parameter adaptation.
Network Expansion We expand the seed network to a super network by introducing more options for architecture elements. For every MBConv layer, we allow for more kernel size settings {3, 5, 7) and more expansion ratios {3, 6). As most differentiable NAS methods (Liu et al., 2019b; Cai et al., 2019; Wu et al., 2018) do, we relax every layer as a weighted sum of all candidate operations.
expla (1)
o (x)
oEO Eveo ( o(x),
Σ
4

---

Published as a conference paper at ICLR 2020
Table 1: Semantic segmentation results on Cityscapes. OS: output stride, the spatial resolution ratio of input image backbone output. The result of DPC in the brackets is our implemented version
under the same settings as FNA. The MAdds of the models are computed with the 1024 x 2048 input resolution.
Method
os
iters
Params
MAdds
mIOU( %
MobileNetV2 (Sandler al., 2018)
2.57M
24.52B
75.5
DPC (Chen al., 2018a)
DeepLabv3
100K
2.51M
24.69B
75.4(75.7)
FNA
2,47M
24.17B
76.6
Auto-DeepLab-S (Liu al., 2019a)
500K
10.15M
333.25B
75.2
FNA
D
100K
5.71M
210.11B
77.2
FNA
100K
5.71M
313.87B
78.0
Table 2: Comparisons of computational cost on the semantic segmentation tasks. ArchAdapt: Ar- chitecture Adaptation. ParamAdapt: Parameter Adaptation. GHs: GPU Hours. * indicates the computational cost calculated under reproducing settings. indicates the cost estimated accord- ing to the description in the original paper.
Method
Total Cost
ArchAdapt Cost
ParamAdapt Cost
DPC (Chen et al., 2018a)
62.2K GHs
62.2K GHs
30.0 GHs
Auto-DeepLab-S (Liu al., 2019a)
244.0 GHs
72.0 GHs
172.0 GHs
FNA
35.8 GHs
1.4 GHs
34.4 GHs
where O denotes the operation set, og denotes the architecture parameter of operation in the ith layer, and æ denotes the input tensor. We set more layers in one stage of the super network and add the identity connection to the candidate operation set for depth search. After expanding the seed network to a super network, we remap the parameters of the seed network to the super network based the paradigm illustrated in Sec. 3.1. This remapping strategy prevents the huge cost of ImageNet pre-training involved in the search space, i.e. the super network differentiable NAS.
Architecture Adaptation We start the differentiable NAS process with the expanded super net- work directly the target task, e.g., semantic segmentation object detection. We first fine-tune operation weights of the super network for some epochs on the training dataset. After the weights are sufficiently trained, we start alternating the optimization of operation weights with a2 and architecture parameters a with ac Za To accelerate the search process and decouple the parameters of different sub- nub-networks, only sample one path in each iteration according to the distribution of architecture parameters for operation weight updating. As the search process terminates, we use the architecture parameters derive the target architecture.
Parameter Adaptation We obtain the target architecture Arch, from architecture adaptation. To accommodate the new segmentation or detection tasks, the target architecture becomes different from that of the seed network N₂ (which is primitively designed for the image classification task). Unlike conventional training strategy, we discard the cumbersome pre-training process of Arch, on ImageNet. We remap the parameters of Nr to Arch, utilizing the method described in Sec. 3.1. Finally, we directly fine-tune Arch; on the target task and obtain the final target network Ng.
EXPERIMENTS
We select the ImageNet pre-trained MobileNetV2 model as the seed network and apply our FNA method on both semantic segmentation and object detection tasks. In this section, we firstly give the implementation details of our experiments; then we report and analyze the network adaptation results; finally, we perform ablation studies to validate the effectiveness of the parameter remapping paradigm and compare different parameter remapping implementations.
4.1 NETWORK ADAPTATION ON SEMANTIC SEGMENTATION
The semantic segmentation experiments are conducted on the Cityscapes (Cordts et al., 2016) dataset. In the architecture adaptation process, map the seed network to the super network,
5

---

Published as a conference paper at ICLR 2020
Table 3: Object detection results on MS-COCO. The MAdds are calculated with 1088 x 800 input.
Method
Params
MAdds
mAP (%)
ShuffleNetV2-20 (Chen et al., 2019b)
13.19M
132.76B
32.1
MobileNetV2 (Sandler et al., 2018)
11.49M
133.05B
32.8
RetinaNet
DetNAS (Chen et al., 2019b)
13.41M
133.26B
33.3
FNA
11.73M
133.03B
33,9
NeVV (Sandler et al., 2018)
4.3M
0.8B
22.1
Mnasnet-92 (Tan et al., 2018)
SSDLite
5.3M
1.0B
22.9
FNA
4.6M
0.9B
23.3
Table 4: Comparison of computational cost on the object detection tasks, All our experiments on object detection are conducted on TITAN-Xp GPUs.
Super Network
Target Network
Method
Total Cost
Pre-training GDs
Finetuning
Search
Pre-training
Finetuning
DetNAS (Chen et al., 2019b)
68 GDs
GDs
20 GDs
12 GDs
12 GDs
FNA (RetinaNet)
9.2 GDs
6 GDs
3.2 GDs
FNA (SSDLite)
21.6 GD>s
6.6 GDDs
15 GDDs
which is used the backbone of DeepLabv3 (Chen al., 2017b). We randomly sample 20% im- ages from the training set as the validation set for architecture parameters updating. The original validation set not used in the search process. To optimize the MAdds of the searched network, we define the loss function in search as L = Ltask + À log, (cost). The first term denotes the cross- entropy loss and the second term controls the MAdds of the network. We set À as 9 10 and r as 45. The search process takes 80 epochs in total. The architecture optimization starts after 30 epochs. The whole search process conducted on a single V100 GPU and takes only 1.4 hours in total.
In the parameter adaptation process, remap the parameters of original MobileNetV2 to the target architecture obtained in the aforementioned architecture adaptation. The whole parameter adaptation process is conducted on TITAN-Xp GPUs and takes 100K iterations, which cost only 8,5 hours in total.
Our semantic segmentation results are shown in Tab. 1. The FNA network achieves 76.6% mIOU on Cityscapes with the DeepLabv3 (Chen et al., 201 framework, 1.1% mIOU better than the man- ually designed seed Network MobileNetV2 (Sandler et al., 2018) with fewer MAdds. Compared with a NAS method DPC (Chen et al., 201 20) (with MobileNetV2 as the backbone) which searches multi-scale module for semantic segmentation tasks, FNA gets 0.9% mIOU promotion with 0.52B fewer MAdds. For fair comparison with auto-DeepLat (Liu et al., 2019a) which searches the back- bone architecture on DeepLabv3 and retrains the searched network on DeepLabv3+ (Chen et al., 2018b), we adapt the parameters of the target architecture Arch, to DeepLabbv3+ framework. Com- paring with Deep-Deep,..--, FNA achieves far better mIOU with fewer MAdds, Params and train- ing iterations. With the remapping mechanism, FNA only takes 35.8 GPU hours, 1737x less than DPC and 6.8x less than Auto-DeepLab.
4.2 NETWORK ADAPTATION ON OBJECT DETECTION
We further implement our FNA method on object detection tasks. We adapt the MobileNetV2 seed network to two commonly used detection systems, RetinaNet (Lin et al., 2017) and a lightweight one SSDLite (Liu et al., 2016; Sandler et al., 2018). The experiments are conducted on the MS-COCO dataset (Lin et al., 2014b). Our implementation is based on the MMDetection (Chen et al., 2019a) framework. In the search process of architecture adaptation, we randomly sample 50% data from the original ra i nva 35k set as the validation set.
We show the results on the COCO dataset in Tab. 3. In the RetinaNet framework, compared with two manually designed networks, ShuffleNetV2-10 (Ma et al., 2018; Chen et al., 2019b) and Mo- bileNetV2 (Sandler et al., 2018), FNA achieves higher mAP with similar MAdds. Compared with DetNAS (Chen et al., 2019b) which searches the backbone of detection network, FNA achieves 0.6% higher mAP with 1. 1.64M fewer Params and 0.2B fewer MAdds. As shown in Tab. 4, our total implement our FNA method the SS- DLite framework. In Tab. 3, FNA surpasses both the manually designed network MobileNetV2 and the NAS searsher network MnasNet-92, where MnasNet (Tan al., 2018) takes around 3.8K GPU
computation cost is only 13.5% of DetNAS. Moreover,

---

Published as a conference paper at ICLR 2020
days search for the backbone network ImageNet. The specific cost FNA takes SSDLite is shown in Tab. 4. is difficult to train the small network due to the simplification (Liu et al., 2019c). Therefore, experiments
SSDLite need long training schedule and take larger computation cost. The experimental results further demonstrate the efficiency and effectiveness of direct adaptation on the target task with parameter remapping.
4.3 EFFECTIVENESS OF PARAMETER REMAPPING
To evaluate the effectiveness of the parameter remapping paradigm in our method, we attempt to op- tionally remove the parameter remapping process before the two stages, i.c. architecture adaptation and parameter adaptation. The experiments are conducted with the DeepLabv3 (Chen al., 2017b) semantic segmentation framework on the Cityscapes dataset (Cordts et al., 2016).
Table 5: Effectiveness evaluation Parameter Remapping. The experiments are conducted with
DeepLabv3 on Cityscapes. Remap: Parameter Remapping. ArchAdapt: Architecture Adaptation.
RandInit: Random Initialization. Pretrain: ImageNet Pretrain. ParamAdapt: Parameter Adaptation.
Row Num Method
MAdds(B) mIOU(%)
Remap + ArchAdapt + Remap + ParamAdapt (FNA)
24.17
76.6
(2)
RandInit -> ArchAdapt - Remap -> ParamAdapt
24.29
76.0
(3)
Remap + ArchAdapt + RandInit + ParamAdapt
24.17
73.0
(4) RandInit -> ArchAdapt - RandInit - ParamAdapt
24.29
72.4
(5) Remap - ArchAdapt - Retrain + ParamAdapt
24.17
76.5
In Row (2) we remove the parameter remapping process before architecture adaptation. In other word, the search is performed from scratch without using the pre-trained network. The mIOU in Row (2) drops by 0.6% compared to FNA in Row (1). Then remove the parameter remapping before parameter adaptation in Row (3), i.e. training the target architecture from scratch on the target task. The mIOU decreases by 3.6% compared the result of FNA. When remove the parameter remapping before both stages in Row (4), it gets the worst performance. In Row (5), we first pre- train the searched architecture on ImageNet and then fine-tune
the target task. It is worth noting
that FNA even achieves a higher mIOU by a narrow margin (0. (0) than the ImageNet pre-trained one Row (5). We conjecture that this may benefit from the regularization effect of parameter remapping before the parameter adaptation stage.
All the experiments are conducted using the same searching and training settings for fair compar- isons. With parameter remapping applied on both stages, the adaptation achieves the best results in Tab. Especially, the remapping process before parameter adaptation tends provide greater performance gains than the remapping before architecture adaptation. All the experimental results demonstrate the importance and effectiveness of the proposed parameter remapping scheme.
Table 6: Results of random search experiments with the RetinaNet framework MS-COCO. Diff- Search: Differentiable NAS. RandSearch: Random Search. The other definitions of abbreviations the same Tab. 5.
Row Num
Method
MAdds(B)
mAP(%)
(1) DetNAS (Chen et al., 2019b)
133.26
33.3
(2)
Remap + DiffSearch + Remap + ParamAdapt (FNA)
133.03
33.9
(3)
Remap + RandSearch + Remap + Papa
133.11
33.5
(4)
RandInit - RandSearch + Remap - ParamAdapt
133.08
31.5
(5) Remap RandSearch RandInit + ParamAdapt
133.11
25.3
RandInit RandSearch RandInit ParamAdapt
133.08
24.9
4.4 RANDOM SEARCH EXPERIMENTS
We carry out the Random Search (RandSearch) experiments with the RetinaNet (Lin et al., 2017) framework on the MS-COCO (Lin et al., 2014a) dataset. All the results are shown in the Tab. 6. We purely replace the original differentiable NAS (DiffSearch) method in FNA with the random search method in Row (3). The random search takes the same computation cost as the search in FNA for fair comparisons. We observe that FNA with RandSearch achieves comparable results with

---

... (., When remapping the parameters on the width-level, we sort the values of |7| and map the parameters with the sorted top-q indices. More specifically, Published as a conference paper at ICLR 2020
original method. It further confirms that FNA is general framework for network adaptation and has great generalization. NAS is only an implementation tool for architecture adaptation. The whole framework of FNA can be treated as a NAS-method agnostic mechanism. It is worth noting that even using random search, our FNA still outperforms DetNAS (Chen et al., 2019h) with 0.2% mAP better and 150M MAdds fewer.
We further conduct similar ablation studies with experiments in Sec. 4.3 about the parameter remap- ping scheme Row (4) (6). All the experiments further support the effectiveness of the parameter remapping scheme.
4.5 STUDY ON PARAMETER REMAPPING
We explore more strategies for the Parameter Remapping paradigm. Similar to Sec. 4.3, all the experiments are conducted with the DeepLabv3 (Chen et al., 2017b) framework on the Cityscapes dataset (Cordts et al., 2016). We make exploration from the following respects. For simplicity, we denote the weights of the seed network and the new network on the remapping dimension (out- put/input channel) as W, (W!"". W!P)) and W
where y S p.
Remapping with BN Statistics on Width-level We review the formulation of batch normaliza- tion (loffe& & Szegedy, 2015) as follows,
Ij PB 0 +
y/s -
- 3,
(5)
where = (x/[") r denotes the p-dimensional input tensor of the ith layer, E Rp denotes the learnable parameter which scales the normalized data on the channel dimension. We compute the absolute values of 7 as |
we define weights remapping function in Algo. where the reference vector is hy]-
Algorithm 1:
Weights Remapping Function
Input: the seed weights W, and the new weights Wn. the reference vector v // get indices of topk values of the vector
2 a < topk-indices(p, k ?)
// sort the indices +sort(a)
for 1,2, ...,, do
( = -Wal())
end
Figure 3: Parameter Remapping on the
Output: W with remapped values
kernel-level with dilation setting.
Remapping with Weight Importance on Width-level We attempt to utilize a canonical form of convolution weights to measure the importance of parameters. Then we remap the seed network parameters with great importance the new network. The remapping operation is conducted based on Algo. as well. We experiment with two canonical forms of weights to compute the reference vector, the standard deviation of W, as (") std..( and the L' norm of W, as ||.... ( [])).
Table 7: Study the methods of Parameter Remapping.
Method
Width-BN
Width-Std
Width-L1
Kernel-Dilate
FNA
mIOU(% (6)
75.8
75.8
75.3
75.6
76.6
Remapping with Dilation Kernel-level We experiment with another strategy of parameter remapping on the kernel-level. Different from the cunction-preserviin method defined in Sec. 3.1, remap the parameters with dilation manner shown in Fig. 3. The values in the convolution
8

---

Published as a conference paper at ICLR 2020
kernel without remapping all assigned 0. It is formulated as
Wax3 sisha
h. w = 1 + k-1 and = 0, 1,2
Wokk Sadh,
(6)
otherwise
where W and w denote the weights of the new network and the seed network respectively, h, w denote the spatial indices.
Tab. 7 shows the experimental results. The network adaptation with the parameter remapping paradigm define in FNA achieves the best results. Furthermore, the remapping operation of FNA easier to implement compared to the several aforementioned ones. However, we explore limited number of methods to implement the parameter remapping paradigm. How to conduct the remap- ping strategy more efficiently remains a significative future work.
5 CONCLUSION
In this paper, we propose a fast neural network adaptation method (FNA) with a parameter remap- ping paradigm and the architecture search method. We adapt the manually designed network Mo- bileNetV2 to semantic segmentation and detection tasks on both architecture- and parameter- level. The parameter remapping strategy takes full advantages of the seed network parameters, which greatly accelerates both the architecture search and parameter fine-tuning process. With our FNA method, researchers and engineers could fast adapt more manually designed networks various frameworks on different tasks. As there are lots of ImageNet pre-trained models available in the community, we could conduct adaptation with low cost and do more applications, e.g., face recogni- tion, pose estimation, depth estimation, etc. We leave more efficient remapping strategies and more applications for future work.
ACKNOWLEDGEMENT
This work was supported by National Natural Science Foundation of China (NSFC) (No. 61876212, No. 61733007 and No. 61572207), National Key R&D Program of China (No. 2018YFB 1018YFB14026G00 1402600) and HUST-Horizon Computer Vision Research Center. We thank Liangchen Song, Yingqing Rao and Jiapei Feng for the discussion and assistance.
REFERENCES
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Under- standing and simplifying one-shot architecture search. In ICML, 2018.
Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. SMASH: one-shot model architecture search through hypernetworks. arXiv:/708.055 2017.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI, 2018.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In ICLR, 2019.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu. Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijic Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, and Dahua Lin. Mmdetection: Open mmlab detection toolbox and benchmark. 1rXiv::906.00 07155, 2019a.
Liang-Chicl Chen, George Papandreou, Iasonas Kokkinos. Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017a.
Liang-Chie Liang-Chich Chen, George Papandreou. Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv: 176.00007 2017b.

---

Published as a conference paper at ICLR 2020
Liang- Ching-Chieh Chen, Maxwell D. Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, and Jonathon Shlens. Searching for efficient multi-scale architectures for dense image prediction. In NeurIPS, 2018a.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018b.
Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. 1rXiv:151..05611 2015.
Yukang Chen, Tong Yang. Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun. Detnas: Neural architecture search on object detection. In NeurIPS, 2019b.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. In ICLR, 2019.
Jiemin Fang, Yukang Chen, Xinbang Zhang, Qian Zhang, Chang Huang, Gaofeng Meng, Wenyu Liu, and Xinggang Wang. EAT-ÑAS: elastic architecture transfer for accelerating large-scale neural architecture search. 0rXiv:/90J.05888 2019a
Jiemin Fang, Yuzhu Sun, Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang. Densely connected search space for more flexible neural architecture search. arXiv: 096077 2019b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Ross B. Girshick, and Piotr Dollár. 181.00811 2018
Rethinking imagenet pre-training.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv: 1704.04411,,, 2017.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Zeming Li, Chao Peng, Gang Yu, Xiangyu Zhang, Yangdong Deng, and Jian Sun. Detnet: Design backbone for object detection. In ECCV, 2018.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014a.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014b.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
10

---

Published as a conference paper at ICLR 2020
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan L. Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018.
Chenxi Liu, Blang-Chiel Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In CVPR, 2019a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019b.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In ECCV, 2016.
Zili Liu, Tu Zheng, Guodong Xu, Zheng Yang. Haifeng Liu, and Deng Cai. Training Phiining-time-t network for real-time object detection. 1909000000 2019c.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. ICLR, 2017.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. In ECCV, 2018.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. urXivvabs//802.01548,, 2018.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. NeurIPS, 2015.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In LNCS, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. urXiv::409. 1556, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. CVPR, 2016.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform- 2018.
aware neural architecture search for mobile. arXiv:
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. arXiv:1908.079) 2019.
J Wang, Xiang Li, and Charles X Ling. Pelee: real-time object detection system on mobile devices. In NewrlPS, 2018.
Bichen Wu, Xiaoliang Dai. Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. arXiv: 0rXiv:/8/2.000 18/2.0344, 2018.
Changqian Yu, Jingbo Wang. Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilat- eral segmentation network for real-time semantic segmentation. In ECCV, 2018.

---

Published as a conference paper at ICLR 2020
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016.
Yiheng Zhang, Zhaofan Qiu, Jingen Liu, Ting Yao, Dong Liu, and Tao Mei. Customizable architec- ture search for semantic segmentation. In CVPR, 2019.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. arXiv:1611.01578 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning transferable architectures for scalable image recognition. arXivv1707.07012, 2017.
A APPENDIX
A.1 IMPLEMENTATION DETAILS ON SEMANTIC SEGMENTATION
For architecture adaptation, the image is first resized to 512 1024 and 321 321 patches are randomly cropped as the input data. The output feature maps are down-sampled by the factor of 16. Depthwise separable convolutions are used in the ASPP module (Chen et al., 2017a;b). The stages where the expansion ratio of MBConv is 6 in the original MobileNetV2 are searched and adjusted. We set the maximum numbers of layers in each searched stage of the super network as (4,4, 6, 6, 1}. We set a warm-up stage in the first 5 epochs to linearly increase the learning rate from 1 X 10-4 to 0.02. Then, the learning rate decays to 1 X 10 with the cosine anncaling schedule (Loshchilov & Hutter, 2017). The batch size is set as 16. We use the SGD optimizer with 0.9 momentum and 5x 10- weight decay for operation weights and the Adam optimizer (Kingma & Ba, 2015) with 4 x 10-5 weight decay and fixed learning rate 1 x 10-3 for architecture parameters.
For parameter adaptation, the training data is cropped as a 769 x 769 patch from the rescaled image. The scale is randomly selected from [0.75, 1.0, 1.25, 1.5, 1.75, 2.0]. The random left-right flipping is used. We update the statistics of the batch normalization (BN) (loffe & Szegedy, 2015) for 2000 iterations before the fine-tuning process. We use the same SGD optimizer as the search process. The learning rate linearly increases from 1 x 10 to 0.01 and then decays to 0 with the polynomial schedule. The batch size is set 16.
A.2 IMPLEMENTATION DETAILS ON OBJECT DETECTION
RetinaNet We describe the details in the search process of architecture adaptation as follows. The depth settings each searched stage are set as (4,4, 4, 4, 4, 4, 1}. For the input image size, the short side is resized to 800 while the maximum long side is set as 1333. For operation weights, we use the SGD optimizer with 1 X 10-4 weight decay and 0.9 momentum. We set a warm-up stage in the first 500 iterations to linearly increase the learning rate from 0 to 0.02. Then we decay the learning rate by a factor of 0.1 at the 8th and 1 1thh epoch. For the architecture parameters, we use the Adam optimizer (Kingma& & Ba, 2015) with 1 x 10 weight decay and fixed learning rate 3x 10-4, For the multi-objective loss function, we set À as 0.02 and as 10. We begin optimizing the architecture parameters after 8 epochs. We remove the random flipping operation input images in the search process. All the other training settings are the same as MMDetection (Chen et al., 2019a) implementation.
For fine-tuning of the parameter adaptation, we use the SGD optimizer with 5 x 10-5 weight decay and 0.9 momentum. similar warm-up procedure is set in the first 500 iterations increase the learning rate from 0 to 0.05. Then we decay the learning rate by 0.1 at the 8th and 1 Itth epoch. The whole architecture search process takes 14 epochs, 18 hours on TITAN-Xp GPUs with the batch size of 8 and the whole parameter fine-tuning takes 12 epochs, 10 hours on 8 TITAN-Xp GPUs with 32 batch size.
SSDLite We resize the input images to 320 x 320. For operation weights in the search process, we use the standard RMSProp optimizer with 4 x 10 weight decay. The warm-up stage in the first 500 iterations increases learning rate from 0 to 0.03. Then decay the learning rate by 0.1 at 16 and 22 epochs. The architecture optimization starts at 12 epochs. We set À as 0.2 and T as 10 for the loss function. The other search settings the same the RetinaNet experiment.
12

---

Published as a conference paper at ICLR 2020
For parameter adaptation, the initial learning rate is 0.2 and decays at 36, 50 and 56 epochs. The training settings follow the SSD (Liu et al., 2016) implementation in MMDetection (Chen et al., 2019a). The search process takes 24 epochs in total, 20 hours 8 TITAN-Xp GPUs with 64 batch size. The parameter adaptation takes 60 epochs, 46 hours on 8 TITAN-Xp GPUs with 512 hatch size.
16
96
160
Con 943
MB K3 85
MB K3 E5
MB K7 66
MB 17 .6
ES
MB 10 ES
MB F6
MB K7 56
MB K2 5G
MB K2 15
M0 86
MB K7 03
MB K3
MB K7
MB RS +
MB
MB
MB
(a) FNA on DeepT.abv3
160
320
32
16
24
32
64
96
MB K3 E1
MB K3 E6
MB F6
MB KS_K5 F6
MB 13 E6
MB K5. E6
MB K3 66
MB E6
MB K7 E6
MB 13
MB K7
MB K7
MB K7
MB K7
MB
(b) FNA on RetinaNet
32
16
24
64
96
160
320
Conv 3x3
MB E1
MB KS Es
MB KS Es
MB KS E6
MB K5 E6
MB K7 E6
MB K7 F6
MB K3
MB K5
MB K3
MB K7
MB K7
MB K7
MB K7
MB MBK5
MBK7
MB K7
(c) FNA on SSDLite
Figure 4: Visualization of our searched architectures different frameworks. MB: inverted residual
block proposed in MobilenetV2 (Sandler et al., 2018). Kx_Ey: the kernel size of the depthwise convolution and the expansion ratio y.
(a) Input
(b) GT
(c) Ours
Figure 5: Visualization of semantic segmentation results on the Cityscapes validation dataset.
13

---

Published as a conference paper at ICLR 2020
stop sign
STOP SORREMUC
blard
Figure 6: Visualization of object detection results the MS-COCO validation dataset.
14


Reviews:
[{"review_id": "r1x08lBvYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a new technique to adapt a source neural network performed well on classification task to image segmentation and objective detection tasks via the author called parameter-remapping trick. The parameter remapping uses weights from the source neural network to the two-stages: architecture adaption phase and parameter adaption phase. The technique results in improvements in both performance and training time.\n\nI like the direction this paper takes, NAS is too expensive and we need faster methods through meta learning/transfer learning. The paper is also clearly organized and written.  To the best of my knowledge, the experiments setting is sensible and the results are good. But I am not in the Computer Vision field and I am not so familiar with NAS, I may missed something. Thus, I am less confident about my rating.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "SkekZOLsKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a method called FNA (fast network adaptation), which takes a pretrained image classification network, and produces a network for the task of object detection/semantic segmentation. The process consists of three phases: Network Expansion, Architecture Adaptation and Parameters Adaptation, and uses the developed parameter remapping scheme twice. Experiments show that it outperforms recent other NAS methods for these two tasks with same or less computation.\n\nConcrete comments\n1. The paper's overall method is a novel one, unifying NAS on det/seg tasks, while prior works mostly only focus on one task. It also \"eliminates\" the need for pretraining each instance of the subnetwork. But no one ever pretrain every classification network for searching on det/seg tasks right? It's an insane amount of computation after all. I'm afraid the emphasis of advantage over prior method here is not very accurate.\n\n2. The concrete parameter remapping scheme is not entirely novel. It is similar to the Net2Net method, while seems more naive than that. It does not preserve the mapping function like Net2Net. It seems like a very coarse effort, since mostly what you do is to copy weights, remove weights or fill in zeros. But it is also interesting to see that this naive method works, and actually beat some of the more advanced alternatives in Section 4. \n\n3. The results are quite impressive. On segmentation, the adapted model achieves ~1% mIOU improvement using  similar or less iterations and similar size of model with the methods it compared to, and GPU hours' saving is more significant. If the authors faithfully compared with state-of-the-art methods in search det/seg architectures, but I'm not super familiar with this literature. On object detection the method does not improve the model size or accuracy, but reduces the search time a lot compared with DetNAS. Could the authors clarify that you compared with every recent high-performance NAS method on seg/det tasks?\n\n4. Though the improvement over prior methods is good, the experiments lack an apple-to-apple comparison. For example, using exactly the same NAS search method and supernet, and comparing the FNA method with that not using a pretrained model (i.e., directly search on det/seg) could be a good experiment to showcase the importance of adaptation.\n\nOverall I find the method is effective and experiments convincing and I recommend weak accept in my rating. I hope authors can address my concerns in the rebuttal. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1xI2WlfYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors take a MobileNet v2 trained for ImageNet classification, and adapt it either (i) semantic segmentation on Cityscapes, or (ii) object detection on COCO. They do this by first expanding the network into a \"supernet\" and copy weights  in an ad-hoc manner, then, they perform DARTS-style architecture search before fine-tuning for the task at hand.\n\nThere is no TLDR for this paper, and I must admit, on reading the abstract and introduction I wasn't entirely sure what this paper was doing at first. Perhaps I was being slow.\n\nFrom a narrative perspective, one of the main selling points is not needing to perform any expensive ImageNet pre-training; however, a pre-trained MobileNetv2 is being utilised. While this was off-the-shelf, it still incurred an initial training cost, so it isn't really fair in e.g. Table 4 to put pre-training cost as zero. On a related note, the authors write that this network is used for its \"generality\". I'd argue that MobileNetv2 is a highly engineered network specialised for mobile computation; a standard ResNet-50 would be more general really.\n\nI would like to see a comparison to a random search, as there are several papers (https://arxiv.org/abs/1902.07638, https://arxiv.org/abs/1902.08142) indicating that this is a very strong baseline. \n\nAs mentioned earlier, the choices for remapping weights seem very ad-hoc. I can't really tell what's going on in Table 5 (why is PR in the NE and PA row?) so the ablation study of how effective this weight mapping is lost on me.  The stuff in Table 6 is pretty interesting however, if convoluted.\n\nI find the odd choices of hyperparameters (tau as 45, gamma as 10, lambda as 9e-3) rather alarming. How important are these? Would this technique work under any other circumstances?\n\nError bars would be a welcome inclusion, particularly in Table 3 where you have 0.1% separating FNA and MNasnet-92. I appreciate that this can be expensive however.\n\nPros:\n- Some promising results\n- Good figures\n\nCons:\n- Ad-hoc design choices\n- Not a fair comparison regarding pre-training. \n- Very specific to one network choice\n- Lack of error bars or comparison to random search.\n\nI am giving this paper a weak reject, as there is insufficient experimental evidence that the technique works, or generalises beyond Mobilenetv2. I am also concerned about the ad-hoc hyperparmaters or weight-mapping. A comprehensive ablation study, along with error bars, and another choice of seed network would do much to strengthen this paper.\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content: Paper proposes a fast network adaptation (FNA) method, which takes a pre-trained image classification network, and produces a network for the task of object detection/semantic segmentation

Summary of discussion:
reviewer1: interesting paper with good results, specifically without the need to do pre-training on Imagenet. Cons are better comparisons to existing methods and run on more datasets. 
reviewer2:  interesting idea on adapting source network network via parameter re-mapping that offers good results in both performance and training time.
reviewer3: novel method overall, though some concerns on the concrete parameter remapping scheme. Results are impressive
Recommendation: Interesting idea and good results. Paper could be improved with better comparison to existing techniques. Overall recommend weak accept.
Decision: Accept (Poster)