Title: Fast Neural Network Adaptation via Parameter Remapping and Architecture Search
Authors: Jiemin Fang*
Yuzhu Sun*
Kangjian Peng*
Qian Zhang
Yuan Li
Wenyu Liu
Xinggang Wang
Venue: ICLR 2020

Abstract: Deep neural networks achieve remarkable performance in many computer vision tasks. Most state-of-the-art~(SOTA) semantic segmentation and object detection approaches reuse neural network architectures designed for image classification as the backbone, commonly pre-trained on ImageNet. However, performance gains can be achieved by designing network architectures specifically for detection and segmentation, as shown by recent neural architecture search (NAS) research for detection and segmentation. One major challenge though, is that ImageNet pre-training of the search space representation (a.k.a. super network) or the searched networks incurs huge computational cost. In this paper, we propose a Fast Neural Network Adaptation (FNA) method, which can adapt both the architecture and parameters of a seed network (e.g. a high performing manually designed backbone) to become a network with different depth, width, or kernels via a Parameter Remapping technique, making it possible to utilize NAS for detection/segmentation tasks a lot more efficiently. In our experiments, we conduct FNA on MobileNetV2 to obtain new networks for both segmentation and detection that clearly out-perform existing networks designed both manually and by NAS. The total computation cost of FNA is significantly less than SOTA segmentation/detection NAS approaches: 1737$\times$ less than DPC, 6.8$\times$ less than Auto-DeepLab and 7.4$\times$ less than DetNAS. The code is available at https://github.com/JaminFong/FNA .


Review discussion:
In this paper, the authors take a MobileNet v2 trained for ImageNet classification, and adapt it either (i) semantic segmentation on Cityscapes, or (ii) object detection on COCO. They do this by first expanding the network into a "supernet" and copy weights  in an ad-hoc manner, then, they perform DARTS-style architecture search before fine-tuning for the task at hand.
There is no TLDR for this paper, and I must admit, on reading the abstract and introduction I wasn't entirely sure what this paper was doing at first. Perhaps I was being slow.
From a narrative perspective, one of the main selling points is not needing to perform any expensive ImageNet pre-training;
however, a pre-trained MobileNetv2 is being utilised.
While this was off-the-shelf, it still incurred an initial training cost, so it isn't really fair in e.g. Table 4 to put pre-training cost as zero. On a related note, the authors write that this network is used for its "generality". I'd argue that MobileNetv2 is a highly engineered network specialised for mobile computation; a standard ResNet-50 would be more general really.
I would like to see a comparison to a random search, as there are several papers (https://arxiv.org/abs/1902.07638, https://arxiv.org/abs/1902.08142) indicating that this is a very strong baseline.
As mentioned earlier, the choices for remapping weights seem very ad-hoc. I can't really tell what's going on in Table 5 (why is PR in the NE and PA row?) so the ablation study of how effective this weight mapping is lost on me.  The stuff in Table 6 is pretty interesting however, if convoluted.
I find the odd choices of hyperparameters (tau as 45, gamma as 10, lambda as 9e-3) rather alarming. How important are these? Would this technique work under any other circumstances?
Error bars would be a welcome inclusion, particularly in Table 3 where you have 0.1% separating FNA and MNasnet-92. I appreciate that this can be expensive however.
Pros:
- Some promising results
- Good figures
Cons:
- Ad-hoc design choices
- Not a fair comparison regarding pre-training.
- Very specific to one network choice
- Lack of error bars or comparison to random search.
I am giving this paper a weak reject, as there is insufficient experimental evidence that the technique works, or generalises beyond Mobilenetv2. I am also concerned about the ad-hoc hyperparmaters or weight-mapping. A comprehensive ablation study, along with error bars, and another choice of seed network would do much to strengthen this paper.
We sincerely thank you for your detailed review and constructive suggestions.
Our proposed FNA aims at adapting the pre-trained neural network to new tasks efficiently with a novel parameter remapping mechanism. Inspired by the influential work Net2Net[1] which proposes an effective method for mapping the parameters of one network to a deeper and wider one, we propose a novel and effective parameter remapping scheme. With the remapping scheme, the parameters of one seed network can be remapped to the super network or the target network. The mapping dimension is further extended to the kernel level. And the proposed parameter remapping method can map parameters to shallower and narrower networks while Net2Net does not cover these perspectives. It is not an ad-hoc design choice as we explore various reasonable strategies in the experiments. The results show that our method achieves superior performances and our mechanism is more convenient to implement compared to others.
Our proposed FNA is not only a NAS method, while it is a general framework to adapt the network to various tasks. FNA demonstrates the effectiveness of both segmentation and detection tasks in the experiments and beats state-of-the-art NAS-based detection and segmentation methods, while prior works mostly focus on only one task as Reviewer1 also mentions this. FNA is meaningful for researchers to carry out network optimization on various new tasks that bear the unaffordable pre-training cost as there are lots of available pre-trained models in the community. We hope our statement can clarify the core value of our work.
>>> Response to "no TLDR for this paper":
We are sorry that our abstract and introduction do not illustrate our main idea or contribution clearly. We will revise the paper detailedly in the next version. We supplement the TLDR as follows.
We propose a fast neural network adaptation (FNA) method to adapt a seed network with pre-trained weights to other new tasks. A parameter remapping mechanism is designed to accelerate the whole adaptation process which takes full advantage of the knowledge from the seed network.
>>> Response to "it isn't really fair in Table 4 to put pre-training cost as zero":
Firstly, our proposed method aims at the available pre-trained models in the community as there are many of them.
Secondly, the seed network in FNA has strong reusability.
For example, if the search space changes due to the need for the task, the super network in other methods, e.g. DetNAS, needs to be pre-trained again.
But the super network of FNA does not need to be pre-trained. All the pre-trained weights of the super network are remapped from the same seed network. It is totally a once-for-all manner.
Even though we take the pre-trained cost of the seed network into consideration, FNA still holds a huge advantage with the perspective of both performance and computation cost. We show the comparison as follows.
| Method     | Total Cost |            Super Network      |                Target Network                    |
|_____________|____________| Pre-training |Finetuning | Search | Pre-training | Finetuning |
| DetNAS[2] | 68 GDs     | 12 GDs          |12 GDs       | 20 GDs | 12 GDs          | 12 GDs        |
| FNA            | 15.9 GDs  | 6.7 GDs         | -                  | 6 GDs   | -                     | 3.2 GDs       |
>>> Response to "the generality of the seed network":
Sorry for this unclearness of the description of the seed network choice. We choose MobileNetV2 because it is widely used for the search space design in many NAS methods [3, 4, 5, 6].
But we would like to try our best to implement the FNA method on the ResNet model before the rebuttal deadline.
>>> Response to "a comparison to a random search":
We will provide the experiment result of the random search. Thanks for your constructive advice.
Besides, our method aims to adapt the off-the-shelf network to other new tasks. NAS is only an implementation tool for architecture adaptation. Which NAS method we use is not the focus of our method indeed.
>>> Response to "what's going on in Table 5":
We feel sorry for the vagueness of Tab. 5 in the paper. We conduct ablation studies to demonstrate the effectiveness of parameter remapping and show the comparison results in Tab. 5. We redraw the table in the following.
Row Num  Method                                                                                      MAdds(B)  mIOU(%)
(1)               Remap -> ArchAdapt -> Remap -> ParamAdapt (FNA)      24.17          76.6
(2)               RandInit -> ArchAdapt -> Remap -> ParamAdapt              24.29          76.0
(3)               Remap -> ArchAdapt -> RandInit -> ParamAdapt              24.17          73.0
(4)               RandInit -> ArchAdapt -> RandInit -> ParamAdapt           24.29           72.4
(5)               Remap -> ArchAdapt -> Retrain -> ParamAdapt                24.17           76.5
Remap: Parameter Remapping. ArchAdapt: Architecture Adaptation. RandInit: Random Initialization. Pretrain: ImageNet Pretrain. ParamAdapt: Parameter Adaptation.
We attempt to optionally remove the parameter remapping process before the two stages, i.e. architecture adaptation and parameter adaptation. In Row (2) we remove the parameter remapping process before architecture adaptation.
In other words, the search is performed from scratch without using the pre-trained network. The mIOU in Row (2) drops by 0.6% compared to FNA in Row (1).
Then we remove the parameter remapping before parameter adaptation in Row (3), i.e. training the target architecture from scratch on the target task. The mIOU decreases by 3.6% compared to the result of FNA.
When we remove the parameter remapping before both stages in Row (4), it gets the worst performance. In Row (5), we first pre-train the searched architecture on ImageNet and then fine-tune it on the target task.
It is worth noting that FNA even achieves a higher mIOU by a narrow margin (0.1%) than the ImageNet pre-trained one in Row (5). We conjecture that this may benefit from the regularization effect of parameter remapping before the parameter adaptation stage.
All the experiments are conducted using the same searching and training settings for fair comparisons. With parameters remapping applied on both stages, the adaptation achieves the best results. Especially, the remapping process before parameter adaptation tends to provide greater performance gains than the remapping before architecture adaptation. All the experimental results demonstrate the importance and effectiveness of the proposed parameter remapping scheme.
We revise this part in the new version of our paper. We hope this revision and explanation can solve your puzzle.
>>> Response to "The stuff in Table 6 ...":
We explore more strategies of parameter remapping in Tab. 6 (Tab. 7 in our new version), as it is an important topic to find more efficient parameter remapping methods. More specifically, we use the importance of the channel to remap the parameters on the width level. The importance evaluation metric is set with the statistics of BN or the standard deviation/L1 norm of the parameters on the channel level. For the kernel-level parameter mapping, we further conduct a dilation manner. The experiments show that the parameter remapping method in FNA achieves the best results.
Moreover, our proposed parameter remapping is the most convenient to implement.
We carry out experiments with the random search (RandSearch) strategy. All the results are shown in the following table. As Row (2) shows, we simplify the FNA process as "Remap -> Differentiable Search (DiffSearch) -> Remap -> ParamsAdapt" for the clear illustration. We purely replace the original differentiable NAS method in FNA with the random search method in Row (3). And FNA with RandSearch achieves comparable results with our original method. It further confirms that FNA is a general framework for network adaptation and has great generality. NAS is only an implementation tool for architecture adaptation. The whole framework of FNA can be treated as a NAS-method agnostic mechanism.
It is worth noting that even using random search, our FNA still outperforms DetNAS with 0.2% mAP better and 150M Flops fewer.
We conduct more ablation studies to demonstrate the effectiveness of the parameter remapping scheme. In Row (4) we remove the parameter remapping process before a random search, the mAP drops by 2.0% compared to Row (3).
Then we remove the parameter remapping before parameter adaptation in Row (5), the mAP decreases by 8.2% compared to Row (3).
When we remove the parameter remapping before both processes in Row (6), it gets the worst performance. All the experiments demonstrate the importance and effectiveness of the parameter remapping scheme. The results will be updated in the revised version of our paper. Thanks a lot for your suggestion of performing this random search ablation study.
Row Num  Method                                                                                           MAdds(B)  map(%)
(1)               DetNAS[1]                                                                                       133.26        33.3
(2)               FNA (Remap -> DiffSearch -> Remap -> ParamsAdapt)  	     133.03         33.9
(3)               FNA (Remap -> RandSearch -> Remap -> ParamsAdapt)       133.11        33.5
(4)               RandInit -> RandSearch -> Remap    -> ParamsAdapt            133.08         31.5
(5)               Remap    -> RandSearch -> RandInit -> ParamsAdapt            133.11         25.3
(6)               RandInit -> RandSearch -> RandInit -> ParamsAdapt            133.08         24.9
[1] Chen Y, Yang T, Zhang X, et al. Detnas: Neural architecture search on object detection[J]. NeurIPS, 2019.
>>> Response to "the odd choices of hyper-parameters":
We revise the loss function as follows, as the $\gamma$ parameter in the original paper is of no use,
$\mathcal{L} = \mathcal{L}_{task} + \lambda \log_{\tau}(cost)$.
It is common to adjust the hyper-parameters of multi-objective optimization in many NAS method [3, 4, 5, 6]. These hyper-parameter settings are often omitted in the main text of most NAS papers. The adjustment of these parameters is to get a similar model size with that of other methods for a fair comparison.
>>> Response to "Error bars ... 0.1% separating FNA and MnasNet-92":
We repeatedly train the network obtained by FNA for three times and it obtains three same results, 23.0% mAP.
It is worth noting that MnasNet [3] takes a huge cost (around 3,800 GPU days) to search the architecture on the ImageNet classification task and achieves SOTA results on the SSDLite framework [3, 7]. The computation cost of FNA is apparently smaller, 176x less than MnasNet. The MAdds of FNA is 100M less than MnasNet-92, while FNA achieves 0.1% higher mAP.
We sincerely thank you for your comprehensive comments and hopefully have cleared your concerns.
[1] Chen T, Goodfellow I, Shlens J. Net2net: Accelerating learning via knowledge transfer[J]. arXiv preprint arXiv:1511.05641, 2015.
[1] Chen Y, Yang T, Zhang X, et al. Detnas: Neural architecture search on object detection[J]. NeurIPS, 2019.
[3] Tan M, Chen B, Pang R, et al. Mnasnet: Platform-aware neural architecture search for mobile[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 2820-2828.
[4] Wu B, Dai X, Zhang P, et al. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 10734-10742.
[5] Cai H, Zhu L, Han S. Proxylessnas: Direct neural architecture search on target task and hardware[J]. arXiv preprint arXiv:1812.00332, 2018.
[6] Chu X, Zhang B, Xu R, et al. Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search[J]. arXiv preprint arXiv:1907.01845, 2019.
[7] Sandler M, Howard A, Zhu M, et al. Mobilenetv2: Inverted residuals and linear bottlenecks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 4510-4520.
Thanks for your suggestions for conducting experiments by choosing ResNet-50 as the seed network. We design the search space of architecture adaptation as follows. We utilize the bottleneck block defined in ResNet-50 [1] to construct our search space. One bottleneck consists of two 1x1 convolutions, one inner 3x3 convolution, and the skip connection.  We search for the depth, width and the kernel sizes of ResNet50. More specifically, we add more layers in the super network for depth search. The kernel size settings include {3x3, 5x5}. We allow several width ratios {1/2, 3/4, 1} of the inner kxk convolutions to search the widths.
It is worth mention that the architecture search with ResNet-based search spaces on segmentation or detection tasks is challenging work. Enlarging kernel sizes in the inner convolutions causes huge computation cost increasing. We allow more width settings to balance the computation cost of the models. Due to the limited time, we do not tune the hyper-parameters for ResNet.
Therefore there is still room for improvement of performance. We conduct experiments on the RetinaNet [2] framework and show our results as follows. FNA promotes the mAP by 0.3% with 20M fewer MAdds compared width ResNet-50. We would like to try more ResNet experiments in the future.
Thanks for your advice again. We hope our answers can clear your concerns.
Method        MAdds(B)  mAP(%)
ResNet-50   202.85         33.3
FNA              202.83         33.6
[1] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.
[2] Lin T Y, Goyal P, Girshick R, et al. Focal loss for dense object detection[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2980-2988.
This paper provides a new technique to adapt a source neural network performed well on classification task to image segmentation and objective detection tasks via the author called parameter-remapping trick. The parameter remapping uses weights from the source neural network to the two-stages: architecture adaption phase and parameter adaption phase. The technique results in improvements in both performance and training time.
I like the direction this paper takes, NAS is too expensive and we need faster methods through meta learning/transfer learning. The paper is also clearly organized and written.  To the best of my knowledge, the experiments setting is sensible and the results are good.
But I am not in the Computer Vision field and I am not so familiar with NAS, I may missed something.
Thus, I am less confident about my rating.
We sincerely thank you for your review and assessment of our work.
Our proposed FNA method can adapt a network with pre-trained weights to other new tasks efficiently. The total computation cost of FNA is far smaller compared to other SOTA methods. We conduct sufficient experiments to demonstrate the effectiveness of our method and FNA achieves superior performances on various tasks.
Moreover, it is convenient to apply FNA on more other tasks, e.g., pose estimation, face detection, NLP, speech recognition, etc. As there are lots of pre-trained models available in the community, FNA can help researchers who cannot afford expensive computation cost explore the model optimization on all kinds of tasks. FNA even makes NAS more accessible for more researchers.
The paper proposes a method called FNA (fast network adaptation), which takes a pretrained image classification network, and produces a network for the task of object detection/semantic segmentation. The process consists of three phases: Network Expansion, Architecture Adaptation and Parameters Adaptation, and uses the developed parameter remapping scheme twice. Experiments show that it outperforms recent other NAS methods for these two tasks with same or less computation.
Concrete comments
1. The paper's overall method is a novel one, unifying NAS on det/seg tasks, while prior works mostly only focus on one task. It also "eliminates" the need for pretraining each instance of the subnetwork.
But no one ever pretrain every classification network for searching on det/seg tasks right? It's an insane amount of computation after all. I'm afraid the emphasis of advantage over prior method here is not very accurate.
2. The concrete parameter remapping scheme is not entirely novel. It is similar to the Net2Net method, while seems more naive than that. It does not preserve the mapping function like Net2Net. It seems like a very coarse effort, since mostly what you do is to copy weights, remove weights or fill in zeros.
But it is also interesting to see that this naive method works, and actually beat some of the more advanced alternatives in Section 4.
3. The results are quite impressive. On segmentation, the adapted model achieves ~1% mIOU improvement using  similar or less iterations and similar size of model with the methods it compared to, and GPU hours' saving is more significant. If the authors faithfully compared with state-of-the-art methods in search det/seg architectures, but I'm not super familiar with this literature. On object detection the method does not improve the model size or accuracy, but reduces the search time a lot compared with DetNAS. Could the authors clarify that you compared with every recent high-performance NAS method on seg/det tasks?
4.
Though the improvement over prior methods is good, the experiments lack an apple-to-apple comparison.
For example, using exactly the same NAS search method and supernet, and comparing the FNA method with that not using a pretrained model (i.e., directly search on det/seg) could be a good experiment to showcase the importance of adaptation.
Overall I find the method is effective and experiments convincing and I recommend weak accept in my rating. I hope authors can address my concerns in the rebuttal.
We sincerely thank you for your detailed and constructive comments.
1. We would like to revise the description part of the advantages over prior methods. Our main point aims at taking full advantage of the pre-trained weights of the seed network, which is essential for fast neural architecture search and parameter adaptation. Our parameter remapping mechanism accelerates the whole procedures greatly which makes it easy to conduct the network adaptation on different tasks.
2. Our parameter remapping method does have some similar effects with Net2Net.
However, they are quite different. Net2Net aims at deepening and widening the network and accelerates the training procedure. In Net2Net, parameters can only be mapped on the depth and width level. We extend the mapping dimension with the kernel level. Parameters can be also mapped to a shallower or narrower network with our remapping scheme, while Net2Net only maps parameters to a deeper and wider network. We deploy FNA in the popular efficient model MobileNetV2. Compared with those seemingly more advanced methods, the parameter remapping mechanism of FNA is easier to implement and achieves the best results.
Moreover, exploring more effective parameter remapping methods is actually a valuable topic, as we do in Sec. 4.4.
3. DetNAS[1] is the latest detection backbone search work that achieves SOTA results and has been accepted in NeurIPS 2019. Our comparison is sufficient and fair.
Furthermore, in RetinaNet, FNA achieves 0.6 accuracy promotion and MAdds is 0.23B smaller compared with DetNAS. In SSDLite, FNA achieves 0.1 accuracy promotion with 100M fewer MAdds compared with MnasNet [2]. MnasNet takes a huge cost (around 3,800 GPU days) to search the architecture on the ImageNet classification task. MnasNet also achieves SOTA results on the SSDLite framework. The computation cost of FNA is apparently smaller, 176x less than MnasNet.
4. We are sorry for the vagueness of Tab. 5 in the paper. Actually, we conduct sufficient apple-to-apple comparison experiments in Sec. 4.3 to showcase the effectiveness of parameter remapping in the network adaptation. We revise Tab. 5 as follows for clearer illustration.
Row Num  Method                                                                                      MAdds(G)  mIOU(%)
(1)               Remap -> ArchAdapt -> Remap -> ParamAdapt (FNA)      24.17          76.6
(2)               RandInit -> ArchAdapt -> Remap -> ParamAdapt              24.29          76.0
(3)               Remap -> ArchAdapt -> RandInit -> ParamAdapt              24.17          73.0
(4)               RandInit -> ArchAdapt -> RandInit -> ParamAdapt           24.29           72.4
(5)               Remap -> ArchAdapt -> Retrain -> ParamAdapt                24.17           76.5
Remap: Parameter Remapping. ArchAdapt: Architecture Adaptation. RandInit: Random Initialization. Pretrain: ImageNet Pretrain. ParamAdapt: Parameter Adaptation.
We attempt to optionally remove the parameter remapping process before the two stages, i.e. architecture adaptation and parameter adaptation. In Row (2) we remove the parameter remapping process before architecture adaptation.
In other words, the search is performed from scratch without using the pre-trained network. The mIOU in Row (2) drops by 0.6% compared to FNA in Row (1).
Then we remove the parameter remapping before parameter adaptation in Row (3), i.e. training the target architecture from scratch on the target task. The mIOU decreases by 3.6% compared to the result of FNA.
When we remove the parameter remapping before both stages in Row (4), it gets the worst performance. In Row (5), we first pre-train the searched architecture on ImageNet and then fine-tune it on the target task.
It is worth noting that FNA even achieves a higher mIOU by a narrow margin (0.1%) than the ImageNet pre-trained one in Row (5). We conjecture that this may benefit from the regularization effect of parameter remapping before the parameter adaptation stage.
All the experiments are conducted using the same searching and training settings for fair comparisons. With parameters remapping applied on both stages, the adaptation achieves the best results. Especially, the remapping process before parameter adaptation tends to provide greater performance gains than the remapping before architecture adaptation. All the experimental results demonstrate the importance and effectiveness of the proposed parameter remapping scheme. We revise this part in the new version of our paper.
We thank for your detailed review once again and hope that our response can address your concerns.
[1] Chen Y, Yang T, Zhang X, et al. Detnas: Neural architecture search on object detection[J]. NeurIPS, 2019.
[2] Tan M, Chen B, Pang R, et al. Mnasnet: Platform-aware neural architecture search for mobile[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 2820-2828.
We thank all the reviewers for their careful comments and constructive suggestions. We revise the paper following the advice and hope our new version of the paper can better illustrate our method. Promised experiments are included.
1. For a clear illustration, we change "the inner network" to "the target architecture".
2. We revise the part of experiments about parameter remapping in Sec. 4.3 and redraw the Tab. 5.
3. We add random search experiments in Sec. 4.4.
4. We move some detailed descriptions of hyper-parameters to the Appendix.
5. We correct some statements and improve the writing parts of the paper.

Meta-review:
role: PC
data: comment: Main content: Paper proposes a fast network adaptation (FNA) method, which takes a pre-trained image classification network, and produces a network for the task of object detection/semantic segmentation

Summary of discussion:
reviewer1: interesting paper with good results, specifically without the need to do pre-training on Imagenet. Cons are better comparisons to existing methods and run on more datasets. 
reviewer2:  interesting idea on adapting source network network via parameter re-mapping that offers good results in both performance and training time.
reviewer3: novel method overall, though some concerns on the concrete parameter remapping scheme. Results are impressive
Recommendation: Interesting idea and good results. Paper could be improved with better comparison to existing techniques. Overall recommend weak accept.

Decision: Accept (Poster)