Fast Neural Network Adaptation via Parameter Remapping and Architecture Search

1 INTRODUCTION . Deep convolutional neural networks have achieved great successes in computer vision tasks such as image classification ( Krizhevsky et al. , 2012 ; He et al. , 2016 ; Howard et al. , 2017 ) , semantic segmentation ( Long et al. , 2015 ; Ronneberger et al. , 2015 ; Chen et al. , 2017b ) and object detection ( Ren et al. , 2015 ; Liu et al. , 2016 ; Lin et al. , 2017 ) etc . Image classification has always served as a fundamental task for neural architecture design . It is common to use networks designed and pre-trained on the classification task as the backbone and fine-tune them for segmentation or detection tasks . However , the backbone plays an important role in the performance on these tasks and the difference between these tasks calls for different design principles of the backbones . For example , segmentation tasks require high-resolution features and object detection tasks need to make both localization and classification predictions from each convolutional feature . Such distinctions make neural architectures designed for classification tasks fall short . Some attempts ( Li et al. , 2018 ; Wang et al. , 2019 ) have been made to tackle this problem . Handcrafted neural architecture design is inefficient , requires a lot of human expertise , and may not find the best-performing networks . Recently , neural architecture search ( NAS ) methods ( Zoph et al. , 2017 ; Pham et al. , 2018 ; Liu et al. , 2018 ) see a rise in popularity . Some works ( Liu et al. , ∗Equal contributions . †The work is performed during an internship at Horizon Robotics . ‡Corresponding author . 2019a ; Zhang et al. , 2019 ; Chen et al. , 2019b ) propose to use NAS to design backbone architectures specifically for segmentation or detection tasks . Nevertheless , pre-training remains a inevitable but costly procedure . Though some ( He et al. , 2018 ) recently demonstrates that pre-training is not always necessary for accuracy considerations , training from scratch on the target task still takes more iterations than fine-tuning from a pre-trained model . For NAS methods , the pre-training cost is non-negligible for evaluating the networks in the search space . One-shot search methods ( Brock et al. , 2017 ; Bender et al. , 2018 ; Chen et al. , 2019b ) integrate all possible architectures in one super network but pre-training the super network and the searched network still bears huge computation cost . As ImageNet ( Deng et al. , 2009 ) pre-training has been a standard practice for many computer vision tasks , there are lots of models trained on ImageNet available in the community . To take full advantages of these models , we propose a Fast Neural Network Adaptation ( FNA ) method based on a novel parameter remapping paradigm . Our method can adapt both the architecture and parameters of one network to a new task with negligible cost . Fig . 1 shows the whole framework . The adaptation is performed on both the architecture- and parameter-level . We adopt the NAS methods ( Zoph et al. , 2017 ; Real et al. , 2018 ; Liu et al. , 2019b ) to implement the architecture-level adaptation . We select a manually designed network ( MobileNetV2 ( Sandler et al. , 2018 ) in our experiments ) as the seed network , which is pre-trained on ImageNet . Then , we expand the seed network to a super network which is the representation of the search space in FNA . New parameters in the super network are initialized by mapping those from the seed network using parameter remapping . Thanks to that , the neural architecture search can be performed efficiently on the detection and segmentation tasks . With FNA we obtain a new optimal target architecture for the new task . Similarly , we remap the parameters of the seed network to the target architecture for initialization and fine-tune it on the target task with no need of pre-training on a large-scale dataset . We demonstrate FNA ’ s effectiveness and efficiency via experiments on both segmentation and detection tasks . We adapt the manually designed network MobileNetV2 ( Sandler et al. , 2018 ) to segmentation framework DeepLabv3 ( Chen et al. , 2017b ) , detection framework RetinaNet ( Lin et al. , 2017 ) and SSDLite ( Liu et al. , 2016 ; Sandler et al. , 2018 ) . Networks adapted by FNA surpass both manually designed and NAS searched networks in terms of both performance and model MAdds . Compared to NAS methods , FNA costs 1737× less than DPC ( Chen et al. , 2018a ) , 6.8× less than Auto-DeepLab ( Liu et al. , 2019a ) and 7.4× less than DetNAS ( Chen et al. , 2019b ) . 2 RELATED WORK . Neural Architecture Search With reinforcement learning ( RL ) and evolutionary algorithm ( EA ) being applied to NAS methods , many works ( Zoph & Le , 2016 ; Zoph et al. , 2017 ; Real et al. , 2018 ) make great progress in promoting the performances of neural networks . Recently , NAS methods based on one-shot model ( Brock et al. , 2017 ; Bender et al. , 2018 ) or differentiable representations ( Liu et al. , 2019b ; Cai et al. , 2019 ; Fang et al. , 2019b ) achieve remarkable results with low search cost . We use the differentiable NAS method to implement architecture adaptation , which adjusts the architecture of the backbone network automatically . Backbone Design As deep neural network designing ( Simonyan & Zisserman , 2014 ; Szegedy et al. , 2016 ; He et al. , 2016 ) develops , the backbones of segmentation or detection networks evolve accordingly . Some works improve the backbone architectures by modifying existing networks . PeleeNet ( Wang et al. , 2018 ) proposes a variant of DenseNet ( Huang et al. , 2017 ) for more realtime object detection on mobile devices . DetNet ( Li et al. , 2018 ) applies dilated convolution ( Yu & Koltun , 2016 ) in the backbone to enlarge the receptive field which helps to detect objects . BiSeNet ( Yu et al. , 2018 ) and HRNet ( Wang et al. , 2019 ) design multiple paths to learn both highand low- resolution representations for better dense prediction . Parameter Remapping Net2Net ( Chen et al. , 2015 ) proposes the function-preserving transformations to remap the parameters of one network to a new deeper or wider network . This remapping mechanism accelerates the training of the new larger network and achieves great performances . Following this manner , EAS ( Cai et al. , 2018 ) extends the parameter remapping concept to neural architecture search . Moreover , some NAS works ( Pham et al. , 2018 ; Fang et al. , 2019a ; Elsken et al. , 2019 ) apply parameters sharing on child models to accelerate the search process . Our parameter remapping paradigm extends the mapping dimension with the kernel level . Parameters can be also mapped to a shallower or narrower network with our scheme , while Net2Net focuses on mapping parameters to a deeper and wider network . The parameter remapping in our FNA largely decreases the computation cost of the network adaptation by taking full advantages of the ImageNet pre-trained parameters . 3 METHOD . As the most commonly used network for designing search spaces in NAS methods ( Tan et al. , 2018 ; Cai et al. , 2019 ; Fang et al. , 2019b ) , MobileNetV2 ( Sandler et al. , 2018 ) is selected as the seed network to give the details of our method . To adapt the network to segmentation and detection tasks , we adjust the architecture elements on three levels , i.e. , convolution kernel size , depth and width of the network . In this section , we first describe the parameter remapping paradigm . Then we explain the whole procedure of the network adaptation . 3.1 PARAMETER REMAPPING . We define parameter remapping as one paradigm which aims at mapping the parameters of one seed network to another one . We denote the seed network as Ns and the new network as Nn , whose parameters are denoted as Ws and Wn respectively . The remapping paradigm is illustrated in the following three aspects . The remapping on the depth-level is firstly carried out and then the remapping on the kernel- and width- level is conducted simultaneously . Moreover , we study different remapping strategies in the experiments ( Sec . 4.5 ) . Remapping on Depth-level We introduce more depth settings in our architecture adaptation process . In other word , we adjust the number of inverted residual blocks ( MBConvs ) ( Sandler et al. , 2018 ) in every stage of the network . We assume that one stage in the seed network Ns has l layers . The parameters of each layer can be denoted as { W ( 1 ) s , W ( 2 ) s , . . . , W ( l ) s } . Similarly , we assume that the corresponding stage with m layers in the new network Nn has parameters { W ( 1 ) n , W ( 2 ) n , . . . , W ( m ) n } . The remapping process on the depth-level is shown in Fig . 2 ( a ) . The parameters of layers in Nn which also exit in Ns are just copied from Ns . The parameters of new layers are all copied from the last layer in the stage of Ns . It is formulated as f ( i ) = min ( i , l ) , W ( i ) n =W ( f ( i ) ) s , ∀1 ≤ i ≤ m. ( 1 ) Remapping on Width-level In the MBConv block of MobileNetV2 ( Sandler et al. , 2018 ) network , the first point-wise convolution expands the low-dimensional features to a high dimension . { ... { ... ( a ) Depth level { ... { ... ( b ) Width level { ... { ... ( c ) Kernel level Figure 2 : Our parameters are remapped on three levels . ( a ) shows the depth-level remapping . The parameters of existing corresponding layers are mapped from the original network . The parameters of new layers are mapped from the last layer in the original network . ( b ) shows the width-level remapping . For each channels-diminished dimension , the parameters are copied from the original existing ones . ( c ) shows the kernel-level remapping . The original parameters are mapped to the central part of the new larger kernel . The values of the other parameters are assigned with 0 . This practice can be utilized for expanding the width and capacity of one neural network . We allow smaller expansion ratios for architecture adaptation . We denote the parameters of one convolution in Ns as W s ∈ Rp×q×h×w and that in Nn as W n ∈ Rr×s×h×w , where r ≤ p and s ≤ q . As shown in Fig . 2 ( b ) , on the width-level , we directly map the first r or s channels of parameters inNs to the narrower one in Nn . It can be formulated as W ni , j , : , : = W s i , j , : , : , ∀1 ≤ i ≤ r , 1 ≤ j ≤ s. ( 2 ) Remapping on Kernel-level The kernel size is commonly set as 3×3 in most artificially-designed networks . To expand the receptive field and capture abundant features in segmentation or detection tasks , we introduce larger kernel size settings in the adaptation process . As Fig . 2 ( c ) shows , to expand the 3 × 3 kernel to a larger one , we assign the parameters of the central 3 × 3 region in the large kernel with the values of the original 3× 3 kernel . The values of the other region surrounding the central part are assigned with 0 . We denote the parameters of the original 3× 3 kernel as W 3×3 and the larger k × k kernel as W k×k . The remapping process on kernel-level can be formulated as follows , W k×k : , : ,h , w = { W 3×3 : , : ,h , w if ( k − 3 ) /2 < h , w ≤ ( k + 3 ) /2 0 otherwise , ( 3 ) where h , w denote the indices of the spatial dimension . This design principle conforms to the function-preserving concept ( Chen et al. , 2015 ) , which accelerates and stabilizes the optimization of the new network .

Summary: This paper provides a new technique to adapt a source neural network performed well on classification task to image segmentation and objective detection tasks via the author called parameter-remapping trick. The parameter remapping uses weights from the source neural network to the two-stages: architecture adaption phase and parameter adaption phase. The technique results in improvements in both performance and training time.