Fast Neural Network Adaptation via Parameter Remapping and Architecture Search

1 INTRODUCTION . Deep convolutional neural networks have achieved great successes in computer vision tasks such as image classification ( Krizhevsky et al. , 2012 ; He et al. , 2016 ; Howard et al. , 2017 ) , semantic segmentation ( Long et al. , 2015 ; Ronneberger et al. , 2015 ; Chen et al. , 2017b ) and object detection ( Ren et al. , 2015 ; Liu et al. , 2016 ; Lin et al. , 2017 ) etc . Image classification has always served as a fundamental task for neural architecture design . It is common to use networks designed and pre-trained on the classification task as the backbone and fine-tune them for segmentation or detection tasks . However , the backbone plays an important role in the performance on these tasks and the difference between these tasks calls for different design principles of the backbones . For example , segmentation tasks require high-resolution features and object detection tasks need to make both localization and classification predictions from each convolutional feature . Such distinctions make neural architectures designed for classification tasks fall short . Some attempts ( Li et al. , 2018 ; Wang et al. , 2019 ) have been made to tackle this problem . Handcrafted neural architecture design is inefficient , requires a lot of human expertise , and may not find the best-performing networks . Recently , neural architecture search ( NAS ) methods ( Zoph et al. , 2017 ; Pham et al. , 2018 ; Liu et al. , 2018 ) see a rise in popularity . Some works ( Liu et al. , ∗Equal contributions . †The work is performed during an internship at Horizon Robotics . ‡Corresponding author . 2019a ; Zhang et al. , 2019 ; Chen et al. , 2019b ) propose to use NAS to design backbone architectures specifically for segmentation or detection tasks . Nevertheless , pre-training remains a inevitable but costly procedure . Though some ( He et al. , 2018 ) recently demonstrates that pre-training is not always necessary for accuracy considerations , training from scratch on the target task still takes more iterations than fine-tuning from a pre-trained model . For NAS methods , the pre-training cost is non-negligible for evaluating the networks in the search space . One-shot search methods ( Brock et al. , 2017 ; Bender et al. , 2018 ; Chen et al. , 2019b ) integrate all possible architectures in one super network but pre-training the super network and the searched network still bears huge computation cost . As ImageNet ( Deng et al. , 2009 ) pre-training has been a standard practice for many computer vision tasks , there are lots of models trained on ImageNet available in the community . To take full advantages of these models , we propose a Fast Neural Network Adaptation ( FNA ) method based on a novel parameter remapping paradigm . Our method can adapt both the architecture and parameters of one network to a new task with negligible cost . Fig . 1 shows the whole framework . The adaptation is performed on both the architecture- and parameter-level . We adopt the NAS methods ( Zoph et al. , 2017 ; Real et al. , 2018 ; Liu et al. , 2019b ) to implement the architecture-level adaptation . We select a manually designed network ( MobileNetV2 ( Sandler et al. , 2018 ) in our experiments ) as the seed network , which is pre-trained on ImageNet . Then , we expand the seed network to a super network which is the representation of the search space in FNA . New parameters in the super network are initialized by mapping those from the seed network using parameter remapping . Thanks to that , the neural architecture search can be performed efficiently on the detection and segmentation tasks . With FNA we obtain a new optimal target architecture for the new task . Similarly , we remap the parameters of the seed network to the target architecture for initialization and fine-tune it on the target task with no need of pre-training on a large-scale dataset . We demonstrate FNA ’ s effectiveness and efficiency via experiments on both segmentation and detection tasks . We adapt the manually designed network MobileNetV2 ( Sandler et al. , 2018 ) to segmentation framework DeepLabv3 ( Chen et al. , 2017b ) , detection framework RetinaNet ( Lin et al. , 2017 ) and SSDLite ( Liu et al. , 2016 ; Sandler et al. , 2018 ) . Networks adapted by FNA surpass both manually designed and NAS searched networks in terms of both performance and model MAdds . Compared to NAS methods , FNA costs 1737× less than DPC ( Chen et al. , 2018a ) , 6.8× less than Auto-DeepLab ( Liu et al. , 2019a ) and 7.4× less than DetNAS ( Chen et al. , 2019b ) . 2 RELATED WORK . Neural Architecture Search With reinforcement learning ( RL ) and evolutionary algorithm ( EA ) being applied to NAS methods , many works ( Zoph & Le , 2016 ; Zoph et al. , 2017 ; Real et al. , 2018 ) make great progress in promoting the performances of neural networks . Recently , NAS methods based on one-shot model ( Brock et al. , 2017 ; Bender et al. , 2018 ) or differentiable representations ( Liu et al. , 2019b ; Cai et al. , 2019 ; Fang et al. , 2019b ) achieve remarkable results with low search cost . We use the differentiable NAS method to implement architecture adaptation , which adjusts the architecture of the backbone network automatically . Backbone Design As deep neural network designing ( Simonyan & Zisserman , 2014 ; Szegedy et al. , 2016 ; He et al. , 2016 ) develops , the backbones of segmentation or detection networks evolve accordingly . Some works improve the backbone architectures by modifying existing networks . PeleeNet ( Wang et al. , 2018 ) proposes a variant of DenseNet ( Huang et al. , 2017 ) for more realtime object detection on mobile devices . DetNet ( Li et al. , 2018 ) applies dilated convolution ( Yu & Koltun , 2016 ) in the backbone to enlarge the receptive field which helps to detect objects . BiSeNet ( Yu et al. , 2018 ) and HRNet ( Wang et al. , 2019 ) design multiple paths to learn both highand low- resolution representations for better dense prediction . Parameter Remapping Net2Net ( Chen et al. , 2015 ) proposes the function-preserving transformations to remap the parameters of one network to a new deeper or wider network . This remapping mechanism accelerates the training of the new larger network and achieves great performances . Following this manner , EAS ( Cai et al. , 2018 ) extends the parameter remapping concept to neural architecture search . Moreover , some NAS works ( Pham et al. , 2018 ; Fang et al. , 2019a ; Elsken et al. , 2019 ) apply parameters sharing on child models to accelerate the search process . Our parameter remapping paradigm extends the mapping dimension with the kernel level . Parameters can be also mapped to a shallower or narrower network with our scheme , while Net2Net focuses on mapping parameters to a deeper and wider network . The parameter remapping in our FNA largely decreases the computation cost of the network adaptation by taking full advantages of the ImageNet pre-trained parameters . 3 METHOD . As the most commonly used network for designing search spaces in NAS methods ( Tan et al. , 2018 ; Cai et al. , 2019 ; Fang et al. , 2019b ) , MobileNetV2 ( Sandler et al. , 2018 ) is selected as the seed network to give the details of our method . To adapt the network to segmentation and detection tasks , we adjust the architecture elements on three levels , i.e. , convolution kernel size , depth and width of the network . In this section , we first describe the parameter remapping paradigm . Then we explain the whole procedure of the network adaptation . 3.1 PARAMETER REMAPPING . We define parameter remapping as one paradigm which aims at mapping the parameters of one seed network to another one . We denote the seed network as Ns and the new network as Nn , whose parameters are denoted as Ws and Wn respectively . The remapping paradigm is illustrated in the following three aspects . The remapping on the depth-level is firstly carried out and then the remapping on the kernel- and width- level is conducted simultaneously . Moreover , we study different remapping strategies in the experiments ( Sec . 4.5 ) . Remapping on Depth-level We introduce more depth settings in our architecture adaptation process . In other word , we adjust the number of inverted residual blocks ( MBConvs ) ( Sandler et al. , 2018 ) in every stage of the network . We assume that one stage in the seed network Ns has l layers . The parameters of each layer can be denoted as { W ( 1 ) s , W ( 2 ) s , . . . , W ( l ) s } . Similarly , we assume that the corresponding stage with m layers in the new network Nn has parameters { W ( 1 ) n , W ( 2 ) n , . . . , W ( m ) n } . The remapping process on the depth-level is shown in Fig . 2 ( a ) . The parameters of layers in Nn which also exit in Ns are just copied from Ns . The parameters of new layers are all copied from the last layer in the stage of Ns . It is formulated as f ( i ) = min ( i , l ) , W ( i ) n =W ( f ( i ) ) s , ∀1 ≤ i ≤ m. ( 1 ) Remapping on Width-level In the MBConv block of MobileNetV2 ( Sandler et al. , 2018 ) network , the first point-wise convolution expands the low-dimensional features to a high dimension . { ... { ... ( a ) Depth level { ... { ... ( b ) Width level { ... { ... ( c ) Kernel level Figure 2 : Our parameters are remapped on three levels . ( a ) shows the depth-level remapping . The parameters of existing corresponding layers are mapped from the original network . The parameters of new layers are mapped from the last layer in the original network . ( b ) shows the width-level remapping . For each channels-diminished dimension , the parameters are copied from the original existing ones . ( c ) shows the kernel-level remapping . The original parameters are mapped to the central part of the new larger kernel . The values of the other parameters are assigned with 0 . This practice can be utilized for expanding the width and capacity of one neural network . We allow smaller expansion ratios for architecture adaptation . We denote the parameters of one convolution in Ns as W s ∈ Rp×q×h×w and that in Nn as W n ∈ Rr×s×h×w , where r ≤ p and s ≤ q . As shown in Fig . 2 ( b ) , on the width-level , we directly map the first r or s channels of parameters inNs to the narrower one in Nn . It can be formulated as W ni , j , : , : = W s i , j , : , : , ∀1 ≤ i ≤ r , 1 ≤ j ≤ s. ( 2 ) Remapping on Kernel-level The kernel size is commonly set as 3×3 in most artificially-designed networks . To expand the receptive field and capture abundant features in segmentation or detection tasks , we introduce larger kernel size settings in the adaptation process . As Fig . 2 ( c ) shows , to expand the 3 × 3 kernel to a larger one , we assign the parameters of the central 3 × 3 region in the large kernel with the values of the original 3× 3 kernel . The values of the other region surrounding the central part are assigned with 0 . We denote the parameters of the original 3× 3 kernel as W 3×3 and the larger k × k kernel as W k×k . The remapping process on kernel-level can be formulated as follows , W k×k : , : ,h , w = { W 3×3 : , : ,h , w if ( k − 3 ) /2 < h , w ≤ ( k + 3 ) /2 0 otherwise , ( 3 ) where h , w denote the indices of the spatial dimension . This design principle conforms to the function-preserving concept ( Chen et al. , 2015 ) , which accelerates and stabilizes the optimization of the new network .

Summary: The paper proposes a method called FNA (fast network adaptation), which takes a pretrained image classification network, and produces a network for the task of object detection/semantic segmentation. The process consists of three phases: Network Expansion, Architecture Adaptation and Parameters Adaptation, and uses the developed parameter remapping scheme twice. Experiments show that it outperforms recent other NAS methods for these two tasks with same or less computation.