Learning Versatile Neural Architectures by Propagating Network Codes

1 INTRODUCTION . Designing a single neural network architecture that adapts to multiple different tasks is challenging . This is because different tasks , such as image segmentation in Cityscapes ( Cordts et al. , 2016 ) and video recognition in HMDB51 ( Kuehne et al. , 2011 ) , have different data distributions and require different granularity of feature representations . For example , although the manually designed networks ResNet ( He et al. , 2016 ) and HRNet ( Wang et al. , 2020a ) work well on certain tasks such as image classification on ImageNet , they deteriorate in the other tasks . Intuitively , manually designing a single neural architecture that is applicable in all these tasks is difficult . Recently , neural architecture search ( NAS ) has achieved great success in searching network architectures automatically . However , existing NAS methods ( Wu et al. , 2019 ; Liang et al. , 2019 ; Liu et al. , 2019b ; Xie et al. , 2018 ; Cai et al. , 2020 ; Yu et al. , 2020 ; Shaw et al. , 2019b ; a ) typically search on a single task . Though works ( Ding et al. , 2021 ; Duan et al. , 2021 ; Zamir et al. , 2018 ) designed NAS algorithms or datasets that can be used for multiple tasks , they still search different architectures for different tasks , indicating that the costly searching procedure needs to be repeated many times . The problem of learning versatile neural architectures capable of adapting to multiple different tasks remains unsolved , i.e. , searching a multitask architecture or transferring architectures between different tasks . In principle , it faces the challenge of task and dataset inconsistency . Different tasks may require different granularity of feature representations , e.g. , the segmentation task requires more multi-scale features and low-level representations than classification . The key to solving task inconsistency is to design a unified architecture search space for multiple tasks . In contrast to most previous works that simply extend search space designed for image classification 1Project page : https : //network-propagation.github.io for other tasks and build NAS benchmarks ( Dong & Yang , 2020 ; Ying et al. , 2019 ; Siems et al. , 2020 ) on small datasets ( e.g. , CIFAR10 , ImageNet-16 ) with unrealistic settings , we design a multiresolution network space and build a multi-task practical NAS benchmark ( NAS-Bench-MR ) on four challenging datasets including ImageNet-224 ( Deng et al. , 2009 ) , Cityscapes ( Cordts et al. , 2016 ) , KITTI ( Geiger et al. , 2012 ) , and HMDB51 ( Kuehne et al. , 2011 ) . Inspired by HRNet ( Wang et al. , 2020a ) , our network space is a multi-branch multi-resolution space that naturally contains various granularities of representations for different tasks , e.g. , high-resolution features ( Wang et al. , 2020a ) for segmentation while low-resolution ones for classification . NAS-Bench-MR closes the gap between existing benchmarks and NAS in multi-task and real-world scenarios . It serves as an important contribution of this work to facilitate future cross-task NAS research . To solve the challenge of dataset inconsistency , in this work , we propose a novel predictorbased NAS algorithm , termed Network Coding Propagation ( NCP ) , for finding versatile and tasktransferable architectures . NCP transforms data-oriented optimization into architecture-oriented by learning to traverse the search space . It works as follows . We formulate all network hyperparameters into a coding space by representing each architecture hyper-parameter as a code , e.g. , ‘ 3,2,64 ’ denotes there are 3 blocks and each block contains 2 residual blocks with a channel width of 64 . We then learn neural predictors to build the mapping between the network coding and its evaluation metrics ( e.g. , Acc , mIoU , FLOPs ) for each task . By setting high-desired accuracy of each task and FLOPs as the target , we back-propagate gradients of the learned predictor to directly update values of network codes to achieve the target . In this way , good architectures can be found in several forward-backward iterations in seconds , as shown in Fig . 1 . NCP has several appealing benefits : ( 1 ) NCP addresses the data mismatch problem in multi-task learning by learning from network coding but not original data . ( 2 ) NCP works in large spaces in seconds by back-propagating the neural predictor and traversing the search space along the gradient direction . ( 3 ) NCP can use multiple neural predictors for architecture transferring across tasks , as shown in Fig . 1 ( a ) , it adapts an architecture to a new task with only a few iterations . ( 4 ) In NCP , the multi-task learning objective is transformed to gradient accumulation across multiple predictors , as shown in Fig . 1 ( b ) , making NCP naturally applicable to various even conflicting objectives , such as multi-task structure optimization , architecture transferring across tasks , and accuracy-efficiency trade-off for specific computational budgets . Our main contributions are three-fold . ( 1 ) We propose Network Coding Propagation ( NCP ) , which back-propagates the gradients of neural predictors to directly update architecture codes along desired gradient directions for various objectives . ( 2 ) We build NAS-Bench-MR on four challenging datasets under practical training settings for learning task-transferable architectures . We believe it will facilitate future NAS research , especially for multi-task NAS and architecture transferring across tasks . ( 3 ) Extensive studies on inter- , intra- , cross-task generalizability show the effectiveness of NCP in finding versatile and transferable architectures among different even conflicting objectives and tasks . 2 RELATED WORK . Neural Architecture Search Spaces and Benchmarks . Most existing NAS spaces ( Jin et al. , 2019 ; Xu et al. , 2019 ; Wu et al. , 2019 ; Cai et al. , 2019 ; Xie et al. , 2018 ; Stamoulis et al. , 2019 ; Mei et al. , 2020 ; Guo et al. , 2020 ; Dai et al. , 2020 ) are designed for image classification using either a single-branch structure with a group of candidate operators in each layer or a repeated cell structure , e.g. , Darts-based ( Liu et al. , 2019b ) and MobileNet-based ( Sandler et al. , 2018 ) search spaces . Based on these spaces , several NAS benchmarks ( Ying et al. , 2019 ; Dong & Yang , 2020 ; Siems et al. , 2020 ; Duan et al. , 2021 ) have been proposed to pre-evaluate the architectures . However , the above search spaces and benchmarks are built either on proxy settings or small datasets , such as Cifar-10 and ImageNet-16 ( 16 × 16 ) , which is less suitable for other tasks that rely on multiscale information . For those tasks , some search space are explored in segmentation ( Shaw et al. , 2019a ; Liu et al. , 2019a ; Nekrasov et al. , 2019 ; Lin et al. , 2020 ; Chen et al. , 2018 ) and object detection ( Chen et al. , 2019 ; Ghiasi et al. , 2019 ; Du et al. , 2020 ; Wang et al. , 2020b ) by introducing feature aggregation heads ( e.g. , ASPP ( Chen et al. , 2017 ) ) for multi-scale information . Nevertheless , the whole network is still organized in a chain-like single branch manner , resulting in sub-optimal performance . Another relevant work to ours is NAS-Bench-NLP Klyuchnikov et al . ( 2020 ) , which constructs a benchmark with 14k trained architectures in search space of recurrent neural networks on two language modeling datasets . Compared to previous spaces , our multi-resolution search space , including searchable numbers of resolutions/blocks/channels , is naturally designed for multiple vision tasks as it contains various granularities of feature representations . Based on our search space , we build NAS-Bench-MR for various vision tasks , including classification , segmentation , 3D detection , and video recognition . Detailed comparisons of NAS benchmarks can be found in Tab . 1 . Neural Architecture Search Methods . Generally , NAS trains numerous candidate architectures from a search space and evaluates their performance to find the optimal architecture , which is costly . To reduce training costs , weight-sharing NAS methods ( Liu et al. , 2019b ; Jin et al. , 2019 ; Xu et al. , 2019 ; Cai et al. , 2019 ; Guo et al. , 2020 ; Li & Talwalkar , 2020 ) are proposed to jointly train a large number of candidate networks within a super-network . Different searching strategies are employed within this framework such as reinforcement learning ( Pham et al. , 2018 ) , importance factor learning ( Liu et al. , 2019b ; Cai et al. , 2019 ; Stamoulis et al. , 2019 ; Xu et al. , 2019 ) , path sampling ( You et al. , 2020 ; Guo et al. , 2020 ; Xie et al. , 2018 ) , and channel pruning ( Mei et al. , 2020 ; Yu & Huang , 2019 ) . However , recent analysis ( Sciuto et al. , 2020 ; Wang et al. , 2021 ) shows that the magnitude of importance parameters in the weight-sharing NAS framework does not reflect the true ranking of the final architectures . Without weight-sharing , hyperparameter optimization methods ( Tan & Le , 2019 ; Radosavovic et al. , 2020 ; Baker et al. , 2017 ; Wen et al. , 2020 ; Lu et al. , 2019 ; Luo et al. , 2020 ; Chau et al. , 2020 ; Yan et al. , 2020 ) has shown its effectiveness by learning the relationship between network hyperparameters and their performance . For example , RegNet ( Radosavovic et al. , 2020 ) explains the widths and depths of good networks by a quantized linear function . Predictor-based methods ( Wen et al. , 2020 ; Luo et al. , 2020 ; Chau et al. , 2020 ; Yan et al. , 2020 ; Luo et al. , 2018 ) learn predictors , such as Gaussian process ( Dai et al. , 2019 ) and graph convolution networks ( Wen et al. , 2020 ) , to predict the performance of all candidate models in the search space . A subset of models with high predicted accuracies is then trained for the final selection . Our Network Coding Propagation ( NCP ) belongs to the predictor-based method , but is different from existing methods in : ( 1 ) NCP searches in a large search space in seconds without evaluating all candidate models by back-propagating the neural predictor and traversing the search space along the gradient direction . ( 2 ) Benefiting from the gradient back-propagation in network coding space , NCP is naturally applicable to various objectives across different tasks , such as multi-task structure optimization , architecture transferring , and accuracy-efficiency trade-offs . ( 3 ) Different from Luo et al . ( 2018 ) ; Baker et al . ( 2017 ) jointly train an encoder , a performance predictor , and a decoder to minimize the combination of performance prediction loss and structure reconstruction loss , we learn and inverse the neural predictor directly in our coding space to make the gradient updating and network editing explicit and transparent . 3 METHODOLOGY . NCP aims to customize specific network hyperparameters for different optimization objectives , such as single- and multi-task learning and accuracy-efficiency trade-offs , in an architecture coding space . An overview of NCP is shown in Fig . 2 . In this section , we first discuss learning efficient models on a single task with two strategies and then show that it can be easily extended to multi-task scenarios . Lastly , we demonstrate our multi-resolution coding space and NAS-Bench-MR .

Summary: This paper organized a multi-task NAS benchmark including 4 widely used datasets and more than 20,000 models. It also proposed a searching architecture named Network Coding Propagation (NCP) to effectively find the optical model for specific tasks. Experimental results indicate that the proposed model can be applied to inter-task, cross-task, and intra-tasks problems and the authors also showed the generalization capability to other benchmarks.