Learning Versatile Neural Architectures by Propagating Network Codes

1 INTRODUCTION . Designing a single neural network architecture that adapts to multiple different tasks is challenging . This is because different tasks , such as image segmentation in Cityscapes ( Cordts et al. , 2016 ) and video recognition in HMDB51 ( Kuehne et al. , 2011 ) , have different data distributions and require different granularity of feature representations . For example , although the manually designed networks ResNet ( He et al. , 2016 ) and HRNet ( Wang et al. , 2020a ) work well on certain tasks such as image classification on ImageNet , they deteriorate in the other tasks . Intuitively , manually designing a single neural architecture that is applicable in all these tasks is difficult . Recently , neural architecture search ( NAS ) has achieved great success in searching network architectures automatically . However , existing NAS methods ( Wu et al. , 2019 ; Liang et al. , 2019 ; Liu et al. , 2019b ; Xie et al. , 2018 ; Cai et al. , 2020 ; Yu et al. , 2020 ; Shaw et al. , 2019b ; a ) typically search on a single task . Though works ( Ding et al. , 2021 ; Duan et al. , 2021 ; Zamir et al. , 2018 ) designed NAS algorithms or datasets that can be used for multiple tasks , they still search different architectures for different tasks , indicating that the costly searching procedure needs to be repeated many times . The problem of learning versatile neural architectures capable of adapting to multiple different tasks remains unsolved , i.e. , searching a multitask architecture or transferring architectures between different tasks . In principle , it faces the challenge of task and dataset inconsistency . Different tasks may require different granularity of feature representations , e.g. , the segmentation task requires more multi-scale features and low-level representations than classification . The key to solving task inconsistency is to design a unified architecture search space for multiple tasks . In contrast to most previous works that simply extend search space designed for image classification 1Project page : https : //network-propagation.github.io for other tasks and build NAS benchmarks ( Dong & Yang , 2020 ; Ying et al. , 2019 ; Siems et al. , 2020 ) on small datasets ( e.g. , CIFAR10 , ImageNet-16 ) with unrealistic settings , we design a multiresolution network space and build a multi-task practical NAS benchmark ( NAS-Bench-MR ) on four challenging datasets including ImageNet-224 ( Deng et al. , 2009 ) , Cityscapes ( Cordts et al. , 2016 ) , KITTI ( Geiger et al. , 2012 ) , and HMDB51 ( Kuehne et al. , 2011 ) . Inspired by HRNet ( Wang et al. , 2020a ) , our network space is a multi-branch multi-resolution space that naturally contains various granularities of representations for different tasks , e.g. , high-resolution features ( Wang et al. , 2020a ) for segmentation while low-resolution ones for classification . NAS-Bench-MR closes the gap between existing benchmarks and NAS in multi-task and real-world scenarios . It serves as an important contribution of this work to facilitate future cross-task NAS research . To solve the challenge of dataset inconsistency , in this work , we propose a novel predictorbased NAS algorithm , termed Network Coding Propagation ( NCP ) , for finding versatile and tasktransferable architectures . NCP transforms data-oriented optimization into architecture-oriented by learning to traverse the search space . It works as follows . We formulate all network hyperparameters into a coding space by representing each architecture hyper-parameter as a code , e.g. , ‘ 3,2,64 ’ denotes there are 3 blocks and each block contains 2 residual blocks with a channel width of 64 . We then learn neural predictors to build the mapping between the network coding and its evaluation metrics ( e.g. , Acc , mIoU , FLOPs ) for each task . By setting high-desired accuracy of each task and FLOPs as the target , we back-propagate gradients of the learned predictor to directly update values of network codes to achieve the target . In this way , good architectures can be found in several forward-backward iterations in seconds , as shown in Fig . 1 . NCP has several appealing benefits : ( 1 ) NCP addresses the data mismatch problem in multi-task learning by learning from network coding but not original data . ( 2 ) NCP works in large spaces in seconds by back-propagating the neural predictor and traversing the search space along the gradient direction . ( 3 ) NCP can use multiple neural predictors for architecture transferring across tasks , as shown in Fig . 1 ( a ) , it adapts an architecture to a new task with only a few iterations . ( 4 ) In NCP , the multi-task learning objective is transformed to gradient accumulation across multiple predictors , as shown in Fig . 1 ( b ) , making NCP naturally applicable to various even conflicting objectives , such as multi-task structure optimization , architecture transferring across tasks , and accuracy-efficiency trade-off for specific computational budgets . Our main contributions are three-fold . ( 1 ) We propose Network Coding Propagation ( NCP ) , which back-propagates the gradients of neural predictors to directly update architecture codes along desired gradient directions for various objectives . ( 2 ) We build NAS-Bench-MR on four challenging datasets under practical training settings for learning task-transferable architectures . We believe it will facilitate future NAS research , especially for multi-task NAS and architecture transferring across tasks . ( 3 ) Extensive studies on inter- , intra- , cross-task generalizability show the effectiveness of NCP in finding versatile and transferable architectures among different even conflicting objectives and tasks . 2 RELATED WORK . Neural Architecture Search Spaces and Benchmarks . Most existing NAS spaces ( Jin et al. , 2019 ; Xu et al. , 2019 ; Wu et al. , 2019 ; Cai et al. , 2019 ; Xie et al. , 2018 ; Stamoulis et al. , 2019 ; Mei et al. , 2020 ; Guo et al. , 2020 ; Dai et al. , 2020 ) are designed for image classification using either a single-branch structure with a group of candidate operators in each layer or a repeated cell structure , e.g. , Darts-based ( Liu et al. , 2019b ) and MobileNet-based ( Sandler et al. , 2018 ) search spaces . Based on these spaces , several NAS benchmarks ( Ying et al. , 2019 ; Dong & Yang , 2020 ; Siems et al. , 2020 ; Duan et al. , 2021 ) have been proposed to pre-evaluate the architectures . However , the above search spaces and benchmarks are built either on proxy settings or small datasets , such as Cifar-10 and ImageNet-16 ( 16 × 16 ) , which is less suitable for other tasks that rely on multiscale information . For those tasks , some search space are explored in segmentation ( Shaw et al. , 2019a ; Liu et al. , 2019a ; Nekrasov et al. , 2019 ; Lin et al. , 2020 ; Chen et al. , 2018 ) and object detection ( Chen et al. , 2019 ; Ghiasi et al. , 2019 ; Du et al. , 2020 ; Wang et al. , 2020b ) by introducing feature aggregation heads ( e.g. , ASPP ( Chen et al. , 2017 ) ) for multi-scale information . Nevertheless , the whole network is still organized in a chain-like single branch manner , resulting in sub-optimal performance . Another relevant work to ours is NAS-Bench-NLP Klyuchnikov et al . ( 2020 ) , which constructs a benchmark with 14k trained architectures in search space of recurrent neural networks on two language modeling datasets . Compared to previous spaces , our multi-resolution search space , including searchable numbers of resolutions/blocks/channels , is naturally designed for multiple vision tasks as it contains various granularities of feature representations . Based on our search space , we build NAS-Bench-MR for various vision tasks , including classification , segmentation , 3D detection , and video recognition . Detailed comparisons of NAS benchmarks can be found in Tab . 1 . Neural Architecture Search Methods . Generally , NAS trains numerous candidate architectures from a search space and evaluates their performance to find the optimal architecture , which is costly . To reduce training costs , weight-sharing NAS methods ( Liu et al. , 2019b ; Jin et al. , 2019 ; Xu et al. , 2019 ; Cai et al. , 2019 ; Guo et al. , 2020 ; Li & Talwalkar , 2020 ) are proposed to jointly train a large number of candidate networks within a super-network . Different searching strategies are employed within this framework such as reinforcement learning ( Pham et al. , 2018 ) , importance factor learning ( Liu et al. , 2019b ; Cai et al. , 2019 ; Stamoulis et al. , 2019 ; Xu et al. , 2019 ) , path sampling ( You et al. , 2020 ; Guo et al. , 2020 ; Xie et al. , 2018 ) , and channel pruning ( Mei et al. , 2020 ; Yu & Huang , 2019 ) . However , recent analysis ( Sciuto et al. , 2020 ; Wang et al. , 2021 ) shows that the magnitude of importance parameters in the weight-sharing NAS framework does not reflect the true ranking of the final architectures . Without weight-sharing , hyperparameter optimization methods ( Tan & Le , 2019 ; Radosavovic et al. , 2020 ; Baker et al. , 2017 ; Wen et al. , 2020 ; Lu et al. , 2019 ; Luo et al. , 2020 ; Chau et al. , 2020 ; Yan et al. , 2020 ) has shown its effectiveness by learning the relationship between network hyperparameters and their performance . For example , RegNet ( Radosavovic et al. , 2020 ) explains the widths and depths of good networks by a quantized linear function . Predictor-based methods ( Wen et al. , 2020 ; Luo et al. , 2020 ; Chau et al. , 2020 ; Yan et al. , 2020 ; Luo et al. , 2018 ) learn predictors , such as Gaussian process ( Dai et al. , 2019 ) and graph convolution networks ( Wen et al. , 2020 ) , to predict the performance of all candidate models in the search space . A subset of models with high predicted accuracies is then trained for the final selection . Our Network Coding Propagation ( NCP ) belongs to the predictor-based method , but is different from existing methods in : ( 1 ) NCP searches in a large search space in seconds without evaluating all candidate models by back-propagating the neural predictor and traversing the search space along the gradient direction . ( 2 ) Benefiting from the gradient back-propagation in network coding space , NCP is naturally applicable to various objectives across different tasks , such as multi-task structure optimization , architecture transferring , and accuracy-efficiency trade-offs . ( 3 ) Different from Luo et al . ( 2018 ) ; Baker et al . ( 2017 ) jointly train an encoder , a performance predictor , and a decoder to minimize the combination of performance prediction loss and structure reconstruction loss , we learn and inverse the neural predictor directly in our coding space to make the gradient updating and network editing explicit and transparent . 3 METHODOLOGY . NCP aims to customize specific network hyperparameters for different optimization objectives , such as single- and multi-task learning and accuracy-efficiency trade-offs , in an architecture coding space . An overview of NCP is shown in Fig . 2 . In this section , we first discuss learning efficient models on a single task with two strategies and then show that it can be easily extended to multi-task scenarios . Lastly , we demonstrate our multi-resolution coding space and NAS-Bench-MR .

Summary: This paper tackles neural architecture search (NAS), addressing the problem of finding architectures suitable for a multitude of vision-related tasks, ranging from object detection to semantic segmentation. To the best of my knowledge, this is the first attempt in NAS for computer vision models that explicitly design the search space that allows for a search of architectures that are broadly applicable to a variety of tasks that usually rely on the different granularity of feature representation.  This paper makes two important steps towards the automated search of such general, multi-purpose architectures: (1) to study this problem, the paper formalizes a multi-task NAS benchmark, covering recognition, detection, semantic segmentation, and activity recognition (video) datasets. This allows for assessing the generality of the sought architecture across vision tasks. (2) to tackle the architecture search in such a setting, this paper propose makes the following contributions. First, it augments the search space with multiple stages that extract feature maps at several resolutions, followed by a feature fusion step (the number of blocks and feat. map resolutions are governed by the network hyperparameters.). This seems to be one of the key features that help to find architectures suitable for tasks that differ as much as image classification and semantic segmentation.  Next, for assessing the performance in this large search space, this paper builds on predictor-based methods, where the idea is to train a network that predicts the performance (i.e., the learned predictor) of the sought-network based on the (encoded) hyperparameters, governing the network architecture. Similarly, the proposed neural coding propagation (NCP) directly encodes the hyperparameters in the network coding space — those can thus be directly updated via back-propagation of the learned predictor. A predictor is learned for each task independently, and this way, gradients can be accumulated separately across different tasks before jointly updating the “architecture codes”.