LEARNING VERSATILE NEURAL ARCHITECTURES BY PROPAGATING NETWORK CODES
Year: 2022

Abstract:
This work explores how to design a single neural network capable of adapting to multiple heterogeneous vision tasks, such as image segmentation, 3D detection, and video recognition. This goal is challenging because both network architecture search (NAS) spaces and methods in different tasks are inconsistent. We solve this challenge from both sides. We first introduce a unified design space for multiple tasks and build a multitask NAS benchmark (NAS-Bench-MR) on many widely used datasets, including ImageNet, Cityscapes, KITTI, and HMDB51. We further propose Network Coding Propagation (NCP), which back-propagates gradients of neural predictors to directly update architecture codes along the desired gradient directions to solve various tasks. In this way, optimal architecture configurations can be found by NCP in our large search space in seconds. Unlike prior arts of NAS that typically focus on a single task, NCP has several unique benefits. (1) NCP transforms architecture optimization from data-driven to architecture-driven, enabling joint search an architecture among multitasks with different data distributions. (2) NCP learns from network codes but not original data, enabling it to update the architecture efficiently across datasets. (3) In addition to our NAS-Bench-MR, NCP performs well on other NAS benchmarks, such as NAS-Bench-201. (4) Thorough studies of NCP on inter-, cross-, and intra-tasks highlight the importance of cross-task neural architecture design, i.e., multitask neural architectures and architecture transferring between different tasks. Code is available at github.com/dingmyu/NCP 1.

heading: 1 INTRODUCTION
text: Designing a single neural network architecture that adapts to multiple different tasks is challenging. This is because different tasks, such as image segmentation in Cityscapes (Cordts et al., 2016) and video recognition in HMDB51 (Kuehne et al., 2011), have different data distributions and require different granularity of feature representations. For example, although the manually designed networks ResNet (He et al., 2016) and HRNet (Wang et al., 2020a) work well on certain tasks such as image classification on ImageNet, they deteriorate in the other tasks. Intuitively, manually designing a single neural architecture that is applicable in all these tasks is difficult.
Recently, neural architecture search (NAS) has achieved great success in searching network architectures automatically. However, existing NAS methods (Wu et al., 2019; Liang et al., 2019; Liu et al., 2019b; Xie et al., 2018; Cai et al., 2020; Yu et al., 2020; Shaw et al., 2019b;a) typically search on a single task. Though works (Ding et al., 2021; Duan et al., 2021; Zamir et al., 2018) designed NAS algorithms or datasets that can be used for multiple tasks, they still search different architectures for different tasks, indicating that the costly searching procedure needs to be repeated many times. The problem of learning versatile neural architectures capable of adapting to multiple different tasks remains unsolved, i.e., searching a multitask architecture or transferring architectures between different tasks. In principle, it faces the challenge of task and dataset inconsistency.
Different tasks may require different granularity of feature representations, e.g., the segmentation task requires more multi-scale features and low-level representations than classification. The key to solving task inconsistency is to design a unified architecture search space for multiple tasks. In contrast to most previous works that simply extend search space designed for image classification
1Project page: https://network-propagation.github.io
for other tasks and build NAS benchmarks (Dong & Yang, 2020; Ying et al., 2019; Siems et al., 2020) on small datasets (e.g., CIFAR10, ImageNet-16) with unrealistic settings, we design a multiresolution network space and build a multi-task practical NAS benchmark (NAS-Bench-MR) on four challenging datasets including ImageNet-224 (Deng et al., 2009), Cityscapes (Cordts et al., 2016), KITTI (Geiger et al., 2012), and HMDB51 (Kuehne et al., 2011). Inspired by HRNet (Wang et al., 2020a), our network space is a multi-branch multi-resolution space that naturally contains various granularities of representations for different tasks, e.g., high-resolution features (Wang et al., 2020a) for segmentation while low-resolution ones for classification. NAS-Bench-MR closes the gap between existing benchmarks and NAS in multi-task and real-world scenarios. It serves as an important contribution of this work to facilitate future cross-task NAS research.
To solve the challenge of dataset inconsistency, in this work, we propose a novel predictorbased NAS algorithm, termed Network Coding Propagation (NCP), for finding versatile and tasktransferable architectures. NCP transforms data-oriented optimization into architecture-oriented by learning to traverse the search space. It works as follows. We formulate all network hyperparameters into a coding space by representing each architecture hyper-parameter as a code, e.g., ‘3,2,64’ denotes there are 3 blocks and each block contains 2 residual blocks with a channel width of 64. We then learn neural predictors to build the mapping between the network coding and its evaluation metrics (e.g., Acc, mIoU, FLOPs) for each task. By setting high-desired accuracy of each task and FLOPs as the target, we back-propagate gradients of the learned predictor to directly update values of network codes to achieve the target. In this way, good architectures can be found in several forward-backward iterations in seconds, as shown in Fig. 1.
NCP has several appealing benefits: (1) NCP addresses the data mismatch problem in multi-task learning by learning from network coding but not original data. (2) NCP works in large spaces in seconds by back-propagating the neural predictor and traversing the search space along the gradient direction. (3) NCP can use multiple neural predictors for architecture transferring across tasks, as shown in Fig. 1(a), it adapts an architecture to a new task with only a few iterations. (4) In NCP, the multi-task learning objective is transformed to gradient accumulation across multiple predictors, as shown in Fig. 1(b), making NCP naturally applicable to various even conflicting objectives, such as multi-task structure optimization, architecture transferring across tasks, and accuracy-efficiency trade-off for specific computational budgets.
Our main contributions are three-fold. (1) We propose Network Coding Propagation (NCP), which back-propagates the gradients of neural predictors to directly update architecture codes along desired gradient directions for various objectives. (2) We build NAS-Bench-MR on four challenging datasets under practical training settings for learning task-transferable architectures. We believe it will facilitate future NAS research, especially for multi-task NAS and architecture transferring across tasks. (3) Extensive studies on inter-, intra-, cross-task generalizability show the effectiveness of NCP in finding versatile and transferable architectures among different even conflicting objectives and tasks.

heading: 2 RELATED WORK
text: Neural Architecture Search Spaces and Benchmarks. Most existing NAS spaces (Jin et al., 2019; Xu et al., 2019; Wu et al., 2019; Cai et al., 2019; Xie et al., 2018; Stamoulis et al., 2019;
Mei et al., 2020; Guo et al., 2020; Dai et al., 2020) are designed for image classification using either a single-branch structure with a group of candidate operators in each layer or a repeated cell structure, e.g., Darts-based (Liu et al., 2019b) and MobileNet-based (Sandler et al., 2018) search spaces. Based on these spaces, several NAS benchmarks (Ying et al., 2019; Dong & Yang, 2020; Siems et al., 2020; Duan et al., 2021) have been proposed to pre-evaluate the architectures. However, the above search spaces and benchmarks are built either on proxy settings or small datasets, such as Cifar-10 and ImageNet-16 (16 × 16), which is less suitable for other tasks that rely on multiscale information. For those tasks, some search space are explored in segmentation (Shaw et al., 2019a; Liu et al., 2019a; Nekrasov et al., 2019; Lin et al., 2020; Chen et al., 2018) and object detection (Chen et al., 2019; Ghiasi et al., 2019; Du et al., 2020; Wang et al., 2020b) by introducing feature aggregation heads (e.g., ASPP (Chen et al., 2017)) for multi-scale information. Nevertheless, the whole network is still organized in a chain-like single branch manner, resulting in sub-optimal performance. Another relevant work to ours is NAS-Bench-NLP Klyuchnikov et al. (2020), which constructs a benchmark with 14k trained architectures in search space of recurrent neural networks on two language modeling datasets.
Compared to previous spaces, our multi-resolution search space, including searchable numbers of resolutions/blocks/channels, is naturally designed for multiple vision tasks as it contains various granularities of feature representations. Based on our search space, we build NAS-Bench-MR for various vision tasks, including classification, segmentation, 3D detection, and video recognition. Detailed comparisons of NAS benchmarks can be found in Tab. 1.
Neural Architecture Search Methods. Generally, NAS trains numerous candidate architectures from a search space and evaluates their performance to find the optimal architecture, which is costly. To reduce training costs, weight-sharing NAS methods (Liu et al., 2019b; Jin et al., 2019; Xu et al., 2019; Cai et al., 2019; Guo et al., 2020; Li & Talwalkar, 2020) are proposed to jointly train a large number of candidate networks within a super-network. Different searching strategies are employed within this framework such as reinforcement learning (Pham et al., 2018), importance factor learning (Liu et al., 2019b; Cai et al., 2019; Stamoulis et al., 2019; Xu et al., 2019), path sampling (You et al., 2020; Guo et al., 2020; Xie et al., 2018), and channel pruning (Mei et al., 2020; Yu & Huang, 2019). However, recent analysis (Sciuto et al., 2020; Wang et al., 2021) shows that the magnitude of importance parameters in the weight-sharing NAS framework does not reflect the true ranking of the final architectures. Without weight-sharing, hyperparameter optimization methods (Tan & Le, 2019; Radosavovic et al., 2020; Baker et al., 2017; Wen et al., 2020; Lu et al., 2019; Luo et al., 2020; Chau et al., 2020; Yan et al., 2020) has shown its effectiveness by learning the relationship between network hyperparameters and their performance. For example, RegNet (Radosavovic et al., 2020) explains the widths and depths of good networks by a quantized linear function. Predictor-based methods (Wen et al., 2020; Luo et al., 2020; Chau et al., 2020; Yan et al., 2020; Luo et al., 2018) learn predictors, such as Gaussian process (Dai et al., 2019) and graph convolution networks (Wen et al., 2020), to predict the performance of all candidate models in the search space. A subset of models with high predicted accuracies is then trained for the final selection.
Our Network Coding Propagation (NCP) belongs to the predictor-based method, but is different from existing methods in: (1) NCP searches in a large search space in seconds without evaluating all candidate models by back-propagating the neural predictor and traversing the search space along the gradient direction. (2) Benefiting from the gradient back-propagation in network coding space, NCP is naturally applicable to various objectives across different tasks, such as multi-task structure
optimization, architecture transferring, and accuracy-efficiency trade-offs. (3) Different from Luo et al. (2018); Baker et al. (2017) jointly train an encoder, a performance predictor, and a decoder to minimize the combination of performance prediction loss and structure reconstruction loss, we learn and inverse the neural predictor directly in our coding space to make the gradient updating and network editing explicit and transparent.

heading: 3 METHODOLOGY
text: NCP aims to customize specific network hyperparameters for different optimization objectives, such as single- and multi-task learning and accuracy-efficiency trade-offs, in an architecture coding space. An overview of NCP is shown in Fig. 2. In this section, we first discuss learning efficient models on a single task with two strategies and then show that it can be easily extended to multi-task scenarios. Lastly, we demonstrate our multi-resolution coding space and NAS-Bench-MR.

heading: 3.1 NETWORK CODING PROPAGATION
text: For each task, we learn a neural predictor F(·) that projects an architecture code e to its ground truth evaluation metrics, such as accuracy gacc and FLOPs gflops. Given an initialized and normalized coding e, the predicted metrics and the loss to learn the neural predictor are represented by:
pacc, pflops, ptask-spec = FW (e), (1) Lpredictor = L2(pacc, gacc) + L2(pflops, gflops) + ... (2)
where W is the weight of the neural predictor, which is fixed after learning. pacc, pflops, and ptask-spec denote the predicted accuracy, FLOPs, and task-specific metrics (if any), L2 is the L2 norm. Note that although the FLOPs can be directly calculated from network codes, we learn it to enable the backward gradient upon FLOPs constraints for accuracy-efficiency trade-offs.
NCP edits the network coding e by back-propagating the learned neural predictor W to maximize/minimize the evaluation metrics (e.g., high accuracy and low FLOPs) along the gradient directions. According to the number of dimensions to be edited, two strategies are proposed to propagate the network coding: continuous propagation and winner-takes-all propagation, where the first one uses all the dimensions of the gradient to update e, while the latter uses only the one dimension with the largest gradient. Fig. 3 visualizes the network update process of our two strategies.
Continuous Propagation. In continuous propagation, we set target metrics as an optimization goal and use gradient descending to automatically update the network coding e. Taking accuracy and FLOPs as an example, we set the target accuracy tacc and target FLOPs tflops for e and calculate the loss between the prediction and the target by:
Lpropagation = L1(pacc, tacc) + λL1(pflops, tflops) (3)
where L1 denotes the smooth L1 loss, λ is the coefficient to balance the trade-off between accuracy and efficiency. The loss is propagated back through the fixed neural predictor FW (·), and the gradient on the coding e is calculated as ∂L∂e (use ∇e in following for simplicity). We update the coding
e iteratively by e ← e − ∇e. In this way, NCP can search for the most efficient model upon a lower-bound accuracy and the most accurate model upon upper-bound FLOPs. In practice, we set tacc = pacc + 1 and tflops = pflops − 1 to find a model with a good efficiency-accuracy trade-off. The propagation is stopped once the prediction reaches the target or after a certain number of iterations (e.g., 70). Finally, we round (the number of channels is a multiple of 8) and decode the coding e to obtain the architecture optimized for the specific goal.
Winner-Takes-All Propagation. Editing different dimensions of the coding often have different degrees of impact on the network. For example, modifying the number of network blocks has a greater impact than the number of channels, i.e., a block costs more computational resources than several channels. In light of this, we select one dimension of the coding e with the best accuracyefficiency for propagation in each iteration, termed as winner-takes-all propagation.
We create a lookup table T (·) to obtain the corresponding FLOPs of any network coding. We then calculate the relative increase ∆r in FLOPs as each dimension of the coding grows:
e′l = el + k ∆rl = T (e′)− T (e) (4)
where l denotes the index of the coding e; k = 8 if el represents the number of convolutional channels, otherwise k = 1. We then use ∆r as a normalization term of backward gradients ∇e and select the optimal dimension l of the normalized gradients∇e/∆r:
l = { arg max(∇e/∆r) if max(∇e/∆r) > 0 arg min(∇e/∆r) if max(∇e/∆r) < 0
(5)
if max(∇e/∆r) > 0, modifying the value of el may improve accuracy and decrease FLOPs simultaneously. While if max(∇e/∆r) < 0, we choose the dimension with the highest improvement in accuracy under the unit FLOPs consumption. The coding e is then updated as follows:
el ← { el − k if max(∇e/∆r) > 0 el + k if max(∇e/∆r) < 0
(6)
Cross-task Learning. Our NCP provides a natural way for cross-task learning by transforming different objectives across tasks into gradient accumulation operations directly in the architecture coding space, facilitating joint architecture search for multiple tasks and cross-task architecture transferring, as shown in Fig. 1. For example, NCP enables joint search for classification and segmentation by accumulating the gradients∇ecls and ∇eseg from the two predictors:
∇e = ∇ecls +∇eseg (7) Note that different weights coefficients can be used to learn multi-task architectures with task preferences. In this work, we set equal weight coefficients for these tasks.

heading: 3.2 NAS-BENCH-MR
text: Multi-Resolution Search Space. Inspired by HRNet (Wang et al., 2020a; Ding et al., 2021), we design a multi-branch multi-resolution search space that contains four branches and maintains both
high-resolution and low-resolution representations throughout the network. As shown in Fig. 2, after two convolutional layers decreasing the feature resolution to 1/4 (4×) of the image size, we start with this high-resolution feature branch and gradually add lower-resolution branches with feature fusions, and connect the multi-resolution branches in parallel. Multi-resolution features are resized and concatenated for the final classification/regression without any additional heads.
The design space follows the following key characteristics. (1) Parallel modules: extend each branch of the corresponding resolutions in parallel using residual units (He et al., 2016); (2) Fusion modules: repeatedly fuse the information across multiple resolutions and generate the new resolution (only between two stages). For each output resolution, the corresponding input features of its neighboring resolutions are gathered by residual units (He et al., 2016) followed by element-wise addition (upsampling is used for the low-to-high resolution feature transformation), e.g., the 8× output feature contains information of 4×, 8×, and 16× input features; (3) A modularized block is then formed by a fusion module and a parallel module.
Coding. We leverage the hyperparameter space of our multi-resolution network by projecting the depth and width values of all branches of each stage into a continuous-valued coding space. Formally, two 3 × 3 stride 2 convolutional layers with i1, i2 number of channels are used to obtain the 4× high-resolution features at the beginning of the network. We then divide the network into four stages with s = {1, 2, 3, 4} branches, respectively. Each stage contains bs modularized blocks, where the fusion module of the first block is used for transition between two stages (e.g., from 2 branches to 3 branches). For the parallel module of the block, we set the number of residual units and the number of convolutional channels to ns = [n1s, . . . , n s s] and cs = [c 1 s, . . . , c s s] respectively, where s denotes the s-th stage containing s branches. Since the fusion module always exists between two parallel modules, once the hyperparameters of the parallel modules are determined, the corresponding number of channels of fusion modules will be set automatically to connect the two parallel modules. The s-th stage comprising a plurality of same modular blocks is represented by:
bs, n 1 s, . . . , n s s, c 1 s, . . . , c s s (8)
where the four stages are represented by 3, 5, 7, and 9-dimensional codes, respectively. By the end of the 4-th stage of the network, we use a 1× 1 stride 1 convolutional layer with a channel number of o1 to fuse the four resized feature representations with different resolutions. In general, the entire architecture can be represented by a 27-dimensional continuous-valued code:
e = i1, i2, b1,n1, c1, . . . , b4,n4, c4, o1 (9)
We then build our coding space by sampling and assigning values to the codes e. For each structure, we randomly choose b, n ∈ 1, 2, 3, 4, and i, c, o ∈ {8, 16, 24, . . . , 128}, resulting in a 27- dimensional coding. In this way, a fine-grained multi-resolution space that enables searching structures for various tasks with different preferences on the granularity of features is constructed.
Dataset. We build our NAS-Bench-MR on four carefully selected visual recognition tasks that require different granularity of features: image classification on ImageNet (Deng et al., 2009), semantic segmentation on Cityscapes (Cordts et al., 2016), 3D object detection on KITTI (Geiger et al., 2012), and video recognition on HMDB51 (Kuehne et al., 2011).
For classification, considering the diversity of the number of classes and data scales in practical applications, we construct three subsets: ImageNet-50-1000 (Cls-A), ImageNet-50-100 (Cls-B), and ImageNet-101000 (Cls-C), where the first number indicates the number of classes and the second one denotes the number of images per class. For segmentation, we follow HRNet (Wang et al., 2020a) by upsampling all branches to the high-resolution one. For 3D detection, we follow PointPillars (Lang et al., 2019) that first converts 3D point points into bird’s-eye view (BEV) featuremaps and then unitizes a 2D network. For video recognition, we replace the last 2D convolutional layer before the final classification layer of our network with a 3D convolutional layer.
We also explore several proxy training settings, such as 10-epoch training used in RegNet (Radosavovic et al., 2020) (Cls-10c), reduced input resolution by four times (Seg-4x), and pretraining instead of training from scratch (Video-p). In summary, we randomly sample 2,500 structures from our search space, train and evaluate these same architectures for those above nine different tasks/settings, resulting in 22,500 trained models. We believe these well-designed and fully-trained models serve as an important contribution of this work to facilitate future NAS research.

heading: 4 EXPERIMENTS

heading: 4.1 IMPLEMENTATION DETAILS
text: Taking a three-layer fully connected network as the neural predictor, we apply NCP to NAS-BenchMR and obtain NCP-Net. To train the neural predictor, 2000 and 500 structures in the benchmark are used as the training and validation sets for each task. Unless specified, we use continuous propagation with an initial code of {b, n = 2; c, i, o = 64} and λ = 0.5 for 70 iterations in all experiments. The optimization goal is set to higher performance and lower FLOPs (tacc = pacc + 1, tflops = pflops − 1). More experiments and details can be found in Appendix.

heading: 4.2 BASELINE EVALUATIONS
text: We make comparisons among different search strategies on our multi-resolution space and segmentation benchmark in Tab. 2. For Random Search (Bergstra & Bengio, 2012) and Neural Predictor (Wen et al., 2020), we sample 100 and 10,000 network codes respectively and report the highest result in top-10 models. For NetAdapt (Yang et al., 2018), we traverse each dimension of the initial codes (increasing or decreasing a unit value) and adapt the codes by greedy strategy.
Effectiveness of NCP. (1) NCP outperforms manually-designed networks, e.g., ResNet (He et al., 2016) and HRNet (Wang et al., 2020a), hyperparameter optimization methods (Bergstra & Bengio, 2012; Wen et al., 2020; Yang et al., 2018), and even well-designed weight-sharing NAS (Liu et al., 2019a; Shaw et al., 2019a). (2) Compared to other hyperparameter optimization methods, NCP is not sensitive to the randomness and noise in model selection by adopting an approximate gradient direction. (3) NCP is much more efficient as it evaluates all dimensions of the codes with only one back-propagation without top-K ranking. (4) NCP is able to search for optimal architectures under different computing budgets by simply setting the coefficient λ.
Effectiveness of our search space. (1) Random Search (Bergstra & Bengio, 2012) works well in our coding space, showing the effectiveness of our multi-resolution space. (2) Models in our search space consist of only 3× 3 convolution in the basic residual unit (He et al., 2016) without complex operators, making it applicable to various computing platforms, such as CPU, GPU, and FPGA.
Search strategies of NCP. Intuitively, the winner-take-all strategy is more like a greedy algorithm. It selects one coding dimension with the best accuracy-efficiency trade-off for propagation, resulting
in efficient models. The continuous strategy updates all dimensions of the coding purely based on gradients. Thus, it reaches better accuracy, though the generated model is not the most efficient. The experiment verifies the above intuition. Unless specified, we use the continuous strategy for all experiments. Fig. 3 shows the code editing process of two strategies.

heading: 4.3 INTER-TASK ARCHITECTURE SEARCH
text: We conduct multi-task architecture learning experiments on four basic benchmarks, including ClsA, Seg, Video, and 3dDet. We use NCP to search models on single (e.g., Cls) or multiple tasks (e.g., Cls + Seg) and evaluate their performance on five datasets, including a new dataset ADE20K (Zhou et al., 2017) which is not included in NAS-Bench-MR. Quantitative comparisons are shown in Tab. 3.
Optimization for various objectives. (1) NCP can customize good architectures for every single task, e.g., the model searched on Cls achieves a top-1 accuracy of 85.56%, better than the models searched on the other three tasks. (2) NCP can learn architectures that achieve good performance on multiple tasks by inverting their predictors simultaneously, e.g., the model searched on Cls + Seg achieves a top1 accuracy of 86.05% and 77.16% mIoU. The joint optimization of all four tasks achieved moderate results on each task. (3) NCP achieves good accuracy-efficiency trade-offs and tunes a proper model size for each task automatically by using the FLOPs constraint.
Relationship among tasks. Two tasks may help each other if they are highly related, and vice versa. For example, jointly optimizing Seg + Cls (correlation is 0.639) improves the performance on both tasks with 17% fewer FLOPs than a single Cls model, while jointly optimizing Seg + Video (correlation is -0.003) hinders the performance of both tasks. When optimizing Seg + 3dDet (correlation is 0.409) simultaneously, the resulted network has better results in 3dDet but worse in Seg, which means accurate semantic information is useful for 3D detection, while the object-level localization may harm the performance of per-pixel classification. The above observations can be verified by Spearman’s rank correlation coefficients in Fig. 4. This way, we can decide which tasks are more suitable for joint optimization, such as CLs + Video, improving both tasks.
Generalizability. The searched models work well on the new dataset ADE20K (Zhou et al., 2017). An interesting observation is that the searched models under multi-task objectives, e.g., Seg + Cls, even conflicting tasks, e.g., Seg + Video, show better performance on ADE20K than those searching on a single task, demonstrating the generalizability of NCP. We also adapt our segmentation model (NCP-Net-L-continuous) in Tab. 2 to the COCO instance segmentation dataset. The model with only 360 GFLOPs (measured using 800x1280 resolution, Mask R-CNN head) achieves 47.2% bounding box AP and 42.3% mask AP on the detection and instance segmentation tasks, respectively. With fewer computational costs, our results outperform HRNet-W48 (46.1% for bbox AP and 41.0% for mask AP) by a large margin.

heading: 4.4 CROSS-TASK ARCHITECTURE TRANSFERRING
text: We experiment on architecture transferring across four different tasks. We first find optimal network codings for each task and then use it as the initial coding to search each of the other three tasks.
Comparison results can be found in Tab. 4. We see that: (1) Compared to searching a network from scratch for one task (e.g., 77.15% on segmentation) or finetuning architectures trained on other tasks (e.g., 75.42% by finetuning a classification model), our transferred architecture are always better (e.g., 78.27% by transferring a classification architecture to segmentation).
(2) The architecture searched on classification achieves better transferable ability on segmentation and video recognition, while the segmentation model is more suitable for transferring to 3D detection. It can be validated by Spearman’s rank correlation in Fig. 4.
(3) NCP is not sensitive to the initialized architecture coding. Models can be searched with different initial codings and achieve good performance.

heading: 4.5 INTRA-TASK ARCHITECTURE SEARCH
text: To show NCP ’s ability to customize architectures according to different data scales and the number of classes, we evaluate the searched architecture on the three subsets of the ImageNet dataset. From Tab. 5, we observe that: (1) NCP can find optimal architectures for each subset, which is essential for practical applications with different magnitudes of data. (2) We perform a joint architecture search for all three subsets. The searched architecture (NCP-Net-ABC) shows surprisingly great results on all three subsets with only 0.77G extra FLOPs and fewer parameters, demonstrating intra-task generalizability of NCP.

heading: 5 CONCLUSION
text: This work provides an initial study of learning task-transferable architectures in the network coding space. We propose an efficient NAS method, namely Network Coding Propagation (NCP), which optimizes the network code to achieve the target constraints with back-propagation on neural predictors. In NCP, the multi-task learning objective is transformed to gradient accumulation across multiple predictors, making NCP naturally applicable to various objectives, such as multi-task structure optimization, architecture transferring across tasks, and accuracy-efficiency trade-offs. To facilitate the research in designing versatile network architectures, we also build a comprehensive NAS benchmark (NAS-Bench-MR) upon a multi-resolution network space on many challenging datasets, enabling NAS methods to spot good architectures across multiple tasks, other than classification as the main focus of prior works. We hope this work and models can advance future NAS research.

heading: ACKNOWLEDGEMENTS
text: We sincerely appreciate all reviewers’ efforts and constructive suggestions in improving our paper. Ping Luo was supported by the General Research Fund of HK No.27208720 and HKU-TCL Joint Research Center for Artificial Intelligence. Zhiwu Lu was supported by National Natural Science Foundation of China (61976220).

heading: A DETAILS OF NAS-BENCH-MR
text: In this section, we provide details of the datasets and settings used to build our NAS-Bench-MR. We follow the common practice and realistic settings for training and evaluation in each task, making the scientific insights generated by our benchmark easier to generalize to real-world scenarios.
A.1 DATASETS AND SETTINGS
We randomly sample 2,500 structures from our network coding space, and train and evaluate these same architectures for each task and setting. Each architecture is represented by a 27-dimensional continuous-valued code. Tab. 7 shows the representation of each dimension of the code. For 9 different datasets and proxy tasks, we train a total of 22,500 models. See below for details.
ImageNet for Image Classification. The ILSVRC 2012 classification dataset (Deng et al., 2009) consists of 1,000 classes, with a number of 1.2 million training images and 50,000 validation images.
Considering the diversity of the number of classes and data scales in practical applications, and saving training time, we construct three subsets: ImageNet-50-1000, ImageNet-50-100, and ImageNet10-1000, where the first number indicates the number of classes and the second one denotes the number of images per class.
In this work, we adopt an SGD optimizer with momentum 0.9 and weight decay 1e-4. The input size is 224 × 224. The initial learning rate is set to 0.1 with a total batch size of 160 on 2 Tesla V100 GPUs for 100 epochs, and decays by cosine annealing with a minimum learning rate of 0. We adopt the basic data augmentation scheme to train the classification models, i.e., random resizing and cropping, and random horizontal flipping (flip ratio 0.5), and use single-crop for evaluation.
We report the top-1 and top-5 accuracies as the evaluation metric on all three benchmark datasets. We print the training log (top-1, top-5, loss, lr) every 20 iterations and evaluate the model every 5 epochs on the ImageNet validation set. We will release four checkpoints of 25, 50, 75, 100 epochs with the optimizer data and all the training logs for each model.
Cityscapes for Semantic Segmentation. The Cityscapes dataset (Cordts et al., 2016) contains highquality pixel-level annotations of 5000 images with size 1024 × 2048 (2975, 500, and 1525 for the training, validation, and test sets respectively) and about 20000 coarsely annotated training images. Following the evaluation protocol (Cordts et al., 2016), 19 semantic labels are used for evaluation without considering the void label.
To study the effect of different image resolutions in segmentation, we conduct two benchmarks with the input size of 512 × 1024 and 128 × 256, respectively. For the small-resolution setting, we pre-resize the Cityscapes dataset to 256× 512 before the data augmentation. In this work, we use an SGD optimizer with momentum 0.9 and weight decay 4e-5. The initial learning rate is set to 0.1 with a total batch size of 64 on 8 Tesla V100 GPUs for 25000 iterations (about 537 epochs). Follows the common practice in (Zhao et al., 2017; Wang et al., 2020a; Ding et al., 2020b), the learning rate and momentum follow the “poly” scheduler with power 0.9 and a minimum learning rate of 1e-4. We use basic data augmentation, i.e., random resizing and cropping, random horizontal flipping, and photometric distortion for training and single-crop testing with the test size of 1024× 2048 and 256× 512 respectively for two settings. We report the mean Intersection over Union (mIoU), mean (macro-averaged) Accuracy (mAcc), and overall (micro-averaged) Accuracy (aAcc) as the evaluation metrics. We print the training log (mAcc, loss, lr) every 50 iterations and evaluate the model every 5000 iterations on the Cityscapes validation set. We will release five checkpoints of 5000, 10000, 15000, 20000, 25000 iterations with the optimizer data and all the training logs for each model.
KITTI for 3D Object Detection. The KITTI 3D object detection dataset (Geiger et al., 2012) is widely used for monocular and LiDAR-based 3D detection. It consists of 7,481 training images and 7,518 test images as well as the corresponding point clouds and the calibration parameters, comprising a total of 80,256 2D-3D labeled objects with three object classes: Car, Pedestrian, and Cyclist. Each 3D ground truth box is assigned to one out of three difficulty classes (easy, moderate, hard) according to the occlusion and truncation levels of objects.
In this work, we follow the train-val split (Chen et al., 2015; Ding et al., 2020a), which contains 3,712 training and 3,769 validation images. The overall framework is based on Pointpillars (Lang et al., 2019). The input point points are projected into bird’s-eye view (BEV) feature maps by a voxel feature encoder (VFE). The projected BEV feature maps (496 × 432) are then used as the input of our 2D network for 3D/BEV detection.
Following (Lang et al., 2019), we set, pillar resolution: 0.16m, max number of pillars: 12000, and max number of points per pillar: 100. We apply the same data augmentation, i.e., random mirroring and flipping, global rotation and scaling, and global translation for 3D point clouds as in Pointpillar (Lang et al., 2019). We use the one-cycle scheduler with an initial learning rate of 2e-3, a minimum learning rate of 2e-4, and batch size 16 on 8 Tesla V100 GPUs for 80 epochs. We use an AdamW optimizer with momentum 0.9 and weight decay 1e-2. At inference time, axis-aligned non-maximum suppression (NMS) with an overlap threshold of 0.5 IoU is used for final selection.
We report standard average precision (AP) on each class as the evaluation metric. We will release five checkpoints of 76, 77, 78, 79, 80 epochs with the optimizer data and detailed evaluation results (Car/Pedestrian/Cyclist easy/moderate/hard 3D/BEV/Orientation detection AP) of each checkpoint for each model. Due to the unstable training of the KITTI dataset, we provide the last five checkpoints for researchers to tune hyperparameters from different criteria, such as the single model best performance, the average best performance, and the last epoch best performance.
HMDB51 for Video recognition. We train video recognition models on the HMDB51 dataset (Kuehne et al., 2011), consisting of 6,766 videos with 51 categories. We use the first training and validation split composing of 3570 and 1530 videos for evaluation, respectively. The video containing less than 64 frames is filtered, resulting in 2649 samples for training.
Considering the difficulty of learning temporal information, in addition to training from scratch, we also conduct training using models of ImageNet-50-1000 as pretraining. To model the temporal
information, we replace the last 2D convolutional layer before the final classification layer of HRNet and its counterparts with a 3D convolutional layer.
In this work, the input size is set to 112 × 112. We adopt an Adam optimizer with momentum 0.9 and weight decay 1e-5. The initial learning rate is set to 0.01 with a total batch size of 80 on 4 Tesla V100 GPUs for 100 epochs, and decays by cosine annealing with a minimum learning rate of 0. We adopt random resizing and cropping, random brightness 0.5, random contrast 0.5, random saturation 0.5, random hue 0.25, and random grayscale 0.3 as data augmentation.
We report the top-1 and top-5 accuracies as the evaluation metric on both two settings. We print the training log (top-1, top-5, loss, lr) every 10 iterations and evaluate the model every 10 epochs on the validation set. We will release the last checkpoint and all the training logs for each model.
A.2 ANALYSIS OF NAS-BENCH-MR
We summarize detailed statistics of our NAS-Bench-MR in Tab. 6. We also give the mean, std, and the validation L1 loss (%) of our neural predictors under the main evaluation metric of each setting.
From Spearman’s rank correlation of nine benchmarks in our main paper, we observe that:
(1) The four main tasks have different preferences for the architecture, and their correlation coefficients are between -0.003 (segmentation and video recognition) and 0.639 (segmentation and classification). This may be because segmentation can be seen as per-pixel image classification.
(2) Different settings in the same task also have different preferences for the architecture, and their correlation coefficients are between 0.3 (Cls-50-1000 and Cls-50-100) and 0.818 (Cls-50-100 and Cls-10c). From Fig. 5 (a) we also see that different training epochs result in a significant performance change. In this way, the proxy 10-epoch training setting (Radosavovic et al., 2020) may not generalize well to real-world scenarios.
(3) The correlation coefficient matrix is related to the model performance of multi-task joint optimization using NP. The higher the correlation coefficient of the two tasks, the greater the gain of joint optimization. If the correlation coefficient of the two tasks is too low, joint optimization may reduce the performance on both tasks.
(4) We show the architecture ranking under different metrics for different tasks in Fig. 5 (b) and (c). We noticed that the three metrics (i.e., mIoU, mAcc, aAcc) in segmentation are positively correlated, which means only one (e.g., mIoU) needs to be optimized to obtain a good model under all three metrics. However, in the 3D object detection task, different metrics are not necessarily related, which makes multi-objective optimization especially important.
B INTRA-TASK GENERALIZABILITY
In this section, we explore the effect of multiple proxy settings such as different input sizes (512 × 1024 and 128× 256), and pretraining strategies (pretraining from classification or training from the scratch for video recognition) in designing network architectures. We search architectures under single (e.g., 512× 1024) or multiple (e.g., both resolutions) to validate the effectiveness of NCP for different intra-task settings.
Tab. 8 and 9 show the intra-task cross-setting generalizability of our searched NCP-Net and some manually-designed networks, such as ResNet (He et al., 2016) and HRNet (Wang et al., 2020a), on the Cityscapes dataset and the HMDB51 dataset, respectively. We see that:
(1) Generally, the network searched on a specific setting performs the best under this setting but poor under other settings, which means that NCP can optimize and customize model structures for a specific setting. For example, Tab. 8 shows NCP customizing structures for different input resolutions on Cityscapes, which is essential for practical applications in real-world scenarios.
(2) Benefiting from the high correlation between different settings of the same task, joint optimization of multiple settings within the same task (for both segmentation and video recognition) usually has a positive effect, i.e., improving the performance on each setting, at the cost of the increasing model size (larger FLOPs). However, different tasks may not necessarily correlate, e.g.segmentation + video recognition. This can be verified by Spearman’s correlation, as discussed in our main paper.
(3) In Tab. 8, we apply the searched segmentation network to the ADE20K dataset (Zhou et al., 2017) to show the generalizability of the searched architectures. The network trained with a large input resolution (512×1024) has better performance than the network trained with a small resolution (128× 256). Moreover, the jointly searched architecture using both resolutions shows better generalizability than those two with a single objective. It also demonstrated that the network searched on our NAS-BenchMR has stronger transfer capability to real-world scenarios, compared to the previous benchmarks such as (Dong & Yang, 2020; Ying et al., 2019) that uses a small input size.
(4) For both the segmentation and the video recognition tasks, our joint searched networks outperform manually-designed networks, such as HRNet (Wang et al., 2020a) and ResNet (He et al., 2016), and achieve better generalizability to other settings and datasets, e.g., ADE20K.

heading: C NCP ON NAS-BENCH-201
text: In this section, we apply our NCP to the NAS-Bench-201 benchmark (Dong & Yang, 2020) to show its effectiveness. We first briefly introduce NAS-Bench-201 and then state the difference between our NAS-Bench-MR and NAS-Bench-201. Lastly, we detail the experiment.
C.1 NAS-BENCH-201
NAS-Bench-201 employs a DARTS-like (Liu et al., 2019b) search space including three stacks of cells, connected by a residual block (He et al., 2016). Each cell is stacked N = 5 times, with the number of output channels as 16, 32, and 64 for the first, second, and third stages, respectively. The intermediate residual block is the basic residual block with a stride of 2. There are 6 searching paths in the space of NAS-Bench-201, where each path contains 5 options: (1) zeroize, (2) skip connection, (3) 1-by-1 convolution, (4) 3-by-3 convolution, and (5) 3-by-3 average pooling layer, resulting in 15,625 different models in total.
NAS-Bench-201 train these models on the ImageNet-16 (with an input size of 16×16) and Cifar10 (with an input size of 32×32) datasets. Since the ImageNet dataset is closer to practical applications, we use the models in ImageNet16 as a comparison in the following section.
C.2 DIFFERENCES
There are some differences between the NAS-Bench-201 benchmark and our NAS-Bench-MR.
(1) NAS-Bench-201 and NAS-Bench-MR are of different magnitudes (15,625 v.s. 1023). (2) The number of channels/blocks/resolutions is fixed in NAS-Bench-201, while it is searchable in our search space. In this way, our search space is more fine-grained and suitable for customizing to tasks with different preferences (high- and low-level features, deeper and shallower networks, wider and narrower networks, etc.). (3) The architecture in NAS-Bench-201 is represented as a one-hot encoding (choose one of five operators). While in our NAS-Bench-MR, the continuous-valued code is more appealing to the learning of the neural predictor. (4) NAS-Bench-201 is built based on a single setting while NAS-Bench-MR contains multiple settings.
We then conduct experiments and show that despite the above many differences, NCP works well on NAS-Bench-201, demonstrating the generalization ability of NCP to other benchmarks.
C.3 EXPERIMENTS
In this work, we formulate each architecture in NAS-Bench-201 to a 6∗5 = 30-dimensional one-hot encoding. We then use our continuous network propagation strategy to traverse architectures in the space with higher-Acc and lower-FLOPs as optimization goals. Given an initial one-hot encoding, our NCP treats it as a continuous-valued coding and utilizes the argmax operation to obtain an edited one-hot coding after every several iterations.
In the experiment, we use the argmax operation after every 10 iterations, termed as a step. As shown in Fig. 6, our NCP finds the optimal structure from a low-Acc and high-FLOPs starting point with only 6 steps (60 iterations). The entire search process takes less than 10 seconds. Compared to other neural predictor-based methods such as (Wen et al., 2020; Luo et al., 2020) that need to predict the accuracy of a large number of architectures, NCP is more efficient.
From Fig. 6 we observe that from the second to fourth steps, the performance of the searched model vibrates. This may be because of the instability caused by one-hot encoding. NCP finds an optimal model with an accuracy of 46.8 and FLOPs of only 90.36 (the highest accuracy in NAS-Bench-201 is 47.33%), showing its effectiveness and generalizability.

heading: D NCP ON DIFFERENT SINGLE TASKS
text: We also use NCP to find the optimal architecture on the four basic vision tasks, i.e., image classification, semantic segmentation, 3D detection, and video recognition. Quantitative comparisons in Fig. 7 show that the architectures found by NCP outperform well-designed networks, such as HRNet (Wang et al., 2020a) and ResNet (He et al., 2016).
We visualize the searched architectures by NCP in Fig. 8 to show it can customize different representations for different tasks (objectives). We see that:
Algorithm 1 The network propagation process. 1. Learn a neural predictor FW (·) and fix it; 2. Initialize an architecture code e; 3. Set the target metrics, such as tacc and tflops; for each iteration do
4. Re-set new target metrics (optional); 5. Forward FW (e) and calculate the loss; 6. Back-propagate and calculate the gradient ∆e; 7. Select an optimal dimension l (optional); 8. update e based on the gradient ∆e;
end for 9. Round and decode e to obtain the final architecture.
(1) Segmentation requires the most high-level information in the last two branches of the last stage, and the video recognition model contains the least low-level semantics in the first two stages.
(2) The classification model contains more channels than other tasks, because the FLOPs constraint for classification is the weakest, i.e., classification costs fewer FLOPs than segmentation under the same architecture.
(3) The 3D detection model mainly utilizes the first two branches, which means it may rely more on high-resolution representations.

heading: E DETAILS OF NCP
text: In this section, we provide supplementary details and complexity analysis of our NCP. The algorithm of Network Coding Propagation (NCP) is shown in Algorithm 1. Code is available.
E.1 IMPLEMENTATION DETAILS
In this work, for each task, 2000 and 500 structures in the benchmark are used as the training set and validation set to train the neural predictor. The optimization goal is set to higher performance (main evaluation metric of each task) and lower FLOPs.
Intuitively, large models can achieve moderate performance on multiple tasks by stacking redundant structures. On the contrary, the lightweight model tends to pay more attention to task-specific structural design due to limited computing resources, making it more suitable for generalizability evaluation. To this end, we set λ = 0.5 to obtain lightweight models unless specified.
We employ a three-layer fully connected network with a dropout ratio of 0.5 as the neural predictor. The first layer is a shared layer for all metrics with a dimension of 256. The second layer is a separated layer for each metric with a dimension of 128. Then for each evaluation metric, a 128- by-1 fully-connected layer is used for final regression. During training, we use an Adam optimizer with a weight decay of 1e-7. The initial learning rate is set to 0.01 with batch size 128 on a single GPU for 200 epochs, and decays by the one cycle scheduler. The metric prediction is learned using the smoothL1 loss. In the network propagation process, unless specified, we use continuous propagation with an initial code of {b, n = 2; c, i, o = 64} and learning rate of 3 for 70 iterations in all experiments.
E.2 TIME COMPLEXITY
Tab. 10 shows the searching time of three predictor-based searching methods. NCP is the fastest as it traverses all dimensions of the code with only one back-propagation without the top-K ranking.
Suffering from the random noise in random search and the neural predictor, existing methods (Wen et al., 2020; Luo et al., 2020) often need to train the top-K models for final evaluation, which is costly (e.g., training a segmentation model on Cityscapes costs 7 hours on 8 Tesla V100 GPUs). NCP uses gradient directions as guidance to alleviate the randomness issue.
F VISUALIZATION AND ANALYSIS
F.1 ANALYSIS OF TASK TRANSFERRING
We visualize the network coding propagation process of our cross-task architecture transferring, e.g., an architecture transferring from the classification task to the segmentation task by using the optimal coding on classification as initialization and using the neural predictor trained on segmentation for optimization. The detailed transferring visualizations of every two of the four tasks are shown in
Fig. 9-12 with many interesting findings. For example, all networks in segmentation try to increase the number of channels of the 4th branch of stage 4, while the networks in classification try to decrease it; the networks in video recognition keep it at an intermediate value.
F.2 ANALYSIS OF SINGLE- AND MULTI-TASK SEARCHING
We visualize the searched architectures (Fig. 13-14) and the network propagation process of each architecture (Fig. 19-24) in Tab. 8-9.
Classification. Manually-designed classification models often use gradually decreasing resolution and gradually increasing the number of channels, because intuitively the learning of classification task requires low-resolution high-level semantic information. However, under different data scales and number of classes, the situation is different, as shown in Fig. 15-18. For example, by reducing the training samples from 1000 to 100 (from Fig. 15 to Fig. 16), the number of convolutional channels and the number of residual units in the 1st branch of stage 4 are increased during propagation, showing the high-resolution low-level information is important when the training samples are insufficient.
Segmentation. From Fig 19, Fig 20, and Fig 21 we observe the differences of the network editing process of segmentation models. For example, compared to the two models optimized at a single resolution, the model optimized at both input resolutions reduces the numbers of residual units of the first two stages and increases the number of blocks of stage 2, resulting in more fusion modules and fewer parallel units.
Segmentation and Video Recognition. From the visualization of found architecture codes in Fig. 13 and 14, we can find, it is not always that the larger the model, the better the performance. Different objectives result in different customized architecture codes. For example, segmentation tends to select architectures with fewer input channels, while video recognition architectures often have more output channels.
0 20 40 60
74
76
Predicted mIoU (%)
0 20 40 601
2 GFLOPs
0 20 40 60 1
2
3
4
Stage1: N_blocks Stage2: N_blocks Stage3: N_blocks Stage4: N_blocks
0 20 40 608
32
56
80
104
128
In-layers: N_channels0 In-layers: N_channels1 Out-layer: N_channels
0 20 40 60 8
32
56
80
104 128 Stage1: N_channels in branch1 Stage2: N_channels in branch1 Stage2: N_channels in branch2
0 20 40 608
32
56
80
104 128 Stage3: N_channels in branch1 Stage3: N_channels in branch2 Stage3: N_channels in branch3
0 20 40 60 8
32
56
80
104
128
Stage4: N_channels in branch1 Stage4: N_channels in branch2 Stage4: N_channels in branch3 Stage4: N_channels in branch4
0 20 40 60 Iterations
1
2
3
4
Stage1: N_units in branch1 Stage2: N_units in branch1 Stage2: N_units in branch2
0 20 40 60 Iterations
1
2
3
4 Stage3: N_units in branch1 Stage3: N_units in branch2 Stage3: N_units in branch3
0 20 40 60 Iterations
1
2
3
4
Cls Seg
Stage4: N_units in branch1 Stage4: N_units in branch2 Stage4: N_units in branch3 Stage4: N_units in branch4
0 20 40 60 25 26 27 Predicted Acc (%)
0 20 40 60 1
2 GFLOPs
0 20 40 601
2
3 4 Stage1: N_blocks Stage2: N_blocks Stage3: N_blocks Stage4: N_blocks
0 20 40 608
32
56
80
104
128
In-layers: N_channels0 In-layers: N_channels1 Out-layer: N_channels
0 20 40 60 8
32
56
80
104 128 Stage1: N_channels in branch1 Stage2: N_channels in branch1 Stage2: N_channels in branch2
0 20 40 608
32
56
80
104
128
Stage3: N_channels in branch1 Stage3: N_channels in branch2 Stage3: N_channels in branch3
0 20 40 60 8
32
56
80
104
128
Stage4: N_channels in branch1 Stage4: N_channels in branch2 Stage4: N_channels in branch3 Stage4: N_channels in branch4
0 20 40 60 Iterations
1
2
3
4
Stage1: N_units in branch1 Stage2: N_units in branch1 Stage2: N_units in branch2
0 20 40 60 Iterations
1
2
3

heading: 4 Stage3: N_units in branch1 Stage3: N_units in branch2 Stage3: N_units in branch3
text: 0 20 40 60 Iterations
1
2
3
4
Cls Video
Stage4: N_units in branch1 Stage4: N_units in branch2 Stage4: N_units in branch3 Stage4: N_units in branch4
0 20 40 60
77.0
77.5 Predicted Car-3D AP (%)
0 20 40 60 1.5 2.0 2.5 GFLOPs 0 20 40 601
2
3 4 Stage1: N_blocks Stage2: N_blocks Stage3: N_blocks Stage4: N_blocks
0 20 40 608
32
56
80
104
128
In-layers: N_channels0 In-layers: N_channels1 Out-layer: N_channels
0 20 40 608
32
56
80
104
128
Stage1: N_channels in branch1 Stage2: N_channels in branch1 Stage2: N_channels in branch2
0 20 40 608
32
56
80
104
128
Stage3: N_channels in branch1 Stage3: N_channels in branch2 Stage3: N_channels in branch3
0 20 40 60 8
32
56
80
104
128
Stage4: N_channels in branch1 Stage4: N_channels in branch2 Stage4: N_channels in branch3 Stage4: N_channels in branch4
0 20 40 60 Iterations
1
2
3
4
Stage1: N_units in branch1 Stage2: N_units in branch1 Stage2: N_units in branch2
0 20 40 60 Iterations
1
2
3
4 Stage3: N_units in branch1 Stage3: N_units in branch2 Stage3: N_units in branch3
0 20 40 60 Iterations
1
2
3
4
Cls 3dDet
Stage4: N_units in branch1 Stage4: N_units in branch2 Stage4: N_units in branch3 Stage4: N_units in branch4
In-layersStage1 Stage2 Stage3 Stage4 Out-layer 0
32 64 96
128
Ch an
ne l
0 1 2 3 4 Bl oc k & Co nv
Seg-512x1024
Channel Block ResUnit
In-layersStage1 Stage2 Stage3 Stage4 Out-layer 0
32 64 96
128
Ch an
ne l
0 1 2 3 4 Bl oc k & Co nv
Seg-128x512
Channel Block ResUnit
In-layersStage1 Stage2 Stage3 Stage4 Out-layer 0
32 64 96
128
Ch an
ne l
0 1 2 3 4 Bl oc k & Co nv
Seg-Both
Channel Block ResUnit
In-layersStage1 Stage2 Stage3 Stage4 Out-layer 0
32 64 96
128
Ch an
ne l
0 1 2 3 4 Bl oc k & Co nv
Video-Scratch
Channel Block ResUnit
In-layersStage1 Stage2 Stage3 Stage4 Out-layer 0
32 64 96
128
Ch an
ne l
0 1 2 3 4 Bl oc k & Co nv
Video-Pretrained
Channel Block ResUnit
In-layersStage1 Stage2 Stage3 Stage4 Out-layer 0
32 64 96
128
Ch an
ne l
0 1 2 3 4 Bl oc k & Co nv
Video-Both
Channel Block ResUnit
0 20 40 60
75.0
77.5
Predicted mIoU (%)
0 20 40 60 1
2 GFLOPs
0 20 40 60 1
2
3
4
Stage1: N_blocks Stage2: N_blocks Stage3: N_blocks Stage4: N_blocks
0 20 40 608
32
56
80
104 128 In-layers: N_channels0 In-layers: N_channels1 Out-layer: N_channels
0 20 40 60 8
32
56
80
104
128
Stage1: N_channels in branch1 Stage2: N_channels in branch1 Stage2: N_channels in branch2
0 20 40 608
32
56
80
104
128
Stage3: N_channels in branch1 Stage3: N_channels in branch2 Stage3: N_channels in branch3
0 20 40 60 8
32
56
80
104
128
Stage4: N_channels in branch1 Stage4: N_channels in branch2 Stage4: N_channels in branch3 Stage4: N_channels in branch4
0 20 40 60 Iterations
1
2
3
4 Stage1: N_units in branch1 Stage2: N_units in branch1 Stage2: N_units in branch2
0 20 40 60 Iterations
1
2
3
4
Stage3: N_units in branch1 Stage3: N_units in branch2 Stage3: N_units in branch3
0 20 40 60 Iterations
1
2
3
4
Stage4: N_units in branch1 Stage4: N_units in branch2 Stage4: N_units in branch3 Stage4: N_units in branch4
0 20 40 60
64
66
Predicted mIoU (%)
0 20 40 60
1
2 GFLOPs
0 20 40 601
2
3 4 Stage1: N_blocks Stage2: N_blocks Stage3: N_blocks Stage4: N_blocks
0 20 40 608
32
56
80
104
128
In-layers: N_channels0 In-layers: N_channels1 Out-layer: N_channels
0 20 40 608
32
56
80
104 128 Stage1: N_channels in branch1 Stage2: N_channels in branch1 Stage2: N_channels in branch2
0 20 40 608
32
56
80
104
128
Stage3: N_channels in branch1 Stage3: N_channels in branch2 Stage3: N_channels in branch3
0 20 40 608
32
56
80
104 128 Stage4: N_channels in branch1 Stage4: N_channels in branch2 Stage4: N_channels in branch3 Stage4: N_channels in branch4
0 20 40 60 Iterations
1
2
3
4
Stage1: N_units in branch1 Stage2: N_units in branch1 Stage2: N_units in branch2
0 20 40 60 Iterations
1
2
3
4
Stage3: N_units in branch1 Stage3: N_units in branch2 Stage3: N_units in branch3
0 20 40 60 Iterations
1
2
3 4 Stage4: N_units in branch1 Stage4: N_units in branch2 Stage4: N_units in branch3 Stage4: N_units in branch4

Review summaries:
This paper tackles neural architecture search (NAS), addressing the problem of finding architectures suitable for a multitude of vision-related tasks, ranging from object detection to semantic segmentation. To the best of my knowledge, this is the first attempt in NAS for computer vision models that explicitly design the search space that allows for a search of architectures that are broadly applicable to a variety of tasks that usually rely on the different granularity of feature representation.  This paper makes two important steps towards the automated search of such general, multi-purpose architectures: (1) to study this problem, the paper formalizes a multi-task NAS benchmark, covering recognition, detection, semantic segmentation, and activity recognition (video) datasets. This allows for assessing the generality of the sought architecture across vision tasks. (2) to tackle the architecture search in such a setting, this paper propose makes the following contributions. First, it augments the search space with multiple stages that extract feature maps at several resolutions, followed by a feature fusion step (the number of blocks and feat. map resolutions are governed by the network hyperparameters.). This seems to be one of the key features that help to find architectures suitable for tasks that differ as much as image classification and semantic segmentation.  Next, for assessing the performance in this large search space, this paper builds on predictor-based methods, where the idea is to train a network that predicts the performance (i.e., the learned predictor) of the sought-network based on the (encoded) hyperparameters, governing the network architecture. Similarly, the proposed neural coding propagation (NCP) directly encodes the hyperparameters in the network coding space — those can thus be directly updated via back-propagation of the learned predictor. A predictor is learned for each task independently, and this way, gradients can be accumulated separately across different tasks before jointly updating the “architecture codes”. 

This paper organized a multi-task NAS benchmark including 4 widely used datasets and more than 20,000 models. It also proposed a searching architecture named Network Coding Propagation (NCP) to effectively find the optical model for specific tasks. Experimental results indicate that the proposed model can be applied to inter-task, cross-task, and intra-tasks problems and the authors also showed the generalization capability to other benchmarks.