Learning Versatile Neural Architectures by Propagating Network Codes
Authors: Mingyu Ding
Yuqi Huo
Haoyu Lu
Linjie Yang
Zhe Wang
Zhiwu Lu
Jingdong Wang
Ping Luo
Venue: ICLR 2022 Poster (2022)
Keywords: Multitask NAS
Task-Transferable Architecture
Neural Predictor
NAS Benchmark

Abstract: This work explores how to design a single neural network capable of adapting to multiple heterogeneous vision tasks, such as image segmentation, 3D detection, and video recognition. This goal is challenging because both network architecture search (NAS) spaces and methods in different tasks are inconsistent. We solve this challenge from both sides. We first introduce a unified design space for multiple tasks and build a multitask NAS benchmark (NAS-Bench-MR) on many widely used datasets, including ImageNet, Cityscapes, KITTI, and HMDB51. We further propose Network Coding Propagation (NCP), which back-propagates gradients of neural predictors to directly update architecture codes along the desired gradient directions to solve various tasks. In this way, optimal architecture configurations can be found by NCP in our large search space in seconds.

Unlike prior arts of NAS that typically focus on a single task, NCP has several unique benefits. (1) NCP transforms architecture optimization from data-driven to architecture-driven, enabling joint search an architecture among multitasks with different data distributions. (2) NCP learns from network codes but not original data, enabling it to update the architecture efficiently across datasets. (3) In addition to our NAS-Bench-MR, NCP performs well on other NAS benchmarks, such as NAS-Bench-201. (4) Thorough studies of NCP on inter-, cross-, and intra-tasks highlight the importance of cross-task neural architecture design, i.e., multitask neural architectures and architecture transferring between different tasks. Code is available at https://github.com/dingmyu/NCP.

Published as a conference paper at ICLR 2022
LEARNING VERSATILE NEURAL ARCHITECTURES BY PROPAGATING NETWORK CODES
Mingyu Ding', Yuqi Huo?, Haoyu Lu", Linjie Yang", Zhe Wang', Wang""Zhive Zhiwu Lu², Jingdong Wang", Ping Luo' University of Hong Kong, Gaoling School of Artificial Intelligence, Renmin University of China, BByteDance Inc., Time Research, "Baidu
mingyading@hku. hk wanggingdonggouttooo..ccom com pluoêcs.hkee hku.hk
ABSTRACT
This work explores how to design single neural network capable of adapting multiple heterogeneous vision tasks, such as image segmentation, 3D detection, and video recognition. This goal is challenging because both network architecture search (NAS) spaces and methods in different tasks are inconsistent. We solve this challenge from both sides. We first introduce unified design space for multiple tasks and build multitask NAS benchmark INAS-Bench. used datasets, including ImageNet, Cityscapes. KITTI, and HMDB51. We further propose Network Coding Propagation (NCP), which back propagates gradients of neural predictors to directly update architecture codes along the desired gradient directions to solve various tasks. In this way, optimal architecture configurations can be found by NCP in our large search space seconds.
on many widely
Unlike prior arts of NAS that typically focus on a single task, NCP has several unique benefits. (1) NCP transforms architecture optimization from data-driven architecture-driven, enabling joint search an architecture among multitasks with different data distributions.
(2) NCP learns from network codes but not original data, enabling it to update the architecture efficiently across datasets. (3) In addi- tion to our NAS-Bench-MR. NCP performs well on other NAS benchmarks, such as
(4) Thorough studies of NCP on inter-, cross-, and intra-tasks highlight the importance of cross-task neural architecture design, i.e., multitask neural architectures and architecture transferring between different tasks. Code is available i thub. com/ di gmyu/MCP.
INTRODUCTION
Designing single neural network architecture that adapts multiple different tasks is challenging. This is because different tasks, such as image segmentation in Cityscapes (Cordts et al., 2016) and video recognition in HMDB51 (Kuehne et al., 201 1), have different data distributions and require different granularity of feature representations. For example, although the manually designed net- works ResNet (He al., 2016) and HRNet (Wang al., 2020a) work well certain tasks such as image classification on ImageNet, they deteriorate in the other tasks. Intuitively, manually designing single neural architecture that is applicable in all these tasks difficult.
Recently, neural architecture search (NAS) has achieved great success in searching network archi- tectures automatically. However, existing NAS methods (Wu et al., 2019; Liang et al., 2019; Liu et al., 2019b; Xie et al., 2018; Cai et al., 2020; Yu et al., 2020; Shaw et al., 2019b;a) typically search on a single task. Though works (Ding et al., 2021; Duan et al., 2021; Zamir et al., 2018) designed datasets that can be used for multiple tasks, they still search different architec-
NAS algorithms tures for different tasks, indicating that the costly searching procedure needs to be repeated many times. The problem of learning versatile neural architectures capable of adapting to multiple dif- ferent tasks remains unsolved, i.e., searching a multitask architecture or transferring architectures between different tasks. In principle, faces the challenge of task and dataset inconsistency.
Different tasks may require different granularity of feature representations, egg., the segmentation task requires more multi-scale features and low-level representations than classification. The key to solving task inconsistency is to design a unified architecture search space for multiple tasks. In contrast to most previous works that simply extend search space designed for image classification
Project page:
wwwp:oootrr.ooaaaa....o. thub.

---

Published as a conference paper at ICLR 2022
Cls Network
Cls Network
Cluster
Cluster
Seg Network
Seg Network
Furward
Backward
Cluster
Cluster
Cls Predictor
Seg Predictor
Cls Predictor
Grudient
Seg Predictor
Acd| FLOPs,
mlouf FLOPs|
Acc|
FLOPs
mloUf
FLOPs]
(a) Structure adaptation across tasks
(b) Joint propagation for multiple tasks
Figure 1: NCP optimizes and propagates the network code in an architecture coding space to achieve the target constraints with propaaropga on the neural predictors. (a) NCP searches for an op- timal structure on classification, then adapts it to segmentation through the segmentation predictor. (b) joint propagation for two tasks by accumulating gradients of two predictors.
for other tasks and build NAS benchmarks (Dong & Yang, 2020; Ying et al., 2019; Siems et al., 2020) small datasets (e.g., CIFARIO, ImageNet-16) with unrealistic settings, design a multi- resolution network space and build multi-task practical NAS benchmark (NAS-Bench-MBR on four challenging datasets including ImageNet-222 (Deng et al., 2009), Cityscapes (Cordts et al., 2016), KITTI (Geiger et al., 2012), and HMDB51 (Kuehne et al., 201 1). Inspired by HRNet (Wang al., 2020a), our network space various granularities of representations for different tasks, e.g., high-resolution features (Wang et al., 2020a) for segmentation while low-resolution ones for classification. NAS-Bench-Mr closes the gap between existing benchmarks and NAS in multi-task and real-world scenarios. It serves as an important contribution of this work to facilitate future cross-task NAS research.
multi-branch multi-resolution space that naturally contains
To solve the challenge of dataset inconsistency, in this work, we propose novel predictor- based NAS algorithm, termed Network Coding Propagation (NCP), for finding versatile and task- transferable architectures. NCP transforms data-oriented optimization into mnchitecture-oriente by learning to traverse the search space. works as follows. We formulate all network hyperparame- ters into coding space by representing each architecture hyper-parameter denotes there are 3 blocks and each block contains 2 residual blocks with a channel width of 64. We then learn neural predictors build the mapping between the network coding and its evaluation metrics (e.g., Acc, mIoU, FLOPs) for each task. By setting high-desired accuracy of each task and FLOPs as the target, we back propagate gradients of the learned predictor directly update val- ues of network codes to achieve the target. In this way, good architectures can be found in several forward-backward iterations in seconds, as shown in Fig. 1.
code, e.g., '3,2,64'
NCP has several appealing benefits: (1) NCP addresses the data mismatch problem in multi-task learning by learning from network coding but not original data. (2) NCP works in large spaces in seconds by back-propagating the neural predictor and traversing the search space along the gradient direction. (3) NCP can use multiple neural predictors for architecture transferring across tasks, as shown in Fig. ((a), it adapts an architecture to a task with only few iterations. (4) In NCP, the multi- task learning objective is transformed to gradient accumulation across multiple predictors, shown in Fig. (b), making NCP naturally applicable various conflicting objectives, such as multi-task structure optimization, architecture transferring across tasks, and accuracy-efficiency trade-off for specific computational budgets.
Our main contributions are three-fold. (1) We propose Network Coding Propagation (NCP), which pock-propagates the gradients of neural predictors to directly update architecture codes along desired gradient directions for various objectives. (2) We build under practical training settings for learning task transferabbl architectures. We believe it will facil- itate future NAS research, especially for multi-task NAS and architecture transferring across tasks. (3) Extensive studies on inter-, intra-, cross-task generalizability show the effectiveness NCP in finding versatile and transferable architectures among different even conflicting objectives and tasks.
on four challenging datasets
RELATED WORK
Neural Architecture Search Spaces and Benchmarks. Most existing NAS spaces (Jin et al,, 2019; Xu et al., 2019; Wu et al., 2019; Cai et al., 2019; Xie et al., 2018; Stamoulis et al., 2019;
2

---

Published as a conference paper at ICLR 2022
Table 1: Comparisons among five NAS benchmarks. Existing benchmarks are either built on small datasets for image classification, or trained with a single simplified setting, In contrast, our NAS- Bench-MR is built on four widely-used visual recognition tasks and various realistic settings. The architectures in our WAS-Bench-MM are trained following the common practices in real-world sce- narios, e.g. 512 x 1024 and 500 epochs on the City Scapes dataset (Cordts al., 2016). It takes
about 400,000 GPU hours to build our benchmark using Nvidia V100 GPUs.
Benchwarks
Datasets
Tisks
Stales
Epuchs
Input Sizes
Settings per Tosk
JMS-Bench-100 101 (Ying et al., 2019)
Cifar- 10
1
10a
4/12/36/108
32 x 32
different training epochs
NAS- Bench-20) 201 (Dong Yang. 2020)
ImageNet-16 16
10*
200
32 x 16
single setting
MAS-Bench-S0l M01 (Siems ot al., 2020)
Ciar--10
1
10ta
100
single setting
TransNAS-Beree ID1 (Duan el il,, 2021)
Taskonomy
7
10a
$30
256 x 236
single setting
NAS-Bench-MIR (Ours)
ImigeNet, Cityscapes,
KITTI, HMDB51
10a0
2000
512 x 1024
224
different image sizes, scale. number of classes, epuchs, pretraining
Mei et al., 2020; Guo et al., 2020; Dai et al., 2020) designed for image classification using either single-branch structure with a group of candidate operators in each layer or a repeated cell structure, e.g., Darts-based (Liu al., 2019b) and MobileNet-based (Sandler et al., 2018) search spaces. Based on these spaces, several NAS benchmarks (Ying et al., 2019; Dong & Yang, 2020; Siems et al., 2020; Duan al., 2021) have been proposed to pre-evaluate the architectures. However, the above search spaces and benchmarks are built either on proxy settings or small datasets, such as Cifar-10 and ImageNet-16 16 (16 16), which is less suitable for other tasks that rely on multi- scale information. For those tasks, some search space are explored in segmentation (Shaw et al., 2019a; Liu et al., 2019a; Nekrasov et al., 2019; Lin et al., 2020; Chen et al., 2018) and object detection (Chen et al., 2019; Ghiasi et al., 2019; Du et al., 2020; Wang et al., 2020b) by introducing feature aggregation heads (e.g., ASPP (Chen et al., 2017)) for multi-scale information. Nevertheless, the whole network is still organized in chain-like single branch manner, resulting in sub-optimal performance. Another relevant work to ours is WAS-Bench-NLE Klyuchnikov et al. (2020), which constructs a benchmark with 14k trained architectures in search space of recurrent neural networks on two language modeling datasets.
Compared to previous spaces, our multi-resolution search space, including searchable numbers of Cooluttoos/blockk/chhnnell, is naturally designed for multiple vision tasks as it contains various granularities of feature representations. Based on our search space, we build NAS-Bench-MR for various vision tasks, including classification, segmentation, 3D detection, and video recognition. Detailed comparisons of NAS benchmarks can be found in Tab.
Generally, NAS trains numerous candidate architectures from a search space and evaluates their performance to find the optimal architecture, which is costly. To reduce training costs, weight-sharing NAS methods (Liu et al., 2019b; Jin et al., 2019; Xu et al., 2019; Cai et al., 2019; Guo et al., 2020; Li & Talwalkar, 2020) are proposed to jointly train large number of candidate networks within super-network. Different searching strategies are employed within this framework such as reinforcement learning (Pham et al., 2018), importance factor learn- ing (Liu et al., 2019b; Cai et al., 2019; Stamoulis et al., 2019; Xu et al., 2019), path sampling (You et al., 2020; Guo et al., 2020; Xie et al., 2018), and channel pruning (Mei et al., 2020; Yu & Huang, 2019). However, recent analysis (Sciuto et al., 2020; Wang et al., 2021) shows that the magnitude of importance parameters in the weight-sharing NAS framework does not reflect the true ranking of the final architectures. Without weight-sharing, hyperparameter optimization methods (Tan & Le, 2019; Radosavovic et al., 2020; Baker et al., 2017; Wen et al., 2020; Lu et al., 2019; Luo et al., 2020; Chau et al., 2020; Yan et al., 2020) has shown its effectiveness by learning the relationship between network hyperparameters and their performance. For example, RegNet (Radosavovic et al., 2020) explains the widths and depths of good networks by a quantized linear function. Predictor-based methods (Wen et al., 2020; Luo et al., 2020; Chau et al., 2020; Yan et al., 2020; Luo et al., 2018) learn predictors, such as Gaussian process (Dai et al., 2019) and graph convolution networks (Wen et al., 2020), to predict the performance of all candidate models in the search space. A subset of models with high predicted accuracies is then trained for the final selection.
Neural Architecture Search Methods.
Our Network Coding Propagation (NCP) belongs to the predictor-based method, but is different from existing methods in: (1) NCP searches in a large search space in seconds without evaluating all candidate models by pack-propagating the neural predictor and traversing the search space along the gradient direction. (2) Benefiting from the gradient back proppagaton in network coding space, NCP is naturally applicable to various objectives across different tasks, such as multi-task structure
3

---

Published as a conference paper at ICLR 2022
4x,
8x
16xt
Out--layer
Ouy-layer
Furward
Increasse
Pack
Parallel
Staged
Sugo4
Fision
Mudall
Modall
Stage3
Coding
Suge3
Stage2
Decreuse Lases
Parallel
Stagez
Editing
Stagel
Plage
Fruge
Module
Fndion
Stagel
In-layers
Fixed
Modale
32
In-layer2 2x
o)
Prodi-spac
Namther
(b) Nasiber of residnal units (o)
by --
In-layeri
Example the Stage
Multi-resolution Architesture
27-dimensional Coding
Neural Predictor
Prediction
Target
Figure 2: Overview of our Network Coding Propagation (NCP) framework. Each architecture in our search space follows multi-resolution paradigm, where each network contains four stages, and each stage is composed of modularized blocks (a parallel module and fusion module). The example on the left shows the 3rd stage consists by modularized blocks with three branches, where each branch contains number of niz residual units with channel number of cj. {1, 2, 3}. We first learn a neural predictor between an architecture coding and its evaluation metrics such as accuracy and FLOPs. After that, we optimization objectives and pack-propagate the predictor to edit the network architecture along the gradient directions.
optimization, architecture transferring, and accuracy-efficien trade-offs. (3) Different from Luo et al. (2018); Baker et al. (2017) jointly train an encoder, performance predictor, and decoder to minimize the combination of performance prediction loss and structure reconstruction loss, learn and inverse the neural predictor directly in our coding space to make the gradient updating and network editing explicit and transparent.
3 METHODOLOGY
NCP aims to customize specific network hyperparameters for different optimization objectives, such single- and multi-task learning and recuracy-efficiere trade-offs, in an architecture coding space. An overview of NCP is shown in Fig. 2. In this section, we first discuss learning efficient models on single task with two strategies and then show that can be easily extended to multi-task scenarios. Lastly, we demonstrate our restutons coding space and NAS- Bench-MR.
3.1 NETWORK CODING PROPAGATION
For each task, we learn neural predictor F(.) that projects architecture code to its ground truth evaluation metries, such accuracy Jace and FL.OPs Gnops- Given initialized and normalized coding e, the predicted metrics and the loss learn the neural predictor are represented by:
Prees Prope: Prasksppe Fw (e),
Ipredicse
1.2(puc, Dwce) + L.2(phups: Blops) +...
(2)
where W is the weight of the neural predictor, which is fixed after learning. Pace: PRops and Pusk-spec denote the predicted accuracy, FLOPs, and task-specific metrics (if any), L2 is the L2 norm. Note that although the FLOPs can be directly calculated from network codes, learn enable the backward gradient upon FLOPs constraints for accuracy-efficiens trade-offs.
NCP edits the network coding by back-propagating the learned neural predictor W to maxi- mize/minimize the evaluation metrics (e.g.. high accuracy and low FLOPs) along the gradient direc- tions. According to the number of dimensions to edited, two strategies are proposed to propagate propagation, where the first one
the network coding: continuous propagation and winner-takes- uses all the dimensions of the gradient to update e, while the latter uses only the one dimension with the largest gradient. Fig. 3 visualizes the network update process of our two strategies.
Continuous Propagation. goal and use gradient descending automatically update the network coding Taking accuracy and FLOPs as an example, we set the target accuracy face and target FLOPs fopp for e and calculate the loss between the prediction and the target by:
continuous propagation, we set target metrics as an optimization
Ipproaaaation L1 (Pace, face) AL1 AL1 (Props= (oops)
where L1 denotes the smooth L1 loss, À is the coefficient to balance the trade-off between accuracy and efficiency. The loss is propagated back through the fixed neural predictor Fw(+), and the gradi- the coding is calculated de
0L (use Ve in following for simplicity). We update the coding
4

---

Published as a conference paper at ICLR 2022
25
Start: 1. and
Friecad not
09.5%
Srper "Losnur saer
Figure 3: Visualization of the network propagation process of our two strategies (left: continuous; right: winner-takes-all) for segmentation. We group 27 dimensions into eight subfigures.
iteratively by e Ve. In this way, NCP can search for the most efficient model upon a lower-bound accuracy and the most accurate model upon upper-bound FLOPs. In practice, we set to find a model with good efficiency- -acuuaccy trade-off.
tace = Puce + and fRpps Piops
The propagation is stopped once the prediction reaches the target or after certain number of itera- tions (e.g., 70). Finally, we round (the number of channels is a multiple of 8) and decode the coding e to obtain the architecture optimized for the specific goal.
Winner-Takes--ll Propagation. Editing different dimensions of the coding often have different degrees of impact on the network. For example, modifying the number of network blocks has greater impact than the number of channels, iee., block costs more computational resources than several channels. In light of this, we select one dimension of the coding with the best accuracy- efficiency for propagation in each iteration, termed as winner-takes-all propagation.
We create a lookup table T()) to obtain the corresponding FLOOPs of any network coding. We then calculate the relative increase Ar in FLOPs as each dimension the coding grows:
ef = el + k
(4)
Ar: T(e') - T(e)
where 1 denotes the index of the coding k = 8 if es represents the number of convolutional channels, otherwise 1. We then use Ar as a normalization term of backward gradients Ve and select the optimal dimension / of the normalized gradients Ve/Ar:
arg mag Atggax(Ve/Ar) Ar) Vrgmin(Ve/AaV)
if max (eex\) >
(5)
max VerAAr <
if max Ve/Ar) O, modifying the value of e may improve accuracy and decrease FLOPs simul- taneously. While if max(Ve/2a) accuracy under the unit FLOPs consumption. The coding e is then updated as follows:
O, we choose the dimension with the highest improvement in
if max (e/AAr) 0 if (ax((Ve/A)) <
(6)
e
Cross-task Learning. Our NCP provides a natural way for cross-task learning by transform- ing different objectives across tasks into gradient accumulation operations directly in the architec- ture coding space, facilitating joint architecture search for multiple tasks and cross-task architecture
transferring, as shown in Fig. 1. For example, NCP enables joint search for classification and seg- mentation by accumulating the gradients el and " from the two predictors:
Ve Ve cls + Ve seg
Note that different weights coefficients can be used to learn multi-task architectures with task pref- erences. In this work, we set equal weight coefficients for these tasks.
3.2 NAS-BENCH-ME
Calti-RRsooltionn Search Space. Inspired by HRNet (Wang et al., 2020a; Ding et al., 2021), we design multi-branch multi-resolution search space that contains four branches and maintains both
5

---

Published as a conference paper at ICLR 2022
high high-resolutton and low-resolution representations throughout the network. As shown in Fig. 2, after two convolutional layers decreasing the feature resolution to 1/4 (4x) of the image size, we start with this high-resolution feature branch and gradually add lower-resolution branches with feature fusions, and connect the multi-resolution branches in parallel. Multi-resolution features are resized and concatenated for the final classification/regressoon without any additional heads.
The design space follows the following key characteristics. (1) Parallel modules: extend each branch of the corresponding resolutions in parallel using residual units (He et al., 2016); (2) Fusion mod- ules: repeatedly fuse the information across multiple resolutions and generate the new resolution (only between two stages). For each output resolution, the corresponding input features of its neigh- boring resolutions are gathered by residual units (He et al., 2016) followed by element- -iiee addition (upsampling used for the low-to-high resolution feature transformation), e.g., the 8x output fea- ture contains information of xx, 1x88x, 8x, and 16x input features; (3) A modularized block is then formed by a fusion module and a parallel module.
Coding. We leverage the hyperparameter space of our multi-resolution network by projecting the depth and width values of all branches of each stage into a continuous-valued coding space. For- mally, two 3 x 3 stride 2 convolutional layers with i number of channels are used to obtain the 4x high-resolution features at the beginning of the network. We then divide the network into four stages with where the fusion module of the first block is used for transition between two stages (e.g., from 2 branches to 3 branches). For the parallel module of the block, set the number of residual units and the number of convolutional channels to n2 = [m],..., .1]] and ca [c!...., ca] respectively, where s denotes the s-th stage containing s branches. Since the fusion module always exists be- tween two parallel modules, once the hyperparameters of the parallel modules are determined, the corresponding number of channels of fusion modules will be set automatically to connect the two parallel modules. The s-th stage comprising a plurality of same modular blocks is represented by:
{1, 2, 3, 4) branches, respectively. Each stage contains b, modularized blocks,
b.y n ....... Tig
(8)
where the four stages are represented by 3, 5, 7, and 9-dimensional codes, respectively. By the end of the 4-th stage of the network, we use a x 1 stride convolutional layer with a channel number of on to fuse the four resized feature representations with different resolutions. In general, the entire architecture can be represented by a 27-dimensional continuous- valued code:
i1, 12, bi, n2, Cr.-..,,u b4, 124
Ca.
(9)
We then build our coding space by sampling and assigning values to the codes e. For each struc- ture, we randomly choose b,n E 1, 3, 4, and i, o E (8, 16, 24,.... 128}. resulting in a 27- dimensional coding. In this way, a fine-grained multi-resolution space that enables searching struc- tures for various tasks with different preferences on the granularity of features is constructed.
Dataset. We build our DAS-Bench-MR on four carefully selected visual recognition tasks that require different granularity of features: image classification on ImageNet (Deng et al., 2009), mantic segmentation on Cityscapes (Cordts et al., 2016), 3D object detection on KITTI (Geiger al., 2012), and video recognition on HMDB51 (Kuehne et al., 2011).
For classification, considering the diversity of the number of classes and data scales in practical applica- tions, we construct three subsets: ImageNer-50010000 (CIs-A), ImageNet-500100 (CIs-B), and ImageNet-10- 1000 (CIs-C), where the first number indicates the number of classes and the second one denotes the num- ber of images per class. For segmentation, we fol- low HRNet (Wang et al., 2020a) by upsampling all branches to the high-resolution one. For 3D detection, we follow PointPillars (Lang et al., 2019) that first converts 3D point points into bird s-eye view (BEV) featuremaps and then unitizes a 2D network. For video recognition, we replace the last 2D convolu- tional layer before the final classification layer of our network with a 3D convolutional layer.
066- 1.000
2.00
3.338
3:537
1.500
E.400
E4424
6224
22:22
0se 0263
2,000
6204
8200
0.52
0.40
2.355
0.224
1000
66-11
600.0
0.643
3.58
9.20
133
2000
6333
01224
0.244
3.602
2.00
0730
Wideo
0255
3.605
3.315
2.038
8.000
6224
0114
6:300
1.000
Figure 4: Spearman's rank correlation of 9 subtasks in
6

---

Published as a conference paper at ICLR 2022
Table 2: Performance comparison (%) on the Cityscapes validation set. All hyperparameter op- timization methods are searched using our NAS-Bench-MR, All models are trained from scratch. FLOPs is measured using an input size of 512 x 1024. À - 0.7, 0.3, 0.1 for 'S', 'M', and 'L'
Model
Type
Params
FLOPs
mloU
mAce aAcc
ResNet34-PSP (Zhao 2017)
manually-designed
23.05M
193.12G
76.17
82.66
95.99
ResNet50-PSP (Zhao 2017)
manually-designee
48.98M
356.91G
76.49
83.18
95.98
WII (Wang et al., 2020a)
manually-designed
9.64M
37.01G
77.73
85.61
96.20
HRNet-W332 (Wang et al., 2020a)
manually-designed
29,55M
90.556
79.28
86.48
96,31
SqueezeNAS (Shaw
2019a)
NAS in SquezzeNAS space
3.00M
32.73G
75.19
Auto-DeepLab (Liu et al., 2019a)
NAS in DeepLab space
10.15M
289.78G
79.74
NetAdapi (Yang et al., 2018)
hyperparameter optimization
10.83M
42.49G
78.02
85.68
96.24
Random Search (Bergstra & Bengio, 2012)
hyperparameter optimization
16.27M
130.50G
77.66
85.35
96.13
Neural Predictor (Wen al., 2020)
hyperparameter optimization
33.49M
182.60G
78.89
86.46
96.21
NCP-Net-M-winnd takes all
hyperparameter optimization
7.18M
32.88G
77.91
$5.69
96.23
hyperparameter optimization
8.39M
36.25G
78.36
86,24
96.29
NCP-Net-S-continoous NCP--ee=L-ccomtiuuuu
hyperparameter optimization
6.04M
20.07G
76.98
85,14
96.07
hyperaarameter optimization
37.70M
179.98G
80.05
87.12
96.49
We also explore several proxy training settings, such as 10-epoch training used in RegNet (Ra- dosavovic et al., 2020) (CIs-10c), reduced input resolution by four times (Seg-4x), and pretrain- ing instead of training from scratch (Video-p). In summary, we randomly sample 2,500 structures from our search space, train and evaluate these same architectures for those above nine different tasks/settings, resulting 22,500 trained models. We believe these well-designed and fully-trained models serve an important contribution of this work to facilitate future NAS research.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Taking a three-layer fully connected network as the neural predictor, we apply NCP to NAS- MR and obtain NCP-Net. To train the neural predictor, 2000 and 500 structures in the benchmark are used as the training and validation sets for each task. Unless specified, we use continuous propagation with an initial code of {6,n n 2; c, i, = 64) and À 0.5 for 70 iterations in all experiments. The optimization goal is set to higher performance and lower FLOPs (lace = Pacc + 1, Pilops 1). More experiments and details can be found in Appendix.
tnops
4.2 BASELINE EVALUATIONS
We make comparisons among different search strategies on our multi-resolltion space and segmen- tation benchmark in Tab. 2. For Random Search (Bergstra & Bengio, 2012) and Neural Predic- tor (Wen et al., 2020), we sample 100 and 10,000 network codes respectively and report the highest result in 10 models. For NetAdapt (Yang al., 2018), we traverse each dimension of the initial codes (increasing or decreasing unit value) and adapt the codes by greedy strategy.
-desigged networks, e.g., ResNet (He al., 2016) and HRNet (Wang et al., 2020a), hyperparameter optimization methods (Bergstra & Bengio, 2012: Wen al., 2020; Yang et al., 2018), and even well-designed sharing NAS (Liu al., 2019a; Shaw et al., 2019a). (2) Compared to other hyperparameter optimization methods, NCP is not sensitive to the randomness and noise in model selection by adopting an approximate gradient direction. (3) NCP is much more efficient as it evaluates all dimensions of the codes with only one back- proppppapggttoon without top-K K ranking. (4) NCP is able to search for optimal architectures under different computing budgets by simply setting the coefficient X.
Effectiveness of NCP. (1) NCP outperforms
Effectiveness of our search space. (1) Random Search (Bergstra & Bengio, 2012) works well in coding space, showing the effectiveness of our muutiirrsoolution space. (2) Models in our search space consist of only 3 x 3 convolution in the basic residual unit (He et al., 2016) without complex operators, making it applicable to various computing platforms, such as CPU, GPU, and FPGA.
Search strategies of NCP. Intuitively, the winner-take strategy is more like a greedy algorithm. It selects one coding dimension with the best accuracy-efficient trade-off for propagation, resulting
7

---

Published as a conference paper at ICLR 2022
Table 3: Performance (%) on five datasets of the models searched on different optimization objec- tives (including single- and multi-task optimization) using our NCP, e.g. "Cls + Seg" denotes the model searched using "CIs-A" and "Seg" benchmarks, "Four Tasks" denotes the model is propa- gated by using the predictors trained on the all four benchmarks (CIs-A, Seg, 3dDet, Video). Note that in addition to cross-task evaluation, we also show the generalizability of NAS by applying the searched model to new dataset, ADE20K, which is not used to train neural predictors. For clearer comparisons, the FLOPs of all networks is measured using input size 128 x 128 under the segmen- tation task. The top-2 results are highlighted in bold.
Classification ImageNet-50--1 500-N-00--00000
Segmentation Cityscupes mloU mAce aAce
Video HMDBSI
3D Object Detection
Segmentation ADE20K
Method
Params FL.PPs
KITTI
lopl
top5
topl
top5
car-3D
car Ba--BVV
ped.3D
ped BEV
mloli
mAce
aAcc
Cls
5.35M
2.05G
88,56
95.36
75.65
84,11
95.23
25.53
59.69
73.60
83,92
35.20
41.39
32.74
42.77
76.99
Seg
7.91M
0,903
82.52
94.00
77.15
84.73
96.01
14.68
44.92
69:99
21.83
31.21
33.32
42.97
77.41
Video
2.56M
0.74G
81.88
93.84
71.09
79.91
95.
28.47
62.18
68.60
81.68
19:02
27.82
25.84
34.07
73.88
3dibet
2.80M
1,216
82.60
94.64
69.43
78.48
95.05
19:84
53.82
75.44
87.37
40.42
48.69
23.28
30.64
72.29
Cls Seg
8.29M
1.70G
86605
95.35
77.36
84.95
96.18
22.95
57.74
74.39
84.72
27.18
35.70
35.58
46.02
78.19
Cl Video
4.32M
1.603
85.64
95.28
72.98
81.51
95.54
27.83
60.84
71.05
84.36
23.84
30.27
30.43
41.29
75.78
Cls 3d12r
4.46M
1.81G
85.76
95.27
72.18
80.81
23
58.40
75,47
87.63
40.61
31.98
42.00
76.67
Seg + Vidco
4.46M
1.163
83.84
94.20
75.64
84.16
95.88
28.46
60.77
71.24
83
34.44
42.00
34.23
41.15
77.28
Seg 3dDet
4.30M
0.91G
83.08
94.57
75.60
83.74
95.85
21.01
56.46
75.54
86.67
41.45
46.32
34.65
46.38
77.71
Video 3dDer
2.37M
0.85G
83.27
06
70.32
79.33
95.
28.20
61.18
37.82
44.17
26,42
36.34
73.69
Four Tasks
4.10M
1.44G
884.36
95.04
75,49
83.86
95.83
25 09.09
58.36
71.90
84,29
27.27
34.34
34.53
45,36
77.45
in efficient models. The continuous strategy updates all dimensions of the coding purely based on gradients. Thus, it reaches better accuracy, though the generated model is not the most efficient. The experiment verifies the above intuition. Unless specified, we use the continuous strategy for all experiments. Fig. shows the code editing process of two strategies.
4.3 INTER-TASK ARCHITECTURE SEARCH
We conduct multi-task architecture learning experiments on four basic benchmarks, including Cls- A, Seg, Video, and 3dDet. We use NCP to search models on single (e.g., Cls) or multiple tasks (e.g., Cls + Seg) and evaluate their performance on five datasets, including new dataset ADE20K (Zhou et al., 2017) which is not included in
Quantitative comparisons are shown in Tab. 3.
Optimization for various objectives. (1) NCP can customize good architectures for every single task, e.g., the model searched on Cls achieves a top-1 accuracy of 85.56% better than the models searched on the other three tasks. (2) NCP can learn architectures that achieve good performance on multiple tasks by inverting their predictors simultaneously, e.g., the model searched on Cls + Seg achieves a topl accuracy of 86.05% and 77.16% mloU. The joint optimization of all four tasks achieved moderate results each task. (3) NCP achieves good accuracy-efficiies tunes a proper model size for each task automatically by using the FLOPs constraint.
trade-offs and
Relationship among tasks. Two tasks may help each other if they are highly related, and vice versa. For example, jointly optimizing Seg + Cls (correlation is 0.639) improves the performance on both tasks with 17% fewer FLOPPs than single Cls model, while jointly optimizing Seg Video (correlation is -0.003) hinders the performance of both tasks. When optimizing Seg + 3dDet (correlation 0.409) simultaneously, the resulted network has better results in 3dDet but worse in Seg, which means accurate semantic information is useful for 3D detection, while the object-level localization may harm the performance of per-pixel classification. The above observations can be verified by Spearman's 's rank correlation coefficients in Fig. 4. This way, we can decide which tasks are more suitable for joint optimization, such as CLs + Video, improving both tasks.
The searched models work well on the new dataset ADE20K (Zhou et al., 2017). An interesting observation is that the searched models under multi-task objectives, e.g., Seg Cls, even conflicting tasks, e.g., Seg + Video, show better performance on ADE20K than those searching on a single task, demonstrating the generalizability of NCP. We also adapt our segmentation model in Tab. to the COCO instance segmentation dataset. The model with only 360 GFLOPs (measured using 800x 1080 resolution, Mask R-CNN head) achieves 47.2% bounding box AP and 42.3% mask AP on the detection and instance segmentation tasks, respectively. With fewer computational costs, our results outperform HRNet- W48 (46.1% 1% for bbox AP and 41.0% for mask AP) by large margin.
Generalizability.
8

---

Published as a conference paper at ICLR 2022
Table 4: Our architecture transfer results between four different tasks. We first find an optimal architecture coding for each task and then use it as the initial coding to search other three tasks. "-FF" and " "-T" denote the architecture finetuning and transferring results, respectively.
Classification
Segmentation
Video Recognition
3D Object Derection
Method
FLOP=F
ImageNat-50-1000
Cityscapes
HMDB51
KITTI
lopl-F|upp1T FLOPs-
mloU-F ImloU-T FLOPs-T
upl-F|jpp-- TLOPs-T
3D-F
can-3D-T FLOPs-
Classification
2.05G
78.27
1.00G
25.53
0.76G
73.60
75.59
1.30G
Segmentation
0.90G
82.52
85.84
1.54G
77.15
14.68
28.44
0.75G
69.99
76.17
1.21G
Video Recognition
0.746
81888
86,03
1.86
71.09
77.56
0356
28.47
68 60
75.78
1,17G
Object Detection
1.21G
85.63
69.43
77.25
1.4G
19.84
28.12
0.78G
73.44
4.4 CROSS-TASK ARCHITECTURE TRANSFERRING
We experiment on architecture transferring across four different tasks. We first find optimal network codings for each task and then use it as the initial coding to search each of the other three tasks.
Comparison results can be found in Tab. We see that: (1) Compared to searching network from scratch for one task (e.g., 77.15% segmentation) or finetuning architectures trained on other tasks (e.g., 75.42% by finetuning a classification model), our transferred architecture are always better (e.g., 78.27% by transferring a classification architecture segmentation).
(2) The architecture searched on classification achieves better transferable ability on segmentation and video recognition, while the segmentation model is more suitable for transferring 3D detec- tion. It can be validated by Spearman's 's rank correlation in Fig. 4.
(3) NCP is not sensitive to the initialized architecture coding. Models can be searched with different initial codings and achieve good performance.
4.5 INTRA-TASK ARCHITECTURE SEARCII
To show NCP 's ability to customize architectures according to different data scales and the number of classes, we evaluate the searched architecture on the three subsets the ImageNet dataset. From Tab. 5, we observe that: (1) NCP can find optimal architectures for each subset, which is essential for practical applications with different magnitudes data. (2) We perform a joint architecture search for all three subsets. The searched architecture (NCP-Net-ABC) shows surprisingly great results on all three subsets with only 0.77G extra FLOPs and fewer parameters, demonstrating intra-task generalizability of NCP.
Table Single-crop top-1 error rates (%) on the ImageNet validation set. 'ImNet-A', 'ImNet-B', 'InNet- C' denote the predictor is trained on benchmarks of ImageNet-50- 1000, ImageNet-50-10m ImageNet- 10-1000, respectively. FLOPss is measured using input size 224 x 224. 0.7. Top-2 results are high- lighted bold.
Model
Params
FL.PPs
ImNet-A
ImNet-B
ImNet-C:
ResNet50
23.61M
4.12G
83.76
50.16
86.00
ResNet101
42.60M
7,85G
84.32
51.92
86.00
HRNet-Wa W32
39.29M
8.99G
84.00
52.00
83.80
HRNet-W48
75.52M
17.36G
84.52
52.88
86.60
NCP-Net-A
4.39M
5.95G
85,80
56.04
86.80
NCP-Net-B
3.33M
3.55G
82,76
56.40
85.60
NCP-Net-C
4.06M
5.51G
84.60
55.60
87.80
NCP-Net-ABC
4.28M
6.72G
85.12
58.12
88.20
CONCLUSION
This work provides an initial study of learning teessfranseeebll architectures the network coding space. We propose an efficient NAS method, namely Network Coding Propagation (NCP), which optimizes the network code achieve the target constraints with back- pack-propagation on neural predic- tors. In NCP, the multi-task learning objective is transformed to gradient accumulation across mul- tiple predictors, making NCP naturally applicable to various objectives, such as multi-task structure optimization, architecture transferring across tasks, and accuracy-efficieet tate the research in designing versatile network architectures, we also build a comprehensive NAS benchmark (NAS-Bench-Min upon multi-resolution network space many challenging datasets, enabling NAS methods to spot good architectures across multiple tasks, other than classification as the main focus of prior works. We hope this work and models can advance future NAS research.
trade-offs. To facili-

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We sincerely appreciate all reviewers' efforts and constructive suggestions in improving our paper. Ping Luo was supported by the General Research Fund of HK No.27208720 and HKU-TCL Joint Research Center for Artificial Intelligence. Zhiwu Lu was supported by National Natural Science Foundation of China (61976220).
REFERENCES
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. arXiv preprint arXiv:1705. 2017. 3,4
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012. 7
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. ICLR, 2019. 2.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In ICLR, 2020.
Thomas Chau, Lukasz Dudziak, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D)
Lane. Brp-nas: Prediction-based nas using gens. arXiv preprint arXiv:2007.08666 2020. 3
Liang-Chiel Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE TPAMI, 40(4))834-848, 2017. 3
Liang-Cue Chigh Chen, Maxwell Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, and Jon Shlens. Searching for efficient multi-scale architectures for dense image prediction. In NeurlPS, pp. 8699 87 10, 2018. 3
Xiaozhi Chen, Kaustav Kundu, Yukun Zhu, Andrew G Berneshawi, Huimin Ma, Sanja Fidler, and Raquel Urtasun. 3d object proposals for accurate object class detection. In NeurIPS, pp. 424-432, 2015. 15
Yukang Chen, Tong Yang. Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun. Detnas: Neural architccture search on object detection. arXiv preprint arXiv:1903.10977 10979, 12):4-1, 2019.
3
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pp. 3213-3223, 2016. 1, 6, 14
Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang. Marat Dukhan, Yunqing Hu, Yiming Wu, Yangqing Jia, et al. Chamnet: Towards efficient network design through platform-aware model adaptation. In CVPR, pp. 11398-11407, 2019,
Xiyang Dai, Dongdong Chen, Mengchen Liu, Yinpeng Chen, and Lu Yuan. Da-nas: Data adapted pruning for efficient neural architecture search. arXiv preprint arXiv:2003. /2563, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, Ieee, 2009. 2. 6, 14
Mingyu Ding, Yugi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo. Learning depth-guided convolutions for monocular 3d object detection. In CVPR, 2020a. 15
Mingyu Ding, Zhe Wang, Bolei Zhou, Jianping Shi, Zhiwu Lu, and Ping Luo. Every frame counts: joint learning of video segmentation and optical flow. In AAAI, volume 34, pp. 10713-10720, 2020b. 15
Mingyu Ding, Xiaochen Lian, Linjie Yang, Peng Wang. Xiaojie Jin, Zhiwu Lu, and Ping Luo. Hr- nas: Searching efficient righ-resolution neural architectures with lightweight transformers. In CVPR, pp. 2982-2992, 2021. 1,5
10

---

Published as a conference paper at ICLR 2022
Xuanyi Dong and Yi Yang. Nas-bench-2011 Extending the scope of reproducible neural architecture search. In ICLR, 2020. 2,3, 18
Xianzhi Du, Tsung-Yi Yi Lin, Pengchong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoc Le, and Xiaodan Song. Spinenet: Learning scale-permuted backbone for recognition and localization. In CVPR. pp. 1 11592-12 1601, 2020. 3
Yawen Duan, Xin Chen, Hang Xu, Zewei Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Li. Transnas-bench- 101: Improving transferability and generalizability cross-task neural architec- ture search. In CVPR, pp. 5251-5260, 2021. 1,
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In CVPR, pp. 3354-3361, 2012. 2, 6, 15
Golnaz Ghiasi, Tsung-Y. Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid archi- tecture for object detection. CVPR, pp- 7036-7045, 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun.
Single path shot neural architecture search with uniform sampling. In ECCV, 2020. 3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016. 1, 6, 7, 17, 18, 19
Xiaojie Jin, Jiang Wang, Joshua Slocum, Ming-Hsuan Yang, Shengyang Dai, Shuicheng Yan, and Jiashi Feng. Rc-darts: Resource constrained differentiable architecture search. arXiv preprint 19Xiv:191..128/1 2019. 2.3
Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, and Evgeny Burnaev. Was-bench-nlp: neural architecture search benchmark for natural language processing. arXiv preprini arXiv:2006.071 2020.
Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: A large video database for human motion recognition. In ICCV, pp. 2556-2563, 2011. 1, 6, 15
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang. and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In CVPR, pp. 12697-12705, 2019. 6, 15
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in Artificial Intelligence, pp. 367-377. PMLR, 2020.
Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping. arXiv preprint www.iv:109.066000 2019.
Peiwen Lin, Peng Sun, Guangliang Cheng, Sirui Xie, Xi Li, and Jianping Shi. Graph-guided archi- tecture search for real-time semantic segmentation. In CVPR, pp. 4203-4212, 2020. 3
Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In CVPR, pp. 82 92, 2019a. 3,
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In ICLR, 2019b. 3, 18
Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. Nsga-net: neural architecture search using multi-objective genetic algorithm. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 419-427, 2019. 3
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimization. In Advances in neural information processing systems, pp. 7816-7827, 2018. 3, 4
Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, and Tic- Yan Liu. Neural architecture search with gbdt. arXiv preprint a0Xiv: 2020. 3, 19, 21

---

Published as a conference paper at ICLR 2022
Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang, Alan Yuille, and Jianchao Yang. Atomnas: Fine- grained end-to-end neural architecture search. In ICLR, 2020.
Vladimir Nekrasov, Hao Chen, Chunhua Shen, and lan Reid. Fast neural architecture search of
compact semantic segmentation models via auxiliary cells. In CVPR, pp. 9126-9135, 2019. 3
Hieu Pham, Melody Guan, Barret Zoph, Quoe V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: TrXXvv/802..32666 2018. 3
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, pp. 10428-104333 2020. 3, 7, 16
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pp. 4510-4520, 2018. 3
Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. ICIR, 2020. 3
Albert Shaw, Daniel Hunter, Forrest Landola, and Sammy Sidhu. Squeezenas: Fast neural architec- ture search for faster semantic segmentation. In ICCVW, 2019a. 3,
Albert Shaw, Wei Wei, Weiyang Liu, Le Song, and Bo Dai. Meta architecture search. In NeurlPS, pp. 11227-11237, 2019b.
Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas- bench-301 and the for surrogate benchmarks for neural architecture search. arXiv preprini arXiv:2008.09777 2020. 2,3
Dimitrios Stamoulis, Ruizhou Ding, Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efficient convnets in less than hours. In ECML-PKDD, pp. 481 197, 2019. 2,3
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, pp. 6105-6114, 2019.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE TPAMI, 2020a. 1, 2. 6, 7, 15, 17, 18, 19
Ning Wang, Yang Gao, Hao Chen, Peng Wang, Zhi Tian, Chunhua Shen, and Yanning Zhang. Nas- fcos: Fast neural architecture search for object detection. In CVPR, pp. 11943 11943-11951, 2020b.
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection in differentiable nas. In ICLR, 2021. 3
Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In ECCV, pp. 660-676. Springer, 2020. 3, 7, 19, 21
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangging Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, pp. 10734-10742, 2019. 1.2
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search.
arXiv preprint arXiv: 18/2.0992, 2018. 1,
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc- darts: Partial channel connections for memory-efficient differentiable architecture search. arXiv preprint arXiv:1907.05770 2019. 2,
Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture represen- tation learning help neural architecture search? arXiv preprint arXiv:2006.069966 2020. 3
12

---

Published as a conference paper at ICLR 2022
Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform-awe aawre neural network adaptation for mobile applications. In ECCV, pp. 285-300, 2018. 7,
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- bench-101: Towards reproducible neural architecture search. In ICML, pp. 7105-7114, 2019. 2, 3, 18
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang. Greedynas: Towards fast one-shot nas with greedy supernet. In CVPR, pp. 1999-2008, 2020. 3
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv: /903. 11728, 2019. 3
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural archi- tecture search with big single-stage models. In ECCV, 2020.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning- In CVPR, pp. 3712-3722, 2018.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. CVPR, pp. 2881-2890, 2017. 7, 15, 17
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pp. 633-641,44 2017. 8, 18
13

---

Published as a conference paper at ICLR 2022
Table Detailed statistics of our MAS-Bench-MR, NAS-Bench-MR contains 4 datasets and 9 set- tings. Considering the diversity and complexity of real-world applications (e. g.different scales/input sizes of training data), we use variety of challenging settings (e.g. full resolution and training epochs) to ensure that the model is fully trained. For classification, train models with different numbers of classes, numbers of training samples, and training epochs. For semantic segmentation, train models under different input sizes. We also evaluated video action recognition models under two settings: trained from scratch and pretrained with ImageNet-50- 1000. f denotes each sample contains multiple classes. denotes there three basic classes (car, pedestrian, cyclist) in KITTI, while for each object we also regress its 3D location (XYZ), dimensions (WHL), and orientation (ox) "N" training from scratch. "Y" training with the ImageNet-50 1000 pretrained model. We also give the mean, std, and the validation LI loss (%) of the neural predictor under the main evaluation metric of each setting.
Cls-50-1000
Clu-50-100
10-1001100
Cla-10
Seg
Seg-4x
3dDeer
Video
Vidao-p
ImNeet50 1000
5m 100
LmNee-50-1000
Cityscapes
Cityscapes
HMDB51
HMDB51
Input Size
224 224
224 x 224
224 x 224
224
224
512
1024
128
256
432 x 496
112 x 112 112 x 112
Eppachs
100
100
100
10
530
530
80
100
100
Number of classes
50
10
19
19
3
51
51
Number of training
samples class
1000
1000
1000
1000
2975'
2975!
3712!
Ro 70
70
Pretmiindd?
Main metric
tupl Acc
lupl Ace
top] Acc
tepl Auc
mloU
mloU
var-5D AP
top) Ace
tupl Acc
Mean
82.31
48.59
85.70
57.07
72.58
62.21
76.
19.56
30.88
Standard Deviation
64
1.51
5.33
4:70
3.69
4.65
Prediction
1.07
0.83
1.07
1,26
A DETAILS OF WAS-BENCH-MO
In this section, we provide details of the datasets and settings used to build our NAS-Bench-MR. We follow the common practice and realistic settings for training and evaluation in each task, making the scientific insights generated by our benchmark casier to generalize to real-world scenarios.
A.1 DATASETS AND SETTINGS
We randomly sample 2,500 structures from network coding space, and train and evaluate these same architectures for each task and setting. Each architecture is represented by a 27-dimensional continuous-valued code. Tab. 7 shows the representation of each dimension of the code. For different datasets and proxy tasks, we train a total of 22,500 models. See below for details.
ImageNet for Image Classification. The ILSVRC 2012 classification dataset (Deng et al., 2009) consists of 1,000 classes, with a number of 1.2 million training images and 50,000 validation images.
Considering the diversity of the number of classes and data scales in practical applications, and sav- ing training time, we construct three subsets: ImageNet-50-100 10000 ImageNet-50-100 100, and ImageNet- 10-1000, where the first number indicates the number of classes and the second one denotes the number of images per class.
In this work, we adopt an SGD optimizer with momentum 0.9 and weight decay Ic-4. The input size is 224 x 224. The initial learning rate is set to 0.1 with a total batch size of 160 on 2 Tesla V100 GPUs for 100 epochs, and decays by cosine annealing with a minimum learning rate of O. We adopt the basic data augmentation scheme to train the classification models, i.e., random resizing and cropping, and random horizontal flipping (flip ratio 0.5), and use single-crop for evaluation.
We report the top-1 and top-5 accuracies as the evaluation metric on all three benchmark datasets. We print the training log (top-1, top-5, loss, Ir) every 20 iterations and evaluate the model every epochs on the ImageNet validation set. We will release four checkpoints of 25, 50, 75, 100 epochs with the optimizer data and all the training logs for each model.
Cityscapes for Semantic Segmentation. The Cityscapes dataset (Cordts et al., 2016) contains high- quality pixel-level annotations of 5000 images with size 1024 x 2048 (2975, 500, and 1525 for the training. validation, and test sets respectively) and about 20000 coarsely annotated training images. Following the evaluation protocol (Cordts et al., 2016), 19 semantic labels are used for evaluation without considering the void label.
14

---

Published as a conference paper at ICLR 2022
Table 7: Representations of our 27- 27-dimensional coding.
Codes
Component
Notocks
Nresidual wnits
Nehannels
Input layers
in b2
Stage 1
b1
ni na, n2 rig. 163. Yég, nt. nig 714
Stage 2
b2
02, Cg G de c cl of 01
Stage 3
ba
Stage 4
bs
Output layer
To study the effect of different image resolutions in segmentation, we conduct two benchmarks with the input size of 512 x 1024 and 128 x 256, respectively. For the small-resolution setting, we pre-resize the Cityscapes dataset to 256 x 512 before the data augmentation.
In this work, we use an SGD optimizer with momentum 0.9 and weight decay 4c-5. The initial learning rate is set to 0.1 with total batch size of 64 on 8 Tesla V100 GPUs for 25000 iterations (about 537 epochs). Follows the common practice in (Zhao et al., 2017: Wang et al., 2020a; Ding et al., 2020b), the learning rate and momentum follow the "poly" scheduler with power 0.9 and minimum learning rate of le-4. We use basic data augmentation, i.e., random resizing and cropping, random horizontal flipping, and photometric distortion for training and single-crop testing with the test size of 1024 x 2048 and 256 x 512 respectively for two settings.
We report the mean Intersection over Union (mloU), mean (macro-averaged) Accuracy (mAcc), and overall mmiero-averaged) Accuracy (aAcc) as the evaluation metrics. We print the training log (mAce, loss, Ir) every 50 iterations and evaluate the model every 5000 iterations on the Cityscapes validation set. We will release five checkpoints of 5000, 10000, 15000, 20000, 25000 iterations with the optimizer data and all the training logs for each model.
KITTI for 3D Object Detection. The KITTI 3D object detection dataset (Geiger et al., 2012) is widely used for monocular and LiDDAR- 3D detection. consists of 7,481 training images and 7,518 test images as well as the corresponding point clouds and the calibration parameters, comprising total of 80,256 2D-3D labeled objects with three object classes: Car, Pedestrian, and Cyclist. Each 3D ground truth box is assigned to out of three difficulty classes (easy, moderate, hard) according to the occlusion and truncation levels of objects.
In this work, we follow the train-val split (Chen et al., 2015; Ding et al., 2020a), which contains 3,712 training and 3,769 validation images. The overall framework is based on Pointpillars (Lang et al., 2019). The input point points projected into bird's s-eye view (BEV) feature maps by voxel feature encoder (VFE). The projected BEV feature maps (496 x 432) are then used as the input of our 2D network for 3D/BEV detection.
(Lang et al., 2019), we set, pillar resolution: 0.16m, max number of pillars: 12000, and max number of points per pillar: 100. We apply the same data augmentation, i.e., random mirroring and flipping, global rotation and scaling, and global translation for 3D point clouds as in Pointpillar (Lang al., 2019). We use the one-cycle scheduler with initial learning rate of 2e-3, a minimum learning rate of 2c-4, and batch size 16 on 8 Tesla V100 GPUs for 80 epochs. We use an AdamW optimizer with momentum 0.9 and weight decay 1c-2. At inference time, axis-aligned non-maximum suppression (NMS) with an overlap threshold of 0.5 IoU is used for final selection.
Following
We report standard average precision (AP) on each class as the evaluation metric. We will release five checkpoints of 76, 77, 78, 79, 80 epochs with the optimizer data and detailed evaluation results CarrPedestrian/CCyliis easy/moderate/hard ID/BEV/Orientatiin detection AP) of each checkpoint for each model. Due to the unstable training of the KITTI dataset, we provide the last five check- points for researchers to tune hyperparameters from different criteria, such as the single model best performance, the average best performance, and the last epoch best performance.
HMDB51 for Video recognition. We train video recognition models on the HMDB51 dataset (Kuehne et al., 201 I), consisting of 6,766 videos with 51 categories. We use the first train- ing and validation split composing of 3570 and 1530 videos for evaluation, respectively. The video containing less than 64 frames is filtered, resulting in 2649 samples for training.
Considering the difficulty of learning temporal information, in addition to training from scratch, also conduct training using models of ImageNet-50-1100 as pretraining. To model the temporal
15

---

Published as a conference paper at ICLR 2022
100
10.4
1.000
0.451
65
mAcc aACC
Car BEV
90-
0.970
1,000
0.843
0.060
0.526
Ped-30
1508
1900
Pad-BEV
50
0.038
0.843
1,000
0.5335
0.577
1008
1000
00.50.45
0.400
0.460
0.535
1,000
0.738
56
0.514 0,77
65
Correlation of epech numbers
Architecture ranking under Seg metrics
Architecture ranking under 3dpet metrics
(b)
Figure 5: (a) Spearman's 's rank correlation of the different number of epochs (checkpoints of 10, 50, 90, 100 epochs during training on ImageNet-50- 1000). '10-C' denotes using the convergent learning rate for 10 epochs, Le., the proxy setting used in RegNet (Radosavovic et al., 2020). (b) Architecture rankings under the evaluation metrics of semantic segmentation (mloU, mAcc, aAcc). (c) Architec- ture rankings under the evaluation metrics of 3D object detection (car/pedestrian 3D/bird's s-eye view detection AP).
information, we replace the last 2D convolutional layer before the final classification layer of HRNet and its counterparts with 3D convolutional layer.
In this work, the input size is set to 112 x 112. We adopt an Adam optimizer with momentum 0.9 and weight decay le-5. The initial learning rate is set to 0.01 with a total batch size of 80 on 4 Tesla V100 GPUs for 100 epochs, and decays by cosine annealing with a minimum learning rate of O. We adopt random resizing and cropping, random brightness 0.5, random contrast 0.5, random saturation 0.5, random hue 0.25, and random grayscale 0.3 data augmentation.
We report the top-1 and top-5 accuracies as the evaluation metric on both two settings. We print the training log (top-1, top-5, loss, Ir) every 10 iterations and evaluate the model every epochs on the validation set. We will release the last checkpoint and all the training logs for each model.
A.2 ANALYSIS OF EAS-BENCH-MR
We summarize detailed statistics of our NAS- MAS--Benc--MR in Tab. 6. We also give the mean, std, and the validation LI loss (%) of our neural predictors under the main evaluation metric of each setting.
From Spearman's 's rank correlation of nine benchmarks in our main paper, we observe that:
(1) The four main tasks have different preferences for the architecture, and their correlation co- efficients between -0.003 (segmentation and video recognition) and 0.639 (segmentation and classification). This may be because segmentation can be seen as per-pixel image classification.
(2) Different settings the same task also have different preferences for the architecture, and their correlation coefficients are between 0.3 (CIs-50- 1000 and Cls-50-100) and 0.818 (CIs-50-100 and Cls-10c). From Fig. 5 (a) we also see that different training epochs result in significant perfor- mance change. In this way. the proxy 10-epoch training setting (Radosavovic et al., 2020) may not generalize well to real-world scenarios.
(3) The correlation coefficient matrix is related to the model performance of multi-task joint opti- mization using NP. The higher the correlation coefficient of the two tasks, the greater the gain of joint optimization. If the correlation coefficient of the two tasks is too low, joint optimization may reduce the performance on both tasks.
(4) We show the architecture ranking under different metrics for different tasks in Fig. (b) and (c). We noticed that the three metrics (i.e., mloU, mAcc, aAcc) in segmentation are positively correlated, which means only (e.g., mloU) needs to be optimized to obtain a good model under all three metrics. However, in the 3D object detection task, different metrics are not necessarily related, which makes multi-objective optimization especially important.
16

---

Published as a conference paper at ICLR 2022
Table 8: Comparative results (%) of NCP on the Cityscapes validation set. The first two architec- tures are searched using neural predictors that are trained on the Cityscapes dataset with an input size of 512x x 1024 (Seg) and the resized Cityscapes dataset with an input size of 128 256 (Seg-4x), respectively. The last model is searched by joint optimization of both the two predictors. À is set to 0.5 during the network codes propagation process. All models are trained from scratch. FLOPSs is measured using 512 x 1024.
Cityscapes 512 x 1024
Cilyscapes 128 x 512
ADE20K
Method
Params
FLOPs
512 x 512
mloUi mACC aACC mIoU mACC aACC mloU mACC aACC
ResNet18-PSP (Zhao ct al., 2017)
12.94M
108.53G
73.21
79.86
95.51
62.66
70.17
93.25
33.45
40.95
77.78
ResNet34-PSP (Zhao et al., 2017)
23.05M
193.12G
76.17
82.66
95.99
64.17
72.37
93.48
32.78
41.23
77.52
HRNet-WW W18s (Wang et al., 2020a)
5.48M
22.45G
76.21
84.43
95.86
64.49
73.58
93.71
34.39
44.65
77.60
W8I8 (Wang el al., 2020n)
9.64M
37.01G
77.73
85.61
96.20
65.19
74.85
93.66
34.84
45.27
77.80
NCP-Net-512 x 1024 (Fig. 19)
7.91M
29.34G
77.15
84.73
96.01
62.40
71.70
93.34
33.32
42.97
77.41
NCP-Net-128 512 (Fig. 20)
2.61M
31.37G
70.91
80.20
95.21
65.89
75.19
93.79
27.52
35.88
73.85
NCP-Net-Both (Fig. 21)
7,82M
50.90G
77.35
84.65
96.23
64.98
73.82
93.83
35.65
45.64
77.94
Table 9: Video recognition results (%) of NCP on the HMDB51 dataset. The first two models are searched using neural predictors that trained on HMDB51 from scratch (Video) and using models of ImageNet-50-1000 of both two predictors. We also show the classification accuracy of the pretrained model the ImageNet-50-1000 dataset (the column of "Cls- "Css-Pretraining"). Note that the last 2D convolutional layer before the final classifier replaced with 3D convolutional layer (3 3 x 3) for all models to capture the temporal information. À is set to 0.5. FLOPs is calculated using an input size of
as pretraining (Video-p). The last model is searched by joint optimization
112 112.
Video-Seratch top5
CIs-Pretraining top5
Method
Params
FLOPs
topl
topl
topl
top5
ResNet18 (Hc al., 2016)
12,10M
0.59G
21.50
$4.93
82.72
93.88
27.84
60.50
ResNet34 (He et al., 2016)
22.21M
1.20G
19.89
52.94
83.40
94.28
31.22
63.53
HRNet-W11 (Wang et al., 2020a)
14.35M
0.86G
24.55
55.87
83.04
95.16
30.96
61.21
HRNet- MiINet-W18// (Wang ct al., 2020a)
20.05M
144G
24.11
55.33
83.48
94.04
32.48
64.30
Ser--ee-Seratch (Fig. 22)
2.56M
0.69G
28.47
62.18
81.88
93.84
37.54
68.50
ACP-Net-Prretraine (Fig. 23)
1.72M
1.00G
27.49
62.01
82.48
94.80
39.14
69.66
NCP-Net-Both (Fig. 24)
2.39M
1.16G
27.14
62.28
83.92
94.76
40.21
70.29
B INTRA-TASK GENERALIZABILITY
In this section, we explore the effect of multiple proxy settings such as different input sizes (512 x 1024 and 128 x 256), and pretraining strategies (pretraining from classification training from the scratch for video recognition) in designing network architectures. We search architectures under single (e.g., 512 x 1024) or multiple (e.g., both resolutions) to validate the effectiveness of NCP for different intra-task settings.
Tab. and 9 show the intra-task cross-setting generalizability of our searched NCP-Net and some manually-designe networks, such as ResNet (He et al., 2016) and HRNet (Wang et al., 2020a), on the Cityscapes dataset and the HMDB51 dataset, respectively. We see that:
(1) Generally, the network searched a specific setting performs the best under this setting but poor under other settings, which means that NCP can optimize and customize model structures for specific setting. For example, Tab. 8 shows NCP customizing structures for different input resolutions on Cityscapes, which is essential for practical applications real-world scenarios.
(2) Benefiting from the high correlation between different settings of the same task, joint optimiza- tion of multiple settings within the same task (for both segmentation and video recognition) usually has positive effect, i... improving the performance model size (larger FLOPs). However, different tasks may not necessarily correlate, e.g. seg.emmentttion video recognition. This can be verified by Spearman's correlation, as discussed in main paper.
each setting, the cost of the increasing
17

---

Published as a conference paper at ICLR 2022
(3) In Tab. 8, apply the searched segmentation network to the ADE20K dataset (Zhou et al., ()1 to show the generalizability of the searched architectures. The network trained with a large input resolution (512x 1024) has better performance than the network trained with small resolution (128 x 256).
Moreover, the jointly searched architecture using both resolutions shows better generalizability than those two with a single objective. also demonstrated that the network searched on our MR has stronger transfer capability real-world scenarios, compared to the previous benchmarks such as (Dong & Yang, 2020; Ying et al., 2019) that uses a small input size.
(4) For both the segmentation and the video recognition tasks, our joint searched networks outper- form caunully-dessggae networks, such as HRNet (Wang et al., 2020a) and ResNet (He et al., 2016), and achieve better generalizability other settings and datasets, e,g., ADE20K.
C NCP ON WAS-BENCH-22
In this section, we apply NCP to the NAS-Bench- benchmark (Dong & Yang, 2020) to show its effectiveness. We first briefly introduce MAS-Bench-201 and then state the difference between our MAS-Bench-MMR and MAS-Bench-201. Lastly, we detail the experiment.
C.1 NAS- AAS-BENCH-201
NAS-Bench-201 employs a DARTS-like (Liu et al., 2019b) search space including three stacks of cells, connected by a residual block (He et al., 2016). Each cell is stacked N = 5 times, with the number of output chan- nels 16, 32, and 64 for the first, second, and third stages, respectively. The intermediate residual block is the basic residual block with a stride of 2. There are 6 searching paths in the space of NAS-Bench-201, where each path contains 5 options: (1) zeroize, (2) skip con- nection, (3) 1-by-1 convolution, (4) 3-by-3 con- volution, and (5) 3-by-3 average pooling layer, resulting 15,625 different models total.
46
Acc (%)
40
FLOPs
Figure 6: Visualization of the architecture prop- agation process on the WAS-Bench-201 bench- mark (Dong & Yang. 2020) (ImageNet-166. * represents the propagated model in each step. NCP finds the optimal structure from low-Ace and high-FLOPs starting point with only 6 steps by optimizing accuracy and FLOPs constraints si- multaneously.
NAS-Bench-201 train these models on the ImageNet- 16 (with an input size 16 x 16) and C0 (with an input size of 32 32) datasets. Since the ImageNet dataset is closer to practical applications, use the models in ImageNet- 16 as a comparison in the following section.
C.2 DIFFERENCES
There are some differences between the
benchmark and our MMS-Bench-MR.
(1) MAS-Bench- and MAS-Bench-MMR are different magnitudes (15,625 v.s. 1023). (2) The number of chameessblockk/resolttiooo is fixed in NAS-Bench-201, while it is searchable in our search space. In this way, our search space more fine-grained and suitable for customizing to tasks with different preferences (high- and low-level features, deeper and shallower networks, wider and narrower networks, etc.). (3) The architecture encoding (choose one of five operators). While in our NAS-Bench the continuous-valuee code more appealing to the learning of the neural predictor. (4) NAS- Bench-201 is built based on a single setting while MAS-Bench-MRR contains multiple settings.
represented as a one-hot
We then conduct experiments and show that despite the above many differences, NCP works well on NAS-Bench-20 demonstrating the generalization ability of NCP other benchmarks.
18

---

Published as a conference paper at ICLR 2022
86
NCP Net L(Ours)
NCP Net(Ours)
Neura Predictor
80
HRNet V32
Auto DeopLab
HRNcs W48
HANet W18
84
NCP Net (eet-5((0ua 5(Ours)
ResMet101-P5P
ResMet50
76
@HRNet W18s
R:chic34
WLB
83
R R.cNett
HiN WILSs
74
Reshett
72
8
10
12
14
16
18
100
200
300
400
500
FLOPs (G)
FLOPs (G)
Image Classification
Semantic Segmentation
NCP Net((Ours)
30
NCP -Ntt M[(Durs)
MCP S(Oura)
HR W32
26
MCC U(Ours)
HRNee-WiBk
196
74
HRNet
ResNu550
72
HRN-S-WA
18
70
RediieM
Re
Restiet101
Recket18
14
0
20
40
60
80
100 120 140 160 160 FLOPs
0
1
3D Detection
Video Recognition
Figure Comparisons of the efficiency (i.e., FLOPs) and the performance (e.g, Acc, mIoU, AP) four computer vision tasks, i.e., image classification (ImageNet), semantic segmentation (CityScapes), 3D detection (KITTI), and video recognition (HMDB51) between the proposed ap- proach and existing methods. Each method is represented by circle, whose size represents the number of parameters. * represents the optimal model with both high performance and low FLOOSs Our approach achieves superior performance compared to its counterparts on all four tasks.
C.3 EXPERIMENTS
In this work, we formulate each architecture in NAS- MAS-Bench-221 to a 6 = 30-dimensional one-hot encoding. We then use our continuous network propagation strategy to traverse architectures the space with higher-Acc and lower-FI.OPs as optimization goals. Given an initial one-hot encoding, our NCP treats as a continuous-valus coding and utilizes the argmax operation to obtain an edited one-hot coding after every several iterations.
In the experiment, we use the argmax operation after every 10 iterations, termed as step. As shown in Fig. 6, our NCP finds the optimal structure from low-Ace and high-FLOPs starting point with only 6 steps (60 iterations). The entire search process takes less than 10 seconds. Compared to other neural predictor- based methods such as (Wen et al., 2020; Luo al., 2020) that need predict the accuracy of a large number architectures, NCP is more efficient.
From Fig. 6 observe that from the second to fourth steps, the performance of the searched model vibrates. This may be because of the instability caused by one-hot encoding. NCP finds an optimal model with accuracy of 46.8 and FLOPs of only 90.36 (the highest accuracy in NAS- -AS-Beech-220 is 47.333333 47,33%) showing its effectiveness and generalizability.
D NCP ON DIFFERENT SINGLE TASKS
We also use NCP to find the optimal architecture on the four basic vision tasks, i.e., image clas- sification, semantic segmentation, 3D detection, and video recognition. Quantitative comparisons in Fig. show that the architectures found by NCP outperform well-designed networks, such as HRNet (Wang et al., 2020a) and ResNet (He et al., 2016).
We visualize the searched architectures by NCP in Fig. 8 to show can customize different repre- sentations for different tasks (objectives). We that:
19

---

Published as a conference paper at ICLR 2022
122
Siock Tosivel
layee Stapei
Stgez
Nayss
Mogel
Out over
In lyves Stgpe
baged
Stagel
Out oyer
Seg
Chemmi
Charmi
Stap22
Stups3
Stapel
Stugel
0cb
Video
3dDet
Figure 8: Visualization of the searched models by NCP for four different tasks (A 0.5). The 27-dimensional array in each row represents network structure.
Algorithm The network propagation process.
1. Learn neural predictor Fw (+) and fix it;
2. Initialize an architecture code e;
3. Set the target metrics, such as face and flopsi
for each iteration do
4. Re-set new target metrics (optional);
5. Forward Fw(e: (e) and calculate the loss;
6. Back-propagate and calculate the gradient Ae;
7. Select optimal dimension (optional);
8. update e based on the gradient Åe;
end for
9. Round and decode e to obtain the final architecture.
(1) Segmentation requires the most high-level information in the last two branches of the last stage, and the video recognition model contains the least low-level semantics in the first stages.
(2) The classification model contains more channels than other tasks, because the FLOPs constraint for classification is the weakest, i.e., classification costs fewer FLOPs than segmentation under the same architecture.
(3) The 3D detection model mainly utilizes the first two branches, which means it may rely more on high-resolution representations.
E DETAILS OF NCP
In this section, we provide supplementary details and complexity analysis of our NCP. The algorithm of Network Coding Propagation (NCP) is shown in Algorithm Code available.
E.1 IMPLEMENTATION DETAILS
In this work, for each task, 2000 and 500 structures in the benchmark used as the training and validation set to train the neural predictor. The optimization goal is set to higher performance (main evaluation metric of each task) and lower FLOPs.
Intuitively, large models can achieve moderate performance on multiple tasks by stacking redun- dant structures. On the contrary, the lightweight model tends pay more attention task-specific structural design due to limited computing resources, making it more suitable for generalizability evaluation. To this end, we set 0.5 obtain lightweight models unless specified.
We employ three-layer fully connected network with a dropout ratio of 0.5 the neural predictor. The first layer shared layer for all metrics with dimension of 256. The second layer separated layer for each metric with dimension of 128. Then for each evaluation metric, 128- by-1 fully-connected layer is used for final regression. During training, we use an Adam optimizer with weight decay of le-7. The initial learning rate is set 0.01 with batch size 128 on a single GPU for 200 epochs, and decays by the one cycle scheduler. The metric prediction continuous using the smoothL1 loss. In the network propagation process, unless specified, propagation with an initial code (b, n 2; c, i, = 64) and learning rate of 3 for 70 iterations in all experiments.
learned
20

---

Published as a conference paper at ICLR 2022
Table 10: Searching time of predictor-based searching methods. Our NCP is the fastest it evalu- ates all dimensions of the code with only one back-propagation without top-K, ranking. The time is measured on Tesla V100 GPU. Searching Time
Model
Notes
Predict validation metric 10,000 random archite- and train the top- models for final evaluation.
Neural Predictor Wen et al. (2020)
GPU day
Traverse each dimension of the code, predict its per-formance,
NetAd Yang (2018)
5min
and then edit the dimension with the highest accuracy improvement.
Maximize evaluation metrics along the gradient dir- ections by propagating architectures in the search space.
NCP (Ours)
10s (70 iterations)
E.2 TIME COMPLEXITY
Tab. 10 shows the searching time of three predictor- based searching methods. NCP is the fastest as it traverses all dimensions of the code with only one back propagationnn without the top-K ranking.
Suffering from the random noise in random search and the neural predictor, existing methods (Wen al., 2020; Luo et al., 2020) often need to train the top-K models for final evaluation, which costly (e.g., training segmentation model on Cityscapes costs 7 hours on 8 Tesla V100 GPUs). NCP uses gradient directions as guidance to alleviate the randomness issue.
F VISUALIZATION AND ANALYSIS
F.1 ANALYSIS OF TASK TRANSFERRING
We visualize the network coding propagation process of our cross-task architecture transferring, e-g., an architecture transferring from the classification task to the segmentation task by using the optimal coding on classification as initialization and using the neural predictor trained on segmentation for optimization. The detailed transferring visualizations of every two of the four tasks are shown in Fig. 9-12 with many interesting findings. For example, all networks in segmentation try increase the number of channels of the 4th branch of stage 4, while the networks in classification try to decrease it; the networks in video recognition keep an intermediate value.
F.2 ANALYSIS OF SINGLE- AND MULTI-TASE SEARCHING
We visualize the searched architectures (Fig. 13-14) and the network propagation process of each architecture (Fig. 19-24) in Tab. 8-9.
Classification. Manually-designer classification models often use gradually decreasing resolution and gradually increasing the number of channels, because intuitively the learning of classification task requires low-resolution high-level semantic information. However, under different data scales and number of classes, the situation is different, as shown in Fig. 15-18. For example, by reduc- ing the training samples from 1000 to 100 (from Fig. 15 to Fig. 16), the number of convolutional channels and the number of residual units in the 1st branch of stage 4 are increased during propaga- tion. showing the high-resolution low-level information is important when the training samples are insufficient.
Segmentation. From Fig 19, Fig 20, and Fig 21 we observe the differences of the network editing process of segmentation models. For example, compared to the two models optimized at a single resolution, the model optimized at both input resolutions reduces the numbers of residual units of the first two stages and increases the number of blocks of stage 2, resulting in more fusion modules and fewer parallel units.
Segmentation and Video Recognition. From the visualization of found architecture codes in Fig. 13 and 14, we can find, it is not always that the larger the model, the better the performance. Different objectives result in different customized architecture codes. For example, segmentation tends to select architectures with fewer input channels, while video recognition architectures often have more output channels.
21

---

Published as a conference paper at ICLR 2022
Cls-Seg
Cla-Video
Cle-JdDet
Figure 9: Visualization of our network propagation process of the optimal model in classification transferring to other three tasks (A = 0.5).
Figure 10: Visualization of our network propagation process of the optimal model in segmentation transferring to other three tasks (A = 0.5).
Video-Cis
Video--Seg
Video-3dDee
Figure 11: Visualization of our network propagation process of the optimal model in video action recognition transferring to other three tasks (A = 0.5).
Figure 12:
Visualization of our network propagation process of the optimal model in 3D object detection transferring to other three tasks (A = 0.5).
22

---

Published as a conference paper at ICLR 2022
126
Contral
126
26
Coatral
Bacc
Mlacc
Rasint
Ratunt
briegesSaage stage2
BrippenSaagei
Stage2
280p0 Video-Scratch
Zages
022-51221024
Coutral
Bec Ove
(iiyersSape)
50092
$3166
InipeeSSape)
Stare
Sbire Video-Pretrained
Saated
byt
Seg-128x512
Contral
Chanrel Macs
BOO
Seg-Both
Video-Both
Figure 13: Visualization of the searched segmen- tation models in Tab. 8 by our NCP for intra-task generalizability (A = 0.5). The 27-dimensional array in each row represents network structure.
Figure 14: Visualization of the searched video recognition models in Tab. 9 by our NCP for intra-task generalizability (A 0.5). The 27- dimensional array in each row represents net- work structure.
heae board
Srans wert: Kunis
Konia
Figure 15: Visualization of our network propaga- tion process of "NCP-Net-A" (A 0.7) for clas- sification.
Figure 16: Visualization of our network propa- gation process of "NCP-Net-B" (A 0.7) for classification.
20/0
135
10
Eupelo
Sapel Lotamne cuacst
Bayth Colered
Seare mharee ineery
Start L-camen #eoums
Sugre maamis teathe
180
105
Eapt urter mram! Bughh-une
Sraad & ioper
Zrank Suerd murld
bayd Toa mitens
Figure 17: Visualization of our network propa- gation process of "NCP-Net-C" (A = 0.7) for classification.
Figure 18: Visualization of our network propa- gation process of "NCP-Net-ABC" (A 0.7) for
classification.
23

---

Published as a conference paper at ICLR 2022
Stipet
Gellope
Suatte dranee nerees
2 Creee to Maet
Sayat 8,5 3,0tire MBC
Sug K.neee klwenss
Barch Lual ainand
Pigch Luatas evend
30
Figure 19: Visualization of our network propaga- tion process of "NCP-Net-512 x 1024" in Tab. 8 tion process of "NCP-Net-128 512" (1 0.5) (A = 0.5).
Figure 20: Visualization of our network propaga-
in Tab. 8 for segmentation.
hedt beteaal No
Brilaye
Brayye
214
20
but laye bohe
Sught Louet oaseee
Sugirt arom
crecn tor
Soeth Litamrns sard
Sucreri Lotenne land
Sopie kurg
10
Erstioms
Figure 21: Visualization of our network propa- gation process of "NCP-Net-Both" (A = 0.5) in Tab. for segmentation.
Figure 22: Visualization of our network propa- gation process of "NCP-Net--Sratcc" in Tab. 9
(A 0.5).
120
Litenns A Stpsl aoands
sagel by asscv
Kagis Schene cesssdds
Eapel Lolane inside
Secr ll of itrunn
Saurd finn
Erart Ladde centrnient
Sualce anio trans)
Hiiel Looi cusss
ransoni
Figure 23: Visualization of our network propaga- tion process of in Tab. 9 for video recognition.
Figure 24: Visualization of our network propa- gation process of "NCP-Net-Botts (A = 0.5) in Tab. 9 for video recognition.
(A = 0.5)
24


Reviews:
[{"review_id": "xYTmOw5DHRm", "reviewer": "Reviewer_Z6eY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors first reorganized a NAS multi-task benchmark and trained and evaluate more than 20,000 models on this benchmark. The performance of these models can greatly inspire other following works regarding NAS tasks. \n\nFor the proposed NCP method, the overall idea is simple and sound. The overall assumption of the NCP is that the search space for model code e is continuously differentiable. Then the whole problem is essentially an optimization problem: given several samples [e_i, y_i], we wanted to find the \\hat{e} so that can maximize y. The authors trained a three-layer MLP as a predictor (function: f: e-->y). Here I feel curious and wish to raise a question, though what I can think about is also training an MLP as a predictor and using gradient descent to find the optimal, if some classic convex optimization algorithms can also solve this problem? And how about the performance? It is okay not to find another solution. \n\nAnother question is about the generalization of this method. If we would like to apply NCP for a new benchmark, e.g. COCO for segmentation, how can we use the knowledge we have on NAS-Bench-MR to find a proper parameter and architecture for the model? Since for NAS-Bench-MR, the authors trained a large number of the models to have the search space, which is not generalizable for each new dataset. \n\nBesides, the writing of this paper needs to be further revised. The current version is a little hard to follow and needs re-read to get the whole idea of the authors. \n\nSome typos with a glance:\n1) abstract: wildly used --> widely used\n2) caption of Fig. : 3th --> 3rd\n----------------------------- \nAfter rebuttal:  \nThanks for answering my questions in detail and the rebuttal fully addressed my concerns. I think this paper can be accepted to ICLR 2022. ", "summary_of_the_paper": "This paper organized a multi-task NAS benchmark including 4 widely used datasets and more than 20,000 models. It also proposed a searching architecture named Network Coding Propagation (NCP) to effectively find the optical model for specific tasks. Experimental results indicate that the proposed model can be applied to inter-task, cross-task, and intra-tasks problems and the authors also showed the generalization capability to other benchmarks. ", "summary_of_the_review": "The NAS benchmark (NAS-Bench-MR) contains valuable large-scale experimental results and be used to benefit other NAS research. The proposed NCP method is sound but kind of naive. The writing of this paper needs to be further revised for both method's details and organizations. Considering all these conditions, I would suggest accepting this paper to ICLR but would not champion it. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qGzEZiFAZfK", "reviewer": "Reviewer_BCgR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strength:\n1. Good idea. The NCP trained on network codes instead of original data. Therefore the trained network can be used to update the architecture efficiently across datasets.\n2. Extensive experiments are conducted. Experiments are conducted on three settings: inter-tasks, cross-tasks, and intra-tasks. Results on these datasets demonstrate the effectiveness of the proposed NCP.\n\nweakness:\n1. The main weakness is about the evaluation of the main idea, network coding spaces. The main contribution of NCP is that NCP converts them into network coding spaces. However, very little understanding or intuition about the network coding space is provided. For example, is it continuous? Can we do interpolation between two codes? It is hard to tell what are the benefits and what are the disadvantages of optimizing on network coding spaces. Properties about the network coding spaces need to be further elaborated.\n2. The paper NAO (Luo et al., 2020) also propose to search on a continuous space. The difference between the proposed method and previous NAO needs more discussion. Why the proposed updating architecture codes along desired gradient directions for various objectives is novel, compared to NAO?\n\n", "summary_of_the_paper": "Network Coding Propagation is proposed in the paper for NAS on multiple heterogeneous vision tasks.\n", "summary_of_the_review": "The idea and the experiments of the paper are good, but some components need more explanation. Also, comparison with existing work needs to be more specific.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kbaVqXDKnGH", "reviewer": "Reviewer_AfiZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n\n\n* This paper tackles a relevant problem of finding versatile neural network architectures that should be suitable for a multitude of vision-related tasks that require different granularities of learned representations. This could bring the community a step closer towards unifying vision tasks and tackling several vision tasks (detection, segmentation …) using a single neural network. This has a potential for significant impact for the community: having general models and backbones could significantly ease the deployment of perception models in real-world scenarios (having several networks trained independently and tackling different sub-problems, and, in turn, requiring multiple forward passes, is not really a practical solution). \n* The problem is well-stated, and the methodology seems sound to me. To study this (cross-task) NAS problem, the paper first formalizes a test-bed, needed to study this problem, defines a search space that seeks representations that extract features at different granularities, and finally, proposes a method that can traverse the search space efficiently (and is applicable across tasks).\nThe paper is well-written and reasonably accessible to a reader, completely unfamiliar with NAS (i.e., this reviewer). \n* The experimental evaluation is thorough and supports the main claims (i.e., the proposed method can find both specialized network architectures as well as architectures that perform very well on recognition and segmentation tasks). \n\n\n**Cons**\n\n* From the eye of someone that is not familiar with NAS methodology and terminology, I was rather confused with a brief description of what the proposed neural coding propagation does (high-level description in the intro). I could only fully understand this part after fully reading Sec. 3. \nFrom the related work, it was unclear whether there have been any similar efforts in other fields (such as NLP) of formalizing NAS for finding versatile (instead of specialized) network architectures? \n* I think it would be fair to make it more explicit that the search space of the network architectures this method can find is still rather constrained and limited to convolutional neural networks and well-established and tested design patterns. It appears to me that the method will not be able to find a radically novel architecture but rather find “good configurations” within a well-constrained space. \n* The paper states that not all candidate models need to be evaluated by the neural predictor, but only those that are “in the search space along the gradient direction”. I would expect a more thorough justification for this. Also, how is exactly the search space along the gradient direction defined?\n", "summary_of_the_paper": "This paper tackles neural architecture search (NAS), addressing the problem of finding architectures suitable for a multitude of vision-related tasks, ranging from object detection to semantic segmentation. To the best of my knowledge, this is the first attempt in NAS for computer vision models that explicitly design the search space that allows for a search of architectures that are broadly applicable to a variety of tasks that usually rely on the different granularity of feature representation. \nThis paper makes two important steps towards the automated search of such general, multi-purpose architectures: (1) to study this problem, the paper formalizes a multi-task NAS benchmark, covering recognition, detection, semantic segmentation, and activity recognition (video) datasets. This allows for assessing the generality of the sought architecture across vision tasks. (2) to tackle the architecture search in such a setting, this paper propose makes the following contributions. First, it augments the search space with multiple stages that extract feature maps at several resolutions, followed by a feature fusion step (the number of blocks and feat. map resolutions are governed by the network hyperparameters.). This seems to be one of the key features that help to find architectures suitable for tasks that differ as much as image classification and semantic segmentation. \nNext, for assessing the performance in this large search space, this paper builds on predictor-based methods, where the idea is to train a network that predicts the performance (i.e., the learned predictor) of the sought-network based on the (encoded) hyperparameters, governing the network architecture. Similarly, the proposed neural coding propagation (NCP) directly encodes the hyperparameters in the network coding space — those can thus be directly updated via back-propagation of the learned predictor. A predictor is learned for each task independently, and this way, gradients can be accumulated separately across different tasks before jointly updating the “architecture codes”. ", "summary_of_the_review": "I am absolutely no expert in the NAS area, so I can only provide an assessment of a reader that is not at all familiar with the related work. However, I find it that this paper tackles a very important problem of general cross-task vision architecture search and makes important steps in this direction by providing an evaluation test-bed and methodology that experimentally validates the claims. In particular, Tab. 2 confirms that the proposed method finds several well-performing network architectures for semantic segmentation, while ‘Cls+seg’ entry in Tab. 3 works very well for both semantic segmentation and ImageNet classification. I see this as evidence that the proposed methodology is indeed capable of finding versatile neural network architectures. \n\nBased on what I can conclude based on the manuscript I see this paper as a very solid contribution, and I think this paper should be accepted. I hope, however, that the other reviewers will be more familiar with NAS and can provide deeper insights (and spot potential issues that were missed by an “untrained eye,” in case they are present). \n\n** After rebuttal **\n\nThanks for answering my questions in the rebuttal! All my doubts were clarified. I have read other reviews and didn't spot any major weaknesses. I am retaining my rating (accept). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper examines neural architecture search for multi-task networks, by associating model hyperparameters with a coding space and building an MLP predictor for mapping codes to task performance.  After the discussion phase, reviewers are marginally in favor or accepting the paper, pointing to the extensive experimental results as a convincing contribution.
Decision: Accept (Poster)