Generic Neural Architecture Search via Regression
Authors: Yuhong Li
Cong Hao
Pan Li
Jinjun Xiong
Deming Chen
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: neural architecture search
convolutional neural network
self-supervised learning

Abstract: Most existing neural architecture search (NAS) algorithms are dedicated to and evaluated by the downstream tasks, e.g., image classification in computer vision. However, extensive experiments have shown that, prominent neural architectures, such as ResNet in computer vision and LSTM in natural language processing, are generally good at extracting patterns from the input data and perform well on different downstream tasks. In this paper, we attempt to answer two fundamental questions related to NAS. (1) Is it necessary to use the performance of specific downstream tasks to evaluate and search for good neural architectures? (2) Can we perform NAS effectively and efficiently while being agnostic to the downstream tasks? To answer these questions, we propose a novel and generic NAS framework, termed Generic NAS (GenNAS). GenNAS does not use task-specific labels but instead adopts regression on a set of manually designed synthetic signal bases for architecture evaluation. Such a self-supervised regression task can effectively evaluate the intrinsic power of an architecture to capture and transform the input signal patterns, and allow more sufficient usage of training samples. Extensive experiments across 13 CNN search spaces and one NLP space demonstrate the remarkable efficiency of GenNAS using regression, in terms of both evaluating the neural architectures (quantified by the ranking correlation Spearman's rho between the approximated performances and the downstream task performances) and the convergence speed for training (within a few seconds). For example, on NAS-Bench-101, GenNAS achieves 0.85 rho while the existing efficient methods only achieve 0.38. We then propose an automatic task search to optimize the combination of synthetic signals using limited downstream-task-specific labels, further improving the performance of GenNAS. We also thoroughly evaluate GenNAS's generality and end-to-end NAS performance on all search spaces, which outperforms almost all existing works with significant speedup. For example, on NASBench-201, GenNAS can find near-optimal architectures within 0.3 GPU hour.

Generic Neural Architecture Search via Regression
Yuhong Li', Cong Hao?, Pan Li Jinjun Xiong', Deming Chen'
University of Illinois at Jrbana-Champal 1 Georgia Institute of Technology',
Purdue University University Buffalo leeyh iwwiioiii.meeeeeooooooooo callic.hao @aeechhh@@ecc ece.
waaa@eeeeaatcc.coo gatech.edu, panli @ purduc.edu, jinjun buffalo.edu, dehen @ illinois.edu
Abstract
Most existing neural architecture search (NAS) algorithms are dedicated to and evaluated by the downstream tasks, e.g., image classification in computer vision. However, extensive experiments have shown that, prominent neural architectures, such as ResNet in computer vision and LSTM in natural language processing, are generally good at extracting patterns from the input data and perform well on different downstream tasks. In this paper, we attempt to answer two fundamental questions related NAS. (1) Is it necessary to the performance of specific downstream tasks to evaluate and search for good neural architectures? (2) Can we perform NAS effectively and efficiently while being agnostic to the downstream tasks? To answer these questions, we propose a novel and generic NAS framework, termed Generic NAS (GenNAS). GenNAS does not use task-specifie labels but instead adopts regression on a set manually designed synthetic signal bases for architecture evaluation. Such self-supervised regression task can effectively evaluate the intrinsic power of an architecture to capture and transform the input signal patterns, and allow more sufficient usage of training samples. Extensive experiments across 13 CNN search spaces and one NLP space demonstrate the remarkable efficiency of GenNAS using regression, terms of both evaluating the neural architectures (quantified by the ranking correlation Spearman's 's be- tween the approximated performances and the downstream task performances) and the convergence speed for training (within few seconds). For example, on NAS-Bench- 101, GenNAS achieves 0.85 p while the existing efficient methods only achieve 0.38. We then propose an automatic task search to optimize the combination of synthetic signals using limited further improving the performance of GenNAS. We also thoroughly evaluate Gen- NAS's generality and end-to-end NAS performance on all search spaces, which outperforms almost all existing works with significant speedup. For example, on NASBench-201, GenNAS can find near-optimal architectures within 0.3 GPU hour. Our code has been made available at: http:///hhbb.commleeeeecc/////ee
waasstreamm-ss-sseeeiiii labels,
Introduction
Most existing neural architecture search (NAS) approaches aim to find top-performing architectures on specific downstream task, such image classification ① 2 3 5) semantic segmentation (6] design [12 13 15
7 8]. neural machine translation [9] I0 [1] or more complex tasks like hardware-softwarc co-
They either directly search on the target task using the target dataset (e.g., classification on CIFAR-I0 2 [7 ) or search on a prexy dataset and then transfer to the target one (e.g. CIFAR-10 ImageNet)
[6].
However, extensive experiments show that prominent neural architectures are generally good at extracting patterns from the input data and perform well to different downstream tasks. For example, ResNet [19] being a prevailing architecture in computer vision, shows outstanding performance across various datasets and tasks (20 21 22], because of its
1
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Samples near the classification boundary
Samples equally contribute to the regression accuracy
heavily affect the classification accuracy
Prediction Groundtruth
OOO
NN
NN
00
F
F
F2
C COONN
NN
O NN
)²
0²
F
3
F
F
(a) Classification
(b) Regression
For classification, only samples near the decision boundary determine the classification accuracy. For regression, all samples equally contribute to the regression accuracy. Therefore, regression is better at leveraging all training samples than classification to achieve faster convergence.
Figure I:
advantageous architecture, the residual blocks. This observation motivates us to ask the first question: Is there a generic way to search for and evaluate neural architectures without using the specific knowledge of downstream tasks?
Meanwhile, we observe that most existing NAS approaches directly use the final classification performance as the metric for architecture evaluation and search, which has several major issues. First, the classification accuracy dominated by the samples along the classification boundary, while other samples have clearer classification outcomes compared to the boundary ones (as illustrated in Fig. Ta). Such phenomena can be observed in the limited number of effective support vectors in SVM [23]. which also applies to neural networks because of the theory of neural tangent kernel 224] Therefore, discriminating performance of classifiers needs many more samples than necessary (the indeed effective ones), causing big waste. Second, a classifier tends to discard a lot of valuable information, such as finer-grained features and spatial information, by transforming input representations into categorical labels. This observation motivates us to ask the second question: Is there a more effective way that can make more sufficient use of input samples and better capture valuable information?
To answer the two fundamental questions for NAS, in this work, we propose a Generic Neural Architecture Search method, termed GenNAS. GenNAS adopts regression-based proxy task using prrnstreammttsk-aggaee
synthetic signals for network training and evaluation. It can efficiently (with near-zero training cost) and accurately approximate the neural architecture performance.
Insights. First, as opposed to classification, regression can efficiently make fully use of all the input samples, which equally contribute to the regression accuracy (Fig. (Fig.Ib). Second, regression on properly-desinnee synthetic signals is essentially evaluating the intrinsic representation power of neural architectures, which is capture and distinguish fundamental data patterns that are agnostic to downstream tasks. Third, such representation power is heavily reflected in the intermediate data of network (as we will show in the experiments), which are regrettably discarded by classification.
Approach. First, we propose a regression proxy task as the supervising task to train, evaluate, and search for neural architectures (Fig. 2 Then, the searched architectures will be used for the target downstream tasks. To the best of our knowledge, we are the first to propose self-supervised regression proxy task instead of classification for NAS. Second, we propose to use unlabeled synthetic data (c.g., sine and random signals) as the groundtruth (Fig. (3] to measure neural architectures' intrinsic capability of capturing fundamental data patterns. Third, to further boost NAS performance, we propose a weakly-supervised automatic proxy task search with only a handful of groundtruth architecture performance (e.g. 20 architectures), to determine the best proxy task, i.e., the combination of synthetic signal bases, targeting a specific downstream task, search space, and/or dataset (Fig. 4.
GenNAS Evaluation. The efficiency and effectiveness of NAS are dominated by neural architecture evaluation, which directs the search algorithm towards tor-perfognnng network architectures. To quantify how accurate the evaluation is, one widely used indicator is the network performance Ranking

---

Correlation [25] between the prediction and groundtruth ranking, defined as Spearman's 's Rho (p) or Kendall's Tau (T). The ideal ranking correlation I when the approximated and groundtruth rankings are exactly the same; achieving large p or can significantly improve NAS quality 226 27 28
Therefore, in the experiments (Sec. D. we evaluate GenNAS using the ranking correlation factors achieves, and then show its end-to-end NAS performance in finding the best architectures. Extensive experiments are done on 13 CNN search spaces and one NLP space [29]. Trained by the regression proxy task using only a single batch of unlabeled data within a few seconds, GenNAS significantly outperforms all existing NAS approaches on almost all the search spaces and datasets. For example, GenNAS achieves 0.87 p NASBench- 101 [3J while Zero-Cost NAS [II, an efficient proxy NAS approach, only achieves 0.38. On end-to-end NAS, GenNAS generally outperforms others with large speedup. This implies that the insights behind GenNAS are plausible and that our proposed regression-based task-agnostic approach is generalizable across tasks, search spaces, and datasets. Contributions, We summarize our contributions follows:
To the best of our knowledge, GenNAS is the first NAS approach using regression as the self- supervised proxy task instead of classification for neural architecture evaluation and search. It is agnostic to the specific downstream tasks and can significantly improve training and evaluation efficiency by fully utilizzng only a handful of unlabeled data.
GenNAS uses synthetic signal bases as the groundtruth to measure the intrinsic capability of networks that captures fundamental signal patterns. Using such unlabeled synthetic data in regression, GenNAS can find the generic task-agnostic top-performing networks and can apply to any new search spaces with zero effort.
An automated proxy task search to further improve GenNAS performance.
Thorough experiments show that GenNAS outperforms existing NAS approaches by large margins in terms of ranking correlation with near-zero training cost, across 13 CNN and one NLP space without proxy task search. GenNAS also achieves state-of-the-an performance for end-to-end NAS with orders of magnitude of speedup over conventional methods.
With proxy task search being optional, GenNAS is fine-tuning-fie highly efficient, and can be easily implemented on single customer-level GPU.
2 Related Work
NAS Evaluation. Network architecture evaluation is critical in guiding the search algorithms of NAS by identifying the top-performing architectures, which is also challenging task with intensive research interests. Early NAS works evaluated the networks by training from scratch with tremendous computation and time cost [18 [ To expedite, weight-sharing among the subnets sampled from However, due to the poor correlation between the weight-sharing and the final performance ranking, weight-sharing NAS can easily fail even in simple search spaces [34]]]]] 35 Yu et al. 36) further pointed out that without accurate evaluation, fashion. Recently, zero-cost NAS methods [37 31 38 39] have been proposed, which score the networks using their initial parameters with only forward and backward propagation. Despite the significant speed up, they fail to identify top rop-performing architectures in large search spaces such NASBench-
supernet is widely adopted [3 28
32, 33
NAS runs in a near-randos
To detach the time-consuming network evaluation from NAS, several benchmarks are developed with fully-trained neural networks within the NAS so that researchers assess the search algorithms alone in the search spaces [40 44.300333 35 29 41. playground.
NAS Transferability. To improve search efficiency, proxy tasks are widely used, on which the architectures are searched and then transferred to target datasets and tasks. For example, the CIFAR- 10 classification dataset seems to be good proxy for ImageNet [8 3). Kornblith et al. [42 studied the transfcrability of 16 classification networks on 12 image classification datasets. NASBench- 201 [35] evaluated the ranking correlations across three popular datasets with 15625 architectures. Liu et al. 4% studied the architecture transferability across supervised and unsupervised tasks. Nevertheless, training downsized proxy dataset is still inefficient (e.g. few epochs of full-blown training (4). In contrast, GenNAS significantly improves the efficiency by using a single batch of data while maintaining extremely good generalizability across different search spaces and datasets.
Self-supervised Learning. Self-supervised learning is form of unsupervised learning, that the neural architectures are trained with automatically generated labels to gain a good degree of com-
3

---

Inpuc
Stage-1
Stage-2 ba. ba
Stage-3 bi
Classifcation (Existing evalustors)
namaated
M.O
M.O
M&O
Groundtruth (Proxy task)
Felly
F5
Regression (GenNAS)
b.c w'
b 02 Rt' W
b-ej N.
(a) CNN regression
(b) RNN regression
Figure 2: Regression architectures CNNs and RNNs. (a) On CNNs, remove the final classifier and extract multiple stages of intermediate feature map for training. (b) On RNNs, we construct a many-to-many regression task, where the input and output tensors have the same size.
Liu et al. 43 recently proposed three unlabeled classification proxy tasks, including rotation prediction, colorization, and solving jigsaw puzzles, for neural network evaluation. Though promising. this approach did not explain why such manually designed proxy tasks are beneficial and still used classification for training with the entire dataset. In contrast, GenNAS uses regression with only a single batch of synthetic data.
prehension or understanding 44 45
47
3 Proposed GenNAS
In Section 3.1 we introduce the main concepts of task-agnostic GenNAS: 1) the proposed regression proxy task for both CNN architectures and recurrent neural network (RNN) architectures; 2) the synthetic signal bases used for representing the fundamental data patterns the proxy task. In Section 3.2 we introduce the automated proxy task search.
3.1
GenNAS
3.1.1
Regression Architectures
Training using unlabeled regression is the key that GenNAS being agnostic downstream tasks. Based on the insights discussed in Section [ the principle of designing the regression architecture is to fully utilize the abundant intermediate information rather than the final classifier.
Regression on CNNs. Empirical studies show that CNNs learn fine-graine high-frequency spatial details in the early layers and produce semantic features in the late layers [48) Following this principle, as shown in Fig. 22, we construct Fully Convolutional Network (FCN) 44 by removing the final classifier of a CNN. and then extract the FCN's intermediate feature maps from multiple stages. We denote the number of stages as N. Inputs. The inputs to the FCN are unlabeled real images, shaped
where is the batch size, and and are the input image size. Outputs. From each stage (1 < < N) of the FCN, we first extract a feature map tensor, denoted by F, E xbxx, and reshape it as F. E TBbxexx xk' x.u' through convolutional layer M, by A M,(F.) (with downsampling if w; > w' or h2 > N²). The outputs are the tensors F, which encapsulate the captured signal patterns from different stages. Groundtruth. We construct a synthetic signal tensor for each stage as the groundtruth, which serves part of the proxy task. A synthetic tensor is a combination multiple synthetic signal bases (more details Section 1.1.2). denoted by Fi- We compare F, with F; for training and evaluating the neural architectures. During training, we use MSE loss defined as L = EN E(FF f.) during validation, we adjust each stage's output importance as M E - -A) since the feature map tensors later stages are more related to the downstream task's performance. The detailed configurations of N, h', w', and of are provided in the experiments.
tensor I xxx33x
Regression on RNNs. The proposed regression proxy task can be similarly applied NL.P tasks using RNNs. Most existing NLP models use sequence of word-classifiers as the final outputs,

---

One stacked syuthetic signal tensor for one batch of data
Sin1D
Toput
One channel
Z
Sin2D
Sin2D Global
fer f''
Dot
Dot Local
2.5%
50%
100%
GDot
1.56%
Combination
(a) Examples of signal bases
(b) Examples of synthetic signal tensors
Figure 3: (a) Examples of synthetic signal bases (2D) feature maps). (b) Examples of the synthetic signal tensors by stacking 21) feature maps along the channel dimension for CNN architectures.
whose evaluations are thus based on the word classification accuracy 500 51 52). Following the same principle for CNNs, we design many-to-many regression task for RNNs shown in Fig. 2b. output, we extract the output tensor of the intermediate
Instead of using the final word-classing layer before it. Inputs. For general RNN model, the input is random tensor I Rlxbxd where / the sequence length, b is the batch size, and is the length of input/output word vectors. Given sequence of length 1, the input the RNN each time is one slice of the tensor I, denoted by z(0) Rxxdd 1 < l. Outputs. The output is F Rixbxdd where slice of f is F(() goxd Groundtruth. Similar to the CNN case, we generate synthetic signal tensor F as the proxy task groundtruth.
3.1.2 Synthetic Signal Bases
The proxy task for regression aims to capture the task-agnostic intrinsic learning capability of the neural architectures, i.e., representing various fundamental data patterns. For example, good CNNs must be able to learn different frequency signals to capture image features [53]. Here, we design four types of synthetic signal basis: (1) 1-D frequency basis (Sin1D); (2) 2-D frequency basis (Sin2D); (3) Spatial basis (Dot and GDot); (4) Resized input signal (Resize). SiniD and Sin2D represent frequency information, Dot and GDot represent spatial information. and Resize reflects the CNN's scale-invariant capability. The combinations of these signal bases, especially with different sine frequencies, can represent wide range of complicated real-world signals [54]. If network architecture is good at learning such signal basis and their simple combinations, it is more likely to be able to capture real-world signals from different downstream tasks.
Fig 3a depicts examples of synthetic signal bases, where each base is a 2D signal feature map. Sin1D generated by sin fx ()) sin (ear////yye fyy+), and Sin2D is generated by sin 2 / +c). where x and are pixel indices. Dot is generated according to biased Rademacher distribution (55 by randomly setting k% pixels 1 zeroed feature maps. GDot is generated by applying Gaussian filter with a = on Dot and normalizing between +1. The synthetic signal tensor F (the proxy task groundtruth) is constructed by stacking the 2D signal feature maps along the channel dimension (CNNs) or the batch dimension (for RNNs). Fig shows examples of stacked synthetic tensor F for CNN architectures. Within one batch of input images, we consider two settings: global and local. The global setting means that the synthetic tensor is the same for all the inputs within the batch, as the Sin2D Global Fig. 3bb aiming to test the network's 's ability to capture invariant features from different inputs; the local setting uses different synthetic signal tensors for different inputs, as the Dot Local in Fig 3b) aiming to test network's ability to distinguish between images. For CNNs, the real images are only used by resize, ze, and both global and local settings used. For RNNs, we only use synthetic signals and the local setting, because resizing natural language or time scries, the typical input of RNNs, does not make much sense resizing images for CNNs.
5

---

3.2 Proxy Task Search
While the synthetic signals can express generic features, the importance of these features for different tasks, NAS search spaces, and datasets may be different. Therefore, we further propose weakly- supervised proxy task search, to automatically find the best synthetic signal tensor, i.e., the best combination of synthetic signal bases. We define the proxy task search space as the parameters when generating the synthetic signal tensors. As illustrated in Fig. 4 first, we randomly sample small subset (e.g., 20) of the neural architectures in the NAS search space and obtain their groundtruth ranking on the target task (e.g.. image classification). We then train these networks using different proxy tasks and calculate the performance ranking correlation p of the proxy and the target task. We use the regularized tournament selection evolutionary algorithm to search for the task that results in the largest p. where p the fitness function.
Proxy Task Search Space. We consider the follow- ing parameters the proxy task search space. (1) Noise. We add noise to the input data following the distribution of parameterized Gaussian
Task search A small subset of all
Ranking on Proxy task
Ranking Target dataset
space
neural architectures
uniform
Network 1
distribution. (2) The number of channels for each synthetic signal tensor (c in F E g xw') can be adjusted. (3) Signal parameters, such as the frequency f and phase é in Sin, can be adjusted. (4) Feature combination. Each synthetic signal tensor uses either local or global, and tensors can be se- lected and summed up. Detailed parameters can be found in the supplemental material.
Network
3
Network
Network
Ranking Correcttion (r
Figure 4: Proxy task search.
4 Experiments
We perform the following evaluations for GenNAS. First, to show the true power of regression, we use manually designed proxy tasks without task search and apply the same proxy task on all datasets and search spaces. We demonstrate that the GenNAS generally excels in all different cases with zero task-specific cost, thanks to unlabeled self-supervised regression proxy task. Specifically, in Section 1 we analyze the effectiveness of the synthetic signal bases and manually construct two sets of synthetic tensors as the baseline proxy tasks; in Section 4.2 we extensively evaluate the proposed regression approach in 13 CNN search spaces and one NLP search space. Second, in Section 3 we evaluate the proxy task search and demonstrate the remarkable generalizability by applying one searched task to all NAS search spaces with no change. Third, in Section 4.4 evaluate GenNAS on end-to-end NAS tasks, which outperforms existing works with significant speedup.
Experiment Setup. We consider 13 CNN NAS search spaces including NASBench-101 30 NASBench-201 [55, Network Design Spaces (NDS) (56), and one NL. search space, NASBench- NLP (9) All the training is conducted using only one batch of data with batch size 16 for 100 iterations. Details NAS search spaces and experiment settings are in the supplemental material.
4.1 Effectiveness of Synthetic Signals
The synthetic signal analysis is performed on NASBench- 101 using CIFAR- 10 dataset. From the whole NAS search space, 500 network architectures are randomly sampled with a known performance ranking provided by NASBench-101. We train the 500 networks using different synthetic signal tensors and calculate their ranking correlations with respect to the groundtruth ranking. Using the CNN architecture discussed in Section 3.1.I we consider three stages, S₁ to S3 for N = 3: the number of channels is 64 for each stage. For Sin1D and Sin2D, set three ranges for frreuuency f: low (L) f E (0, 0.125). medium (M) f E (0.125, 0.375), and high (H) f E (0.375, 0.5). Within each frequency range, 10 signals are generated using uniformly sampled frequencies. For Dot and GDot, we randomly set 50% and 100% pixels to +1 on the zeroized feature maps.
The results of ranking correlations are shown in Table The three stages are evaluated independently and then used together. Among the three stages, Sin1D and Sin2D within medium and high frequency work better in S₁ and S2, while the high frequency Dot and resize work better in S3. The low frequency signals, such as GDot, Sin1D-L. Sin2D-L. and the extreme case zero tensors, result in low ranking correlations; we attribute their poor distinguishing ability. We also observe that the

---

Table 1: Ranking correlation (Spearman's 's p) analysis of different synthetic signals on NASBench- 1011
Sim1D
Sim20
Dot
GDot
Stage
Resize
M
H
50%
50%
100%
S1
0.13
0.43
0.64
0.14
0.53
0.63
0.55
0.62
0.18
0.16
0.56
0.17
0.03
0.52
0.79
0.05
0.73
0.72
0.64
0.69
0.03
0.02
0.73
0.18
Sg
0.08
0.77
0.80
0.23
0.78
0.72
0.76
0.81
0.16
0.17
0.80
0.22
GenNAS-combe: 0.85
best task in S3 (0.81) achieves higher p than Si (0.64) and S2 (0.79), which is consistent with the intuition that the features learned in deeper stages have more impact to the final network performance.
When all three stages are used, where each stage uses its top-3 signal bases, the ranking correlation can achieve 0.85, higher than the individual stages. This supports our assumption in Section 3.1.] that utiliizig more intermediate information of a network beneficial. From this analysis, we choose two top- cop-peformmin proxy tasks in the following evaluations to demonstrate the effectiveness of regression: GenNAS-single the best proxy task with single signal tensor Dot%100 used only in S3, and GenNAS-combo- the combination of the three top-pen up-performing tasks in three stages.
4.2 Effectiveness and Efficiency of Regression without Proxy Task Search
To quantify how well the proposed regression can approximate the neural architecture performance with only one batch of data within seconds, we use the ranking correlation, Spearman's 's p. as the metric 331
We use the two manually designed proxy tasks (GenNAS-single and GenNAS-combo without proxy task search to demonstrate that GenNAS is generic and can be directly applied to any new search spaces with zero task-specific search efforts. The evaluation is extensively conducted on 13 CNN search spaces and I NLP search space, and the results are summarized in Table
On NASBench-101. GenNAS is compared with zero-cost NAS 31 37 and the latest classi- fication based approaches [43]. Specifically, NASWOT (37] is zero-training approach that predicts network's trained accuracy from its initial state by examining the overlap of ac- tivations between datapoints. Abdelfattah et al. [1 proposed proxies such as synflow to evaluate the networks, where the synflow computes the summation of all the weights multi- plied by their gradients and has the best reported performance in the paper. Liu et al. [4 used three unsupervised classification training proxies, namely rotation prediction (rot), col- orization (col), and solving jigsaw puzzles (jig), and one supervised classification proxy (cls). We report their results after 10 epochs (@ep10) for each proxy. The results show that GenNAS- single and GenNAS-combo achieve 0.81 and 0.85 p on CIFAR-10. 10, and achive 0.73 on Im- ageNet, respectively, much higher than NAS- WOT and synflow. It is also comparable and even higher comparing with the classification proxies, cls @ els@ep5 and cls@ep10 10. Notably, the
Epochs
10
12
0.9
Gen seanch--search-N (0.a7)
GenNAS single
0.7
snio
Spearmat's
grasp
0.5
fisher
classification proxies need to train for 10 epochs using all training data, while GenNAS requires only a few seconds, more than 40x faster. On NASBench-201, we further compare with vote 331 and EcoNAS (26). EcoNAS is a re- cently proposed reduced-training proxy NAS approach. Vote (31) adopts the majority vote between three zero-training proxies including synflow, jacob_coor cov, and snip, Clearly, GenNAS- combo outperforms all these methods regard- ing ranking correlation, and is also 60x faster than EcoNAS and 40x faster than cls@ep10. On Neural Design Spaces, we evaluate GenNAS on both CIFAR-10 10 and ImageNet datasets. Compar- ing with NASWOT and synflow, GenNAS-single and GenNAS-comtl achieve higher p almost all cases. Also, synflow performs poorly on most of the NDS search spaces especially on ImageNet dataset, while GenNAS achieves even higher p. Extending to NIP search space, NASBench-NI.P, MonNAS-singll and GenNAS-come achieve 0.73 and 0.74 p. respectively, surpassing the best
4
synf
jacob
NASWOT
0.3
cls
0.1
500 10000
3500
Evaluation Cast (number minibatchos)
Figure 5: The effectiveness of regression- bbsed proxy task. GenNAS significantly outperforms all the existing NAS evaluation approaches regarding ranking correlation, with near-zero training cost.
7

---

Table 2: GenNAS ranking correlation evaluation using the correlation Spearman's p. GenNAS-single and GenNAS-combo use single or a combination of synthetic signals that are manually designed without proxy lask search. GenNAS search-N, -D, -R mean the proxy task is searched NASBench-101 101, NDS DARTS design space, and NDS ResNet design space, respectively. The top-1/2/3 results of GenNAS and efficient NAS baselines are highlighted by 1/1/$ respectively for each task. The values with superscripts are obtained after task search (*) or transferred (*) from a previous searched task. Methods like jig@epll which is 40x slower compared to the GenNAS in prediction are not considered as efficient oncs.
NASBench-101
Dulaset
NASWOT
synllow
jig@epl
mi@ep10
cal@ep cls@Pep10
GenNAS
>- 40x slower
single
combo
search-N
CIFAR-10
0.34
0.38
0.69
0.85
0.71
0.81
0.81
0.85
0.87
ImgNet
0.21
0.09
0.72
0.82
0.67
0.79
0.65
0.731
0.71
VANHeench. 201
Dataset
NASWOT
syndow jacob_cov snip
cls
vote
EsoNAS
GenNAN
> 40 slower
> 60% slower
single comnho
Search-N
CI
0.791
0.22
0.76
0.75
0.81
0.877
0.90
CIFAR-100
0.81
0.76
0.70
0.61
0.75
0.834
0.75
0.69
0.825
0.84"
No116
0.78
0.73
0.73
0.59
0.68
081
0.77
0.70
0.811
0,87
Neural Design Spaces
Dataset
NAS Space
NASWOT
synßoow
cls@ep10
GenNAS
40x slower
sangle
combo
search
search-D
search-R
CIFAR-10 10
DARTS
0.65
0.41
0.63
0.43
0.68
0.71
0.86
0.82
D
0.31
0.09
0.82
0.51
0.591
0.53
0.5g
0.52
Amoeha
0.33
0.06
0.67
0.64
0.683+
0,78
0.72
ENAS
0.55
0.19
0.66
0.56
0,70
0.67*
0.82
0,78
ENAS-f
0.43
0.26
0.86
0:65
0.65
0.6751
0.73
0.67
NASNet
0.40
0.00
0.64
0.56
0,66
0.65
0.77
0.71
PNAS
0.51
0.26
0.50
0.32
0.5g
0.5y
0.76 14
0.71
PN
0.10
0.32
0.85
0.48
0.56"*
0.55
0,47
ResNet
0.26
0.22
0.65
0.34
0.52
0.55
0.54
0.83*
ResNeXt-A
0.65
0.48
0.86
0.57
0.61
0.80"*
0.63
0.84
ResNeXi-B
0.60
0.60
0.66
0.26
0,30
0.53*
0,5s
0.71
ImageNet
DARTS
0.66
0.21
0.61
0.75
0,703
0.84
0.55
DARTS-f
0.20
0.37
0.68g
0.691
0.67*
0.69 tr
0.59
Amocha
0:42
0.25
0.63
0,72
0.73
0.80
0.67
ENAS
0.69
0.17
0.59
0.70*
0.58
0.81
0.65
NASNet
0.51
0.01
0.52
0
0.52
0.70
0
PNAS
0
0.14
0.28
0.39
0.45
0.62
0.40
ResNeXt-A
0.72
0.42
0
0.84
0.75*
0.62
0.87
ResNeXt-B
0.63
0.31
0.71
0.791
0.51
0.60
0.64
NASBench-NLPP
Dataset
grad_norm norm snip
grasp
synllow
jucob_cov cov
ppi@ep3
GenNAS
192x slower
single
combo
seurch
PTB
0.21
0.19
0.16
0.34
0,56
0.79
0.73
0.741
0.81
zero-proxy method (0.56). Comparing with the ppl@ep3, the architectures trained on PTB dataset after three epochs, GenNAS is 192x faster in prediction.
Fig. visualizes the comparisons between GenNAS and existing NAS approaches on NASBench- 101, CIFAR-10. Clearly, regression-based GenNAS (single, combo) significantly outperforms the existing NAS with near-zero training cost, showing remarkable effectiveness and high efficiency.
4.3 Effectiveness of Proxy Task Search and Transferability
Effectiveness of Proxy Task Search. While the un- searched proxy tasks can already significantly outper- form all existing approaches (shown in Section 4.2). we demonstrate that the proxy task search described in Sec- tion 3.2 can further improve the ranking correlation. We adopt the regularized evolutionary algorithm [[]. The population size is 50: the tournament sample size is 10; the search runs 400 iterations. We randomly select 20 architectures with groundtruth for calculating the p. More settings can be found in the supplemental mate- rial. Fig. [6) shows the search results averaged from 10 runs with different seeds. shows that the regularized evolutionary algorithm is more effective comparing with random search, where the correlations of 20 architec- tures are 0.86 + 0.02 and 0.82 + 0.01, respectively.
Regularized Evolutionary Random Search
0.55
Spearman's Rho
DUNG
034
100
2001
of Samples
Figure 6: Proxy task search.
8

---

Table 3: GenNAS end-to-end NAS results comparing with the state-of-the-art NAS approaches, showing test accuracy (%) on different NAS spaces and datasets. denotes a method that is replicated with the same regularized evolutionary algorithm in Section 4.4 for fair comparison. On NASBench- 201, the GPU hours do not include task search since GenNAS-N transferred from NASBench- -01. The values with superscripts are obtained after task search (*) or transferred (*) from a previous searched task.
MASBench-101(CIFAAN
Optimal
NASWOT
synßow"
Hallway"
GenNAS-N
94.32
03.30:+0.002
91.31
93,28::0.002 93.92==0.004'
NASBench-201
Dataset
Optimal
RSPS
DARTS-V2
GDAS
SETN
ENAS
NASWOT
GenNAS-N
CIFAR-10
94.37
84.07.1.361
54.30.10.00
93.61110.09
87.64.10.00
53.89-10.58
92.96.10.80
94.18
CIFAR-100
73.49
58.33:1:.334
15.61:10.00
70.61::0.26
56.87:17.77
15.00::0.00
1.10
72.56:10.4'
ImgNet16
47.31
26.28.13.09
16.32.0.000
41.71110.98
32.52.10.21
14.84.12.10
44.43.12007
4..59.10.54
hours
8.8
Neural Design Spaces (CIFAR-10)
NAS-Space
Optimal
NASWOT
synflow"
cl<@ep10"
GenNAS-N
GenNAS-R
GenNAS-D
RosNer
95,30
93,52=00311 02.8100.10
94.51+0.20
94.48:2::24
04.63:+::23
94.77:10.13"
ResNeXt-A
94,99
93,39:40.67
94.05=0048
94.24: 0.22
94.25=0021*
94.12:::020f
94.37-10.14
ResNeXt-B
95.12
03.56:00.3
93.65:00.4
94.33::0.26
94.29=00.4
94.26::0.35
94,23+:0.22
In the following experiments, we evaluate three searched proxy tasks, denoted by GenNAS search-N, -D, and -R, meaning that the task is searched on NASBench-101, NDS DARTS search space, and NDS ResNet search space, respectively. We study the performance and transferability of the searched tasks on all NAS search spaces. Proxy task search is done on a single GPU GTX 1080Ti. On NASBench- 101 NGenNAS-N), ResNeXt (GenNAS-R), and DARTS (GenNAS-D), the search time is 5.75, 4, and 12.25 GPU hours, respectively. Once the proxy tasks is searched, it can be used to evaluate any architectures in the target search space and can be transferred to other search spaces.
Proxy Task. The performance of GenNAS scarchh, -D, and -R proxy tasks is shown in Table 2 First, in most cases, proxy task search improves the ranking correlation. For example, in NDS, using the proxy task searched on DARTS space (search-D) outperforms other GenNAS settings ResNet-like spaces outperforms others as well. In NASBench-NLP, the proxy task search can push the ranking correlation 0.81, surpassing the ppl@ep3 (0.79). Such results demonstrate the effectiveness of our proposed proxy task search. Second, the searched proxy task shows great transferability: the proxy task searched on NASBench-101 (search-N) generally works well for other search spaces, i.e., NASBench-201, NDS, and NASBench-NLP This further emphasizes that the fundamental principles for top-performing neural architectures are similar across different tasks and datasets. Fig. 5 visualizes the performance of GenNAS comparing with others.
GenNAS with
DARTS-like spaces, while using proxy task search-R on
4.4 GenNAS for End-to-End NAS
Finally, we evaluate GenNAS the end-to-end NAS tasks, aiming to find the best neural architec- tures within the search space. Table 3 summarizes the comparisons with the tfate-of-the-ar art NAS approaches, including previously used NASWOT, synflow, cls@ep10, and additionally Halfway (30),
RSPS 288, DARTS-VI R DARTS-V2, GDAS (58]. SETN [59], and ENAS [60]. Halfway is the NASBench- 101 releasedd result using half of the total epochs for network training. In all the searches during NAS, we do not use any tricks such as warmup selection [3T] or groundtruth query to compensate the low rank correlations. We fairly use a simple yet effective regularized evolutionary algorithm [1] and adopt the proposed regression loss as the fitness function. The population size 50 and the tournament sample size is 10 with 400 iterations. On NASBench- 101, GenNAS finds better architectures than NASWOT and Halfway while being up to 200 faster. On NASBench-201 GenNAS finds better architectures than the state-of-the-a GDAS within 0.3 GPU hours, being 30x faster. Note that GenNAS uses the proxy task searched on NASBench- 101 and transferred to NASBench-200 201, demonstrating remarkable transferability. On Neural Design Spaces, GenNAS finds better architectures than the cls@ep 10 using labeled classification while being 40x faster. On NASBench-NI.P, GenNAS finds architectures that achieve 0.246 (the lower the better) average final regret score т, outperforming the ppl@ cp3 (0.268) with 192x speed up. The regret score at the

---

Table Comparisons with state-of- NAS methods on ImageNet under the mobile setting. is the time for proxy task search.
Method
Test Fir. (%)
Params (M)
FLOPS(M)
Search Cost (GPU days)
Searched Method
Seurched dataset
top-1
top-5
(M)
NASNet-A 1
26.0
8.4
5.3
564
2000
RL evolution
CIFAR-10
AmoebuNet-CC
24.3
7.6
6.4
570
3150
CIFAR-I0
PNAS 63
25.8
8.1
5.1
588
225
SMBO
CIEAR 10
DARTS order)
26.7
4.7
574
gradient-based
SNAS 05
27.3
9.2
4.3
522
1,5
gradient-based
CIFAR-10
GDAS 58
26.0
8.5
5,3
581
0.21
gradient-bised
CIFAR-10
PADARTS
24.4
7.4
4.9
557
gradiien based
CIFAR-I
P-DARTS
24.7
7.5
5.1
577
gradient- hased
CIFAR-100 100
gradiint- training. free
TE.NAS 02
26.2
8.3
6.3
0.05
CIFAR-10
PC-DARTS
24.2
7.3
5.3
597
gradieni- based grafient hased
ImageNet
ProxyleseNAS 4
24.9
7.5
7.1
465
ImageNet
UNNAS-jig
M3
567
gradient-
ImageNet
TE.NAS
24.5
7.5
5.4
599
0.17
training free
ImugeNet
GenNAS-combo
25.1
7.8
4.8
559
0.04
evolution+lew-te
CIFAR-10
GenNAS-D14
24.3
7.2
5.3
599
0.7 0.7"+0.04
evolution+few shot
CIFAR 10
moment t is rt) = L(t) L*, where L(t) is the testing log perplexity of the best found architecture according to the prediction by the moment, and L 4.36 is the lowest testing log perplexity in the whole dataset achieved by LSTM [50] architecture.
On DARTS search space, we also perform the end-to-end search on ImageNet- 1k 61 dataset. We fix the depth (layer) of the networks to be 14 and adjust the width (channel) so that the of FLOPs is between 500M 600M. We evaluate two strategies: one without task search using GenNAS-combo (see Table [T. and the other GenNAS-D14 with proxy task search DARTS search space with depth 14 and initial channel 16. The training settings follow TENAS 62) The results are shown in Table 4 We achieve top- 1/5 test eπrоr of 25.1/7.8 1/7.8 using GenNAS-comee
and top- 1/5 test error of 24.3/7.2 using GenNAS-DIA 114, which are on par with existing NAS architectures. GenNAS-combo is much faster than existing works, while GenNAS-DD -14 pays extra search time cost. Our next step to investigate the searched tasks and demonstrate the generalization and transferrability of those searched tasks to further reduce the extra search time cost.
These end-to-end NAS experiments strongly suggest that GenNAS is generically efficient and effective across different search spaces and datasets.
5 Conclusion
In this work, we proposed GenNAS, self-supervised regression-battt approach for neural archi- tecture training, evaluation, and search. GenNAS successfully answered the two questions at the beginning. (1) GenNAS is generic task-agnostic method, using synthetic signals to capture neural networks' fundamental learning capability without specific downstream task knowledge. (2) GenNAS extremely effective and efficient method using regression, fully utilizing all the training samples and better capturing valued information. We show the true power self-supervised regression via manually designed proxy tasks that do not need to search. With proxy search, GenNAS can deliver even better results. Extensive experiments confirmed that GenNAS able to deliver state-of-the-a performance with near-zero search time, in terms of both ranking correlation and the end-to-end NAS tasks with great generalizability.
6 Acknowledgement
We thank IBM-Illinois Center for Cognitive Computing Systems Research (C3SR) for supporting this research. We thank all reviewers and the area chair for valuable discussions and feedback. This work utilizes resources (66) supported by the National Science Foundation's 's Major Research Instrumentation program, grant #1 #1725729, well as the University of Illinois at Jrbana-Champall P.I.. is also partly supported by the 2021 JP Morgan Faculty Award and the National Science Foundation award HDR-2117997.
10

---

References
[1] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference artificial intelligence. volume 33, pages 4780-4789, 2019.
[2] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv: 1.01578, 2016.
[3] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint /rXiv:
2018.
[4] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:/8/2.003:: 2018.
[5] Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang. Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vision transformer architecture search. arXiv preprint 2tt1:21006.17000 2021.
[6] Chenxi Liu, Liang-Chien Chigh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 82 92, 2019.
[7] Vladimir Nekrasov, Hao Chen, Chunhua Shen, and Ian Reid. Fast neural architecture search of compact semantic segmentation models via auxiliary cells. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9016--933,, 2019.
[8] Albert Shaw, Daniel Hunter, Forrest Landola, and Sammy Sidhu. Squeezenas: Fast neural archi- tecture search for faster semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[9] Changlin Li, Tao Tang, Guangrun Wang, Jiefeng Peng. Bing Wang, Xiaodan Liang, and Xiaojun Chang. Bossnas: Exploring hybrid con-transformers with block-wisely self-supervised neural architecture search. arXiv preprint arXiv:2103.
2021.
[10] Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hwwde transformers for efficient natural language processing. arXiv preprint arXiv:2005. /4187, 2020.
[11] David So, Quoe Le, and Chen Liang. The evolved transformer. In International Conference Machine Learning, pages 5877-5886. PMLR, 2019.
[12] Cong Hao and Deming Chen. Deep neural network model and fpga accelerator co-design: Opportunities and challenges. In 2018 14th IEEE International Conference Solid-State and Integrated Circuit Technology (ICSICT), pages 1-4. IEEE, 2018.
[13] Cong Hao, Xiaofan Zhang, Yuhong Li, Sitao Huang, Jinjun Xiong, Kyle Rupnow, Wen-mei Hwu, and Deming Chen. Fpga/dnn co-design: An efficient design methodology for lot intelligence on the edge. In 2019 56th ACM/IEEE Design Automation Conference (DAC), pages 1-6. IEEE, 2019.
[14] Xiaofan Zhang, Haoming Lu, Cong Hao, Jiachen Li, Bowen Cheng, Yuhong Li, Kyle Rupnow, Jinjum Xiong, Thomas Huang, Honghui Shi, et al. Skynet: a hardware-efficient method for object detection and tracking on embedded systems. arXiv preprint arXiv: 1909. 09709, 2019.
[15] Ji Lin, Wei-Ming Chen, Yujun Lin, John Cohn, Chuang Gan, and Song Han. Mcunet: Tiny deep learning on iot devices. arXiv preprint arXiv:2007.10319 2020.
[16] Lin, Wei-Ming Chen, Han Cai, Chuang Gan, and Song Han. Mcunetv2: Memory-efficient patch-based inference for tiny deep learning. arXiv preprint arXiv:2110.15352 2021.
[17] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoe Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International Conference Machine Learning, pages 2902-2911. PMLR, 2017.

---

[18] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818 2018.
[21] Mark Marsden, Kevin McGuinness, Suzanne Little, and Noel E O'Connor. Resnetcrowd: A residual deep learning architecture for crowd counting, violent behaviour detection and crowd density level classification. In 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pages 1-7. IEEE, 2017.
[22] Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. arXiv preprint arXiv: 1902.07208, 2019.
[23] Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18-25 1998.
[24] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. arXiv preprint arXiv:1806.07572 2018.
[25] Wayne W Daniel al. Applied nonparametric statistics. 1990.
[26] Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. Econas: Finding proxies for economical neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11396-11404, 2020.
[27] Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairnas: Rethinking evaluation faimesss of weight sharing neural architecture search. arXiv preprint arXiv:/907 997.00/455 2019.
[28] Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in Artificial Intelligence, pages 367-377. PMLR, 2020.
[29] Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, and Evgeny Burnaev. Nas-bench-nlp: neural architecture search benchmark for natural language processing. arXiv preprini arXiv:2006.00 16,,16622000 2020.
[30] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pages 7105-7114. PMLR, 2019.
[31] Mohamed S Abdelfattah, Abhinay Mehrotra, Lukasz Dudziak, and Nicholas D Lane. Zero-cost proxies for lightweight nas. arXiv preprint
2021.
[32] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pe-darts: Partial channel connections for memory-efficiem architecture search. arXiv preprint 1OXXv:1977.05731 2019.
[33] Yuhong Li, Cong Hao, Xiaofan Zhang, Xinheng Liu, Yao Chen, Jinjun Xiong, Wen-mei Hwu, and Deming Chen. Edd: Efficient differentiable dnn architecture and implementation co-search for embedded ai solutions. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pages 1-6. IEEE, 2020.
[34] Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakehi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. arXiv preprint arXiv:1909.0 09656, 2019.
[35] Xuanyi Dong and Yi Yang. Nas-hench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326, 2020.
12

---

[36] Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. arXiv preprint arXlv:1902.08142, 2019.
[37] Joseph Mellor, Jack Turner, Amos Storkey, and Elliot Crowley. Neural architecture search without training. arXiv preprint arXiv:2006.046-7 2020.
[38] Debadeepta Dey, Shital Shah, and Sebastien Bubeck. Fear: A simple lightweight method to rank architectures. arXiv preprint arXiv:2106.04010 2021.
[39] Guihong Li, Sumit K Mandal, Umit Y Ogras, and Radu Marculescu. Flash: Fast neural architecture search with hardware optimization. ACM Transactions on Embedded Computing Systems (TECS), 20(5s): 2021.
[40] Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXii:2008.00977 2020.
[41] Xiu Su, Tao Huang. Yanxi Li, Shan You, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Prioritized architecture sampling with monto-carlo tree search. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 10968-10977, 2021.
[42] Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2661-2677 2019.
[43] Chenxi Liu, Piotr Dollár, Kaiming He, Ross Girshick, Alan Yuille, and Saining Xie. Are labels necessary for neural architecture search? In European Conference on Computer Vision, pages 793-813. Springer, 2020.
[44] Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[45] Alexey Dosovitskiy, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. Citeseer, 2014.
[46] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. Proceedings of the IEEE international conference on computer vision, pages 1422-1430, 2015.
[47] Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using vidcos. In Proceedings of the IEEE international conference on computer vision, pages 2794- 2802, 2015.
[48] Haohan Wang, Xindi Wu, Zeyi Huang, and Eric Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 8684-8694, 2020.
[49] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[50] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1735:173- 1997.
[51] Tomáš Mikolov, Martin Karafíát, Lukáš Burget, Jan Černocky, and Sanjeev Khudanpur. Recur- rent neural network based language model. In Eleventh annual conference of the international speech communication association, 2010.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit. Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint a7X.v:1706.03762, 2017.
[53] Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06523, 2019.
13

---

[54] Georgi P Tolstov. Fourier series. Courier Corporation, 2012.
[55] Stephen J Montgonery-Smith,
The distribution of rademacher sums. Proceedings of the American Mathematical Sociery, 109(2)517-522) 1990.
[56] Iliaa Radosavovic, Justin Johnson, Saining Xie, Wan- Yen Lo, and Piotr Dollár. On network design spaces for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1882-1890, 2019.
[57] Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of english: The penn treebank. 1993.
[58] Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1761-1770, 2019.
[59] Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3681-3690, 2019.
[60] Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, pages 4095 4104. PMLR, 2018.
[61] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[62] Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.1 1535, 2021.
[63] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European conference computer vision (ECCV), pages 19-34, 2018.
[64] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint
2018
[65] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1294-1303, 2019.
[66] Volodymyr Kindratenko, Dawei Mu, Yan Zhan, John Maloney, Sayed Hadi Hashemi, Benjamin Rabe, Ke Xu, Roy Campbell, Jian Peng, and William Gropp. Hal: Computer system for scalable deep learning. In Practice and Experience in Advanced Research Computing, pages 41-48. 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
14

---

(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "MdqXcWzSRU4", "reviewer": "Reviewer_aBAs", "summary": "This article introduces a novel low-cost regression proxy task, which has a high correlation with the test performance of the final model. Its contributions include the following:\n\n1. Introducing a novel low-cost regression agent task, which greatly reduces the cost of evaluation.\n2. This proxy task is basically independent of downstream tasks, and can make the searched model achieve approximately consistent results in multiple search spaces and datasets.\n3. Competitive results can be achieved in End-to-End NAS.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ **Originality**: This article designs a very interesting proxy task, which allows the network to fit a set of hand-designed bases through regression to determine whether the network performance is good or bad.\n+ **Quality**:  There are a few questions:\n1. In the proxy task, why the input of CNN is unlabeled real data, but the input of RNN is random data?\n2. The image does contain information of different frequencies, so the basis of the design looks more reasonable for the image. But for RNN, is it reasonable to choose the same base?\n\n+ **Clarity**: This article is basically written in a well-organized manner. From the introduction of proxy task, to why such a proxy task was designed, to experimental verification and proxy task improvement (proxy task search), the whole logical line is very clear.\n\n+ **Significance**: Designing a low-cost proxy task is very important for NAS. The proxy task designed in this article uses a set of bases that have nothing to do with downstream data to evaluate the performance of the network at a very low cost. This is relatively novel. Previous work focused on network structure and data. This work introduces a new direction.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "92aSNoWs2CI", "reviewer": "Reviewer_CCjK", "summary": "This work designed synthetic signals as proxy tasks for Neural Architecture Search. The author claimed that self-supervised regression tasks on the synthetic signals can deliver task-agnositc NAS, and make searched results more highly correlated with ground truth accuracy of architectures.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. \"thus\" in the first item in the contribution list on page 2: I don't think it is the \"switching from classification to regression in NAS\" that leads to being \"agnostic to the speciﬁc downstream tasks\". This description could be misleading.\n2. Making NAS agnostic to tasks is a reasonable motivation, but this is irrelevant using regression or classification as the loss function, but more related to whether or not using supervision signals. Many places (e.g. line 56-57, 139) make a reader feel that regression is vital, but I disagree. Introducing regression tasks in NAS is not a novelty. Properly designing proxy tasks is the root.\n3. I could not see the necessity of using manually designed signals. If this is true, we should use large amount of manually designed signals for contrastive learning, to learn meaningful representations before downstream fine-tuning. Actually self-supervised learning on large-scale datasets like ImageNet, JFT can work very well recently [1,2]. Not to say the effort and uncertainty of manually designing these signals.\n4. I think the correlation comparison in Table 2 is not fair. GenNAS can use combo of signals, but all other methods only measure the correlation on a single dataset. Maybe using combination of CIFAR and ImageNet-16 can further boost their correlation.\n5. Another efficient & w.o. proxy task NAS work is not considered [3].\n\n[1] Improved Baselines with Momentum Contrastive Learning\n\n[2] A simple framework for contrastive learning of visual representations\n\n[3] Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "My main concern is the usage of the regression tasks on manually designed signals, which may not widely adoptable (than unlabeld natural images) and further cost design efforts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hour"}, "extra_text": {}}, {"review_id": "7qIryk-y7p0", "reviewer": "Reviewer_SYQY", "summary": "This work made an interesting attempt to solve the generic neural architecture search (NAS) based on regression. A self-supervised regression task based on synthetic signal is proposed to evaluate the performance of a neural network architecture without the true labels of the images as in most previous NAS algorithms, leading to an efficient and effective search scheme. The ranking correlation coefficient, i.e., Spearman coefficient is taken to quantitatively evaluate the efficacy of this novel scheme on extensive search space.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally speaking, the whole idea seems a good attempt for the hot topic of NAS, leading to possible new insight to task agnostic NAS. Comprehensive tests across 13 CNN search space and 1 NLP space illustrate a promising ranking correlation between the ground truth and the network evaluated based on regression tasks.  However, some big concerns remain for its current form, which are shown in the below section.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1.  Though for one-shot NAS, the ranking correlation seems to be a good metric for its performance evaluation. There is still some performance gap between the searched one and the stand-alone model. So the authors should better provide more justification on this aspect, that is, why the ranking correlation is enough to validate the effectiveness of their NAS scheme.\n\n2. No strong analysis or no insight on why such a synthetic signal works effectively for evaluation of a network architecture.\n\n3. The authors claimed that the proposed NAS is task agnostic, however, in all the experiments, only classification related experiments are provided to show the performance. As such, this seems not convincing to support such a claim.  More tasks such as detection, segmentation should be provided on this point.\n\n4. Only CIFAR-10 results are reported for end-to-end NAS comparisons, which is not convincing to show the efficacy of the proposed scheme. ImageNet based results should be provided to further support this point.\n\n5. Representation of this manuscript needs a lot of polish.", "review_text": "", "extra_scores": {"ethical_concerns": "\n", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "about 3 hours"}, "extra_text": {}}, {"review_id": "6oph3RrzyiC", "reviewer": "Reviewer_p1m2", "summary": "In this paper, the authors propose a generic NAS method named GenNAS, which uses regression as the self-supervised task. Moreover, the proposed method is task-agnostic and highly efficient. Experiments demonstrate the effectiveness of the proposed method. My detailed comments are as follows. \n\nPositive points:\n1.\tThe authors proposed a new regression-based NAS method. \n2.\tThe results from experiments show the proposed method achieves promising performance.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Negative points:\n1. In section 3.1.1, it is not clear how to divide a network into N stages. More details should be provided.\n2. In the last line of Table 3, cls@ep10 method yields better performance than GenNAS-N. More explanations are required. Moreover, it seems that GenNAS should not be highlighted.\n3. In section 4.4, line 352, it’s not clear that how to obtain L^*=4.36.\n4. Several state-of-the-art NAS methods [1,2] should be compared in the paper.\n5. Kendall Tau (KTau) is a widely used metric to measure the ranking ability. It would be stronger to use KTau to compare different methods.\n6. Since this paper is a NAS paper, the results of the final searched architectures on CIFAR and ImageNet should be reported.\n\nReferences:\n[1] Proxylessnas: Direct neural architecture search on target task and hardware, ICLR 2019.\n[2] Once-for-all: Train one network and specialize it for efficient deployment, ICLR 2020.\n\n--Post Rebuttal--\n\nI am happy to raise my score based on the additional results. I hope that all the results can be included in the paper as promised by the authors.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and potential negative societal impact of their work should be clarified.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel approach to ranking architectures using only one minibatch of data by constructing unsupervised regression tasks for CNNs and RNNs via passing the inputs through manually designed feature extractors. The authors have shown the efficacy of the results on a large number of search spaces and baselines.

During discussion period a number of extra results and nuances have been elicited. The authors are strongly recommended to include these in the manuscript. Examples include the precision@K bin of architectures, Kendall-Tau scores and also properly reporting the GenNAS-N gpu hours since it otherwise hides the time for the extra task-specific search.

This ranking method can be quite valuable to the community, so it is strongy recommended that the authors  release high-quality code such that subsequent papers on the topic can utilize this very well.
Decision: Accept (Spotlight)