Generic Neural Architecture Search via Regression

1 Introduction . Most existing neural architecture search ( NAS ) approaches aim to find top-performing architectures on a specific downstream task , such as image classification [ 1 , 2 , 3 , 4 , 5 ] , semantic segmentation [ 6 , 7 , 8 ] , neural machine translation [ 9 , 10 , 11 ] or more complex tasks like hardware-software codesign [ 12 , 13 , 14 , 15 , 16 ] . They either directly search on the target task using the target dataset ( e.g. , classification on CIFAR-10 [ 2 , 17 ] ) , or search on a proxy dataset and then transfer to the target one ( e.g . CIFAR-10 to ImageNet ) [ 18 , 3 ] . However , extensive experiments show that prominent neural architectures are generally good at extracting patterns from the input data and perform well to different downstream tasks . For example , ResNet [ 19 ] being a prevailing architecture in computer vision , shows outstanding performance across various datasets and tasks [ 20 , 21 , 22 ] , because of its 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . advantageous architecture , the residual blocks . This observation motivates us to ask the first question : Is there a generic way to search for and evaluate neural architectures without using the specific knowledge of downstream tasks ? Meanwhile , we observe that most existing NAS approaches directly use the final classification performance as the metric for architecture evaluation and search , which has several major issues . First , the classification accuracy is dominated by the samples along the classification boundary , while other samples have clearer classification outcomes compared to the boundary ones ( as illustrated in Fig . 1a ) . Such phenomena can be observed in the limited number of effective support vectors in SVM [ 23 ] , which also applies to neural networks because of the theory of neural tangent kernel [ 24 ] . Therefore , discriminating performance of classifiers needs many more samples than necessary ( the indeed effective ones ) , causing a big waste . Second , a classifier tends to discard a lot of valuable information , such as finer-grained features and spatial information , by transforming input representations into categorical labels . This observation motivates us to ask the second question : Is there a more effective way that can make more sufficient use of input samples and better capture valuable information ? To answer the two fundamental questions for NAS , in this work , we propose a Generic Neural Architecture Search method , termed GenNAS . GenNAS adopts a regression-based proxy task using downstream-task-agnostic synthetic signals for network training and evaluation . It can efficiently ( with near-zero training cost ) and accurately approximate the neural architecture performance . Insights . First , as opposed to classification , regression can efficiently make fully use of all the input samples , which equally contribute to the regression accuracy ( Fig . 1b ) . Second , regression on properly-designed synthetic signals is essentially evaluating the intrinsic representation power of neural architectures , which is to capture and distinguish fundamental data patterns that are agnostic to downstream tasks . Third , such representation power is heavily reflected in the intermediate data of a network ( as we will show in the experiments ) , which are regrettably discarded by classification . Approach . First , we propose a regression proxy task as the supervising task to train , evaluate , and search for neural architectures ( Fig . 2 ) . Then , the searched architectures will be used for the target downstream tasks . To the best of our knowledge , we are the first to propose self-supervised regression proxy task instead of classification for NAS . Second , we propose to use unlabeled synthetic data ( e.g. , sine and random signals ) as the groundtruth ( Fig . 3 ) to measure neural architectures ’ intrinsic capability of capturing fundamental data patterns . Third , to further boost NAS performance , we propose a weakly-supervised automatic proxy task search with only a handful of groundtruth architecture performance ( e.g . 20 architectures ) , to determine the best proxy task , i.e. , the combination of synthetic signal bases , targeting a specific downstream task , search space , and/or dataset ( Fig . 4 ) . GenNAS Evaluation . The efficiency and effectiveness of NAS are dominated by neural architecture evaluation , which directs the search algorithm towards top-performing network architectures . To quantify how accurate the evaluation is , one widely used indicator is the network performance Ranking Correlation [ 25 ] between the prediction and groundtruth ranking , defined as Spearman ’ s Rho ( ρ ) or Kendall ’ s Tau ( τ ) . The ideal ranking correlation is 1 when the approximated and groundtruth rankings are exactly the same ; achieving large ρ or τ can significantly improve NAS quality [ 26 , 27 , 28 ] . Therefore , in the experiments ( Sec . 4 ) , we evaluate GenNAS using the ranking correlation factors it achieves , and then show its end-to-end NAS performance in finding the best architectures . Extensive experiments are done on 13 CNN search spaces and one NLP space [ 29 ] . Trained by the regression proxy task using only a single batch of unlabeled data within a few seconds , GenNAS significantly outperforms all existing NAS approaches on almost all the search spaces and datasets . For example , GenNAS achieves 0.87 ρ on NASBench-101 [ 30 ] , while Zero-Cost NAS [ 31 ] , an efficient proxy NAS approach , only achieves 0.38 . On end-to-end NAS , GenNAS generally outperforms others with large speedup . This implies that the insights behind GenNAS are plausible and that our proposed regression-based task-agnostic approach is generalizable across tasks , search spaces , and datasets . Contributions . We summarize our contributions as follows : • To the best of our knowledge , GenNAS is the first NAS approach using regression as the self- supervised proxy task instead of classification for neural architecture evaluation and search . It is agnostic to the specific downstream tasks and can significantly improve training and evaluation efficiency by fully utilizing only a handful of unlabeled data . • GenNAS uses synthetic signal bases as the groundtruth to measure the intrinsic capability of networks that captures fundamental signal patterns . Using such unlabeled synthetic data in regression , GenNAS can find the generic task-agnostic top-performing networks and can apply to any new search spaces with zero effort . • An automated proxy task search to further improve GenNAS performance . • Thorough experiments show that GenNAS outperforms existing NAS approaches by large margins in terms of ranking correlation with near-zero training cost , across 13 CNN and one NLP space without proxy task search . GenNAS also achieves state-of-the-art performance for end-to-end NAS with orders of magnitude of speedup over conventional methods . • With proxy task search being optional , GenNAS is fine-tuning-free , highly efficient , and can be easily implemented on a single customer-level GPU . 2 Related Work . NAS Evaluation . Network architecture evaluation is critical in guiding the search algorithms of NAS by identifying the top-performing architectures , which is also a challenging task with intensive research interests . Early NAS works evaluated the networks by training from scratch with tremendous computation and time cost [ 18 , 1 ] . To expedite , weight-sharing among the subnets sampled from a supernet is widely adopted [ 3 , 28 , 4 , 32 , 33 ] . However , due to the poor correlation between the weight-sharing and the final performance ranking , weight-sharing NAS can easily fail even in simple search spaces [ 34 , 35 ] . Yu et al . [ 36 ] further pointed out that without accurate evaluation , NAS runs in a near-random fashion . Recently , zero-cost NAS methods [ 37 , 31 , 38 , 39 ] have been proposed , which score the networks using their initial parameters with only one forward and backward propagation . Despite the significant speed up , they fail to identify top-performing architectures in large search spaces such as NASBench-101 . To detach the time-consuming network evaluation from NAS , several benchmarks are developed with fully-trained neural networks within the NAS search spaces [ 40 , 30 , 35 , 29 , 41 ] , so that researchers can assess the search algorithms alone in the playground . NAS Transferability . To improve search efficiency , proxy tasks are widely used , on which the architectures are searched and then transferred to target datasets and tasks . For example , the CIFAR10 classification dataset seems to be a good proxy for ImageNet [ 18 , 3 ] . Kornblith et al . [ 42 ] studied the transferability of 16 classification networks on 12 image classification datasets . NASBench201 [ 35 ] evaluated the ranking correlations across three popular datasets with 15625 architectures . Liu et al . [ 43 ] studied the architecture transferability across supervised and unsupervised tasks . Nevertheless , training on a downsized proxy dataset is still inefficient ( e.g . a few epochs of full-blown training [ 43 ] ) . In contrast , GenNAS significantly improves the efficiency by using a single batch of data while maintaining extremely good generalizability across different search spaces and datasets . Self-supervised Learning . Self-supervised learning is a form of unsupervised learning , that the neural architectures are trained with automatically generated labels to gain a good degree of com- prehension or understanding [ 44 , 45 , 46 , 47 , 43 ] . Liu et al . [ 43 ] recently proposed three unlabeled classification proxy tasks , including rotation prediction , colorization , and solving jigsaw puzzles , for neural network evaluation . Though promising , this approach did not explain why such manually designed proxy tasks are beneficial and still used classification for training with the entire dataset . In contrast , GenNAS uses regression with only a single batch of synthetic data . 3 Proposed GenNAS . In Section 3.1 , we introduce the main concepts of task-agnostic GenNAS : 1 ) the proposed regression proxy task for both CNN architectures and recurrent neural network ( RNN ) architectures ; 2 ) the synthetic signal bases used for representing the fundamental data patterns as the proxy task . In Section 3.2 , we introduce the automated proxy task search . 3.1 GenNAS . 3.1.1 Regression Architectures . Training using unlabeled regression is the key that GenNAS being agnostic to downstream tasks . Based on the insights discussed in Section 1 , the principle of designing the regression architecture is to fully utilize the abundant intermediate information rather than the final classifier . Regression on CNNs . Empirical studies show that CNNs learn fine-grained high-frequency spatial details in the early layers and produce semantic features in the late layers [ 48 ] . Following this principle , as shown in Fig . 2a , we construct a Fully Convolutional Network ( FCN ) [ 49 ] by removing the final classifier of a CNN , and then extract the FCN ’ s intermediate feature maps from multiple stages . We denote the number of stages as N . Inputs . The inputs to the FCN are unlabeled real images , shaped as a tensor I ∈ Rb×3×h×w , where b is the batch size , and h and w are the input image size . Outputs . From each stage i ( 1 ≤ i ≤ N ) of the FCN , we first extract a feature map tensor , denoted by Fi ∈ Rb×ci×hi×wi , and reshape it as F̂i ∈ Rb×c ′ i×h ′×w′ through a convolutional layer Mi by F̂i = Mi ( Fi ) ( with downsampling if wi > w′ or hi > h′ ) . The outputs are the tensors F̂i , which encapsulate the captured signal patterns from different stages . Groundtruth . We construct a synthetic signal tensor for each stage as the groundtruth , which serves as part of the proxy task . A synthetic tensor is a combination of multiple synthetic signal bases ( more details in Section 3.1.2 ) , denoted by F∗i . We compare F̂i with F∗i for training and evaluating the neural architectures . During training , we use MSE loss defined as L = ∑N i=1 E [ ( F∗i − F̂i ) 2 ] ; during validation , we adjust each stage ’ s output importance as L = ∑N i=1 1 2N−i E [ ( F∗i − F̂i ) 2 ] since the feature map tensors of later stages are more related to the downstream task ’ s performance . The detailed configurations of N , h′ , w′ , and c′i are provided in the experiments . Regression on RNNs . The proposed regression proxy task can be similarly applied to NLP tasks using RNNs . Most existing NLP models use a sequence of word-classifiers as the final outputs , Sin1D Sin2D Dot GDot ( a ) Examples of signal bases … Sin2D Global Dot Local Combination … … Input Batch One stacked synthetic signal tensor for one batch of data One channel ( b ) Examples of synthetic signal tensors Figure 3 : ( a ) Examples of synthetic signal bases ( 2D feature maps ) . ( b ) Examples of the synthetic signal tensors by stacking 2D feature maps along the channel dimension for CNN architectures . whose evaluations are thus based on the word classification accuracy [ 50 , 51 , 52 ] . Following the same principle for CNNs , we design a many-to-many regression task for RNNs as shown in Fig . 2b . Instead of using the final word-classifier ’ s output , we extract the output tensor of the intermediate layer before it . Inputs . For a general RNN model , the input is a random tensor I ∈ Rl×b×d , where l is the sequence length , b is the batch size , and d is the length of input/output word vectors . Given a sequence of length l , the input to the RNN each time is one slice of the tensor I , denoted by I ( i ) ∈ Rb×d , 1 ≤ i ≤ l. Outputs . The output is F̂ ∈ Rl×b×d , where a slice of F̂ is F̂ ( i ) ∈ Rb×d . Groundtruth . Similar to the CNN case , we generate a synthetic signal tensor F∗ as the proxy task groundtruth .

Summary: This article introduces a novel low-cost regression proxy task, which has a high correlation with the test performance of the final model. Its contributions include the following:  1. Introducing a novel low-cost regression agent task, which greatly reduces the cost of evaluation. 2. This proxy task is basically independent of downstream tasks, and can make the searched model achieve approximately consistent results in multiple search spaces and datasets. 3. Competitive results can be achieved in End-to-End NAS.