Generic Neural Architecture Search via Regression

1 Introduction . Most existing neural architecture search ( NAS ) approaches aim to find top-performing architectures on a specific downstream task , such as image classification [ 1 , 2 , 3 , 4 , 5 ] , semantic segmentation [ 6 , 7 , 8 ] , neural machine translation [ 9 , 10 , 11 ] or more complex tasks like hardware-software codesign [ 12 , 13 , 14 , 15 , 16 ] . They either directly search on the target task using the target dataset ( e.g. , classification on CIFAR-10 [ 2 , 17 ] ) , or search on a proxy dataset and then transfer to the target one ( e.g . CIFAR-10 to ImageNet ) [ 18 , 3 ] . However , extensive experiments show that prominent neural architectures are generally good at extracting patterns from the input data and perform well to different downstream tasks . For example , ResNet [ 19 ] being a prevailing architecture in computer vision , shows outstanding performance across various datasets and tasks [ 20 , 21 , 22 ] , because of its 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . advantageous architecture , the residual blocks . This observation motivates us to ask the first question : Is there a generic way to search for and evaluate neural architectures without using the specific knowledge of downstream tasks ? Meanwhile , we observe that most existing NAS approaches directly use the final classification performance as the metric for architecture evaluation and search , which has several major issues . First , the classification accuracy is dominated by the samples along the classification boundary , while other samples have clearer classification outcomes compared to the boundary ones ( as illustrated in Fig . 1a ) . Such phenomena can be observed in the limited number of effective support vectors in SVM [ 23 ] , which also applies to neural networks because of the theory of neural tangent kernel [ 24 ] . Therefore , discriminating performance of classifiers needs many more samples than necessary ( the indeed effective ones ) , causing a big waste . Second , a classifier tends to discard a lot of valuable information , such as finer-grained features and spatial information , by transforming input representations into categorical labels . This observation motivates us to ask the second question : Is there a more effective way that can make more sufficient use of input samples and better capture valuable information ? To answer the two fundamental questions for NAS , in this work , we propose a Generic Neural Architecture Search method , termed GenNAS . GenNAS adopts a regression-based proxy task using downstream-task-agnostic synthetic signals for network training and evaluation . It can efficiently ( with near-zero training cost ) and accurately approximate the neural architecture performance . Insights . First , as opposed to classification , regression can efficiently make fully use of all the input samples , which equally contribute to the regression accuracy ( Fig . 1b ) . Second , regression on properly-designed synthetic signals is essentially evaluating the intrinsic representation power of neural architectures , which is to capture and distinguish fundamental data patterns that are agnostic to downstream tasks . Third , such representation power is heavily reflected in the intermediate data of a network ( as we will show in the experiments ) , which are regrettably discarded by classification . Approach . First , we propose a regression proxy task as the supervising task to train , evaluate , and search for neural architectures ( Fig . 2 ) . Then , the searched architectures will be used for the target downstream tasks . To the best of our knowledge , we are the first to propose self-supervised regression proxy task instead of classification for NAS . Second , we propose to use unlabeled synthetic data ( e.g. , sine and random signals ) as the groundtruth ( Fig . 3 ) to measure neural architectures ’ intrinsic capability of capturing fundamental data patterns . Third , to further boost NAS performance , we propose a weakly-supervised automatic proxy task search with only a handful of groundtruth architecture performance ( e.g . 20 architectures ) , to determine the best proxy task , i.e. , the combination of synthetic signal bases , targeting a specific downstream task , search space , and/or dataset ( Fig . 4 ) . GenNAS Evaluation . The efficiency and effectiveness of NAS are dominated by neural architecture evaluation , which directs the search algorithm towards top-performing network architectures . To quantify how accurate the evaluation is , one widely used indicator is the network performance Ranking Correlation [ 25 ] between the prediction and groundtruth ranking , defined as Spearman ’ s Rho ( ρ ) or Kendall ’ s Tau ( τ ) . The ideal ranking correlation is 1 when the approximated and groundtruth rankings are exactly the same ; achieving large ρ or τ can significantly improve NAS quality [ 26 , 27 , 28 ] . Therefore , in the experiments ( Sec . 4 ) , we evaluate GenNAS using the ranking correlation factors it achieves , and then show its end-to-end NAS performance in finding the best architectures . Extensive experiments are done on 13 CNN search spaces and one NLP space [ 29 ] . Trained by the regression proxy task using only a single batch of unlabeled data within a few seconds , GenNAS significantly outperforms all existing NAS approaches on almost all the search spaces and datasets . For example , GenNAS achieves 0.87 ρ on NASBench-101 [ 30 ] , while Zero-Cost NAS [ 31 ] , an efficient proxy NAS approach , only achieves 0.38 . On end-to-end NAS , GenNAS generally outperforms others with large speedup . This implies that the insights behind GenNAS are plausible and that our proposed regression-based task-agnostic approach is generalizable across tasks , search spaces , and datasets . Contributions . We summarize our contributions as follows : • To the best of our knowledge , GenNAS is the first NAS approach using regression as the self- supervised proxy task instead of classification for neural architecture evaluation and search . It is agnostic to the specific downstream tasks and can significantly improve training and evaluation efficiency by fully utilizing only a handful of unlabeled data . • GenNAS uses synthetic signal bases as the groundtruth to measure the intrinsic capability of networks that captures fundamental signal patterns . Using such unlabeled synthetic data in regression , GenNAS can find the generic task-agnostic top-performing networks and can apply to any new search spaces with zero effort . • An automated proxy task search to further improve GenNAS performance . • Thorough experiments show that GenNAS outperforms existing NAS approaches by large margins in terms of ranking correlation with near-zero training cost , across 13 CNN and one NLP space without proxy task search . GenNAS also achieves state-of-the-art performance for end-to-end NAS with orders of magnitude of speedup over conventional methods . • With proxy task search being optional , GenNAS is fine-tuning-free , highly efficient , and can be easily implemented on a single customer-level GPU . 2 Related Work . NAS Evaluation . Network architecture evaluation is critical in guiding the search algorithms of NAS by identifying the top-performing architectures , which is also a challenging task with intensive research interests . Early NAS works evaluated the networks by training from scratch with tremendous computation and time cost [ 18 , 1 ] . To expedite , weight-sharing among the subnets sampled from a supernet is widely adopted [ 3 , 28 , 4 , 32 , 33 ] . However , due to the poor correlation between the weight-sharing and the final performance ranking , weight-sharing NAS can easily fail even in simple search spaces [ 34 , 35 ] . Yu et al . [ 36 ] further pointed out that without accurate evaluation , NAS runs in a near-random fashion . Recently , zero-cost NAS methods [ 37 , 31 , 38 , 39 ] have been proposed , which score the networks using their initial parameters with only one forward and backward propagation . Despite the significant speed up , they fail to identify top-performing architectures in large search spaces such as NASBench-101 . To detach the time-consuming network evaluation from NAS , several benchmarks are developed with fully-trained neural networks within the NAS search spaces [ 40 , 30 , 35 , 29 , 41 ] , so that researchers can assess the search algorithms alone in the playground . NAS Transferability . To improve search efficiency , proxy tasks are widely used , on which the architectures are searched and then transferred to target datasets and tasks . For example , the CIFAR10 classification dataset seems to be a good proxy for ImageNet [ 18 , 3 ] . Kornblith et al . [ 42 ] studied the transferability of 16 classification networks on 12 image classification datasets . NASBench201 [ 35 ] evaluated the ranking correlations across three popular datasets with 15625 architectures . Liu et al . [ 43 ] studied the architecture transferability across supervised and unsupervised tasks . Nevertheless , training on a downsized proxy dataset is still inefficient ( e.g . a few epochs of full-blown training [ 43 ] ) . In contrast , GenNAS significantly improves the efficiency by using a single batch of data while maintaining extremely good generalizability across different search spaces and datasets . Self-supervised Learning . Self-supervised learning is a form of unsupervised learning , that the neural architectures are trained with automatically generated labels to gain a good degree of com- prehension or understanding [ 44 , 45 , 46 , 47 , 43 ] . Liu et al . [ 43 ] recently proposed three unlabeled classification proxy tasks , including rotation prediction , colorization , and solving jigsaw puzzles , for neural network evaluation . Though promising , this approach did not explain why such manually designed proxy tasks are beneficial and still used classification for training with the entire dataset . In contrast , GenNAS uses regression with only a single batch of synthetic data . 3 Proposed GenNAS . In Section 3.1 , we introduce the main concepts of task-agnostic GenNAS : 1 ) the proposed regression proxy task for both CNN architectures and recurrent neural network ( RNN ) architectures ; 2 ) the synthetic signal bases used for representing the fundamental data patterns as the proxy task . In Section 3.2 , we introduce the automated proxy task search . 3.1 GenNAS . 3.1.1 Regression Architectures . Training using unlabeled regression is the key that GenNAS being agnostic to downstream tasks . Based on the insights discussed in Section 1 , the principle of designing the regression architecture is to fully utilize the abundant intermediate information rather than the final classifier . Regression on CNNs . Empirical studies show that CNNs learn fine-grained high-frequency spatial details in the early layers and produce semantic features in the late layers [ 48 ] . Following this principle , as shown in Fig . 2a , we construct a Fully Convolutional Network ( FCN ) [ 49 ] by removing the final classifier of a CNN , and then extract the FCN ’ s intermediate feature maps from multiple stages . We denote the number of stages as N . Inputs . The inputs to the FCN are unlabeled real images , shaped as a tensor I ∈ Rb×3×h×w , where b is the batch size , and h and w are the input image size . Outputs . From each stage i ( 1 ≤ i ≤ N ) of the FCN , we first extract a feature map tensor , denoted by Fi ∈ Rb×ci×hi×wi , and reshape it as F̂i ∈ Rb×c ′ i×h ′×w′ through a convolutional layer Mi by F̂i = Mi ( Fi ) ( with downsampling if wi > w′ or hi > h′ ) . The outputs are the tensors F̂i , which encapsulate the captured signal patterns from different stages . Groundtruth . We construct a synthetic signal tensor for each stage as the groundtruth , which serves as part of the proxy task . A synthetic tensor is a combination of multiple synthetic signal bases ( more details in Section 3.1.2 ) , denoted by F∗i . We compare F̂i with F∗i for training and evaluating the neural architectures . During training , we use MSE loss defined as L = ∑N i=1 E [ ( F∗i − F̂i ) 2 ] ; during validation , we adjust each stage ’ s output importance as L = ∑N i=1 1 2N−i E [ ( F∗i − F̂i ) 2 ] since the feature map tensors of later stages are more related to the downstream task ’ s performance . The detailed configurations of N , h′ , w′ , and c′i are provided in the experiments . Regression on RNNs . The proposed regression proxy task can be similarly applied to NLP tasks using RNNs . Most existing NLP models use a sequence of word-classifiers as the final outputs , Sin1D Sin2D Dot GDot ( a ) Examples of signal bases … Sin2D Global Dot Local Combination … … Input Batch One stacked synthetic signal tensor for one batch of data One channel ( b ) Examples of synthetic signal tensors Figure 3 : ( a ) Examples of synthetic signal bases ( 2D feature maps ) . ( b ) Examples of the synthetic signal tensors by stacking 2D feature maps along the channel dimension for CNN architectures . whose evaluations are thus based on the word classification accuracy [ 50 , 51 , 52 ] . Following the same principle for CNNs , we design a many-to-many regression task for RNNs as shown in Fig . 2b . Instead of using the final word-classifier ’ s output , we extract the output tensor of the intermediate layer before it . Inputs . For a general RNN model , the input is a random tensor I ∈ Rl×b×d , where l is the sequence length , b is the batch size , and d is the length of input/output word vectors . Given a sequence of length l , the input to the RNN each time is one slice of the tensor I , denoted by I ( i ) ∈ Rb×d , 1 ≤ i ≤ l. Outputs . The output is F̂ ∈ Rl×b×d , where a slice of F̂ is F̂ ( i ) ∈ Rb×d . Groundtruth . Similar to the CNN case , we generate a synthetic signal tensor F∗ as the proxy task groundtruth .

Summary: In this paper, the authors propose a generic NAS method named GenNAS, which uses regression as the self-supervised task. Moreover, the proposed method is task-agnostic and highly efficient. Experiments demonstrate the effectiveness of the proposed method. My detailed comments are as follows.   Positive points: 1.	The authors proposed a new regression-based NAS method.  2.	The results from experiments show the proposed method achieves promising performance.