NAS-Bench-360: Benchmarking Diverse Tasks for Neural Architecture Search

1 INTRODUCTION . Neural architecture search ( NAS ) aims to automate the design of deep neural networks , ensuring performance on par with hand-crafted architectures while reducing human labor devoted to tedious architecture tuning ( Elsken et al. , 2019 ) . With the growing number of application areas of ML , and thus of use-cases for automating it , NAS has experienced an intense amount of study , with significant progress in search space design ( Zoph et al. , 2018 ; Liu et al. , 2019b ; Cai et al. , 2019 ) , search efficiency ( Pham et al. , 2018 ) , and search algorithms ( Xu et al. , 2020 ; Li et al. , 2021a ; White et al. , 2021 ) . While the use of NAS techniques may be especially impactful in under-explored or under-resourced domains where less expert help is available , the field has largely been dominated by methods designed for and evaluated on benchmarks in computer vision ( Liu et al. , 2019b ; Ying et al. , 2019 ; Dong & Yang , 2020 ) . There have been a few recent efforts to diversify these benchmarks to settings such as vision-based transfer learning ( Duan et al. , 2021 ) and speech and language processing Mehrotra et al . ( 2021 ) ; Klyuchnikov et al . ( 2020 ) ; however , evaluating NAS methods on such wellstudied tasks using traditional CNN search spaces does not give a good indication of their utility on more far-afield applications , which have often necessitated the design of custom neural operations ( Cohen et al. , 2018 ; Li et al. , 2021b ) . We make progress towards studying NAS on more diverse tasks by introducing a suite of benchmark datasets drawn from various data domains that we collectively call NAS-Perf-360 . This benchmark consists of an organized setup of ten suitable datasets that ( a ) can be evaluated in a unified way using existing NAS approaches and ( b ) represent diverse application domains , dataset sizes , problem 1In this work , NAS method refers to a combined search space and algorithm pair , not the algorithm alone . dimensionalities , and learning objectives . We also include standard image classification evaluations as a baseline point of comparison , as many new methods continue to be designed for such tasks . Following our construction of this suite of tasks , we demonstrate both the usefulness of and need for NAS-Perf-360 by using it to investigate whether modern NAS is useful to practitioners faced with diverse tasks , i.e. , whether its success in computer vision is indicative of strong performance on the much broader set of problems to which NAS can conceivably be applied . To address this question , we start with the fact that a common first approach when applying deep learning to a new domain is to try an off-the-shelf CNN ; in our case , this will be the Wide ResNet ( WRN ) ( Zagoruyko & Komodakis , 2016 ) . We then consider the scenario of two practitioners : one with only the resources to train one WRN using the default settings and another that has enough to tune WRN using an off-the-shelf hyperparameter optimizer ( Li et al. , 2018 ) . Both are faced with a decision : should they use these fixed-architecture baselines or try out the best NAS has to offer ? Overall , our empirical investigation suggests the following : 1 . The less-constrained practitioner might usually do better using NAS—20 % relative improvement over WRN on the median task—but risks catastrophic results on specific non-vision applications . 2 . The robustness of NAS in the constrained case may be worse : the practitioner is likely better-off simply using the simple off-the-shelf WRN , as its median rank across NAS-Perf-360 ’ s ten tasks is the same as that of our candidate NAS method . These results are obtained via experiments using two well-studied modern search spaces : the cellbased DARTS space ( Liu et al. , 2019b ) and the efficiency-focused DenseNAS space ( Fang et al. , 2020 ) . Each space is paired with a search method known to find well-performing architectures on ImageNet , specifically the state-of-the-art GAEA PC-DARTS ( Li et al. , 2021a ) for the former and the original weight-sharing algorithm used by DenseNAS for the latter . Note that our assessment includes a more holistic comparison using performance profiles ( c.f . Figure 1 ) to reinforce these ranking-based comparisons , which are useful but can miss a lack of robustness or exaggerate minor differences between methods . The initial experimental results enabled by NAS-Perf-360 suggest that the robustness of modern search methods to diverse tasks beyond image classification is mixed at best . At the same time , our set of tasks can serve as a crucial tool for investigating and rectifying this issue , and it is thus important for moving towards a truly automated pipeline containing NAS . In particular , NAS-Perf-360 will facilitate such progress via a diverse array of tasks for validating NAS methods that are not only challenging , real-life problem settings but also computationally accessible for academic researchers with limited budgets . We demonstrate this potential via further studies on the comparative importance of search spaces v. search algorithms and the usefulness of more-customized approaches , specifically by studying a random search ( RS ) baseline over the DenseNAS space as well as two domain-specific methods : Auto-DeepLab ( Auto-DL ) ( Liu et al. , 2019a ) for dense prediction and AMBER ( Zhang et al. , 2021b ) for prediction from 1D data . Among other insights , these experiments provide evidence that a more robust NAS may require better search spaces with a wider variety of operations . The associated datasets and experiment code will remain open-source and accessible at a temporarily anonymized repository https : //anonymous.4open.science/r/ NAS-Bench-360-26D1 . Reproducibility of all experiments is assured from open-sourcing all relevant code for the end-to-end procedure , with Docker containers and random seeds provided . 2 RELATED WORK . Benchmarks have been critical to the development of NAS in recent years . This includes standard evaluation datasets and protocols , of which the most popular are the CIFAR-10 and ImageNet routines used by DARTS ( Liu et al. , 2019b ) . Another important type of benchmark has been tabular benchmarks such as NAS-Bench-101 ( Ying et al. , 2019 ) , NAS-Bench-201 ( Dong & Yang , 2020 ) , and NAS-Bench-1Shot1 ( Zela et al. , 2020 ) ; these benchmarks exhaustively evaluate all architectures in their search spaces , which is made computationally feasible by defining simple searched cells . Consequently , they are less expressive than the DARTS cell ( Liu et al. , 2019b ) , often regarded as the most powerful search space in the cell-based regime . Notably , our benchmark is not a tabular benchmark , i.e. , we do not evaluate every architecture from a fixed search space ; instead , the focus is on the organization of a suite of tasks for assessing both NAS algorithms and search spaces , which would necessarily be restricted by fixing a search space for a tabular benchmark . Pre-computing on an expansive search space such as DARTS , with 1018 possible architectures , is computationally intractable . Architectures found on lesser search spaces are most likely suboptimal : the vanilla WRN outperforms all networks in the NAS-Bench-201 search space on CIFAR-100 . While NAS methods and benchmarks have generally been focused on computer vision , recent work such as AutoML-Zero ( Real et al. , 2020 ) and XD-operations ( Roberts et al. , 2021 ) has started moving towards a more generically applicable set of tools for AutoML . However , even more recent benchmarks that do go beyond the most popular vision datasets have continued to focus on well-studied tasks , including vision-based transfer learning ( Duan et al. , 2021 ) , speech recognition ( Mehrotra et al. , 2021 ) , and natural language processing ( Klyuchnikov et al. , 2020 ) . We aim to go beyond such areas to evaluate the potential of NAS to automate the application of ML in truly under-explored domains . One analogous work to ours in the field of meta-learning is the Meta-Dataset benchmark of few-shot tasks ( Triantafillou et al. , 2020 ) , which similarly aimed to establish a wide-ranging set of evaluations for that field . For our inclusion of diverse tasks , we title our benchmark NAS-Perf-360 to resemble the idea of a 360-degree camera that covers all possible directions . 3 NAS-PERF-360 : A SUITE OF DIVERSE AND PRACTICAL TASKS . In this section , we introduce the NAS setting targeted by our benchmark , our motivation for organizing a new set of diverse tasks as a NAS evaluation suite , and our task-selection methodology . We report evaluations of specific algorithms on this new benchmark in the next section . 3.1 NEURAL ARCHITECTURE SEARCH : PROBLEM FORMULATION AND BASELINES . For completeness and clarity , we first formally discuss the architecture search problem itself , starting with the extended hypothesis class formulation Li et al . ( 2021a ) . Here the goal is to use a dataset of points x 2 X to find parameters w 2 W and a 2 A of a parameterized function fw , a : X 7 ! R 0 that minimize the expectation Ex⇠Dfw , a ( x ) for some test distribution D over X ; here X is the input space , W is the space of model weights , and A is the set of architectures . For generality , we do not require the training points to be drawn from D to allow for domain adaptation , as is the case for one of our tasks , and we do not require the loss to be supervised . Note also that the goal here does not depend on computational or memory efficiency , which we do not focus on in our evaluations ; our restriction is only that the entire pipeline can be run on an NVIDIA V100 GPU . Notably , this formulation makes no distinction between the model weights w and architectures a , treating both as parameters of a larger model . Indeed , the goal of NAS may be seen as similar to model design , except now we include the design of an ( often-discrete ) architecture space A such that it is easy to find an architecture a 2 A and model weights w 2 W whose test loss EDfw , a is low using a search algorithm . This can be done in a one-shot manner—simultaneously optimizing a and w—or using the standard approach of first finding an architecture a and then keeping it fixed while training model weights w using a pre-specified algorithm such as stochastic gradient descent ( SGD ) . This formulation also includes non-NAS methods by allowing the architecture search space to be a singleton . When the sole architecture is a standard and common network such as WRN ( Zagoruyko & Komodakis , 2016 ) , this yields a natural baseline with an algorithm searching for training hyperparameters , not architectures . On the other hand , any architecture space A allows for non-one-shot methods to search for architectures , such as random search through repeatedly sampling architectures and evaluating them from partial training . We adopt this simple method as our random baseline . For our empirical investigation , we compare the performance of state-of-the-art NAS approaches against that of the two baselines . 3.2 TASK SELECTION : MOTIVATION AND METHODOLOGY . Curating a diverse , practical set of tasks for the study of NAS is our primary motivation behind this work . We observe that past NAS benchmarks focused on creating larger search spaces and more sophisticated search methods for neural networks . However , the utility of these search spaces and methods are only evaluated on canonical computer vision datasets . On a broader range of problems , whether these new methods can improve upon simple baselines remains an open question . This calls for the introduction of new datasets lest NAS research overfits to the biases of CIFAR-10 and ImageNet . By identifying these possible biases , future directions in NAS research can be better primed to suit the needs of practitioners and to increase the deployment of NAS . Summarized in Table 1 , NAS-Perf-360 consists of problems that are conducive to processing by convolutional neural networks , which includes a trove of applications associated with spatial and temporal data , spanning single and multiple dimensions . Most current NAS methods are not implemented to search for other types of architectures to process tabular data and graph data . Therefore , we have set this scope for our investigation . During the selection of tasks , diversity is our primary consideration . We define the following axes of diversity to govern our task-filtering process : the first is problem dimensionality , including both 2D with matrix inputs and 1D with sequence inputs ; the second is dataset size , for which our selection spans the scale from 1,000 to 1,000,000 ; the third is problem type , divisible into tasks requiring a singular prediction ( point prediction ) and multiple predictions ( dense prediction ) ; fourth and finally , diversity is achieved through selecting tasks from various learning objectives from applications of deep learning , where introducing NAS could improve upon the performance of handcrafted neural networks . In lieu of providing raw data , we perform data pre-processing locally and store the processed data on a public Amazon Web Service ’ s S3 data bucket with download links available on our website . Our data treatment largely follows the procedure defined by the researchers who provided them . This enhances reproducibility by ensuring the uniformity of input data for different pipelines . Specific pre-processing and augmentation steps are described below . CIFAR-100 : Standard image classification As a starting point of comparison to existing benchmarks , we include the CIFAR-100 task ( Krizhevksy , 2009 ) , which contains RGB images from natural settings to be classified into 100 fine-grained categories . CIFAR-100 is preferred over CIFAR-10 because it is more challenging and suffers less from over-fitting in previous research . Spherical : Classifying spherically projected CIFAR-100 images To test NAS methods applied to natural-image-like data , we consider the task of classifying spherical projections of the CIFAR-100 images , which we call Spherical . In addition to scientific interest , spherical image data is also present in various applications , such as omnidirectional vision in robotics and weather modeling in meteorology , as sensors usually produce distorted image signals in real-life settings . To create Spherical CIFAR , we project the planar signals of the CIFAR images to the northern hemisphere and add a random rotation to produce spherical signals for each channel following the procedure specified in Cohen et al . ( 2018 ) . The resulting images are 60⇥60 pixels with RGB channels . NinaPro : Classifying electromyography signals NinaPro moves away from the image domain to classify hand gestures indicated by electromyography signals . For this , we use a subset of the NinaPro DB5 dataset ( Atzori et al. , 2012 ) in which two Myo armbands collect EMG signals from 10 test individuals who hold 18 different hand gestures to be classified . These armbands leverage data from muscle movement , which is collected using electrodes in the form of wave signals . Each wave signal is then sampled using a wavelength and frequency prescribed in Côté-Allard et al . ( 2019 ) to produce 2D signals . FSD50K : Labeling sound events FSD50K ( Fonseca et al. , 2020 ) is derived from the larger Freesound dataset ( Fonseca et al. , 2017 ) of Youtube videos with 51,000 clips totaling more than 100 hours of sound . These clips are manually labeled and equally distributed in 200 classes from the AudioSet ontology ( Gemmeke et al. , 2017 ) . Each clip could receive multiple labels . Unlike TIMIT ( Garofolo , 1993 ) , FSD50K does not focus exclusively on sounds of spoken language but includes sound events from physical sources and production mechanisms . The mean average precision ( mAP ) is used to evaluate classification results . Darcy Flow : Solving partial differential equations ( PDEs ) Our first regression task , Darcy Flow , focuses on learning a map from the initial conditions of a PDE to the solution at a later timestep . This application aims to replace traditional solvers with learned neural networks , which can output a result in a single forward pass . The input is a 2d grid specifying the initial conditions of a fluid , and the output is a 2d grid specifying the fluid state at a later time , with the ground truth being the result computed by a traditional solver . We report the mean square error ( MSE or ` 2 ) . PSICOV : Protein distance prediction PSICOV studies the use of neural networks in the protein folding prediction pipeline , which has recently received significant attention to the success of methods like AlphaFold ( Jumper et al. , 2020 ) . While the dataset and method they use are too large-scale for our purposes , we consider a smaller set of protein structures to tackle the specific problem of inter-residual distance predictions outlined in Adhikari ( 2020b ) . 2D large-scale features are extracted from protein sequences , resulting in input feature maps with a massive number of channels . Correspondingly , the labels are pairwise-distance matrices with the same spatial dimension . The evaluation metric is mean absolute error ( MAE or ` 1 ) computed on distances below 8 Å , referred to as MAE8 . Cosmic : Identifying cosmic ray contamination Images from space-based facilities are prone to corruption by charged particles collectively referred to as `` cosmic rays . '' Cosmic rays on images should be identified and masked before the images are used for further analysis ( Zhang & Bloom , 2020 ) . The Cosmic task uses imaging data of local resolved galaxies collected from the Hubble Space Telescope . Inputs and outputs are same-size 2D matrices , with the output predicting whether each pixel in the input is an artifact of cosmic rays . We report the false-negative rate ( FNR ) of identification results . ECG : Detecting heart disease Electrocardiograms are frequently used in medicine to diagnose sinus rhythm irregularities . The ECG task is based on the 2017 PhysioNet Challenge ( Clifford et al. , 2017 ) , with 9 to 60-second ECG recordings sampled at 300 Hz and labeled using four classes : normal , disease , other , or noisy rhythms . Recordings are processed using a fixed sliding window of 1,000 ms and stride of 500 ms. We report the F1-score according to the challenge ’ s guidelines . Satellite : Satellite image time series analysis Satellite image time series ( SITS ) are becoming more widely available in earth monitoring applications . Our dataset comes from Formosat-2 satellite images acquired over Toulouse , France ( Petitjean et al. , 2012 ) . Available in multiple channels , SITS track the land cover changes over several years as each pixel in the image represents a geographical region . The goal of the Satellite task is to generate land cover maps for geo-surveying . Specifically , a series of pixels in a given color channel constitute a time series to be classified into 46 land cover types . DeepSEA : Predicting functional effects from genetic sequences Predicting chromatin effects of genetic sequence alterations is a significant challenge in the field to understand genetic diseases . DeepSEA ( Zhou & Troyanskaya , 2015 ) , provides a compendium of genomic profiles from the Encyclopedia of DNA Elements ( ENCODE ) project ( Consortium et al. , 2004 ) to train a predictive model estimating the behavior of chromatin proteins , divided into 919 categories . Due to computation constraints , we subsample 36 of these categories as per Zhang et al . ( 2021a ) and further take 5 % of the training data for prediction . We report the area under the receiver operating characteristic ( AUROC ) following the previous work .

Summary: This paper proposes a benchmark to test the performance of NAS algorithms and search spaces on a diverse set of tasks. The benchmark consists of 10 different datasets across different modalities. On these tasks, a variety of NAS algorithm as well as search spaces are allowed a fixed amount of compute resources (in terms of GPU hours) to explore and train, and reach the final performance. The search space are not limited to architecture topologies, but also hyper-parameters. On this new benchmark, authors find that existing SoTA NAS methods may not generalize to different tasks, especially with low compute budgets.