NAS-Bench-360: Benchmarking Diverse Tasks for Neural Architecture Search
Authors: Renbo Tu
Mikhail Khodak
Nicholas Carl Roberts
Ameet Talwalkar
Venue: ICLR 2022 Submitted (2022)
Keywords: automated machine learning
neural architecture search

Abstract: Most existing neural architecture search (NAS) benchmarks and algorithms prioritize performance on well-studied tasks, e.g., image classification on CIFAR and ImageNet. This makes the applicability of NAS approaches in more diverse areas inadequately understood.
In this paper, we present NAS-Bench-360, a benchmark suite for evaluating state-of-the-art NAS methods for convolutional neural networks (CNNs). To construct it, we curate a collection of ten tasks spanning a diverse array of application domains, dataset sizes, problem dimensionalities, and learning objectives. By carefully selecting tasks that can both interoperate with modern CNN-based search methods but that are also far-afield from their original development domain, we can use NAS-Bench-360 to investigate the following central question: do existing state-of-the-art NAS methods perform well on diverse tasks? Our experiments show that a modern NAS procedure designed for image classification can indeed find good architectures for tasks with other dimensionalities and learning objectives; however, the same method struggles against more task-specific methods and performs catastrophically poorly on classification in non-vision domains. The case for NAS robustness becomes even more dire in a resource-constrained setting, where a recent NAS method provides little-to-no benefit over much simpler baselines. These results demonstrate the need for a benchmark such as NAS-Bench-360 to help develop NAS approaches that work well on a variety of tasks, a crucial component of a truly robust and automated pipeline. We conclude with a demonstration of the kind of future research our suite of tasks will enable. All data and code is made publicly available.

Under review as a conference paper at ICLR 2022
NAS-PERF-360: J DIVERSE TASKS FOR NEURAL ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Most existing neural architecture search (NAS) benchmarks and algorithms priori- tize performance on well-studied tasks, e.g., image classification on CIFAR and ImageNet. This makes the applicability of NAS approaches in more diverse areas inadequately understood. In this paper, we present NAS-Perf-366 a performance benchmark suite for statc-of-the tar NAS methods for convolutional neural net- works (CNNs) !1 To construct it, we curate collection of ten tasks spanning a diverse array of application domains, dataset sizes, problem dimensionalities, and learning objectives. By carefully selecting tasks that can both interoperate with modern CNN-based search methods but that are also far-afield from their original development domain, we can use NAS-Perf Perf-360 to investigate the following cen- tral question: do existing sfateoof-the-ar NAS methods perform well on diverse tasks? Our experiments show that a modern NAS procedure designed for image classification can indeed find good architectures for tasks with other dimension- alities and learning objectives; however, the same method struggles against more task-specific methods and performs catastrophically poorly classification in non-vision domains. The case for NAS robustness becomes even more dire in a resource-constrained setting, where recent NAS method provides little-to-no benefit over much simpler baselines. These results demonstrate the need for a performance benchmark such as NAS-Perf-360 that work well on a variety of tasks, crucial component of truly robust and auto- mated pipeline. We conclude with a demonstration of the kind of future research our suite of tasks will enable. All data and code is made publicly available.
help develop NAS approaches
INTRODUCTION
Neural architecture search (NAS) aims to automate the design of deep neural networks, ensuring performance par with architecture tuning (Elsken et al. 2019). With the growing number of application areas of ML,
architectures while reducing human labor devoted to tedious
and thus of use-cases for automating it, NAS has experienced an intense amount of study, with
significant progress in search space design Zoph et al. 2018 Liu et al. 2019b Cai et al. 2019). ) search efficiency (Pham et al. 2018). and search algorithms (Xu et al. 2020 Li al. 2021a White jet al. 2021). While the use of NAS techniques may be especially impactful in under-explored or
under-resourced domains where less expert help is available, the field has largely been dominated by methods designed for and evaluated on benchmarks in computer vision (Liu et al. 2019b Ying et al. 2019 Dong & Yang 2020). There have been a few recent efforts diversify these benchmarks to settings such as vision- based transfer learning (Duan et al. 2021) and speech and language processing Mehrotra al. (2021): Klyuchnikov et al. (2020); however, evaluating NAS methods on such well- studied tasks using traditional CNN search spaces does not give a good indication of their utility on more far-afield applications, which have often necessitated the design custom neural operations
(Cohen et al. 2018 Li et al. 2021b).
We make progress towards studying NAS on more diverse tasks by introducing suite of benchmark datasets drawn from various data domains that we collectively call NAS-Perf-360. This benchmark consists of an organized setup of ten suitable datasets that (a) can be evaluated in a unified way using existing NAS approaches and (b) represent diverse application domains, dataset sizes, problem
In this work, NAS method refers combined search space and algorithm pair, the algorithm alone.

---

Under review as a conference paper at ICLR 2022
refemmnne fo the cessraree Sctting
perforonce modles for the Los co staieed setting
DenseNAS WRN (default)
DARTS (GAEA)
WRN (tuned)
Figure 1:
Performance profiles for two settings on all ten tasks in NAS-Perf-360. The y-value indicates the fraction of tasks which plotted method's error is within multiplicative factor T of the lowest error achieved by all plotted methods..
dimensionalities, and learning objectives. We also include standard image classification evaluations baseline point of comparison, many new methods continue to be designed for such tasks.
Following our construction of this suite of tasks, we demonstrate both the usefulness and need for NAS-PerI-360 by using it to investigate whether modern NAS is useful to practitioners faced with diverse tasks, i.e., whether its success in computer vision is indicative of strong performance on the much broader set of problems to which NAS can conceivably be applied. To address this question, start with the fact that a common first approach when applying deep learning to new domain is to try off-the-shelff CNN; in our case, this will be the Wide ResNet (WRN) Zagoruyko & Komodakis) 2016 We then consider the scenario of two practitioners: one with only the resources to train one WRN using the default settings and another that has enough tune WRN using an
hyperparameter optimizer (Li et al. 2018). Both are faced with a decision: should they use these
fixed-achitectture baselines or try out the best NAS has to offer?
Overall, our empirical investigation suggests the following:
1. The less-constrained practitioner might usually do better using NAS- relative improvement over WRN on the median task-but but risks catastrophic results on specific non-vision applications.
2. The robustness of NAS in the constrained case may be worse: the practitioner is likely better-off' simply using the simple off-the-shelf WRN, as its median rank across NAS-Perf-360's tasks is the same as that of our candidate NAS method.
These results are obtained via experiments using two well-studied modern search spaces: the cell- based DARTS space (Liu et al. 2019b) and the efficiency-focused DenseNAS space (Fang et al.) 2020). Each space is paired with search method known to find well-performing architectures ImageNet, specifically the state-of-the-art GAEA PC-DARTS (Li et al. 2021a) for the former and the original weight-sharing algorithm used by DenseNAS for the latter. Note that assessment includes a more holistic comparison using performance profiles (c.f. Figure I to reinforce these ranking-based comparisons, which are useful but can miss a lack of robustness or exaggerate minor differences between methods.
The initial experimental results enabled by NAS-Perf-360 suggest that the robustness of modern search methods to diverse tasks beyond image classification is mixed at best. At the same time, our set of tasks can serve as a crucial tool for investigating and rectifying this issue, and it is thus important for moving towards a truly automated pipeline containing NAS. In particular, NAS-Perf-B facilitate such progress via diverse array of tasks for validating NAS methods that are not only challenging. real-life problem settings but also computationally accessible for academic researchers with limited budgets. We demonstrate this potential via further studies on the comparative importance of search spaces v. search algorithms and the usefulness of more-customizes approaches, specifically by studying a random search (RS) baseline over the DenseNAS space as well as two domain specifcc methods: -eet-DDeeLaab Auto-DL)(Liu et al.] (019a) for dense prediction and AMBER ZZang şt al. 2021b) for prediction from ID data. Among other insights, these experiments provide evidence that robust NAS may require better search spaces with wider variety of operations.
will
2

---

Under review as a conference paper at ICLR 2022
The associated datasets and experiment code will remain open-source and accessi- 4open. science//iee////ee
ble at a temporarily anonymized repository Itttps://anonynnn NAS-Bench- 3 Reproducibility of all experiments is assured from open-soureing all relevant code for the end-to-end procedure, with Docker containers and random seeds provided.
2 RELATED WORK
Benchmarks have been critical to the development of NAS in recent years. This includes standard evaluation datasets and protocols, of which the most popular are the CIFAR-10 10 and ImageNet
routines used by DARTS (Liu et al., 2019b). Another important type of benchmark has been tabular
benchmarks such NAS-Bench- 101 (Ying et al. 2019). NAS-Bench-201 (Dong & Yang 2020), and NAS- WASS-BenchIae Shotl (Zel et al. 2020); these benchmarks exhaustively evaluate architectures in their search spaces, which is made computationally feasible by defining simple searched cells. Consequently, they are less expressive than the DARTS cell (Liu et al. 2019b), often regarded as the most powerful search space in the cell-based regime. Notably, benchmark, i.e., we do not evaluate every architecture from a fixed search space; instead, the focus is on the organization of suite of tasks for assessing both NAS algorithms and search spaces, which would necessarily be restricted by fixing a search space for tabular benchmark. Pre-computing on an expansive search space such as DARTS, with 101s possible architectures, computationally intractable. Architectures found on lesser search spaces are most likely suboptimal: the vanilla WRN outperforms all networks in the MAS-Bench-200 search space on CIFAR-100.
benchmark is not tabular
While NAS methods and benchmarks have generally been focused on computer vision, recent work such as AutoMI- (Real et al. 2020) and XID-operations (Roberts et al. 2021 has started moving towards a more generically applicable set of tools for AutoML. However, even more recent bench- marks that do go beyond the most popular vision datasets have continued to focus on well-studied tasks, including vision-based transfer learning (Duan et al., (02I), speech recognition (Mehrotral et al. 2021). and natural language processing (Klycchnikov et al. (2200). We aim to go beyond such areas evaluate the potential of NAS to automate the application of ML in truly under-explored domains. One analogous work to ours in the field of meta-learning is the Meta-Datase benchmark of few-shot tasks (Triantffilluu et al. 2020), which similarly aimed establish wide-ranging set of evaluations for that field. For our inclusion of diverse tasks, we title our benchmark NAS-Perf-360 to resemble the idea of a 360-degree camera that covers all possible directions.
SUITE OF DIVERSE AND PRACTICAL TASKS
In this section, we introduce the NAS setting targeted by our benchmark, our motivation for organizing a new set diverse tasks as a NAS evaluation suite, and our task-selection methodology. We report evaluations of specific algorithms on this new benchmark in the next section.
3.1 NEURAL ARCHITECTURE SEARCH: PROBLEM FORMULATION AND BASELINES
For completeness and clarity, we first formally discuss the architecture search problem itself, starting with the extended hypothesis class formulation Li et al. (021a). Here the goal is to use dataset of points x E X' to find parameters E W and A of parameterized function fw. X + Rzo (oo.)ff.a(()) for some test distribution D over X;; here X' is the imput
that minimize the expectation space, W the space of model weights, and AA is the set of architectures. For generality, we do not require the training points to be drawn from D to allow for domain adaptation, as is the case for one of our tasks, and we do not require the loss to be supervised. Note also that the goal here does not depend computational or memory efficiency, which we do not focus in evaluations; restriction is only that the entire pipeline can be run on an NVIDIA V100 GPU.
Notably, this formulation makes distinction between the model weights and architectures treating both as parameters model design, except we include the design of an (often-discrete) architecture space AA such that it is easy to find an architecture e A and model weights w e W whose test loss Epf is low using a search algorithm. This can be done in one-shot manner- simuttannnuul optimizing a and
larger model. Indeed, the goal of NAS may be seen as similar to
3

---

Under review as a conference paper at ICLR 2022
or-or using the standard approach of first finding architecture and then keeping it fixed while training model weights w using a pre-specified algorithm such as stochastic gradient descent (SGD).
This formulation also includes non-NAS methods by allowing the architecture search space to be a singleton. When the sole architecture is a standard and common network such as WRN ZZgoruyko & Komodakis] 2016), this yields natural baseline with algorithm searching for training hyperparameters, not architectures. On the other hand, any architecture space A allows for non-one-shot methods to search for architectures, such as random search through repeatedly sampling architectures and evaluating them from partial training. We adopt this simple method as our random baseline. For our empirical investigation, we compare the performance of state-of-tht of-thh--arr NAS approaches against that of the two baselines.
3.2
TASK SELECTION: MOTIVATION AND METHODOLOGY
Curating a diverse, practical set of tasks for the study of NAS our primary motivation behind this work. We observe that past NAS benchmarks focused creating larger search spaces and more sophisticated search methods for neural networks. However, the utility of these search spaces and methods are only evaluated on canonical computer vision datasets. On broader range of problems, whether these new methods can improve upon simple baselines remains an open question. This calls for the introduction of new datasets lest NAS research overfits to the biases of CIFAR-10 and ImageNet. By identifying these possible biases, future directions in NAS research can be better primed suit the needs of practitioners and to increase the deployment of NAS.
Summarized in Table D NAS-Perf-360 consists of problems that are conducive to processing by convolutional neural networks, which includes trove of applications associated with spatial and temporal data, spanning single and multiple dimensions. Most current NAS methods are not imple- mented to search for other types of architectures to process tabular data and graph data. Therefore, we have set this scope for our investigation. During the selection of tasks, diversity is our primary consideration. We define the following axes diversity to govern our fask-filtrring process: the first is problem dimensionality, including both 2D with matrix inputs and ID with sequence inputs; the second is dataset size, for which our selection spans the scale from 1,000 to 1,000,000; the third is problem type. , divisible into tasks requiring singular prediction (point prediction) and multiple predictions (dense prediction); fourth and finally, diversity is achieved through selecting tasks from various learning objectives from applications of deep learning, where introducing NAS could improve upon the performance of handerafted neural networks.
In lieu of providing raw data, we perform data pre-prooe processein locally and store the processed data on a public Amazon Web Service's S3 data bucket with download links available on our website. Our data treatment largely follows the procedure defined by the researchers who provided them. This enhances reproducibility by ensuring the uniformity of input data for different pipelines. Specific pre- pe-pprocessing and augmentation steps are described below.
Standard image classification As a starting point of comparison to existing bench- marks, we include the CIFAR- 1-FAR-100 100 task (Krizhevksy) 2009), which contains RGB images from natural settings to be classified into 100 fine-grained categories. CIFAR-100 is preferred over CIFAR-10 because it is more challenging and suffers less from over-fitting in previous research.
CIFAR-100:
Spherical:
Classifying spherically projected CIFAR-100 images To test NAS methods applied to natural-image-like data, we consider the task of classifying spherical projections of the CIFAR-100 images, which we call Spherical. In addition to scientific interest, spherical image data is also present in various applications, such as omnidirectional vision in robotics and weather modeling in meteorology, as sensors usually produce distorted image signals in real-life settings. To create Spherical CIFAR, we project the planar signals of the CIFAR images to the northern hemisphere and add random rotation to produce spherical signals for each channel following the procedure specified in Cohen et al. 2018). The resulting images are x0 pixels with RGB channels.
NinaPro: Classifying electromyography signals NinaPro moves away from the image domain
to classify hand gestures indicated by electromyography signals. For this, we use a subset of the NinaPro DB5 dataset (Atzori et al. 2012) in which two Myo armbands collect EMG signals from 10 test individuals who hold 18 different hand gestures be classified. These armbands leverage data
4

---

Under review as a conference paper at ICLR 2022
Table I: Information of tasks in NAS-Perf-360
Task name
Size
Dim.
Type
Learning objective
New to NAS
CIFAR-100
60K
2D
Point
Classify natural images into 100 classes
Classify spherically projected images into 100 classes
Spherical
60K
2D
Point
Classify sEMG signals into 18 classes corresponding hand gestures
NinaPro
3956
2D
Point
Point
Classify sound events in log-mel spectrograms with labels
FSDSOK
51K
2D
(multi-label)
Predict the final state of fluid from its initial conditions
Darey Flow
1100
2D
Dense
Predict pairwise distances between resi- duals from 2D protein sequence features
PSICOV
3606
2D
Dense
Predict propablistic maps identify cos- mic rays in telescope images
Cosmic
5250
2D
Dense
Detecting atrial cardiac disease from a ECG recording via classification
ECG
330K
1D
Point
Classify satellite image pixels' time series into 24 land cover types
Satellite
IM
ID
Point
Point
Predicting chromatin states and binding states of RNA-binding sequences
DeepSEA
250K
1D
(multi-label)
from muscle movement, which is collected using electrodes in the form of wave signals. Each wave signal is then sampled using a wavelength and frequency prescribed in Côté-Allard al. (2019) to produce 2D signals.
FSD50K: Labeling sound events FSD50K (oonseca et al., 2020) is derived from the larger Freesound dataset (Fonseca et al.) 2017) of Youtube videos with 51,000 clips totaling more than 100 hours of sound. These clips manually labeled and equally distributed in 200 classes from the AudioSet ontology (Gemmeke et al. 2017). Each clip could receive multiple labels. Unlike TIMIT GGaoofolo 1993), FSD50K does not focus exclusively sounds of spoken language but includes sound events from physical sources and production mechanisms. The mean average precision (mAP) is used to evaluate classification results.
Darey Flow: Solving partial differential equations (PDEs) Our first regression task, Darcy Flow, focuses on learning a map from the initial conditions of a PDE to the solution at a later timestep. This application aims to replace traditional solvers with learned neural networks, which can output result in a single forward pass. The input is 2d grid specifying the initial conditions of fluid, and the output is 2d grid specifying the fluid state later time, with the ground truth being the result computed by traditional solver. We report the mean square error (MSE or E2).
PSICOV: Protein distance prediction PSICOV studies the use of neural networks in the protein folding prediction pipeline, which has recently received significant attention to the success of methods like AlphaFold (Jumper et al. 2020). While the dataset and method they
too large-scale for purposes, we consider a smaller set of protein structures to tackle the specific problem of inter-residual distance predictions outlined in Adhikari (220b). 2D large-scale features are extracted from protein sequences, resulting in input feature maps with a massive number channels. Correspondingly, the labels pairwise-dae -istiine- matrices with the same spatial dimension. The evaluation metric mean absolute eитor (MAE & computed distances below 8 Å, referred as MAEg
Cosmie: Identifying cosmic ray contamination Images from space-based facilities are prone to corruption by charged particles collectively referred to as "cosmic rays." Cosmic rays on images should be identified and masked before the images used for further analysis (Zhang & Bloom
5

---

Under review as a conference paper at ICLR 2022
Table 2: Performance of NAS and the WRN baselines across the tasks of NAS-Perf-360. Methods are divided into efficient methods (DenseNAS and fixed WRN) that take 1-10 GPU-hours, more expensive methods (DARTS and WRN tuned by ASHA) that take 10- 100+ GPU-hours, and specialized methods (Auto-DL and AMBER). All results are averages of three random seeds.
Scarch
Search
CIFAR-100
Spherical 0-1 error
Darcy Flow
PSICOV MAEs
Cosmic FNR
space
algorithm
0-1 error
relative &
WRN
default
23.35:10.05
85.7710.71
0.073110.001
3.84:10.053
51.76:12.09
DenseNAS
random
25.49.10.41
71.23:1..65
0.07111.006
3.70:10.06
70.42:16.07
DenseNAS
original
25.98.100.8
72.99110.95
0.10110.01
3.84110.15
79.52:12.20
WRN
ASHA
23.39:+0.01
75.46=+0:4
0.066+00000
3.84+0005
37.53+
DARTS
GAEA
24.02.11.92
48.23
0.026.10.001
2.94.10.13
31.1513.448
Auto-DL
DARTS
n/a
0.049+0.005
6.73++.73
99.79=0.02
Search
Search
NinaPro
FSD50K
ECG
Satellite
DeepSEA
space
algorithm
0-1 error"
mAP
Fl score
0-1 error"
AUROC
WRN
default
6.78110.26
0.08100001
0.57110.01
15.4911.:33
0.60.10.001
DenseNAS
random
8.45+00.56
0,40+000001
0.58+0001
13.91=0.13
0.60+00001
DenseNAS
original
10.17:11.31
0.36+0.002
0.60+10.01
13.81=0.69
0.60+0.001
WRN
ASHA
7.34:10.76
0.09.10.03
0.57
15.84.10.52
0.59100.002
DARTS
GAEA
17.67:11.39
0.06.10.02
0.66110.01
12.51
0.64:10.02
AMBER
ENAS
n/a
n/a
0.67#0.015
12.97=0.07
0.68+0.01
h/t a higher / lower value the metric indicates better performance.
2020). The Cosmic task uses imaging data of local resolved galaxies collected from the Hubble Space Telescope. Inputs and outputs are same-size 2D matrices, with the output predicting whether each pixel in the input is an artifact of cosmic rays. We report the false-negative rate (FNR) of identification results.
ECG: Detecting heart disease Electrocardiograas are frequently used in medicine diagnose sinus rhythm irregularities. The ECG task is based on the 2017 PhysioNet Challenge (Clifford et al.) 2017). with 9 to 60-second ECG recordings sampled at 300 Hz and labeled using four classes: normal, disease, other, or noisy rhythms. Recordings are processed using a fixed sliding window of 1,000 ms and stride of 500 ms. We report the Fl-score according the challenge's guidelines.
Satellite: Satellite image time series analysis Satellite image time series (SITS) are becoming more widely available earth monitoring applications. Our dataset comes from Formosat-2 satellite images acquired over Toulouse, France (Petitjean et al. 2012). Available in multiple channels, SITS track the land cover changes over several years as each pixel in the image represents a geographical region. The goal of the Satellite task is to generate land cover maps for geo-surveying. Specifically, a series of pixels in a given color channel constitute a time series to be classified into 46 land cover types.
DeepSEA: Predicting functional effects from genetic sequences Predicting chromatin effects of genetic sequence alterations a significant challenge the field to understand genetic diseases. DeepSEA (/hou & Troyanskaya 2015). provides compendium of genomic profiles from the Encyclopedia of DNA Elements (ENCODE) project (Consortium et al. 2004) to train a predictive model estimating the behavior of chromatin proteins, divided into 919 categories. Due to computation constraints, we subsample 36 of these categories as per Zhang et al. (2021a) and further take 5% of the training data for prediction. We report the area under the receiver operating characteristic (AUROC) following the previous work.
4 EXPERIMENTAL DESIGN
Having detailed our construction of strate its usefulness for evaluating NAS methods and guiding research diverse tasks. In this section,
we now describe a set of experiments to demon-

---

Under review as a conference paper at ICLR 2022
we first specify the different NAS methods and baselines compare, followed by the experimental and reproducibility setup we follow. The resulting evaluations are reported in Table aggregate performance in Table and performance profiles in Figure 2
4.1 BASELINES AND SEARCH PROCEDURES
As noted in Section D our initial experiments focus on two practitioners with different resource settings, one with enough compute to tune a WRN and another who can only train it once with the default hyperparameters. In matching these settings, focus two well-known search paradigms: cell-based NAS (using DARTS (Liu et al. 2 and macro NAS (using DenseNAS (Fang et al.) 20200). We further compare these approaches tWO customized NAS methods: Auto-Deepl.ab (Liu et al. 2019a) for 2D dense prediction and AMBER (Zhang et al. 202 for 1D prediction. We detail
these approaches below.
Wide ResNet with Hyperparameter Tuning Architectures based on ResNet He et al. common first choice by practitioners faced with a new domain (Fonseca et al. 2020 Adhikari 2020b); it is thus a natural source of fixed- -rrhitcciee baselines for our study. We use the Wide ResNet variant Zagoruyko & Komodakis 2016) with 16 layers and a widen factor of 4, and apply its original training routine directly for the constramed practitioner. For the other practitioner, we wrap the training procedure with a hyperparameter tuner, ASHA (Li et al., 2018), an asynchronous version of Hyperband (Li et al. 2017). Given a range for each hyperparameter, ASHA uniformly samples configurations and uses brackets of climination: at each round, each configuration is trained for some epochs, before the algorithm selects the best-performing portion based on validation metrics. This procedure is useful for finding suitable hyperparameters in an easy-to-use, automated fashion.
2016) are
Cell-based Search Using DARTS The first NAS paradigm we consider is cell-based NAS. These methods first search for a genotype, a cell containing neural operations. During evaluation, an architecture is constructed by replicating the searched cell and stacking them together. The most popular search space for this approach is DARTS (Liu et al., 2019b), which assigns one of eight operations six edges in two types of cells: "normal" cells preserve the shape of the input while "reduction" cells downsample it. For dense tasks, we do not use the reduction cell to prevent introducing bottleneck. For ID tasks, all convolutions in the cell are converted from 2D to ID. Finally, to adhere to standard MI. practices, we do not adapt the standard DARTS pipeline from the original paper. As this search space has been heavily studied, we use as a search routine a recent approach, GAEA PC-DARTS (GAEA), that achieves some of the best-known results CIFAR-10 and ImageNet (Li al. 202 a This NAS approach, due to its heavy retraining routine, is compared to the tuned WRN baseline of the less-resource-constrriiie practitioner.
Maero NAS Using DenseNAS The second NAS paradigm we consider is macro NAS. Instead of building from a fixed cell, it requires the specification of supernet with different inter-connected blocks. These blocks and connections are then pruned to construct an architecture. For our benchmark, choose recent search space, Nen (Fang et al. 2020). which has state-of-the-an results on ImageNet. DenseNAS searches for architectures with densely-connectee blocks emulate DenseNet (Huang et al. 2017). In our experiments, we use the ResNet-based search space, DenseNAS-RI. which contains all neural operations of the WRN backbone. For 2D tasks, we adapt two super networks from the one used for ImageNet inputs the search algorithm. The super network for dense tasks maintains the same spatial dimensions without downsampling to avoid bottlenecks, and we lower the learning rate for evaluating architectures to prevent divergence. For transferring to 1D tasks, all network operations are switched from 2D to ID. Other training and evaluation procedures are identical to those in the original paper and uniform across all tasks. DenseNAS is quick to search and evaluate, making it comparable to the fixed WRN bascline.
customizable routing
We apply another search method to the fixed DenseNAS space to study the relative importance of search algorithms. Random search is implemented through randomly sampling architectures from the DenseNAS space and validating them after a brief training period of 10 epochs before evaluating the best performer. To ensure faimess of comparison, we allot equal GPU hours to random search and regular DenseNAS search, additionally applying a soft constraint that random architecture model sizes should not surpass DenseNAS searched architecture sizes significantly.
7

---

Under review as a conference paper at ICLR 2022
Table Median rank and performance improvement over WRN across NAS-Perf-360.
Search space
WRN default
DenseNAS original
DenseNAS random
WRN ASHA
DARTS GAEA
Auto-DL DARTS
AMBER
Search algorithm
ENAS
Median rank % better than WRN
4.0
4.0
4.0
3.5
1.5
6.0
1.0
0.0%
2.53%
0.0%
0.0%
20.1%
-75.3%
20.0%
relative improvement over the default (untuned) WRN baseline
metric computed only the subsct of three tasks which the method was evaluated
To study the importance of search spaces, we further handpick two domain-specific NAS approaches applicable only to a subset of the tasks. Using an encoder-decoder architecture, Auto-Deepl (Auto-DI.) (Lau et al. (0T9a) is designed for dense prediction, e.g., semantic segmentation. While the decoder is fixed. AL searches for an encoder via first-order DARTS. We evaluate Auto-DL on Darcy Flow, PSICOV, and Cosmic tasks.
Domain-specific NAS Baselines: Auto-DL and AMBER
AMBER (Zhang et al., 2021b) aims to automate neural network design for ID genomic data. This framework establishes 12-layer network backbone and parametrizes long-short term memory network (LSTM) as a controller to search for suitable 1D operations and residual connections, following the ENAS (Pham et al. 2018) optimization protocol. At each step, the controller samples architectures to compute reward based on area under the receiver operating characteristics (AUROC) before outputting the highest-reward architecture. We evaluate AMBER on the ECG, Satellite, and DeepSEA tasks.
4.2 EXPERIMENTAL SETUP
Below we discuss the main reporting details our empirical evaluation.
the same hyperparameter ranges across all tasks to tune WRN, The tuning budget is selected to match that of DARTS (GAEA).
Hyperparameter tuning: As detailed in the Appendix,
A gggreattion Metrics: Table B contains the median rank and relative improvement over WRN of each method for direct comparison via a single number. We also employ performance profiles (Dolan & Moré 2002) in Figure 2 approach that allows analysis taking into account outliers while not allowing small differences in performance to dominate; as described in Figure | these curves denote for each the fraction of tasks which method is worse than 7-factor from the optimal.
Software and hardware: We adopt the free, open-source software Determined for experiment management, hyperparameter tuning, and cloud deployment. All experiments are performed on single p3. 2xlage instance with an NVIDIA V100 GPU. Costs in GPU-hours are in the appendix.
5 DISCUSSION
We conclude our presentation of NAS-Perf-360 via an analysis that (a) reveals new insights about the capabilities and robustness of current NAS methods and (b) demonstrates how our benchmark can enable critical next steps in NAS research. In particular, we start by considering our two practitioners faced with a choice of spending their limited compute on (possibly tuned) fixed- architecture CNN or trying to find better architecture using NAS. With this study, we investigate whether modern NAS methods perform well beyond the tasks for which they were designed.
A surface-level analysis suggests that under light resource constraints, modern NAS the form of DARTS (GAEA) is quite robust to wide variety of tasks: Table 3 shows it is the highest- ranked Nomain-indepenae WRN baseline. The performance profile in Figure (left) also seems favorable, although it is overtaken by tuned WRN at a higher r-suboptimality. However, closer look at 2D point tasks in Figure 2 (right) reveals that DARTS is quite poor there, despite its design domain being image classification; in particular, it performs very poorly on NinaPro and FSD50K. Furthermore, on tasks where it performs well, it can still lag behind expert architectures; for example, on Darcy Flow, networks that use FNO (i et al. 2021b) or XD (Roberts et al. 2021 operations do much
method and attains the most significant improvement over the fixed
GGitHub repository: httpt:: htttp:///iituub.coo
thub.
ned ai /eti//eeerminnnn
8

---

Under review as a conference paper at ICLR 2022
hor Sommancn mofies for All kn tosls
perfomance noffes for nour 20 rojet mediction lasks
DenseNAS
Dense NAS (RS)
DenseNAS (RS)
DARTSS (GAEA)
DARTS (GAEA)
WRN (tuned)
WRN Itured)
WRN (default)
WRN Iole' fault)
Performance profiles on all tasks (left) and 2D point tasks (right). The y-value indicates the fraction of tasks which plotted method's error is within multiplicative factor of the lowest error achieved by all plotted methods..
Figure 2:
better. Overall, our results suggest that this practitioner can apply NAS and expect to see some improvement, but also risks catastrophically poor performance (e.g., FSD50K) or not getting truly sfate-of-the-rr art results (c.g.. Darcy Flow).
2. Under stronger budget constraints, our experiments strongly suggest that practitioner should simply apply the default WRN to their problem rather than undergo the additional complexity of using DenseNAS, as the latter attains little-to to-le-to-no no improvement over the former in Table and has a usually- worse performance profiles Figure 2 One bright point, however, is the strong performance of DenseNAS on FSD5OK, where it outperforms all methods even while DARTS (GAEA) fails.
These first experiments suggest that the modern NAS methods are not always robust to diverse tasks, especially under essurrc-comstraiiee settings. We believe that NAS-Perf-360's main roles as future benchmark include developing an understanding of the multi-domain performance of existing approaches and guiding research into better NAS methods. While the latter is beyond the scope of this paper. our additional experiments demonstrate how NAS-Perfs facilitates the former.
Notably, several results address the question of the relative importance of search space v. search algorithm. For example, Table J that on DenseNAS, random search is nearly identical the more sophisticated weight-sharing scheme of the original paper; the two algorithms' performance profiles are also difficult to distinguish in Figure y Furthermore, AMBER-a a ID NAS method whose search space includes Aarger-kernel convolutions for handling such tasks does better than GAEA even though it uses an older search algorithm (ENAS). These both suggest that search space design, including the use of wider variety of operations, may be least as crucial for success as the search algorithm. This point is reinforced by example tasks such as Darcy Flow, where architectures with more exotic operations substantially outperform our best results, discussed earlier.
NAS-Perf-360 also reveals failure points of several methods, not just general ones that usually perform quite well such as DARTS (GAEA) but also the objective-specin approach Auto-DL, which despite being designed for dense prediction tasks, does poorly on all those considered here. Understanding when and why these performance drops happen is critical to developing a more robust NAS that is useful not just on average but more challenging settings.
6 CONCLUSION
NAS-Perf-3600 is new performance benchmark consisting of ten diverse tasks derived from various fields of research and practice. It is designed for reproducible research on an academic budget that will guide the development of NAS methods and other automated approaches towards more robust performance across different domains. In initial results, we have demonstrated both the need for such benchmark and the utility of NAS-Perf-366 specifically for developing search spaces and algorithms; we welcome researchers to use its tasks to develop new procedures for automating MI.

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
Within our array of tasks, only NinaPro, ECG, and DeepSEA contain human-derived data. Our chosen subset of NinaPro contains only muscle movement data without any exposure of personal information. The original experiments to acquire NinaPro data are approved by the ethics com- mission of the canton of Valais, Switzerland (Atzori al. 2012). The ECG data derives from an open challenge and is provided by the medical device company AliveCor, under the GPL license allowing it for public use. The DeepSEA data derived from ENCODE part of an international collaborative effort, which is overseen and funded by the National Human Genome Research Institute (NHGRI), For other datasets, we have listed the data licenses the appendix. While we do not view the specific datasets in NAS-Perf Perf-360 as potential candidates for misuse, the broader goal of applying NAS to new domains comes with inherent risks that may require mitigation on an coplication-by-appleeee applicattioon basis.
8 REPRODUCIBILITY STATEMENT
The following measures are taken ensure reproducibility:
1. We store the processed data in AWS S3, and data splits are the same for all experiments.
2. Code is always executed in fixed Docker container using a pre-built image on Docker Hub. This guarantees a uniform execution environment and saves users from configuring dependencies.
3. Via the specification of a random seed, Determined controls several essential sources of randomness during execution, including hyperparameter sampling and training data shuffling.
4. During training, we validate on the entire validation set, not on mini-batch, limit stochas- ticity.
5. Code and download links of all datasets are available at the anonymized repository: https
//anonymous 4open. www.:sscceccccc..ee...... /AS Bench 360-2601
REFERENCES
Badri Adhikari. Deepcon: protein contact prediction using dilated convolutional neural networks with dropout. Bioinformatics, 36(2):470-477, 2020a.
Badri Adhikari. A fully open-source framework for deep learning protein real-valued distances. Scientific reports, 10(1):1-10, 2020b.
Manfredo Atzori, Arjan Gijsberts, Simone Heynen, Anne-Gabrielle Mittaz Hager, Olivier Deriaz, Patrick Van Der Smagt, Claudio Castellini, Barbara Caputo, and Henning Müller. Building the ninapro database: A resource for the biorobotics community. In 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), pp. 1258-1265. IEEE, 2012.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In Proceedings of the 7th International Conference on Learning Representations, 2019.
Gari D Clifford, Chengyu Liu, Benjamin Moody, H Lehman Li-wei, Ikaro Silva, Qiao Li, AE John- son, and Roger G Mark. Af classification from short single lead ecg recording: The phys- ionet/computing in cardiology challenge 2017. In 2017 Computing in Cardiology (CinC), pp. 1-4. IEEE, 2017.
Taco Cohen, Mario Geiger, Jonas Koehler, and Max Welling. Spherical CNNs. In Proceedings of the 6th International Conference Learning Representations, 2018.
ENCODE Project Consortium et al. The encode (encyclopedia of dna elements) project. Science, 306(5696):636-440 640, 2004.
Ulysse CôlérAllard, Cheikh Latyr Fall, Alexandre Drouin, Alexandre Campeau-Lecour Clément Gosselin, Kyrre Glette, François Laviolette, and Benoit Gosselin. Deep learning for electromyo- graphic hand gesture signal classification using transfer learning. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 7704)770--771, 2019.
10

---

Under review as a conference paper at ICLR 2022
Angus Dempster, François Petitjean, and Geoffrey I Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5):1454-1495 2020.
Elizabeth Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical programminng 91(2):201 201.11,,.11 2002.
Xuanyi Dong and Yi Yang. NAS- MAS--Bnch-2001 201: Extending the scope of reproducible neural architecture search. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Yawen Duan, Xin Chen, Hang Xu, Zewei Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Li. SeersNAA-Beecch--- Improving transferability and generalizability of cross-task neural architec- ture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Jiemin Fang, Yuzhu Sun, Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang. Densely connected search space for more flexible neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Eduardo Fonseca, Jordi Pons Puig, Xavier Favory, Frederic Font Corbera, Dmitry Bogdanov, Andres Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra. Freesound datasets: platform for the creation of open audio datasets. In Hu X, Cunningham SJ, Turnbulll D. Duan Z, editors. Proceedings of the 18th ISMIR Conference; 2017 oct 23-27; Suzhou, China. (Canadaa: International Society for Music Information Retrieval; 2017. p. 486-93. International Society for Music Information Retrieval (ISMIR), 2017.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. arXiv preprint arXiv:2010.00475, 2020.
2020.
John Garofolo. Timit acoustic phonetic continuous speech corpus. Linguistic Data Consortium, 1993, 1993.
Jort Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 20/7 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780, IEEE, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Shenda Hong. Yanbo Xu, Alind Khare, Satria Priambada, Kevin Maher, Alaa Aljiffry, Jimeng Sun, and Alexey Tumanov. Holmes: health online model ensemble serving for deep learning models in intensive care units. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1614-1624, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4705 2017.
David Josephs, Carson Drake, Andy Heroy, and John Santerre. semg gesture recognition with simple model of attention. In Machine Learning for Health, pp. 126-138. PMLR, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Kathryn Tunya- suvunakool, Olaf Ronneberger, Russ Bates, Augustin Židek, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Anna Potapenko, Andrew Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Martin Steinegger, Michalina Pacholska, David Silver, Oriol Vinyals, Andrew W Senior, Koray Kavukeuoglu, Pushmeet Kohli, and Demis Hassabis. High accuracy protein structure prediction
11

---

Under review as a conference paper at ICLR 2022
o using deep learning. In Fourteenth Assesssenn Techniques for Protein Structure Pre- diction (Abstract Book), 2020. URL UKL/Apppee/prrccccciooeeeeeeee.. wwtp:/eetteeeoeecaaaaeaeeeeeeeee cttp:/asp11////
CASP14 14 Abstracts.pdl
Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, and Evgeny Burnaev. B.S--Bench-NL. Neural architecture search benchmark for natural language processing. arXiv, 2020.
Alex Krizhevksy. Learning multiple layers of features from tiny images. Technical report, 2009.
I Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning. arXiv preprint arXiv:1810.05933 2018.
Liam Li, Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Geometry-aware gradient algorithms for neural architecture search. Proceedings of the 9th International Conference on Learning Representations, 2021a.
Lisha Li, Kevin G Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: Bandit-based configuration evaluation for hyperparameter optimization. In ICLR (Poster), 2017.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In Proceedings of the 9th International Conference on Learning Representa- lions, 2021b.
Chenxi Liu, Liang- Lhang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Li Fei- Fei. Auto-deeplah: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 82-92, 2019a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In Proceedings of the 7th International Conference Learning Representations, 2019b.
Abhinav Mehrotra, Alberto Gil, C. P. Ramos, Souray Bhattacharya, Lukasz Dudziak, Ravichander Vipperla, Thomas Chau, Samin Ishtiaq, Mohamed Abdelfattah, and Nicholas D. Lane. NAS- Bench-ASR: Reproducible neural architecture search for speech recognition. In Proceedings of the 8th International Conference on Learning Representations, 2021.
François Petitjean, Jordi Inglada, and Pierre Gançarski. Satellite image time series analysis under time warping. IEEE transactions on geoscience and remote sensing, (0(8):3081-3093 2012.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Proceedings of the 35th International Conference on Machine Learning, 2018
Esteban Real, Chen Liang, David R. So, and Quoc Le. AutoML-Zeroo Evolving machine learning algorithms from scratch. Proceedings of the 37th International Conference on Machine Learning, 2020.
Nicholas Roberts, Mikhail Khodak, Tri Dao, Liam Li, Chris Ré, and Ameet Talwalkar. Rethinking neural operations for diverse tasks. arXiv, 2021.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Antrre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: A dataset of datasets for learning to learn from few examples. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Colin White, Willie Neiswanger, and Yash Savani. BANANAS: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, 2021.
12

---

Under review as a conference paper at ICLR 2022
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang. Xin Chen, Guo-Jun Qi. Qi Tian, and Hongkai Xiong. PC-DARTS: Partial channel connections for memory-efficien architecture search. Proceedings of the 8th International Conference Learning Representations, 2020.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- Bench-101: 101: Towards reproducible neural architecture search. In Proceedings of the 36th Interna- tional Conference on Machine Learning, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference, 2016.
Arber Zela, Julien Siems, and Frank Hutter. NAS-Bench-1Shotf Benchmarking and dissecting one- shot neural architecture search. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Keming Zhang and Joshua S Bloom. deeper: Cosmic ray rejection with deep learning. The Astrophysical Journal, 889(1):24, 2020.
Zijun Zhang, Evan M Cofer, and Olga G T'royasskaya. Ambient:
accelerated convolutional neural network architecture search for regulatory genomies. bioRxiv, 2021a.
Zijun Zhang, Christopher Y Park, Chandra Theesfeld, and Olga G Troyanskaya. An automated framework for efficiently designing deep convolutional neural networks in genomies. Nature Machine Intelligence, 365):392-400, 2021b.
Jian Zhou and Olga G Troyanskaya, Predicting effects of noncoding variants with deep tearning-basee sequence model. Nature methods, 12(10))931-99) 9344, 2015.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, 2018
13


Reviews:
[{"review_id": "wFGrH5W-pzx", "reviewer": "Reviewer_zB8j", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1/ I think the problem this paper raises is very meaningful. Currently, NAS has been over-focused on a few problems, especially, image classification on ImageNet or CIFAR10 datasets. It is not clear wether progress achieved on NAS will transfer to other problems and applications.\n \n2/ The NAS benchmark proposed in this paper is flexible and general enough to include architecture search, hyper-parameter tuning, and different base architectures. This is better than previous NAS Benchmarks, where the scope is limited to cell-level architecture space on a specific problem. Also, the computational cost for this NAS benchmark is much more manageable. \n\n3/ The conclusion or take-away message of this paper is useful for practitioners. Especially, depending on the computational budgets, whether to choose to use an off-the-shelf model with HP tuning or to search architectures. \n\nWeaknesses:\n1/ Despite that authors clarified their principles of choosing different tasks in this benchmark, the exact choice is still quite arbitrary. Eventually, the reason to have a benchmark is that 1) the benchmark itself measure some metrics we care about, or 2) performance on this benchmark can generalize to a wider range of tasks that may arise in practical applications. In this paper, scenario-1) obviously does not hold. But for scenario-2), this has not been demonstrated in the paper. Specifically, does a NAS method that performs well on this benchmark can generalize better to new unseen tasks? There is no experimental or theoretical support for this, which limits the contribution of this work.\n\n2/ The technical contribution of this paper is limited. This paper did not propose new methods or other novel contributions. An example of a more significant contribution might be to propose a better NAS algorithm that generalizes better than existing ones. \n\n3/ The paper is not well written and organized. For example, the metrics in Figure 1 is not clearly explained. The caption explains the metric as \"A larger value indicates a larger fraction of tasks on which the method is within a multiplicative factor \\tau of the best.\" The question is, the best of what? The best performance in terms of accuracy or error rate? What is \"multiplicative factor \\tau of the best\"? I don't think this is clearly explained anywhere in the paper, which makes the results confusing. ", "summary_of_the_paper": "This paper proposes a benchmark to test the performance of NAS algorithms and search spaces on a diverse set of tasks. The benchmark consists of 10 different datasets across different modalities. On these tasks, a variety of NAS algorithm as well as search spaces are allowed a fixed amount of compute resources (in terms of GPU hours) to explore and train, and reach the final performance. The search space are not limited to architecture topologies, but also hyper-parameters. On this new benchmark, authors find that existing SoTA NAS methods may not generalize to different tasks, especially with low compute budgets. ", "summary_of_the_review": "The paper proposes a meaningful direction and benchmark for NAS. But whether this benchmark can measure the generalization of NAS method is not clear, the technical contribution of the paper is limited, and the writing of the paper should be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gVib5cXAVW9", "reviewer": "Reviewer_oZbZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a 'so-called' NAS-Bench-360, which is quite misleading. The original NASBench idea (brought by NASBench-101 Ying et al.) is to exhaustively train all (or a substantial amount of) architectures within a search space and record their training / evaluation statistics for researcher or NAS practitioners to evaluate their search algorithm in a fair manner. However, this work mis-use the concept. After reading the entire paper for multiple times, it seems this work is merely evaluating some standard NAS algorithms on various datasets and present some empricial analysis.  \n\nIn my humble opinion, the usefulness of NAS-Bench is to leverage some groups' massive compute-power to obtain a massive pool of architecture statistics on various tasks, to benefit people without such power to conduct NAS research. Evaluting some algorithms, even across drastically different tasks, is interesting but not enough to be accepted to a top tier conference. \n\n", "summary_of_the_paper": "This paper proposes to evaluate some standard NAS algorithms over various tasks, ranging from image classification, sattllite image time series analysis or ECG detecting hard diseases. \n\n", "summary_of_the_review": "See above, this paper mis-use the concept of NAS-Bench that is common to NAS researcher. After reading the entire paper, I do not find it interesting as a NAS practitioner. Evaluating some algorithms on different tasks is a good empirical report but cannot be qualified as a top-tier conference paper due to lack of novelty and effort.\n\n ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "exUnCx0WQya", "reviewer": "Reviewer_8TV7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general this paper is really well-written and the motivation behind its idea is quite valid in my opinion. In general I think the NAS community will benefit from this suite of different tasks to run their NAS methods. Below I list some more details comments:\n\n**Pros:**\n\n(+) I think the set of collected tasks will be a useful contribution to the NAS community in order to shift the attention to other tasks that do not only involve image classification.\n\n(+) Useful insights on the usefulness of NAS by evaluating different methods or/and search space across 10 different diverse tasks.\n\n(+) The paper is well-structured and easy to follow.\n\n(+) Available codebase that allows reproducing the experiments and evaluating new methods by using the same settings.\n\n**Cons:**\n\n(-) I think the paper could benefit more by experiments that provide further insights for the NAS practitioner. For instance, performance predictors are commonly used nowadays in NAS [1]. It would be useful to evaluate a couple of these methods (e.g. some of the ones used in [1]) on the proposed suite, both in isolation (reporting the rank correlation coefficient) and inside a black-box algorithm (e.g. Bayesian optimization).\n\n(-) I think there is some text and results in the main paper that can be moved to the appendix and make some space for additional empirical results. For instance, the authors can move Table 3 to the appendix, since it is somewhat redundant with Table 2. Some examples of such additional experiments can be for instance:\n- A more detailed investigation on why the NAS methods fail in many of the tasks in NAS-Bench-360; \n- Why some of these NAS methods perform better in some tasks and some others not? Is this because of a sub-optimal search space (operation choices and topology) or an underperforming NAS optimizer?\n- How much does the inductive bias affect the ranking of different NAS methods evaluated throughout the 10 tasks? By NAS methods I mean: Different NAS optimizers evaluated on a fixed search space.\n\n**Other comments:**\n\n- I think the term \"NAS-Bench\" is reserved for tabular/surrogate benchmarks and using it for this submission might be confusing at first for many readers, since NAS-Bench-360 in principle is only a collection of datasets and scripts for running methods (search space + NAS algorithm) on them. I would suggest the authors to reconsider changing the name of their suite.\n\n- I would also add plots showing the performance over time of the different evaluated NAS methods on every task.\n\n- I agree with the authors that as an AutoML practitioner, in general I am also concerned which will provide the best output: 1) optimizing the architecture when keeping the hyperparameters fixed or 2) the hyperparameters of a fixed well-performing architecture. However, one can also do both, i.e. given a fixed computational budget, start with 1) and then move to 2), or the other way around. It would be interesting to see this experiment with a NAS search space that includes the WRN architecture (does the DenseNAS space include WRN? If yes, one can very well use that.).\n\n- Why Table 5 is in the Appendix and not in the main paper instead of Table 2?\n\n**References**\n\n[1] White et al. How powerful are performance predictors in neural architecture search?, In NeurIPS 2021", "summary_of_the_paper": "This paper presents NAS-Bench-360, which consists of collection of 10 diverse tasks, carefully selected to include datasets with different number of classes, total number of datapoints, dimensionality and application domain. By using a fair evaluation protocol, the authors benchmark different NAS methods on various search spaces, with various degrees of inductive bias, and provide some interesting insights on the effectiveness of NAS under time constraints, showing that there is almost no benefit compared to using a fixed manually engineered architecture.", "summary_of_the_review": "I really like the contributions and the insights offered by this paper. Even though there is no novelty per se, I think the NAS community will benefit from this curated collection of diverse tasks. I lean towards acceptance and I will increase my score after most of my concerns are addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TUuuvSypdIi", "reviewer": "Reviewer_ofGn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this work, the authors thrown out an issue that the existing NAS benchmarks focused on the popular tasks -- image classification and needs more diverse tasks for the NAS evaluation. First of all, the existing NAS benchmarks have covered at least 6 different tasks/datasets and if we counting the dataset/tasks explored in the HPO works (which also try to find some architecture options), we have more architecture datasets/benchmarks in the NAS community, which is strong and diverse. So that some claims in the paper may be exaggerated.\n\nAs for the proposed \"benchmark\", one important feature for the NAS benchmark is the associated architecture datasets, which provides tabuler or suggrate performance data for the architectures. Unfortunately, this work does not provide such architecture dataset.\n\nAs for the collected tasks, some may be too simple and not suitable for evaluating the DL or NAS methods. For example, some datasets only has about 1K data but try to do some 2D dense prediction. It is not supuring some approaches, which require enough data, perform poorly on such dataset.\n\nFrom the technical part, the authors compared a few NAS method and basic DL models. As the NAS techniques have been applied in many many different domains/tasks. If we collect the results from the papers that use DARTS in a non-CIFAR/classification application, the effect seems to be the similar. Compared to this option, the only benefit of this NAS-Bench-360 would be the calibration the NAS methods in the existing application papers.", "summary_of_the_paper": "The authors argued that existing NAS benchmarks targets the well-studied tasks, and thus proposed that we should evaluate NAS on diverse tasks. And thus the authors collected 10 tasks and did some analysis on these ten tasks/datasets to compare several NAS methods and some strong baseline DL models, such as wide resnet.", "summary_of_the_review": "This NAS benchmark lacks architecture datasets for the collected ten datasets and technical novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KKFDjV9vSaf", "reviewer": "Reviewer_JYCo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The attempt of creating new benchmarks or datasets is very promising for the NAS field. \n+ The proposed benchmark is much diverse compared with the existing ones.\n\nConcerns:\n- The attempt is important but this paper seems to be just providing the existing datasets. For NAS evaluation, it is also important to provide the rank of each candidate architecture in the search space because recent studies show that NAS methods find architectures achieving good performance, but their ranks in the search space are far from the best [Yu+, ICLR'20]. It indicates that it is necessary to provide the rank or something related to the rank to appropriately evaluate NAS methods, not only performance.\n\n- Depending on a given task, an optimal search space would change. But this paper uses the limited search space for the tasks, thus I am wondering if this benchmark can appropriately evaluate NAS methods. It seems that designing search space itself is needed for diverse tasks and evaluation of NAS methods on them.\n\n- It would be better to describe the difference between the existing NAS benchmarks and the proposed one in more detail.\n- It would be nice to provide the detail of the search space in the main paper. ", "summary_of_the_paper": "This paper proposes a new benchmark for NAS methods, which is called NAS-Bench-360. Unlike the existing benchmark datasets for NAS, the proposed benchmark contains ten diverse tasks derived from various fields of research. This paper has tested several standard NAS methods on the proposed benchmark and confirmed that there are many gaps among the ten tasks and NAS methods.", "summary_of_the_review": "The attempt of creating a new benchmark is very important and this paper shows promising results. However, my major concern is about the design of the proposed benchmark (see concerns above). Hopefully, the authors can address my concern in the rebuttal period.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper at first used the name NAS-Bench-360 for the benchmark, which confused several reviewers (who expected a tabular benchmark behind this name). The authors renamed the benchmark, which removed this issue, emphasizing that the contribution does not lie in proposed a new tabular NAS benchmark, but a new performance evaluation of NAS on different data sets.
One reviewer recommended acceptance, but 3 reviewers stuck with their rejection scores, the reasons being that 
- there are by now several papers applying NAS outside of computer vision, with a seemingly more comprehensive analysis 
- more analysis would be useful
- it is unclear how general the conclusions are that can be drawn from performance on the included datasets.
(Low technical novelty was also mentioned, but I do not believe that this type of paper can be very impactful even if it has no technical novelty.)

Overall, although I agree with the accepting reviewer that this type of work can be very useful to the community, the rejecting reviewers have too many criticisms to accept the paper in its current form. I encourage the authors to address them and to resubmit.
One note (which did not affect the decision, but which I'd like to notify the authors about) is that a reviewer found that the author identity was revealed in the anonymous codes provided by the authors (https://anonymous.4open.science/r/NAS-Bench-360-26D1).
Decision: Reject