user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search

Paper content:
Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years.
It could be time to take a step back and analyze the good and bad aspects in the field of NAS.
A variety of algorithms search architectures under different search space.
These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization.
This raises a comparability problem when comparing the performance of various NAS algorithms.
NAS-Bench-101 has shown success to alleviate this problem.
In this work, we propose an extension to NAS-Bench-101: NAS-Bench-201 with a different search space, results on multiple datasets, and more diagnostic information.
NAS-Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms.
The design of our search space is inspired by the one used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph.
Each edge here is associated with an operation selected from a predefined operation set.
For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-201 includes all possible architectures generated by 4 nodes and 5 associated operation options, which results in 15,625 neural cell candidates in total.
The training log using the same setup and the performance for each architecture candidate are provided for three datasets.
This allows researchers to avoid unnecessary repetitive training for selected architecture and focus solely on the search algorithm itself.
The training time saved for every architecture also largely improves the efficiency of most NAS algorithms and presents a more computational cost friendly NAS community for a broader range of researchers.
We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms.
In further support of the proposed NAS-Bench-102, we have analyzed it from many aspects and benchmarked 10 recent NAS algorithms, which verify its applicability.
The deep learning community is undergoing a transition from hand-designed neural architecture (He et al., 2016; Krizhevsky et al., 2012; to automatically designed neural architecture (Zoph & Le, 2017; Pham et al., 2018; Dong & Yang, 2019b; Liu et al., 2019) .
In its early era, the great success of deep learning was promoted by novel neural architectures, such as ResNet (He et al., 2016) , Inception , VGGNet (Simonyan & Zisserman, 2015) , and Transformer (Vaswani et al., 2017) .
However, manually designing one architecture requires human experts to try numerous different operation and connection choices (Zoph & Le, 2017) .
In contrast to architectures that are manually designed, those automatically found by neural architecture search (NAS) algorithms require much less human interaction and expert effort.
These NAS-generated architectures have shown promising results in many domains, such as image recognition (Zoph & Le, 2017; Pham et al., 2018; , sequence modeling (Pham et al., 2018; Dong & Yang, 2019b; Liu et al., 2019) , etc.
Recently, a variety of NAS algorithms have been increasingly proposed.
While these NAS methods are methodically designed and show promising improvements, many setups in their algorithms are different.
(1) Different search space is utilized, e.g., different macro skeletons of the whole architecture Tan et al., 2019 ) and a different operation set for the micro cell within the skeleton (Pham et al., 2018) , etc. (2) After a good architecture is selected, various strategies can be employed to train this architecture and report the performance, e.g., different data augmentation (Ghiasi et al., 2018; , different regularization , different scheduler , and different selections of hyper-parameters (Liu et al., 2018; Dong & Yang, 2019a) .
(3) The validation set for testing the performance of the selected architecture is not split in the same way (Liu et al., 2019; Pham et al., 2018) .
These discrepancies raise a comparability problem when comparing the performance of various NAS algorithms, making it difficult to conclude their contributions.
In response to this problem, NAS-Bench-101 (Ying et al., 2019) and NAS-HPO-Bench are proposed.
However, some NAS algorithms can not be applied directly on NASBench-101, and NAS-HPO-Bench only has 144 candidate architectures, which maybe insufficient to evaluate NAS algorithms.
To extend these two benchmarks and towards better reproducibility of NAS methods 1 , we propose NAS-Bench-201 with a fixed cell search space, inspired from the search space used in the most popular neural cell-based searching algorithms Liu et al., 2019) .
As shown in Figure 1 , each architecture consists of a predefined skeleton with a stack of the searched cell.
In this way, architecture search is transformed into the problem of searching a good cell.
Each cell is represented as a densely-connected directed acyclic graph (DAG) as shown in the bottom section of Figure 1 .
Here the node represents the sum of the feature maps and each edge is associated with an operation transforming the feature maps from the source node to the target node.
The size of the search space is related to the number of nodes defined for the DAG and the size of the operation set.
In NAS-Bench-201, we choose 4 nodes and 5 representative operation candidates for the operation set, which generates a total search space of 15,625 cells/architectures.
Each architecture is trained multiple times on three different datasets.
The training log and performance of each architecture are provided for each run.
The training accuracy/test accuracy/training loss/test loss after every training epoch for each architecture plus the number of parameters and floating point operations (FLOPs) are accessible.
Hopefully, NAS-Bench-201 will show its value in the field of NAS research.
(1) It provides a unified benchmark for most up-to-date NAS algorithms including all cell-based NAS methods.
With NASBench-201, researchers can focus on designing robust searching algorithm while avoiding tedious hyper-parameter tuning of the searched architecture.
Thus, NAS-Bench-201 provides a relatively fair benchmark for the comparison of different NAS algorithms.
(2) It provides the full training log of each architecture.
Unnecessary repetitive training procedure of each selected architecture can be avoided (Liu et al., 2018; Zoph & Le, 2017) so that researchers can target on the essence of NAS, i.e., search algorithm.
Another benefit is that the validation time for NAS largely decreases when testing in NAS-Bench-201, which provides a computational power friendly environment for more participations in NAS.
(3) It provides results of each architecture on multiple datasets.
The model transferability can be thoroughly evaluated for most NAS algorithms.
(4) In NAS-Bench-201, we provide systematic analysis of the proposed search space.
We also evaluate 10 recent advanced NAS algorithms including reinforcement learning (RL)-based methods, evolutionary strategy (ES)-based methods, differentiable-based methods, etc.
We hope our empirical analysis can bring some insights to the future designs of NAS algorithms.
In this paper, we introduce NAS-Bench-201 that extends the scope of reproducible NAS.
In NASBench-201, almost any NAS algorithms can be directly evaluated.
We train and evaluate 15,625 architecture on three different datasets, and we provide results regarding different metrics.
We comprehensively analyze our dataset and test some recent NAS algorithms on NAS-Bench-201 to serve as baselines for future works.
In future, we will (1) consider HPO and NAS together and (2) much larger search space.
We welcome researchers to try their NAS algorithms on our NAS-Bench-201 and would update the paper to include their results.
Table 6 : We compare the correlation of different training strategies.
The correlation coefficient between the validation accuracy after several training epochs on CIFAR-10 and (1) the validation accuracy of full trained models on the CIFAR-10 training set, (2) the test accuracy on CIFAR-10 trained with the training and validation sets, (3) the validation/test accuracy on CIFAR-100 trained with the CIFAR-100 training set, (4) the validation/test accuracy on ImageNet-16-120 trained with the ImageNet-16-120 training set.
We use the validation accuracy after "EPOCHS" training epochs, where the the cosine annealing converged after "TOTAL" epochs.
Parameter sharing (Pham et al., 2018 ) becomes a common technique to improve the efficiency of differentiable neural architecture search methods (Liu et al., 2019; Dong & Yang, 2019b; a) .
The shared parameters are shared over millions of architecture candidates.
It is almost impossible for the shared parameters to be optimal for all candidates.
We hope to evaluate the trained shared parameters quantitatively.
Specially, we use DARTS, GDAS, and SETN to optimize the shared parameters and the architecture encoding on CIFAR-10.
For each architecture candidate, we can calculate its probability of being a good architecture from the architecture encoding following SETN (Dong & Yang, 2019a) .
In addition, we can also evaluate a candidate using the shared parameters on the validation set to obtain "the one-shot validation accuracy".
It is computationally expensive to evaluate all candidates on the whole validation set.
To accelerate this procedure, we evaluate each architecture on a mini-batch with the size of 2048, and use the accuracy on this mini-batch to approximate "the one-shot validation accuracy".
Ideally, the architecture ranking sorted by the probability or the one-shot validation accuracy should be similar to the ground truth ranking.
We show the correlation between the proxy metric and the ground truth validation accuracy in Table 7 .
There are several observations: (1) The correlation between the probability (encoded by the architecture encoding) and the ground truth accuracy is low.
It suggests that the argmax-based deriving strategy (Liu et al., 2019) can not secure a good architecture.
It remains open on how to derive a good architecture after optimizing the shared parameters.
(2) The behavior of BN layers is important to one-shot validation accuracy.
The accumulated mean and variance from the training set are harmful to one-shot accuracy.
Instead, each architecture candidate should re-calculate the mean and variance of the BN layers.
(3) GDAS introduced Gumbel-softmax sampling when optimizing the architecture encoding.
This strategy leads to a high correlation for the learned probability than that of DARTS.
(4) The uniform sampling strategy for training the shared parameters (Dong & Yang, 2019a) can increase the correlation for one-shot accuracy compared to the strategy of the joint optimizing strategy (Dong & Yang, 2019b; Liu et al., 2019) .

assistant: text: A NAS benchmark applicable to almost any NAS algorithms.