Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Edit after rebuttals: I have read all other reviews and rebuttals and maintain my assessment.
----
Summary: Comparison of neural architecture search algorithms is hindered by the lack of a common measurement procedure. This paper describes a publicly available benchmark on which most recent types of NAS algorithms can be evaluated. It does so by exhaustive calculation of performance metrics on the full combinatorial space of select architectures, on two select datasets. NAS algorithms can then perform search without having to perform evaluation on each node, which shrinks the computational cost of experimentation and benchmarking drastically.

I recommend acceptance, as the resource described in the paper has been created thoughtfully and is useful to the research community, as well as to users of NAS algorithms. The paper is clear about restrictions too, which doesn't hurt.

The technical details are laid out clearly especially in sec 2.1. It would be interesting to know the computational cost of producing the data. It is useful in practice to have access to different metrics (validation, training and test) for each node, as well as extra diagnostic information.  

The usefulness of the resources hinges on a few elements, which make its strength and also weakness:
- choice of tasks and datasets
- choice of skeleton architecture, fig 1
- choice of hyperparameters, sec 2.3 (I note there is no regularisation, as discussed in the paper)
All of these seem reasonable to me. It is clearly a limitation that hyperparameter search is infeasible to conduct in parallel with architecture search, as pointed out sec 6.

The principal competitor NAS-Bench-101 is only applicable to specific NAS algorithms, which evidences the need for the present resource. The discussion and comparison in sec3 is fair.

The discussion of weaknesses, such as possible overfitting patterns, or technical choices, is balanced. 

# Minor
English proofreading is required. 
- Maybe you can attempt a pun on Ananas in the naming?
- I'm not sure "fairness" as in the abstract is the exact core problem; I would call this comparability.
- sec2 head: "side information", I usggest diagnostic information
- sec2.2 "and etc" is a redundant: etc stands for "and the others"
- sec2.4 almost involves almost; target on computation cost; stabability
- sec 4: has impacts on, parameters keeps the same -> stays, which serves as testing -> to test
- sec6 tricky ways-> insidious?
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: --- Updated during response period ---

Authors successfully answers all my questions. I revise my rating to Accept.


-----

Summary:

This paper proposes another benchmark dataset for neural architecture search. The idea is following the NASBench-101 dataset, that in a given search space, densely sampled all existing architectures and train each of them on three tasks for multiple times, and using the obtained metrics as a tool to evaluate an arbitrary neural architecture search algorithm. The paper also presents comprehensive reports on the statistics, revealing a strong performance correlation between tasks, and evaluate some baseline NAS algorithms.

I think this paper will be valuable to the research community for these reasons: (1) the dataset contains a more geologically complex search space comparing to the original NASBench-101, whose search space is restrained in certain ways; (2) released metrics include more meaningful information rather than single point value in NASbench; (3) it uses 3 datasets rather than 1.
My major concerns, which I will detail later, is the phrasing "algorithm-agnostic" does not truly reflect the difference between their approach and NASBench-101, and about the architecture search space design details. 

Altogether, I think even the technical novelty is incremental, the work is not trivial considering the computational cost. I am willing to improve my score if my concerns are addressed during the rebuttal period. Nevertheless, this dataset is a strong subsidy of existing NASBench-101 and can benefit the research community and serves as an important baseline to evaluate a NAS algorithm. 


Strength

+ Clear motivation to use an operation-on-the-edge search space that is widely used in NAS domain.
+ Extensive experiments on evaluating 15K architectures over 3 datasets
+ Detailed statistics on the search space
+ Good baseline experiments comparison

Main concerns about this dataset:

- Comparing to NASBench-101 in terms of "Algorithm Agnostic", it is in a "more-or-less" game but not a "yes-or-no" one, so that AA-NAS-Bench does not seem appropriate. In my perspective, this dataset has not shown significant differences for the following reasons.

1. With proper adaptation, both NASBench-101 and the one in this paper are "algorithm agnostic". For example, original ENAS is training a reinforcement learning sampler that learns to predict a string with encoding [id1, op1, id2, op2] for each node, where id1, id2 is the IDs of the previous node to connect, op1, op2 is the operation choice for each edge. Since NASBench has operation on output node, one could simply make RL sampler to predict [id1, id2, op1], or another string encoding that suits the search space better. In my perspective, Ying et al. mentioned that many NAS algorithms cannot be directly evaluated on NASBench-101 are because the search space is different, but it does not mean using NASBench-101 is impossible. On the other hand, for some other state-of-the-art algorithms, like Proxyless-NAS on ImageNet, the search space is also different from the one proposed in this paper, but likewise, it does not indicate evaluating Proxyless-NAS on this dataset is impossible. 

2. NASBench-101 does impose constraints on maximum edge number equals to 9 with 7 nodes in their space and results in 423K architectures. However, this constraint is no longer applied if you reduce the number of node to 6 (i.e. all possible architectures can be sampled), yet it still contains around 64K architectures, which is more than 15K in the proposed dataset. In this perspective, NASBench is a larger dataset and "algorithm agnostic".

To summarize, I acknowledge the paper's contribution is using an operation-on-the-edge search space that is widely used in previous NAS algorithms while NASBench-101 is using operation-on-the-node space. However, it only makes the proposed dataset "more algorithm agnostic" with less effort, and it does not make the previous NASBench-101 "not" algorithm agnostic. If using the current name AA-NAS-Bench, I think it is not fair for the NASBench-101, specifically they are 4 times larger after removing the edge number constraints. 

- Questions about architecture space design
1. Why using average pooling instead of max pooling? 
2. How do you compute the total architecture number 15,625 in Table 3? In your setting, with the number of node V=4 densely connected DAG, it should have 6 edges as depicted in Figure 1, and each edge has 5 possible operations, i.e. total number = 6^5 = 7776. I am confused about this point, could author comment more on this number?
3. Is there any topologically equal architectures in this space? For example, let's name the node 1,2,3,4, and the following two architectures should be the same since input edges are summed before passed to the next node. I listed **non-zeroed** edge as, id1->id2: op

Architecture 1:
1->2: conv3x3
2->4: skip
1->3: conv1x1
3->4: skip

Architecture 2:
1->2: conv1x1
2->4: skip
1->3: conv3x3
3->4: skip

If the pruning is not effectively conducted, my worry is the actual number of architectures is smaller.

Minor comments:
1. DARTS results on the are quite poor as mentioned in this paper that, DARTS will eventually converge to an architecture with all skip connection. However, it could be a simple fix, by tracking the architecture evolution during the search and report the best like early-stopping. Will this improve DARTS results? 

2. Since ENAS is the first work using parameter sharing on the NAS problem, could the author add it to the baseline?

3. In table 4, what is the average (94.37 for CIFAR-10) mean in the "optimal" column? Is this the mean performance of all architectures? If so, it is quite strange to see all the baselines are selecting architectures worse than the average performance. Or it is the best architecture performance as indicated in the caption? This "average" column for "optimal" seems confusing.

4. The dynamic ranking of architecture in Figure 5 is very interesting. Architecture ranking seems stable after the 190th epoch. Could the author provide another visualization, showing when stabilization happens in between epoch number 150 and 190? 

5. Figure 4, correlation matrix for top 4743 architectures are significantly lower than the full and 1387 ones, is this possible because of repetitive architectures in the space are not pruned? And, what is the reason for number 4743 and 1387?

6. ResNet (star in Figure 2) seems to perform very well. Does this indicates the proposed search space is not much meaningful, considering there are only 1~2% for NAS to improve? 
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Summary:

Research into Neural Architecture Search (NAS) has exploded in recent times. But unfortunately the entry barrier into the field is high due to the computational demands of running experiments on even cifar10/100 let alone ImageNet sized datasets. Furthermore there is a reproducibility and fair comparision crisis due to differences in search spaces, training routine hyperparameters, stochasticity in gpu training, etc. This paper proposes a benchmark cell-search space (resnet backbone, 4-node cell space, 5 possible operations) which is algorithm agnostic. They train all possible architectures (15625) in this search space on cifar10/100/Imagenet-16-120 (a reduced version of ImageNet with 120 classes). Thus anyone can now use this pretrained lookup-table to benchmark their search algorithm in seconds on a tiny laptop instead of having to get access to a cluster with hundreds of gpus. By also proposing reference implementations of training architectures the community can use this to fairly benchmark their search algorithms. 

The other such benchmark is NASBench-101 which uses a much more expansive search space but by imposing a limit on the number of edges in the cell (to keep the search space manageable with respect to how many of them they have to train) they leave out algorithms which do weight-sharing (ENAS, DARTS, RANDNas) from being able to use their benchmark. This paper alleviates those constraints and thus brings important algorithm classes to their fold.

Comments:

- The paper is very well written. Thanks!

- Minor clarification question: One nice thing of the NASBench paper was the fact that they also reported variance in training with differnt random seeds. I see a line in the 'Metrics' section saying that this is also done but did not find any details on number of trials and whether this was part of the benchmark lookup. I might have missed it somewhere. 

- There is another class of search algorithms which grow from small to big cells (if using a cell search space) like EFAS (Efficient Forward Architecture Search by Dey et al and AutoGrow by Wen et al.). Can such algorithms take advantage of this benchmark? I think the answer is yes, because of the 'zeroise' operation but wanted to get the authors' answer.

- Overall I think this is an important contribution to the field and I am assuming that the authors plan to release the benchmark and reference implementations if accepted?
review_title: Official Blind Review #1

Meta-review: This paper presents a new benchmark for architecture search. Reviewers put this paper in the top tier. I encourage the authors to also cite https://openreview.net/forum?id=SJx9ngStPH in their final version. 
Decision: Accept (Spotlight)