Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a macro search space, which allows blocks in a model to be different to promote performance. To provide a systematic study of the performance of NAS algorithms in a macro search space named Blox – a benchmark that consists of 91k unique models trained on the CIFAR-100 dataset. The dataset also includes other hardware latency information.
strengths: 1. Macro search space and benchmark for NAS with 91k unique architecture. The accuracy improvement with the different blocks is remarkable according to Figure 4. 
2. The analysis includes different NAS algorithms on Blox in terms of block signatures, accuracy predictors, and training methodologies. These factors include 2 existing work named DONNA and HANT.

weaknesses: 1. The training process is extremely complex with three stage settings including Normal setting, Distillation and Fine-tuning according to Figure 6. 

2. Also, with the results provided in Figure 7, the good or bad teacher results in different distilled and fine-tuned performance and has a relatively low spearman with accuracy when trained from scratch. Besides, the number of distilled epochs also affects the spearman correlation.
(1) Is it hard to choose a teacher? Or simply use the best teacher? How to choose the best teacher?
(2) The rank of accuracy distilled and fine-tuned by the bad teacher of all these models is not highly correlated with the rank of the good teacher as shown in Figure 10(b). 
(3) which choice of distilled epoch makes more sense?

3. With these problems, is it hard to apply this pipeline for other researchers to use your Blockwise Search Space to perform architecture search in other network architectures? The complicated training pipeline and factors affected by teachers.
(1) what's the most important items?
(2) does there exist a more simplified procedure?

4. Can we use the Blox dataset to evaluate other commonly seen NAS algorithms besides DONNA and HANT?

correctness: The dataset is constructed in a sound way. 
clarity: The paper is well-written and easy to follow.
relation to prior work: The difference from the previous contributions is clearly discussed.
additional feedback: Please check the Weakness part.
review_title: Official Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: For computational efficiency, most existing NAS algorithms search for cell-based architectures. However, this paper argues that the search space of NAS algorithms should be “macro”, i.e., by including the full network topology in the search space and allowing operations to differ in each block. To systematically evaluate the performance of NAS algorithms in macro search spaces, the paper proposes BLOX, a NAS benchmark that consists of 91k CNN models with better block diversity trained on CIFAR100. Compared to previous macro NAS benchmarks, the BLOX search space is larger as it contains more block options. The paper also evaluates two recent blockwise NAS algorithms on the proposed search space and provides a detailed analysis of their performance and the implications on the efficacy of different block signatures, accuracy predictors, and training strategies.

strengths: - The proposed benchmark focuses on macro NAS, which is a relatively new and less-studied field compared to cell-based NAS. The proposed search space of block operations is expansive compared to that of NAS-Bench-Macro. It contains more operations (up to 45) and includes the commonly seen ones such as residual blocks and inverted bottleneck blocks.

- The paper includes a detailed study on two blockwise NAS methods which have not appeared in previous NAS benchmarks. 

- The analysis in the paper provides two valuable insights. First, the Pareto-frontier of searched models of macro NAS dominates that of models of micro NAS. This suggests that a macro search space contains higher performing models, thus pointing out a promising direction for future NAS research, i.e., to develop efficient methods that search for macro structures and allow heterogeneity between network blocks. Second, for blockwise NAS evaluation, the paper considers two novel settings—distillation and fine-tuning—in addition to the standard training setting. Performance results and the analysis provided can be helpful for the development of training methodology for future algorithms.
 
- Full experiment details, hyperparameters, and code are provided for reproducibility. 

weaknesses: - The search space design has limited novelty and capacity. The set of block options trivial expands that of NAS-Bench-Macro. Moreover, as BLOX contains more block options, it only evaluates architecture with 3 stages to reduce computational cost (note that NAS-Bench-Macro search space has 8 stages). This design choice limits the depth and size of the candidate architectures, and consequently affects the learning capacity of the searched model. Nowadays, commonly used CNNs for image classification (even the simplest ones such as ResNet50) are much larger than 3-stage CNNs. This hinders the practical significance of the benchmark as few ML developers will actually use a network model that has a similar scale to that evaluated in the benchmark for research or industry purposes. Also, it is unsure whether the evaluated networks can work for more complicated tasks beyond CIFAR-100, and the analysis based on the searched results also might not generalize to more complicated architectures and difficult learning problems.
 
- The paper only evaluates two blockwise NAS algorithms. There are many other NAS methods that can be considered "macro", i.e., the ones that generate networks in a global view rather than stacking cells. Examples are morphism-based methods as mentioned in section 4.1.4 of [1]. Do these methods also count towards macro NAS? If not, can you give a better and clearer definition of macro NAS?

- The comparison of macro NAS and cell-based NAS in Fig. 4 might be unfair. When constructing the “uniform block” search space, it seems that the authors consider models generated with the same block options and the three-stage structure so that the number of candidate networks is far less than that of the macro search space. However, micro NAS methods are often more computationally efficient and work for significantly larger search spaces. For a fair comparison, I think the “uniform block” search space should be constructed in a way that either the size of the “uniform block” and the “different block” search space is identical, or the computational cost of exploring the two search spaces are similar. Right now, I find the conclusions of section 2.3 less convincing. 

[1] He, Xin et al. “AutoML: A Survey of the State-of-the-Art.” Knowl. Based Syst. 212 (2021): 106622.

correctness: I’m unsure about the claims made in section 2.3. The evaluation and experimental protocols in later parts seem correct and reasonable.

clarity: The paper is mostly well-written but is less accessible to researchers working outside of the NAS domain. 

relation to prior work: The authors discuss several related works in Introduction and Table 1 and make it clear that benchmarks and studies on macro NAS are lacking.

additional feedback: Which blockwise NAS algorithm is used to obtain the results in Table 2?
review_title: Benchmark for macro NAS

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The Authors have introduced a new search space for NAS which allows for different cell architectures at each block, contrary to cell-based search spaces where a single cell architecture is repeated throughout the network. 

The benchmark (referred to as BLOX) collects in particular information regarding the training and validation of all architectures on CIFAR-100 into a dataset, which can be queried through an API.

They further evaluate several conventional NAS algorithms, together with two recent block-wise NAS algorithms in the common setting provided by BLOX, and use the benchmark to analyze and address a number of questions regarding different components of block-wise NAS methods.
strengths: Having a diverse set of search spaces is crucial for a proper assessment of NAS methods, and the macro search space provided by BLOX will be useful in this respect, especially given that most prior work focus on cell-based search spaces. The related queryable dataset will also allow fast evaluations of NAS methods and can therefore help speed up NAS research.
weaknesses: Despite the fact that “The search space is designed to be compatible with all NAS methods, including differentiable architecture search” as mentioned at the beginning of section 2, no differentiable NAS methods are applied to the search space. This would be useful to support the claim “1) Conventional NAS achieves worse results than standard blockwise (FT200) when a good teacher is used” made in section 3.3.

The fact that the Pareto-front of models with different blocks dominates that of models with uniform blocks, as highlighted towards the end of section 2 (highlight 1), is expected given the three orders of magnitude larger number of models with different blocks (45^3-45) compared to models with similar blocks (45) in the BLOX search space. Rather than comparing different and uniform-block architectures within the same search space, in order to motivate the study further, it would be useful to compare BLOX with a cell-based search space of roughly the same size (see also additional feedback below).
correctness: I cannot identify any flaws in the paper, including the proposed search space, the construction of the dataset, and the performed experiments. 
clarity: The paper is well written and presented in a clear way.
relation to prior work: Relation to prior work is discussed briefly in the introduction.
additional feedback: Given that the BLOX search space is compatible with differentiable NAS methods as well, it would be useful to apply at least one well-known differentiable NAS method (preferably other than DARTS) to support (or disprove) the claim made in section 3.3 that “1) Conventional NAS achieves worse results than standard blockwise (FT200) when a good teacher is used”

If feasible, to motivate the creation of the BLOX search space further, it would be useful to include in figure 4 the Pareto-front of a cell-based search space (or perhaps sampled subsets of existing cell-based benchmarks, such as NAS-Bench-101 etc.) with roughly the same number of architectures and roughly the same range of number of parameters / flops.
review_title: New NAS benchmark allowing independent searchable blocks

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: BLOX introduces a new macro NAS search space and explores NAS block algorithms through research questions regarding distillation. The contribution consists of a precomputed benchmark and insights from controlled experiments. 
strengths: - The benchmark precomputes 95K architectures that accelerates architecture evaluation for future NAS research. 
- Experiments make a strong case for fine-tuning, which is effective with a good teacher. 
- Block signatures show promise as a performance predictor. 
- Runtime under different hardware settings is measured and provided. 
- Code is open-source in anonymized repository, alongside good benchmark documentation in the appendix. 
weaknesses: - Benchmark is only evaluated on CIFAR-100. This choice should be justified. Typically CIFAR-10 is the common denominator for NAS benchmark datasets. 
- Another advantage of multiple dataset is the possibility of studying distilling the teacher on a new dataset, which could demonstrate this methodology’s usefulness. This study is not enabled in the current setting. 
- I remain doubtful of the search space’s quality, given it’s best accuracy of 76.6% on CIFAR-100. P-DARTS reports more than 80% accuracy for most methods in DARTS-like search space on this dataset (see [this link](https://arxiv.org/pdf/1904.12760v1.pdf)). Therefore, the superiority of macro search spaces is not clearly established. 
- Only a limited number of NAS methods are evaluated on this tabular benchmark, which excludes weight-sharing (one-shot) methods that are state-of-the-art. 
- Additional seeds for the experiments are forthcoming. 
- The code repository does not seem to contain scripts for experiments in Sec 3. 
correctness: Claims are correct except for those discussed in the weaknesses above. 
clarity: Yes. 
relation to prior work: Yes. 
additional feedback: I'm looking forward to engage with the authors on the issues mentioned.
review_title: Interesting NAS study but lacking as a benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Authors release a macro block based NAS benchmark called BLOX that includes 91k models trained on cifar100 dataset. This benchmark includes the performance and runtime measurements on different hardware platforms. Authors also perform comprehensive comparison between the block-wise NAS and cell-based NAS.
strengths: This is a timely work on the important research topic (NAS), and the paper is well written and organized. Authors conduced comprehensive experiments to characterize the blockwise NAS.
weaknesses: I don't think this paper has any obvious shortcomings, but it can be improved in some small parts:
(1) The authors should clarify the reason or insight of your model architecture search space. Why do you design this way and what are the benefits?
(2) I appreciate the comprehensive experiments conducted against Q1-Q6, though they are interesting, but a little bit hard to follow, for example, why do you ask these questions and how are they connected, and please also highlight the concrete answer (bold) for each question.

correctness: The claims made in the paper are mainly correct.
clarity: Well written.
relation to prior work: Yes
additional feedback: Nope.
review_title: An interesting benchmark on macro NAS algorithms

Meta-review: metareview: This paper introduces a new NAS benchmark with 95k architectures evaluated on a single dataset. 
Criticisms were quite diverse, including the novelty and design of the search space, the use cell-based baselines with much smaller search spaces, the limited number of algorithms being benchmarked, and the limitation to a single dataset, CIFAR-100. However, most of the criticisms were addressed during the rebuttal, leading to several reviewers increase their scores.
Overall, all reviewers are in favour of acceptance, some of them clearly so. I therefore recommend acceptance as a poster.
confidence: 3: The area chair is somewhat confident
Decision: Accept