Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the relationship of correlation of ranking of networks sampled from SuperNet and that of stand-alone networks under various settings. They also study the how masking some operations in the search space and different ways of training effect the ranking correlation.

Pros:
The paper has a lot of experiments to substantiate the claims.
Figure 3 where every operation is systematically masked, provides more insights about which operations are effective and how NAS behaves if one of the operation is masked.

Cons:
Several other papers have already published similar findings. Overall the paper is very incremental.
More specifics in the questions

Questions

1. How is the SuperNet trained?
2. Figure2: Yu et al [1] have already explored the correlation of ranks of networks sampled from SuperNet and that of stand-alone networks. How is Figure 2 different from that? 
3. RobustDarts [2] has explored the possibility of how subset of NASBENCH search spaces behave. FAIRDarts [3] also explored the influence of skip connection by running DARTS without skip connection, running random search by limiting skip connection to 2 etc. Figure 4 seems to be inspired by that. While it is interesting, this might be a slight extension to the work done by Yu et al [1]
4. Bender et al [4] postulate that the operations of a SuperNet are subject to co-adaptation and recommended techniques such as regularization, drop path etc to alleviate the same. RobustDarts also suggest some recommendations such as L2 regularization, drop path etc although in the context of DARTS. So while Figure 6 demonstrates this empirically, it is not a new finding.

Overall, the empirical results in the paper are very useful for the NAS community. But the work is still very incremental. This might be better received as a workshop paper instead.
review_title: Official Blind Review #2

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces an empirical study on the ranking correlation in the singe-path setup. Following the paradigm of NAS-Bench-201,  the authors test the Kendall Tau correlation of networks from NAS training and networks from standalone training. 

In general, I appreciate the authors' effort in bringing more infrastructure to the community of NAS. As a recently emerged community, we do need works like this one, as well as previous ones such as NAS-Bench-101 and NAS-Bench-201, to make the evaluation protocol more scientific. NAS problems are non-trivial as the search space is notoriously large. Colleagues who would like to invest their time and resources in exploring and manifesting this search space to uncover more phenomena are thus worth respect. 

However, this respectable responsibility also comes with a higher standard to evaluate works attempting to fulfill it. My major concern with this work is that the manuscript is not organized well. Although authors provide substantial details on their empirical study, they did not form a coherent logic flow to present these empirical findings, which makes this work more like a technical report than an academic paper. Readers may find these phenomena interesting but may not get interesting insights after reading this paper. Hence the technical contribution, especially on novelty, seems quite limited, even if there may be some intriguing points in the authors' discovery. I would recommend the authors to pick some phenomena e.g., masking Zero, masking Skip, etc., as examples to provide more analysis, so as to demonstrate to colleagues in our community that these findings can indeed lead to interesting research topics. 

Some minors: There are some works missed in the literature review. For example, the authors did not give adequate credits to colleagues who pioneered in using Kendall Tau to evaluate NAS training. As far as I know, Sciuto et al., 2019 was one of the earliest works. In the third paragraph, when reviewing recent progress, the authors did not distinguish the ranking correlation between NAS searching and retraining from the correlation between NAS searching results and stand-alone training. The former one was discussed and addressed in Hu et al., 2020. As the community has not fully realized the subtle but crucial difference between these two correlations, I believe a better framing of this work can be more helpful to other colleagues, especially those new comers. 

Sciuto et al. 2019, Evaluating the search phase of neural architecture search.

Hu et al. 2020, DSNAS: Direct Neural Architecture Search without Parameter Retraining
review_title: An empirical study on the ranking correlation of several NAS mechanisms in the singe-path setup

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
+ This paper studies the single-path one-shot super-network predictions and ranking correlation throughout an entire search space, as all stand-alone model results are known in advance. This is a crucial step in NAS. As we know, inaccurate architecture rating is the cause of ineffective NAS in almost all existing NAS methods. It makes nearly all previous NAS methods not better the random architecture selection (suggested by two ICLR 2020 papers and many ICLR 2021 submissions). Therefore, analyzing the architecture rating problem is of most importance in NAS. This paper takes a deep insight into the architecture rating problem, which provides a timely metric for evaluating NAS's effectiveness. (+)


- In the following text, another paper entitled "Block-wisely Supervised Neural Architecture Search with Knowledge Distillation" should be discussed: "Recent efforts have shown improvements by strictly fair operation sampling in the super-network training phase (Chu et al. (2019b)), by adding a linear 1×1 convolution to skip connections, improving training stability (Chu et al. (2019a)), or by dividing the search space (Zhao et al. (2020))," (-)


- Kendall's Tau is a good metric. As shown in EagleEye, Spearman Correlation Coefficient(SCC) and Pearson Correlation Coefficient (PCC) are also good metrics. Could the authors also provide a comparison using these two metrics? (-)

EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning


- I think NAS-Bench-201 is not enough. As we know, CIFAR-10 is sometimes considered a toy benchmark, and the sole result on CIFAR-10 is not convincing. Could the authors provide more results in addition to CIFAR-10? (-)


- As we know, there may be a gap between the small-channel supernet and the large-channel finally-adopted architecture. We are quite interested in the ranking correlations between a subnet obtained from the small-channel supernet and a channel-expanded version of the subnet trained from scratch. Could the authors provide such a ranking correlation analysis? (-)


- Could the authors provide more details in Figure 3. Figure 3 shows that the lines on the top mean the operation is used more frequently. But I am not sure what the value of the y-axis means. (-)


- Could the authors present some comments on "Perhaps the most surprising is the low importance of Average Pooling, even lower than Zero, an operation that does absolutely nothing"? (-)


+ The following observation is believed to be crucial in NAS: "The baseline for small networks (top left, red) has the same averaged prediction accuracy for the top 10 as for the top 500 networks". This validates the inefficiency of SPOS in architecture search. (+)


+ The following observation is also important in NAS: "Masking Skip (blue, left) is the most harmful to τa (=1). As seen in Figure 4, the top-N networks have a worse average predicted accuracy than the top-M (for N < M) networks, and sometimes even below the random sample, which is terrible. Interestingly, \tau may improve within the predictions for the top-N architectures." Especially, the phenomenon that masking skip connection reduces the ranking correlations is interesting. As is shown in SCARLET-NAS, the supernet training with skip connection is not fair. But in this paper, we can see that skip connection benefits the ranking correlation. We are interested in this opposite opinion. Specifically, it is fascinating to see that "Although the additional transformers seem to stabilize training, as seen by the lower standard deviation, they also worsen the τa problem." Besides, the phenomenon of "\tao may improve within the predictions for the top-N architectures" indicates that the metric for ranking correlations maybe not perfect. A more reasonable metric may be desirable. (+-)


+ The following observation is important: "medium-sized super-networks require additional care." As shown by Figure 4, the averaged predicted accuracy of top-N networks in several subsets is lower than that of a random subset of networks. This is consistent with previous work like DNA, which shows a large search space may be harmful to the architecture rating. Even if a medium-sized supernet has a bad architecture rating, the ranking correlation should be worse in a large-sized supernet. (+)

DNA: Block-wisely Supervised Neural Architecture Search with Knowledge Distillation


- The following description is questionable: "After the architecture search, all Linear Transformers can safely be removed, as they do not impact the network capacity". Actually, stacking many fully connected layers without non-linear activations could lead to only one fully connected layer. It is an open question of whether optimizing loss(ABCx, y) is as difficult as optimizing loss(Dx, y) using stochastic gradient descent. (-)


+ The results providing evidence against disabling cell topology sharing during the training phase are exciting and new to the public. (+)


+ The following observation is fascinating: "The absolute validation accuracy value is increased by uniform sampling. However, this is not relevant, as only the correct ranking matters". This is against FairNAS. (+)


+ It is interesting and convincing that many tricks such as learning rate warm-up, gradient clipping, and regularization do not work to improve the ranking correlation. We are pleased that the authors provide so many experiments to point out some misleading approaches in NAS. I think this paper is very important in the context of AutoML. (+)


- The analysis is based on medium-sized and small-sized search space. It would be good to see some analysis of large-sized search space. (-)


Overall, this paper provides a timely analysis of the current NAS's ineffectiveness caused by the inaccurate architecture rating problem. As there are many NAS papers published every year and their ineffectiveness may still be not widely recognized by the reviewers and the public, I recommend a strong acceptance for this paper to promote the analysis of the NAS's architecture rating problem.




------------------------------post rebuttal------------------------------------------


-------------------Response to the authors' response----------------------


Thank you for the hard work in responding.

I have read other reviewers' reviews and the response from the authors. The authors have addressed most of my concerns.

I believe this paper deserves acceptance. As we know, variants of efforts have been made to improve NAS's effectiveness since 2016, and a great process has been reached. Despite the high expectation and solemn devotion, NAS's effectiveness is believed to be still low. This is inconsistent with many pioneer researchers' expectations four years ago, in which NAS is expected to be another revolutionary technique similar to 2012's deep learning. Currently, there are many NAS papers published every year. But their effectivenesses are unclear due to the lack of ranking correlation analysis. Differently, this paper comprehensively analyzes the architecture rating problem, which provides a timely analysis of the current NAS's ineffectiveness caused by inaccurate architecture rating. I think this paper can attract the community's attention, encouraging the community to pay attention to the architecture rating in NAS, especially when reviewing a NAS paper. Therefore, I recommend an acceptance for this paper to promote the analysis of the NAS's architecture rating problem.

I agree with R2 that Yu et al. have proposed a similar idea (I assume R2 refers to "Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, Mathieu Salzmann, Evaluating the Search Phase of Neural Architecture Search"). But the analysis in this paper is more comprehensive than Yu et al.'s article. Many findings are new (at least they are not in published papers).

I agree with R4 that the authors did not form a coherent logic flow to present these empirical findings, and the paper was similar to a technique report. However, many important articles, e.g., "Designing Network Design Spaces," "Exploring Simple Siamese Representation Learning," "Is Faster R-CNN Doing Well for Pedestrian Detection?" are also technique-report-like.

I appreciate R1 for his devotion to finding similar observations in his experiments. I believe these observations are important and deserve publication. I agree with R1 that removing any operation leads to a smaller search space and a higher ranking.

In summary, I will keep my rating as an acceptance. 

Undoubtedly, I also believe the comments from other reviewers can benefit the improvement of your paper.









review_title: A timely analysis of the current NAS's ineffectiveness caused by the inaccurate architecture rating problem

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors proposed several findings on the single-path training strategy. The ranking correlation is the main issue. The experiments are conducted on NASBench201. 

Introduction.

'However, the aforementioned weight sharing xxx', there are a number of efficient multi-objective (Pareto front) NAS methods.

'However, since the single-path', please cite some literature related to the ranking issue.

Summarize the main findings in the introduction section.

Method.

Define $\tau_{\alpha}$ in math. 'describing the ranking correlation of the average prediction accuracy depending on N', what is the average prediction accuracy for two lists?

Experiments.

4.1 'masking the Zero operation (bottom row) significantly reduces this portion and thus improves the ranking correlation $\tau$ (KT)'. If removing other operations other than zero, will the $\tau$ be lower? In my opinion, removing any operation leads to smaller search space, and they all have a higher ranking.

4.2, 4.2 examine some of the training strategies proposed by previous works.

References.

Please make the references clear. Add the venues for all the papers. 




This paper tries to explore the single-path training strategy by studying the search space, the supernet, the linear transformer, the strict uniform sampling, the topology sharing, the LR warmup, the regularization, the clipping. The authors have done lots of experiments to clarify the important reasons for the ranking. However, most of the findings are not new to me. They have been discussed more or less by previous works and discovered by my own experiments. So the contribution is not significant. The paper is mostly clear, some paragraphs and references need to be polished, more related works should be added.
review_title: Review

Meta-review: This submission received reviews with a very wide range of scores (initially 3,5,5,9; then 5,5,5,9). In the discussion, all reviewers maintained their general position (although a private message by the reviewer giving a score of 9 said he/she would consider going down to an 8).

Because of the high variance, I read the paper in detail myself. I agree with all reviewers that NAS is a very important field of study, that the experiments are interesting, and that purely empirical papers studying what works and what doesn't work (rather than introducing a new method) are definitely needed in the NAS community. But overall, for this particular paper, I agree with the 3 rejecting reviewers. The paper presents a lot of experiments, but I am missing novel deep insights or lasting overarching take-aways. The papers reads a bit like a log book of all the experiments the authors did, before having gone through the next iteration in the process to consolidate findings and gain lasting insight.

In a bit more detail, half the results in Section 4 use medium-sized super networks, which seem broken to me, yielding much worse performance than small super networks. I did not find any motivation for studying these medium-sized networks, no reason given for them to perform poorly, and none stating why the results are still interesting when the networks perform so poorly (apologies if I overlooked these). The poor performance may be due to using a training pipeline that works poorly for these larger networks, but this is hard to know exactly without further experiments. I would either try to fix these networks' performance or drop them from the paper entirely, as I do not see any insights that can be reliable gained from the current results. As is, I believe these results (accounting for half the plots in the paper) only muddy the water and are preventing a crisp presentation of insightful results.

Another factor that I find unfortunate about the paper is that it only uses NAS-Bench-201 for its empirical study, and even for that dataset, mostly only the CIFAR-10 part. After getting rid of isomorphic graphs from the original 15625 architectures, NAS-Bench-201 only has 6466 unique architectures (see Appendix A of NAS-Bench-201), while, e.g., NAS-Bench-101 has 423k unique architectures. As the authors indicate themselves in their section "Grains of Salt", it is unclear whether insights gained on the very small NAS-Bench-201 space generalize to larger spaces. I therefore believe that there should also be some experiments on another, larger space, to study how well some of the findings generalize. An additional benchmark that the authors could have directly used without performing additional experiments themselves is the NAS benchmark NAS-Bench-1shot1 (ICLR 2020: https://openreview.net/forum?id=SJx9ngStPH), which studies 3 different subsets of NAS-Bench-101, and which was created to allow one-shot methods to use the larger space of evaluated architectures in NAS-Bench-101. 

Minor comments:
- It reads as if the authors performed 5 runs, computed averages of the outcomes, and then computed correlation coefficients. That would be a suboptimal experimental setup, though; in practical applications, only one run of the super network would be run, and therefore, in order to assess performance reliably, one should compute correlation coefficients for one run at a time, and then obtain a measurement of reliability of these correlation coefficients across the 5 runs.
- The y axis in Figure 2 appears to be broken: for example, in the left column it goes from 99.978 to 99.994, and the caption says these should be accuracy predictions of NAS-Bench201. However, even the best architectures in NAS-Bench201 only achieve around 95% accuracy.


Overall, I recommend rejection for the current version of the paper.
Going forward, I encourage the authors to continue this line of work and recommend that they iterate over their experiments and extract crisp insights from their experiments. I also recommend performing experiments with a much larger search space than that of NAS-Bench-201 to assess whether the findings generalize.
Decision: Reject