Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work analyzes commonly used heuristics for training the supernet in weight sharing NAS. The authors first proposes a new metric, sparse Kendall-Tau, to measure the quality of the supernet. Then extensive experiments are conducted on three NAS benchmarks to empirically evaluate the heuristics, and pick the best settings. To highlight the significance of the training quality of supernet, the author showed that random search, when combined with the best settings, can performs competitive to SOTA results. 

Strength:

1. The supernet training quality is an important factor in weight sharing NAS, which, as argued in this paper, has not received enough attention. This work raises the concern about this issue and empirically shows that it can indeed affect the search result significantly. 

2. The systematic benchmarking of different heuristics is valuable and serves as a useful guide to apply weight sharing NAS effectively. 

Weakness:

1. The experiments are all conducted on relatively small search spaces. However, the search spaces in more realistic settings are usually much larger and might have different properties. For example, a recent work (Bender et al, 2020) showed that, contrary to the results on the small search space, random search significantly underperforms weight sharing results on large search space. So it would be interesting to see if the conclusions in this work still holds on large search spaces. 

2. The proposed sparse Kendall-Tau metric is shown to be more robust. But it also seems to be more coarse grained. For example, if the differences of top models are small, they would be put into one group by sparse Kd-T thus neglecting their relative ranking. However, those relatively rankings is also important in getting the best search outcome / selecting the best model. 

3. It is probably expected that the low-fidelity estimates would make search results worse since they are trading off search quality with efficiency. It would help to calibrate the comparison based on the search budget. For example, the comparisons in Figure 11 should be done while controlling the search budget for each setting and it would be even better to show multiple comparisons under different search budgets. 

4. More information about the loss landscape visualization would be appreciated, especially how the losses are computed for the supernet. Since the supernet can be seen as a larger standalone network, it is unintuitive to me why its loss landscape would be so different.  

5. The presentation could be improved to make the main message more clear. Some specific points below:

Figure 1 is quite confusing. From the description, it seems the main message is that, from (a) to (c), more and more parts in the whole process are controlled, so it would help to align the elements from different sub-figures to make it easier to compare and see the highlighted differences. Right now, it is a bit cluttered and hard to see the main point. It would also help to simplify the elements. For example, the sub-figures seem to be also describing the differences in the search algorithm's relationship with the other components, but it is not very clear and there isn't any discussion about it. 

The use of "the proxy task" seems confusing. From my understanding, this term is usually used to refer to a surrogate task, when the target task is too expensive to search on. An example from (Cai et al., 2018): "they need to utilize~\emph{proxy} tasks, such as training on a smaller dataset, or learning with only a few blocks, or training just for a few epochs. These architectures optimized on proxy tasks are not guaranteed to be optimal on the target task.". However, this paper seems to use "proxy task" to refer to the target task that the NAS is optimizing for. It would help to clarify the difference in the paper. 

The term "super-net accuracy" and how it is computed should be defined early on since it is used quite frequently and could be ambiguous to the reader. 

Typo:

Appendix B. "This is not a reliable metric, as shown in Fig. 9 in the main paper." ==> I guess it should be "Fig. 8" instead of "Fig. 9" since Fig. 9 is discussing "Low fidelity estimates on NASBench-201".


Bender, Gabriel, et al. "Can Weight Sharing Outperform Random Architecture Search? An Investigation With TuNAS." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

Cai, Han, Ligeng Zhu, and Song Han. "Proxylessnas: Direct neural architecture search on target task and hardware." arXiv preprint arXiv:1812.00332 (2018).

review_title: Interesting empirical study of supernet training. Some more information needed and the writing could be improved.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
The paper analyzes the effect of different components of the supernet training process w.r.t performance of the final NAS models. Finally it makes some recommendations for good practices for training supernets that eventually lead to better NAS designed models.

Strengths:
+ The performance of many NAS approaches is critically dependent on the effectiveness of Supernets. As such, a thorough study of supernet training process is timely and useful.
+ The final recommendations are useful, albeit unsurprising. Standalone or supernet, training hyper-parameters usually play a big role.
+ The discussion on batch norm is helpful. This is indeed a problem with some kinds of supernet training methods.

Weaknesses:
- I did not find information on the supernet training process that is used. There are many supernet training method proposed in the literature but it is not clear what is similar or different in the paper training process. I looked for this but could not find it, it is possible I missed it, but this seems to be important information that should not be hard to find.
- As existing supernet models (e.g., OFA, SPOS, BigNAS) show, sampled subnets can be directly used, with the same level of accuracy compared to training from scratch. This seems to run counter to the main premise of the current paper, which is that the supernet only provides an architecture, which then needs to be trained properly. As such it is not clear how much of the analysis in this paper is applicable to current supernets, and consequently how useful this analysis is.
- Following from the above, the OnceForAll supernet is one of the most effective one, which does incorporate some of the suggestions from this paper. There does not seem to be much discussion of the OFA supernet apart from the OFA conv.
- Parts of the paper are very confusing and not easy to understand. As an example, in section 2, $f_{ws}$ is introduced as a mapping but not defined until much later. 

Overall, the paper analyzes the effect of different components of the supernet training process. While parts of the paper are interesting, it does not discuss the training process of some of the latest and most effective supernet models. As such it is not clear how useful the findings  of this paper are.
review_title: Analysis of Supernet Training Process for NAS

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper aims to analyze 14 commonly used training tricks for weight-sharing based NAS algorithms. The empirical analysis is evaluated on three existing NAS datasets.

Pros:
- A comprehensive analysis of commonly used NAS training tricks.
- A new metric -- sparse Kendall-Tau -- for the correlation between architectures.

Cons:
- No new ideas are proposed, as the authors are evaluating existing tricks.
- Most of the conclusions are somehow already revealed in previous NAS works.
- As an emprical analysis paper, the authors do have some interesting finds. However, could these finds be used to improve existing NAS algorithms (which mostly already integrate multiple different training tricks)? A more concrete example is could the authors use the proposed findings to outperform the recent TuNAS?
- The authors claimed to evaluate and analyse 14 factors, and use one page to explain the behavior of BN in Sec.4.2. However, "the proposed modification" is already mentioned and analyzed in previous papers, such as https://arxiv.org/abs/2001.00326 and https://arxiv.org/abs/2008.06120.
- The explanation in Sec.4.3 is unclear to me. The learning rate schedule does not consider the commonly used warmup strategy. From Figure 8, it seems 0.01 results in the best correlation, but the authors choose 0.025 without further explanation. The authors mentioned "requires a smaller lr", whereas 0.005 is worse than 0.01. Overall speaking, in Sec.4.3, the loss landscapes looks interesting to me, but other analysis is not convincing to me.
- All the experiments in this paper focus on the DARTS-like search space. There are also some other popular search spaces, such as MobileNet-like search space. It remains a question on how does the conclusion generalize to other search spaces.


Minor issues:
- In Figure 1, to make it self-contained, I would suggest to explain P_{ws}, f_{xxx} in the caption.
- In Figure 1, "NAS-Bench-102" -> "NAS-Bench-201"
- In Figure 1, why use "Proxy-task Training protocol" for the evaluation phase? In my opinion, the search is to proxy the evaluation task?
- Regarding the presentation, f_{xxx} and P_{xxx} in Sec 2 take me a lot of time to understand. If the authors could add some examples when introducing these concepts, it could help readers understand.
- In Sec.4.4, "The impact of this approach on the super-net quality, however, has never been studied." is somewhat arbitrary. The results of #cells and #channels for DARTS is reported in their rebuttal, and I believe many researchers try these hyperparameters in their experiments but just not mentioned in the paper.
- In Sec.4.5, the reference before "linearly interpolate" should be "Network Pruning via Transformable Architecture Search" instead of the cited one.
- "SNAS" is accepted by ICLR 2019, "FBNet" is accepted by CVPR,  "Auto-DeepLab", etc; I would like to suggest authors cite their official version instead of arxiv version.
- I'm confused about "disabling dynamic channeling". Would the authors mind to explain more or share the codes on how to do it? In addition, it seems this strategy can only be used for NAS-Bench-101 search space, instead of others.

Some questions:
- I'm confused about Figure 4 - 2/3. How could the x-axis be "sparse KdT"? If x-axis is "sparse KdT", it means each data architecture has a "sparse KdT" value? But if I understand correctly, "sparse KdT" is evaluated on a set of pairs, instead of a single data.
- In Sec.3.2, why does less smoothed loss landscape indicate a smaller learning rate?
- Would the authors mind adding the results of #channels=32 / #batch size=32/512 / #cells=1,5 in Figure 9?


**Post Rebuttal Comments**:
I appreciate the authors' efforts on the rebuttal, most of my questions are answered and addressed. However, I still have a major concerns. As agreed by the authors' response, this paper lies in a detailed analysis of existing tricks. I thus hope the paper could bring some new insights, but the mentioned "other important conclusions" in response is somehow commonly known in the NAS community. This makes the paper like some empirical supplementary material for existing papers instead of a new one. Therefore, I keep my original rating.
review_title: [Initial Rating -- Borderline] Lack evidence on large-scale dataset

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Summary
Neural Architecture Search (NAS) aims to find a model with the best possible accuracy (or best possible accuracy/size tradeoff) from within a human-defined search space. One popular strategy for speeding up NAS is to train a one-shot model -- a single set of shared weights -- that can then be used to evaluate any candidate architecture within the search space without retraining or fine-tuning. The submission investigates how various decisions made when training a one-shot model can affect its ability to rank different candidate architectures within a search space.

The submission presents an empirical study, and I believe its main contribution is to provide information about (i) how sensitive one-shot model quality is to hyper-parameter tuning, and (ii) how to tune the hyper-parameters of a one-shot model in order to produce high-quality search results. These results are likely to be of practical interest to people trying to train high-quality one-shot models for neural architecture search, as well as for researchers trying to benchmark and compare different NAS algorithms.

The submission presents several sets of experiments. The most significant set of experiments (in my opinion) is Figure 11 ("influence of factors on the final model"), which shows the effect on the quality of the final searched model when we tune different hyper-parameters of the one-shot model, such as the learning rate, number of training epochs, weight decay, sampling scheme, and batch norm implementation, or apply various heuristics for decreasing the training time and memory usage of the one-shot model. Several additional sets of experiments are presented throughout Sections 4 and 5 of the paper.

In general, the paper is clear and well-organized, and most of the experiments are well-described; two relatively minor exception are (i) Figure 1, which I had trouble making sense of initially, and (ii) some of the experiments on the right-hand side of Figure 11 (e.g., "ofa kernel," "op-edge"), which include citations but where the authors' precise setup doesn't seem to be clearly defined in the main paper.

**Pros:**
* Conducts a broad set of experiments evaluating the effect of different training decisions on the quality of a one-shot model.
* The experiments and concrete recommendations made by the paper are likely to be of practical value for practitioners/researchers who want to train high-quality one-shot models.
* Experiments reported on a variety of different search spaces/benchmark tasks.

**Cons:**
* Experiments are limited to CIFAR-10.
* Some unresolved questions about a few of the experiments. (Discussed below.)

**Notes on Paper Rating**: I have some concerns about how a few of the experiments are presented  (detailed below) which would need to be addressed before I felt comfortable advocating for the paper's acceptance. If they are adequately addressed, however, I think the paper could be a valuable resource for NAS practitioners.

## Notes on Empirical Evaluation
The submission conducts experiments on (certain subsets of) the NASBench-101 and NASBench-201 benchmark tasks, as well as a third task (attributed to Radosavovic et al., 2019) which the submission refers to as DARTS-NDS. One caveat is that all the conclusions in the paper appear to be drawn based on experiments on the CIFAR-10 image classification dataset. While it's unclear how well the results would generalize to other datasets, I think this is an acceptable limitation, given (i) the broad set of experiments conducted on this dataset, and (ii) most existing NAS Benchmark tasks I'm aware of are built around CIFAR-10.

At various points in the paper, the quality of the one-shot model is measured according to one of three different metrics: (i) "sparse Kendall Tau", which measures the correlation between accuracies measured according to the one-shot model and ground-truth accuracies obtain by training network architectures from scratch, (ii) the ground-truth accuracy of the network architecture which is ranked highest according to the one-shot model, and (iii) the probability that a network architecture found using a one-shot model will be better than one found using a random search baseline without weight sharing. What this means in practice is that different sets of experiments use different metrics. This leads to a slightly inconsistent presentation. For example, the experiments in Section 4.4 ("lower fidelity estimates") focus on Sparse Kendall Tau, while the experiments in Section 5 / Figure 11 ("Influence of factors on the final model") focus on ground-truth accuracies. However, experiments within the same section/plot are typically comparable to each other.

The weakest part of the paper is Table 3 ("Final Results") which compares the results of the authors' well-tuned one-shot model against previously published results. The raw numbers are compelling, but I had trouble verifying the comparison between the "Random NAS" and "Random NAS (Ours)" rows of the table. In particular:
1. The 87.66 accuracy number for "Random NAS" in NASBench-201 was obtained by sampling 100 random architectures from the search space and retaining only the one with the highest one-shot model accuracy. It was unclear to me whether "Random NAS (Ours)" used the same evaluation protocol.
2. It was unclear to me how the 91.33 accuracy number for DARTS-NDS Random NAS was obtained, since I was unable to find that number when I skimmed Radosavovic's paper.
3. If I understand the table caption correctly, the "NASBench-101" table header is misleading and should be changed to "NASBench-101 (n=7)".
**Because the improvement of "Random NAS (Ours)" over "Random NAS" is highlighted in Section 5, these points would need to be addressed before I felt comfortable recommending the paper for acceptance.**

## Additional Notes
In Section 4.4: It would be helpful to explicitly state how the learning rate is adjusted when you decrease the batch size, since this can significantly affect the final results.

In Section 4.5: It would be helpful to update the submission to clarify: if you disable dynamic channels and train a separate model for each possible number of incoming edges, do you include comparisons between candidate architectures with different numbers of incoming edges when you compute the sparse Kendall-Tau? (If not, it seems like the sparse Sparse Kendall Tau numbers for different methods might not be directly comparable to each other.)
review_title: Paper Review

Meta-review: This paper presents an analysis of different tricks for training the super-network in NAS. While all reviewers see value in some of the many experiments, all reviewers also have substantial criticisms of the paper, and all reviewers gave weak rejection scores.

Looking at the paper myself, I agree with this assessment. Several of the experiments are valuable, but there are also several substantial issues.

One question that confused two reviewers and myself is about using sparse Kendall's tau as a metric that the authors in the rebuttal again state can be computed during super-net training to evaluate the quality, just like super-net accuracy. I don't see how that is possible. Kendall's tau measures the correlation between the ranks of the performances of the stand-alone architectures and the ordering implied by the super-net. Computing this requires access to the performance ranks of the stand-alone architectures. For tabular benchmarks this is of course available, but not in practical NAS applications.

I would also like to echo the concern of AnonReviewer2 that too little information is given to fully understand what is shown in Figure 10.

Some reviewers also questioned inhowfar the results generalize to the setting of the Once-for-all-network or BigNAS. This was not a deciding factor for me, since insights based on NAS-Bench101, 201 and a DARTS-like search spaces are already very useful.

I agree with the reviewers that the authors' use of "proxy" is highly misleading. It is standard to refer to the low-fidelity model used for training as the proxy model. In contrast, the authors use it for the final evaluation model.

Concerning the authors' five final take-aways:
1) I don't see how sparse Kendall's tau is actionable.
2) The batch normalization part is interesting, and I agree with the authors that it is useful to spell this out and analyze it, rather than just having one sentence in the paper as NB-201 and TuNAS, but the attribution that this has been done before is broken. "In contrast to X", rather than "Like X"
3) This is interesting, although I agree with AnonReviewer3 that I'm lacking intuition why a smaller learning rate should be useful for a less smooth space
4) The experiment on low fidelity estimates is very misleading. The proxy settings used during training are already low fidelity evaluations -- for the final evaluation, you would increase the number of channels, number of layers and number of epochs. Stating that the use of low fidelities is not useful is highly misleading. The authors' experiments only shows that the proxy model is already well chosen, and that if you reduce #layers or #channels and proportionally increase #epochs, performance gets worse. I encourage the authors to try searching without this proxy model, and I'm sure they will find that (which correlations might increase) the search process will be far too slow.
5) The insight on dynamic channeling appears very useful to me.

In summary, I recommend rejection and encourage the authors to address the points raised by the reviewers and in this meta-review.
Decision: Reject