Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Differentiable neural architecture search (NAS), and more recently, perturbation-based operation selection in differentiable NAS, is a popular recent area of study in NAS. In parallel, zero-cost proxies are also gaining in popularity in NAS. In this paper, the authors combine the insights from perturbation-based operation selection and zero-cost proxies, to create a new method that sees substantial time speedups. Specifically, they formalize "operation scoring", and use zero-cost proxies to score operations with the perturbation paradigm. This leads them to a new NAS method (Zero-Cost-PT) that outperforms existing methods. They evaluate their proposed algorithm on the DARTS search space (and its subsets) and NAS-Bench-201, showing that it can achieve up to 40x speedups.
main_review: ## Strengths
- This is a great combination of two popular areas: zero-cost proxies, and differentiable NAS via perturbation-based operation scoring. It is a natural idea to combine these two things, but also not trivial to do it well.
- These ideas can have a high impact.
- It seems that they give new insights for the celebrated “Rethinking Architecture Selection in Differentiable NAS” paper, including more proof of why DARTS-PT does well, and a refutation of the claim that disc-acc does well.
- The appendix is pretty good and has a lot of information for reproducibility (but for code, see weaknesses).
- Sections 3.1 and 3.2 are nicely done and make sense, and Fig 2 in particular is very interesting.

## Weaknesses
- Over-reliance on NAS-Bench-201. Almost all of Sections 3 and 4 focus on NAS-Bench-201. But this is the smallest search space, and in particular, it is known that zero-cost proxies do well on NAS-Bench-201 and bad on larger search spaces (the authors mention this as well). Although the authors also test their full algorithm on the DARTS search space in Section 5, it would strengthen the paper to do the analysis of Sections 3 and 4 on at least one other search space. NAS-Bench-1shot1 is the best option because it is size 360k and can run one-shot algorithms. TransNAS-Bench-101 can also run one-shot, and it is smaller but would add diversity. Finally, it might be possible to run some of the experiments using the 60k trained architectures from NAS-Bench-301).
- Better ablation between the search and the validation stage. Sections 3 and 4 motivate the search part of the algorithm, but adding in the validation stage makes it less clear what leads to strong performance, especially on DARTS where the search step was never run in isolation. I think it makes it less apples-to-apples comparisons, e.g. because DARTS, DARTS-PT etc do not have the final validation stage. The authors did have an ablation, but it is only on NAS-Bench-201 and does not cover search only ("validation only" is basically NASWOT). In order to have a better comparison between ZC-PT and DARTS-PT, the authors could run ZC-PT once with no validation stage. Or, run ZC-PT and DARTS-PT both V times and do the validation stage for both of them. For Tables 2, 4, 5. As a result, I think the claim in the intro that "Our novel Zero-Cost-PT improves searching time and accuracy compared to the best available differentiable architecture search for many search space sizes, including very large ones." is misleading. 
- The authors claim in Section 7 that they released their code in the supplementary material, but unfortunately there was no supplementary material submitted. If the authors provide the code during the rebuttal period, I will feel more positive about the paper (for example, anonymous github).

summary_of_the_review: This paper combines two popular techniques in a nice way. I think the ideas in this paper can be pretty impactful. I do have three weaknesses about reliance on NAS-Bench-201, a fairer experimental setup, and code release, so I will give a weak accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors introduce new training-free proxies to the context of differentiable NAS, which can speed up the search process while improving accuracy. Further, the authors propose, evaluate and compare perturbation-based zero-cost operation scoring (Zero-Cost-PT) for differentiable NAS. Extensive experiments empirically show the effectiveness of Zero-Cost-PT in six search spaces and 3 datasets. 
main_review: Pros:
This paper introduces the lightweight operation scoring based on zero-cost proxies, which empirically outperform existing operation scoring functions. The experiments show that the perturbation is more effective than discretization and propose Zero-Cost-PT. Extensive experiments empirically show that the proposed method outperforms the best available differentiable architecture search in terms of searching time and accuracy. 

Cons:
The zero-cost proxies method has been used in the previous work. This work only borrows it into differentiable architecture search, which is only a combination. There exist some typos and it is a little difficult to understand Figure 1. Some annotations can be moved to the caption. 

summary_of_the_review: Generally speaking, I think it is incremental work. Although there is a lack of theoretical contributions, a large number of experiments provide empirical contributions. It is difficult for me to put forward some new opinions about these large numbers of experiments. I give the above acceptance threshold, but reject is OK.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use the zero-cost proxy in the operation selection in differentiable neural architecture search. Specifically, the paper formalizes the selection procedure into perturbation and discretization. By introducing the zero-cost proxy, they first find the proxy could yield a better ranking compared to both Darts and Darts-PT in the Nas-Bench 201. They also experiment with different proxies proposed by previous work and find they all perform better than the Darts-PT. In the Darts-CNN and S1-S4 experiments, they select one of the best-performed proxies and find it could yield better accuracy than other baselines.
main_review: Pros:
1. The paper is well-written and easy to follow. However, analyze part is slightly too long however the paper is in general in good shape.
2. The experiment results look promising and interesting. It could achieve better accuracy across several benchmarks and some spaces, which could be useful to the community.

Cons:
1. The paper's novelty is quite limited. It simply combines the differentiable neural architecture search with the one-shot one. Although it has been shown and addressed across the paper the zero-cost proxies are quite useful, however, it lacks insight on how it is correlated with the final performance ranking. 
2. At the same time, all the zero-cost proxies are proposed by other works and there is limited understanding of why some specific proxies are clearly better than others in the differentiable setting but not in the one-shot setting. 
3. Although the paper formalizes the problem into perturbation and discretion, I find only discretization is used in the final experiments, where the formalized framework seems not that important.
summary_of_the_review: Although the paper has shown us the zero-cost proxy has great power in operation search in the differentiable neural architecture search, the paper stills lack insight on analyzing how it works. Therefore, I rate it as a borderline paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work combines training-free proxies and perturbation-based subnetwork evaluation to achieve efficient NAS. The author also conducted comprehensive study of training-free proxies under different training iterations.
main_review: Strength:
This work studied different training-free proxies and leveraged perturbation-based methods for NAS.

Weakness:
My main concern is the novelty of this work. Specifically,
1. This work did not propose any new proxies for evaluating networks performance. If I understand correctly, the proxies listed in Table 2 are all from previous works. The authors did not claim any novelty in these training-free proxies either. Although the authors mentioned "opportunity to use a new class of training-free proxies" in Sec. 3.1, there is no concrete proposal in this work.
2. The operation sorting method (Eq. 5) is also similar to the perturbation method proposed by [1]. The main difference is that the authors adopt this perturbation at supernet's initialization.

[1] Rethinking architecture selection in differentiable NAS.
summary_of_the_review: In addition to the novelty issue mentioned above, some results and description of this work also makes me a bit confusing. For example, when demonstrating Figure 2, the author did not mention which concrete training-free proxy they used for $S$ in Eq. 5. Also, the purpose of "retrain for 5 epochs" is unclear to me. Why does the author want to conduct this experiment given the "zero-cost" motivation in this work?
review_title: null

Meta-review: This paper received scores of 5,5,6,8. The reviewer giving a score of 8 stated that they would've given a 7, but that that is not an option in the system. The other reviewer giving an acceptance scores mentioned that they would also be OK with a rejection. The details of the assessment are thus less enthusiastic than could be assumed with an overall average score of 6. I am therefore weakly recommending rejection.

The main criticisms of the reviewers are lack of novelty, lack of deeper analyses that really provide insights into why zero-cost operation scoring works, and lack of the number of NAS benchmarks tested. Out of these, personally, I would not criticize a lack of novelty, since it is not trivial to put together zero-cost and one-shot methods and the results appear promising.
However, even the most positive reviewer criticized that the work focuses on NAS-Bench-201 heavily (which is particularly problematic given that NAS-Bench-201 uses a fixed wiring and only allows the choice of operations; this may make the proposed method particularly applicable). During the rebuttal, the authors added NAS-Bench-1shot, which is a very good step, but the proposed technique does not actually work as well there. While this may be due to the special nature of operations in the nodes rather than in the edges for NAS-Bench-1shot1, for a revision, it would be good to add additional experiments on further NAS benchmarks in order to allow for a better understanding under which circumstances the proposed method works well. In particular, it would be interesting how well the method works on a quite different search space, such as the one of MobileNet.
Decision: Reject