Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This study proposes to use zero-order optimization methods for neural architecture search based on the empirical observation that the approximation of differentiability will distort the loss landscape and lead to the biased objective and inaccurate gradient. Experiments on multiple datasets are conducted to show the improved stability and performance of the search results. 
main_review: The empirical analyses are reasonable and sound. The proposal to use zero-order optimization methods is novel and interesting. The paper is well-written. I mainly have the following concerns.

1.	Since the authors use zero-order methods to optimize the architecture parameters, there is no requirement that the loss is differentiable wrt the architecture parameters. In this case, can we optimize the binary architecture parameters directly without the need to introduce the continuous softmax relaxation? 
2.	The zero-order optimization methods introduce much more computational cost than the first-order gradient-based method, which causes the high search cost. So, it would be difficult to directly search on ImageNet. Considering many NAS methods have already been able to directly search on ImageNet, this drawback limits the potential applicability of the proposed methods.  
3.	I think the stability of a NAS method can be claimed only when the method is able to produce similar searc results stably. Can the authors prove that multiple implementations of the proposed search method would lead to the same architecture or similar architectures with close performances? The result of Figure 2 is interesting. But I think the trends in Figure 2 are of randomness and difficult to reproduce. It is better to report the averaged trends of multiple implementations.  
4.	The search results are not significantly better than some baselines. On ImageNet, the result is not better than PC-DARTS (ImageNet) that has 24.2% err. as reported in [1]. Besides, many studies are not cited and compared with, such as [2,3,4,5]. 
5.	Is the proposed method applicable to the chain-based search space adopted in [6,7,8]? 


[1] Xu et al., PC-DARTS: PARTIAL CHANNEL CONNECTIONS FOR MEMORY-EFFICIENT ARCHITECTURE SEARCH

[2] Liang et al., DARTS+: Improved Differentiable Architecture Search with Early Stopping

[3] Chu et al., DARTS-: ROBUSTLY STEPPING OUT OF PERFORMANCE COLLAPSE WITHOUT INDICATORS

[4] Yang et al., ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding

[5] Zhou et al., Theory-Inspired Path-Regularized Differential Network Architecture Search

[6] Chu et al., FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search

[7] Yu et al., BigNAS: Scaling up Neural Architecture Search with Big Single-stage Models

[8] Mei et al., AtomNAS: FineGrained End-to-End Neural Architecture Search

summary_of_the_review: I vote for a weak accept currently considering the novelty of the proposed methods. But the concerns above need to be well addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents ZARTS, a zero-order optimization method for DARTS, to search without enforcing the approximation of the network weights. It conducts in-depth analysis on the first/second order approximation in DARTS, and points out that such approximation leads to bias and instability. Then the work proposes three zero-order optimization methods to solve the issue.
main_review: The instability of DARTS has always been an topic in the NAS area, and has drawn a lot of attention. Many works have investigated the issue and propose methods to alleviate it.

This paper, conducts analysis on the first/second order approximation of network weights and points out that such approximation leads to the instable optimization of architecture parameters. Consequently it proposes to solve the problem via zero-order optimization, and propose three zero-order optimization methods, which is very novel. Experiments demonstrate the effectiveness of the proposed methods.

The weakness:
1. The increased search cost due to the sampling and estimation process.
2. Though the experimental results improve against original DARTS, however these tasks are explored by plenty of works and an error rate of around 2.50 and 24.3 on CIFAR-10 and ImageNet are achieved by many works and are not that outstanding. More experiments may strengthen the results.

Some questions for the authors:
1. In ZARTS-GLD, it seems that the network weight $w^{*}$ is estimated by the current weight $w$ and the method searches within a small area around current architecture parameter $\alpha$. Is this estimation accurate and why?
2. In section 5.1, convergence analysis, do the architectures derived every 25 epochs change or do they keep the same after several epochs?

Overall, I think this is a good work.
summary_of_the_review: This work presents zero-order optimization methods for DARTS, which is very novel in the area. The paper is well-organized, with sufficient analysis and sound technical contribution. I think it is a good work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper (ZARTS) proposes to apply gradient-estimation-based zero-order optimization methods to tackle neural architecture search (NAS). Two major contributions are made by this paper: (1) it is the first to borrow the methods of zero-order optimization to solve NAS  problem; (2) it shows that zero-order optimization methods can greatly avoid the training instability of first- or second-order optimization NAS methods like DARTS (which sharpens the loss landscape). Moreover, this paper also provides an explanation on how ZARTS is connected with DARTS. Extensive experiments demonstrate the efficacy of ZARTS.
main_review: Pros
++ The analysis of how first-order optimization sharpens the landscape of loss (fig 1) is interesting. It sheds some lights on why first-order DARTS faces the problem of instability during the search. 
++ The paper is the first to apply zero-order optimization methods to solve NAS, which could open a new research direction of NAS.
++ The analysis on connecting ZARTS and DARTS provides a theoretical explanation on why ZARTS can work better than DARTS.
++ The experiments are extensive and can demonstrate the effectiveness of the proposed ZARTS.

Cons
-- The three zero-order optimization methods are all proposed by other works, which weakens the novelty of this paper.
-- The search costs of ZARTS are not appealing comparing to other differentiable NAS methods like P-DARTS or FairDARTS (Table 4). Moreover, the top-1 error rate of ZARTS is similar to that of P-DARTS, while being slower at search.
- The superiority of ZARTS is not obvious in Table 7 comparing to SDARTS.
summary_of_the_review: Overall, it is a good paper that opens a new research direction of differentiable NAS family. The quality in terms of visualization, theoretical proof and empirical evaluation are also good. So, I recommend an 'accept' for this paper. The concern preventing me from giving higher rating is that all the optimization methods are directly obtained from previous works.
review_title: null

Meta-review: The paper proposes a series of zeroth order optimization approaches to stabilize DARTS training. Although the reviewers think that zeroth order approach is novel to the NAS community, they also point out several weaknesses. In particular, the method will introduce extra computation time and the results are not really standing out comparing with other state-of-the-art methods. Therefore, despite some interesting ideas are presented in the paper, we decide to reject the paper and encourage the authors to address those weaknesses in their future revision.
Decision: Reject