Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Thanks for your informative response addressing my comments. After the revision, the description of the method is clearer (Sec 3.2), and the experimental results are clearer (Sec 4). I'll stay with my original accept-score.

===============

Summary:

The paper provides interesting results for neural architecture search. In particular, this paper proposes a search strategy for NAS problems, Generative Adversarial NAS (GA-NAS), using importance sampling, which can be applied to micro/macro, constrained/unconstrained search problems. GA-NAS beats the state-of-the-art search algorithms proposed for NAS on public benchmarks, including NAS-Bench-101, NAS-Bench-201, and NAS-Bench-301. Also, on the EfficientNet macro search space, GA-NAS finds a new architecture with higher ImageNet accuracy and a lower number of parameters than EfficientNet-B0.


Pros: 

1. The proposed method achieves higher performance to compare to previous methods with better robustness, reproducibility, and efficiency.

2. The idea of NAS based on importance sampling for rare event simulation in the method seems interesting. The proposed method at the same time could be broadly applied to micro/macro, constrained/unconstrained search problems. 

3. This paper provides comprehensive experiments, including various ablation studies, to show the effectiveness of the proposed framework. 


Cons: 

1. I suggest the authors conduct further ablation studies to enhance the understanding of the approach and readability of the paper: 

(1) Comparison of computational resources (e.g. wall clock inference time) required for each query in Table 1 and Table 3. To be a fair comparison, it would be better to compare [number of queries * resource consumed per query].

(2) For the update algorithm of the generator, the proposed method uses JS-divergence minimization referring to [29]. Section 3.1 mentions that JS-divergence is more robust than KL-divergence, but I think a further analysis could strengthen the point.

(3) Adding FLOPs or inference speed to Table 5 and Table 6 would be helpful in explaining the performance of the new architecture found by GA-NAS.

2. Section 3, 4 need to be polished for better readability. For example, an explanation about the method and the concept of evaluation metric "rank" should be improved for the readers.


Some typos: 

(1) In the section 3.2 “\tau=\{C_0,C_1,\ldotsC_{N-1}\}” -> “\tau=\{C_0,C_1,\ldots,C_{N-1}\}” 
(2) In the equation (10) of appendix, “-(1-\rho) + P(X\leq \zeta^*)\geq 0.” -> “-(1-\rho) + P(X\leq \zeta^*)\geq 0,”

Some suggestions:
"\cdots"s are used in the expression such as "X_1,\cdots, X_N" and "S(X_1),\cdots, S(X_N)" in the proof of the theorem 6.2 in the appendix. I think "\ldots" is more syntactically typical here.

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: A flexible but complex and expensive NAS method.

Summary:
The authors introduce a method for NAS that repeatedly trains a generator to sample candidate architectures. The method is evaluated on three NAS oracle benchmarks as well as constrained NAS settings. While there are some promising experimental results, I lean slightly against acceptance due to poor presentation, limited comparisons on most evaluations, and what seems like fairly limited benefits of the approach given its complexity and cost.

Strengths:
1. The method can easily incorporate constraints on computation and memory.
2. The method outperforms existing non-weight-sharing methods on several benchmarks.

Weaknesses:
1. The method introduces a lot of complexity such as a graph NN, a recurrent NN, and a full round of generative adversarial training in each search iteration. The computational cost of the latter is not discussed.
2. As with most non-weight-sharing methods, GA-NAS requires several hundred queries on each benchmark, which translates to GPU-weeks of search time. It is not clear that the benefits over weight-sharing methods, which are not quantified for most cases, outweigh this large search cost.
3. The results section for unconstrained search is confusing and it is hard to make comparisons (see notes 4-7 below).
4. From a look at the NAS-Bench-301 paper, it seems that BANANAS was the best non-weight-sharing method evaluated, but the authors compare only to EA and RS.
5. In the constrained search section, there are no comparisons to any other NAS methods. For example, random search is just as easy to apply to constrained problems as GA-NAS and should be used as a baseline.
6. There is no code in the supplementary materials. Will code be released?

Notes:
1. “Remain hard to be assessed” -> “Remain hard to assess”
2. The citation style does not follow ICLR guidelines.
3. “architectures are sampled, which are discretized graphs” -> “architectures, which are discretized graphs, are sampled”
4. What does it mean to “discover the Nth best architecture in Q queries”? Is “best” here according to test or validation? Why is Q a good metric for speed given that the algorithm doesn’t know to stop after query Q since in practice it won’t know the rank N of the current architecture?
5. Many numbers in paragraph 4 for Section 4.1 do not correspond to any number in any table.
6. Why isn’t the BANANAS performance bolded in Table 1?
7. Table 4 should include at least one weight-sharing method such as GDAS (Dong & Yang, 2019), which is much faster and performs reasonably well. 

# Post-response update
Thank you to the authors for answering some of my questions and clarifying the search and evaluation of GA-NAS. I believe my original assessment that the contributed method was complex remains accurate; while the authors note that other methods like ENAS also use an RNN controller, in my view those methods are also complex. This paper increases this complexity with a GNN and an adversarial training setup. Use of such additions require showing significant improvements over baselines like random search, which I do not believe is achieved. I thus stand by my initial rating.
review_title: A flexible but complex and expensive NAS method.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a Neural Architecture Search algorithm (GA-NAS) based on adversarial learning. The generator constructs architectures auto-regressively, which receives feedback from a GNN discriminator. Reinforcement learning (PPO) is used for training, to solve non-differentiability. GA-NAS’s effectiveness is demonstrated on several architecture search benchmarks for CIFAR-10 and 100, and is shown to improve EfficientNet for ImageNet.

Disclosure: I’m only vaguely familiar with the neural architecture search literature.

Pros:
1.	GA-NAS appears to consistently find high ranking architectures compared to the baselines, often at the cost of fewer queries.  
2.	A fairly extensive set of experiments are included in the Experiments section and Appendix.
3.	GA-NAS is able to incorporate constraints into search. The authors demonstrate that GA-NAS is able to find a variation that slightly outperforms EfficientNet-B0.

Cons:
1.	As the name implies, the goal of NAS is search. GANs have proven excellent over the years at interpolation, but not extrapolation, which is what search/exploration requires. I’m concerned that a GAN-based approach is therefore limited in its search ability. How would GA-NAS compare with a random search policy constrained to be close to known “good” architectures?
2.	The GA-NAS generator and discriminator are initialized with an initial set of good architectures X_0. In the experiments, X_0 takes on the value of 50 and 100. The assumption that such a large number of “good” architectures are available ahead of time seems rather strong.
3.	The authors state that previous work has determined that other architecture search methods are not any better than random search. The gains in accuracies of the architectures produced by GA-NAS over the baseline seem moderate at best.

Questions:
1.	Where in the f-GAN paper is it stated that the JS divergence is more robust than the assymetric KL? Can you demonstrate this claimed advantage in GA-NAS by comparing objectives? 
2.	Is X_0 counted in the number of queries reported? How were these initial architectures chosen? What rank are they?

Miscellaneous:
1.	This paper doesn’t adhere to the ICLR citation guidelines; citations in the text should include the first author’s last name and year. Additionally, the entire author list should appear in the references, not just “[First author] et al.” Please correct this during the rebuttal phase.
2.	Table 1: Typically, values in a table are bolded to represent the best values. BANANAS has the same accuracy value as GA-NAS.
3.	Why is the variance of GA-NAS’s accuracy so small in Table 1? 
4.	Table 2 caption could be more descriptive. Besides the columns, how is this table differ from Table 1?
5.	There’s an extra space between “ImageNet” and footnote 2.

Rating:
While the results seem good, I’m not entirely convinced that a generative adversarial approach can effectively explore a neural architecture space, as the discriminator will inherently disincentivize deviating from previously seen architectures. I’m also concerned about the validity of the assumption that an initial set of known “good” architectures would be available to a search algorithm; the authors should clarify how these were selected. I lean toward rejection for now, but would be willing to raise my score if the authors could address my concerns.


========
Post-rebuttal
========

I thank the authors for answering my questions. While I am satisfied with the authors response to my concern about the "initial good architectures" assumption, I still remain unconvinced that adversarial learning can help search find new good architectures. I keep my score.

I also encourage the authors to carefully re-read the f-GAN paper, which explains exactly how any f-divergence (including the KL) can be implemented for adversarial learning. Switching to any f-divergence requires only a simple change to the loss function. It also appears that the authors significantly misunderstand VAEs. The difference between GANs and VAEs is not JS-divergence vs KL-divergence. Using a KL loss for adversarial learning does not require switching to a VAE. Given the central role they play in this paper's motivation, a better understanding of these subjects is important.
review_title: Not entirely convinced by the approach or premise

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors introduce a NAS technique with an adversarial component. The discriminator learns to tell the difference between a set of good networks and randomly generated ones. This is quite a nice idea.

A few comments. I don’t think Section 3.1 adds anything and would be better off in an appendix. The connection between Algorithm 1 and Algorithms 2 seems fairly tenuous to me (although I could be wrong).

The bibliography is unacceptable. All papers with more than two authors are written as “et al.”, and there are glaring inconsistencies. arXiv is mentioned in multiple different ways and different fonts are used for different entries.
On a formatting issue, the text in the paper doesn’t look right compared to other ICLR submissions (it’s too pale). It would be worth looking in to this. The paper is otherwise fairly well written. Referring to EfficientNet as “Google’s EfficientNet” is quite odd. ResNet is not written as “Microsoft’s ResNet”.  I would recommend crediting the authors and not the institution. Table 2 appears above Table 1.

The details regarding training and the generator architecture are relegated to the appendix. These are very elaborate, which makes it very difficult to tell what is exactly contributing to the algorithm working. Table 7 seems to indicate that the discriminator itself can be removed for quite a small change in mean accuracy (94.2ish to 94.1ish). Without the discriminator the algorithm appears to be REINFORCE but with a more complicated generator network. On a related note, the comparison to REINFORCE is missing in table 1 which makes me suspect that this algorithm is basically the same thing in outcome, if not in effect.

Table 9 in the appendix (minor note — it’s not really an appendix when it’s in a separate file!) seems to show that different means of varying the size of the pool of networks over training has very little effect. (Having 94.227 in bold above 94.22 doesn’t change the fact that we are talking about 0.007%!)

The evaluation in Table 1 is very odd, as the authors are reporting the best acc instead of mean+-std as is common practice. This is unreliable, as when deploying these algorithms in the wild we are far more interested in how they do in expectation (particularly if they fail completely some of the time).

 Although the idea of using an adversarial framework to tell apart architectures in a search space is nice, the implementation has many moving parts, and doesn’t appear noticeably different to the standard REINFORCE NAS approach (which just has a generator as an RNN).  The presence of a discriminator has a very minimal effect, which is a shame. Some evaluation choices are very questionable. I am inclined towards rejection.


review_title: Review

Meta-review: This paper proposes a method for neural architecture search (NAS) based on adversarial methods. It uses a discriminator trained to distinguish between random vs. good architectures, letting the discriminator's scores serve as a reward signal for an autoregressive generator. I agree with AR1: this is a nice and clever idea. Reviewers generally agreed that the method was interesting, e.g. it's quite flexible in that it's able to incorporate constraints, and that the evaluation is rather extensive and shows that the method performs well across the board. Many minor criticisms were raised and addressed well by the authors in their responses and manuscript updates.

The major criticism shared by most reviewers was the high methodological complexity of the proposed approach, and the proportionally small gains shown over much simpler baselines. This criticism remained despite the authors' responses. The method is indeed complex: the same method without any adversarial component already performs well, and many important details of the model are relegated to Appendix A.2. (I would recommend, for example, moving Fig. 2 to the main text if at all possible. Also, the Appendix can/should be included in the main PDF for ICLR, rather than in supplementary material, as AR1 mentions.) It was not clear to reviewers that the adversarial component of the approach has a significant benefit. The authors respond by pointing to Table 7 showing that the discriminator reduces the number of queries and points out that in reality these queries correspond to expensive evaluations. If this is a major selling point of the method (it sounds like it could be), it should be highlighted and analyzed far more -- at least moved to the main text rather than an Appendix -- ideally with a real-world evaluation showing a practical large improvement in overall wall-clock time, rather than a benchmark where these evaluations are free. Perhaps the exclusive reliance on these benchmarks, though undoubtedly useful for quick experimentation, in the end holds back the paper and prevents the method's benefits from becoming apparent to the readers.

As a minor point (also raised by AR1), the paper is formatted incorrectly for ICLR: the font color is off, and more importantly the PDF is unsearchable (text cannot be selected, ctrl-F does not work), which makes it very difficult to quickly reference and review. Please try not to stray from the conference-provided style file for future submissions.

I appreciate the cleverness of the method, the extent of the evaluation, and the thorough responses to the reviews. However, unfortunately with the current presentation, it is too difficult to discern the benefit of the proposed approach from the manuscript. The approach is nonetheless intuitively appealing and seems quite promising, and I hope the authors will take the reviewers' good feedback into account and resubmit the paper in the future.
Decision: Reject