Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes to use the variational auto-encoder (VAE) to sample the network architectures. The VAE is applied to both one-short and gradient descent scenarios, and shows consistent improvement on different NAS tasks. The proposed method is reasonable, but I have two major concerns:

- I wonder whether the VAE based approach will consistently converge to a good local minimum. It will be very helpful if the authors could provide robust analysis or at least the variations of testing errors.

- I understand the motivation of VAE + one shot, but I am not very convinced by VAE + gradient-based. In the last paragraph in Section 4, the paper claims (1) VAENAS-G can increase the diversity of architectures, which can be also achieved by sampling the data set. Also, it claims (2) VAENAS-G helps to search for large models, which  I do not see experimental supports. 

Detailed comments:
- Algorithm 1: it is confusing to use S_K and S_k without explanation
- Table 1: most of the other methods provide test error with standard variations. To be fair, I'd see VAENAS's test error variation. 
- Tables 2 and 3: Why is VAENASNet (table 3) different from VAENAS-G and VAENAS-OS?

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposed to use VAE to learn a sampling strategy in neural architecture search. The main idea is to use the currently high-performing networks to train a VAE from which the sampled architectures for the next iteration will likely supply both high-performing networks and better diversity coverage. The experiments are extensive, including results under various settings.

The idea is straightforward and reasonable. I do not work on neural architecture search myself, so I'm not sure how significant the experimental results are. Would the 0.1% (1.1%) absolute improvement over the second best in Table 2 (Table 3) be considered significant enough to justify the effectiveness of the proposed approach? 

I'm a little concerned about the fairness of the comparison experiments. A fairly heavy computation overhead is required to train the VAE models in the proposed method. Instead of taking this overhead, wouldn't it be easier to randomly sample more architectures? Intuitively, if we spend the cost of training a VAE model instead on sampling more architectures, the end  effects could be the same. 

Are the numbers in Table 2 and Table 3 swapped? 


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: ============ comments after rebuttal
I would like to thank the authors for addressing some of my concerns. I believe the new results under Q2 and Q3 are useful additions to strengthen the paper.

As for the authors' comments for Q1, I'd like to point out that a "larger" search space is not necessarily more difficult (a more meaningful metric would be the average accuracy of random architectures). It is still possible that the current search space is putting the proposed method at advantage, especially if having dense connections is a useful prior.

Given the above, I would like to increase my score from 1 to 3 (weak reject).

============ previous comments
Neural architecture search can be formulated as learning a distribution of promising architectures (the sampling policy). Such a distribution is usually represented in a fully factorized fashion (e.g., as a set of multinational distributions as in DARTS). This paper proposes to model the architecture distribution using a VAE instead, where the encoder and decoder are implemented using LSTMs. The authors argue that the increased flexibility of the sampling policy leads to improved performance on CIFAR-10, NASBench and ImageNet.

The idea of representing the architecture distribution using VAEs is very natural, which in principle could offer better coverage over interesting regions in the search space as compared to traditional factorized distribution representation (which has a single mode only).

While the method itself is interesting, I do not think it has been properly backed up by controlled experiments. This is largely due to the fact that the authors are comparing their method against baselines in fundamentally different search spaces. For instance:
* For CIFAR-10 experiments, the authors mentioned in the appendix: "Different from DARTS, in our search space, one node could have more than two predecessors in one cell". This makes the search space very different from the existing ones as used by NASNet/AmoebaNet/DARTS/SNAS, and it hence remains unclear to what degree the resulting architecture has benefited from the increased in-degrees per node. Note the searched densely connected cells in Figure 4 & 5 in Appendix A.4 are clearly not part of the search space for many of the baselines.
* For ImageNet experiments, the authors are using a ShuffleNet-like search space which has fundamentally different building blocks than other architecture search baselines (commonly built on top of inverted bottleneck layers). It is unclear to what degree the 77.4 top-1 accuracy @ 365 MFlops results have benefited from this different search space.

Without fair comparisons in a controlled setup, it is impossible for readers to draw any solid conclusion about the true empirical advantages of the method. I'm therefore unable to recommend acceptance for the paper at the moment, but am willing to raise my score if the authors can properly address those issues in the rebuttal.

Additional question: How can we isolate it to tell whether the gains come from the LSTMs or the VAEs? Is there any intuition why incorporating a generative sampler based on VAEs is potentially superior to method like ENAS (which involves LSTMs decoders only)?
review_title: Official Blind Review #1

Meta-review: This paper proposes to represent the distribution w.r.t. which neural architecture search (NAS) samples architectures through a variational autoencoder, rather than through a fully factorized distribution (as previous work did). 

In the discussion, a few things improved (causing one reviewer to increase his/her score from 1 to 3), but it became clear that the empirical evaluation has issues, with a different search space being used for the method than for the baselines. There was unanimous agreement for rejection. I agree with this judgement and thus recommend rejection.
Decision: Reject