Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes an interesting idea to perform Neural Architecture Search: first, an auto-encoder is pre-trained to encode/decode an neural architecture to/from a continuous low-dimensional embedding space; then the decoder is fixed but the encoder is copied as an agent controller for reinforcement learning. The controller is optimized by taking actions in the embedding space. The reward is also different from previous works which usually only considered validation accuracy but this work also considers the generalization gap.

The idea is interesting, but there are some problems on both the method's and the experimental sides:
1. NAO [1] also embeds neural architectures to a continuous space. Different from NAO which applies gradient descent in the embedded space, this paper uses RL. I double that RL can work better than gradient descent in a continuous space. The paper should compare with NAO. Ideally, this paper might work better than NAO if the accuracy predictor in the NAO is not accurate, while this paper uses real accuracy as a reward for search. However, this is not soundly compared.
2. It is unreasonable to discretize continuous actions to a Bernoulli distribution. Many RL methods, such as DDPG, can handle continuous actions;
3. The paper uses Eq. 1 as a reward. It's interesting, but it's unclear why the generalization error is needed. Ablation study is required.
4. As the community makes more progresses in AutoML, a better and better (smaller and smaller) search space is used. It doesn't make much sense to compare the search time under different search spaces. Comparison under the same setting (e.g. NASBench-101) is required. 


Minors:
1. missing a number in "and T0 = epochs"
2. missing "x" in "32 32 RGB in 100 classes", and "100" should be "10"

[1] Luo, Renqian, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. "Neural architecture optimization." In Advances in neural information processing systems, pp. 7816-7827. 2018.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper borrows the idea of word-to-vector from NLP and applies it in reinforcement learning based Neural Architecture Search (NAS). It suggests a pretrained encoder to transform the search space to a dense and continuous architecture-embedding space. First it trains the architecture-embedding encoder and decoder with self-supervision learning like Auto-Encoder.  Then it performs reinforcement learning based Neural Architecture Search(NAS) in the architecture-embedding space.

Strength:
There is no architecture prior, such as cell, in the searching process. Thus it's more general and can explore more architectures possibilities.
Because it performs architecture search in a continuous space, a CNN based controller is used instead of a RNN controller. 
The result of the proposed method on CIFAR-10 is comparable with other popular NAS approaches.
It reduces the number of searching architectures to <100 in <12 GPU hours without using tricks such as cell or parameter sharing.

Weakness:
The evaluations are highly insufficient. It only performs experiment on CIFAR-10, and the generalization ability on other datasets is unclear. In many NAS works. CIFAR-100 and ImageNet are commonly used to evaluate the performance. 
Besides, there is no comparison with more recent and related important methods such as DARTS and the method proposed by Luo et al. (2018). Actually its performance is not as good as Darts or the best performance reported in ENAS.

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This work searches neural architectures in an embedding space, which is continuous. Overall, it lacks of innovation and experimental results are not strong enough. 

	1. The innovation and technical contribution are weak. The key idea of searching in embedding space has already been proposed and applied in  Luo et al. 2018. The authors do not differentiate this work from Luo et al. 2018, and I didn't find any essential differences between them.

	2. The proposed method does work well. It is not compared with latest algorithms including NAO and DARTS. The reported numbers in Table 2 are poor, far from SOTA numbers.

Besides, there are many writing issues and typos.
review_title: Official Blind Review #3

Meta-review: This paper proposes a method for neural architecture search in embedding space. This is an interesting idea, but its novelty is limited due to its similarity to the NAO approach. Also, the empirical evaluation is too limited; comparisons should have been performed to NAO and other contemporary NAS methods, such as DARTS. 

Due the factors above, all reviewers gave rejecting scores (3,3,1). The rebuttal did not remove the main issues, resulting in the reviewers sticking to their scores. I therefore recommend rejection.
Decision: Reject