Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors proposed to combine both NN-based NAS and Aging EVO to get the benefit of both world: good global and local sample efficiency. The main idea is to use Aging EVO algorithm to guide the overall search process, and a NN predicting the final performance is used to guide the mutation process. The combined EVO-NAS has showed consistent good performance over a range of tasks.

Overall, while the novelty of the paper is not exceptional, since it is a rather straightforward combination of two existing approaches, the end-results is promising. I would like to see more in-depth analysis on the combined algorithm to validate authors' hypothesis on why EVO-NAS works better. More detailed comments can be found below.

1. The experiment on the synthetic task is not very helpful, since the domain can be far apart from the real NAS applications. One evidence is that the NN significantly outperform EVO in this task but not in the other tasks.

2. In difficult tasks, the proposed EVO-NAS and the original Aging EVO are very close in the first few hundreds of trials, however, later the gap remains the same. It would be interesting to see if we can eliminate the gap by adding the NN component in the middle of the Aging EVO experiment (e.g., at the point of 2000 trials).

3. I am also curious about the difference between the NN learned from Neural agent vs. those learned from EVO-NAS agent. Moreover, do we need a NN for the EVO-NAS? Or something simpler would be sufficient to guide the search.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper is well organized. The applied methods are introduced in detail. But it lacks some more detailed analysis.

My concerns are as follows.
1. The performance differences between Evolutionary agent and EVO-NAS agent seem not significant. Please conduct additional statistical tests such as the Wilcoxon signed-rank test to verify the performance improvements are significant.
2. Many studies have been conducted to automatically adjust control parameters such as crossover and mutation probabilities in evolutionary algorithm literature. It would be better to compare one of these approaches in the experiments.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: It is a nice paper that combines the deep reinforcement learning and evolutionary learning techniques to neural architecture search problem. Experimental results are promising. However, I still have some concerns on the current submission.
1.In Fig 1,2 &3, it seems that the performances of Neural (PQT) keeps increasing. For better compassion, we recommend the authors reports the performances of compared algorithms until they are convergent.
2.The different training algorithms (Reinforce and PQT) have difference performances whether because different training algorithms converge to difference local minima or stationary points.

review_title: Official Blind Review #1

Meta-review: Thanks to the authors for the revision and discussion. This paper provides a neural architecture search (NAS) method, called Evolutionary-Neural hybrid agents (Evo-NAS), which combines NN-based NAS and Aging EVO. While the authors' response addressed some of the reviewers' comments, during discussion period there is a new concern that the idea proposed here highly overlaps with the method of RENAS, which stands for Reinforced Evolutionary Neural Architecture Search. Reviewers acknowledge that this might discount the novelty of the paper. Overall, there is not sufficient support for acceptance.
Decision: Reject