Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors present a novel evolution scheme applied to neural network architecture search. It relies on defining an expressive search space for conducting optimization, with a constrained search space that leads to a lighter and more efficient algorithm. To balance these constraints, they grow sub-modules in a hierarchical way to form more and more complex cells. Hence, each level is limited to a small search space while the system as a whole converges toward a complex structure. To speed up the search, they focus on finding cells instead of an entire network. In evaluation time, they insert these cells between layers of a network comparable in size to known networks. They find complex cells that lead to state-of-the-art performance on benchmark dataset CIFAR-10 and ImageNet. They also claim that their method is reaching a new milestone in evolutionary search strategies performance.

The method proposed for an hierarchical representation for optimizing over neural network designs is well thought and sound. It could lead to new insight on automating design of neural networks for given problems. In addition, the authors present results that appear to be on par with the state-of-the-art with architecture search on CIFAR-10 and ImageNet benchmark datasets. The paper presents a good work and is well articulated. However, it could benefit from additional details and a deeper analysis of the results.

The key idea is a smart evolution scheme. It circumvents the traditional tradeoff between search space size and complexity of the found models. The method is also appealing for its use of some kind of emergence between two levels of hierarchy. In fact, it could be argued that nature tends to exploit the same phenomenon when building more and more complex molecules. Thought, the paper could benefit from a more detailed analysis of the architectures found by the algorithm. Do the modules always become more complex as they jump from a level to another or there is some kind of inter-level redundancy? Are the cells found interpretable? The authors should try to give their opinion about the design obtained.

The implementation seems technically sound. The experiments and results section shows that the authors are confident and the evaluation seems correct. However, paragraphs on the architectures could be a bit clearer for the reader. The diagram could be more complete and reflect better the description. During evaluation, what is a step? A batch or an epoch or other?

The method seems relatively efficient as it took 36 hours to converge in a field traditionally considered as heavy in terms of computation, but at the requirement of using 200 GPU. It raises questions on the usability of the method for small labs. At some point, we will have to use insights from this search to stop early, when no improvement is expected. Also, authors claim that their method consume less computation time than reinforcement learning. This should be supported by some quantitative results.

The paper would greatly benefit from a deeper comparison over other techniques. For instance, it could describe more the advantages over reinforcement learning. An important contribution is to show that a well-defined architecture representation could lead to efficient cells with a simple randomized search. It could have taken more spaces in the paper.

I am also concerned the computational efficiency of the results obtained with this method on current processors. Indeed, the randomness of the found cells could be less efficient in terms of computation that what we can get from a well-structured network designed by hand. Exploiting the structure of the GPUs (cache size, sequential accesses, etc.) allows to get best possible performance from the hardware at hand. Does the solution obtained with the optimization can be run as efficiently? A short analysis forward pass time of optimized cells vs. popular models could be an interesting addition to the paper. This is a general comment over this kind of approach, but I think it should be addressed. 

review_title: Good paper on searching space of network design

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work fits well into a growing body of research concerning the encoding of network topologies and training of topology via evolution or RL. The experimentation and basic results are probably sufficient for acceptance, but to this reviewer, the paper spins the actual experiments and results a too strongly.

The biggest two nitpicks:

> In our work we pursue an alternative approach: instead of restricting the search space directly, we allow the architectures to have flexible network topologies (arbitrary directed acyclic graphs)

This is a gross overstatement. The architectures considered in this paper are heavily restricted to be a stack of cells of uniform content interspersed with specifically and manually designed convolution, separable convolution, and pooling layers. Only the topology of the cells themselves are designed. The work is still great, but this misleading statement in the beginning of the paper left the rest of the paper with a dishonest aftertaste. As an exercise to the authors, count the hyperparameters used just to set up the learning problem in this paper and compare them to those used in describing the entire VGG-16 network. It seems fewer hyperparameters are needed to describe VGG-16, making this paper hardly an alternative to the "[common solution] to restrict the search space to reduce complexity and increase efficiency of architecture search."

> Table 1

Why is the second best method on CIFAR (“Hier. repr-n, random search (7000 samples)”) never tested on ImageNet? The omission is conspicuous. Just test it and report.

Smaller nitpicks:

> “New state of the art for evolutionary strategies on this task”

“Evolutionary Strategies”, at least as used in Salimans 2017, has a specific connotation of estimating and then following a gradient using random perturbations which this paper does not do. It may be more clear to change this phrase to “evolutionary methods” or similar.

> Our evolution algorithm is similar but more generic than the binary tournament selection (K = 2) used in a recent large-scale evolutionary method (Real et al., 2017).

A K=5% tournament does not seem more generic than a binary K=2 tournament. They’re just different.
review_title: Great results, needlessly overstated.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The fundamental contribution of the article is the explicit use of compositionality in the definition of the search space. Instead of merely defining an architecture as a Directed Acyclic Graph (DAG), with nodes corresponding to feature maps and edges to primitive operations, the approach in this paper introduces a hierarchy of architectures of this form. Each level of the hierarchy utilises the existing architectures in the preceding level as candidate operations to be applied in the edges of the DAG. As a result, this would allow the evolutionary search algorithm to design modules which might be then reused in different edges of the DAG corresponding to the final architecture, which is located at the top level in the hierarchy.

Manually designing novel neural architectures is a laborious, time-consuming process. Therefore, exploring new approaches to automatise this task is a problem of great relevance for the field. 

Overall, the paper is well-written, clear in its exposition and technically sound. While some hyperparameter and design choices could perhaps have been justified in greater detail, the paper is mostly self-contained and provides enough information to be reproducible. 

The fundamental contribution of this article, when put into the context of the many recent publications on the topic of automatic neural architecture search, is the introduction of a hierarchy of architectures as a way to build the search space. Compared to existing work, this approach should emphasise modularity, making it easier for the evolutionary search algorithm to discover architectures that extensively reuse simpler blocks as part of the model. Exploiting compositionality in model design is not novel per se (e.g. [1,2]), but it is to the best of my knowledge the first explicit application of this idea in neural architecture search. 

Nevertheless, while the idea behind the proposed approach is definitely interesting, I believe that the experimental results do not provide sufficiently compelling evidence that the resulting method substantially outperforms the non-hierarchical, flat representation of architectures used in other publications. In particular, the results highlighted in Figure 3 and Table 1 seem to indicate that the difference in performance between both paradigms is rather small. Moreover, the performance gap between the flat and hierarchical representations of the search space, as reported in Table 1, remains smaller than the performance gap between the best performing of the approaches proposed in this article and NASNet-A (Zoph et al., 2017), as reported in Tables 2 and 3.

Another concern I have is regarding the definition of the mutation operators in Section 3.1. While not explicitly stated, I assume that all sampling steps are performed uniformly at random (otherwise please clarify it). If that was indeed the case, there is a systematic asymmetry between the probability to add and remove an edge, making the former considerably more likely. This could bias the architectures towards fully-connected DAGs, as indeed seems to occur based on the motifs reported in Appendix A.

Finally, while the main motivation behind neural architecture search is to automatise the design of new models, the approach here presented introduces a non-negligible number of hyperparameters that could potentially have a considerable impact and need to be selected somehow. This includes, for instance, the number of levels in the hierarchy (L), the number of motifs at each level in the hierarchy (M_l), the number of nodes in each graph at each level in the hierarchy (| G^{(l)} |), as well as the set of primitive operations. I believe the paper would be substantially strengthened if the authors explored how robust the resulting approach is with respect to perturbations of these hyperparameters, and/or provided users with a principled approach to select reasonable values.

References:

[1] Grosse, Roger, et al. "Exploiting compositionality to explore a large space of model structures." UAI (2012).
[2] Duvenaud, David, et al. "Structure discovery in nonparametric regression through compositional kernel search." ICML (2013).

review_title: In this paper, the authors propose a novel evolutionary algorithm for neural architecture search. 

Meta-review: PROS:
1. Overall, the paper is well-written, clear in its exposition and technically sound.
2. With some caveats, an independent team concluded that the results were "largely reproducible"
3. The key idea is a smart evolution scheme. It circumvents the traditional tradeoff between search space size and complexity of the found models.
4. The implementation seems technically sound.

CONS:
1. The results were a bit over-stated (the authors promise to correct)
2. Could benefit from more comparison with other approaches (e.g. RL)
Decision: Accept (Poster)