Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work tries to cast the search of good RNN Cell architectures as a black-box optimization problem where examples are represented as an operator tree and are either 1. Sampled randomly and scored based on a learnt function OR 2. Generated by a RL agent.
While the overall approach appears to generalize previous work, I see a few serious flaws in this work:

Limited scope
As far as I can tell this work only tries to come up with a design for a single RNN cell, and then claims that the optimality of the design will carry over to stacking of such modules, not to mention more complicated network designs. 
Even the design of a single cell is heavily biased by human intuition (section 4.1) It would have been more convincing to see that the system learn heuristics such as “don’t stack two matrix mults” rather than have these hard coded.

No generalization guarantees:
No attempt is made to optimize hyperparameters of the candidate architectures. This leads one to wonder if the winning architecture has won only because of the specific parameters that were used in the evaluation.
Indeed, the experiments in the paper show that a cell which was successful on one task isn’t necessary successful on a different one, which questions the competence of the scoring function / RL agent.

On the experimental side:
No comparison is made between the two optimization strategies, which leaves the reader wondering which one is better.
Control for number of network variables is missing when comparing candidate architectures. 

review_title: Nice work, but limited in scope and does not appear to generalize well

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors introduce a new method to generate RNNs architectures. The authors propose a domain-specific language two types of generators (random and RL-based) together with a ranking function and evaluator. The results are promising, and this research introduces a framework that might enable the community to find new interesting models. However, it's not clear how these framework compare with previous ones (see below). Also, the clarity of the text could be improved.

Pros:
1. An interesting automatic method for generating RNNs is introduced by the authors (but is not entirely clear how does ir compare with previous different approaches)
2. The approach is tested in a number of tasks: Language modelling (PTB and wikipedia-2) and machine translation (
3. In these work the authors tested a wide range of different RNNs
3. It is interesting that some of the best performing architectures (e.g. LSTM, residual nets) are found during the automatic search


Cons:
1. It would be nice if the method didn’t rely on defining a specific set of functions and operators upon which the proposed method works.
2. The text has some typos: for example: “that were used to optimize the generator each batch”
3. In section 5, the authors briefly discuss other techniques using RL and neuroevolution, but they never contrast these approaches with theirs. Overall, it would be nice if the authors had made a more direct comparison with other methods for generating RNNs.
4. The description of the ranking function is not clear. What kind of networks were used? This appears to introduce a ranking-network-specific bias in the search process.

Minor comments:
1. The authors study the use of subtractive operators. Recently a new model has considered the use of subtractive gates in LSTMs as a more biologically plausible implementation (Cortical microcircuits as gated-recurrent neural networks, NIPS 2017).
2. Figure 4 missing label on x-axis
3. End of Related Work period is missing.
3. The authors state that some of the networks generated do not follow human intuition, but this doesn’t appear to discussed. What exactly do the authors mean?
4. Not clear what happens in Figure 4 in epoch 19k or so, why such an abrupt change?
5. Initial conditions are key for such systems, could the init itself be included in this framework?
review_title: The authors present an interesting framework to search for new RNN models, with some promising results.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper investigates meta-learning strategy for automated architecture search in the context of RNN. To constraint the architecture search space, authors propose a DSL that specifies the RNN recurrent operations. This DSL allows to explore RNN architectures using either random search or a reinforcement-learning strategy. Candidate architectures are ranked using a TreeLSTM that tries to predict the architecture performances. The top-k architectures are then evaluated by fully training them on a given task.

Authors evaluate their approach on  PTB/Wikitext 2 language modeling and Multi30k/IWSLT'16  machine translation. In both experiments, authors show that their approach obtains competitive results and can sometime outperforms RNN cells such as GRU/LSTM. In the PTB experiment, their architecture however underperforms other LSTM variant in the literatures.


- Quality/Clarity
The paper is overall well written and pleasant to read.

Few details can be clarified. In particular how did you initialize the weight and bias for both the LSTM/GRU baselines and the found architectures? Is there other works leveraging RNN that report results on the Multi30k/IWSLT datasets?

You state in paragraph 3.2 that human experts can inject the previous best known architecture when training the ranking networks. Did you use this in the experiments? If yes, what was the impact of this online learning strategy on the final results? 


- Originality
The idea of using DSL + ranking for architecture search seems novel.


- Significance
Automated architecture search is a promising way to design new networks. However, it is not clear why the proposed approach is not able to outperforms other LSTM-based architectures on the PTB task. Could the problem arise from the DSL that constraint too much the search space ? It would be nice to have other tasks that are commonly used as benchmark for RNN to see where this approach stand.

In addition, authors propose both a DSL, a random and RL generator and a ranking function. It would be nice to disentangle the contributions of the different components. In particular, did the authors compare the random search vs the RL based generator or the performances of the RL-based generator when the ranking network is not used?

Although authors do show that they outperform NAScell in one setting, it would be nice to have an extended evaluation (using character level PTB for instance).
review_title: Review

Meta-review: The paper presents a domain-specific language for RNN architecture search, which can be used in combination with learned ranking function or RL-based search. While the approach is interesting and novel, the paper would benefit from an improved evaluation, as pointed out by reviewers. For example, the paper currently evaluated coreDSL+ranking for language modelling and extendedDSL+RL for machine translation. The authors should use the same evaluation protocol on all tasks, and also compare with the state-of-the-art MT approaches.
Decision: Invite to Workshop Track