Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper proposes to use reinforcement learning to learn how to compose the words in a sentence, i.e. parse tree, that can be helpful for the downstream tasks. To do that, the shift-reduce framework is employed and RL is used to learn the policy of the two actions SHIFT and REDUCE. The experiments on four datasets (SST, SICK, IMDB, and SNLI) show that the proposed approach outperformed the approach using predefined tree structures (e.g. left-to-right, right-to-left). 

The paper is well written and has two good points. Firstly, the idea of using RL to learn parse trees using downstream tasks is very interesting and novel. And employing the shift-reduce framework is a very smart choice because the set of actions is minimal (shift and reduce). Secondly, what shown in the paper somewhat confirms the need of parse trees. This is indeed interesting because of the current debate on whether syntax is helpful.

I have the following comments:
- it seems that the authors weren't aware of some recent work using RL to learn structures for composition, e.g. Andreas et al (2016).
- because different composition functions (e.g. LSTM, GRU, or classical recursive neural net) have different inductive biases, I was wondering if the tree structures found by the model would be independent from the composition function choice.
- because RNNs in theory are equivalent to Turing machines, I was wondering if restricting the expressiveness of the model (e.g. reducing the dimension) can help the model focus on discovering more helpful tree structures.

Ref:
Andreas et al. Learning to Compose Neural Networks for Question Answering. NAACL 2016
review_title: official review

final_score: rating: 7: Good paper, accept
review_content: I have not much to add to my pre-review comments.
It's a very well written paper with an interesting idea.
Lots of people currently want to combine RL with NLP. It is very en vogue.
Nobody has gotten that to work yet in any really groundbreaking or influential way that results in actually superior performance on any highly relevant or competitive NLP task.
Most people struggle with the fact that NLP requires very efficient methods on very large datasets and RL is super slow.
Hence, I believe this direction hasn't shown much promise yet and it's not yet clear it ever will due to the slowness of RL.
But many directions need to be explored and maybe eventually they will reach a point where they become relevant.

It is interesting to learn the obviously inherent grammatical structure in language though sadly again, the trees here do not yet capture much of what our intuitions are.

Regardless, it's an interesting exploration, worthy of being discussed at the conference.

review_title: Accept

final_score: rating: null
review_content: I like the idea of learning structure or representation of text which is useful for a task of interest. I have a few questions regarding the experiments.

Would it make sense to compare the proposed approach to methods that learn use differentiable data structure, such as the stack RNNs introduced by Grefenstette et al. (2015) and Joulin & Mikolov (2015)?

The authors state that the best model (out of 3 random initialization) is kept. What is the importance of the initialization? In particular, what is the variance between the different initializations?

Finally, if I understand the model correctly, the shift operation correspond to a regular LSTM update (with pushing the resulting hidden/memory representation on the stack). Is that correct?

Grefenstette et al. (2015). Learning to Transduce with Unbounded Memory. In NIPS.
Joulin & Mikolov (2015). Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets. In NIPS.

review_title: Experiments

final_score: rating: null
review_content: How did you train your model on SST? Because SST was annotated at phrase level (every internal node in a parse tree was given a sentiment label), I think you can train your model such that a reward is given after every (or few) action(s)
review_title: Experiments on SST

final_score: rating: null
review_content: I like this paper a lot. Great and clear model description.
I find the excuse for the not so stellar performance not convincing though.
I'm sure DeepMind has the resources to train a slightly larger model with more parameters, even if it takes a few days.
The paper would really be stronger if it didn't include the excuse that your model would just be better if it was bigger. Show at least a plot with increasing performance based on increasing # of parameters. Or ideally just run with larger models to be really convincing.
Maybe the model will start to overfit if it's bigger and not get better?

review_title: Feedback

final_score: rating: null
review_content: It was slow to train the model because you had to build a different new computational graph for each sentence. I was wondering if the slow speed is due to the way you implemented (e.g. tensorflow, python). Dynet is designed especially for this kind of problem, did you try it? 
review_title: implementation 

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors propose a new method to learn hierarchical representations of sentences, based on reinforcement learning. They propose to learn a neural shift-reduce parser, such that the induced tree structures lead to good performance on a downstream task. They use reinforcement learning (more specifically, the policy gradient method REINFORCE) to learn their model. The reward of the algorithm is the evaluation metric of the downstream task. The authors compare two settings, (1) no structure information is given (hence, the only supervision comes from the downstream task) and (2) actions from an external parser is used as supervision to train the policy network, in addition to the supervision from the downstream task. The proposed approach is evaluated on four tasks: sentiment analysis, semantic relatedness, textual entailment and sentence generation.

I like the idea of learning tree representations of text which are useful for a downstream task. The paper is clear and well written. However, I am not convinced by the experimental results presented in the paper. Indeed, on most tasks, the proposed model is far from state-of-the-art models:
 - sentiment analysis, 86.5 v.s. 89.7 (accuracy);
 - semantic relatedness, 0.32 v.s. 0.25 (MSE);
 - textual entailment, 80.5 v.s. 84.6 (accuracy).
From the results presented in the paper, it is hard to know if these results are due to the model, or because of the reinforcement learning algorithm.

PROS:
 - interesting idea: learning structures of sentences adapted for a downstream task.
 - well written paper.
CONS:
 - weak experimental results (do not really support the claim of the authors).

Minor comments:
In the second paragraph of the introduction, one might argue that bag-of-words is also a predominant approach to represent sentences.
Paragraph titles (e.g. in section 3.2) should have a period at the end.

----------------------------------------------------------------------------------------------------------------------
UPDATE

I am still not convinced by the results presented in the paper, and in particular by the fact that one must combine the words in a different way than left-to-right to obtain state of the art results.
However, I do agree that this is an interesting research direction, and that the results presented in the paper are promising. I am thus updating my score from 5 to 6.
review_title: Weak experimental results

Meta-review: All the reviewers agreed that the research direction is very interesting, and generally find the results promising. We could quibble a bit about the results not being really state-of-the-art and the choice of baselines, but I think the main claims are well supported by the experiments (i.e. the induce grammar appears to be useful for the problem in question, within the specific class of models). There are still clearly many issues unresolved, and we are yet to see if this class of methods (RL / implicit structure-based) can lead to state-of-the-art results on any important NLP problem. But it is too much to ask from a submission. I see the paper as a strong contribution to the conference.
Decision: Accept (Poster)