Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes Parsing-Reading-Predict Networks (PRPN), a new model jointly learns syntax and lexicon. The main idea of this model is to add skip-connections to integrate syntax relationships into the context of predicting the next word (i.e. language modeling task).

To model this, the authors introduce hidden variable l_t, which break down to the decisions of a soft version of gate variable values in the previous possible positions. These variables are then parameterized using syntactic distance to ensure that the final structure inferred by the model has no overlapping ranges so that it will be a valid syntax tree.

I think the paper is in general clearly written. The model is interesting and the experiment section is quite solid. The model reaches state-of-the-art level performance in language modeling and the performance on unsupervised parsing task (which is a by-product of the model) is also quite promising.

My main question is that the motivation/intuition of introducing the syntactic distance variable. I understand that they basically make sure the tree is valid, but the paper did not explain too much about what's the intuition behind this or is there a good way to interpret this. What motivates these d variables?
review_title: solid experiments and interesting model

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: the paper proposes a novel method to leverage tree structures in an unsupervised learning manner. The key idea is to make use of “syntactic distance” to identify phrases, thus building up a tree for input sentence. The proposed model achieves SOTA on a char-level language modeling task and is demonstrated to yield reasonable tree structures.

Comment: I like the paper a lot. The idea is very creative and interesting. The paper is well written.

Besides the official comment that the authors already replied, I have some more:
- I was still wondering how to compute the left hand side of eq 3 by marginalizing over all possible unfinished structures so far. (Of course, what the authors do is showed to be a fast and good approximation.)
- Using CNN to compute d has a disadvantage that the range of look-back must be predefined. Looking at fig 3, in order to make sure that d6 is smaller than d2, the look-back should have a wide coverage so that the computation for d6 has some knowledge about d2 (in some cases the local information can help to avoid it, but not always). I therefore think that using an RNN is more suitable than using a CNN.
- Is it possible to extend this framework to dependency structure?
- It would be great if the authors show whether the model can leverage given tree structures (like SPINN) (for instance we can do a multitask learning where a task is parsing given a treebank to train)

review_title: review 

final_score: rating: null
review_content: The paper proposes a very cute idea. My questions/comment: 

1. can you compute p(x_t+1|x0...x_t) (eq 3) by marginalising over g?

2. is the idea of using "syntactic distance" related to any linguistic theory?  

3. I think eq 5 has a typo: is it g_i or g_t'? 

4. the last line on page 4: d_{K-1} (capital K). Shouldn't d index start from 1? (you say that there are K-1 variables)

5. Eq 11: I think d doesn't need to be in [0, 1]. Did you try other activation functions (e.g Tanh)?

6. The line right after Eq 12: shouldn't t+1 be superscript? (It's better to be coherent with the notations above)

7. In 6.3, did you try any word embedding? As suggested by [1], word embeddings can be very helpful. 


1. Le & Zuidema. Unsupervised Dependency Parsing: Let’s Use Supervised Parsers


review_title: questions / comment 

final_score: rating: 7: Good paper, accept
review_content: ** UPDATE ** upgraded my score to 7 based on the new version of the paper.

The main contribution of this paper is to introduce a new recurrent neural network for language modeling, which incorporates a tree structure More precisely, the model learns constituency trees (without any supervision), to capture syntactic information. This information is then used to define skip connections in the language model, to capture longer dependencies between words. The update of the hidden state does not depend only on the previous hidden state, but also on the hidden states corresponding to the following words: all the previous words belonging to the smallest subtree containing the current word, such that the current word is not the left-most one. The authors propose to parametrize trees using "syntactic distances" between adjacent words (a scalar value for each pair of adjacent words w_t, w_{t+1}). Given these distances, it is possible to obtain the constituents and the corresponding gating activations for the skip connections. These different operations can be relaxed to differentiable operations, so that stochastic gradient descent can be used to learn the parameters. The model is evaluated on three language modeling benchmarks: character level PTB, word level PTB and word level text8. The induced constituency trees are also evaluated, for sentence of length 10 or less (which is the standard setting for unsupervised parsing).

Overall, I really like the main idea of the paper. The use of "syntactic distances" to parametrize the trees is clever, as they can easily be computed using only partial information up to time t. From these distances, it is also relatively straightforward to obtain which constituents (or subtrees) a word belongs to (and thus, the corresponding gating activations). Moreover, the operations can easily be relaxed to obtain a differentiable model, which can easily be trained using stochastic gradient descent.

The results reported on the language modeling experiments are strong. One minor comment here is that it would be nice to have an ablation analysis, as it is possible to obtain similarly strong results with simpler models (such as plain LSTM).

My main concern regarding the paper is that it is a bit hard to understand. In particular in section 4, the authors alternates between discrete and relaxed values: end of section 4.1, it is implied that alpha are in [0, 1], but in equation 6, alpha are in {0, 1}, then relaxed in equation 9 to [0, 1] again. I am also wondering whether it would make more sense to start by introducing the syntactic distances, then the alphas and finally the gates? I also found the section 5 to be quite confusing. While I get the	general idea, I am not sure what is the relation between hidden states h and m (section 5.1). Is there a mixup between h defined in equation 10 and h from section 5.1? I am aware that it is not straightforward to describe the proposed method, but believe it would be a much stronger paper if written more clearly.

To conclude, I really like the method proposed in this paper, and believe that the experimental results are quite strong.
My main concern	regarding the paper is its clarity: I will gladly increase my score if the authors can improve the writing.
review_title: REVIEW

Meta-review: Nice language modeling paper with consistently high scores. The model structure is neat and the results are solid. Good ICLR-type paper with contributions mostly on the ML side and experiments on a (simple) NLP task.
Decision: Accept (Poster)