Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposed to analyze several recently developed machine readers and found that some machine readers could potentially take advantages of the entity marker (given that the same marker points out to the same entity). I usually like analysis papers, but I found the argument proposed in this paper not very clear.

I like the experiments on the Stanford reader, which shows that the entity marker in fact helps the Stanford reader on WDW. I found that results rather interesting.

However, I found the organization and the overall message of this paper quite confusing. First of all, it feels that the authors want to explain the above behavior with some definition of the “structures”. However, I am not sure that how successful the attempt is. For me, it is still not clear what the structures are. This makes reading section 4 a bit frustrating. 

I am also not sure what is the take home message of this paper. Does it mean that the entity marking should be used in the MR models? Should we design models that can also model the entity reference at the same time? What are the roles of the linguistic features here? Should we use linguistic structure to overcome the reference issue?

Overall, I feel that the analysis is interesting, but I feel that the paper can benefit from having a more focused argument.

review_title: Review

final_score: rating: null
review_content: In what sense is this a logical structure? A main finding of the paper seems to be the section after equation 11 where it is shown that the aggregation readers effectively learn to mimic the explicit reference readers. This is an emergent structure, but what sort of logic does it encode? Is the term "logic" in reference to the symbolic nature of the learned anonymized entity embeddings? 

(11) and (12) imply a H = S (+) E direct sum structure for the representations? This is because the entity identifiers, being semantics-free, must be permutable (approximately) without changing the model scores, correct? Do the authors have a sense of how big the E subspace must be to handle a given number of entity vectors? Is it equal to the maximum number of entities in a document?

Regarding equation 11, it is unclear why the inner-product of h_{t} and the answer embedding would be a fixed positive constant 'c' - especially since h_{t} is the embedding obtained by running a bi-LSTM/GRU encoder on the paragraph. i.e. for all t in R(a,p), the h_t's will encode everything it has seen before. However it is intuitive that since all t's in R(a,p) refers to positions of answers in the document, the inner product would be high (but not necessarily fixed).

Also in equation 11, it is also very counter-intuitive that for all t not in R(a,p) the inner product of h_{t} and the answer embedding would be zero. For example, consider the embedding of h_{t+1}. Since this word is just after the answer word, will it have a very low inner product? It would be nice to add this number after row 2 in table 1.

It is hard to follow equation 12. A new variable h' has been introduced just before which has not been used in the equation which makes it even more confusing. I think the paper would benefit a lot if section 4 is re-visited and written more clearly because I think it is the most important section of the paper (especially explanation/intuition behind equation 12 and 13).

Equation 13 suggests the existence of a sum-structure between two orthogonal subspaces S (subspace of statement vectors) and E(subspace of entity pointers). Do the experiment in section 5 explicitly capture this? It is clear to me that the one-hot and general pointer encoding are ways to inject coreference information in the model but does it explain the additive nature of the embeddings (since the two vectors are concatenated).

review_title: Questions

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper aims to provide an insightful and analytic survey over the recent literature on reading comprehension with the distinct goal of investigating whether logical structure (or predication, as the authors rephrased in their response) arises in many of the recent models. I really like the spirit of the paper and appreciate the efforts to organize rather chaotic recent literature into two unified themes: "aggregation readers" and "explicit reference models”. Overall the quality of writing is great and section 3 was especially nice to read. I’m also happy with the proposed rewording from "logical structure" to “predication", and the clarification by the authors was detailed and helpful.

I think I still have slight mixed feelings about the contribution of the work. First, I wonder whether the choice of the dataset was ideal in the first place to accomplish the desired goal of the paper. There have been concerns about CNN/DailyMail dataset (Chen et al. ACL’16) and it is not clear to me whether the dataset supports investigation on logical structure of interesting kinds. Maybe it is bound to be rather about lack of logical structure.

Second, I wish the discussion on predication sheds more practical insights into dataset design or model design to better tackle reading comprehension challenges. In that sense, it may have been more helpful if the authors could make more precise analysis on different types of reading comprehension challenges, what types of logical structure are lacking in various existing models and datasets, and point to specific directions where the community needs to focus more.


review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper aims to consolidate some recent literature in simple types of "reading comprehension" tasks involving matching questions to answers to be found in a passage, and then to explore the types of structure learned by these models and propose modifications. These reading comprehension datasets such as CNN/Daily Mail are on the simpler side because they do not generally involve chains of reasoning over multiple pieces of supporting evidence as can be found in datasets like MCTest. Many models have been proposed for this task, and the paper breaks down these models into "aggregation readers" and "explicit reference readers." The authors show that the aggregation readers organize their hidden states into a predicate structure which allows them to mimic the explicit reference readers. The authors then experiment with adding linguistic features, including reference features, to the existing models to improve performance.

I appreciate the re-naming and re-writing of the paper to make it more clear that the aggregation readers are specifically learning a predicate structure, as well as the inclusion of results about dimensionality of the symbol space. Further, I think the effort to organize and categorize several different reading comprehension models into broader classes is useful, as the field has been producing many such models and the landscape is unclear. 

The concerns with this paper are that the predicate structure demonstrated is fairly simple, and it is not clear that it provides insight towards the development of better models in the future, since the "explicit reference readers" need not learn it, and the CNN/Daily Mail dataset has very little headroom left as demonstrated by Chen et al. 2016. The desire for "dramatic improvements in performance" mentioned in the discussion section probably cannot be achieved on these datasets. More complex datasets would probably involve multi-hop inference which this paper does not discuss. Further, the message of the paper is a bit scattered and hard to parse, and could benefit from a bit more focus.

I think that with the explosion of various competing neural network models for NLP tasks, contributions like this one which attempt to organize and analyze the landscape are valuable, but that this paper might be better suited for an NLP conference or journal such as TACL.

review_title: Simple predicate structure and data set

final_score: rating: null
review_content: 1.	In table 2, what gain did you get by using NER and POS features? How much gain can be obtained from just the first three features?
2.	I found table 1 and figure 1 are very confusing to me. Could the authors explain to me what kind of logical structure is being found by the network?  

review_title: Pre-review Questions

final_score: rating: null
review_content: In section 4, the authors describe how logical structure is emergent in a variety of neural readers that were proposed for machine comprehension tasks like CNN/DailyMail. Is it correct to interpret that the logical structure summarized in equation 11 and Table 1 boils down to semantic similarity between the (attended) text snippet and the question? Or am I missing something here?

(I think I was rather expecting to see conjunction, disjunction, if-then etc for logical structure...)

Also, can any of the neural readers discussed in the paper handle inference or reasoning that requires synthesizing information taken from more than one snippets from the given text?
review_title: logical structure

Meta-review: The reviewers mostly agree that this paper offers valuable insights about a family of problems in automatic "reading" of text, as well as current solutions. The paper seems to fail to generate excitement because it doesn't really point the way forward. I disagree with some reviewers about the work's suitability for ICLR (as opposed to an ACL venue) since ML researchers are also thinking about these tasks now. The consensus is that the paper will have greater impact (wherever it is published) with a clearer message.
Decision: Reject