Title: Emergent Predication Structure in Vector Representations of Neural Readers
Authors: Hai Wang
Takeshi Onishi
Kevin Gimpel
David McAllester
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning
Applications

Abstract: Reading comprehension is a question answering task where the answer is to be found in a given passage about entities and events not mentioned in general knowledge sources. A significant number of neural architectures for this task (neural readers) have recently been developed and evaluated on large cloze-style datasets. We present experiments supporting the emergence of “predication structure” in the hidden state vectors of a class of neural readers including the Attentive Reader and Stanford Reader. We posits that the hidden state vectors can be viewed as (a representation of) a concatenation [P, c] of a “predicate vector” P and a “constant symbol vector” c and that the hidden state represents the atomic formula P(c). This predication structure plays a conceptual role in relating “aggregation readers” such as the Attentive Reader and the Stanford Reader to “explicit reference readers” such as the Attention-Sum Reader, the Gated-Attention Reader and the Attention-over-Attention Reader. In an independent contribution, we show that the addition of linguistics features to the input to existing neural readers significantly boosts performance yielding the best results to date on the Who-did-What dataset.

TL;DR: Provide some novel insights on reading comprehension models and boost the performance of those models

Review discussion:
1.	In table 2, what gain did you get by using NER and POS features? How much gain can be obtained from just the first three features?
2.	I found table 1 and figure 1 are very confusing to me. Could the authors explain to me what kind of logical structure is being found by the network?
Question: 1.	In table 2, what gain did you get by using NER and POS features? How much gain can be obtained from just the first three features?
Reply:
NER and POS features are only used for the CBT experiments. For the anonymized WDW experiments, only the first three features were used. We will update the paper to make it more clear.
Question: 2.	I found table 1 and figure 1 are very confusing to me. Could the authors explain to me what kind of logical structure is being found by the network?
Reply: You are not the only one. :) Please see the detailed explanation in the answer to the first question of reviewer 1.  We will rework section 4 along the lines of the response to reviewer 1.
In section 4, the authors describe how logical structure is emergent in a variety of neural readers that were proposed for machine comprehension tasks like CNN/DailyMail. Is it correct to interpret that the logical structure summarized in equation 11 and Table 1 boils down to semantic similarity between the (attended) text snippet and the question? Or am I missing something here?
(I think I was rather expecting to see conjunction, disjunction, if-then etc for logical structure...)
Also, can any of the neural readers discussed in the paper handle inference or reasoning that requires synthesizing information taken from more than one snippets from the given text?
All reviewers asked for clarification of section 4.  Please see the detailed response to the first question of reviewer 1.  This response also addresses integration of information from different parts of the passage. We will rework section 4.
In what sense is this a logical structure? A main finding of the paper seems to be the section after equation 11 where it is shown that the aggregation readers effectively learn to mimic the explicit reference readers. This is an emergent structure, but what sort of logic does it encode? Is the term "logic" in reference to the symbolic nature of the learned anonymized entity embeddings?
(11) and (12) imply a H = S (+) E direct sum structure for the representations? This is because the entity identifiers, being semantics-free, must be permutable (approximately) without changing the model scores, correct? Do the authors have a sense of how big the E subspace must be to handle a given number of entity vectors? Is it equal to the maximum number of entities in a document?
Regarding equation 11, it is unclear why the inner-product of h_{t} and the answer embedding would be a fixed positive constant 'c' - especially since h_{t} is the embedding obtained by running a bi-LSTM/GRU encoder on the paragraph. i.e. for all t in R(a,p), the h_t's will encode everything it has seen before.
However it is intuitive that since all t's in R(a,p) refers to positions of answers in the document, the inner product would be high (but not necessarily fixed).
Also in equation 11, it is also very counter-intuitive that for all t not in R(a,p) the inner product of h_{t} and the answer embedding would be zero.
For example, consider the embedding of h_{t+1}. Since this word is just after the answer word, will it have a very low inner product? It would be nice to add this number after row 2 in table 1.
It is hard to follow equation 12. A new variable h' has been introduced just before which has not been used in the equation which makes it even more confusing. I think the paper would benefit a lot if section 4 is re-visited and written more clearly because I think it is the most important section of the paper (especially explanation/intuition behind equation 12 and 13).
Equation 13 suggests the existence of a sum-structure between two orthogonal subspaces S (subspace of statement vectors) and E(subspace of entity pointers). Do the experiment in section 5 explicitly capture this? It is clear to me that the one-hot and general pointer encoding are ways to inject coreference information in the model but does it explain the additive nature of the embeddings (since the two vectors are concatenated).
Reviewer 1, we greatly appreciate your careful analysis of our paper.
Question: In what sense is this a logical structure? A main finding of the paper seems to be the section after equation 11 where it is shown that the aggregation readers effectively learn to mimic the explicit reference readers. This is an emergent structure, but what sort of logic does it encode? Is the term "logic" in reference to the symbolic nature of the learned anonymized entity embeddings?
Reply: We are arguing for the emergence of “predication”.  We will change the title to “Emergent Predication Structure …”.  By predication we mean that the hidden state h_t of the passage at position (or word) t can be viewed as a pair (Phi_t, c_t) where Phi_t is a property (or statement or predicate) being stated of a particular constant symbol c_t at position t in the passage.  A logician might write this as h_t = Phi_t[c_t].  We present evidence that the hidden state vector h_t can be “parsed” into its parts Phi_t and c_t and that these parts can be interpreted as a predicate and a constant symbol.
A question has the form Psi[x] --- a question asks for a candidate answer c such that the passage implies Psi[c].  At a simplified level we are looking for a place t in the passage such that the statement Phi_t[x] being expressed at position t implies Psi[x] (for any x).  In this case the answer is the constant symbol c_t being “talked about” at position t.
To allow for a combination of evidence from different statements in the passage the answer selection might be
(1) argmax_c   sum_t    <e(Phi_t),e(Psi)>  <e(c_t),e(c)>.
The first inner product is interpreted as measuring the extent to which Phi_t[x] implies Psi[x] for any x.  The second inner product should be near zero if c_t is different from c and near a large constant if c_t = c.
Selection rule (1) is not built into aggregation readers. In an aggregation reader the training objective has the form
(2)  argmax_c  sum_t   <h_t,q>  <h_t,e(c)>.
Here q is the vector representation of the question.  Equations (1) and (2) agree if q is interpreted as e(Psi) and h_t is a direct sum of e(Phi_t) and e(c_t).  We claim that this structure of q and h_t emerges from training on (2).  This claim is supported in table 1 and figure 1.
Selection rules (1) and (2) also address a question of reviewer 3 --- the support for an answer can indeed combine several statements in the passage.
We will revise the paper shortly to make the intended structure clearer.
Question: (11) and (12) imply a H = S (+) E direct sum structure for the representations? This is because the entity identifiers, being semantics-free, must be permutable (approximately) without changing the model scores, correct? Do the authors have a sense of how big the E subspace must be to handle a given number of entity vectors? Is it equal to the maximum number of entities in a document?
Reply: Yes, our intuition started from the the fact that the entity identifiers can be permuted and therefore must be semantics free.  Even for non-anonymized data such as Who-Did-What, annotating the passage with which places in the passage refer to which answers improves performance.  We believe that even in this case an “answer pointer” is coded in the hidden state and equations (11) and (12) appear to hold.  This is supported by the performance of the one-hot pointer reader in table 2.
In the case of CNN there are about 500 entities but the entity embedding is only 100 dimensional.  We did some experiments with “two-sparse” entity id embeddings which were designed and not trained.  These did reasonably well.  These codings give 2d(d-1) distinct embeddings in dimension d (there are d choose 2 sparsity patterns and 4 choices of the sign of the non-zero values). We looked at the literature on packing vectors on the surface of a sphere where we found the suggestion of two-sparse vectors.  This generalizes to k-sparse vectors but we have no experiments for k > 2. We will add some discussion of this in the next revision.
Question: Regarding equation 11, it is unclear why the inner-product of h_{t} and the answer embedding would be a fixed positive constant 'c' - especially since h_{t} is the embedding obtained by running a bi-LSTM/GRU encoder on the paragraph. i.e. for all t in R(a,p), the h_t's will encode everything it has seen before.
However it is intuitive that since all t's in R(a,p) refers to positions of answers in the document, the inner product would be high (but not necessarily fixed).
Reply: We select the entity identifier with the largest inner product with the weighted hidden state vector.   If one of the entity identifier embeddings had larger norm than the others then the selection process would be biased toward that entity identifier.
But the system should be invariant to permuting the identifiers.  The caption of figure 1 gives the mean and variance of the square norm of the entity embeddings and indeed the variance (17) is small compared to the mean (169).  We will say something about this in the revised version.
Question: Also in equation 11, it is also very counter-intuitive that for all t not in R(a,p) the inner product of h_{t} and the answer embedding would be zero.
For example, consider the embedding of h_{t+1}. Since this word is just after the answer word, will it have a very low inner product? It would be nice to add this number after row 2 in table 1.
Reply: We will add this to table 1.
It should be noted that a precise word position for each candidate answer is given to the system. In the case of CNN it is the position of the anonymized entity identifier.  In other cases it is given by explicit reference annotation.  The attention heatmaps show that the attention is highly focused on the precise word of the given candidate answer positions.
So the equivalence of aggregation readers and explicit reference readers does not require that (11) holds at non-answer positions adjacent to answer positions.
But if the attention is highly focused on the precise answer positions there is no reason to remember answer embeddings --- the LSTM (or GRU) could immediately forget them to leave room in the hidden state for more useful information.
But we will do the measurement.
Question: It is hard to follow equation 12. A new variable h' has been introduced just before which has not been used in the equation which makes it even more confusing. I think the paper would benefit a lot if section 4 is re-visited and written more clearly because I think it is the most important section of the paper (especially explanation/intuition behind equation 12 and 13).
Reply: The introduction of h’ in the discussion is simply to formalize the invariance to permutation of the entity identifiers.  We may drop this and just say “because the system should be invariant to entity identifier permutation …”.  We will definitely rework section 4.
Question: Equation 13 suggests the existence of a sum-structure between two orthogonal subspaces S (subspace of statement vectors) and E(subspace of entity pointers). Do the experiment in section 5 explicitly capture this? It is clear to me that the one-hot and general pointer encoding are ways to inject coreference information in the model but does it explain the additive nature of the embeddings (since the two vectors are concatenated).
Reply:  In general concatenation guarantees sum-structure --- [x,y] can be uniquely decomposed as [x,0] + [0,y] where the addends are orthogonal.  It would be simpler to work just with concatenation.
However, the learning system need not make the orthogonal subspaces S and E axis-aligned.
The paper proposed to analyze several recently developed machine readers and found that some machine readers could potentially take advantages of the entity marker (given that the same marker points out to the same entity). I usually like analysis papers, but I found the argument proposed in this paper not very clear.
I like the experiments on the Stanford reader, which shows that the entity marker in fact helps the Stanford reader on WDW. I found that results rather interesting.
However, I found the organization and the overall message of this paper quite confusing.
First of all, it feels that the authors want to explain the above behavior with some definition of the “structures”.
However, I am not sure that how successful the attempt is. For me, it is still not clear what the structures are. This makes reading section 4 a bit frustrating.
I am also not sure what is the take home message of this paper. Does it mean that the entity marking should be used in the MR models? Should we design models that can also model the entity reference at the same time? What are the roles of the linguistic features here? Should we use linguistic structure to overcome the reference issue?
Overall, I feel that the analysis is interesting, but I feel that the paper can benefit from having a more focused argument.
First of all, thanks for this valuable suggestion. Below are our answers for the question.
Reviewer: However, I found the organization and the overall message of this paper quite confusing.
First of all, it feels that the authors want to explain the above behavior with some definition of the “structures”.
However, I am not sure that how successful the attempt is. For me, it is still not clear what the structures are. This makes reading section 4 a bit frustrating.
Response: Our claim is that in the aggregation reader the hidden state vector can be decomposed into two parts --- a semantics-free entity pointer and a property being stated of that pointer.  See our earlier comments on predication structure.
Reviewer: I am also not sure what is the take home message of this paper. Does it mean that the entity marking should be used in the MR models? Should we design models that can also model the entity reference at the same time? What are the roles of the linguistic features here? Should we use linguistic structure to overcome the reference issue?
Response:
Q: Does it mean that the entity marking should be used in the MR models?
Q: Should we design models that can also model the entity reference at the same time?
Entity pointers contain reference information.  In the case of multiple choice reading comprehension questions the pointers indicate which answer choice is being referred to.  All high performing reading comprehension systems rely on reference annotation of some sort.  Anonymization provides reference annotation because it replaces entities in both the answer choices and the passage with entity identifiers making the coreference apparent.
But if we eventually want to move to doing the coreference within the deep model then entity pointers seem essential. It seems important to note that aggregation readers are already constructing entity pointers.
Q: What are the roles of the linguistic features here? Should we use linguistic structure to overcome the reference issue?
Our linguistic features provide some simple information that seems orthogonal to the issue of entity pointers and predication structure. The fact that we observe consistent improvement by adding these features suggests that the readers are not being learned to capture this information otherwise. We find improvements from features with both an aggregation reader and an explicit reference reader (see Table 3).
Like the above point about modeling coreference better in the deep models themselves, these findings suggest that another future research direction is to seek to capture simple beneficial linguistics features in the readers themselves. We will make this more clear in the paper.
Hi, Dear reviewers,
Thanks for the valuable suggestions on our paper and sorry to make you confused about the notations, we have updated the paper to make it more clear. Please see the latest version.
This paper aims to provide an insightful and analytic survey over the recent literature on reading comprehension with the distinct goal of investigating whether logical structure (or predication, as the authors rephrased in their response) arises in many of the recent models. I really like the spirit of the paper and appreciate the efforts to organize rather chaotic recent literature into two unified themes: "aggregation readers" and "explicit reference models”.
Overall the quality of writing is great and section 3 was especially nice to read. I’m also happy with the proposed rewording from "logical structure" to “predication", and the clarification by the authors was detailed and helpful.
I think I still have slight mixed feelings about the contribution of the work.
First, I wonder whether the choice of the dataset was ideal in the first place to accomplish the desired goal of the paper. There have been concerns about CNN/DailyMail dataset (Chen et al. ACL’16) and it is not clear to me whether the dataset supports investigation on logical structure of interesting kinds. Maybe it is bound to be rather about lack of logical structure.
Second, I wish the discussion on predication sheds more practical insights into dataset design or model design to better tackle reading comprehension challenges. In that sense, it may have been more helpful if the authors could make more precise analysis on different types of reading comprehension challenges, what types of logical structure are lacking in various existing models and datasets, and point to specific directions where the community needs to focus more.
The paper aims to consolidate some recent literature in simple types of "reading comprehension" tasks involving matching questions to answers to be found in a passage, and then to explore the types of structure learned by these models and propose modifications. These reading comprehension datasets such as CNN/Daily Mail are on the simpler side because they do not generally involve chains of reasoning over multiple pieces of supporting evidence as can be found in datasets like MCTest. Many models have been proposed for this task, and the paper breaks down these models into "aggregation readers" and "explicit reference readers." The authors show that the aggregation readers organize their hidden states into a predicate structure which allows them to mimic the explicit reference readers. The authors then experiment with adding linguistic features, including reference features, to the existing models to improve performance.
I appreciate the re-naming and re-writing of the paper to make it more clear that the aggregation readers are specifically learning a predicate structure, as well as the inclusion of results about dimensionality of the symbol space.
Further, I think the effort to organize and categorize several different reading comprehension models into broader classes is useful, as the field has been producing many such models and the landscape is unclear.
The concerns with this paper are that the predicate structure demonstrated is fairly simple, and it is not clear that it provides insight towards the development of better models in the future, since the "explicit reference readers" need not learn it, and the CNN/Daily Mail dataset has very little headroom left as demonstrated by Chen et al. 2016. The desire for "dramatic improvements in performance" mentioned in the discussion section probably cannot be achieved on these datasets. More complex datasets would probably involve multi-hop inference which this paper does not discuss.
Further, the message of the paper is a bit scattered and hard to parse, and could benefit from a bit more focus.
I think that with the explosion of various competing neural network models for NLP tasks, contributions like this one which attempt to organize and analyze the landscape are valuable, but that this paper might be better suited for an NLP conference or journal such as TACL.

Meta-review:
role: PC
data: comment: The reviewers mostly agree that this paper offers valuable insights about a family of problems in automatic "reading" of text, as well as current solutions. The paper seems to fail to generate excitement because it doesn't really point the way forward. I disagree with some reviewers about the work's suitability for ICLR (as opposed to an ACL venue) since ML researchers are also thinking about these tasks now. The consensus is that the paper will have greater impact (wherever it is published) with a clearer message.

Decision: Reject