Title: Machine Comprehension Using Match-LSTM and Answer Pointer
Authors: Shuohang Wang
Jing Jiang
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning

Abstract: Machine comprehension of text is an important problem in natural language processing. A recently released dataset, the Stanford Question Answering Dataset (SQuAD), offers a large number of real questions and their answers created by humans through crowdsourcing. SQuAD provides a challenging testbed for evaluating machine comprehension algorithms, partly because compared with previous datasets, in SQuAD the answers do not come from a small set of candidate answers and they have variable lengths. We propose an end-to-end neural architecture for the task. The architecture is based on match-LSTM, a model we proposed previously for textual entailment, and Pointer Net, a sequence-to-sequence model proposed by Vinyals et al. (2015) to constrain the output tokens to be from the input sequences. We propose two ways of using Pointer Net for our tasks. Our experiments show that both of our two models substantially outperform the best results obtained by Rajpurkar et al. (2016) using logistic regression and manually crafted features. Besides, our boundary model also achieves the best performance on the MSMARCO dataset (Nguyen et al. 2016).

TL;DR: Using Match-LSTM and Answer Pointer to select a variable length answer from a paragraph

Review discussion:
Hi,
In the introduction you  mentioned some state-of-the-art results (Salesforce Resarch?) that are not discussed in the result section.
I have some trouble to understand equation (2) is it possible that you missed the index j in both \alpha and G?
I am probably missing something because in this way the shapes do not sum up for me.
Same thing at equation (8)
Could you elaborate more on the statement of footnote 2?
The description of global search on the spans is not very clear, could you add something to it?
We thank you for your valuable comments!
For the state-of-the-art results, by the time of submission, the related paper wasn't released, so we didn't go into further discussion. The results were collected from the data set Leaderboard (https://rajpurkar.github.io/SQuAD-explorer/) which keeps updating the best performance on the hidden test data set. We will cite this link instead in the revision.
For the equation (2),  $H^q$ is a matrix, so $G_i$ represents a matrix with the dimension of l*Q here. $wG_i$ and $b \bigotimes e_Q$ would be the vectors with the dimension of Q. We think it's clearer to show the softmax function with a vector as input in equation (2).
So is the equation (8). We will clarify the dimension of G and F in the revision.
For the footnote 2, usually LSTM will get each hidden state through the equation "h=o*tanh(c)" where "o"is the output gate and "c" is the memory cell. We find the output gate in the pre-processing LSTM doesn't influence the performance, so we simplify it as "h=tanh(c)" to save some training time. We will add this in the revision.
About the global search: After getting the probabilities of the boundary points, we need to select a_s and a_e which make P(a_s) * P(a_e|a_s) largest as the boundary points. Actually, the selection of a_s will not influence the value of P(a_e|a_s), as the hidden representation of a_s is computed through $H^r \beta$ in equation (9). Without other restrictions, we can simply select a_s and a_e which make P(a_s) and P(a_e) largest respectively with the complexity of O(p), where "p" is the length of the paragraph sequence.
While the boundary model could point to a too long span covering several sentences, so we manually limit the length of the predicted span. Under this restriction, we need to search for the best a_s and a_e with the complexity of O(s*p), where "s" is the longest span we limit for the prediction. We will add more explanation about it in the revision.
Dear Authors,
Could you please clarify the following --
1. In equation 3, weighted question (which is a matrix, right?) is being combined with the current token of the passage (which is a vector, right?).
So, could you please clarify how is a vector being combined with matrix? What exact operation is being used?
2. How is the match-LSTM expected to work when the questions are not paraphrases of sentences from the original text?
For instance, for the questions which require world knowledge to resolve the correspondences with the sentences from the text (see type-2 in table 3 in the SQuAD paper (https://arxiv.org/pdf/1606.05250.pdf)).
Thanks,
Dear reviewer,
Thank you for your valuable comments!
1. In equation 3, the weighted sum of the question hidden states is a vector computed by the multiplication between $H^q$ and $\alpha_i$, where $H^q$ is a matrix with the dimension of "l*Q" and $\alpha_i$ is a vector with the dimension of "Q". We will clarify the dimension of $\alpha$ in the revision.
2. We think match-LSTM also has the ability to find the world knowledge alignment as long as similar knowledge appeared in the training data set. We just visualized the $\alpha$ values for this instance, and we find that "European", "Parliament", "Council", "European" and "Union" have larger probabilities aligned to the word "governing" in the question.
Besides, we also find that "and" has larger probability aligned to the word "two" in the question. We relist this raw instance in the development data set here:
Question: Which two governing bodies have legislative veto power?
Sen: ... the European Parliament and the Council of the European Union have powers of amendment and veto during the legislative process ...
We will show this visualization in the revision and add the analyse of the world knowledge issue soon.
Thanks,
Shuohang
Hi,
it would we great if you could clarify in words how your work integrates match-LSTM in pointr network, to clarify your contribution in that respect.
[I think the reader would benefit from this if you clarify that in the introduction]
A minor comment:
Table 2 would be clearer if you directly cite the corresponding works there if the numbers you report are identical to the ones the authors report in their work.
Thanks.
Dear reviewer,
We thank you for your valuable comments and will make the revisions as your comments soon!
We will clarify the integration of match-LSTM in pointer network in the introduction!
Yes, the reported numbers are identical and we will directly cite the corresponding works of the baselines in Table 2.
Thanks,
Shuohang
Dear reviewers,
Thank you for your valuable comments again! We have made the corresponding revisions and updated a new pdf version. We briefly list the changes here:
AnonReviewer1:
1.We clarify the dimension of row vector $\alpha_i$ to be $1\times Q$.
2.We add the visualization of the $\alpha$ values for the question requiring world knowledge in Figure 2 and add the corresponding analysis at the end of the section "Experiments".
AnonReviewer2:
We revise the description of the state-of-the-art results in the last paragraph in "Introduction".
We clarify the dimension of $G$ to be $l\time Q$, the row vector $\alpha_i$ to be $1\times Q$, the column vector $w$ to be $l\times 1$ for equation (2).
So is the equation (8).
We clarify the statement of footnote 3 about the output gates in the pre-processing layer.
We clarify the description of global search on the spans in both the boundary model description part and the Table 2.
AnonReviewer3:
We clarify the integration of match-LSTM and pointer network in the last two paragraphs of the "Introduction".
We directly cite the works of the baselines in Table 2.
Thanks,
Shuohang
SUMMARY.
This paper proposes a new neural network architectures for solving the task of reading comprehension question answering where the goal is answering a questions regarding a given text passage.
The proposed model combines two well-know neural network architectures match-lstm and pointer nets.
First the passage and the questions are encoded with a unidirectional LSTM.
Then the encoded words in the passage and the encoded words in the questions are combined with an attention mechanism so that each word of the passage has a certain degree of compatibility with the question.
For each word in the passage the word representation and the weighted representation of the query is concatenated and passed to an forward lstm.
The same process is done in the opposite direction with a backward lstm.
The final representation is a concatenation of the two lstms.
As a decoded a pointer network is used.
The authors tried with two approaches: generating the answer word by word, and generating the first index and the last index of the answer.
The proposed model is tested on the Stanford Question Answering Dataset.
An ensemble of the proposed model achieves performance close to state-of-the-art models.
OVERALL JUDGMENT
I think the model is interesting mainly because of the use of pointer networks as a decoder.
One thing that the authors could have tried is a multi-hop approach. It has been shown in many works to be extremely beneficial in the joint encoding of passage and query. The authors can think of it as a deep match-lstm.
The analysis of the model is interesting and insightful.
The sharing of the code is good.
We thank you for your valuable comments!
As our models on the SQuAD dataset was released through arXiv in August 2016, around two months earlier than the other studies on the SQuAD dataset, we believe we were the first to bring the following two useful insights into this task:
(1) Using attention mechanism to make each word of the passage to have a certain degree of compatibility with the question is important for the SQuAD dataset.
(2) Only predicting the start and the end points of the answer span is better than predicting the answer span word by word.
The other studies on the SQuAD dataset (Bidirectional Attention Flow for Machine Comprehension (https://openreview.net/forum?id=HJ0UKP9ge) and Dynamic Co-attention Networks For Question Answering (https://openreview.net/forum?id=rJeKjwvclx)) further explored these two insights and achieved the state-of-art performance.
In the latest version of the paper, we further added experiments on the dataset “MS MARCO: A Human Generated Machine Reading Comprehension Dataset”. We believe we are the first to explore this dataset where the question is a real submission to the Bing search engine. As the average length of the answer in this dataset is 16 words,  which is much longer than the average (3 words) for the SQuAD dataset, we further showed that our boundary model could significantly outperform the sequence model. The sequence model will always stop the prediction early, while the boundary model can somehow overcome this. Our boundary model can outperform the "Golden Passage" baseline, which uses the human selected passage from the 10 candidate passages as the answer. We only predict one span in these passages. We've updated the paper by adding this experiment and the analysis, and several prediction cases in the appendix.
We will further explore deep match-LSTM. Thank you for the suggestion!
Summary:
The paper presents a deep neural network for the task of machine comprehension on the SQuAD dataset. The proposed model is based on two previous works -- match-LSTM and Pointer Net. Match-LSTM produces attention over each word in the given question for each word in the given passage, and sequentially aggregates this matching of each word in the passage with the words in the question. The pointer net is used to generate the answer by either generating each word in the answer or by predicting the starting and ending tokens in the answer from the provided passage. The experimental results show that both the variants of the proposed model outperform the baseline presented in the SQuAD paper. The paper also shows some analysis of the results obtained such as variation of performance across answer lengths and question types.
Strengths:
1. A novel end-to-end model for the task of machine comprehension rather than using hand-crafted features.
2. Significant performance boost over the baseline presented in the SQuAD paper.
3. Some insightful analyses of the results such as performance is better when answers are short, "why" questions are difficult to answer.
Weaknesses/Questions/Suggestions:
1. The paper does not show quantitatively how much modelling attention in match-LSTM and answer pointer layer helps.
So, it would be insightful if authors could compare the model performance with and without attention in match-LSTM, and with and without attention in answer pointer layer.
2. It would be good if the paper could provide some insights into why there is a huge performance gap between boundary model and sequence model in the answer pointer layer.
3. I would like to see the variation in the performance of the proposed model for questions that require different types of reasoning (table 3 in SQuAD paper). This would provide insights into what are the strengths and weaknesses of the proposed model w.r.t the type reasoning required.
4. Could authors please explain why the activations resulting from {h^p}_i and {h^r}_{i-1} in G_i in equation 2 are being repeated across dimension of Q. Why not learn different activations for each dimension?
5. I wonder why Bi-Ans-Ptr is not used in the ensemble model (last row in table 2) when it is shown that Bi-Ans-Ptr improves performance by 1.2% in F1.
6. Could authors please discuss and compare the DCR model (in table 2) in the paper in more detail?
Review Summary: The paper presents a reasonable end-to-end model for the task of machine comprehension on the SQuAD dataset, which outperforms the baseline model significantly.
However, it would be good if more analyses / ablation studies / insights are included regarding -- how much attention helps, why is boundary model better than sequence model, how does the performance change when the reasoning required becomes difficult.
We thank you for your valuable comments!
Q: The paper does not show quantitatively how much modelling attention in match-LSTM and answer pointer layer helps.
So, it would be insightful if authors could compare the model performance with and without attention in match-LSTM, and with and without attention in answer pointer layer.
A: We have added these baselines in Table 1 with the model name "LSTM with Ans-Ptr". We use the final state of the question LSTM to replace the representation computed by the attention mechanism and the performance drop heavily without attention.
Q: It would be good if the paper could provide some insights into why there is a huge performance gap between boundary model and sequence model in the answer pointer layer.
A: The sequence model has a problem of early stop prediction, while the boundary model can somehow overcome this. We did another experiment on the "MS MARCO: A Human Generated MAchine Reading Comprehension Dataset" , where the question is a real submission to the Bing search engine. As the average length of the answer in this dataset is 16 words which is much longer than the 3 words in average for the SQuAD,  our boundary model can significantly outperform the sequence model. The average length of the answers generated by the sequence model is 7 words, while the boundary model is 21 words. We further show comparison between the predictions of the boundary and the sequence models in the appendix B.
Q: I would like to see the variation in the performance of the proposed model for questions that require different types of reasoning (table 3 in SQuAD paper). This would provide insights into what are the strengths and weaknesses of the proposed model w.r.t the type reasoning required.
A: We show the predictions of different models on different types of questions in Appendix B. Our model can't solve the questions that need several sentences reasoning.
Q: Could authors please explain why the activations resulting from {h^p}_i and {h^r}_{i-1} in G_i in equation 2 are being repeated across
dimension of Q. Why not learn different activations for each dimension?
A: We think that it is somehow introducing the word position information of the question when computing the attention weights, if each dimension learns
a diffenrt {W^p}. We think the semantic meaning of the word is more important than the position information for the attention mechanism. We will explore it in the future. Thank you for the suggestion!
Q: I wonder why Bi-Ans-Ptr is not used in the ensemble model (last row in table 2) when it is shown that Bi-Ans-Ptr improves performance by 1.2% in F1.
A: I'm sorry about it. Our model still runs on CPUs and that takes time to train a single model. We haven’t successfully built an ensemble model on it yet. We will further explore it and update it.
Q: Could authors please discuss and compare the DCR model (in table 2) in the paper in more detail?
A: Our model was first posted in August 2017 through arXiv, two months earlier than the DCR model and other similar work was posted. Our model has some similarities with DCR, such as the attention parts inspired by match-LSTM.
While they maximize the correct span from all the candidate spans, we predict the start and the end points of the span. We've added this in the revision.
Thanks for incorporating the comments. It is informative to know that the model can't solve questions that require reasoning across multiple sentences. I am changing the review score to "Good Paper, Accept."
The paper looks at the problem of locating the answer to a question in a text (For this task the answer is always part of the input text). For this the paper proposes to combine two existing works: Match-LSTM to relate question and text representations and Pointer Net to predict the location of the answer in the text.
Strength:
-	The suggested approach makes sense for the task and achieves good performance, (although as the authors mention, recent concurrent works achieve better results)
-	The paper is evaluated on the SQuAD dataset and achieves significant improvements over prior work.
Weaknesses:
1.	It is unclear from the paper how well it is applicable to other problem scenarios where the answer is not a subset of the input text.
2.	Experimental evaluation
2.1.	It is not clear why the Bi-Ans-Ptr in Table 2 is not used for the ensemble although it achieves the best performance.
2.2.	It would be interested if this approach generalizes to other datasets.
Other (minor/discussion points)
-	The task and approach seem to have some similarity of locating queries in images and visual question answering. The authors might want to consider pointing to related works in this direction.
-	I am wondering how much this task can be seen as a “guided extractive summarization”, i.e. where the question guides the summarization process.
-	Page 6, last paragraph: missing “.”: “… searching This…”
Summary:
While the paper presents an interesting combination of two approaches for the task of answer extraction, the novelty is moderate.
While the experimental results are encouraging, it remains unclear how well this approach generalizes to other scenarios as it seems a rather artificial task.
We thank you for your valuable comments!
Q: It is not clear why the Bi-Ans-Ptr in Table 2 is not used for the ensemble although it achieves the best performance.
A: I'm sorry about it. Our model still runs on CPUs and that takes time to train a single model. We haven’t successfully built an ensemble model on it yet. We will further explore it and update it.
Q: It would be interested if this approach generalizes to other datasets.
A: We further explored the dataset "MS MARCO: A Human Generated MAchine Reading COmprehension Dataset", where the question is a real submission to the Bing search. Our experiment is shown in Table 2. Our boundary model can outperform the "Golden Passage" baseline, which uses the human selected passage from the 10 candidate passages as the answer. We only predict one span in these passages. Our performance on the hidden test data is still in submission and we will update the paper when we get feedback.
Q: The task and approach seem to have some similarity of locating queries in images and visual question answering. The authors might want to consider pointing to related works in this direction.
A: We added the analysis of the difference between our work and VQA problem in the related work. Thank you for the suggestion!
Q: I am wondering how much this task can be seen as a “guided extractive summarization”, i.e. where the question guides the summarization process.
A: It's an quite interesting problem! We will further explore it! Thank you for the suggestion!
Q: Page 6, last paragraph: missing “.”: “… searching This…”
A: We have fixed the typo in the updated version. Thank you!
1. Thanks for the clarifications.
2. It is surprising that the long time of revision (1 month) was not sufficient to run the best ensemble model, even on CPU.
3. I encourage the authors to update the pdf with the official evaluation on MSMARCO, it is surprising that it takes so long (more than a week).
4. The additional experiments with the MSMARCO dataset is interesting and encouraging. There some questions though:
- "We select the span that is most similar the answer for training and only predict the span in the passages": this sentence is not clear to me.
Also please clarify how this done at test time. How is the selection made?
- Are there no other prior work results on this task/dataset?
- You show the human selected passage, but what be the upper bound for selecting a passage, according to the evaluation metrics?
- The computation for bleu 1 bleu 2 bleu 3 bleu 4 seems wrong as blue 4 should be much lower than blue 3,2,1 etc, by the definition of the measure. How is it computed?
Overall I keep my original judgment of 6.
We thank you for your valuable comments again!
Q: 2. It is surprising that the long time of revision (1 month) was not sufficient to run the best ensemble model, even on CPU.
A: I'm so sorry about it. I didn't have much experience in training ensemble model and made a mistake during the training. As we have GPU now, we have submitted better tuned models with F1 score 77% for a single model and 80% for its ensemble model on the development data set. We will update it after getting the feedback.
Q: 3. I encourage the authors to update the pdf with the official evaluation on MSMARCO, it is surprising that it takes so long (more than a week).
A: I'm sorry about it. We haven't received the official evaluation yet.We will update it as soon as we get it.
Q:4. The additional experiments with the MSMARCO dataset is interesting and encouraging. There some questions though:
- "We select the span that is most similar the answer for training and only predict the span in the passages": this sentence is not clear to me.
Also please clarify how this done at test time. How is the selection made?
A: We select the span that has the largest F1 score with the golden answer for training. During the test time, we only select one span in the passages as the answer.
Q: - Are there no other prior work results on this task/dataset?
A: We haven't found other paper exploring this data set besides the dataset paper. The original paper provides several baselines on a subset of the data set as follows:
(1)Best Passage         / Best ROUGE-L of any passage / 0.351
(2)Passage Ranking      / A DSSM-alike passage ranking model / 0.177
(3)Sequence to Sequence / Vanilla seq2seq model predicting answers from questions / 0.089
(4)Memory Network       / Seq2seq model with MemNN for passages / 0.119
As this subset is not released, we didn't add these performance into our pdf version. What we can see is that the baselines are much worse to the Golden(Best) Passage performance.
But our boundary model can outperform the Golden(Best) Passage performance.
Q:- You show the human selected passage, but what be the upper bound for selecting a passage, according to the evaluation metrics?
A:  The upper bound hasn't been provided by the authors yet. The corresponding baselines and the performance of our models will be released through their official website (http://www.msmarco.org/) soon.
Q: - The computation for bleu 1 bleu 2 bleu 3 bleu 4 seems wrong as blue 4 should be much lower than blue 3,2,1 etc, by the definition of the measure. How is it computed?
A: As the answer could just be a span in the passages, I think the exact match could make blue4 score much higher. We will further explore it. Thank you!
Sorry for such a late PDF update!
We just received the official evaluation on the MSMARCO (Microsoft MAchine Reading COmprehension Dataset) test data set:
bleu_1: 0.4071608785489091
bleu_2: 0.3388083705901471
bleu_3: 0.3064746094507665
bleu_4: 0.2872954087295379
rouge_l: 0.373298227899
And that is consistent with the performance on the development data set. The leaderboard (http://www.msmarco.org/) of this task is coming soon!
We also added the performance of the ensemble model on SQuAD dataset.
Thank you!

Meta-review:
role: PC
data: comment: This paper provides two approaches to question answering: pointing to spans, and use of match-LSTM. The models are evaluated on SQuAD and MSMARCO. The reviewers we satisfied that, with the provision of additional comparisons and ablation studies submitted during discussion, the paper was acceptable to the conference, albeit marginally so.

Decision: Accept (Poster)