Venue: NAACL 2018

# Leveraging Context Information for Natural Question Generation

Linfeng Song\({}^{1}\), Zhiguo Wang\({}^{2}\), Wael Hamza\({}^{2}\), Yue Zhang\({}^{3}\) and Daniel Gildea\({}^{1}\)

\({}^{1}\)Department of Computer Science, University of Rochester, Rochester, NY 14627

\({}^{2}\)IBM T.J. Watson Research Center, Yorktown Heights, NY 10598

\({}^{3}\)Singapore University of Technology and Design

Work done during an internship at IBM.

###### Abstract

The task of natural question generation is to generate a corresponding question given the input passage (fact) and answer. It is useful for enlarging the training set of QA systems. Previous work has adopted sequence-to-sequence models that take a passage with an additional bit to indicate answer position as input. However, they do not explicitly model the information between answer and other context within the passage. We propose a model that matches the answer with the passage before generating the question. Experiments show that our model outperforms the existing state of the art using rich features.

## 1 Introduction

The task of natural question generation (NQG) is to generate a fluent and relevant question given a passage and a target answer. Recently NQG has received increasing attention from both the industrial and academic communities because of its values for improving QA systems by automatically increasing the training data. It can also be used for educational purposes such as language learning Heilman and Smith (2010).

One example is shown in Table 1, where a question "when was nikola tesla born ?" is generated given a passage and a fact "1856". Existing work for NQG uses a sequence-to-sequence model Sutskever et al. (2014), which takes a passage as input for generating a question. They either entirely ignore the target answer Du et al. (2017), or directly hard-code answer positions Zhou et al. (2017); Yang et al. (2017); Subramanian et al. (2017); Tang et al. (2017); Wang et al. (2017); Yuan et al. (2017). These methods can neglect rich potential interactions between the passage and the target answer. In addition, they fail when the target answer does not occur in the passage verbatim. In Table 1 the answer "1856" is the year when nikola tesla was born. This can be easily determined by leveraging the contextual information of "10 july 1856 - 7 January 1943", while it is relatively hard when only the answer position information is adopted.

We investigate explicit interaction between the target answer and the passage, so that contextual information can be better considered by the encoder. In particular, matching is used between the target answer and the passage for collecting relevant contextual information. We adopt the multi-perspective context matching (MPCM) algorithm Wang et al. (2017b), which takes two texts as input before producing a vector of numbers, representing similarity under different perspectives.

Results on SQuAD Rajpurkar et al. (2016) show that our model gives better BLEU scores than the state of the art. Furthermore, the questions generated by our model help to improve a strong extractive QA system. Our code is available at [https://github.com/freesunshine0316/MPQG](https://github.com/freesunshine0316/MPQG).

\begin{table}
\begin{tabular}{p{227.6pt} p{227.6pt}}  & **Passage:** nikola tesla ( Serbian cyrillic : Hinkolta Tecla ; 10 july 1856 – 7 january 1943 ) was a serbian american inventor , electrical engineer , mechanical engineer , physicist , and futurist best known for his contributions to the design of the modern alternating current ( ac ) electricity supply system . \\  & **Question:** when was nikola tesla born ? \\  & \\ \end{tabular}
\end{table}
Table 1: A QG example, where answer is underlined.



Baseline: sequence-to-sequence

Our baseline is a sequence-to-sequence model (Bahdanau et al., 2015) with the copy mechanism (Gulcehre et al., 2016; Gu et al., 2016). It uses an LSTM encoder to encode a passage and an LSTM decoder to synthesize a question.

### Encoder

The encoder is a bi-directional LSTM (Hochreiter and Schmidhuber, 1997), whose input \(x_{j}\) at step \(j\) is \([e_{j};b_{j}]\), the concatenation of the current word embedding \(e_{j}\) with additional bit \(b_{j}\) indicating whether it belongs to the answer.

### Decoder with the copy mechanism

The decoder is an attentional LSTM model, with the attention memory \(H\) being the concatenation of all encoder states. Each encoder state \(h_{j}\) is the concatenation of two bi-directional LSTM states:

\[h_{j}=[\overleftarrow{h_{j}};\overrightarrow{h_{j}}] \tag{1}\] \[H=[h_{0};\ldots;h_{N}]\text{,} \tag{2}\]

where \(N\) is the number of encoder states. At each step \(t\), the decoder state \(s_{t}\) and context vector \(c_{t}\) are generated from the previous decoder state \(s_{t-1}\), context vector \(c_{t-1}\) and output \(x_{t-1}\) in the same way as Bahdanau et al. (2015). The output distribution over a vocabulary is calculated via:

\[P_{vocab}=\text{softmax}(V_{1}[s_{t};c_{t}]+b_{1})\text{,}\]

where \(V_{1}\) and \(b_{1}\) are model parameters, and the number of rows in \(V_{1}\) is the size of the vocabulary.

Since many passage words also appear in the question, we adopt the copy mechanism (Gulcehre et al., 2016; Gu et al., 2016), which integrates the attention over input words into the final vocabulary distribution. The probability distribution is defined as the interpolation:

\[P_{final}=g_{t}P_{vocab}+(1-g_{t})P_{attn}\text{,}\]

where \(g_{t}\) is the switch for controlling generating a word from the vocabulary or directly copying it from the passage. \(P_{vocab}\) is the vocabulary probability distribution as defined above, and \(P_{attn}\) is calculated based on the current attention distribution by merging probabilities of duplicated words. Finally, \(g_{t}\) is defined as:

\[g_{t}=\sigma(w^{T}_{c}c_{t}+w^{T}_{s}s_{t}+w^{T}_{x}x_{t-1}+b_{2})\text{,}\]

where the vectors \(w_{c}\), \(w_{s}\), \(w_{x}\) and scalar \(b_{2}\) are model parameters.

## 3 Method

Our model follows the baseline encoder-decoder framework. The encoder reads a passage \(P=(p_{1},\dots,p_{N})\) and an answer \(A=(a_{1},\dots,q_{M})\); the decoder generates a question \(Q=(q_{1},\dots,q_{L})\) word by word.

### Multi-perspective encoder

Different from the baseline, we first encode both the passage and the answer by using two separate bi-directional LSTMs:

\[h^{p}_{j}=[\overleftarrow{h^{p}_{j}},\overleftarrow{h^{p}_{j}}] =\mathrm{BiLSTM}(\overleftarrow{h^{p}_{j+1}},\overleftarrow{h^{ p}_{j-1}},p_{j})\] \[h^{a}_{i}=[\overleftarrow{h^{a}_{i}},\overleftarrow{h^{a}_{i}}] =\mathrm{BiLSTM}(\overleftarrow{h^{a}_{i+1}},\overleftarrow{h^{a }_{i-1}},a_{i})\]

We use the multi-perspective context matching algorithm (Wang et al., 2017) on top of the BiLSTM outputs, matching each hidden state \(h^{p}_{j}\) of the passage against all hidden states \(h^{a}_{1}\dots h^{a}_{M}\) of the answer. The goal is to detect whether each passage word belongs to the relevant context of the answer. Shown in Figure 1, we adopt three strategies to match the passage with the answer, each investigating different sources of information.

_Full-matching_ considers the last hidden state of the answer, which encodes all words and the word order. _Attentive-matching_ synthesizes a vector by computing a weighted sum of all answer states against the passage state, then compares the vector with the passage state. It also considers all words in the answer but without word order. Finally, _max-attentive-matching_ only considers the most relevant answer state to the passage state.

**Multi-perspective matching** These strategies require a function \(f_{m}\) to match two vectors \(v_{1}\) and \(v_{2}\). It is defined as:

\[m=f_{m}(v_{1},v_{2};\mathbf{W})\text{,}\]

where \(\mathbf{W}\) is a tunable weight matrix. Each row \(W_{k}\in\mathbf{W}\) represents the weights associated with

Figure 1: Matching strategies.

 

one perspective, and the similarity according to that perspective is defined as:

\[m_{k}=\cos(W_{k}\odot v_{1},W_{k}\odot v_{2})\text{,}\]

where \(\odot\) is the element-wise multiplication operation. So \(f_{m}(v_{1},v_{2};\mathbf{W})\) represents the matching results between \(v_{1}\) and \(v_{2}\) from all perspectives. Intuitively, each _perspective_ calculates the cosine similarity between two reweighted input vectors, associated with a weight vector trained to highlight different dimensions of the input vectors. This can be regarded as considering a different part of the semantics captured in the vector.

The final matching vector \(m_{j}\) for the \(j\)-th word in the passage is the concatenation of the matching results of all three strategies. We employ another BiLSTM layer on top of the matching layer:

\[h^{m}_{j}=\overleftarrow{h^{m}_{j}},\overrightarrow{h^{m}_{j}}=\mathrm{BiLSTM }(\overleftarrow{h^{m}_{j+1}},\overrightarrow{h^{m}_{j-1}},m_{j})\]

Comparison with the baselineThe encoder states (\(h_{j}\)) of the baseline only contains the answer position information in addition to the passage content. The matching states (\(h^{m}_{j}\)) of our model includes the matching information of all passage words, and potentially contains the answer position information. The rich matching information can guide the decoder to generate more accurate questions.

### Decoder with the copy mechanism

The decoder is identical to the one described in Section 2.2, except that matching information (\(\overleftarrow{h^{m}_{j}},\overrightarrow{h^{m}_{j}}\)) is added to the attention memory:

\[h_{j} =[\overleftarrow{h^{p}_{j}};\overrightarrow{h^{p}_{j}}; \overleftarrow{h^{m}_{j}};\overrightarrow{h^{m}_{j}}] \tag{3}\] \[H =[h_{0};\ldots;h_{N}] \tag{4}\]

The attention memory contains not only the passage content, but also the matching information, which helps generate more accurate questions.

## 4 Experiments

Following existing work Du et al. (2017); Zhou et al. (2017), experiments are conducted on the publically accessible part of SQuAD Rajpurkar et al. (2016). The dataset contains 536 articles and over 100k questions, and around 10% is held by the organizer for fair evaluation.

### Settings

We evaluate our model for question quality against gold questions, as well as their effectiveness in improving an extractive QA system. Since Du et al. (2017) and Zhou et al. (2017) conducted their experiments using different training/dev/test split, we conduct experiments on both splits, and compare with their reported performance. For improving an extractive QA system, we use the data split of Du et al. (2017), and conduct experiments on low-resource settings, where only (10%, 20%, or 50%) of the human-labeled questions in the training data are available. We choose Wang et al. (2016) as the extractive QA system.

Both the baseline and our model are trained with cross-entropy loss. Greedy search is adopted for generating questions.

### Development experiments

Matching strategiesIn Table 2, we analyze the importance of each matching strategy by performing an ablation experiment on the devset according to the data split of Du et al. (2017). We can see that there is a performance decrease when removing each of the three matching strategies, which means that all three strategies are complementary. In addition, _w/o max-attentive-matching_ shows the least performance decrease. One likely reason is that _max-attentive-matching_ considers only the most similar hidden state of the answer, while the other two consider all hidden states. Finally, _w/o full-matching_ shows more performance decrease than _w/o attentive-matching_. A reason

Figure 2: Effectiveness on the number of perspectives.

 

may be that _full-matching_ captures word order information, while _attentive-matching_ does not.

Number of perspectivesFigure 2 shows the performance changes with different numbers of perspectives. There is a large performance improvement when increasing the number from 1 to 3, which becomes small when further increasing the number from 3 to 5. This shows that our multi-perspective matching algorithm is effective, as we do not need a large number of perspectives for reaching our reported performance.

### Results

In Table 3, we compare our model with the previous state of the art: _S2S-ans_[4] and _S2S+cp+f_[60]. Both methods use the sequence-to-sequence model. _S2S-ans_ encodes only the passage, yet does not use answer position information. _S2S+cp+f_ uses both answer position and rich features (NE and POS tags) by concatenating their embeddings with the word embedding on the encoder side [79], adopting the copy mechanism for their decoder. _S2S+cp_ is our sequence-to-sequence baseline with the copy mechanism, and _M2S+cp_ is our model, which further uses multi-perspective encoder.

_M2S+cp_ outperforms _S2S+cp_ on both data splits, showing that modeling contextual information is helpful for generating better questions. In addition, only taking word embedding features, _M2S+cp_ shows better performance than _S2S+cp+f_. Both multi-perspective matching and rich features play a similar role of leveraging more information than the answer position information. However, _M2S+cp_ can be applied to low-resource languages and domains, where there is not sufficient labeled data for training the taggers for generating rich features. _M2S+cp_ is also free from feature engineering, which is necessary for _S2S+cp+f_ on new domains.

Finally, unlike _S2S-ans_, _S2S+cp+f_ and _S2S+cp_, _M2S+cp_ can be useful when the answer is not explicitly contained in the passage, as it matches the target answer against the passage rather than using the answer position information.

### Example Output

Table 4 shows example outputs of _M2S+cp_ and _S2S+cp_. For the first case, _M2S+cp_ recognizes that "1856" is the year when "nikola tesla" is born, while _S2S+cp_ fails to. The matching algorithm of _M2S+cp_ gives high matching numbers for the phrase "10 july 1856 - 7 January 1943" with "1856" having the highest matching number, while _S2S+cp_ only highlights "1856". Simply highlighting "1856" can be ambiguous, while recognizing a pattern "day month year - day month year" with the first year being the answer is more definite. It is similar in the second case, where _M2S+cp_ recognize "zheng meaning right", which fits into the pattern "\(A\) meaning \(B\)" with \(B\) being the answer.

The third example is a case where the answer is not explicitly contained in the passage.1_M2S+cp_ generates a precise question, even though the answer "in illinois" does not appear in the passage. On the other hand, _S2S+cp_ fails in this case, as the answer position information can not be obtained from the input.

Footnote 1: This is modified from SQuAD, as all the original answers in the SQuAD dataset are explicitly contained in the passage.

### Question generation for extractive QA

Table 5 shows data augmentation results for extractive QA, where the gold questions of only a part of the training data are available. _Only-gold_ uses only the available gold questions to train the

\begin{table}
\begin{tabular}{l|c c|c} \hline \multirow{2}{*}{Models} & \multicolumn{2}{c|}{Split 1} & Split 2 \\  & BLEU & METEOR & ROUGE-L & BLEU \\ \hline S2S-ans & 12.28 & 16.62 & 39.75 & – \\ S2S+cp+f & – & – & – & 13.29 \\ \hline S2S+cp & 12.22 & 17.38 & 39.03 & 12.59 \\ M2S+cp & **13.98** & **18.77** & **42.72** & **13.91** \\ \hline \end{tabular}
\end{table}
Table 3: Test results.

 

extractive QA model, while _S2S+cp_ and _M2S+cp_ use all training data, adopting the model-generated questions if the gold question is not available. For evaluation metrics, _F1 score_ treats the prediction and ground-truth answer as bags of tokens, and compute their F1 score; _Exact Match_ measures the percentage of predictions that match the ground truth answer exactly Rajpurkar et al. (2016).

_M2S+cp_ is consistently better than _S2S+cp_ both under F1 score and Exact Match, showing that contextual information helps to generate more accurate questions. Besides, using 10% gold data, the automatically generated questions from _M2S+cp_ help to reach a better performance than that using only 20% gold data, and it is 11 points better than that using only 10% gold data.

## 5 Conclusion

We demonstrated that natural question generation can benefit from contextual information. Leveraging a multi-perspective matching algorithm, our model outperforms the existing state of the art, and our automatically generated questions help to improve a strong extractive QA system.

## Acknowledgments

We would like to thank the anonymous reviewers for their feedback.

## References

* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _ICLR 2015_.
* Du et al. (2017) Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017)_. pages 1342-1352.
* Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016)_.
* Gulcehre et al. (2016) Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016)_. Berlin, Germany.
* Heilman and Smith (2010) Michael Heilman and Noah A. Smith. 2010. Good Question! Statistical ranking for question generation. In _The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-2010)_. pages 609-617.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_ 9(8):1735-1780.
* Peng et al. (2016) Xi Peng, Rogerio S Feris, Xiaoyu Wang, and Dimitris N Metaxas. 2016. A recurrent encoder-decoder network for sequential face alignment. In _European Conference on Computer Vision_. pages 38-56.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of EMNLP 2016_. Austin, Texas, pages 2383-2392.
* Subramanian et al. (2017) Sandeep Subramanian, Tong Wang, Xingdi Yuan, and Adam Trischler. 2017. Neural models for key phrase detection and question generation. _arXiv preprint arXiv:1706.04560_ .
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _NIPS 2014_. pages 3104-3112.
* Tang et al. (2017) Duyu Tang, Nan Duan, Tao Qin, and Ming Zhou. 2017. Question answering and question generation as dual tasks. _arXiv preprint arXiv:1706.02027_ .
* Wang et al. (2017a) Tong Wang, Xingdi Yuan, and Adam Trischler. 2017a. A joint model for question answering and question generation. _arXiv preprint arXiv:1706.01450_ .
* Wang et al. (2017b) Zhiguo Wang, Wael Hamza, and Radu Florian. 2017b. Bilateral multi-perspective matching for natural language sentences. In _IJCAI 2017_.
* Wang et al. (2016) Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2016. Multi-perspective context matching for machine comprehension. _arXiv preprint arXiv:1612.04211_ .
* Yang et al. (2017) Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William Cohen. 2017. Semi-supervised QA with generative domain-adaptive nets. In _Proceedings of the 55th Annual Meeting of the Association for

\begin{table}
\begin{tabular}{l|c c c|c c c} \hline \multirow{2}{*}{Methods} & \multicolumn{2}{c|}{Exact Match (EM)} & \multicolumn{3}{c}{F1 score} \\ \cline{2-7}  & 10\% & 20\% & 50\% & 10\% & 20\% & 50\% \\ \hline only-gold & 47.87 & 57.98 & 63.60 & 59.64 & 68.05 & 73.02 \\ S2S+cp & 57.80 & 60.26 & 64.79 & 67.01 & 69.71 & 73.74 \\ M2S+cp & **59.11** & **61.40** & **65.95** & **67.73** & **70.60** & **75.08** \\ \hline \end{tabular}
\end{table}
Table 5: Results on improving extractive QA with automatically generated questions.

 

Computational Linguistics (ACL-2017)_. Vancouver, Canada, pages 1040-1050.
* Yuan et al. [2017] Xingdi Yuan, Tong Wang, Caglar Gulcehre, Alessandro Sordoni, Philip Bachman, Saizheng Zhang, Sandeep Subramanian, and Adam Trischler. 2017. Machine comprehension by text-to-text neural question generation. In _Proceedings of the 2nd Workshop on Representation Learning for NLP_. Vancouver, Canada, pages 15-25.
* Zhou et al. [2017] Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. _arXiv preprint arXiv:1704.01792_ .