SlowGuess/ACL_2020_OCR huggingface dataset:
To evaluate the model's ability to generate deep questions, we conduct experiments on HotpotQA (Yang et al., 2018), containing  $\sim 100\mathrm{K}$  crowd-sourced questions that require reasoning over separate Wikipedia articles. Each question is paired with two supporting documents that contain the evidence necessary to infer the answer. In the DQG task, we take the supporting documents along with the answer as inputs to generate the question. However, state-of-the-art semantic parsing models have difficulty in producing accurate semantic graphs for very long documents. We therefore pre-process the original dataset to select relevant sentences, i.e., the evidence statements and the sentences that overlap with the ground-truth question, as the input document. We follow the original data split of HotpotQA to pre-process the data, resulting in 90,440 / 6,072 examples for training and evaluation, respectively.
Following previous works, we employ BLEU 1-4 (Papineni et al., 2002), METEOR (Lavie and Agarwal, 2007), and ROUGE-L (Lin, 2004) as automated evaluation metrics. BLEU measures the average  $n$ -gram overlap on a set of reference sentences. Both METEOR and ROUGE-L specialize BLEU's  $n$ -gram overlap idea for machine translation and text summarization evaluation, respectively. Critically, we also conduct human evaluation, where annotators evaluate the generation quality from three important aspects of deep questions: fluency, relevance, and complexity.
We compare our proposed model against several strong baselines on question generation.
- Seq2Seq + Attn (Bahdanau et al., 2014): the basic Seq2Seq model with attention, which takes the document as input to decode the question.
- NQG++ (Zhou et al., 2017): which enhances the Seq2Seq model with a feature-rich encoder containing answer position, POS and NER information.
- ASs2s (Kim et al., 2019): learns to decode questions from an answer-separated passage encoder together with a keyword-net based answer encoder.
- S2sa-at-mp-gsa (Zhao et al., 2018): an enhanced Seq2Seq model incorporating gated self-attention and maxout-pointers to encode richer passage-level contexts (B4 in Table 1). We also implement a ver
sion that uses coverage mechanism and our answer encoder for fair comparison, labeled B5.
- CGC-QG (Liu et al., 2019a): another enhanced Seq2Seq model that performs word-level content selection before generation; i.e., making decisions on which words to generate and to copy using rich syntactic features, such as NER, POS, and DEP.
Implementation Details. For fair comparison, we use the original implementations of ASs2s and CGC-QG to apply them on HotpotQA. All baselines share a 1-layer GRU document encoder and question decoder with hidden units of 512 dimensions. Word embeddings are initialized with 300-dimensional pre-trained GloVe (Pennington et al., 2014). For the graph encoder, the node embedding size is 256, plus the POS and answer tag embeddings with 32-D for each. The number of layers  $K$  is set to 3 and hidden state size is 256. Other settings for training follow standard best practice<sup>2</sup>.
4.3 Comparison with Baseline Models
The top two parts of Table 1 show the experimental results comparing against all baseline methods. We make three main observations:
1. The two versions of our model — P1 and P2 — consistently outperform all other baselines in BLEU. Specifically, our model with DP-based semantic graph (P2) achieves an absolute improvement of 2.05 in BLEU-4  $(+15.2\%)$ , compared to the document-level QG model which employs gated self-attention and has been enhanced with the same decoder as ours (B5). This shows the significant effect of semantic-enriched document representations, equipped with auxiliary content selection for generating deep questions.
2. The results of CGC-QG (B6) exhibits an unusual pattern compared with other methods, achieving the best METEOR and ROUGE-L but worst BLEU-1 among all baselines. As CGC-QG performs word-level content selection, we observe that it tends to include many irrelevant words in the question, leading to lengthy questions (33.7 tokens on average, while 17.7 for ground-truth questions and 19.3 for our model) that are unanswerable or with semantic errors. Our model greatly reduces the error with node-level content selection based on semantic relations (shown in Table 3).
2All models are trained using Adam (Kingma and Ba, 2015) with mini-batch size 32. The learning rate is initially set to 0.001, and adaptive learning rate decay applied. We adopt early stopping and the dropout rate is set to 0.3 for both encoder and decoder and 0.1 for all attention mechanisms.
Model	BLEU1	BLEU2	BLEU3	BLEU4	Meteor	Rouge-L
Baselines	B1. Seq2Seq + Attn	32.97	21.11	15.41	11.81	18.19	33.48
B2. NQG++	35.31	22.12	15.53	11.50	16.96	32.01
B3. ASs2s	34.60	22.77	15.21	11.29	16.78	32.88
B4. S2s-at-mp-gsa	35.36	22.38	15.88	11.85	17.63	33.02
B5. S2s-at-mp-gsa (+cov, +ans)	38.74	24.89	17.88	13.48	18.39	34.51
B6. CGC-QG	31.18	22.55	17.69	14.36	25.20	40.94
Proposed	P1. SRL-Graph	40.40	26.83	19.66	15.03	19.73	36.24
P2. DP-Graph	40.55	27.21	20.13	15.53	20.15	36.94
Ablation	A1. -w/o Contexts	36.48	20.56	12.89	8.46	15.43	30.86
A2. -w/o Semantic Graph	37.63	24.81	18.14	13.85	19.24	34.93
A3. -w/o Multi-Relation & Attention	38.50	25.37	18.54	14.15	19.15	35.12
A4. -w/o Multi-Task	39.43	26.10	19.14	14.66	19.25	35.76
Table 1: Performance comparison with baselines and the ablation study. The best performance is in bold.
Model	Short Contexts	Medium Contexts	Long Contexts	Average
Flu.	Rel.	Cpx.	Flu.	Rel.	Cpx.	Flu.	Rel.	Cpx.	Flu.	Rel.	Cpx.
B4. S2sa-at-mp-gsa	3.76	4.25	3.98	3.43	4.35	4.13	3.17	3.86	3.57	3.45	4.15	3.89
B6. CGC-QG	3.91	4.43	3.60	3.63	4.17	4.10	3.69	3.85	4.13	3.75	4.15	3.94
A2. -w/o Semantic Graph	4.01	4.43	4.15	3.65	4.41	4.12	3.54	3.88	3.55	3.73	4.24	3.94
A4. -w/o Multi-Task	4.11	4.58	4.28	3.81	4.27	4.38	3.44	3.91	3.84	3.79	4.25	4.17
P2. DP-Graph	4.34	4.64	4.33	3.83	4.51	4.28	3.55	4.08	4.04	3.91	4.41	4.22
G1. Ground Truth	4.75	4.87	4.74	4.65	4.73	4.73	4.46	4.61	4.55	4.62	4.74	4.67
Table 2: Human evaluation results for different methods on inputs with different lengths.  $Flu$ ,  $Rel$ , and  $Cpx$ . denote the Fluency, Relevance, and Complexity, respectively. Each metric is rated on a 1–5 scale (5 for the best).
3. While both SRL-based and DP-based semantic graph models (P1 and P2) achieve state-of-the-art BLEU, DP-based graph (P2) performs slightly better  $(+3.3\%)$  in BLEU-4). A possible explanation is that SRL fails to include fine-grained semantic information into the graph, as the parsing often results in nodes containing a long sequence of tokens.
We also perform ablation studies to assess the impact of different components on the model performance against our DP-based semantic graph (P2) model. These are shown as Rows A1-4 in Table 1. Similar results are observed for the SRL-version.
- Impact of semantic graph. When we do not employ the semantic graph (A2, -w/o Semantic Graph), the BLEU-4 score of our model dramatically drops to 13.85, which indicates the necessity of building semantic graphs to model semantic relations between relevant content for deep QG. Despite its vital role, result of A1 shows that generating questions purely from the semantic graph is unsatisfactory. We posit three reasons: 1) the semantic graph alone is insufficient to convey the meaning of the entire document, 2) sequential information in the passage is not captured by the graph, and that 3) the automatically built semantic graph inevitably contains much noise. These reasons ne
cessitate the composite document representation.
- Impact of Att-GGNN. Using a normal GGNN (A3, -w/o Multi-Relation & Attention) to encode the semantic graph, performance drops to 14.15  $(-3.61\%)$  in BLEU-4 compared to the model with Att-GGNN (A4, -w/o Multi-Task). This reveals that different entity types and their semantic relations provide auxiliary information needed to generate meaningful questions. Our Att-GGNN model (P2) incorporates attention into the normal GGNN, effectively leverages the information across multiple node and edge types.
- Impact of joint training. By turning off the content selection task (A4, -w/o Multi-Task), the BLEU-4 score drops from 15.53 to 14.66, showing the contribution of joint training with the auxiliary task of content selection. We further show that content selection helps to learn a QG-aware graph representation in Section 4.7, which trains the model to focus on the question-worthy content and form a correct reasoning chain in question decoding.
We conduct human evaluation on 300 random test samples consisting of: 100 short (<50 tokens), 100 medium (50-200 tokens), and 100 long (>200 tokens) documents. We ask three workers to rate the 300 generated questions as well as the ground-truth
Types	Examples	S2sa-at-mp-gsa	CGC-QG	DP-Graph
Correct	(Pred.) (G.T.)	Between Kemess Mine and Colomac Mine, which mine was operated earlier? What mine was operated at an earlier date, Kemess Mine or Colomac Mine?	56.5%	52.9%	67.4%
Semantic Error	(Pred.) (G.T.)	Lawrence Ferlinghetti is an American poet, he is a short story written by who? Lawrence Ferlinghetti is an American poet, he wrote a short story named what?	17.7%	26.4%	8.3%
Answer Revealing	(Pred.) (G.T.)	What is the release date of this game released on 17 October 2006? What is the release date of this game named Hurricane?	2.1%	5.7%	1.4%
Ghost Entity	(Pred.) (G.T.)	When was the video game on which Michael Gelling plays Dr. Promoter? When was the video game on which Drew Gelling plays Dr. Promoter?	6.8%	0.7%	4.9%
Redundant	(Pred.) (G.T.)	What town did Walcha and Walcha belong to? What town did Walcha belong to?	16.3%	14.3%	13.9%
Unanswerable	(Pred.) (G.T.)	What is the population of the city Barack Obama was born? What was the ranking of the population of the city Barack Obama was born in 1999?	8.2%	18.6%	8.3%
Table 3: Error analysis on 3 different methods, with respects to 5 major error types (excluding the "Correct"). Pred. and G.T. show the example of the predicted question and the ground-truth question, respectively. Semantic Error: the question has logic or commonsense error; Answer Revealing: the question reveals the answer; Ghost Entity: the question refers to entities that do not occur in the document; Redundant: the question contains unnecessary repetition; Unanswerable: the question does not have the above errors but cannot be answered by the document.
questions between 1 (poor) and 5 (good) on three criteria: (1) Fluency, which indicates whether the question follows the grammar and accords with the correct logic; (2) Relevance, which indicates whether the question is answerable and relevant to the passage; (3) Complexity, which indicates whether the question involves reasoning over multiple sentences from the document. We average the scores from raters on each question and report the performance over five top models from Table 1. Raters were unaware of the identity of the models in advance. Table 2 shows our human evaluation results, which further validate that our model generates questions of better quality than the baselines. Let us explain two observations in detail:
- Compared against B4 (S2sa-at-mp-gsa), improvements are more salient in terms of "Fluency"  $(+13.33\%)$  and "Complexity"  $(+8.48\%)$  than that of "Relevance"  $(+6.27\%)$ . The reason is that the baseline produces more shallow questions (affecting complexity) or questions with semantic errors (affecting fluency). We observe similar results when removing the semantic graph (A2. - w/o Semantic Graph). These demonstrate that our model, by incorporating the semantic graph, produces questions with fewer semantic errors and utilizes more context.
- All metrics decrease in general when the input document becomes longer, with the most obvious drop in "Fluency". When input contexts is long, it becomes difficult for models to capture question-worthy points and conduct correct reasoning, leading to more semantic errors. Our model tries to alleviate this problem by introducing semantic graph and content selection, but question quality drops as noise increases in the semantic graph when the
document becomes longer.
In order to better understand the question generation quality, we manually check the sampled outputs, and list the 5 main error sources in Table 3. Among them, "Semantic Error", "Redundant", and "Unanswerable" are noticeable errors for all models. However, we find that baselines have more unreasonable subject-predicate-object collocations (semantic errors) than our model. Especially, CGC-QG (B6) has the largest semantic error rate of  $26.4\%$  among the three methods; it tends to copy irrelevant contents from the input document. Our model greatly reduces such semantic errors to  $8.3\%$ , as we explicitly model the semantic relations between entities by introducing typed semantic graphs. The other noticeable error type is "Unanswerable"; i.e., the question is correct itself but cannot be answered by the passage. Again, CGC-QG remarkably produces more unanswerable questions than the other two models, and our model achieves comparable results with S2sa-at-mp-gsa (B4), likely due to the fact that answerability requires a deeper understanding of the document as well as commonsense knowledge. These issues cannot be fully addressed by incorporating semantic relations. Examples of questions generated by different models are shown in Figure 3.
4.7 Analysis of Content Selection
We introduced the content selection task to guide the model to select relevant content and form proper reasoning chains in the semantic graph. To quantitatively validate the relevant content selection, we calculate the alignment of node attention
Passage 1) Last One Picked is the second studio album by the Christian rock band Superchic[k].
2) " Na Na " appeared on the Disney film , " Confessions of a Teenage Drama Queen "
3) Confessions of a Teenage Drama Queen is a 2004 American teen musical comedy film directed by Sara Sugarman and produced by Robert Shapiro and Matthew Hart for Walt Disney Pictures.
Semantic Graph
Figure 3: An example of generated questions and average attention distribution on the semantic graph, with nodes colored darker for more attention (best viewed in color).
$\alpha_{v_i}$  with respect to the relevant nodes  $\sum_{v_i\in RN}\alpha_{v_i}$  and irrelevant nodes  $\sum_{v_i\notin RN}\alpha_{v_i}$ , respectively, under the conditions of both single training and joint training, where  $RN$  represents the ground-truth we set for content selection. Ideally, a successful model should focus on relevant nodes and ignore irrelevant ones; this is reflected by the ratio between  $\sum_{v_i\in RN}\alpha_{v_i}$  and  $\sum_{v_i\notin RN}\alpha_{v_i}$ .
When jointly training with content selection, this ratio is 1.214 compared with 1.067 under single-task training, consistent with our intuition about content selection. Ideally, a successful model should concentrate on parts of the graph that help to form proper reasoning. To quantitatively validate this, we compare the concentration of attention in single- and multi-task settings by computing the entropy  $H = -\sum \alpha_{v_i} \log \alpha_{v_i}$  of the attention distributions. We find that content selection increases the entropy from 3.51 to 3.57 on average. To gain better insight, in Figure 3, we visualize the semantic graph attention distribution of an example. We see that the model pays more attention (is darker) to the nodes that form the reasoning chain (the highlighted paths in purple), consistent with the quantitative analysis.
We propose the problem of DQG to generate questions that requires reasoning over multiple disjoint pieces of information. To this end, we propose a novel framework which incorporates semantic
graphs to enhance the input document representations and generate questions by jointly training with the task of content selection. Experiments on the HotpotQA dataset demonstrate that introducing semantic graph significantly reduces the semantic errors, and content selection benefits the selection and reasoning over disjoint relevant contents, leading to questions with better quality.
There are at least two potential future directions. First, graph structure that can accurately represent the semantic meaning of the document is crucial for our model. Although DP-based and SRL-based semantic parsing are widely used, more advanced semantic representations could also be explored, such as discourse structure representation (van Noord et al., 2018; Liu et al., 2019b) and knowledge graph-enhanced text representations (Cao et al., 2017; Yang et al., 2019). Second, our method can be improved by explicitly modeling the reasoning chains in generation of deep questions, inspired by related methods (Lin et al., 2018; Jiang and Bansal, 2019) in multi-hop question answering.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2019. Question answering by reasoning across documents with graph convolutional networks. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 2306-2317.
Yixin Cao, Lifu Huang, Heng Ji, Xu Chen, and Juanzi Li. 2017. Bridge text and knowledge by learning multi-prototype entity mention embedding. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1623-1633.
Yllias Chali and Sadid A. Hasan. 2012. Towards automatic topical question generation. In International Conference on Computational Linguistics (COLING), pages 475-492.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In International Conference on Learning Representations (ICLR).
Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from wikipedia. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1907-1917.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1342-1352.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 866-874.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Annual Meeting of the Association for Computational Linguistics (ACL).
Michael Heilman. 2011. Automatic factual question generation from text. Language Technologies Institute School of Computer Science Carnegie Mellon University, 195.
Yichen Jiang and Mohit Bansal. 2019. Self-assembling modular networks for interpretable multi-hop reasoning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4473-4483.
Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. 2019. Improving neural question generation using answer separation. In AAAI Conference on Artificial Intelligence (AAAI), pages 6602-6609.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR).
Alon Lavie and Abhaya Agarwal. 2007. METEOR: an automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation (WMT@ACL), pages 228-231.
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. 2016. Gated graph sequence neural networks. In International Conference on Learning Representations (ICLR).
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2018. Multi-hop knowledge graph reasoning with reward shaping. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3243-3253.
Bang Liu, Mingjun Zhao, Di Niu, Kunfeng Lai, Yancheng He, Haojie Wei, and Yu Xu. 2019a. Learning to generate questions by learning what not to generate. In International World Wide Web Conference (WWW), pages 1106-1118.
Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2019b. Discourse representation parsing for sentences and documents. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 6248-6262.
Lluis Marquez, Xavier Carreras, Kenneth C. Litkowski, and Suzanne Stevenson. 2008. Semantic role labeling: An introduction to the special issue. Computational Linguistics, 34(2):145-159.
Lluis Marquez, Xavier Carreras, Kenneth C. Litkowski, and Suzanne Stevenson. 2008. Semantic role labeling: An introduction to the special issue. Computational Linguistics, 34(2):145-159.
Rik van Noord, Lasha Abzianidze, Antonio Toral, and Johan Bos. 2018. Exploring neural methods for parsing discourse representation structures. Transactions of the Association for Computational Linguistics (TACL), 6:619-633.
Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. 2019. Recent advances in neural question generation. CoRR, abs/1905.08949.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 311-318.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4932-4942.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In *Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 2383-2392.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1073-1083.
Peng Shi and Jimmy Lin. 2019. Simple BERT models for relation extraction and semantic role labeling. CoRR, abs/1904.05255.
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. 2018. Answer-focused and position-aware neural question generation. In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3930-3939.
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL).
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. CoRR, abs/1710.10903.
Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. 2019. Auto-encoding scene graphs for image captioning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10685–10694.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2369-2380.