SlowGuess/ACL_2020_OCR huggingface dataset:
Liangming Pan $^{1,2}$  Yuxi Xie $^{3}$  Yansong Feng $^{3}$
Tat-Seng Chua² Min-Yen Kan²
$^{1}$ NUS Graduate School for Integrative Sciences and Engineering
$^{2}$ School of Computing, National University of Singapore, Singapore
$^{3}$ Wangxuan Institute of Computer Technology, Peking University
e0272310@u.nus.edu, {xieyuxi, fengyansong}@pku.edu.cn
{dcscts@, kanmy@comp.}nus.edu.sg
This paper proposes the problem of Deep Question Generation (DQG), which aims to generate complex questions that require reasoning over multiple pieces of information of the input passage. In order to capture the global structure of the document and facilitate reasoning, we propose a novel framework which first constructs a semantic-level graph for the input document and then encodes the semantic graph by introducing an attention-based GGNN (Att-GGNN). Afterwards, we fuse the document-level and graph-level representations to perform joint training of content selection and question decoding. On the HotpotQA deep-question centric dataset, our model greatly improves performance over questions requiring reasoning over multiple facts, leading to state-of-the-art performance. The code is publicly available at https://github.com/WING-NUS/SG-Deep-Question-Generation.
Question Generation (QG) systems play a vital role in question answering (QA), dialogue system, and automated tutoring applications – by enriching the training QA corpora, helping chatbots start conversations with intriguing questions, and automatically generating assessment questions, respectively. Existing QG research has typically focused on generating factoid questions relevant to one fact obtainable from a single sentence (Duan et al., 2017; Zhao et al., 2018; Kim et al., 2019), as exemplified in Figure 1 a). However, less explored has been the comprehension and reasoning aspects of questioning, resulting in questions that are shallow and not reflective of the true creative human process.
People have the ability to ask deep questions about events, evaluation, opinions, synthesis, or reasons, usually in the form of Why, Why-not, How,
Input Sentence:
Oxygen is used in cellular respiration and released by photosynthesis, which uses
the energy of sunlight to produce oxygen from water.
Question: What life process produces oxygen in the presence of light?
Answer: Photosynthesis
a) Example of Shallow Question Generation
Input Paragraph A: Pago Pago International Airport
Pago Pago International Airport, also known as Tafuna Airport, is a public airport located 7 miles (11.3 km) southwest of the central business district of Pago Pago, in the village and plains of Tafuna on the island of Tutuila in American Samoa, an unincorporated territory of the United States.
Input Paragraph B: Hoonah Airport
Hoonah Airport is a state-owned public-use airport located one nautical mile (2 km) southeast of the central business district of Hoonah, Alaska.
Question: Are Pago Pago International Airport and Hoonah Airport both on American territory?
b) Example of Deep Question Generation
Figure 1: Examples of shallow/deep QG. The evidence needed to generate the question are highlighted.
What-if, which requires an in-depth understanding of the input source and the ability to reason over disjoint relevant contexts; e.g., asking Why did Gollum betray his master Frodo Baggins? after reading the fantasy novel The Lord of the Rings. Learning to ask such deep questions has intrinsic research value concerning how human intelligence embodies the skills of curiosity and integration, and will have broad application in future intelligent systems. Despite a clear push towards answering deep questions (exemplified by multi-hop reading comprehension (Cao et al., 2019) and commonsense QA (Rajani et al., 2019)), generating deep questions remains un-investigated. There is thus a clear need to push QG research towards generating deep questions that demand higher cognitive skills.
In this paper, we propose the problem of Deep Question Generation (DQG), which aims to generate questions that require reasoning over multiple pieces of information in the passage. Figure 1 b) shows an example of deep question which requires a comparative reasoning over two disjoint pieces of evidences. DQG introduces three additional challenges that are not captured by traditional QG systems. First, unlike generating questions from a single sentence, DQG requires document-level
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1463-1475
understanding, which may introduce long-range dependencies when the passage is long. Second, we must be able to select relevant contexts to ask meaningful questions; this is non-trivial as it involves understanding the relation between disjoint pieces of information in the passage. Third, we need to ensure correct reasoning over multiple pieces of information so that the generated question is answerable by information in the passage.
To facilitate the selection and reasoning over disjoint relevant contexts, we distill important information from the passage and organize them as a semantic graph, in which the nodes are extracted based on semantic role labeling or dependency parsing, and connected by different intra- and intersemantic relations (Figure 2). Semantic relations provide important clues about what contents are question-worthy and what reasoning should be performed; e.g., in Figure 1, both the entities Pago Pago International Airport and Hoonah Airport have the located_at relation with a city in United States. It is then natural to ask a comparative question: e.g., Are Pago Pago International Airport and Hoonah Airport both on American territory?. To efficiently leverage the semantic graph for DQG, we introduce three novel mechanisms: (1) proposing a novel graph encoder, which incorporates an attention mechanism into the Gated Graph Neural Network (GGNN) (Li et al., 2016), to dynamically model the interactions between different semantic relations; (2) enhancing the word-level passage embeddings and the node-level semantic graph representations to obtain an unified semantic-aware passage representations for question decoding; and (3) introducing an auxiliary content selection task that jointly trains with question decoding, which assists the model in selecting relevant contexts in the semantic graph to form a proper reasoning chain.
We evaluate our model on HotpotQA (Yang et al., 2018), a challenging dataset in which the questions are generated by reasoning over text from separate Wikipedia pages. Experimental results show that our model — incorporating both the use of the semantic graph and the content selection task — improves performance by a large margin, in terms of both automated metrics (Section 4.3) and human evaluation (Section 4.5). Error analysis (Section 4.6) validates that our use of the semantic graph greatly reduces the amount of semantic errors in generated questions. In summary, our contributions are: (1) the very first work, to the best of
our knowledge, to investigate deep question generation, (2) a novel framework which combines a semantic graph with the input passage to generate deep questions, and (3) a novel graph encoder that incorporates attention into a GGNN approach.
Question generation aims to automatically generate questions from textual inputs. Rule-based techniques for QG usually rely on manually-designed rules or templates to transform a piece of given text to questions (Heilman, 2011; Chali and Hasan, 2012). These methods are confined to a variety of transformation rules or templates, making the approach difficult to generalize. Neural-based approaches take advantage of the sequence-to-sequence (Seq2Seq) framework with attention (Bahdanau et al., 2014). These models are trained in an end-to-end manner, requiring far less labor and enabling better language flexibility, compared against rule-based methods. A comprehensive survey of QG can be found in Pan et al. (2019).
Many improvements have been proposed since the first Seq2Seq model of Du et al. (2017): applying various techniques to encode the answer information, thus allowing for better quality answer-focused questions (Zhou et al., 2017; Sun et al., 2018; Kim et al., 2019); improving the training via combining supervised and reinforcement learning to maximize question-specific rewards (Yuan et al., 2017); and incorporating various linguistic features into the QG process (Liu et al., 2019a). However, these approaches only consider sentence-level QG. In contrast, our work focuses on the challenge of generating deep questions with multi-hop reasoning over document-level contexts.
Recently, work has started to leverage paragraph-level contexts to produce better questions. Du and Cardie (2018) incorporated coreference knowledge to better encode entity connections across documents. Zhao et al. (2018) applied a gated self-attention mechanism to encode contextual information. However, in practice, semantic structure is difficult to distil solely via self-attention over the entire document. Moreover, despite considering longer contexts, these works are trained and evaluated on SQuAD (Rajpurkar et al., 2016), which we argue as insufficient to evaluate deep QG because more than  $80\%$  of its questions are shallow and only relevant to information confined to a single sentence (Du et al., 2017).
Figure 2: The framework of our proposed model (on the right) together with an input example (on the left). The model consists of four parts: (1) a document encoder to encode the input document, (2) a semantic graph encoder to embed the document-level semantic graph via Att-GGNN, (3) a content selector to select relevant question-worthy contents from the semantic graph, and (4) a question decoder to generate question from the semantic-enriched document representation. The left figure shows an input example and its semantic graph. Dark-colored nodes in the semantic graph are question-worthy nodes that are labeled to train the content selection task.
Given the document  $\mathcal{D}$  and the answer  $\mathcal{A}$ , the objective is to generate a question  $\bar{\mathcal{Q}}$  that satisfies:
$$
\bar {\mathcal {Q}} = \arg \max  _ {\mathcal {Q}} P (\mathcal {Q} | \mathcal {D}, \mathcal {A}) \tag {1}
$$
where document  $\mathcal{D}$  and answer  $\mathcal{A}$  are both sequences of words. Different from previous works, we aim to generate a  $\bar{\mathcal{Q}}$  which involves reasoning over multiple evidence sentences  $\mathcal{E} = \{s_i\}_{i=1}^n$ , where  $s_i$  is a sentence in  $\mathcal{D}$ . Also, unlike traditional settings,  $\mathcal{A}$  may not be a sub-span of  $\mathcal{D}$  because reasoning is involved to obtain the answer.
We propose an encoder-decoder framework with two novel features specific to DQG: (1) a fused word-level document and node-level semantic graph representation to better utilize and aggregate the semantic information among the relevant disjoint document contexts, and (2) joint training over the question decoding and content selection tasks to improve selection and reasoning over relevant information. Figure 2 shows the general architecture of the proposed model, including three modules: semantic graph construction, which builds the DP- or SRL-based semantic graph for the given input; semantic-enriched document representation, employing a novel Attention-enhanced Gated Graph Neural Network (Att-GGNN) to learn the semantic graph representations, which are then fused with
the input document to obtain graph-enhanced document representations; and joint-task question generation, which generates deep questions via joint training of node-level content selection and word-level question decoding. In the following, we describe the details of each module.
3.2 Semantic Graph Construction
As illustrated in the introduction, the semantic relations between entities serve as strong clues in determining what to ask about and the reasoning types it includes. To distill such semantic information in the document, we explore both SRL-(Semantic Role Labelling) and DP- (Dependency Parsing) based methods to construct the semantic graph. Refer to Appendix A for the details of graph construction.
- SRL-based Semantic Graph. The task of Semantic Role Labeling (SRL) is to identify what semantic relations hold among a predicate and its associated participants and properties (Márquez et al., 2008), including "who" did "what" to "whom", etc. For each sentence, we extract predicate-argument tuples via SRL toolkits<sup>1</sup>. Each tuple forms a subgraph where each tuple element (e.g., arguments, location, and temporal) is a node. We add intertuple edges between nodes from different tuples if they have an inclusive relationship or potentially mention the same entity.
<sup>1</sup>We employ the state-of-the-art BERT-based model (Shi and Lin, 2019) in the AllenNLP toolkit to perform SRL.
- DP-based Semantic Graph. We employ the bi-affine attention model (Dozat and Manning, 2017) for each sentence to obtain its dependency parse tree, which is further revised by removing unimportant constituents (e.g., punctuation) and merging consecutive nodes that form a complete semantic unit. Afterwards, we add inter-tree edges between similar nodes from different parse trees to construct a connected semantic graph.
The left side of Figure 2 shows an example of the DP-based semantic graph. Compared with SRL-based graphs, DP-based ones typically model more fine-grained and sparse semantic relations, as discussed in Appendix A.3. Section 4.3 gives a performance comparison on these two formalisms.
3.3 Semantic-Enriched Document Representations
We separately encode the document  $\mathcal{D}$  and the semantic graph  $\mathcal{G}$  via an RNN-based passage encoder and a novel Att-GGNN graph encoder, respectively, then fuse them to obtain the semantic-enriched document representations for question generation.
Document Encoding. Given the input document  $\mathcal{D} = [w_1, \dots, w_l]$ , we employ the bi-directional Gated Recurrent Unit (GRU) (Cho et al., 2014) to encode its contexts. We represent the encoder hidden states as  $\mathbf{X}_{\mathcal{D}} = [\mathbf{x}_1, \dots, \mathbf{x}_l]$ , where  $\mathbf{x}_i = [\vec{\mathbf{x}}_i; \vec{\mathbf{x}}_i]$  is the context embedding of  $w_i$  as a concatenation of its bi-directional hidden states.
Node Initialization. We define the SRL- and DP-based semantic graphs in an unified way. The semantic graph of the document  $\mathcal{D}$  is a heterogeneous multi-relation graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , where  $\mathcal{V} = \{v_{i}\}_{i = 1:N^{v}}$  and  $\mathcal{E} = \{e_k\}_{k = 1:N^e}$  denote graph nodes and the edges connecting them, where  $N^v$  and  $N^e$  are the numbers of nodes and edges in the graph, respectively. Each node  $v = \{w_j\}_{j = m_v}$  is a text span in  $\mathcal{D}$  with an associated node type  $t_v$ , where  $m_v / n_v$  is the starting / ending position of the text span. Each edge also has a type  $t_e$  that represents the semantic relation between nodes.
We obtain the initial representation  $\mathbf{h}_v^0$  for each node  $v = \{w_j\}_{j=m_v}^{n_v}$  by computing the word-to-node attention. First, we concatenate the last hidden states of the document encoder in both directions as the document representation  $\mathbf{d}_{\mathcal{D}} = [\vec{\mathbf{x}}_l; \vec{\mathbf{x}}_1]$ . Afterwards, for a node  $v$ , we calculate the attention distribution of  $\mathbf{d}_{\mathcal{D}}$  over all the words
$\{w_{m_v}, \dots, w_j, \dots, w_{n_v}\}$  in  $v$  as follows:
$$
\beta_ {j} ^ {v} = \frac {\exp (\operatorname {A t t n} \left(\mathbf {d} _ {\mathcal {D}} , \mathbf {x} _ {j}\right))}{\sum_ {k = m _ {n}} ^ {n _ {v}} \exp (\operatorname {A t t n} \left(\mathbf {d} _ {\mathcal {D}}, \mathbf {x} _ {k}\right))} \tag {2}
$$
where  $\beta_j^v$  is the attention coefficient of the document embedding  $\mathbf{d}_{\mathcal{D}}$  over a word  $w_{j}$  in the node  $v$ . The initial node representation  $\mathbf{h}_v^0$  is then given by the attention-weighed sum of the embeddings of its constituent words, i.e.,  $\mathbf{h}_v^0 = \sum_{j = m_v}^{n_v}\beta_j^v\mathbf{x}_j$ . Word-to-node attention ensures each node to capture not only the meaning of its constituting part but also the semantics of the entire document. The node representation is then enhanced with two additional features: the POS embedding  $\mathbf{p}_v$  and the answer tag embedding  $\mathbf{a}_v$  to obtain the enhanced initial node representations  $\tilde{\mathbf{h}}_v^0 = [\mathbf{h}_v^0;\mathbf{p}_v;\mathbf{a}_v]$ .
Graph Encoding. We then employ a novel Att-GGNN to update the node representations by aggregating information from their neighbors. To represent multiple relations in the edge, we base our model on the multi-relation Gated Graph Neural Network (GGNN) (Li et al., 2016), which provides a separate transformation matrix for each edge type. For DQG, it is essential for each node to pay attention to different neighboring nodes when performing different types of reasoning. To this end, we adopt the idea of Graph Attention Networks (Velickovic et al., 2017) to dynamically determine the weights of neighboring nodes in message passing using an attention mechanism.
Formally, given the initial hidden states of graph  $\mathbf{H}^0 = \{\tilde{\mathbf{h}}_i^0\} |_{v_i\in \nu}$ , Att-GGNN conducts  $K$  layers of state transitions, leading to a sequence of graph hidden states  $\mathbf{H}^0,\mathbf{H}^1,\dots ,\mathbf{H}^K$ , where  $\mathbf{H}^k = \{\mathbf{h}_j^{(k)}\} |_{v_j\in \nu}$ . At each state transition, an aggregation function is applied to each node  $v_{i}$  to collect messages from the nodes directly connected to  $v_{i}$ . The neighbors are distinguished by their incoming and outgoing edges as follows:
$$
\mathbf {h} _ {\mathcal {N} _ {\vdash (i)}} ^ {(k)} = \sum_ {v _ {j} \in \mathcal {N} _ {\vdash (i)}} \alpha_ {i j} ^ {(k)} \mathbf {W} ^ {t _ {e _ {i j}}} \mathbf {h} _ {j} ^ {(k)} \tag {3}
$$
$$
\mathbf {h} _ {\mathcal {N} _ {- 1 (i)}} ^ {(k)} = \sum_ {v _ {j} \in \mathcal {N} _ {- 1 (i)}} \alpha_ {i j} ^ {(k)} \mathbf {W} ^ {t _ {e _ {j i}}} \mathbf {h} _ {j} ^ {(k)} \tag {4}
$$
where  $\mathcal{N}_{\vdash(i)}$  and  $\mathcal{N}_{\vdash(i)}$  denote the sets of incoming and outgoing edges of  $v_{i}$ , respectively.  $\mathbf{W}^{t_{e_{ij}}}$  denotes the weight matrix corresponding to the edge type  $t_{e_{ij}}$  from  $v_{i}$  to  $v_{j}$ , and  $\alpha_{ij}^{(k)}$  is the attention
coefficient of  $v_{i}$  over  $v_{j}$ , derived as follows:
$$
\alpha_ {i j} ^ {(k)} = \frac {\exp \left(\operatorname {A t t n} \left(\mathbf {h} _ {i} ^ {(k)} , \mathbf {h} _ {j} ^ {(k)}\right)\right)}{\sum_ {t \in \mathcal {N} _ {(i)}} \exp \left(\operatorname {A t t n} \left(\mathbf {h} _ {i} ^ {(k)} , \mathbf {h} _ {t} ^ {(k)}\right)\right)} \tag {5}
$$
where  $\mathrm{Attn}(\cdot, \cdot)$  is a single-layer neural network implemented as  $\mathbf{a}^T[\mathbf{W}^A\mathbf{h}_i^{(k)}; \mathbf{W}^A\mathbf{h}_j^{(k)}]$ , here  $\mathbf{a}$  and  $\mathbf{W}^A$  are learnable parameters. Finally, an GRU is used to update the node state by incorporating the aggregated neighboring information.
$$
\mathbf {h} _ {i} ^ {(k + 1)} = \operatorname {G R U} \left(\mathbf {h} _ {i} ^ {(k)}, \left[ \mathbf {h} _ {\mathcal {N} _ {- (i)}} ^ {(k)} ; \mathbf {h} _ {\mathcal {N} _ {- (i)}} ^ {(k)} \right]\right) \tag {6}
$$
After the  $K$ -th state transition, we denote the final structure-aware representation of node  $v$  as  $\mathbf{h}_v^K$ .
Feature Aggregation. Finally, we fuse the semantic graph representations  $\mathbf{H}^K$  with the document representations  $\mathbf{X}_{\mathcal{D}}$  to obtain the semantic-enriched document representations  $\mathbf{E}_{\mathcal{D}}$  for question decoding, as follows:
$$
\mathbf {E} _ {\mathcal {D}} = \operatorname {F u s e} \left(\mathbf {X} _ {\mathcal {D}}, \mathbf {H} ^ {K}\right) \tag {7}
$$
We employ a simple matching-based strategy for the feature fusion function Fuse. For a word  $w_{i}\in \mathcal{D}$  , we match it to the smallest granularity node that contains the word  $w_{i}$  , denoted as  $v_{M(i)}$  . We then concatenate the word representation  $\mathbf{x}_i$  with the node representation  $\mathbf{h}_{v_{M(i)}}^{K}$  , i.e.,  $\mathbf{e}_i = [\mathbf{x}_i;\mathbf{h}_{v_{M(i)}}^K ]$  When there is no corresponding node  $v_{M(i)}$  , we concatenate  $\mathbf{x}_i$  with a special vector close to  $\vec{0}$
The semantic-enriched representation  $\mathbf{E}_{\mathcal{D}}$  provides the following important information to benefit question generation: (1) semantic information: the document incorporates semantic information explicitly through concatenating with semantic graph encoding; (2) phrase information: a phrase is often represented as a single node in the semantic graph (cf Figure 2 as an example); therefore its constituting words are aligned with the same node representation; (3) keyword information: a word (e.g., a preposition) not appearing in the semantic graph is aligned with the special node vector mentioned before, indicating the word does not carry important information.
3.4 Joint Task Question Generation
Based on the semantic-rich input representations, we generate questions via jointly training on two tasks: Question Decoding and Content Selection.
Question Decoding. We adopt an attention-based GRU model (Bahdanau et al., 2014) with copying (Gu et al., 2016; See et al., 2017) and coverage mechanisms (Tu et al., 2016) as the question decoder. The decoder takes the semantic-enriched representations  $\mathbf{E}_{\mathcal{D}} = \{\mathbf{e}_i, \forall w_i \in \mathcal{D}\}$  from the encoders as the attention memory to generate the output sequence one word at a time. To make the decoder aware of the answer, we use the average word embeddings in the answer to initialize the decoder hidden states.
At each decoding step  $t$ , the model learns to attend over the input representations  $\mathbf{E}_{\mathcal{D}}$  and compute a context vector  $\mathbf{c}_t$  based on  $\mathbf{E}_{\mathcal{D}}$  and the current decoding state  $\mathbf{s}_t$ . Next, the copying probability  $P_{cpy} \in [0,1]$  is calculated from the context vector  $\mathbf{c}_t$ , the decoder state  $\mathbf{s}_t$  and the decoder input  $y_{t-1}$ .  $P_{cpy}$  is used as a soft switch to choose between generating from the vocabulary, or copying from the input document. Finally, we incorporate the coverage mechanisms (Tu et al., 2016) to encourage the decoder to utilize diverse components of the input document. Specifically, at each step, we maintain a coverage vector  $\mathbf{cov}_t$ , which is the sum of attention distributions over all previous decoder steps. A coverage loss is computed to penalize repeatedly attending to the same locations of the input document.
Content Selection. To raise a deep question, humans select and reason over relevant content. To mimic this, we propose an auxiliary task of content selection to jointly train with question decoding. We formulate this as a node classification task, i.e., deciding whether each node should be involved in the process of asking, i.e., appearing in the reasoning chain for raising a deep question, exemplified by the dark-colored nodes in Figure 2.
To this end, we add one feed-forward layer on top of the final-layer of the graph encoder, taking the output node representations  $\mathbf{H}^K$  for classification. We deem a node as positive ground-truth to train the content selection task if its contents appear in the ground-truth question or act as a bridge entity between two sentences.
Content selection helps the model to identify the question-worthy parts that form a proper reasoning chain in the semantic graph. This synergizes with the question decoding task which focuses on the fluency of the generated question. We jointly train these two tasks with weight sharing on the input representations.
4.1 Data and Metrics