SlowGuess/ACL_2020_OCR huggingface dataset:
Zi Chai, Xiaojun Wan
{chaizi, wanxiaojun}@pku.edu.cn
Traditional Question Generation (TQG) aims to generate a question given an input passage and an answer. When there is a sequence of answers, we can perform Sequential Question Generation (SQG) to produce a series of interconnected questions. Since the frequently occurred information omission and coreference between questions, SQG is rather challenging. Prior works regarded SQG as a dialog generation task and recurrently produced each question. However, they suffered from problems caused by error cascades and could only capture limited context dependencies. To this end, we generate questions in a semi-autoregressive way. Our model divides questions into different groups and generates each group of them in parallel. During this process, it builds two graphs focusing on information from passages, answers respectively and performs dual-graph interaction to get information for generation. Besides, we design an answer-aware attention mechanism and the coarse-to-fine generation scenario. Experiments on our new dataset containing 81.9K questions show that our model substantially outperforms prior works.
Question Generation (QG) aims to teach machines to ask human-like questions from a range of inputs such as natural language texts (Du et al., 2017), images (Mostafazadeh et al., 2016) and knowledge bases (Serban et al., 2016). In recent years, QG has received increasing attention due to its wide applications. Asking questions in dialog systems can enhance the interactiveness and persistence of human-machine interactions (Wang et al., 2018). QG benefits Question Answering (QA) models through data augmentation (Duan et al., 2017) and joint learning (Sun et al., 2019). It also plays an important role in education (Heilman and Smith, 2010) and clinical (Weizenbaum et al., 1966) systems.
Traditional Question Generation (TQG) is defined as the reverse task of QA, i.e., a passage and an answer (often a certain span from the passage) are provided as inputs, and the output is a question grounded in the input passage targeting on the given answer. When there is a sequence of answers, we can perform Sequential Question Generation (SQG) to produce a series of interconnected questions. Table 1 shows an example comparing the two tasks. Intuitively, questions in SQG are much more concise and we can regard them with given answers as QA-style conversations. Since it is more natural for human beings to test knowledge or seek information through coherent questions (Reddy et al., 2019), SQG has wide applications, e.g., enabling virtual assistants to ask questions based on previous discussions to get better user experiences.
SQG is a challenging task in two aspects. First, information omissions between questions lead to complex context dependencies. Second, there are frequently occurred coreference between questions. Prior works regarded SQG as a dialog generation task (namely conversational QG) where questions are generated autoregressively (recurrently), i.e., a new question is produced based on previous outputs. Although many powerful dialog generation models can be adopted to address the challenges mentioned above, there are two major obstacles. First, these models suffer from problems caused by error cascades. Empirical results from experiments reveal that the later generated questions tend to become shorter with lower quality, especially becoming more irrelevant to given answers, e.g., "Why?", "What else?". Second, models recurrently generating each question struggle to capture complex context dependencies, e.g., long-distance coreference. Essentially, SQG is rather different from dialog generation since all answers are given in advance and they act as strict semantic constraints during text generation.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 225-237 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
(1) A small boy named  $[\mathrm{John}]_1$  was at the park one day.
(2) He was [swinging]2 [on the swings]3 and [his friend]4 named [Tim]5 [played on the slide]6.
(3) John wanted to play on the slide now.
(4) He asked Tim [if he could play on the slide]7.
(5) Tim said [no]s, and he cried.
Turn	TQG	SQG	Answer
1	Who was at the park?	Who was at the park?	John
2	What was John doing at the park?	What was he doing there?	swinging
3	Where was John swinging?	On what?	on the wings
4	Who was with John at the park?	Who was he with?	his friend
5	What is the name of John's friend?	Named?	Tim
6	What was Tim doing?	What was he doing?	played on the side
7	What did John asked Tim?	What did John asked him?	if he could play on the slide
8	What did Tim say to John?	What did he say?	no
Table 1: Comparison of Traditional Question Generation (TQG) and Sequential Question Generation (SQG). The given passage contains five sentences, and we mark the given answers in the passage as blue.
To deal with these problems, we perform SQG in a semi-autoregressive way. More specifically, we divide target questions into different groups (questions in the same group are closely-related) and generate all groups in parallel. Especially, our scenario becomes non-autoregressive if each group only contains a single question. Since we eliminate the recurrent dependencies between questions in different groups, the generation process is much faster and our model can better deal with the problems caused by error cascades. To get information for the generation process, we perform dual-graph interaction where a passage-info graph and an answer-info graph are constructed and iteratively updated with each other. The passage-info graph is used for better capturing context dependencies, and the answer-info graph is used to make generated questions more relevant to given answers with the help of our answer-aware attention mechanism. Besides, a coarse-to-fine text generation scenario is adopted for the coreference resolution between questions.
Prior works performed SQG on CoQA (Reddy et al., 2019), a high-quality dataset for conversational QA. As will be further illustrated, a number of data in CoQA are not suitable for SQG. Some researchers (Gao et al., 2019) directly discarded these data, but the remaining questions may become incoherent, e.g., the antecedent words for many pronouns are unclear. To this end, we build a new dataset from CoQA containing 81.9K relabeled questions. Above all, the main contributions of our work are:
- We build a new dataset containing 7.2K passages and 81.9K questions from CoQA. It is the first dataset specially built for SQG as far as we know.
- We perform semi-autoregressive SQG under dual-graph interaction. This is the first time that SQG is not regarded as a dialog generation task. We also propose an answer-aware attention mechanism and a coarse-to-fine generation scenario for better performance.
- We use extensive experiments to show that our model outperforms previous work by a substantial margin. Further analysis illustrated the impact of different components.
Dataset for this paper is available at https://github.com/ChaiZ-pku/Sequential-QG.
2.1 Traditional Question Generation
TQG was traditionally tackled by rule-based methods (Lindberg et al., 2013; Mazidi and Nielsen, 2014; Hussein et al., 2014; Labutov et al., 2015), e.g., filling handcrafted templates under certain transformation rules. With the rise of data-driven learning approaches, neural networks (NN) have gradually taken the mainstream. Du et al. (2017) pioneered NN-based QG by adopting the Seq2seq architecture (Sutskever et al., 2014). Many ideas were proposed since then to make it more powerful, including answer position features (Zhou et al., 2017), specialized pointer mechanism (Zhao et al., 2018), self-attention (Scialom et al., 2019), answer separation (Kim et al., 2019), etc. In addition, enhancing the Seq2seq model into more complicated structures using variational inference, adversarial training and reinforcement learning (Yao et al., 2018; Kumar et al., 2019) have also gained much attention. There are also some works performing TQG under certain constraints, e.g., controlling the
topic (Hu et al., 2018) and difficulty (Gao et al., 2018) of questions. Besides, combining QG with QA (Wang et al., 2017; Tang et al., 2017; Sun et al., 2019) is also focused by many researchers.
2.2 Sequential Question Generation
As human beings tend to use coherent questions for knowledge testing or information seeking, SQG plays an important role in many applications. Prior works regarded SQG as a dialog generation task (namely conversational QA). Pan et al. (2019) pretrained a model performing dialog generation, and then fine-tuned its parameters by reinforcement learning to make generated questions relevant to given answers. Gao et al. (2019) iteratively generated questions from previous outputs and leveraged off-the-shelf coreference resolution models to introduce a coreference loss. Besides, additional human annotations were performed on sentences from input passages for conversation flow modeling.
Since SQG is essentially different from dialog generation, we discard its dialog view and propose the first semi-autoregressive SQG model. Compared with using the additional human annotation in Gao et al. (2019), our dual-graph interaction deals with context dependencies automatically. Besides, our answer-aware attention mechanism is much simpler than the fine-tuning process in Pan et al. (2019) to make outputs more answer-relevant.
As the reverse task of QA, QG is often performed on existing QA datasets, e.g., SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), etc. However, questions are independent in most QA datasets, making TQG the only choice. In recent years, the appearance of large-scale conversational QA datasets like CoQA (Reddy et al., 2019) and QuAC (Choi et al., 2018) makes it possible to train data-driven SQG models, and the CoQA dataset was widely adopted by prior works. Since the test set of CoQA is not released to the public, its training set (7.2K passages with 108.6K questions) was split into new training and validation set, and its validation set (0.5K passages with 8.0K questions) was used as the new test set.
Different from traditional QA datasets where the answers are certain spans from given passages, answers in CoQA are free-form text<sup>1</sup> with cor
responding evidence highlighted in the passage. This brings a big trouble for QG. As an example, consider the yes/no questions counting for  $19.8\%$  among all questions. Given the answer “yes” and a corresponding evidence “…the group first met on July 5, 1967 on the campus of the Ohio state university…”, there are many potential outputs, e.g., “Did the group first met in July?”, “Was the group first met in Ohio state?”. When considering the context formed by previous questions, the potential outputs become even more (the original question in CoQA is “Was it founded the same year?”). When there are too many potential outputs with significantly different semantic meanings, training a converged QG model becomes extremely difficult. For this reason, Gao et al. (2019) directly discarded questions that cannot be answered by spans from passages. However, the remaining questions can become incoherent, e.g., antecedent words for many pronouns become unclear.
To this end, we build a new dataset from CoQA by preserving all 7.7K passages and rewriting all questions and answers. More specifically, we first discarded questions that are unsuitable for SQG. To do so, three annotators were hired to vote for the preservation/deletion of each question. A question is preserved if and only if it can be answered by a certain span from the input passage<sup>2</sup>. As a result, most deleted questions were yes/no questions and unanswerable questions. Besides, the kappa score between results given by different annotators was 0.83, indicating that there was a strong inter-agreement between annotators. For the remaining QA-pairs, we preserved their original order and replaced all answers by spans from input passages. After that, we rewrote all questions to make them coherent. To avoid over-editing, annotators were asked to modify as little as possible. It turned out that in most cases, they only needed to deal with coreference since the prototype of pronouns were no longer existed. To further guarantee the annotation quality, we hired another project manager who daily examined  $10\%$  of the annotations from each annotator and provided feedbacks. The annotation was considered valid only when the accuracy of examined results surpasses  $95\%$ . Our annotation process took 2 months, and we finally got a dataset containing 7.7K passage with 81.9K QA-pairs.
2Using certain spans from input passages (instead of free-formed text) as answers is a conversion in QG. In this way, the number of potential output questions is greatly reduced.
Only  $66.8\%$  of the answers overlap with the passage after ignoring punctuations and case mismatches.
Figure 1: Architecture of our model. The example is corresponding with Table 1
In this section, we formalize the SQG task and introduce our model in details. As shown in Figure 1, the model first builds a passage-info graph and an answer-info graph by its passage-info encoder and answer-info encoder respectively. After that, it performs dual-graph interaction to get representations for the decoder. Finally, different groups of questions are generated in parallel under a coarse-to-fine scenario. Both encoders and decoder take the form of Transformer architecture (Vaswani et al., 2017).
4.1 Problem Formalization
In SQG, we input a passage composed by  $n$  sentences  $P = \{S_{i}\}_{i=1}^{n}$  and a sequence of  $l$  answers  $\{A_{i}\}_{i=1}^{l}$ , each  $A_{i}$  is a certain span of  $P$ . The target output is a series of questions  $\{Q_{i}\}_{i=1}^{l}$ , where  $Q_{i}$  can be answered by  $A_{i}$  according to the input passage  $P$  and previous QA-pairs.
As mentioned above, we perform SQG in an semi-autoregressive way, i.e., target questions are divided into different groups. Ideally, questions in the same group are expected to be closely-related, while questions in different groups should be as independent as possible. Our model takes a simple but effective unsupervised question clustering method. The intuition is: if two answers come from the same sentence, the two corresponding questions are likely to be closely-related. More specifically, if the  $k$ -th sentence  $S_{k}$  contains  $p$  answers from  $\{A_{i}\}_{i=1}^{l}$ , we cluster them into an answer-group  $\mathbb{G}_k^{ans} = \{A_{j_1}, A_{j_2}, \ldots, A_{j_p}\}$  where  $j_1 < j_2 < \ldots < j_p$  are continuous indexes from
$\{1,2,\dots,l\}$ . By replacing each answer in  $\mathbb{G}_k^{ans}$  with its corresponding question, we get a question-group  $\mathbb{G}_k^{ques} = \{Q_{j_1},Q_{j_2},\ldots ,Q_{j_p}\}$ , and we further define a corresponding target-output  $T_{k}$  as "  $Q_{j_1}$  [sep]  $Q_{j_2}$  [sep] ... [sep]  $Q_{j_p}$ " where "[sep]" is a special token. In Table 1, there are four target outputs  $T_{1},T_{2},T_{4},T_{5}$  (no  $T_{3}$  since the third sentence in Table 1 do not contain any answer),  $T_{2}$  is "What was he doing there? [sep] On What? [sep] ... [sep] What was Tim doing?" corresponding with the second sentence, and  $T_{5}$  is "What did he say?" corresponding with the last sentence. Supposing there are  $m$  answers- and question-groups, then our model generates all the  $m$  target-outputs in parallel, i.e., all questions are generated in a semi-autoregressive way.
4.2 Passage-Info encoder
As shown in Figure 1, our passage-info encoder maps input sentences  $\{S_i\}_{i=1}^n$  into their sentence representations  $\{s_i\}_{i=1}^n$  where every  $s_i \in \mathbb{R}^{2d_s}$ . We regard each sentence as a sequence of words and replace each word by its pre-trained word embeddings (Mikolov et al., 2013) which is a dense vector. After that, the sequence of word embeddings is sent to a Transformer-encoder that outputs a corresponding sequence of vectors. By averaging these vectors, we get the local representation  $s_i^{local} \in \mathbb{R}^{d_s}$  of  $S_i$ .
After we get the local representations of all sentences  $\{S_i\}_{i=1}^n$  in passage  $P$ , another Transformer-encoder is adopted to map the sequence  $\{\pmb{s}_i^{local}\}_{i=1}^n$  into  $\{\pmb{s}_i^{global}\}_{i=1}^n$ , where  $\pmb{s}_i^{global} \in \mathbb{R}^{d_s}$  is called the
Figure 2: Illustration of answer embeddings and an answer-attention head for the forth sentence in Table 1.
global representation for  $S_{i}$ . In other words, the passage-info encoder takes a hierarchical structure. We expect the local and global representations capture intra- and inter- sentence context dependencies respectively, and the final representation for  $S_{i}$  is  $\pmb{s}_i = [\pmb{s}_i^{\text{local}}; \pmb{s}_i^{\text{global}}] \in \mathbb{R}^{2d_s}$ .
4.3 Answer-Info Encoder
As described in Section 4.1, the input answers are split into  $m$  answer-groups. For  $\mathbb{G}_k^{ans}$  corresponding with the  $k$ -th sentence of the input passage, we define  $\{\mathbb{G}_k^{ans}, S_k\}$  as a "rationale"  $R_k$ , and further obtain its representation  $\boldsymbol{r}_k \in \mathbb{R}^{2d_r}$  by our answer-info encoder, which is based on a Transformer-encoder regarding sentence  $S_k$  as its input.
To further consider information from  $\mathbb{G}_k^{ans}$ , two more components are added into the answer-info encoder, as shown in Figure 2. First, we adopt the answer-tag features. For each word  $w_i$  in sentence  $S_k$ , the embedding layer computes  $[x_i^w; x_i^a] \in \mathbb{R}^{d_r}$  as its final embedding, where  $x_i^w$  is the pre-trained word embedding and  $x_i^a$  contains answer-tag features. More specifically, we give  $w_i$  a label from  $\{\mathrm{O}, \mathrm{B}, \mathrm{I}\}$  if it is "outside", "the beginning of", "inside of" any answer from  $\mathbb{G}_k^{ans}$ , and use a vector corresponding with this label as  $x_i^a$ . Second, we design the answer-aware attention mechanism. In the multi-head attention layer, there are not only  $l_h$  vanilla "self-attention heads", but also  $l_a$  "answer-aware heads" for each answer in  $\mathbb{G}_k^{ans}$ . In an answer-aware head corresponding with answer  $A$ , words not belonging to  $A$  are masked out during the attention mechanism. The output of the Transformer-encoder is a sequence of vectors  $H_k^{enc} = \{h_k^{enc}\} (h_k^{enc} \in \mathbb{R}^{d_r})$  corresponding with the input word sequence from  $S_k$ .
After getting  $H_{k}^{enc}$ , we further send the se
quence of vectors to a bi-directional GRU network (Chung et al., 2014) and take its last hidden state as the final rationale embedding  $\pmb{r}_k\in \mathbb{R}^{2d_r}$ .
4.4 Graph Construction
In our SQG task, the input passage contain  $n$  sentences, which can be represented by  $\{s_i\}_{i = 1}^n\in$ $\mathbb{R}^{2d_s}$  leveraging the passage-info encoder. Among all input sentences, only  $m$  of them contain certain answers  $(m\leq n)$ , and we further define  $m$  rationales based on these sentences,  $\{\mathbb{G}_{F(j)}^{ans},S_{F(j)}\}_{j = 1}^{m}$  where the  $j$ -th rationale  $(j\in \{1,2,\dots,m\})$  corresponds with the  $F(j)$ -th sentence of the input passage  $(F(j)\in \{1,2,\dots,n\})$ . For the example in Table 1,  $n = 5$ ,  $m = 4$ ,  $F(j)$  maps  $\{1,2,3,4\}$  into  $\{1,2,4,5\}$  respectively. Using the answer-info encoder, we can get representations  $\{\pmb {r}_{F(j)}\}_{j = 1}^{m}\in$ $\mathbb{R}^{2d_s}$  for all rationales.
We further build a passage-info graph  $\mathcal{V}$  and an answer-info graph  $\mathcal{U}$  based on these representations. For the rationale corresponding with the  $k$ -th sentence of the input passage, we add node  $u_{k}, v_{k}$  in graph  $\mathcal{U}, \mathcal{V}$  respectively. For the example in Table 1,  $\mathcal{U}$  is composed by  $\{u_{1}, u_{2}, u_{4}, u_{5}\}$  and  $\mathcal{V}$  is composed by  $\{v_{1}, v_{2}, v_{4}, v_{5}\}$ , as shown in Figure 1. The initial representation for  $u_{k}$  is computed by:
$$
\boldsymbol {u} _ {k} ^ {(0)} = \operatorname {R e L U} \left(\boldsymbol {W} _ {u} \left[ \boldsymbol {r} _ {k}; \boldsymbol {e} _ {k} \right] + \boldsymbol {b} _ {u}\right) \in \mathbb {R} ^ {d _ {g}} \tag {1}
$$
where  $\pmb{r}_k\in \mathbb{R}^{2d_r}$  is the rationale representation,  $\pmb {e}_k\in \mathbb{R}^{d_e}$  is the embedding of index  $k$  , and  $\pmb {W}_u\in$ $\mathbb{R}^{(d_e + 2d_r)\times d_g},\pmb {b}_u\in \mathbb{R}^{d_g}$  are trainable parameters. And the initial representation for  $v_{k}$  is:
$$
\boldsymbol {v} _ {k} ^ {(0)} = R e L U \left(\boldsymbol {W} _ {v} \left[ \boldsymbol {s} _ {k}; \boldsymbol {e} _ {k} \right] + \boldsymbol {b} _ {v}\right) \in \mathbb {R} ^ {d _ {g}} \tag {2}
$$
where  $\pmb{s}_k\in \mathbb{R}^{2d_s}$  is the sentence representation and  $\pmb {W}_v\in \mathbb{R}^{(d_e + 2d_s)\times d_g},\pmb {b}_v\in \mathbb{R}^{d_g}$  are parameters.
After adding these points, there are  $m$  nodes in  $\mathcal{U}$  and  $\nu$  respectively. For  $u_{i}, u_{j} \in \mathcal{U}$  corresponding with the  $i$ -th,  $j$ -th input sentences respectively, we add an edge between them if  $|i - j| < \delta$  ( $\delta$  is a hyper-parameter). Similarly, we add edges into  $\mathcal{V}$  and the two graphs are isomorphic.
4.5 Dual-Graph Interaction
In our answer-info graph  $\mathcal{U}$ , node representations contain information focused on input answers. In the passage-info graph  $\mathcal{V}$ , node representations capture inter- and intra-sentence context dependencies. As mentioned above, a good question should be
answer-relevant as well as capturing complex context dependencies. So we should combine information in both  $\mathcal{U}$  and  $\nu$ . Our dual-graph interaction is a process where  $\mathcal{U}$  and  $\nu$  iteratively update node representations with each other. At time step  $t$ , representations  $\pmb{u}_i^{(t - 1)},\pmb{v}_i^{(t - 1)}$  are updated into  $\pmb{u}_i^{(t)},\pmb{v}_i^{(t)}$  respectively under three steps.
First, we introduce the information transfer step. Taking  $\mathcal{U}$  as an example. Each  $\pmb{u}_i^{(t - 1)}$  receives  $\pmb{a}_i^{(t)}$  from its neighbors (two nodes are neighbors if there is an edge between them) by:
$$
\boldsymbol {a} _ {i} ^ {(t)} = \sum_ {u _ {j} \in \mathcal {N} (u _ {i})} \boldsymbol {W} _ {i j} \boldsymbol {u} _ {j} ^ {(t - 1)} + \boldsymbol {b} _ {i j} \tag {3}
$$
where  $\mathcal{N}(u_i)$  is composed by all neighbors of node  $u_{i}$  and  $\pmb {W}_{ij}\in \mathbb{R}^{d_g\times d_g},\pmb {b}_{ij}\in \mathbb{R}^{d_g}$  are parameters controlling the information transfer. For  $u_{i},u_{j}$  and  $u_{i^{\prime}},u_{j^{\prime}}$  whose  $|i - j| = |i^{\prime} - j^{\prime}|$ , we use the same  $\pmb{W}$  and  $\pmb{b}$ . In other words, we can first create a sequence of matrices  $\{\pmb {W}_1,\pmb {W}_2,\dots \} \in \mathbb{R}^{d_g\times d_g}$  and vectors  $\{\pmb {b}_1,\pmb {b}_2,\dots \} \in \mathbb{R}^{d_g}$ , and then use  $|i - j|$  as the index to retrieve the corresponding  $\pmb {W}_{ij},\pmb {b}_{ij}$ . For graph  $\nu$ , we similarly compute
$$
\tilde {\boldsymbol {a}} _ {i} ^ {(t)} = \sum_ {v _ {j} \in \mathcal {N} (v _ {i})} \tilde {\boldsymbol {W}} _ {i j} \boldsymbol {v} _ {j} ^ {(t - 1)} + \tilde {\boldsymbol {b}} _ {i j} \tag {4}
$$
In the second step, we compute multiple gates. For each  $\pmb{u}_i^{(t - 1)}$  in  $\mathcal{U}$ , we compute an "update gate"  $\pmb{y}_i^{(t)}$  and a "reset gate"  $\pmb{z}_i^{(t)}$  by:
$$
\begin{array}{l} \boldsymbol {y} _ {i} ^ {(t)} = \sigma \left(\boldsymbol {W} _ {y} \left[ \boldsymbol {a} _ {i} ^ {(t)}; \boldsymbol {u} _ {i} ^ {(t - 1)} \right]\right) \\ \boldsymbol {z} _ {i} ^ {(t)} = \sigma \left(\boldsymbol {W} _ {z} \left[ \boldsymbol {a} _ {i} ^ {(t)}; \boldsymbol {u} _ {i} ^ {(t - 1)} \right]\right) \tag {5} \\ \end{array}
$$
where  $W_{y}, W_{z} \in \mathbb{R}^{2d_{g} \times d_{g}}$  are parameters. Similarly, for each  $\pmb{v}_{i}^{(t-1)}$  in  $\mathcal{V}$  we compute: