Learning to Generate Questions by Recovering Answer-containing Sentences

1 INTRODUCTION . Machine reading comprehension ( MRC ) , which finds the answer to a given question from its accompanying paragraphs ( called context ) , is an essential task in natural language processing . With the release of high-quality human-annotated datasets for this task , such as SQuAD-v1.1 ( Rajpurkar et al. , 2016 ) , SQuAD-v2.0 ( Rajpurkar et al. , 2018 ) , and KorQuAD ( Lim et al. , 2019 ) , researchers have proposed MRC models even surpassing human performance . These datasets commonly involve finding a snippet within a context as an answer to a given question . However , these datasets require significant amount of human effort to create questions and their relevant answers from given contexts . Often the size of the annotated data is relatively small compared to that of data used in other self-supervised tasks such as language modeling , limiting the accuracy . To overcome this issue , researchers have studied models for generating synthetic questions from a given context along with annotated ( or generated ) answers on large corpora such as Wikipedia . Golub et al . ( 2017 ) suggest a two-stage network of generating question-answer pairs which first chooses answers conditioned on the paragraph and then generates a question conditioned on the chosen answer . Dong et al . ( 2019 ) showed that pre-training on unified language modeling from large corpora including Wikipedia improves the question generation capability . Similarly , Alberti et al . ( 2019 ) introduced a self-supervised pre-training technique for question generation via the nextsentence generation task . However , self-supervised pre-training techniques such as language modeling or next sentence generation are not specifically conditioned on the candidate answer and instead treat it like any other phrase , despite the candidate answer being a strong conditional restriction for the question generation task . Also , not all sentences from a paragraph may be relevant to the questions or answers , so task of their generation may not be an ideal candidate as a pre-training method for question generation tasks . Moreover , in question generation it is important to determine which part of a given context can be a suitable answer for generating questions . To address these issues , we propose a novel training method called Answer-containing Sentence Generation ( ASGen ) for a question generator . ASGen is composed of two steps : ( 1 ) dynamically predicting K answers to generate diverse questions and ( 2 ) pre-training the question generator on the answer-containing sentence generation task . We evaluate our method against existing ones in terms of the generated question quality as well as the fine-tuned MRC model accuracy after training on the data synthetically generated by our method . Experimental results demonstrate that our approach consistently improves the question generation quality of existing models such as T5 ( Raffel et al. , 2020 ) and UniLM ( Dong et al. , 2019 ) , and shows state-of-the-art results on MS MARCO ( Nguyen et al. , 2016 ) , NewsQA ( Trischler et al. , 2017 ) , as well as comparable results to the state-of-the-art on SQuAD . Additionally , we demonstrate that the synthetically generated data by our approach can boost up downstream MRC accuracy across a wide range of datasets , such as SQuAD-v1.1 , v2.0 , and KorQuAD , without any modification to the existing MRC models . Furthermore , our experiments highlight that our method shines especially when a limited amount of training data is given , in terms of both pre-training and downstream MRC data . 2 PROPOSED METHOD . This section discusses our proposed training method called Answer-containing Sentence Generation ( ASGen ) . While ASGen can be applied to any generative model , we use a simple Transformer ( Vaswani et al. , 2017 ) based generative model as our baseline , which we call BertGen . First , we will describe how the BertGen model generates synthetic questions and answers from a context . Next , we will explain the novel components of our methods and how we pre-trained the question generator in BertGen based on them . BertGen encodes given paragraphs with two networks , the answer generator and the question generator . Answer Generator . To make the contextual embeddings and to predict answer spans for a given context without the question , we utilize a BERT ( Devlin et al. , 2019 ) encoder ( Fig . 1- ( 1 ) , BERT Encoder-A ) . We estimate the number of answer candidates K by applying a fully connected layer on the contextual embedding of BERT ’ s classification token “ [ CLS ] ” . Depending on the estimated number K , we select the K top candidate answer spans from the context . We use the K selected answer spans as input to the question generator . Question Generator . Next , we generate a question conditioned on each answer predicted from the answer generator . Specifically , we give as input to a BERT encoder the context and an indicator for the answer span location in the context ( Fig . 1- ( 2 ) , BERT Encoder-Q ) . Next , a Transformer decoder generates the question word-by-word based on the encoded representation of the context and the answer span . When pre-training the question generator on an answer-containing sentence generation task , we exclude the answer-containing sentence from the original context and train the model to generate the excluded sentence given the modified context and the answer span as input . Finally , we generate synthetic questions and answers from a large corpus , e.g. , all the paragraphs in Wikipedia . After generating this data , we train the MRC model on the generated data in the first phase and then fine-tune on the downstream MRC dataset ( e.g. , SQuAD ) in the second phase . In this paper , we use BERT as the default MRC model , since BERT or its variants achieve state-of-the-art performance across numerous MRC tasks . 2.1 DYNAMIC ANSWER PREDICTION . In question generation , it is important to determine which part of a given context can be a suitable answer for generating questions . To this end , we predict the number of answer K in a given context W = { wt } Tt=0 to obtain a more appropriate set of “ answer-like ” phrases , i.e. , { wenct } Tt=0 = BERT Encoder-A ( W ) , K = bfk ( wenc0 ) c , where T is the number of word tokens in the context , and position 0 reserved for classification token ‘ [ CLS ] ’ . fk represents a fully connected unit with two hidden layers that have hidden dimensions equal to H and 1 , respectively , where H is the hidden dimension of BERT Encoder-A . For training , we use the mean squared error loss between the output value of fk and ground-truth number of answers Kgt . To calculate the score si for start index i of a predicted answer span , we compute the dot product of the encoder output with a trainable vector vs. For each start index i , we calculate the span end index score ei , j for index j in a similar manner with a trainable vector ve , i.e. , si = vs ◦wenci , ei , j = ve ◦ fs ( wencj ⊕wenci ) , where fs represents a fully connected layer with hidden dimension H and ⊕ indicates the concatenation operation . For training , we use cross-entropy loss on the si , ei , j and ground truth start , end of the answer span for each token . Predicting the number of answers and the answer span are jointly trained to minimize the sum of their respective losses . During inference , we choose the K top answer spans with the highest score summation of start index score and end index score , i.e. , Aspan = { ( i , j ) | 1 ≤ i < T and i ≤ j < T } , ak = max ( { a | # { ( i , j ) | ( i , j ) ∈ Aspan and si + ei , j ≥ a } = K } ) , Aspank = { ( i , j ) | ( i , j ) ∈ A span and si + ei , j ≥ ak } . The K selected answer spans Aspank are then given to the question generator as input in the form of an indication of the answer span location in the given context . 2.2 PRE-TRAINING QUESTION GENERATOR . In order to generate questions conditioned on different answers that may arise in a context , we generate a question for each of the K answers . Alberti et al . ( 2019 ) proposed a pre-training method for this generative model using the self-supervised task of generating the next-sentence . We identify several issues with this approach . This technique is not specifically conditioned on the answer , despite the answer being a strong condition for the question generation task . Also , not all sentences from a paragraph may be relevant to the questions or answers from within that paragraph , so their generation is not an ideal candidate for pre-training question generation model . To address these issues , we modify the context to exclude the sentence containing the previously generated answer and pre-train the question generation model on the task of generating this excluded answer-containing sentence , conditioned on the answer and the modified context . Specifically , we exclude answer-containing sentence Sans while retaining the answer , modifying the original context D to Dans as Sstart = { p | sentence start index = p } ∪ { T } , Sans = { ( ps , pe , i , j ) | max ( { p≤i } ) s , min ( { p≥j } ) e } , Dans = [ D [ : ps ] ; D [ i : j ] ; D [ pe : ] ] , ( ps , pe , i , j ) ∈ S ans , where ( i , j ) ∈ Aspank . Note that we change Sans to not exclude the answer-containing sentence for fine-tuning the question generator , i.e. , Sans = { ( ps , pe , i , j ) |ps = i , pe = j } . In BertGen , we pass the previously generated answer to the generation model in the form of an additional position encoding Mans that indicates the answer location within the context , i.e. , Mans = [ m0 ∗ ps ; m1 ∗ ( j − i ) ; m0 ∗ ( T − pe ) ] , where m0 and m1 indicate trainable vectors corresponding to encoding id 0 and 1 , respectively . That is , we assign the encoding id for each word in the context as 0 and each word in the answer as 1 . A ∗B indicates the operation of stacking vector A for B many times . Next , we generate answer-containing sentence output words probability W o = { wot } T0 as Cenc = BERT Encoder-Q ( Dans , Mans ) , wgt = Transformer Decoder ( { w g i } t−1 i=0 , C enc ) , { wot } Tt=0 = { Softmax ( w g tE ) } Tt=0 , where Cenc is encoded representation of the context and E ∈ Rd×D represents a word embedding matrix with vocabulary size D shared between the BERT Encoder-Q and the Transformer decoder . Finally , we calculate the loss of the generated words using the cross-entropy loss as L = − ( T∑ t=1 D∑ i=1 yt , ilog ( wot , i ) ) /T , where y indicates a ground-truth one-hot vector of the answer-containing sentence word . Note that y is the question word in the case of fine-tuning . In this manner , we pre-train the question generation model using a task similar to the final task of conditionally generating the question from a given answer and a context .

Summary: This paper presents a model for unsupervised pre-training for the task of question generation. The model first predicts the number of answer present in a given paragraph and then selects the top-K answer spans from the paragraph. After selecting the answer spans, the model, then tries to generate the answer containing sentence by inputting the paragraphs less the answer-containing sentence and also the answer span. The key idea is that this unsupervised pre-training strategy is close to the actual task of generating question given the context and the answer. This is also the key differentiator between this work and other existing pre-training strategies for question generation (e.g. Alberti et al 2019).