Learning to Generate Questions by Recovering Answer-containing Sentences

1 INTRODUCTION . Machine reading comprehension ( MRC ) , which finds the answer to a given question from its accompanying paragraphs ( called context ) , is an essential task in natural language processing . With the release of high-quality human-annotated datasets for this task , such as SQuAD-v1.1 ( Rajpurkar et al. , 2016 ) , SQuAD-v2.0 ( Rajpurkar et al. , 2018 ) , and KorQuAD ( Lim et al. , 2019 ) , researchers have proposed MRC models even surpassing human performance . These datasets commonly involve finding a snippet within a context as an answer to a given question . However , these datasets require significant amount of human effort to create questions and their relevant answers from given contexts . Often the size of the annotated data is relatively small compared to that of data used in other self-supervised tasks such as language modeling , limiting the accuracy . To overcome this issue , researchers have studied models for generating synthetic questions from a given context along with annotated ( or generated ) answers on large corpora such as Wikipedia . Golub et al . ( 2017 ) suggest a two-stage network of generating question-answer pairs which first chooses answers conditioned on the paragraph and then generates a question conditioned on the chosen answer . Dong et al . ( 2019 ) showed that pre-training on unified language modeling from large corpora including Wikipedia improves the question generation capability . Similarly , Alberti et al . ( 2019 ) introduced a self-supervised pre-training technique for question generation via the nextsentence generation task . However , self-supervised pre-training techniques such as language modeling or next sentence generation are not specifically conditioned on the candidate answer and instead treat it like any other phrase , despite the candidate answer being a strong conditional restriction for the question generation task . Also , not all sentences from a paragraph may be relevant to the questions or answers , so task of their generation may not be an ideal candidate as a pre-training method for question generation tasks . Moreover , in question generation it is important to determine which part of a given context can be a suitable answer for generating questions . To address these issues , we propose a novel training method called Answer-containing Sentence Generation ( ASGen ) for a question generator . ASGen is composed of two steps : ( 1 ) dynamically predicting K answers to generate diverse questions and ( 2 ) pre-training the question generator on the answer-containing sentence generation task . We evaluate our method against existing ones in terms of the generated question quality as well as the fine-tuned MRC model accuracy after training on the data synthetically generated by our method . Experimental results demonstrate that our approach consistently improves the question generation quality of existing models such as T5 ( Raffel et al. , 2020 ) and UniLM ( Dong et al. , 2019 ) , and shows state-of-the-art results on MS MARCO ( Nguyen et al. , 2016 ) , NewsQA ( Trischler et al. , 2017 ) , as well as comparable results to the state-of-the-art on SQuAD . Additionally , we demonstrate that the synthetically generated data by our approach can boost up downstream MRC accuracy across a wide range of datasets , such as SQuAD-v1.1 , v2.0 , and KorQuAD , without any modification to the existing MRC models . Furthermore , our experiments highlight that our method shines especially when a limited amount of training data is given , in terms of both pre-training and downstream MRC data . 2 PROPOSED METHOD . This section discusses our proposed training method called Answer-containing Sentence Generation ( ASGen ) . While ASGen can be applied to any generative model , we use a simple Transformer ( Vaswani et al. , 2017 ) based generative model as our baseline , which we call BertGen . First , we will describe how the BertGen model generates synthetic questions and answers from a context . Next , we will explain the novel components of our methods and how we pre-trained the question generator in BertGen based on them . BertGen encodes given paragraphs with two networks , the answer generator and the question generator . Answer Generator . To make the contextual embeddings and to predict answer spans for a given context without the question , we utilize a BERT ( Devlin et al. , 2019 ) encoder ( Fig . 1- ( 1 ) , BERT Encoder-A ) . We estimate the number of answer candidates K by applying a fully connected layer on the contextual embedding of BERT ’ s classification token “ [ CLS ] ” . Depending on the estimated number K , we select the K top candidate answer spans from the context . We use the K selected answer spans as input to the question generator . Question Generator . Next , we generate a question conditioned on each answer predicted from the answer generator . Specifically , we give as input to a BERT encoder the context and an indicator for the answer span location in the context ( Fig . 1- ( 2 ) , BERT Encoder-Q ) . Next , a Transformer decoder generates the question word-by-word based on the encoded representation of the context and the answer span . When pre-training the question generator on an answer-containing sentence generation task , we exclude the answer-containing sentence from the original context and train the model to generate the excluded sentence given the modified context and the answer span as input . Finally , we generate synthetic questions and answers from a large corpus , e.g. , all the paragraphs in Wikipedia . After generating this data , we train the MRC model on the generated data in the first phase and then fine-tune on the downstream MRC dataset ( e.g. , SQuAD ) in the second phase . In this paper , we use BERT as the default MRC model , since BERT or its variants achieve state-of-the-art performance across numerous MRC tasks . 2.1 DYNAMIC ANSWER PREDICTION . In question generation , it is important to determine which part of a given context can be a suitable answer for generating questions . To this end , we predict the number of answer K in a given context W = { wt } Tt=0 to obtain a more appropriate set of “ answer-like ” phrases , i.e. , { wenct } Tt=0 = BERT Encoder-A ( W ) , K = bfk ( wenc0 ) c , where T is the number of word tokens in the context , and position 0 reserved for classification token ‘ [ CLS ] ’ . fk represents a fully connected unit with two hidden layers that have hidden dimensions equal to H and 1 , respectively , where H is the hidden dimension of BERT Encoder-A . For training , we use the mean squared error loss between the output value of fk and ground-truth number of answers Kgt . To calculate the score si for start index i of a predicted answer span , we compute the dot product of the encoder output with a trainable vector vs. For each start index i , we calculate the span end index score ei , j for index j in a similar manner with a trainable vector ve , i.e. , si = vs ◦wenci , ei , j = ve ◦ fs ( wencj ⊕wenci ) , where fs represents a fully connected layer with hidden dimension H and ⊕ indicates the concatenation operation . For training , we use cross-entropy loss on the si , ei , j and ground truth start , end of the answer span for each token . Predicting the number of answers and the answer span are jointly trained to minimize the sum of their respective losses . During inference , we choose the K top answer spans with the highest score summation of start index score and end index score , i.e. , Aspan = { ( i , j ) | 1 ≤ i < T and i ≤ j < T } , ak = max ( { a | # { ( i , j ) | ( i , j ) ∈ Aspan and si + ei , j ≥ a } = K } ) , Aspank = { ( i , j ) | ( i , j ) ∈ A span and si + ei , j ≥ ak } . The K selected answer spans Aspank are then given to the question generator as input in the form of an indication of the answer span location in the given context . 2.2 PRE-TRAINING QUESTION GENERATOR . In order to generate questions conditioned on different answers that may arise in a context , we generate a question for each of the K answers . Alberti et al . ( 2019 ) proposed a pre-training method for this generative model using the self-supervised task of generating the next-sentence . We identify several issues with this approach . This technique is not specifically conditioned on the answer , despite the answer being a strong condition for the question generation task . Also , not all sentences from a paragraph may be relevant to the questions or answers from within that paragraph , so their generation is not an ideal candidate for pre-training question generation model . To address these issues , we modify the context to exclude the sentence containing the previously generated answer and pre-train the question generation model on the task of generating this excluded answer-containing sentence , conditioned on the answer and the modified context . Specifically , we exclude answer-containing sentence Sans while retaining the answer , modifying the original context D to Dans as Sstart = { p | sentence start index = p } ∪ { T } , Sans = { ( ps , pe , i , j ) | max ( { p≤i } ) s , min ( { p≥j } ) e } , Dans = [ D [ : ps ] ; D [ i : j ] ; D [ pe : ] ] , ( ps , pe , i , j ) ∈ S ans , where ( i , j ) ∈ Aspank . Note that we change Sans to not exclude the answer-containing sentence for fine-tuning the question generator , i.e. , Sans = { ( ps , pe , i , j ) |ps = i , pe = j } . In BertGen , we pass the previously generated answer to the generation model in the form of an additional position encoding Mans that indicates the answer location within the context , i.e. , Mans = [ m0 ∗ ps ; m1 ∗ ( j − i ) ; m0 ∗ ( T − pe ) ] , where m0 and m1 indicate trainable vectors corresponding to encoding id 0 and 1 , respectively . That is , we assign the encoding id for each word in the context as 0 and each word in the answer as 1 . A ∗B indicates the operation of stacking vector A for B many times . Next , we generate answer-containing sentence output words probability W o = { wot } T0 as Cenc = BERT Encoder-Q ( Dans , Mans ) , wgt = Transformer Decoder ( { w g i } t−1 i=0 , C enc ) , { wot } Tt=0 = { Softmax ( w g tE ) } Tt=0 , where Cenc is encoded representation of the context and E ∈ Rd×D represents a word embedding matrix with vocabulary size D shared between the BERT Encoder-Q and the Transformer decoder . Finally , we calculate the loss of the generated words using the cross-entropy loss as L = − ( T∑ t=1 D∑ i=1 yt , ilog ( wot , i ) ) /T , where y indicates a ground-truth one-hot vector of the answer-containing sentence word . Note that y is the question word in the case of fine-tuning . In this manner , we pre-train the question generation model using a task similar to the final task of conditionally generating the question from a given answer and a context .

Summary: This paper proposes a pretraining technique for question-generation models. The pretraining involves generating the sentence which contains an answer candidate from the context around it. Through several well-executed experiments the paper shows that this type of pretraining can improve performance of several existing question generation models, and the resulting synthetic questions generated help in augmenting reading comprehension datasets.