ASGen: Answer-containing Sentence Generation to Pre-Train Question Generator for Scale-up Data in Question Answering

1 INTRODUCTION . Machine reading comprehension ( MRC ) , which finds an answer to a given question from given paragraphs called context , is an essential task in natural language processing . With the use of high-quality human-annotated datasets for this task , such as SQuAD-v1.1 ( Rajpurkar et al. , 2016 ) , SQuAD-v2.0 ( Rajpurkar et al. , 2018 ) , and KorQuAD ( Lim et al. , 2019 ) , researchers have proposed MRC models , often surpassing human performance on these datasets . These datasets commonly involve finding a short snippet within a paragraph as an answer to a given question . However , these datasets require a significant amount of human annotation to create pairs of a question and its relevant answer from a given context . Often the size of the annotated data is relatively small compared to that of data used in other unsupervised tasks such as language modeling . Hence , researchers often rely on the two-phase training method of transfer learning , i.e. , pre-training the model using large corpora from another domain in the first phase , followed by fine-tuning it using the main MRC dataset in the second phase . Most state-of-the-art models for MRC tasks involve such pre-training methods . Peters et al . ( 2018 ) present a bidirectional contextual word representation method called ELMo , which is pre-trained on a large corpus , and its learned contextual embedding layer has been widely adapted to many other MRC models . Devlin et al . ( 2019a ) show that pre-training with a masked language model on a large corpus and then fine-tuning on a downstream dataset results in significant performance improvements . However , pre-training on another domain task and then fine-tuning on a downstream task may suffer from performance degradation , depending on which pre-training task is used in the first phase . For example , Yang et al . ( 2019 ) show that the pre-training task of next sentence classification decreases performance on the downstream MRC tasks . To handle this problem , generating synthetic data similar to the those of a downstream task is crucial to obtain a properly pre-trained model . Recently , researchers have studied a model for generating synthetic MRC data from large corpora such as Wikipedia . This is essentially a form of transfer learning , by training a generation model and using this model to create synthetic data for training the MRC model , before fine-tuning on the downstream MRC dataset . Golub et al . ( 2017 ) suggest a two-stage synthesis network that decomposes the process of generating question-answer pairs into two steps , generating a fixed number ( K ) of answers conditioned on the paragraph , and question generation conditioned on the paragraph and the generated answer . Devlin et al . ( 2019b ) introduced a pre-training technique for the question generator of this method by pretraining on the generation of next-sentence that follows the paragraph . However , choosing a fixed number ( K ) of candidate answers from each paragraph will lead to missing candidates if K is too small , and will lead to having lower-quality candidates if K is too big . Moreover , the next sentence generation task is not conditioned on the answer , despite the answer being a strong conditional restriction for question generation task . Also , the next sentence that follows a paragraph may have little relevance to the questions or answers from within the paragraph , and hence is not the ideal candidate for pre-training question generation . To address these issues , we propose Answer-containing Sentence Generation ( ASGen ) , a novel method for a synthetic data generator with two novel processes , ( 1 ) dynamically predicting K answers to generate diverse questions and ( 2 ) pre-training the question generator on answer-containing sentence generation task . We evaluate the question generation capability of our method by comparing the BLEU score with existing methods and test our method by fine-tuning the MRC model on downstream MRC datasets after training on the generated data . Experimental results show that our approach outperforms existing generation methods , increasing the performance of the state-ofthe-art MRC models across a wide range of MRC datasets such as SQuAD-v1.1 , SQuAD-v2.0 , KorQuAD , and QUASAR-T ( Dhingra et al. , 2017 ) without any architectural modifications to the MRC model . 2 PROPOSED METHOD . This section discusses the details of our proposed ASGen method . ASGen consists of a BERT-based generative model ( BertGen ) and answer-containing sentence generation pre-training ( AS ) . First , we will describe how BertGen model generates synthetic data from Wikipedia . Next , we will explain the novel components of our methods and how we pre-trained the question generator in BertGen based on them . BertGen encodes paragraphs in Wikipedia with two separate generation networks , the answer generator and the question generator . Answer Generator . As shown in Fig . 2- ( 1 ) , we generate the number of answer candidates K for a given context without the question by applying a fully connected feed-forward layer on the contextual embedding of classification token “ [ CLS ] ” . To make the contextual embeddings and to predict answer spans , we utilize a BERT ( Devlin et al. , 2019a ) encoder ( Fig . 2-BERT Encoder-A ) . Depending on the predicted number K , we select the K top candidate answer spans from the context . As shown in Fig . 2- ( 2 ) , we use the K selected candidate answer spans as input to the question generator . Question Generator . Next , as shown in Fig . 2- ( 2 ) , we generate a question conditioned on each answer predicted from the answer generator . Specifically , we pass as input to a BERT encoder the context and an indicator for the answer span location in the context ( Fig . 2-BERT Encoder-Q ) . Next , a Transformer decoder ( Vaswani et al. , 2017 ) generates the question word-by-word based on the encoded representation of the context and the answer span . For pre-training such a question generator on an answer-containing sentence generation task , we exclude the answer-containing sentence from the original context and train the model to generate the excluded sentence given the modified context and the answer span as input . Finally , we generate questions and answers from a large corpus , e.g. , all the paragraphs in Wikipedia in this paper . After generating such data , we train the MRC model on the generated data in the first phase and then fine-tune on the downstream MRC dataset ( such as SQuAD ) in the second phase . In this paper , we use BERT as the default MRC model , since it exhibits state-of-the-art performance in many MRC datasets . 2.1 DYNAMIC ANSWER PREDICTION . The most natural method for humans to create a question-answer pair from a given context is to select the answer first and then create a corresponding question . In this situation , we conjecture that a human is more likely to choose as an answer a phrase that is “ answer-like ” , such as keyphrases , nouns , dates , names , etc . There may be several answers in the context that are likely to be selected by humans as answers , especially if the context is lengthy or if it contains multiple nouns , dates , names , etc . For example , the context “ Barack Hussein Obama II is an American attorney and politician who served as the 44th president of the United States from 2009 to 2017 ” can have as possible answers “ Barack Hussein Obama ” , “ 44th ” , “ United States ” , “ 2009 to 2017 ” , etc . As shown in Fig . 4 , to see these characteristics , we examine the distribution of the number of answers in the SQuAD dataset and hypothesize that there exists an underlying pattern in the number of answers that occur in a context . The conventional method to generate multiple answers from a context is to draw a fixed number ( K ) of answers . However , this approach can generate low-quality answers if K is too big , and it can impact the number and diversity of the generated answers if K is too small . Therefore , we predict the number of answers K in a given context W = { wt } T0 using regression as , { wenct } Tt=0 = BERT Encoder-A ( W ) t , K = bfk ( wenc0 ) c , where T is the number of word tokens in the context with position 0 reserved for classification token ‘ [ CLS ] ’ , and fk represents a fully connected unit with two hidden layers that have hidden dimensions equal to H and 1 , respectively , where H is the hidden dimension of BERT Encoder-A . To calculate the score si for start index i of a predicted answer span , we compute the dot product of the encoder output with a trainable start vector S. For each start index i , we calculate the span end index score ei , j for end index j in a similar manner with a trainable end vector E , but conditioned on i , i.e. , { si } Ti=0 = S ◦wenci { ei , j } T , Ti , j=0 = E ◦ fs ( w enc j ⊕wenci ) , where fs represents a fully connected layer with hidden dimension H and ⊕ indicates the concatenation operation . For training , we use the mean squared error loss between K and ground-truth number of answers . We also use cross-entropy loss on the si , ei , j and ground truth start/end of the answer span for each token . Predicting the number of answers and predicting the span are jointly trained to minimize the sum of their respective losses . During inference , we choose the K top answer spans with the highest score summation of start index score and end index score , i.e. , Aspan = { ( i , j ) | 1 ≤ i < T and i ≤ j < T } , ak = max ( { a | # { ( i , j ) | ( i , j ) ∈ Aspan and si + ei , j ≥ a } = K } ) , Aspank = { ( i , j ) | ( i , j ) ∈ A span and si + ei , j ≥ ak } . The K selected answer spans Aspank are then given to the question generator as input in the form of an indication of the answer span location . 2.2 PRE-TRAINING QUESTION GENERATOR . In order to generate questions conditioned on different answers that may arise in a context , we generate a question for each of the K answers . Devlin et al . ( 2019b ) previously proposed to pre-train this generation model with an unsupervised task that generates the next sentence following a given paragraph to improve generation performance . We identify several issues with this approach . The final question generation task has the form of sentence generation given an answer and a context , while the next-sentence generation has no answer component . The next-sentence generation task is not conditioned on the answer , despite the answer being a strong conditional constraint for the question generation task . Also , the next sentence that follows a paragraph may have little relevance to the questions or answers from within the paragraph , and hence is not the ideal candidate for pre-training question generation . To address these issues , we modify the context to exclude the sentence containing our previously generated answer and pre-train our generator on the task of generating this excluded answercontaining sentence , conditioned on the answer and the modified context . Specifically , we exclude answer-containing sentence Sans while leaving the answer and modify the original context D to Dans as Sstart = { p | sentence start index = p } , Sans = { ( p , q , i , j ) | max ( { p|p≤i } ) , min ( { q|q≥j } ) , ( i , j ) ∈ Aspank , p ∈ S start , q ∈ Sstart } , Dans = [ D [ : p ] ; D [ i : j ] ; D [ q : ] ] , ( p , q , i , j ) ∈ Sans . Note that we change Sans to not exclude the answer-containing sentence in the case of fine-tuning on the question generation , i.e. , Sans = { ( p , q , i , j ) |p = i , q = j } . Afterwards , we pass the previously generated answer to the sequence-to-sequence generation model as a segmentation encoding Mans that identifies the answer part within the context , i.e. , Mans = [ m0 ∗ p ; m1 ∗ ( j − i ) ; m0 ∗ ( T − q ) ] , ( p , q , i , j ) ∈ Sans , where m0 and m1 indicate trainable vectors corresponding to segmentation id 0 and 1 , respectively . Here we tag the segmentation id for each word in the context as 0 and each word in the answer as 1 . A ∗B indicates the operation of concatenating vector A for B many times . Next , we generate answer-containing sentence embedding W g = { wgt } T0 using a Transformer sequence-to-sequence model ( the encoder part is initialized with BERT ) as wgt = Transformer Decoder ( { w g i } t−1 i=0 , BERT Encoder-Q ( D ans , Mans ) ) . Finally , we calculate the loss of the generation model with cross-entropy over generated sentence words , i.e. , { wot } Tt=0 = { Softmax ( w g tE ) } Tt=0 , L = − ( T∑ t=1 D∑ i=1 yt , ilog ( wot , i ) + ( 1− yt , i ) log ( ( 1−wot , i ) ) ) /T , where y indicates a ground-truth one-hot vector of the answer-containing sentence word ( the question word in the case of fine-tuning ) , D is the vocabulary size , and E ∈ Rd×D represents a word embedding matrix shared between the BERT Encoder-Q and the Transformer decoder . In this manner , we pre-train the question generation model using a task similar to the final task of conditionally generating the question from a given answer and a context .

Summary: This paper proposes a pretraining technique for question generation, where an answer candidate is chosen beforehand, and the objective is to predict the answer containing sentence given a paragraph excluding this sentence and the target answer candidate. The intuition of this method is that question generation requires to generate a sentence which contains the information about the answer while being conditioned on the given paragraph. In particular, the paper compares its approach to Devlin’s presentation (https://nlp.stanford.edu/seminar/details/jdevlin.pdf according to the references; is it not a published work?) which uses next sentence generation for pretraining, that is less related to the downstream question generation task.