Question Generation using a Scratchpad Encoder

1 INTRODUCTION . Given the data driven nature of today ’ s question answering ( QA ) systems , the size and quality of training data play a major role in a system ’ s ability to answer questions correctly . In a knowledgebased question answering system , where the input is a natural language question and the answer generated is retrieved from the knowledge base ( KB ) , the standard approach is to parse the question into a logical form that can query the knowledge base . ( See Fig . 1 for an example . ) This intermediate logical form could be Lambda-DCS ( Liang ( 2013 ) ) , SPARQL ( Harris et al . ( 2013 ) ) , or any other form interpretable by the knowledge base ’ s query engine . One of the main challenges in training robust knowledge-based QA models is acquiring a large amount of diverse labeled data . Currently , the largest publicly available datasets for knowledgebased QA are on the order of 5-6 thousand queries 1 ( tau Yih et al . ( 2016 ) ; Berant et al . ( 2013 ) ; Su et al . ( 2016 ) ) , which is relatively small when compared with reading comprehension datasets like SQuAD ( Rajpurkar et al . ( 2016 ) ) that have on the order of 100,000 examples , or machine translation datasets containing millions to tens of millions of parallel sentences . Several complications arise when constructing labeled corpora for knowledge-based QA . Namely , since non-expert crowd workers are not familiar with logical form languages , collecting ( query , logical form ) pairs can be a difficult and slow process ( Liang et al . ( 2016 ) ; Reddy et al . ( 2014 ) ) 2 . One could bypass crowd sourcing logical forms by training a model on ( question , answer ) pairs , treating the intermediate logical as a latent variable ( Berant et al . ( 2013 ) ; Kwiatkowski et al . ( 2013 ) ; Yao & Van Durme ( 2014 ) . Unfortunately , collecting only ( query , answer ) pairs is also difficult . To create the WebQuestions dataset Berant et al . ( 2013 ) , judges were given 100,000 questions and asked to find the answers in Freebase . However , in only∼6,000 questions of these 100,000 ( 6 % ) were two judges able to arrive at the same answer . In order to collect arbitrarily large datasets , we need methods requiring less human intervention and expertise . In this paper we explore the feasibility of generating meaningful questions given a logical form ( i.e . SPARQL ) . The intuition is that it is easier to have a programmer with domain expertise generate a large set of programs , rather than train an average human judge to master SPARQL . 1The SimpleQuestions dataset ( Bordes et al . ( 2015 ) ) contains 100,000 examples , but can not be considered as a general knowledge-base QA or semantic parsing dataset because all the questions have one logical form only : “ Select Entity.Attribute from KB ” . Here the parser ’ s job is to only fill out the values of Entity and Attribute . Our goal is to be able to answer more complex questions that require logical forms beyond those that can be described as only triple selection ( See Fig 1 ) . 2WebQuestionsSP tau Yih et al . ( 2016 ) dataset used crowdsourcing to collect ( query , logical form ) pairs , but the judges were familiar with Freebase , and a special interface was created for them Recent work has even shown question generation from a logical form leads to improvements in semantic parsing ( Guo et al . ( 2018 ) ) . We adapt the sequence to sequence ( seq2seq ) framework ( Sutskever et al . ( 2014 ) ) given its success in tasks such as machine translation ( Bahdanau et al . ( 2014 ) ) and traditional semantic parsing ( Dong & Lapata ( 2016 ) ) . Despite their success , seq2seq with attention models fail to produce fluent output to the level of specificity and quality necessary for our task . Furthermore , they often fail to keep track of what has already been generated by the decoder , or copied from the input tokens in the case of Copynet ( He et al . ( 2017 ) ( see Table 7 ) often leading to erroneous repetitions . Our approach is aimed at mitigating these issues . We introduce a novel write mechanism to the seq2seq framework that significantly outperforms several baselines termed the Scratchpad Encoder . Put simply , we allow the decoder to keep notes as it decodes ; we keep the standard seq2seq encoder ( Sutskever et al . ( 2014 ) ) , but at each time step allow the decoder to attentively write to the encoder outputs . In this way , we use the encoder as a “ scratchpad ” to direct future generation . In both quantitative evaluations of the generated questions and human judgments of their fluency and adequacy , our model attains significant improvement over standard seq2seq , Copynet ( Gu et al . ( 2016 ) ) , and a coverage-enhanced approaches Tu et al . ( 2016 ) ; See et al . ( 2017 ) . Furthermore , human judges strongly prefer questions generated using the Scratchpad Encoder over those produced using Copynet and a coverage-enhanced approaches — 89.44 % and 61.36 % of the time , respectively . The contributions of this work are two-fold : 1 . We introduce Scratchpad Encoder , a novel enhancement to the seq2seq framework allowing the decoder to see what has been generated thus far , and outperforming multiple baselines on quantitative and qualitative metrics . 2 . We use Scratchpad Encoder to automatically generate questions given a corresponding SPARQL query , making it possible to generate a large high quality dataset of knowledge base ( query , logical form ) pairs . 2 MODEL BASICS/BACKGROUND . We build off of a standard seq2seq setup with ( a ) an encoder operating off the logical form , ( b ) a decoder outputting the generated question , and ( c ) a task-specific attention mechanism so that the decoder may focus on different parts of the logical form as it decodes . We describe these components below before detailing the copy mechanism and Scratchpad Encoder . Encoder Let the input sequence ( tokens of the logical form ) of length T be indexed by the subscript t , so that the sequence of word vectors input into the model is x0 ... xT = x0 .. T. We encode the input via a bidirectional 2 layer GRU ( Cho et al . ( 2014 ) ) , resulting in encoder outputs h0 .. T. Decoder Let the decoding sequence ( tokens of the question ) be indexed by the superscript i such that si is the state of the decoder at decoding timestep i . The decoder ’ s initial state s0 is set to the final output of the encoder in both directions . The decoder uses a recurrence new state = f ( 2 ) ( current state , input ) , where we use f ( l ) to refer to an l-layer GRU from now on . Attention At every decoding step i , the decoder computes an attentive read ( attniread ) over the encoder states ( h0 .. T ) ( as in Bahdanau et al . ( 2014 ) ) . This is computed by first computing a score for each encoder output ( ht ) via an MLP as follows . ( We denote the concatenation of vectors z0 , ... , zn as [ z0 ; ... ; zn ] . ) scoreit = W 1 ( W2 [ si ; ht ] T ) ( 1 ) where W1 and W2 are learned parameters . These scores ( scorei0 .. T ) are then normalized into a probability distribution and used as the weights to compute a weighted average of encoder outputs ( termed the attentive read ) , allowing the decoder to focus on different parts of the input at different timesteps i : ai0 .. T = softmax ( score i 0 .. T ) ( 2 ) attniread = T∑ t=0 ( ait ∗ ht ) ( 3 ) Update The decoder then computes its new state using the word vector ŷi−1 for the previous output yi−1 along with the attentive read ( attniread ) to obtain the post-read state ( s i post read ) : si+1 = sipost read = f ( 2 ) ( si , [ ŷi−1 ; attniread ] ) ( 4 ) Generation At every step , the decoder computes a distribution over output tokens from the postread state ( sipost read ) : yi = softmax ( Woutsipost read ) ( 5 ) where Wout is a learned parameter . Cross-Vocabulary Copying We noticed that many tokens that appear in the logical form are also present in the natural language form for each example . In fact , nearly half of the tokens in the question appear in the corresponding SPARQL of the WebQuestionSP dataset ( tau Yih et al . ( 2016 ) ) Hence , we give the network the ability to copy ( Gu et al . ( 2016 ) ) from its input by redefining the output distribution yi as a mixture between the generation distribution described previously ( Eq . 5 ) and a copy distribution ( Eq . 7 ) . yi = pigen ∗ softmax ( Woutsipost read ) + ( 1− pigen ) ∗ copy ( sipost read , h0 .. T , x0 .. T ) ( 6 ) where copy is a distribution over the vocabulary V. We only consider “ copyable ” tokens ( Vc ) that appear in both the logical form input vocabulary ( Vi ) and question output vocabulary ( Vo ) ( Vc = Vi ∩Vo ) . We define copy as : copy ( sipost read , h0 .. T , x0 .. T ) v = T∑ t=0 ( cit ∗ 1xt==v ) ( 7 ) The copy-attention distribution ci0 .. T is computed in a similar fashion to the attentive-read distribution ( ai0 .. T ) computed earlier ( Eq . 3 ) , except we use the post-read decoder state ( s i post read ) so the network can take into account the attentive read ( attniread ) as well as the previously generated token ( ŷi−1 ) when computing copy scoreit . copy scoreit = W 1 ( W2 [ sipost read ; ht ] T ) ) ( 8 ) ci0 .. T = softmax ( copy score i 0 .. T ) ( 9 ) attnicopy = T∑ t=0 ( cit ∗ ht ) ( 10 ) where W1 and W2 are learned parameters with the same dimensionalities as in the previous case . Finally , we need to compute “ how much ” we are generating vs. copying ( pigen ) , again using the postread decoder state ( sipost read ) and the copy read ( attn i copy ) so the network can take the attentive read ( attniread ) and previously generated token ( ŷ i−1 ) into account . pigen = σ ( W 1ReLU ( W2 [ sipost read ; attn i copy ] T ) ) ( 11 ) 3 KEEPING NOTES WITH A SCRATCHPAD ENCODER . Although modern seq2seq models ( including common extensions like Copynet tend to do well on multiple tasks and have led to promising improvements across the board ( Bahdanau et al . ( 2014 ) ; Sutskever et al . ( 2014 ) ; Dong & Lapata ( 2016 ) ) , they often have issues with over and undergeneration , particularly with regards to repetition or copying ( See et al . ( 2017 ) ; Tu et al . ( 2016 ) , as well as integrating new information ( See Eric & Manning ( 2017 ) ) . We propose a method that allows the network to update its encoding of the input at every step of decoding . Intuitively , we add one simple step to the decoder : treat the encoder states as a scratchpad , writing to it as if it were an external memory . Up until now , the decoder ’ s workflow at every step i is as follows : 1 . Reads attentively from the encoder outputs ( attniread ) ( See Eq . 3 ) 2 . Updates its state ( sipost read ) ( See Eq . 4 ) 3 . Outputs a distribution ( yi ) over the output vocabulary ( See Eq . 6 ) With the Scratchpad Encoder we add a fourth step : 4 . Write an update ( ui ) to the encoder states ( h0 .. T ) in an attentive fashion ( αi0 .. T ) using the post-read decoder state ( sipost read ) , treating the encoder states ( h0 .. T ) as if they were cells in an external memory : hi+1t = α i th i t + ( 1− αit ) ui ( 12 ) αit = σ ( W 1 ( W2 [ sipost read ; attn i copy ; h i t ] T ) ) ( 13 ) ui = Tanh ( W3ReLU ( W4 [ sipost read ; attn i copy ] T ) ) ( 14 ) Tanh is used to ensure that hi+1t remains in the range [ −1 , 1 ] , since hit ∈ [ −1 , 1 ] as h0t is the output of a GRU . αit can be understood as an update gate for the representation of the input sequence h0 .. T , signifying how much to overwrite a cell versus keeping past information . While decoding , it is advantageous for the network to keep track of which tokens have been generated and which locations have been attended . By allowing the decoder to write to the encoder we can easily track this information . By keeping the information outside of the decoder GRU we also preserve capacity in the decoder for other subtasks like smoothing . The Scratchpad Encoder is independent of the copy mechanism that we present here , meaning it can be an addition to any seq2seq with attention framework . To use the Scratchpad Encoder without the copy mechanism replace attnicopy with attn i read in the equations above .

Summary: The paper studies the problem of question generation from sparql queries. The motivation is to generate more training data for knowledge base question answering systems to be trained on. However, this task is an instance of natural language generation: given a meaning representation (quite often a database record), generate the natural language text correspoding to it. And previous work on this topic has proposed very similar ideas to the scratchpad proposed here in order to keep track of what the neural decoder has already generated, here are two of them: