Assumption Questioning: Latent Copying and Reward Exploitation in Question Generation

1 INTRODUCTION . Posing questions about a document in natural language is a crucial aspect of the effort to automatically process natural language data , enabling machines to ask clarification questions ( Saeidi et al. , 2018 ) , become more robust to queries ( Yu et al. , 2018 ) , and to act as automatic tutors ( Heilman & Smith , 2010 ) . While questions often have a unique answer , the inverse is rarely true ; there are multiple ways of phrasing a question , and there can be semantically different questions with the same answer . The ability to generate questions therefore provides a mechanism for augmenting existing question answering datasets or automatically annotating new ones , improving the resilience of question answering models . Recent approaches to question generation or answer questioning ( AQ ) have used Seq2Seq ( Sutskever et al. , 2014 ) models with attention ( Bahdanau et al. , 2014 ) and a form of copy mechanism ( Vinyals et al. , 2015 ; Gulcehre et al. , 2016 ) . Such models are trained to generate a plausible question , conditioned on an input document and answer span within that document ( Zhou et al. , 2018 ; Du et al. , 2017 ; Du & Cardie , 2018 ; Yuan et al. , 2017 ) . Many innovations in sequence generation have been led by a motivation to improve neural machine translation ( NMT ) models , where the input and output vocabularies are largely orthogonal and the correct lengths of the input and output sequences are often similar . Shi et al . ( 2016 ) showed that specific units within NMT models learn to count the sequence length during decoding , to help achieve the correct output length . In this way , NMT can be viewed as separate alignment and per-token transformation tasks . There is no particular intuition that the lengths of a context document and a question about that document should be related ; there is a correlation of only 0.009 between the lengths of contexts and related questions in the popular SQuAD question answering dataset ( Rajpurkar et al. , 2016 ) . The document and the question are also written in the same language , and so use overlapping vocabu- laries ; 88 % of the words from all questions in the training set also appear in at least one context document . AQ therefore presents significantly different challenges to NMT . Recent works on AQ have used various formulations of copy mechanisms , but have not investigated which approach should be preferred . We show that by removing inductive bias from the model and allowing the choice of generation path to become latent , we achieve substantial improvements over both a naive and more principled biased implementation . There are currently no dedicated question generation datasets , and recent work has used the contextquestion-answer triples available in SQuAD . Only a single question is available for each contextanswer pair , and models are trained using teacher forcing ( Williams & Zipser , 1989 ) . The combination of these factors exacerbates the problem of exposure bias ( Ranzato et al. , 2015 ) , whereby the model does not learn how to distribute probability mass over sequences that are valid but different to the ground truth . Recent work has investigated training the models directly on a performance based objective , either by optimising for BLEU score ( Kumar et al. , 2018a ) or other quality metrics ( Yuan et al. , 2017 ) . There is an implicit assumption that these metrics are in fact good proxies for question quality . We perform fine tuning using a range of rewards , including an adversarial objective . We show that although this leads to increases in those scores , the resulting models perform worse when evaluated by human workers , and the generated questions exploit weaknesses in the reward models . 2 RELATED WORK . Neural machine translation ( NMT ) has led to a number of advances in sequence modelling ( Sutskever et al. , 2014 ; Bahdanau et al. , 2014 ; Gulcehre et al. , 2016 ; Gehring et al. , 2017 ) , but for NMT the input and output sequences often have comparable lengths ( Shi et al. , 2016 ) . This means that the attention weights correspond closely to per-token alignments , and the task becomes closer to a context-aware per-token transformation . Question generation requires identifying the important sections of the context before reordering some phrases and constructing some new ones , and so poses somewhat different challenges . Summarisation involves taking a context document as input and generating a summary that should be considerably shorter , and is therefore more similar to the task of question generation . Cheng & Lapata ( 2016 ) propose an extractive neural summarisation framework that makes use of a Seq2Seq architecture with an attention mechanism , and Gu et al . ( 2016a ) extend this by adding a pointer network . Nallapati et al . ( 2016a ; b ) present an abstractive model , and See et al . ( 2017 ) again extend these approaches by adding a pointer network . Paulus et al . ( 2017 ) propose a framework for fine tuning a summarisation model using reinforcement learning . Early systems for generating questions were generally based around the use of some sort of templating or rule-based reordering of the context tokens ( Heilman & Smith , 2010 ; Heilman , 2011 ; Agarwal et al. , 2011 ; Ali et al. , 2010 ; Danon & Last , 2017 ; Popowich & Winne , 2013 ; Chali & Golestanirad , 2016 ; Labutov et al. , 2015 ; Mazidi & Nielsen , 2014 ) . Similar to these template based approaches , neural techniques have been used to generate questions from entities and relations in a knowledge base ( Serban et al. , 2016 ; Indurthi et al. , 2017 ) , but these require knowledge of the relations in advance and do not work from the raw textual input . AQ systems can be used to augment datasets for training QA models , and Wang et al . ( 2017 ) and Tang et al . ( 2017 ) approach the task with this in mind . They generate questions using a Seq2Seq model , but primarily focus on the resulting improvement to the QA model . Yang et al . ( 2017 ) take a similar approach , using an AQ model to facilitate semi-supervised training of a QA model on a range of different domains . The AQ task has also been approached by using only the document to generate questions , without conditioning on a specific answer . Subramanian et al . ( 2017 ) used named entity recognition to identify key phrases in the context before feeding this reduced input to a Seq2Seq model . They report an improved rating by human evaluators , but do not give any automated evaluation metrics for the generated questions . Kumar et al . ( 2018b ) use a similar two-stage approach , adding attention and a pointer network to the decoder . Kumar et al . ( 2018a ) further update this model by performing secondary policy gradient training , using BLEU and other automatic metrics as the rewards . Du et al . ( 2017 ) use a Seq2Seq based model to generate questions conditioned on context-answer pairs , and build on this work by preprocessing the context to resolve coreferences and adding a pointer network ( Du & Cardie , 2018 ) . Similarly , Zhou et al . ( 2018 ) use a part-of-speech tagger to augment the embedding vectors . Both works perform a human evaluation of their models , and show significant improvement over their baseline . Song et al . ( 2018 ) use a modified context encoder based on multi-perspective context matching ( Wang et al. , 2016 ) , similar to cross attention . Bahuleyan et al . ( 2017 ) used a variational encoder to generate multiple questions from a single context sentence . Gao et al . ( 2018 ) propose splitting the training data by the difficulty of question , and including this difficulty as part of the conditioning on the decoder . Yuan et al . ( 2017 ) describe a Seq2Seq model with attention and a pointer network , with an additional encoding layer for the answer . They also describe a method for further tuning their model on a language model and question answering reward objective using policy gradient , but unfortunately do not perform any human evaluation . 3 MODEL DESCRIPTION . 3.1 TASK DEFINITION . The training data consists of context-question-answer triples ( D , Q , A ) , that have been tokenised such that D = { d1 , d2 , . . . , d|D| } where |D| is the number of tokens in the document , and similarly for the question and answer . The task is to generate a natural language question Ŷ , conditioned on a document D and answer A , by sampling from the parameterised conditional distribution at each time step given by p ( ŷt ) = pθ ( ŷt|ŷ < t , D , A ) . For example , given the input document D = ” this paper investigates assumptions in question generation ” and A = ” question generation ” , the model should produce a question such as Ŷ = ” what is investigated in the paper ? ” . 3.2 ENCODER . The context tokens are transformed into an embedding representation dt , by looking up the relevant entry in the word embedding matrix . We initialise with vectors from GloVe ( Pennington et al. , 2014 ) if the word exists in the GloVe vocabulary ; otherwise we initialise with a random vector ( Glorot & Bengio , 2010 ) . We limit these embeddings to the top 2000 words in the SQuAD contexts and questions combined , ranked by frequency . Context words not in this vocabulary are mapped to the out-of-vocabulary token for the purpose of embedding . The embedded tokens dt are augmented with an additional binary feature , indicating whether that token comprises part of the answer or not , so that d̃t = [ dt ; I ( dt ∈ A ) ] . The sequence of augmented embeddings is passed through a bidirectional LSTM layer ( Hochreiter & Schmidhuber , 1997 ) to generate the context encodings hdt . For the model proposed by Yuan et al . ( 2017 ) with additional condition encoding , the encodings at the time steps corresponding to the answer span are concatenated with the word embeddings of these tokens , and this ( shorter ) sequence is passed through a second bidirectional LSTM layer . The condition encoding vector ha is given by the RNN output at the last time step . For the standard Seq2Seq model , the condition encoding is calculated as the mean context encoding of the answer span , ha = 1|A| ∑ t∈A hdt . 3.3 DECODER . At each time step , a weighted context vector vt is computed using an attention mechanism to calculate soft alignments with the context document , and taking a weighted sum of the context encodings . This vector vt = |D|∑ t′ αt , t′ h d t′ is used as the input to a unidirectional LSTM layer , giving the outputs ot . The initial state for the decoder RNN is calculated according to s0 = tanh ( W0r + b0 ) where r = Lha + 1n |D|∑ t hdt , and L , W0 , and b0 are learned parameters . Alignment scores αt are calculated using an attention mechanism ( Bahdanau et al. , 2014 ) . Briefly , this takes the form of a fully connected network with a single hidden layer and a softmax output layer , that takes the current context encoding and previous hidden state as inputs , and produces a distribution over context time steps αt , i = exp ( et , i ) ∑ j exp ( et , j ) as output , where et = f ( vt , st−1 ) is a fully connected neural network with tanh activation .

Summary: The paper studies question generation, which is an important problem in many real applications. The authors propose to use better caching model and more evalution methods to deal with the problem. However, the paper is poorly written and hard to follow, and the proposed model lacks of novelty. The main reasons are as below: