Reinforcement Learning Based Graph-to-Sequence Model for Natural Question Generation

1 INTRODUCTION . Natural question generation ( QG ) has many useful applications such as improving the question answering task ( Chen et al. , 2017 ; 2019a ) by providing more training data ( Tang et al. , 2017 ; Yuan et al. , 2017 ) , generating practice exercises and assessments for educational purposes ( Heilman & Smith , 2010 ; Danon & Last , 2017 ) , and helping dialog systems to kick-start and continue a conversation with human users ( Mostafazadeh et al. , 2016 ) . While many existing works focus on QG from images ( Fan et al. , 2018 ; Li et al. , 2018 ) or knowledge bases ( Serban et al. , 2016 ; Elsahar et al. , 2018 ) , in this work , we focus on QG from text . Conventional methods ( Mostow & Chen , 2009 ; Heilman & Smith , 2010 ; Heilman , 2011 ) for QG rely on heuristic rules or hand-crafted templates , leading to the issues of low generalizability and scalability . Recent attempts have been focused on exploiting Neural Network ( NN ) based approaches that do not require manually-designed rules and are end-to-end trainable . Encouraged by the huge success of neural machine translation , these approaches formulate the QG task as a sequence-tosequence ( Seq2Seq ) learning problem . Specifically , attention-based Seq2Seq models ( Bahdanau et al. , 2014 ; Luong et al. , 2015 ) and their enhanced versions with copy ( Vinyals et al. , 2015 ; Gu et al. , 2016 ) and coverage ( Tu et al. , 2016 ) mechanisms have been widely applied and show promising results on this task ( Du et al. , 2017 ; Zhou et al. , 2017 ; Song et al. , 2018a ; Kumar et al. , 2018a ) . However , these methods typically ignore the hidden structural information associated with a word ˚Corresponding author . sequence such as the syntactic parsing tree . Failing to utilize the rich text structure information beyond the simple word sequence may limit the effectiveness of these models for QG . It has been observed that in general , cross-entropy based sequence training has several limitations like exposure bias and inconsistency between train/test measurement ( Ranzato et al. , 2015 ; Wu et al. , 2016 ) . As a result , they do not always produce the best results on discrete evaluation metrics on sequence generation tasks such as text summarization ( Paulus et al. , 2017 ) or question generation ( Song et al. , 2017 ) . To cope with these issues , some recent QG approaches ( Song et al. , 2017 ; Kumar et al. , 2018b ) directly optimize evaluation metrics using Reinforcement Learning ( RL ) ( Williams , 1992 ) . However , existing approaches usually only employ evaluation metrics like BLEU and ROUGE-L as rewards for RL training . More importantly , they fail to exploit other important metrics such as syntactic and semantic constraints for guiding high-quality text generation . Early works on neural QG did not take into account the answer information when generating a question . Recent works have started to explore various means of utilizing the answer information . When question generation is guided by the semantics of an answer , the resulting questions become more relevant and readable . Conceptually , there are three different ways to incorporate the answer information by simply marking the answer location in the passage ( Zhou et al. , 2017 ; Zhao et al. , 2018 ; Liu et al. , 2019 ) , or using complex passage-answer matching strategies ( Song et al. , 2017 ) , or separating answers from passages when applying a Seq2Seq model ( Kim et al. , 2018 ; Sun et al. , 2018 ) . However , they neglect potential semantic relations between passage words and answer words , and thus fail to explicitly model the global interactions among them in the embedding space . To address these aforementioned issues , in this paper , we present a novel reinforcement learning based generator-evaluator architecture that aims to : i ) make full use of rich hidden structure information beyond the simple word sequence ; ii ) generate syntactically and semantically valid text while maintaining the consistency of train/test measurement ; iii ) model explicitly the global interactions of semantic relationships between passage and answer at both word-level and contextual-level . In particular , to achieve the first goal , we explore two different means to either construct a syntaxbased static graph or a semantics-aware dynamic graph from the text sequence , as well as its rich hidden structure information . Then , we design a graph-to-sequence ( Graph2Seq ) model based generator that encodes the graph representation of a text passage and decodes a question sequence using a Recurrent Neural Network ( RNN ) . Our Graph2Seq model is based on a novel bidirectional gated graph neural network , which extends the gated graph neural network ( Li et al. , 2015 ) by considering both incoming and outgoing edges , and fusing them during the graph embedding learning . To achieve the second goal , we design a hybrid evaluator which is trained by optimizing a mixed objective function that combines both cross-entropy and RL loss . We use not only discrete evaluation metrics like BLEU , but also semantic metrics like word mover ’ s distance ( Kusner et al. , 2015 ) to encourage both syntactically and semantically valid text generation . To achieve the third goal , we propose a novel Deep Alignment Network ( DAN ) for effectively incorporating answer information into the passage at multiple granularity levels . Our main contributions are as follows : • We propose a novel RL-based Graph2Seq model for natural question generation . To the best of our knowledge , we are the first to introduce the Graph2Seq architecture for QG . • We explore both static and dynamic ways of constructing graph from text and are the first to systematically investigate their performance impacts on a GNN encoder . • The proposed model is end-to-end trainable , achieves new state-of-the-art scores , and outperforms existing methods by a significant margin on the standard SQuAD benchmark for QG . Our human evaluation study also corroborates that the questions generated by our model are more natural ( semantically and syntactically ) compared to other baselines . 2 AN RL-BASED GENERATOR-EVALUATOR ARCHITECTURE . In this section , we define the question generation task , and then present our RL-based Graph2Seq model for question generation . We first motivate the design , and then present the details of each component as shown in Fig . 1 . 2.1 PROBLEM FORMULATION . The goal of question generation is to generate natural language questions based on a given form of data , such as knowledge base triples or tables ( Bao et al. , 2018 ) , sentences ( Du et al. , 2017 ; Song et al. , 2018a ) , or images ( Li et al. , 2018 ) , where the generated questions need to be answerable from the input data . In this paper , we focus on QG from a given text passage , along with a target answer . We assume that a text passage is a collection of word tokens Xp “ txp1 , x p 2 , ... , x p Nu , and a target answer is also a collection of word tokens Xa “ txa1 , xa2 , ... , xaLu . The task of natural question generation is to generate the best natural language question consisting of a sequence of word tokens Ŷ “ ty1 , y2 , ... , yT u which maximizes the conditional likelihood Ŷ “ arg maxY P pY |Xp , Xaq . Here N , L , and T are the lengths of the passage , answer and question , respectively . We focus on the problem setting where we have a set of passage ( and answers ) and target questions pairs , to learn the mapping ; existing QG approaches ( Du et al. , 2017 ; Song et al. , 2018a ; Zhao et al. , 2018 ; Kim et al. , 2018 ) make a similar assumption . 2.2 DEEP ALIGNMENT NETWORK . Answer information is crucial for generating relevant and high quality questions from a passage . Unlike previous methods that neglect potential semantic relations between passage and answer words , we explicitly model the global interactions among them in the embedding space . To this end , we propose a novel Deep Alignment Network ( DAN ) component for effectively incorporating answer information into the passage with multiple granularity levels . Specifically , we perform attentionbased soft-alignment at the word-level , as well as at the contextual-level , so that multiple levels of alignments can help learn hierarchical representations . Let Xp P RFˆN and rXp P R rFpˆN denote two embeddings associated with passage text . Similarly , let Xa P RFˆL and rXa P R rFaˆL denote two embeddings associated with answer text . Conceptually , as shown in Fig . 2 , the soft-alignment mechanism consists of three steps : i ) compute the attention score βi , j for each pair of passage word x p i and answer word x a j : ii ) multiply the attention matrix β with the answer embeddings rXa to obtain the aligned answer embeddings Hp for the passage ; iii ) concatenate the resulting aligned answer embeddings Hp with the passage embeddings rXp to get the final passage embeddings rHp P Rp rFp ` rFaqˆN . Formally , we define our soft-alignment function as following : rHp “ AlignpXp , Xa , rXp , rXaq “ CATprXp ; Hpq “ CATprXp ; rXaβT q ( 1 ) where the matrix rHp is the final passage embedding , the function CAT is a simple concatenation operation , and β is a N ˆ L attention score matrix , computed by β 9 exp ´ ReLUpWXpqT ReLUpWXaq ¯ ( 2 ) where W P RdˆF is a trainable weight matrix , with d being the hidden state size and ReLU is the rectified linear unit ( Nair & Hinton , 2010 ) . After introducing the general soft-alignment mechanism , we next introduce how we do soft-alignment at both word-level and contextual-level . 2.2.1 WORD-LEVEL ALIGNMENT . In the word-level alignment stage , we first perform a soft-alignment between the passage and the answer based only on their pretrained GloVe embeddings and compute the final passage embeddings by rHp “ AlignpGp , Ga , rGp ; Bp ; Lps , Gaq , where Gp , Bp , and Lp are the corresponding GloVe embedding ( Pennington et al. , 2014 ) , BERT embedding ( Devlin et al. , 2018 ) , and linguistic feature ( i.e. , case , NER and POS ) embedding of the passage text , respectively . Then a bidirectional LSTM ( Hochreiter & Schmidhuber , 1997 ) is applied to the final passage embeddings rHp “ trhpi uNi “ 1 to obtain contextualized passage embeddings sHp P R sFˆN . On the other hand , for the answer text Xa , we simply concatenate its GloVe embedding Ga and its BERT embedding Ba to obtain its word embedding matrix Ha P Rd1ˆL . Another BiLSTM is then applied to the concatenated answer embedding sequence to obtain the contextualized answer embeddings sHa P R sFˆL . 2.2.2 CONTEXTUAL-LEVEL ALIGNMENT . In the contextual-level alignment stage , we perform another soft-alignment based on the contextualized passage and answer embeddings . Similarly , we compute the aligned answer embedding , and concatenate it with the contextualized passage embedding to obtain the final passage embedding matrix AlignprGp ; Bp ; sHps , rGa ; Ba ; sHas , sHp , sHaq . Finally , we apply another BiLSTM to the above concatenated embedding to get a sF ˆN passage embedding matrix X .

Summary: The authors propose a Graph-to-Sequence Reinforcement Learning Model for Natural Question Generation, evaluated on SQuAD benchmark in for Question Generation. An interesting aspect of the work is related to the Graph2Seq model, and the use of the Reinforcement Learning to fine-tune the model. The latter stage seems to improve the structure of the answers considerably. An interesting use of RL algorithm and apparently a good choice of reward functions.