GraphSeq2Seq: Graph-Sequence-to-Sequence for Neural Machine Translation

Sequence-to-Sequence ( Seq2Seq ) neural models have become popular for text generation problems , e.g . neural machine translation ( NMT ) ( Bahdanau et al. , 2014 ; Britz et al. , 2017 ) , text summarization ( Nallapati et al. , 2017 ; Wang & Ling , 2016 ) , and image captioning ( Venugopalan et al. , 2015 ; Liu et al. , 2017 ) . Though sequential modeling has been shown to be effective , the dependency graph among words contains additional semantic information , and thus can be utilized for sentence modeling . In this paper , we propose a Graph-Sequence-to-Sequence ( GraphSeq2Seq ) model to fuse the dependency graph among words into the traditional Seq2Seq framework . For each sample , the sub-graph of each word is encoded to a graph representation , which is then utilized to sequential encoding . At last , a sequence decoder is leveraged for output generation . Since above model fuses different features by contacting them together to encode , we also propose a variant of our model that regards the graph representations as additional annotations in attention mechanism ( Bahdanau et al. , 2014 ) by separately encoding different features . Experiments on several translation benchmarks show that our models can outperform existing state-of-the-art methods , demonstrating the effectiveness of the combination of Graph2Seq and Seq2Seq . 1 INTRODUCTION . Neural machine translation ( NMT ) is a hot topic in Natural Language Processing . Most of NMT models belong to the encoder-decoder framework ( Sutskever et al. , 2014 ; Cho et al. , 2014 ) which encodes the source language to a representation vector and then decodes the vector to the target language . When both the input and output are sequential words , the models based on this framework are also called Sequence-to-Sequence ( Seq2Seq ) models ( Sutskever et al. , 2014 ; Bahdanau et al. , 2014 ) . More importantly , the dependency graph among words is also critical for sentence modeling in the machine translation , since it contains additional semantic information ( Bastings et al. , 2017 ) . Even the sequential sentence may serialize closely-related elements far away , the dependency graph can involve all of the elements closely , which is helpful for sentence modeling . However , most of existing Seq2Seq models do not well exploit this dependency graph information . How to effectively combine the dependency graph and the sequential information to improve the translation quality , therefore , becomes a challenging problem . To address this problem , we propose a Graph-Sequence-to-Sequence ( GraphSeq2Seq ) model by fusing the dependency graph among words into the Seq2Seq framework . For each sentence , we first utilize a dependency parser to obtain the dependency graph among words . Each sentence has a graph and each word has a sub-graph . Then the sub-graph is embedded to a representation for the word , and it is passed to a bidirectional sequence encoder . Finally , a decoder with attention mechanism generates the outputs . Since above model fuses different features by contacting them together to encode , we propose a variant of our model that tries another strategy by separately encoding different features . We evaluate GraphSeq2Seq model and its variant model on several machine translation benchmarks . Experiment results show our models outperforms existing stateof-the-art methods , demonstrating the effectiveness of the combination of Graph2Seq and Seq2Seq . The main contributions of this paper are as follows . • We propose a GraphSeq2Seq model by combining Graph2Seq model and Seq2Seq model . The dependency graph information and sequential information are helpful for sentence modeling to generate high-quality translations . • Our GraphSeq2Seq model passes the embedding of the sub-graph of a word to a bidirec- tional sequence encoder . Besides , a variant of GraphSeq2Seq model is also deployed which leverages features separately to calculate the context vectors in attention mechanism . • Experiment results on four translation benchmarks show the better performance of our models than state-of-the-arts , demonstrating the effectiveness of combining dependency graph information and sequence information . 2 BACKGROUND . We consider the problem that this paper is trying to address as how to combine Seq2Seq model and Graph2Seq model to improve the translation quality . Given an input sentence X = { x1 , x2 , ... , xn } and its dependency graph for each word G = { g1 , g2 , ... , gn } , our models combine X and G to generate the output sequence Y = { y1 , y2 , ... , ym } . Three sets of input-output pairs ( X , Y ) are assumed to be available for training , validating and testing . The graphG is parsed from the sentence X . The trained model is evaluated by computing the average task-specific scoreR ( Ŷ , Y ) on the test set , where Ŷ is the prediction . 2.1 SEQUENCE-TO-SEQUENCE MODELS . Seq2Seq Models ( Bahdanau et al. , 2014 ; Neubig , 2017 ) are general based on Recurrent Neural Networks ( RNN ) encoder-decoder framework ( Sutskever et al. , 2014 ; Cho et al. , 2014 ) . Seq2Seq models is widely used for the Machine Translation task , but it has been also used for a variety of other tasks , including Summarization , Conversational Modeling , and Image Captioning etc . Rush et al . ( 2015 ) utilized sequence-to-sequence encoder-decoder Long Short-Term Memory ( LSTM ) with attention to train a neural model for summarization task . Shang et al . ( 2015 ) proposed a neural network-based response generator for Short Text Conversation using the encoder-decoder framework . As long as the problem can be phrased as encoding input data in one format and decoding it into another format , Seq2Seq can be utilized to address it . Given a sequence X , an encoder reads it into hidden state vector h. Generally , a bidirectional RNN ( Schuster & Paliwal , 1997 ) is utilized . For the input sequence with ordering from x1 to xT , the forward RNN calculates a sequence of its forward hidden states { −→ h1 , · · · , −→ hT } . Meanwhile , reversing the input as the order from xT to x1 , the backward RNN calculates a sequence of its backward hidden states { ←− h1 , · · · , ←− hT } . Then we obtain the final hidden states by concatenating them as hj = { −→ hj , ←− hj } which saves the summaries of both the preceding words and the following words . Attention mechanism ( Bahdanau et al. , 2014 ) aims to find the parts of inputs that should be focused . Thus , the context vector c is calculated by a weighted sum of the final hidden states : ct = T∑ j=1 αt , jhj , ( 1 ) where the weight αt , j is computed by an alignment model . Its details can be found in Bahdanau et al . ( 2014 ) . Given the predicted preceding words { y1 , y2 , · · · , yt−1 } , context vector ct , and the RNN hidden state st , the decoder calculates a probability of the next word yt : p ( yt|X , y1 , y2 , · · · , yt−1 ) = g ( yt−1 , st , ct ) , ( 2 ) where g ( · ) is a softmax activation function and st = f ( yt−1 , st−1 , ct ) , ( 3 ) here , f is a non-linear function . It can be a logistic function , an LSTM unit , or a Gated Recurrent Unit ( GRU ) . Recently , Byte Pair Encoding ( BPE ) is utilized for word segmentation Sennrich et al . ( 2016 ) to overcome the out-of-vocabulary problem and rare in-vocabulary problem . Both of Edunov et al . ( 2018 ) and Deng et al . ( 2018 ) achieve state-of-the-art results , where they utilize BPE on both source side and target side . However , our GraphSeq2Seq can not apply BPE on the source side . Thus , we do not choose this kind of methods for comparison . 2.2 GRAPH-TO-SEQUENCE MODELS . Graph2Seq models ( Cohn et al. , 2018 ; Gildea et al. , 2018 ) are generally utilized to address the generation of graph-structured data ( such as Abstract Meaning Representation ( AMR ) ( Banarescu et al. , 2013 ) and dependency graph ( Chen et al. , 2017 ) ) to text . It is to recover a text representing the same meaning as an input graph . Flanigan et al . ( 2016 ) converted input graphs to trees by splitting reentrances , and then translated the trees into sentences with a tree-to-string transducer . Graph convolutional networks ( GCN ) ( Kipf & Welling , 2016 ) are used for semantic role labeling ( Marcheggiani & Titov , 2017 ) and neural machine translation ( Bastings et al. , 2017 ) . Graph state LSTM ( Gildea et al. , 2018 ; Song et al. , 2018 ) adopts gated operations for making updates , while GCN uses a linear transformation . A graph consists of triples ( i , j , l ) , where i and j are indices of the incoming and outgoing nodes , l is their edge label . For a sub-graph g ( n ) of the n-th node xn , Gildea et al . ( 2018 ) described incoming representation and outgoing representation as : xinn = ∑ ( i , n , l ) ∈gin ( n ) W [ xi , el ] + b , ( 4 ) xoutn = ∑ ( n , j , l ) ∈gout ( n ) W [ xj , el ] + b , ( 5 ) where xi is the node representation and el is the edge representation , [ · ] is a concat operation for them . W and b are the weight and bias to encode the representations . Then Gildea et al . ( 2018 ) adopted a graph state LSTM to encode each sub-graph as : in ( t ) = σ ( Wix in n + Ŵix out n + Uih in n + Ûih out n + bi ) , ( 6 ) on ( t ) = σ ( Wox in n + Ŵox out n + Uoh in n + Ûoh out n + bo ) , ( 7 ) fn ( t ) = σ ( Wfx in n + Ŵfx out n + Ufh in n + Ûfh out n + bf ) , ( 8 ) un ( t ) = σ ( Wux in n + Ŵux out n + Uuh in n + Ûuh out n + bu ) , ( 9 ) cn ( t ) = fn ( t ) cn ( t− 1 ) + in ( t ) un ( t ) , ( 10 ) hn ( t ) = on ( t ) tanh ( cn ( t ) ) , ( 11 ) where in ( t ) , on ( t ) and fn ( t ) are the input , output and forget gates . W , Ŵ , U , Û and b are model parameters . Besides , the incoming hidden and outgoing hidden also consider the graph structure so that they are represented by ∑ ( i , n , l ) ∈gin ( n ) hi ( t − 1 ) and ∑ ( n , j , l ) ∈gout ( n ) hj ( t − 1 ) . Then the hidden vectors are adopted in the decoder .

Summary: This paper proposes a seq2seq model which incorporates dependency parse information from the source side by embedding each word's subgraph (according to a predetermined dependency parse) using the Graph2Seq model of Song et al. (2018); the authors propose two variants for achieving an encoded source-side representation from the subgraph embeddings, involving bidirectional lstms. The authors show that the proposed approach leads to good results on three translation datasets.