Neural Phrase-to-Phrase Machine Translation

1 INTRODUCTION . Segmental structures ( i.e. , phrasal information ) are abstractions that have been shown to be useful in machine translation . State-of-the-art machine translation systems prior to the deep learning revolution were dominated by phrase-based models ( Koehn et al. , 2003 ; 2007 ; Lopez , 2008 ; Koehn , 2009 ) . In contrast to word-based approaches , these methods consider explicit phrase structures in both source and target sentences and model their alignments . Recent advances in sequence-to-sequence learning ( Sutskever et al. , 2014 ) and attention-based mechanisms ( Bahdanau et al. , 2014 ) have driven impressive progress on a new generation of machine translation models . They typically consist of three main components : an encoder that summarizes the source sentence into vectors , a decoder that generates the translation word by word , and a wordbased attention module that is used by the decoder to capture relevant source information when decoding . Innovations in terms of model architectures ( Vaswani et al. , 2017 ; Gehring et al. , 2017a ) and optimization algorithms ( Ba et al. , 2016 ) push these models to become the de facto state-of-the-art machine translation systems . These models operate at the word-level , since discovering phrases for both the source and target sentences and incorporating this phrasal information into a neural sequence to sequence model is not straightforward ( i.e. , it requires marginalizing over all possible phrase alignments in a neural model ) . Huang et al . ( 2018 ) show a first-step toward a neural phrase-based machine translation model by designing a model that is aware of phrase structures in the target sentence . They show that such an approach leads to better results compared to baseline sequence to sequence models on several language pairs . In this paper , we present the first neural phrase-based machine translation model ( NP2MT ) that considers phrasal information in both the source and target sentences . Our model uses segmental recurrent neural networks ( Kong et al. , 2015 ) to discover phrases on the source sentence and a novel phrase-attention model to align them to generated target phrases ( §2.1 ) . We design a dynamic programming algorithm to efficiently marginalize over all possible segments ( phrases ) in the source and target sentences in our model for training ( §2.3 ) and use a greedy algorithm or beam search for decoding ( §2.4 ) . We also show how to augment this model with a memory module consisting of phrase-to-phrase translation ( i.e. , a phrase dictionary ) in a setup when there are many unseen words ( e.g. , proper nouns , cross-domain translation ) to improve performance ( §2.5 ) . Experiments on benchmark datasets show that our model outperforms existing phrase-based machine translation models both in terms of performance and speed and is competitive with state-of-the-art word-based models ( §3 ) . 2 NP2MT Consider a source sentence x1 : S = { x1 , x2 , . . . , xS } and a target sentence y1 : T = { y1 , y2 , . . . , yT } . Our model is based on the sequence-to-sequence model ( Sutskever et al. , 2014 ) with two main components : a source sentence encoder and a target sentence decoder . Figure 1 shows an overview of NP2MT , we describe each component in details below . 2.1 ENCODER . We represent each word in x1 : S as a vector and use a bidirectional LSTM to obtain token representations r1 , . . . , rS by concatenating the forward and backward hidden states . We then obtain phrase representations using another bidirectional LSTM that takes token representations above as inputs based on Segmental RNNs ( Kong et al. , 2015 ) . Specifically , in order to compute the representation of a phrase spanning from the i-th token to the j-th token , denoted by si : j , we run a bidirectional LSTM with ri , . . . , rj as inputs . Denote the hidden states of the forward LSTM by −→ h and the hidden states of the backward LSTM by ←− h . We have si : j = concat ( −→ h j , ←− h i ) . Given the maximum segment length P ( a hyperparameter of the model ) , we compute representations for all phrases : si , j , ∀ ( i , j ) ∈ { ( i , j ) : 1 ≤ i ≤ j ≤ i+ P − 1 ≤ S } Since we use a bidirectional LSTM as the phrase encoder and all phrases are contiguous segments , we can efficiently compute these representations in O ( SP ) . 2.2 DECODER . Let z ∈ Zy1 : T be a valid of segmentation of y1 : T = { y1 , y2 , . . . , yT } . Denote each segment in z by zk and the number of segments |zk| . The generative process of the target sentence is : • For each segment k = 1 , . . . , |zk| : – Sample a word yki until we get the end of segment symbol ( i.e. , yki = $ ) or the end of sentence symbol in the last segment ( i.e. , yki = $ $ ) . Our decoder consists of two main components : • A unidirectional ( forward ) LSTM to compute a contextual representation gz < zk of previously generated segments z < k . In training time , gz < k can be obtained simply by running the LSTM on all target words from previous segments token by token . • A segment decoder that generates words in the current segment . The segment decoder is an attention-based model over all possible source phrases si , j , gz < zk , and previously generated words in the current segment yzk < t . We compute the phrase-attention score azki , j once for every segment zk , using gz < zk to query all possible source phrase representations si , j : azki , j = attn ( si , j , gz < zk ) ( 1 ) The phrase-attention score azki , j gives a soft alignment 1 between the target phrase zk and the source phrases si , j . The attention value is then computed as the sum over si , j weighted by the attention score azki , j . We use it as the initial state of the segment decoder . The probability of generating the target sentence is computed as : p ( y1 : T | x1 : S ) = ∑ z∈Zy1 : T ∏ t∈zk p ( yzkt | si , j , gz < zk , y zk < t ) 2.3 TRAINING . For a given pair of source and target sentence , the objective function that we would like to maximize in Eq . 2 is intractable due to the summation over Zy1 : T ( i.e. , all possible target segmentations ) . Here , we show a dynamic programming algorithm to compute it efficiently . Denote the probability of generating all valid segmentations ending in the n-th target words by α ( n ) . The probability of generating the entire sentence is therefore α ( T ) . We have the following recursive function : α ( n ) = ∑ m < n α ( m ) ∏ t∈z ( m , n ) p ( yz ( m , n ) t | si , j , gz < z ( m , n ) , y z ( m , n ) < t ) , where α ( 0 ) = 1 , and z ( m , n ) denotes a segment from m to n. The computational complexity of this dynamic programming algorithm is O ( TP ) , where P is the maximum ( target ) segment length . 2.4 DECODING . For inference at test time ( i.e . decoding ) , we can not use dynamic programming in §2.3 since the target sentence is unknown . We either use a greedy algorithm or beam search for decoding . Greedy algorithm For the greedy method , we follow the generative process in §2.2 and greedily pick the word yki , until we reach the end of sentence symbol ( i.e . y k i = $ $ ) . Beam search Our beam search algorithm is different from the standard left-to-right beam search algorithm used in vanilla sequence-to-sequence models since the probability for the next word is conditioned not only on the local context with in the segment , but also on all previously generated segments . The main idea of our algorithm is to keep track of both during the beam search process . We show our Algorithm in Appendix A . We maintain a set Yo to store the incomplete ( open ) sentences during the search . An open sentence is represented as a tuple consisting of three elements : previously generated segments , previously 1Enforcing a hard attention similar as in ( Raffel et al. , 2017 ) may be beneficial in the context of phrase-level attention . We leave that for future work . generated words in the current segment , and the probability of generating the open sentence so far . This tuple contains sufficient information to compute the probability distribution of the next output word . We update each element ( each open sentence ) in Yo after we generate a new word , so the open sentences in Yo will always have the same length . 2.5 DECODING WITH A MEMORY MODULE . In language modeling , augmenting a recurrent neural network with a memory module at evaluation time has been shown to improve performance ( Grave et al. , 2016 ; Krause et al. , 2017 ; Grave et al. , 2017 ) , especially for rare word predictions and in an open vocabulary setup . We explore the possibility of augmenting a neural machine translation model with a persistent memory module derived from external sources ( i.e. , a phrase-to-phrase dictionary ) to improve decoding . In this setup , we first build a phrase-to-phrase dictionary using a phrase extractor from Moses ( Koehn et al. , 2007 ) . During decoding , at each timestep , we compute attention scores forr all possible source segments azki , j . We then choose the source phrase with the highest score to be translated . Given the attended source phrase , we decide whether to use a dictionary to translate the phrase or not . We perform a memory lookup and check whether the phrase is in the dictionary . If it is , we directly translate using the dictionary ; otherwise , we proceed as in the greedy decoding case described above . When there are multiple translations for a phrase in the dictionary , we use the model to score all candidates , and then choose the best one as follows : yzk ← argmax dzk∈dzk ∏ t∈zk p ( dzkt | sm : n , gz < zk , d zk < t ) ( 2 ) We summarize our memory-augmented decoding method in Algorithm 1 and illustrate it in Figure 2 . We note that word-based models can also benefit from the augmented memory module . However , it is more straightforward to use the phrase-level information in NP2MT . The word-based models only lookup phrases with source with length 1 using attention weights . Experiments show that when compared to models not augmented with memory modules , NP2MT improves significantly when there is a larger amount of OOV ( See Appendix Table 6 ) . In addition , it is also possible to augment beam search decoding with a memory module . In this case , we perform a lookup and we add the translated phrase to Yo after the beam reaches the length of the translated phrase . We present the greedy case for simplicity . It should be straight-forward for the readers to extend the algorithm into beam search cases by combining algorithms 1 and 2 . We leave exploration of beam search and memory to future work .

Summary: This paper presents a phrase-based encoder-decoder model for machine translation. The encoder considers all possible phrases (i.e. word sequences) up to a certain length and compute phrase representations using bidirectional LSTMs from contextual word embeddings computed with another bidirectional LSTM layer. The decoder also considers possible segmentations and computes contextual representations for the previously generated segments. Each word in the current segment is generated by a Transformer model by attending to all phrases in the source sentence. The authors present a dynamic programming method for considering all possible segmentations in decoding. They also present a method for incorporating a phrase-to-phrase dictionary built by Moses into the decoding process.