Efficient Neural Machine Translation with Prior Word Alignment

1 INTRODUCTION . As neural machine translation ( NMT ) models have become the dominant approach in the machine translation task , the explicit word alignment model , which is an essential intermediary result from the training of most statistical machine translation ( SMT ) models ( Koehn et al. , 2003 ; Och & Ney , 2004 ; Ganchev et al. , 2008 ) , seems becoming increasingly obsolete . Prior research suggests that the attention mechanism of NMT systems takes over the word alignment model of SMT systems ( Bahdanau et al. , 2014 ) . However , the word alignment information extracted from the attention mechanism is far from gold alignment and even performs much worse than automatic word aligner such as FastAlign or GIZA++ . In this study , we focus on the use of prior word alignment in the NMT system to improve translation performance . With the guidance of good enough known word alignment , replacing some words in the source sentence with semantically corresponding words in the target language leads to better translation or user-desired translation , and it is also known as a tip to use the translator well . As in Figure 1 , we can see that an open translation system 1 generates a better translation closer to the target sentence when some words of the target sentence are provided in the source sentence . In order words , a user can use specific alignment , such as공개↔ released and사진↔ picture , to get a desired translation . The case in Figure 1 happens because word alignment between source and target sentences more or less holds no matter how the model acquires such alignment . Besides , not all word alignment may help and only those good enough word alignment can truly enhance the model . When the concerned language pair shares a large vocabulary , such good enough alignment may be easily obtained and then conveniently works in the proposed early substituting way . This work will right explore an effective way to figure out those ’ good ’ enough certain alignment for neural machine translation enhancement . Previous studies of similar approaches can be largely divided into two categories : constraint decoding and augmenting MT input with its corresponding target information . The former is to leverage 1https : //translate.google.com pre-specified translations to guide the decoding procedure in the modified NMT architecture , such as an additional attention layer ( Alkhouli et al. , 2018 ; Song et al. , 2020 ) and a modification to beam search ( Post & Vilar , 2018 ; Hu et al. , 2019 ) . They can be useful in certain applications , where the user wants to enforce specific translation of certain words . However , if they do not use information ( e.g. , alignment and dictionary ) extracted from a ground truth sentence pair , they can instead lead to lower translation quality due to strict enforcement of constraints ( Dinu et al. , 2019 ) . Also , the constrained decoding methods using a modified beam search cause computational overhead in translation time . The latter is to augment MT input with pre-defined dictionary entries for a specific domain and let NMT models learn , at training time , how to use the corresponding target term when provided in the source sentence ( Dinu et al. , 2019 ; Park & Zhao , 2019 ) . Although they may gain small but constant performance improvement , they require a ’ suitable ’ pre-defined dictionary for translation tasks with a set of constraints . Otherwise , the translation performance is significantly degraded because it relies only on a fixed dictionary without considering the context . This paper focuses on integrating word alignment information into NMT systems in an effective and efficient way , without building a ’ suitable ’ dictionary beforehand . To this end , we exploit the alignment learned by SMT model , insert an Alignment-Based Word Substitution ( ABWS ) model into NMT system , and the goal of the model is to learn how to replace the input word with the target one at training time . Specifically , the ABWS model ’ s input is the source sentence , and the reference of the ABWS model is the modified source sentence , in which some words are replaced with their corresponding target words according to the prior word alignment . Note that our model only requires alignment information during training , so unlike the previously proposed models , there is no need to process the test dataset . At inference time , the final hidden state of the trained ABWS model is integrated into NMT models to provide additional target information for MT input . The benefits of our model are twofold : ( 1 ) Different from constraint decoding , which add modifications to the decoding algorithm , our method does not cause computational overhead in inference time . ( 2 ) Although the augmenting MT input requires a ’ good ’ pre-defined dictionary , our proposed method does not need to construct it separately because the ABWS model can efficiently perform the predefined dictionary role in our model . Furthermore , several previous studies inject alignment signal directly into an attention head of Transformer for constraint decoding or better alignment extraction , but they do not lower or change the translation performance ( Alkhouli et al. , 2018 ; Garg et al. , 2019 ; Song et al. , 2020 ) . To summarize , we make the following contributions . ( 1 ) We propose a novel augmenting MT input method that leverages only prior word alignment without pre-defined dictionaries to improve translation performance in NMT system . Therefore , prior alignment information from the automatic word aligner can be effectively injected into the NMT system in any bilingual corpus through our method . ( 2 ) In our experiment , our model outperforms strong baseline models such as vanilla Transformer , constraint decoding , augmenting MT input method on Romanian-English , English to German , and Korean-English translation tasks . 2 RELATED WORK . Word alignment is no longer an indispensable component in training NMT models , but recently there has been a resurgence of interest in the community to study word alignment for NMT models due to its better interpretability . For example , previous works used word alignment information to interpret the end-to-end NMT system and analyze translation errors or to extract better alignments from the learned NMT models . There have been also several studies that leverage word alignment to guide NMT decoding directly , and especially Alkhouli et al . ( 2018 ) described this approach as a new downstream task of leveraging word alignment ( dictionary-guided decoding task ) . In this study , we focus on infusing SMT offered word alignment into NMT system to improve translation performance . Alkhouli et al . ( 2018 ) added a special alignment head , which conditions a lexical model on the alignment information , to the multi-head source-to-target attention module of Transformer decoder . The use of a separate alignment model adds significant computational overhead to the decoding process , requiring special handling to optimize speed . And Song et al . ( 2020 ) proposed an approach that introduces a dedicated head in the multi-head Transformer architecture to capture external supervision signals . Besides , these two studies constraint the decoding process to correct translation errors with pre-defined dictionaries and achieve significant improvement on translation tasks . Dinu et al . ( 2019 ) ; Park & Zhao ( 2019 ) also used pre-defined dictionaries for a specific domain to let NMT model learn how to use dictionary entries when provided with the input , but they do not constraint the decoding procedure and adopt a non-coercive method that augments MT input with additional information . Furthermore , Dinu et al . ( 2019 ) showed that constrained decoding hurt translation performance in their experiments . While our goal resembles that of Dinu et al . ( 2019 ) ; Park & Zhao ( 2019 ) ( converting MT input to augment it with pre-defined dictionaries for a specific domain ) , our proposed method does not need to build the pre-defined dictionaries and not limit to a specific domain , all it needs is to feed bilingual word alignment during training . After NMT has become the dominant MT approach , there have been a variety of studies that integrate external resources ( e.g. , lemmas , POS tags , named entity , and other linguistic features ) into NMT systems . Sennrich & Haddow ( 2016 ) proposed a simple but novel method that augments input embedding with its corresponding linguistic features through concatenation operation . Li et al . ( 2020 ) presented three ways to integrate the inputs ’ compressed sentence into NMT systems . Correspondingly , we propose four integration methods to infuse the target information generated from the ABWS model into NMT models . 3 MODEL . In this section , we first describe the Transformer ( Vaswani et al. , 2017 ) for machine translation . Then we propose our core model , Alignment-Based Word Substitution ( ABWS ) model that replaces part of the source sentence with the corresponding target one according to prior word alignment . Furthermore , we introduce four strategies to fuse ABWS output into NMT system : Source-side Addition ( Src-Add ) , Source-side Context Gate ( Src-Gate ) , Soruce-side Fusion ( Src-Fusion ) and Target-side Fusion ( Tgt-Fusion ) . Figure 2 shows the architecture of our proposed model with Target-side Fusion integration strategy . 3.1 TRANSFORMER . A Transformer architecture follows the encoder-decoder paradigm ( Sutskever et al. , 2014 ) and hasN stacked encoder layers and decoder layers that rely entirely on self-attention networks . A sequence of input words is first fed into a word embedding layer to get word embeddings . Then positional information is injected into the embeddings . The word embeddings are fed into the encoder layer that consists of two sub-modules , namely a self-attention module and a feed-forward module . The self-attention module first creates a query matrix Q , a key matrix K , and a matrix-vector V from each of the word embeddings and product an output matrix as follows : SelfAttn ( Q , K , V ) = Softmax ( QK dmodel ) V , ( 1 ) where dmodel denotes the dimensions of the model . To get a better meaning and context , the Transformer uses different attention heads that are computed parallelly and independently . Multi-head attention is computed from the concatenation of n attention heads output headi : MultiHead ( Q , K , V ) = Concat ( head1 , ... , headi ) W O , ( 2 ) headi = SelfAttn ( QW Q i , KW K i , V W V i ) , ( 3 ) where WO ∈ Rd×d , WQi , WKi , WVi ∈ Rd× d n are parameter matrices . The Multi-head attention network is a core one in Transformer . Each encoder layer consists of a self-attention module and a feed-forward module . To preserve auto-regressive property of translation tasks , masked multi-head attention is added to each decoder layer . Finally , a softmax layer based on the decoder ’ s last layer HNfinal produces a probability distribution over the target vocabulary : p ( yt|y1 , ... , yt−1 , x ) = Softmax ( HNfinalW F ) , ( 4 ) where W F is the learned weight matrix , x is the source sentence , and y1 , y2 , ... , yt is the target words . 3.2 ALIGNMENT-BASED WORD SUBSTITUTION MODEL . We propose Alignment-Based Word Substitution ( ABWS ) model that learns , at training time , how to align the source word with the target one while target words are provided in the source sentence . The model consists of 2 stacked base Transformer ’ s encoder layers and is jointly trained with the NMT model , and the ABWS model input and reference are as follows : • Input IL×V : Original sentence identical to the input of the NMT model , • Reference RAbwsL×V : Modified sentence that generated by replacing some source words with the target one according to the prior alignment information , where L is the length of the source sentence and V is the size of the shared vocabulary . At the inference time , the last hidden state of the trained model is incorporated into the input of the NMT model . Formally , one input sentence x = { x1 , x2 , x3 , x4 , x5 } is fed into the ABWS model . The model learned to perform suitable replacement with the reference r = { x1 , y5 , x3 , x4 , y1 } , where two source words x2 , x5 are replaced with its corresponding target word y5 , y1 according to the alignment . As our heuristics are applied to the use of alignment in the word substitution and model training , we distinguish three cases of bilingual word alignment : one-to-null ( unaligned word ) , one-to-one , and one-to-many ( multi-aligned word ) . For one-to-one alignment , we simply replace the source word with the target one in the sentence . Since unaligned source words are not replaced , this model is simply learning to copy from source word to target one . However , there are some difficulties in processing a multi-aligned word . Unlike normal single-label classification models where class labels are mutually exclusive , the model should be able to classify multi-label data . In other words , if one source word is replaced with multiple target words , this problem can not be approached with the single-label classification task like the typical sequence generation model . Furthermore , some of the target words may not be the key meaning of the corresponding source one . Therefore , we present three solutions to these issues : ( 1 ) Like the noising method of the existing pre-training language models ( Devlin et al. , 2019 ; Yang et al. , 2019 ) , the replacement process is performed with randomly sampled pairs at a certain ratio from all alignment pairs for each batch . The ratio p for sampling alignment pairs will be empirically determined . ( 2 ) It gives the model the flexibility to either learn core meaning from aligned target words or to deviate from fixed source-target word substitution pairs . The parameter of the output linear layer and the parameter of word embedding are shared , and only the parameters of the model are trained . At the inference time , the final hidden state of the model , not the output generated by the argmax operation , is directly integrated into the NMT model . This can preserve information about multi-aligned words . In preliminary experiments , we observed a drop of about 2-3 BLUE score when the output generated from the argmax is fed into the NMT model through the embedding layer . ( 3 ) Inspired by Garg et al . ( 2019 ) , we implement multilabel classification for the multi-aligned words . Formally , let OAbwsL×V be the output of the model . We minimize the Kullback-Leibler divergence between RAbwsl , v and O Abws l , v which is equivalent to optimizing the following cross-entropy loss La : La ( OAbws ) = 1 V L∑ l=1 V∑ v=1 dl , vR Abws l , v log ( O Abws l , v ) , ( 5 ) where dl , v ( duplicate alignment penalty ) is the inverse frequency of each target index for each alignment sequence , and the goal of this penalty is to exclude words that are repeated too often but have little meaning ( e.g. , the , a , and so on ) from model training . We train our model to minimize La in conjunction with the standard translation loss Lt . The overall loss L is : L = Lt + λLa ( OAbws ) , ( 6 ) where λ is a hyper-parameter , which is set to 0.1 in our experiments .

Summary: This paper proposes to integrate word alignment obtained from SMT into an NMT system. This is an exciting topic not only because it can help interpretability, but also because the same mechanism could be used e.g. for imposing a specific terminology in translations, something that was relatively easy to do with SMT but is much harder to achieve with NMT. The proposed method involves computing word-word alignment using existing SMT models (GIZA and FastAlign in the experiments) and integrating that information in the decoder of a transformer-based NMT model. Experiments on English-Romanian and English Korean show small improvement over a standard baseline.