Efficient Neural Machine Translation with Prior Word Alignment

1 INTRODUCTION . As neural machine translation ( NMT ) models have become the dominant approach in the machine translation task , the explicit word alignment model , which is an essential intermediary result from the training of most statistical machine translation ( SMT ) models ( Koehn et al. , 2003 ; Och & Ney , 2004 ; Ganchev et al. , 2008 ) , seems becoming increasingly obsolete . Prior research suggests that the attention mechanism of NMT systems takes over the word alignment model of SMT systems ( Bahdanau et al. , 2014 ) . However , the word alignment information extracted from the attention mechanism is far from gold alignment and even performs much worse than automatic word aligner such as FastAlign or GIZA++ . In this study , we focus on the use of prior word alignment in the NMT system to improve translation performance . With the guidance of good enough known word alignment , replacing some words in the source sentence with semantically corresponding words in the target language leads to better translation or user-desired translation , and it is also known as a tip to use the translator well . As in Figure 1 , we can see that an open translation system 1 generates a better translation closer to the target sentence when some words of the target sentence are provided in the source sentence . In order words , a user can use specific alignment , such as공개↔ released and사진↔ picture , to get a desired translation . The case in Figure 1 happens because word alignment between source and target sentences more or less holds no matter how the model acquires such alignment . Besides , not all word alignment may help and only those good enough word alignment can truly enhance the model . When the concerned language pair shares a large vocabulary , such good enough alignment may be easily obtained and then conveniently works in the proposed early substituting way . This work will right explore an effective way to figure out those ’ good ’ enough certain alignment for neural machine translation enhancement . Previous studies of similar approaches can be largely divided into two categories : constraint decoding and augmenting MT input with its corresponding target information . The former is to leverage 1https : //translate.google.com pre-specified translations to guide the decoding procedure in the modified NMT architecture , such as an additional attention layer ( Alkhouli et al. , 2018 ; Song et al. , 2020 ) and a modification to beam search ( Post & Vilar , 2018 ; Hu et al. , 2019 ) . They can be useful in certain applications , where the user wants to enforce specific translation of certain words . However , if they do not use information ( e.g. , alignment and dictionary ) extracted from a ground truth sentence pair , they can instead lead to lower translation quality due to strict enforcement of constraints ( Dinu et al. , 2019 ) . Also , the constrained decoding methods using a modified beam search cause computational overhead in translation time . The latter is to augment MT input with pre-defined dictionary entries for a specific domain and let NMT models learn , at training time , how to use the corresponding target term when provided in the source sentence ( Dinu et al. , 2019 ; Park & Zhao , 2019 ) . Although they may gain small but constant performance improvement , they require a ’ suitable ’ pre-defined dictionary for translation tasks with a set of constraints . Otherwise , the translation performance is significantly degraded because it relies only on a fixed dictionary without considering the context . This paper focuses on integrating word alignment information into NMT systems in an effective and efficient way , without building a ’ suitable ’ dictionary beforehand . To this end , we exploit the alignment learned by SMT model , insert an Alignment-Based Word Substitution ( ABWS ) model into NMT system , and the goal of the model is to learn how to replace the input word with the target one at training time . Specifically , the ABWS model ’ s input is the source sentence , and the reference of the ABWS model is the modified source sentence , in which some words are replaced with their corresponding target words according to the prior word alignment . Note that our model only requires alignment information during training , so unlike the previously proposed models , there is no need to process the test dataset . At inference time , the final hidden state of the trained ABWS model is integrated into NMT models to provide additional target information for MT input . The benefits of our model are twofold : ( 1 ) Different from constraint decoding , which add modifications to the decoding algorithm , our method does not cause computational overhead in inference time . ( 2 ) Although the augmenting MT input requires a ’ good ’ pre-defined dictionary , our proposed method does not need to construct it separately because the ABWS model can efficiently perform the predefined dictionary role in our model . Furthermore , several previous studies inject alignment signal directly into an attention head of Transformer for constraint decoding or better alignment extraction , but they do not lower or change the translation performance ( Alkhouli et al. , 2018 ; Garg et al. , 2019 ; Song et al. , 2020 ) . To summarize , we make the following contributions . ( 1 ) We propose a novel augmenting MT input method that leverages only prior word alignment without pre-defined dictionaries to improve translation performance in NMT system . Therefore , prior alignment information from the automatic word aligner can be effectively injected into the NMT system in any bilingual corpus through our method . ( 2 ) In our experiment , our model outperforms strong baseline models such as vanilla Transformer , constraint decoding , augmenting MT input method on Romanian-English , English to German , and Korean-English translation tasks . 2 RELATED WORK . Word alignment is no longer an indispensable component in training NMT models , but recently there has been a resurgence of interest in the community to study word alignment for NMT models due to its better interpretability . For example , previous works used word alignment information to interpret the end-to-end NMT system and analyze translation errors or to extract better alignments from the learned NMT models . There have been also several studies that leverage word alignment to guide NMT decoding directly , and especially Alkhouli et al . ( 2018 ) described this approach as a new downstream task of leveraging word alignment ( dictionary-guided decoding task ) . In this study , we focus on infusing SMT offered word alignment into NMT system to improve translation performance . Alkhouli et al . ( 2018 ) added a special alignment head , which conditions a lexical model on the alignment information , to the multi-head source-to-target attention module of Transformer decoder . The use of a separate alignment model adds significant computational overhead to the decoding process , requiring special handling to optimize speed . And Song et al . ( 2020 ) proposed an approach that introduces a dedicated head in the multi-head Transformer architecture to capture external supervision signals . Besides , these two studies constraint the decoding process to correct translation errors with pre-defined dictionaries and achieve significant improvement on translation tasks . Dinu et al . ( 2019 ) ; Park & Zhao ( 2019 ) also used pre-defined dictionaries for a specific domain to let NMT model learn how to use dictionary entries when provided with the input , but they do not constraint the decoding procedure and adopt a non-coercive method that augments MT input with additional information . Furthermore , Dinu et al . ( 2019 ) showed that constrained decoding hurt translation performance in their experiments . While our goal resembles that of Dinu et al . ( 2019 ) ; Park & Zhao ( 2019 ) ( converting MT input to augment it with pre-defined dictionaries for a specific domain ) , our proposed method does not need to build the pre-defined dictionaries and not limit to a specific domain , all it needs is to feed bilingual word alignment during training . After NMT has become the dominant MT approach , there have been a variety of studies that integrate external resources ( e.g. , lemmas , POS tags , named entity , and other linguistic features ) into NMT systems . Sennrich & Haddow ( 2016 ) proposed a simple but novel method that augments input embedding with its corresponding linguistic features through concatenation operation . Li et al . ( 2020 ) presented three ways to integrate the inputs ’ compressed sentence into NMT systems . Correspondingly , we propose four integration methods to infuse the target information generated from the ABWS model into NMT models . 3 MODEL . In this section , we first describe the Transformer ( Vaswani et al. , 2017 ) for machine translation . Then we propose our core model , Alignment-Based Word Substitution ( ABWS ) model that replaces part of the source sentence with the corresponding target one according to prior word alignment . Furthermore , we introduce four strategies to fuse ABWS output into NMT system : Source-side Addition ( Src-Add ) , Source-side Context Gate ( Src-Gate ) , Soruce-side Fusion ( Src-Fusion ) and Target-side Fusion ( Tgt-Fusion ) . Figure 2 shows the architecture of our proposed model with Target-side Fusion integration strategy . 3.1 TRANSFORMER . A Transformer architecture follows the encoder-decoder paradigm ( Sutskever et al. , 2014 ) and hasN stacked encoder layers and decoder layers that rely entirely on self-attention networks . A sequence of input words is first fed into a word embedding layer to get word embeddings . Then positional information is injected into the embeddings . The word embeddings are fed into the encoder layer that consists of two sub-modules , namely a self-attention module and a feed-forward module . The self-attention module first creates a query matrix Q , a key matrix K , and a matrix-vector V from each of the word embeddings and product an output matrix as follows : SelfAttn ( Q , K , V ) = Softmax ( QK dmodel ) V , ( 1 ) where dmodel denotes the dimensions of the model . To get a better meaning and context , the Transformer uses different attention heads that are computed parallelly and independently . Multi-head attention is computed from the concatenation of n attention heads output headi : MultiHead ( Q , K , V ) = Concat ( head1 , ... , headi ) W O , ( 2 ) headi = SelfAttn ( QW Q i , KW K i , V W V i ) , ( 3 ) where WO ∈ Rd×d , WQi , WKi , WVi ∈ Rd× d n are parameter matrices . The Multi-head attention network is a core one in Transformer . Each encoder layer consists of a self-attention module and a feed-forward module . To preserve auto-regressive property of translation tasks , masked multi-head attention is added to each decoder layer . Finally , a softmax layer based on the decoder ’ s last layer HNfinal produces a probability distribution over the target vocabulary : p ( yt|y1 , ... , yt−1 , x ) = Softmax ( HNfinalW F ) , ( 4 ) where W F is the learned weight matrix , x is the source sentence , and y1 , y2 , ... , yt is the target words . 3.2 ALIGNMENT-BASED WORD SUBSTITUTION MODEL . We propose Alignment-Based Word Substitution ( ABWS ) model that learns , at training time , how to align the source word with the target one while target words are provided in the source sentence . The model consists of 2 stacked base Transformer ’ s encoder layers and is jointly trained with the NMT model , and the ABWS model input and reference are as follows : • Input IL×V : Original sentence identical to the input of the NMT model , • Reference RAbwsL×V : Modified sentence that generated by replacing some source words with the target one according to the prior alignment information , where L is the length of the source sentence and V is the size of the shared vocabulary . At the inference time , the last hidden state of the trained model is incorporated into the input of the NMT model . Formally , one input sentence x = { x1 , x2 , x3 , x4 , x5 } is fed into the ABWS model . The model learned to perform suitable replacement with the reference r = { x1 , y5 , x3 , x4 , y1 } , where two source words x2 , x5 are replaced with its corresponding target word y5 , y1 according to the alignment . As our heuristics are applied to the use of alignment in the word substitution and model training , we distinguish three cases of bilingual word alignment : one-to-null ( unaligned word ) , one-to-one , and one-to-many ( multi-aligned word ) . For one-to-one alignment , we simply replace the source word with the target one in the sentence . Since unaligned source words are not replaced , this model is simply learning to copy from source word to target one . However , there are some difficulties in processing a multi-aligned word . Unlike normal single-label classification models where class labels are mutually exclusive , the model should be able to classify multi-label data . In other words , if one source word is replaced with multiple target words , this problem can not be approached with the single-label classification task like the typical sequence generation model . Furthermore , some of the target words may not be the key meaning of the corresponding source one . Therefore , we present three solutions to these issues : ( 1 ) Like the noising method of the existing pre-training language models ( Devlin et al. , 2019 ; Yang et al. , 2019 ) , the replacement process is performed with randomly sampled pairs at a certain ratio from all alignment pairs for each batch . The ratio p for sampling alignment pairs will be empirically determined . ( 2 ) It gives the model the flexibility to either learn core meaning from aligned target words or to deviate from fixed source-target word substitution pairs . The parameter of the output linear layer and the parameter of word embedding are shared , and only the parameters of the model are trained . At the inference time , the final hidden state of the model , not the output generated by the argmax operation , is directly integrated into the NMT model . This can preserve information about multi-aligned words . In preliminary experiments , we observed a drop of about 2-3 BLUE score when the output generated from the argmax is fed into the NMT model through the embedding layer . ( 3 ) Inspired by Garg et al . ( 2019 ) , we implement multilabel classification for the multi-aligned words . Formally , let OAbwsL×V be the output of the model . We minimize the Kullback-Leibler divergence between RAbwsl , v and O Abws l , v which is equivalent to optimizing the following cross-entropy loss La : La ( OAbws ) = 1 V L∑ l=1 V∑ v=1 dl , vR Abws l , v log ( O Abws l , v ) , ( 5 ) where dl , v ( duplicate alignment penalty ) is the inverse frequency of each target index for each alignment sequence , and the goal of this penalty is to exclude words that are repeated too often but have little meaning ( e.g. , the , a , and so on ) from model training . We train our model to minimize La in conjunction with the standard translation loss Lt . The overall loss L is : L = Lt + λLa ( OAbws ) , ( 6 ) where λ is a hyper-parameter , which is set to 0.1 in our experiments .

Summary: This work proposes to incorporate word alignment information as a word substitution model. Basic idea is to jointly train a separate encoder using a cross entropy loss which predicts a source input sequence with words substituted by aligned target words. The learned representation is combined with a Transformer either by simple summation, gating or joint attention mechanism. Experimental results on Romanian/English and Korean/English tasks show very marginal gains over the baseline Transformer.