Year: 2020

Abstract:
Prior word alignment has been shown indeed helpful for better translation, if such prior is good enough and can be acquired in a convenient way at the same time. Traditionally, word alignment can be learned through statistical machine translation (SMT) models. In this paper, we propose a novel method that infuses prior word alignment information into neural machine translation (NMT) to provide hints or guidelines for the target sentence at running time. To this end, previous works of similar approaches should build dictionaries for specific domains, or constraint the decoding process, or both. While being effective to some extent, these methods may greatly affect decoding speed and hurt translation flexibility and efficiency. Instead, this paper introduces an enhancement learning model, which can learn how to directly replace specific source words with their target counterparts according to prior alignment information. The proposed model is then inserted into a neural MT model and augments MT input with the additional target information from the learning model in an effective and more efficient way. Our novel method achieves BLEU improvements (up to 1.1) over a strong baseline model on English-Korean, English-to-German and English-Romanian translation tasks.

heading: 1 INTRODUCTION
text: As neural machine translation (NMT) models have become the dominant approach in the machine translation task, the explicit word alignment model, which is an essential intermediary result from the training of most statistical machine translation (SMT) models (Koehn et al., 2003; Och & Ney, 2004; Ganchev et al., 2008), seems becoming increasingly obsolete. Prior research suggests that the attention mechanism of NMT systems takes over the word alignment model of SMT systems (Bahdanau et al., 2014). However, the word alignment information extracted from the attention mechanism is far from gold alignment and even performs much worse than automatic word aligner such as FastAlign or GIZA++. In this study, we focus on the use of prior word alignment in the NMT system to improve translation performance.
With the guidance of good enough known word alignment, replacing some words in the source sentence with semantically corresponding words in the target language leads to better translation or user-desired translation, and it is also known as a tip to use the translator well. As in Figure 1, we can see that an open translation system 1 generates a better translation closer to the target sentence when some words of the target sentence are provided in the source sentence. In order words, a user can use specific alignment, such as공개↔ released and사진↔ picture, to get a desired translation. The case in Figure 1 happens because word alignment between source and target sentences more or less holds no matter how the model acquires such alignment. Besides, not all word alignment may help and only those good enough word alignment can truly enhance the model. When the concerned language pair shares a large vocabulary, such good enough alignment may be easily obtained and then conveniently works in the proposed early substituting way. This work will right explore an effective way to figure out those ’good’ enough certain alignment for neural machine translation enhancement.
Previous studies of similar approaches can be largely divided into two categories: constraint decoding and augmenting MT input with its corresponding target information. The former is to leverage
1https://translate.google.com
pre-specified translations to guide the decoding procedure in the modified NMT architecture, such as an additional attention layer (Alkhouli et al., 2018; Song et al., 2020) and a modification to beam search (Post & Vilar, 2018; Hu et al., 2019). They can be useful in certain applications, where the user wants to enforce specific translation of certain words. However, if they do not use information (e.g., alignment and dictionary) extracted from a ground truth sentence pair, they can instead lead to lower translation quality due to strict enforcement of constraints (Dinu et al., 2019). Also, the constrained decoding methods using a modified beam search cause computational overhead in translation time. The latter is to augment MT input with pre-defined dictionary entries for a specific domain and let NMT models learn, at training time, how to use the corresponding target term when provided in the source sentence (Dinu et al., 2019; Park & Zhao, 2019). Although they may gain small but constant performance improvement, they require a ’suitable’ pre-defined dictionary for translation tasks with a set of constraints. Otherwise, the translation performance is significantly degraded because it relies only on a fixed dictionary without considering the context.
This paper focuses on integrating word alignment information into NMT systems in an effective and efficient way, without building a ’suitable’ dictionary beforehand. To this end, we exploit the alignment learned by SMT model, insert an Alignment-Based Word Substitution (ABWS) model into NMT system, and the goal of the model is to learn how to replace the input word with the target one at training time. Specifically, the ABWS model’s input is the source sentence, and the reference of the ABWS model is the modified source sentence, in which some words are replaced with their corresponding target words according to the prior word alignment. Note that our model only requires alignment information during training, so unlike the previously proposed models, there is no need to process the test dataset. At inference time, the final hidden state of the trained ABWS model is integrated into NMT models to provide additional target information for MT input. The benefits of our model are twofold: (1) Different from constraint decoding, which add modifications to the decoding algorithm, our method does not cause computational overhead in inference time. (2) Although the augmenting MT input requires a ’good’ pre-defined dictionary, our proposed method does not need to construct it separately because the ABWS model can efficiently perform the predefined dictionary role in our model. Furthermore, several previous studies inject alignment signal directly into an attention head of Transformer for constraint decoding or better alignment extraction, but they do not lower or change the translation performance (Alkhouli et al., 2018; Garg et al., 2019; Song et al., 2020).
To summarize, we make the following contributions. (1) We propose a novel augmenting MT input method that leverages only prior word alignment without pre-defined dictionaries to improve translation performance in NMT system. Therefore, prior alignment information from the automatic word aligner can be effectively injected into the NMT system in any bilingual corpus through our method. (2) In our experiment, our model outperforms strong baseline models such as vanilla Transformer, constraint decoding, augmenting MT input method on Romanian-English, English to German, and Korean-English translation tasks.

heading: 2 RELATED WORK
text: Word alignment is no longer an indispensable component in training NMT models, but recently there has been a resurgence of interest in the community to study word alignment for NMT models due to its better interpretability. For example, previous works used word alignment information to interpret the end-to-end NMT system and analyze translation errors or to extract better alignments from the learned NMT models. There have been also several studies that leverage word alignment to guide NMT decoding directly, and especially Alkhouli et al. (2018) described this approach as a new downstream task of leveraging word alignment (dictionary-guided decoding task). In this study, we focus on infusing SMT offered word alignment into NMT system to improve translation performance.
Alkhouli et al. (2018) added a special alignment head, which conditions a lexical model on the alignment information, to the multi-head source-to-target attention module of Transformer decoder. The use of a separate alignment model adds significant computational overhead to the decoding process, requiring special handling to optimize speed. And Song et al. (2020) proposed an approach that introduces a dedicated head in the multi-head Transformer architecture to capture external supervision signals. Besides, these two studies constraint the decoding process to correct translation errors with
pre-defined dictionaries and achieve significant improvement on translation tasks. Dinu et al. (2019); Park & Zhao (2019) also used pre-defined dictionaries for a specific domain to let NMT model learn how to use dictionary entries when provided with the input, but they do not constraint the decoding procedure and adopt a non-coercive method that augments MT input with additional information. Furthermore, Dinu et al. (2019) showed that constrained decoding hurt translation performance in their experiments. While our goal resembles that of Dinu et al. (2019); Park & Zhao (2019) (converting MT input to augment it with pre-defined dictionaries for a specific domain), our proposed method does not need to build the pre-defined dictionaries and not limit to a specific domain, all it needs is to feed bilingual word alignment during training.
After NMT has become the dominant MT approach, there have been a variety of studies that integrate external resources (e.g., lemmas, POS tags, named entity, and other linguistic features) into NMT systems. Sennrich & Haddow (2016) proposed a simple but novel method that augments input embedding with its corresponding linguistic features through concatenation operation. Li et al. (2020) presented three ways to integrate the inputs’ compressed sentence into NMT systems. Correspondingly, we propose four integration methods to infuse the target information generated from the ABWS model into NMT models.

heading: 3 MODEL
text: In this section, we first describe the Transformer (Vaswani et al., 2017) for machine translation. Then we propose our core model, Alignment-Based Word Substitution (ABWS) model that replaces part of the source sentence with the corresponding target one according to prior word alignment. Furthermore, we introduce four strategies to fuse ABWS output into NMT system: Source-side Addition (Src-Add), Source-side Context Gate (Src-Gate), Soruce-side Fusion (Src-Fusion) and Target-side Fusion (Tgt-Fusion). Figure 2 shows the architecture of our proposed model with Target-side Fusion integration strategy.

heading: 3.1 TRANSFORMER
text: A Transformer architecture follows the encoder-decoder paradigm (Sutskever et al., 2014) and hasN stacked encoder layers and decoder layers that rely entirely on self-attention networks. A sequence of input words is first fed into a word embedding layer to get word embeddings. Then positional information is injected into the embeddings. The word embeddings are fed into the encoder layer that consists of two sub-modules, namely a self-attention module and a feed-forward module. The self-attention module first creates a query matrix Q, a key matrix K, and a matrix-vector V from
each of the word embeddings and product an output matrix as follows:
SelfAttn(Q,K,V ) = Softmax( QK
dmodel )V , (1)
where dmodel denotes the dimensions of the model. To get a better meaning and context, the Transformer uses different attention heads that are computed parallelly and independently. Multi-head attention is computed from the concatenation of n attention heads output headi:
MultiHead(Q,K,V ) = Concat(head1, ..., headi)W O, (2)
headi = SelfAttn(QW Q i ,KW K i ,V W V i ), (3)
where WO ∈ Rd×d, WQi ,WKi ,WVi ∈ Rd× d n are parameter matrices. The Multi-head attention network is a core one in Transformer. Each encoder layer consists of a self-attention module and a feed-forward module. To preserve auto-regressive property of translation tasks, masked multi-head attention is added to each decoder layer. Finally, a softmax layer based on the decoder’s last layer HNfinal produces a probability distribution over the target vocabulary:
p(yt|y1, ...,yt−1,x) = Softmax(HNfinalW F ), (4)
where W F is the learned weight matrix, x is the source sentence, and y1,y2, ...,yt is the target words.

heading: 3.2 ALIGNMENT-BASED WORD SUBSTITUTION MODEL
text: We propose Alignment-Based Word Substitution (ABWS) model that learns, at training time, how to align the source word with the target one while target words are provided in the source sentence. The model consists of 2 stacked base Transformer’s encoder layers and is jointly trained with the NMT model, and the ABWS model input and reference are as follows:
• Input IL×V : Original sentence identical to the input of the NMT model,
• Reference RAbwsL×V : Modified sentence that generated by replacing some source words with the target one according to the prior alignment information,
where L is the length of the source sentence and V is the size of the shared vocabulary. At the inference time, the last hidden state of the trained model is incorporated into the input of the NMT model. Formally, one input sentence x = {x1,x2,x3,x4,x5} is fed into the ABWS model. The model learned to perform suitable replacement with the reference r = {x1,y5,x3,x4,y1}, where two source words x2,x5 are replaced with its corresponding target word y5,y1 according to the alignment.
As our heuristics are applied to the use of alignment in the word substitution and model training, we distinguish three cases of bilingual word alignment: one-to-null (unaligned word), one-to-one, and one-to-many (multi-aligned word). For one-to-one alignment, we simply replace the source word with the target one in the sentence. Since unaligned source words are not replaced, this model is simply learning to copy from source word to target one. However, there are some difficulties in processing a multi-aligned word. Unlike normal single-label classification models where class labels are mutually exclusive, the model should be able to classify multi-label data. In other words, if one source word is replaced with multiple target words, this problem cannot be approached with the single-label classification task like the typical sequence generation model. Furthermore, some of the target words may not be the key meaning of the corresponding source one. Therefore, we present three solutions to these issues: (1) Like the noising method of the existing pre-training language models (Devlin et al., 2019; Yang et al., 2019), the replacement process is performed with randomly sampled pairs at a certain ratio from all alignment pairs for each batch. The ratio p for sampling alignment pairs will be empirically determined. (2) It gives the model the flexibility to either learn core meaning from aligned target words or to deviate from fixed source-target word substitution pairs. The parameter of the output linear layer and the parameter of word embedding are shared, and only the parameters of the model are trained. At the inference time, the final hidden state of the model, not the output generated by the argmax operation, is directly integrated into the NMT model. This can preserve information about multi-aligned words. In preliminary experiments, we observed a drop of about 2-3 BLUE score when the output generated from the argmax is fed into the NMT model through the embedding layer. (3) Inspired by Garg et al. (2019), we implement multilabel classification for the multi-aligned words. Formally, let OAbwsL×V be the output of the model. We minimize the Kullback-Leibler divergence between RAbwsl,v and O Abws l,v which is equivalent to optimizing the following cross-entropy loss La:
La(OAbws) = 1
V L∑ l=1 V∑ v=1 dl,vR Abws l,v log(O Abws l,v ), (5)
where dl,v (duplicate alignment penalty) is the inverse frequency of each target index for each alignment sequence, and the goal of this penalty is to exclude words that are repeated too often but have little meaning (e.g., the, a, and so on) from model training. We train our model to minimize La in conjunction with the standard translation loss Lt. The overall loss L is:
L = Lt + λLa(OAbws), (6) where λ is a hyper-parameter, which is set to 0.1 in our experiments.

heading: 3.3 INTEGRATION STRATEGIES
text: While many different ways have been explored to augment MT input with additional information, we consider four strategies to incorporate the last hidden state of the ABWS model into NMT system: Src-Add, Src-Gate, Src-Fusion and Tgt-Fusion. Given the last hidden state of the ABWS model HAbws ∈ Rl×d and input word embeddings S ∈ Rl×d, where l is the length of source sentence , these four integration strategies are performed as follows.
Source-side Addition (Src-Add) is to add S to HAbws: SrcAdd(S,HAbws) = S +HAbws. (7)
Source-side Context Gate (Src-Gate) is to use a context gate G ∈ Rl×d for fusing S and HAbws: SrcGate(S,HAbws) = ConGate(S,HAbws) = G S + (1.−G) HAbws, (8)
G = σ(MLP ([S;HAbws])), (9)
where σ is the logistic sigmoid function, is the element-wise multiplication, and [; ] denotes the concatenation operation. Source-side Context Gate (Src-Fusion) is to introduce an additional dedicated multi-head attention layer for integrating HNAbws into the encoder layer:
SrcFusion(S,HAbws) = ConGate(H n src,H n Align), (10) HnAlign = AlignEncMultiHead(S,HAbws,HAbws), (11)
Hnsrc = EncMultiHead(S,S,S), (12)
where SrcFusion is fed to the FFN module of the encoder layer, and AlignEncMultiHead is an additional dedicated multi-head attention layer that is identical with the original one. Target-side Context Gate (Tgt-Fusion) is to introduce an additional dedicated encoder-decoder multi-head attention layer for integrating HNAbws into the decoder layer:
TgtFusion(S,HAbws) = ConGate(TAlign,TOri), (13)
TAlign = AlignEncDecMultiHead(H n tgt,H N Abws,H N Abws)), (14)
TOri = OriEncDecMultiHead(H n tgt,H N src,H N src)), (15)
HNAbws = FFN(EncMultiHead(HAbws,HAbws,HAbws)), (16)
HNsrc = FFN(EncMultiHead(S,S,S)), (17)
where TgtFusion is fed to the FFN module of the decoder layer, and AlignEncDecMultiHead is an additional dedicated encoder-decoder multi-head attention layer.

heading: 4 EXPERIMENTS
text: To verify that the proposed method is effective, we perform experiments on three language pairs: Romanian-English (EN↔RO), English to German (EN→DE) and Korean-English (KO↔EN). For all translation tasks, we use BLEU (Papineni et al., 2002) for the evaluation of translation quality. All the model training is on a single NVIDIA Tesla V100 GPU.

heading: 4.1 DATASETS
text: Training data and test data for EN↔RO translation are Europarl v8 corpus and newstest2016, respectively. For EN→DE, we use the Europarl v7 news datasets as training data, newstest2016 as validation data, and newstest2014 and newstest2015 as test data. In order to evaluate alignment quality on two automatic word aligner, we use the the gold alignments for EN↔RO 2 and EN→DE 3 For KO↔EN translation, we use news dataset that provided by AIHub 4 and split the dataset for the validation and test data. For all dataset, we first tokenize three languages (English, Romanian, German) using Moses (Koehn et al., 2007) and Korean data using KoNLPy 5 toolkit and apply BytePair-Encoding (Sennrich et al., 2016). Table 1 shows the statistics of datasets and alignment error rate (Och & Ney, 2000) (AER) for EN↔RO and EN→DE.
2http://web.eecs.umich.edu/˜mihalcea/ wpt/index.html#resources 3https://www-i6.informatik.rwth-aachen.de/goldAlignment/ 4http://www.aihub.or.kr/ 5https://konlpy.org/en/latest/

heading: 4.2 SETUP AND BASELINES
text: For Bilingual word aligner, we use the MGIZA++ 6 (Gao & Vogel, 2008), a parallel implementation of GIZA++, and FastAlign 7 toolkit with default parameters. We align the bilingual training corpora with FastAlign for all language pairs. For RO↔EN translation task, we additionally use word alignments produced by MGIZA++ to compare the different word aligners. Both FastAlign and GIZA are used with default settings and all the training corpora are in subword format. Furthermore, the pairs of sentences with an error (e.g., zero sentences) are pruned in the process of generating alignment.
In all experiments for this task, we train all models using a base Transformer configuration with an embedding size of 512, 6 encoder and decoder layers, 8 attention heads, shared source and target embeddings, the standard relu activation function and sinusoidal positional embedding. We train with a batch size of 3500 tokens and use the validation translation loss for early stopping and update parameters every 8 batches. Furthermore, We optimize the model parameters using Adam optimizer with a learning rate 7e-4 β1= 0.9 and β2 = 0.98, learning rate warm-up over the first 4000 steps. Additionally, we apply label smoothing with a factor of 0.1. We average over the last 5 checkpoints and run inference with a beam size of 5. All our experiments were performed using the Torch-based toolkit, Fairseq(-py) (Ott et al., 2019).
In our experiment, we use Transformer (Vaswani et al., 2017) as the baseline model for all language pairs. For EN→DE translation, we compare our approach to the following baselines:
• Constraint decoding 8: A vectorized lexically constrained decoding with dynamic beam allocation reported in Post & Vilar (2018); Hu et al. (2019). • Training-by-replacing: An augment MT input method directly replacing the original term with the target one according to a dictionary (Dinu et al., 2019; Park & Zhao, 2019). • Training-by-appending: An augment MT input method appending the target term to its source version according to a pre-defined dictionary (Dinu et al., 2019).
6https://github.com/moses-smt/mgiza 7https://github.com/clab/fast align 8https://github.com/pytorch/fairseq/blob/master/examples/constrained decoding
They all require a pre-defined dictionary, so we extracted the dictionary from the GIZA++ alignment for a fair comparison. In order to avoid spurious matches, we used the following pruning methods: (1) We first removed meaningless word with several POS tags9 (e.g., auxiliary, determiner, punctuation, stop words and so on). (2) To make an appropriate one-to-one matching set, we counted the occurrence frequency of each target term matched to the source one, and then adopt the occurrence frequency of the top 1 target term if it is more than 90% of the occurrence frequency of the total target term. (3) Finally, we filtered out entries occurring in the top 500 most frequent English words.

heading: 4.3 EXPERIMENTAL RESULTS
text: Table 2 shows the BLEU evaluation of our systems. For the experimental results, we made the following observation: (1) For the integration strategy, Most of them outperformed the baseline Transformer. We can see that each integration strategy has a gap in improving translation performance depending on the language pair or translation direction. For example, Tgt-Fusion shows relatively high performance on RO↔EN and KO↔EN translation, but Src-Fusion is the best on EN→DE translation. (2) For the automatic word aligner, the results showed that word alignment information of GIZA++ yields a better performance improvement than FastAlign on RO↔EN and EN→DE translation. This means that better alignment information can lead to better translation. (3) For other baseline models, which use pre-defined dictionary building with prior word alignment, the performance is degraded or maintained. This means that our model makes good use of the alignment information. (4) The parameters of our proposed model with the four integration strategies increased 6M to 16M over the baseline Transformer. For inference speed, the decoding time of our model increased by only 1.2 times over the baseline Transformer.

heading: 5 ANALYSIS
text: 5.1 EVALUATING ALIGNMENT SAMPLING RATIO p
In order to verify the impact of different ratios p for sampling alignment pairs on translation quality, we conducted the corresponding experiments on RO→EN translation task with our proposed model. When the sampling ratio p = 0.0, it means no use alignment information, and when the sampling ratio p = 1.0, it is equivalent to leveraging all word alignment information. Table 3a shows the experimental results. Hence, the alignment sampling ratio is set to 0.9 for the best performance in all our experiments.

heading: 5.2 WHICH NEURAL NETWORK IS SUITABLE AS THE ABWS MODEL?
text: We experimented with different neural networks to determine which neural network could represent the replaced word well for RO→EN translation task. As shown in Table 3b, there was little change when using Bi-LSTM as the ABWS model, and the transformer encoder improved the translation
9https://spacy.io/
performance. Therefore, we adopted the 2 stacked transformer encoder as our model, considering its size and performance.

heading: 5.3 EVALUATION OF ALIGNMENT USAGE ON LOW-RESOURCE CASES
text: We evaluated our proposed model’s performance on a low-resource case that might lead to poor word alignment and our models’ lower performance. We first sampled 100,000 sentences from the EN→DE data set, and the AER for that small data set was 34.0. Contrary to our expectations, Table 3c shows that our models achieve even more remarkable performance gains (up to 2 BLEU) on the low-resource case. This means that our model is also useful in low-resource cases.

heading: 5.4 ANALYSIS OF ALIGNMENT USAGE IN TRANSLATION
text: In this subsection, We observed how the ABWS model utilizes prior word alignment in the NMT model, which applies Tgt-Fusion integration strategy on the KO→EN translation task. Table 4 shows some translation examples that include top 3 ABWS output. We can see that the ABWS model replaces some source words with target one and the replaced target words appears in the translation output. Moreover, from the Korean word 시장 (market, mayor) in the first and second translation examples, we can see that the ABWS model distinguishes the homophone in each sentence. This means that the model’s replacement process takes context into account. Another observation is that the ABWS model learned well for multi-aligned words. For example, in the third example,인 권(human rigths) is aligned with two target words (’human’ and ’rights’), and the ABWS model catches them well.

heading: 6 CONCLUSION
text: This work presents a novel solution for the effective and efficient fusion of NMT and SMT. Especially, we explore an efficient way of exploiting prior word alignment offered by SMT models for NMT enhancement during the model training phase instead of constraint decoding in previous work which may slow down the inference. In detail, to augment NMT input, we design an extra model that learns how to replace specific source words with their corresponding target ones according to the prior word alignment. Our method helps yield significant performance improvement compared to a strong baseline on three translation tasks, which verifies the effectiveness of the proposed method.

Review summaries:
This paper proposes to integrate word alignment obtained from SMT into an NMT system. This is an exciting topic not only because it can help interpretability, but also because the same mechanism could be used e.g. for imposing a specific terminology in translations, something that was relatively easy to do with SMT but is much harder to achieve with NMT. The proposed method involves computing word-word alignment using existing SMT models (GIZA and FastAlign in the experiments) and integrating that information in the decoder of a transformer-based NMT model. Experiments on English-Romanian and English Korean show small improvement over a standard baseline.

This work proposes to incorporate word alignment information as a word substitution model. Basic idea is to jointly train a separate encoder using a cross entropy loss which predicts a source input sequence with words substituted by aligned target words. The learned representation is combined with a Transformer either by simple summation, gating or joint attention mechanism. Experimental results on Romanian/English and Korean/English tasks show very marginal gains over the baseline Transformer.

The paper presents a strategy to integrate prior word alignments into NMT models. It is not clear the motivation for this in the NMT context, especially why the prior alignments are crucial information that is necessary to be given a-priori to the Transformer. Besides this, the description of the method and the discussion of related work is given, SMT methods are briefly mentioned but the usage of the idea in previous work, also SMT literature is necessary.