Efficient Neural Machine Translation with Prior Word Alignment
Authors: Jeonghyeok Park
hai zhao
Venue:  (2021)
Keywords: Word Alignment
Neural Machine Translation

Abstract: Prior word alignment has been shown indeed helpful for a better translation if such prior is good enough and can be acquired in a convenient way at the same time. Traditionally, word alignment can be learned through statistical machine translation (SMT) models. In this paper, we propose a novel method that infuses prior word alignment information into neural machine translation (NMT) to provide hints or guidelines for the target sentence at running time. To this end, previous works of similar approaches should build dictionaries for specific domains, or constraint the decoding process, or both. While being effective to some extent, these methods may greatly affect decoding speed and hurt translation flexibility and efficiency.  Instead, this paper introduces an enhancement learning model, which can learn how to directly replace specific source words with their target counterparts according to prior alignment information. The proposed model is then inserted into a neural MT model and augments MT input with the additional target information from the learning model in an effective and more efficient way. Our novel method achieves BLEU improvements (up to 1.1) over a strong baseline model on English-Korean and English-Romanian translation tasks.

Under review as a conference paper at ICLR 2021
EFFICIENT NEURAL MACHINE TRANSLATION WITH PRIOR WORD ALIGNMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Prior word alignment has been shown indeed helpful for better translation, if such prior good enough and can be acquired in convenient way at the same time. Traditionally, word alignment can be learned through statistical machine transla- tion (SMT) models. In this paper, we propose a novel method that infuses prior word alignment information into neural machine translation (NMT) provide hints or guidelines for the target sentence at running time. To this end, previous works of similar approaches should build dictionaries for specific domains, con- straint the decoding process, or both. While being effective to some extent, these methods may greatly affect decoding speed and hurt translation flexibility and ef- ficiency. Instead, this paper introduces an enhancement learning model, which can learn how to directly replace specific source words with their target counterparts according to prior alignment information. The proposed model is then inserted into a neural MT model and augments MT input with the additional target infor- mation from the learning model in an effective and more efficient way. Our novel method achieves BLEU improvements (up 1.1) over strong baseline model on Inglish-Korean Engrrsn and English- Romshnnn translation tasks.
INTRODUCTION
As neural machine translation (NMT) models have become the dominant approach in the machine translation task, the explicit word alignment model, which is an essential intermediary result from the training of most statistical machine translation (SMT) models (Koehn al. 20033 Och & Ney 2004 Ganchev et al. 2008), seems becoming increasingly obsolete. Prior research suggests that the attention mechanism of NMT systems takes over the word alignment model of SMT systems (Bah- danau et al. 2014). However, the word alignment information extracted from the attention mecha- nism is far from gold alignment and even performs much worse than automatic word aligner such as FastAlign or GIZA++. In this study, we focus the use of prior word alignment in the NMT system to improve translation performance.
With the guidance of good enough known word alignment, replacing some words in the source sentence with semantically corresponding words in the target language leads to better translation or user-desired translation, and it is also known tip see that an open translation system generates a better translation closer to the target sentence when some words of the target sentence are provided in the source sentence. In order words, a user can use specific alignment, such as - 7 ++ released and 사ト ++ picture, to get a desired translation. The case in Figure [egure[[apper happens because word alignment between source and target sentences more or less holds no matter how the model acquires such alignment. Besides, not all word alignment may help and only those good enough word alignment can truly enhance the model. When the concerned language pair shares a large vocabulary, such good enough alignment may be easily obtained and then conveniently works in the proposed early substituting way. This work will right explore an effective way to figure out those 'good' enough certain alignment for neural machine translation enhancement.
the translator well. As in Figure [ we
Previous studies of similar approaches can be largely divided into two categories: constraint decod- ing and augmenting MT input with its corresponding target information. The former is to leverage
hhaps//ttanslate.gogge.cco

---

Under review as a conference paper at ICLR 2021
pre-specified translations to guide the decoding procedure in the modified NMT architecture, such as an additional attention layer (Alkhouli et al. 2018 Song et al. 2020) and a modification to beam search (Post & Vilar 2018 Hu et al.] 2019)) They can be useful in certain applications, where the user wants to enforce specific translation of certain words. However, if they do not use informa- tion (e.g., alignment and dictionary) extracted from ground truth sentence pair, they can instead lead to lower translation quality due to strict enforcement of constraints (Dinu et al., 2019). Also, the constrained decoding methods using modified beam search cause computational overhead in translation time. The latter is to augment MT input with pre-defined dictionary entries for a specific domain and let NMT models learn at training time, how use the corresponding target term when provided in the source sentence (Dinu et al. 2019 Park & Zhao 2019). Although they may gain small but constant performance improvement, they require a 'suitable" pre-defined dictionary for translation tasks with a set of constraints. Otherwise, the translation performance is significantly degraded because it relies only on a fixed dictionary without considering the context.
This paper focuses on integrating word alignment information into NMT systems in effective and efficient way, without building a 'suitable" dictionary beforehand. To this end, we exploit the alignment learned by SMT model, insert Alignment- Baseed Word Substitution (ABWS) model into NMT system, and the goal of the model is to learn how to replace the input word with the target one training time. Specifically, the ABWS model's input is the source sentence, and the reference of the ABWS model is the modified source sentence, in which some words are replaced with their corresponding target words according to the prior word alignment. Note that our model only requires alignment information during training, so unlike the previously proposed models, there no need to process the test dataset. At inference time, the final hidden state of the trained ABWS model is integrated into NMT models to provide additional target information for MT input. The benefits of our model are twofold: (1) Different from constraint decoding, which add modifications to the decoding algorithm, our method does not cause computational overhead in inference time. (2) Although the augmenting MT inpul requires ''ood' pre-defined dictionary, our proposed method does not need to construct it separately because the ABWS model can efficiently perform the pre- defined dictionary role in our model. Furthermore, several previous studies inject alignment signal directly into attention head of Transformer for constraint decoding or better alignment extraction, but they do not lower or change the translation performance (Alkhouli et al. 2018 Garg et al. 2019 (ong et al. 2020).
To summarize, we make the following contributions. (1) We propose a novel augmenting MT input method that leverages only prior word alignment without pre-defined dictionaries improve transla- tion performance in NMT system. Therefore, prior alignment information from the automatic word aligner can be effectively injected into the NMT system in any bilingual corpus through our method. (2) In our experiment, our model outperforms strong baseline models such as vanilla Transformer, constraint decoding, augmenting MT input method on Romanian Englishh, English to German, and Korean-Englist translation tasks.
2 RELATED WORK
Word alignment is no longer an indispensable component in training NMT models, but recently there has been resurgence of interest in the community study word alignment for NMT models due to its better interpretability. For example, previous works used word alignment information to interpret the end-to-end NMT system and analyze translation errors or to extract better alignments from the learned NMT models. There have been also several studies that leverage word alignment to guide NMT decoding directly, and especially Alkhouli et al. (2018) described this approach as new downstream task of leveraging word alignment (dictionary-guiin study, we focus on infusing SMT offered word alignment into NMT system to improve translation performance.
decoding task). In this
Alkhouli et al. (2018) added a special alignment head, which conditions a lexical model on the align- ment information, to the multi-head source-to-target attention module of Transformer decoder. The use of a separate alignment model adds significant computational overhead to the decoding process, requiring special handling to optimize speed. And Song et al. (2020) proposed an approach that in- troduces a dedicated head in the multi-head Transformer architecture to capture external supervision signals. Besides, these two studies constraint the decoding process to correct translation errors with
2

---

Under review as a conference paper at ICLR 2021
Source:
--
&1 t
3 /
x
Target:
crystal
released
picture
of
the
black-aut white
filter
with
an
atmosphere
Systems
Sentences
Source
=
oo x
=
$ m J
14 2 8' ct.
Translator Out
crystal unveiled* an atmospheric black white filter photo*.
Source with Targtt
7 E 71
21
+ uf
=| picture & released
Translator Out
crystal released picture atmospheric black and white filter.
Figure 1:
Korean-to-Ennlish translation example of inputting a source sentence, that replaces the input word with its corresponding target one, in an open translation system. The colorful line between the source and target the top represents 'good' alignment, and the gray dotted line is trivial one. The underline denotes the target word corresponding to the source one, and the asterisk* the word to correct using the 'good' alignment.
pre-defined dictionaries and achieve significant improvement translation tasks. Dinu et al. (019); Park & Zhao (2019) also used pre-defined dictionaries for a specific domain let NMT model learn how to use dictionary entries when provided with the input, but they do not constraint the decoding procedure and adopt a non-coercive method that augments MT input with additional information. Furthermore, Dinu et al. (2019) showed that constrained decoding hurt translation performance in their experiments. While our goal resembles that of Dinu et al. (2019): Park Zhao (2019) (con- verting MT input to augment with pre-defined dictionaries for specific domain), proposed method does not need to build the pre-defined dictionaries and not limit to a specific domain, all needs is to feed bilingual word alignment during training.
After NMT has become the dominant MT approach, there have been a variety of studies that inte- grate external resources (e.g., lemmas, POS tags, named entity, and other linguistic features) into NMT systems. (Sennrich & Haddow (2016) proposed a simple but novel method that augments in- put embedding with its corresponding linguístic features through concatenation operation. [Li et al. (2020) presented three ways to integrate the inputs' compressed sentence into NMT systems. Cor- respondingly. we propose four integration methods infuse the target information generated from the ABWS model into NMT models.
3 MODEL
In this section, we first describe the Transformer Vaswann et al. 2017) for machine translation. Then we propose our core model, Alignment- -ased Word Substitution (ABWS) model that replaces part of the source sentence with the corresponding target according to prior word alignment. Further- more, we introduce four strategies to fuse ABWS output into NMT system: Source-side Addition (Sre-Add), Source-side Context Gate (Sre-Gate), Soruce-side Fusion (Sre-Fusion) and Target-side Fusion (Tgt-Fusion) Figure shows the architecture of our proposed model with Target-side Fu- sion integration strategy.
3.1 TRANSFORMER
Transformer architecture follows the encoder-decoder paradigm (Sutskever et al. 2014) and has N stacked encoder layers and decoder layers that rely entirely self-attention networks. A sequence of input words is first fed into a word embedding layer to get word embeddings. Then positional information is injected into the embeddings. The word embeddings that consists of two sub-mode modules, namely a self-attention module and a feed-forward module. The self-attention module first creates query matrix Q, key matrix K, and matrix-vector V from
fed into the encoder layer
3

---

Under review as a conference paper at ICLR 2021
Word Alignment
Output Probabilities
Source
Source
Referenees
Softmax
Targer
Sampling
Linear
XN
Add & Norm
Food
Forward
Reference Replacing
Hilwo
Add & Norm
gettax
Ouiput Probabilities
Inputs including References-
Tatgs
Original Multi-Head Attention
Aligmment Multi-Head Attention
Add & Norm
Softmax
Feed
Forward
Higs
Linear
Hyger
Add & Norm
Add & Norm
Masked Multi-Head Attention
Alignment-Based Based Word Substitution Model
Multi-Head Atlention
Positional Encoding
Positional Encoding
Outpur Eibhaddng
Input Embedding
Outputs (shifted right)
Inputs
Figure 2: The architecture of our proposed model with Target-side Fusion integration strategy.
each of the word embeddings and product output matrix follows:
Self Altm(l (Q, K, V) Softmar( QK V
dmodel
where dmmode denotes the dimensions of the model. To get better meaning and context, the Trans- former uses different attention heads that are computed parallelly and independently. Multi-head attention is computed from the concatenation of n attention heads output head,;
MultiHcad(Q.. K, V) = Concat(headd,.... head,)
(2)
head, Sel Attm KW. VW.).
(3)
where W E Radxd W W.K W.V RR are parameter matrices. The Multi-head attention network is one in Transformer. Each encoder layer consists of self-attention module and foed-forward module. To preserve auto-regressive property of translation tasks, masked multi-head attention is added each decoder layer. Finally, softmax layer based the decoder's ' last layer final produces a probability distribution over the target vocabulary:
H
p(y ly ....., 1)) = Softmar(( H finat W²).
where F is the learned weight matrix, is the source sentence, and yı y2. .... Ur is the target words.
3.2 ALIGNMENT-BASHE WORD SUBSTITUTION MODEL
We propose Alignment- Based Word Substitution (ABWS) model that learns, training time, how to align the source word with the target one while target words are provided the source sentence. The model consists of 2 stacked base Transformer's 's encoder layers and is jointly trained with the NMT model, and the ABWS model input and reference are as follows:
Input TxxV: Original sentence identical to the input of the NMT model,
4

---

Under review as a conference paper at ICLR 2021
Reference R Modified sentence that generated by replacing some source words with the target one according to the prior alignment information,
where L is the length of the source sentence and V is the size of the shared vocabulary. At the inference time, the last hidden state of the trained model is incorporated into the imput of the NMT model. Formally, one input sentence model learned to perform suitable replacement with the reference r = { Us: 23. L4. y ). where two source words Ø2, 25 are replaced with its corresponding target word ys, yı according to the alignment.
{21, 22, 23 24, 255 } is fed into the ABWS model. The
As our heuristies are applied the use of alignment in the word substitution and model training. we distinguish three cases of bilingual word alignment: one-to-null (unaligned word), one-to-one, and one-to-many (multi-aligned word). For one-to-one alignment, we simply replace the source word with the target one in the sentence. Since unaligned source words are not replaced, this model simply learning to copy from source word to target one. However, there are some difficulties in processing multi-aligned word. Unlike normal single-label classification models where class labels are mutually exclusive, the model should be able to classify multi-label data. In other words, if one source word is replaced with multiple target words, this problem cannot be approached with the single-label classification task like the typical sequence generation model. Furthermore, some of the target words may not be the key meaning of the corresponding source one. Therefore, present three solutions to these issues: (1) Like the noising method of the existing pre-training language models (Devlin et al. 2019 Yang al. 2019), the replacement process is performed with randomly sampled pairs at a certain ratio from all alignment pairs for each batch. The ratio p for sampling alignment pairs will be empirically determined. (2) It gives the model the flexibility either learn core meaning from aligned target words or to deviate from fixed source-target word substitution pairs. The parameter of the output linear layer and the parameter of word embedding are shared, and only the parameters of the model are trained. At the inference time, the final hidden state of the model, not the output generated by the a argmax operation, is directly integrated into the NMT model. This can preserve information about multi-aligned words. In preliminary experiments, we observed a drop of about 2-3 BLUE score when the output generated from the a grgax is fed into the NMT model through the embedding layer. (3) Inspired by Garg et al. (2019), we implement multi- label classification for the multi-aligned words. Formally, let 0, be the output of the model. We minimize the Kullback-Lein divergence between RAbbu and Obbus which is equivalent to optimizing the following cross-entropy loss La:
L
V
La (O
ΣΣ 1=1 v=1
di Re RAAbwa log log(O (b
(5)
V
where di.e (duplicate alignment penalty) is the inverse frequency of each target index for each align- ment sequence, and the goal of this penalty is to exclude words that are repeated too often but have little meaning (e.g., the, and so on) from model training. We train our model to minimize L. in conjunction with the standard translation loss L.. The overall loss C is:
L L. + (O )
(6)
where \ is hyper-parameter, which set to 0.1 in our experiments.
3.3 INTEGRATION STRATEGIES
While many different ways have been explored to augment MT input with additional information, we consider four strategics to incorporate the last hidden state of the ABWS model into NMT system: Sre-Add, Sre-Gate, Sre-Fusion and Tgt-Fusion. Given the last hidden state of the ABWS model H Abws E Rixd and input word embeddings S E Ri where I is the length of source sentence, these four integration strategies are performed as follows.
Source-side Addition (Sre-Add) is to add to Abbws:
SreAdd(S, H. Abwss) S + H Abwss
Source-side Context Gate (Sre-Gate) is to use context gate G E gixd for fusing S and A
SreGate(S, H Abwws) = ConGate(S S, Abwws) = G o S G) H Hbwss,
(8)
G = a (MLP(([S: H Abwss)) 1));
5

---

Under review as a conference paper at ICLR 2021
Table
Corpora statistics and AER [%) w.r. FastAlign and GIZA++.
Corpus
Train
Valid
Test
BPE
FastAlign
GIZA+
Romanian--Ennlish
399K
1999
1999
50K
35.4/35.5
30.9/31.8
English-+ 1.84M
2999
3003/2196
40K
30.9
20.7
Korean English
767K
2000
2000
32K/32K
where a is the logistic sigmoid function, O is the element-wise multiplication, and [:] denotes the concatenation operation.
Source-side Context Gate (Sre-Fusion) to introduce an additional dedicated multi-heal attention layer for integrating H
Abwa into the encoder layer:
SreF PreFusion(S. (S, S.H. Abwu)) ConGate H Srct H Httig)::
(10)
H Align Aliig ulti Heai H Abwss H Abbss)s
(11)
H
eadi
(12)
where SrcFusion fed to the FFN module of the encoder layer, and EnngnnEnnuultccaa an additional dedicated multi-head attention layer that is identical with the original Target-side Context Gate (Tgt-Fusion) is to introduce an additional dedicated encoder-decoder multi-head attention layer for integrating H N
Abws into the decoder layer:
T Tt usion H Abws) = ConGate(TAigga Tori),
(13)
T Tllign = MlignEnnDeeMultiHH H iggh H HAoss H Aboss),
(14)
Tor OriiEncDecMulliin ulti Higt: H sree H&))),
(15)
H
Abws FFF DEFN(Encai EncM ulti ead Abwsy H Abwsy H Hbwws))
(16)
Abws
H
src = FFN (EncM eod(H
N
(17)
where T Fusion is fed to the FFN module of the decoder layer, and AlignEncDecMulttHHee an additional dedicated encoder-decoder multi-head attention layer.
4 EXPERIMENTS
To verify that the proposed method is effective, perform experiments three language pairs: Romanian- English (EN+RO), English to German (EN-DEE and Korean-English (KO+=N), For all translation tasks, BLEU (Papineni al. 2002) for the evaluation of translation quality. All the model training is on single NVIDIA Tesla V100 GPU.
4.1 DATASETS
Training data and test data for EN++RO translation Europarl v8 corpus and newstest2016. spectively. For EN-DE, we use the Europarl v7 news datasets as training data, newstest2016 as validation data, and newstest2014 and newstest2015 as test data. In order evaluate alignment qual- ity on two automatic word aligner, we use the the gold alignments for EN+RO 2 and EN->D For KO++NN translation,
use news dataset that provided by AlHub and split the dataset for the validation and test data. For all dataset, we first tokenize three languages (English, Romanian, German) using Moses (Koehn et al. 2007) and Korean data using KoNL toolkit and apply Byte- Pair-Encoding (Sennrich al. 2016). 6).. Table [ shows the statistics of datasets and alignment error rate (Och Ney] 2000) (AER) for EN+RO and EN-
'http//web.eecs. umiicc.ddu/iihaaceaa wpt/index. html@resuuress 'htps://wwwii6.inn
"http///www.ahhu..or.t kr/
https://konlyy.org/e/aaaaee

---

Under review as a conference paper at ICLR 2021
Table 2: Experimental results on three translation task. "+" represent significantly better systems
than the corresponding baseline Transformer at p-value 0.05. Time(s) denotes the average translation time (second) per sentence.
Systems
RO-EN
EN-<RO
KO-EN
EN-
Transformer
30.84
32.20
39.50
35.63
Fast
GIZA
Fast
GIZA
Fast
GIZA
Fast
GIZA
Sre-Add
31.26
31.33
32.55
32.60
40.37+
40.34
36.01
36.04
Sre-Gate
31.17
31.29
32.21
32.25
40.46
40.42+
36.11
36.01
Sre-Fusion
31.22
31.33
31.90
31.88
40.35+
40.44+
35.91
35.03
Tgt-Fusion
31.41
31.62
32.44
32.46
40.63+
40.50+
36.19+
36.21
(a) RO++EN (WMT16) and KO++NN (AlHub)
Systems
WMT14
Time(s)
WMT15
Time(s)
#Params
Transtormer
26.87
0.336
28.98
0.304
64.4M
Constr. dec.
26.47
0.495
28.59
0.472
64.4M
Train-by-rep.
26.78
0.340
29.06
0.314
64.4M
Train-by-app
26.88
0.343
28.93
0.319
64.4M
Fast
GTZA
Time(s)
Fast
GIZA
Time(s)
#Params
Sre-Add
26.85
26.99
0.352
29.11
29.23
0.314
70.7M
Sre-Gate
27.12
27.28
0.341
29.10
29.33
0.320
71.2M
Sre-Fusion
27.36
27.55
0.377
29.75+
29.88+
0.349
80.IM
Tgt-Fusion
26.70
27.23
0.402
28.54
29.15
0.362
80.
(b) EN +DE
(WMT
14 and WMT 15)
4.2 SETUP AND BASELINES
For Bilingual word aligner, we use the MGIZA++ (Gao Vogel 2008), a parallel implementation of GIZA++, and PastAlign ttolkitt with default parameters. We align the bilingual training corpora with FastAlign for all language pairs. For RO+EN translation task, we additionally use word align- ments produced by MGIZA+ compare the different word aligners. Both FastAlign and GIZA used with default settings and all the training corpora are in subword format. Furthermore, the pairs of sentences with an error (e.g., zero sentences) are pruned in the process of generating alignment.
In all experiments for this task, we train all models using a base Transformer configuration with an embedding size of 512, 6 encoder and decoder layers, attention heads, shared source and target embeddings, the standard relu activation function and sinusoidal positional embedding. We train with batch size of 3500 tokens use the validation translation loss for early stopping and update parameters every 8 batches. Furthermore, We optimize the model parameters using Adam optimizer with a learning rate 7e-4 31= 0.9 and 82 0.98, learning rate warm-up over the first 4000 steps. Additionally, apply label smoothing with factor of 0.1. We average over the last 5 checkpoints and run inference with a beam size of All our experiments were performed using the Torch-based toolkit, Fairseq(-py) (Ot et al. 2019).
In our experiment, we use Transformer Vaswani et al. 2017 as the baseline model for all language pairs. For EN-DE translation, compare approach to the following baselines:
Constraint decoding 3 vectorized lexically constrained decoding with dynamic beam
allocation reported in Post & Vilar (2018); 3): Hu al. (2019).
Training-by-replann
An augment MT input method directly replacing the original term
with the target one according
dictionary (Dinu et al. 2019 Park & Zhao 2019). Training-by-ae by-a-peni-ngpg: An augment MT imput method appending the target term to its source version according a pre-defined dictionary (Dinu al. 2019).
http:///hhti...oo//////s//////// hhtpp:///github... http:///iiuuu..o//c/aaaafa/t/ align
oat.e decoding

---

Under review as a conference paper at ICLR 2021
They all require a pre-defined dictionary, for a fair comparison. In order to avoid spurious matches, we used the following pruning methods: (1) We first removed meaningless word with several POS (e.g. auxiliary, determiner, punctu- ation, stop words and so on). (2) To make an appropriate one-to-one matching set, we counted the occurrence frequency of each target term matched to the source one, and then adopt the occurrence frequency of the top I target term if it is more than 90% of the occurrence frequency of the total target term. (3) Finally, we filtered entries occurring in the top 500 most frequent English words.
extracted the dictionary from the GIZA++ alignment
4.3 EXPERIMENTAL RESULTS
Table shows the BLEU evaluation of our systems. For the experimental results, made the following observation: (1) For the integration strategy, Most of them outperformed the baseline Transformer. We can see that each integration strategy has gap in improving translation perfor- mance depending on the language pair or translation direction. For example. Tgt-Fusion shows relatively high performance on RO+EN and KO++EN translation, but Sre-Fusion is the best on EN-DE translation. (2) For the automatic word aligner, the results showed that word alignment information of GIZA++ yields a better performance improvement than FastAlign on RO++EN and EN-DE translation. This means that better alignment information can lead to better translation. (3) For other baseline models, which use pre-defined dictionary building with prior word alignment, the performance is degraded or maintained. This means that our model makes good use of the align- ment information. (4) The parameters of our proposed model with the four integration strategies increased 6M to 16M over the baseline Transformer. For inference speed, the decoding time of our model increased by only 1.2 times over the baseline Transformer.
Table 3: Performances on translation tasks, where our proposed model uses Tgt-Fusion integration strategy and word alignment from GIZA++. The (a) and (b) are translation performances (BLEU score) with different alignment sampling ratios and different ABWS models, respectively. In (a) and (b), the asterisk denotes our proposed ABWS model setting. The (c) to evaluate our model on low-resource cases.
SR
BLEU
SR
BLEU
Systems
BLEU(A)
Systems
BLEU
0.0
30.94
w/o ABWS
30.84(0)
14
15
0.1
30.94
0.6
31.43
Bi-LSTM
30.80(-0.04)
Trans.
11.37
13.14
0.2
30.87
0.7
31.50
Trans Enc
31.10(0.26)
S. Add
11.55
13.55
0.3
31.02
0.8
31.53
Trans Enc
31.62(0.78)
S. Gate
11.94
13.71
0.4
31.05
0.9
31.62
Trans Enc
31.43(0.59)
S. Fus.
13.25
15.03
0.5
31.25
1.0
31.26
Trans Enc
31.49(0.65)
T. Fus.
12.90
14.78
(a) Alignment SR (RO-ENN
(b) ABWS Models (RO-EN)
(c) Now-Resource (EN->DE)
ANALYSIS
5.1 EVALUATING ALIGNMENT SAMPLING RATIO P
In order to verify the impact of different ratios p for sampling alignment pairs translation quality, we conducted the corresponding experiments on RO-EN translation task with our proposed model. When the sampling ratio p 0.0, it means no use alignment information, and when the sampling ratio = 1.0, it is equivalent to leveraging all word alignment information. Table 3a shows the experimental results. Hence, the alignment sampling ratio is set to 0.9 for the best performance in all our experiments.
5.2 WHICH NEURAL NETWORK IS SUITABLE THE ABWS MODEL?
We experimented with different neural networks to determine which neural network could represent the replaced word well for translation task. As shown Table 3b] there was little change when using Bi-LSTM as the ABWS model, and the transformer encoder improved the translation
htpps///pacc.io
8

---

Under review as a conference paper at ICLR 2021
Table 4: Translation examples in which the ABWS model's output augments the MT input. The underline denotes source word and its corresponding target one in the ABWS output.
Source
4 a of + X real market of A] H] regulated x
2 9 o) $ x t 2
ABWS top
top 2
estate 2 in non- @ @ チ areas has attention E receiving is
top 3
# 1 markets at un @ @ regulatory area are drawing its drawing are &quot;
Output
u regulated areas are drawing attention in the real estate market.
Target
quantitative un @ @ regulated areas gaining attention in the real estate market
Source
+ a & A o H E If of
H
ABWS top
mayor election L E fel f ofl park Al the elections next day from work on began...
top
top 3
= @ mayoral t2 A > following same starting business in started &quot;
Output
mayor park began his work the day after the election.
Target
one day after the election, park began his job.
Source
d 0
ス @
E
o I a of
ABWS top
rights - @ ス| -
human violations e T ol 2
top top 3
human they teach &apos; does what also rights violation says says this are
d he teaching hurt do things ol X all say say is being and it is also said that not teaching human rights is violation of human rights
Output Target
some say that not teaching human rights also a violation of human rights
performance. Therefore, adopted the 2 stacked transformer encoder as model, considering its size and performance.
5.3 EVALUATION OF ALIGNMENT USAGE ON LOW-RESOURCE CASES
low-resource case that might lead to poor word alignment and our models' lower performance. We first sampled 100,000 sentences from the expectations,
We evaluated our proposed model's performance
EN-+E data set, and the AER for that small data set 34.0. Contrary Table 3c shows that our models achieve even more remarkable performance gains (up to 2 BLEU) on the low-resource case. This means that model is also useful in low-resource cases.
5.4 ANALYSIS OF ALIGNMENT USAGE IN TRANSLATION
In this subsection, We observed how the ABWS model utilizes prior word alignment in the NMT model, which applies Tgt-Fusion integration strategy the KO~EN translation task. Table 4 shows some translation examples that include top ABWS output. We can see that the ABWS model replaces some source words with target one and the replaced target words appears in the translation output. Morcover, from the Korean word A] z (market, mayor) in the first and second translation examples,
that the ABWS model distinguishes the homophone in each sentence. This means that the model's replacement process takes context into account. Another observation is that the ABWS model learned well for multi-aligned words. For example, in the third example, el (human rigths) is aligned with two target words (Chuman' and 'rights"), ) and the ABWS model catches them well.
CONCLUSION
This work presents a novel solution for the effective and efficient fusion of NMT and SMT. Espe- cially, we explore an efficient way of exploiting prior word alignment offered by SMT models for NMT enhancement during the model training phase instead of constraint decoding in previous work which may slow down the inference. In detail, to augment NMT input, we design an extra model that leams how to replace specific source words with their corresponding target ones according to the prior word alignment. Our method helps yield significant performance improvement compared to a strong baseline three translation tasks, which verifies the effectiveness of the proposed method.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Tamer Alkhouli, Gabriel Bretschner, and Hermann Ney. On the alignment problem in multi-head attention-based neural machine translation. In Proceedings of the Third Conference Machine Translation: Research Papers, pp. 177-185, Brussels, Belgium, 2018. Association for Com- putational Linguistics. doi: 10.18653/1/W114-1
URL https::/wwwtt aww.eb.
Muthology/W1BB63//
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate, 2014. URL http://an /ttp://axxiv..o wrg//bsg//a.//ee..ee.eo/eee abs/1403 0473 cite Sociv:1409.0473 Comment: Accepted at ICLR 2015 as oral presentation.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10. 00.18653/11/N19-1425 1423. URLhttps://ww/
aclweb. org/ angh N1 /11-0yyy/199122233
Georgiana Dinu, Prashant Mathur, Marcello Federico, and Yaser Al-Onaizan. Training neural ma- chine translation to apply terminology constraints. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3063-3068, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P194 1294. URL https: www ac web.
oxc/antthology/PI9-11
Kuzman Ganchev, João V. Graça, and Ben Taskar. Better aligmments better translations? In Proceedings of ACL-08: HLT, pp. 986-993, Columbus, Ohio, June 2008. Association for Com- putational Linguistics. URI. htttt:://wwwt http://wwwwwww....c/// org/
Qin Gao and Stephan Vogel. Parallel implementations of word aligument tool. In Software Engi- neering. Testing, and Quality Assurance for Natural Language Processing, pp. 49,57, Columbus, Ohio, 2008. Association for Computational Linguistics. URL https: www aclweb org/
enthology/W033 0509
Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik. Jointly learning to align and translate with transformer models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Hong Kong, November 2019. URL htRL:tts:://// httpp:ss//axxxv.rrr..
abs/1909 02074
Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Ben- jamin Van Durme. Improved lexically constrained decoding for translation and monolingual rewriting. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Humun Language Technologies, Volume (Long and Short Papers), pp. 839-850, Minneapolis, Minnesota, June 2019. Association for Computational Lin- guisties. doi: 10.18653/v1/N19- 1/999/100.0 URL https: wwww www.wb. wwwp:ottogaaaooooooooooo
N19 N19-1090]
Philipp Koehn, Jranz. Och, and Daniel Marcu. Statistical phrase-based translation. Proceedings of the 2003 Human Language Technology Conference of the North American Chupter of the Association for Computational Linguistics, pp. 127-133, 2003. URL http://wwww/...o...o wwtp://wwwwww.wbb.. aclweb. NN03-100
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bo- jar. Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical ma- chine translation. In Proceedings of the 45th Annual Meeting of the Association for Com- putational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pp- 177-180, Prague, Czech Republic, 2007. Association for Computational Linguistics. URL. https: // www. acl www.wwwb.oo org/ antaannooooooo ogy/ P07-2045
Zuchao Li. Rui Wang, Kehai Chen, Masao Utiyama, Eicchrro Sumita, Zhuosheng Zhang, and Hai Zhao. Explicit sentence compression for neural machine translation. In Proceedings of AAAI- 2020, 2020.
11980
10

---

Under review as a conference paper at ICLR 2021
Franz Josef Och and Hermann Ney. Improved statistical alignment models. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, ACL '00, pp. 440-447, USA, 2000. Association for Computational Linguisties. doi: 10.3115/1075211:17555744 URL
ht.pss:/dooi.
3111///1111111111111111 1055774
Franz Josef Och and Hermann Ney. The alignment template approach to statistical machine transla- tion. Computational Linguistics, 30(4))417-449, 2004. doi: 10. 10.1162/0911200445448884 URL
www
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, 2002. Association for Computational Linguistics. doi: 1031115/107303.10733333 URL https:
//www. aclweb. org/ ant wttp://annnoooooyyo---04400
Jeonghyeok Park and Hai Zhao. Korean-to-chinese machine translation using chinese character as pivot clue. In Proceedings of 33rd Pacific Asia Conference Language, Information and Computation (PACLIC 33), pp. 558-566, Hakodate, Japan., 2019.
Matt Post and David Vilar. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1314-1324, New Orleans, Louisiana, June 2018. Association for Com- putational Linguistics. doi: 10.18653/v//118- 1119. URL https: /www ac aclweb. web.org/
anthology/N1B-1I1
Rico Sennrich and Barry Haddow. Linguistic input features improve neural machine translation. In Proceedings of the First Conference on Machine Translation; Volume 1, Research Papers, pp- 83-91, Berlin, Germany, 2016. Association for Computational Linguistics. doi: 10.18653/v1/
/www. wwwwww.www.c..... www.wwwwwwwwwww.....o....w.....a http://nnhhhoooooaoaae W16 22009
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, 2016. Association for Computational Linguistics. doi: 10.18653/v 1/t15/3//11111111 162. Utthttps: /wwwwwwww...... aclweb. org/
anthologgy/Pl6-162
Kai Song, Kun Wang, Heng Yu, Yue Zhang, and Min Zhang. Alignment-enhanced transformer for constraining nmt with pre-specified translations. In Proceedings of the AAAI Conference Artificial Intelligence, 2020. doi: http:://doi.or//10..699// www./////.............o.. 434105.6418.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. 1409. 3215
CoRR, 4bs/1409.3215, 2014. URL
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In In Advances in Neural Informa- fion Processing Systems 30: Annual Conference on Neural Information Processing Systems, pp- 2017. URL httplitttp///aaxv.ooogg. /tt.aa/bb.//.../ 03762
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances Neural Information Processing Systems 32, pp. 5753-5763. Curran Associates, Inc., 2019.
11


Reviews:
[{"review_id": "hNjyPWos7Bl", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to integrate word alignment obtained from SMT into an NMT system. This is an exciting topic not only because it can help interpretability, but also because the same mechanism could be used e.g. for imposing a specific terminology in translations, something that was relatively easy to do with SMT but is much harder to achieve with NMT. The proposed method involves computing word-word alignment using existing SMT models (GIZA and FastAlign in the experiments) and integrating that information in the decoder of a transformer-based NMT model. Experiments on English-Romanian and English Korean show small improvement over a standard baseline.\n\nModeling is described at a high level and would require significant guesswork to re-implement. The core of the method is in the word substitution model (Sec 3.2) and how it is integrated in the decoder (Sec 3.3). Unfortunately neither is described with much detail. For example, there are a dozen operators involved in describing the model in Eqs. 1-17, some of them defined, some of them relatively easily guessable, other entirely unclear.\nIt would greatly help to clearly define the input and outputs of the model (e.g. vectors of context, matrices of probabilities or activation, appropriate dimensions, etc.)\n\nRegarding evaluation, it is a positive to have results on two language pairs, in both directions. In the English-Romanian pair, GIZA seems to produce slightly better results, although the difference is unlikely to be significant. Still, it is odd that only  FastAlign was used for Korean. Although gains seem fairly consistent, they are also very limited and unlikely to be significant (no significance test seems to have been performed). Also the variability of performance (e.g. due to sampling) is not assessed. Finally, the only comparison is to a straight transformer baseline. None of the methods mentioned in the prior art have been tested.\n\nThe analysis of the impact of the alignment on the attention weights in Sec 5.2 is interesting. What is the motivation for using different sentences and epoch #, rather than show how alignment is modeled as training progresses (as claimed in the text) on the same sentence?\n\nMisc:\n[Sec 5.1]: why is the sampling rate suddenly called the « compression ratio »?\n\n** UPDATE**\nRead the author answers, thanks. There is a significant amount of new material added in response to the reviews, including some interesting new findings (e.g. Tab. 3, 4). Unfortunately that did not help the clarity issue. I did not see a new example in Section 3.2.\nBasically, the stronger points of the paper (the results) are stronger, but the weaknesses (clarity, motivation) are not really addressed. The new version uploaded by the authors is an improvement but I still lean towards rejection.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9gX-2zg1iX", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to incorporate word alignment information as a word substitution model. Basic idea is to jointly train a separate encoder using a cross entropy loss which predicts a source input sequence with words substituted by aligned target words. The learned representation is combined with a Transformer either by simple summation, gating or joint attention mechanism. Experimental results on Romanian/English and Korean/English tasks show very marginal gains over the baseline Transformer.\n\n# Pros\n\n* It is a very interesting work on combining word alignment information in a model by predicting aligned target words by following alignment links.\n\n* Small gains are observed on Romanian/English and Korean/English.\n\n* The attention weight visualization especially for word substituted encoder presents sharper distribution, which might indicate that useful information might be learned in the attention mechanism.\n\n# Cons\n\n* The gains are very small and thus, I suspect that the difference might be negligible when running statistical significance tests.\n\n* Analysis is a bit weak in that it does not compare  the quality of alignment against the translation qualities.\n\n# Details\n\nThis work presents an interesting idea, though, I have a couple of concerns to this submission. Thus, I have some hesitation for full acceptance.\n\n* I'd like to see any tradeoffs of word alignment qualities and translation qualities by randomly distorting word alignment links. This work simply tweak sampling rate to vary the ratio of using word alignment links, not distorting links, and thus, it is not a direct measure to verify the quality tradeoff.\n\n  * Thank you very much for adding the comparison of GIZA and fastalign. However, simply swapping a model for alilgnment does not given us details about the tradeoff of alignment quality and the end-to-end results, since alignment models have different characteristic to capture the correspondence in two langauges, e.g., assuming linearlity and/or fertility. I'd rather like to see a much simpler approach of distroting alignment to avoid influence of the models employed for word alignment.\n\n* Given that non-aligned words are also predicted in the word substituted model in Equation 8, I suspect the model is simply learning to copy from input to output for non-aligned words. I think the loss in Equation 8 should be applied only to those words substituted by alignment links.\n  \n  * Thanks for the explanation. I'd like to see ablation studies regarding the loss.\n\n* Please verify the experimental results by statistical significance tests.\n\n  * Thank you for adding the tests. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5R_j-mLAX4x", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a strategy to integrate prior word alignments into NMT models. It is not clear the motivation for this in the NMT context, especially why the prior alignments are crucial information that is necessary to be given a-priori to the Transformer. Besides this, the description of the method and the discussion of related work is given, SMT methods are briefly mentioned but the usage of the idea in previous work, also SMT literature is necessary.\nThe applicability of the method is discussed very generally for NMT, however most languages do not have a one-to-one mapping in words and even in extremely low-resource cases it is difficult to see how the alignment information can be useful except for translating for instance foreign words and named entities (in languages where transliteration is not necessary).\nThe description of the method seems sound and the experiments are performed for two languages with improvements in BLEU scores. Have the authors analyzed what is exactly being improved in the translations?\nThe paper is mostly clear but has too many grammatical errors and can benefit from revisions and more editing.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers suggested to reject and the other reviewer also thought it below the threshold.
Decision: Reject