Translation Memory Guided Neural Machine Translation
Authors: Shaohui Kuang
Heng Yu
Weihua Luo
Qiang Wang
Venue:  (2021)
Keywords: neural machine translation
translation memory
pre-train language model

Abstract: Many studies have proven that Translation Memory (TM) can help improve the translation quality of neural machine translation (NMT). Existing ways either employ extra encoder to encode information from TM or concatenate source sentence and TM sentences as encoder's input. These previous methods don't model the semantic relationship between the source sentence and TM sentences. Meanwhile, the training corpus related to TM is limited, and the sentence level retrieval approach further limits its scale. 
In this paper, we propose a novel method to combine the strengths of both TM and NMT. We treat the matched sentence pair of TM as the additional signal and apply one encoder enhanced by the pre-trained language model (PLM) to encode the TM information and source sentence together. Additionally, we extend the sentence level retrieval method to the n-gram retrieval method that we don't need to calculate the similarity score. Further, we explore new methods to manipulate the information flow from TM to the NMT decoder. We validate our proposed methods on a mixed test set of multiple domains. Experiment results demonstrate that the proposed methods can significantly improve the translation quality and show strong adaptation for an unknown or new domain.

Under review as a conference paper at ICLR 2021
TRANSLATION MEMORY GUIDED NEURAL MACHINE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Many studies have proven that Translation Memory (TM) can help improve the translation quality of neural machine translation (NMT). Existing ways either em- play extra encoder to encode information from TM or concatenate source sen- tence and TM sentences as encoder's input. These previous methods don't model the semantic relationship between the source sentence and TM sentences. Mcan- while, the training corpus related to TM is limited, and the sentence level retrieval approach further limits its scale. In this paper, we propose novel method to bine the strengths of both TM and NMT. We treat the matched sentence pair of TM the additional signal and apply guage model (PLM) to encode the TM information and source sentence together. Additionally, extend the sentence level retrieval method the n-gram retrieval method that we don't tt need to calculate the similarity score. Further, we explore new methods manipulate the information flow from TM to the NMT decoder. We validate our proposed methods on a mixed test set of multiple domains. Ex- periment results demonstrate that the proposed methods can significantly improve the translation quality and show strong adaptation for an unknown new domain.
encoder enhanced by the pre-trained lan-
INTRODUCTION
Neural machine translation (NMT), an end-to-end approach, has achieved state-of-the tit-thhe-rttt art translation performance on many language pairs (Vaswani et al., 2017 Wang al.] (201)). Usually, trained NMT model translates new sentence into the target language from scratch. However, human translators can quickly and accurately translate a sentence by reuse existing repetitive translation fragments in the translation memory (TM). Therefore, we naturally think of using TM to improve the translation quality NMT. Typically, a TM consists bilingual parallel sentence pairs(TM- source and TM-target) that are similar to the current sentence to be translated (Koehn & Senellart 2010 Cao & Xiong 2018). And from statistical machine translation (SMT) to NMT, a variety of efforts have been made to integrate TM into machine translation.
The process of integrating TM information and NMT mainly includes two steps: TM retricval and fusion of TM information and NMT network. For the fusion of TM and NMT, such attempts have already been conducted. And a commonly used integration way is to employ the multi-encode structure. Cao & Xiong (2018) propose a simple method that employs new encoder encode the TM information to guide the decoding process and Xia et al. (2019) also use a graph-based encoder pack the TM sentences into graph. Their methods all require additional encoder structure, ignore the TM-source information, and only encode the TM-target information. These will cause series of problems. On the one hand, it will significantly increase the parameter scale of the network. On the other hand, the encoding process of T'M-target information and source sentence are isolated from each other, so the semantic connection between them lost.
About TM retrieval, various metrics can be used to estimate the similarity score of two sentences. We select the sentences with the highest similarity from the TM database for the current source sentence by calculating the similarity score. The existing retrieval approaches used in previous work are usually to calculate the sentence level similarity score, such as Edit-Distance (Gu et al.] 2017 Xia et al., 2019), IDF-based similarity score (Bapna & Firat 2019), and cosine similarity (Xu al. 2020). TM's current work is experimenting with relatively small data sets that are usually only hundreds of thousands of sentences. One main reason is that use sentence-level similarity. When

---

Under review as a conference paper at ICLR 2021
also set relatively high similarity threshold for souree sentence, have high probability that we will not find a similar sentence in the TM database. Although Bapna & Firat (2019) and Xu al. (2020) also use the n-gram method to search, they still need to select the corresponding sentence that maximizes the n-gram similarity with the source sentence. Meanwhile, a small training set will also lead to insufficient training of network parameters.
In this paper, to address the problem presented above, we proposed a novel and effective method for the combination of TM and NMT. The proposed approach's key idea is to treat the matched TM-source and TM-target as the additional signal and try to encode them with the source sentence together. Specifically, we first find the matched TM-source and TM-target sentence pairs from our training corpus. To enhance the semantic relationship between source sentence, TM-source, and TM-target, we use a universal encoder to encode the three sentences and obtain context representa- tion information simultaneously. Then we explore and try four methods to incorporate the context information into the decoding network. To further strengthen the ability to copy useful information from TM context information and alleviate the rare-word problem, we integrate pointer network Gulcehre et al. 2016 Gu et al. 2016 See et al., 2017) into the decoder.
To obtain sufficient training corpus and train network parameters more fully and effectively, in this paper, we also modify the retrieval algorithm and use a pre-trained language model (PLM) to ini- tialize the encoder's 's parameters. Partially inspired by phrase-based basse SMT, don't compute the sentence level similarity score hetween two sentences in our retrieved method. If two sentences have a common n-gram segment, we assume that they are similar, and the sentence pairs of the TM database can provide a useful segment to help improve the translation quality. Currently, many stud- ies have proven that PLM can offer valuable prior knowledge to enhance the translation performance of NMT (Weng et al. 2020 [Song et al. 2019), So we also employ PLM to initialize the parameters of the encoder and give encoder well-trained parameters as a starting point.
To validate the proposed approach's 's effectiveness, we implement our idea on top of the state-of- the-art model Transformer (Vaswani et al.) 2017). A series of experiments on the Eeglish-to-Frenche translation task demonstrate that the proposed method can significantly improve NMT with the TM information. In summary, we make three main contributions:
We employ the n- n-gram retrieval to find similar sentence, and this very simple and fast. does not need the complicated fuzzy matches algorithm to calculate the similarity between the source sentence and TM-source from TM or training data.
Does not need an extra encoder to encode the retrieved sentence from TM and use only an encoder enhanced by PLM to model the semantic relationship between TM sentences and the source sentence and obtain their context representation information simultaneously.
apply the Copy Mechanism to alleviate the rare-word problem, especially when we do not have sufficient training corpus.
2 RELATED WORK
TRANSLATION MEMORY
Our work aim at the studies that integrate Translation Memory into machine translation. Many methods have been proposed to combine TM and MT. For example, Koehn & Senellart (2010) applies the matched segments from TM SMT in decoding. However, the integration of TM and NMT is more complicated, and limited efforts method is explored so far compared with the fusion of TM and SMT. Cao & Xiong (2018) identify this as a multi-input problem and use the multi-encode framework to encode the retneved TM-target sentence and current source sentence. On this basis, Baapna & Firat (2019) propose new approach that incorporates information from source sentence and TM-source while encoding the TM-target. Gu et al. (2017) encode all similar sentences from TM into context-vectors and use context-vectors to decode the target word by an additional attention mechanism. Xia et al. (2019) further extend the method of (Gu et al.) 2017) in which they pack the sequential TM into a graph, leading to a more efficient attention computation. Our proposed method does not require extra encoder, so additional encoder parameters are introduced. Additionally, Different from the method proposed by BaBana & Firat (2019), we encode the three sentences (source, TM-source, TM-target) and obtain their context representation information simultaneously.
2

---

Under review as a conference paper at ICLR 2021
SOURCE AUGAENTEEE
Our work is related to studies that use similar sentences to improve translation quality. For exam- ple. Niehues et al. (2016) employ the pre-translations obtained by phrase-based SMT system to augment the input sentences, and then concatenate the pre-translations and input sentences as the final input of the NMT model. This method is maintaining separate SMT system which might introduce errors of its own. A similar principle, to dynamically adapt individual input sentences, Li al. (018)) retrieve similar sentences from the training data finetune the general NMT model. But this approach requires running expensive gradient descent steps before every translation. Xu al. (2000) and Bulte & Tezcan (219) employ the various experimenting to introduce how re- trieve similar sentences from TM and use the selected sentence as the extra information of source sentence. However, these studies only focus on how to enhance the context information of source sentences. How to use enhanced context information more effectively in the decoder is ignored. In our proposed methods, we try different ways manipulate the extra context information from TM-source and TM-ttaret.
EXTERNAL KNOWLEDGE FOR NMT
Our work is also related to previous works that incorporate external knowledge or information into NMT. Recently, a series of studies have explored the integration of SMT knowledge into the nmt system. For example, Zhou et al.] (2017) and Wang al.] (2017) propose to integrate the SMT recommendations into NMT and improve the translation quality of NMT. Besides, there exist also studies on the learning of global context by the aid of discourse-leve level methods in document machine translation (Kuang & Xiong. 2018 Kuang al., 2018 Zhang al., 2018). Additionally, several successful attempts have been made to apply the pre-traimed model for NMT (Ramachandran et al., 2017 Song et al., 2019 Weng et al. 2020). In addition to these, some studies also integrate external dictionaries into NMT or employ the force-decoding methods to constrain the decoding process of NMT by given words/phrases in target translations (Hokamp & Liu 2017
Post Vilar 2018
Hasler et al. 2018).
PROBLEM FORMULATION
3.1
ATTENTIONAL NMT
Here, we will introduce neural machine translation based on the Transformer (Vaswani et al. 2017
which has achieved state-of-the-art performance in several language pairs. Formally, let T (2), ... ) be a source sentence and y = (y) ....... yr) and target sentence are parallel sentence pair. x, denotes the i-th word vector representation in source sentence and y denotes the j-th word vector representation in target sentence.
target sentence. The source
By default, the encoder is composed of stack Lf identical layers. The output of I-th encoder layer can be computed as follows:
O LN(ATT(Q',, K. V +HH H'
H' = LN((FNNO(OO") O'))
(2)
where ATT, LN and FFN are self-attention mechanism, layer normalization, and feed-forward net- works with ReLU activation in between, respectively. Q' K' and V are the query, key and value matrix that are transformed from the (I 1)-th encoder layer output H'-1, We define the output of the 1-th encoder layer as He
In the decoding stage, the decoder is also composed of stack of L identical layers. the output Z' of 1-th decoder layer can be calculated as follows:
O LN(ATT(Q' K", V') + 2
(3)
S' LN(ATT(OO), Ko, Vo)+ + +'))
(4)
z' LN(FFN(S") + +S))
(5)
3

---

Under review as a conference paper at ICLR 2021
where Q', K' and V are transformed from the (L 11tt decoder layer Z'-1 Ke and Ve transformed from the He. For attention mechanism ATT, the computed process is as follows:
C₁
at * 8/
(6)
j=1
Etj
at =
(7)
Ek=1 Ctk
Etj = q:ki
(8)
where is computed context vector of the t-th words, and the is transformed from hidden repre- sentation of the (-th words, k and v is obtained from the previous layer of the decoder or the context representation of the encoder.
TRANSLATION MEMORY GUIDED NMT
In this section, first introduce how TM sentence pairs retrieved. Then describe the proposed Translation Memory Guided NMT (TMG-NMT) model in detail. Our TMG-NMT consists of two parts, universal memory encoder for encoding context information of TM sentences and source sentence and TM-guided decoder for using the TM context information to improve the translation quality.
4.1 SEGMENT-BASER TM RETRIEVAL
Given a source sentence r to be translated, we find a matched example (tz, ty) from a translation memory database D {ar, })M Existing approaches proposed by previous work employ off-the- shelf search engines for the retrieval stage, and the retrieval source part for has the highest similarity score to the æ. In general, when calculating similarity scores between i and fr, the sentence level similarity score is employed, such as IDF-Based sentence score (Bapna & Firat 2019), Fuzzy Match Score (Cao & Xiong 2018), and Miiit-Ditannc (Li et al. 2018).
Motivated by phrase-baset SMT, in this paper, propose the segment- retrieval method that uses n-gram as the segment representation. When the retrieved sentence tz has the same segment consider that (tx, ty) may be helpful improve the translation quality. Given source sentence æ, we denote its segments set as (81, 82, Sx), where 8k is one n-gram in the sentence For each 8ş in s, we try find matched example (tz. ty) from D what tz contains the 8μ. So, for sentence æ, we can obtain a example set {(tx by } with N sentence pairs. The advantages of the segment-based method are as follows:
as the
It doesn't need to compute the sentence level score, which greatly saves matching time.
For (x. y) pair in training data, we can find N matched sentence pairs, this dramatically increases the size of our training corpus.
4.2 UNIVERSAL MEMORY ENCODER
Figure illustrates the proposed universal memory encoder (UM-encoder). We concatenate the source sentence, TM-source sentence, and TM-target sentence the input of the encoder. Formally, We use tr = (tæı,tæz,.... tæ and ty = (ty₁. ty,,...,tyy to represent the re- trieval TM-source sentence and TM-target sentence. The input of encoder H is concatenation of all vector representations of source words, TM-source words and TM-target words. H [m:; 2;; taz ...: f. by:;;..; tyy.)
Then, we split the H' to get the source context H TM-source context H and TM-ttrget context Hty as follows:
H Ht. H = split(H' I,
where split is function to split the matrix according the length parameters NN.
Context control Unlike the previous work, we don't employ additional context encoder or self- attention layer (Cao & Xiong 2018 Kuang & Xiong 2018) encode and integrate contexts. And
4

---

Under review as a conference paper at ICLR 2021
conternt
IM-ssurre contenat
TM- TM-targel contenst
A
Ap
Ax
Aup
Aug
Ass
Ray
Transformer-encoder
Exp
Exp
Sourco
TM-sourcee
TM-tarret
Figure I: Architecture of the proposed UM-encoder model. A separation mark "SEP" is inserted between them. Position embedding is introduced distinguish the word order in each sentence;
FFN
FFN
concat-context
+
cross-attention
TM-larget context
gated cross attention
self-attention
self-attention
source context
TM-target context
target
source context
target
Figure 2: Architecture TM-concat decoder
Figure 3: Architecture of TM-gate decoder
only use one encoder to model the concatenated input. In this way, additional parameters will be introduced and can also keep the same source sentence's semantic consistency. Meanwhile, distinguish the context information from different sentences, a separation mark "SEP" is inserted between sentences. We arrange its position embedding separately to keep the word order for each sentence.
lingsll Encoding The TM information contains two different languages: the source language and target language. we need the UM-encoder to learn the ability to encode the semantics of different languages. For this goal, introduce two methods into the UM-encoder:
Firstly, we use pre-trained cross-lingual language model (PLM) to initialize the UM- encoder's parameters, which provides an excellent cross- lross-language modeling ability in ad- vance.
Secondly, We use TM-source as bridge establish the connection between the source sentence and TM-target. Because the TM-source and TM-target are the parallel sentence pair while the source sentence and TM-source are the same languages, it is easier model the connection.
4.3 TM-GUIDED DECODER
By the UM-encoder, we can obtain the source context vectors, "o--suurce context vectors, and TM- target context vectors. So, we can employ the TM context information to guide the decoder to predict next words. is worth mentioning that we ignore TM-source context information because the words in the TM-source also appear in the source sentence, and we don't need to re-encode them while the words in TM-source but not in the source sentence are not helpful for translation. Additionally, we have used TM-source bridge to establish the semantic association between TM- target and source. There may exist different ways to use TM context information. We concentrate on the following four methods.
TM-implicit Decoder As we use UM-encoder for encoding. the source context already contains TM-target context information. So, only the source context H, as input H, of the decoder.
TM-concat Decoder Following Xu et al. (2020), we also concatenate the source context and TM- target context the input of the decoder. Figure 2 illustrates the TM-coneat decoder model. And
5

---

Under review as a conference paper at ICLR 2021
Porpy
Piredis
Pcopy trnget Til-orget steedion wnight
Ppredict vice denniilyy
FFN
TM-target context
gated cross attention
self-attention
source context
target
Figure 4: Architecture of TM-point decoder.
the He can be obtained as follows:
H = H Hlg
(10)
TM-gate Decoder Following Bapma & Firat (2019). We also try to use the gate attention mechanism for information fusion and use the H. and Hly the decoder's input. Figure 4 schematically illustrates the TM-gate model. The St of the decoder recalculated as follows:
S ATT(O KA, Vo)
(11)
S = ATT(O', Keg, Viy)
(12)
g = sigmoid(FFN(S5,. m. Sty Z z1-1))
(13)
St IN(g' S + -') Sty +
(14)
where g' is the gated context vector obtained by sigmoid function and FFN. The K2 and V, are transformed from the H., and the Kry and Vty are transformed from the Hty.
TM-pointer Decoder We further integrate the pointer-network into the decoder to enhance the model's copy ability based the TM-gate model. Figure is schematic representation of the TM-pointer model. At each decoding step t, we can get the current gated scalar 9t by Equation (1I) and attention distribution vectors a computed by Equation (10) from (-th decoder layer, where Σd a; = 1 and N the length of TM-target sentence. The final word prediction probability for ytis calculated follows:
Pluiueer x) = Ppredict (ye (eleee, x) + (1 - 9c) Peopy (V: ())
(15)
where Ppredict (ye 1ec. 22 is the vocabulary distribution computed by decoder. The Peopy (3e Vee, the probability what copying a word from the TM-target sequence and we use the attention distri- bution vectors represent this probability. Notice that if is not in the TM-target sentence, we set Peopy (ye Uete x) 0.
5 EXPERIMENTS
We carried out a series of Eoglish-to-Frende translation experiments to evaluate the effectiveness of the proposed TMG-NMT and conducted in-depth analyses experiment results and translations.
5.1 DATA
We used the following corpora in this work to validate the performance of the proposed model: the WMT training set (36M pairs), Oppen-Subiitles (33M pairs), the IWSLT bilingual corpus (237k pairs) and JRC-As (797k pairs). For OpenSubtitles and JRC-Acquis, there is no publicly available test set. We create our own splits for validation and test. After segmentation, the JRC-Acquis test set contains 2000 sentence pairs and the OpenSubtitles test set has 2000 sentence pairs. For WMT,

---

Under review as a conference paper at ICLR 2021
Table 1: BLEU scores on the English-to-French translation tasks and the baseline trained using the concatenation of all the original corpus. OST indicate the OpenSubTitle test. Avg means the average scores on all sets. Higher BLEU scores indicate better translation quality.
Model
IWSLT15
OST
JRC
WMT14
Avg
A
Deep- Transformer
43.07
34.76
54.31
42.02
43.57
Baseline
Transformer-Hee
44.25
35.18
56.43
43.53
44.84
w/ pretrain
44.88
35.02
56.99
43.23
45.03
TM-implicit
44.85
35.57
61.31
43.94
46.41
+1.38
TM-concat
45.45
35.86
61.69
43.77
46.69
+1.66
TMG-NMT
TM-gate
46.18
35.44
61.71
43.97
46.82
+1.79
TM-pointer
46.97
36.49
62.60
44.64
47.67
+2.64
use newstest 13 for validation and the newstest 14 for test. For IWSLT, we use test 2014 as validation and test 2015 as test.
In the training stage, for each sentence pair (x, y) in the training corpus, we find up to N similar sentence pairs læx, by from the training corpus. And we set the N 10. In the test stage, for each source sentence in the test set, we randomly select one of them when finding N similar sentence pairs. Additionally, we restrict the retrieval to the in-domain training corpus.
Following M-BERT (Pires et al. 2019)) and for source vocabulary, we use I 10k shared WordPiece vocabulary. For the target vocabulary, we extract French-related character representations from source vocabulary, and the target vocabulary size is 39000, We used the case-insensitive 4-gram NIST BLEU (Papineni et al. 2002) score as our evaluation metric.
5.2
EXPERIMENTAL SETTINGS
We implemented our TMG-NMT model based on standard Transformer Base model VVaswani et al. (017). For the pre-trained cross cross-lingual language model, We take M-BERT trained on 104 languages as our first choice because its structure consistent with the encoder of the Transformer.
For the encoder of the TMG-NMT, to conveniently load the parameters from M-BERT, we set the dimension of the hidden layer as 768 and set the size of the feed-forward layer is 3072. The number of layer and attention heads are 12 for the encoder. For the decoder of the TMG-NMT, the hidden dimension of all layers set to 512, and the size of the feed-forward layer is set to 2048. We employ 8 parallel attention heads. The number of layers for the decoder is 6. Sentence pairs are batched together by approximate sequence length, and the maximum length of a sentence limited to 100. We use label smoothing with value 0.1 and dropout with rate of O.1. We Adam (Kingma & Ba 2015) to update the parameters, and the learning rate was varied under a warm-up strategy with 6000 steps. Other settings of Transformer follow (Vaswaní et al. 2017).
In order to compare the performance our proposed model more fairly, we also use the same parameter settings train the following two baseline systems. Our proposed models maintain the same configuration as Transformer-Bert.
Deep-Transforme The baseline model that encoder contains 12 layers, hidden size is set to 512, the size of the feed-forwar layer is 2048, and the number of attention head in the encoder are 12.
Transformer- Bert The enhanced baseline model that the encoder has the same configura- tion with TMG-NMT. The encoder contains 12 layers, hidden size is set to 768, the size of the feed-forward layer is 3072, and the number of attention head in the encoder are 12.
5.3 EXPERIMENTAL RESULTS
Main Results Table displays the translation performance measured in terms of BLEU scores. The results the IWSL, OpenSubTitle, JRC and WMT14 indicate that every of proposed TMG-NMT models improves the translation accuracy in comparison to the transformer baseline. With respect BLEU scores, observe consistent trend that the TM-gate model works bet-

---

Under review as a conference paper at ICLR 2021
Table 2: BLEU scores when the baseline model and TMG-NMT with TM-pointer model are only trained on WMT corpus.
Model
IWSLTI5
OST
JRC
WMT14
Avg
A
Transformer- Bert with pretrain TM-pointer
39.32
22.33
52.59
42.76
39.25
41.04
24.98
58.60
43.76
42.09
+2.84
ter than the TM-concat model and TM-implicit model, while the TM-pointer model achieves the best accuracy over all test sets. On all test sets, the TM-pointer outperforms the baseline Deep- Transformer by 4.1 BLEU points and outperforms the other three proposed methods by 0.54~11.26 BLEU points.
compared with Enhanced baseline We also trained an enhanced transformer model (Transformer- Bert), which has the same encoder parameters configuration as TMG-NMT. From Table 1, we can find that as the scale of parameters increases, Transformer-Be Bert achieves an average gain of 1.27 BLEU points over the heep-transformer. When we use M-BERT's parameters to initialize the en- coder, it further improves 0. 19 BLEU points compared with Transformer-Bi However, compared with the enhanced baseline, the proposed TMG-NMT still achieves average improvement of 1.38~2.64 BLEU points.
5.4 ADAPTATION
In the section, we consider a different translation scenario to NMT. In this scenario, we have a new TM that has never been seen when training the TMG-NMT model. Or the test sets need to translate are a new domain, which is entirely different from the corpus for training the TMG- NMT model. So, we need to test whether the TMG-NMT model can be adapted to any new dataset by updating the retrieval TMM database. In document-leve translation tasks and online translation services, it's common phenomenon that new sentence will be added continuously to the TM cache, and should be applied immediately to translate the following sentence.
So, following [Bapna & Firat 2019). on the WMT training corpus, we train baseline model and TMG-NMT model. Then we valid the translation performance in JRC, IWSLT15, and OpenSubTitle test sets. We retrieval similar T'Msource and TM-target from their respective training corpus when evaluating the adaptation of The proposed TMG-NMT.
Table shows the corresponding results. Note that since the model is only trained on the WMT corpus, all test BLEU scores are lower than those shown in Table The TM-pointer model sig- nificantly outperforms the base model on all test sets and achieves an average gain of 2.84 BLEU points. The results three test sets indicate that the proposed methods show good adaptability to the unknown or new domain.
CONCLUSION
In this paper, we have presented TMG-NMT, an effective method to encode translation memory into NMT. Firstly, we explore new framework that employs the universal memory encoder to simultaneously encode the TM information and source sentence, allowing the encoder to obtain the semantic relationship easily. Secondly, the TM-guided decoder proposed to manipulate the information flow from TM to NMT decoder. Especially, we incorporate the pointer-network into the TM-guided decoder to further strengthen copying. Finally, We only use the n- n-gram matching algorithm to find similar sentences, making easier for obtain the training corpus of the TMG- NMT model and expand the model's application scenarios.
Experiments English to French translation shows that the proposed models can significantly improve translation quality. Further in-depth analysis demonstrate that our models show excellent adaptability to the unknown new domain.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ankur Bapna and Orhan Firat. Non- Kon-parametric adaptation for neural machine translation. In Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long and Short Papers), PP- 1921-1931, 2019.
Bram Bulte and Arda Tezcan. Neural fuzzy repair: Integrating fuzzy matches into neural machine translation. In Proceedings the 57th Annual Meeting of the Association for Computational Lin- guistics, pp. 1800-1809, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/P19- 1175. URLhttts::/ www. ac web org/ an /P10p1//1991111
Qian Cao and Deyi Xiong. Encoding gated translation memory into neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 3042 3047, 2018.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. Incorporating copying mechanism in sequence-to-seye seeuunecc learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1631-1640, 2016.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. Search engine guided non- paraamaeettic neural machine translation. arXiv preprint 10X.::1705.072000 2017.
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. Pointing the unknown words. In Proceedings of the 54th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers). pp. 140-149, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/P16- 1014. URL https: www. ac web.
org/ htrp//nntholooo//l6-1111
Eva Hasler, Adrià de Gispert, Gonzalo Iglesias, and Bill Bymee Neural machine translation decod- ing with terminology constraints. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Short Papers), volume 2, pp. 506-512, 2018
Chris Hokamp and Qun Liu. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), volume 1, pp. 1535-1546, 2017.
Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR 2015), 2015.
Philipp Koehn and Jean Senellart. Convergence translation memory and statistical machine trans- lation. In Proceedings of AMTA Workshop MT Research and the Translation Industry, pp- 21-31, 2010.
Shaohui Kuang and Deyi Xiong. Fusing recency into neural machine translation with an inter- sentence gate model. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 607-617, 2018
Shaohui Kuang. Deyi Xiong, Weihua Luo, and Guodong Zhou. Modeling coherence for neural ma- chine translation with dynamic and topic caches. In Proceedings the the 27th International Confer- Computational Linguistics, pp. 596-606, Santa Fe, New Mexico, USA, August 2018. As- sociation for Computational Linguistics. URL https: www.
C18-1055
Xiaoqing Li, Jiajun Zhang, and Chengqing Zong. One sentence one model for neural machine translation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European I Language Resources Associa- tion (ELRA). URL htt.ttttps.... www. ac web. erge has///nthoolgygg hoypoologyyy//l
1146
Jan Niehues, Eunah Cho, Thanh-Le Ha, and Alex Waibel. Pre-translation for neural machine trans- lation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 1828-1836 2016

---

Under review as a conference paper at ICLR 2021
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311-318. Association for Computational Linguistics, 2002.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual bert? In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4996-5001, 2019.
Matt Post and David Vilar. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), volume 1. pp. 1314-1324, 2018.
Prajit Ramachandran, Peter J Liu, and Quoc Le. Unsupervised pretraining for sequence sequence learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 383-391, 2017.
Abigail See, Peter J Liu, and Christopher D Manning. Get the point: Summarization with pointer- generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computa- lional Linguistics (Volume 1: Long Pupers), pp. 1073-1083, 2017.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mass: Masked sequence to sequence pre-training for language generation. In ICML, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Qiang Wang, Bei Li, Tong Xiao, Jingho Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1810-1822, 2019.
Xing Wang, Zhengdong Lu, Zhaopeng Tu, Hang Li, Deyi Xiong, and Min Zhang. Neural machine translation advised by statistical machine translation. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp. 3330-3336. AAAI Press, 2017.
Rongxiang Weng, Heng Yu. Shujian Huang, Shanbo Cheng, and Weihua Luo. Acquiring knowledge from pre-trained model to neural machine translation. AAAI, 2020.
Mengzhou Xia, Guoping Huang, Lemao Liu, and Shuming Shi. Graph based translation memory for neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7297-7304, 2019.
Jitao Xu, Josep Crego, and Jean Senellart. Boosting neural machine translation with similar trans- lations. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pp. 1590, Online, July 2020. Association for Computational Linguistics. doi:
10. 10.18653/11/002..acc acl-main. 144. URL https:: /wwp:://wwww... aclweb org wwt.///tnoooogy//2222... acl 1al1mmann... 144
Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. Improving the transformer translation model with document-level context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 533-542, 2018.
Long Zhou, Wenpeng Hu, Jiajun Zhang, and Chengqing Zong. Neural system combination for ma- chine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), volume pp. 378-384, 2017.
10


Reviews:
[{"review_id": "nt7WTftPnWQ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a way to integrate a translation memory into a neural machine translation model. They introduce a novel method for finding matching sentences (which they do not properly describe) and they also have 4 methods from including this information into an NMT model, with the best approach using multiple sources combined with gated attention and a pointer network for copying rare words. They show improvements of up to 2.64 BLEU over a reasonable but not SOTA baseline for English-French task. \n\nMy major problem with this paper is that the model and the experiments are not adequately described. \n\nThe segment based retrieval function is one of the two core contributions of the paper, but it cannot be fully described. They claim that any sentence in the TM that matches any n-gram in the sentence x. This would result in a huge set of matching sentences, where 1-gram that matches could be the work \"the\". So there must be a way of ordering the matches or scoring the matches to result in a good set of N matches where N=10 here. How do they do this? \n\nAt no point to they describe where they get the translation memory from and/or how big it is. This is a fundamental part of the model. I have searched and can't find this - did I miss it? \nI find this to be an extremely worrying omission - especially as there should be a discussion about the properties of this TM and how it compares to for instance using a phrase-table/lexicon etc. \n\nTheir translation baselines are reasonable but probably significantly lower than SOTA (44.6 BLEU vs highest published 45.6 on WMT14 from http://nlpprogress.com/english/machine_translation.html)\nThey are not directly comparable however because they should be reporting detokenized BLEU using sacreBLEU. Their scores are tokenised lowercase text which means that their numbers are artificially elevates when compared to true cased tokenised text. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tu1a3WkxQ14", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes several improvements on using information from a Translation Memory (TM) in Neural Machine Translation (NMT). In the spirit of several prior work, the approach relies on 1) a retrieval step to obtain TM content that is related to the current source sentence to translate, 2) an encoder combining the source sentence with retrieved TM content, and 3) a decoder using the joint encoded information to produce a (target) translation. Experiments are conducted on benchmark French-English data, showing consistent improvement over classical baselines.\n\nTranslation Memories are important Computer-Aided Translation (CAT) tools, likely the most widely used CAT tools by translation professionals and agencies. As such, it is important to study how they can be used to improve translation quality for example through inclusion in NMT. This study is therefore a welcome addition to the relatively limited work investigating this topic. I personally wish there was more work on integrating existing translation resources in MT. However, the experimental setup does not really correspond to a typical TM use. It essentially leverages the idea of reusing close matches to a source sentence in the training data. The idea is interesting, but only loosely related to TM. On the other hand, the experiments in Section 5.4 are much closer to a TM, it is unfortunate that these experiments are quite limited.\n\nDespite the general relevant and interesting focus of this work, there are a number of issues discussed below, related mainly to modeling and to the experimental evaluation.\n\nMODELLING:\nEach of the three components on the method (retrieval, encoding, and decoding) introduces some novelty. There are also a number of issues to resolve.\n\n1) The retrieval (sec. 4.1) uses an n-gram matching technique, which is contrasted with the usual computation of sentence similarities (edit distance, fuzzy match or idf-based score). I basically don’t buy the advantages put forward in the paper:\n- The cost of retrieval in a TM is dominated by the requirement to go over the entire memory for each source sentence, not by the computation of the score. The n-gram matching would still incur that cost, unless some smart way to retrieve matching sentences (such as an inverted n-gram index) is implemented. Unfortunately there is no detail at all in the paper about how the ngram matching is performed.\n- The fact that one can retrieve N matched sentence pairs for each (x,y) pair in the training data is no different from retrieving the top-N sentence pairs using any of the usual similarity metrics.\n\nAdditionally, when retrieving N>1 sentence pairs from the TM, it is not entirely clear how the N pairs are used. One interpretation of the second paragraph of 5.1 is that this actually yields N different training instances at training time, while one match is randomly picked at prediction time. This should be clarified. In addition, this would introduce additional randomness at prediction time, producing possibly different target translations. It would be good to assess the impact of this choice on the performance, and compare against the obvious choice of picking the closest match.\n\n2) The encoding is straightforward but clever. It is not entirely clear how the encoder keeps track of the split of the context into I+M+N (one assumes here that N is no longer the number of matched pairs, but the length of the TM-target context) — is it through propagating the separation marker, hardcoded in the encoder, through some other way?\n\n3) The decoding is done in four different manners, offering various ways to integrate the TM-target information into the prediction. However, the description of « TM-pointer Decoder » in Section 4.3 seems faulty: Eq. 11 shows how to get S_x^l through the self-attention mechanism and Eq. 10 illustrates the concatenation mechanism in the TM-concat decoder, they can’t help get g_t and the attention distribution vector a.\n\nEVALUATION:\n\nStrengths:\n+ Shows consistent non trivial gains\n+ Uses a large corpus, with several domains\n+ Interesting « domain adaptation » mode shows good results [Sec. 5.4]\n\nWeaknesses:\n- Limited to French-English (close languages with lots of cognates, lots of resources, high performance) — it would be interesting to show how this works on radically different languages, especially in a lower resource setting where an existing TM may greatly help.\n- Limited comparison to a couple baselines. None of the methods cited in the related work is tested against.\n- It is unclear what significance test was used, if any, to back the claim of « significantly outperforms » (e.g. end Section 5.4). \n\nSmall typos and clarifications:\n« by reuse existing » (Sec 1)\n« nmt » (Sec 2) -> NMT\n«  Formally, We » (Sec 4.2)\nWhat is the sentence after Eq. 15 (« The p_copy… ») trying to tell us. This is not clear.\n« we set the N to 10 » (Sec 5.1) \n« set the size … is » (Sec 5.2)\n« we valid the translation performance » (Sec 5.4)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FV8mgwcEn_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "`The paper argues that the existing way of using Translation Memory (TM) in neural machine translation (NMT) is sub-optimal. Therefore it proposes TMG-NMT, namely Translation Memory Guided NMT, which consists of two parts, a universal memory encoder and a TM guided decoder. Experiments are performed to demonstrate that their method can significantly improve the translation quality and show strong adaptation for a new domain. \n\nPros: None\n\nCons:\n\n1. The main concern of this paper is that, the contributions are quite limited. The authors claimed three contributions: n-gram retrieval, universal encoder, and using the copy mechanism. Basically, none of them is novel. \n  - Bapna & Firat (2019) and Xu et al. (2020) have used n-gram matching for retrieval. \n  - In late 2020, what is the novelty of using a multilingual BERT when encoding source sentences and retrieved TM sentences? Very little if any. \n  - Likewise, using the copy mechanism to tackle rare word problems seems a regular approach.\n\n Overall, I don't see any of these so-called contributions are truly technically original. This paper seems a very hurry combination of some existing techniques. I basically learn nothing new from reading this submission. \n\n2. Another one of the key concerns about the paper is the lack of rigorous experimentation to study the usefulness of the proposed method. Despite the paper stating that there have been earlier work (Gu et al. 2017, Can & Xiong. 2018, Xia et al. 2019) that explore Translation Memory in NMT, the paper does not compare with them and only compare to non-TM-guided baselines, making the improvement less convincing. In addition, what is the language pair evaluated in this paper, which was not even mentioned...\n\n3. Considering the limited results, a deeper analysis of the proposed method would have been nice. Is the semantic relationship between the source sentence and TM sentences well learned in TMG-NMT? What kind of translation error can be well addressed with the help of TM? Further analysis of the proposed model would provide greater insight to the community.\n\n4. Section 5.4: The results would have been more complete if another setting is considered where the transformer is adapted to the target domain without using the TM mechanism, such as fine-tuning the vanilla transformer on the provided TM parallel sentences. In this way, the adaptation ability of TMG-NMT could be better proven.  \n\n5. To be honest, the writing of the current version seems a disaster. Not to mention the impressive amount of grammar errors, many parts of the storytelling are logically incoherent. For example,\n  - \"Although Bapna & Firat (2019) and Xu et al. (2020) **also** use the n-gram method to search, they still need to select the corresponding sentence that maximizes the n-gram similarity with the source sentence. \" - The usage of \"also\" is so weird, when you didn't even mention you are using n-gram method in advance... And by the way, what is the difference between your ngram matching and theirs? You should've made it clear.\n  - \"To obtain sufficient training corpus and train network parameters more fully and effectively, in this paper, we also modify the retrieval algorithm and use a pre-trained language model (PLM) to initialize the encoder’s parameters. Partially inspired by phrase-based SMT, we don’t compute the sentence level similarity score between two sentences in our retrieved method. If two sentences have a common n-gram segment, we assume that they are similar, and the sentence pairs of the TM database can provide a useful segment to help improve the translation quality. Currently, many studies have proven that PLM can offer valuable prior knowledge to enhance the translation performance of NMT (Weng et al., 2020; Song et al., 2019), So we also employ PLM to initialize the parameters of the encoder and give encoder well-trained parameters as a starting point.\" - What is the logical relationship b/w the first sentence and the second one? \n  - Please check minors for more details.\n\n\n*****\nMinors:\n1) It would have been nice to see that the format of the reference are unified.\n2) Khandelwal et al, 2020 [1] propose a novel way to incorporate Translation Memory into NMT which may bring you more thoughts towards using TM.\n\nTypos: too many. \n1. Equation (2): a redundant close paren\n2. Section 3.1 penultimate paragraph: l-th encoder layer -> L-th encoder layer\n3. Section 4.2 First paragraph: N->M n->m\n\nGrammar errors:\n\nToo many. E.g., In the contribution part in the intro, the second and the third items start with \"does\" and \"apply\". What are the subj of these two verbs?\n\nPlease try to properly use Grammarly to check your writing before submission.\n\n\n[1] \"Nearest Neighbor Machine Translation. Khandelwal et al. 2020 arXiv.\"\n\n\n******\nReasons for score:\nThe novelty of this paper is basically none. The experimental results are limited and the comparison with prior work is none, which cannot fully demonstrate the effectiveness of the proposed method.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8Dx8xypI6k", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Summary **\n\n(1) The authors proposed a translation system with an external memory. Given a sentence $x$ to be translated, they first retrieve a $(tx, ty)$ sentence pair from the training set through ``SEGMENT-BASED TM RETRIEVAL’’ defined in Section 4.1, where $tx$ and $ty$ are from source and target languages respectively. Then the $(x,tx,ty)$ are fused together to get the eventual translation, where authors design several ways to achieve that. \n\n(2) Specifically, in the encoder side, the M-BERT model is leveraged to jointly encode the $x,tx,ty$.\n\n(3) The improvement in Table 1 is significant. \n\n\n** Clarity **\n1.\tSection 4.1 is unclear to me. In the 2nd paragraph of Section 4.1, ``For each s_k in s, we try to find a matched example (tx, ty) from D what tx contains the s_k’’: (1) There should be many sentence pairs (tx,ty) that tx contains s_k. Which ones should be kept? (2) What’is more, if tx contains s_k, can we say that the selected $tx$ is similar to $x$? (3) in experiments, the $n$ in  n-gram is set as? And what if we choose different $n$?\n2.\tWhich script did you choose to evaluate BLEU score?\n\n** Significance **\n1.\tThe idea itself is not novel. Compared to the related work, the novel part of this work is: (i) a new retrieval way, which is not quite clear and convincing to me. (ii) a new way to aggregate multiple inputs (using M-BERT) and several different decoding methods. In experiments, there is no comparison with previous retrieval based methods. Similar idea also exists in [R3], which is missing from this paper. The differences with [R3] should be discussed. \n2.\tThe authors did not provide what the retrieved sentences are like. Given a validation corpus (X,Y), and the corresponding retrieved (tX, tY), the authors should at least show the similarity between (X,tX), (Y,tY), which measures the retrieval quality. \n3.\tPlease report the total training and total inference time, and make a comparison with standard Transformer model. Specifically, for Table 1, the inference time of each algorithm should be reported (retrieval time included).\n4.\tWhy do you choose case-insensitive BLEU score for En->Fr, which is not commonly used in previous baselines.\n5.\tConsidering the BERT is leveraged, you should discuss the relation with BERT + NMT [R1,R2]. \n6.     The authors should conduct experiments beyond English-to-French. More languages pairs should be verified.\n\nTypos:\n1.\tcompared with Enhanced baseline… -> Comparison with enhanced baseline\n\n\n** Refereces **\n\nR1: Zhu, Jinhua, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. \"Incorporating bert into neural machine translation.\" ICLR’20, https://arxiv.org/pdf/2002.06823.pdf\n\nR2: Yang, Jiacheng, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Yong Yu, Weinan Zhang, and Lei Li. \"Towards making the most of bert in neural machine translation.\" AAAI’20, https://arxiv.org/pdf/1908.05672.pdf\n\nR3. Eriguchi, A., Rarrick, S. and Matsushita, H., 2019, November. Combining Translation Memory with Neural Machine Translation. In Proceedings of the 6th Workshop on Asian Translation (pp. 123-130).\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a way to use a translation memory (TM) to improve neural machine translation.  Basically the proposed model uses a n-gram retrieve matching sentence （or pieces） and takes advantage of the useful parts using gated attention and copying mechanism.  Although the idea of leveraging TM in the context of NMT is not new,  this work seems to be a fair contribution. My major concerns are the following
1. The retrieval part  is not clearly presented, raising questions about  complexities and the noise brought by the common words. The authors should give a better exposition on the ranking mechanism. 
2. The experiments are not convincing enough since the proposed model is not compared to the SOTA and the competitive models described in the prior work.

In conclusion I would suggest to reject this paper.
Decision: Reject