Rethinking Translation Memory Augmented Neural Machine Translation
Year: 2023

Abstract:
This paper rethinks translation memory augmented neural machine translation (TMaugmented NMT) from two perspectives, i.e., a probabilistic view of retrieval and the variancebias decomposition principle. The finding demonstrates that TM-augmented NMT is good at the ability of fitting data (i.e., lower bias) but is more sensitive to the fluctuations in the training data (i.e., higher variance), which provides an explanation to a recently reported contradictory phenomenon on the same translation task: TM-augmented NMT substantially advances vanilla NMT under the high-resource scenario whereas it fails under the low-resource scenario. Then we propose a simple yet effective TM-augmented NMT model to promote the variance and address the contradictory phenomenon. Extensive experiments show that the proposed TMaugmented NMT achieves consistent gains over both conventional NMT and existing TM-augmented NMT under two variancepreferable (low-resource and plug-and-play) scenarios as well as the high-resource scenario.

heading: 1 Introduction
text: The effectiveness of Translation Memory (TM) in Machine Translation has long been recognized (Garcia, 2009; Koehn and Senellart, 2010; Utiyama et al., 2011; Wang et al., 2013; Liu et al., 2019), because a TM retrieved from a bilingual dataset (i.e., training data or an external dataset) may provide valuable knowledge for the source sentence to be translated. Many notable approaches recently have been proposed to enhance neural machine translation (NMT) by using a TM (Feng et al., 2017; Gu et al., 2018; Cao et al., 2020; Hoang et al., 2022b; Cai et al., 2021; Huang et al., 2021).
For example, on the standard JRC-Acquis task, TM-augmented NMT achieves substantial
∗Partial work was done when Hongkun Hao was interning at Tencent AI Lab.
†Lemao Liu and Rui Wang are corresponding authors.
gains over the vanilla NMT (without TM) under the conventional high-resource training scenario. Unfortunately, Cai et al. (2021) surprisingly find that TM-augmented NMT fails to advance the vanilla NMT model on the same task under a lowresource scenario, if a quarter of the full data is used for training and TM retrieval, as reproduced in Table 1. Due to the lack of theoretical insights, it is unclear the reason why such a contradictory phenomenon happens. This motivates us to rethink the working mechanism of TM-augmented NMT as well as its statistical principle.
In this paper, we first cast TM-augmented NMT as an approximation of a latent variable model where the retrieved TM is the latent variable through a probabilistic view of retrieval. From this statistical viewpoint, we identify that the success of such an approximation depends on the variance of TM-augmented NMT with respect to the latent variable. Then, we empirically estimate the variance of TM-augmented NMT from the principle of variance-bias decomposition in the learning theory. Our findings demonstrate that TMaugmented NMT is worse than the vanilla NMT in terms of variance which indicates the sensitivity to fluctuations in the training set, although TMaugmented NMT is better in terms of the bias which indicates the ability of fitting data. The finding about the variance takes the responsibility of the contradictory phenomenon in Table 1, because
2589
limited training data may amplify its negative effect on variance (Vapnik, 1999; Niyogi and Girosi, 1996; Bishop and Nasrabadi, 2006).
To better trade off the variance and the bias, we further propose a simple yet effective method for TM-augmented NMT. The proposed method is general to be applied on top of any TM-augmented NMT models. To validate the effectiveness of the proposed approach, we conduct extensive experiments on several translation tasks under different scenarios including lowresource scenario, plug-and-play scenario, and high-resource scenario.
Contributions of this paper are three-fold: • It rethinks and analyzes the variance of TMaugmented NMT models from the probabilistic view of retrieval and the bias-variance decomposition perspective. • It proposes a simple yet effective lightweight network to ensemble TM-augmented NMT models, which better trades off variance and bias. • Its experiments show the effectiveness of the aforementioned approach, which outperforms both vanilla Transformer and baseline TM-augmented NMT models under the low-resource scenario, plug-and-play scenario, and conventional highresource scenario.

heading: 2 Preliminary

heading: 2.1 NMT
text: Suppose x = {x1, · · · , xn} is a source sentence and y = {y1, · · · , ym} is the corresponding target sentence. NMT builds a probabilistic model with neural networks parameterized by θ, which is used to translate x in the source language to y in the target language. Formally, NMT aims to generate output y given x according to the conditional probability defined by Eq. (1):
P (y|x;θ) = m∏
t=1
P (yt|x,y<t;θ)
=
m∏
t=1
Softmax ( f(Ht) ) [yt],
(1)
where Ht denotes the NMT decoding state.

heading: 2.2 TM-Augmented NMT
text: In general, TM-augmented NMT works in the following two-step paradigm. It first retrieves topK TM bilingual sentences Z = {zk}Kk=1, where zk = (x tm k ,y tm k ) is the k-th TM; Then it generates
the translation y by using the information from the source sentence x and its retrieved TMs Z.
Retrieval Model Following previous works (Gu et al., 2018; Zhang et al., 2018; Xia et al., 2019; He et al., 2021), for x we employ Apache Lucene (Białecki et al., 2012) to retrieve top-100 similar bilingual sentences from datastore. Then we adopt the similarity function in Eq. (2) to re-rank the retrieved bilingual sentences and maintain top-K (e.g. K = 5) bilingual sentences as the TMs for x:
sim(x, zk) = 1− dist(x,xtmk )
max(|x|, |xtmk |) , (2)
where dist denotes the edit-distance.
Generation Model Given a source sentence x and a small set of relevant TMs Z = {zk}Kk=1, the generation model defines the conditional probability P (y|x,Z;θ):
P (y|x,Z;θ) = m∏
t=1
P (yt|x,y<t,Z;θ)
=
m∏
t=1
Softmax ( f(Ht,Z) ) [yt],
(3)
where Ht,Z denotes the decoding state of TM-augmented NMT. There are different TMaugmented NMT models, and accordingly there are different instantiations of Ht,Z. We refer readers to Gu et al. (2018); Bulte and Tezcan (2019); Cai et al. (2021); He et al. (2021) for its detailed definitions.

heading: 3 Rethinking TM-Augmented NMT

heading: 3.1 Probabilistic View of Retrieval
text: Given the source sentence x, the top-K retrieval aforementioned actually can be considered as a probabilistic retrieval model (i.e., P (Z|x)), from which a translation memory Z = {zk}Kk=1 is sampled. Mathematically, such a retrieval model P (Z|x) is defined as follows:
P (Z|x) = ∏
k
P (zk|x) ∝ exp(sim(x, zk)/T ),
(4) where sim is defined as in Eq. (2), and T > 0 is a temperature. Note that if T is a sufficiently small number, sampling zk from the above probabilistic retrieval model is similar to the deterministic argmax retrieval widely used in prior studies (Gu et al., 2018; Zhang et al., 2018; Xia et al., 2019).
By using the probabilistic retrieval model P (Z|x), the translation model P (y|x) is related to the variable Z theoretically through the following latent variable model:
P (y|x) = ∑
Z
P (Z|x)P (y|x,Z) = EZP (y|x,Z).
(5) In practice, it is impossible to perform the summation over all possible Z. Instead it can be estimated by the Monte Carlo sampling:
P (y|x) ≈ P (y|x,Z), with Z ∼ P (Z|x). (6)
As a result, according to Eq. (6), we can see the following statement: the TM-augmented NMT model P (y|x,Z) can be considered as an approximation of P (y|x) via Monte Carlo sampling over a latent variable model in Eq. (5). In particular, whether TM-augmented NMT P (y|x,Z) is a good estimator depends on the expected approximate error defined by EZ ( P (y|x,Z) − P (y|x)
)2. In other words, P (y|x,Z) is a good estimator of P (y|x) if the expected approximate error is small; otherwise P (y|x,Z) is not a good estimator (Voinov and Nikulin, 2012).
Because of the Equation (5), the expected estimation error is actually derived by the variance of P (y|x,Z) with respect to Z as follows:
EZ ( P (y|x,Z)− P (y|x) )2 = EZ ( P (y|x,Z)− EZP (y|x,Z) )2 := VZP (y|x,Z). (7) From the above equation, it is easy to observe that VZP (y|x,Z) actually controls the approximate effect of P (y|x,Z). Therefore, VZP (y|x,Z) would have a negative effect on TM-augmented NMT due to the fluctuations with respect to the variable Z if VZP (y|x,Z) is relatively large.
It is worth noting that the above analysis on the variance is only related to the variable Z, but has nothing to do with the variables x and y and it is agnostic to neural network architecture. Moreover, it is intractable to estimate the approximate error, i.e., the variance VZP (y|x,Z) with respect to Z because P (y|x) requires the summation over all possible Z. In the next subsection, we will employ the variance-bias decomposition principle to quantify the variance with respect to all variables given specific models.

heading: 3.2 Variance-Bias Decomposition
text: Estimation of Bias and Variance Bias-variance trade-off is a fundamental principle for understanding the generalization of predictive learning models and larger variance may induce the lower generalization ability of models (Geman et al., 1992; Hastie et al., 2009; Yang et al., 2020).
The bias-variance decomposition is typically defined in terms of the Mean Squared Error at the example level for classification tasks and Yang et al. (2020) reorganize its definition in terms of the Cross-Entropy loss. In the machine translation task the optimization loss is the Cross-Entropy loss at the token level, we hence simply extend the variance and bias decomposition in terms of cross-entropy (Yang et al., 2020) to the token level.1 Specifically, assume P0(y|x,y<t) is the empirical distribution (i.e., P0(y|x,y<t) is one if y = yt is the ground-truth word yt and 0 otherwise), and P (y|x,y<t) is the model output distribution. Then the expected cross entropy with respect to a random variable τ = {⟨x,y⟩} (i.e., the training data) can be defined:
Eτ ( H(P0, P ) ) =
− Eτ (∑
t
∑
y
P0(y|x,y<t) logP (y|x,y<t) ) .
(8)
Further, it can be decomposed as:
Eτ ( H(P0, P ) ) = DKL(P0||P )︸ ︷︷ ︸
Bias2 + Eτ ( DKL(P ||P ) ) ︸ ︷︷ ︸
Variance
, (9)
where P is the expected probability after normalization:
P (y|x,y<t) ∝ exp ( Eτ ( logP (y|x,y<t) )) .
(10) Generally speaking, the bias indicates the ability of the model P to fit the data whereas the variance measures the sensitivity of the model P to fluctuations in the training data.
We follow classic methods (Hastie et al., 2009; Yang et al., 2020) to estimate the variance in Eq. (9),
1Since it is not clear how to extend the variance-bias decomposition on top of the BLEU score used as an evaluation metric, we instead consider it on top of the token-level crossentropy loss.
which is shown in Algorithm 1 of Appendix B. The key idea is to estimate the expectation over the random variable τ and it can be achieved by randomly splitting the given training dataset into several parts and training several models on each part for average. The above bias and variance estimation method is defined for P (y|x,y<t) but it is similar to estimate the bias and variance for the TM-augmented NMT P (y|x,y<t,Z) by retrieving a TM Z for each x via top-K retrieval as default rather than sampling as in §3.1.
Experiments We conduct experiments on JRCAcquis German⇒English task to estimate variance and bias of vanilla Transformer and TM-augmented NMT models. Similar to the preliminary experiment in §1, we retrieve top-5 TMs for TMaugmented NMT. In order to eliminate the effect of model architecture, we use both TM-augmented NMT backbones, which respectively involve a single encoder (Bulte and Tezcan, 2019) and dual encoder (Cai et al., 2021), see details in Appendix C. Following Yang et al. (2020), Bias2 is estimated by subtracting the variance from the loss.
Table 2 shows the variance-bias decomposition results of different models. Firstly, within each backbone, we can find that the variance of TMaugmented NMT model is larger than that of vanilla Transformer, which verifies our hypothesis that variance of TM-augmented NMT model is worse than that of vanilla Transformer, which results in poor performance under the low-resource scenario indicated in §1. Secondly, the bias of TM-augmented NMT model within each backbone is smaller than that of vanilla Transformer, which explains the better performance under the highresource scenario indicated in §1. In addition, since the variance is highly dependent on the scale of training data and the limited training data may even amplify its negative effect on variance (Niyogi and Girosi, 1996; Vapnik, 1999; Bishop and Nasrabadi,
2006), the higher variance of TM-augmented NMT in Table 2 gives an explanation for the contradictory phenomenon in Table 1.
In summary, although the TM-augmented NMT model has lower bias, resulting in fitting on the training data better especially when the training data size is large, it has non-negligible higher variance, which leads to its poor performance with less training data. Therefore, the inherent flaw drives us to find ways to reduce the variance of TM-augmented NMT models, as shown in the next section. Note that our proposed methods shown below can all theoretically reduce the variance with respect to the variable Z (i.e. TM) in §3.1, and our experiments below empirically show that reducing the variance with respect to Z can actually reduce the model variance introduced here.

heading: 4 Proposed Approach for Lower Variance
text: In this section, we first propose two techniques to reduce the variance of the TM-augmented NMT model. Then we propose a new TM-augmented NMT on the basis of the two techniques to address the contradictory phenomenon as presented in §1. Finally we empirically quantify the variance and bias for the proposed TM-augmented NMT. Note that the proposed TM-augmented NMT is general enough to be applied on top of any specific TMaugmented NMT models.

heading: 4.1 Two Techniques to Reduce Variance
text: Technique 1: Conditioning on One TM Sentence In conventional TM-augmented NMT, the translation model P (y|x,Z) is conditioned on Z consisting of five2 retrieved TMs {z1, z2, z3, z4, z5} with zi = (x tm i ,y tm i ). Similar to the analysis in Eq.(5-6), we can obtain the following equations:
P (y|x, z1) = ∑
z2,··· ,z5 P (y|x, z1,Z>1)P (Z>1|x, z1)
= ∑
z2,··· ,z5 P (y|x, z1,Z>1)P (Z>1|x)
≈P (y|x,Z), with Z>1 ∼ P (Z>1|x),
(11)
where Z>1 = {z2, z3, z4, z5}, and the second equation holds due to the conditional independence assumption in Eq. (6). In this sense, we can see that P (y|x,Z) conditioned on five retrieved
2Since many studies (Gu et al., 2018; Xia et al., 2019; Cai et al., 2021) find that using five TMs is suitable.
TMs is actually an approximation of P (y|x, z1) conditioned on a single retrieved TM z1. As a result, as analysed in §3.1, whether P (y|x,Z) is a good estimator depends on the variance with respect to Z>1, i.e., VZ>1P (y|x, z1,Z>1).
Based on the above analyses, to alleviate the variance, we directly estimate P (y|x) by using a single sampled sentence pair. Formally, suppose z = ⟨xtm,ytm⟩ is a single bilingual sentence sampled from P (z|x), we adopt the following TMaugmented NMT conditioned on a single retrieved sentence z:
P (yt|x,y<t) = EzP (yt|x,y<t, z) ≈ P (yt|x,y<t, z),
P (y|x) = ∏
t
P (yt|x,y<t)
≈ ∏
t
P (yt|x,y<t, z),
(12)
where P (yt|x,y<t, z) can be any model architecture of TM-augmented NMT models P (yt|x,y<t,Z) by replacing top-K TMs Z with a single top-1 TM z. In addition, the training of the above model P (yt|x,y<t, z) is the same as the training of the conventional model P (yt|x,y<t,Z). Technique 2: Enlarging the Sample Size In Eq. (12), EzP (yt|x,y<t, z) is approximated by one sample z, which still induces some potential estimation errors due to the variance VzP (yt|x,y<t, z). In fact, the estimation errors can be further reduced by the estimation using multiple samples as follows.
Proposition 1. If z1, · · · , zK are independent and identically distributed random variables sampled from the P (z|x), then the following inequality holds (The proof is presented in Appendix A.):
VzP (yt|x,y<t, z) ≥
Vz1,··· ,zK ( 1 K ∑
k
P (yt|x,y<t, zk) ) . (13)
According to the above Proposition, we propose the method to approximate P (yt|x,y<t) through the average ensemble of all P (yt|x,y<t, zk):
P (yt|x,y<t) ≈ 1
K
K∑
k=1
P (yt|x,y<t, zk),
P (y|x) ≈ m∏
t=1
K∑
k=1
P (yt|x,y<t, zk). (14)
Note that we do not particularly retrain the averaged ensemble model by directly taking the parameters from the model P (yt|x,y<t, z) trained in Technique 1 above in this paper.

heading: 4.2 TM-Augmented NMT via Weighted Ensemble
text: In our experiments as shown in §4.3 later, we found that the both techniques deliver better variance but sacrifice its ability of fitting data (i.e., bias) compared to the standard TM-augmented NMT conditioning on all retrieved Z = {zk}Kk=1. In order to further improve the bias for a better ability to fit data, we propose the weighted ensemble to establish a stronger relationship between the source sentence and each TM zk by endowing a more powerful representation ability via the weighting coefficient w(x,y<t, zk) in Eq. (15). Formally, the weighted ensemble model is defined as:
P (y|x) ≈ m∏
t=1
K∑
k=1
w(x,y<t, zk)P (yt|x,y<t, zk),
(15) where P (yt|x,y<t, zk) can be any TM-augmented model architecture as claimed in §4.1, and w(x,y<t, zk) is a weighting network via the following equation:
w(x,y<t, zk) = Softmax ( f(Ht, Ht,k) ) [k], (16)
where f consists of two linear layers with residual connection and layer normalization (Vaswani et al., 2017), Ht is the decoding state of a vanilla translation model P (yt|x,y<t), Ht,k is the decoding state for any TM-augmented model P (yt|x,y<t, zk) similar to Ht,Z in Eq. (3). For example, if we implement the weighted ensemble model on top of the model architecture of Cai et al. (2021), then the source sentence and the k-th TM are encoded by two separate encoders respectively, resulting in a hidden state Ht for current time step and a contextualized TM representation Ht,k for the k-th TM in current time step.
Similar to the average ensemble method, we do not train the whole network from scratch. Instead, we start from the trained model parameters of P (yt|x,y<t, zk) in §4.1 and then we just fine-tune the whole parameters including those from both P (yt|x,y<t, zk) and w(x,y<t, zk) for only about 2,000 updates on 90% part of valid data while the other 10% part of valid data are used to select the checkpoint for testing.
Remark The above average ensemble model involves K retrieved sentence pairs similar to the standard translation memory augmented models (Gu et al., 2018; Zhang et al., 2018; Cai et al., 2021), but the notable difference is that our model P (yt|x,y<t, zk) conditions on a single zk whereas those standard TM-augmented NMT models (see Eq. (3)) directly condition on K retrieved sentence pairs Z = {zk}Kk=1.

heading: 4.3 Empirical Analysis on Bias and Variance
text: To verify the effectiveness of the proposed methods empirically, we conduct the variancebias decomposition experiments similar to §3.2. Specifically, we estimate the variance and bias of both three proposed models and two baselines (please refer to §5.1 for detailed settings).
Table 3 shows the overall results of estimated variance and bias. By comparing results of different models, we can get the following two observations: Firstly, through the results of variance, we can find that the three proposed methods all achieve lower variance compared with the default TM-augmented NMT model. It is notable that the weighted ensemble method achieves the best variance within all the TM-augmented NMT models. Secondly, through the results of bias, we can find that all TM-augmented NMT models achieve lower bias compared to vanilla Transformer (Vaswani et al., 2017) without TM. Although our methods have a slightly higher bias, the proposed weighted ensemble method can achieve comparable bias with regard to the default TM-augmented NMT model.

heading: 5 Experiments
text: In this section, we validate the effectiveness of the proposed methods in three scenarios: (1) the lowresource scenario where training pairs are scarce, (2) the plug-and-play scenario where additional bilingual pairs are added as the data store and the model is not re-trained any more as the data store is enlarged, and (3) the conventional high-resource
scenario where the entire training data are used for training and retrieval. We use BLEU score (Papineni et al., 2002) as the automatic metric for the translation quality evaluation.

heading: 5.1 Settings
text: Data We use the JRC-Acquis corpus (Steinberger et al., 2006) and select four translation directions including Spanish⇒English (Es⇒En), En⇒Es, German⇒English (De⇒En), and En⇒De, for evaluation. Besides, we also use the re-split version of the Multi-Domain data set in Aharoni and Goldberg (2020) originally collected by Koehn and Knowles (2017) for our experiments, which includes five domains: Medical, Law, IT, Koran and Subtitle. Detailed data statistics and descriptions are shown in Appendix E.
Models To study the effect of the proposed methods in §4, we implement a series of model variants by using the fairseq toolkit (Ott et al., 2019). #1 vanilla NMT without TM (Vaswani et al., 2017). We remove the model components related to TM, and only employ the encoder-decoder architecture for NMT. #2 Default TM-augmented NMT with top-5 TMs. We use top-5 TMs to train and test the model. Note that this is also a baseline model in Cai et al. (2021). #3 TM-augmented NMT with single TM. To study the effect of technique 1 in §4.1 which conditions only one TM, we use top-1 TM to test the model. In order to avoid overfitting, during each epoch we use training pairs with top-5 TMs and empty TM to train the model six times, which is similar to He et al. (2021). #4 TM-augmented NMT with the average ensemble. To study the effect of technique 2 in §4.1 which enlarges the sample size, we use the trained model in #3 directly and average ensemble top-5 TMs. #5 TM-augmented NMT with the weighted ensemble. We fine-tune the trained model in #3 with weighted ensemble top-5 TMs. Detailed model descriptions and experimental settings are in Appendix C and D respectively.

heading: 5.2 Low-Resource Scenario
text: One of the major advantages of our proposed weighted ensemble is that it has a lower variance, which means that it is less sensitive to fluctuations in the training data. This motivates us to conduct experiments in low-resource scenarios, where we use only a part of the training data to train models. Specifically, we create low-resource scenario by
randomly partitioning each training set in JRCAcquis corpus and Multi-Domain dataset into four subsets of equal size. Then we only use the training pairs in the first subset to train each model.
Results The test results of the above models on Multi-Domain dataset and JRC-Acquis corpus are presented in Table 4 and Table 5 respectively. We can get the following observations: (1) Our proposed weighted ensemble method delivers the best performance on test sets across all translation tasks, outperforming the vanilla Transformer by a large margin. (2) The performance of default TM-augmented NMT with top-5 TMs is degraded compared with vanilla Transformer, while single TM method and average ensemble method make up for the degradation issue to some extent.

heading: 5.3 Plug-and-Play Scenario
text: “Plug-and-play” is one of the most remarkable properties of TM (Cai et al., 2021), that is to say, the corpus used for TM retrieval is different during training and testing. This is useful especially in online products because we can adapt a trained model to a new corpus without retraining by adding or using a new TM. Specifically, we directly use models trained in §5.2, and add the second, the third, and the last subset to the TM data store gradually. At each datastore size, we retrieve for the test set again and test the performance of models with the newly retrieved TMs.
Results Figure 1 and Figure 2 show the main results on the test sets of JRC-Acquis corpus and
Multi-Domain dataset respectively. The general patterns are consistent across all experiments: The larger the TM becomes, the better translation performance the model achieves. When using all training data (4/4), the translation quality is boosted significantly. At the same time, our proposed weighted ensemble method achieves the best performance all the time.

heading: 5.4 High-Resource Scenario
text: Following prior works (He et al., 2021; Cai et al., 2021) in TM-augmented NMT, we also conduct experiments in the high-resource scenario where all the training pairs of JRC-Acquis corpus are used to train models in order to prove the practicability and universality of our proposed methods.
Results The results are presented in Table 6. We can see that our proposed weighted ensemble method still has the best performance even under the high-resource scenario, where default TMaugmented NMT using top-5 TMs is also relatively good because in this condition the bias plays an important role in fitting the whole training data to achieve good performance.

heading: 6 Related Work
text: This work mainly contributes to the research line of Translation Memory (TM) augmented Neural Machine Translation (NMT). Early works mainly concentrated on model architecture designing (Feng et al., 2017; Gu et al., 2018; Cao and Xiong, 2018; Cao et al., 2020; Zhang et al., 2018; Xia et al., 2019; Bulte and Tezcan, 2019; Xu
et al., 2020; He et al., 2021; Wang et al., 2022d; Hoang et al., 2022a; Cai et al., 2022). Recently, Cai et al. (2021) used a dense retrieval method with a dual encoder to compute the similarity between the source sentence and TMs, which can use monolingual translation memory instead of bilingual ones. Hoang et al. (2022b) proposed to shuffle the retrieved suggestions to improve the robustness of TM-augmented NMT models. Cheng et al. (2022) proposed to contrastively retrieve translation memories that are holistically similar to the source sentence while individually contrastive to each other providing maximal information gains.
The distinctions between our work and prior works are obvious: (1) We rethink TM-augmented NMT from a probabilistic view of retrieval and the variance-bias decomposition principle; (2) We consider the performance of TM-augmented NMT under low-resource scenario, plug-and-play scenario, and conventional high-resource scenario, instead of only high-resource scenario. (3) Our methods are agnostic to specific model and retrieval method and thus can be applied on top of any advanced architecture.
Another research line highly related to our work is kNN-MT (Khandelwal et al., 2021; Zheng et al., 2021a,b; Meng et al., 2022; Wang et al., 2021, 2022a; Zhu et al., 2022; Wang et al., 2022c; Du
et al., 2022, 2023; Wang et al., 2022b). However, kNN-MT retrieves similar key-value pairs on the token level, resulting in slow retrieval speed (Meng et al., 2022; Dai et al., 2023) and large storage space (Wang et al., 2022a; Zhu et al., 2022), whereas TM-augmented NMT retrieves similar pairs on the sentence level, which has more efficient retrieval speed and takes up less storage space.

heading: 7 Conclusion
text: Existing work surprisingly finds that TMaugmented NMT fails to advance the NMT model under low-resource scenario, but it is unclear the reason why such a contradictory phenomenon happens. This paper rethinks TM-augmented NMT from latent variable probabilistic model view and variance-bias decomposition view respectively, giving the reason for the failure under low-resource scenario. Estimation for variance and bias indicates that TM-augmented NMT is better at fitting data (bias) yet worse at sensitivity to fluctuations in the training data (variance). To better trade off the bias and variance, this paper proposes a simple yet effective weighted ensemble method for TM-augmented NMT. Experiments under three scenarios demonstrate that the proposed method outperforms both vanilla Transformer and baseline
TM-augmented models consistently. Future work could aim to find factors that influence the bias of TM-augmented NMT models, such as the quality of retrieved TMs.
Limitations
Compared with the standard NMT, TM-augmented NMT models incur extra retrieval time during both training and inference. Besides, since both TMs and the source sentence need to be encoded, the speed of TM-augmented NMT is slower than that of standard NMT. These characteristics indeed slightly induce some overhead. Specifically, experiments show that the latency time cost for all TM-augmented NMT models is higher than for vanilla Transformers. And the decoding time of our proposed weighted ensemble method is twice as much as that of the standard NMT. This issue is one limitation of TM-augmented NMT, which can be further studied and addressed.
Ethics Statement
This paper will not pose any ethical problems. First, machine translation is a standard task in natural language processing. Second, the datasets used in this paper have been already used in previous papers.

heading: Acknowledgements
text: Hongkun and Rui are with MT-Lab, Department of Computer Science and Engineering, School of Electronic Information and Electrical Engineering, and also with the MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200204, China. Rui is supported by the General Program of National Natural Science Foundation of China (6217020129), Shanghai Pujiang Program (21PJ1406800), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), Beijing Academy of Artificial Intelligence (BAAI) (No. 4), CCF-Baidu Open Fund (F2022018), and the Alibaba-AIR Program (22088682).

heading: A Proof of Proposition 1 in §4
text: Throughout this section, we prove the Proposition 1 in §4. For simplicity, we use f(z) and f(zk) to denote P (yt|x,y<t, z) and P (yt|x,y<t, zk) in Proposition 1, respectively. Note that each zk are i.i.d sampled from P (z|x).
Proof. Firstly, for the relationship between Ezf(z) and Ez1,··· ,zK ( 1 K ∑ k f(zk) ) we can get the following equations:
Ez1,··· ,zK ( 1 K ∑
k
f(zk) )
= 1
K Ez1,··· ,zK
(∑
k
f(zk) )
= 1
K
∑
k
( Ezkf(zk) )
= 1
K
∑
k
( Ezf(z) )
= 1
K ∗K ∗ Ezf(z)
= Ezf(z).
Secondly, we start from the relationship between
Vzf(z) and Vz1,z2 ( f(z1) + f(z2) ) :
Vz1,z2 ( f(z1) + f(z2) )
= Ez1,z2 ( f(z1) + f(z2)
− Ez1,z2 ( f(z1) + f(z2)
))2
= Ez1,z2 ( f(z1) + f(z2) )2
− ( Ez1,z2 ( f(z1) + f(z2) ))2
= Ez1 ( f(z1) )2 + Ez2 ( f(z2) )2
+ 2Ez1,z2 ( f(z1)f(z2) ) − ( Ez1f(z1) )2 − ( Ez2f(z2) )2 − 2 ( Ez1f(z1) )( Ez2f(z2) )
= Ez1 ( f(z1) )2 + Ez2 ( f(z2) )2
+ 2 ( Ez1f(z1) )( Ez2f(z2) ) − ( Ez1f(z1) )2 − ( Ez2f(z2) )2 − 2 ( Ez1f(z1) )( Ez2f(z2) )
= Ez1 ( f(z1) )2 − ( Ez1f(z1) )2
+ Ez2 ( f(z2) )2 − ( Ez2f(z2) )2
= Vz1f(z1) + Vz2f(z2) = 2Vzf(z).
Then, we can get the relationship between Vzf(z) and Vz1,z2 ( 1 2 ( f(z1)+f(z2) )) as follows:
Vz1,z2 (1 2 ( f(z1) + f(z2) ))
= 1
22 Vz1,z2
( f(z1) + f(z2) )
= 1
4 ∗ 2 ∗ Vzf(z)
= 1
2 Vzf(z)
< Vzf(z).
Similarly, for the relationship between Vzf(z) and Vz1,··· ,zK ( 1 K ∑ k f(zk) ) we can get the
following equations:
Vz1,··· ,zK ( 1 K ∑
k
f(zk) )
= 1
K2 Vz1,··· ,zK
(∑
k
f(zk) )
= 1
K2
(∑
k
( Ezk ( f(zk) )2 − ( Ezkf(zk) )2) )
= 1
K2
∑
k
( Vzkf(zk) )
= 1
K2
∑
k
( Vzf(z) )
= 1
K2 ∗K ∗ Vzf(z)
= 1
K Vzf(z)
≤ Vzf(z), where the equality holds if and only if K = 1.
B Variance-Bias Estimation Method
In §3.2, we mention that we follow classic methods (Hastie et al., 2009; Yang et al., 2020) to estimate the variance and bias in Eq. (9). Here we detail the method to estimate the variance, which is shown in Algorithm 1. Then Bias2 is estimated by subtracting the variance from the loss (Yang et al., 2020). Specifically, in this paper we set N to be 1 and k to be 4 in Algorithm 1. Besides, we compute the variance for each test point (x,y<t) in test set, and average them to get the final variance for each model.

heading: C Details about Architecture
text: In §3.2, we use single encoder backbone (Bulte and Tezcan, 2019) and dual encoder backbone (Cai et al., 2021) respectively. Here we provide a detailed description of these two architectures.
Single Encoder Architecture For this architecture, we follow Bulte and Tezcan (2019). Specifically, the model architecture is the same as vanilla Transformer, which consists of an encoder and a decoder. The encoder encodes the concatenation of the source sentence and TM, relying on the encoder’s self-attention to compare the source sentence to the TM and determine which TM phrases are relevant for the translation(Hoang et al., 2022a). Therefore, the only difference is that we need to change the input format to the concatenation of the source sentence and TM.
Algorithm 1 Estimating Variance of NMT Models Input: Test point (x,y<t), Training data τ .
1: for i = 1 to k do 2: Split the τ into τ (i)1 , · · · , τ (i) N . 3: for j = 1 to N do 4: Train the model using τ (i)j ; 5: Evaluate the model at (x,y<t); call the result P (i)j (y|x,y<t); 6: Normalize the top 100 probability in
P (i) j (y|x,y<t) and setting others to be 0
(to reduce computation complexity); 7: end for 8: end for 9: Compute P̂ (y|x,y<t) =
exp (
1 N ·k ∑ i,j logP (i) j (y|x,y<t) )
(P̂ (y|x,y<t) estimates P (y|x,y<t)). 10: Normalize P̂ (y|x,y<t) to get a probability
distribution. 11: Compute the variance var =
1 N ·k ∑ ij DKL ( P (i) j (y|x,y<t)||P̂ (y|x,y<t) )
Output: var
Dual Encoder Architecture This architecture is similar to that used by Cai et al. (2021), which achieves relatively good performance and thus can serve as a strong baseline (Cai et al., 2021).
On the encoder side, the source sentence and TM are encoded by two separate encoders respectively, resulting in a set of contextualized token representations {zkj}Lkj=1, where Lk is the length of the k-th TM.
On the decoder side, after using y<t and x to get the hidden state Ht at current time step t, it can produce the original NMT probability Pnmt(yt|x,y<t) and the TM probability Ptm(yt|x,y<t,Z) concurrently, and finally get the probability P (yt|x,y<t,Z).
Specifically, the original NMT probability Pnmt(yt|x,y<t) is defined by as follows:
Pnmt(yt|x,y<t) = Softmax ( f(Ht) ) [yt], (17)
where f is a linear layer that maps hidden state Ht to a probability distribution. And for the TM probability Ptm(yt|x,y<t,Z), the decoder firstly computes a cross attention score α = {αkj} over all tokens of the k-th TM sentence where αkj is the attention score of Ht to the j-th token in the k-th
TM. The computation is shown as follows:
αkj = exp(HTt Wtmzkj)∑K
k=1 ∑Lk j=1 exp(H T t Wtmzkj) , (18)
where Wtm are parameters. Then we can get the contextualized TM representation Ht,Z for the TM Z in current time step, as shown in Eq. (19):
Ht,Z = Wh
L∑
k=1
Lk∑
j=1
αkjzkj , (19)
where Wh are parameters. Then, the TM probability Ptm(yt|x,y<t,Z) can be computed as follows:
Ptm(yt|x,y<t,Z) = K∑
k=1
Lk∑
j=1
αkjIzkj=yt , (20)
where I is the indicator function. Finally, the decoder computes the next-token probability as follows:
P (yt|x,y<t,Z) = (1− λt)Pnmt(yt|x,y<t)+ λtPtm(yt|x,y<t,Z) (21)
where λt is a gating variable computed by another linear layer whose input is Ht,Z.

heading: D Detailed Settings for All Experiments
text: Here we provide a detailed description of our configuration settings in the paper for the
preliminary experiment (§1), the variance-bias decomposition experiment (§3.2 and §4) and the main experiment (§5) respectively.
Settings for Preliminary Experiment in §1 In this part, we implement vanilla Transformer base model (Vaswani et al., 2017) for the standard NMT. The learning rate schedule, dropout, and label smoothing are the same as Vaswani et al. (2017). For the TM-augmented NMT, we follow Cai et al. (2021) and implement the dual encoder architecture aforementioned. We conduct experiments on the JRC-Acquis German⇒English task. For highresource scenario we use the full training data and train models with up to 100k steps. For lowresource scenario we randomly select a quarter of training data and train models with up to 30k steps.
Settings for Variance-Bias Decomposition Experiment in §3.2 and §4 For the experiment in §3.2, we use both single encoder and dual encoder architecture introduced above in order to eliminate the effect of model architecture. For dual encoder, since the amount of parameters in vanilla Transformer is originally smaller than that of TM-augmented NMT model, which makes the two models non-comparable (Yang et al., 2020), we use empty TMs to simulate vanilla Transformer and make the two models comparable. We set N = 1 and k = 4 in Algorithm 1 to estimate the variance, so we train models with up to 30k steps. The other settings are the same as the preliminary experiment
in §1. For single encoder, the configuration is the same as vanilla Transformer base model (Vaswani et al., 2017). And we also train models with up to 30k steps. For the experiment in §4, we use dual encoder architecture as in Table 3.
Settings for Main Experiment in §5 We build our model using Transformer blocks with the same configuration as Transformer Base (Vaswani et al., 2017) (8 attention heads, 512 dimensional hidden state, and 2048 dimensional feed-forward state). The number of Transformer blocks is 4 for the memory encoder, 6 for the source sentence encoder, and 6 for the decoder side. We retrieve the top 5 TM sentences. The FAISS index code is "IVF1024_HNSW32,SQ8" and the search depth is 64.
We follow the learning rate schedule, dropout, and label smoothing settings described in Vaswani et al. (2017). We use Adam optimizer (Kingma and Ba, 2015) and train models with up to 100K steps throughout all experiments. When fine-tuning models with the proposed weighted ensemble method, we only need to use 90% valid pairs to finetune with up to 2K steps and use the remaining 10% valid pairs to choose the checkpoint for testing.
Table 7 provides the number of parameters, training budget, and hyperparameters of each model. All experiments were performed on 8 V100 GPUs. We report the result of a single run for each experiment.
Packages Table 8 shows the packages we used for preprocessing, model training and evaluation.

heading: E Details about Data Statistics
text: In this paper, we use the JRC-Acquis corpus (Steinberger et al., 2006) and the re-split version of the Multi-Domain data set in Aharoni and Goldberg (2020) for our experiments.
JRC-Acquis For the JRC-Acquis corpus, it contains the total body of European Union (EU) law applicable to the EU member states. This corpus was also used by Gu et al. (2018); Zhang et al. (2018); Xia et al. (2019); Cai et al. (2021) and we managed to get the datasets originally preprocessed by Gu et al. (2018). Specifically, we select four translation directions, namely, Spanish⇒English (Es⇒En), En⇒Es, German⇒English (De⇒En), and En⇒De, for evaluation. Table 9 shows the detailed number of train/dev/test pairs for each language pair.
Multi-Domain Data Set For the Multi-Domain data set, it includes German-English parallel data in five domains: Medical, Law, IT, Koran, and Subtitle. Table 10 shows the detailed number of train/dev/test pairs for each domain.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section Limitations and Section Ethics Statement.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 1, 3, 4, 5.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 1, 3, 4, 5 and Appendix B, C, D, E.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 1, 3, 4, 5 and Appendix B, C, D, E.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 1, 3, 4, 5 and Appendix B, C, D, E.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section Ethics Statement.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 1, 3, 4, 5 and Appendix B, C, D, E.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5 and Appendix E.
C 3 Did you run computational experiments? Section 1, 3, 4, 5.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix D.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 1, 3, 4, 5 and Appendix D.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 1, 3, 4, 5.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 1, 3, 4, 5 and Appendix D.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work could aim to find factors that influence the bias of TM-augmented NMT models, such as the quality of retrieved TMs. Limitations
Compared with the standard NMT, TM-augmented NMT models incur extra retrieval time during both training and inference. Besides, since both TMs and the source sentence need to be encoded, the speed of TM-augmented NMT is slower than that of standard NMT. These characteristics indeed slightly induce some overhead. Specifically, experiments show that the latency time cost for all TM-augmented NMT models is higher than for vanilla Transformers. And the decoding time of our proposed weighted ensemble method is twice as much as that of the standard NMT. This issue is one limitation of TM-augmented NMT, which can be further studied and addressed