Revisiting Source Context in Nearest Neighbor Machine Translation
Authors: Xuanhong Li
Peng Li
Po Hu
Venue: EMNLP 2023 Main (2023)
Keywords: nearest neighbor machine translation; source context; retrieval-augmented machine translation

Abstract: Nearest neighbor machine translation ($k$NN-MT), which interpolates target token probabilities with estimates derived from additional examples, has achieved significant improvements and attracted extensive interest in recent years. However, existing research does not explicitly consider the source context when retrieving similar examples, potentially leading to suboptimal performance. To address this, we comprehensively revisit the role of source context and propose a simple and effective method for improving neural machine translation via source context enhancement, demonstrating its crucial role in both retrieving superior examples and determining more suitable interpolation coefficients. Furthermore, we reveal that the probability estimation can be further optimized by incorporating a source-aware distance calibration module. Comprehensive experiments show that our proposed approach can be seamlessly integrated with representative $k$NN-MT baselines, resulting in substantial improvements over these strong baselines across a number of settings and domains. Remarkably, these improvements can reach up to 1.6 BLEU points.

Revisiting Source Context in Nearest Neighbor Machine Translation
Xuanhong Li Peng Li Po H2
Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning, Central China Normal University, Wuhan, Hubei, China School of Computer Science, Central China Normal University, Wuhan, Hubei, China 3 National Language Resources Monitoring & Research Center for Network Media, Central China Normal University, Wuhan, Hubei, China Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China xuanhong. li@mails..................mmll.l ccnu. edu. cn, lipeng@air. tsinghua. edu. cn
phu@mail ccnu. edu.
Abstract
0.81
0.81
0.74
Nearest neighbor machine translation (&NN- MT), which interpolates target token probabili- ties with estimates derived from additional ex- amples, has achieved significant improvements and attracted extensive interest recent years. However, existing research daes explicitly consider the source context when retrieving similar examples, potentially leading to subop- timal performance. To address this, compre- hensively revisit the role of source context and propose simple and effective method for im- proving neural machine translation via source context enhancement, demonstrating its crucial role in both retrieving superior examples and determining more suitable interpolation coef- ficients. Furthermore, we reveal that the prob- ability estimation be further optimized by incorporating a source-aware distance calibra- tion module. Comprehensive experiments show that proposed approach be seamlessly integrated with representative ANN-MT base- lines, resulting in substantial improvements over these strong baselines across a number settings and domains. Remarkably, these im- provements can reach up to 1.6 BLEU points.!
0.64
0.6
Ratio
0.34
0.25
0.28
0.2
0.0
[0. 0.25]
(0.25, 0.5]
(0.5, 0.75J
(0.75, 1.0]
(1.0,
(1.25,
(1.5, tinf)
1.25)
1.5]
Source Context Distance
The ratio of the top-k retrieved examples whose value equal to the ground-truie token, which is roughly inversely correlated with the source context distance, indicating the potential benifit of source con- text for İNN-MT. The development set of IT domain from (Zheng et al., 202 la) is used and l: = 32, Please refer to introduction for more detials.
Figure 1:
main adaptation (Zheng et al., 2021b; Du et al.. 2022), and online learning (Wang et al., 2022b), it has attracted increasing attention in recent years.
Due to its success, extensive efforts have been made improve the effectiveness of ANN-MT. Zheng al. (2021a) propose dynamic approach determine the number of examples to be retrieved. In contrast, Wang et (2022c) concentrate on de- veloping retrieval-aware example representations. Jiang et al. (2022) suggest leveraging NMT pre- dicted results to calibrate kNN distribution. Ad- ditionally, Zheng et al. (2021a) improve the esti- mation of probability distribution by employing ensemble of multiple estimations derived from different numbers of examples. Despite these ad- vances, the source context, e.g. the souree side of a parallel sentence pair, has not been explicitly considered in these works.
1 Introduction
Nearest neighbor machine translation (Khandel- wal et al., 2021, ANN-MT) enhances conventional neural machine translation (NMT) by adopting a retrieval-based strategy to interpolate the target to- ken probability distribution produced by the NMT model with the one estimated from similar exam- ples of an auxiliary datastore. Since demonstrat- ing promising results in wide range of machine translation scenarios such as multilingual transla- tion (Khandelwal al., 2021; Li et al., 2022), do-
Peng (lipeng @ air. tir..stinnua.... edu.en) and Po Hu (phu camiil.ccu.eedue edu.cn) are corresponding authors.
Preliminary study shows that source context may have significant potiential ANN-MT. Typically,
'Our code is available at http://gituub.com/iixaaaaa source-context-kun..e

---

an example in KNN-MTT is denoted as a key-value pair (Khandelwal al., 2021), where key is the de- coder intermediate representation given the source sentence and the target prefix, and value is the next target token observed in training data. The top- k examples ranked descendingly according to the distance between the keys and the current decoder hidden state are used to estimate the probability distribution be interpolated. Figure shows the ratio of the top-k examples whose value is equal to the ground-truth target token v.s. the source con- text distance duing inference, where source con- text distance defined as the L2 distance between the representations of the input sentence and the corresponding source sentence of an example (see Sec. 3.1 for more details). It is evident that the ratio roughly inversely correlates to the source context distance, indicating potential usefulness of source context in ANN-MT. Moreover, Dai et al. (2023) show that source context is effective in selecting subset of parallel sentence pairs for improving the efficiency ANN-MT. Therefore, source context deserves further attention in AT.
other sequence--t-ssequeecee tasks that are challeng- ing for existing kNN-MT methods, indicating that source context merits further consideration within the kNN-MT framework.
2 Background
This section briefly introduces ANN-MT (Khandel- wal al., 2021), formulated as the following two main steps: datastore construction and prediction with datastore.
Datastore Construction A datastore for ANN- MT usually contains decoder intermediate repre- sentations and corresponding target tokens key- value pairs. Specially, given sentence pair (x,y) from the training dataset T = { {(x(1, , (0)) ())), where x = I1, F2, I' and y = y/, 3/, ... Un. it is fed into pre-trained NMT model MM in a teacher forcing way. The decoder representation at position l is denoted as f(x, y <t), and the corresponding target token is yr. then the datastore D is follows:
D
e T E y). (1)
In this work, we revisit the source context in ANN-MT and reveal that it is effective in enhanc- ing the three fundamental components of 6NN-MT, namely, example retrieval, probability distribution estimation and interpolation. By leveraging dy- namic datastore, built with data selected based on source context similarities, we manage to recall ad- ditional candidate examples that hold potential util- ity, yet might be overlooked by the existing kNN- MT methods. We then perform a distance calibra- tion of all candidates based on both the source and target contexts, thereby achieving more precise probability distribution estimation. Additionally, we propose to determine the interpolation coeffi- cient based on the similarity of the source context.
Prediction with Datastore Given a source sen- tence x, the NMT model .M generates the pre- diction ý token by token. Considering the t-th time step. the decoder representation is denoted as h, = ((xx)<)). A ANN-MT model uses h, as query to retrieve the most similar 1 entries N' {(h(") "( in the datastore D. Then the ANN distribution PANN x. y<t)) is calculated based on the L2 distances d(h,, h() between query ht and the key h;:
PANN(UElX (ul Fice) x
(2)
d(h h(:))
)
Σ (h),))
1 1= exp
Since an advantage of our proposed method is its seamless integration capability with existing kNN-MT methods, we integrate our method with representative kNN-MT baselines and conduct ex- periments on five benchmark datasets from differ- ent domains. Experimental results demonstrate that our method consistently outperforms all the hase- lines across all these datasets, with improvements of up 1.6 BLEU points. Further ablation stud- ies justify that source context indeed plays a cru- cial role in all the three fundamental components. Furthermore, our source context-cnhancee kNN- MT method achieves significant improvements in
where i'is the temperature used smooth the ANN distribution, and the final probability distribution is obtained by interpolating the kNN distribution with the NMT distribution:
PHNN(U, & + (1 PNMT (
(3)
where X is the interpolation coefficient, and a larger means a larger percentage of the ANN distribu- tion in the final distribution.

---

Source-E Dource-Ebbaceed Retrieval
Source-Aware Calibration
Source-Angmented Interpolation
Construct
Train
Vinil sinon hoch Vogel tbegen schnell.
lastore
Estimate Interpolaticn Hyperrparamttrr
Meta-k Network
Set
Vögel fleege hoch.
Source Context
Retzieee Similar
Calibre
Sentenoes
Audiary Datastore
KNN Distribution
Retrieve
Vidte fleat
Vilue
03
float
75
fly
67
fly
6B
Pinel Distibuuiinn
Boat By
Original Datastore
Encoder
Decoder
NMT Distiluttion
By
Vógel fliegnn
Birds fiz
(1-
Sent Bl
Figure 2: An overview the proposed method. Source-context enhanced ANN-MT from three aspects, namely
retrieval, calibration, and interpolation.
3 ANN-MT via Enhanced Source Context
used to construct an auxiliary datastore D'. The construction method is identical to the approach described in Sec. 2. Note that we construct an adaptive auxiliary datastore independently for each input sentence.
In this section, we will detail the proposed method, which improves three erucial processes of ANN- MT, including retrieval, calibration, and interpola- tion with source context enhancement. The com- plcte process is illustrated Figure 2.
Finally, use h4 the query retrieve k near- entries from each of the datastores 'D and D', and combine them to obtain an entries list whose size 2k for constructing the ANN distribution.
3.1 Source-Enhanced Retrieval
the Vanilla ANN-MT (Khandelwal et al., 2021), the target-side representation is used as a query to retrieve entries similar to it from the constructed datastore, but the entries retrieved in this way may be affected by noise (Wang et al., 2022a,c) and some desired entries may be missed. Therefore. we adopt an adaptive approach by utilizing text similar to the input text to construct auxiliary datastores. The intuition is that if the source text similar, there is higher probability that the target text will also be similar. Consequently, the entries con- structed from similar source texts are more likely to be the ground truth than others (Figure 1). Specifically, we first encode the source texts x(6) from the training T using the encoder of the NMT model M, resulting in the representation r ()) E R where is the sentence length and H is the hidden size, then we take the average over the length dimension to get the final representation
3.2 Source-Aware Calibration
After source-enhancer KNN retrieval, a list of en- tries N = {(h"), ( is retrieved. These en- tries are now ranked based their target-side dis- tance d(h,, (()) However, since the NMT model used to compute the keys is not optimized for re- trieval, using target-side distance alone may result in suboptimal performance (Wang et al., 2022c). Therefore, we employ source-side distance to cal- ibrate the potentially inaccurate original distance (target-side distance), prioritizing entries that ex- hibit distance in both the target and source sides. Accordingly, the &NN distribution in Eq. 2 is also calibrated.
Specifically, we calculate the sum of the target- side distance d(,,, h()) and the source-side dis- tance d(r, r()) as a new distance metric de. To (6) E " Correspondingly, the input sentence to be translated is encoded into _
better control the weight assigned to the source- side distance in d,, we introduce a hyperparameter μ. and de is defined as
Then, calculate the L2 distance d(r, r() be- tween x and ( score the similarity between them. The set of sentences [x(") with the smallest L2 distance are selected, and their corre- sponding sentence pairs {x((1, y00) will be
(4) In order not to affect the speed of inference, we store the target token and its corresponding encoder
de d (1) d(r,r(

---

output I; when constructing the datastore?. The entries in the datastore are like (h, Ve, rt).
Dotaset
IT
Medical
Koran
Laws
JRC
JRC
Train
222.927
248,009
17,982
467,309
679,088
Dev Test
2,000
2,000
2,000
2,454
2,533
Since the hyperparameter μ directly affects the performance of the model, setting f as a fixed value may hinder the ability of the model to generalize effectively. Therefore, we propose lightweight module consisting of a two-layer feed-forward net- work to dynamically adjust the hyperparameter based on the retrieved entries:
2,000
2,000
2,000
2,000
2,433
2,966
Table
Statistics of datasets.
This ensures that the lightweight modules can dy- namically allocate the appropriate μ and À values based on the quality of the retrieval results.
2 = [s(1] (2R). (1)) .... d(2k), (2) ..... u (
Experiments
4.1 Experimental Settings
# sigmoid W (2((tnn(((() W Wiz)) + b1), (5)
Datasets and Evaluation In order to conduct more comprehensive comparison, we selected datasets separately from the direction of domain adaptation and in-domain translation. For do- main adaptation, we follow the same configura- tion as previous studics (Zheng et al., 202 la; Jiang et al., 2022), conducting experiments on four com- monly used benchmark domain datasets (Koehn and Knowles, 2017), including ITT Medical, Ko- ran, and Law. We use the split version from Aha- roni and Goldberg (2020) and the translation di- rection is En. For in-domain translation, we perform experiments in four directions on the JRC- Acquis (Steinberger et al., 2006) dataset, includ- ing En->De, De-En, En-+Es, and En-+En. The statistics of these datasets are shown in Table 1. All data are tokenized by Moses toolkit 3, and we use SacreBLEU 4 to evaluate the Performance
where (00 is source-side distance, de is our new distance (Eq. 4), 2(0 is the number of unique values in neighhors, and W1, 1- W2 parameter matrices. The distance metric de we propose here consid- ers both the target-side distance and the source-side distance. As the distances the retrieved entries are calibrated, the ANN distriction in Eq. is also calibrated accordingly. Subsequent experimental results further demonstrate that the new distance can effectively mitigate the noise and suboptimality issues caused by the target-only distance.
3.3 Source-Augmented Interpolation
Previous work has demonstrated that in the interpo- lation process of kNN-MT, using fixed interpola- tion coefficients À or retrieval quantities & will de- crease the model performance (Zheng et al., 2021 Jiang et ul., 2022) as they tend to treat retrieved information of different quality equally. Inspired by these works, propose source-augmentee lightweight module to adaptively compute À in the interpolation process:
Implementation We use kNN-BOX (Zhu et al., 2023b) implement multiple ANN-MT models. Faiss (Johnson et al., 2021) is employed for en- tries retrieval. For the four domain datasets, we use WMT 19 winner model (Ng et al., 2019) as the base translation model. For the JRC-Acquis dataset, we train Transformer (Vaswani et al., 2017) model for each translation direction, which served the base model. During the training pro- cess of the İNN-MT model, following the settings of the previous research, we use Adam (Kingma and Ba, 2015) as the optimizer and use the cross- entropy loss for optimization. The learning rate is set to 3 x 10-4, and the batch size is 32. The train- ing is stopped when the loss no longer decreases, and hyperparameters are adjusted based on the sults from the validation set. The specific model
2' = [s (1)
(2k), (1) .....
(2k) : (1) ..... u (2h))
A (igmoid (ettaann([3330) W42')) + b2), (6)
where Wa. W4 are parameter matrices, and s0,
de and ( are the same Eq. 5.
3.4 Training Procedure
During training, we freeze the parameters of the NMT model M and only update the parameters of the lightweight module (like W, and W2), min- imizing the cross-entropy loss until convergence.
In theory, this approach could require doubling the stor- age capacity. However, considering that tokens in the same sentence share the same source-side representation, we only utilize an additional 3% of storage space in our implementa- tion.
htt thub. "ht hhttps://gi wtt.getht..oc.oooooooeooooo com/mjposst/sarreblee. *The signature of ScareBl EEU is BI BLEUraase mi numrefs. I+smooth.cho sammotthexp=ttok 1 13atversion. 5.

---

configuration is presented in Appendix A. All ex- periments are conducted on an Ubuntu server with a RTX3090 GPU.
surpassing the strong baseline Robust &NN-MT 78). Furthermore, after enhancement, Robust kNN-MT can achieve higher BLEU score (49.66) on this dataset. Through an overall analysis of the four datasets, our method demonstrates a signif- icant improvement, with an average increase of 1.1 and 0.7 BLEU scores for Adaptive kNN-MT and Robust KNN-MT, respectively. These results demonstrate the effectiveness and the generality of our method, as it can be applied enhance many ANN-MT models and bring consistent performance improvements.
We use the following models our baselines:
Baselines
Base NMT (Ng et al., 2019): the pre-trained NMT model that is commonly used as the base model of kNN-MT.
Vanilla kNN-MT (Khandelwal et al., 2021): the base kNN-MT model that introduces the token level nearest neighbor method into NMT for the first time.
A question worth contemplating is why the en- hancement effect on the Koran dataset is not significant
Adaptive ANN-MT (Zheng et al., 2021a): the model innovatively introduces a lightweight trainable module to adaptively determine hy- perparameters such as k: and f during distri- bution interpolation.
other datasets, and the notable per- formance of the CL.NNN (Wang et al., 2022c) on this dataset provides some insights. By employing supervised contrastive learning, CLKNN enhances the distinctiveness of the representations keys in the datastore. However, we still utilize the outputs of the original NMT model as keys. Considering the sparsity of correct entries in such small datasets, retrieving the correct entries becomes even more challenging when using these indistinct keys for retrieval. Conversely, for larger-scale datasets, the abundance of correct entries may make it easier to retrieve the ground truth, and it can be observed that our enhancement method performs well on large-scale datasets.
KSTAR (Jiang et al., 2021): the model that adds learnable kernel to dynamically calcu- late the similarity of retrieved instances and smooth the obtained ANN distribution.
CLKNNN (Wang et al., 2022c): the model that enhances the distinetiveness of the represen- tation of keys in the datastore by employing supervised contrastive learning.
Robust ANN-MT (Jiang et al., 2022): the model that calibrates the kNN distribution and adds noise training to enhance the robustness.
In- In-Domain Translation The results on four do- main datasets demonstrate the ability of our method to enhance the domain knowledge transfer of kNN- MT. Furthermore, we aim demonstrate that the KNN-MT model can enhance the utilization of in- domain knowledge through token-level informa- tion retrieval, thereby mitigating issues such as knowledge forgetting, and our proposed method further strengthens this ability. To achieve this, in contrast using pre-trained NMT models do- main datasets, we utilized NMT models trained on the source domain specifically for the JRC-Acquis dataset.
4.2 Main Results
Domain Adaptation
The main results the four domain datasets are shown in Table 2, where the performance of the strong baseline before and after the enhancement of our method is listed. Note that due to Adaptive ANN-MT simultaneous adaptively determine both À and k, when enhancing Adap- tive &NN-MT using method, additionally incorporate the source-side distance as the input dynamically determine This differs from the approach described in Scc. 3.3, which focuses on adaptively calculating A. The same rationale ap- plies to Robust ANN-MT enhancement as well.
The results in Table 3 indicate that the ANN- MT model can effectively enhance the original NMT model by retrieving and utilizing in-domain knowledge but still lags behind several Translation Memory (TM) base strong baselines. After being enhanced by our method, the performance of ANN- MT has been significantly improved. For exam- ple, the Adaptive ANN-MT and Robust kNN-MT achieve 1.5 and 0.8 average BLEU improvement
It can be found that after being enhanced by method, two strong ANN-MT baselines (Zheng al., 2021a; Jiang et al., 2022) have gained con- sistent and significant performance improvement on all datasets. For example, the enhanced Adap- tive ANN-MT achieves an average increase of 1.57 BLEU points (47.74 to 49.31) 33))) on the IT dataset,

---

Model
Medical
Koran
Laws
Avg
Base-NMT
38.35
40.06
16.26
45.48
35.03
Vanilla ANN-MT
45.72
$4.26
20.29
61.27
45.38
KSTAR
47.72
56.81
19.97
63.34
46.96
CLKNNN
47.84
55.87
21.81
62.01
46.88
Adaptive KNN-MT
47.74
56.12
20.31
62.87
46.76
Ours
49.31 (+1.57)
57.721 (+1.60)
20.38 (+0.07)
64.14 (+1.27)
47.89 (+1.13)
Robust ANN-MT
48.78
57.11
20.50
63.61
47,50
Ours
49.66 (+0.88)
57.78| (+0.67)
20.72 (+0.22)
64.52' (+0,91)
48.17 (+0.67)
Table The BLEU scores on test sets of four domain benchmarks. "e" indicates the results from original papers. and "1" denotes statistically significant improvement (p 0.05).
Model
Es-+En
En-+Es
De-tEn
En-+De
Avg
(Zhang al., 2018)"
64.30
61.56
60.26
55.14
60.31
(Xia al., 2019)*
66.21
62.76
61.72
56.88
61.89
(Cai al., 2021)*
66.48
62.76
63.85
57.53
62.65
(Cheng al., 2022a))
67.76
64.04
64.33
58.69
63.70
Transformer
62.71
60.54
58.93
53.32
58.87
Vanilla ANN-MT
65.13
62.78
63.03
54.08
61.25
Adaptive ANN-MT
66.01
64.09
64,59
58.54
63.30
Ours
67.871 (+1.86)
65.53 (+1.44)
65.67! (+1.08)
60.31 (+1.47)
64.84 (+1.54)
Robust KNN-MT
67.08
64.97
64.93
59.52
64.13
Ours
67.95! (+0.87)
65.60' (+0.63)
65.73! (+0.80)
60.48'
(+0.96)
64.94 (+0.81)
Table The BLEU scores on test sets of JRC-As "x" highlights results provided by Cheng et al. (2022b), and "f" marks an improvement that is statistically significant 0.05).
Adaptive KNN-MT
47.74
the calibration module. This is because the model can only rely on target-side distances to determine the weights of retrieved entries without the source- aware calibration module, which makes the model susceptible to noise and leads to suboptimal per- formance. This finding further highlights the im- portance of comprehensive consideration of the source-side context ANN-MT.
w/ Retrieval
48.08 (+0.34)
w/ Calibration
48.63 (+0.89)
w/ Interpolation
48.25 (+0.51)
w/ All
48.89 (+1.15)
Table 4: The impact of each enhanced component on the overall performance.
It should noted that, for fair comparison, we set the value of & to that which performs optimally in Adaptive ANN-MT (k 8), rather than the value most suited for our approach. As result, the value of the "ww/ All" term in Table (48.89) does not attain our highest reported performance (49.31), Consequently, owing to the constraints imposed by smaller & value, the retrieval and interpolation modules, despite exhibiting performance improve- ments, have not been able to fully manifest their capabilities. Analysis
in four directions of JRC-Acquis and successfully surpassing these TM-base strong baselines, thus demonstrating the effectiveness of source-side enhancement.
4.3 Ablation Study
Our proposed method enhances three ANN-MT pro- cesses (retrieval, calibration, and interpolation). To investigate the utility of each enhanced module, we conducted ablation experiments on the IT dataset. Specially, we individually add each enhanced com- ponent and observed the performance changes. The results shown in Table 4.
In this section, we attempt to explore several im- portant and meaningful questions through experi- mental analysis.
can be observed that the performance of the model has improved after incorporating each en- hancement module, particularly with the most sig- nificant improvement seen after the inclusion of
Q1: Does the model enhanced with our

---

method demonstrate robustness when setting different values of k? (Sec. 5.1)
ENN
51
-US
Adaptive
Veiilh INN-MI
Vinilla ONN
Q2: Why does our method deliver perfor- mance gains? 5.2)
20
40.33
45
Can our proposed enhanced method be applied to other text generation tasks besides translation? (Sec.
Q3:
17
BLEU
5.1 Robustness (Q1)
41,0
Previous studies have found that the Vanilla kNN- MT model is highly sensitive to the setting of the retrieval entries number often achieving opti- mal performance only when & is set moderate value. This may limit the capacity of retrieval in- formation that KNN-MT can utilize, resulting in suboptimal performance. Some research enhanced the robustness of the ANN-MT model through su- pervised training to adaptively determine hyperpa- rameters (Zheng ct al., 2021a) a) or calibrate the kNN distribution (Jiang al., 2022).
Namber ofA
Nasion of
Figure 3: BLEU scores v.s. & on the IT dataset.
Robest INN 3ET Ours
Robaa LMS
Atlipinee 2814-TI
Aciepine 2NN6-MT
40
32
Number of
16
127
To demonstrate that the model enhanced by our method can achieve better robustness while improv- ing performance, following previous studies, we conducted experiments on the IT dataset, consider- ing different settings of k. Note that our enhanced model retrieves the number of entries from both the original datastore and the auxiliary datas- tore. To ensure fair comparison, in this section, when we mention the enhanced model retrieves k entries, it refers to the following scenario: the model retrieves k/2 entries from both the original and the auxiliary datastore and concatenates them to form a list of size k. The experimental results are shown in Figure 3.
1.5
155
20 Inference Speed (sents/s)
100
120
140
160
Figure 4: Inference speed v.s. k on the dataset.
cost remains roughly unchanged. a result, the overall additional computational overhead is min- imal. In Figure 4, we present the inference speed of various ÀNN-MT models and their correspond- ing enhanced models. We report the average speed of five trials for each setting. It can be observed that the enhanced model and the original model have comparable inference speeds, which justifies the theoretical analysis. Therefore, our proposed method achieves sustained and stable performance growth at a remarkably low cost.
We can observe that across all value settings of k. the model enhanced by the proposed enhanced method consistently surpasses the original model, demonstrating the effectiveness of our method, Moreover, the enhanced model can achieve robust performance A: increases. For example, the en- hanced Adaptive :NN-MT model maintains high- performance improvement increases, while the original model slowly declines after l = 8. This suggests that our method can effectively filter out noise in retrieval entries and appropriately utilize the relevant knowledge to enhance the translation model.
5.2 Accuracy (Q2)
To investigate the source of performance gains due to the source-side enhancement method, we ana- lyze it from the perspective of retrieval accuracy. A retrieved entry is considered correctly re- trieved its value matched the ground-truth token. The experiments are performed on the validation set of the dataset. Several metrics like P@n
Another key issue is the inference speed. Theo- retically, given the same value k; calibration and interpolation introduce only a marginal increase in computation, while the retrieval computational

---

Model
P@1
P@5
P@20
MAP@5
MAP@20
BLEU
Vanilla kNN-MT
0.3382
0.3176
0.2968
0.3262
0.3097
42.17
Adaptive KNN-MT
0.3408
0.3188
0.2971
0.3279
0.3106
43,98
Ours
0.3504
0.3405
0.3132
0.3492
0.3296
46.43
Robust KNN-MT
0.3496
0.3267
0.3038
0.3365
0.3181 0.3334
45.15
+ Ours
0.3621
0.3433
0.3174
0.3523
46.72
Table 5: The accuracy of retrieved entries of different ANN-MT models on IT validation set.
and MAP@n (the average of AP@ scores across all queries) are employed to evaluate retrieval ac- curacy. The results are shown in Table 5. It can be observed that the enhanced model surpasses the original model in all retrieval metrics, demon- strating that our method enables the ANN-MT to retrieve more correct entries, thereby leading to better translation results.
Model
DG
QG
Transformer
39.86
29.56
Vanilla ANN-MT
41.09
30.85
Adaptive ANN-MT +Ours
41.67
31.16
43.18 (+1.51)
32.09 (+0.93)
Robust ANN-MT +Ours
42.64
31.48
43,51 (+0.87)
32.26 (+0.78)
Table 6: The experimental results dialogue genera- tion (DG) and question generation (QG) tasks.
An interesting observation is that Adaptive kNN- MT, compared to Vanilla kNN-MT, shows min- imal improvement in retrieval metrics but a sig- nificant increase in the BLEU score. This might be attributed the content of the Adaptive &NN- MT, which enhances performance by adaptively adopting retrieval entries, and this improvement not related to retrieval accuracy. However, after our method enhancement, Adaptive KNN-MT's re- trieval accuracy is significantly improved and the performance is further enhanced. Combined with our proposed method like improved calibration dis- tance, suspect that the performance improve- ment is likely due to enhanced retrieval accuracy.
eration tasks. Additionally, by enhancing with our method, the performance of these ANN-MT models can be further improved, thus demonstrating the effectiveness and generality of our model.
Related Work
Existing approaches commonly improve ANN- MT (Khandelwal et al., 2021) from effectiveness and efficiency perspectives.
Effectiveness To enhance the effectiveness of ANN-MT, Zheng et al. (2021a) adaptively deter- mine the retrieved entries be utilized. Jiang al. (2021) incorporates kernel smoothing methods ÀNN-MT. Jiang al. (2022) calibrate &NN distri- bution and make more robust by noise-resistant training. Wang et al. (2022c) employs contrastive learning to make the keys of entries more discrimi- native to improve retrieval accuracy, while Li et al. (2022) utilizes representations from pre-trained models construet higher-quality datastore. It can be observed that these works lack explicit sideration of the source context. In contrast, we incorporate source context in all three processes of ANN-MT: retrieval, calibration, and interpolation, resulting in consistent performance improvements.
5.3 Generality (Q3)
To validate the generality of our proposed method, we conducted experiments on several text genera- tion tasks other than translation. We collect task- oriented dialogue generation and question genera- tion datasets from (Tang et al., 2022). Since these datasets did not provide testing set, we split the training set into a new training set and valida- tion set in 9 ratio and report the experimental results the original validation set. We trained a Transformer model as the base model on each dataset. The BLEUU metric is used to evaluate the performance of the models, and the results are re- ported in Table 6.
Efficiency Constrained by the performance quirements imposed by large datastore for infer- ence, researchers explore approaches enhance the efficiency of kNN-MT. Wang et al. (2022a) reduces the dimensionality of entry features and prunes redundant entries by clustering. Zhu et al.
Based on the results presented in Table 6, it can be observed that the kNN-MT method effectively enhances the performance of the model on these text generation datasets, indicating the potential of the &NN-MT method in non-tram con-translation text gen-

---

(2023a) explores the knowledge required by kNN- MT and filter out redundant entries according local correctness. Deguchi al. (2023) retrieves tokens from neighbor sentences instead of all sentences and using a lookup table for distance calculation. Dai et al. (2023) constructs a mini-datastor using sentences similar to the input during each inference instead of the original datastore and inspired to introduce the auxiliary datastore in our work. As shown in Figure 4, our method introdues limited extra computational cost, achieving good balance between effectiveness and efficiency.
Acknowledgement
This work is supported by the National Social Sci- ence Fund of China under Grant No. 20BTQ068. We thank all anonymous reviewers for their insight- ful suggestions on this work.
References
Roee Aharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In Proceedings the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7747- 7763, Online. Association for Computational Lin- guistics.
7 Conclusion
Deng Cai, Yan Wang. Huayang Li, Wai Lam, and Lemao Liu. 2021. Neural machine translation with monolingual translation memory. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1sth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7307-7318, 18, Online. Association for Computational Linguisties.
In this paper, we revisit the role of source con- text and propose a novel method for improving neural machine translation via source context en- hancement, which augments the key processes of KNN-MT (i.c., retrieval, calibration, and interpola- tion). Experimental results demonstrate that it can simply and effectively enhance most mainstream kNN-MT models, achieving consistent improve- ments. Morcover, our method can significantly improve the robustness of the ENN-MT model and help improve its effectiveness in dialogue genera- tion and question generation. We look forward to further exploring the practical utility and efficiency of this method in more natural language generation tasks in the future.
Xin Cheng, Shen Gao, Lemao Liu, Dongyan Zhao, and Rui Yan. 2022a. Neural machine translation with contrastive translation memories. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 3591-3601, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Xin Cheng, Shen Gao, Lemao Liu, Dongyan Zhao, and Rui Yan. 2022b. Neural machine translation with contrastive translation memories. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 3591-3601, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Limitation
While our method has exhibited effectiveness, its exploration beyond machine translation in terms of generality has been confined to just two additional text generation datasets, namely dialogue genera- tion and question generation. Consequently, it is imperative to investigate our method across a more diverse set of tasks. Existing research (Dai et al., 2023) indicates that the improvement brought by kNN-MT on certain large-scale translation datasets are not sufficiently significant. In future work, we aim to rigorously assess the effectiveness of our method on these large-scale translation datasets. Although our work achieves good balance be- tween effectiveness and efficiency, we believe there still room for improving efficiency without compro- mising effectiveness by leveraging source context, which will be explored in the future.
Yuhan Dai, Zhirui Zhang, Qiuzhi Liu, Qu Cui, Weihua Li, Yichao Du, and Tong Xu. 2023. Simple and scalable nearest neighbor machine translation. In Proceedings of the Eleventh International Conference on Learning Representations.
Hiroyuki Deguchi, Taro Watanabe, Yusuke Matsui, Masao Utiyama, Hideki Tanaka, and Eicchroo Sumita. 2023. Subset retrieval nearest neighbor machine translation. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 174-189, Toronto, Canada. Association for Computational Linguistics.
Yichao Du Weizhi Wang, Zhirui Zhang, Boxing Chen, Tong Xu. Jun Xie. and Enhong Chen. 2022. Non- parametric domain adaptation for end-to-end speech translation. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Process- ing, pages 306-320, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.

---

Hui Jiang, Ziyao Lu, Fandong Meng, Chulun Zhou, Jie Zhou, Degen Huang, and Jinsong Su. 2022. To- wards robust k-nearest-neighhor machine translation. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 1468-5477, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Demonstrations, pages 435-444, Abu Dhabi, UAE, Association for Computational Linguistics.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, Curran Associates, Inc.
Qingnan Jiang, Mingxuan Wang. Jun Cao. Shanbo Cheng, Shujian Huang, and Lei Li. 2021. Learning kernel-smoothed machine translation with retrieved examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7280-7290, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Dexin Wang. Kai Fan, Boxing Chen, and Deyi Xiong- 2022a. Efficient cluster-based k-nearest-neighhor machine translation. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2175- 2187, Dublin, Ireland. Association for Computational Linguistics.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535-547.
Dongqi Wang. Haoran Wei, Zhirui Zhang, Shujian Huang, Jun Xie, and Jiajun Chen. 2022b. Non- parametric online learning from human feedback for neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10): 14411 11439.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest neigh- bor machine translation. In International Conference on Learning Representations.
Qiang Wang, Rongxiang Weng, and Ming Chen. 2022c. Learning decoupled retrieval representation for near- est neighhour neural machine translation. In Proceed- ings of the 29th International Conference on Com- putational Linguistics, pages 5142-5147, Gyeongju, Republic of Korea, International Committee on Com- putational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego CA. USA, May 7-9, 2015, Conference Track Proceedings.
Philipp Koehn and Rebecca Knowles. 2017. Six chal- lenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39, Vancouver. Association for Computa- tional Linguistics.
Mengzhou Xia, Guoping Huang, Lemao Liu, and Shum- ing Shi. 2019. Graph based translation memory for neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):7297- 7304.
Jiahuan Li, Shanbo Cheng. Zewei Sun. Mingxuan Wang. and Shujian Huang. 2022. Better datastore, better translation: Generating datastores from pre-trained models for nearest neural machine translation. CoRR, abs/2212.08822.
Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Gra- ham Neubig, and Satoshi Nakamura. 2018. Guiding neural machine translation with retrieved translation pieces. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long Papers), pages 1335-1335, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Out, Michael Auli, and Sergey Fddunov. 2019. Facebook FAIR's ' WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). pages 314-319, Florence, Italy, Association for Computational Linguistics.
Xin Zheng, Zhirui Zhang, Junliang Guo, Shujian Huang, Boxing Chen, Weihua Luo, and Jiajun Chen. 2021a. Adaptive nearest neighbor machine translation. Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pages 668-374, Online. Association for Computational Linguistics.
Ralf Steinberger, Bruno Pouliquen, Anna Widiger, Camelia Ignat, Tomaž Frjavee, Dan Tuñiş, and Dániel Varga. 2006. The JRC-Acquis: A multilingual aligned parallel corpus with 20+ languages. In Pro- ceedings of the Fifth International Conference on Language Resources and Evaluation (LREC*06))
Xin Zheng, Zhirui Zhang, Shujian Huang, Boxing Chen, Jun Xie, Weihua Luo, and Jiajun Chen. 2021b. Non- parametric unsupervised domain adaptation for neu- ral machine translation. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2021, pages 4234 4241, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tianyi Tang, Junyi Li, Zhipeng Chen, Yiwen Hu, Zhuo- hao Yu, Wenxun Dai, Wayne Xin Zhao, Jian-yun Nie, and Ji-rong Wen. 2022. TextBox 2.0: A text gener- ation library with pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natraal Language Processing: System

---

Wenhao Zhu, Shujian Huang, Yunzhe Lv, Xin Zheng, and Jiajun Chen. 2023a. What knowledge is needed? towards explainable memory for kNN-MT domain adaptation. In Findings of the Association for Com- putational Linguistics: ACL 2023, pages 2824-2833, Toronto, Canada. Association for Computational Lin- guistics.
Wenhao Zhu, Qianfeng Zhao, Yunzhe Lv, Shujian Huang, Siheng Zhao, Sizhe Liu, and Jiajun Chen. 2023b. knn-box: A unified framework for nearest neighbor generation. CoRR, abs/2302.13574.

---

A Model Configuration
2017) model from scratch using the Subtitle train- ing set as the base model. The experimental results are reported in Table 9. can see that the two strong KNN-MT baselines remain effective on this larger dataset. Moreover, our method introduces significant improvements under both the Domain Adaptation and In Domain settings.
The specific hyperparameter settings of the Trans- former model trained on the JRC-Acquis dataset are presented in Table 7.
Hyperparameter
Value
Embedding Dim
512
Attention Head
8
Model
DA
II
Encoder Layer
6
Decoder Layer
6
Base Model
29.58
29.96
Dropout Hidden Dim
512
Adaptive &NN-MT
31.23
31.38
+ Ours
32.08" (+0.85)
32.47' (+1.09)
Table 7: The configuration of our Transformer model.
Robust ANN-MT
31.68
31.91
+ Ours
32.391 (+0.71)
32.66' (+0.75)
B Datastore
Table 9: The BLEU scores for the Subtitle dataset test set in the Domain Adaptation (DA) and In Domain (ID)
Following previous work (Zheng ct al., 2021a), we employ numpy. array 6 to store key-value pairs and utilized faiss' for index construction. The cluster centroids are set to 4, 096 and 32 during training and retrieval, respectively, while the vector code size is set to 64. The specific disk space occupied is presented in Table 8. can be observed that the storage occupied by the source context about 3% the total original storage, demonstrating the low space requirement of our work.
settings. "1" denotes statistically significant improve- ment (p < 0.05).
D Auxiliary Datastore
Note that the auxiliary datastore is constructed from the top n similar sentence pairs ranked by the source-side similarity between the input sen- tence and the source side of the sentence pairs. Therefore, we will recall different entries from the original and auxiliary datastores even if the same query is used. Experiments the development set of the IT dataset reveal that only 7.38% of tokens retrieved from the auxiliary datastore overlap with those from the original datastore. Here, "overlap" means the two tokens are identical and from the same sentence pairs. This result indicates that the auxiliary datastore has the potential to (1) introduce more unique tokens and (2) recall complementary references for the same tokens. In conjunction with our ablation study results on the retrieval mod- ule, this underscores the importance of considering source context when constructing datastore.
Datastore
IT
Medical
Koran
Law
Key-Value
13.22
1.00
Source Context
0.42
0.47
0.05
0.89
Table 8: Disk space (GB) occupied by different datas-
tores.
Experiments on Larger Datasets
Following the settings in recent strong base- lines (Zheng et al., 2021a; Jiang et al., 2022), we conduct experiments on the four datasets, i.e., IT, Medical, Koran, and Law, to make our work di- rectly comparable with these recent strong base- lines. We also conduct experiments on the Sub- titles dataset (Koehn and Knowles, 2017), whose training set consists of about 14 million parallel sentence pairs, two orders of magnitude larger than the previous four datasets. The translation direction is De-+En. For domain adaptation setting, we use the WMT19 De-> En direction winner model (Ng et al., 2019) as the base model. For in domain setting, we trained Transformer (Vaswani et al.,
Another noteworthy issue that the size of the aux- iliary datastore is about 4-5 orders of magnitude smaller than the size of the original datastore, for instance, 285 v.s. 3,602,862 on the IT dataset. This means that the construction and retrieval of the aux- iliary datastore necessitate minimal computational expenditure.
"httpss//nunmp"" org hattts://ggituub.cmm/aaaeeeeeeee http:///eaaaaabboeeeeereccceeeee


Reviews:
[{"review_id": "19iHOpFEZV", "reviewer": "Reviewer_aAZW", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question A: Experiments on larger datasets.**\n\n**Response A:**\n\n- Following the settings in recent strong baselines[1, 2], we conduct experiments on the four datasets, i.e., IT, Medical, Koran, and Law, to make our work directly comparable with these recent strong baselines. We fully agree with the reviewer's opinion that experiments on larger datasets will make the results more solid. Therefore, we conduct experiments on the Subtitles dataset[3], and use the split version from Aharoni and Goldberg[4], whose training set consists of **14M** parallel sentence pairs and is leveraged in the original kNN-MT paper[5].  **In summary, the results show that our method introduces significant improvements under both the Domain Adaptation and In Domain settings on this larger dataset**, justifying the effectiveness of our work. Note that we try our best but fail to provide experimental results under the WMT19 data setting[5] due to both resource limitation and reproduction problems, which will be explained at the end of this response.\n\n- **Dataset**: We conduct the experiments on the Subtitle dataset[3], which consists of 14M parallel sentence pairs, two orders of magnitude larger than the four datasets we originally used. The translation direction is de=>en.\n\n- **Results under Domain Adaptation Setting:** Experimental results on two versions of base models will be reported: \n\n  - The WMT19 de=>en direction winner model[6] (**WMT19-winner**), which is the standard setting leveraged in kNN-MT baselines;\n\n  - A model finetuned from WMT19-winner on the Subtitle training set (**WMT19-finetuned**), which simulates the scenario where the training data of the target domain is large.\n\n  The following table shows the results on the Subtitle evaluation set. We can see that the two strong kNN-MT baselines are still effective on the larger dataset. And our method demonstrates statistically significant improvement  ($p < 0.05$, indicated by \"*\") over the two baselines.\n\n  | Model                  | BLEU           |\n  | ---------------------- | -------------- |\n  | WMT19-winner           | 29.58          |\n  | Adaptive kNN-MT        | 31.23          |\n  | Adaptive kNN-MT + Ours | 32.08* (+0.85) |\n  | Robust kNN-MT          | 31.68          |\n  | Robust kNN-MT + Ours   | 32.39* (+0.71) |\n\n  Moreover, from the following table we can observe that finetuning a base model on the large training set of the target domain can achieve better results (29.58 v.s. 31.75). Both kNN-MT baselines surpass the performance of the stronger base model, which underscores the efficacy of the kNN-MT approaches. Furthermore, our methods also outperforms these kNN-MT baselines significantly ($p < 0.05$).\n\n  | Model                  | BLEU           |\n  | ---------------------- | -------------- |\n  | WMT19-finetuned        | 31.75          |\n  | Adaptive kNN-MT        | 32.66          |\n  | Adaptive kNN-MT + Ours | 33.35* (+0.69) |\n  | Robust kNN-MT          | 32.95          |\n  | Robust kNN-MT + Ours   | 33.57* (+0.62) |\n\n  **Above all, the results on larger datasets justifies the effectiveness of our method.**\n\n- **Results under In Domain Setting:** As the WMT19-finetuned model is finetuned on the Subtitle training set, it can also be viewed as an in-domain model of Subtitle. Consequently, the results in the second table in last section also justify that our method is effective for the in domain setting with large training data. Additionally, we trained a Transformer model from scratch using the Subtitle training set. The findings from this experiment, presented in the subsequent table, align with our earlier observations. Therefore, we think we can safely conclude that our method is also effective for the in domain setting.\n\n  | Model                  | BLEU           |\n  | ---------------------- | -------------- |\n  | Transformer            | 29.96          |\n  | Adaptive kNN-MT        | 31.38          |\n  | Adaptive kNN-MT + Ours | 32.47* (+1.09) |\n  | Robust kNN-MT          | 31.91          |\n  | Robust kNN-MT + Ours   | 32.66* (+0.75) |\n\n- **Comments on WMT19 experiments**: The WMT19 dataset leveraged in the orginal kNN-MT paper[5] is a much larger dataset. However, we fail to reproduce the result due to both resource and code problems. On the one hand, the raw datastore is about 2TB, and the corresponding index built by [faiss](https://github.com/facebookresearch/faiss) for retrieval would exceed 50GB, making it impossible to be loaded into our available GPUs. On the other hand, the code provided by the authors of kNN-MT[5] does not contain the code to reproduce the results on WMT19. We find other researchers also faced the above problems and struggle to reproduce the results (https://openreview.net/forum?id=uu1GBD9SlLe, ICLR 2023).\n\n**Question B:** **More details on the four domain datasets.**\n\n**Response B:**\n\n- These datasets are from [3]. We follow the same settings as Adaptive KNN-MT[1] and Robust kNN-MT[2], and use the split version from Aharoni and Goldberg[4]. The translation directions are German to English.\n\n**Question C: Overlap between the Law and JRC-Acquis datasets.**\n\n**Response C:**\n\n- Thank you for pointing out the overlap between the Law dataset and the JRC Acquis dataset. We will annotate this in our revised version, but we believe this may not affect our argumentation. In the Domain Adaptation setting, we use the WMT19 winner model as our base model, build a datastore using the Law dataset, and compare the performance differences between several baselines and the proposed method. This is to demonstrate that source context can enhance the Domain Adaptation capability of the kNN-MT model. In the In-Domain setting, we use the Transformer model trained on the JRC Acquis dataset as our base model and continue to construct a datastore using the JRC Acquis dataset. This is to justify that context can enhance the In-Domain capability of the kNN-MT model. It can be seen that due to different settings, the overlap between the Law and JRC Acquis datasets does not affect our argumentation.\n\n**Question D: Lack of significance test for experimental results.**\n\n**Response D:**\n\n- Thank you for the reminder. We have added a significance test as shown on the following table. The symbol * represents significant improvement (p<0.05). For all comparisons except for the Koran dataset, our proposed method is significantly better than the strong baselines, demonstrating the effectiveness of our method. We explained the reasons for insignificant performance improvement on the Koran dataset in Section 4.2 of our original submission.\n\n  | Model                  | IT     | Medical | Koran | Law    |\n  | ---------------------- | ------ | ------- | ----- | ------ |\n  | Adaptive kNN-MT        | 47.74  | 56.12   | 20.31 | 62.87  |\n  | Adaptive kNN-MT + Ours | 49.31* | 57.72*  | 20.38 | 64.14* |\n  | Robust kNN-MT          | 48.78  | 57.11   | 20.50 | 63.61  |\n  | Robust kNN-MT + Ours   | 49.66* | 57.78*  | 20.72 | 64.52* |\n\n  | Model                  | Es->En | En->Es | De->En | En->De |\n  | ---------------------- | ------ | ------ | ------ | ------ |\n  | Adaptive kNN-MT        | 66.01  | 64.09  | 64.59  | 58.54  |\n  | Adaptive kNN-MT + Ours | 67.87* | 65.53* | 65.67* | 60.31* |\n  | Robust kNN-MT          | 67.08  | 64.97  | 64.93  | 59.52  |\n  | Robust kNN-MT + Ours   | 67.95* | 65.60* | 65.73* | 60.48* |\n\n**Question E: Lack of relevant references like context-aware NMT.**\n\n**Response E:**\n\n- Thank you for the reminder. We will add relevant context-aware references to the revised version.  \n\n&nbsp;\n\n[1] Zheng X, Zhang Z, Guo J, et al. Adaptive Nearest Neighbor Machine Translation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 2021: 368-374. \n\n[2] Jiang H, Lu Z, Meng F, et al. Towards Robust k-Nearest-Neighbor Machine Translation[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 5468-5477. \n\n[3] Koehn P, Knowles R. Six Challenges for Neural Machine Translation[C]//Proceedings of the First Workshop on Neural Machine Translation. 2017: 28-39. \n\n[4] Aharoni R, Goldberg Y. Unsupervised Domain Clusters in Pretrained Language Models[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020: 7747-7763.\n\n[5] Khandelwal U, Fan A, Jurafsky D, et al. Nearest Neighbor Machine Translation[C]//International Conference on Learning Representations. 2021. \n\n[6] Ng N, Yee K, Baevski A, et al. Facebook FAIR’s WMT19 News Translation Task Submission[C]//Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). 2019: 314-319.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Perhaps an interesting add-on to the series of previous kNN-MT papers.", "reasons_to_reject": "Training models for high-resource languages like English, German and Spanish using tiny fractions of the available data and reporting any improvements over such models is not appropriate. All datasets used have few hundred thousands of parallel sentences while both EN-DE and EN-ES have close to a billion parallel sentences available on Opus. This makes the reader question if the proposed method would have changed anything at all when trained on reasonable scale data. There's even no need to go large since strong models can be trained in days on a single GPU with at least a few tens of millions parallel sentences.\n\nIt is very unclear what are the `four commonly benchmark domain datasets` mentioned on line 257. They are not common at all and the citation from line 255 is not the correct source here. The authors also do not specify translation directions or even languages for the four datasets. Only for JRC they do. \nWould this be the correct citation https://aclanthology.org/W17-3204/ ? In that case, the *law* part there is the same JRC Acquis. \n\nIn addition, when reporting BLEU score improvements of around 1-2 it would always be a good idea to perform statistical significance testing to make sure that the results are not just by chance. This has not been done in this paper. Not that it would offset using tiny amounts of training data for resource-rich languages, but would still make the results more credible.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a method for using source context in nearest neighbor machine translation (kNN-MT). The authors compare their proposed method against other kNN-MT models and the vanilla Transformer model on several languages and domains, and report constant improvements in BLEU scores.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Some related work in general context-aware NMT. There are papers by Sennrich, Voita, and many others."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Having the BLEU score improvement mentioned in the abstract does not say much without knowing the involved languages, translation direction and domain. Either add more details or remove. The introduction is also not the best place for mentioning it (line 103)."}}, {"review_id": "83Xs6Rp2qM", "reviewer": "Reviewer_qmbQ", "summary": "", "questions": "- A.  In line 170, which encoder state is used to create the datastore? Is it a sentence or token-level representation? If it is a sentence-level representation, how can you generate it from encoder states? If it is token-level, which encoder state do you use as a query during inference?\n- B. In line 223, what does \"the number of unique values in neighbors\" mean? Does it mean unique tokens?\n- C. In line 489, please clarify or cite how P@n and MAP@n are calculated.\n- D. Have you conducted experiments with distant language pair such as English-Chinese, or English-Japanese? It would be nice to see whether the proposed method works well even if the source and target representation can be largely different.\n- E. How do the tokens retrieved from the auxiliary datastore and original datastore overlap? I'm curious how unique tokens can be retrieved by using the auxiliary datastore.\n- F. I understand that your method constructs the auxiliary datastore for each sentence. How heavy is it to construct?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question A: In line 170, which encoder state is used to create the datastore? Is it a sentence or token-level representation? If it is a sentence-level representation, how can you generate it from encoder states? If it is token-level, which encoder state do you use as a query during inference?**\n\n**Response A:**\n\n- In our implementation, we use the encoder of the NMT model to encode the source sentence, resulting in a tensor of size ($L$, $H$), where L is the sentence length and $H$ is the hidden size. We take the average over the length dimension to get a representation of size ($1$, $H$) denoted as $r$. This serves as the context representation of the source sentence and is used as a query. We will provide more details in the revised version.\n\n**Question B: In line 223, what does \"the number of unique values in neighbors\" mean? Does it mean unique tokens?**\n\n**Response B:**\n\n- This concept comes from a previous work Adaptive KNN-MT[1]. It refers to the number of unique tokens. For example, if $k$=4, and the retrieved set of entries is (203, 203, 203, 837), the unique tokens are 203 and 837. Another set of entries, (203, 203, 508, 837), would have unique tokens 203, 508, and 837. Previous research[1] found that unique tokens affect confidence calculations. We will elaborate on the revised version.\n\n**Question C: In line 489, please clarify or cite how P@n and MAP@n are calculated.**\n\n**Response C:**\n\n- P@n refers to precision at $n$, which is widely used in information retrieval[2]. Suppose there are $c$ ground truth tokens in the top $n$ retrieved entries, then we have $P@n=\\frac{c}{n}$. And the average value of P@n over the entire evaluation dataset is reported in our paper.\n- AP@n is defined as $AP@n = \\frac{1}{NF} \\sum_{i=1}^{n} \\operatorname{rel}(i) \\times P@i$, where $rel(i)$ is an indicator function that equals 1 when the $i$-th retrieved entry is a ground truth token, and 0 otherwise, and $NF$ is the total number of ground truth tokens in the top $n$ retrieved entries[3]. MAP@n is defined as the mean of AP@n values across $Q$ retrievals[4]: $MAP@n = \\frac{1}{Q} \\sum_{q=1}^{Q} (AP@n)_q$\n\n- Thanks for your suggestion. We will provide more details in the revised version.\n\n**Question D: Have you conducted experiments with distant language pairs such as English-Chinese, or English-Japanese? It would be nice to see whether the proposed method works well even if the source and target representation can be largely different.**\n\n**Response D:**\n\n- This is an excellent suggestion. Due to resource limitations, we were unable to conduct these experiments at the time of submission. We will try our best to add these experiments to the revised version.\n\n**Question E: How do the tokens retrieved from the** **auxiliary datastore and original datastore overlap? I'm curious how unique tokens can be retrieved by using the auxiliary datastore.**\n\n**Response E:**\n\n- Note that the auxiliary datastore is constructed from the top $m$ similar sentence pairs **ranked by the source-side similarity** between the input sentence and the source side of the sentence pairs. Therefore, we will recall different entries from these two datastores even if the same query is used. Experiments on the development set of the IT dataset reveal that only 7.38% of tokens retrieved from the auxiliary datastore overlap with those from the original datastore. Here, \"overlap\" means the two tokens are identical and from the same sentence pairs. This result indicates that the auxiliary datastore has the potential to (1) introduce more unique tokens and (2) recall complementary reference for the same tokens. In conjunction with our ablation study results on this module, this underscores the importance of considering source context when constructing datastore.\n\n**Question F: I understand that your method constructs the auxiliary datastore for each sentence. How heavy is it to construct?**\n\n**Response F:**\n\n- In our implementation, for an input source sentence, we retrieve $m$ similar sentences to construct the auxiliary datastore (to ensure enough entries for retrieval, we usually set $m$=$k$, where $k$ is the number of entries retrieved). As shown in Section 5.1 of our paper, its impact on efficiency is negligible. This is because the size of the auxiliary datastore is about 4-5 orders of magnitude smaller than different from the original datastore, for instance 285 vs 3,602,862 on the IT dataset.\n\n**Question G: It might be nice to cite this concurrent work, which uses source-side information to filter the datastore. https://aclanthology.org/2023.acl-long.10/**\n\n**Response G:**\n\n- We will add an introduction to this related work in the revised version. Thanks for the suggestion.\n\n&nbsp;\n\n[1] Zheng X, Zhang Z, Guo J, et al. Adaptive Nearest Neighbor Machine Translation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 2021: 368-374. \n\n[2] Craswell, N. (2009). Precision at n. In: LIU, L., ÖZSU, M.T. (eds) Encyclopedia of Database Systems. Springer, Boston, MA. \n\n[3] Craswell, N., Robertson, S. (2009). Average Precision at n. In: LIU, L., ÖZSU, M.T. (eds) Encyclopedia of Database Systems. Springer, Boston, MA.  \n\n[4] Beitzel, S.M., Jensen, E.C., Frieder, O. (2009). MAP. In: LIU, L., ÖZSU, M.T. (eds) Encyclopedia of Database Systems. Springer, Boston, MA.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed source-context aware kNN-MT method is exciting and consistently improves performance on machine translation, dialogue generation, and question generation tasks.\n- The proposed method also improves the robustness of the number of retrieved tokens (k).\n- The comparison of inference speed shows the proposed method does not drastically slow the inference, which is essential for practical use.", "reasons_to_reject": "- Some details of the proposed method are unclear to me, and it might make it difficult to reproduce. See Questions for more details.\n", "questions_to_authors": "- A.  In line 170, which encoder state is used to create the datastore? Is it a sentence or token-level representation? If it is a sentence-level representation, how can you generate it from encoder states? If it is token-level, which encoder state do you use as a query during inference?\n- B. In line 223, what does \"the number of unique values in neighbors\" mean? Does it mean unique tokens?\n- C. In line 489, please clarify or cite how P@n and MAP@n are calculated.\n- D. Have you conducted experiments with distant language pair such as English-Chinese, or English-Japanese? It would be nice to see whether the proposed method works well even if the source and target representation can be largely different.\n- E. How do the tokens retrieved from the auxiliary datastore and original datastore overlap? I'm curious how unique tokens can be retrieved by using the auxiliary datastore.\n- F. I understand that your method constructs the auxiliary datastore for each sentence. How heavy is it to construct?", "paper_topic_and_main_contributions": "The conventional kNN-MT only uses decoder-side information to retrieve the kNN tokens from the datastore. However, the source-side context should be considered as well.\nTherefore, this paper proposes a new method to incorporate the source-side contexts into kNN-MT.\n\nThe proposed method comprises source-enhanced retrieval, source-aware calibration, and source-augmented interpolation.\nThe source-enhanced retrieval adds a new auxiliary datastore, constructed and searched by encoder-side information.\nDuring the inference, the model uses the conventional decoder-based and the new encoder-based auxiliary datastore.\nThe source-aware calibration modifies the distance function, which is used to score the retrieved kNN tokens, to consider source-side distance.\nThe source-augmented interpolation adaptively computes $\\lambda$, which weighs the kNN tokens and original NMT scores.\n\nThey conducted experiments mainly on machine translation tasks. They confirmed that the new source-context aware module can combine with several kNN-MT models and consistently improves both the domain adaptation and in-domain translation tasks.\nAblation study shows the calibration module most contributes to the accuracy.\n\nAnalysis reveals how the new method is robust to different hyper-parameter settings, comparison of inference speed, retrieved tokens accuracy, and the results on the other two tasks.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- It might be nice to cite this concurrent work, which uses source-side information to filter the datastore. https://aclanthology.org/2023.acl-long.10/"}, "extra_text": {}}, {"review_id": "ODZ5lx2Zpm", "reviewer": "Reviewer_4RbX", "summary": "", "questions": "A. Can you clarify details regarding evaluation?\n\nB. Can you clarify details regarding method?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question A: Fairness of the comparisons**\n\n**Response A:**\n\n- We find the reviewer has big concern on the fairness of the comparisons in our paper. **We believe that they are indeed fair.** We will delve into the specifics in the subsequent paragraphs.\n\n- **Evaluation Metric is Aligned**\n\n  - The signature of ScareBLEU is `BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.5.1`, which is the same as the baselines. \n\n- **Baseline Results can be Compared Directly:** \n\n  - **Domain Adaptation (Refer to Table 2 in the original submission):** Apart from CLKNN, the results for all other models come from our implementation. Notably, the original KSTAR paper trained the KSTAR model using Transformer, we implement it with the WMT19 winner model[1] to ensure a fair comparison, which might explain the improved performance over the original paper. The subsequent table show the differences between our implement results and the original papers.\n\n    | Model                                                   | IT    | Medical | Koran | Law   | Avg   |\n    | ------------------------------------------------------- | ----- | ------- | ----- | ----- | ----- |\n    | Base NMT                                                | 37.98 | 39.91   | 16.30 | 45.71 | 34.97 |\n    | Base NMT (Our Implementation)                           | 38.35 | 40.06   | 16.26 | 45.48 | 35.03 |\n    |                                                         |       |         |       |       |       |\n    | Vanilla kNN-MT                                          | 45.82 | 54.35   | 19.45 | 61.78 | 45.35 |\n    | Vanilla kNN-MT (Our Implementation)                     | 45.72 | 54.26   | 20.29 | 61.27 | 45.38 |\n    |                                                         |       |         |       |       |       |\n    | KSTAR                                                   | 35.74 | 53.40   | 16.97 | 59.41 | 41.38 |\n    | KSTAR (Our Implementation)                              | 47.72 | 56.81   | 19.97 | 63.34 | 46.96 |\n    |                                                         |       |         |       |       |       |\n    | Adaptive kNN-MT[2]                                      | 48.04 | 56.41   | 21.09 | 63.21 | 47.18 |\n    | Adaptive kNN-MT (Robust kNN-MT Author's Implementation) | 47.88 | 56.10   | 20.43 | 63.20 | 46.90 |\n    | Adaptive kNN-MT (INK Author's Implementation)           | 47.37 | 56.21   | 20.44 | 63.13 | 46.79 |\n    | Adaptive kNN-MT (Our Implementation)                    | 47.74 | 56.12   | 20.31 | 62.87 | 46.76 |\n    |                                                         |       |         |       |       |       |\n    | Robust kNN-MT[3]                                        | 48.90 | 57.28   | 20.71 | 64.07 | 47.74 |\n    | Robust kNN-MT (INK Author's Implementation)             | 48.50 | 57.12   | 20.81 | 63.74 | 47.54 |\n    | Robust kNN-MT (Our Implementation)                      | 48.78 | 57.11   | 20.50 | 63.61 | 47.50 |\n\n  - It can be seen the average BLEU score difference between our results and the original paper across the four datasets is within 0.2 BLEU score. Except for Adaptive kNN-MT, for which we additionally cited Robust kNN-MT[3] and INK[4] for their implementation results. Their results are similar to ours. For example, INK's implementation average BLEU is 46.79, and ours is 46.76.\n\n  - **In Domain (Refer to Table 3 in the original submission):** The first four baselines reused the results from Cheng et al.[5]. The other methods, including Transformer, Vanilla kNN-MT, Adaptive kNN-MT, Adaptive kNN-MT + Ours, Robust kNN-MT, and Robust kNN-MT + Ours, were implemented by us. We intend to showcase the performance improvement our method can bring by comparing model with or without our method. It can be seen that under the same implementation conditions, Adaptive kNN-MT + Ours has an average increase of 1.54 BLEU in four directions, and Robust kNN-MT + Ours has an average increase of 0.81 BLEU, showing the effectiveness of our proposed method.\n\n- **Hyperparameters for the Baselines are also Optimized:** \n\n  - For Adaptive kNN-MT, the best values for $k$ and $T$ were determined through grid search on the development set. For Adaptive kNN-MT + Ours, the best values for $k$ and $T$ were also determined through grid search on the development set.\n  - For Robust kNN-MT, the best value for $k$ was determined through search on the development set ($T$  value is dynamically calculated). For Robust kNN-MT + Ours, the best value for $k$ was determined through search on the development set ($T$ value is dynamically calculated).\n\n- Therefore, we are confident that our comparisons are fair, the results are trustworthy, and the observed improvements are substantial.\n\n**Question B: Can you clarify details regarding method (how to calculate the distance between representations r in the auxiliary datastore, how the lightweight module is trained)?**\n\n**Response B:**\n\n- **Distance Computation between Representations in Auxiliary Datastore:** We encode the source sentence $x$ using the NMT model's Encoder module to obtain a tensor of size ($L$, $H$), where $L$ is the sentence length, and H is the hidden size. We average over the length dimension to get a representation $r$ of size ($1$, $H$), which serves as the context representation of the source sentence. Similarly, the retrieved sentence can also be transformed into a representation $r^j$ of size ($1$, $H$) in a similar manner. We then calculate the L2 distance between them to obtain similarity.\n- **Training of the Lightweight Module:** During training, we freeze the parameters of the NMT model and only update the parameters of the lightweight module ($W_1$, $W_2$, and $b_1$), minimizing the cross-entropy loss until convergence. This ensures that the lightweight module can dynamically allocate the appropriate lambda value based on the quality of the retrieval results.\n- We will add these details to the revised version. Thank you for the suggestion. \n\n**Question C: Grammatical errors and presentation improvements.**\n\n**Response C:**\n\n- Thank you for your detailed suggestions. We will fix them in the revised version.\n\n&nbsp;\n\n[1] Ng N, Yee K, Baevski A, et al. Facebook FAIR’s WMT19 News Translation Task Submission[C]//Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). 2019: 314-319. \n\n[2] Zheng X, Zhang Z, Guo J, et al. Adaptive Nearest Neighbor Machine Translation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 2021: 368-374. \n\n[3] Jiang H, Lu Z, Meng F, et al. Towards Robust k-Nearest-Neighbor Machine Translation[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 5468-5477. \n\n[4] Zhu W, Xu J, Huang S, et al. INK: Injecting kNN Knowledge in Nearest Neighbor Machine Translation[J]//Proceedings of the 2023 Annual Meeting of the Association for Computational Linguistics. 2023: 15948–15959. \n\n[5] Cheng X, Gao S, Liu L, et al. Neural Machine Translation with Contrastive Translation Memories[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 3591-3601.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Using source context for kNN-MT makes a lot of sense, but is an under-explored direction. The authors provide good motivation for this direction, and results seem promising.\n* The method is quite comprehensive. Source context is used in multiple kNN-MT components, and an ablation indicates that it brings benefits to every component, and more so when combined.\n* Inference speed and memory requirements are only marginally increased compared to earlier work.\n* The paper is well-written and easy to follow, thanks to clear structuring and a clear overview figure.", "reasons_to_reject": "* Evaluation misses important details\n\nThe SacreBLEU signature is missing, so it is not clear whether the comparison with the baselines is fair. From the text I understand that for all models reported in Tables 2 and 3, except Transformer* and your own models, you report scores from the original papers? But these scores do not match the numbers reported in the papers. For instance, the numbers reported for adaptive kNN-MT do not match the numbers in the original adaptive kNN-MT paper.\n\nWithout additional information and context, it is hard to draw any conclusions from these numbers.\n\n* Some important method details are missing.\n\nIt is unclear how distance d(r,r^j) (line 173) between x and x^j is calculated, given that x and x^j in general do not have the same length. Similar for d(h_t, h^i) (line 202).\n\nIt is unclear how the lightweight module that adjusts hyperparameter \\mu is optimized/learned. In particular, how are the weights of W_1, W_2, and b_1 learned? Similar for equation 6.\n\n\\lambda is computed adaptively for your method. what about other hyperparameters such as number of neighbors k and temperature T, are they found using a hyperparameter search? If so, which values do you search over? What about for the other methods you compare to, which sets do you use there? Without this information, it is not possible to determine the amount of hyperparameter tuning for your method vs the baselines.", "questions_to_authors": "A. Can you clarify details regarding evaluation?\n\nB. Can you clarify details regarding method?", "paper_topic_and_main_contributions": "This paper investigates the role of source context in kNN-MT. The authors argue that source context warrants further attention, since it plays an important role in kNN-MT but is mostly overlooked by earlier work. \n\nThe authors propose methods to incorporate source context in multiple kNN-MT components:\n* Retrieval. An auxiliary datastore based on similarity with current input sentence is created, and used in conjunction with the vanilla datastore.\n* Calibration. Instead of relying solely on target side distance, the method also takes into account source side distance with the goal of removing noisy matches.\n* Interpolation. \\lambda is computed adaptively based on source similarity (amongst other factors).\n\nResults seem to improve over strong baselines, while inference speed and memory requirements only marginally increase.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Line 123: \"it is feed into\" -> \"it is fed into\"\n* Line 135: \"is calculates\" -> \"is calculated\"\n* Line 168-171: it's confusing to use x^{(j)} and x to both denote a source sentence.\n* Line 240: \"these work\" -> \"these works\"\n* Line 248: \"Experiment\" -> Experiments\n* Line 388: \"To deeply investigate\" -> \"to investigate\"\n* Line 420-427: it would be clearer to choose better names for the subsections instead of this bulleted list. E.g., \"5.1 How robust is the model for different values of k?\". Similarly for the other subsections. Then the bulleted list is not necessary anymore and can be removed.\n* Line 462-463: \"This proves\" -> \"This suggests\" (an empirical observation is not a proof.)\n* Line 751: \"employe\" -> \"employ\""}}]

Author rebuttal:

Meta-review: The work augments kNN-MT with source context enhancement, including a distance calibration module. Such a method has the benefit of being easily integrated into existing kN-MT setups, and the authors test this across various settings. The reviewers express some concerns, such as more competitive baselines (especially in non-artificial low-resource settings), and the authors present interesting results in the rebuttal. I'd recommend the authors also include some sense of the variation in BLEU, to better understand if improvements like +0.6 are significant.
Decision: Accept-Main