Simple and Scalable Nearest Neighbor Machine Translation
Authors: Yuhan Dai
Zhirui Zhang
Qiuzhi Liu
Qu Cui
Weihua Li
Yichao Du
Tong Xu
Venue: ICLR 2023 poster (2023)
Keywords: Nearest Neighbor
Machine Translation

Abstract: $k$NN-MT is a straightforward yet powerful approach for fast domain adaptation, which directly plugs the pre-trained neural machine translation (NMT) models with domain-specific token-level $k$-nearest-neighbor ($k$NN) retrieval to achieve domain adaptation without retraining. Despite being conceptually attractive, $k$NN-MT is burdened with massive storage requirements and high computational complexity since it conducts nearest neighbor searches over the entire reference corpus. In this paper, we propose a simple and scalable nearest neighbor machine translation framework to drastically promote the decoding and storage efficiency of $k$NN-based models while maintaining the translation performance. To this end, we dynamically construct a extremely small datastore for each input via sentence-level retrieval to avoid searching the entire datastore in vanilla $k$NN-MT, based on which we further introduce a distance-aware adapter to adaptively incorporate the $k$NN retrieval results into the pre-trained NMT models. Experiments on machine translation in two general settings, static domain adaptation, and online learning, demonstrate that our proposed approach not only achieves almost 90% speed as the NMT model without performance degradation, but also significantly reduces the storage requirements of $k$NN-MT. 

Published as a conference paper at ICLR 2023
SIMPLE AND SCALABLE NEAREST NEIGHBOR MA- CHINE TRANSLATION
Yuhan D Zhirui Zhangi,, Qiuzhi Liu', Qu Cui', Weihua Li", Yichao Du' and Tong Xu Universitty of Science and Technology of China 'Tencent Al Lab fdirkiedyel gmail. com, duyichao uuscc.euu en, tongxuluste. edu. cn A #zrustcll 1 @mmallll....... com, {qiuzhiliu, qucui, weihuali Setennentttteeeettteemeem com.
ABSTRACT
&NN-MT (Khandelwal et al. 2021 straightforward yet powerful approach for fast domain adaptation, which directly plugs pre-trained neural machine trans- lation (NMT) models with domain-specific token-level k-nearest-neighhor (kNN) retrieval to achieve domain adaptation without retraining. Despite being conceptu- ally attractive, kNN-MT is burdened with massive storage requirements and high computational complexity since it conducts nearest neighbor searches over the entire reference corpus. In this paper, we propose a simple and scalable nearest neighbor machine translation framework to drastically promote the decoding and storage efficiency of ANN-based models while maintaining the translation perfor- mance. To this end, we dynamically construct an extremely small datastore for each input via sentence-level retrieval to avoid searching the entire datastore in vanilla ANN-MT, based on which we further introduce distance-awe aware adapter to adaptively incorporate the &NN retrieval results into the pre- tre-trained NMT mod- els. Experiments on machine translation in two general settings, static domain adaptation, and online learning, demonstrate that our proposed approach not only achieves almost 90% speed as the NMT model without performance degradation, but also significantly reduces the storage requirements of ANN-MT.
INTRODUCTION
Domain adaptation is one of the fundamental challenges in machine learning which aspires to cope with the discrepancy across domain distributions and improve the generality of the trained models. has attracted wide attention in the neural machine translation (NMT) area (Britz et al.
2017 7 Chen
et al. 2017]CCu & |ang] BBapaa& & Firat. 2019 Bapna et al.] 2019 Wei et al. 2020). Recently, MNN-MT and its variants (Khandelwal et al., 2021; Zheng et al. 202 bj Wang et al., 2022a)
provide new paradigm and have achieved remarkable performance for fast domain adaptation by retrieval pipelines. These approaches combine traditional NMT models (Bahdanau et al., 2015
Vaswani et al., 2017) with a token-level n--narest-neiihhbuur (kNN) retrieval mechanism, allowing to directly access the domain-specific datastore to improve translation accuracy without fine-tuning the entire model. By virtue of this promising ability, single kNN-MT can be seamlessly generalized to other domains by simply altering the external knowledge it attends to.
In spite of significant achievements and potential benefits, the critical bottleneck of KNN-MT is its large token-level external knowledge (also called a datastore), which brings massive storage overhead and high latency during inference. For instance, |Khandelwal et al. 2021 found that KNN-MT is two orders of magnitude slower than the base NMT system in generation speed when retrieving 64 keys from datastore containing billions of records. To case this drawback and make KNN search more efficient, line of works (Martins et al.) 2022b) Wang et al. (022a) proposed methods to reduce the volume of datastore, such as pruning the redundant records and reducing the dimension of keys. On another line, Meng al. (2022) designed Fast ANN-MT to construct a smaller datastore for each source sentence instead of consulting the entire datastore. Typically, the small datastore is constructed by searching for the nearest token-level neighbors of the source tokens and mapping them to the corresponding target tokens. However, in essence, Fast &NN-MT
"Equal contribution.

---

Published as a conference paper at ICLR 2023
Table 1: Analysis on samples involved in KNN retrieval (k=8) when translating multi-domain German-to Germannto-Englishh datasets (IT, Medical and Law). The left column includes the statistics of full data and correspondent kNN-MT performance, while the right column shows the averaged number of sentence pairs (Sents) and target tokens (Tokens), the storage overhead of datastore (Datastore) and correspondent performance of our proposed method (SK-MT) on these samples, respectively.
Full
Involved Samples During Inference
Domain
Sents
Tokens
Datastore
kNN-MT
Sents
Tokens
Datastore
SK-MT
IT Medical
223k
3.6M
6,9G
45.9
7.1
249
0.46M
46.3
248k
6.9M
14,0G
54.2
9.0
358
0.68M
57.8
Law
467k
19.0M
37.0G
61.3
14.2
730
L.5M
62.7
migrates the inefficient &NN retrieval from the target side to the source side, resulting in still two times slower decoding speed than the standard NMT model. Despite its failure to speed up, the insights behind Fast 6NN-MT inspire that it is avoidable to leverage the entire datastore for nearest neighbor search. In addition to decoding acceleration, little attention has been paid to the storage overhead of &NN-MT, which also limits its practicability and promotion on real-world online services. Concretely, ANN-MT requires nearly 37GB of disk storage for creating a datastore from 467k law-domain sentence pairs, while the plain text takes less than 100MB, meaning prohibitive storage requirements when applied to a larger corpus. Both factors enlighten us to reflect on the necessity of constructing a complete datastore for kNN-MT.
To this end, for each test input, we deeply investigate the involved sentence pairs during the decoding process of &NN-MT, and the results are shown Table We collect the minimum sentence pairs that cover all decoding steps of kNN-MT on multi-domain German-to-Engliss datasets, including IT, Medical and Law. Interestingly, we discover that during the whole decoding process, scarce number of training samples (i.e., less than 16 samples on average) in the domain-specifer reference corpus are required for each test sentence, while the storage overhead of the datastore built by these samples is negligible. Moreover, combining the dynamic datastore in ANN-MT is capable of achieving superior performance than vanilla 6NN-MT. This phenomenon provides a new perspective that it is feasible to perform sentence-level retrieval to obtain small number of similar samples for ENN-MT generation.
Following this line, propose simple and scalable nearest neighbor machine translation frame- work (SK-MT) to handle the limitations of A-N-MT while keeping its performance. This approach first leverages current efficient text retrieval mechanisms, such as BM25 (Robertson & Zaragoza 2009), to obtain a small number of reference samples that have high overlaps with the input sen- tence, and then dynamically construct a tiny datastore by forwarding the samples the pre-trained model. In this way, we avoid searching through the whole datastore for nearest neighbors and dras- tically improve the decoding and storage efficiency of ANN-MT. Based on that, distance-aware adapter is introduced to adaptively incorporate the ANN retrieval results into the pre-trained NMT models. We conduct experiments in two settings, including static domain adaptation and online learning. Experimental results demonstrate that our proposed approach not only achieves almost 90% speed as the NMT model without performance degradation, but also significantly loosens the storage requirements of ANN-MT. Our code is open-sourced on httpss/giitub.com/ddikkedaaa/
2 BACKGROUND: ANN-MT
Recently, Khandelwal et al. 2021 proposed KNN-MT that augments pre-trained NMT models with a translation memory retriever, empowering models to attend to external knowledge and improve translation performance. It is generally formulated in two processes: datastore construction and inference with ANN retrieval.
Datastore Construction. The datastore is a translation memory which converts bilingual sentence pairs into a set of key-value pairs. Given a reference corpus (z, v) E (X.3), the pre-trained NMT model generates the context representation fo v-ce) at each timestep t. Then collect the output hidden state fo, Vee) as key and ye as value to construct the whole datastore (K,, V):
(C. = U (Uo(I, u.). Vy, E y
(1)
(x.))((2)))
2

---

Published as a conference paper at ICLR 2023
(4) Dynamic Dalastore Creation
Bilingusi Reference Corpus
Datastore
Nearest Neighbors
PENN
L2
Value
Multiple
NMT Model
10
Samples
80
Text Retrieval
Carvaris
Impfstoff
Source Sentence
query
A 9(46)
Context
Representation
PNMT
Prediction
x
Cervarix Impfstoff
Cevaric
NMT Model
vaccine
y/1
(b) Inference with Adaptive ANN Retrieval
Figure 1: An illustration of our method consisting of two modules, including dynamic datastore construction and inference with adaptive kNN retrieval.
Inference with &NN Retrieval. At the 1-th decoding step, given the already generated words Dec. the current context representation [a(2, (<x) is leveraged to generate a retrieval distribution PANN(yol2, (<t) over the entire vocabulary:
d(hs, fo fo )2)))
(eN(()l2,0e))) x
Σ Iy exp(
(Nu,
where the .).,.) stands for Euclidean distance function and is the temperature to control the sharp- ness of softmax function. The final prediction distribution enhances vanilla NMT distribution pNMt with the retrieval distribution PENN, and formally calculated as:
2 )c)) (p (c.) (1 X)pNMT(Jelz, Pcr),
where À is tuned interpolation coefficient. In consideration of beam search in the inference phase, the ANN search over the entire datastore (K., V) applied each decoding step of all beams, resulting in the total time complexity of ( Bl), where B denotes the beam size and I denotes the target length. Moreover, the storage requirement of datastore leads to the space complexity of OIK]))), where h stands for the dimension of hidden representations. The decisive factor in time and space complexity the datastore size /C],, therefore the growing of |K] will engender massive storage overhead and high latency. In practice, kNN-MT usually adopts the FAISS (Johnson et al.) 2021) toolkit for efficient similarity approximate search and clustering of dense vectors. However, when given extensive datastore (e.g., billions of records), decoding and storage efficiency is still unsatisfactory Khandelwal et al. 2021)
METHODOLOGY
In this work, we provide a new perspective that leverages sentence-level retrieval to dynamically build extremely small datastore for each test sentence, instead of constructing complete data- store for kNN-MT.. It is inspired by an interesting phenomenon that only few training samples in the domain-specifie reference corpus involved during the decoding process of &NN-MT for each test input, as shown in Table T We design a simple and scalable nearest neighbor machine transla- tion framework (SK-MT) handle the shortcoming of kNN-MT while keeping its performance in fast domain adaptation. As illustrated in Figure I the overall framework can be decomposed into two main processes, i.e. dynamic datastore construction and inference with adaptive &NN retrieval. Next, we will give detailed view of these two modules.
3.1 DYNAMIC DATASTORE CONSTRUCTION
In our framework, adopting a dynamic datastore instead of static and extensive datastore used in conventional ANN-MT plays vital role in fast domain adaptation. Its benefits are mainly two-fold: 1) it filters out the majority of noises in the original datastore advance and forces the model to attend the most helpful records, and 2) it limits the nearest neighbor search to extremely small
3

---

Published as a conference paper at ICLR 2023
datastore, thus improving inference efficiency and reducing storage requirements to large extent. Specifically, for each test sentence z, we leverage an existing search engine (c.g., ElasticScarct") to retrieve the top-64 (default setup) bilingual sentence pairs {(x', y² ] with the highest relevance score from the training corpus. We implement the relevance score as BM25 (Robertson & Zaragoza 2000), popular and competitive approach to text ranking tasks. Then in pursuit of selecting better output hypothesis, we employ the following similarity to re-rank the retrieved bilingual sentences and maintain top-m bilingual reference samples {(x), y)))
sim (x,x") = =1 dist(ar, x²)
max(|a], (=d)))
where dist(.;.) denotes the edit-distance, and æf, y are the retrieved source and target sentences from the training data, respectively. These sentence pairs are utilized to create a small datastore
(Kug Va) for the source sentence x as Equation by forwarding the trimmed corpus to the pre- trained NMT model:
(K.,, V.) - U (e).) E'}..
(5)
15ism
In this way, the time and space complexity of KNN-MT reduced to O[[K, and O(K,, respectively, where |Kzz << IC].
3.2 INFERENCE WITH ADAPTIVE KNN RETRIEVAL
Since we build significantly small datastore from several similar bilingual sentences, is inevitable to introduce irrelevant sub-units (chunks or words) for the nearest neighbor search. Under this circumstance, arbitrarily introducing kNN distribution with fixed coefficient À will take the risk of being interfered by the noises in retrieved neighbors. Moreover, it is unnecessary to incorporate ANN distribution when the pre-trained NMT model has higher confidence in predicting the next word, otherwise, it tends to harm the performance. To ease the bias of using a fixed interpolation coefficient in kNN-MT, prior works (Zheng al., (021a) train light-weigh network to predict the coefficient X, which decides the engagement of the ENN module in an adaptive manner. In this work, extremely simplify the process by proposing to determine À linear the normalized distance, which is effective but requires no further training. Precisely, we explicitly calculate À by:
x = g(do) = ReLU(1 do
(6)
where is the temperature parameter defined in Equation 2 and da represents the top-1 distance during nearest neighbor search. In this way, the engagement of kNN distribution is ignored when the distance is larger than the temperature value т, and it is magnified when records are relevant enough to the current query.
3.3 DISCUSSIONS
To our knowledge, the most relevant work in ANN-MT area to our study is Fast İNN-MT (Meng al. 2022), which proposes to construct different datastores for each source sentence by token-level prefiltering, However, it still suffers from constructing the entire datastore in advance, resulting in less decoding and storage efficiency than our SK-MT method that adopts a dynamic datastore. SK- MT can also be viewed pruning approaches, such as random pruning (Khandelwal et al. (02T), greedy merging (He et al, 2021) and bbased pruning (Wang et al. 2022a).. Our approach benefits from the fact that the storage overhead of text data is extremely more negligible than dense vectors, and thus we provide a practical version of ANN-MT in real applications. Another main advantage of our method is its scalability in terms of reference fetching (owing to the advanced text retrieval system) and translation (owing the scalability of reference samples). such as insertions, deletions, or substitutions by operating the reference samples, while the same process is quite costly for a k-Ne- method that requires a pre-defined datastore.
novel sentence-level pruning strategy, parallel to previous token-level
effortless perform operations on a datastore
4

---

as SK- Published as a conference paper at ICLR 2023
Interestingly, the retrieval-then generation paradigm inspired by ANN-MT analysis also falls into translation-mematt memory area (Gu et al. 2018 Zhang et al.] 2018a Xia et al. 2019, Cai et al. 2021). Our approach is somewhat in a similar framework of Gu et al. (2018) but introduces ENN retrieval to achieve shallow fusion. In this manner, we inherit the advantage of kNN-MT that we do not need extra training, and leverage sentence-level text retrieval from the conslation-memon area to improve KNN-MT's efficiency. We also find that SK-MT achieves better or comparable performance than ofate-of-the-ar memsmry methods, which are included in the Appendix B
EXPERIMENTS
We evaluate the effectiveness of our proposed SK-MT in two settings: 1) domain adaptation, in which pre-trained general-domain NMT model is used to translate domain-specific sentences with KNN searching over an in-domain datastore; 2) online learning from human feedback in the human- in-the-loop machine translation, where the pre-trained NMT model incrementally takes human- corrected samples into account during the decoding process.
4.1 DOMAIN ADAPTATION
Dataset and Evaluation. For the domain adaptation task, we use the same multi-domain dataset as the baseline (Khandelwal et al. 2021) and consider IT,, Medical, Koran, and Law in our experiments. The statistics of the dataset are shown in Appendix A. For performance evaluation, we adopt
SacreBLEU (Post 2018 2 and ChrF (Popovic_201) 2015).
Baselines.
We compare our method (SK-MT) with several baselines: 1 NMT: We adopt the
1M -nnglshh news translation task winner model (Ng et al. 2019) as the pre-trained
NMT model. 2 ANN-MT: Following Khandelwal et al. (2021), we incorporate the datastore built
from domain datasets into the pre-trained model. 3 AK-MT: variant of KNN-MT (Zheng al.
2021a). where the aperrparameeer & is adaptively selected. ④4) FK-MT: Fast KNN-MT proposed by
Meng et al. (2022) constructs smaller datastore for each source sentence by searching for the near- est token-level neighbors. S EK-MT: An efficient ANN-MT proposed by Martins et al. (202bb)
which explores several approaches for acceleration. 6 CK-MT: Chunk-based KNN-MT (Martins
et al. 2022a) retrieves chunks of tokens from the datastore, instead of single token.
Implementation Details. In our experiments, we adopt the THUMT (Zhang et al. 2020 toolkit for our model implementation. For each sentence in the test set, we retrieve 64 sentences with the highest BM25 scores from the training corpus and perform re-ranking according to the metric defined in Equation 4 We maintain the top-m bilingual sentences as our reference corpus, which are utilized to build dynamic datastore with the hidden dimension set to 1024. During inference, we carefully tune the hyper-parameters on the development set by performing a grid search on k E 21, 2, 3, 4), m (1, 4, 8, 16) and r E (5. 10, 20, 50, 100, 150, 200). Based on the validation results, we select two widely-used model architectures in our experiments, m 2, k 16, k 2 as SK-MT,, where the temperature
MT, and and length penalty are set to 4 and 0.6 for all datasets. To replicate other k-N- based baselines, we utilize FAISS (Johnson et al. 2021) for efficient &NN retrieval, and we learn 4096 cluster centroids for ANN retrieval and search 32 clusters for each target token.
are both set to 100. The beam size
T Selection
Table Grid search on and A on IT development set with the temperature r fixed to 100. The * marks the two selected models (SK-MT, and SK-MT,) in our experiments.
64
BLEU Chrf
62
BLEU
ChiF
61
0
2
3
4
1
4
60
42.7
42.2
41.9
40.8
61.7
61.3
61.0
60.5
43.4"
43.0
42.5
41.6
62.1"
61.7
61.5
60.9
36
100
200
43.5
43.8
43.0
42.3
62.2
62.1
61.7
61.3
T
43.3
43.9
43.4
42.7
62.1
62.3
62.0
61.4
Figure 2: Temperature selection on 1T
16
43.3
43.9
43.6
43.0
62.0
62.3*
62.1
61.8
development set.
http://ghthhb.cmmmijjsssaecehe
5

---

Published as a conference paper at ICLR 2023
Table 2: BLEU(1) and ChrF(t) on multi-domain sets, including IT, Medical, Koran and Law.
BLEU
ChrF Koran
Model
IT
Medical
Koran
Law
Avg.
IT
Medical
Law
Avg.
NMT
39.1
41.8
16.9
45.9
35.9
58.9
61.4
39.8
66.0
56.5
KNN-MT
45.9
54.2
20.4
61.3
45.5
63.3
69.5
41.3
76.0
62.5
AK-MT
46.9
56.4
20.3
62.6
46.6
64.4
71.0
41.7
76.9
63.5
FK-MT
45.5
53.6
21.2
56.0
44.1
EK-MT
44.4
51.9
20.1
57.8
43.6
CK-MT
44.2
53.1
19.3
59.7
44.1
SK-MT,
46.1
56.8
17.6
60.7
45.4
63.6
70.7
40.6
75.5
62.5
w/o adapter
40.6
47.0
18.4
52.3
39.6
59.7
63.4
40.9
69.1
58.3
SK-MT,
46.2
57.6
19.5
62.3
46.4
64.0
71.3
41.5
76.4
63.3
w/o adapter
41.3
51.2
20.5
56.3
42.3
60.4
66.0
42.3
71.0
59.9
Table 3: Storage overhead and inference speed on Law test set.
Storage Overhead
Inference Speed (ms/sentence)
Model
Datastore
FAISS Index
GPU
batch=1
batch=4
batch=8
batch=16
NMT
300.8 (x1.00)
158.2 (x1.00)
85.1 (x1.00)
79.1 (x1.00)
1986.4 (x0.15)
749.5 (x0.21)
467.4 (x0.18)
410.5 (x0.19)
ANN-MT
37.0G
1.3G
409.6 (x0.73)
195.3 [x0.81)
110.5 (x0.77)
100.5 (x0.79)
SK-MT;
0.16M
344.4 (x0.87)
184.8 (x0.86)
94.6 (x0.90)
85.0 (x0.93)
SK-MT,
1.34M
430.5 (x0.70)
225.1 (x0.70)
131.1 (x0.65)
136.2 (x0.58)
Main Results. In Table we verify the performance of SK-MT on the multi-domain dataset. As for SK-MT, that refers to only = 2 samples instead of the whole training set as vanilla NNN- MT does, we do not notice significant performance degradation both in terms of BLEU and ChrF. Surprisingly, with a higher amount of reference samples, SK-MT, substantially outperforms vanilla ANN-MT by point of both BLEU and ChrF on average and achieves comparable performance to AK-MT, which is regarded the state-of-the SK-MT, and SK-MT, surpass all the efficient methods (FK-MT, EK-MT, and CK-MT) by a large margin, illustrating that framework is capable of attaining high effectiveness while is designed for efficient decoding as those three approaches. Additionally, we conduct an ablation study on our adapter with dynamic parameter strategy, which indicates the benefit of adjusting À in adaptive manner. We also carry out experiments on a more general setting, i.c., WMT' 14 news translation task, but find that kNN-based methods could not achieve significant improvements. We believe that low sentence-lewe similarity greatly attributes to the unfavourable performance on the WMT' 14 dataset. More results and discussions are included in Appendix C
6NN-MT method. It is remarkable outcome that
The Effect of Hyper-pane parameters. The translation quality of ANN-MT is susceptible to its hyper- parameters X, T, k, which are reduced to 7 and only our SK-MT framework. The number of reference samples m is also an essential hyper-parameter that determines the richness of reference samples and the size of the dynamic datastore.
a straightforward intuition that the increment of tends to generate better translations but requires more decoding time. Therefore, carefully selecting m is of vital importance because it balances the effectiveness and efficiency of our pro- posed method. To achieve better trade-off, search on the hyper-parameters, as illustrated in Table 4 and Figure we can see that our proposed SK-MT, achieves the best performance on the IT development set. More details can be found in Appendix A..3 It is worthwhile to notice that SK-MT, where k and m 2 achieves compara- ble performance the best SK-MT, with conceptually improved efficiency, which will be verified experimentally in the next section.
consider 16 the maximum of m.. With grid
Decoding Speed and Storage Overhead. In this section, we conduct experiments on comparing the inference speed" and storage overhead with other basclines. Considering the completeness and fairness of speed comparison, we test the speeds of all the models with various batch sizes, including 1,4,8, and 16. The hardware use is 11 12 cores of Intel(R) Xeon(R) Gold 6258R CPU and single
In experiment, the preliminary time spent text retrieval of SK-MT is negligible.

---

Published as a conference paper at ICLR 2023
IT
Medical
Koran
Law
210k
2:10
210
2:10
(10"20")
p00.10'y
[10P
[L0.10%)
[10.10%)
(10.10))
10%)
10)
[5.10]
[5.10]
[5.10]
3-
KMT
1OIT SKAMT
MMT
NMT
JAN
ANMMT
MT:
SK-MI
SK-MT;
S6ATT
0.5
0.6
0.7 Accuracy
0.8
0.9
0.5
0.7
0.8
0.9
1.0
0.5
0.2
01.
0.6
0.8
0.
0.6
0.7
0.8
0.9
1.0
Accuracy
Accuracy
Accuracy
Figure 3: Word accuracy on multi-domain dataset.
GeForce RTX 2080 Ti GPU. As shown in Table 3 our proposed light- SK-MT, achieves almost 90% decoding speed as the NMT model and has higher speed than ANN-MT with GPU acceleration. Not surprisingly, the decoding speed SK-MT will decrease along with the increment of m, because typically all the reference samples are passed forward to the NMT models to construct a datastore. In addition, kNN-MT typically takes 37G of space to create its datastore, while this storage requirement is avoidable for SK-MT. Both the time and space efficiency of our method further amplified when given an extensive datastore (e.g., datastore built from 14). More details can be found in Appendix C
Word Accuracy Analysis. It is observed that directly utilizing general models to translate out- of-domain sentences or rare words can cause severe performance degradation (Koehn & Knowles) 2017 Dou et al.] 2019). We suspect ANN-based models alleviate this issue by introducing external information of oft-of-domain and rare words to the pre-trained NMT models. To analyze this prob- lem, we adopt COMPARE-MT (Neubig al.. 2019) compute the accuracy of words at different frequencies (y-axis). As illustrated in Figure 3 from the perspective of word accuracy, kNN-based methods substantially outperform NMT, especially for the rare words at the frequency
10. It indicates that the excellent adaptation ability of kNN-based models is partially owing to their richer knowledge of low-frequency words. On average. SK-MT, performs slightly better than SK-MT,, and perhaps it is because the increment of reference samples covers more rare words in the datastore.
4.2 ONLINE LEARNING
Task Description. We verify the feasibility of applying our proposed approach to machine translation with human feed- back, a canonical incremental adaptation task. In this scenario, human experts are involved in post-editing translation hypothe- ses, and in turn, the revised translations are fed to the MT system for adaptation and improvement. One of the greatest challenges to previous work is that those approaches gener- ally require adapting the online models to new new-coming samples constantly, which conduces to significant inefficiency in prac- tice. Worse still, they are suffering from catastrophic forgetting problems. As for our framework, we follow the research line of non-part non-parametric online learning methods (Wang et al. 2022b). which adopt external knowledge to memorize human feedback.
SRC: Ambri tet et Impletot MT. Amboor is vitus
Translate
NHT Model
Adapt
Ambrlx oh PE: Ambbor is vacoin
Figure 4: An illustration of the sce- nario of machine translation with human feedback.
Dataset and Baselines. For the online learning task, we adopt two widely-used document-level datasets, i.c., European Medicines Agency (EMEA) dataset (Tiedemann) 2009) and JRCAcquis cor- pus (Steinberger et al. 2006). Following previous work (Wang et al. 2022b). we divide the doc- uments into five buckets based on their lengths (0-50, 50-100, 100-200, 200-500 and 500-1000). We compare our method with several representative kinds of research, including NMT (Ng et al.) 2019). ANN-MT (Khandelwal et al., (021)) and KoK (Wang et al.] (022b). More details of dataset and implementation are shown in Appendix A. and A.2
7

---

Published as a conference paper at ICLR 2023
Table 5: B1EU(1) and ChrF(t) on EMEA and JRC-Acquis datasets.
BLEU
ChrF
Model
[0,
[50,
[100,
1200,
[500,
[0,
[50.
[100,
1200)
[500.
Full
Full
50)
100)
200)
500)
1000)
50)
100)
200)
500)
1000)
EMEA
NMT
44,2
43.2
38.3
42.4
40.6
41.7
64.8
63.6
61.5
63.4
64.8
64.1
ANN-MT
43.6
43.4
39.9
43.8
43.8
43.4
63.5
63.7
61.8
64
65.9
64.6
KoK
44.4
44.6
44.1
45.7
53.7
49.2
65.0
65.4
64.4
65.5
71.5
68.2
SK-MT1
45.5
44.8
43.4
45.6
53.2
49.2
65.3
64.7
64.3
65.5
71.6
68.3
SK-MT
46.1
45.6
43.8
46.3
53.6
49.7
65.8
65.1
64.6
65.8
71.8
68.6
JRC-Accs AcC-us
NMT
54.1
50.0
42.2
39.9
43.4
44.5
72.2
70.2
65.9
62.9
65.6
66.4
ANN-MT
55.5
52.2
45.7
43.6
47.7
48.1
72.0
70.7
67.8
65.1
68.3
68.3
KoK
56.3
52.4
47.7
44.7
50.1
49.8
73.9
72.0
69.2
66.1
70.2
69.9
SK-MT,
56.6
52.8
47.2
43.5
47.8
48.5
74.0
72.0
69.0
65.33
68.8
69.1
SK-MT,
57.4
53.7
48.2
44.7
49.5
49.8
74.5
72.5
69.4
66.0
69.7
69.8
EMEA: (50,100)
EMEA: (100.200)
EMEA: (200,500)
EMEA: [500.1 1000)
30
00
NH1T
180
NNIT
180
HNT
2004
JNN-
JAN
Rok
Noc
SK-MTT
SKATT:
EIAMT:
R valuu
SK-VTT
value
SKATT
value
DIC-TT
70
70-
00
INRUMT
60
60
60
16K
SKATT
SKATT
30
50-
R
R
Rs.
Ri Ro.
Re..
Ri Ro-s Ro..
Re.
JRC: (50,100)
JRC: (100.200)
JRC:
(200.500)
JRC: (500,1000
NVTT
90-
SI1T
UNNAIT
1N84AI
KUK
R valuu
R valuue
R valuu
SK-MF;
71-
70-
WMT
NVT
60-1
UNVATI
60-
KGK
MT.:
BK VT. SPM
GH-MT
Ry-0
Ro.
Ro
R
Re-
Ry.
Rg.
R.
Ro-s Ro-s
Ro.
17.
Ro-s Re- Ro.
Figure 5: Results of R-indicator on documents with (50, 100), (100,200), [200, 500), and (500.1000)
buckets from EMEA and JRC-Acquis.
Main Results. We adopt the same performance evaluation as domain adaptation: BLEU and ChrF. Considering the fairness and completeness of the results, we also measure corpus-level BLEU and ChrF by concatenating the translations all the documents in each bucket. As illustrated in Table 5 the improvement of KoK over the ANN-MT baseline suggests that using dynamic coefficient X for interpolation helps produce better translation quality, which is also claimed in Wang eet al. (202bb). With the benefit of an adaptive X, SK-MT,, not only exceeds ANN-MT by large margin as expected, but achieves equivalent or even better performance than KoK on both EMEA and JRC-Acquis datasets. Similar domain adaptation, SK-MT, is better than ANN-MT in terms of BLEU and ChrF when utilized in online learning setting. Moreover, the improvements in various buckets of length demonstrate the effectiveness and generalization of our method.
Zero-Shot and Few-Shot Ability. Following Wang et al. (2022b), we evaluate the adaptation
speed of different approaches to human feedback using R R-indicator (Simianer al. 2019), which measures the translation recall of words with different occurrence times in users' feedback. For the j-th pair of bilingual pairs sentences from the corpus (æzz Vj) (X, )²), we let R.,; represent unique words in the reference yj that are their (i + 1)-th occurrence in the whole document. We denote RG as the recall of tokens that have appeared times in the previous corrected sentences:
Σ
IN 1=1
HH RR
R₂
Rayl
8

---

Published as a conference paper at ICLR 2023
where N denotes the corpus size (X, } and H, stands for unique words in the j-th hypothesis 8j- Speeifically, Ru evaluates the tokens that first appear in the translating document and R1 considers those that have appeared once. We conduct experiments on documents with [50, 100),[100, 200), (200, 500) and (500, 1000) buckets from EMEA and JRC-Acquis datasets, and compute Ro. R1. R2-55 Re and Rg+. As shown in Figure [ in all the settings, SK-MT shows roughly the same trend as KoK, where Rt value of SK-MT improves rapidly and outperforms NMT and ANN-MT baselines. It indicates that SK-MT and KoK adapt to helpful human feedback faster.
5 RELATED WORK
Domain Adaptation for NMT. Domain adaptation aspires to cope with the discrepancy across domain distributions and improve the generality of the trained models. The domain adaptation approaches in the MT field are mainly divided into two categories: 1) architecture-
which lypically adds trainable parameters to the NMT model for adaptation. (Wang et al. 2017 Wuebker et al. 2018 [Bapna& & Firat 2019] Guo et al. 2021) 2) data-centric, which fine-tunes the NMT model using the domain-specific corpora. In the absence of a pre-defined in-domain corpus, the parallel data can be selected from larger generie corpus (Del et al., (021), generated from monolingual corpus through forward- or back-translation (Zhang et al.] 2018b) Poncelas & Way [219 Wei et al.)
2020), or synthesized from lexicon or template (Hu et al. [019 Peng al.) 2020). Recently, non-parametric methods provide a new paradigm for domain adaptation by retrieving the datastore of similar instances (Khandelwal al. [221] Zheng et al.) 202 Jiang et al. (202I). We follow this
research line and propose a more effective and efficient kNN-based framework.
Nearest Neighbor Translation. Non-parametric approaches that incorporate external knowledge into the pre-trained models through a retrieval pipeline have attracted wide attention from natural language processing areas, including language modeling (Khandelwal et al., 2020), machine transla-
tion KKandeewal et al. 2002 Zheng et al. 202 Jiang et 2021]). question answering (Guu et al,
2020
2021 Thulke al. 2021 For the NMT system, Khandelwal et al. (2021) first propose ANN- -T. a non- -on-paraett
Lewis et al. 2020 Xiong et al. 2022 and dialogue generation (Fan et al.
approach that plugs ANN classifier over large datastore with traditional NMT models (Bahdanau jet al. 2015; Vaswani et al., 2017: Hassan et al.) 2018) to achieve significant improvement. In ad- al. (2021a) propose to dynamically determine the number of retrieved tokens to consider at each step. Martins et al. (2022a) attempt to retrieve chunks of tokens from the datastore, instead of a single token. Due to its scalability and adaptability, ANN-MT have also shown great potential for unsupervised domain adaptation (Zheng et al., 2021b Du et al. 2022) and online learn- ing (Wang et al., 2022b). Despite its great success, ANN- suffers from large storage overhead and low decoding speed. Recently, several works are proposed to promote its practicality. |Meng jet al.] (2022) design Fast ANN-MT to reduce the decoding time, in which they construct different datastores for each source sentence by searching for the neighbours of the source tokens. Martins ctt al. (2022b) investigate adaptive retrieval, datastore pruning and dimension reduction proposed in He et al. 202II, and further design a retrieval distributions cache to speed-up decoding. |Wang et al. (2022a) adopt lightweight neural network to reduce the dimension of keys and further leverage cluster-based pruning reduce retrieval redundancy. Different from previous methods, we propose to construct a dynamic datastore with a extremely minor size, which avoids ANN search over the entire datastore and thus dramatically improves the time and space efficiency.
dition, ZZheng
6 CONCLUSION AND FUTURE WORK
In this paper, we present SK-MT, simple and scalable nearest neighbor machine translation ap- proach for fast domain adaptation. By constructing a dynamic datastore and introducing a distance- aware adapter for inference, we are able to produce equivalent or even superior performance than previous kNN-based approaches. Moreover, experimental results on domain adaptation and online learning settings demonstrate that our framework does not require any extra training and is efficient in both decoding time and storage overhead. It is promising that our proposed SK-MT has a wide range of applications not limited ANN-MT discussed in the paper. In the future, we would like to explore the fcasibility of SK-MT when applied in baN-based methods such as RNN- (Khandel- wal et al. 2020), or other sequence-to-sequence tasks.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their helpful feedback. We appreciate Lemao Liu and Hongkun Hao for the fruitful discussions and dataset sharing. This work was done when the first author was an intern at Tencent AI Lab and supported by the grants from National Natural Sci- ence Foundation of China (No.62222213, 62072423), and the USTC Research Funds of the Double First-Class Initiative (No. MDD11000200090 Tong Xu is the corresponding author.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In ICLR, 2015.
Ankur Bapna and Orhan Firat. Non-parametric adaptation for neural machine translation. In NAACL, 2019.
Ankur Bapna, N. Arivazhagan, and Orhan Firat. Simple, scalable adaptation for neural machine translation. In EMNLP, 2019.
Denny Britz, Quoc Le, and Reid Pryzant. Effective domain mixing for neural machine translation. In WMT, 2017.
Deng Cai, Yan Wang, Huayang Li, Wai Lam, and Lemao Liu. Neural machine translation with monolingual translation memory. In ACL, 2021.
Boxing Chen, Colin Cherry, George Foster, and Samuel Larkin. Cost weighting for neural ma- chine translation domain adaptation. In NMT@ACL, 2017.
Chenhui Chu and Rui Wang. A survey of domain adaptation for neural machine translation. In COLING, 2018.
Maksym Del, Elizaveta Korotkova, and Mark Fishel. Translation transformers rediscover inherent data domains. In WMT, 2021.
Zi-Yi Dou, Junjie Hu, Antonios Anastasopoulos. and Graham Neubig, Unsupervised domain adap- tation for neural machine translation with domain-aware feature embeddings. EMNLP, 2019.
Yichao Du, Weizhi Wang. Zhirui Zhang, Boxing Chen, Tong Xu, Jun Xie, and Enhong Chen. Non- parametric domain adaptation for end-to-end speech translation. In EMNLP, 2022.
Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. Augmenting transformers with
banebased composite memory for dialog. TACL, 2021.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li. Search engine guided neural machine translation. In AAAI, 2018.
Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In ACI. 2021.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wes Wei Chang. Realm: Retrieval- augmented language model pre-training. ArXiv, abs/2002.08909, 2020.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt William D. Lewis, Mu Li, Shujie Liu, Tie- Van Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingee Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. Achieving human parity automatic chinese to english news translation. ArXiv, abs/ 180300560557, 2018.
Junxian He, Graham Neubig, and Taylor Kirg-Kirkpatrick. models. In EMNLP, 2021.
Efficient nearest neighbor language
Junjie Hu, M. Xia, Graham Neubig, and Jaime G. Carbonell. Domain adaptation neural machine translation by lexicon induction. In ACI, 2019.
10

---

Published as a conference paper at ICLR 2023
Qingnan Jiang, Mingxuan Wang, Jun Cao, Shanbo Cheng, Shujian Huang, and Lei Li. Learning smoothed machine translation with retrieved examples. In EMNLP, 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Trans. Big Data, 7:535-547, 547, 2021.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In ICLR, 2020.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Nearest neigh- bor machine translation. ICLR, 2021.
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. In NMT@ACL, 2017.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. detrieval-augmennt generation for knowledge knowledge-intensive intensive nlp tasks. ArXiv, abs/2005, 14011, 2020.
Pedro Henrique Martins, Zita Marinho, and André Martins. Chunk-basee nearest neighbor machine translation. ArXiv, abs/2205. 12230, 2022a.
Pedro Henrique Martins, Zita Marinho, and André Martins, Efficient machine translation domain adaptation. ArXiv, 2tss/2204.11608, 2022b.
Yuxian Meng, Xiaoya Li, Xiayu Zheng, Fei Wu, Xiaofei Sun, Tianwei Zhang, and Jiwei Li. Fast nearest neighbor machine translation. In ACL (Findings), 2022.
Graham Neubig, Zi-Y Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. compare-mt: A tool for holistic comparison of language generation systems, ArXiv, abs/1903.07926, 2019.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. Facebook fair's Wmt19 news translation task submission. ArXiv, abs/1907.06611 2019.
Wei Peng, Chongxuan Huang, Tianhao Li, Yun Chen, and Qun Liu. Dictionary- based data augmen- tation for domminn neural machine translation. ArXiv, abs/2004.02577, 2020.
Alberto Poncelas and Andy Way. Selecting saificialll-generatte sentences for fine-tuning neural machine translation. ArXiv, abs/1909.1201 2019.
Maja Popovic. chrf: character n-gram f-score for automatic mt evaluation. In
2015. Matt Post. A call for clarity in reporting bleu scores. ArXiv, abs/1804.08771, 2018.
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and be- yond. Found. Trends Inf. Retr, 3:333-389, 2009.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2016.
Patrick Simianer, Joern Wuebker, and John DeNero. Measuring immediate adaptation performance for neural machine translation. In NAACL, 2019,
Ralf Steinberger, Bruno Pouliquen, Anna Widiger, Camelia Ignat, T. Erjavec, Dan Tufis, and Dániel Varga. The jrc-aequis: A multilingual aligned parallel corpus with 20+ languages. ArXiv,
abs/cs/0609058, 2006.
David Thulke, Nico Daheim, Christian Dugast, and Hermann Ney. Efficient retrieval augmented generation from unstructured knowledge for task-oriented dialog. ArXiv, ab//210000444 2021.
Jörg Tiedemann. News from opus - a collection of multilingual parallel corpora with tools and interfaces. 2009.
11

---

Published as a conference paper at ICLR 2023
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Dexin Wang, Kai Fan, Boxing Chen, and Deyi Xiong. Efficient cluster-based k-nearest-neighhor machine translation. In ACL, 2022a.
Dongqi Wang, Hao-Ran Wei, Zhirui Zhang. Shujian Huang, Jun Xie, Weihua Luo, and Jiajun Chen. non-parametric online learning from human feedback for neural machine translation. In AAAI, 2022b.
Rui Wang, Masao Utyaama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. Instance weighting for neural machine translation domain adaptation. In EMNLP. 2017.
Hao-Ran Wei, Zhirui Zhang, Boxing Chen, and Weihua Luo. Iterative domain-repaired back- translation. In EMNLP. 2020.
Joern Wuebker, Patrick Simianer, and John DeNero. Compact personalized models for neural ma- chine translation. EMNLP, 2018.
M. Xia, Guoping Huang, Lemao Liu, and Shuming Shi. Graph based translation memory for neural machine translation. In AAAI, 2019.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. ArXiv, 000/2007.00800 2021.
Jiacheng Zhang, Yanzhuo Ding, Shiqi Shen, Yong Cheng, Maosong Sun, Huanbo Luan, and Yang Liu. Thumt: An open-source toolkit for neural machine translation. In AMTA, 2020.
Jingyi Zhang. Masao Utiyama, Eiichiro Sumita, Graham Neubig. and Satoshi Nakamura. Guiding neural machine translation with retrieved translation pieces. In NAACL-HLT, 2018a.
Zhirui Zhang, Shujie Liu, Mu Li, M. Zhou, and Enhong Chen. Joint training for neural machine translation models with monolingual data. In AAAI, 2018b.
Xin Zheng, Zhirui Zhang, Junliang Guo, Shujian Huang, Boxing Chen, Weihua Luo, and Jiajun Chen. Adaptive nearest neighbor machine translation. In ACL, 2021a.
Xin Zheng, Zhirui Zhang, Shujian Huang, Boxing Chen, Jun Xie, Weihua Luo, and Jiajun Chen. Non-parametric unsupervised domain adaptation for neural machine translation. In EMNLP (Findings), 2021b.
12

---

Published as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS AND MORE ANALYSIS
A.1 DATASET STATISTICS
The statistics of the datasets included in domain adaptation (multi- dataset) and online learn- ing (EMEA and JRC-Acquis datasets) tasks listed in Table 6 and 7 respectively. The Moses toolkit is used to tokenize the sentences and split the words into sub-word units (Sennrich et al.) 2016) with the bpe-codes provided by Ng et al. (2019).
Table 6: The statistics of Medical
dataset.
Koran
IT
Law
Train Sents
18k
223k
248k
467k
Dev Sents
2000
2000
2000
2000
Test Sents
2000
2000
2000
2000
Table 7: The statistics of EMEA and JRC-Acquis datasets for online learning.
Bucket
0-50
50-100
100-200
200-500
500-1000
EMEA
Documents 22
Ave sentences
38.4
73.0
157.9
392.8
759.2
Ave tokens
1174.7
1938.9
3466.1
9334.5
22725.6
JRC-Acqure
Documents Ave sentences
22
38.1
73.1
158.5
373.8
734.8
Ave tokens
1347.1
2466.7
5345.4
12518.2
26409.2
A.2 BASELINE SETUP AND IMPLEMENTATION
For the domain adaptation task, we carefully tune hyper-parameters A, T k of kNN-MT (Khan- delwal et al. 202T) and report the best scores for each domain. In the implementation of AK- MT Zheng et al. 2021a), we train a basic model on IT development set about 2500 steps and generalize to Medical, Koran and Law domains, in which the syper-parameters kmaz, 7 are set to 8 and 10, respectively.
As for the online learning task, consider ANN-MT and KoK (Wang al. 2022b). KoK intro- duces two --nearest-neighhoo (kNN) modules: ToKen-KNN memorizes the human feedback, which the correct sentence provided by human translators, while Policy-ANN balances the usage of the history of human feedback and original NMT models adaptively. To replicate KoK, we follow the setup Wang al. (2022b) and the K for Token-ANI and Policy-&NN to The whole process of the online learning task is as follows: we initialize our reference corpus as empty and incrementally add the corresponding bilingual sentences to the corpus after every source sentence is translated. In this manner, we simulate the human-in-the-lone tem can only attend the previous numan-correeteed sentences. When translating the following sentence, we perform the same text retrieval procedure as in the domain adaptation setting for our method. As for kNN-MT and KoK, we add the corresponding bilingual sentences to the datastore after every source sentence is translated, which is also simulation of the online learning scenario.
scenario where the translation sys-
A.3 HYPER- PAPER-PARAMETERS SELECTION
We report the results of adopting a variety of byrre-parameters to our proposed SK-MT model on the development set. We consider grid search {5, 10, 20, 50, 100, 150, 200), {1, 2, and m E 21,2..., 16). The optimal choices of the multi-domani dataset, including IT, Medical, Koran and Law, shown in Table and
13

---

Published as a conference paper at ICLR 2023
Table 8: Grid search on temperature with & fixed to 2 and fixed to 16.
Temperature
Domain
Metric
5
10
20
50
100
150
200
BLEU
38.2
40.7
42.9
43.9
40.9
IT
ChrF
58.7
59.3
60.2
61.6
62.3
61.7
59.8
BLEU
45.9
48.3
51.2
54.0
55.2
54.4
52.4
Medical
ChrF
64.0
65.4
67.2
69.1
70.1
69.7
68.4
BLEU
16.6
16.5
16.8
17.6
18.2
18.5
18.5
Koran
ChrF
39.6
39.6
39.8
40.3
40.9
40.8
40.8
BLEU
51.0
53.0
56.1
59.7
61.3
61.1
59.8
Law
ChrF
68.9
70.1
72.1
74.5
75.8
75.9
75.0
Table 9: Grid search on m and with temperature 7 fixed to 100.
BLEU
ChrF
Domain
m
16
2
4
8
16
k
42.7
43.4
43.5
43.3
43.3
61.7
62.1
62.2
62.1
62.0
42.4
43.0
43.8
43.9
43.9
61.3
61.7
62.1
62.3
62.3
IT
41.9
42.5
43.0
43.4
43.6
61.0
61.5
61.7
62.0
62.1
4
40.8
41.6
42.3
42.7
43.0
60.5
60.9
61.3
61.4
61.8
53.2
54.0
54.7
54.9
55.2
68.7
69.3
69.7
69.8
69.9
52.1
53.5
54.6
54.8
55.2
68.0
68.8
69.6
69.7
70.1
Medical
50.6
52.2
53.4
54.0
54.2
67.0
68.1
69.0
69.4
69.6
4
49.3
51.1
52.3
53.2
53.2
66.5
67.5
68.4
69.0
69.0
16.3
16.9
17.2
17.4
17.4
39.4
39.8
40.1
40.2
40.2
16.4
16.9
17.6
18.0
18.2
39.5
39.9
40.4
40.8
40.9
Koran
16.3
16.9
17.9
18.5
18.8
39.5
39.8
40.6
41.1
41.3
4
16.2
16.6
17.9
18.6
18.9
39.5
39.7
40.5
41.1
41.1
59.6
60.6
61.0
61.1
61.6
74.7
75.3
75.6
75.7
76.0
58.3
59.7
60.6
61.2
61.4
73.9
74.7
75.4
75.8
75.8
Law
57.1
58.9
59.6
60.0
60.2
73.3
74.3
74.7
75.0
75.1
4
55.8
57.6
58.3
58.9
59.4
72.6
73.6
74.0
74.5
74.6
Table 10: BLEU(t) and ChrF(1) on
test sets, including Medical, Koran and Law.
BLEU
ChrF Koran
Model
IT
Medical
Koran
Law
Avg.
IT
Medical
Law
Avg.
NMT
39.1
41.8
16.9
45.9
35.9
58.9
61.4
39.8
66.0
56.5
NMT+FT
48.8
57.7
21.2
62.9
47.7
66.0
71.6
42.6
77.2
64.4
ÀNN-MT
45.9
54.2
20.4
61.3
45.5
63.3
69.5
41.3
76.0
62.5
AK-MT
46.9
56.4
20.3
62.6
46.6
64.4
71.0
41.7
76.9
63.5
SK-MT,
46.1
56.8
17.6
60.7
45.3
63.6
70.7
40.3
75.5
62.5
SK-MT,
46.2
57.6
19.5
62.3
46.4
64.0
71.3
41.5
76.4
63.3
A.4 COMPARISON WITH FINE-TUNED MODELS
Table 10 shows the performance comparisons of kNN-based models with the base model fine-tuned be seen that i-N--based methods do not maintain equivalent translation quality to fine-tuned models, underperforming roughly 1 point of BLEU and ChrF on average. However, their domain adaptation performance acceptable because they do not adapt the pre-trained models for better generation. As for the machine translation with human feedback task. we online update the pre-trained NMT model with coman-corrected sentences, and the results are illustrated in Table II We can observe that KoK and SK-MT outperform NMT with online tuning in all the buckets of length with no extra constant model updating, which verifies the effectiveness and efficiency of involving non-parametric approaches in the online learning scenario.
on the comain-specifee datasets.
14

---

Published as a conference paper at ICLR 2023
Table 11: BLE on EMEA and JRC-Acquis datasets.
EMEA
JRC-Acquis
Model
[0.
150.
[100,
[200,
[500.
[50,
[100,
[200.
[500.
Full
Full
50)
100)
200)
500)
1000)
100)
200)
500)
1000)
NMT
44.2
43.2
38.3
42.4
40.6
41.7
54.1
50.0
42.2
39.9
43.4
44.5
NMT+FT
44.0
43.5
39.6
43.8
44.7
43.8
54.4
50.9
43.8
42.8
47.5
47.3
ANN-MT
43.6
43.4
39.9
43.8
43.8
43.4
55.5
52.2
45.7
43.6
47.7
48.1
KoK
44.4
44.6
44.1
45.7
53.7
49.2
56.3
52.4
47.7
44.7
50.1
49.8
SK-MT,
45.5
44.8
43.4
45.6
53.2
49.2
56.6
52.8
47.2
43.5
47.8
48.5
SK-MT2
46.1
45.6
43.8
46.3
53.6
49.7
57.4
53.7
48.2
44.7
49.5
49.8
Table 12: The statistics of JRC-Acquis datasets for translation memory methods.
Dataset
Train Sents
Dev Sents
Test Sents
Dec>En
699,596 679,088
2454 2533
2483
Es
2596
A.5 PERFORMANCE ON DIFFERENT CORPUS SCALES
We further investigate the performance discrep- ancies brought by different volumes of refer- ence corpus for text retrieval on the Law dataset, and the detailed results are shown in Figure 6 Specifically, we adopt ratio range of (0.2, 0.4, 0.6, 0.8, 1.0) randomly sample from the train- ing corpus as our reference corpus for quick experiments, and the same corpus used build a datastore for vanilla KNN-MT. These results demonstrate that the translation quality improves steadily along with the increment of corpus scale, which verifies the effectiveness of leveraging external information in NMT. SK- MT with m = 16 achieves the highest BLEU score among the SK-MT methods and is com- petitive with the state-of-the-an AK-MT model in corpora with different scales (less than point on average). It is noteworthy to discover that the performance gain between m 8 and gradually reduced, meaning that using ex- tensive reference samples when combined with rich external information is unnecessary.
Law
64
INN-MT
SK MT (T
62
SK-MT
SK-MTmm-16.---
60-
AX-MI
BLEU
58
56
54
52-
50
20%
40%
60%
80%
100%
Ratio of Corpus
Figure 6: Translation performance of kNN-based methods on Law development set given reference corpus of different scales.
16
B COMPARISON WITH TRANSLATION-MEMORY METHODS
In order to make a comparison with previous translation memory (TM) approaches (uu et al. 2018 Zhang et al., 2018a Xia al.) 2019 Cai al.) (02I). we follow up their experimental setup and evaluate the translation performance JRC-Acquis corpus (Steinberger et al. 2006). We obtain the datasets originally preprocessed by Gu et (2018) and earry out translation experiments on four language pairs, i.c., Farmann-Englise (Dee+En) and Spanish-English (Es++En). The statistics of datasets are shown in T2
We adopt the same transformer structure as Xia et al. (2019). which contains a 6-layer Transformer encoder and a 6-layer Transformer decoder. The input dimension, FFN layer dimension and at- tention heads are set to 512, 2048 and 8, respectively. As shown in Table 13] our proposed ap- proach achieves significant improvements or comparable performance to previous TM-based meth- ods, which also indicates the effectiveness of the &NN retrieval to achieve translation memory fusion.
15

---

Published as a conference paper at ICLR 2023
Table 13: BLEU score of our method and TM-based approaches on JRC-Acquis corpus.
Es= En
=s
De=
En=+De
Model
dev
test
dev
test
dev
test
dev
test
NMT
62.8
62.7
60.4
60.5
58.5
58.9
53.3
53.3
Gu et al.
2018
63.2
62.9
Zhang cf al. 2018a
64.0
64.3
61.5
61.6
60.1
60.3
55.5
55.1
Xia 2019
66.4
66.2
62.5
62.8
61.9
61.7
57.4
56.9
NMT (our implementation)
64.3
64.1
62.3
61.5
59.8
60.8
55.0
54,9
Cai et al. 2021
67.0
66.5
63.0
62.8
63.6
63.9
57.9
57.5
SK-MI1 MT2
66.6
66.0
63.3
63.4
63.5
63.7
58.6
58.2
64.2
63.9
63.8
64.2
59.0
59.0
Table 14: Model Performance of different methods on WMT 14 test set, where kNN-MT adopts FAISS-GPU to speedup the kNN retrieval. For inference speed, we set the batch size 8. The hardware we use is a single NVIDIA Tesla V100 GPU and 96 cores Intel(R) Xeon(R) Platinum 8255C CPU.
De=
En=
Model
FAISS Index
Speed (ms/sent)
FAISS Index
Speed (ms/sent)
BLEU
ChrF
Datastore
BLEU
ChrF
Datastore
NMT
31.4
58.1
36.7
27.2
57.8
32.8
ANN-MT
31.3
58.0
145G
9.0G
252.4
27.3
57.8
128G
7.9G
343.2
SK-MT,
31.4
58.0
0.06M
50.1
27.0
57.8
0.06M
43.3
SK-MT,
31.3
58.0
0.46M
63.4
27.0
57.8
0.46M
50.4
Table 15: Statistics for sentence similarities between each test sentence and the training set for the WMT' De<+En (3003 sentences), JRC-Acquis De<>En (2483 sentences) and IT De->En (2000 sentences) datasets.
WMT 14
JRC-us
IT
De=
En=DDe
Dn=En
En= De
De=
Similarity
Sent
Percent
Sent
Percent
Sent
Percent
Sent
Percent
Sent
Percent
[0, 0.1)
1
0%
0%
2
0%
0%
164
8.2%
0.2)
70
2.3%
61
2%
116
4.7%
70
2.8%
111
5.6%
(0.2, 0.3)
1210
40.3%
1101
36.7%
344
13.6%
288
11.4%
174
8.7%
[0.3,
L096
36.5%
1125
37.5%
346
13.7%
353
14.1%
229
11.5%
0.5)
411
13.7%
468
15.6%
226
0.9%
225
8.9%
141
7.1%
0.6,
173
5.8%
198
6.6%
233
9.1%
246
9.7%
543
27.2%
0.76
28
0.9%
35
1.2%
223
8.7%
213
8.4%
296
14.8%
0.87,088
9
0.3%
11
0.4%
216
0.5%
228
9%
151
7.6%
0.9)
4
0.1%
0.1%
369
14.8%
418
16.8%
148
7.4%
00.9. 1]
0%
0%
441
16.6%
442
17.7%
43
2.2%
C EXPERIMENTS ON WMT' 14 DATASET
In this section, we carry out experiments on the WMT' 14 English-German dataset consisting of 4.5M sentence pairs for model training and consider bidirectional translation this setting. We learn joint bpe-codes at the length of 45K types and adopt the Moses toolkit to tokenize the sentences and split the words into sub-word units. For each translation direction, train separate Transformer- based model, and select the best model on the development set, a split of the entire sentence pairs with a ratio of 1%. We adopt SacreBLEU (Post 2018) and ChrF (Popovic) 2015) for performance evaluation and report final results newstest2014, which is composed of 3003 bilingual sentence pairs.
As listed in Table [4 quality when utilized the WMT' 14 translation task. We suspect that it is partially because the pre-trained model is trained the same reference corpus, which dramatically limits the richness of external information. Regardless, these results prove that SK-MT has much smaller latency than vanilla &NN-MT if the datastore is considerable, which reveals that constructing dynamic datastore
find that ANN-based methods show little power in boosting the translation
16

---

Published as a conference paper at ICLR 2023
very helpful in promoting kNN-MT's efficiency. Furthermore, to give profound insight into this phenomenon, following Zhang et al. (2018a), we measure the similarity between a test sentence and the training corpus Draain by computing the sentence similarities between and the retrieved source sentences as
sim (a, Derain)
max sim(a, 2x))
(8)
EDraain
The analysis listed in Table 15 demonstrates that most of the test sentences (nearly 75%) have simi- larities of less than 0.4 to the training set, suggesting the similarity between training and test sets in the WMT 14 is considerably low. We believe the low similarity greatly attributes to the unfavourable performance of the WMT" dataset. And also, the different distribution of similarities between the WMT JRC-Acquis and IT datasets can be used to explain the performance discrepancy listed in Table 13 and [4] Thus, high similarity contributes much to ANN-basee methods, which are simi- lar TM-based methods, and it also gives some insights into the working mechanism of NNN-based methods.
17


Reviews:
[{"review_id": "iH-K7jL-kU9", "reviewer": "Reviewer_wAt2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work is to improve the efficiency of the KNN-MT framework. In this work, the author proposed a simple and scalable framework to speed up the KNN search. By utilizing sentence level retrieval approach (BM25), the author reduced the search pool for each input. The experiment results shows comparable performance with KNN-MT and reduced time/storage cost.", "summary_of_the_review": "Overall, the work is simple but effective but needs further justification mentioned in the k setup to verify if the method is solid or it is due to the task dataset.", "strength_and_weaknesses": "Strength: The proposed method is reasonable and effective to improve the kNN-MT framework by reducing the search space. \nWeakness: More analysis on m and k choice is needed. If k=1 or 2, is the always best choice for kNN-MT, which means the rest examples are all far away, then we just keep top-1 token pair is ok and does not need sentence retrievel.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Kr8R1aIpS8", "reviewer": "Reviewer_XTZD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "$k$NN-NMT is a straightforward yet powerful approach for fast domain adaptation, but is burdened with massive storage requirements\nand high computational complexity. This paper propose an efficient variant of $k$NN-MT that drastically increase speed in two folds:\n1) search similar sentences by search engine instead of searching over all tokens in training datasets\n2) use a distance-aware adaptor to adaptively incorporate the kNN retrieval results into the pre-trained NMT models.\n\n", "summary_of_the_review": "Overall, the experimental results on adaptaion tasks seems good, but lack of experiments on large-scale open-domain dataset like WMT. ", "strength_and_weaknesses": "**Strength**\n* The method is simple yet effective on multiple adaptation tasks.\n* The writing is clear and the paper is easy to follow.\n\n**Weaknesses**\n* $k$NN-MT is not only good at (narrow-)domain adaptation, but also good at improving performance on large-scale open-domain task like WMT en-de, however there is no corresponding experiments in this paper. Actually, I suspect that the sentence-level retrieval may only work well on narrow-domain data as reported in [2].\n* Regarding the two major contribution of this paper, i.e. 1) search similar sentences by search engine and 2) distance-aware adaptor, there are existed works [1][2][3]. I would give authors the credit of combining of them together in $k$NN-MT setting, but I think the contribution may not be significant enough. Directly comparing the proposed method with [1] and [2] and siginificant improvements could address my concern.\n* Although the compuation cost of searching $k$NN is alleviated, the proposed method still need to compute hidden representation of $k$ nearest sentences on-the-fly, this is actually another heavy compuational burden on GPU.\n\n**References**\n\n[1] Search engine guided neural machine translation. Jiatao Gu et al., AAAI 2018\n\n[2] Guiding Neural Machine Translation with Retrieved Translation Pieces. Zhang et al., NAACL 2018 \n\n[3] Adaptive nearest neighbor machine translation. Zheng et al., ACL2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9y5SzosprJ", "reviewer": "Reviewer_n9iv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents SK-MT which makes storage efficiency improvements over the kNN-MT model. The efficiency comes in the form of a dynamic datastore construction strategy that uses token-level retrieval (BM25) to first get a set of relevant translation pairs, and then construct a datastore from this smaller subset. To alleviate noise from a smaller datastore, an adaptive interpolation parameter is computed based on the ratio of the top-1 neighbor’s distance and the softmax temperature used with kNN-MT. This is a simpler adaptation approach than adaptive kNN-MT that involves learning a separate network for adaptation.\n\nExperiments are conducted on two MT tasks, reporting BLEU and character n-gram f-scores.  \n- Domain adaptation using the multi-domains data: the proposed SK-MT models are on par with the full-datastore baselines (kNN-MT and adaptive kNN-MT). The proposed SK-MT method reduces storage size of the datastore considerably and eliminates the need for a faiss index, but doesn’t necessarily run faster than the full datastore baselines when they search the faiss index on the GPU.\n- MT with humans in the loop providing post-edits and corrections: this involves growing the datastores dynamically as the human edits become available. The two corpora include EMA and JRCAquis. The proposed SK-MT method is on par with the KoK baseline that operates over the full datastore with an adaptive interpolation parameter.\n", "summary_of_the_review": "Recommending rejection because the current version of the paper is missing an important connection to prior work and a number of claims are unclear and/or unjustified. Open to revising the recommendation after the discussion period.\n\nEDIT after author response:  \nThe authors did a good job of addressing the primary concerns raised in the initial review. While the novelty of their approach is still limited, this paper makes contributions on the empirical side and provides data points that could be valuable for the community. For this reason, I’m raising my score to a 6.\n", "strength_and_weaknesses": "Strengths:  \n1. This paper brings the approach from Gu et al. (2018) to modern transformers and kNN-MT style approaches that don’t require training the model, while having much lower storage costs for the datastore which makes brute-force search faster.\n2. It proposes a simplification to adaptive kNN-MT that does not require training a separate network. Ablations show that the proposed adaptation is especially important for the dynamically constructed datastores.\n3. The proposed approach is on par with the full datastore kNN-MT and variants.\n4. The paper provides latency scores as well as an analysis of word accuracy across methods based on the frequency of the words.\n5. The paper provides an analysis of the rate at which various approaches adapt to domain-specific words.\n\nWeaknesses: \n1. The paper’s primary proposed approach for building the index dynamically seems very similar to Gu et al. (2018) and specifically their shallow fusion variant. Yet, the paper doesn’t really explore this connection or even attribute this approach to them. The results and analyses are interesting, but a discussion on this connection seems important and is currently missing. \n\n2. While the decrease in storage costs is nice, the decoding speed improvements aren’t clearly there due to faiss-gpu, especially when the number of retrieved translation pairs increases from 2 to 16. The current story and discussion should be revised to remove the claim that this approach improves decoding speed, or it should be revised to clearly state that a trade-off exists – better decoding speed comes at the cost of lower performance. Adding a plot on latency vs. performance would go nicely with a discussion on the trade-off. \n\n3. The paper calls this approach scalable but it is unclear how the number of retrieved translation pairs (m) scales with the size of the corpus. Is this claim justified somewhere in the paper through trends based on corpus size? The original kNN-MT paper shows that larger datastores provide diminishing returns so perhaps there is some evidence of this in the literature but missed the discussion in the paper.\n\n4. The discussion on online learning experiments needs more detail. It’s not very clear what the setup is for the different approaches, especially SK-MT. Is there some constraint that selects human provided translations, or are the token-level retrievals only over the human provided translations?\n\n5. Related to the previous discussion, the analysis on R-values is also a bit confusing. Given the current explanation of the online learning experiments, shouldn’t the R-value for all kNN approaches converge to the same value for tokens that have appeared 9+ times? Why do the full datastore retrievals ignore the human feedback but SK-MT doesn’t, or is something else going on? Also how important is it to upweight the retrievals from human feedback? What if there were a completely separate kNN function operating on the human feedback combined with kNN over the dataset and the base MT system?\n\n6. Overall, given the connection to Gu et al. (2018) and the lack of decoding speed improvements, it isn’t clear if the approach makes enough of a case for added value even though the results and analysis are interesting – at least not in the current narrative of the paper. Why is this approach better than just adaptive kNN-MT if it doesn’t run faster? Both sets of experiments show similar performance to full datastore but adaptive versions of kNN-MT. Maybe one argument is that lower storage makes this viable for running kNN-MT on-device where there are storage constraints? \n\nQuestions:\n1. Why are numbers in Tables 2 and 4 for IT different?\n\nJiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li. Search engine guided neural machine translation. In AAAI, 2018.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4jFUgFq7AKw", "reviewer": "Reviewer_TZ41", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes two very simple methods to improve the efficiency and effectiveness of k-nearest neighbor machine translation by 1) dynamically creating a datastore of bilingual sentences given an input sentence using BM25, and 2) dynamically adjusting a coefficient to scale the contribution from the similarity scores of extracted instances.\n\nExperiments are carried out in two scenarios, MT domain adaptation and online learning. In the domain adaptation task, the proposed work achieved significant speed up without loosing the MT qualities. In the online learning task, e.g., iteratively enlarging the training data, the proposed method also achieved better results when compared with prior baselines.", "summary_of_the_review": "This work presents an interesting and very simple approach to solve the efficiency problem in kNN MT. Experiments are well designed, but it is missing an important question on how well the proposed approach might work on a standard setting, i.e., training and testing on the same domain with large data. Also, the dynamic scaling sounds a nice contribution to the field, but it is not clear whether the approach could be employed in other retrieval methods or not.\n\n- Additional experiments will be useful for this community.", "strength_and_weaknesses": "Strength\n\n* The use of data retrieval based on BM25 sounds a simple yet efficient method to solve the computational problem in kNN-MT. The experiments in Table 3 shows reduced latencies by extracting only relevant bilingual sentences without loss in translation qualities as shown in Table 2.\n\n* The dynamic adaptation for the coefficient in Equation 6 is very simple and experiments demonstrate the gains.\n\nWeakness\n\n* This work presents experiments in two tasks, domain adaptation and online learning, and does not present any results regarding translation in general, e.g., training data on news domain and testing on news domain. This work should also include those results to discuss whether the proposed method also works in a standard task setting. Even if the results were negative, they should be provided for further discussion.\n\n  - The additional experimental results comparing for generic dataset sounds reasonable to me. \n\n* The proposed dynamic coefficient scaling seems to be very effective in adjusting the tradeoff of the extracted dataset, but it is not clear whether the approach is only effective to the BM25-based filtering or not. This work should compare the use of the dynamic scaling for other baselines, e.g., the original kNN MT and its variations, to investigate whether the approach is orthogonal to the retrieval methods or not.\n\n  - I also like the additional experiment, and that should be included in the main body.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster