Multilingual \textit{k}-Nearest-Neighbor Machine Translation
Authors: David Stap
Christof Monz
Venue: EMNLP 2023 Main (2023)
Keywords: multilingual machine translation
semi-parametric
kNN-MT

Abstract: \textit{k}-nearest-neighbor machine translation has demonstrated remarkable improvements in machine translation quality by creating a datastore of cached examples. However, these improvements have been limited to high-resource language pairs, with large datastores, and remain a challenge for low-resource languages. In this paper, we address this issue by combining representations from multiple languages into a single datastore. Our results consistently demonstrate substantial improvements not only in low-resource translation quality (up to $+3.6$ BLEU), but also for high-resource translation quality (up to $+0.5$ BLEU). Our experiments show that it is possible to create multilingual datastores that are a quarter of the size, achieving a 5.3x speed improvement, by using linguistic similarities for datastore creation.\footnote{We will release our code upon acceptance.}

Multilingual k- k-Nearest-Neighhor Machine Translation
David Stap Christof Monz Language Technology Lab University of Amsterdam stas, C. c.monz } @uva. n1
Abstract
and Bojar, 2018) and multilingual NMT (mNMT) (Johnson et al., 2017; Arivazhagan al., 2019; Stap et al., 2023). Preliminary findings on combining kNN-MT with mNMT suggest that mNMT representations generalize sufficiently well across languages to make cross-lingual retrieval effective (Khandelwal et al., 2021). However, its effectiveness for low- resource languages remains an open question. what extent
k-nearest-neighhor machine translation has demonstrated remarkable improvements in ma- chine translation quality by creating datas- tore of cached examples. However, these im- provements have been limited to high-resource language pairs, with large datastores, and re- main a challenge for low-resource languages. this paper, we address this issue by com- bining representations from multiple languages into a single datastore. Our results consistently demonstrate substantial improvements in low-resource translation quality (up to +3.66 BLEU), but also for high-resource translation quality (up +0.5 BLEUU). Our experiments show that it is possible to create multilingual datastores that are quarter the size, achiev- ing 5.3x speed improvement, using linguis- tic similarities for datastore creation."
In this paper, we investigate mNMT can useful for improving low-resource kNN-MT translation quality. First, we experi- ment with cross-lingual datastores from related languages and find that low-resource languages generally benefit from larger cross-lingual data- stores. We then propose simple yet effective approach, multilingual ANN-MT, which uses mul- tilingual datastores that constructed by merg- ing bilingual datastores. Our results show substan- tial improvements for fow-resource languages, and also noticeable improvements for high-resource languages. Finally, we show that it is possible to create multilingual datastores that are significantly therler- resulting in substantially faster decoding times by relying linguistic similari- ties when creating multilingual datastores.
only
1 Introduction
Recently, cemi-parametric approaches such k- nearest-neighhor machine translation (KNN-MT) (Khandelwal et al., 2021) have attracted interest due to series of impressive results in language modeling and machine translation (Guu et al., 2018; Bapna and Firat, 2019; Khandelwal et al., 2020). These techniques capitalize information retrieved from extensive repository of translation examples cached in datastore. One of the most important limitations of kNN-MT is that the extent quality improvements strongly depends on the size of the datastore (Khandelwal et al., 2020, 2021; Zhu et al., 2023). This dependence on datastore size is problematic for low-resource languages and the improvements that ANN-MT can offer in low- resource settings modest at best (Vardhan et al., 2022). On the other hand, there general meth- ods that can improve low-resource performance, such as transfer learning (Zoph al., 2016; Koemi
k-nearest neighbor machine translation
ANN-MT combines parametric component with nearest neighbor retrieval mechanism that al- lows direct access to datastore of cached ex- amples (Khandelwal et al., 2021). The datastore D consists of key-value pairs, where each key is translation context, i.e., decoder output repre- sentation, y<t), and the value the corre- sponding target token yi- At inference time, the model searches the datastore to retrieve the set of k nearest neighbors N. Using their distances d(-) to the current translation context, retrieval dis- tribution (Nel(ye x) is computed. The final probability distribution is obtained by combining
'We release our code at https: //gitttt....g.. com/ davidstap/ htt/mmuiiiii tilingual kNN-mt.

---

(N) and PANN(wlyctt
To further align multilingual representations, we learn linear mapping between two languages. Our goal is to let language e more effectively query from a la datastore. For our training data TT, we include translation contexts from the Ll datastore D(e e)) and the l datastore D(e ((e2,e)) ,e) that correspond to the same target sentence y and target token y;:
3 Multilingual k-nearest-neighhon MT
Despite the potential benefits, the integration of mNMT with ANN-MT has only been rarely ex- plored. Using a datastore with English on the source side improves performance for other source languages (Khandelwal et al., 2021), but it is not known to what extent this holds for low-resource languages. Pre-trained multilingual language mod- els can be used to build monolingual datastores of target language (Li al., 2022), but the required alignment training does not work for low-resource languages due to data scarcity. Our goal is to im- prove performance for low-resource languages by constructing cross-lingual and multilingual datas- tores. These datastores consist of keys generated from mNMT representations, allowing semanti- cally related sentences from different languages to cluster together (Johnson et al., 2017; Escolano al., 2019).
T = (Piaa (ia P(Ge))) [ map to y E (y E ()
(3) where (())) {Bie 81); B( is the com- bined data from e and 12 into l" Subsequently, we minimize min Σ = || -TT using the normal equation approach, where Tý, and Tj originate from tuples of T. We then use A to map a translation context from e to l We also learn the inverse relation, i.e., e to E and create an op- timized multilingual datastore for ll by applying the L to El mapping prior to storing the datastore.
Experiments
4.1 Setup
The 418M parameter version of the M2M100 mul- tilingual translation model (Fan et al., 2021) is used for all experiments. It is a Transformer (Vaswani et al.. 2017) with 24 layers, 16 heads and hidden dimensionality of 1024 supporting 100 languages. We conduct our experiments on the widely used TED Talks corpus (Qi et al., 2018). We use the train set to create datastores, the development set for tuning kNN-MT hyperparameters, and the test set to report results. We use 51 languages into English, from 23 different language families, that supported by both TED and M2M100.
3.1 Bilingual and cross-lingual datastores A biliiguull datastore is defined as follows:
Dan, = y()(x.)))...)) Vy y (x.y E B),,)))
(1) where bi-text data B(eee) originates from a single source language E into target language é'. When we use bilingual datastore Dee,e) to augment the translation direction of another source language E* + l into target language é we call the datastore cross-lingua.. For instance, a Russian-English data- store ((w,,n) may be used to enhance Belarusian- English translation. An important advantage of cross-ling lingull datastores is that they can be signifi- cantly larger than their bilingual counterparts, and therefore may result in better translation quality.
We use kNN-BOX (Zhu et al., 2023) for our experiments. Following Khandelwal et al. (2021), we tune the number of neighbors E {16, 32, 64). interpolation À C {0.2, 0......, 0.7) and softmax temperature T E {10, 100} hyperparameters on the development set. We use a beam size of 5. We evaluate our models using sacreBLEU (Post, 2018; Papineni et al., 2002).
3.2 Multilingual datastores
Earlier work is limited to monolingual or bilingual datastores (Khandelwal et al., 2021; Cai et al., 2021: Li et al., 2022). In contrast, we create multilingual datastores consisting of multiple source languages, resulting in larger datastores.
4.2 Cross-lingual and multilingual datastores
We construct datastores for languages from three M2M100 language groupings into English: Slavic (12 languages), Germanic (5 languages), and Greek (4 languages). We then generate translations for all possible combinations of source language and data- store, with the goal of investigating the potential of
We construct a multilingual datastoreD (M...)) by considering set of source languages LMI, that map to a target language É::
D(aan.e (((xx, vt)). Vyr y yx e Bus ey)
(2)
where BiL = Uset Bleeer is the combined data from all LML source languages into target !..
(oddmm/ Ferefs: I case:mixed] offinottok [tok: 13a smoochoxppversion22...

---

Teeat
The
Thu
my
Des
Da
Dhr
The
12tt
Due
Pun
DALL
[D]
116K
149K
D20K
G83K
1.6M
2.7M
2.9M
3.3M
8.GM
4.7M
4.7M
5.6M
30.6M
86 4M
120M
15.2
20.3
20.9
20.6
21.0
22.4
21.4
21.3
21.3
21.3
22.2
31.5
34.0
54.7
35.2
36.
26.4
27-3
27.0
27.0 8
27.9
27.9
28. 3
28.4
27.9
28.1
29.2
20.2
20.
23.
82.0
92.1 1
$2.5
32.5
33. 2
33.8
33.3
34.8
33.9
34.1
33.4
33.
36.6
85.5
35.6
28.
30.3
30.5
91.3
4
392 6
33.0
32.1
32
3.2.4
32.8
32.4
32.6
34.1
33.6
34. 1
27.
29.3 3
219.
50.0
30
31.4
30.7
30
31.22
30.8
31.0
31.8
32.1
25.6
28.3
28.8
34.9
35.5
36.0
35.5
33.1
301-7
32.2
32.7
33.3
33.6
33.4
34.2
34.0
35.
35.
34.8
34.3
34.6 6
36.3
35.7
36.
bg
334.4
385
36.2 2
97.1
37
35 4
37.9
37.6
38. 1
35.
39.5
36. 0
38.3
39.
39.
39
21.1
22.8
22.8
23.4
23.5
23.4
24.
23.9
24.0
24.
24.5
25.0
24.3
25.4
25 4
25.4
21.6
23.5
25.3
23.8
24.1
24.3 3
24.7
24.9
24.7
24.8
25.0
24.9
25.6
26.0
26.0
25.4
27.1
Duo
DAR
DALL
D
bose
Da
17g
DALL
1:204
1.4M
4.9M
12M
128M
351K
644K
1.2M
3.5M
6.6M
125M
02.2
15.0
48.7
45.0
48.4
48.3
87.7
17.4
87.8
10.8
14.7
12.8
15.2
15:2
da
30.0
43.1
43.5
43.3
44.0
44.0
15.3
15.0
46.7
by
16.8
20.1
19.0
20.0
20.3
20.7
37.3
89.6
40.4
11.0
40.8
40.8
41.8
42.1
42.0
31.9
3652
33.4
36.3
34.6
360
36.5
366
35.00
0
37.1
37.2
87.3
d
32.6
84.8
35.3
85.8
38.3
88. 8
38.7
38.8
34.4
34.6
33.22
36.0
avg
36.7
30.3
40.2
40.0
40.7
40.7
41.7
41.5
Table X-+en BLEU scores for three language groupings: Slavic (top), Germanic (bottom left) and Greek (hottom right). We display results for all combinations of translation directions and datastores DD within each language grouping. (For brevity, we write e.g. the Relarusian- Beharusian-Fnglish datastores as De instead of D (e..n))) Datastore size depicted as [D). We refer to languages for which [D] < 1M as low-resource languages. These languages are separated from high resource languages by a dashed line. We color the bilingual datastores on the diagonal grey. The three rightmost columns are multilingual datastores, built from language grouping languages (Pus). bridge languages (Das), or all languages (PALL). BLEU scores obtained without ANN-MT MT are listed in the column labeled base. We underline the best cross-lingual or bilingual results. Overal best scores are depicted in bold.
cross-lingual datastores to improve low-resource performance.
4.3 Results
Translation results for bilingual, cross-lingual, and multilingual datastores are shown in Table 1.
Additionally, we construct several multilingual
datastores:
P(ALL.en): We created comprehensive datas- tore, integrating languages that occur in both TED and M2M100 into English, resulting in 125M entries.
Bilingual datastores work to a limited extent We observe that bilingual datastores can bring limited improvements in performance for low- resource languages, even though their datastores small. For instance, low-resource languages from the Slavic language grouping improve with +.3 BLEU on average. High-resource languages have more improvements, e.g., the Slavic grouping gains +4.5 BLEU on average.
[BRR. mNMT is typically English-centric, i.e., English occurs on the source or target side in the training data. M2M100 instead uses a set of bridge languages, which leads greater cover- age of direct translation directions. To align with these languages, we additionally create a smaller datastore of size 86.4M, consisting of 24 bridge languages.
Cross-lingual is better than bilingual In gen- cral, low-resource languages benefit from cross- lingual datastores. For instance, Belarusian- English (be-en) with bilingual datastore (116K instances) results in +1.7 BLEU, whereas Celurusian-Engliste with a substantially larger Eknainian-Ennlish (uk-en) datastore (2.9M in- stances) leads to a further improvement of +1.5 BLEU. However, while datastore size and perfor- mance do correlate', the additional quality im- provements can not be fully explained by the size increase. For instance, Bosnian-Engliss (bs-en) augmented with a cross-lingual (hr-en) datastore, which is only 60% of the size of Russian- -nnglihh (ru-en), Ieads to +1.0 BLEU compared to using Russian-English. We conclude
Di..G.n): We investigate to what extent datastore size can be further decreased. We hypothesize that more similar multilingual representations result in better cross- lins-lln retrieval, and that representa- tions within the same language grouping are more similar. line with this, we create three multilin- gual datastores consisting of languages within a language grouping: Slavic (datastore size 30.6M), Germanic (datastore size 20M), and Greek (datas- tore size 5.6M).
Oroatian-Engliin
See Appendix A for more details on the datastores.
"p = 0.88 with < 0.001 for Slavic language grouping.

---

that it is difficult to predict which cross-lingual datastore will perform best.
D
[E]
Tw
ATw
BLEU
D (bsar
23K
20.5
20.4
0.1
((s.n)
95K
20.9
21.3
0.4
In contrast, for bigh-resource languages is al- ways a better choice to use the bilingual datastore, even when larger cross-lingual datastores are avail- able. For instance, Serbian- Serbian-English (sr-en) with Serbian-Engliss datastore of 3.6M tokens performs better (35.9 BLEU) than Serbian- Englishh with the substantially larger cross- lingull datastore Russian- English (5. 656M tokens, 34.6 BLEU).
P (o
73K
20.8
21.2
0.4
(ak,es)
202K
21.9
21.3
0.3
D (esun)
305K
21.5
21.7
0.2
(a
317K
22.4
22. 2
-0.2
D Drrm))
359K
21.1
21.7
0.3
Dir.as)
417K
21.3
21.8
0.5
D (rem)
431K
21.3
21.8
0.5
P(t.n))
421K
21.3
22.1
0.8
P(n.a)
533K
21.7
22.3
0.6
291K
21.3
21.6
0.3
((c,
23.1
28.4
0.3
When considering cross-lingual datastores that come from a more distant language family, using a bilingual datastore leads to better results, even for low-resource languages.. Considering Georgian- English (ka-en), the bilingual datastore improve- ment (433.9 BLEU) is larger than the improvement for its best cross- lingual datastore Greek-Englist (el-en, +2.1 BLEU)
scorec for Slavic grouping, with cross- lingual mapping (ATTb) and without (Tw)). Training data size for mapping is shown as [TT] Best results shown in bold.
Table 2: he-ten BL.U scores
datastores lead to best results for all languages we tested, including higher-resource directions such as Polish-Englin (pl-en, +0.44 BLEU compared to bilingual) and Ukrainian-Enggln (uk-en, +0.8 BLEU).
Multilingual datastores perform best Since is unclear which cross-lingual datastore performs best, we use as many languages as possible as a first attempt. This results in our largest datas- tore (nA... which has 125M entries. For al- most all languages, except Russian- (ru- en), this leads to better results than bilingual data- stores. Low-resource languages show the largest improvements, where Bosnian -angish (bs-en) has the largest improvement of +3.6 BLEU compared to bs-en datastore, or +0.7 BLEU compared to the best cross-lingual datastore. A problem for P(ALLee)) is slow inference speed, because kNN lookup in large datastore is expensive. When de- creasing the datastore size by focusing on bridge languages, we can construct a smaller datastore of size 16, but results are worse in almost all cases which is clearly reflected in the average scores. Finally, we consider a datastore that is con- structed using linguistic similarity. It consists of languages from the same language grouping. We observe that this multilingual datastore is par with the largest one, while significantly smaller, which is clearly reflected the averages. For some languages, such as Belarusian -inllss (be-en) and Bosnian-Ennn Engllshh (bs-en), this even brings improve- ments of +0.6 BLEU and +0.5 BLEU compared to using D(A) We emphasize that multilingual
Effectiveness of cross-lingual mapping We cre- ated a cross- cross-lingual mapping from Belarusian (be) to other languages in the Slavic language group- ing. We also created the inverse mapping, and constructed Slavic language grouping datastore mapped to Belarusian representations.
Results are presented in Table 2. We observe that generally, Belarusian Belarusian-Englishh (be-en) performance is improved, especially for larger cross-lingual data- stores such as Polish-English (pl-en, +0.8 BLEU), For bs-en and uk-en the mapping results in a slight quality 5ecrease.
4.4 Analysis
Which languages used? We explore the lan- guage origin of KNN-MT target token suggestions when using ((LLLeen) augment the Norwegian- English (no-en) translation direction. The top 15 origins with highest probability mass are shown in Table 3. Surprisingly, despite consisting of only 5 out of 51 languages, the Germanic language group accounts for 23.2% of the suggestions. This helps to explain why D D(LG,en) performs on par with D (ALL,,en)- even though it is more than ten times smaller. Full results are in Appendix B.
Multilingual datastore speed Using the smaller datastore D results in significantly faster de-
"This is likely because the Greek language grouping com- bines different families: Greek (el) Hellenie language, whereas Georgian (ka) is from the Kartvelian language family. Therefore, their representations likely have larger differences than these of Bengali-Engee English (be-en) Ukrainian Ukrainian-English (uk-en), which come from the same family.
"This can possibly be explained by the small data size (for bs-en), and because uk-en and be-en are already relatively well aligned, since uk-en the cross cross-lingual datastore be-en (see Table 4).

---

Pas
Fin
Limitations
411K
6.05%
0.34%
i-cn
5.5M
5.74%
4.62%
We use an mNMT model that is non- non-English- centric, which may present limitations in the gener- alizability of our multilingual KNN-MT results to other multilingual settings such as English-centric. A limitation of KNN-MT, which has become the central focus of most subsequent KNN-MT re- search, is the steep increase in decoding time in- troduced by KNN-MT, as each decoding step re- quires a computationally expensive nearest neigh- bor search (Zheng et al., 2021; Martins et al., 2022a.b; Yang et al., 2022: Meng et al., 2022; Jiang et al., 2022).
fr-en
5.1M
5 6.6%%
4.29%
nl-en
4.9M
5.47%
4,00%
en
5.2M
5.31%
4.38%
he-en
5.7M
8.88%
4,80%
4.7M
4.77%
3.95%
r0-cn 1.8M
4.40%
4,05%
dn-en
1.2M
3.64%%
0.99%
el-en
2.96%
hn 3.3M
3.15%
2.73%
nu-nn
sr-en
3.6M
3.11%
3.02%
1.4M
3.08%
1.18%
Table 3: Bilingual origins for the 15 datastore lan- guages with the highest occurrence when augmenting Norwegian-Enge (no-en) with multilingual datastore DeALL.en) datastore. D denotes bilingual datastore, and [D] the corresponding size. Pabs are the observed origin percentages when decoding the no-en test Puni are the uniform origin percentages, when taking into account the bilingual datastore sizes. Bold datastores indicate that they are from the no-en language group- ing, and underlined means they are a bridge language. Darker colors indicate more probability mass.
While we built multilingual datastores using up to 51 languages, we did not investigate to what ex- tent even larger multilingual datastores can further improve performance.
Broader Impact
Machine translation poses potential risks, such as errors in translation. This risk is particularly high for low-resource languages. Adding RNN-MT likely decreases this risk, since translation qual- ity improves. Using multilingual datastores likely further decreases this risk.
coding speeds of up to 5.3x for Ellarusian-English (be-en) compared to using (ALL,en)) More results are in Appendix C.
5 Conclusion
We have proposed a simple and effective approach to enhance quality for low-resource languages in KNN-MT. We augmented an mNMT model with cross-lingual and multilingual datastores of related and unrelated languages. We show that using mul- tilingual datastores substantially improves trans- lation quality for low-resource languages, while high-resource languages also improve. We find that by harnessing linguistic similarity, we can limit multilingual datastore size while preserving quality and significantly increasing inference speed. Fi- nally, we show that by further aligning multilingual representations, we can more effectively use cross- lingual and multilingual datastores.
References
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: findings and chal- lenges. arXiv: 10X7v 1907.05019 [cs]. ArXiv: 1907.05019.
Ankur Bapna and Orhan Firat. 2019. -orrmaraett adaptation for neural machine translation. Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics, pages 1921-1931, Minneapolis, Min- nesota. Association for Computational Linguistics.
Deng Cai, Yan Wang, Huayang Li, Wai Lam, and Lemao Liu. 2021. Neural Machine Translation with Proceedings
Acknowledgements
Monolingual Translation Memory. of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1tth International Joint Conference on Natural Language Processing (Voiume 1: Long Papers), ), pages 7307-7318, Online. Association for Computational Linguistics.
This research was funded in part by the Netherlands Organization for Scientific Research (NWO) under project numbers VI.C.192.08 and VI. Veni.212.228 We thank Ali Araabi, Vlad Nic- ulac, Yan Meng, Shaomu Tan, Ke Tran, Sony Tre- nous and Di Wu for their helpful suggestions and insights.
Carlos Escolano, Marta R. Costa-jussà, and José A. R. Fonollosa. 2019. From bilingual to multilingual neu- ral machine translation by incremental training. In

---

Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguisties: Student Re- search Workshop, pages 236-242, Florence, Italy. Association for Computational Linguisties.
Pedro Henrique Martins, Zita Marinho, and André F. T. Martins. 2022b. Efficient Machine Translation Do- main Adaptation. In Proceedings of the 1st Work- shop on Semiparametric Methods in NLP: Decou- pling from Knowledge, pages 23-29, Dublin, Treland and Online. Association for Computational Linguisties.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Man- deep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vi- taliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021. Be- yond English-centric multilingual machine trans- lation. Journal of Machine Learning Research, 12(107):1-48.
Yuxian Meng. Xiaoya Li, Xiayu Zheng, Fei Wu, Xiaofei Sun, Tianwei Zhang, and Jiwei Li. 2022. Fast Near- est Neighbor Machine Translation. In Findings of the Association for Computational Linguisties: ACL 2022, pages 555-565, Dublin, Ireland. Association for Computational Linguistics.
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating Sentences by Editing Prototypes. Transactions of the Association for Computational Linguistics, 6:437-4504
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: a method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics. page 311, Philadelphia. Pennsyl- vania. Association for Computational Linguistics.
Hui Jiang, Ziyao Lu, Fandong Meng. Chulun Zhou, Jie Zhou, Degen Huang, and Jinsong Su. 2022. Towards robust k-Nearest-Neighhoo machine translation. In Proceedings of the 2022 conference on empirical methods in natural language processing, pages 5468- 5477, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference Machine Translation: Research Pupers, pages 186- 191, Belgium, Brussels. Association for Computa- tional Linguistics.
Melvin Johnson, Mike Schuster, Quae V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Femanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: en- abling zero-shot translation. Transactions of the As- sociation for Computational Linguisties, 1:339-351.
Ye Qi. Devendra Sachan, Matthieu Felix, Sarguna Pad- manabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neu- ral machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the As- sociation for Computational Linguistics, pages 529- 535, New Orleans, Louisiana. Association for Com- putational Linguistics.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest Neigh- bor Machine Translation. In International Confer- ence on Learning Representations.
David Stap, Vlad Niculae, and Christof Monz. 2023. Viewing Knowledge Transfer in Multilingual Ma- chine Translation Through a Representational Lens. ArXiv:2305. 1550 [cs].
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through Memorization: Nearest Neighbor Language Models. In International Conference Learning Representations.
Harsha Vardhan, Anurag Beniwal, Narayaman Sadagopan, and Swair Shah. 2022. Low resource retrieval augmented adaptive neural machine transla- tion. In NeurlPS 2022 workshop on trustworthy and socially responsible machine learning (TSRML).
Tom Koemi and Ondřej Bojar. 2018 Trivial transfer learning for low-resource neural machine translation. In Proceedings of the Third Conference on Machine Translation, pages 244, Belgium, Brussels. As- sociation for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), Long Beach, Califor- nia. Neural Information Processing Systems (NIPS). ArXiv: 1706.03762.
Jiahuan Li, Shanbo Cheng, Zewei Sun, Mingxuan Wang. and Shujian Huang. 2022. Better Datastore, Bet- ter Translation: Generating Datastores from Pre- Trained Models for Nearest Neural Machine Transla- tion. ArXiv:2212.08822 [cs].
Zhixian Yang, Renliang Sun, and Xiaojun Wan. 2022. Nearest Neighbor Knowledge Distillation for Neural Machine Translation. ArXiv:2205.0047 00407 [cs].
Pedro Henrique Martins, Zita Marinho, and André F. T. Martins. 2022a. Chunk- Nearest Neighbor Ma- chine Translation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Lnguage Processing, pages 0228-4245 Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Xin Zheng, Zhirui Zhang, Junliang Guo, Shujian Huang. Boxing Chen, Weihua Luo, and Jiajun Chen. 2021. Adaptive Nearest Neighbor Machine Translation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th

---

International Joint Conference on Natural Language Processing (Volune Short Papers), pages -68-374, Online. Association for Computational Linguisties.
C Multilingual datastore inference speed
We present multilingual datastore inference speeds in Figure 1. We set to 64, and present results for all multilingual datastores, using a single source language into English for each language group- ing. We average results over 3 runs. We ob- serve clear trend: smaller datastores result in substantially faster decoding times. For be-en, using the Language Grouping instead of the All multilingual datastore leads to a 5.3x speed im- provement. For no-en and ka-en, the improvements are 3.0x and 2.6x. In terms of quality, be-en im- proves when using the Language Grouping datas- tore (+0.6 BLEU), while no-en and ka-en have sim- ilar performance (-0.1 BLEU and +0.0 BLEU).
Wenhao 7hu, Qianfeng Zhao, Yunzhe Lv, Shujian Huang, Siheng Zhao, Sizhe Liu, and Jiajun Chen. 2023. KNN-BOX: A Unified Framework for Nearest Neighbor Generation. ArXiv:2302.1 I 13574 [es].
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neu- ral machine translation. In Proceedings of the 2016 Conference Empirical Methods in Natural Lan- guage Processing, pages 1568-1575 1575, Austin, Texas. Association for Computational Linguistics.
A Datastore information
We list information about all 51 datastores in Ta- ble 4. Note that often, language family and lan- guage grouping are consistent, i.c., all languages in a language grouping are from the same language family. However, there are exceptions such the language grouping Greek, which includes languages from the Hellenic, Kartvelian, Armenian, and A Albanian families.
Inference mutilingua datastores
52x
Tokera second
B Multilingual datastore language origins
Brigg OZ (86
Sunk 130
Gemmnn
Stoett sa
We show language origins for the no-en translation direction augmented with a multilingual datastore consisting of all 51 languages in Table We track the bilingual datastore origin of target token sug- gestions from ANN-MT during inference on the test set, and calculate the percentage based the total number of suggested tokens. It should be noted that not all target token suggestions from KNN-MT are included the generated target sentence.
Forskdioe Anecley
122
Datastore
200
Figure Inference speed for Greek, Germanic, Slavic, Bridge, and All multilingual datastores. The x-axis displays the datastore size (large to small), and the y- axis shows the corresponding tokens per second. clear linear trend be observed: smaller datastores result in substantially faster decoding times.
From Table 5 we observe that the distribution of observed language origins generally follows a uni- form distribution based on bilingual datastore size. The largest outlier is the no-en datastore, which is used for 6.05% of the generations, opposed to the 0.34% that would be expected from uniform distribution.
Additionally, we find that all five languages from the Germanic datastore, to which no-en belongs, are oversampled compared to the uniform distribu- tion. This set of just five languages is responsible for 23.2% of the matches.
Furthermore, we observe that datastores from several bridge languages, including ar-en, pl-en, hu- en, ko-en and ja-en, are undersampled compared the uniform expectation. This discrepancy can likely be attributed to the fact that these languages are relatively distant from Norwegian, resulting in dissimilar representations.

---

datastore
size
family
grouping
script
bridge
M2M100
+&NN-MT
diff
kk-en
84K
Turkic
Turkic
Cyrillic
2.1
2.6
he-en
116K
Slavic
Slavic
Cyrillic
19.2
20.9
bn-en
127K
Indo-Aryan
Indo
Eastem-Nagari
9.3
13.9
ms-en
132K
Malayo-Polyn.
Malayo
Latin
28.8
30.6
bs-en
146K
Slavic
Slavic
Latin
31.5
33.1
1.6
az-en
153K
Turkic
Turkic
Cyrillic
8.8
10.0
ta-en
156K
Dravidian
Indo
Tamil
0.4
0.9
0.5
ur-en
158K
Indo-Aryan
Indo
Arabic
14.4
16.9
2.5
mn-en
181K
Mongolic
Mongolic
Cyrillic
5.1
7.1
2.0
mr-en
241K
Indo-Aryan
Indo
Devanagari
3.9
6.2
2.3
gl-en
254K
Romance
Romance
Latin
32.4
34.0
1.6
et-en
280K
Uralic
Uralic
Latin
23.5
25.3
1.8
ka-en
332K
Kartvelian
Greek
Georgian
10.8
14.7
3.9
no-en
411K
Germanic
Germanic
Latin
42.8
45.6
2.8
hi-en
481K
Indo-Aryan
Indo
Devanagari
17.9
23.3
5.4
sl-en
520K
Slavic
Slavic
Latin
24.9
27.3
2.4
hy-en
544K
Armeian
Greek
Armenian
16.8
20.1
3.3
my-en
558K
Sino-Tibbeta
Mongolic
Burmese
0.4
1.5
1.1
fi-en
623K
Uralic
Uralic
Latin
21.0
22.8
1.8
mk-en
683K
Slavic
Slavic
Cyrillic
29.3
32.8
3.5
It-en
1.1M
Baltic
Uralic
Latin
24.7
28.2
3.5
sq-en
1.2M
Alhanian
Greck
Latin
31.9
35.8
3.9
da-en
1.2M
Germanic
Germanie
Latin
40.0
44.5
4.5
pt-en
1.2
Romance
Romance
Latin
38.9
42.0
3.1
1.4M
Germanic
Germanie
Latin
37.3
41.0
3.7
sk-en
1.6M
Slavic
Slavic
Latin
28.4
32.6
4.2
id-en
2.3M
Malayo-Polyn.
Malayo
Latin
29.1
32.5
3.4
th-en
2.6M
Kra-Dai
Mongolic
Thai
2.3
8.5
6.2
2.7M
Slavic
Slavic
Latin
27.5
3144
3.9
uk-en
2.9M
Slavic
Slavie
Cyrillie
24.7
29.1
hr-en
3.3M
Slavic
Slavie
Latin
32.2
37.0
3.5M
Hellenic
Greek
Greek
32.6
38.3
sr-en
3.6M
Slavic
Slavic
Cyrillic
30.7
35.9
hu-en
3.9M
Uralic
Uralic
Latin
23.3
26.8
3.5
4.0M
Iranian
Arabic
Arabic
22.7
27.6
de-en
4.5M
Germanic
Germanic
Latin
31.7
36.9
vi-en
4.6M
Vietic
Chinese
23.7
27.2
bg-en
4.7M
Slavic
Slavic
Cyrillic
34.4
39.5
5.1
pl-en
4.7M
Slavic
Slavic
Latin
21.1
25.0
3.9
ro-en
4.8M
Romance
Romance
Latin
30.6
35.4
nl-en
4.9M
Germanic
Germanic
Latin
31.9
36.2
tr-en
4.9M
Turkic
Turkie
Latin
22.4
26.5
4.1
fr-en
5.1M
Romance
Romance
Latin
35.1
40.3
es-en
5.2M
Romance
Romance
Latin
36.6
41.9
5.3
zh-en
5.4M
Chinese
Chinese
Chinese
16.3
20.2
3.9
ja-en
5.5M
Japonic
Chinese
Kanji
10.6
13.5
2.9
it-en
5.5M
Romance
Romance
Latin
33.4
38.4
5.0
ko-en
5.5M
Koreanic
Chinese
Hangul
16.2
19.5
3.3
ru-en
5.6M
Slavic
Slavic
Cyrillic
21.6
25.8
4.2
he-en
5.7M
Semitic
Arahic
Hehrew
31.2
36.8
5.6
ar-en
5.8M
Arabic
Arabic
Arabic
26.2
31.4
5.2
average
2.5M
23.4
27.0
3.6
total
125M
Table Datastore information for all languages into English that we use. Column grouping indicates language grouping from M2M100. Column bridge indicates whether the language is bridge language in M2M100. The three final rows show BLEU scores the base model (M2M100), augmented with KNN-MT (+NNN-MT), and their difference (diff).

---

D
[D]
Pa-
Piri
111K
6.05%
0.34%
fr-en
5.1M
5.62%
4.29%
os-in
5. 2M
5.31%
4.38%
deven
4.7M
4.92%
3.73%
he-en
5.7M
4.88%
4.80%
bg-on
4.7M
4.77%
3.91%
da-en
1.2M
3.64%
0.99%
3.5M
3.48%
2.96%
hr-en
3.3M
3.15%
2.73%
5.6M
3.15%
4.71%
se-cn
3.11%
3,025%
sv-en
1.4M
3,08%
1.18%
ar-en
5. 8M
3.05%
4.87%
plaen
4.7M
2.60%
3.95%
cseen
2. 7M
2.35%
2.27%
3.9M
2.34%
3,28%
tr-nn
2.21%
id en
2.3M
2.%
1.90%
pr-en
1.2M
1.81%
1.03%
ke en
5.5M
1.66%
4.63%
ja-en
5.5M
1.17%
4.60%
sqren
mk-en
683K
0.83%
0.57%
Ir-en
slien
520K
0.55%
0.44%
l-cn
623K
0.48%
0.52%
gl-en
254KK
0.31%
0.21%
ct-on
280K
20.0%%
0.23%
hi-en
481K
19%
0.40%
bs-ea
146K
0.14%
0.12%
ka-on
332K
14%
0.28%
my
mi-en
132K
0.08%
0.11%
0.07%
0.20%
be-en
116K
0.07%
0.10%
158K
0.05%
0,13%
bu-cn
127K
0.04%
0.11%
mn-en
181K
0.04%
0.15%
a2-en
153K
0.03%
0.13%
84K
0.02%
0.07%
Table Bilingual origins when augmenting no-en with multilingual datastore D (ALL,cc) datastore. 'D denotes bilingual datastore, and [D] the corresponding size. Pats are the observed origin percentages when decoding on the no-en test set. Puni are the uniform origin percent- ages, when taking into account the bilingual datastore sizes. Bold datastores indicate that they are from the no-en language grouping, and underlined means they bridge language. Darker colors indicate more prob- ability mass.


Reviews:
[{"review_id": "XbL1rqaKhu", "reviewer": "Reviewer_oGvF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n* The contribution can be deemed minor, previous work used similar methods but did not focus on low resource languages\n\nOur work is different from previous work in important ways, and the difference is not limited to our focus on low-resource languages. There is related work that combines *k*NN-MT with multilingual models: [1] augments a multilingual *model* with *bilingual datastores*; [2] uses a pre-trained multilingual model to create a *monolingual datastore* of the target language. In contrast, we develop methods to carefully construct *multilingual datastores*, which we show outperforms bilingual datastores. Our method is simple, which we see as a strength. The method shows strong improvements for low-resource languages (and smaller improvements for high-resource languages) compared to strong baselines.\n\nWe’ll make sure to fix the typo and the issues with spacing, thanks for pointing it out.\n\n[1] Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer and Mike Lewis. Nearest Neighbor Machine Translation. ICLR 2021.\n\n[2] Jiahuan Li, Shanbo Cheng, Zewei Sun, Mingxuan Wang and Shujian Huang. Better Datastore, Better Translation: Generating Datastores from Pre-Trained Models for Nearest Neural Machine Translation. arXiv 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well written, the method proposed of multilingual datastore is simple yet effective\n* The method results in substantial improvements on TED across different languages\n* Latency analysis is provided\n* Additional meaningful analysis such as which datastore are mostly used is also provided\n* The authors mention they will release the code upon publication", "reasons_to_reject": "* The contribution can be deemed minor, previous work used similar methods but did not focus on low resource languages", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores multilingual k-nearest neighbor (KNN) approach to machine translation (MT), with focus on low resource languages. The authors compare multiple datastore types for KNN, including bilingual and multilingual ones with different selection of language groups. Results on TED show large improvements on low resource languages when using the proposed multilingual datastore as opposed to the bilingual ones, in addition to smaller improvements for high resource languages. The authors also provide reliable language group selection to reduce the datastore size and report latency analysis of the inference stage. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* lines 116,129 issue in spacing\n* Table 1: build -> built\n"}, "extra_text": {}}, {"review_id": "l1v6gFUkf1", "reviewer": "Reviewer_oA3E", "summary": "", "questions": "Have you compared with a multilingual model?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n* The improvement of cross-lingual mapping is relatively small (0.3 BLEU). The contribution of this part is a little bit minor.\n* Similarly, the innovation is not strong enough.\n\nRegarding your comment that the innovation is not strong enough: our method is simple, which we see as a strength, and it shows significant improvements, like you mentioned. (This is without the additional improvement of the cross-lingual mapping, which brings an additional small benefit and is not the main result of our work.) Different from prior work, that augments a *multilingual* model with *bilingual datastores* [1] or uses a pre-trained multilingual model to create a *monolingual datastore* [2], we are the first to develop methods to carefully construct *multilingual datastores*, and we show that this significantly outperforms bilingual datastores. We further empirically show that multilingual datastore size can be limited while preserving quality compared to our largest multilingual datastore, which significantly increases inference speed.\n\n* There should be a baseline of a native multilingual datastore, that is, a multilingual model.\n* Have you compared with a multilingual model?\n\nRegarding your comment about the absence of a multilingual datastore / multilingual model baseline: this is incorrect, we *do* have these results in the paper, and we therefore disagree that this is a reason to reject. Please see Table 1 for these numbers: scores for the base multilingual model (i.e., the M2M100 model without datastore) are listed in the *first columns* (“base”), with datastore sizes of 0; scores for the base multilingual model + a complete multilingual datastore (i.e., the M2M100 model augmented with datastores consisting of all TED languages into English) are listed in the *last columns* (D_{ALL}), with datastore size of 125M. We discuss these results in Section 4.3. With regards to the baselines, we observe that using any kind of datastore always outperforms the base model (as one would expect). More importantly, we show that using our complete multilingual datastore (D_{ALL}) works better than bilingual (cross-lingual) datastores for all languages except Ru-En. We then show that datastores constructed using linguistic similarity (D_{LG}) allows for significantly smaller datastores (resulting in faster inference of up to 5.3x, see Figure 1 in Appendix), while being on par with the complete datastore in terms of BLEU.\n\n[1] Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer and Mike Lewis. Nearest Neighbor Machine Translation. ICLR 2021.\n\n[2] Jiahuan Li, Shanbo Cheng, Zewei Sun, Mingxuan Wang and Shujian Huang. Better Datastore, Better Translation: Generating Datastores from Pre-Trained Models for Nearest Neural Machine Translation. arXiv 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea is clear and straightforward. The motivation is solid.\n2. The experiments show significant improvements.\n3. The detailed analysis is convincing.", "reasons_to_reject": "1. The improvement of cross-lingual mapping is relatively small (0.3 BLEU). The contribution of this part is a little bit minor. \n2. Similarly, the innovation is not strong enough.\n3. There should be a baseline of a native multilingual datastore, that is, a multilingual model.", "questions_to_authors": "Have you compared with a multilingual model?", "paper_topic_and_main_contributions": "The paper proposes a multilingual method for kNN NMT. The author mixes datastores of different language pairs. And use a linear mapping method to make the representation closer.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BRJEI46trv", "reviewer": "Reviewer_aszQ", "summary": "", "questions": "- the paper only shows xx->en results which might be different from en->xx results, so it would be good to also show some results on en->xx. Could you also add en->xx results too?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n* the proposed approach is a bit complicated, as it combines multilingual NMT with kNN-based NMT, so it might be hard for other people to follow;\n\nRegarding your comment that our approach seems a bit complicated; we respectfully disagree. We agree with Reviewer oGvF and Reviewer oA3E that our idea is clear and straightforward, resulting in a simple yet effective method. A multilingual NMT system is a single system that can serve a large number of languages. Adding kNN is an extra step, which requires the construction of datastores. In contrast to [1], our approach allows us to use a *single multilingual datastore* for a large number of languages, which is much simpler than creating *separate bilingual datastores* per language direction. Note that we will release our code (which is based on the commonly used fairseq framework) to the community, which makes reproducing results and building on our approach very simple. In addition, we will expand Section 2 (i.e., the section that explains kNN-MT) when our paper gets accepted, since we will get additional space. We hope this change will make our paper even easier to follow.\n\n* the paper only shows xx->en results which might be different from en->xx results, so it would be good to also show some results on en->xx;\n* the paper only shows xx->en results which might be different from en->xx results, so it would be good to also show some results on en->xx. Could you also add en->xx results too?\n\nRegarding your question about showing en->xx results in addition to xx->en; this is a good point. We note that *k*NN-MT builds datastores consisting of source context (hidden representations) and target tokens. Therefore, creating cross-lingual/multilingual datastores that (partly) consist of target tokens that are from a different target language will not lead to good results. As an extreme example, if we create a multilingual datastore consisting of {En-Fr, En-Zh} data, the En-Zh portion is not beneficial for En-Fr translations because of lack of overlap. We will add some en->xx results plus this explanation in the updated version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- this paper should have enough contents and results for a short paper;\n- the improvements on low-resource languages look good in BLEU scores;\n- the experiments are comprehensive;\n- the paper is well written;", "reasons_to_reject": "- the proposed approach is a bit complicated, as it combines multilingual NMT with kNN-based NMT, so it might be hard for other people to follow;\n- the paper only shows xx->en results which might be different from en->xx results, so it would be good to also show some results on en->xx;", "questions_to_authors": "- the paper only shows xx->en results which might be different from en->xx results, so it would be good to also show some results on en->xx. Could you also add en->xx results too?", "paper_topic_and_main_contributions": "This paper tries to combine multilingual NMT with kNN-based NMT to achieve better translation quality for low-resource languages.\n\nMajor contributions:\n- proposed an approach to augment a multilingual NMT model with datastores of related and unrelated languages;\n- showed the approach can improve low-resource translation quality;\n- a tradeoff between quality and speed;", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 240: \"a first attempt\" should be \"the first attempt\";"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes using kNN-MT for low-resource languages. Most prominently, it combines representations from multiple different languages into the same datastore --- traditional kNN-MT uses huge datastores built for one language. The work contains extensive experimentation and reviewers are in consensus for acceptance.
Decision: Accept-Main