Nearest Neighbor Machine Translation
Authors: Urvashi Khandelwal
Angela Fan
Dan Jurafsky
Luke Zettlemoyer
Mike Lewis
Venue: ICLR 2021 Poster (2021)
Keywords: nearest neighbors
machine translation

Abstract: We introduce $k$-nearest-neighbor machine translation ($k$NN-MT), which predicts tokens with a nearest-neighbor classifier over a large datastore of cached examples, using representations from a neural translation model for similarity search. This approach requires no additional training and scales to give the decoder direct access to billions of examples at test time, resulting in a highly expressive model that consistently improves performance across many settings. Simply adding nearest-neighbor search improves a state-of-the-art German-English translation model by 1.5 BLEU. $k$NN-MT allows a single model to be adapted to diverse domains by using a domain-specific datastore, improving results by an average of 9.2 BLEU over zero-shot transfer, and achieving new state-of-the-art results---without training on these domains. A massively multilingual model can also be specialized for particular language pairs, with improvements of 3 BLEU for translating from English into German and Chinese. Qualitatively, $k$NN-MT is easily interpretable; it combines source and target context to retrieve highly relevant examples.

Published as a conference paper at ICLR 2021
NEAREST NEIGHBOR MACHINE TRANSLATION
Urvashi Khandelwal' Angela Fan', Dan Jurafsky', Luke Zettlemoyer', Mike Lewis!
1 Stanford University
Faceboook AI Research
(urvashik, jurafsky (errssky@sstafooott edu angelafan, 1sz, mikelewiss sateelewiseee 8 com
ABSTRACT
We introduce nheihbrtt machine translation (ANN-MT), which predicts tokens with a nearest neighbor classifier over a large datastore of cached examples, using representations from neural translation model for similarity search. This approach requires no additional training and scales to give the decoder direct ac- cess to billions of examples at test time, resulting in a highly expressive model that consistently improves performance across many settings. Simply adding nearest neighbor search improves state-of-the-an 1.5 BLEU. KNN-MT allows single model to be adapted to diverse domains by using domain-specific datastore, improving results by an average of 9.2 BLEU over zero-shot transfer, and achieving new state-of-the-art results cesuhts-without train- ing on these domains. A massively multilingual model can also be specialized for particular language pairs, with improvements of BLEU for translating from English into German and Chinese. Qualitatively, &NN-MT easily interpretable; it combines source and target context to retrieve highly relevant examples.
Ferman-Engliss translation model by
INTRODUCTION
Non-parametric methods have recently been successfully applied to tasks such as language modeling
(Khandelwal et al. 2020) and question answering (Guu et al., 2020 Lewis et al. 2020). They allow models that are (1) expressive, because they can
arbitrary amount of data at test time; (2)
adaptable, because predictions can be controlled by changing the datastore, and (3) interpretable, because the data used to make the prediction simple con-paraettric method for machine translation (MT) using nearest neighbor retrieval. ANN- MT' be added to any pre-trained neural translation model without further training, and signifi- cantly improves performance for in-domain, out-of-domain, and multi-lingual evaluations.
be directly inspected. We introduce &NN-MT,
More specifically, ANN-MT interpolates the target-token softmax distribution from neural MT model with a multinomial generated using nearest neighbor search over examples cached in a data store. The cache is over translation contexts (i.e. the complete source and prefix of the target). and is indexed by hidden states computed from the base MT model. We hypothesize that contexts which are close in representation space are more likely to be followed by the same target word. We show this is not only true for the original training data, thereby improving base model performance, but across a range of different bi-text corpora, allowing for simple and effective model adaptation.
Our work builds upon recent results showing the effectiveness of nearest neighbor methods in uncon- ditional language models (Khandelwal et al. 2020). We generalize to conditional language models, by using both source and target context, and show nearest neighbour models can be effective for generation in addition to density estimation. Compared to prior work on non-parame parammettii methods for MT. our approach is arguably simpler (in that it requires training, compared to Gu et al. (018)) and more expressive (in that it provides access to billions of key-value pairs during infer- ence, compared to Zhang al. (2018); Gu al. (2018))
Extensive experiments show that ANN-MT scales to datastores containing billions of tokens, im- proving results across range of settings. For example, improves state-of-the-art German- English translation model by 1.5 BLEU, ANN-MT can also be used to adapt a single model to Work done while the first author was interning Facehook AT Research.

---

(s SM,). Published as a conference paper at ICLR 2021
Training Transiation Contexts
Representation f(
Datastore
Target
Distances
Nearest
Temperature d' 6,7r
Normalization
(6 (
dj d
p(l,) rx expl (
J'ei
been
my been been
my been been
my been
l'aveis éte maison d'apprdcie Vone.
been
0.3
0.32
I enjoy
summner
100
0.4
been
0.28
J'al chambre.
I have
Aggregation
Input
Generated
Representation
Target
PaNN((s)
pumay ((8))
tokens
Dh:r-1
Jal dans ma propro chambre.
have
Deen
0.6
Figure I: An illustration of how the KNN distribution computed. The datastore, which con- structed offline, consists of representations of training set translation contexts and corresponding larget tokens for every example in the parallel data. During generation, the query representation, conditioned on the test imput as well as previously generated tokens, used retrieve the nearest neighbors from the datastore, along with the corresponding target tokens. The distance from the query used to compute distribution over the retrieved targets after applying a softmax tempera- ture. This distribution is the final kNN distribution.
diverse domains by simply adding a domain-specific datastore-impronn improving results by average of 9.2 BLEU over the base model out-of-domain, and even outperforming existing models that train on these domains. Finally, Language-pair-speceee particular language pairs, with improvements of 3 BLEU for translating English into German and Chinese. We find that retrievals from &NN-MT are typically highly contextually relevant.
datastores are used to adapt multilingual model to
2 NEAREST NEIGHBOR MACHINE TRANSLATION
ANN-MT involves augmenting the decoder of a pre-trained machine translation model with a nearest neighbor retrieval mechanism, allowing the model direct access to a datastore of cached examples. The translation is generated word-by-word at each time step, we find the most similar contexts in the datastore, and compute a distribution over the corresponding target tokens, as shown in Figure This distribution is then interpolated with the output distribution from the pre- pre-trained MT model.
More specifically, given an input sequence tokens in a source language neural MT model outputs a sequence of tokens f = (f1,..., (.M,) in the target language. When using autoregressive decoders, the output distribution for each token & in the target sequence is conditioned on the entire source sequence as well as the previous target tokens, p(t,|s, t1.-1). Let (s, (1 be the translation context and t be the target token.
Datastore creation Our datastore is constructed offline and consists of set of key-value pairs. The key is nigh-dimensional representation of the entire translation context computed by the MT decoder, f(s f1:-1), where f represents mapping from input of the decoder. The value is the corresponding ground truth target token f, For a parallel text collection (S, 7), the representations are generated by single forward pass each example and the complete datastore is defined as follows:
intermediate representation
(AC, V) = {(f(s, t1x-1), t.), Vt, Et (s, E (S,T)}
(1)
Tokens from the source language are not stored directly as values in the datastore. Conditioning on the source is implicit via the keys, and the values are only target language tokens.
Generation At test time, given a source x, the model outputs a distribution over the vocabulary PMr(y (1s-1) for the target y, every step of generation, where represents the generated tokens. The model also outputs the representation f [x, W)*-1), which is used to query the datastore for the l nearest neighbors V according to squared-LZ distance, d. In practice, the search over billions of key-value pairs is carried out using FAISS (Johnson et al. 2017). a library for fast nearest neighbor search in sigh-dimensional spaces.
The retrieved set is converted into a distribution over the vocabulary by applying a softmax with temperature T' to the negative distances and aggregating over multiple occurrences of the same
2

---

Published as a conference paper at ICLR 2021
cabulary item. Using temperature greater than one flattens the distribution, and prevents overfitting to the most similar retrievals.
1 mu exp (=d(ki, f(z, (1:--1))
PaNn(yilor, 11 x
Σ
(2)
T
(Agi05)EN
While a pure ANN approach is effective, improve results by interpolating with the base model distribution, which is more robust in cases without relevant cached examples. The model and ANN distributions are interpolated with tuned parameter A, resulting in the final &NN-MT distribution:
p(ul2,, 1) > PINN(Wi -1:i1) .) (1 - X) pwT(Vi D11-1)
The complete translation is generated using beam search.
ANN-MT ANN-LM ANN-MT is a generalization of KNN-LM applied conditional sequence generation, with a few important differences. First, the keys are not only conditioned on prior con- text, but also the source sequence (here, in different language). This means that the representa- tions must encode both source and target context; we show examples in Section 6 Second, there is an additional tuned parameter, the softmax temperature. Higher temperatures flatten the distribution and allow for greater diversity without overfitting to the retrieved contexts, shown in Section 6
3 EXPERIMENTAL SETUP
We experiment with ANN-MT in three settings: (1) single language- pariag translation, (2) multilingual MT and (3) domain adaptation.
Data We use the following datasets for training and evaluation.
19 For the single language-pain experiments, we use WMT' 19 data for German-Ein Englishh.
CCMATRIX: We train our multilingual model on CCMatrix (Schwenk et al. 2019), containing parallel data for 79 languages and 1,546 language pairs. The parallel sentences are mined from cleaned monolingual commoncrawl data created using the ccNet pipeline (Wenzek et al. (019). Semantically similar sentences in different languages are aligned using learned distance measure; examples where the distance measure is at least 1.06, resulting in billion sentence-pairs
NEWSTEST: The newstest2018 and newstest2019 test sets from WMT (Bojar et al. 2018 Barrault al. 2019) are used as validation and test sets for the multilingual experiments. The same German- English validation and test sets are also used for evaluation in the single language-pal and domain adaptation experiments.
We use the Ted Talks data prepared by Qi et al. (2018) for evaluation in the multilingual setting, particularly explore performance for language pairs that do include English.
TED TALKS:
MULTI-DOMAINS: We use the multi-domains dataset (Koehn & Knowles 2017). re-split by Aharoni & Goldberg 2020 for the domain adaptation experiments. It includes German- German-English parallel data for train//alidatiion/te sets in five domains: Medical, Law, IT, Koran and Subtitles.
Models For the single language-pair and domain adaptation experiments, we use the WMT' 19 German-Englist news translation task winner (Ng et al. 2019). available via the FAIRSEQ library (Ot al. 2019 is a Transformer encoder-decoder model Vaswani al. 2017) with 6 layers, 1,024 dimensional representations, 8, 8.192 dimensional feedforward layers and 8 attention heads. Apart from WMT' 19 training data, this model trained on over 10 billion tokens of backtranslation data and fine-tuned newstest test sets from years prior to 2018. In this work, we do not use ensembles or n-best reranking.
For multilingual MT, trained 418M parameter Transformer- based encoder-decoder model on the CCMatrix data for 100K updates. The model has embedding dimension 1,024, hidden dimen- sion 4,096, 12 layers in both the encoder and decoder, with 16 attention heads. To balance the

---

Published as a conference paper at ICLR 2021
training of different language pairs, which have various resource levels, we apply temperature up- sampling with T = 5 (Ariazhagan et al. 2019). The vocabulary is shared across all languages and consists of 128K subwords extracted using sentencepiece (Kudo & Richardson, 2018 All results use case-sensitive detokenized BLEU, measured using SACREBLHU (Post 2018). We pro- vide the SACREBLEU signatures, along with details on the statistical power of our experiments, in Appendix [
KNN-MT In this work, we use FAISS index to represent the datastore and search for nearest neighbors. The keys are stored in clusters to speed up search and quantized to 64-bytes for space efficiency (the full-precision keys are discarded). The index constructed offline via single for- ward pass over every example in the given parallel text collection. We use the 1024-dimensional representation input to the final layer feedforward network as the key. Building the index involves training phase to learn the cluster centroids. We use 5M keys for learning 13 cluster centroids for the multilingual experiments, and IM keys for 4K clusters for in-domain data in the domain adaptation experiments. During inference, we query the datastore for 64 neighbors while searching 32 clusters. The interpolation and softmax temperature parameters are tuned on the validation sets
Computational Cost While ANN-MT does not add trainable model parameters, it does add some computational overhead. The primary cost of building the datastore is a single forward pass over all examples in the datastore, which is fraction of the cost for training on the same examples for one epoch. During inference, retrieving 64 keys from a datastore containing billions of items results in a generation speed that two orders of magnitude slower than the base MT system. Generation speed can be improved by searching fewer clusters, using smaller beams, or querying smaller datastores, with relatively minor trade-offs in performance, as we will see in Section 5 Developing faster nearest neighbor search tools remains an active area of research (Guo et al. 2020).
4 EXPERIMENTS
4.1 SINGLE LANGUAGE-PAIE TRANSLATION
To test whether ANN-MT can improve model's ability to generalize from its training data, we first apply it to a state-of-thee -o-thhe-rt translation model, using datastore containing only the original training set. We use a state-of-the-ar German- German-English model as our base MT system, which scores 37.59 BLEU on the newstest2019 test set ff This is a highly competitive baseline apart from the WMT' 19 training data, the base model has also been trained on over 10 billion tokens of extra backtranslation data as well fine-tuned on newstest test sets from previous years. Providing this heavily tuned base model with a datastore containing about 770M tokens of WMT' 19 training data improves performance by 1.5 BLEU to 39.08, without any additional training. This result shows that even very strong translation models can be improved with test-time access to training sets.
4.2 MULTILINGUAL MACHINE TRANSLATION
Next, we apply ANN-MT multilingual machine translation, to measure its ability to add capacity to model when using very large training sets. For these experiments, we create datastores using subsets of the CCMatrix parallel data that the model has been trained on.
Retrieving neighbors from same source language data Here, we build one datastore per uuguugee-pirr being tested, using the training examples for that language-pair Table[]shows perfor- mance for the baseline and kNN-MT on 17 language-pairs from newstest2019. Retrieving neighbors results in up to 3 BLEU improvements for English-German English-Chinese and Chinese-English, with average improvement of 1.4 BLEU across all 17 pairs, without any additional training.
Table 0 also shows the sizes of each of the datastores. Datastore size and the increase in BLEU are only weakly correlated across languages, though within language, a larger datastore is decidedly
Evaluating our model the recently released OPUS100 1000 /hhang et al. 2020) corpus improves upon the result in Zhang al. (2020) by 0.4 BLEU, suggesting that is very strong baseline.
*Code for ANN-MT will be available at https: /// / t thub. com/ urr vashiki k/kkknn 1m
"The winning system (scoring 40.8 BI. BI.FU) extends this model with ensembling and n-best reranking.
4

---

Published as a conference paper at ICLR 2021
de-en
ru-en
zh-en
ja-en
fi-en
It-en
de-fr
de-cs
en-cs
Test set sizes
2,000
2,000
2,000
993
1,996
1,000
1,701
1,997
2,000
Base MT
34.45
36.42
24.23
12.79
25.92
29.59
32.75
21.15
22.78
+KNN-MT
35.74
37.83
27.51
13.14
26.55
29.98
33.68
21.62
23.76
Datastore Size
5.56B
3.80B
1.19B
360M
318M
168M
4.21B
696M
533M
en-de 1,997
en-ru
en-zh
en-ja
en-fi
en-It
fr-de
cs-de
Avg.
Test set sizes
1,997
1,997
1,000
1,997
998
1,701
1,997
Base MT
36.47
26.28
30.22
21.35
21.37
17.41
26.04
22.78
26.00
+KNN-MT
39.49
27.91
33.63
23.23
22.20
18.25
27.81
23.55
27.40
Datastore Size
6.50B
4.23B
1.13B
433M
375M
204M
3.98B
689M
Table I: Multilingual machine translation with ANN-MT. All test sets are from newstest2019, except ja-enlen-ja which are from newstest2020. Adding ANN-MT increases BLEU scores in all cases, and by over 3 points for en-de, zh-en and en-zh. Bold scores indicate significant results based on statistically powered experiments (Card et al. 2020).
Ted Talks
Newstest2019
Avg.
de-ja
ru-ja
uk-ja de-ru
de-zh
fr-de
es-de
de-cs
Test set sizes
4,442
5,090
3,560
4,288
4,349
1,701
1,997
1,997
Base MT
10.11
9.69
8.36
17.24
20.48
26.04
22.78
21.15
16.98
+KNN-MT (en-*)
11.08
10.42
9.64
18.02
21.22
27.85
23.71
21.74
17.96
Datastore Size
433M
433M
433M
4.23B
1.13B
6.50B
6.50B
533M
Table 2: Adding datastores with English source-side data can improve translation from other lan- guages by average of I BLEU, suggesting that our representations generalize over different source langauges. The model's representations of the source generalize across languages and make cross- lingual retrieval effective.
better, as shown in Section This suggests that underlying factors, such as the quality of the parallel data used populate the datastore, may also factor into the size of the improvements from I-N-MT..
Finally, we also observe that improvements for languages translated into English, on average 1.23 BLEU, are lower than improvements for languages translated from English, average 1.94 BLEU. As English the most frequent language in the base model's training data, this suggests that kNN- MT is particularly useful for improving decoders languages that may be underfit during training.
Retrieving neighbors using English as the source language Here, we construct datastores from training examples where English is the source language, and the target language the language being tested. This setting is useful for rarer language pairs with less bi-text, and is related to pivoting (Utiyama & Isahara 2007| Cohn & Lapatal 2007). Table shows that on five pairs from the Ted Talks data and three from newstest2019, we find that KNN-MT improves performance by 1 BLEU on average. This result shows that the model's representations across languages to make cross-lingual retrieval effective. Further investigation is needed to study the extent to which multilingual representations from related and unrelated languages can improve translation performance via MNN-MT.
the source generalize well enough
4.3 DOMAIN ADAPTATION
We also measure the effectiveness of ANN-MT for domain adaptation, in which we use a domain- specific datastore to adapt the model, without further training. For these experiments, we use the Engllsh translation model from Section 4.1 base MT system and provide domain- specific data in the datastores. We also explore the effects of retrieving neighbors from a large amount of out-of-domain data well as from single multi-domain datastore.
5

---

Published as a conference paper at ICLR 2021
Newstest
Medical
Law
IT
Koran
Subtitles
Avg.
2019
Test set sizes
2,000
2,000
2,000
2,000
2,000
2,000
Aharoni & Goldberg (2020):
one model per domain
56.5
59.0
43.0
15.9
27.3
40.34
one model for all domains best data selection method
53.3
$7.2
42.1
20.9
27.6
40.22
54.8
58.8
43.5
21.8
27.4
41.26
Base MT
37.59
39.91
45.71
37.98
16.30
29.21
33.82
+
in-domain datastore
39.08
54.35
61.78
45.82
19.45
31.73
42.63
WMT" datastore
39.08
40.22
46.74
40.27
17.99
29.23
34.89
all-domains datastore
38.88
54.54
61.11
48.63
19.22
31.70
43.04
Datastore Size (in-domain)
770M
5.70M
18.3M
3. 3.10M
450K
159M
Domain adaptation using ANN-MT. The base MT system is trained WMT' data which is also treated as the in-domain data for newstest2019, kNN-MT improves the base model by an average of 9.2 BLEU, without training, achieve the best reported results this task.
Table 3:
Domain-specific datastores Table 3 shows the base MT system's 's in-domain performance on new- stest2019, as well as zero-shot transfer five other domains. ::N-MT significantly outperforms the base MT system in all settings. For the multi-domains dataset, kNN-MT improves the base MT model performance by an average of 9.2 BLEU, with improvements as large as 16 BLEU on Law and 14.5 BLEU on Medical, all without any further training. We also provide scores from Aharoni & Goldberg (2020) for models trained on in-domain data, those trained on all domains jointly, and those trained using the best-performing data selection method proposed by the authors. We find that ENN-MT also outperforms the best reported average on the multi-domains dataset by 1.4 BLEU.
Out-of-domain and multi-domain datastores Table also shows performance for retrieving neighbors from 770M tokens of WMT 19 data that the model has been trained on. While the average BLEU for the multi-domain data point higher, the improvements are much smaller compared to using in in-domain data. This illustrates the value of adding domain-specific data to the datastore over adding a large amount of arbitrary data. We also measure the effectiveness of building single multi-domai datastore containing parallel data from all six settings. Performance on IT improves by 3 BLEU but scores for the other domains are mostly the same. This shows that ANN-MT robust to the presence of out-of-domain examples since retrieving neighbors from datastore where large amounts data is out-of-domain does not hurt performance relative to using only in-domain data.
TUNING KNN-MT
We investigate how key hyperparameters affect multilingual kNN-MT on validation data. We pro- vide validation set BLEU scores as well as hyperparameter choices for our experiments in Ap- pendix A
Softmax temperature A softmax temperature is used when estimating the nearest neighbor distri- bution in order to prevent the model from assigning most of the probability mass to single neighbor, thus hurting diversity. Values greater than 1 will flatten the distribution, which can improve ANN- MT performance. Figure that temperature of results in significantly lower BLEU scores. For all of our experiments, values of either 10 or 100 prove to be optimal.
Number of neighbors per query our experiments, we the value of k. the number of neigh- bors retrieved per query, to 64. For fixed temperature and interpolation parameter,
find that performance does not improve when retrieving larger number of neighbors, and in some cases, per- formance deteriorates. This suggests that retrieving generation process. (sgurs that in some cases, performance improves when retrieving fewer neighbors, and further gains may be possible by tuning this parameter.
neighbors add noise the sequence

---

Published as a conference paper at ICLR 2021
en-2h
Datastore size: en-2h=1.138 ru-en=3.88
Diferrece in BLEU score
813 5000
Base MT
---
Temperature=1
35
Temperature-11
Temperature=110
Temperature-
33
KNN MT for en KNN-MT for
32
0.5 Fraction Size Datastore
Number neightors
Figure 2: Effect of the number of neighbors trieved and the softmax temperature on the val- idation BLEU score for en-zh. Temperatures greater than 1 are important to prevent the model from overfitting to the most similar neighbor. For higher temperatures, more neighbors do not al-
Figure 3: Effect of datastore size the valida- tion BLEU score for ru-en and en-zh. Perfor- mance improves monotonically with size but re- trieval can be slow for datastores containing bil- lions of tokens. Smaller datastores, which ac- count for large fraction of the improvement, can be used for faster retrieval.
ways result in improvements.
Datastore size Figure shows increasing the size of the datastore improves translation perfor- mance. However, larger datastores result in slower retrieval, indicating a ppeed-performance trade- off. Much of the benefit can be realized with much smaller, and correspondingly faster, datastores.
QUALITATIVE ANALYSIS
To better understand MNT, we examine the retrievals for several examples. We use the German- English model and generate with only the kNN distribution (A = 1) with beam size 1, retrieving neighbors from the News Commentary and Common Crawl subsets of WMT' 19 data.
Figure 4 shows an example from newstest2018 where all the retrieved neighbors map to the same target, military. Many of the retrieved examples include phrases similar to tamed the military such as Autorität gegenüber dem Militär, Kontrolle des Militärs and das Militär gezwungen on the source side and authority over the military, control over the military and forced the military given the local target side context, but differ sharply in their longer context, often describing different nations and centuries. We provide additional examples illustrating this point in Appendix
Another interesting observation is that ANN, able to reconstruct named entities that are split into multiple subword tokens, even if that particular named entity does not appear in the datastore. One such example is the name Haysom that is split into subwords Hay and som. The retrieved neighbors for the first subword token include examples that contain the names Hayes and Haydn, while those for the second include Grissom and Folsom, showing subword representations are used effectively in the nearest neighbor search.
when not interpolated with the base model, is
7 RELATED WORK
Retrieval in Translation Recent work has integrated retrieval of words and phrases into neural translation, to gain some of the advantages of the previous generation of word- and phrase-based methods (Brown et al. 1993 Koehn et al.] 2003). For example, Zhang et al. (2018) proposed guiding models by retrieving 7-grams and up- weightttting the probabilities of retrieved tokens. [Tu et al. (2018) use cache-based models (Grave al. 2017a save and retrieve translation histories, so models can adapt to changing contexts. Compared to these, ANN-MT has several advantages for instance, the external datastore only needs to be created once, whereas the cache model requires constant writes. Further, ANN-MT scales retrieval to orders-on of of-mannitude larger datastores, while taking advantage of neural context representations.
Other work has retrieved complete example translation sentences at test time. Nagao 1984) pro- posed example-hased MT' for translating sequences by analogy. Before deep learning widely
7

---

Published as a conference paper at ICLR 2021
Dabei schien es, habe Erdogan das Militär gezähmtt Generated tokens: In doing so, it seems as if Endogan has tamed the
Test Input:
Training Set Target
Context Probability
Training Set Translation Context (source and target)
Dem charismatischen Minis- terpräsidenten Recep Tayyip Erdogan, der drei aufeinanderfol- gende Wahlen für sich entscheiden konnte, ist es gelungen seine Autorität gegenübber dem Militär geltend zu machen.
The charismatic prime minister, Re- cep Tayyip Erdogan, having three consecutive elections, has been able to exert authority over the
military
0.132
Fin bemerkenswerter Fall war die Ermordung des gemüßligten Pre- mierministers Inukai Tsuyoshi Jahre 1932. die das Ende jeder wirklichen zivilen Kontrolle des Militürs murkiert.
One notable case was the assas- sination of moderate Prime Minis- Inukai Tsuyoshi in 1932, which marked the end of any real civilian control of the
military
0.130
Sie sind Teil eines Normal- isterungsprozesses und der Her- stellung der absoluten zivilen Kontrolle üher das Militär und bestätigen das Prinzip. dass niemand über dem Gesetz stelt.
They are part of a process of nor- malization, of the establishment of absolute civilian control of the
military
0.129
Diese hart formulierte Erklärung wurde als verschleierte, jedoch un- missverstündllihe Warrung ange- sehen, dass das Militär bereit wäre einzuschreiten...
That toughly worded statement was seen as veiled but unmistakable warning that the
military
0.123
Final ANN distribution: military = 1.0
Final Translation: In doing so, Erdogan seemed to have tamed the military. Reference: In doing so, it seems Erdogan has tamed the military.
Example retrievals using kNN-MT. Not only do the retrievals all correctly predict the target word military, but the local contexts tend to be semantically related. Both the source and the three nearest retrievals express the concept of control over the military.
Figure 4:
adopted, this approach was extended to identifying portions of the training data that could be trans- lation based edit distance (Doi et al.) 2005), matching training examples based on local trigram contexts (yan den Bosch et al. 2007), using phrase-based memories (van Gompel et al. 2010) and incorporating syntactic features when retrieving similar examples (Stroppa et al. 2007 Haque al, 2009]. Recently, Gu et al. (0118) proposed a model that retrieves examples similar to the test source sequence and then attends over this subset of retrieved source-target pairs the token level, while generating translations. Bulte Tezcan (2019) and Xu et al.) (2020) use fuzzy-matching with trans- lation memories and augment source sequences with retrieved marce-ttrrgt pairs. These techniques face challenges in identifying relevant retrieval candidates, as they focus on sentence-level retrieval. In contrast, ÀNN-MT focuses on token level retrieval from billions of key-value pairs, meaning that each word can retrieve the most relevant examples for its translation.
Finally, various studies have explored retrieving additional information improve domain adap- tation, often using lexicons (Hu et al.) 2019), Nomainn-ddaphe training (Farajian et al., (017)) or attending over neighbors similar n-grams in the source (Bapna & Firat] 2019). These modifica- tions require additional training, whereas kNN-M'T provides the flexibility to use different datastores when decoding in different domains, keeping the model fixed.
Retrieval in Text Generation Retrieval mechanisms have also been applied to generation tasks more broadly. WWstton et (2018) and et al. (2020) improve dialogue response generation sys- tems by retrieving examples and concatenating them to model inputs. Lewis et al. (2020) improve
8

---

Published as a conference paper at ICLR 2021
open-domain question answering systems by retrieving relevant contexts from Wikipedia and con- catenating them to the inputs. Hashimoto et al. (2018) use a -ed framework to generate structured outputs such as code, by jointly training the editor and retriever. For kNN-MT, retrieval results in a distribution over the vocabulary that is used for generation directly and does not require further training or providing the retrieval candidates input.
CONCLUSION
We introduced a simple and effective method that can be applied to any neural MT model without further training. We show that similar contexts in a model's embedding space are more likely to be followed by similar next words, allowing the model to be improved by interpolation with a nearest neighbor classifier. The approach improves a state-of-the-art model in-domain, leads to large gains out-of-domaij and can specialize a multilingual model for specific language-paiie paana.....rs......... Future work should improve efficiency, for example by sown-sampling frequent target words in the datastore.
ACKNOWLEDGMENTS
The authors thank Kartikay Khandelwal for thoughtful discussions, Holger Schwenk and Sergey Edunov for sharing data and model details, and Matthijs Douze for answering questions about FAISS.
REFERENCES
Roee Aharoni and Yoav Goldberg. Unsupervised domain clusters in pretrained language models. In Association of Computational Linguistics (ACL), 2020.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint 100iv:9907.0500 2019.
Ankur Bapna and Orhan Firat. Non-parametric adaptation for neural machine translation. In North American Association of Computational Linguistics (NAACL), 2019.
Loic Barrault, Ondfej Bojar, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 con- ference machine translation (WMT19). In Proceedings of the Fourth Conference on Ma- chine Translation (Volume 2: Shared Task Papers, Day 1), pp. 1-61, Florence, Italy, Au- gust 2019. Association for Computational Linguistics. doi: 10. 10.18663/11//119553011 URL https: www.aclleeb..orr/amtooooooooo www.aahoooogggoooo//o/e3 W19-5301]
Ondfej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. Findings of the 2018 conference on machine translation (WMT18). In Pro- ceedings of the Third Conference on Machine Translation: Shared Task Papers, pp. 272-303, Belgium, Brussels, October 2018. Association for Computational Linguistics. doi: 10. 18653vvv W18-6401. htRLhttps:/ www. ac wtwp://wwb/r/////.oooo.oooo/oo/.
8 111611111116111
Peter F Brown, Stephen A Della Pietra, Vincent J Della Pictra, and Robert L Mercer. The mathe- matics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2): 263-311, 1993.
Bram Bulte and Arda Tezcan. Neural fuzzy repair: Integrating fuzzy matches into neural machine translation. In Association of Computational Linguistics (ACL), 2019.
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Juraf- sky. With little power comes great responsibility. Empirical Methods in Natural Language Processing (EMNLP), 2020.
Trevor Cohn and Mirella Lapata. Machine translation by triangulation: Making effective use of multi-paralle corpora. In Proceedings of the 45th Annual Meeting of the Association of Compu- tational Linguistics, pp. 728-735, Prague, Czech Republic, June 2007. Association for Computa- tional Linguistics.
: www. a web. hww.bee.rr///ataooooo/////011000

---

Published as a conference paper at ICLR 2021
Takao Doi, Hirofumi Yamamoto, and Eiichiro Sumita. Example-base based machine translation using efficient sentence retrieval based on edit-distance. ACM Transactions on Asian Language Infor- mation Processing, 4(4):377-399, 2005.
Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. Augmenting transformers with knn-based composite memory for dialogue. arXiv preprinl htti::0004.2244:5 2020.
M Amin Farajian, Marco Turchi, Matteo Negri, and Marcello Federico. Multi-domain neural ma- chine translation through unsupervised adaptation. In Proceedings of the Second Conference on Machine Translation, pp. 127-137, 2017.
Edouard Grave, Moustapha M Cisse, and Armand Joulin. Unbounded cache model for online lan- guage modeling with open vocabulary. In NIPS, pp. 6042-6052, 2017a.
Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous cache. In International Conference on Learning Representations (ICI.R), 2017b.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. Search engine guided neural machine translation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Ku- mar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning (ICML), 2020.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint MiXiv:2022.88909 2020.
Rejwanul Haque, Sudip Naskar, Yanjun Ma, and Andy Way. Using supertags as source language context in smt. 13th Annual Conference of the European Association for Machine Translation, (EAMT), 01 2009.
Tatsunori B. Hashimoto, Kelvin Guu, Yonatan Oren, and Percy Liang. A retrieve-and-edit frame- work for predicting structured outputs. Advances Neural Information Processing Systems (NeurIPS), 2018.
Junjie Hu, Mengzhou Xia, Graham Neubig, and Jaime G Carbonell. Domain adaptation of neural machine translation by lexicon induction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2989-3001, 2019.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. arXiv preprint arXiv: 2017.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generaliza- tion through memorization: Nearest neighbor language models. In International Conference on Learning Representations
( (ICLR), 2020.
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. In First Work-
shop on Neural Machine Translation (WNMT), 2017.
Philipp Koehn, Franz Och, and Daniel Marcu. Statistical phrase-based translation. Technical report, UNIVERSITY OF SOUTHERN CALIFORNIA MARINA DEL REY INFORMATION SCIENCES INST, 2003.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP), 2018.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-inlensive nlp tasks. arXiv preprint arXiv:2005.11401, 2020.
Makoto Nagao. A framework of a mechanical translation between japanese and english by analogy principle. In Artificial and human intelligence, 1984.
10

---

Published as a conference paper at ICLR 2021
Nathan Ng. Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. Facebook fair's wmt19 news translation task submission. In Fourth Conference on Machine Translation (WMT), 2019.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In North American Association of Computational Linguistics: System Demonstrations (NAACL), 2019.
Matt Post. A call for clarity reporting BLEU scores. Proceedings of the Third Conference Machine Translation; Research Papers, pp. 186-191, Belgium, Brussels, October 2018. As- acl www.bb.ooogaatoooog.....
sociation for Computational Linguistics. URL wwtpt:ttp///cc /ttp:://wwwww.. http://wwwwwwlllll.
Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. When and why are pre-trained word embeddings useful for neural machine translation? In North American Association of Computational Linguistics (NAACL). 2018.
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin. Cematrix: Mining billions of high-quality parallel sentences on the web. arXiv preprint arXiv:1911.04944, 2019.
Nicolas Stroppa, Antal van den Bosch, and Andy Way. Exploiting source similarity for smt using context-informed features. Proceedings of The 11th Conference Theoretical and Methodolog- ical Issues in Machine Translation, 01 2007.
Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. Learning to remember translation history with a continuous cache. In Transactions of the Association of Computational Linguistics (TACL), 2018.
Masao Utiyama and Hitoshi Isahara. A comparison of pivot methods for phrase-based statistical ma- chine translation. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pp. 484 491, Rochester, New York, April 2007. Associatioonnnoooo for Computational Linguistics. URL
https: www wttpw//wwwwww..w...oooooo///////
Antal van den Bosch, Nicolas Stroppa, and Andy Way. A wvemory-based classification approach to marker-based ebmt. METIS-II Workshop on New Approaches to Machine Translation, 04 2007.
Maarten van Gompel, Antal van den Bosch, and Peter Berck. Extending memory-based machine translation to phrases. In Computational Linguistics in the Netherlands (CIJN), pp. 79-86, 2010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz. Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NIPS), 2017.
Guillaume Wenzek. Marie-Anne Lachaux. Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Cenet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019.
Jason Weston, Emily Dinan, and Alexander Miller. Retrieve and refine: Improved sequence generation models for dialogue. arXiv preprint
2018.
Jitao Xu, Josep Crego, and Jean Senellart. Boosting neural machine translation with similar transla- tions. In Association of Computational Linguistics (ACL), 2020.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilingual neural machine translation and zero-shot translation. arXiv preprint arXhv:2004.11880 11867, 2020.
Jingyi Zhang, Masao Utiyama, Fiichro Sumita, Graham Neubig, and Satoshi Nakamura. Guiding neural machine translation with retrieved translation pieces. In North American Association of Computational Linguistics (NAACL), 2018

---

Published as a conference paper at ICLR 2021
de-en
ru-en
zh-en 3,981
ja-en
fi-en
It-en
de-fr
de-cs
en-cs
Test set sizes
2,998
3,000
1,998
3,000
2,000
1,512
3,000
2,983
Base MT
38.21
30.51
21.93
14.44
21.50
28.68
28.46
21.97
20.66
+NNN-MT
40.48
32.6
24.49
15.62
22.11
29.41
29.74
23.01
21.79
Datastore Size
5.56B
3.80B
1.19B
360M
318M
168M
4.21B
696M
533M
Interpolation (A)
0.6
0.5
0.4
0.4
0.2
0.2
0.6
0.4
0.3
Temperature (T)
10
10
10
10
10
10
100
100
10
en-de
en-zh 3,981
en-ja 1,998
en-fi 3,000
en-lt 2,000
fr-de 1,512
cs-de 3,000
Avg.
Test set sizes
2,998
3,000
Base MT
39.07
26.00
32.72
16.31
16.02
21.11
25.16
24.16
25.11
+ENN-MT
42.22
29.52
37.96
18.28
17.22
22.84
26.39
24.5
26.95
Datastore Size
6,50B
4.23B
1.13B
433M
375M
204M
3,98B
689M
Interpolation (A)
0.6
0.7
0.7
0.6
0.4
0.4
0.5
0.4
Temperature (T)
100
100
10
10
10
10
100
Table 4: Multilingual machine translation with ANN-MT the validation set. We show the the tuned interpolation parameter (A) as well as the tuned softmax temperature (T) for each language pair.
Newstest
Medical
Law
IT
Koran
Subtitles
Avg.
2019
Test set sizes
2,000
2,000
2,000
2,000
2,000
2,000
Base MT +ANN-MT:
48.07
39.94
45.78
35.78
16.30
29.74
33.51
in-domain datastore
48.57
53.12
61.58
42.41
19.67
32.28
41.81
Datastore Size (in-domain)
770M
5.70M
18.3M
3.10M
450K
159M
Interpolation (A)
0.4
0.8
0.8
0.7
0.8
0.7
Temperature (7)
100
10
10
10
100
10
Table 5: Domain adaptation using MNN-MT on the multi-domains validation data and newstest201 The base MT system trained on WMT' 19 data which also treated as the in-domain data for newstest2018. We present the interpolation (A) and softmax temperature (2") hyperparameter choices for each domain.
A HYPERPARAMETER TUNING
In this section, we present validation set results as well as the hyperparameter choices for the mul- tilingual machine translation and domain adaptation experiments. Only two hyperparameters have been tuned on the validation sets, the interpolation parameter À and the softmax temperature T', The number of neighbors k has been fixed 64, the number of clusters searched has been set 32 and the beam size has been set to 5. For the number of clusters in the FAISS index, preliminary experiments showed that for larger datastores, while using more clusters does hurt performance, it does significantly speed up the search process since searching within the clusters is exhaustive. Hence, we use 131K clusters for the multilingual experiments.
shows the validation set BLEU scores for the multilingual experiments well as the hyper- parameter choices, and Table 5 shows the same for the domain adaptation experiments using only the in-domain datastores. Values for the interpolation parameter lie between 0 and 1. We also note that for fixed value = 0.5, using KNN-MT either performs similarly to or improves the base MT model's performance, but never hurts, validation sets across the 17 language pairs evaluated in Section 4.2) For the temperature, we find that values of either 10 or 100 are optimal for all of our experiments.
12

---

Published as a conference paper at ICLR 2021
Test Input: Aber Papua sich wenig verändert, und heute fühlen sich die Finheimischen betrogen.
Generated tokens: But not much has
Context
Training Set Translation Context
Training Set Target
Probability
Nach einem schwer umkämpften Wahlkampf der deutlich über zwei Milliarden Dollar kostete, sieht für viele Beobachter aus, hätte sich in der amerikanischen Politik nicht viel geandert...
After hard-fought elec- tion campaign, costing well in excess $2 billion, it seems to many observers that not much has
changed
0.143
Geändert freilich hat sich wenig: Schlecht gehandhabte Kriege...
But not much has
changed
0.137
Kaum etwas sich veründert außer dass es jetzt nicht mehr die Bewohner des Appartement- Gebüuudds..
Not much has
changed
0.130
Es ist zwar richtig, dass sich seit dem Ausbruch der globalen Finanzkrise vor iüber vier Jahren und der schon 2010 verabschiedeten Dodd-Franke Fiaazzarkreformen den Vereinigten Staaten kaum etwas daran geändert hat...
True, while the global fi- nancial crisis erupted more than four years ago, and the Drad-Frank financial reforms were adopted the United States back in 2010, not much has
changed
0.121
Final ANN distribution: changed 1.0
Final Translation: But not much has changed in Papua, and locals feel betrayed today. Reference: But precious little has changed in Papua, and today local people feel betrayed.
An example where 6NN-MT retrieves the same target token, changed, across the neighbors. It matches local contexts on both the source and target sides, but not the global context regarding Papua.
Figure 5:
B ADDITIONAL EXAMPLES
Figure 5
further illustrates the behavior of ANN-MT using local contexts in both the source and target to retrieve nearest neighbors. ( a case where model has very little target-side prior context and mainly relies on the source context to retrieve the best neighbors.
C BLEU SCORES
In this paper, all results use case-sensitive detokenized BLEU, measured using SACREBLEU Posi 2018), with the following signatures:
General: BLEU+case mixed-numrefs www.ttt......oooo..... www.::t.........................
For chinese: BLEU+case mi htw.:eeeeeiueeeee....e.eeeeeeeee wtt.:/tto....tteeees..o.......
For japanese:
BLEU+case wumrrfftuuuff....mmmmmmmmmmm 1ts.otttt
m-cab- 0 996 IPAtversion. 1.113
For Table and other experiments, follow advice from Card al. 2020 regarding the statisti- cal power of machine translation experiments given the improvements in BLEU scores and the size of the dataset. The authors present results for a single language pair and we verify that their as- sumptions hold for couple of other language pairs. Specifically, we find that for Chinese-English Po 0.13 and bn 12, and for Bingishh-hhnnes P. 0.07 and 6, 16. This indicates that these experiments, with the test sets containing about 2,000 examples, have close to 100% power which was verified using the notebooks provided by Card et al. (2020). We refer the reader to the original paper for more details. More generally, experiments examples, with improvements of about BLEU or higher, are statistically powered.
datasets which contain about 2,000
13

---

Published as a conference paper at ICLR 2021
Test Input: Wir werden das Beste tun, mit dem, was wir haben.
Generated tokens: We
Training Set Target
Context Probability
Training Set Translation Context (source and target)
Wir werden versuchen zu beweisen, dass die We ein
will
0.145
Vermutung falsch ist, dies zu Gegenbeispiel Leinwände, dass die Aussage falsch ist, zu finden.
sich diese große Na- tion, wird diese Nation aussehen zu weit, wir werden das tun, was wichtig und wertvoll, Identity Wiederherstellen der Nation.
Allerdings, wenn
However, you look this great nation, this nation will look too wide and we
will
0.132
Wir werden alles tun, um Dinge für die Anfänger sehr einfach zu machen, während wir es den Experten erlauben, Dinge zu verändern, falls sie wollen.
will
0.127
"Wir werden ihre Fülle und die Fülle anderer politischer Gefangener vor die Gerichte brin- gen und das falsche Bild zerstören...
"We
are
0.127
Final &NN distribution: will = 0.639, are = 0.238, intend= 0.123 Final Translation: We will do the best
with what have. Reference: We'll the best we can with what we got.
Figure 6: An example where the model has access to very short amount of target-side context that ambiguous. A-NNMT is able rely on source context resolve this and generate the correct target token, will.
14


Reviews:
[{"review_id": "T_3Kyhbxvg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a nearest-neighbor enhancement to NMT, where internal token-level context representations are used to index into a large data store to find relevant (source, target prefix) pairs. Since the index representation is taken from a pre-softmax representation in the decoder network, no additional training of the NMT model is required. The authors show a diverse range of strong results, from improvements using a data store over the model’s own training data, to improvement from using a collection of domain-specific corpora not present during training used for domain adaptation, to language specific collections to improve capacity of multilingual models. They are also able to show by example how the model makes MT more interpretable.\n\nThis is a very strong paper. It's well-written and easy to read, the method is very novel to MT, and the results are great. The method isn’t practical right now (decoding is two orders of magnitude slower), but it’s very interesting and thought-provoking. I can imagine it influencing a lot of work, even if the actual method doesn’t see a lot of use.\n\nThe only complaint that I could imagine raising against this paper is that the method is not particularly novel in light of recent work on nearest-neighbor language modeling, but in this day and age, with so many papers available, I think it’s actually very important to make these incremental stops in neighboring fields to make the connections explicitly clear. All the great experiments on multilingual MT and domain adaptation also help a lot. To their credit, the authors provide a concise section discussing the changes that needed to be made for the conversion to conditional language modeling (MT).\n\nSmall concerns:\n\nThe exp(d) in Figure 1 is missing a negative: exp(-d).\n\nTable 1: what does the bolding indicate? It looks like statistical significance, but if so, please be clear about what test was used.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ey3paOnNj7_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents an approach to exploits at decoding time a very large translation memory to improve NMT. An extensive evaluation is performed along with some detailed analysis on the important parameters.\n\n\n\nStrengths:\n- the idea is very simple, very easy to understand, and intuitive\n- can be added to existing pre-trained NMT model\n- many interesting applications (domain adaptation, multilingual model specialization for instance) are presented and are mainly the reason why I think paper can be accepted for publication.\n- the paper is easy to read and well-written\n\nWeaknesses:\n-  exploiting a translation memory at test time is not novel (exploitation of billions of tokens is rather impressive but in my opinion making this possible is more an engineering problem)\n- the approach is described within one page, the remainder of the paper is about evaluation and analysis. For ICLR, the paper lacks of substance.\n- the improvements over SOTA English-German are very small considering that billions of tokens are exploited and the high decoding cost.\n- the experiments presented in this paper are not reproducible since unpublishable data are exploited to train the system (eg. CCMatrix)\n- computational cost at test-time is extremely high, as expected. This is probably why nobody tried it before. I do not see how it could be used for real-world applications. Focusing on reducing the computational cost would greatly improve the paper.\n\n\nQuestions/suggestions:\n- \"we also provide scores from Aharoni & Goldberg (2020)\": did you check that these scores are comparable with yours? It is unclear in the paper whether they also used sacreBLEU (insert the sacreBLEU signature in a footnote in your paper to help future work reusing your scores)\n- I recommend to add the decoding time in the tables and a description of the hardware used. Since the major issue of the proposed approach is its computational cost, adding the decoding time would probably encourage future work to try to improve it.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4ax9gDZt8gA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis submission introduces the kNN-MT approach for neural machine translation, which incorporates the memorize-catching spirit with the nearest neighbor classifier on a large datastore when generating the decoding sentences, together with the neural machine translation model for similarity search. No additional parameters are needed, but the inference cost increases. The authors conduct experiments are different settings, the single language pair translation, the multi-lingual machine translation, and the domain adaption translation. Results show that kNN-MT can easily improve translation performances by searching out related test sentences with non-trivial scores. \n\nComments:\n\nGenerally speaking, the submission is okay and the proposed approach has no big flaws, however, I feel hard to make this submission to be accepted. The main reasons or concerns are:\n1. It is clear that this submission is a direct and straightforward extension of the previously published ICLR-2020 paper: kNN-LM. As the authors also clearly stated in the abstraction. Therefore, in terms of the contributions and differences, they are quite limited. The technique is almost the same, except the key is added with the source language sentence. The presentation of this paper is also similar to kNN-LM. The direct extension of the kNN approach from Language Model to Neural Machine Translation makes me feel hard to recommend, and this makes much more like a technique report of the method extension. \n2. To say about the approach, I acknowledge that this method is effective, as the authors have done with multiple experiments. However, the computation cost is also high. The authors also discussed this in Section 3. It is hard for real-time systems to afford the increased inference cost as this approach made. The improved results with a little increased cost are okay, but too much is not a good choice. Though the authors mentioned there is a trade-off and I also acknowledge this, but it is still not clear what is a good trade-off. \n3. Also, this method highly depends on the scale of the dataset, also the similarity between training and test dataset, if I understand correctly. This assumption can hold for high resource translations, but for low resource translation, this would be limited. This is another drawback of these search-based algorithms. \n\nMinor question: What is the effect if $\\lambda$ is varied?\n\nTherefore, shortly speaking, I feel this paper is straightforward to extend from the previous paper (indeed this is the future work and the answer from the review comments of previous work). This concerns me a lot for another one in ICLR-2021. \n\n---------------\n\nUpdate:\nI thank the authors to give responses to my points, especially the discussion about novelty. But I still feel the success of KNN for NMT is similar for LM, that's why a lot of works study on NMT are also work on LM. Since this KNN method only targets at the decoder side, same as LM model. Therefore, I still feel not novel enough. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "0fHOaZiowT9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is an extension of [1]. The task in [1] is Language Modeling, while this paper is doing machine translation with the similar idea. The authors propose a non-parametric method for machine translation via a k-nearest-neighbor (KNN) classifier. Specifically, it predicts tokens with a KNN classifier over examples cached in a so-called datastore and this method can be applied to any pre-trained neural machine translation model without further training. The experiments show that it improves results across a range of settings (in-domain, out-of-domain, and multi-lingual evaluations).\n\nStrengths: \n+ The method is simple and can be applied to pre-trained neural machine translation model without further training.\n+ The experimental results across a range of settings are effective.\n\nWeaknesses:\n- Although the method is simple and does not add trainable parameters, it add the computational cost. The authors mentioned the computational cost briefly but there are no detailed experiments. It would be good to see the authors add more analysis on the computational cost, for example, how it varies with k.\n- Technical novelty over [1] seems to be incremental, where a large portion of the work is essentially regarding machine translation as a language modeling and applying the method in [1] to machine translation.\n\n\n[1] Khandelwal, Urvashi, et al. \"Generalization through memorization: Nearest neighbor language models.\" arXiv preprint arXiv:1911.00172 (2019).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends past work on kNN-augmentation for language modeling to the task of machine translation: a classic parametric NMT model is augmented with kNN retrieval from an external datastore. Decoder-internal token-level representations are used to index and retrieve relevant contexts (source + target prefix) that weigh-in during the final probability calculation for the next target word. Results are extremely positive across a range of MT setups including both in-domain evaluation and domain transfer. Reviews are thorough, but quite divergent. There is general agreement that the proposed approach is reasonable, well-motivated, and clearly described -- and further, that experimental results are both solid and relatively extensive. However, the strongest criticism concerns the paper's relationship with past work.  In terms of ML novelty, everyone agrees (including the paper itself) that the proposed methodology is a relatively simple extension of past work on non-conditional language modeling. However, two of the four reviewers strongly feel that, in light of the potentially prohibitive decoding costs, the positive experimental results are not sufficient to make this paper relevant to an ICLR audience given the lack of ML novelty. In contrast, another reviewer strongly takes an opposite stand-point:  rather, that the results will be extremely impactful to the MT subcommunity at ICLR since they are unexpected (i.e. that a non-parametric model might compete with highly-tuned NMT systems) and very positive across a range of domains and settings (i.e. in-domain, out-of-domain, multilingual) -- further, that the approach has substantial novelty in the context of MT where parametric models are the norm and that it might inspire substantial future work  (e.g. on efficient decoding techniques and further non-parametric techniques) given that it so drastically breaks the current MT mold. The final reviewer shares the concern of the former two about novelty, but is swayed by the experimental results and potential uses for the model (given kNN augmentation is possible without further training) and therefore votes for a marginal accept. After thorough, well-reasoned, and well-intentioned discussion between all four reviewers, the reviews land just barely in favor of acceptance, but with substantial divide. After considering the paper, reviews, rebuttal, and discussion I am swayed by the argument that (a) these experimental results are largely unexpected, (b) they are both extremely positive and offer a new trade-off between test and train compute in MT, and (c) that the paper may therefore inspire substantial discussion and follow-up work in the community. Thus I lean in favor of acceptance overall.
Decision: Accept (Poster)