Nearest Neighbor Machine Translation is Meta-Optimizer on Output Projection Layer
Authors: Ruize Gao
Zhirui Zhang
Yichao Du
Lemao Liu
Rui Wang
Venue: EMNLP 2023 Main (2023)
Keywords: Nearest Neighbor Machine Translation
meta-optimization
domain adaptation
Neural Machine Translation
TL;DR: We provide insights into the working mechanism of kNN-MT as a specific case of fine-tuning, which implicitly execute gradient descent on output projection layer of NMT and conduct experiments to compare the performance of kNN-MT and fine-tuning.

Abstract: Nearest Neighbor Machine Translation ($k$NN-MT) has achieved great success in domain adaptation tasks by integrating pre-trained Neural Machine Translation (NMT) models with domain-specific token-level retrieval. However, the reasons underlying its success have not been thoroughly investigated. 
In this paper, we comprehensively analyze $k$NN-MT through theoretical and empirical studies. Initially, we provide new insights into the working mechanism of $k$NN-MT as an efficient technique to implicitly execute gradient descent on the output projection layer of NMT, indicating that it is a specific case of model fine-tuning.  Subsequently, we conduct multi-domain experiments and word-level analysis to examine the differences in performance between $k$NN-MT and entire-model fine-tuning. Our findings suggest that: ($i$) Incorporating $k$NN-MT with adapters yields comparable translation performance to fine-tuning on in-domain test sets, while achieving better performance on out-of-domain test sets; 
($ii$) Fine-tuning significantly outperforms $k$NN-MT on the recall of in-domain low-frequency words, but this gap could be bridged by optimizing the context representations with additional adapter layers.

Nearest Neighbor Machine Translation is Meta-Optimizer on Output
Projection Layer
Ruize Gao
Zhirui Zhangat Yichao Du
Lemao Liu?
Rui Wang"!
Shanghai Jiao Tong University TTeneett AI Lab
3 University of Science and Technology of China
ruizzgaonlp@gaiil com, wangrui 12@sjtu. edu. 2zrustc1 1@gmai com, redmonndliutennent.cco com "duyichao@ne wst..sttcoooo... edu. cn
Abstract
et al., 2015; Vaswani et al., 2017; Hassan et al., 2018) with token-level n-nearest-niighhour re- trieval mechanism. This allows for direct access to domain-specific datastores, improving transla- tion accuracy without the need for supervised fine- tuning. Although ANN-MT has achieved great suc- cess in domain adaptation tasks, its working mech- anism is still open problem that has not been thoroughly investigated.
Nearest Neighbor Machine Translation (ANN- MT) has achieved great success in domain adaptation tasks by integrating pre-trained Neu- ral Machine Translation (NMT) models with domain-specific token-level retrieval. How- ever, the reasons underlying its success have not been thoroughly investigated. In this pa- per, comprehensively analyze &NN-MT through theoretical and empirical studies. Ini- tially, we provide new insights into the working mechanism of ANN-MT as an efficient tech- nique to implicitly execute gradient descent on the output projection layer of NMT, indicating that it is specific case of model fine-tuning. Subsequently, we conduct multi-domain ex- periments and word-level analysis exam- ine the differences in performance between &NN-MT and entire-model fine-tuning. Our findings suggest that: (i) Incorporating İNN- MT with adapters yields comparable transla- tion performance fine-tuning on in-domain test sets, while achieving better performance on out-of-domain test sets; (ii) Fine-tuning signif- icantly outperforms &NN-MT the recall of in in-domain low-frequent frequency words, but this gap could be bridged by optimizing the context rep- resentations with additional adapter layers.'
In this paper, propose novel perspective to understand ANN-MT by describing it as spe- cial case of fine-tuning, specifically a process of meta-optimization on the Output Projection Layer (OPL) of NMT, and establish connections between ANN-MT and model fine-tuning (Section 3). Our novel perspective on ANN-MT posits that (i) the working mechanism of kNN-MT is to implicitly execute gradient descent OPL, producing meta- gradients via forward computation based on k- nearest-neighhors, and (ii) explicit fine-tuning on OPL shares a similar gradient format with the meta- gradients obtained by ANN-MT, according to the derivation of back-propagation. As illustrated in Figure 1, ANN-MT and explicit OPL fine-tuning share dual view of gradient descent-based opti- mization. The key difference between them lies in the method for computing gradients: kNN-MT produces meta-gradients through forward compu- tation and interpolation, while fine-tuning method computes gradients of OPL via back back- propagation. Hence, it is reasonable to understand ANN-MT as an implicit form of model fine-tuning.
1 Introduction
In recent years, Nearest Neighbor Machine Trans- lation (ANN-MT) and its variants (Khandelwal al., 2021; Zheng al., 2021a,b; Jiang et al., 2021; Wang et al., 2022a) have provided a new paradigm and achieved strong performance for fast domain adaptation through retrieval pipelines. Unlike model fine-tuning, which requires addi- tional parameter updates or introduces external adapter layers, &NN-MT combines traditional Neu- ral Machine Translation (NMT) models (Bahdanau
To provide empirical evidence for our under- standing, we carry out experiments based on multi- domain datasets (Section 4.1). Specifically, we compare the model predictions of KNN-MT and explicit OPL fine-tuning on five domain adaptation tasks. As expected, the predictions of kNN-MT is highly similar to that of explicit OPL. fine-tuning. These findings support our understanding that ANN- MT performs implicit OPL. fine-tuning.
Thii work was done when Ruize Gao was interning
Tencent Al Lab.
Corresponding authors.
OOr code open-sourced at https://github.cem com/ RuizGao/knnmt mette///nnnnttttaaa-eeeee met.tnttteettt-iiiieeeeeee

---

cm
X
Interpolation
Meta-Gradients
p(ym |e, &cm)
9m
A AWAND
Encoder
РNMT
PINN
ANN-MT
AWrr
Computation
Decoder Layer
Wo
Dual View
XN
Wo
Query
Decoder
(hh) [(C)) VTT)
Back-Propagation
Fine-tuning OPL
Datastore
A Nearest Neighbors
Figure 1: ANN-MT implicitly executes gradient descent on the Output Projection Layer (OPL) of NMT and produces meta-gradients via forward computation based on k-nearest-neighhan The meta-optimization process of kNN-MT shares a dual view with explicit OPL fine-tune ane-tuning that updates the parameters of OPL with back proppprppeette gradients.
Next, we conduct comprehensive multi-domain experiments and word-level analysis to examine the differences in translation performance between kNN-MT and other popular fine-tuning methods, such entire-model fine-tuning and adapter-based fine-tuning (Section 4.2 and 4.3). Our empirical results suggest that: (i) Introducing &NN-MT on top of adapter-bast based fine-tumi tinettuning obtains compara- ble translation performance to entire-model fine- tuning on in-domain test sets, while achieving bet- ter performance on out-of-domain test sets. (ii) The entire-model fine-tumi fine-tuning significantly outper- forms &NN-MT in terms of the recall of in-domain fow-freqyency words, but this difference can be mitigated by optimizing the context representations with lightweight adapter layers.
2.2 Nearest Neighbor Machine Translation
Khandelwal et al. (2021) propose KNN-MT that enhances pre-trained NMT models the general domain by incorporating translation memory re- triever. It enables the models to leverage external in-domain knowledge and improve the quality in-domain translations. This approach is generally formulated in two processes; datastore construction and inference with &NN retrieval.
The datastore is a translation memory that con- verts bilingual sentence pairs into set of key-value pairs. For given target domain bilingual corpus {(x, y)}, the context representation fx(x, y<m) generated by the pre-trained NMT model at each timestep m is used as the key, and the m-th tar- get token ym is treated as the corresponding value, resulting in a key-value pair. The entire corpus contributes to the datastore D, which is comprised of all key-value pairs:
2 Background
2.1 Neural Machine Translation
D U ((o(xxy<<)) Hm). Vym y.. (2) (x,))
NMT employs an encoder-decoder model with neu- ral networks that are parameterized by fe to estab- lish the mapping between the source sentence x and its corresponding target sentence y. For the decoding stage, at time step m, NMT utilizes the context representation h Rd which is generated from the source sentence and the current target context <m. to predict the next-token probability:
During inference, the model utilizes the current context representation h fø(x, 8cm) at the m-th decoding step to produce a probability distribution over restricted vocabulary obtained through a nearest-neighbor approach:
PUN Fcm) x
h = fe(x, & vem), (yM x, & &mm) (altmax(Woh)))
d(K.m. h)
Σ
(1)
1 yy==Vo exp(
T
(3)
where Wo RD xd. represents the parameter matrix of OPL in the NMT model and | is the vocabulary size.
where T denotes the temperature control the sharpness of the softmax function and M(hh {(KT) V) is the set of k nearest- neighbrss

---

retrieved from D using pre-defined distance func- tion d(...) .). In practice, we can use either the dot- product function or negative l2 distance to imple- ment d(...). Xu et al. (2023) have demonstrated that the performance of these two functions is al- most identical, so we adopt the dot- dot-product func- tion for theoretical analysis in this paper. Finally, kNN-MT interpolates the vanilla NMT prediction PNMT with the &NN prediction PŁNN to obtain the final next-token probability:
current input q as the query. Instead of using the regular voftmax-normalinn which is Attention V, q) = softmax (K q). we investigate the working mechanism of kNN-MT under a relaxed linear attention form, following the approach of Irie et al. (2022).
dot product attention,
3 KNN-MT Performs Implicit Gradient Descent on Output Projection Layer
In this section, we first demonstrate that probability distribution in kNN-MT, including PaNN and PNMt, is equivalent to Transformer attention. On top of that, we argue that ANN-MT implicitly performs gradient descent on OPL, producing ineaa-gradions via forward computation and interpolation based k-nearest--eighhoon Next, we draw comparisons between ANN-MT and explicit OPL fine-tuning, establishing connections between these two forms.
p(ym X, y 2m) = X PWNN(Um Xx, y <m)
+ (1 X) (Mnr(ym|X, cm),
(4)
where À is a tuned interpolation coefficient. In addition, this prediction way could also be substi- tuted with other kNN variants (Zheng et al., 2021a; Wang et al., 2022a; Dai et al., 2023b) to achieve better model performance or inference speed.
3.1 Output Distributions are Attentions
2.3
Dual Form Between Gradient Descent Based Optimization and Attention
Let h - (x, {mm) be the context representation at each timestep m, and we obtain the nearest neigh-
Irie et al. (2022) present that linear layers optimized by gradient descent have dual form of linear at- tention, which motivates us to view KNN-MT as meta-optimizers. Concretely, linear layer opti- mized via gradient descent can be formulated as:
bors set set (hh) = {(C)) DTTh }} from the datas- tore D. Let Km = [KT K Km Rd and Vm = [Tm VTT V E Di denote matrices representing all key and value vectors N (h). in which we replace the original token value with a one-hot vector for VIr. Then, we reformu- late the computation of PLNN in Equation (3):
F(q) (Wo + AW)q.
(5)
where q E dh the input representation, and Wo, AW E Rd are the initialized parameter matrix and the updated matrix, respectively. In the back propaaaopng algorithm, AW is computed by accumulating n training inputs to this layer Q (qI...., 9n) E pduxx and corresponding (back- propagation) error signals E (e1, ..., ea)
h
[ (m) =Vmsoftmax
Km Vm h),
AAtentionn
(8)
where the dot-product function for the dis- tance metric d(...). According to the above equa- tion, PENN is a key-value memory with attention storing all nearest neighbors from the datastore.
Rddan xn obtained by gradient descent:
AW Σ
R q - EQ
(6)
For the computation of PNMT, we introduce an identity matrix 4|>i and convert it into attention format:
The dual form of a linear layer trained by gradi- ent descent is a key-value memory with attention storing the entire training experience:
PyMT(Um) ycm) roftmax(Woh)
[Iyssfftma(Wooh)
F(q) (Wo + AW)q
= Attention(Wo. 4pr) h),
Woq+ + EQJS q
(9)
where Wo. = Emb, Emby, Embpy] C <dx1 the matrix that represents key vectors for each token in vocabulary. Similarly, PNMT a key-value memory with attention storing all repre- sentations of the entire vocabulary.
Woq + LinearAttn E.
where LinearAttn V, q) denotes the linear at- tention operation, and we regard the training inputs Q as keys, the error signals E as values, and the

---

3.2 ANN-MT as Meta-Optimization
In summary, introduce new perspective explain &NN-MT as a process of meta-oppimizzttinn on the output projection layer of NMT, in which kNN- MT produces meta- gradients via the computa- tion of PŁNN РNMT based on k-nearest-neighhors (hh) = { Dr and implicitly applies gradients to the original output projection layer.
For the ease of qualitative analysis, we follow Irie et al. (2022) to understand the working mechanism of KNN-MT under relaxed lincar attention form, i.e., remove the softmax operation in the compu- tation of PaNN and PNMT. resulting in the following rewritten expressions for PLNN and PNMT:
3.3
Comparing kNN-MT with Fine-tuning
(UMr(Um/Xe X. & <m) 2 (h) =LinearAttn(Wo. / h) = Woh, (NN x, X, {m)) ~ FFNN(h)
As the Equation (12) indicates that the nearest- neighbors set (h) {(C) D } serves the training inputs the output projection layer in the dual form of ANN-MT, we proceed compare the meta-optimization of ANN-MT with explicit OPL fine-tuning, This explicit OPL fine-tuning ap- proach maximizes the bogllikel of the nearest- neighbors set:
(10)
Kmm Vm,, h)
VnnKKhh
LinearAtttn(
T
T
Then the next-token prediction probability of kNN- MT is the weighted sum of two attentions:
p(uml x, {mm) = A PENN + (1 X) PNMT = PNMT + X- (PWNN - PNNTT
((Wo) log PNMT (V (7(/TT a
|Wol
j=1
R (h) + X (FRNN(h) - -FNMT(h) (h)).
(11)
yin log(sofmmax(ook7))
2 |Woll
Combing Equation (7), (10) and (11), we derive the dual form between gradient descent-based opti- mization and 6NN-MT:
(13)
where a is the objective and we optimize the parameter matrix of OPL using KGr and Vj"" as input and label, respec- tively. By applying the back proppappaanttonn algorithm, we obtain the updated matrix Á WFF as follows:
of 12-regularization
(h) =FNMT(h) + A. (F) - FNMr(h))
Vm
-Woh +A- A T (Vmn, h T Woh) -Woh+ +
T -Woh)
AWIT WW aWo
-Woh+ A
(LinearAltn(Xnn Em, h)
k (VT softmax a- Wo
T
2 aWo h)
=Woh + T AWRNNh
Σ (VT -P K, a- Wo
1=1
-(Wo + T
AWANN h = F (h),
(Vm - P...) KT Wo.
(12)
(14)
where AWANN ET - ?(ll)
where Pjn softmax(Woky") is the prediction probability of NMT for the context representa- tion KTT P... [PTT, P ..., P!. D rep- resents all prediction probabilities for the entire nearcst- neighbous set, and the complete deriva- tion process is presented in Appendix A.1. In the case of standard gradient descent, the new parame- ter matrix of OPL, i.e., W is computed as:
2 aWo repre- sents the total gradient including a linear layer (dual form) and /2-regularization objective, Kmm stands for nearest-neighbors training inputs to the output projection layer in NMT, and Em - Vm is the corresponding error signals obtained by gradient descent. As shown in the above equations, the in- troduced probability difference, i.e., PŁNN PNMT. is equivalent to parameter updates AW ANN that af- fect Wo. We can also regard A, = K, as some meta meta--gadiennts which are leveraged to compute the updated parameter matrix A AWNNN
W6 = Wo + n AWWF
= Wo + Fn (Vmm -P a Wo)
(15)

---

Methods
Training Data
Error Signals
Gradients
Optimizer
ANN-MT
(Kms Vm) Vm
(Vi T Wo)
Computation & Interpolation
OPI.-TT
(Km- Vm)
Vm
Pm
((vm Pm) Km Wo)
SGD
Table 1:
The similarities and differences between ANN-MT and explicit OPL fine-tuning, where error signals and gradients are provided in Equation (12) and (14).
where 1) is the learning rate. Similar to Equa- tion (12), Km denotes training inputs and Em
as the basic model for ANN-MT and OPL-FT. We employ both inner-product (IP) and negative /2- distance (1.2) as distance metrics, which the data- store size and parameer settings for ANN- MT are included in Appendix A.3 and we main- tain consistency with previous work (Zheng el al.. 2021 for most details. As for OPL-FT, the pa- rameter of OPL is trained with the same k-nearest- neighbors retrieved by kNN-MT via either IP L2 at each timestep. We perform grid search and use the perplexity (PPL) on the validation set to determine the optimal learning rate and hyper- parameter for SGD optimization. More details are presented in Appendix A.3. As kNN-MT and OPL- FT only involve the optimization of OPL, we adopt reacher-forcing decoding strategy and evaluate the similarity between them by measuring the mean and variance of the difference between their model predictions on the golden label. Specifically, for the test set containing n target tokens, the mean M (-) and variance (-) are computed as:
Vm Pm the corresponding error signals via
explicit OPL fine-tuning.
Table displays similarities and differences be- tween ANN-MT and explicit OPL fine-tum tne-tuning, both of which aim to maximize the log-likelihood of
nearest-neighhor set (Vhh) {OC)) V ) J=== The main distinction lies in the fact that kNN-MT generates mela- geeadggaddeets through forward computa- tion and interpolation, while fine-tuning computes gradients of OPL through back- prok-propagation. More- over, we discover that explicit OPL fine-tuning produces gradient formats that are so similar to meta- geta-ggadseets acquired through kNN-MT. There- fore, it is reasonable to view kNN-MT plicit model fine-tuning process OPI., in which ANN-MT produces distinet parameter matrix W at each decoding time step. As ANN-MT only in- volves the optimization of OPL compared to entire- model fine-tuning, its performance evidently con- strained by the context representations produced by the base NMT model.
im-
x (Pa(3)) (3()))
M(A B)
n i=1
4 Experiments
V(A- B)
1
Σ (PA(3)) pB(n)
this section, we begin by comparing the model predictions of KNN-MT and explicit OPL fine- tuming (OPL-FT) using multi-domain datasets to verify our earlier analysis. Then we carry out com- prehensive
(n - 1)
i=1
M(A B))?
where A, E (NMT, ANN-MT, OPL-FT, FT) and p(v)) denotes the model prediction probability on each golden label y/-
experiments and word- level analysis to gain better understanding of the translation performance differences between ANN- MT and current popular fine-tuning methods.
Results. As shown in Table 2, we find that ANN- MT has a more similar model prediction with OPL- FT (lower mean/variance) compared to the base NMT model or entire model fine-tuning (FT). The experimental results indicate that ANN-MT and OPL-FT are closer than other tuned models. These findings provide empirical evidence supporting our understanding that kNN-MT performs implicit OPL fine-tuning. Additionally, observe that MNN-MT achieves slightly higher mean of model predictions than OPL-FT on average. We suspect that this is because KNN-MT solely utilizes the
4.1 ANN-MT v.s. Explicit OPL Fine-tuning
Setup. We mainly compare &NN-MT and OPL- FT on five domain adaptation datasets, including multi-domain
Englishh datasets in Khan- delwal et al. (2021) (IT, Law, Medical, and Ko- ran), and the IWSLT' 14 German-English transla- tion dataset. The details of multi-domain datasets are listed Appendix A.2. The pre-trained NMT model from the WMT' 19 German- German-English news translation task winner (Ng et al., 2019) is used

---

Law
Medical
Koran
IWSLT
Avg.
ANN-MT- NMT
.073 / 037
.137 .055
.133/ .057
.064 (44
.008 / 014
.083 041
OPI-FT-N NMT
.044 .043
.098 / .055
.100 1.09
.061 /044
.026 0011
.070/ /.42
FT NMT
.20 102
.147 /.077
.152/ .093
.107 /.666
.038 .024
.113/ 1.72
FT KNN-MT
047 /.066
.010/ / 044
.019 046
043 / 041
.034/ /.04
631 .048
FT OPI-FT
056/ .079
.049 .049
.051 /,56
.046 / .048
012 .027
.043 .052
ENN-MT OPL-FT
.110 .024
639 .023
033 .022
.003 / .026
-,018 .011
.013/.021
ENN-MT NMT
.811/ / 037
.135/ .049
.137 / .056
.052 037
.017 /024
.082/ 041
OPL-FT- -MTT
.644 / .043
.098 / .055
.100 .059
.061 / .043
0266/.01
.070/ /.42
FT - NMT
.702 .133
.147 .077
.152 .093
.107 ..666
.038 .024
.113 .072
L2
FT AT--ANN-JE ANN-MT
039 .064
.012 042
.016 044
.055 0.40
011 0.1/.040
027 .046
FT OPL-FF
.056 .079
.049 / .049
.051 /.066
046 .048
.012/ .027
.043 / .052
KNN-MT OPL-FT
.17 / .024
635 .018
037 / .022
-0199/022
-.001 / 022
.014/ /022
Table 2: The mcan/variance (U) of the golden label probability differences between base NMT, entire-model fine-tuning (FT), &NN-MT and explicit OPL fine-tuning (OPL-FT) over each multi-domain test set.
Model
Params
IT
Law
Medical
Koran
IWSLT
Avg.
OOD Avg.
# Speed
NMT
38.35
45.48
40.06
16.26
39.12
35.85
35.36
1.00x
OPL-FT
43.03M
41,26
51.51
47.56
21.27
40.50
40.42
19.62
1.00
&NN-MT
45.60
61.64
53.77
20.66
39.90
44.31
17.79
0.74x
AK-MT
1.2K
47.40
63.32
56.38
20.77
40.04
45.58
31.69
0.72x
Adapter(r = 64)
3.96M
43.55
52.46
48.32
21.62
41.65
41.52
31,74
0.97
Adapter(r 128)
7.90M
44.17
53.98
49.05
21.91
41.54
42.13
31.28
0.97
Adapter(r- = 256)
15.77M
45.27
55.55
51.32
22.38
41.57
43.22
31.06
0.95
FT
269.75M
49.08
63.61
58.43
22.99
41.57
47.06
22.84
1.00x
AK-MTT Adpptr((==256)
15.77M
49.34
64.42
57.27
23.04
41.52
47.12
29.50
0.72x
Table 3: The BLEU (%) and decoding speed of all models multi-domain test sets, including IT, Law, Medical, Koran, and IWSLT. "# Params" refers to the number of fine-tuned parameters. The test sets of the other four domains are integrated out-of-domain (OOD) test sets for each domain and "OOD Avg." represents the average performance of all models OOD test sets. For detailed results on the OOD test sets, please refer to Appendix A.4. "# Speed" indicates the relative inference speed using vanilla NMT as a baseline with batch size of 50k tokens.
label of --iiabost-neigghbors as error signals to up- date the models, without considering the prediction of the NMT model, which may weaken the label signal.
kenized BLEU score (Papineni et al., 2002).
All experiments are conducted based the Fairseq toolkit (Ott al., 2019). For the Adapter, build adapter layers according the approach proposed in Houlsby et al. (2019), with intermedi- ate dimensions selected from (64, 128, 256). For kNNed models, we adopt L2 as the distance metric and the same hyper parameters the pre- vious section. We also explore the performance of combining AK-MT and Adapter (AK-MT, Adapter), which keeps the same hyper-parameters AK-MT. The Adam algorithm (Kingma and Ba, 2015) is used for FT, Adapter and OPL-FT2 with a learning rate of lc-4 and a batch size of 32k tokens. The training process is executed on 4 NVIDIA Tesla
4.2 Translation Performance
Setup. As ANN-MT could be viewed as spe- cial case of model fine-tuning, we further com- pare the translation performance of two &NN-based models, i.e., traditional kNN-MT and adaptive ANN-MT (AK-MT) (Zheng et al., 2021a), with other popular fine-tuning methods, including entire- model fine-tuning (FT) and adapter-based fine- tuning (Adapter). We adopt the previous multi- domain datasets for this experiment but integrate the test sets of the other domains as the out-of- domain (OOD) test set for each domain. The eval- uation metric BacreBLEU, a case-sensitive deto-
PIt is difficult to dynamically update the parameter matrix of OPL. during Ouridiugg beam search, so we directly use all training data to optimize the parameter matrix of OPL. in this experiment.

---

V100 GPUs and the maximum number of training steps set to 100k with validation occurring every 500 steps. During decoding, the beam size set to 4 with a length penalty of 0.6.
[en(w)> where fID(.) and fGD(.) are the word frequencies in domain-specific and general- domain training data, respectively. Then we split the target words into four buckets based on y {0 < 7w)) < 1,1 S 7(w) < 2, S 7(w) < 5, (,)(w) 2 5), with words having a higher do- main frequency ratio y indicating a higher degree of domain-specificity To better illustrate the gap between kNN-based methods and FT, we define incremental word recall AR for ANN-MT, AK-MT and AK-MT as the difference in word recall compared to FT: AR (w) = (ww) (F)(
7(w) fiD(w)
Results. As illustrated in Table 3, we evaluate the translation performance of all models and obtain the following findings:
OPL-FT, which optimizes the parameter matrix of OPL, also brings significant improvements. This proves that only updating the parameter of OPL could achieve relatively high domain adap- tation performance for NMT since it already pro- duces precise context representation due to the large-scale model pre-training.
Figure 2a presents AR values for words in different buckets, indicating that compared to FT, KNN-MT and AK-MT have poor word recalls for words with 7ww) 2 2, particularly when y(w) 2 5. However, AK-MT Adapter achieves comparable per- formance to FT, suggesting that enhancing the con- text representations with adapter layers could han- dle this issue. Moreover, we focus on words with (w) 2 5 and evaluate word recalls in different buckets based on word frequency, dividing words into four buckets based on their in-domain fre- quency ranking: top 1%, top 1-5%, top 5-20%, and top 20-100% As shown in Figure 2b, for in- domain low-frequency words, particularly those ranking behind top 20% &NN-MT and A-MM perform significantly worse than FT in terms of word recall. Similarly, AK-MT Adapter yields com- parable word recall to FT. These results demon- strate that the performance differences between kNN-based models and FT mainly lie in the low re- call of in-domain wwwfrreuency words, which be alleviated by optimizing context representations with additional adapter layers.
Results.
During inference, kNN-MT dynamically select the most appropriate training data for optimiza- tion at each step, resulting in better performance than OPL-FT. However, kNN-MT falls short of FT by 2.75 BLEU score. despite outperforming the Adapter on most domain adaptation tasks. AK-MT achieves better performance than kNN- MT but is still weaker than FT, highlighting the necessity of tuning the context representations generated by the original NMT model.
Combining Adapter and AK-MT achieves com- parable translation quality to FT with better per- formance on OOD test sets (average gain of 6.66 BLEU score). It indicates optimizing the con- text representations with additional adapter lay- ers could further improve kNN-MT.
All in all, as a meta-optimizer on OPL, ANN-MT works quite well on domain adaptation tasks but still requires tuming of the context representations generated by the original model to achieve compa- rable performance to FT.
Nearest Neighbors Analysis. We verify the per- formance of &NN retrieval for the words with (w) 2 5 to better understand the quality of con- text representations. We use the teacher-forcing decoding strategy to calculate the non-reirieval rate of words in each bucket, where word is de- fined non-retrieval if any sub- aub-word of it is not retrieved in the n-iihbostt of AK-MT and AK-MT Adapter- The eenearest-neighhoee of ANN- MT and AK-MT are exactly the same. Figure 3 shows that the non-retrieval rate (Unretrieved%) of AK-MT increases as word frequency decreases, consistent with the results of word recall in Figure
4.3 Word-Level Empirical Analysis
Setup. Apart from the BLEU score, we conduct wevd-level analysis to investigate the translation differences between &NN-MT and FT, and deter- mine the bottleneck of KNN-MT. Specifically, we analyze the translation results of ANN-MT, AK- MT, FT, and AK-MT by calculating the re- call of different target 3ords.3 3 We first use spaces as delimiters to extract target words and define the Nomain--peciffe degree of each word w as
'As shown in Appendix A.6, we calculate the precision. recall, and Fl score (P/R/F1) 1)1 for each word in the translation results and observe that the correlation hetween translation performance and word recall is strongest.
"We manually check the entire dictionary with and find that most words with y 2 2 are real in-domain words.

---

AR 0.02
AR 0.02
0.00
0.00
-0.02
-0.02
0.04
-0.04-
KNNAM
00.05
+ KNN-MT
-0.06
AK-MT
AK-MI
-0.88
AK-MI
AK-MI
A3p
-0.08
-0.10
0-1
1-2
2-5
5- 7
-1%
1-5%
5-20%
20-% Freq
(a)
(h)
Figure 2: Incremental word recall AR of different words on multi-domain test sets. plot the mean AR five datasets with standard deviation in both figures. For the left figure (a). count word recalls in different buckets based on while for the right figure (b), we focus on words with 7(w) 2 5 and calculate word recalls in different buckets based on word frequency.
AK-MI
2020; Xiong et al., 2021), and dialogue genera- tion (Fan et al., 2021; Thulke et al., 2021).
Unretriever%
AK-MT Adaptett=200)
For the NMT system, Khandelwal et al. (2021) propose ANN-MT that utilizes ANN classifier large datastore with traditional NMT mod- cls (Bahdanau et al., 2015; Vaswani et al., 2017; Hassan et al., 2018) to achieve significant improve- ments. Recently, several attempts have been made by most researchers to improve the robustness and scalability. Meng et al. (2022) and Martins et al. (2022a) propose fast versions of kNN-MT. Zheng et al. (2021a) develop adaptive kNN-MT by dynam- ically determining the number of retrieved tokens and interpolation
10-
0-
41%
1-5%
5-20%
20-% Freq
r (Unretrieved%) of the words with ((w) on multi-domain test sets. We plot the mean of five datasets with standard deviation.
Figure 3: The non-retrieval rate
each step, while Martins et al. (2022h) attempt to retrieve chunks of tokens from the datastore instead of a single token. Wang et al. (2022a) adopt a lightweight neural network and the cluster-based pruning method to reduce re- trieval redundancy. Dai et al. (2023b) improve both decoding speed and storage overhead by dynami- cally constructing an extremely small datastore and introducing distance-aware adapter for inference, and further observe the similar behaviours between ANN-based methods and translation memory ap- proaches (Gu et al., 2018; Zhang et al., 2018; Hao et al., 2023).
2b. indicates that the context representations of in-domain www-freuency words are not ef- fectively trained at the pre-training stage, re- sulting in poor word recalls. With adapter- udapter-based fine-tuning, we enhance the context representations for in-domain low- fww-freuynccy words and thus im- prove the word recall of AK-MT. We also intro- duce more metrics evaluate the property of the nearest-neighhors set and please refer more details in Appendix A.6.
5 Related Work
Retrieval-augmented methods have attracted much attention from the community and achieved re- markable performance on various tasks, includ- ing language modeling (Khandelwal et al., 2020; He al., 2021; Nie al., 2022; Xu al., 2023; Wang et al., 2023), machine translation (Khandel-
Despite the great success of the kNN-MT family, the working mechanism of these methods remains an open question. Zhu et al. (2023) analyze the relationship between the datastore and NMT model to better understand the behaviour of kNN-MT. To the best of our knowledge, are the first pro- vide a meta-optimizzation perspective for I-N-MT,, i..,, kNN-MT performs implicit gradient descent on the output projection layer.
wal et al., 2021; Zheng al., 2021a.b; Jiang et al., 2021; Wang et al., 2022b; Du et al., 2022, 2023), question answering (Guu et al., 2020; Lewis et al.,

---

6 Conclusion
these methods actually share similar composi- tion function optimize the context representa- tions (He et al., 2022). We leave this exploration as the future work.
this paper, we present a new meta-optimization perspective to understand kNN-MT and establish connections between kNN-MT and model fine- tuming. Our results on multi- dalli-domain datasets pro- vide strong evidence for the reasonability of this perspective. Additional experiments indicate that (i) incorporating kNN-MT with adapter-based fine- tuning achieves comparable translation quality to entire-model fine-tuning, with better performance on out-of-domain test sets; (ii) kNN-based mod- els suffer from the low recall of in-domain low- frequency words, which could be mitigated by opti- mizing the representation vectors with lightweight adapter layers. We hope our understanding would have more potential to enlighten kNN-based appli- cations and model design in the future.
The word-level empirical analysis indicates that kNN-based models suffer from the low recall of in-domain wowffrequnncy words. Apart from adapter-based fine-tuning, this issue may be mitigated by enhancing the context representa- tions of low-frequency words via more efficient approaches, e.g., introducing frequency-awan token-level contrastive learning method (Zhang et al., 2022) at the pre-training stage and leverag- ing large-scale pre-trained models (Devlin et al., 2019 Brown et al., 2020; Guo et al., 2020: Li al., 2022).
Theoretical and empirical analysis on MNN-MT actually could be directly applied to nearest neighbor language models (kNN-LM) (Khandel- wal et al., 2020). In the future, we would like to follow this research line and do more in-depth explorations KNN-LM. Moreover, the theoret- ical analysis in this paper is limited to the last hidden states of NMT and we are also interested investigating the effectiveness of our analysis on other hidden states of NMT, such as the out- put of the last attention layer in the decoder (Xu et al., 2023).
A Acknowldgements
We would like to thank the anonymous reviewers for their insightful comments. Ruize and Rui are with the MT-Lab, Department of Computer Sci- ence and Engineering, School of Electronic Infor- mation and Electrical Engineering, and also with the MoE Key Lab of Artificial Intelligence, Al In- stitute, Shanghai Jiao Tong University, Shanghai 200204, China. Rui is supported by the General Program of National Natural Science Foundation of China (62176153), Shanghai Pujiang Program (21PJ1406800), Shanghai Municipal Science and Technology Major Project (2021 1001SHZZZX0100 the Alibaba-AIR Program (22088682), and the Ten- cent Lab Fund RBFR2023012.
References
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representatinss, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Limitations
this section, we discuss the limitations and future research directions of our work:
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voor Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
In the theoretical interpretation of ANN-MT, we adopt a relaxed form of attention in the compu- tation of РKNN and PNMT for qualitative analysis, following the approach of preview work (Irie et al., 2022; Garg et al., 2022: Dai et al., 2023a). Whether this conclusion is suitable for normal attention is not rigorously proven, but empirical results provide strong evidence of the plausibility of this perspective.
This paper does not include the results of com- bining other parameter-efficient fine-tuning meth-
ods, such as Prefix-tuning (Li and Liang, 2021) and LoRA (Hu et al., 2022), with &N. But
Damai Dai, Yutao Sun, Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. 2023a. Why can

---

GPT learn in-context? language models secretly per- form gradient descent as meta- meta-optimizers. In Find- ings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages Association for Computational Linguis-
Kelvin Guu, Kenton Lee. Zora Tung, Panupong Pasupat. and Ming- Wei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMI.R
tics.
Yuhan Dai, Zhirui Zhang, Qiuzhi Liu, Qu Cui, Weihua Li, Yichao Du and Tong Xu. 2023b. Simple and scalable nearest neighbor machine translation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
Hongkun Hao, Guoping Huang, Lemao Liu, Zhirui Zhang, Shuming Shi, and Rui Wang. 2023. Rethink- ing translation memory augmented neural machine translation. In Findings of the Association for Com- putational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 2589-2605. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary. Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin uunzys--owmmunt, William Lewis, Mu Li, Shujie Liu, Tie- Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingee Xia. Dong- dong Zhang. Zhirui Zhang, and Ming Zhou. 2018. Achieving human parity on automatic chinese to en- glish news translation. CoRR, abs/1803.05567
Yichao Du, Weizhi Wang, Zhirui Zhang, Boxing Chen, Tong Xu, Jun Xie, and Enhong Chen. 2022. Non- parametric domain adaptation for end-to-end speech translation. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7, 2022, pages 306-320, Association for Computational Linguistics.
Junxian Hc, Grabam Neubig, and Taylor Berg- Kirkpatrick. 2021. Efficient nearest neighbor lan- guage models. In Conference Empirical Methods in Natural Language Processing.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Towards unified view of parameter-officient transfer learning. In The Tenth International Conference on Learning Representations, ICIR 2022, Virtual Tvent, April 25- 29, 2022. OpenReview.net.
Yichao Du, Zhirui Zhang, Bingzhe Wu, Lemao Liu, Tong Xu, and Enhong Chen. 2023. Federated near- est neighbor machine translation. In The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo. Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICML. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR
Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. 2021. Augmenting Transformers with KNN- Based Composite Memory for Dialog. Transactions of the Association for Computational Linguistics. 9:82-99.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gre- gory Valiant. 2022. What can transformers learn in-context? A case study of simple function classes. In NeurlPS.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li. 2018 Search engine guided neural ma- chine translation. In AAAI Conference on Artificial Intelligence.
Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. 2022. The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In International Confer- ence on Machine Learning, ICMI 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 9639-9659. PMLR.
Junliang Guo, Zhirui Zhang, Linli Xu, Hao-Ran Wei, Boxing Chen, and Enhong Chen. 2020. Incorpo- rating BERT into parallel sequence decoding with adapters. In Advances in Neural Information Pro- cessing Systems 33: Annual Conference Neural Information Processing Systems 2020, NewrIPS 2020. December 6- 12, 2020, viriual.
Qingnan Jiang, Mingxuan Wang. Jun Cao, Shanbo Cheng, Shujian Huang, and Lei Li. 2021. Learning

---

kemee--mmoothed machine translation with retrieved examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7280-7290, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
nearest neighbor machine translation. In Findings of the Association for Computational Linguisties: ACL 2022, pages 555-565, Dublin, Ireland, Association for Computational Linguistics.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ou,, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's s WMT19 news translation task submission. In Proceedings of the Fourth Conference Machine Translation (Volume 2: Shared Task Pupers, Day 1), pages 314-319, Florence, Italy, Association for Computational Linguistics.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest neigh- bor machine translation. In 9th International Confer- ence on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In 8th International Conference on Learning Representations. ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net.
Feng Nie, Meixi Chen, Zhirui Zhang, and Xu Cheng. 2022. Improving few-shot performance of language models via nearest neighhor calibration. CoRR, ahs/2212.02216.
Mylc Ott. Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling, In Proceedings of the 2019 Con- ference of the American Chapter of the Associa- tion for Computational Linguistics (Demonstrations)) pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik- tus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe liriivval--ummented generation for knowledge-intensive NLP tasks. In Advances in Neu- ral Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguisties.
Kicla. 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Jiahuan Li, Shanbo Cheng. Zewei Sun, Mingxuan Wang, and Shujian Huang. 2022. Better datastore, better translation: Generating datastores from pre-trained models for nearest neural machine translation. CoRR, abs/2212.0882
Xiang Lisa Li and Percy Liang. 2021. -tenin-tuning: Optimizing continuous prompts for generation. Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 1 1sh Inter- national Joint Conference Natural Language Pro- cessing (Volume 1: Long Papers), abs/2101. 1000900.
David Thulke, Nico Daheim, Christian Dugast, and Hermann Ney. 2021. Efficient retrieval augmented generation from unstructured knowledge for task- oriented dialog. CoRR, abs/2102.00443..
Pedro Martins, Zita Marinho, and André FT Martins. 2022a. Efficientt machine translation domain adap- tation. In Proceedings of the 1st Workshop on Semi- parametric Methods in NLP: Decoupling Logic from Knowledge, pages 23-29.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, NeurIPS 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998 6008.
Pedro Henrique Martins, Zita Marinho, and André F. T. Martins. 2022b. Chunk- based nearest neighbor ma- chine translation. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, pages 4228-4245, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Dexin Wang. Kai Fan, Boxing Chen, and Deyi Xiong. 2022a. Efficient cluster-based k- d-nearest-neighe machine translation. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2175- 2187, Dublin, Ireland. Association for Computational Linguistics.
Yuxian Meng, Xiaoya Li, Xiayu Zheng, Fei Wu, Xi- aofei Sun, Tianwei Zhang, and Jiwei Li, 2022. Fast

---

Dongqi Wang, Haoran Wei, Zhirui Zhang, Shujian Huang, Jun Xie, and Jiajun Chen. 2022b. Non- parametric online learning from human feedback for neural machine translation. In Thitty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022. Thirty-Fouthh Conference on Innovative Applications of Artificial Intelligence, TAAI 2022, The Twelveth Symposium on Educational Advances in Artificial In- lelligence, EAAI 2022 Virtual Event, February 22 March I, 2022, pages 1431-11449, AAAI Press.
parametric unsupervised domain adaptation for neu- ral machine translation. In Findings of the Associa- fion for Computational Linguistics: EMNLP 2021, pages 4234-4241, Punta Cana, Dominican Republic. Association for Computational Linguisties.
Wenhao Zhu, Shujian Huang, Yunzhe Lv, Xin Zheng, and Jiajun Chen. 2023. What knowledge is needed? towards explainable memory for knn-mt domain adaptation. In Findings of the Association for Com- putational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 2824-2836. Association for Computational Linguisties.
Weizhi Wang, Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. 2023. Aug- menting language models with long-tent term memory. ArXiv, abs/2306.07174.
Lee Xiong, Chenyan Xiong, Li, Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. In 9th International Conference on Learning Representations, ICLR 2021. Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Frank F. Xu, Uri Alon, and Graham Neubig. 2023. Why do nearest neighbor language models work? In In- ternational Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, vol- ume 202 of Proceedings of Machine Learning Re- search, pages 38325-38341 PMLR.
Jingyi Zhang. Masao Utiyama, Eiichiro Sumita, Gra- ham Neubig, and Satoshi Nakamura. 2018. Guiding neural machine translation with retrieved translation pieces. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Pupers), pages 1325 1335. Association for Computational Linguis- tics.
Tong Zhang, Wei Ye, Baosong Yang, Long Zhang, Xingzhang Ren, Dayiheng Liu, Jinan Sun, Shikun Zhang, Haibo Zhang. and Wen Zhao. 2022. Frequency- aware contrastive learning for neural ma- chine translation. In Thirty-Sixthl AAAI Conference on Artificial Intelligence, AAAI 2022, Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances Artificial Intelligence, FAAI 2022 Virtual Event, February 22 March 1, 2022, pages 1711-11720 AAAI Press
Xin Zheng. Zhirui Zhang, Junliang Guo, Shujian Huang. Boxing Chen, Weihua Luo, and Jiajun Chen. 2021 Adaptive nearest neighbor machine translation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 368 -34, Online. Association for Computational Linguistics.
Xin Zheng, Zhirui Zhang, Shujian Huang, Boxing Chen, Jun Xie, Weihua Luo, and Jiajun Chen, 202 Non-

---

A Appendix
Law
Medical
Koran
IWSLT
Train
222,927
467,309
248,009
17,982
160,239
A.1 Derivation Process of AWFT
Dev
2,000
2,000
2,000
2,000
7,283
Test
2,000
2,000
2,000
2,000
6,750
According to the chain rule, the updated matrix
AWrT is calculated as follows:
Table 4: Sentence statisties of multi-domain datasets.
AWIT = aC(Wo)
IT
Law
Medical
Koran
IWSLI
aWo
Datastore Size
3.84M
19.5M
7.15M
542K
3.96M
b a(V (oggssotmaxWooA
k
8
4
4
16
32
A
0.6
0.8
0.7
aWo
T
20
10
20
30
20
k
8
4
4
16
32
o(1l)
L2
X
0.7
0.8
0.8
0.8
0.5
10
aWo
log log(softmax (2]"))
az,n
(16)
Table 5: The datastore size (number of tokens) and hyper-parameter choices (i.e., k, and 7') of ANN-MT (IP) and ANN-MT (L2) in each domain.
dz
aWo
a. Wo
b 107T
log( (asSstamaxx2))) (2)
where D is the one-hot vector whose the 1-th value is 1, and PrT softmax softmax(WooK(1) is the whole vector of prediction probability. Finally, the Equa- tion 16 is re-written
KmT
j=1
a Wo:
where 2mm Wo and aw Km Then we provide the derivation process for the rest part. Assume that denotes the vocabulary index of Vmm. pt the i-th probability computed by softmax m and 2; stand for the i-th value of the vector T.rh The calculation of
02
oy log(softmax (2.")) K.
AW
02n
j=1
a Wo
Σ (V - PT KTT a Wo
1 log(softmax (s0ftmax(2111))
j=1
can be re-written as F = log(pr). When the partial derivative of F to 2; is calculated as:
-V - PP aa Wo.
(20)
0
OF 1 Öpt
Σ
e*k
A.2 Dataset Statisties
0z,
Di
02:
P1
dz,
We adopt multi-domain dataset and consider do- mains including IT, Medical, Koran and Law, to- gether with IWSLT" 14 German-English (DE-EN) dataset in all our experiments. The sentence statis- tics of datasets are illustrated in Table 4. For the data preprocessing, we use the Moses toolkit to tokenize the sentences and split the words into sub- word units (Sennrich al., 2016) using the bpe- codes provided by Ng et al. (2019).
1
eth
k= c²x)
V
(e
(17)
PL
(E e²k)²
1
(pi - př = Pi-
Pl
i f we have:
0
0F
1
Opi
Σ M
e
0z4
PI
02:
Pl
Oz,
A.3 Datastore Size and Hyper-parameters
1
e 6
(18)
The datastore size of each domain and the choices of hyper-parameterd in &NN-MT are shown in Table 5, in which
PI
(E )2
1
consider grid search on E {2, 1,8, 16, 32). À E {0.1, 0.2, 0.8, 0.9) and T E (5, 10, 20, 50, 100, 150, 200). We main- tain the same hyper-parameter for AK-MT but set kmax = 16. For OPL-FT, we perform a grid search to find the best learning rate Ir. The search range
Pt Pj-
Pi-
pt
Combining the above equations, we have:
OF
1Mm pmm
(19)
mzm

---

Dataset
Law
Medical
Koran
IWSLT
Ir
4c-3
6c-3
6o-3
2e-3
1o-3
Table 6: The optimal learning rates for explicit OPL fine-tuning based on the perplexity of the validation set.
for all datasets is the same. The search base values are (1, 4, 5, 6, 8, 9} and we scale them to le- 1, le-2. le-3 and le-4 times, i.e., we have X 4 36 values to search. Table 6, we present the de- tails of the selected learning rates on five datasets. Once we obtain the optimal learning rate, the hyper- parameter C {0, 0.01, 0.05, 0.1, 0.5, 1,5, 10) is further selected by the perplexity on the validation set of each domain.
A.4 Translation Performance on Out-of-Domain Test Sets
As shown Table we report the whole out-of- domain results for the experiment in Section 4.2.
A.5 Translation Performance of Recent Advancements in kNN-MT
We provide a comprehensive comparison of trans- lation performance between recent advancements in ANN-MT and the methods mentioned in sec- tion 4.2. The results are shown in Table 8. The results of FK-MT, EK-MT, CK-MT and SK-MT are excerpted from Dai al. (2023b).
A.6 More Details of Word-Level Analysis
We report the overall P/R/F1 results multi- domain test sets Table 9. Compared with preci- sion and Fl score, the defect of KNN-MT more obvious on word recall. In addition, as shown in Table 10, we focus on words with (w) 2 5 and calculate word recalls in different buckets based on word frequency. For the nearest-neighbors analy- sis, in addition to the non-retrieval rate mentioned in section 4.3, we evaluate the following metrics: ① Gold Rank/Gold Dist: the average gold label rank/distance in the top-k list, while taking the rank and distance of the last word in the top-k list (i.e., the farthest neighbor) unretrieved; @ #Gold Labels: the average number of gold labels in the top-k list; 3 #Labels: the average distinct labels in the top-k list, indicating the diversity. For in- domain words (7ww) 5), the detailed results of n-nighosst analysis in above metrics shown in Table 11. We observe that after adapter- based fine-tuning, the non-retrieval rate is reduced as the average distance of the gold label increases.

---

Model
# Params
IT
Law
Medical
Koran
IWSLT
Avg.
NMT
35.10
32.54
34.74
38.55
35.87
35.36
OPL-FT
43.03M
23.56
21.01
22.81
6.97
23.73
19.62
ANN-MT
22.55
13.51
12.69
8.49
31.70
17.79
AK-MT
32.87
27.79
29.36
33,80
34.62
31.69
Adapter(r = 64)
3.96M
33.08
30.06
30.63
32.33
32.60
31.74
Adapter(r = 128)
7.90M
32.87
29.51
30.23
31.58
32.20
31.28
Adapter(r = 256)
15.77M
32.67
28.45
30.09
31.95
32.14
31.06
FT
269.75M
759
14.92
19.06
21.87
27.60
30.75
22.84
AK-MT Adapter(r=256)
15.77M
31.70
26.89
27.51
30.00
31.40
29.50
Table The BLEU score (%) of all models on out-of-domain (OOD) test sets, including IT, Law, Medical, Koran, and IWSLT. "# Params" refers to the number of fine-tuned parameters. The test sets of the other four domains are integrated as OOD test sets each domain and "Avg." represents average performance all models on OOD test sets.
Model
IT
Law
Medical
Koran
Avg.
NMT
38.4
45.5
40.1
16.3
35.0
kNN-MT (Khandelwal al., 2021)
45.6
61.6
53.8
20.7
45.4
FK-MT* (Meng al., 2022)
45.5
56.0
53.6
21.2
44.1
EK-MT* (Martins et al., 2022a)
44.4
57.8
51.9
20.1
43.6
CK-MT (Martins et al., 2022b)
44.2
59.7
53.1
19.3
44.1
SK-MT* (Dai et al., 2023b)
46.2
62.3
57.6
19.5
46.4
AK-MT (Zheng et al., 2021a)
47.4
63.3
56.4
20.8
47.0
AK-MT Adapter(==256)
49.3
64.4
57.3
23.0
48.6
Table 8: The BLEU score (%) of recent advancements &NN-MT and the methods mentioned in section 4.2 on multi-domain sets, including IT, Law, Medical and Koran.

---

7(w)
NMT
ANN-MT
AK-MT
AK-MTA
FT
IT
0~~1
0.597/0.632/0.611
0.672/0.656/0.666
0.709/0.681/0.6%
0.743/0.676/0.700
0.730/0.673/0.700
1~2
0.764/0.746/0.555
08815/0.64/00.888
18815/0.75500 795
0.837/0.778/0.660
0.81880.77330.799
2-5
0.680/0. 6600066006677 670
0.724/0. 701/0.712
1.740/0.710/00770
0.761/0.7260.77 744
0.757/0.726/0.1
Sm
0. (.69//6688/00 180666990000
0.714/0.707/0.77
0.735/0.75000.7 742
0.746/0.75000.744
SUM
0.676/0.668/0.672
0.736/0.702/0.711
0.750/0.724/0.757
0.774/0.738/0.55
0.767/0.735/0.751
Law
0~1
0.724/0.7300.727
0.784/0. 788/0.786
0.830/0.80880 808/0.818
0.850/0.8 8 10/0.829
0.835/0.81
l~2
0.820/0.805/0.812
0.875/0.862/0.866
0.884/0.87200 872/0.875
0.891/0.87550.888
0.885/0.869/0.877
2
0.792/0.756/0.77 7744
0.85 1/511/0.80 80040.80885555
0.869/0.88 2.869/0.8480... 848/0.859
0.868/0.860/0.38
0.867/0.8558
5~
0.787/0.70040.743
0.838/0.88 /1/ 816
0.855200.2008 820/0.8308888 836
0.854/0.83338
844
1.856/0.83500.844
SUM
0.78220.75300.767
0.839/0.828/0.88
0.860/0.8408 0000008400088500 850
0.867/0.846/0.857
0.862/0.84570.68
Medical
0~1
0.644/0.651/0.664
0.695/0.716/0.77
0.770/0.133/0.74
0.797/0.11500.755
0.772/0.725/0.
1~22
0.737/0.729/0.77
7099/00.64/0.7777
0,822/0.7899 789/0,808 805
0.837/0.795/0.66
0.814/0.795/0.. 804
2
0.777/0.73 0.819/0. 113900.771/07999
0.848/0.7
1848//0.99/00.0 794/0.820
0.853/0.796/0.55
0.838/0.801//.8/
5~
0.732/0.6544/0666
0.792/0. 7.500.552
0.817/0. 770/0. 793
0.815500781/1 798
0.809/0.790/0.799
SUM
0.716/0.684/00666
0.77440.72700.75.
0.812/0.763/0.787
0.822/0.769/0.79
0.806/0.776/0.799
Koran
0~1
0.261/0.252/0.256
0.677/0.570/0.619
0.645/0.553/0.99
0,677/0.556/0.61
0.679/0.562/0.115
1~2
0.29920.2.59/0.27
2.6800.598/0...3..
..673/0559/00663
0.699/0.60s. 60566.0500999999
0.693/0.616/0.665
2~5
0.070/0.067/0.000 0688
...62/00558/0...
3366/0.547/0.555
0.596/0.5669 56690..588
0.590/0.577/0.55
0.082/0.078/0. .080
0.554/0. 5.544/0.521/05337
0.548/0.525/0.544
1.582/0.549/0.1
0.575/0.556/0.56
SUM
0.175/0.165/001
0.61220.55770.53
0.604/0.55400.78
0.633/00.6880.6MM
0.63000.577/0.662
IWSLT
0~11
0.687/0.732/0.700
0.722/0.714/0.711
0.710/0.724/0.71
0.746/0.7 1.//0000.6/0..311
(.742/0.718/0.77
1~2
0.789/0.786/0778
0.801/0.792/0.799
7980079.793/0796
0.813/0.797/0.66
0.809/00.99/00.0
2~55
0.724/0.685/00 704
0.728/06688/0.700
0.731/0.690/0.710
0.733/0.700/0.00
0.729/0.70490.716
5~-
0.798/0.591/0 679
0.776/0. 07776/0.64500.1 704
0.788/0.63500 735
0.745/0.70300.777
0.736/0.705/0.72
SUM
0.744/0.7 11////0371/0
0..57/0.722/0.733
0. 0.7560..24/0.739
0.765/0.73600.550
0.760/0.738/0.744
Table 9: Overall P/R/FI of all models multi-domain test sets, in which we count P/R/FI in different buckets
based on the Nomain--peccif degree each word y(w). AK-MTA is the brief description of AK-MT, Adapter(==556)

---

# Words
NMT
ANN-MT
AK-MT
AK-MTA
FT
IT
top
993
0.751
0.767
0.786
0.809
0.802
top 1-5%
1,223
0.707
0.748
0.761
0.764
0.772
top 5-20%
1,492
0.591
0.705
0.718
0.747
0.743
top 20-100%
2,239
0.501
0.638
0.651
0.720
0.722
Law
top
4,896
0.937
0.944
0.948
0.943
top 1-5%
4,013
0.684
0.806
0.816
0.833
0.836
top 5-20%
3,473
0.671
0,789
0.799
0.809
0.806
top 20-100%
2,736
0.492
0.636
0.652
0.674
0.695
Medical
top 1%
2,896
0.765
0.815
0.838
0.844
0.844
top 1-5%
2.676
0.621
0.740
0.755
0.768
0.779
top 5-20%
2,937
0.633
0.748
0.772
0.769
0.774
top 20-100%
4,339
0.618
0.717
0.738
0.761
0.776
Koran
top 1%
5,727
0.183
0.749
0.722
0.726
0.737
top 1-5%
3,502
0.023
0.505
0.486
0.521
0.545
top 5-20%
2.669
0.003
0.430
0.432
0.468
0.468
top 20-100%
2,685
0,234
0.255
0.282
IWSLT
top top 1-5%
9,774
0.669
0.751
0.718
0.790
0.791
5,856
0.519
0.599
0.566
0.634
0.640
top 5-20%
2,243
0.540
0.565
0.563
0.608
0.603
top 20-100%%
1,497
0.522
0.547
0.552
0.623
0.631
Table 10: The word recall of all models on multi-domain test sets, which we focus on words with (w) 2 5 and calculate word recalls in different buckets based word frequency. "# Words" denotes the total number of examples in different buckets. AK-MTA is the brief description of AK-MT, Adapte((==256)
Unretrieved % ()) Gold Rank (1) Gold Dist ())
#Gold Labels
#Labels (L)
AK-MT AK-MTA
AK-MT
AK-MTA
AK-MT AK-MTA
AK-MT AK-MTA
IT
top
8.26%
7.55%
2,89 59.20
2.55/
11.43
12.18
3.27
2.68
top 1-51%
12.35%
11.61%
/6.46
83.58
3.75
3.28
top 5-20%
14.75%
13.74%
3.19/ 69.25
2.98 79.91
3.13
top 20-100%
20.63%
15.36%
2,94 99.12
2,52 100.98
9.55
10.88
3.56
2.79
Law
top
1.90%
1.92%
19.4/29.28
1.42 /3..56
14.19
14.46
1.73
1.59
top 1-5%
5.14%
5.17%
2.16/50.90
2.08 51.74
12.34
2.61
2.36
top 5-20%
6.06%
5.54%
2.12 59.40
2.01 /6144
12.41
2.56
2.30
top 20-100%%
10.72%
8.44%
1,94 / 89.54
1.75 / 90,28
12.05
12.83
2.34
1.98
Medical
top
5.52%
4.97%
2.17 53.94
2.07 62.05
12.15
12.56
2.89
2.55
top 1-55%
9.68%
7.59%
2.46
2.22 72.97
2.72
top 5-20%
9.87%
8.41%
2,24 60.43
2.13 68.29
12.08
12.58
2.62
2.37
top 20-100%
16.80%
13.55%
2,31 / 82.42
2.08 91.77
11.38
12,20
2.64
2.28
Koran
top
7.81%
6.95%
3.00 62.56
2.68 92.52
9.85
10.79
4.07
3.21
top 1-5%
18.10%
15.13%
744.3
16.09
7.39
8.26
5.01
4.21
top 5-20%
20.57%
16.90%
4.31 83.25
124.37
4.86
4.15
top 20-100%
32.85%
27.15%
125.08 4.33
3.87 176.16
5.63
6.50
5.38
4.66
IWSLT
top
8.44%
8.28%
3.11 54.43
2.99 /5.16
10.92
11.27
3.08
2.81
top 1-5%
16.62%
16.26%
4.53
4.45 82.59
4.48
4.10
top 5-20%
17.92%
17.79%
4.74 83.92
4.60 / 88.23
8.01
8.26
4.55
4.20
top 20-100%
25.78%
23.11%
3.29 117.77
3.09/129.62
8.41
8.92
4.00
3.67
Table 11: Detailed results of k-nearest-neighhon analysis of in-domain words (7(w) 5) multi-domain test sets. AK-MI --MT is the brief description of AK-MT AK-MTxapur(==250)) Adhpte((==556).


Reviews:
[{"review_id": "Ww2Z3tc2Ce", "reviewer": "Reviewer_Gf9v", "summary": "", "questions": "1) A comprehensive comparison is requested between the presented method and the most recent advancements in Fast kNN-MT (https://aclanthology.org/2022.spanlp-1.3.pdf), considering both the translation quality and generation speed as critical parameters for evaluation.\n2) An inquiry is raised regarding the effectiveness of the proposed method when applied to kNN-LM (k-Nearest Neighbors Language Models). Should this method prove effective in this context, a thorough discussion on the subject would be deemed valuable and pertinent.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide a comprehensive review of our work. We appreciate your thoughtful feedback and constructive comments. Below, we address the concerns and questions you raised.\n\n> Q1: A comprehensive comparison considering both the translation quality and generation speed as critical parameters for evaluation.\n\nThank you for pointing out the missing reference [https://aclanthology.org/2022.spanlp-1.3.pdf] and we will ensure to include it in our final version for proper citation. \nWhile we follow the same experimental setting as the previous paper, we could directly append more comparisons in our final version, including FK-MT, EK-MT, CK-MT and SK-MT (as named in https://openreview.net/pdf?id=uu1GBD9SlLe). \nIt is important to note that these methods will be implemented solely on the IWSLT domain. Due to the limited time for response, we are unable to provide the results of these methods on the IWSLT domain.\nWe supplement the decoding speed results while batch size is 50k tokens computed based on decoding tokens per second in this Table. We will include it in our final version.\n\n| **Model**                  | **Decoding Speed** |\n|---------------------------------|-------------------------|\n| NMT                             | 1.00x                   |\n| OPL-FT                          | 1.00x                   |\n| $k$NN-MT                        | 0.74x                   |\n| AK-MT                           | 0.72x                   |\n| Adapter($r=64$)                 | 0.97x                   |\n| Adapter($r=128$)                | 0.97x                   |\n| Adapter($r=256$)                | 0.95x                   |\n| FT                              | 1.00x                   |\n| AK-MT$_{\\text{Adapter}(r=256)}$ | 0.72x                   |\n\n> Q2: The effectiveness of the proposed method when applied to $k$NN-LM.\n\nAs mentioned in our paper, theoretical and empirical analysis conducted on $k$NN-MT actually could be directly applied to $k$NN-LM. \nSpecifically, the theoretical reasoning process outlined in Section 3 is fully applicable to $k$NN-LM by replacing $p_{\\text{NMT}}$ with $p_{\\text{LM}}$.\nAlthough we currently lack empirical results for $k$NN-LM, we anticipate that it may provide intriguing insights. \nSince this paper primarily focuses on the $k$NN-MT system, we consider the exploration of $k$NN-LM as a potential avenue for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces a novel meta-optimization perspective for comprehending kNN-MT (k-Nearest Neighbors Machine Translation), while also forging connections between kNN-MT and general model fine-tuning. Through experimentation on multi-domain datasets, substantial evidence is provided to affirm the validity of this viewpoint. Key findings include: (i) the combination of kNN-MT with adapter-based fine-tuning, which yields translation quality on par with entire-model fine-tuning and exhibits superior performance on out-of-domain test sets; (ii) the identification of a shortcoming in kNN-based models pertaining to low recall of in-domain, low-frequency words, a challenge that can be alleviated through the optimization of representation vectors using lightweight adapter layers.", "reasons_to_reject": "The authors do not compare their proposed approach with enough recent methods.", "questions_to_authors": "1) A comprehensive comparison is requested between the presented method and the most recent advancements in Fast kNN-MT (https://aclanthology.org/2022.spanlp-1.3.pdf), considering both the translation quality and generation speed as critical parameters for evaluation.\n2) An inquiry is raised regarding the effectiveness of the proposed method when applied to kNN-LM (k-Nearest Neighbors Language Models). Should this method prove effective in this context, a thorough discussion on the subject would be deemed valuable and pertinent.", "paper_topic_and_main_contributions": "This paper proposes a novel perspective to understand kNN-MT by describing it as a special case of fine-tuning, specifically a process of meta-optimization on the Output Projection Layer (OPL) of NMT, and establish connections between kNN-MT and model fine-tuning. The\nnovel perspective on kNN-MT posits that (i) the working mechanism of kNN-MT is to implicitly execute gradient descent on OPL, producing meta-gradients via forward computation based on k-nearest-neighbors, and (ii) explicit fine-tuning on OPL shares a similar gradient format with the meta-gradients obtained by kNN-MT, according to the derivation of back-propagation.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mrtzf7fUcJ", "reviewer": "Reviewer_4kop", "summary": "", "questions": "Q1\nIn Table 3, AK-MT and AK-MT with adapter models need to train Meta-k network before starting inference. So is not #Params for those must be stated?\n\nQ2\nIn Table 2, it seems a bit obvious that probability distance between kNN-MT and OPL-FT are closer than distance between NMT and each of those models, since they uses in-domain training data to fine-tune (or modify) OPL. In my opinion, if the authors want to claim that kNN-MT and OPL-FT are very similar, the authors should provide probability distance between kNN-MT or OPL-FT and another in-domain tuned model (ex. fully Finetuned one) to show kNN-MT and OPL-FT are closer than other tuned models.\n\nQ3.\nIn Table 2, the authors claim that OPL-FT and kNN-MT similary behave. But in Table 3, the in-domain test scores of those models are quite different (44.31 vs 40.42) Do you have any suggestion for reason to this incongruity?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide a comprehensive review of our work. We appreciate your thoughtful feedback and constructive comments. Below, we address the concerns and questions you raised.\n\n> Q1: \\#Params for AK-MT.\n\nThanks for pointing out our omission. The amount of training parameters of AK-MT is 1216, which is much smaller than that of other methods. We will add \\#Params for AK-MT and AK-MT with adapter models in our final version.\n\n> Q2: Probability distance between $k$NN-MT or OPL-FT and another in-domain tuned model.\n\nThank you for providing these valuable comments.\nWe have conducted a thorough analysis of the probability distances between $k$NN-MT, OPL-FT, and another in-domain tuned model (i.e., fully fine-tuning, FT). \nAs shown in the following two tables, we find that $k$NN-MT and OPL-FT are closer (lower mean/variance) than other tuned models on average.\nThese results serve as empirical evidence that supports our understanding: **$k$NN-MT implicitly performs OPL fine-tuning**. \nWe will include these results in our final version to further strengthen our argument.\n\n|  Inner Product   | IT                           | Law                                       | Medical                                   | Koran                         | IWSLT                                      | AVG                           |\n|----------------------------------------|----------------------------------------|----------------------------------------------------|----------------------------------------------------|-----------------------------------------|-----------------------------------------------------|----------------------------------------|\n|                      $k$NN-MT - NMT    |$ .073 / \\underline{.037}                $|$ .137 / .055                                        $|$ .133 / .057                                        $|$ .064 / \\underline{.041}                 $|$ \\textcolor{blue}{\\textbf{.008}} / \\underline{.014}  $|$ .083 / \\underline{.041}                $|\n|                      OPL-FT - NMT      |$ .064 / .043                            $|$ .098 / .055                                        $|$ .100 / .059                                        $|$ .061 / .044                             $|$ .026 / \\textcolor{blue}{\\textbf{.011}}              $|$ .070 / .042                            $|\n|                       FT - NMT          |$ .120 / .102                            $|$ .147 / .077                                        $|$ .152 / .093                                        $|$ .107 / .066                             $|$ .038 / .024                                         $|$ .113 / .072                            $|\n|                       FT - $k$NN-MT     |$ \\underline{.047} / .066                $|$ \\textcolor{blue}{\\textbf{.010}} / \\underline{.044} $|$ \\textcolor{blue}{\\textbf{.019}} / \\underline{.046} $|$ \\underline{.043} / \\underline{.041}     $|$ .034 / .044                                         $|$ \\underline{.031} / .048                $|\n|                       FT - OPL-FT       |$ .056 / .079                            $|$ .049 / .049                                        $|$ .051 / .056                                        $|$ .046 / .048                             $|$ \\underline{.012} / .027                             $|$ .043 / .052                            $|\n| $k$NN-MT - OPL-FT |$ \\textcolor{blue}{\\textbf{.010/.024}} $|$ \\underline{.039} / \\textcolor{blue}{\\textbf{.023}} $|$ \\underline{.033} / \\textcolor{blue}{\\textbf{.022}} $|$ \\textcolor{blue}{\\textbf{.003/.026}}  $|$ -.018 / \\textcolor{blue}{\\textbf{.011}}             $|$ \\textcolor{blue}{\\textbf{.013/.021}} $|\n\n|  L2 Similarity  | IT                           | Law                                       | Medical                                   | Koran                         | IWSLT                                      | AVG                           |\n|----------------------------------------|----------------------------------------|----------------------------------------------------|----------------------------------------------------|-----------------------------------------|-----------------------------------------------------|----------------------------------------|                          \n|                      $k$NN-MT - NMT    |$ .081 / \\underline{.037}                $|$ .135 / .049                                        $|$ .137 / .056                                        $|$ .052 / \\underline{.037}                 $|$ .017 / .024                                         $|$ .082 / \\underline{.041}                $|\n|                      OPL-FT  - NMT     |$ .064 / .043                            $|$ .098 / .055                                        $|$ .100 / .059                                        $|$ .061 / .043                             $|$ .026/\\textcolor{blue}{\\textbf{ .011}}          $|$ .070 / .042                            $|\n|                       FT - NMT          |$ .702 / .133                            $|$ .147 / .077                                        $|$ .152 / .093                                        $|$ .107 / .066                             $|$ .038 / .024                                         $|$ .113 / .072                            $|\n|                       FT - $k$NN-MT     |$ \\underline{.039} / .064                $|$ \\textcolor{blue}{\\textbf{.012}} / \\underline{.042} $|$ \\textcolor{blue}{\\textbf{.016}} / \\underline{.044} $|$ .055 / .040                             $|$ \\underline{.011} / .040                             $|$ \\underline{.027} / .046                $|\n|                       FT - OPL-FT      |$ .056 / .079                            $|$ .049 / .049                                        $|$ .051 / .056                                        $|$ \\underline{.046} / .048                 $|$ .012 / .027                                         $|$ .043 / .052                            $|\n| $k$NN-MT - OPL-FT |$\\textcolor{blue}{\\textbf{.017/.024}}$|$ \\underline{.035} / \\textcolor{blue}{\\textbf{.018}} $|$ \\underline{.037}/ \\textcolor{blue}{\\textbf{.022}} $|$ \\textcolor{blue}{\\textbf{-.019/.023}} $|$ \\textcolor{blue}{\\textbf{-.001}} / \\underline{.022} $|$ \\textcolor{blue}{\\textbf{.014/.022}} $|\n\n> Q3: Why are the in-domain test scores of those models quite different?\n\nThis issue is discussed in footnote 1 on page 6 of this paper. \nWhile evaluating translation performance in Section 4.2, it is difficult to dynamically update the parameter matrix of OPL during beam search like in Section 4.1, so we directly use all training data to optimize the parameter matrix of OPL in this experiment.\n\nSpecifically, in Section 3, we conclude that $k$NN-MT and OPL-FT exhibit similar behavior. \nIn this context, \"OPL-FT\" refers to fine-tuning a different output projection layer at each decoding step using the key-value pairs of $k$-nearest neighbors as training data. To ensure consistency in the $k$-nearest neighbors searched by $k$NN-MT and OPL-FT, we employ the teacher-forcing decoding method in the experiments conducted in Section 4.1. However, in Section 4.2, OPL-FT is trained with the entire training set, similar to FT. The reason why $k$NN-MT outperforms OPL-FT in this scenario is that the training data of \n$k$NN-MT dynamically changes at each step.\n\n> Q4: The limitation of AK-MT with adapter.\n\nWe will add the limitations of AK-MT with the adapter in Section 7 of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provided an easy-to-understand summary of previous research, theoretical hypotheses, and various experiments to support them. The Appendix contains additional experimental results to help further understand the analysis. This is also the first theoretical paper to understand how NNMT works, which is currently being studied a lot, and it will serve as a basis for further analysis.", "reasons_to_reject": "I think some of the claims in the paper need to be supported by more logical evidence. It would be great if the following questions were answered sufficiently. Also AK-MT with adapter achieves best performance among all models in this paper, but it would be worth to note that AK-MT with adapter model has limitations which it has slow inference speed AND must be fine tune adapter beforehand.", "questions_to_authors": "Q1\nIn Table 3, AK-MT and AK-MT with adapter models need to train Meta-k network before starting inference. So is not #Params for those must be stated?\n\nQ2\nIn Table 2, it seems a bit obvious that probability distance between kNN-MT and OPL-FT are closer than distance between NMT and each of those models, since they uses in-domain training data to fine-tune (or modify) OPL. In my opinion, if the authors want to claim that kNN-MT and OPL-FT are very similar, the authors should provide probability distance between kNN-MT or OPL-FT and another in-domain tuned model (ex. fully Finetuned one) to show kNN-MT and OPL-FT are closer than other tuned models.\n\nQ3.\nIn Table 2, the authors claim that OPL-FT and kNN-MT similary behave. But in Table 3, the in-domain test scores of those models are quite different (44.31 vs 40.42) Do you have any suggestion for reason to this incongruity?", "paper_topic_and_main_contributions": "Nearest Neighbor Machine Translation (NNMT) is one of the new NLP methodologies that has been actively researched in recent years. For sequence generation problems such as language modeling or NMT, each token is usually generated autoregressively during inference. NNMT is a method that creates a datastore with the hidden representation and target token generated with training data, and retrieves the datastore items similar to the current hidden state at each step of inference to supplement the prediction of the NMT model. NNMT has the advantage of improving general translation performance by using a general training datastore, or domain adaptation without fine-tuning by using an in-domain datastore. One of the distinctive variants of NNMT is Adaptive KNN Machine Translation (AK-MT). NNMT is not robust in performance since it is highly dependent on the kNN algorithm hyperparameters. AK-MT shows stable performance by adding a meta-k network to adapt to hyperparameters.\n\nAs shown above, NNMT has been widely studied as a useful way to improve inference performance non-parametrically, but there has never been a theoretical discussion of the success of NNMT. In this paper, the authors follow the work of Irie et al. (2022) to show that NNMT is theoretically similar to output projection layer (OPL) fine-tuning, and provide empirical learning results. The authors also provide directions for further improving the NNMT performance through word-level analysis on NNMT's output sentences. (In this paper, they focus on the domain adaptation effect of NNMT).\n\nFollowing Irie et al. (2022), the authors first characterize that OPL forward propagation (with appropriate relaxation) has a dual form of attention to embedding vectors. The authors also show that the computation of $P_{kNN}$ is equivalent to the computation of the attention to the datastore, so the interpolation of $P_{NMT}$ and $P_{kNN}$ is essentially the computation of the meta-gradient for OPL using the attention to the datastore. To prove this experimentally, they calculate the probability of test set tokens for base NMT, OPL-finetuned model, kNN-MT and show that OPL-FT and kNN-MT produce similar results.\n\nFurthermore, the authors measure the translation performance of kNN-MT and AK-MT with in-domain and out-of-domain test sets. They show that kNN-MT, OPL-FT, and AK-MT obtained better in-domain test BLEUs compared to the adapter models, but their OOD test scores were poor, and their in-domain test BLEUs were also poor compared to FT. These weaknesses are addressed by introducing adapters to AK-MT, which not only improved the in-domain test score but also improved the OOD test BLEU.\n\nAs a further analysis of the above learning results, the authors also conduct a word-level analysis of the test set output sequence of each model. The results show that kNN-MT and AK-MT had poorer in-domain vocabulary recall than FT, especially for words with low in-domain frequency. In contrast, AK-MT with adapters achieve almost the same recall as FT. Furthermore, AK-MT show a worse in kNN retrieval rate for in-domain low-frequency words, and the retrieval rate could be improved by introducing an adapter to AK-MT.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qfAJj0i0bi", "reviewer": "Reviewer_QnkT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide a comprehensive review of our work. We appreciate your thoughtful feedback and constructive comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors provide new insights into the working mechanism of kNN-MT as an efficient technique to implicitly execute gradient descent on the output projection layer of NMT, indicating that it is a specific case of model fine-tuning. They establish connections between kNN-MT and model fine-tuning. I think it is a interesting find.\n2. They give a very detailed mathematical explanation of the methods.\n3. They conduct solid experiments and provide detailed analysis.\n4. They conduct word-level empirical analysis in Section 4.3, which is a good complement to the main experiment.", "reasons_to_reject": "I don't see any reason to reject this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper provides new insights into the working mechanism of kNN-MT as an efficient technique to implicitly execute gradient descent on the output projection layer of NMT, indicating that it is a specific case of model fine-tuning. Then, they conduct multi-domain experiments and word-level analysis to examine the differences in performance between kNN-MT and entire-model fine-tuning.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the Nearest Neighbor Machine Translation (kNN-MT) method, which has shown success in domain adaptation tasks. The authors aim to provide a theoretical understanding of why kNN-MT works and compare it to entire-model fine-tuning through empirical experiments. 

Pros:

* Novel Perspective: The paper introduces a novel perspective by describing kNN-MT as a specific case of model fine-tuning, shedding light on the underlying mechanisms of kNN-MT. This theoretical insight adds depth to the understanding of the method.
		
* Detailed Analysis: Reviewer 1 appreciates the detailed mathematical explanation of the methods, and Reviewer 3 commends the authors for providing new insights, conducting solid experiments, and offering a comprehensive analysis. The paper's in-depth analysis contributes to its strength.
		
* Empirical Results: The paper includes extensive empirical experiments, comparing kNN-MT and entire-model fine-tuning. The results are well-presented and provide valuable insights into the performance of these methods in different domains.
		
* Word-Level Analysis: The paper goes further by conducting word-level analysis, focusing on the recall of in-domain low-frequency words. This adds granularity to the evaluation and highlights the limitations of kNN-MT, which can be addressed with additional adapter layers.


Cons:
		
* Limited Comparison: Reviewer 1 suggests that the authors could have compared their proposed approach with more recent methods, which could provide a more comprehensive understanding of its strengths and weaknesses.
		
* Performance Trade-offs: Reviewer 2 notes that while the proposed approach with adapters improves performance, it comes at the cost of slower inference speed and the need for adapter fine-tuning, which are important practical considerations that should be mentioned.


In summary, the paper offers valuable insights into kNN-MT, providing a novel perspective on its working mechanism and conducting thorough empirical and word-level analysis. The paper's strengths lie in its detailed analysis and experimental results. Addressing the suggestions regarding further comparisons and practical considerations could enhance the paper's completeness.
Decision: Accept-Main