Domain-Aware k-Nearest-Neighbor Knowledge Distillation for Machine Translation (2024)

Abstract: kNN-MT has utilized neighborhood knowledge for auxiliary decoding, significantly improving translation performance. Subsequently, kNNKD transitions the use of neighborhood knowledge from the decoding phase to the training phase, to address the temporal and spatial inefficiencies inherent in kNN-MT. However, kNN-KD transfers all the kNN knowledge arbitrarily, which has the potential to restrict the learning of student models. In this paper, we propose a novel domain-aware kNN-KD method, which filters out domainrelevant neighborhood knowledge for learning in the distillation process. Notably, this entire process exclusively utilizes the neighborhood knowledge of the original model, eliminating the need for establishing any additional datastores. Experiments on four domain translation tasks demonstrate that our method achieves state-of-the-art performance, realizing an average gain of 1.55 COMET and 1.42 BLEU scores, by further enhancing the translation of rare words. Source code can be accessed at https://github.com/wangzx1219/Dk-KD.

heading: 1 Introduction
text: The field of neural machine translation (NMT, Vaswani et al., 2017; Ng et al., 2019) has witnessed significant advancements, resulting in noteworthy improvements in various translation tasks. Among these innovations, the introduction of kNNMT (Khandelwal et al., 2020) stands out as a pioneering approach. This method leverages neighborhood knowledge for assisted decoding, enhancing the translation capabilities of the model. In kNN-MT, the translation process benefits from an expanded contextual understanding, allowing for more accurate and contextually relevant translations. This integration effectively bridges the
* Equal contribution † Corresponding Author
gap between raw translation output and nuanced, context-aware language understanding.
Although the enhancement of model translation capability by kNN-MT is significant, its retrieval cost in the decoding process and consumption of storage space are non-negligible, which limits the application of kNN-MT in practical scenarios. The existing methods try to transfer knowledge from the kNN datastore into new models. kNN-KD (Yang et al., 2022) employs knowledge distillation (KD, Hinton et al., 2015) for training the network from scratch, which, in some cases, leads to suboptimal and unstable outcomes. INK (Zhu et al., 2023) propose to migrate knowledge into the adapter module (Bapna and Firat, 2019b), which is more efficient due to its smaller parameter size and generally yields better results. However, existing methods indiscriminately transfer knowledge from the datastore, a practice we believe may have its limitations.
Building on the pros and cons of the previous methods, this paper introduces domain-aware kNNKD (Dk-KD), aimed at extracting the most valuable knowledge from the kNN datastore within the domain. The approach begins by training a more refined teacher model, specifically optimized for storing kNN knowledge, which facilitates the learning process of the adapter. Subsequently, kNN knowledge is selectively distilled from the teacher model into the adapter, ensuring targeted and efficient knowledge transfer. Specifically, we first construct a datastore using the representations from the original NMT model. Subsequently, we utilize domain-relevant knowledge to aid in training the teacher model. Finally, we leverage both the teacher and domain-relevant knowledge to assist in training the final model. Domain-aware knowledge selection is employed to filter domain-relevant knowledge from the retrieved representations.
Experiments on four domain translation tasks indicate that Dk-KD outperforms other advanced models that utilize kNN knowledge for KD. Our
9458
primary contributions are as follows:
• We introduce Dk-KD, which extracts domainrelevant knowledge from a general-domain NMT model by establishing a kNN datastore.
• We employ a two-step distillation process to extract domain-relevant knowledge from the datastore. Experiments show that domainaware knowledge selection within the datastore is beneficial for model learning.
• Dk-KD focuses on the learning of domain knowledge, improving the translation of domain-specific low-frequency words.
2 Background
2.1 kNN-MT
kNN-MT (Khandelwal et al., 2020) is a method that employs retrieval-augmented techniques in text generation. It operates by retrieving the k closest data points from a vast datastore during the decoding phase, thereby assisting a pre-trained Neural Machine Translation (NMT) model by providing contextually relevant information. The process of kNN-MT can be divided into two main steps:
Building kNN Datastore The datastore is a crucial aspect of the kNN-MT system, where it stores the knowledge of a pre-trained NMT model in the form of key-value pairs. In this context, the key represents the output representation at each time step, and the value corresponds to the target token, which is the accurate translation. Specifically, given a set of training data (X ,Y), we process each sentence pair (x,y) to construct the datastore (K,V). This process can be understood as structuring the knowledge learned by the model in a way that facilitates efficient retrieval and utilization later on. The construction for the datastore is:
(K,V) = ⋃
(x,y)∈(X ,Y) {(f(x,y<i),yi) ∀yi ∈ y} .
(1) The output representation of the NMT model at a specific time step i, denoted as f(x, y<i), serves as the key, while the corresponding reference target token yi is the value. This is because each time step’s output representation corresponds to a target token, and all these representations need to be stored for subsequent retrieval and utilization.
Model Inference During the inference process, at each decoding step i, kNN-MT transforms the current translation context into a representation, f(x, y<i). This representation, f , is then used to query the k nearest neighbors N ik = {(kj , vj) | j ∈ {1, 2, . . . , k}} from the datastore by comparing the l2 distance, effectively identifying the k most similar historical contexts to the current one. The distribution for kNN is: pkNN(yi|x,y<i) ∝
∑
(kj ,vj)∈N ik
1yi=vj exp
( −d(kj , f(x, yi))
τ
) ,
(2) where τ is the temperature, and d(·, ·) is the l2 distance function. The final probability distribution predicting the next token yi is the interpolation of two distributions with a tuning parameter λ:
p(yi|x,y<i) = (1− λ)pMT(yi|x,y<i) + λpkNN(yi|x,y<i).
(3) By incorporating external knowledge, the retrieval distribution has been adjusted to refine the initial NMT distribution, thus better performance.
2.2 kNN-KD kNN-KD was proposed to address the slow decoding speed and large storage space issues of kNNMT. It shifts the phase of utilizing kNN knowledge from decoding to training. Specifically, this is achieved through the KD method, during the model training process, the kNN distribution stored in the datastore acts as the teacher. For all translation contexts (x,y<i) during the training process, kNNKD treats them as queries and performs retrieval in the datastore, subsequently obtaining the retrieval results N . Apply kNN to KD Let pkNN represent the kNN distribution retrieved from the datastore. To train an NMT model from scratch using KD, the loss during distillation is:
LkNN−KD = ∑
yi∈V pkNN(yi|x,y<i)
× log ( pkNN(yi|x,y<i) pMT(yi|x,y<i) ) ,
(4)
where V represents the target language vocabulary. The final loss is obtained by balancing the crossentropy (CE) loss and the distillation loss through the parameter δ:
L = (1− δ)LCE + δLkNN−KD. (5)
3 Domain-Aware kNN-KD

heading: 3.1 Motivation
text: Our objective is to maximize the utilization of the datastore established by existing models and inject domain-relevant knowledge into adapter layers through a dual KD process, a technique not implemented in previous methodologies. Recent studies, such as kNN-KD (Yang et al., 2022) and INK (Zhu et al., 2023), have shifted the kNN retrieval from the datastore from the decoding phase to the training process, utilizing kNN representations to facilitate the model’s learning of knowledge. However, they overlook the need to tailor knowledge learning to specific domains, which may lead to inadequate learning outcomes for the model regarding domain-specific vocabulary. Therefore, we propose Domain-Aware kNN-KD, which focuses on learning domain-relevant knowledge from kNN representations during the training process. The overall training process is depicted in Figure 1.

heading: 3.2 Domain-Aware Datastore Construction
text: As illustrated in the first part of Figure 1, for a specific domain, we employ the original NMT model to perform forced decoding on sentences from the domain’s training set (X ,Y). For each sentence, we obtain multiple context representations, which, along with the corresponding target tokens, are saved into a domain-aware kNN datastore (K,V).

heading: 3.3 Domain-Aware Teacher Finetuning
text: The purpose of this step is to train a teacher model that possesses a profound understanding of specific
domain knowledge, termed a domain-aware model. Beginning with a pre-trained NMT model, which lacks a deep understanding of any specific domain, we tackle this issue by adopting a fine-tuning strategy and incorporating a domain-aware datastore. The key aspect here is to perform domainaware knowledge selection on the kNN representations, enabling us to accurately identify and extract domain-relevant knowledge through this process. This domain-specific knowledge is instrumental in training a teacher model that comprehends the nuances and demands of the specific domain.
During fine-tuning, upon obtaining the kNN distribution, we assess the magnitude of the target token ȳi within this distribution. If the probability pDTkNN(ȳi) is greater than or equal to a threshold α, it is considered domain-relevant knowledge. Conversely, if pDTkNN(ȳi) is less than α, it is deemed domain-irrelevant knowledge. We calculate the Kullback-Leibler (KL) divergence between the domain-relevant kNN distribution and the output distribution of the NMT model:
LKL = ∑
yi∈V pDTkNN(yi|x,y<i)
× log ( pDTkNN(yi|x,y<i) pRTMT(yi|x,y<i) ) ,
(6)
Furthermore, the acronyms DT and RT refer to the Distant Teacher and the Relevant Teacher, corresponding to the original NMT model and the domain-aware teacher model, respectively. The final loss for the entire sentence can be formulated
as:
LTeacher = LCE + I∑
i=1
1(pDTkNN(ȳi)≥α)L i KL, (7)
where I represents the length of the sentence. The indicator function therein indicates the screening process, where only valid kNN knowledge is included in the calculation of the loss.
The parameters of the original NMT model are defined as θN, and the parameters of the teacher model during fine-tuning are defined as θT. The training objective for this step is formulated as: θ̃T = argminLTeacher ( (X ,Y); (K,V); θN ; θT ) )
(8)

heading: 3.4 Domain-Aware Adapter Distillation
text: The aim of this step is to refine the domain-aware teacher model through fine-tuning and to further extract domain-relevant knowledge from the domainaware datastore. This process begins with the adapter layers attached to the original NMT model. These adapter layers are additional layers introduced into the NMT model architecture, including the encoder and decoder, aimed at enhancing the model’s adaptability and understanding of specific domain knowledge without significantly altering the overall structure of the model. During this process, domain-aware knowledge selection is still performed on the kNN representations, allowing for the precise extraction of domain-relevant knowledge from the datastore through this filtering mechanism. This domain-relevant knowledge is then used to optimize the adapter layers.
We utilize the relevant teacher model to retrieve kNN distribution from the datastore. Subsequently, we compare the probability of the golden label in this distribution with α and filter out distributions where the probability is less than α, in order to isolate knowledge with stronger domain-relevant. The distilled distributions are then integrated with the distribution output by the teacher model. The final representation of the teacher probability distribution pT is:
pT = { λpRTMT + (1− λ)pRTkNN, if pRTkNN(ȳi) ≥ α, pRTMT, if p RT kNN(ȳi) < α. (9) The method for calculating the final loss is: LKL = ∑
yi∈V pT(yi|x,y<i) log
( pT(yi|x,y<i)
pSTUMT (yi|x,y<i)
) .
(10)
LStudent = LCE + β I∑
i=1
LiKL. (11)
where STU denotes the final student model and β is a balanced coefficient.
The parameters of the adapter layer attached to the NMT model are defined as θA, hence the training objective for this step can be represented as: θ̂A = argminLStudent ( (X ,Y); (K,V); θ̃T ; θA ) )
(12)

heading: 3.5 Discussion
text: The original NMT model is generally not specialized to a specific domain since it is trained on data from various domains. However, after training on a dataset specific to a domain, we consider the teacher model to be focused on that domain. Similarly, the distribution obtained from the datastore, after filtering, should also be focused on the domain. This can make our final model align with the domain as closely as possible. The final model includes the original model with adapter layers added, while keeping the parameters of the original model’s embedding layer, encoder, and decoder unchanged. Therefore, our model can outperform vanilla kNN-MT systems in terms of inference time and memory costs, achieving a balance between time, memory space, and model effectiveness.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Testbed Model and Dataset We selected the winning model from the WMT’191 German-toEnglish (Ng et al., 2019) news translation task as the basis for constructing our NMT model, utilizing it for both translation and datastore. In this paper, we refer to it as the domain-agnostic model. We conduct experiments on the multi-domain datasets, which include Koran, IT, Medical, and Law. The statistics of these datasets are shown in Table 1. We follow Ng et al. (2019) to tokenize the sentence into subword units.
1https://github.com/facebookresearch/fairseq/ tree/main/examples/wmt19
Baselines For the purpose of comparison, we outline the performance of the original model here and also test two methods of applying neighborhood knowledge during the decoding phase. Moreover, we also compare with the methods distilling neighborhood knowledge into the model.
• Vanilla NMT (Vaswani et al., 2017) introduces the Transformer model, which adopts a selfattention mechanism.
• Vanilla kNN-MT (Khandelwal et al., 2020) utilizes neighborhood knowledge to enhance translation during the decoding phase.
• Robust kNN-MT (Jiang et al., 2022) enhances kNN-MT performance by dynamically adjusting decoding hyperparameters.
• kNN-KD (Yang et al., 2022) aims to train an NMT model from scratch that distills kNN knowledge into it.
• INK (Zhu et al., 2023) smooths the representation space using kNN knowledge from an asynchronously refreshed datastore.
Evaluation During inference, we set beam size as 4 and length penalty as 0.6. We used the following two metrics for MT evaluation:
• COMET (Rei et al., 2020), a machine learningbased evaluation tool specifically designed for assessing the quality of machine translations. It quantifies the accuracy and fluency of translations by comparing the translated text with reference texts translated by humans. We report
the COMET score, calculated using the publicly available wmt20-comet-da2 model.
• BLEU (Papineni et al., 2002), a widely-used metric for automatically evaluating the quality of machine translations. We present our results using case-sensitive, detokenized sacrebleu3.

heading: 4.2 Implementation Details
text: We employed the fairseq4 toolkit for model implementation, leveraging knn-box5 (Zhu et al., 2024) for the construction and kNN retrieval of the datastore. Furthermore, we utilized FAISS6 for efficient search operations. In the first phase, we set the learning rates for the Koran, IT, and Medical domains to 1e-4, respectively. For the Law domain, the learning rate was adjusted to 3e-4.In the second phase, during the training of the adapter layers, we standardized the learning rates across all four domains to 3e-4, respectively, and set β to 2.0.All Dk-KD systems were trained on A100 GPUs. In the two-stage distillation, we set the threshold α to 0.5. Throughout the entire training process, we set the label smoothing parameter to 0.1, the weight decay to 0.0001, used the Adam optimizer with betas of (0.9, 0.98), and adopted an inverse square root learning rate scheduler with a warm-up of 4000 updates. We selected the model with the highest validation BLEU score for testing.
The parameters involved in the distillation process of Dk-KD are outlined in Table 2, including those for kNN retrieval, max-epochs, and batchsize for training. During the fine-tuning phase of the teacher model, due to the small size of the Koran dataset, we appropriately reduced the maxepochs to prevent the model from overfitting to the validation set, which could lead to suboptimal results. With the larger Law dataset, to ensure thorough learning, we adjusted its batch size. Similarly, during the distillation adapter phase, we also reduced the dimension of the adapter layers specifically for the characteristics of the Koran dataset. To ensure the learning effectiveness and training speed of the adapter layers, we standardized the max-tokens to 8192.
2https://github.com/Unbabel/COMET 3https://github.com/mjpost/sacrebleu 4https://github.com/facebookresearch/fairseq 5https://github.com/NJUNLP/knn-box 6https://github.com/facebookresearch/faiss

heading: 4.3 Main Results
text: The comparative results of different systems are presented in Table 3. Given the original NMT model’s lack of domain sensitivity, its performance on tests across the four domains was relatively moderate. While the conventional kNN-MT model demonstrates improved translation quality, its decoding phase incurs significant time and storage costs, which limit its widespread application. The kNN-KD approach requires training from scratch, resulting in unstable outcomes. Meanwhile, the INK training framework demands asynchronous updates to the datastore, resulting in a high demand for disk interaction during training. Dk-KD optimizes the balance between memory usage and method effectiveness, eliminating the need for extra disk operations during training, ensuring a more stable and consistent learning environment, thereby validating the method’s rationality. Through testing on four domain tasks, Dk-KD achieved commendable translation outcomes, demonstrating the effectiveness of Dk-KD.

heading: 5 Analysis

heading: 5.1 Method Justification
text: Effect of Domain-Aware Teacher Finetuning To validate the significance of training the teacher
model, we omitted the distillation process in Step 1. Specifically, we directly extracted the kNN probability distribution from the datastore during the training process, applied the same threshold to select domain-relevant knowledge, and used the filtered results for KD on the adapter layer. Although there was an improvement compared to adapter baseline, the results were still not sufficiently excellent and stable. Table 4 indicates that distilling the adapter layer using a teacher model significantly enhances translation quality. These findings underscore the effectiveness of integrating domain-specific knowledge into a teacher model, highlighting it as an effective strategy for extracting domain-relevant knowledge from the datastore.
Effect of Domain-Aware Knowledge Selection To validate the effectiveness of the domain-aware distillation approach, we conducted four sets of experiments for two-stage KD on the Koran and IT datasets, with results depicted in Figure 2. Here, Dk-KD denotes “Learn ≥ α”, wherein the learning objective targets the distribution of neighborhood knowledge with golden labels greater than or equal to a threshold, representing domain-relevant knowledge; “Learn < α” represents the opposite scenario, targeting distributions where golden labels are below the threshold, representing domainirrelevant knowledge; “Learn all” refers to using
all retrieved neighbor knowledge kNN distributions for distillation without filtering, meaning all retrieved probability distributions are utilized for distillation; “Learn none” indicates that neighbor knowledge kNN distributions are not used for distillation. It can be intuitively observed that learning from domain-relevant knowledge can enhance the performance of the final model compared to learning from the complete set of neighborhood knowledge, while learning from domain-irrelevant knowledge can diminish the performance of the final model. Furthermore, the effectiveness and rationality of selectively filtering knowledge are highlighted by the improved performance of the final model after executing domain-aware knowledge selection, compared to indiscriminate learning.
In addition, the choice of different thresholds is also noteworthy, we conducted experiments on different α while fixing other parameters and the results of the tests are shown in Table 5. As can be seen by their average scores in the four domains, for both assessment metrics, all test results are higher than the current state-of-the-art methods. These
experiments demonstrate that, despite the variability introduced by different threshold settings, the method consistently enhances performance compared to the state-of-the-art (namely, INK), indicating flexibility and robustness in threshold selection.
5.2 How Does Dk-KD Enhance Domain Translation?
Enhanced Domain-Aware Sentence Representation We input the test set sentences into the model, extract the outputs from the model’s encoder, and perform average pooling. Then, by applying t-SNE for dimensionality reduction on the sentence representations, we obtain the clustering results as shown in Figure 3. The Koran dataset exhibits the best clustering effect, which may be attributed to the presence of a larger number of unique terms within the Koran domain. Conversely, in the clustering of the Law, Medical, and IT datasets, there is always a degree of overlap, likely due to the intersecting content across these domains. We also calculated the average intracluster distance for each method, and the results indicate that Dk-KD has the smallest average cluster distance, demonstrating the best clustering effect for distinguishing domain shifts. Given that our sentence representations are derived from the average pooling of individual word representations, the model’s effectiveness in clustering sentence representations is directly related to its performance in representing words. Low-frequency words, due to their sparse occurrence, may have vector representations that are less accurate or robust, leading to suboptimal clustering performance.
Enhanced Rare Word Translation In multidomain datasets, low-frequency words are often terminologies specific to their respective domains. We examined word frequencies of WMT19 GermanEnglish training sets across domains, classifying words as low-frequency (less than 50 occurrences),
Low Mid High 0.0
0.2
0.4
0.6
0.8
Fsc
or e
Koran Adapter INK Dk-KD
Low Mid High
IT
Low Mid High
Medical
Low Mid High
Law
Figure 4: Comparing the translation performance of tokens with varying frequencies across four datasets.
medium-frequency (51-999 occurrences), and highfrequency (over 1000 occurrences). To evaluate our approach, we used a baseline that involves direct training of adapters and employed comparemt (Neubig et al., 2019) to calculate F-scores, as shown in Figure 4. Our research demonstrates that, in comparison to the adapter baseline and INK, Dk-KD exhibits a significant improvement in the inference accuracy of low-frequency words across various domains. This improvement is particularly notable in the medical and law fields, where the enhancement in inference accuracy for low-frequency words exceeds that of medium and high-frequency words. We hypothesize that the improvement of translating low-frequency words is associated with the size of the datastore, suggesting that a larger and more comprehensive datastore facilitates better learning outcomes for low-frequency words. The robust performance of Dk-KD across different domains, especially the translation of low-frequency words, validates the efficacy of our approach.
5.3 Dk-KD with kNN Datastore Retrieval
Following the training process of Dk-KD, we constructed a new datastore using the representations from the final model and applied Robust kNN-MT to it. The resulting COMET and BLEU scores are shown in Table 6. We found that applying R-kNN to the Koran dataset actually deteriorates translation performance, which may be attributed to the smaller size of the Koran dataset leading to a sparser datastore, making it challenging to extract an effective kNN distribution to assist in genera-
tion. The sparse datastore for the Koran dataset indicates a need for further methodological adjustments or enhancements when dealing with limited or highly specialized data collections. Despite the challenges observed with the Koran dataset, the overall results illustrate that, in most scenarios, the combined application of Dk-KD with R-kNN still leads to an improvement in translation quality.

heading: 6 Related Work

heading: 6.1 Domain Adaptation for NMT
text: The prevailing approaches in this area can generally be categorized into model-centric and datacentric methods. The former focuses on carefully designing NMT model architectures to learn translation knowledge of target domain (Wang et al., 2017; Zeng et al., 2018; Bapna and Firat, 2019a; Guo et al., 2021) or improving the training process to better utilize relevant context (Wuebker et al., 2018; Bapna and Firat, 2019b; Lin et al., 2021; Liang et al., 2021). The latter, on the other hand, involves leveraging target domain monolingual corpora (Zhang and Zong, 2016; Zhang et al., 2018b), synthetic corpora (Hoang et al., 2018; Hu et al., 2019; Wei et al., 2020), parallel corpora (Chu et al., 2017), or learn the similar data various domains to intensify the model for low-resource machine translation (Zhan et al., 2021), to enhance machine translation models through model fine-tuning. In this paper, we propose a novel approach by starting with building a datastore of domain knowledge, extracting domain-relevant knowledge from stored representations.
6.2 kNN-MT
Example-based approaches have been introduced into machine translation, demonstrating the utility of leveraging token-level or sentence-level examples to improve translation performance (Zhang et al., 2018a; Gu et al., 2018). kNN-MT significantly improves performance by retrieving examples, serving as a non-parametric retrievalaugmented method that offers an alternative to traditional fine-tuning methods (Khandelwal et al., 2020). Adaptive kNN-MT further enhances performance through the training of a network that dynamically adjusts parameters (Zheng et al., 2021a; Jiang et al., 2022). kNN-MT has shown promising progress in various machine translation tasks, including domain adaptation (Zheng et al., 2021b; Cao et al., 2023), interactive machine translation (Wang et al., 2022b), Transfer Learning (Li et al., 2022; Liu et al., 2023) and speech translation (Du et al., 2022). Other researchers have enhanced kNN-MT retrieval efficiency by pruning the datastore (Wang et al., 2022a), dynamically constructing the datastore (Meng et al., 2022; Wang et al., 2021; Dai et al., 2023), and reducing the number of steps required for retrieval (Martins et al., 2022a,b). To eliminate the need for retrieval during inference, the approach of distilling the knowledge from the datastore into the model parameters has been proposed, presenting a novel alternative to kNN retrieval (Yang et al., 2022; Zhu et al., 2023). We observe that not all the knowledge contained in a domain-specific datastore is beneficial for the learning of student models. In this paper, we present a two-step distillation process to extract domain-relevant knowledge from the neighboring datastore, effectively distilling the knowledge into the adapter.

heading: 7 Conclusion and Future Work
text: In this paper, we introduce a method named domain-aware kNN-KD, which extracts domainrelevant knowledge from a domain-aware datastore constructed by a domain-agnostic model. In
Dk-KD, we design a two-stage knowledge distillation process that initially trains a domain-relevant teacher with the aid of domain-relevant knowledge, and then utilizes the teacher model to distill domain-relevant knowledge into the adapter layers, achieving improvements in translation performance. Notably, Dk-KD significantly enhances the source representation and the translation accuracy of domain-specific low-frequency words.
Future work includes: 1) Attempting to shift the filtering of domain-focused knowledge from the training phase to the datastore construction phase, aiming to reduce the storage footprint of the datastore and alleviate retrieval times; 2) Identifying more effective and efficient methods for filtering domain-relevant knowledge to enhance the efficacy of kNN knowledge distillation.
Limitations
While the effectiveness of our model is promising, employing the entire datastore for knowledge distillation during the training phase results in longer training times. Furthermore, we need to conduct two rounds of training for each domain, thereby increasing the overall training expense. Additionally, we observed that even with the final model, kNN knowledge continues to aid in inference performance. This suggests that our selection of domainrelevant knowledge is not sufficiently precise, or the distillation method may be unsuitable, indicating potential areas for improvement. Due to the large scale of large language models, current methods of kNN knowledge distillation, including Dk-KD, are not readily transferable to machine translation based on large language models, awaiting further development in GPU memory capacity.

heading: Acknowledgment
text: This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 62206076, 62306084, 62261160648), Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024A1515011491),
Shenzhen Science and Technology Program (Grant Nos. ZDSYS20230626091203008, KJZD20231023094700001, RCBS20221008093121053), Shenzhen College Stability Support Plan (Grant Nos. GXWD20220811173340003, GXWD20220817123150002), the Science and Technology Development Fund, Macau SAR (Grant Nos. FDCT/060/2022/AFJ, FDCT/0070/2022/AMJ), the Multi-year Research Grant from the University of Macau (Grant No. MYRG-GRG2023-00006-FST-UMDF), and Tencent AI Lab Rhino-Bird Gift Fund (Grant No. EF2023-00151-FST). Xuebo Liu was sponsored by CCF-Tencent Rhino-Bird Open Research Fund. We would like to thank the anonymous reviewers and meta-reviewer for their insightful suggestions.

heading: Future Work
text: Future work includes: 1) Attempting to shift the filtering of domain-focused knowledge from the training phase to the datastore construction phase, aiming to reduce the storage footprint of the datastore and alleviate retrieval times; 2) Identifying more effective and efficient methods for filtering domain-relevant knowledge to enhance the efficacy of kNN knowledge distillation. Limitations
While the effectiveness of our model is promising, employing the entire datastore for knowledge distillation during the training phase results in longer training times. Furthermore, we need to conduct two rounds of training for each domain, thereby increasing the overall training expense. Additionally, we observed that even with the final model, kNN knowledge continues to aid in inference performance. This suggests that our selection of domainrelevant knowledge is not sufficiently precise, or the distillation method may be unsuitable, indicating potential areas for improvement. Due to the large scale of large language models, current methods of kNN knowledge distillation, including Dk-KD, are not readily transferable to machine translation based on large language models, awaiting further development in GPU memory capacity