kNN-TL: k-Nearest-Neighbor Transfer Learning for Low-Resource Neural Machine Translation
Year: 2023

Abstract:
Transfer learning has been shown to be an effective technique for enhancing the performance of low-resource neural machine translation (NMT). This is typically achieved through either fine-tuning a child model with a pretrained parent model, or by utilizing the output of the parent model during the training of the child model. However, these methods do not make use of the parent knowledge during the child inference, which may limit the translation performance. In this paper, we propose a k-Nearest-Neighbor Transfer Learning (kNN-TL) approach for low-resource NMT, which leverages the parent knowledge throughout the entire developing process of the child model. Our approach includes a parent-child representation alignment method, which ensures consistency in the output representations between the two models, and a child-aware datastore construction method that improves inference efficiency by selectively distilling the parent datastore based on relevance to the child model. Experimental results on four lowresource translation tasks show that kNN-TL outperforms strong baselines. Extensive analyses further demonstrate the effectiveness of our approach. Code and scripts are freely available at https://github.com/NLP2CT/kNN-TL.

heading: 1 Introduction
text: Although deep learning has significantly advanced the field of neural machine translation (NMT, Bahdanau et al., 2015; Vaswani et al., 2017; Liu et al., 2019, 2020), the standard training procedure of NMT is not well-suited for languages with only a small amount of bilingual data, leading to challenges in developing NMT models for low-resource languages (Zhan et al., 2021; Wang et al., 2022d). To overcome this limitation, transfer learning has been proposed as an effective method to enhance low-resource NMT through the parent-child framework. This framework transfers knowledge from a
∗Co-corresponding author
high-resource parent model to a low-resource child model (Zoph et al., 2016).
Previous works in transfer learning, such as Kim et al. (2019a) and Aji et al. (2020), have aimed to address the problem of vocabulary mismatch for more effective knowledge transfer. These works, referred to as Vanilla TL, primarily focus on transferring knowledge during the initialization stage of the child model and do not consider other stages of the development of the child model. Recently, Li et al. (2022) propose a novel transfer learning method, namely ConsistTL, which models consistency between the parent model and the child model to facilitate the continual transfer of knowledge from the parent model during the child training. While ConsistTL considers both the initialization and training stages of the child model, it does not address the inference stage, which may limit the overall transferability of knowledge from the parent model. The effective utilization of parent knowledge during the inference stage is an intuitive strategy to improve the performance of low-resource child models.
This paper presents a novel k-nearest-neighbor transfer learning (kNN-TL) method for lowresource NMT. The proposed method aims to fully utilize the knowledge from the parent model to provide more comprehensive guidance throughout the entire development process of the child model,
1878
as shown in Table 1. To achieve this, kNN-TL aligns the parent and child representations during the child training to ensure the retrieval of relevant and useful knowledge from the parent datastore during the child inference. Additionally, to accelerate inference, kNN-TL selectively distills relevant knowledge from the parent datastore to construct a child-aware datastore. At each step of the model prediction, kNN-TL considers both the probability distributions retrieved from the parent datastore and predicted by the child NMT model. Experimental results on four low-resource translation tasks, guided by two high-resource parent models, confirm the effectiveness and efficiency of the proposed kNN-TL method. Further analysis reveals that kNN-TL can effectively align the representations of the parent and child models, providing a reasonable explanation for the performance improvement. Our main contributions are as follows:
• We propose kNN-TL to transfer knowledge from the parent model throughout the entire developing process of the child model, including the initialization, training, and inference.
• We propose a child-aware datastore construction method by selectively distilling the parent datastore, which improves inference speed while maintaining comparable performance.
• Experimental results demonstrate that kNNTL can achieve non-trivial improvements over strong transfer learning methods on four lowresource translation tasks, as measured by widely-used automatic evaluation metrics.

heading: 2 Background

heading: 2.1 Transfer Learning for NMT
text: The parent-child framework has been widely used in previous studies (Zoph et al., 2016; Kim et al., 2019b; Aji et al., 2020) to conduct transfer learning, which transfers the knowledge of a high-resource NMT model (i.e., parent) to a low-resource NMT model (i.e., child). Generally, the framework involves the following two steps.
Parameter Initialization The first step is to initialize the child model by the parent model:
θc = R(θp), (1)
where θp is the pre-trained parameters of the parent model, θc is the parameters of the child model, and
R denotes the initialization strategy. Part or all of the parent parameters can be used for initialization.
Fine-tuning The second step is to train the child model on the low-resource child data (xc,yc) ∈ (X c,Yc), starting from the pre-initialized parameters. The child model is optimized by minimizing the cross-entropy (CE) loss function:
LCE = − T∑
t=1
log(p(yct |xc,yc<t, θc)), (2)
where T denotes the length of the target sentence.
2.2 kNN-MT
To incorporate the knowledge of the parent model into the inference phase, we borrow ideas from the k-nearest-neighbor machine translation (kNN-MT, Khandelwal et al., 2021) which has been shown to be effective in improving domain-specific translation tasks. kNN-MT is a retrieval-augmented text generation paradigm that assists the pretrained NMT model by retrieving the k nearest neighbors from a large-scale datastore for relevant knowledge in the decoding stage. Formally, kNN-MT mainly includes the following two stages.
Datastore Building The datastore is the core component of kNN-MT that stores the knowledge of a pretrained NMT model explicitly through keyvalue pairs, where the key is the output representation at each time step and the value is the corresponding gold target token. Given the training data (X ,Y), the datastore is constructed over all the sentence pairs (x,y) as follows:
(K,V) = ⋃
(x,y)∈(X ,Y) {(f (x,y<t) , yt) ,∀yt ∈ y} ,
(3) where f (x,y<t) is output representation of the NMT model at t step, and yt is the gold target token. It is worth noting that the size of the datastore is proportional to the number of tokens in the target sentences, which could be very large.
Inference with Retrieval In kNN-MT, the NMT model generates two probability distributions for prediction during inference, namely, the one by the output representation (i.e., pNMT) and the extra one by the retrieved representation from the datastore (i.e., pkNN). Specifically, at each inference step t, the output representation f (x,y<t) is used to query the datastore and obtain the k nearest
neighbors as N kt = {(kj , vj) , j ∈ {1, 2, . . . , k}}. Then, the retrieval distribution can be computed as:
pkNN (yt|x,y<t) ∝ k∑
j=1
1yt=vj exp (−d (kj , f (x,y<t))/τ) , (4)
where τ is the softmax temperature and d(·, ·) is the l2 distance function.
The final probability distribution for predicting the next token yt is the interpolation of the two distributions with a tuned parameter λ:
p (yt|x,y<t) = λpkNN (yt|x,y<t) + (1− λ)pNMT (yt|x,y<t) . (5)
The retrieval distribution refines the original NMT distribution with external knowledge, which improves the prediction accuracy.
3 kNN-TL
This section introduces the kNN-TL method in detail. It begins by clarifying the motivation for the work by comparing kNN-TL to previous methods. The training process of kNN-TL is then presented with a specific focus on the parent-child representation alignment component for subsequent kNN retrieval. After that, the steps for building a childaware datastore to improve inference speed are described. Finally, the method of incorporating knowledge from the parent datastore to guide the child model during inference is presented.

heading: 3.1 Motivation
text: We aim at exploiting the knowledge of the parent model throughout the whole development process of the child model based on the parent-child framework, which has not been accomplished in previous methods. As shown in Table 1, vanilla transfer learning (Kim et al., 2019a; Aji et al., 2020) initializes the child model by the optimized parameters of the parent model, and then continues the training of the child model on the low-resource translation dataset. Recent studies, such as ConsistTL (Li et al., 2022), have found that incorporating knowledge of the high-resource parent models to provide continuous guidance for the child models during training can significantly improve the performance of low-resource translation tasks. However, these studies ignore the high-resource parent models in inference, which does not make full use of the
parent model and potentially limits the translation performance. Therefore, we propose kNN-TL to fully exploit the high-resource parent models at initialization, training and inference process.

heading: 3.2 Parent-Child Representation Alignment
text: Due to the discrepancy in feature representations between the child model and the parent model, building the datastore solely from the parent data may not provide sufficient and relevant knowledge, leading to poor performance of the child model. To address this issue, we propose to align the representations of the child and parent models.
Pseudo Parent Data Construction In order to align the feature representations of the parent and child models, we generate a set of paired samples. We adopt the approach proposed by Li et al. (2022) to generate pseudo parent source sentences for the entire child data. Specifically, for each instance (xc,yc) ∈ (X c,Yc), we use a well-trained reversed parent model to back-translate the target sentence yc to a pseudo parent source sentence x̃p and obtain the pseudo parent data (x̃p,yc) ∈ (X̃ p,Yc).
Representation-based Consistency Learning In ConsistTL and other consistency learning methods (Wang et al., 2022d; Li et al., 2023), the consistency between the parent and child models is encouraged over the probability distributions, but this approach does not impose strong constraints on the feature representations. To address this issue, we propose to utilize the child data and the pseudo parent data to learn consistent output rep-
resentations for the same target sentences. Specifically, for each instance of the pseudo parent data (x̃p,yc) ∈ (X̃ p,Yc), the parent model generates the output representation as fθp (x̃p,yc<t) for every target token yct , while the child model generates the output representation as fθc (xc,yc<t) for the same target token. Then we minimize the squared Euclidean distance of these two output representations with the MSE loss:
LMSE = T∑
t=1
∥fθp (x̃p,yc<t)− fθc (xc,yc<t)∥2 ,
(6) where θp and θc represent the parameters of the parent and child models, respectively. The final loss is a combination of the CE loss and the MSE loss, with a balancing hyper-parameter α:
L = LCE + αLMSE. (7)

heading: 3.3 Child-Aware Datastore Construction
text: The aim of kNN-TL is to improve the performance of the child model by utilizing relevant knowledge from the parent data. However, using a large amount of parent data leads to a large datastore that can slow down the retrieval speed during inference. To address this issue, we propose a method to selectively prune the high-resource parent datastore by pre-retrieving relevant entries using the pseudo parent data. Specifically, we first utilize the welltrained parent model to forward pass the parent data (X p,Yp) and obtain the intermediate representation fθp (x̃p;yc<t) to construct a large parent datastore as Eq.(3). For each instance of the pseudo
parent data (x̃p,yc), we use the parent model to forward pass it and conduct kNN retrieval from the large parent datastore with a large value of k̄. The obtained k̄ nearest neighbors is expressed as:
Nyc = { (kj , vj) , j ∈ {1, 2, . . . , k̄},∀yct ∈ yc } .
(8) As the pseudo parent data is semantically equivalent to the child data, the pre-retrieved subset will include entries that are more relevant to the child data. Besides, our method only needs to retrieve through the parent datastore, rather than accessing the parent data which may not be available in industrial applications. Finally, we merge all retrieved entries to build the child-aware parent datastore:
(K,V) = { Nyc , ∀(x̃p,yc) ∈ (X̃ p,Yc) } . (9)

heading: 3.4 Parent-Enhanced Model Prediction
text: During inference, the child model generates the intermediate representation fθc (xc;yc<t) to query from the child-aware parent datastore. The retrieval distribution from the child-aware parent datastore can be computed as:
pparent−kNN (y c t |xc,yc<t) ∝ k∑
j=1
1yct=vj exp (−d (kj , f (xc,yc<t))/τ) .
(10)
The final probability distribution for predicting the next token yt is the interpolation of the child NMT distribution and the retrieval distribu-
tion weighted by the hyper-parameter λ:
p (yct |xc,yc<t) = λpparent−kNN (yct |xc,yc<t) + (1− λ)pchild−NMT (yct |xc,yc<t) .
(11) Different from vanilla kNN-MT that generates the two distributions from a same NMT model, kNNTL makes use of the parent model rather than the child model to build high-quality datastore, which will generate a more accurate retrieval distribution, and thus better translation performance.

heading: 4 Experiments

heading: 4.1 Setup
text: Parent Language Pairs Our method is independently evaluated using German-English (De-En) and French-English (Fr-En) as the parent language pairs in our experiments. For De-En task, we follow the dataset settings of Li et al. (2022) to train on WMT17 De-En and validate on newstest2013. The training set consists of 5.8M sentences. For Fr-En task, we train on WMT14 Fr-En dataset and validate on newstest2013. we follow the data process of fairseq1 and also randomly select 5.8M samples as the training set. The vocabularies are learned using the joint source-target BPE with 40K merge operations (Sennrich et al., 2016b).
Child Language Pairs We conduct experiments on four low-resource translation benchmarks. We use three translation benchmarks from Global Voices (Tiedemann, 2012; Khayrallah et al., 2020): Hungarian (Hu-En), Indonesian (Id-En), and Catalan (Ca-En). The subset splits follow Khayrallah et al. (2020). The training set contains 15,176, 8,448, and 7,712 instances respectively. Both the validation set and the test set are 2000 instances. We adopt WMT17 Turkish-English (Tr-En) benchmark as the fourth language pair and use newstest2016 as the validation set. We carry out a series of data processing procedures including normalization, tokenization by Moses (Koehn et al., 2007). To enhance the quality of the Tr-En training data, sentences exceeding 60 words in length and with a length ratio greater than 1.5 are removed. The settings of the joint source-target BPE to the child language pairs follow Li et al. (2022).
Baselines We mainly compare our method with the following baselines:
1https://github.com/facebookresearch/fairseq/blob/main/ examples/translation/prepare-wmt14en2fr.sh
• Vanilla NMT (Vaswani et al., 2017) proposes Transformer that significantly improves the performance of NMT. However, its performance is severely limited when applied to the scenario of low-resource machine translation.
• TL (Zoph et al., 2016) is the earliest work on transfer learning, which initializes the child model with copied parameters from the parent model except for the embedding layers of the encoder. For the embedding layers of the encoder, this method initialized it using the embeddings randomized from the parent model. After the initialization stage, the child model is trained on the child data as the usual NMT models.
• TM-TL (Aji et al., 2020) proposes “Token Matching” to conduct transfer learning, which is similar to TL except for the initialization of the embedding layers in the encoder of the child model. For the initialization of the embedding layers, this method assigns the embeddings of common tokens from the parent models to the child model. The embeddings of the rest tokens are initialized as the usual NMT models.
• ConsistTL (Li et al., 2022) enhances the consistency between the predictions of the parent model and the child model during the training stage of the child model. The initialization stage of this method follows TM-TL.

heading: 4.2 Implementation Details
text: Training We adopt the fairseq toolkit for model implementation (Ott et al., 2019). We train the parent model for 80K steps with 460K tokens per batch, a dropout rate of 0.1, a peak learning rate of 0.001, and linear warmup steps of 10K. We tie all embedding layers of the parent models. For child models, we tie the input embedding layers of the decoder and the output projection. We also follow the embedding initialization as TM-TL. We train all the child models for 200 epochs with 16K max tokens per batch for Tr-En and 1K for other language pairs. For child training, we set the warm-up steps to 1K, the label smoothing to 0.1 and the dropout rate to 0.3. Both the attention and activation dropout rates are set to 0.1. To prevent overfitting, a lower peak learning rate of 0.0003 is employed. The α is set to 0.01. The Adam (Kingma and Ba, 2015) optimizer is set to β1 = 0.9, β2 = 0.98. We choose the model with the best validation BLEU for testing.
Inference We use the kNN-box2 (Zhu et al., 2023) to implement kNN retrieval and the FAISS (Johnson et al., 2021) for efficient search. For the child-aware datastore, we tune the hyper-parameters by performing grid search on k̄ ∈ {16, 32, 64, 128} for the Tr-En and k̄ ∈ {256, 512, 1024, 1536} for the other language pairs. During inference, we empirically perform grid search on k ∈ {8, 12, 16, 20, 24, 28} and λ ∈ {0.2, 0.25, 0.3, 0.35, 0.4} and T ∈ {1, 10, 30, 50, 70, 100} to choose the optimal value. All the selected hyper-parameter values for each model and dataset are based on validation sets. As a reference, the hyper-parameters (k, λ and T ) of four language pairs with De-En parent are Id: 28/0.35/10, Ca: 28/0.4/100, Hu: 20/0.4/70, and Tr: 16/0.35/100.
Evaluation We use beam search with a beam width of 5 and a length penalty of 1 for model inference. To fully validate the effectiveness of our proposed method, we use SacreBLEU (Post, 2018), BLEURT (Sellam et al., 2020) and BERTScore (Zhang et al., 2020) to evaluate the generation quality.

heading: 4.3 Main Results
text: Table 2 reports the results on the four low-resource tasks. The results of transfer learning could be divided into two parts according to the usage of the parent language pair. When using De-En as the parent, our method kNN-TL achieves the best performance consistently on all child language pairs in all metrics. Compared with the strong baseline TM-TL that uses the same initialization strategy, kNN-TL achieves large improvements. Moreover, we observe that kNN-TL could still outperform
2https://github.com/NJUNLP/knn-box
the strongest baseline ConsistTL with significant gains. Similar observations can be drawn when we switch the parent to Fr-En, which indicates that kNN-TL brings consistent improvements across different parent language pairs. In summary, the experimental results demonstrate the superiority of our proposed kNN-TL method, as it conducts more comprehensive transfer learning.

heading: 5 Analysis
text: In this section, we conduct extensive analyses to demonstrate the effectiveness of each component in kNN-TL. By default, we choose Ca-En and Tr-En for the child model with the De-En parent model.
Loss for Imposing Consistency Constraints We investigate the effectiveness of MSE that imposes constraints on the output representation, compared with JS loss that encourages consistency over probability distributions. Table 3 demonstrates the impact of learning a consistent representation of translation context on kNN retrieval. Without consistency constraints, the model performs worst on kNN retrieval. When using JS loss, the utilization of kNN retrieval lead to moderate improvements. In contrast, the performance of the kNN retrieval is significantly enhanced using MSE loss. These observations reveal the necessity of learning consistent representations for kNN-TL.
Representation Type for Training and Inference We conduct an empirical study to investigate the impact of representation type for training (consistency learning) and inference (retrieval) respectively. Output and Intermediate respectively represent the output representation and the representation of feed-forward input of the last decoder layer follow Khandelwal et al. (2021). Table 4 lists all the setups and corresponding results. We can observe that utilizing output representation for the training stage while intermediate representation for the inference stages yields the optimal performance. We leave further investigation of the representation type for training and inference as our future work.
Importance of Parent Datastore To verify the importance of the parent datastore in kNN-TL, we compare the parent datastore with the child datastore and the pure NMT model. Table 5 compares the results caused by the pure NMT model and different datastores. Compared with the pure NMT model, the child datastore achieves weak improvements with an average increase of only 0.2 BLEU. This shows that for the low-resource child data, the child model can already learn most of the knowledge in the data well. In contrast to the child datastore, the model is significantly improved with an increase of 1.3 and 0.6 BLEU when using the childaware parent datastore. These findings demonstrate that for low-resource NMT models, fully leveraging the knowledge from high-resource parents is a more effective means of improvement.
Inference Speed-up by Child-Aware Datastore To investigate the impact of the child-aware datastore construction, we analyze the performance of the original parent datastore and child-aware data-
store in terms of BLEU and inference speed, as shown in Table 6. The experimental results show that the implementation of the child-aware datastore leads to an improvement in inference speed, with a 1.5 and 1.7-fold increase observed in two language pairs. This enhancement in speed is achieved while maintaining a comparable performance of using the whole parent datastore. Nonetheless, the decoding speed of kNN-TL remains three times lower than conventional NMT models, which can be mitigated by utilizing other accelerated methods of kNN-based retrieval.
We also analyze the quality-speed trade-off on the Tr-En language pair using the child-aware datastore in Figure 3. The horizontal axis in the figure represents the different values of k̄ used and “ALL” (original parent datastore). It can be observed that as the pre-retrieval k̄ value decreases, there is a corresponding increase in inference speed. When the k̄ is set to 16 (resulting in a reduction of the datastore to less than 30%), the model exhibits a 2.6 times increase in inference speed with a degradation of 0.2 BLEU. The results illustrate that our proposed method can effectively balance the tradeoff between inference speed and performance.
Visualization of Representation Alignment In order to verify the consistency of the intermediate representation of child and parent models, we visualize the representation of the child model and parent model on the target side of the child data. Figure 4 shows intermediate representations generated by the De-En parent model and different Ca-En child models respectively. We can see that there exists a significant discrepancy in the representation of the parent and child model of the
TM-TL. ConsistTL slightly brings the two representations closer but still remains a notable discrepancy. Compared to the previous two models, the representations of the parent model and child model of kNN-TL are highly similar, indicating the effectiveness of our parent-child representation alignment method during training. The utilization of consistency learning via the output distribution serves as an effective constraint on the intermediate distribution. Simultaneously, this provides a sound justification for the ability of the kNN-TL method to effectively retrieve knowledge across parent and child models. In conjunction with the results presented in Table 3, we can conclude that proper alignment of the intermediate representation can optimize the performance of the child model through effective knowledge retrieval.
Effect of Back-translation Back-translation (BT, Sennrich et al., 2016a) is a frequently employed technique in contemporary NMT systems, particularly for low-resource language pairs that suffer from a scarcity of parallel data. To verify the complementarity of our method with BT, we conduct a performance analysis on augmented training data, obtained through BT from News Crawl 2015 English monolingual data. We adopted the experiment settings of Li et al. (2022) to sample 200k English monolingual data at a ratio of approximately 1:1. Table 7 displays the Tr-En results of kNN-TL and baseline methods. By incorporating supplementary back-translated data, kNN-TL can achieve an improvement of 2.7 BLEU and also outperforms
the baseline transfer learning methods. These findings demonstrate the generality of kNN-TL and its complementarity with BT, which facilitates the integration into practical NMT systems with other mainstream approaches.
Model Calibration While ConsistTL (Li et al., 2022) uses the prediction distribution of the parent model, we further incorporate the probability distribution retrieved from the parent datastore during inference. In order to investigate the impact of kNN distribution on inference calibration, we analyze the gap between the confidence and accuracy of the model.3 The smaller gap between the prediction probability (i.e., confidence) and the correctness of generated tokens (i.e., accuracy) indicated better calibration performance (Wang et al., 2020). Figure 5 shows the averaged confidence and accuracy of different methods. Compared with baseline methods, kNN-TL effectively reduces the over-confidence of the model while improving the accuracy. Specifically, kNN-TL exhibits a significant improvement in the model’s calibration performance as it produces a decrease in the gap of 3.1 and 1.8 for the two language pairs, respectively. According to the prior work (Yang et al., 2022), the knowledge of kNN retrieval can prevent the over-confidence of the model on the one-hot labeling, ultimately resulting in elevated generalizability for inference. kNN-TL incorporates the distribution and knowledge from diverse perspectives, thus leading to a more comprehensive transfer learning framework for low-resource NMT.
3https://github.com/shuo-git/InfECE

heading: 6 Related Works

heading: 6.1 Transfer Learning for NMT
text: Transfer learning is an efficient method to boost low-resource NMT models based on the parentchild framework (Wang et al., 2021a; Zoph et al., 2016; Liu et al., 2021a,b), which transfers knowledge from the high-resource parent model to the low-resource child model. Recent works propose to cope with the vocabulary between the parent model and the child model for the initialization of the child model, including using extra transformation (Kim et al., 2019a) and transfer partial embeddings from the parent model (Aji et al., 2020). These works mainly focus on the initialization stage of the child model. ConsistTL revisits the relationship between the parent and child models and proposes to receive continual guidance from the parent model during the child training (Li et al., 2022).
However, the above works still ignore the continual transfer from the parent model during the child inference. To this end, inspired by the kNN mechanism (Khandelwal et al., 2020; He et al., 2021), this paper proposes to conduct cross-model transfer from the parent model throughout the developing process of a child model, which includes the stages of initialization, training and inference.
6.2 k-Nearest-Neighbor Retrieval Recently, non-parametric retrieval-augmented methods have promoted the progress of many fields of NLP, including language modeling (Khandelwal et al., 2020; He et al., 2021), NMT (Khandelwal et al., 2021; Zheng et al., 2021a), named entity recognition (Wang et al., 2022c), question answering (Kassner and Schütze, 2020; Xiong et al., 2021), text classification (Su et al., 2022) and so on. For NMT, A series of approaches incorporate the external knowledge into NMT systems through kNN retrieval from the datastore built with the training data. Some works improve the performance by dynamically adjusting the ratio λ between NMT and kNN (Zheng et al., 2021a; Jiang et al., 2021). Some researchers improve the efficiency of kNN-MT retrieval by pruning the datastore (Wang et al., 2022a), dynamically constructing the datastore (Meng et al., 2022; Wang et al., 2021b; Dai et al., 2023), and reducing the number of steps to be retrieved (Martins et al., 2022a,b). kNN-MT is also applied to various sub-areas of MT, including domain adaptation in MT (Khandelwal et al., 2021; Zheng et al., 2021b), interactive
MT (Wang et al., 2022b), domain adaptation in speech translation (Du et al., 2022), and so on.
It is important to note that when constructing a datastore utilizing a low-resource NMT model, the interpolation of kNN retrieval methodologies may not result in a significant enhancement in performance. In this paper, we propose an extension of the kNN retrieval method to transfer learning, which allows child models to acquire knowledge from a well-trained parent model, instead of relying solely on their limited internal datastores. This enhances the capability of the child models to perform accurate retrieval in low-resource settings.

heading: 7 Conclusion and Future Works
text: In this paper, we propose kNN-TL to transfer knowledge from the parent throughout the entire developing process of child models. kNN-TL aligns the output representations of parent and child during training, allowing for efficient retrieval of useful knowledge from the parent datastore. In addition, kNN-MT builds a child-aware datastore by selectively distilling relevant entries of the largescale parent datastore, thereby improving the inference efficiency. Experimental results on four low-resource NMT benchmarks show a continuous improvement over the other powerful transfer learning methods for NMT. Further analysis reveals the effectiveness and importance to align the output representations for better model improvement. Future works include:1) integrating parent datastores from different high-resource language pairs to improve the performance of the child model, and 2) analyzing the transferability of the parent model through the child-aware datastore construction.
Limitation
In comparison to other transfer learning methods of NMT, kNN-TL incurs extra time costs and more processes to transfer knowledge from the parent model. This is a result of the requirement to construct a high-resource datastore utilizing large-scale parent data and retrieve it. On the other hand, kNNTL requires a substantial amount of storage capacity due to the storage of a datastore containing millions of entries. We employ the output representation layer for the alignment and the intermediate representation layer for the retrieval. This method justification is mainly supported by the results of model validation (Table 4), which might deserve further investigation.

heading: Acknowledgments
text: This work was supported in part by the Science and Technology Development Fund, Macau SAR (Grant Nos. FDCT/060/2022/AFJ, FDCT/0070/2022/AMJ), the National Natural Science Foundation of China (Grant No. 62206076), the Research Program of Guangdong Province (Grant No. 2220004002576), Shenzhen College Stability Support Plan (Grant Nos. GXWD20220811173340003, GXWD20220817123150002), Shenzhen Science and Technology Program (Grant No. RCBS20221008093121053) and the Multi-year Research Grant from the University of Macau (Grant No. MYRG2020-00054-FST). This work was performed in part at SICC which is supported by SKL-IOTSC, and HPCC supported by ICTO of the University of Macau. We would like to thank the anonymous reviewers and meta-reviewer for their insightful suggestions.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitation
7 A2. Did you discuss any potential risks of your work? There is no potential risk.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3&4
3 B1. Did you cite the creators of artifacts you used? 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The data and code used in the paper are publicly available.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The data and code used in the paper are publicly available.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The data and code used in the paper are publicly available.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4
C 3 Did you run computational experiments? 4
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future works include:1) integrating parent datastores from different high-resource language pairs to improve the performance of the child model, and 2) analyzing the transferability of the parent model through the child-aware datastore construction. Limitation
In comparison to other transfer learning methods of NMT, kNN-TL incurs extra time costs and more processes to transfer knowledge from the parent model. This is a result of the requirement to construct a high-resource datastore utilizing large-scale parent data and retrieve it. On the other hand, kNNTL requires a substantial amount of storage capacity due to the storage of a datastore containing millions of entries. We employ the output representation layer for the alignment and the intermediate representation layer for the retrieval. This method justification is mainly supported by the results of model validation (Table 4), which might deserve further investigation