Iterative Nearest Neighbour Machine Translation for Unsupervised Domain Adaptation
Year: 2023

Abstract:
Unsupervised domain adaptation of machine translation, which adapts a pre-trained translation model to a specific domain without indomain parallel data, has drawn extensive attention in recent years. However, most existing methods focus on the fine-tuning based techniques, which is non-extensible. In this paper, we propose a new method to perform unsupervised domain adaptation in a non-parametric manner. Employing only indomain monolingual data, this method jointly perform nearest neighbour inference on both forward and backward translation directions. The forward translation model creates nearest neighbour datastore for the backward direction, and vice versa, strengthening each other in an iterative style. Experiments on multidomain datasets demonstrate that our method significantly improves the in-domain translation performance and achieves state-of-the-art results among non-parametric methods.

heading: 1 Introduction
text: Neural machine translation (NMT) has demonstrated impressive performance when trained on large-scale corpora. However, despite the abundance of general-domain parallel data, domainspecific parallel data is not readily available (Chu and Wang, 2018). Therefore, how to adapt a general-domain NMT model via in-domain monolingual data has become the research focus in the community. Since no annotated data is involved, this effort is also generally known as the unsupervised domain adaptation of NMT.
Existing approaches are mostly focused on the data selection and finetuning techniques (Pourdamghani et al., 2019; Aharoni and Goldberg,
†Contribution during internship at ByteDance Inc. ‡Corresponding Authors.
2020a; Hu et al., 2019; Zhang et al., 2022). For example, Hu et al. (2019) proposed to induce indomain lexicon pairs as synthetic data for finetuning. Zhang et al. (2022) proposed to use constrained back-translation model to generate synthetic in-domain data for fine-tuning. Despite the progress they have made, the cumbersome finetuning process would lead to catastrophic forgetting (Thompson et al., 2019) and decrease the performance on general domain. Besides, a copy of parameters is required for each domain, which is not flexible facing multi-domain scenario. Therefore, Zheng et al. (2021) propose to perform adaptation based on non-parametric nearest neighbour inference, and they introduce an autoencoder task based on target language to enable in-domain data construction with monolingual data. However, adapter layers are still required to be fine-tuned in their method, which is not fully non-parametric.
In this work, we propose an iterative nearest neighbour approach named Iter-kNNMT to achieve fully non-parametric unsupervised domain adaptation. Our framework is built based on the recently proposed kNN-MT (Khandelwal et al., 2021). We employ two pre-trained generaldomain NMT models in both forward and backward directions, and the datastores are constructed by the model in the reverse direction for each other. The forward model performs nearest neighbour inference to the source language sentences, and the results serve as the datastore for the backward model. Then, the backward model performs nearest neighbour inference to the target sentences and generate better datastore for the forward model. This process is iteratively performed, making the most of monolingual data for nonparametric inference.
We evaluate the proposed Iter-kNNMT on multidatasets, including IT, Medical, Law and Koran
13294
domains. Experimental results show that, without introducing any extra parameters, we are able to achieve 6 BLEU improvement on in-domain translation, bringing a new state-of-the-art results among non-parametric adaptation methods.
2 Approach
2.1 Preliminary: kNN-MT
kNN-MT can be formulated as the following two steps, namely datastore construction and nearest neighbour decoding.

heading: 2.1.1 Datastore Construction
text: Given a pretrained NMT model and an in-domain parallel corpus (X ,Y), kNN-MT first constructs a key-value datastore as follows:
D(X ,Y) = ∪
(x,y)∈(X ,Y) {(f(x, y<i), yi), ∀yi ∈ y}
where the keys are the mapping representations of all the translation contexts in the training set using the model representation f(·), and the values are corresponding ground-truth tokens, and (x, y) is a parallel sentence pair.

heading: 2.1.2 Nearest Neighbour Decoding
text: During inference, on each step i, kNN-MT models the decoding probability PkNN(ŷi|x, ŷ<i) by measuring the distance between query f(x, ŷ<t) and its k-nearest representations in D(X ,Y). Denote the retrieved neighbors as N i = {(hj , yj), j ∈ {1, 2, ..., k}}, and then a kNN distribution over vocabulary is computed as:
PkNN(ŷi|x, ŷ<i) ∝ ∑ (hj ,yj)∈N i Iŷi=yjexp (−d(hj , f(x, ŷ<i)) τ )
where τ is the temperature, and d(·, ·) is the L2 distance function. The final probability for the next token is the interpolation of PNMT(yi|x, y<i) and PkNN(yi|x, y<i) with a tunable weight λ:
P (ŷi|x, ŷ<i) = (1− λ)PNMT(ŷi|x, ŷ<i)+ λPkNN(ŷi|x, ŷ<i)
2.2 kNN-MT with monolingual data
An effective method to improve domain-specific machine translation with monolingual data is to augment the parallel training corpus with backtranslations of target language sentences (Sennrich et al., 2016). In the case of kNN-MT, the utilization of monolingual data can follow this style. Specially, given a set of sentences Y in target language, a pre-constructed NMT model is used to automatically generate their translations X̃ in source language. Then the datastore can be created based on the synthetic data (X̃ ,Y):
D(X̃ ,Y) = ∪
(x̃,y)∈(X̃ ,Y) {(f(x̃, y<i), yi), ∀yi ∈ y}
This datastore can then be retrieved to interpolate the prediction of forward translation model. Although the source language sentences X̃ are synthetic, the target sentences Y are fluent and intact. Therefore, when performing nearest neighbour retrieval, the best tokens can be retrieved largely based on the fitness to target context. As discussed in previous research (Edunov et al., 2018), target fluency is one of the major factors to hinder the performance on domain-specific translation. Therefore, the interpolated probability distribution would be more inclined to the target-domain.
2.3 Iter-kNNMT
In most applications, if the target monolingual data Y is accessible, a monolingual source data X would be accessible, too. To make the most of monolingual data for nearest neighbour inference, we extend the task setting from solely improving the forward NMT model augmented with target monolingual data into a paired one.
As shown in Figure 1, our method runs nearest neighbour inference bidirectionally and refines the datastore iteratively. At each iteration step j:
1) Model fx→y performs kNN inference on datastore Dj(X̃ ,Y) to decode the monolingual data X into Ỹj+1, which is combined into the datastore Dj+1(Ỹ,X ) for backward translation. 2) Then model fy→x performs kNN inference on Dj+1(Ỹ,X ) to decode the monolingual data Y into X̃j+1, forming datastore Dj+1(X̃ ,Y) for the next iteration step. The newly generated datastore would contain more diverse and fluent source context, serving as a better memory base for the next kNN retrieval. Notice no further parameter is introduced during the whole procedure, only the datastore is updated, therefore our method is totally non-parametric.

heading: 3 Experiments

heading: 3.1 Setup
text: We use the same multi-domain dataset as Aharoni and Goldberg (2020b) to evaluate the effectiveness of our proposed method. We mainly experiment on the adaptation of four domains of IT, Medical, Law and Koran. To exclude the influence of parallel sentence pairs, we divide the training set into two halves, and fetch the source side of first half and target side of second half, forming two unaligned monolingual in-domain datasets X and Y , and dev and test set are kept unchanged.
The WMT19 German-English News translation task winner model (Ng et al., 2019) is chosen as our general domain model, and the same setting is applied to train the English-German model1. Faiss2 is used to build the in-domain datastore to carry out fast nearest neighbor search, and 4096 cluster centroids are learned for each domain. We
1We do not introduce monolingual data as augmentation, only following the setting of their basic model.
2https://github.com/facebookresearch/faiss
set the hyper-parameter τ as 4 for IT, Medical, Law, and 40 for Koran. The λ is tuned on the indomain dev sets for different methods, and we use (4, 8, 16, 24, 32) as the value for k.
To avoid the influence of random data partition, all results are the average of 5 runs with different random seeds when partitioning the data.

heading: 3.2 Baselines
text: We mainly compare with the following methods for translation domain-adaptation:
• Basic NMT The general-domain model is directly evaluated on the target domain;
• Copy-kNNMT In-domain datastore is created based on (Y,Y) for kNN-MT inference;
• BT-kNNMT In-domain datastore is created based on back-translated data (X̂ ,Y) for kNNMT inference;
• Parallel-kNNMT Ground-truth in-domain parallel data (X ,Y) is used to generate the datastore, which can be regarded as the upper bound of the kNN retrieval based methods;
• UDA-kNNMT (Zheng et al., 2021) This method first creates datastore based on (Y,Y), and then introduces lightweight adapters to map the token-level representation to the ideal representation of translation task. Notice due to the introduction of adapter layers, this method is not fully non-parametric.

heading: 3.3 Main Results
text: As shown in Table 1, our methods can surpass all baselines and achieve the state-of-the-art among non-parametric methods. Especially, we are able to surpass the result of UDA-kNN by 1 BLEU
score without the introduction of an extra training phase. Copy-kNNMT could hardly bring any improvement. BT-kNNMT, which is actually our method without iteration, could introduce notable improvement, but still underperforms our method. We attribute this to the iteration process, where domain-specific knowledge is written into datastore and further refined during the iteration. While not introducing any extra training parameters, the datastore itself is able to memorize and update the domain-related lexical and syntactic knowledge, and this can be actually deemed as a nonparametric learning process.
We also illustrate the variation of BLEU score according the iteration number on EN-DE direction. As shown in Figure 2, the BLEU score increases rapidly in the first two iterations, but the improvement would be marginal or even negative in the following iterations. Therefore, we set the max iteration number as 5 in all experiments.

heading: 4 Ablation Studies

heading: 4.1 The influence of datastore size
text: In this section, we want to investigate the influence of the datastore size. To this end, we split the indomain data into different folds, and fetch two unaligned folds for datastore building.
As shown in Figure 3, on EN-DE direction, our Iter-kNNMT can bring consistent improvement and surpass UDA-kNNMT among different data scales. Despite the monolingual data being limited, we are able to create various key-value pairs with different kNNMT models in different iterations. With different back translations for a single target sequence, the most suitable key-value pair would have more possibility to be retrieved.

heading: 4.2 Refinement and Accumulation
text: To verify that the improvement comes from both the refinement during iteration and the accumulation of datastore, we perform two contrast experiments on EN-DE direction. Firstly, we perform nbest decoding with random sampling on the back translation model, and generate 5 different source sentences for each target sequence, forming an accumulated datastore without refinement. Secondly, we perform iteration without accumulation, only using the newest datastore each time.
As can be seen in Table 2, both refinement and accumulation play an important role in the IterkNNMT. While datastore is indeed refined during the iteration, accumulation is necessary to keep the variety and robustness of datastore. While the refined back translation can function as more accurate datastore, the comparably noisy back translation can also improve the robustness of the retrieval. Also, if there is no refinement from iteration, a single back-translation model only has limited decoding space induced by pre-train data, therefore the constructed datastore can also provide limited domain-specific guidance.

heading: 5 Conclusion
text: This paper proposes a simple yet effective method to perform unsupervised domain adaptation of ma-
chine translation in a non-parametric manner. We perform nearest neighbour inference on both forward and backward directions, strengthening each other in an iterative manner. While accumulated datastore is more robust and effective than datastore generated in a single pass, the accumulated datastore introduce extra retrieval overhead. In the future, we would investigate how to compress the datastore and improve the decoding efficiency.
Limitations
Due to the lack of research in this area, there is only one direct related paper to our work, which serves as the main baseline in our experiments. We hope we can compare our method with more related works to verify its effectiveness in the future. Also, the domain adaptation problem not only exits in the machine translation filed, but also various generation and understanding NLP tasks, where we should evaluate our method on if we are not limited by time and resource.

heading: Acknowledgements
text: This work is supported by National Key RD Program of China (2020AAA0108000), National Natural Science Foundation of China (62276077, U1908216), Key RD Program of Yunnan (202203AA080004) and Shenzhen College Stability Support Plan (No. GXWD20220811170358002).

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Left blank.
3 A2. Did you discuss any potential risks of your work? Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we would investigate how to compress the datastore and improve the decoding efficiency. Limitations
Due to the lack of research in this area, there is only one direct related paper to our work, which serves as the main baseline in our experiments. We hope we can compare our method with more related works to verify its effectiveness in the future. Also, the domain adaptation problem not only exits in the machine translation filed, but also various generation and understanding NLP tasks, where we should evaluate our method on if we are not limited by time and resource