What Knowledge Is Needed? Towards Explainable Memory for kNN-MT Domain Adaptation
Year: 2023

Abstract:
kNN-MT presents a new paradigm for domain adaptation by building an external datastore, which usually saves all target language token occurrences in the parallel corpus. As a result, the constructed datastore is usually large and possibly redundant. In this paper, we investigate the interpretability issue of this approach: what knowledge does the NMT model need? We propose the notion of local correctness (LAC) as a new angle, which describes the potential translation correctness for a single entry and for a given neighborhood. Empirical study shows that our investigation successfully finds the conditions where the NMT model could easily fail and need related knowledge. Experiments on six diverse target domains and two language-pairs show that pruning according to local correctness brings a light and more explainable memory for kNN-MT domain adaptation1.

heading: 1 Introduction
text: Domain adaptation in neural machine translation (NMT) aims at adapting pre-trained NMT models to a target domain (Chu et al., 2017; Thompson et al., 2019; Hu et al., 2019; Zhao et al., 2020; Zheng et al., 2021). Fine-tuning (Luong and Manning, 2015) has been the de facto standard for adaptation. However, fine-tuning suffers from the catastrophic forgetting problem (McCloskey and Cohen, 1989; French, 1999).
Recently, Khandelwal et al. (2021) propose kNN-MT, showing a new paradigm for domain adaptation. kNN-MT first explicitly extracts translation knowledge in the target domain training data into a key-value datastore with a pre-trained NMT model. For each datastore entry, the key is a continuous representation and the value is a symbolic token. The datastore is then used to assist the NMT
1Code will be released at https://github.com/NJUNLP/ knn-box
model during translation. The kNN-MT framework circumvents the necessity to disturb the parameters of the pre-trained NMT model and enables quick adaptation by switching datastores. kNN-MT incorporates the symbolic datastore to assist the neural model (Khandelwal et al., 2021; Zheng et al., 2021; Jiang et al., 2021). However, the datastore usually stores all the target tokens in the parallel data, without considering the capability of the neural model. As a result, the datastore is usually huge in size and possibly redundant.
To understand the relationship between the datastore and the NMT model, this paper conducts investigations on the interpretability issue: what knowledge does the NMT model need? Intuitively, the pre-trained NMT model only needs knowledge that remedies its weaknesses. Thus, we propose to explore this issue from the point of local correctness (Section 3). Our local correctness includes two aspects, the correctness of translating a given entry (entry correctness) and, more importantly, the correctness of performing translation in a given neighborhood in the representation space (neighborhood correctness).
For the entry correctness, we check whether the NMT could make correct translation for the entry itself and accordingly split the datastore entries into two categories, namely known and unknown. Based on entry correctness, we examine neighborhood correctness to more comprehensively evaluate the NMT model’s underlying capability. Specifically, we propose a knowledge margin metric to evaluate the maximum size of the neighborhood where the NMT could make correct translation. Intuitively, the NMT model may fail when the knowledge margin is small.
To verify our interpretation, we devise a datastore pruning algorithm PLAC (Pruning with LocAl Correctness), which simply removes entries with a higher knowledge margin value (Section 4). These entries are less useful for adaptation, because the
2824
NMT model translates well in their neighborhood. We conduct experiments on six diverse target domains in two language pairs (Section 6). Compared with existing pruning baselines (Martins et al., 2022; Wang et al., 2022), PLAC prunes more entries (up to 45%) in four OPUS domains’ datastore without hurting translation performance. Through ablation study, we reveal that simply relying on entry correctness is not enough, showing that the novel metric knowledge margin for the neighborhood correctness could be the key to build a light and more explainable memory for kNN-MT domain adaptation.

heading: 2 Background
text: For NMT domain adaptation, kNN-MT constructs a datastore D based on the given target domain bilingual corpus C and use it to provide helpful target domain translation knowledge for the pretrained NMT model M. In this section, we briefly introduce kNN-MT and its advanced variant, adaptive kNN-MT (Zheng et al., 2021).

heading: 2.1 Building a Domain Specific Datastore
text: Given target domain bilingual corpus C, all translation pairs in C are fed into the frozen pre-trained NMT model for decoding with teacher-forcing (Williams and Zipser, 1989). At decoding time step t, the hidden state from the last decoder layer h(x,y<t) is taken as key and the t-th target token yt is taken as value, resulting in a key-value pair. For the entire corpus, the datastore D is consisted of key-value pairs:
D = {(h(x,y<t), yt) | ∀yt ∈ y, (x,y) ∈ C}, (1) where y<t denotes previous tokens in the sequence y. Each entry in the datastore explicitly memorizes the following translation knowledge: generating the value token at the decoder hidden state key. And the datastore covers all target language token occurrences.

heading: 2.2 Translating with the Datastore
text: During inference, given a source language sentence x, kNN-MT simultaneously leverages M and D to generate target language translation y={y1, y2, · · · , y|y|}. More specifically, at decoding time step t, kNN-MT queries the datastore with the decoder hidden state h(x,y<t) generated by M. The k nearest neighbors of the query
Nk = {(hj , yj)}k1 are retrieved, which are k entries with keys closest to the query according to squared-L2 distance, d. These retrieved knowledge are converted into a distribution over the vocabulary:
pkNN(yt|x,y<t) ∝ (2) ∑
(hj ,yj)∈Nk 1yt=yj exp(
−d(hj , h(x,y<t)) T ),
where T is the temperature. Then, kNN-MT interpolates pkNN with the pre-trained NMT model’s output distribution as the final translation distribution:
p(yt|x,y<t) = λ pkNN(yt|x,y<t) + (1− λ) pNMT(yt|x,y<t) (3)
The complete translation y can be generated by beam search.
2.3 Adaptive kNN-MT
For vanilla kNN-MT, the selection of hyperparameters, such as k or λ, highly affect the final translation performance, which is less stable across languages or domains. Adaptive kNN-MT uses a lightweight meta-k neural network to dynamically determine the usage of retrieved entries, which avoids the tuning of hyper-parameters and achieves a more stable performance (Zheng et al., 2021).

heading: 3 What Knowledge Does the NMT Model Need?
text: Although less accurate, the pre-trained NMT model could perform translation without the datastore. This fact suggests that the NMT model knows some bilingual knowledge of the target domain. However, the construction of datastore dismisses this point and results in a huge amount of entries being stored.
Intuitively, the pre-trained NMT model only needs knowledge that remedies its weaknesses. To find out these weaknesses and build more explainable memory, we start from investigating entry correctness. Based on this basic concept, we further study neighborhood correctness and find that it precisely reflects the NMT model’s strengths and weaknesses.

heading: 3.1 Known v.s. Unknown for Entry Correctness
text: The capability of the NMT model in target domain is difficult to describe directly. However, as the datastore consists of entries constructed on training set, it is easier to check whether the NMT model could make correct translation for them.
This can be efficiently accomplished by an extra evaluation during the teacher-forcing decoding. More specifically, at each time step t of the teacherforcing process, we not only record the hidden states h(x,y<t) and the correct target token yt, but also evaluate the prediction of the NMT model y′t, which is the target token with the highest probability pNMT(y′t|x,y<t) . Then we call an entry as a known entry if the NMT model could predict it correctly; and unknown, otherwise (Equation 4).
(h(x,y<t), yt) is { known, if y′t = yt unknown, o.w. (4)
Obviously, the unknown entries in the datastore are important, because these are the points where the NMT model tends to make a mistake.

heading: 3.2 The Knowledge Margin Metric for Neighborhood Correctness
text: However, entry correctness alone could not fully reveal the NMT model’s weaknesses. Because for known entries, the NMT model may still fail during inference where the context could be similar but different. Considering that the contextualized representations of similar context stay close in the representation space (Peters et al., 2018), we propose to investigate the NMT model’s translation performance in a neighborhood.
We propose a metric called knowledge margin, denoted as km, to measure the neighborhood correctness. Given an entry (h, y), its neighborhood is defined by its k nearest neighbors2 in the datastore Nk(h) = {(hj , yj)}k1 . The knowledge margin of the entry, i.e. km(h), is defined as:
argmax k (hj , yj) is known, ∀(hj , yj) ∈ Nk(h). (5)
Intuitively, km is the maximum size of the neighborhood of the entry h where the NMT could make
2In our implementation, we do not consider the given entry itself as its neighbor.
correct translation. If considering at most k̄ nearest neighbors of h, its knowledge margin will be a number between 0 and k̄.
Please note that the definition of knowledge margin applies for any point in the representation space, because for each point (e.g. an actual query q during inference), its neighborhood Nk(q) could be defined by querying the datastore. This extension allows the investigation of the NMT model at any given point in the representation space.

heading: 3.3 Empirical Analysis
text: We now present an empirical analysis of the relationship between the NMT model and the datastore, and reveal the NMT model’s weaknesses.
Settings We follow Zheng et al. (2021) and consider four domains in German-English OPUS dataset (Tiedemann, 2012) as target domains3. Table 1 lists statistics of four domains4. For pretrained NMT model, we use the winner model of WMT’19 German-English news translation task 5 (Ng et al., 2019). The datastore for each domain is constructed on the corresponding training set with the pre-trained NMT model.
Entry Correctness We collect statistics about the two categories of entries and report results in Table 2. The results show that 56%∼73% (averaging 66.7%) of datastore entries are known by the pre-trained NMT model. This high ratio strongly indicates that a large amount of datastore entries may be redundant.
Neighborhood Correctness We measure neighborhood correctness of each datastore entries and plot the distribution of knowledge margin for known and unknown entries in Figure 1 (k̄ =
3Detailed description of OPUS domains can be found in Appendix A.
4We use the dataset re-split by Aharoni and Goldberg (2020), which removes the overlap between training, development and test sets.
5https://github.com/pytorch/fairseq/tree/main/ examples/wmt19
2048). The distributions on four OPUS domains show the same trends. Most unknown entries has a very low knowledge margin, e.g., around 90% of unknown entries have a margin value between 0 and 4. In contrast, the distribution for known entries is more diverse. The results indicate that the neighborhood correctness is consistent with the entry correctness, but may provide more information for known entries.
To verify the relation between knowledge margin and NMT model’s translation ability, we conduct experiments on the development set for each domain, where translation context are unseen. For each token yt in the dev set, we perform teacherforcing until time step t−1 and query the datastore for the neighborhood at time step t. We evaluate the knowledge margin of the query and the prediction accuracy of the NMT model.
Figure 2 shows the results. For tokens with higher margins, e.g. km ≥ 32, the prediction accuracy of the NMT model is higher than 95%. In contrast, for tokens with lower margins, e.g. km < 4, the accuracy is lower than 50%. This is a strong evidence that the NMT model could easily fail when knowledge margin is small.
In Table 3, we also show a translation example for such a condition, where knowledge margin of the current query is 0 and the NMT model fails to
generate the last subword of “Cyanokit”6.

heading: 4 Building Explainable Memory Based on Local Correctness
text: Because local correctness are good indicators for translation failures. It could also be interpreted as the importance of datastore entries. To verify this interpretation, we propose a pruning algorithm, i.e. Pruning with LocAl Correctness (PLAC), to cut off entries with a high knowledge margin (Algorithm 1).
There are two steps in the algorithm. In the first step, each entry (h, y) in the datastore D is checked for their local correctness. If knowledge margin of (h, y) is greater than or equal to the threshold kp, the entry is collected as the pruning candidates 7.
In the second step, these pruning candidates are randomly selected and get removed from D until the required pruning ratio is reached. Since our method does not need to train any additional neural networks, it can be easily implemented. The pruned
6“Cyanokit” is a drug name, which contains the active substance hydroxocobalamin (vitamin B12).
7In practice, there exists a very small amount of unknown entries with high knowledge margin. In our implementation, we keep them in the datastore because the NMT model cannot make correct translation about these entries.
datastore can be used in different kNN-MT models, such as adaptive kNN-MT.
Algorithm 1 Datastore Pruning by PLAC Input: datastore D, the knowledge margin threshold kp, the
pruning ratio r Output: pruned datastore D
1: candidates← ∅ ▷ step 1: collect 2: for each entry (h, y) in D do 3: if km(h) ≥ kp then: 4: candidates← candidates ∪ (h, y) 5: end if 6: end for 7: repeat ▷ step 2: drop 8: randomly select entry (h, y) from candidates 9: remove (h, y) from D
10: until pruning ratio r is satisfied 11: return D

heading: 5 Experiment Setup
text: This section introduces general experiment setup for evaluating pruning effect. More implementation details can be found in Appendix C.

heading: 5.1 Data and Processing
text: We conduct datastore pruning for 6 different domains from 2 language pairs. Specifically, we take 4 OPUS domains for De-En experiments and 2 UM domains8 for Zh-En experiments (Tian et al., 2014),
8We split the original training set into training, development, test set because there is no development set provided in the original dataset and there exists an overlap between original training and test sets. Detailed description of UM domains can be found in Appendix A.
which are all benchmark dataset for NMT domain adaptation research.
For preprocessing, we use moses9 toolkit to tokenize German and English corpus and jieba10 to tokenize Chinese corpus. Byte pair encoding11 (BPE) is applied for subword segmentation.

heading: 5.2 Pre-trained NMT Model
text: For De-En tasks, we use the winner model of WMT’19 De-En news translation task, which is based on the Transformer architecture (Vaswani et al., 2017). For Zh-En tasks, we train a base Transformer model from scratch on CWMT’17 ZhEn Dataset12 (9 million sentence pairs), since we do not find any publicly available Zh-En pre-trained NMT model on the website.
The pre-trained NMT model is the unadapted general domain model for each language pair, which is the starting point for domain adaptation.
9https://github.com/moses-smt/mosesdecoder 10https://github.com/fxsjy/jieba 11https://github.com/rsennrich/subword-nmt 12http://nlp.nju.edu.cn/cwmt-wmt
For kNN methods, it also serves as the base for building the datastore.

heading: 5.3 Systems for Comparison
text: We report the performance of the following systems for reference: the pre-trained NMT model (Base), the pre-trained model finetuned on each target domain (Finetune) (Luong and Manning, 2015), adaptive kNN-MT with full datastores built for each target domain on their training set (Adaptive kNN) (Zheng et al., 2021). Finetuning and Adaptive kNN are two popular alternatives for adaptation.
The following pruning methods are applied to the datastore of Adaptive kNN for comparison: randomly pruning (Random), cluster-based pruning (Cluster) (Wang et al., 2022), merging similar entries (Merge) (Martins et al., 2022), randomly pruning known entries (Known), pruning all known entries (All Known). Among them, Cluster and Merge are pruning methods based on the context similarity of different entries (Wang et al., 2022; Martins et al., 2022).
We report case-sensitive detokenized BLEU (Papineni et al., 2002) calculated by sacrebleu13 and COMET (Rei et al., 2020) calculated by publicly available wmt20-comet-da14 model. For the prunign methods, statistical significance test (Koehn, 2004) against the full datastore (Adaptive kNN) are conducted as well.

heading: 6 Experiment Results and Analysis

heading: 6.1 Safely Pruning with PLAC
text: Experiment results on OPUS domains are presented in Table 5. For the reference, the pre-trained NMT model usually does not translate well on target domains. Finetuning and Adaptive kNN have
13https://github.com/mjpost/sacrebleu 14https://github.com/Unbabel/COMET
comparable performances. We perform datastore pruning with PLAC for different domains and report the largest pruning ratio without significant performance degradation on the test set.
Compared with using full datastore (Adaptive kNN), our method (PLAC) cutting off 25%-45%15 entries of the datastore while achieving comparable performance. On the two largest domains, “OPUSMedical” and “OPUS-Law”, our method successfully prunes 45% datastore (millions of key-value pairs). Excellent pruning performance validates our analysis concerning with local correctness.
Cluster and Merge lead to a larger degradation of translation performance16, showing that entries with identical target tokens indeed have different importance in assisting the NMT model. Simply pruning all known entries results in a significant drop of performance (All Known). Pruning known entries to the same ratio as PLAC also lead to degradation (Known), although it outperforms Cluster and Merge. These comparisons indicates that the entry correctness only partially reflects entry importance, demonstrating the necessity of the neighborhood correctness analysis with knowledge margin.
The results on UM domains are presented in Table 6. The datastore could be pruned by 30% for “UM-Law” and 15% for “UM-Thesis” Datastore without any sacrifice in translation performance. The other findings are similar with those in GermanEnglish experiments.
15The best pruning ratios are different because for different target domains, the translation knowledge inside the pretrained NMT model is different. For the target domain which is more distant away, the best pruning ratio is naturally smaller.
16Here we show the pruning effect of the two methods under a large pruning ratio. According to the original published results, these methods may suffer less performance degeneration when the pruning ratio is lower, e.g. 10% (Wang et al., 2022).

heading: 6.2 How Knowledge Margin Affects Pruning Performance?
text: In this section, we examine how knowledge margin affects pruning performance and provide more insight into our proposed method. Figure 3 plots BLEU scores of adaptive kNN-MT models with pruned datastore under different pruning ratios on development sets. We can observe that trends are mostly similar in different domains. Pruning by PLAC achieves the best performance over the other baselines and the performance is more stable even with a higher pruning ratio.
Note that Known is a case where neighborhood correctness is dismissed during entry pruning. Although it outperforms Random, Cluster and Merge in most scenarios, its performance is still unstable.
When tuning the hyperparameter kp among {4, 8, 16, 32}, we can see a trade-off between BLEU score and the pruning ratio. Large kp leads to a small sacrifice of BLEU score but a lower pruning ratio. Small kp allows us to prune more entries but causes significant BLEU scores decline after a specific threshold ratio. For example, when kp = 4, it is allowed to prune 55% “OPUS-Medical” datastore, but translation performance declines drastically after the pruning ratio reaches 50%. Finally, we choose the top-right point17 in each subfigure as the best-performed setting for each domain, which are used in other experiments.

heading: 6.3 Datastore Entries With Lower Knowledge Margin Are Indeed Valuable
text: In this section, we want to verify that entries with low knowledge margin are truly important for NMT adaptation. For this purpose, we remove entries
17Hyper-parameter values of these points are reported in Appendix C.
from datastore with a reversed strategy, i.e. the knowledge margin of (h, y) is less than kp.
Table 7 shows pruning effect. We can see that pruning entries with reverse strategy suffers significant performance decline at even a small pruning ratio, demonstrating the importance of these entries for domain adaptation. We also show some cases for each domain in Table 8. We can see that target tokens of these valuable entries are more domainspecific, e.g. “dose” and “Executive”.
6.4 PLAC Is Applicable to Different kNN-MT Variants
For more comprehensive evaluation, we plug our pruned datastore into different kNN-MT variants, i.e. vanilla kNN (Khandelwal et al., 2021), KSTER (Jiang et al., 2021) and adaptive kNN. Experiment results on OPUS-Law domain show that our pruned datastore does almost no harm to the translation performance of different variants, demonstrating the effectiveness of PLAC.

heading: 6.5 Pruned Datastore Occupies Less Memory Space
text: In practice, the datastore must be loaded to CPU and GPU memory during inference. So its size affects the efficiency. Since Faiss index is used to index and represent the datastore, we compare the size of index file before and after pruning (Table 9). For all the domains, our pruning method PLAC significantly reduces the memory occupation. The ratio of saved memory space is roughly identical with the PLAC pruning ratio. For the largest datastore, “OPUS-Law”, the memory space can be reduced by 44%.

heading: 7 Related Work
text: Less attention have been paid to the research of interpretability of kNN-MT. To the best of our knowledge, we are the first to systematically study the relationship between the NMT model and the datastore. As for datastore pruning, Wang et al. (2022)
and Martins et al. (2022) prune the datastore based on the hypothesise that entries with similar translation are redundant. Actually, entries with similar translations may have different importance to the translation. Our analysis suggests one way to understand these differences.

heading: 8 Conclusion
text: It is interesting to explore how a neural model and a symbolic model works together. In this paper, we propose to analyze the local correctness of the neural model’s predictions to identify the conditions where the neural model may fail. By introducing a knowledge margin metric to measure the local correctness, we find that the NMT model often fails when the knowledge margin is small. These results provide support for building a more explainable machine translation system.
Based on analyses, we can safely prune the datastore with the proposed PLAC method. Empirically, the datastore could be successfully pruned up to 45% while retaining translation performance. This results validate our earlier findings about the local correctness and translation failures.
Our method is general to different kNN-MT variants and easy to implement. Future directions maybe using local correctness to explore more interpretability issue of NMT domain adaptation, e.g. catastrophic forgetting.

heading: 9 Limitation
text: During inference, kNN-MT have to query the datastore at each decoding step, which is timeconsuming. Although up to 45% datastore entries can be safely pruned by our method, deploying a high-quality kNN-MT system with fast inference speed is still an open challenge.

heading: 10 Ethical Considerations
text: In kNN-MT works, the symbolic datastore helps adaptation but also introduce privacy concerns. Since kNN-MT explicitly saves all target language tokens in the datastore, there is a risk of privacy leakage. In the future, more efforts may be put into addressing this issue.

heading: Acknowledgement
text: We would like to thank the anonymous reviewers for their insightful comments. Shujian Huang is the corresponding author. This work is supported by National Science Foundation of China (No. 62176120), the Liaoning Provincial Research Foundation for Basic Research (No. 2022-KF-2602).

heading: A Detailed descriptions of Target Domains
text: “OPUS-Medical” domain is made out of PDF documents from the European Medicines Agency. “OPUS-Law” domain is a collection of the legislative text of the European Union. “OPUS-IT” domain is constructed from localization files and documents of GNOME, KDE, PHP, Ubuntu, and OpenOffice. “OPUS-Koran” domain is a collection of Quran translations complied by the Tanzil project. “UM-Law” domain contains law statements from mainland China, Hong Kong, and Macau. “UM-Thesis” domain is composed of journal topics in the research area, including electronics, agriculture, biology, economy, etc.
B Involved Scientific Artifacts
In this section, we list the artifact used in our project:
Moses (LGPL-2.1-License): It is a statistical machine translation system that allows you to automatically train translation models for any language pair.
Jieba (MIT-License): it is a library for chinese word segmentation.
Subword-nmt (MIT-License): Subword-nmt is a package containing preprocessing scripts to segment text into subword units.
Fairseq (MIT-license): It is a sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling and other text generation tasks.
Faiss (MIT-license): It is a library for efficient similarity search and clustering of dense vectors.
For the sake of ethic, our use of these artifacts is consistent with their intended use.
C Implementation Details
We implement adaptive kNN-MT with Zheng et al. (2021)’s released code and script18 based on fairseq19 (Ott et al., 2019). Due to the large space of hyper-parameters, we follow Zheng et al. (2021) to set the number of retrieved entries (ka) as 8 when training adaptive kNN-MT models for
18https://github.com/zhengxxn/adaptive-knn-mt 19https://github.com/pytorch/fairseq
most experiments, and report pruning performance under different ka in Appendix D. During inference, we set beam size as 5 and length penalty as 1.0.
For implementing PLAC, the hyper-parameter kp in Algorithm 1 implicitly determines the maximum number of entries that are allowed to be pruned. So we tune kp among the subset of {4, 8, 16, 32} when given different pruning ratio r.
After buiding the datastore, we follow previous kNN-MT works (Khandelwal et al., 2021; Zheng et al., 2021) and use Faiss20 index (Johnson et al., 2019) to represent the datastore and accelerate nearest neighbors search.
In Table 11, we report hyperparameters to reproduce our main results in Table 5 and 6. In our experiments, it takes at most 1.5 GPU hours to train adaptive kNN-MT models on a single NVIDIA Titan RTX.

heading: D Pruning Effect is Insensitive to Hyperparameter ka
text: To demonstrate the reliability of our pruned datastore, after pruning datastore, we train adaptive kNN-MT models with different hyperparameter ka and evaluate their translation performance (BLEU) on “OPUS-Law” domain’s test set (Table 12). Results show that our pruning method enjoys consistent performance under different ka.
20https://github.com/facebookresearch/faiss
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section 9

heading: 3 A2. Did you discuss any potential risks of your work?
text: section 10

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: appendix B

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 5

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: appendix B

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? appendix B
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Left blank.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? appendix C
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, more efforts may be put into addressing this issue