XDailyDialog: A Multilingual Parallel Dialogue Corpus
Year: 2023

Abstract:
High-quality corpora are significant to the development of dialogue models. However, most existing corpora for open-domain dialogue modeling are limited to a single language. The absence of multilingual open-domain dialog corpora not only limits the research on multilingual or cross-lingual transfer learning but also hinders the development of robust opendomain dialogue systems that can be deployed in other parts of the world. In this paper, we provide a multilingual parallel open-domain dialog dataset, XDailyDialog,1 to enable researchers to explore the challenging task of multilingual and cross-lingual open-domain dialogue. XDailyDialog includes 13K dialogues aligned across 4 languages (52K dialogues and 410K utterances in total). We then propose a dialogue generation model, kNN-Chat, which has a novel kNN-search mechanism to support unified response retrieval for monolingual, multilingual, and cross-lingual dialogue. Experiment results show the effectiveness of this framework.

heading: 1 Introduction
text: Developing high-quality open-domain dialogue systems is one of the key challenges in Artificial Intelligence. Unlike closed-domain dialogue systems which deal with specific kinds of conversations (like a chatbot for customer support), open-domain dialogue systems can engage in conversation on any topic. In recent years, there has been a significant increase in research on dialogue due to the rise of voice-based bots, such as Meena (Adiwardana et al., 2020), BlenderBot (Roller et al., 2021) and XiaoIce (Zhou et al., 2020). To advance the quality of open-domain dialogue systems, many largescale corpora have been created (Sordoni et al., 2015; See et al., 2019; Yang et al., 2018; Mazaré et al., 2018; Keskar et al., 2019).
∗ Equal contribution † Corresponding author: Haifeng Wang.
1https://github.com/liuzeming01/XDailyDialog
However, to the best of our knowledge, almost all existing large-scale corpora for open-domain dialogue modeling are limited to a single language, such as English (Sordoni et al., 2015; See et al., 2019; Yang et al., 2018; Mazaré et al., 2018; Keskar et al., 2019), or Chinese (Shang et al., 2015; Wu et al., 2017; Cai et al., 2019; Wang et al., 2020). The absence of multilingual open-domain dialogue corpora not only limits the research on multilingual or cross-lingual transfer learning (Lin et al., 2021) but also hinders the development of robust opendomain dialogue systems that can be deployed in other parts of the world. Previous work on various NLP tasks has shown that multilingual corpora can bring performance improvements in multilingual
12240
or cross-lingual transfer learning. This includes tasks such as conversational recommendation (Liu et al., 2021), task-oriented dialog (Schuster et al., 2019b), semantic parsing (Li et al., 2021), QA and reading comprehension (Jing et al., 2019; Lewis et al., 2020; Artetxe et al., 2020; Clark et al., 2020; Hu et al., 2020; Hardalov et al., 2020), machine translation (Johnson et al., 2017b), document classification (Lewis et al., 2004; Klementiev et al., 2012; Schwenk and Li, 2018), semantic role labelling (Akbik et al., 2015) and NLI (Conneau et al., 2018). Thus, we believe that multilingual training data might enhance multilingual or crosslingual transfer learning for open-domain dialogue as well.
To facilitate the study of multilingual and crosslingual dialogue, we present a multilingual parallel dialog dataset, XDailyDialog, for multilingual and cross-lingual open-domain dialogue. XDailyDialog consists of 13K dialogues aligned across 4 languages (52K dialogues and 410k utterances in total). The most significant advantage of parallel data over non-parallel data, such as XPersona(Lin et al., 2020), is that it can support crosslingual tasks.
We define 3 task settings using XDailyDialog. As shown in Figure 1, the first task is monolingual dialogue, where dialogue context and response are in the same language. It aims at investigating the performance variation of the same model across different languages. The figure also illustrates another task that is called multilingual dialogue. Here we directly mix training instances of the 4 languages into a single training set and train a single model to handle multilingual dialogue at the same time. Finally, the last task is cross-lingual dialogue, where model input and output are in different languages, e.g. dialog context is in English and the generated response is in Chinese.
To address these tasks, we build a model using kNearest Neighbors (kNN) and mBART (Liu et al., 2020a). We conduct an empirical study of the proposed model on XDailyDialog. Our experimental results indicate that the use of cross-lingual data can bring performance improvement in the monolingual dialogue.
Our work makes the following contributions:
• To facilitate the study of multilingual and cross-lingual dialogue, we create a novel corpus XDailyDialog, the first publicly available multilingual parallel open-domain dialogue
corpus.
• We define 3 tasks, including monolingual, multilingual, and crosslingual dialog, based on XDailyDialog. Automatic evaluation and human evaluation results confirm the benefits of this corpus for monolingual dialogue.
• We propose a dialog generation framework, kNN-Chat, with a novel kNN-search mechanism that can support unified token retrieval for monolingual, multilingual, and crosslingual dialogue generation. Our experimental results confirm the effectiveness of this framework.

heading: 2 Related Work
text: Multilingual and Cross-lingual Datasets for dialog Multilingual dialog datasets are relatively scarce. Lin et al. (Lin et al., 2020) propose a Multilingual Persona-Chat dataset, XPersona, by extending the Persona-Chat corpora (Dinan et al., 2019) to 6 languages: Chinese, French, Indonesian, Italian, Korean, and Japanese. In XPersona, the training sets are automatically translated using translation APIs, while the validation and test sets are annotated by humans. XPersona focuses on cross-lingual personalized dialog and is not parallel, while XDailyDialog focuses on multilingual and cross-lingual dialog generation and is parallel. Liu et al(Liu et al., 2021) construct a Multilingual conversational recommendation dataset, DuRecDial 2.0, by Crowdsourcing translation based on DuRecdial (Liu et al., 2020b) to 2 languages: Chinese, and English. DuRecDial 2.0 focuses on conversational recommendation, while XDailyDialog focuses on dialog generation and has more languages.
Multilingual and Cross-lingual Datasets for Task-oriented Dialog Several multilingual taskoriented dialogue datasets have been published (Mrkšić et al., 2017b; Schuster et al., 2019a), enabling evaluation of the approaches for crosslingual dialogue systems. (Lin et al., 2021) introduces the first bilingual multi-domain dataset for task-oriented dialogue modeling, which has only two languages and is not parallel. mrksi et al .(Mrkšić et al., 2017b) annotated two languages (German and Italian) for the dialogue state tracking dataset WOZ 2.0 (Mrkšić et al., 2017a) and trained a unified framework to cope with multiple languages. Meanwhile, Schuster et al. (Schuster et al., 2019a) introduced a multilingual NLU dataset and
highlighted the need for more sophisticated crosslingual methods. Those datasets mainly focus on multilingual NLU and DST for task-oriented dialogue and are not parallel. In comparison with them, XDailyDialog is a multilingual dataset for open-domain dialog, which has 4 languages and is parallel.

heading: 3 Dataset Collection
text: XDailyDialog aims to collect high-quality parallel data for the research of monolingual, multilingual, and cross-lingual open-domain dialog. It is translated from DailyDialog (Li et al., 2017), which is a human-written, well-formatted English dataset. In this section, we describe how XDailyDialog is collected.

heading: 3.1 Data Collection
text: Human Translation We select 3 common languages (Italian, Chinese, and German) from 3 different language families to be translated by professional translators.2 To guarantee the quality of translation, we use a strict quality control procedure.
First, we randomly sample 100 conversations from DailyDialog and assign them to more than 50 professional translators. Following (Liu et al., 2021), all translation results are assessed by 3 data specialists with translation experience after translation. Specifically, data specialists randomly select 20% of each translator’s translation results for assessment. The assessment is done at a wordlevel, utterance-level, and session-level. For wordlevel assessment, they assess whether the choice of words is appropriate, and whether there are typos. For utterance-level assessment, they assess whether the utterance is accurate and colloquial. For session-level assessment, they assess whether the session is coherent and parallel to DailyDialog. Only if the error rate is less than 5%, the translators can pass. Finally, we pick 20 translators.
Then, the 20 translators translate about 500 utterances at a time. After data translation, data specialists randomly select 10-20% of each translator’s translation results for assessment in the same way as above. The translators can continue to translate only after they pass the assessment.
2http://www.ethnologue.com

heading: 3.2 Dataset Quality Analysis and Statistics
text: Quality Analysis of Human Translation We conduct human evaluations for data quality. A dialog will be rated “1” if all utterances are accurate and colloquial and the dialogue session is coherent., otherwise “0”. Then we ask 3 data specialists with translation experience to judge the quality of 100 randomly sampled dialogues (about 800 dialogue utterances). 3 Finally, we obtain an average score of 0.96, 0.97, and 0.98 for German, Italian, and Chinese, respectively.
Dataset Statistics Table 1 provides statistics of XDailyDialog, indicating rich dialog languages. We believe that XDailyDialog would better facilitate the study of multilingual and cross-lingual dialog.
4 Task Formulation on XDailyDialog
Let D = {(Xi, Ti,Ai, Ei, Yi)}ni=1 denote a set of dialogues in XDailyDialog, where ∀i ∈ {1, . . . , n}, Xi refers to a dialog context, Yi is a response to Xi, and Ti, Ai, Ei is the dialogue topic, dialogue act, dialogue emotion corresponding to Yi, respectively. Given a context X = {uj}i−1j=0 associated with a dialogue topic T , a dialogue act A and a dialogue emotion E , the aim is to produce a proper response Y = ui, where uj and ui are dialogue utterances. As shown in Table 1, the dialogues in XDailyDialog include languages L = (De,En, It, Zh).
Monolingual dialog: Task 1: (Xl, Tl, Al, , El, Yl), where ∀l ∈ L. With these 4 monolingual dialogue forms, we can investigate the performance variation of the same model trained on 4 separate datasets in different languages. In our experiments, we train 4 models respectively for the 4 monolingual tasks. Then we can evaluate their performance variation across all languages to see how the changes between languages can affect model performance.
3We calculate the averaged weighted kappa value for all sampled dialogues and get a high score of 0.81, demonstrating good agreements between data specialists.
Multilingual dialog: Task 2: (XL, TL, AL, , EL, YL). Similar to multilingual conversational recommendation (Liu et al., 2021), multilingual neural machine translation (Johnson et al., 2017b) and multilingual reading comprehension (Jing et al., 2019), we directly mix training instances of the 4 languages into a single training set and train a single model to handle the 4 languages dialogue at the same time. This task setting can help us investigate if the use of additional training data in other languages can bring performance benefits for a model of the current language.
Cross-lingual dialog: The cross-lingual dialogue is Task 3: (Xl2, Tl1, Al1, El1, Yl1), where l1 and l2 are two different languages in L. In Task 3, when given a related dialogue topic, dialogue act, and dialogue emotion (e.g., in Engish), the model takes dialog context in one language (e.g., in Chinese) as input, and then produces responses in another language (e.g., in Engish) as output. Understanding the mixed-language dialog context is a desirable skill for end-to-end dialog systems. This task setting can help evaluate if a model has the capability to perform cross-lingual tasks.

heading: 5 Our Approach
text: We propose a non-parametric method named kNNChat for monolingual, multilingual, and crosslingual dialogue generation, as shown in Figure 2. kNN-Chat is inspired by kNN language models such as kNN-LM (Khandelwal et al., 2020b) and kNN-MT(Khandelwal et al., 2020a) and implemented for monolingual, multilingual, and crosslingual dialogue by using dialogue context, response, and extra information (dialogue emotion, topic, action, target language). Compared to recent dialogue systems with kNN (Fan et al., 2020), our experimental results show that kNN-Chat, a nonparametric method (without extra training except for pre-training mBART), could also be effective for monolingual, multilingual, and cross-lingual dialogue generation.

heading: 5.1 Model Architecture
text: We utilize a unified generative framework for the given 4 monolingual tasks, 4 cross-lingual tasks, and 1 multilingual dialogue task. We choose the unified generative framework as it has a) interpretable responses, b) flexible language control, and c) efficient training and inference.
Specifically, kNN-Chat, contains three modules: (1) a generative module to encode multiple languages and decode coarse-grained target language responses (mBART is used in our experiments). (2) a datastore module to store key-value pairs, where the key is the representation of dialogue context and extra information, and the value is the corresponding response token. (3) a kNN-search module (faiss (Johnson et al., 2017a) is used in our experiments) to search similar dialogue tokens from the datastore according to the generative model’s representation of the next token.
Moreover, kNN-Chat is interpretable as similar dialogue contexts are retrieved for generation and is flexible as it is adaptable to different language settings with corresponding arbitrary amount of data size by using faiss (Johnson et al., 2017a). Besides, it is efficient as it can extend to any trained generative model such as mBART or GPT without extra training processes.

heading: 5.2 Generative Module
text: As shown in Figure 2, we use mBART (Liu et al., 2020a) as our generative model for all tasks. We do not modify the mBART encoder-decoder architecture but design the input-output for mBART. Specifically, we concatenate dialogue emotion, dialogue topic, and dialogue act with dialogue context for better response generation as the input of mBART. Since the response could be in different languages, we also append a language identifier of the response to the end of the input. The output of mBART is the coarse-grain dialogue response in the target language specified by the language identifier.
We use the pre-trained mBART model from (Liu et al., 2020a) and fine-tune it on task-specific corpus using the faiseq (Ott et al., 2019). When mBART is finetuned, mBART can be directly used to generate dialogue response and served as our baseline model.

heading: 5.3 Datastore Module
text: After mBART is finetuned, we construct the datastore of kNN-Chat based on trained mBART before response generation. Datastore consists of a set of key-value pairs. Let I = {X,A, T , E}, given an input-response pair (in, y) ∈ (I,Y) from the training set, we use the trained mBART model to generate the t-th token yt based on the input and generated tokens (in, y<t). When mBART generates t-th token yt, it also produces a high-dimensional
representation vector. The vector corresponding to yt can be denoted as f(in, y<t). Finally, each token of response in the training set has a representation vector. The representation f(in, y<t) and response token yt are then used as key and value in the datastores respectively:
(K,V) = ⋃
(in,y)∈(I,Y)) {(f(in, y<t), y<t), ∀yt ∈ y}
Datastore can be created by one forward pass of mBART from the training set. For monolingual and cross-lingual tasks, we build one datastore for one language setting. For multilingual tasks, we build datastores for each language separately. The datastore size (number of training set response tokens) is set to 2 million for monolingual and cross-lingual experiments and 20 million for the multilingual experiment. We train faiss (Johnson et al., 2017a) index on datastore and then search potential response tokens from these datastores by kNN-search module.

heading: 5.4 KNN-search Module
text: When kNN-Chat predicts yt at time step t, mBART produces the representation f(in, y<t) for yt according to the generated tokens y<t and the dialogue input. kNN-Chat uses the representation
of yt to search k nearest neighbors. Suppose the queried neighbors for f(in, y<t) are N t = {(ki, vi) ∈ (K,V ), i ∈ {1, 2, . . . , k}}, where ki and vi are i-th key vector and i-th tokens in nearest neighbors. Then the distribution of neighbors over the decoder vocabulary can be calculated as:
pknn(yt|in, y<t) ∝ ∑
(ki,vi)
1 yt=viexp( −d(ki,f(in,y<t)) T ))
where T is the temperature and d() denotes the distance between representation vectors. After we get the pknn, the final probability of yt can be computed as the interpolation of two distributions with a hyper-parameter λ:
p(yt|in, y<t) = λpknn(yt|in, y<t)+(1−λ)pmBART (yt|in, y<t)
where pmBART (yt|in, y<t) denotes the vanilla mBART prediction probability in section 5.2.

heading: 6 Experiments and Results

heading: 6.1 Experiment Setting
text: Dataset For the train/development/test set of XDailyDialog, we follow the split of (Li et al., 2017),
with one notable difference that we filtered the duplicate data in (Li et al., 2017).
We conduct both automatic and human evaluation for kNN-Chat and baselines on XDailyDialog.
Automatic Evaluation Metrics: For automatic evaluation, we follow the setting in previous work (Li et al., 2017) to use several common metrics such as F1, BLEU (Papineni et al., 2002) (BLEU14), and DISTINCT (DIST-1 and DIST-2) (Li et al., 2016) to measure the relevance, fluency, and diversity of generated responses.
Human Evaluation Metrics: The human evaluation is conducted at the level of both turns and dialogues. For turn-level human evaluation, we ask each model to produce a response conditioned on a given context. The generated responses are evaluated by 4 evaluators in terms of fluency, appropriateness, and informativeness. For dialoguelevel human evaluation, we let each model converse with evaluators. For each model, we collect 30 dialogues. These dialogues are then evaluated by 4 evaluators in terms of coherence that examines fluency, relevancy, and logical consistency of each response when given context. The evaluators rate the dialogues on a scale of 0 (poor) to 2 (good) in terms of each human metric.4

heading: 6.2 Baselines
text: We carefully select two strong baselines for multilingual and crosslingual natural language generation.
mBART (Liu et al., 2020a) is a multilingual sequence-to-sequence denoising auto-encoder pretrained on CC25 (Wenzek et al., 2020; Conneau et al., 2020). It provides a set of parameters that can be fine-tuned for any of the language pairs in CC25, including all languages in XDailyDialog. We treat our 3 tasks as Machine Translation tasks as (Liu et al., 2021). The mBART model can serve as a strong baseline for multilingual and cross-lingual dialogue generation.
mT5 (Xue et al., 2021) is a massively multilingual pre-trained text-to-text transformer model, trained following a similar recipe as T5 (Raffel et al., 2020). It can be fine-tuned for 100 languages, including all languages in XDailyDialog.

heading: 6.3 Experiment Results
text: Table 2, 3, 4, 5, 6, and 7 report automatic evaluation results and human evaluation results of our model
4Please see Appendix A.2 and A.3 for more details.
and all baselines.

heading: 6.3.1 Experiment Results for Monolingual dialog
text: According to the evaluation results in Table 2, and Table 3, kNN-Chat outperforms mBART and mT5 across almost all the tasks or metrics, which confirms the effectiveness of kNN-Chat for monolingual dialog generation. The possible reason is that kNN-search can retrieval more appropriate tokens for dialog generation. Furthermore, Chinese dialog(1(Zh→Zh)) is more challenging than other tasks. One possible reason is that both kNN-Chat and baselines can not well model the dialogue context of those two languages.

heading: 6.3.2 Experiment Results for Multilingual dialog
text: Our model vs. Baselines: According to the evaluation results in Table 4, and Table 5, kNN-Chat outperforms mBART and mT5 across almost all the tasks or metrics, which also demonstrates the effectiveness of kNN-Chat for multilingual dialog generation and the flexibility for different language settings.
Monolingual vs. Multilingual: Based on the results in Table 2, 4, 3, and 5, we can find that all languages get worse results in the multilingual task. It indicates that multilingual dialogue generation is a more challenging task than monolingual dialogue generation. The possible reason is that it is more difficult to train the model to generate in multiple languages than to generate in a single language. Moreover, we can find that multilingual tasks are better than monolingual tasks for mT5. It indicates that the use of additional corpora can improve mT5’s performance for multilingual dialog. But the other two models for other language multilingual tasks can not outperform the monolingual tasks. The possible reason is that the pre-trained models can not perform well in the modeling of 4 languages dialog utterances, resulting in poor model performance.

heading: 6.3.3 Experiment Results for Cross-lingual dialog
text: Our model vs. Baselines: According to the evaluation results in Table 6 and Table 7, kNN-Chat outperforms mBART and mT5 across almost all the tasks or metrics, which indicates the effectiveness of kNN-Chat for cross-lingual dialog generation and the flexibility for different language settings.
Monolingual vs. Cross-lingual:
According to the results in Table 2, 3, 6, and 7, the model performs surprisingly better on almost all cross-lingual tasks than the corresponding monolingual tasks (for example, 3(En→De) is better than 1(De→De)) in terms of almost all the automatic and human metrics, which is similar to (Liu et al., 2021). It indicates that the use of multilingual corpora can consistently bring performance improvement for monolingual dialog. One possible reason is that kNN-Chat or mBART can fully
exploit the multilingual dataset, resulting in better model performance.

heading: 6.3.4 Human Evaluation
text: Our human evaluation results are reported in Table 3, 5, and 7, which show that all the models can generate fluent responses. In general, kNNChat is also better than mBART and mT5 in this evaluation, which further confirms the effectiveness of kNN-Chat for monolingual, multilingual, and cross-lingual dialog generation. However, the
models score much lower on appropriateness, informativeness, and coherence. These results highlight the challenges in building a multilingual or crosslingual dialog system and opportunities for future progress.

heading: 7 Conclusion
text: To facilitate the study of multilingual and crosslingual dialog, we created the first publicly available multilingual parallel dataset, XDailyDialog,
for dialog and proposed three challenging tasks for the community based on it. Furthermore, We also built a new conversation generation framework, kNN-Chat, with a novel kNN-search mechanism that can support unified response retrieval for monolingual, multilingual, and cross-lingual dialog generation. Expensive experiment results confirm the effectiveness of this framework. We hope that XDailyDialog would help push forward research in the unified end-to-end monolingual, multilingual,
and cross-lingual conversational modeling.
Limitations
The main limitation of this work is that the pretraining model can not deal with numerical knowledge well. In future work, we will try to enhance the ability of the pre-training model to better deal with numerical knowledge.
Ethics Statement
We make sure that XDailyDialog has been collected in a manner that is consistent with the terms of use of any sources and the intellectual property and privacy rights of the original authors of the texts. And crowd workers were treated fairly. This includes but is not limited to, compensating them fairly and ensuring that they were able to give informed consent, which includes but is not limited to, ensuring that they were voluntary participants who were aware of any risks of harm associated with their participation. In this paper, we propose a novel multilingual corpus for end-to-end dialog training and evaluation. Our corpus neither introduces any social/ethical, since we generate data by human translation or machine translation. We do not foresee any direct social consequences or ethical issues.

heading: Acknowledgments
text: Thanks for the insightful comments from the reviewers. This work was supported by the National Key R&D Program of China (2021ZD0110501).

heading: A Appendix
text: A.1 Training and Inference Parameters of kNN-Chat
We train and inference all our experiments on NVIDIA-SMI Quadro RTX 8000 GPU. The CUDA version is 11.4. Experiments are conducted with fairseq and faiss tool. The parameters we used are shown in table 8.
A.2 Turn-level Human Evaluation Guideline Fluency measures fluency of each response:
• score 0 (bad): unfluent and difficult to understand.
• score 1 (fair): there are some errors in the response text but still can be understood.
• score 2 (good): fluent and easy to understand.
Appropriateness examines relevancy of each response when given the context:
• score 0 (bad): not relevant to the current context.
• score 1 (fair): relevant to the current context, but using some irrelevant knowledge.
• score 2 (good): otherwise.
Informativeness
• score 0 (bad): safe response / universal response, and not relevant to the current context.
• score 1 (fair): safe response / universal response, but relevant to the current context.
• score 2 (good): otherwise.
A.3 Dialogue-level Human Evaluation Guideline
Coherence measures fluency, relevancy, and logical consistency of each response when given the current context:
• score 0 (bad): more than two-thirds responses irrelevant or logical contradictory to the given context.
• score 1 (fair): more than one-third of responses are irrelevant or logical contradictory to the given current context.
• score 2 (good): otherwise.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8

heading: 3 A2. Did you discuss any potential risks of your work?
text: Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: 1 and 2
7 A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Not applicable. Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3 and 6
C 7 Did you run computational experiments? Left blank.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A.1
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Ethics Statement

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? 3

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Ethics Statement
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: In future work, we will try to enhance the ability of the pre-training model to better deal with numerical knowledge