I run as fast as a rabbit, can you? A Multilingual Simile Dialogue Dataset
Year: 2023

Abstract:
A simile is a figure of speech that compares two different things (called the tenor and the vehicle) via shared properties. The tenor and the vehicle are usually connected with comparator words such as "like" or "as". The simile phenomena are unique and complex in a real-life dialogue scene where the tenor and the vehicle can be verbal phrases or sentences, mentioned by different speakers, exist in different sentences, or occur in reversed order. However, the current simile research usually focuses on similes in a triplet tuple (tenor, property, vehicle) or a single sentence where the tenor and vehicle are usually entities or noun phrases, which could not reflect complex simile phenomena in real scenarios. In this paper, we propose a novel and high-quality multilingual simile dialogue (MSD) dataset to facilitate the study of complex simile phenomena. The MSD is the largest manually annotated simile data (∼20K) and it contains both English and Chinese data. Meanwhile, the MSD data can also be used on dialogue tasks to test the ability of dialogue systems when using similes. We design 3 simile tasks (recognition, interpretation, and generation) and 2 dialogue tasks (retrieval and generation) with MSD. For each task, we provide experimental results from strong pretrained or state-of-the-art models. The experiments demonstrate the challenge of MSD and we will release the data/code on GitHub.

heading: 1 Introduction
text: Simile plays an important role in human language to make utterances more vivid, interesting, and graspable (Zhang et al., 2021; He et al., 2022) and is an increasingly studied phenomenon in computational linguistics (Song et al., 2021; He et al., 2022). A simile is a figure of speech that compares two things from different categories (called the tenor and the vehicle) via shared properties (Paul, 1970). A tenor and a vehicle are usually connected with
∗*Corresponding author
comparator words such as "like" or "as". For example, in the first example of Table 1, the tenor is "The boy", the vehicle is "a rabbit", the event is "run", the comparator is "as ... as" and the shared property is "fast".
The current simile research usually focuses on the simile in a triplet (tenor, shared property, vehicle) (Song et al., 2021) or a single sentence (Bizzoni and Lappin, 2018; Liu et al., 2018; Li et al., 2022). For example, the simile recognition (Birke and Sarkar, 2006; Liu et al., 2018) task is judging whether a sentence contains a simile, such as distinguishing which of the first and second examples in Table 1 contains a simile. However, a simile in a triplet or a single sentence is not enough to reflect the complex simile phenomena in the real scenario. In this paper, we study similes in reallife dialogue where a tenor and a vehicle can be mentioned by different speakers, exist in different sentences, or occur in reversed order. The third example in Table 1 shows a simile dialogue where the tenor "That fireman" and the vehicle "a bull" are in different utterances. The fourth example in Table 1 shows a simile where the tenor and the vehicle are mentioned by different speakers and the vehicle occurs before the tenor. What is more, different from previous research where the tenor and vehicle are usually single entities (Song et al., 2021) or simple
7223
nominal phrases (Bizzoni and Lappin, 2018), the tenor and vehicle in a real-life dialogue may be a verbal phrase or a long sentence. A verbal phrase can function as the subject or object of a verb, such as the fifth example in Table 1. A sentence is a set of words expressing a statement, a question, or an order, usually containing a subject and a verb. The sixth example in Table 1 shows sentences as the tenor and vehicle. The verbal phrase and sentences can convey richer content and emotions, making the real-life dialogue more vivid and interesting. Studying these complex simile phenomena in a dialogue scenario needs to consider both the dialogue context and the various forms of the tenor and vehicle, and will lead the simile research to a brand new level. However, similes in real-life dialogue scenarios have not been studied by previous research so there are no public benchmarks available nowadays.
To facilitate the simile study, we release a humanannotated, high-quality simile dialogue dataset, which contains both English and Chinese data. The complex simile phenomena in real-life dialogue scenarios not only bring more difficulties to traditional simile tasks such as recognition, interpretation (Su et al., 2016), and generation (Li et al., 2022) but also raise challenges for dialogue research, e.g. generation and retrieval tasks. To address the simile dialogue tasks, dialogue models need to understand the simile relations between entities/phrases/sentences. Our contributions are:
• To the best of our knowledge, we are the first to study the simile phenomenon in dialogue and propose a high-quality multi-lingual simile dialogue (MSD) dataset to assist both the simile and dialogue research.
• There are 5 tasks with the proposed MSD dataset. For simile research, we design the dialogue simile recognition/interpretation/generation tasks. For dialogue research, we design the response retrieval and generation tasks.
• We verify how strong pre-trained models and the state-of-the-art simile models perform on the 5 tasks we designed. Experimental results reveal that simile in dialogue is a difficult task and requires further study. Our code and data will be released on GitHub1.
1https://github.com/malongxuan/MSD

heading: 2 Related Work

heading: 2.1 Simile and Metaphor
text: The simile is a kind of metaphor that is frequently used in human languages to make utterances more vivid and graspable (Niculae and DanescuNiculescu-Mizil, 2014) and expresses human sentiments (Li et al., 2012; Mohammad et al., 2016). Previous researchers defined different metaphor categories. We present examples for these categories in the first four lines of Table 2. For example, Bizzoni and Lappin (2018) categorized metaphor into Noun phrases, Adjectives, Verbs, and Multi-word; Li et al. (2022) categorized metaphor into Nominal, Verbal (Subject-Verb-Object), Adjective-Noun, and Adverb-Verb. Previous work usually denoted the Noun phrase metaphor as a simile (Li et al., 2022; He et al., 2022; Chen et al., 2022). Following previous work, we also categorize Noun phrase metaphor as a simile. Meanwhile, we extend the tenor and vehicle to verbal phrases and sentences according to the simile phenomena in dialogue. The examples of verbal phrases and sentences in simile are shown in the last two lines of Table 2.

heading: 2.2 Tasks in Metaphor/Simile
text: The tasks in metaphor are also suitable for simile, such as recognition (Birke and Sarkar, 2006; Liu et al., 2018), interpretation (Su et al., 2016), and generation (Li et al., 2022). The recognition task is also called identification (Steen, 2010; Li et al., 2022) or detection (Tsvetkov et al., 2014; Mohler et al., 2016), which aims to identify whether a given phrase or sentence contains a metaphor/simile. The interpretation is also called explanation (Liu et al., 2018) which usually assigns an appropriate inter-
pretation to a metaphorical expression (Bizzoni and Lappin, 2018) or infers the shared properties of the tenor and the vehicle (Song et al., 2021; He et al., 2022; Chen et al., 2022). The generation task also has different forms. For example, when giving an input tenor, it can generate a simile sentence conditioned on the input tenor (Li et al., 2022); when giving both the tenor and the shared property in simile, it can generate the vehicle (Song et al., 2021; Chen et al., 2022); when providing a literal sentence, it can generate a metaphoric sentence which paraphrases that input (Chakrabarty et al., 2020; Stowe et al., 2021), or generating a specific simile according to the location where the simile interpolation should happen (Zhang et al., 2021). In this paper, we also define recognition, interpretation, and generation tasks. However, different from previous work that only focused on similes in a triplet tuple or a sentence, we investigate a more challenging scenario where the simile happens in a multi-turn dialogue.

heading: 2.3 Survey of Simile Datasets
text: Table 3 shows the comparison between our MSD dataset with the existing simile datasets. Su et al. (2016) constructed a small Chinese Metaphor (CM) data with 85 nominal and 35 verbal metaphors for the interpretation task. Liu et al. (2018) introduced Simile Recognition in Chinese (SRC) data containing sentences with a special comparator像 (like). The Chinese Nominal Metaphor Corpus (CMC) (Li et al., 2022) data merges other Chinese metaphor datasets (Liu et al., 2018) for simile generation. He et al. (2022) proposed a simile property probing task and constructed Multi-choice Probing (MCP) datasets. Chakrabarty et al. (2020) collected Reddit comments containing similes and then autoconstructed a parallel simile corpus with a pretrained model powered by commonsense knowledge (Bosselut et al., 2019). However, their Selflabeled Similes (SLS) dataset is limited to a “like a”
pattern which appears only at the end of a sentence. Zhang et al. (2021) introduced the Writing Polishment with Similes (WPS) dataset where models need to locate the simile position in a sentence and then generate a simile in that position. The SLS and WPS are much larger than other existing data but they are not manually annotated. Our MSD data is extracted from more than 166M dialogue data (shown in Table 4). It is the first multi-lingual simile dialogue data and the largest manually annotated simile data so far. What’s more, benefiting from the strict annotation schedule, the MSD contains necessary simile components so that it can be used for simile recognition/interpretation/generation simultaneously.

heading: 3 Multilingual Simile Dialogue Dataset
text: In this section, we introduce the collection, annotation, and statistics of our MSD data.

heading: 3.1 Data Collection
text: Since we aim to extract the simile in a real-life dialogue, we adopt the existing open-domain dialogue corpus collected from social platforms such as Reddit.com and Weibo.com. For English similes, we use the 3 turns version Reddit Dialogue dataset (Dziri et al., 2018) which contains more than 15 million dialogues. For Chinese similes, we use two datasets: PchatbotW and LCCC. The PchatbotW (Qian et al., 2021) is the largest dialogue data we can find and contains 139 million 2 turns dialogues from Weibo. The LCCC (Wang et al., 2020) is also from Weibo and contains 12 million 2 or 3 turns dialogues. We treat the last utterance in a dialogue as a response and the utterances in front of the response as a dialogue context. We extract dialogues from these large-scale datasets with a rigorous data collection pipeline, which is built based on a set of rules we will introduce in this section. Notice that we do not make any changes to the original dialogue data and only extract those dialogues with comparators in the response.
In the first step, we select the dialogue examples where the responses contain comparators such as
Data Annotation
非结构化文本增强的开放域对话（绪论） 对话理解 知识库 对话生成对话管理
回复 对话语料库 对话上文 检索未来
回复生成 筛选的知识是已 知的，可去掉 当前query 对方的上文 候选知识句 知识筛选的联想 和类比能力 候选回复 知识筛选的联想 和类比能力 自己的上文 筛选的知识是已 知的，可去掉
回复 对话语料库 对话上文 检索未来
回复 筛选的知识是已 知的，可去掉 当前query 对方的上文 候选知识句 知识筛选的联想 和类比能力 候选回复 知识筛选的联想 和类比能力 自己的上文 筛选的知识是已 知的，可去掉 回复 对话上文 检索未来 回复生成 候选知识句 知识筛选的联想 和类比能力 候选回复 知识筛选的联想 和类比能力 解决孤独？ 为什么结婚 为了爱情 传宗接代（不孝有 三，无后为大） 观念错误。 整个人类都是由最开始的几个猴 子演变过来的，所谓的传宗接代 也是在延续猴子祖宗的血脉。全 世界这么多人，不差你这一个。 结婚能幸福吗？为了幸福生活 观念错误。 总所周知，婚姻是爱情的坟墓。最好的婚 姻也只是从恋人变成朋友和家人。所以现 在大家都称呼配偶为“队友”。爱情只是 短暂的冲动，结婚不是因为爱情，结婚是 找到一起奋斗一起变老的队友。 为什么要找队 友？ 联想和类比能力 （高级应用） 对话理解指导KS （能筛选知识） 知识为中心的视角 （改善因稀疏性和 PTM导致的KS和 RG对知识利用不一 致问题）（同时做 KS和RG） DA指导知识注入 （能利用知识） （提升可解释性） 联想和类比能力 （高级应用） 对话理解指导KS （能筛选知识） （提升准确性） 知识为中心的视角 （改善因稀疏性和 PTM导致的KS和 RG对知识利用不一 致问题）（同时做 KS和RG） DA指导知识注入 （能利用知识） （提升可解释性） 非结构化文本增强的开放域 对话关键技术研究 对话上文： 你知道哪些碳元素构成的物质？ 非结构化文本增强 的对话系统 预期回复： 金刚石和石墨 幻觉回复： 木炭燃烧产生二氧化碳 非结构化文本知识： 金刚石和石墨都是碳元素构成的。
基于知识类比能力
的对话建模 （第五章）
基于对话意图理解 的知识筛选 （第二章）
基于知识为中心视
角的对话建模
（第四章）
基于对话动作指导 的知识筛选和注入 （第三章）
融合知识类比能力的对话建模方法（第五章）
基于对话冗余信息过滤的知识筛选
（第二章）
基于知识为中心视角的对话建模
（第四章）
基于显式信号指导的知识筛选和注入
（第三章）
非结构化文本增强的开放域对话
（第一章）
A Compare Aggregate Transformer for Understanding Document-grounded Dialogue. EMNLP (Findings) 2020（一作）
Knowledge-centric Response Selection for Document-grounded Dialogue. EMNLP 2022. Review scores: 4/1.5/3.5/3（一作）
Exploiting Dialogue Act for Document-grounded Dialogue. EMNLP 2022. Review scores: 4/3/2（一作）
Unstructured Text Enhanced Open-Domain Dialogue System: A Systematic Survey. ACM TOIS (2022)（一作）
2022年11月-2023年2月 任务自适应的知识注入方法（第三章补充）
Exploiting Side-information for Understanding Document-grounded Dialogue 拟10月份投稿 TOIS（一作）
基于知识类比能力的对话建模
（第5章）
基于对话意图理解的知识筛选
（第2章）
基于知识为中心视角的对话建模
（第4章）
基于对话动作指导的知识筛选和注入
（第3章）
知识稀疏性
知识利用的
可解释性
从语义级别到语用级别应用
对话冗余信息影响知识筛选准确率
知识筛选和知识注入不一致现象
知识筛选和注入缺少显式信号的指导
研究问题 面临挑战 研究内容
基于知识类比能力的对话建模
（第5章）
基于对话冗余信息过滤的知识筛选
（第2章）
基于知识为中心视角的对话建模
（第4章）
基于显式信号指导的知识筛选和注入
（第3章）
知识增强从语义到语用级别的扩展
知识稀疏条件下的知识筛选
知识筛选和知识注入不一致现象
知识筛选和知识注入的融合
研究问题 研究内容
语义级别
语用级别 非 结 构 化 文 本 增 强 的 开 放 域 对 话
撰写论文，参加预答辩和答辩2023年7-9月
2023年5-6月 完成论文全部章节工作
比喻 类比 类比 推理
子任务N
子任务 1
方法 1.1
输入 输出方法 2.1
问题
方法 3.1
...
子任务N子任务 1 方法 1 输入 输出
方法 2
问题
问题
方法 3
... 改进
改进
子任务 N.1
子任务1.1
方法 1
输入 输出 方法 2
问题
问题
方法 3
...
改进
改进
子任务1.x...
方法 1
方法 2
问题
问题
方法 3
改进
改进
子任务 N.x...
方法 1.2
方法 2.2
问题 改进
问题 改进
Simile Vehicle candidate is correct? Annotate Vehicle
Literal
Has sensitive words
Annotate LiteralDialogue example
Annotate Delete
Tenor exists? Annotate TenorYes
Dialogue Corpus
Length Control
Comparator Check
Semantic Check
Data Collection
Figure 1: The data collection and annotation process.
"像...一样"/"like"/"as...as"2. We only select dialogue examples with context lengths between 15 and 30 words so that the dialogue context is both informative and not too long for the annotators to read. These examples are denoted by the coarse version of the simile dialogue data and the statistics are shown in Table 4.
In the second step, we use machine translation3
to ensure that a sentence contains a comparator. We only reserve the dialogue examples that still contain comparator when they are translated into another language. For example, an English simile candidate sentence "I run as fast as a rabbit" contains a comparator "as...as". When translating it into Chinese, this sentence is "我跑得像兔子一样快" and still contains a comparator "像"(like). After the machine translation checking, we got the fine version of the simile dialogue candidates. The fine version needs further improvement since the candidate tenor/vehicle connected by the comparator is not always a simile. For example, the sentence "The Poodle is as tall as a Corgi" is not a simile since the sentence compares the height of two different kinds of dogs. So we conduct a third step to further remove examples that are not similes.
In the third step, we adopt a semantic dependency tool4 to locate the candidate tenor/vehicle, then we compute the similarity between them to retain the examples with low similarity so that the remaining candidate tenor/vehicle are from different categories. The similarity is computed with dense representations of the candidate tenor/vehicle from BERT (Devlin et al., 2019). After the above pipeline, we obtain the final version of simile dialogue data for annotation. The statistics of the fine/final version we obtained are also shown in Table 4.
2The full Chinese comparators are listed in Appendix 10 3We use https://ai.youdao.com/ to conduct the translation. 4https://stanfordnlp.github.io/CoreNLP

heading: 3.2 Data Annotation
text: We recruited 7 students majoring in English for annotating the English data and recruit other 6 welleducated native speakers (graduate students) for annotating the Chinese data. We randomly select 100 examples in the final version, finding that the vehicle candidates we extracted have an acceptable accuracy (above 80%). However, the accuracy of the tenor candidate is not good (below 60%). Hence, we provided annotators with "dialogue context", "response", "comparator", and "vehicle candidate" for each dialogue. We use the annotation tool proposed by Yang et al. (2018) to simplify the operation so that the annotators can use a mouse and a few shortcuts on the keyboard to annotate.
There are some difficulties when annotating similes in the dialogue scenario apart from the fact that the tenor may exist in different sentences or occur after the vehicle. For example, the tenor may not exist in the dialogue even if the response is a simile. We ask the annotators to delete these examples. There are other situations that a dialogue that contains commonly used phrases or slang that makes the dialogue seem like a simile but not. For instance, "make like a tree" is not a simile but slang means "leave". Besides, English words usually have different meanings. For example, according to the Oxford Dictionary, the word "body" means "the whole physical structure of a human or an animal" as well as "a group of people who work or act together, often for an official purpose". So the sentence "This association is like the body that represents its members." is not a simile. Furthermore, there are many abbreviations used on social platforms such as FTW (for the win) and OP (original poster). These difficult linguistic phenomena require the annotators to have a good understanding of the dialogue context so that they could determine whether a response contains a simile.
We conduct preliminary training for the recruited annotators so that they are aware of the professional
standards. We ask the annotators to first check whether the response in this dialogue example contains a simile. The example will be annotated "Literal" if the response is not a simile. Otherwise, they should check whether the vehicle candidate in the response is correct. They need to annotate the correct vehicle (can be word/phrase/sentence) if the candidate is not accurate. If the candidate vehicle is correct, they can annotate the tenor (can be word/phrase/sentence) if it exists. We present the annotation schedule in Figure 1. Our annotation schedule ensures that the tenor and vehicle are in the data.
Quality Evaluation. During the annotation, each time we send a small "*.txt" file containing hundreds of dialogue examples to the annotators and conduct a random sampling test after they return the annotated data5. The annotator who returns a low-quality file will be asked to check their annotation again before we send the next file. The whole annotation takes 35 days, and each dialogue is annotated by 3 annotators. When determining the final result, the majority will be adopted when there is a disagreement among the three annotators6. The overall inter-rater agreement measured by Fliess’ Kappa is 0.61, indicating a substantial agreement among the annotators.

heading: 3.3 Data Statistics
text: After the annotation, we get a total of 19,565 (8,146 English and 11,419 Chinese) dialogues. The MSD has multiple comparators for both English and Chinese data. In MSD English data, the "like" mode is around 52.4% and the "as" mode is around 47.6%. In MSD Chinese data, "像...一样" accounts for the
5During annotation, we randomly selected 5% of the examples from one annotated file and checked if the annotator made accurate annotations for these random examples. The annotators were preliminary trained so that they were expected to make as few errors as possible. We expected no more than 1 error per 20 examples in the random sampling test. Otherwise, the file will be sent back for revision.
6There are a few cases where the three annotators disagree with each other, we decide these cases by ourselves.
most7. The proportion of each comparator is similar in simile and literal data. Table 5 shows some of the statistics of the MSD data. Please refer to the data link for more details.

heading: 4 Tasks and Results
text: In this section, we introduce the 5 tasks defined with our MSD dataset. Including the definition of the task, the baselines, evaluation metrics, experimental results, and analysis. The implementation details are shown in the Appendix A.

heading: 4.1 Simile Recognition Task
text: Following previous work (Liu et al., 2018; Li et al., 2022), we define simile recognition as a binary classification task where the model needs to distinguish whether an input sequence contains a simile. The input is a multi-turn dialogue and the output is True (simile) or False (literal).

heading: 4.1.1 Baselines and Evaluation Metrics
text: We use two baselines: 1) BERT is widely used and proven to be effective in classification tasks. We randomly split our MSD-En/Ch data into train/validation/test (8:1:1) sets and use the train set to fine-tune BERT. We use the output vector of the first input token <cls> of BERT to calculate the classification score for the input dialogue (see Appendix A); 2) a large language model (ChatGLM8). The input to ChatGLM is a concatenation of three parts: the definition of simile "A simile is a figure of speech that compares two different things via their shared properties."; a requirement "answer yes or no to this question: is the following dialogue example contains a simile?"; a simile dialogue examples such as in Table 1. Then we calculate the results according to the prediction of the baselines. Following previous work (Liu et al., 2018), we use Precision/Recall/F1 to measure the results.
7There are total of 11 comparators in Chinese data. Please refer to Appendix B for more details.
8https://www.datalearner.com/ai-models/pretrainedmodels/ChatGLM-6B

heading: 4.1.2 Results and Analysis
text: Table 6 shows the simile recognition results. We can see that BERT(fine-tuned) performs much better on Precision and F1 than ChatGLM on both MSD-En and MSD-Ch9. It is reasonable since the BERT models are fine-tuned on our training set. On the other hand, the ChatGLM is much better on Recall with a zero-shot setting. Overall, the classification results on both BERT and ChatGLM still have a lot of room to improve. Using syntactic structure information to locate simile components may help this task.

heading: 4.2 Simile Interpretation/Generation Tasks
text: Following the previous simile interpretation task (Song et al., 2021; He et al., 2022) and simile generation task (Song et al., 2021), we define Simile Interpretation/Generation (SI/SG) as a Multi-choice task with the "as...as" mode in our MSD-En10 data (we test with 450 examples) since the shared property naturally exists in the comparator.
For interpretation task, we have a simile dialogue where the shared property between two "as"s is removed and replaced with a blank. The model needs to select a property from 4 choices (one correct answer and three distractors) for the blank. We construct the distractors with ConceptNet (Speer et al., 2017). In particular, we first use the tenor and some relations to find the related concept to the tenor and then use the HasProperty relation to find the distractors. Notice that for the examples where the tenor is a phrase of a sentence we could not find in ConceptNet, we use keywords (e.g. the subject of the sentence, the noun in the phrase) as the tenor to search ConceptNet.
Similar to the simile interpretation task, we remove the vehicle in a simile dialogue and leave a blank for the simile generation task. The model needs to select a proper vehicle for this blank from 4 candidates (one correct answer and three distractors). We also construct the distractors with ConceptNet. We use the vehicle and certain relations in the ConceptNet to find the related concepts to the vehicle as the distractors. Notice that for the examples where the vehicle is a phrase or sentence that we could not find in ConceptNet, we use the vehicles from other dialogues in MSD dataset as
9For Chinese, we use https://huggingface.co/bert-basechinese
10We did not conduct simile interpretation/generation on MSD-Ch in this paper since we did not annotate the shared property in Chinese data and we leave it for future work.
the distractors. To ensure the distractors are true-negative, we randomly select 50 dialogue examples and manually check the quality of the distractors. We find that 92% of the distractors are well selected and the rest 8% are not as ideal as we expected but can still serve as distractors. More details about using ConceptNets are shown in Appendix C.

heading: 4.2.1 Baselines and Evaluation Metrics
text: The first baseline is a BERT-large model which takes the whole dialogue with the shared property or the vehicle masked and predicts the masked words. The second baseline is the BERT-Probe (He et al., 2022) that fine-tunes BERT with the simile interpretation task. To compare both SI and SG tasks with this baseline. We further finetune the BERT-Probe model with the SG task using the data proposed by He et al. (2022). The third baseline is BERT-ANT (Chen et al., 2022) which is trained with masked word prediction with metaphor data and can solve the Simile Interpretation and Generation tasks in a unified framework of simile triple completion. For example, when giving tenor=fireman and vehicle=bull, BERT-ANT can generate a list of words including the shared property like "strong" or "brave". All baselines are based on a BERT-large-uncased model. Since there are multiple masked words in our SI/SG experiments. We encode the predicted words and the candidates into dense vectors with a sentence-transformer (huggingface.co/sentencetransformers/all-MiniLM-L6-v2). Then we compute the cosine similarity between the predicted words and each of the candidates. The candidate with the highest similarity is chosen as the answer. We use Hit@1 to measure the accuracy.

heading: 4.2.2 Results and Analysis
text: Table 7 shows the results of simile interpretation/generation tasks. We can see that BERT-Probe performs better than BERT-large in this task, showing that a model pre-trained on simile data can better align the simile components in an input sequence and predict the missing component, even though the training data is much different from our
proposed data. The BERT-ANT performs similarly to the other two models on SG tasks but not as well at SI. It is because the training data of BERTANT is more of a metaphor data rather than simile data, a large portion of the metaphor data does not have shared properties. Hence, BERT-ANT is more powerful in connecting tenor and vehicle but is less powerful when predicting shared properties. Overall, the results on both simile interpretations/generations still have a lot of room to improve. How to exploit the semantic information in context to help these tasks requires further study.

heading: 4.3 Response Retrieval Task
text: Following previous work in retrieval (Guo et al., 2016), we define Response Retrieval as a ranking task. The input is a multi-turn dialogue context and multiple response candidates (including the correct one) and the model needs to rank all the candidates so that the correct one has the highest score. In particular, for each "dialogue context" in MSD simile data (both English and Chinese), we randomly select 19 responses from other dialogue as the negative examples.

heading: 4.3.1 Baselines and Evaluation Metrics
text: We use BERT-base for our baseline in response retrieval since it is widely used and proven to be effective in retrieval tasks. We concatenate dialogue context and each of the response candidates as the input sequence to the pre-trained model. Then we use the output of the first input token <cls> to compute the score for the input sequence as in Appendix A. Finally, the response candidate with the highest score will be chosen as the answer.
We first randomly split the Reddit dialogue data into train/validation/test (14.99M/5K/5K) sets. Then we used the BERT model to train an English dialogue retrieval model with this train/validation data. The model is denoted by BERT(Reddit). We choose a checkpoint with the best performance on the validation set. Then we use this checkpoint to compare its performance on both the Reddit Test set and the MSD-En set. Similarly, we combine LCCC and PchatbotW and randomly select 12M/5K/5K from the combined data as train/validation/test sets and train a Chinese dialogue retrieval model. The trained BERT11 model is denoted by BERT(Ch) and used to do the comparison of the performance on the LCCC+PchatbotW
11https://huggingface.co/bert-base-chinese
Test set and the MSD-Ch set. We measure the accuracy of the retrieval with Recall@1/2/5.

heading: 4.3.2 Results and Analysis
text: Table 8 shows the results of the response retrieval task. The performance of BERT(Reddit) and BERT(LCCC) on MSD is lower than their performance on Reddit and LCCC+PchatbotW Test sets, respectively. The results show that the data distribution in MSD is different from the data used to extract it and selecting a simile response is much harder than selecting a proper response. The low Recall results show that the dialogue retrieval task on MSD simile data needs further study. This requires a model that judges not only the relevance between context and response but also the plausibility of similes.

heading: 4.4 Response Generation Task
text: The traditional response generation task uses dialogue context as input and outputs the response of the context. In this section, we also introduce a new generation task that completes the response sentence behind the comparator. Taking the fifth simile dialogue "Arguing with parents is not wise. It is like throwing an egg at a rock." as an example, we give the model "Arguing with parents is not wise. It is like" as input and ask the model to generate the rest "throwing an egg at a rock.". This is different from the Writing Polishment with Similes Zhang et al. (2021) task since our task is a dialogue scene. The model needs to understand the difference between different speakers and complete the simile sentence. We use the simile data in MSD for the generation experiments. We conduct comparative experiments on the Reddit-dialogue Test set and the LCCC+PchatbotW Test set we used in the response retrieval task to show the difference between datasets.

heading: 4.4.1 Baselines and Evaluation Metrics
text: For the traditional response generation task, we use the DialoGPT (Zhang et al., 2020) and GODEL
Model PPL BLEU(1/2/3/4)(%) ROUGE(1/2/L)(%) METEOR(%) Distinct(1/2)(%) Reddit-dialogue Test set (En) DialoGPT 236.74 0.01 / 0.00 / 0.00 / 0.00 2.05 / 0.00 /1.79 1.24 6.67 / 23.84 GODEL 3.70 0.53 / 0.02 / 0.00 / 0.00 2.80 / 0.00 / 1.98 2.41 6.54 / 36.01 MSD-En (simile data) DialoGPT 329.55 11.29 / 3.58 / 1.45 / 0.70 7.53 / 0.57 / 6.39 8.48 8.39 / 28.16 GODEL 6.10 17.10 / 5.99 / 2.61 / 1.37 10.91 / 0.87 / 8.94 11.78 7.00 / 23.37 MSD-En (simile data) on Response Completion DialoGPT - 17.29 / 8.50 / 5.24 / 3.35 23.71 / 5.13 / 23.04 12.85 14.64 / 43.51 LCCC+PchatbotW Test set (Ch) CDialGPT(Ch) 102.00 3.01 / 0.64 / 0.16 / 0.05 5.42 / 0.21 / 4.77 2.24 11.10 / 40.41 GPT-2(Ch) 129.28 5.20 / 1.50 / 0.59 / 0.26 7.09 / 0.87 / 6.14 3.04 23.23 / 66.14 MSD-Ch (simile data) CDialGPT(Ch) 113.75 3.07 / 0.72 / 0.26 / 0.09 5.46 / 0.24 / 4.85 2.30 11.36 / 40.58 GPT-2(Ch) 101.24 5.89 / 1.11 / 0.27 / 0.10 6.35 / 0.19 / 5.47 2.98 12.15 / 48.18 T5-base(Ch) 118.60 7.61 / 2.57 / 1.40 / 0.94 8.66 / 0.94 / 7.66 4.25 22.15 / 66.59 BART-large(Ch) 44.28 10.16 / 3.34 / 1.64 / 1.00 11.13 / 1.09 / 8.82 6.56 15.26 / 51.91
Table 9: Dialogue generation and completion results.
(Peng et al., 2022) for English data; use T5-base12, BART-large13 (Lewis et al., 2020), GPT-214 (Radford et al., 2019), and CDialGPT15 (Wang et al., 2020) for Chinese data. We choose these baselines since 1) they are widely used and proven to be effective in dialogue generation tasks. For example, GODEL (Grounded Open Dialogue Language Model) is pre-trained for dialogue and is initiated from T5 (Raffel et al., 2020). CDialGPT and BARTlarge are pre-trained with LCCC-large; 2) the different size models can provide more insight into the experiments. For our proposed response generation (completion) task, we conduct the experiment on English data with DialoGPT.
We use the following automatic evaluation metrics employed by dialogue research. Perplexity (PPL), BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Lavie and Agarwal, 2007), and Distinct (Li et al., 2016). PPL measures the probability of the model predicting the real response. BLEU measures the n-gram overlap between the generated response and the reference one. ROUGE is based on the calculation of the recall rate of the common sub-sequence of generating response and the real one. METEOR further considers the alignment between the generated and the real responses to improve BLEU. Distinct measures the diversity of responses by calculating the proportion of distinct n-grams in the total number of n-grams. Higher BLEU/ROUGE/METEOR/Distinct means better performance. The PPL is provided for comparing models with the same vocabularies, and the results are also useful for future research.
12huggingface.co/shibing624/prompt-t5-base-chinese 13huggingface.co/HIT-TMG/dialogue-bart-large-chinese 14huggingface.co/shibing624/gpt2-dialogbot-base-chinese 15huggingface.co/thu-coai/CDial-GPT_LCCC-large

heading: 4.4.2 Results and Analysis
text: Table 9 shows the generation and completion results. On most metrics of English data, DialoGPT and GODEL perform better on MSD-En than on Reddit-dialogue. CDialoGPT and GPT-2 have comparable performance on the LCCC+PchatbotW Test set and MSD-Ch. This is different from the response retrieval tasks where the MSD data is more difficult than the original data used to extract MSD. The reason may be the dialogue context in MSD provides more information than the context in the original data, so the generation models could leverage the rich context information to construct an informative response. Experiments also verify that larger models (GODEL/T5/BART) have a better performance. However, even the performance of the best baseline can still be improved. We analyze the generation results. Although there are some interesting cases, most of the results are not similes. It means the simile dialogue generation task requires a specific model design to capture the simile relations in context. We provide a case study in Appendix D.
For the response completion task, when giving the comparator, DialoGPT has a big performance gain. It proves that the simile generation can benefit from the guide. Please refer to our code/data link for more experimental results about this simile dialogue completion task.

heading: 5 Conclusion
text: We propose manually annotated multilingual simile dialogue (MSD) data for both simile and dialogue research. We design 3 simile tasks (recognition, interpretation, and generation) and 2 dialogue tasks (retrieval and generation) with MSD. Experiments
with strong baselines show the challenge of each task. Future works include but are not limited to 1) Dataset enlargement (e.g., more annotated examples with more kinds of comparators); 2) Model designing (e.g., models with a specific structure to address the proposed tasks); 3) New task designing (e.g., detecting tenor in the coarse/fine data). We encourage using the MSD in future simile and dialogue research.
Limitations
Due to time constraints, we were unable to implement some unreleased models as baselines for the proposed tasks. We did not conduct simile interpretation/generation on MSD-Ch in this paper since we could not automatically annotate the shared property in Chinese data like the "as...as" mode in English. We are currently working on this annotation and plan to release the Chinese simile interpretation/generation results on the data link. The coarse/fine version data we introduced in this paper can still be used for enlarging the MSD data. We will study to utilize them for more simile data and richer language phenomena.
Ethics Statement
We provide and emphasize some details of our work to address potential ethical concerns. First, all the data sources used in the data collection process are publicly available. We did not make any changes to the data sources and only extracted dialogue examples from these data. We carried out strict quality control during the extraction and annotation process. We made sure that there are no sensitive words even though the original data sources have already conducted this kind of checking. However, using our data to train or fine-tune a pre-trained generation model may still generate semantic errors or unpleasant similes or responses. One reason is that simile is a difficult task that compares two different things, mistakes could happen even when humans use similes. The other reason is that the knowledge stored in the original parameters of the pre-trained models may dominate the generation. We protect the privacy rights of annotators and paid 0.55 Chinese Yuan for annotating each dialogue data. The income of each annotator was above 100 Chinese Yuan per hour (On January 20, 2023, 100 yuan can be converted into 14.73 dollars).

heading: Acknowledgements
text: This paper is supported by the Science and Technology Innovation 2030 Major Project of China (No. 2021ZD0113302) and the National Natural Science Foundation of China (No. 62076081, No. 61772153, and No. 61936010).

heading: B Statistic Appendix
text: In Table 10, we present all the comparators and their proportions in MSD-Chinese.

heading: C ConceptNet Appendix
text: We use ConceptNet to construct the distractors in simile interpretation and generation tasks. ConceptNet is a knowledge graph that connects words and phrases of natural language with labeled edges (Speer et al., 2017). Two concepts (A and B) are connected with relations such as "IsA" or "PartOf". In the simile interpretation task, we need to find three distractors for the shared property of tenor and vehicle. We use the "Antonym" relation to extract the antonym of the property as the first distractor. We adopt the nine relations in Table 11 to find the related concepts to the tenor and then use the "HasProperty" relation to find the properties of these related concepts. Finally, we randomly
choose two of the nine properties as the distractors. For example, if the (tenor, property, vehicle) is (fireman, strong, bull). We first have "weak" as the Antonym and the first distractor of "strong". Then we find that "fireman" is related to (RelateTo) "fire" and "fire" has a property (HasProperty) "hot". So "hot" is the second distractor for "strong". We can get up to nine distractors and choose two of them along with the Antonym as the final three distractors. Notice that for the examples where the tenor is a phrase or sentence that we could not find in ConceptNet, we use keywords (e.g. the subject of the sentence, the noun in the phrase) as the tenor to do the search in ConceptNet. In the simile generation task, we also use the relations in Table 11 to find the related concepts to the vehicle as the distractors. Different from the interpretation task, for the examples where the vehicle is a phrase or sentence that we could not find in ConceptNet, we use the vehicles from other dialogues in MSD dataset as the distractors.

heading: D Generation Cases Appendix
text: As we introduced in the simile response generation section, most of the generated results are not similes since the baselines are not designed for this task.
In Table 12, we provide two cases to show the dialogue cases in MSD and the generation results from different models. In the first English example, both DialoGPT and GODEL generate fluent responses and contain the comparator "like" or "as". However, both models fail to generate a simile response like the ground truth one. The Chinese example is extracted from the LCCC data, we can see BARTlarge performs the best and gives an informative response with a simile in it. The GPT-2 gives a general response and T5-base gives an informative response. The CDialGPT also gives a general response even if it is trained with the LCCC dataset. The two cases in Table 12 further verify that simile dialogue generation is challenging. However, in the response completion task, when adding the comparator in the input, we can see the DialoGPT outputs a simile and makes the dialogue more vivid and interesting.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: Future works include but are not limited to 1) Dataset enlargement (e. g. , more annotated examples with more kinds of comparators); 2) Model designing (e. g. , models with a specific structure to address the proposed tasks); 3) New task designing (e. g. , detecting tenor in the coarse/fine data). We encourage using the MSD in future simile and dialogue research. Limitations
Due to time constraints, we were unable to implement some unreleased models as baselines for the proposed tasks. We did not conduct simile interpretation/generation on MSD-Ch in this paper since we could not automatically annotate the shared property in Chinese data like the "as. as" mode in English. We are currently working on this annotation and plan to release the Chinese simile interpretation/generation results on the data link. The coarse/fine version data we introduced in this paper can still be used for enlarging the MSD data. We will study to utilize them for more simile data and richer language phenomena