HAUSER: Towards Holistic and Automatic Evaluation of Simile Generation
Year: 2023

Abstract:
Similes play an imperative role in creative writing such as story and dialogue generation. Proper evaluation metrics are like a beacon guiding the research of simile generation (SG). However, it remains under-explored as to what criteria should be considered, how to quantify each criterion into metrics, and whether the metrics are effective for comprehensive, efficient, and reliable SG evaluation. To address the issues, we establish HAUSER, a holistic and automatic evaluation system for the SG task, which consists of five criteria from three perspectives and automatic metrics for each criterion. Through extensive experiments, we verify that our metrics are significantly more correlated with human ratings from each perspective compared with prior automatic metrics. Resources of HAUSER are publicly available at https://github.com/Abbey4799/HAUSER.

heading: 1 Introduction
text: Similes play a vital role in human expression, making literal sentences imaginative and graspable. For example, Robert Burns famously wrote “My Luve is like a red, red rose” to metaphorically depict the beloved as being beautiful. In this simile, “Luve” (a.k.a. topic) is compared with “red rose” (a.k.a. vehicle) via the implicit property “beautiful” and the event “is”. Here, topic, vehicle, property, and event are four main simile components (Hanks, 2013). As a figure of speech, similes have been widely used in literature and conversations (Zheng et al., 2019; Chakrabarty et al., 2022).
Simile generation (SG) is a crucial task in natural language processing (Chakrabarty et al., 2020; Zhang et al., 2021; Lai and Nissim, 2022), with the aim of polishing literal sentences into similes. In Fig. 1, the literal sentence “He yelps and howls.” is polished into a simile by inserting the phrase “like a wolf ”, resulting in “He yelps and
∗Corresponding author.
Literal Sentence: He yelps and howls . GroundTruth: He yelps and howls like a wolf.

heading: 1. He yelps and howls like a wounded buffalo.

heading: 2. He yelps and howls like a chicken.
text: 3. He yelps and howls like hell.
Generated Simile Candidates
… Human
BLEU HAUSER Quality Top1 : #1 Creativity Top1: #1 Informativeness Top1 : #1 Quality Top1 : #2 ::
what else criteria should I use…
#1 The most high-quality and is
Figure 1: An example of Simile Generation (SG) Evaluation. The commonly used automatic metric BLEU deems the second candidate as the most high-quality one among all the generated similes, while our proposed metrics HAUSER deem the first candidate as the best one regarding its quality, creativity and informativeness, which better correlates with human ratings and also provides more criteria for SG evaluation.
howls like a wolf ”. The ability to generate similes can assist various downstream tasks, such as making the generations more imaginative in story or poet generation task (Tartakovsky and Shen, 2018; Chakrabarty et al., 2022) and the generated response more human-like in dialogue generation task (Zheng et al., 2019).
Automatic evaluation is critical for the SG task since it enables efficient, systematic, and scalable comparisons between models in general (Celikyilmaz et al., 2020). However, existing studies are inadequate for effective SG evaluation. Task-agnostic automatic metrics (Papineni et al., 2002; Zhang et al., 2019; Li et al., 2016) are widely adopted for SG evaluation (Zhang et al., 2021; Lai and Nissim, 2022), which have several limitations: (1) The simile components should receive more attention than other words during SG evaluation (e.g. “he” and “wolf ” in Fig. 1), while there are no automatic metrics that consider the key components. (2) The SG task is open-ended, allowing for multiple plausible generations for the same input (Chakrabarty et al., 2020) (e.g. the howling man can be compared to “wolf ”, “buffalo”, or “tiger” in Fig. 1). Hence, the metrics based on word overlap with a few references are inadequate to accurately mea-
12557
sure the overall quality of generated similes. As shown in Fig. 1, the commonly used metric BLEU deems the second candidate as the highest quality, as it has more overlapped words with the only referenced groundtruth, while human deems the first candidate as the most coherent one. (3) The existing metrics are inadequate to provide fine-grained and comprehensive SG evaluation, considering that the creative generation tasks have distinct criteria for desired generations (Celikyilmaz et al., 2020), such as novelty and complexity for story generation (Chhun et al., 2022) and logical consistency for dialogue generation (Pang et al., 2020).
However, establishing a comprehensive, efficient, and reliable evaluation system for SG is nontrivial, which raises three main concerns: (1) What criteria should be adopted to evaluate the SG task in a comprehensive and non-redundant fashion? (2) How to quantify each criterion into a metric thus enabling efficient and objective SG evaluation, given that the human evaluation of creative generation task is not only time-consuming but also subjective and blurred (Niculae and Danescu-Niculescu-Mizil, 2014; Celikyilmaz et al., 2020)? (3) Whether the proposed metrics are effective in providing useful scores to guide actual improvements in the realworld application of the SG model?
In this paper, we establish HAUSER, a Holistic and AUtomatic evaluation system for Simile gEneRation task, consisting of five criteria (Tab. 1): (1) The relevance between topic and vehicle, as the foundation of a simile is to compare the two via their shared properties (Paul, 1970). (2) The logical consistency between the literal sentence and generated simile, since the aim of SG task is to polish the original sentence without altering its semantics (Tversky, 1977). (3) The sentiment consistency between the literal sentence and generated simile,
since similes generally transmit certain sentiment polarity (Qadir et al., 2015). (4,5) The creativity and informativeness of the simile, since novel similes or those with richer content can enhance the literary experience (Jones and Estes, 2006; Roncero and de Almeida, 2015; Addison, 2001). Overall, these five criteria can be categorized into three perspectives: quality (which considers relevance, logical, and sentiment consistency jointly), creativity, and informativeness. We further quantify each criterion into automatic metrics (Fig. 2) and prove their effectiveness through extensive experiments.
To the best of our knowledge, we are the first to systematically investigate the automatic evaluation of the SG task. To summarize, our contributions are mainly three-fold: (1) We establish a holistic and automatic evaluation system for the SG task, consisting of five criteria based on linguistic theories, facilitating both human and automatic evaluation of this task. (2) We design automatic metrics for each criterion, facilitating efficient and objective comparisons between SG models. (3) We conduct extensive experiments to verify that our metrics are significantly more correlated with human ratings than prior metrics.

heading: 2 Related Work

heading: 2.1 Simile Generation Task
text: There are two primary forms of the simile generation (SG) task: simile triplet completion and literal sentence polishing. For simile triplet completion, a model receives simile components, topic and property, and is required to generate the vehicle (Roncero and de Almeida, 2015; Zheng et al., 2019; Chen et al., 2022; He et al., 2022). For literal sentence polishing, a model receives a literal sentence and is expected to convert it into similes (Zhang
et al., 2021; Stowe et al., 2020; Chakrabarty et al., 2020; Lai and Nissim, 2022). We focus on the latter. However, prior works mainly adopt task-agnostic automatic metrics to evaluate the SG task, raising concern as to whether the claimed improvements are comprehensive and reliable.

heading: 2.2 Automatic Evaluation for NLG Systems
text: Existing automatic metrics for Natural Language Generation (NLG) evaluation can be categorized into task-agnostic and task-specific metrics. Taskagnostic metrics can be applied to various NLG tasks, which generally focus on the coherence of generations (Papineni et al., 2002; Zhang et al., 2019), including n-gram-based metrics (Papineni et al., 2002; Lin, 2004; Denkowski and Lavie, 2014) and embedding-based metrics (Zhang et al., 2019; Zhao et al., 2019). There are also many metrics for evaluating the diversity of generations (Li et al., 2016; Zhu et al., 2018; Tevet and Berant, 2021). Task-specific metrics are proposed to evaluate NLG systems on specific tasks (Tao et al., 2018; Dhingra et al., 2019; Ren et al., 2020). Specifically, various works systematically study the evaluation of the creative generation task (Pang et al., 2020; Tevet and Berant, 2021; Chhun et al., 2022). Different from these works, we revisit SG evaluation, propose holistic criteria based on linguistic theories, and design effective automatic metrics for it.
3 HAUSER for SG evaluation
We establish HAUSER, a holistic and automatic evaluation system for SG evaluation, containing five criteria from three perspectives, and further design automatic metrics for each criterion (Fig. 2).

heading: 3.1 Quality
text: We measure the overall quality of generated similes using three criteria: relevance, logical consistency, sentiment consistency. The key simile components - topic and vehicle - should be relevant, as the foundation of a simile is to compare the two via their shared properties (relevance) (Paul, 1970). In Tab. 1, comparing “raindrops” to “tears” is more coherent than to “arrows”. Additionally, the generated simile should remain logically consistent with the original sentence (logical consistency), as the SG task aims to polish the plain text without changing its semantics (Tversky, 1977). In Tab. 1, comparing “Stefan” to “dancer” better depicts his controlled and easy movement than to “lightning”. Furthermore, as similes generally transmit certain sentiment polarity (Qadir et al., 2015), the generated simile should enhance the sentiment polarity of the original sentence (sentiment consistency). In Tab. 1, the vehicle “thunderous wave” enhances the positive polarity of the original sentence, while the vehicle “earthquake” brings a negative sentiment polarity in opposition to the original sentence.

heading: 3.1.1 Relevance
text: For the relevance score, if the components of one simile are relevant, they tend to co-occur in simile sentences (Xiao et al., 2016; He et al., 2022) and possess shared properties (Paul, 1970; Tversky, 1977). Hence, obtaining the relevance score requires large-scale simile sentences as references, as well as knowledge about the properties (adjectives) of each simile component. For a simile s, the relevance score is defined as follows:
r = 1
mp
∑
(t,v)∈s
∑
e∈Γ(t,v) Pe(t, v), (1)
where there are mp topic-vehicle pairs extracted from simile s, each denoted as (t, v)1. Γ(t, v) is the set of similes containing (t, v) as simile components, each denoted as e. Pe(t, v) is the probability that the simile components (t, v) share properties in the context of the simile sentence e.
An effective way to obtain the frequency information Γ(t, v) and property knowledge Pe(t, v) is to utilize the large-scale probabilistic simile knowledge base MAPS-KB (He et al., 2022), which contains millions of simile triplets in the form of (topic, property, vehicle), along with frequency and two probabilistic metrics to model each triplet2. Specifically, the probabilistic metric Plausibility is calculated based on the confidence score of the simile instance (topic, property, vehicle, simile sentence) supporting the triplet, indicating the probability that the topic and vehicle share the property. The relevance score r can be calculated as follow:
r = 1
mp
∑
(t,v)∈s
∑
(t,p,v)∈G(t,v)
n(t, p, v) · P(t, p, v), (2)
where G(t,v) is the set of triplets (t, p ,v) containing the (t, v) pair in MAPS-KB, with p referring to the property. n and P are the metrics provided by MAPS-KB, where n and P denote the frequency and the plausibility of the triplet respectively.
It is noticed that the metric is not coupled with MAPS-KB, as the frequency information can be obtained by referencing a large set of simile sentences and the property knowledge can be contained via other knowledge bases. More methods are beyond the scope of this paper. However, we additionally provide a method to approximate the relevance score. If we assume the probability that the simile components (t, v) share properties in each sentence is 1, the relevance score can be approximated as:
r ≈ 1 mp
∑
(t,v)∈s n(t, v), (3)
where n(t, v) denotes the number of samples that contain the simile components (t, v) in large-scale simile sentences. We discuss the effects of the referenced dataset size in Sec. 4.2.1.

heading: 3.1.2 Logical Consistency
text: The literal sentence and the generated simile that are logically inconsistent generally exhibit contra-
1All the simile components in our work are extracted and cleaned using rules from (He et al., 2022) which determines the optimal semantics a component should carry, e.g., “a kid in a candy store” instead of just “a kid”.
2More details of MAPS-KB is provided in Appx. D
dictory logic. Hence, for a generated simile, we input the <literal text(l), simile(s)> sentence pair into existing pre-trained Multi-Genre Natural Language Inference (MNLI) model3, which determines the relation between them is entailment, neutral, or contradiction. The logical consistency score cl of this simile is defined as follows (Pang et al., 2020):
cl = 1− P (h<l,s> = c), (4)
where P (h<l,s> = c) represents the probability that the model predicts the relation of the sentence pair < l, s > to be contradiction (denoted as c).

heading: 3.1.3 Sentiment Consistency
text: Better similes tend to enhance the sentiment polarity of the original sentence (Qadir et al., 2015). Hence, we first apply the model fine-tuned on the GLUE SST-2 dataset4 to classify each simile as being either positive or negative. Then, the sentiment consistency score cs is defined as follows:
cs = P (hs = a)− P (hl = a), (5)
where a is the sentiment polarity of the literal sentence (positive or negative) predicted by the model. P (hs = a) and P (hl = a) denote the probabilities that the model predicts the sentiment polarity of the simile s and the literal sentence l to be a, respectively.
It is noticed that different <topic, vehicle> pairs within a sentence may have distinct sentiment polarities, such as <She, scared rabbit> and <I, bird> in the simile “If she escapes like a scared rabbit, I will fly like a bird to catch her.”. Directly inputting text containing multiple topic-vehicle pairs into the sentiment classification model will result in inferior performance. Therefore, for each simile, only the text from the beginning up to the first vehicle is input into the model (i.e. “If she escapes like a scared rabbit” in the given example), and for each literal sentence, the text from the beginning up to the first event (i.e. “If she escapes” in the given example) is input into the model.
3We use the checkpoint of the model (robertabase_mnli_bc) that achieves the SOTA performance on the GLUE (Wang et al., 2018) MNLI dataset at the time of submission, according to https://paperswithcode.com/sota/textclassification-on-glue-mnli.
4We apply the checkpoint of the model (distilbert-baseuncased-finetuned-sst-2-english) with the most download times on the GLUE SST-2 dataset at the time of submission, according to https://huggingface.co/models.

heading: 3.1.4 Combination
text: Since the aim of the SG task is to polish the plain text, the quality of similes generated from different texts can not be compared. Therefore, the normalized score among the simile candidates for each original text is utilized. Suppose there are m simile candidates S = {s1, s2, ..., sm} for the literal text l, the original relevance scores of R is R = {r1, r2, ..., rm} respectively. The normalized relevance score r′i of si is formulated as follows:
r′i = ri −min(R)
max(R)−min(R) , (6)
which ranges from 0 to 1. Then, the normalized logical and sentiment consistency score c′li, c ′ si for each simile si are obtained in the same manner5. Finally, the quality for simile si is defined as the weighted combination of three parts as follows:
Qi = α · r′i + β · c′li + γ · c′si, (7)
where α, β, and γ are hyperparameters.

heading: 3.2 Creativity
text: Creative similes can provide a better literary experience (Jones and Estes, 2006). In Tab. 1, comparing “sarcasm” to “vitriol” is less common than to “fire”, yet it better conveys the intensity of a person’s sarcasm. Hence, we design creativity score.
Previous studies mainly evaluate the creativity of text generation tasks via human evaluation (Sai et al., 2022), since measuring the creativity of openended text is a relatively difficult task (Celikyilmaz et al., 2020). Although there have been many works evaluating the diversity of open-ended text generation (Li et al., 2016; Zhu et al., 2018; Tevet and Berant, 2021), these metrics are not suitable for measuring the creativity of the text. Because the diversity metrics take a set of generated text as input and output one score, while a creativity metric is required to measure each text individually and output a set of corresponding scores.
Different from other open-ended generation tasks, the components of the generated similes enable us to evaluate creativity automatically. According to linguists, the creativity of a simile is determined by vehicles (Pierce and Chiappe, 2008; Roncero and de Almeida, 2015). Intuitively, the generated simile may be less creative if its extracted
5If all the relevance scores ri in R are the same, the normalized relevance scores r′i in R′ are set to 0.5 uniformly.
topic-vehicle pair co-occurs frequently, or if many topics are compared to its vehicle in the corpus. Therefore, we adopt large-scale corpora as references when designing our creativity metric. The creativity score of s is calculated as follows:
Ci = −log( 1 mv
∑ v∈s Nv + 1), (8)
where there are mv vehicles extracted from the simile s, each denoted as v. Nv denotes the frequency of the vehicles appearing in the similes in the corpora. The log transformation aims to reduce the influence of extreme values.
An effective way to obtain the adequate frequency information Nv is to utilize the millionscale simile knowledge base MAPS-KB, where the Nv can be defined as follows:
Nv = ∑
(t,p,v)∈Gv
n(t, p, v), (9)
Gv is the set of triplets containing the vehicle v in MAPS-KB, n denotes the frequency of the triplet.
It is noticed that the metric is not coupled with MAPS-KB, as Nv can also be obtained by counting the samples containing the vehicle v in largescale simile sentences. The method of obtaining the simile sentences is beyond the scope of this paper. Nevertheless, we discuss the effects of the referenced dataset size in Sec. 4.2.2.

heading: 3.3 Informativeness
text: The vehicle with richer content can create a more impact and vivid impression(Addison, 2001). In the example from Tab. 1, the addition of the word “angry” makes the similes more expressive. Therefore, we design the metric informativeness to measure the content richness of the vehicles.
Intuitively, the more words a vehicle contains, the richer its content will be. Hence, for a given simile s, we adopt the average length of the extracted vehicles to be the informativeness score6 (Chakrabarty et al., 2020; Zhang et al., 2021), defined as Ii = 1mv ∑ v∈s len(v), where there are mv vehicles extracted from simile s.
4 HAUSER Analysis
In this section, we conduct experiments to verify the effectiveness of our automatic metrics.
6Different from the quality metric, we do not use a normalized score for creativity and informativeness, since they mainly depend on the generated vehicles, rather than the original text.

heading: 4.1 Experiment Setup

heading: 4.1.1 Simile Generation
text: The existing datasets for the SG task are either Chinese (Zhang et al., 2021), limited to the simile triplet completion (Roncero and de Almeida, 2015; Chen et al., 2022), or having all vehicles located at end of the sentence (Chakrabarty et al., 2022; Lai and Nissim, 2022), which are not practical for English simile generation in a real-world application. To bridge the gap, we construct a large-scale English dataset for SG task based on simile sentences from (He et al., 2022), which contains 524k simile sentences labeled with topic and vehicle. The output decoder target is the simile sentence s and the input encoder source is s rewritten to drop the comparator “like” and the vehicle. For example, given s = “The idea resounded like a thunderclap throughout the land.”, the encoder source would be “The idea resounded throughout the land.”. In particular, we remove the simile sentences whose event is a linking verb (e.g. be, seem, turn) as they would be meaningless after the vehicle is removed. The final train, validation and test sets contain 139k, 2.5k, and 2.5k sentence pairs, respectively.
Based on our constructed dataset, we finetune a pre-trained sequence-to-sequence model, BART (Lewis et al., 2020), for the SG task, which has been demonstrated to be an effective framework for various figurative language generation (Zhang and Wan, 2021; Chakrabarty et al., 2022; He et al., 2022; Lai and Nissim, 2022). The experiments are run on RTX3090 GPU and the implementation of BART is based on the HuggingFace Transformers7. The experiments are run with a batch size of 16, a max sequence length of 128, and a learning rate of 4e-5 for 10 epochs.

heading: 4.1.2 Evaluation Dataset Construction
text: Firstly, we randomly sample 50 literal sentences from the test set and adopt the trained SG model to generate five candidates for each one. Then, for each perspective, three raters are asked to rate each
7https://github.com/huggingface/transformers/
simile from 1 to 5, where 1 denotes the worst and 5 denotes the best8. Since evaluating the quality of generated similes is subjective and blurred (Niculae and Danescu-Niculescu-Mizil, 2014), we remove the simile-literal sentence pairs if (1) raters argue that the pairs lack context and are difficult to rate (e.g. “Nobody can shoot.”) or (2) some raters rate them as low quality (quality score of 1- 2), while others rate them as high quality (scores of 4-5) (Niculae and Danescu-Niculescu-Mizil, 2014). Moreover, we measure the inter-rater agreement by holding out the ratings of one rater at a time, calculating the correlations with the average of the other rater’s ratings, and finally calculating the average or maximum of all the held-out correlations (denoted as “Mean” and “Max”, respectively). The inter-rater agreement before and after applying the filtering strategies is shown in Tab. 2. Overall, the final inter-rater agreement ensures the reliability of our evaluation of automatic metrics and the filtering strategies improve the inter-rater agreement generally. We finally get 150 simile candidates generated from 44 literal sentences.

heading: 4.2 Results

heading: 4.2.1 Quality
text: We compare our quality metric with the following automatic metrics9: (1) BLEU (Papineni et al.,
8The details about human ratings, including the instructions provided to raters and examples of human ratings are provided in Appx. A.
9These metrics are normalized among simile candidates for a literal sentence, since the quality score between the similes
2002) calculates the precision of n-gram matches, (2) RougeL (Lin, 2004) is a recall-oriented metric, (3) METEOR (Denkowski and Lavie, 2014) proposes a set of linguistic rules to compare the hypothesis with the reference, (4) BERTScore (Zhang et al., 2019) calculates the cosine similarity between the BERT embeddings, (5) Perplexity (Pang et al., 2020) measures the proximity of a language model, the inverse of which is utilized.
Correlations with Human Ratings. Tab. 3 shows the correlation coefficients between automatic metrics and human ratings. Firstly, our metrics are significantly more correlated with human ratings than prior automatic metrics. Moreover, all the sentence-level metrics, which consider the semantics of the entire sentence, perform worse than almost all the n-gram-level metrics, which compare the n-grams between the hypothesis and the reference, which reveals that simile components need to be specifically considered during SG evaluation.
According to the visualized correlation result in Fig. 3, datapoints from prior automatic metrics tend to scatter at 0 or 1, while the datapoints from our metric are distributed closer to the fitter line, proving that our metric can better measure the quality.
Recommendation Task. We compare the rankings given by automatic metrics with human rankings10. We adopt the following metrics: Hit Ratio at rank K (HR@K(K=1,3)), Nor-
generated from different literal sentences can not be compared. Please refer to Appx. C for the implementation of them.
10We remove the literal sentences with fewer than three valid simile candidates in this task, as they are too simple to rank. We finally get 134 sentences from 35 literal sentences.
Metrics HR@1 HR@3 nDCG@1 nDCG@3 MRR
N-gram-level Metrics
BLEU1 0.429 0.857 0.893 0.945 0.662 BLEU2 0.314 0.838 0.892 0.936 0.600 BLEU3 0.286 0.838 0.859 0.924 0.648 BLEU4 0.286 0.838 0.882 0.929 0.581 Rouge1 0.400 0.848 0.907 0.941 0.655 Rouge2 0.400 0.848 0.905 0.937 0.650 RougeL 0.429 0.848 0.901 0.937 0.670
METEOR 0.286 0.857 0.884 0.936 0.589
Sentence-level Metrics
BERTSbase 0.314 0.829 0.870 0.934 0.585 BERTSlarge 0.257 0.838 0.895 0.939 0.570 Perplexity 0.257 0.810 0.898 0.940 0.549
HAUSER
Quality 0.457 0.848 0.915 0.937 0.688
Table 4: Comparison of automatic metrics ranking and human ranking when evaluating quality.
malized Discounted Cumulative Gain at rank K (NDCG@K(K=1,3))11, and Mean Reciprocal Rank (MRR). From Tab. 4, our metric achieves significant improvement compared to other metrics, indicating that our metric can yield more accurate rankings for quality. Also, the n-gram-level metrics generally outperform sentence-level metrics, which is consistent with the result in Tab. 3.
Ablation Study. To investigate the importance of different sub-metrics in quality metric, we compare the correlation between quality metric and human ratings after removing each sub-metric individually. From Tab. 3, the removal of any sub-metric leads to a decline in performance, which proves the effectiveness of each sub-metric. Among three components, the removal of the relevance results in the largest performance drop, which reveals that relevance is the most important sub-metric.
The Effects of Hyperparameters. Since different sub-metrics have varying levels of importance, we study the correlation results when gradually increasing the weight of relevance component and decreasing the weight of sentiment consistency component (as in Tab. 5). From Fig. 4 (left), increasing the weight of the relevance component consistently results in improved performance, peaking at the combination [7](α, β, γ = 3/6, 2/6, 1/6), before eventually causing a decline in performance. This reveals that although relevance is the most important sub-metric, too much weight on it can be detrimental.
The Effects of Referenced Dataset Size. We sample different numbers of simile sentences from (He et al., 2022) as references for relevance
11The formulated NDCG@K in our setting is provided in Appx. B, with the optimal ranking being human rankings.
score and study the correlation between the quality metric and human ratings12. From Fig. 4 (right)13, correlations grow linearly with exponential growth in referenced dataset size, indicating that using datasets larger than 100k will improve the correlation coefficients. Moreover, the performance at the peak surpasses the prior automatic metrics, proving the effectiveness of our approximation method.

heading: 4.2.2 Creativity
text: We compare our creativity metric with the following automatic metrics: (1) Perplexity which is often utilized to measure diversity as well (Tevet and Berant, 2021), (2) Self-BLEU (Zhu et al., 2018) calculates the BLEU score of each generation against all other generations as references, (3) Distinct n-grams(Dist) (Tevet and Berant, 2021), which is the fraction of distinct n-grams from all possible n-grams across all generations.
Correlations with Human Ratings. From Tab. 6, our metric creativity is significantly more correlated with human evaluation scores compared with prior diversity metrics. According to the visualized correlation result in Fig. 5, the prior diversity metrics have either wide confidence intervals (Perplexity, Dist) or scattered datapoints (self-BLEU), whereas our creativity metrics exhibit stronger linear correlation and narrower confidence intervals (Creativty w/ Log), implying higher reliability.
Recommendation Task. We compare the rankings given by automatic metrics with human rank-
12The results are averaged over three random seeds. 13The best hyper-parameter combination is applied.
Metrics Pearson Spearman
Prior Diversity Metrics
Perplexity 0.088 0.041 Self-BLEU3 0.118 0.076 Self-BLEU4 0.196 0.175 Self-BLEU5 0.128 0.077
Dist1 0.278 0.311 Dist2 0.319 0.369 Dist3 0.299 0.379
HAUSER
Metrics HR@1 HR@3 nDCG@1 nDCG@3 MRR
Prior Diversity Metrics
Perplexity 0.314 0.800 0.800 0.903 0.566 Self-BLEU3 0.257 0.771 0.765 0.892 0.520 Self-BLEU4 0.257 0.762 0.756 0.889 0.518 Self-BLEU5 0.229 0.762 0.751 0.882 0.504
Dist1 0.486 0.800 0.862 0.927 0.671 Dist2 0.571 0.810 0.893 0.939 0.737 Dist3 0.543 0.838 0.877 0.938 0.725
HAUSER
Creativty 0.629 0.914 0.944 0.976 0.784
Table 7: Comparison of automatic metrics ranking and human ranking when evaluating creativity.
ings. According to Tab. 7, our creativity metric outperforms prior automatic metrics, which proves our metric can better measure the creativity of simile candidates given a literal sentence, which is consistent with the results in Tab. 6.
Ablation Study. According to Tab. 6, removing the log transformation leads to significant performance drops. According to the visualized correlation result in Fig. 5, the datapoints are distributed closer to the fitter line and exhibit narrower confidence intervals after applying the log transformation, which further proves that log transformation is essential for our creativity metric.
The Effects of Referenced Dataset Size. According to Fig. 6 (left), the correlation coefficients increase continuously and eventually converge as the number of referenced sentences increases. Moreover, the performance after convergence is comparable to that given by the creativity metric based on the simile KB. The trend reveals that our metric referencing 10k similes can achieve a promising correlation with human ratings.

heading: 4.2.3 Informativeness
text: The Pearson and Spearman correlation coefficients between our informativeness metric and human ratings are 0.798 and 0.882, respectively. According to Fig. 6 (right), the strong linear correlation between the metric and human ratings proves that our
informativeness metric is simple yet quite effective.

heading: 4.2.4 Relation between Metrics
text: We present pair-wise correlations between the three automatic metrics in Tab. 8 and also visualize them in Fig. 7. Among the three metrics, creativity correlates with informativeness moderately, mainly because shorter vehicles tend to be less creative than longer ones. The correlations of all other pairwise metrics are relatively weak. Thus, it is evident that the three metrics are independent of each other and it is necessary to measure each one of them to obtain a holistic view of SG evaluation.
5 HAUSER Application
We perform a case study to prove that our designed automatic metrics are effective for various methods. Here, we apply our metrics to a retrieval method (Zhang et al., 2021) (denoted as BM25),
which utilizes the 20 context words around the insertion position given by groundtruth to retrieve the 5 most similar samples based on the BM25 ranking score from the training set, and adopts the vehicles from these samples to be those of simile candidates. This method ensures the diversity of generated similes. The method introduced in Sec. 4.1 is denoted as Ours. Given the candidates generated by each method, we rerank them using a weighted combination of quality, creativity, and informativeness rankings obtained by HAUSER, with a ratio of 2:2:1.
From Tab. 11 in Appendix, the candidates generated by various methods can be more correlated with human rankings after being ranked by our metrics, thus proving the generality of our metrics. It is noticed that the insertion position for BM25 is provided by the groundtruth, while the insertion position for Ours is predicted by the model, thus proving the effectiveness of our generation method.

heading: 6 Conclusion
text: In this work, we systematically investigate the evaluation of the Simile Generation (SG) task. We establish a holistic and automatic evaluation system for the SG task, containing five criteria from three perspectives, and propose holistic automatic metrics for each criterion. Extensive experiments verify the effectiveness of our metrics.

heading: Acknowledgements
text: This research is funded by the Science and Technology Commission of Shanghai Municipality Grant (No. 22511105902), Shanghai Municipal Science and Technology Major Project (No. 2021SHZDZX0103), National Natural Science Foundation of China (No. 62102095).
Limitations
We analyze the limitations of our work as follows. Firstly, although applying a million-scale simile knowledge base or large-scale simile sentences as reference makes our designed metric significantly
more correlated with humans than prior referencebased metrics (e.g. BLEU, Rouge, BERTScore), our metrics are still reference-based and rely on the quality and scale of referenced data. We have discussed the effect of referenced dataset size in our paper and will design reference-free metrics to further complement our metrics in future work. Additionally, since our metrics utilize a million-scale simile knowledge base or large-scale simile sentences as references, the efficiency of our method is slightly lower than the automatic metrics based on a few references. Nevertheless, this limitation does not prevent our metrics from performing systematic and scalable comparisons between SG models.
Ethical Considerations
We provide details of our work to address potential ethical considerations. In our work, we propose holistic and automatic metrics for SG evaluation and construct an evaluation dataset to verify their effectiveness (Sec. 4.1). All the data sources used in our evaluation dataset are publicly available. The details about human ratings, such as the instructions provided to raters, are provided in Appx. A. In our case study (Sec. 5), the human rankings are discussed by three raters. We protect the privacy rights of raters. All raters have been paid above the local minimum wage and consented to use the evaluation dataset for research purposes covered in our paper. Our work does not raise any ethical considerations regarding potential risks and does not involve the research of human subjects.

heading: A Human Ratings
text: The instructions given to raters are detailed as follows:
1. All raters are provided with the necessary background information on similes and the simile generation task, including the definition of similes, the main simile components, and the motivation of our proposed criteria.
2. To ensure the quality of ratings, all the raters label a small set of 20 samples to reach an agreement on the labeling criteria for each metric before the formal labeling.
3. For each perspective (i.e. quality, creativity, informativeness), three raters are asked to rate each simile from 1 to 5, where 1 denotes the worst and 5 denotes the best. The examples of our human ratings are provided in Tab. 10.
4. During the rating, raters are asked to specifically label the simile-literal sentence pairs which lack context and are thus difficult to rate (e.g. “Nobody can shoot.”).

heading: B NDCG Formulation
text: In our setting, the optimal rankings are human rankings. Hence, given m simile candidates S = {s1, s2, ..., sm}, the NDCG@k given by each automatic metric is defined as follows:
NDCG(k) = DCG(Ohypo, k) DCG(Oref, k)
(10)
DCG(O, k) = k∑
i=1
O[I(i)] log2(1 + i)
(11)
where Oref and Ohypo represent the score list given by humans and each automatic metric respectively, O[j] denote the score of sj , I(i) denotes the index of the i-th largest score in O.

heading: C The Implementation of Prior Metrics
text: We report the packages used to implement prior automatic metrics in Tab. 9. For the metric denoted with an asterisk(*), we apply the corresponding package to implement the key parts, based on the definition from the cited papers. The formulation of NDCG in our setting is provided in Appx. B. The rest of the metrics are entirely implemented by us according to the cited papers.

heading: D The Details of MAPS-KB
text: MAPS-KB (He et al., 2022) is a million-scale probabilistic simile knowledge, containing 4.3 million simile triplets from 70 GB corpora, along with frequency and two probabilistic metrics, plausibility and typicality, to model each triplet. The simile triplet is in the form of (topic, property, vehicle)(t, p, v).
In our paper, we specifically adopt the frequency and plausibility information from MAPS-KB to implement our relevance metric. With regard to plausibility, it evaluates the quality of simile triplets based on the confidence score of their supporting simile instances (simile sentence, topic, property, vehicle)(si, t, p, v). In each simile instance, the topic and vehicle are extracted from the simile sentence, while the property is generated via generative commonsense model COMET (Bosselut et al., 2019) and prompting the PLMs. MAPS-KB adopt the noisy-or model to measure the plausibility of the triplet (t, p, v), which is defined as follows:
P(t, p, v) = 1− η∏
i=1
(1− S(si, t, p, v)),
where S(si, t, p, v) = P (p|si, t, v) is the confidence score of each simile instance during generation and η is the number of simile instances supporting the simile triplet (t, p, v).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
the “Limitations” Section.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? the “Abstract” Section and the Section 1.
3 A4. Have you used AI writing assistants when working on this paper? I adopt the free api from text-davinci-003 to polish the language of the whole paper.
B 3 Did you use or create scientific artifacts? Section 4,5
3 B1. Did you cite the creators of artifacts you used? Section 4,5 and Appendix A, C, D
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 4, 5, the “Ethical Consideration” Section, and Appendix C, D
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3, 4, 5, and Appendix D
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? the “Ethical Consideration” Section
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4, Appendix A
C 3 Did you run computational experiments? Section 4, 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4, 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3, 4, Appendix C, D
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix A
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? the “Ethical Consideration” Section.
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? the “Ethical Consideration” Section.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We have discussed the effect of referenced dataset size in our paper and will design reference-free metrics to further complement our metrics in future work. Additionally, since our metrics utilize a million-scale simile knowledge base or large-scale simile sentences as references, the efficiency of our method is slightly lower than the automatic metrics based on a few references. Nevertheless, this limitation does not prevent our metrics from performing systematic and scalable comparisons between SG models. Ethical Considerations
We provide details of our work to address potential ethical considerations. In our work, we propose holistic and automatic metrics for SG evaluation and construct an evaluation dataset to verify their effectiveness (Sec. 4. 1). All the data sources used in our evaluation dataset are publicly available. The details about human ratings, such as the instructions provided to raters, are provided in Appx. A. In our case study (Sec. 5), the human rankings are discussed by three raters. We protect the privacy rights of raters. All raters have been paid above the local minimum wage and consented to use the evaluation dataset for research purposes covered in our paper. Our work does not raise any ethical considerations regarding potential risks and does not involve the research of human subjects