Venue: EMNLP 2023

StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding

 Cheng Jiayang\({}^{\bullet}\)  Lin Qiu   Tsz Ho Chan\({}^{\bullet}\) Tianqing Fang\({}^{\blackloq}\) Weiqi Wang\({}^{\blackloq}\)

Chunkit Chan\({}^{\blackloq}\) Dongyu Ru\({}^{\diamondsuit}\) Qipeng Guo\({}^{\diamondsuit}\) Hongming Zhang\({}^{\blackloq}\)

Yangqiu Song\({}^{\blackloq}\) Yue Zhang\({}^{\dagger}\) Zheng Zhang\({}^{\diamondsuit}\)

\({}^{\blackloq}\)The Hong Kong University of Science and Technology

\({}^{\dagger}\)Westlake University Amazon AWS AI {jchengaj, yqsong}@cse.ust.hk zhangyue@westlake.edu.cn zhaz@amazon.com

###### Abstract

Analogy-making between narratives is crucial for human reasoning. In this paper, we evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus, StoryAnalogy, which contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory. We design a set of tests on StoryAnalogy, presenting the first evaluation of story-level analogy identification and generation. Interestingly, we find that the analogy identification tasks are incredibly difficult not only for sentence embedding models but also for the recent large language models (LLMs) such as ChatGPT and LLaMa. ChatGPT, for example, only achieved around 30% accuracy in multiple-choice questions (compared to over 85% accuracy for humans). Furthermore, we observe that the data in StoryAnalogy can improve the quality of analogy generation in LLMs, where a fine-tuned FlanT5-xxI model achieves comparable performance to zero-shot ChatGPT.1

Footnote 1: This work was done when Jiayang was an intern at Amazon AWS AI Lab. Code and data are released at: [https://github.com/loginaway/StoryAnalogy](https://github.com/loginaway/StoryAnalogy).

## 1 Introduction

Analogy-making plays a central role in human reasoning abilities. By drawing similarities between seemingly unrelated concepts (e.g., in Figure 1, "virus" v.s. "burglar") and processes ("the virus invades cells" v.s. "the burglar breaks into the house"), we can infer that the virus infiltrates and damages cells in a similar way to how a burglar breaks into a house to steal or cause harm. These story-level analogies, which involve comparing entire narratives or coherent sequences of events, enable intelligent agents to gain insights Boden (2009); Ding et al. (2023); Bhavya et al. (2023) and understand complex phenomena Webb et al. (2022).

Despite its significance, there has been limited research on story analogies. One of the reasons is the lack of available data and evaluation benchmarks. In contrast, the community has predominantly focused on word-level analogies, which involve identifying relational similarities between pairs of concepts (e.g., _king_ to _man_ is like _queen_ to _woman_) Mikolov et al. (2013); Gladkova et al. (2016); Czinczoll et al. (2022).

In this work, we introduce StoryAnalogy, a large-scale story-level analogy corpus derived from various domains: scientific scripts, social narratives, word analogies, and knowledge graph triples, to facilitate the study of complex analogies. The story-level analogies we examine contain richer relational details, such as relations between entities (e.g., virus, _invades_, cells) and between events (e.g., the virus invades cells, _as a result_, the virus damages DNAs).

One of the challenges in building StoryAnalogy is establishing a clear and specific way to evaluate story analogies. To address this problem, we extend the Structure-Mapping Theory (SMT; Gentner (1983) to evaluate on longer texts. According to SMT, analogies hold (e.g., the _hydrogen atom_ vs. the _Solar System_) because of the similarity in _relational_ information (e.g., the relative motion between objects), rather than _attributive_ information (e.g., size), between the source and target. Conversely, if both types of information are similar, the source and target

Figure 1: An example of analogy between story **S1**: the invasion of cells by a virus, and **S2**: a burglar breaking into a house.

 

exhibit a literal similarity (e.g., the _X12 star system_ v.s. the _Solar System_). Inspired by this notion, we extend SMT to the story level (SS 2.1). We use _entity_ and _relation similarity_ to assess the level of similarity in _attributes_ and _relations_ between the source and target stories. Additionally, we propose an _analogy score_ based on these two similarities to quantify the degree of analogy between stories. Figure 2 provides a visual representation of the similarity space spanned by the two similarities.

We then collect candidate story analogies for similarity annotations. Since story analogies are scarce in free texts2, we use large language models (LLMs) to generate story pairs that are likely to be analogies. The stories are sourced from various domains, including scientific scripts Dalvi et al. (2018), social commonsense stories Mostafazadeh et al. (2016), word-level analogies Turney et al. (2003); Czinczoll et al. (2022), and knowledge graphs Speer et al. (2017). Next, we conduct crowd-sourcing to obtain similarity annotations for each candidate story pair. As a result, we create StoryAnalogy, which consists of 24K diverse story pairs, each with human annotation guided by the extended SMT.

Footnote 2: Analogies are only present in approximately 3% of a scientific corpus Sultan and Shahf (2023), and the prevalence is expected to be even lower in general texts.

Based on StoryAnalogy, we curate a set of tests to evaluate the analogy identification ability of models. Our findings indicate that both competitive encoder models (such as SimCSE Gao et al. (2021) and OpenAI's text-embedding-002) and LLMs (such as ChatGPT OpenAI (2022) and LLMa Touvron et al. (2023)) have a significant gap compared to human performance in terms of predicting the level of analogy between stories. We further evaluate LLMs using multiple choice questions derived from the story candidates. Even the best-performing LLM still falls short of human performance by 37.7%. Furthermore, we discover that using stories in StoryAnalogy can enhance models' ability to identify and generate analogies. By employing few-shot in-context learning and finetuning on StoryAnalogy, baseline models achieve a considerable performance boost. For instance, a fine-tuned FlanT5-xxl model exhibits generation quality on par with zero-shot ChatGPT. We hope that the data and evaluation settings we proposed in this study will benefit the research community in the area of story analogies.

## 2 StoryAnalogy

Conventional benchmarks in computational analogy primarily focus on word-level analogies (e.g. _word_ to _language_ is like _note_ to _music_). However, less attention has been given to more sophisticated analogies. We introduce StoryAnalogy, a dataset of 24,388 pairs of stories (e.g., _"The virus invades cells and DNAs are damaged."_ versus _"A burglar breaks into the house and smashes the valuables inside."_), each annotated with two dimensions of similarity based to SMT.

### Evaluating story analogies

To assess the degree of analogy between a pair of instances, recent studies classify story pairs using a set of labels. For instance, Sultan and Shahaf (2023) use 5 labels including not-analogy, self-analog, close-analogy, far-analogy, and sub-analogy. Nagarajah et al. (2022) use 6 labels: shallow attribute analogy, deep attribute analogy, relational analogy, event analogy, structural analogy, and moral/purpose. However, they observed very poor agreement among annotators for most labels, which indicates a vague understanding of the task. Making comparisons across these studies are challenging due to the vastly different settings.

In cognitive psychology, the Structure Mapping Theory (SMT; Gentner (1983) is well-known for its explanation of the cognitive process of making analogies between objects. SMT evaluates object comparisons from two perspectives: (a) the attributes of objects and (b) the relational structures between objects. Analogies between objects

Figure 2: The similarity space, showing different kinds of matches in terms of the degree of relation similarity versus entity similarity . According to SMT, we can classify the type of matches Analogy, Literal similarity, Anomaly, and Mere-appearance) between the source and target story by the two similarities. The figure is an extension with story examples based on the visualization in Gentner and Markman (1997).

 

occur when they have similar relational structures but dissimilar attributes (e.g., the _hydrogen atom_ v.s. the _Solar System_). In contrast, literal similarity occurs when objects have both similar relational structures and attributes (e.g., the _X12 star system_ v.s. the _Solar System_).

Based on SMT, we propose to compare stories by their _entity_ and _relation similarity_. These measures assess the degree of similarity in terms of attributive and relational structures, respectively. We provide necessary extensions to their definitions:

**Entity similarity** (EntSim  ). The similarity of entity and topics discussed between a pair of stories, ranging from 0 (unrelated) to 3 (almost equivalent). This score should be high if the two stories are both discussing apples and pears, even if they differ greatly in the details.

**Relation similarity** (RelSim  ). The similarity of relational structures between a pair of stories, ranging from 0 (very poor alignment) to 3 (alignment). In this context, the relational structures refer to the connections between elements at different levels. For instance, first-order relations can be regarded as the relationship between entities, such as predicates. Second-order relations, on the other hand, represent connections between higher granularity elements, such as the logical connection between events or sentences. We encourage annotators to also consider higher-order relational similarity, such as the moral or purpose behind the stories.

We present the established similarity space with example source and target stories in Figure 2.

Modeling the _analogy score_ (\(\alpha\)).We discuss possible definitions of the _analogy score_ (\(\alpha\)). The score \(\alpha\) should be proportional to the level of analogy between a pair of stories. Defining \(\alpha\) to be equivalent with RelSim has been adopted in word analogy [45]. However, this definition cannot distinguish analogy from literal similarity, as both of them have high RelSim (Figure 2). We can alleviate this problem by introducing EntSim to the definition of \(\alpha\): according to SMT, analogy happens when the RelSim between the source and target story is high and the EntSim is low3. Therefore, in the rest of this paper, we define \(\alpha\) as RelSim/EntSim4.

Footnote 3: An analogy is a comparison in which relational predicates, but few or no object attributes, can be mapped from base to target.” [67]

Footnote 4: In practice, we compute it by RelSim/(1+EntSim) to ensure numerical stability.

### Distilling story analogies from LLMs

Obtaining a large number of story analogy by retrieval is difficult. Evidence from Sultan and Shahaf [2023] shows that the prevalence of analogies within a categorized dataset is around 3%. It is expected that the ratio is much lower in general corpora. Identifying analogies by retrieving from general corpora would thus require huge human efforts, making it unrealistic to build a large-scale story 

analogy collection in this way. Recently observations suggest that LLMs are capable of understanding and predicting analogies for problem-solving (Webb et al., 2022), cross-domain creativities (Ding et al., 2023), and generating explanations for word analogies (Bhavya et al., 2022). In addition to these findings, we discover that LLMs can generate high-quality story analogies (i.e., with more than a half generations being analogies). Here, we introduce the pipeline for generating story analogies. The generated analogies are further annotated by crowd annotators for verification. (Details are in SS A.1.)

**Curating seed examples.** The first step is to curate a seed set of story analogies. We ask experts from our team to write story analogies. To ensure diversity, the experts are required to consider multiple domains and are allowed to search in corpora or on the Internet. They then determine whether these story pairs are indeed analogies Examples that are not considered analogies are removed from the gold set. As a result, we obtained a total of 28 story analogy examples, each containing a pair of stories and the corresponding entities.

**Source data.** To guarantee the coverage of topics, we sample from corpora of four domains to generate stories, including (1) scientific scripts ProPara (Dalvi et al., 2018), (2) social commonsense stories ROCStories (Mostafazadeh et al., 2016), (3) word analogy evaluation sets5 SAT (Turney et al., 2003), U2 and U46, and SCAN (Czinczoll et al., 2022). , and (4) the commonsense KG ConceptNet7(Speer et al., 2017). Note that, source data (1) and (2) consist of stories, while (3) and (4) consist of word pairs.

Footnote 5: After manual inspection of all word analogy datasets, we do not include classic datasets such as Google (Mikolov et al., 2013), where the relations between words are relatively easier syntactic or shallow semantic relations, such as (“similar: similarly”, “rare: rarely”).

Footnote 6: [https://englishforeveryone.org/Topics/Analogies.html](https://englishforeveryone.org/Topics/Analogies.html)

Footnote 7: We consider entity pairs in triples that share the same relations from ConceptNet. [https://huggingface.co/datasets/relbert/analogy_questions](https://huggingface.co/datasets/relbert/analogy_questions)

**Generating story candidates.** Using the seed examples and source data, we prompt LLMs8 to generate analogies. Due to the different formats of the source data, the story pairs are generated using two different paradigms (Details are in SSA A.1.):

Footnote 8: OpenAI’s text-davinci-003 is used in the generation.

_Generating from story pairs._ Given a source story and several source-target story pairs sampled from the seed examples, we prompt an LLM to generate the target story.

_Generating from word pairs._ Given a word analogy pair (e.g. _"word"_, _"language"_ and _"note"_, _"musicic"_), together with source-target analogies with the corresponding entities from seed examples, an LLM is prompted to generate both the source and target stories.

### Annotation

To evaluate each candidate story pair under the extended SMT, we conduct crowd annotations on Amazon Mechanical Turk9. We recruit crowd workers to annotate the entity and relation similarities for the collected pairs. In addition, workers are required to label an instance as "poor quality" if they find the generated content broken or toxic. The annotation consists of the following two rounds:

**(i) Qualification round.** We first annotate 80 candidate story pairs (20 from each domain) to curate a qualification set. Three domain experts from our team are asked to read through the annotation instruction and independently annotate EntSim and RelSim for these pairs. The Spearman's \(\rho\) between each annotator's prediction with the average scores of the others ranges from 93% to 96% on EntSim, and from 89% to 95% on RelSim.

Footnote 9: [https://www.mturk.com/](https://www.mturk.com/)

We invite crowd workers who have >=90% history approval rates and have >=1K HITs approved to attend the qualification. Workers whose predictions achieve >=70% Spearman's \(\rho\) with the average scores from three experts pass the qualification. As a result, 158 and 80 workers passed the qualification for EntSim and RelSim, respectively.

**(ii) Main round.** Qualified crowd workers are invited to attend the main round annotations. We assign 5 different annotators to give predictions for each similarity of a story pair. To guarantee the annotation quality, we follow the annotation setting in (Agirre et al., 2012). We split the main round into multiple mini-rounds, each with 1K-2K candidate pairs. After each mini-round, we filter out and disqualify workers who do not show significant correlations with the average scores of the others. They are paid more than what is required by the local wage law. In addition, experts from our team manually check the quality of annotations and write feedback to workers correspondingly.

The generated contents sometimes contain hallucinations or toxic contents. We filter out story pairs labeled as "poor quality" by more than 10% anno 

notators, which accounts for 142 instances. For each story pair, we adopt the average scores from workers as the predicted EntSim and RelSim.

### Analysis of StoryAnalogy

To assess inter-annotator agreement, we randomly sampled 1K instances with 3 independent annotations from our dataset. The Fleiss's kappa Fleiss (1971) on the binarized annotations of EntSim are 47%, and 42% on RelSim, indicating moderate agreement among annotators. In addition, we additionally obtained expert annotations on 200 randomly sampled instances. The averaged Spearman's correlation between crowd and expert annotations on EntSim and RelSim is 64.7% and 69.9%, respectively.

The final dataset consists of 24,388 story pairs on four domains: ProPara (6.9K), ROCStories (4.9K), Word-Analogy (7.5K), and ConceptNet (5.0K). Stories in StoryAnalogy have 19.94 tokens on average. The distributions of EntSim and RelSim are presented in Figure 3. We randomly select 500 instances from each domain as the test set, and another 500 instances from each domain as the validation set. Examples of StoryAnalogy are shown in Table 1.

## 3 Story Analogy Identification

We begin by assessing the ability of models to _identify_ story analogies using two different setups. The first evaluation setup is similar to Semantic Textual Similarity (STS) tasks Agirre et al. (2012), where we calculate the Spearman's correlation between models' predicted similarity and the _analogy scores_ (\(\alpha\)) derived from annotations (SS 3.1). For the second evaluation, we reframe our dataset as multiple-choice questions and evaluate LLMs on this set (SS 3.2).

### Correlation with the _analogy score_\(\alpha\)

Similar to the STS-style evaluation Agirre et al. (2012), we assess whether models can predict analogy scores based on embeddings (for encoder models) or by generation (for LLMs). We use a model to predict the similarity \(f(\cdot,\cdot)\) for two stories. For encoder models, \(f(s1,sz)={\tt Cosine}({\tt Encoder}({\tt s1}),{\tt Encoder}({\tt s2}))\). For LLMs, we prompt them to predict the EntSim and RelSim for the two stories. Finally, Spearman's correlations between the predicted similarity and the respective scores are reported.

Setups.We consider both encoder models and LLMs as baselines. Details are in SS A.2.

The encoder models we evaluate include RoBERTa Liu et al. (2019), SimCSE Gao et al. (2021), OpenAI-ada (text-embedding-ada-002), Discourse Marker Representation (DMR) Ru et al. (2023), RelBERT Ushio et al. (2021), and GloVe embeddings Pennington et al. (2014) on nouns, verbs, or all words10. In addition to the unsupervised encoder models, we also fine-tune two models on the training set: a regression model, RoBERTa-Reg, which has a multilayer perceptron on top of the RoBERTa model that predicts EntSim and RelSim, and a contrastive learning-based model, RoBERTa-CL, which uses a contrastive learning objective to optimize its representations.

Footnote 10: We use Stanza Qi et al. (2020) to conduct part-of-speech tagging for words.

For LLMs, we test FlanT5 Chung et al. (2022), LLMa Touvron et al. (2023), ChatGPT OpenAI (2022), and GPT-3.5 (text-davinci-003). Each model input is composed of three parts: the instructions, which give explanations to the similarity scores; \(N\) examples, and the query story pair. We evaluate models with two instructions (short and long, where short instructions only contain the labels, and long instructions additionally have label definitions), and \(N\) is set to 0, 1, or 3.

Results.The overall evaluation results are presented in Table 2. Generally, the models perform relatively poorly on the analogy score \(\alpha\), indicating that there is still room for improvement on StoryAnalogy.

Figure 3: Distributions of EntSim and RelSim on four data domains in StoryAnalogy. Notably, the distributions of EntSim and RelSim on ROCStories tend to skew towards higher values. This could be attributed to the fact that stories from this source primarily revolve human-focused social narratives.

 

We have the following observations: (1) Similarities from state-of-the-art sentence embedding models are not good indicators for story analogy. Encoders such as RoBERTa, SimCSE, and OpenAI-ada show relatively good correlation with EntSim and RelSim, but they perform poorly on the analogy score \(\alpha\). This suggests that their embeddings are suitable for literal similarity retrieval but not analogy retrieval. (2) Relational feature-aware models are better at analogy identification. Additionally, we find that encoder models aware of relation information, such as DMR (discourse relation), RelBERT (inter-word relation), and GloVeVerb (predicates), correlate better with the analogy score \(\alpha\). (3) Finetuning improves models' analogy identification ability. The finetuned models, RoBERTa-Reg and RoBERTa-CL, are the top-performing models that significantly outperform all the other baselines on \(\alpha\). (4) Generally, LLMs do not perform well on the analogy score \(\alpha\). As shown in Figure 4, most LLMs can benefit from longer instructions as the extra definitions help in understanding the scores. Moreover, we find that despite its size, FlanT5-xxl is one of the best-performing LLMs in terms of predicting EntSim and RelSim.

### Multiple choice evaluation

We construct a multiple-choice evaluation set using the annotated story pairs. First, we gather story pairs with EntSim < 1.0 and RelSim > 2.0. For each target story, we choose 3 negative choices to form the candidates. Out of these, two (easy) negative choices are randomly selected, while one (hard) negative example is chosen by retrieving stories with high nounal similarity (measured by the cosine similarity of the noun GloVe embeddings) and < 50% token overlap. An example question is provided in Table 3. To assess human performance, we conduct human annotations.

We assess LLMs on multiple-choice questions. Each model input consists of an instruction, \(N\) examples of multiple-choice questions, and the query

Figure 4: Spearman’s \(\rho\) (%) of LLMs, averaged across data domains. Here E, R, and R/E indicate EntSim, RelSim, and RelSim/EntSim (\(\alpha\)).

 

multiple-choice question. We evaluate the models using three different instructions, such as "Which candidate story is the best creative analogy for the source story?", where **N** can be 0, 1, or 3. As a baseline, we obtain the performance of the analogy retrieval model in Sultan and Shahaf (2023) on our multiple choice questions, which achieves an accuracy of 44.9%.

**Results.** The results are presented in Table 4. Interestingly, while annotators can answer the questions correctly at an accuracy of 85.7%, LLMs struggle on selecting the most analogous story (the averaged accuracy for text-davinci-003 is merely 37.1%). Increasing the number of demonstrations does not show consistent benefits to model prediction. Also, we find that explicitly instructing models to choose the "creative analogy" (SS A.3, question template B) or to provide a definition of SMT when explaining analogies (template C) yields better performance compared to simply asking models to select the best analogy (template A).

We present the breakdown ratio of the percentage of types of choices selected in Table 5. We have the following observations: (1) LLMs can can differentiate between randomly sampled easy negatives and other choices. The proportion of easy negatives they select is less than 20%, whereas random chance would be 50%. Furthermore, more powerful LLMs like GPT-3.5 are better at this judgement compared to LLaMa and FlanT5-xxl. (2) LLMs can be easily distracted by hard negatives, as they often have a similar or higher chance of selecting hard negative choices instead of the targets. This suggests that the models prioritize surface similarity over structural similarity, despite the latter being more important in identifying analogies.11 (3) In comparison, the baseline model from Sultan and Shahaf (2023) is more resilient against hard negative distractions. This is likely due to its framework design, which captures the structural similarity between stories by clustering entities and finding the mappings between clusters.

Footnote 11: This phenomenon was also observed in visual analogies Bitton et al. (2023), where they found that that models can solve visual analogies well when the distractors are random, but struggle with difficult distractors.

## 4 Story Analogy Generation

We examine whether the dataset StoryAnalogy can enhance the ability of analogy generation. We evaluate FlanT5 Chung et al. (2022), LLaMa Touvron et al. (2023), ChatGPT OpenAI (2022), and GPT-3.5 in zero-shot and few-shot settings using 40 source stories from the test set. To explore the potential of smaller models in generating high-quality analogies, we fine-tuned FlanT5-xl (3B parameters) and FlanT5-xxl (11B parameters) using the same template.

A crowd annotation is conducted to evaluate the quality of the generated stories from the models mentioned above. Workers are provided with a source story and its corresponding generated target story. They are then asked to assess the following: (1) Whether the target story is an analogy for the

\begin{table}
\begin{tabular}{c c c c} \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**N-shot**} \\ \cline{2-4}  & 0 & 1 & 3 \\ \hline FlanT5-xxl & 45.0 & 46.3 & 45.0 \\ LiLAM+GB & 27.7 & 28.6 & 29.5 \\ ChaMPT & 35.8 & 29.2 & 23.3 \\ GPT-3.5 & 44.2 & 34.1 & 33.1 \\ \hline \end{tabular} 
\begin{tabular}{c c c c} \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Question template**} \\ \cline{2-4}  & A & B & C \\ \hline FlanT5-xxl & 41.2 & 47.1 & 48.0 \\ LiLAM+GB & 28.6 & 59.7 & 11.7 \\ ChatGPT & 31.0 & 33.3 & 33.9 \\ GPT-3.5 & 33.7 & 33.6 & 44.0 \\ \hline \end{tabular}
\end{table}
Table 4: Multiple choice evaluation results. Each value represents the average accuracy (%) across different number of demonstrations (left) or across three question templates (right). The random and human performance are 25% and 85.7%, respectively.

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**N-shot**} \\ \cline{2-4}  & 0 & 1 & 3 \\ \hline FlanT5-xxl & 45.0 & 46.3 & 45.0 \\ LiLAM+GB & 27.7 & 28.6 & 29.5 \\ ChaMPT & 35.8 & 29.2 & 23.3 \\ GPT-3.5 & 44.2 & 34.1 & 33.1 \\ \hline \end{tabular} 
\begin{tabular}{c c c c} \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Question template**} \\ \cline{2-4}  & A & B & C \\ \hline FlanT5-xxl & 41.2 & 47.1 & 48.0 \\ LiLAM+GB & 28.6 & 59.7 & 11.7 \\ ChatGPT & 32.4 & 59.5 & 8.1 \\ GPT-3.5 & 37.1 & 55.8 & 7.1 \\ \hline \end{tabular}
\end{table}
Table 3: An example of the multiple choice question. The goal is to select a candidate story that is the best analogy for the source story.

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Target**} & \multicolumn{3}{c}{**Hard**} & **Easy** \\ \hline Random & 25.0 & 25.0 & 50.0 \\ \hline (Sultan and Shahaf, 2023) & 44.9 & 17.8 & 37.2 \\ FlanT5-xxl & 45.4 & 37.2 & 17.4 \\ LiLAM+GB & 28.6 & 59.7 & 11.7 \\ ChatGPT & 32.4 & 59.5 & 8.1 \\ GPT-3.5 & 37.1 & 55.8 & 7.1 \\ \hline \end{tabular}
\end{table}
Table 5: Breakdown ratio (%) of model predictions: The table presents the percentage of different types of choices selected. “Target” refers to the ground-truth target, which is the analogous story. “Hard” and “Easy” refer to the negative examples sampled using nounal similarity and random sampling, respectively.

 

source (as opposed to being a literal similarity or something else); (2) whether the target story is novel compared to the source; and (3) whether the target is plausible (More details can be found in SS A.4). The average scores from three annotators are reported in Table 6. Example generations are shown in Table 7.

Under the zero-shot setting, we observe that FlanT5 and LLaMa struggle to generate meaningful analogies. They often tend to repeat patterns from the source stories (e.g., only replacing one word). In contrast, ChatGPT and GPT-3.5 produce more flexible stories that are frequently considered as analogies and novel.

Stories in StoryAnalogy can help models generate better analogies. With a few demonstrations, we observe a significant improvement in the generation quality of LLaMa (+28.4% and +27.5%). Moderate improvement on ChatGPT and GPT-3.5 is also observed. Notably, finentuning smaller LMs enhanced their generation quality. The finetuned FlanT5-xxl model performs better than the zero-shot ChatGPT and is comparable to the few-shot ChatGPT and GPT-3.5, despite having fewer parameters. Furthermore, while models become more creative through finetuning and in-context learning, their generation plausibility decreases, indicating an increase in hallucination.

## 5 Related Work

**Word-level analogy.** One of the famous works on word-level computational analogy was Mikolov et al. (2013), where they found that word analogies can be predicted by word vector offsets. For instance, \(\overset{\rightarrow}{King}-\overset{\rightarrow}{Man}\) + \(Woman*Queen\). With the development of pretrained language models (PLMs) such as BERT Devlin et al. (2018), there have been works utilizing PLMs to solve word analogies by LM perplexity Ushio et al. (2021), pretrain relational embedding on certain prompt templates Usho et al. (2021), or use word analogies as latent restriction to implicitly probe relational knowledge Rezaee and Camacho-Collados (2022).

In this line of work, a typical evaluation setting is ranking word pairs based on their relational similarity with the source pair Mikolov et al. (2013); Czinczoll et al. (2022). For instance, given a word pair A:B, the aim is to select a target pair C:D such that the relation between C and D is the most similar to A:B among all candidates. This is similar to our multiple-choice evaluation setting.

In comparison, only a handful of research has been done in sentence or paragraph-level analogy: **Analogous text retrieval.** Built on the famous structure mapping theory, SME Falkenhainer et al. (1989) and LRME Turney (2008) model the analogy retrieval problem as an entity-mapping problem. They then solve this problem through web mining Sultan and Shahaf (2023) develops a QASRL based analogy retrieval method to conduct

\begin{table}
\begin{tabular}{c l c c c} \hline \multirow{2}{*}{**Setting**} & \multicolumn{3}{c}{Generation quality} \\ \cline{2-5}  & \multicolumn{1}{c}{**Analogy**} & \multicolumn{1}{c}{**Novelty**} & \multicolumn{1}{c}{**Plausibility**} \\ \hline  & FlanT5-xl & 52.5 & 48.3 & 92.5 \\  & FlanT5-xxl & 46.7 & 49.2 & 92.5 \\ Zero & LLaMa-65B & 38.3 & 39.2 & **93.3** \\  & ChatGPT & 70.0 & 72.5 & 90.8 \\  & GPT-3.5 & 75.8 & 81.7 & 87.5 \\ \hline  & FlanT5-xl & 48.3 & 50.0 & 91.7 \\  & FlanT5-xl & 40.0 & 43.3 & 85.0 \\ Few & LLaMa-65B & 66.7 & 66.7 & 92.5 \\  & ChatGPT & **78.3** & **83.3** & 86.7 \\  & GPT-3.5 & 77.5 & 79.2 & 88.3 \\ \hline Tuned & FlanT5-xl & 65.8 & 79.2 & 88.3 \\  & FlanT5-xl & 72.5 & 81.7 & 86.7 \\ \hline \end{tabular}
\end{table}
Table 6: The crowd-annotated generation quality (%) in terms of (1) Whether the target story is considered an analogy to the source; (2) Novelty of the target story; (3) Plausibility of the generations.

\begin{table}
\begin{tabular}{p{42.7pt} p{170.7pt}} \hline Source & A projectile is affected by gravity. It falls and picks up speed. \\ \hline ZemGrad & Just as a sided sliding down a steep hill gains momentum as it accelerates, so does a projectile as it falls under the force of gravity. \\ GPT-3.5 & An Olympic runner is running a middle distance race. \\ LLaMa-65B & A projectile is affected by gravity. It falls and picks up speed. \\ FlanT5-xl & A rocket is affected by gravity. It picks up speed and goes faster. \\ FlowStep & \\ ChatGPT & A seed is planted in fertile soil. It takes root and grows, flourishing into a healthy plant. \\ GPF-3.5 & An alpha particle is buffered by the atmosphere. It spirals and eventually finds the ground. \\ LLaMa-65B & A projectile is affected by gravity. It falls and picks up speed. \\ FlanT5-xl & A rockist is affected by gravity. It falls and picks up speed. \\ FlanT5-xl & A car is affected by gravity. It accelerates and picks up speed. \\ FlanT5-xl & A meter is propelled by a rug. It moves and finds its way. An idea is planted by brainstorm. It takes shape and comes to fruition.

\begin{table}
\begin{tabular}{p{42.7pt} p{170.7pt}} \hline Source & A projectile is affected by gravity. It falls and picks up speed. \\ \hline ZenGrad & Just as a sided sliding down a steep hill gains momentum as it accelerates, so does a projectile as it falls under the force of gravity. \\ GPT-3.5 & An Olympic runner is running a middle distance race. \\ LLaMa-65B & A projectile is affected by gravity. It falls and picks up speed. \\ FlanT5-xl & A rockist is affected by gravity. It falls and picks up speed. \\ FlanT5-xl & A car is affected by gravity. It picks up speed and goes faster. \\ Footnote 1: footnotemark: \\ ChatGPT & A seed is planted in fertile soil. It takes root and grows, flourishing into a healthy plant. \\ GPT-3.5 & An alpha particle is buffered by the atmosphere. It spirals and eventually finds the ground. \\ LLaMa-65B & A projectile is affected by gravity. It falls and picks up speed. \\ FlanT5-xl & A rockist is affected by gravity. It falls and picks up speed. \\ FlanT5-xl & A car is affected by gravity. It accelerates and picks up speed. \\ FlanT5-xl & A meter is propelled by a rug. It moves and finds its way. An idea is planted by brainstorm. It takes shape and comes to fruition. \\ \hline \end{tabular}
\end{table}
Table 7: Examples showing the source story and model generations under zero-shot, few-shot, and finetuning settings.

 

entity mapping. However, these works evaluate their methods by annotating the precision of the top-ranked results, leaving no large-scale analogy evaluation benchmarks to date.

**Analogy generation.** Recently, there have been attempts at pretraining or prompting LMs for analogy generation. Bhavya et al. (2022); Webb et al. (2022) evaluated LLMs' ability on solving word analogy tasks, where they found that large language models such as GPT-3.5 can surpass human performance on certain word analogy tasks. Ding et al. (2023) evaluated LLMs' creativity in terms of crossdomain analogies. Bhavya et al. (2022); Chen et al. (2022) evaluated LMs' ability on generating explanations for word analogies. Bhavya et al. (2023) proposed a novel analogy mining framework based on generation.

**Analogy benchmarks.** There are many word-level analogy datasets. Google Mikolov et al. (2013), BATS Gladkova et al. (2016) contain relatively easier syntactic or shallow semantic relations. In contrast, U2 and U4, and Czinczoll et al. (2022) include examples with relatively more abstract relations. To the best of our knowledge, there is no large-scale story-level analogy data or resources as of the time of writing. The only related works here are Li and Zhao (2021); Zhu and de Melo (2020), which transform word analogy pairs into sentence pairs with a few templates. Nagarajah et al. (2022) tried to annotate a tiny scale story analogy benchmark based on fables, but they failed to achieve. Wijesiriwardene et al. (2023) re-organized sentence relation datasets, where they viewed such relations (e.g., entailment, negation) are analogy, which is fundamentally different from our settings.

**Analogy in other domains.** In addition to analogies on word pairs and stories, there have been related studies on other topics. Hope et al. (2017) contribute a method for analogy mining over products. Chan et al. (2018) mine analogies from research papers with respect to their background, purpose, mechanism, and findings. Gilon et al. (2018) develop a search engine for expressing and abstracting specific design needs. Recently, Bitton et al. (2023) propose a visual analogies dataset, VASR, where they found that models struggle to find out analogies when given carefully chosen distractors.

## 6 Conclusion

We introduce StoryAnalogy, a multi-domain story-level analogy corpus with 24K story analogies pairs annotated on two similarities under the extended SMT. To assess the analogy identification and generation capabilities of various models, we have devised a series of tests based on StoryAnalogy. The experimental findings indicate that current encoder models and LLMs still fall short of human performance in analogy identification. Additionally, we demonstrate that generative models can greatly benefit from our dataset.

### Limitations

We attempted to ensure dataset coverage by utilizing seed data from various sources. However, there are still specific domains that we were unable to include, such as biomedical stories or academic articles. We can extend the annotation to these domains using the annotation framework and evaluation metrics mentioned in this paper. Additionally, we have explored applications such as analogy identification (Section 3) and generation (Section 4). The potential of StoryAnalogy to be applied for creativity generation tasks (such as poetry, lyrics and humor generation) has not been fully investigated. Further development on other sources and applications is left as future work.

### Ethics Statement

The generated knowledge in StoryAnalogy have been carefully evaluated by crowd annotators to remove any possible toxic or counterfactual content. We set the threshold to be as low as 10%, such that any annotator's labeling an instance as toxic will lead to its removal. 142 instances are removed during this process. We conformed to recognized privacy practices and rigorously followed the data usage policy. We declare that all authors of this paper acknowledge the _ACM Code of Ethics_ and honor the code of conduct.

## Acknowledgements

The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20) and the GRF (16211520 and 16205322) from RGC of Hong Kong. We also thank the support from the UGC Research Matching Grants (RMGS20EG01-D, RMGS20CR11, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08) 



## References

* Agirre et al. (2012) Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: a pilot on semantic textual similarity. In
* SEM 2012: The First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385-393. Cited by: SS4.
* Bhavyya Bhavya et al. (2022) Analogy generation by prompting large language models: a case study of instructept. arXiv preprint arXiv:2210.04186. Cited by: SS1.
* B. Bhavya, J. Xiong, and C. Zhai (2023) Cam: a large language model-based creative analogy mining framework. In Proceedings of the ACM Web Conference 2023, pp. 3903-3914. Cited by: SS1.
* Y. Bitton, R. Yosef, E. Strugo, D. Shahaf, R. Schwartz, and G. Stanovsky (2023) Vasr: visual analogies of situation recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, pp. 241-249. Cited by: SS1.
* M. A. Boden (2009) Computer models of creativity. AI Magazine30 (3), pp. 23-23. External Links: ISSN 0021-907X, Document, Link Cited by: SS1.
* Chunkit Chan and T. H. Chan (2023) Discourse-aware prompt for argument impact classification. In Proceedings of the 15th International Conference on Machine Learning and Computing, ICMLC 2023, Zhuhai, China, February 17-20, 2023, pp. 165-171. Cited by: SS1.
* C. Chan, J. Cheng, W. Wang, Y. Jiang, T. Fang, X. Liu, and Y. Song (2023) Chatgpt evaluation on sentence level relations: a focus on temporal, causal, and discourse relations. CoRRabs/2304.14827. Cited by: SS1.
* C. Chan, X. Liu, T. H. Chan, J. Cheng, Y. Song, G. Y. Wong, and S. See (2023) Self-consistent narrative prompts on abductive natural language inference. CoRRabs/2309.08303. Cited by: SS1.
* C. Chan, X. Liu, J. Cheng, Z. Li, Y. Song, G. Y. Wong, and S. See (2023) Discoprompt: path prediction prompt tuning for implicit discourse relation recognition. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 35-57. Cited by: SS1.
* J. Chen, J. C. Chang, T. Hope, D. Shahaf, and A. Kittur (2018) Solvent: a mixed initiative system for finding analogies between research papers. Proceedings of the ACM on Human-Computer Interaction2 (CSCW), pp. 1-21. Cited by: SS1.
* J. Chen, R. Xu, Z. Fu, W. Shi, Z. Li, X. Zhang, C. Sun, L. Li, Y. Xiao, and H. Zhou (2022) E-kar: a benchmark for rationalizing natural language analogical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 3941-3955. Cited by: SS1.
* Y. Chen, J. Cheng, H. Jiang, L. Liu, H. Zhang, S. Shi, and R. Xu (2022) Learning from sibling mentions with scalable graph inference in fine-grained entity typing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2076-2087. Cited by: SS1.
* J. Cheng, H. Jiang, Q. Yang, and Y. Xiao (2021) A question-answering based framework for relation extraction validation. CoRRabs/2104.02934. Cited by: SS1.
* Wei-Lin Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, I. Stoica, and E. P. Xing (2023) Vicuna: an open-source chatbot impressing gpt-4 with 90%* chatgp quality. Cited by: SS1.
* H. Won Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. (2022) Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. Cited by: SS1.
* L. Cui, D. Yang, J. Cheng, and Y. Xiao (2021) Incorporating syntactic information into relation representations for enhanced relation extraction. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 416-428. Cited by: SS1.
* L. Cui, D. Yang, J. Yu, C. Hu, J. Cheng, J. Yi, and Y. Xiao (2021) Refining sample embeddings with relation prototypes to enhance continual relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 232-243. Cited by: SS1.
* T. Czinczoll, H. Yannakoudakis, P. Mishra, and E. Shutova (2022) Scientific and creative analogies in pretrained language models. arXiv preprint arXiv:2211.15268. Cited by: SS1.
* B. Dalvi, L. Huang, N. Tandon, W. Yih, and P. Clark (2018) Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1595-1604. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.



Zijian Ding, Arvind Srinivasan, Stephen MacNeil, and Joel Chan. 2023. Fluid transformers and creative analogies: Exploring large language models' capacity for augmenting cross-domain analogical creativity. _arXiv preprint arXiv:2302.12832_.
* Falkenhainer et al. (1989) Brian Falkenhainer, Kenneth D Forbus, and Dedre Gentner. 1989. The structure-mapping engine: Algorithm and examples. _Artificial intelligence_, 41(1):1-63.
* Fleiss (1971) Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. _Psychological bulletin_, 76(5):378.
* Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 6894-6910.
* Gentner (1983) Dedre Gentner. 1983. Structure-mapping: A theoretical framework for analogy. _Cognitive science_, 7(2):155-170.
* Gentner and Markman (1997) Dedre Gentner and Arthur B Markman. 1997. Structure mapping in analogy and similarity. _American psychologist_, 52(1):45.
* Gilon et al. (2018) Karni Gilon, Joel Chan, Felicia Y Ng, Hila LifshitzAssaf, Aniket Kittur, and Defna Shahaf. 2018. Analogy mining for specific design needs. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_, pages 1-11.
* Gladkova et al. (2016) Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016. Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn't. In _Proceedings of the NAACL Student Research Workshop_, pages 8-15.
* Hope et al. (2017) Tom Hope, Joel Chan, Aniket Kittur, and Dafna Shahaf. 2017. Accelerating innovation through analogy mining. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, pages 235-243.
* Jiang et al. (2023) Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. Lion: Adversarial distillation of closed-source large language model. _CoRR_, abs/2305.12870.
* Jiang et al. (2022) Yuxin Jiang, Linhan Zhang, and Wei Wang. 2022. Improved universal sentence embeddings with prompt-based contrastive learning and energy-based learning. In _Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 3021-3035. Association for Computational Linguistics.
* Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. _CoRR_, abs/2001.08361.
* Li et al. (2023a) Haoran Li, Yulin Chen, Jinglong Luo, Yan Kang, Xiaojin Zhang, Qi Hu, Chunkit Chan, and Yangqiu Song. 2023a. Privacy in large language models: Attacks, defenses and future directions.
* Li et al. (2023b) Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. 2023b. Multi-step jailbreaking privacy attacks on chatgpt. _CoRR_, abs/2304.05197.
* Li and Zhao (2021)Yian Li and Hai Zhao. 2021. Pre-training universal language representation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 5122-5133.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. _Advances in neural information processing systems_, 26.
* Mo et al. (2021) Haoran Mo, Edgar Simo-Serra, Chengying Gao, Changqing Zou, and Ruomei Wang. 2021. General virtual sketching framework for vector line art. _ACM Trans. Graph._, 40(4).
* Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. _arXiv preprint arXiv:1604.01696_.
* Nagarajah et al. (2022) Thiloshn Nagarajah, Filip Ilievski, and Jay Pujara. 2022. Understanding narratives through dimensions of analogy. _arXiv preprint arXiv:2206.07167_.
* OpenAI (2022)OpenAI. 2022. Introducing chatgpt.
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report. _CoRR_, abs/2303.08774.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543.
* Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. _arXiv preprint arXiv:2003.07082_.
* Ray (2023) Partha Pratim Ray. 2023. Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. _Internet of Things and Cyber-Physical Systems_ 

Klamehr Rezaee and Jose Camacho-Collados. 2022. Probing relational knowledge in language models via word analogies. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 3930-3936.
* Ru et al. (2023) Dongyu Ru, Lin Qiu, Xipeng Qiu, Yue Zhang, and Zheng Zhang. 2023. Distributed marker representation for ambiguous discourse markers and entangled relations. _CoRR_, abs/2306.10658.
* Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In _Proceedings of the AAAI conference on artificial intelligence_, volume 31.
* Sultan and Shahaf (2023) Oren Sultan and Dafna Shahaf. 2023. Life is a circus and we are the clowns: Automatically finding analogies between situations and processes.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Navier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Turney (2008) Peter D Turney. 2008. The latent relation mapping engine: Algorithm and experiments. _Journal of Artificial Intelligence Research_, 33:615-655.
* Turney et al. (2003) Peter D Turney, Michael L Littman, Jeffrey Biham, and Victor Shnayder. 2003. Combining independent modules in lexical multiple-choice problems. In _RANLP_, pages 101-110.
* Ushio et al. (2021a) Asaihi Ushio, Luis Espinosa Anke, Steven Schockaert, and Jose Camacho-Collados. 2021a. Bert is to nilp what alexnet is to cv: Can pre-trained language models identify analogies? In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3609-3624.
* Usho et al. (2021b) Asaihi Usho, Jose Camacho-Collados, and Steven Schockaert. 2021b. Distilling relation embeddings from pretrained language models. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 9044-9062.
* Wang et al. (2023) Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. 2023. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. _arXiv preprint arXiv:2310.07521_.
* Webb et al. (2022) Taylor Webb, Keith J Holyoak, and Hongjing Lu. 2022. Emergent analogical reasoning in large language models. _arXiv preprint arXiv:2212.09196_.
* A novel benchmark for long text analogy evaluation in large language models. _CoRR_, abs/2305.05050.
* Zhenyuan et al. (2023) Liu Zhenyuan, Michal Piovarci, Christian Hafner, Raphael Charrondiere, and Bernd Bickel. 2023. Directionality-aware design of embroidery patterns. In _Computer Graphics Forum_, volume 42, pages 397-409. Wiley Online Library.
* Zhu and de Melo (2020) Xunjie Zhu and Gerard de Melo. 2020. Sentence analogies: Linguistic regularities in sentence embeddings. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 3389-3400.

## Appendix A Appendix

### Details in creating StoryAnalogy.

#### a.1.1 Details in generating candidates.

We prompt text-davinci-003 model to collect the story analogy candidates. The prompt template is

Demonstrations.The in-context learning seed examples are presented in Table 8. In addition to the golden story analogies, we also curated the corresponding keyword pairs with regard to each story pair. This keyword pairs are useful for prompting to generate candidate stories from the Word Analogy and ConceptNet inputs, where their input data format are word-pairs (e.g. word: language :: note: music).

To construct a list of demonstrations for each data source, we ask experts to construct a set of analogous story pairs by web-searching and revising the results. Then, to ensure the diversity of the analogy, we make a list of orthogonal topics in each dataset, and randomly sampled demonstrations from these subtopics every time we construct a prompt.

Prompt templates for "generating from story pairs".The template for the demonstration is: "Example:\n(1){sourcestory(i)}nAn analogy for story (1) can be:\n(2){target story(i)}" It is concatenated with a prompt at the end: "Example:\n(1){sourcestory}nAn analogy for story (1) can be:" Prompt templates for "generating from word pairs".The prompt template for generating from word pairs is: "Write group of 2-sentence 



\begin{table}
\begin{tabular}{p{142.3pt} p{190.7pt}} \hline
**Source story** & **Target story** \\ \hline The stream becomes a river. The river continues to flow along the same path for a long time. ENTITY: stream, river & A person grows from a child into an adult. As time passes, the person experiences ongoing growth and maturation. ENTITY: child, adult \\ \hline Magma rises from deep in the earth. The magma goes into volcanos. ENTITY: magma, volcanoes & Food goes up from the stomach. The food enters the esophagus. ENTITY: food, esophagus \\ \hline The plasma membrane encloses the animal cell. It controls the movement of materials into and out of the cell. ENTITY: plasmamembrane, cell & Security guards monitor the doors of the factory. They manage the entry and exit of personnel to and from the factory. ENTITY: security guard, factory \\ \hline The tadpole begins storing food in the tail. The tadpole develops hind legs and lives off food stored in the it’s tail. ENTITY: tadpole, food & A person saves money in a savings account. The person relies on the saved funds to meet future financial obligations and sustain their lifestyle. ENTITY: human, money \\ \hline The sediment near the bottom is compressed by the weight of newer sediment. The sediment becomes sedimentary rock as it is pushed together by the heavy weight. ENTITY: sediment, sedimentary rock & A person’s ideas and beliefs are shaped by their experiences and influences. The person’s thoughts and opinions become more solidified and defined as they are influenced by outside forces. ENTITY: belief, solidified belief \\ \hline Morgan enjoyed long walks on the beach. She and her boyfriend decided to go for a long walk. ENTITY: beach, walking & Lenny liked to climb trees. He embarked on a tree-climbing expedition in the woods. ENTITY: woods, climbing trees \\ \hline He got a call from his girlfriend, asking where he was. Frank suddenly realized he had a date that night. ENTITY: call, date & She received a notification on her phone, reminding her of an upcoming meeting. Jane suddenly remembered there was an important presentation to give. ENTITY: notification, presentation \\ \hline She was petrified and prayed to get out of the test. On the last day of lessons, the bus broke down and she was spared. ENTITY: test, fear & He was terrified of the upcoming job interview. Due to oversleeping on the day of the interview, he missed the appointment and thus avoided the stress. ENTITY: job interview, stress \\ \hline He is only two weeks into his job and he is nervous. Every time he responds to calls he gets very worried. ENTITY: job, nervous & Having recently started a relationship, she is grappling with anxiety. She becomes highly anxious whenever they have a disagreement. ENTITY: relationship, anxious \\ \hline She made sure she was quiet and respected others’ space. It was strange that on Wednesday, she came to the office hung over. ENTITY: introverted, getting drunk & James took care to comply with the rules and demonstrate deference towards authority figures. Surprisingly, he was caught shoplifting on a Friday. ENTITY: disciplined,shoplifting \\ \hline \end{tabular}
\end{table}
Table 8: Seed analogy examples for generation candidates StoryAnalogy. We sample 5 pairs from propara and 5 paris from rocstories.



stories in around 30 words given the keyword(s).\Hint: Story 2 should be analogous to story 1.\n Example 0:\n Keywords forstory 0: {keywords}"

In addition, we notice that the story pairs generated this way tend to have low entity similarity (since their entities are pre-given). Therefore, we additionally prompt LLMs to write a set of similar keywords for the source first, and then write a corresponding story, which does not need to be similar to the source.

"Write a group of 2-sentence stories in around 30 words given the keyword(s). Keywords 1: {} Story 1: {} Give a set of keywords similar to keywords 1, and then write a corresponding story (the stories do not have to be similar):"

#### a.1.2 Annotation templates.

In this section, we showcase our templates used for annotation on the Amazon Mechanical Turk platform. The instructions used for evaluating entity and relation similarities are presented in Figure 6 and Figure 8, respectively. Followed by these instructions, questions are presented using templates shown in Figure 7 and Figure 9.

### Details in SS 3.1.

#### a.2.1 Baselines

DMRNote that DMR requires two sentences as input. We use the NLTK toolkit to tokenize the stories into two sentences. In cases where there are not enough two sentences, we try to split the story by the first comma. If there is no comma in the story, the DMV vector is computed between the story and an empty string " ". This only accounts for a small number of stories (10-100).

GloVeWe use the glove-840B-300D version12. We first use the _Stanza_ part-of-speech (PoS) annotation tool to parse the PoS of words. Then, we return the summation of embeddings for all the words with the corresponding PoS. Specifically, we determine nouns if a word's upos is in {"PROPN", "NOUN"}. We detect verbs if its upos is "VERB" or its xpos starts with "VB".

Footnote 12: [https://nlp.stanford.edu/data/glovee.840B.300d.zip](https://nlp.stanford.edu/data/glovee.840B.300d.zip)

RoBERTa-RegWe apply an MLP on top of the RoBERTa model and output two digits, which correspond to the EntSim and RelSim, respectively.

RoBERTa-CLDetails for training the contrastive learning model. We adopt the SimCSE training script13 for training our contrastive learning model. The positive pairs are filtered according to the EntSim<= 1.0 and RelSim>= 2.0.

Footnote 13: [https://github.com/princeton-nlp/SimCSE](https://github.com/princeton-nlp/SimCSE)

#### a.2.2 Prompt templates for similarity prediction.

The templates used for prompting LLMs to generate similarity predictions are presented below.

The "long instruction" template for EntSim prediction."

"Evaluate the entity similarity between a pair of stories. Assign the pair a score between 0 and 3 as follows:

0 : Unrelated. The two stories are talking about different topics and entities of different types.

1 : Somewhat related. The two stories talk about different entities, and some of them have similar or related types.

2 : Somewhat equivalent. The two stories have different entities, but they have the same types.

3 : Almost equivalent. The entities in the two stories are overlapped or synonymous.

Following the above instruction, evaluate the entity / topic similarity for S1 and S2 (only answer by a score from 0, 1, 2, 3):

{N-DEMONSTRATIONS HERE} Q:

S1 - {INPUT-S1}

S2 - {INPUT-S2}

Score :"

**The "long instruction" template for RelSim prediction.**

"Evaluate the relation similarity between a pair of stories. Assign the pair a score between 0 and 3 as follows:

0 : Very poor alignment. Most if not all relationships do not align.

1 : Alignment with significant mismatches. Some of the relationships align, but there are some significant mismatches.

2 : Alignment with insignificant mismatches. Most of the relationships align except for some insignificant mismatches.

 

align very well between the two stories. Following the above instruction, evaluate the relational similarity for S1 and S2 (only answer by a score from 0, 1, 2, 3): {N-DEMONSTRATIONS HERE} Q: S1 - {INPUT-S1} S2 - {INPUT-S2} Score :" Here, we insert NE\(\{0,1,3\}\) demonstrations at the "N-DEMONSTRATION HEE" and fill in the story pairs at "INPUT-S1" and "INPUT-S2". The "short instruction" templates are similar, with the only difference that the detailed definition of scores is removed. For instance, "@ : Unrelated. The two stories are talking about different topics and entities of different types." is replaced with "@ : Unrelated."

### Details in SS 3.2.

To construct the multiple-choice evaluation set, we gather story analogy pairs with EntSim < 1.0 and RelSim > 2.0. Next, we sample negatives for each story analogy pairs to form multiple choice questions. Similar to the GloVe baseline in SSA.2, we obtain the nounal embedding for each story, and retrieve stories with high cosine similarities while have <50% overlapped tokens as the hard negative choices. We manually inspect the overall quality of the multiple choice questions constructed in this manner. We excluded the questions generated from the ROCStories split due to their lower quality, likely because the unusual distribution of EntSim in this split made it difficult to use the same method for creating the dataset as in the other splits (Figure 3). The resulting multiple choice dataset consists of 360 questions.

Baselines in Sultan and Shahaf [2023]We apply both the FMQ and FMV models, as suggested in Sultan and Shahaf [2023], to our story analogy identification task. To be precise, we gather the intermediate story pair similarities generated by their models. Afterwards, we choose the option that exhibits the highest similarity to the source story. Notably, the lengths of the stories in our dataset are considerably shorter than datasets used in its paper. Therefore, when running the baseline on our dataset, we adjusted the threshold of the similarity filter to better suit our settings. We selected a threshold of 0.3 for FMQ and 0.2 for FMV. For the other implementation details, we follow the original settings in their code repo at [https://github.com/orensul/analogics_mining](https://github.com/orensul/analogics_mining). As for the result, we discovered that FMQ and FMV exhibited comparable performance (44.9% versus 44.7%) on the multiple choice dataset. The result from FMQ are reported in the main paper.

#### A.3.1 Prompt templates for multiple-choice evaluation.

The prompt template used in multiple-choice evaluation is:

" {QUESTION} Sourcestory: {} Candidate stories: {0}: {} {1}  {2}: {} {3}: {} Answer: " where "QUESTION" is replaced with one of the following questions:

A: "Select the candidate that best matches the source story as an analogy." B: "Which candidate story is the best creative analogy for the source story?" C: "A creative analogy should have fewer similar entities but similar relational structures to the source story. Which candidate story is the best creative analogy for the source story?"

### Details in the evaluation of story analogy generation.

#### A.4.1 Generation setups.

The models are evaluated under zero-shot, few-shot, and instruction-tuning settings. For zero-shot and few-shot prompting, the templates are: "Write an analogy for story 1.n n Story 1: {}nStory 2:" This template is also used in the finetuning setting. For finetuning, we employ DeepSpeed14 to accelerate the training on a single 8*V100 (32GB) instance.

Footnote 14: [https://www.microsoft.com/en-us/research/project/deepspeed/](https://www.microsoft.com/en-us/research/project/deepspeed/)

#### A.4.2 Annotation.

We conduct crowd annotation on AMT to evaluate the generation quality. The annotation instruction is presented in Figure 5. During the annotation, the meta information of the target generation is hidden from the annotators and the requesters. In addition, the target stories are shuffled such that 

annotators cannot find out which models are used to generate the stories based on the order.

### Miscellaneous

In this section, we present some discussions that took place during the reviewing process.

#### a.5.1 Potential applications of this work.

_Analogy Mining for Art and Design._ There have been various studies focusing on building analogical search engines. Hope et al. (2017) contribute a method for analogy mining over products. Chan et al. (2018) mine analogies from research papers with respect to their background, purpose, mechanism, and findings. Gilon et al. (2018) develop a search engine for expressing and abstracting specific design needs. Recently, Bitton et al. (2023) propose a visual analogies dataset, VASR, where they found that models struggle to find out analogies when given carefully chosen distractors. In computer graphics, some graphics design algorithms take as input an image from the user, and transform it to some other types of visual designs that are similar to the given image, such as embroidery patterns Zhenyuan et al. (2023) and vector line arts Mo et al. (2021). This category of work establishes connections between images and application-specific graphics patterns. With images as a guidance, the complicated visual design processes are made easy and intuitive for nonprofessional users.

_Analyological Reasoning._ Large language models (LLMs) have demonstrated impressive abilities in few-shot and zero-shot learning Kaplan et al. (2020); OpenAI (2022, 2023). Recently, Chat GPT OpenAI (2022), GPT-4 OpenAI (2023), Alpaca Taori et al. (2023) and their following works Chiang et al. (2023); Jiang et al. (2023) have achieved remarkable performance on a wide range of benchmarks. It is believed that they have acquired certain kind of analogical reasoning ability that are not only task-specific Webb et al. (2022); Ding et al. (2023) , but also omnipresent throughout the prompting process of LLMs, and there are a lot of prompt engineering work to leverage this characteristic for downstream tasks Jiang et al. (2022); Chan et al. (2023); Chan and Chan (2023). Meanwhile, it is important to note that LLMs also exhibit potential issues related to hallucination, biases, and privacy Ray (2023); Li et al. (2023, 2023); Wang et al. (2023). Mitigating such issues often requires building up knowledge bases Cheng et al. (2021); Cui et al. (2021b, a), where analogy could be a useful angle to improve automatic building performance Chen et al. (2022). The data and evaluation metrics in this work may serve as a benchmark in evaluating one of the analogical reasoning abilities.

#### a.5.2 Why the predictions of individual scores are good, but the prediction of \(\alpha\) is bad.

Original question: _How is it that models are so good at individually predicting EntSim and RelSim (in Section 3.1), but they are not that good at predicting the analogy score \(\alpha\)?_

Since the analogy score is computed from both EntSim and RelSim, the prediction of the analogy score relies on predicting the gap between EntSim and RelSim, which is harder than predicting each similarity alone. A case is presented below to illustrate this.

Suppose we have four story pairs, and their ground-truth scores are: EntSim = [2, 1, 3, 0], RelSim = [0, 1, 2, 3].

The corresponding predictions are: EntSim'= [0, 2, 3, 0], RelSim'= [1, 0, 2, 1].

Then, the analogy scores and the predicted analogy scores can be computed from the above values (using \(\alpha=\frac{\texttt{RelSim}}{\texttt{EntSim}+1}\)): \(\alpha\) = [0, 0.5, 0.5, 3], \(\alpha\)' = [1, 0, 0.5, 1].

Finally, we can compute the Spearman's correlation coefficients as:

Corr(RelSim, RelSim')=0.316 Corr(EntSim, EntSim')=0.632 Corr(\(\alpha\), \(\alpha\)') = 0

Here, though the predictions of the respective scores have medium correlation with the ground-truths, the prediction of \(\alpha\) has zero correlation 

Source story: The virus invades cells. As a result, the DNAs are damaged.

and several target stories:

Target story 1: As virus infiltrates the body, it attacks healthy cells and causes a disturbance in the immune system.

Target story 2: Parasites invade their hosts, feed on nutrients, and weaken the immune system, causing damage to the host's body.

Target story 3: The burglar breaks into the house. As a result, the valuables inside are smashed.

Target story 4: A swarm of space bees invaded the atmosphere and caused the planets to spin backwards. As a result, the gravitational path of the universe was disrupted and time travel became possible.

Your task is to answer three YES/NO questions for each target stor:

**1.****Danalogy? Is the target stop an analogy for the source story? Here, an analogy should have similar structures rather than details. For instance, target story 2 and 3 are analogies to the source story, while story 1 is not.**

**2.****[Novel]? Is the target story novel (compared to the source)? This measures the novelty of the target story. Think about whether the target is engaging and interesting to you. For instance, many may found target story 3 novel, while 1 and 2 not.**

**3.****[Plausible!] Is the target story plausible? Check whether the target story is plausible and makes sense. For instance, target story 4 does not really make sense and should be labeled as plausible.**

Figure 5: The annotation instruction for generation quality evaluation.

Figure 6: The instructions used for evaluating entity similarity in human annotations.

 



\begin{table}
\begin{tabular}{p{142.3pt} p{56.9pt} p{78.0pt} p{78.0pt} p{78.0pt} p{78.0pt}}
**Source story** & **Target story** & **Scores** & \(\alpha\) & **Domain** \\ \hline The stream becomes a river. The river continues to flow along the same path for a long time. & A person grows from a child into an adult. As time passes, the person experiences ongoing growth and maturation. & : 0.6 & : 2.8 & 1.6 & PP \\ \hline Fertilage the soil. Mix seeds into the fertilized soil. & Apply lotion to the skin. Massage the lotion into the skin. & : 0.0 & : 2.6 & 2.6 & PP \\ \hline Fill the tray with cool water. Place the tray in the freezer. & Fill the bucket with warm water. Place the bucket in the refrigerator. & Fill the bucket with warm water. Place the bucket in the refrigerator. & : 3.0 & : 3.0 & 0.8 & PP \\ \hline The resulting material disappears. The plant becomes one with the soil. & The water evaporates. The liquid turns into vapor and disperses into the air. & : 2.0 & : 0.4 & 0.1 & PP \\ \hline The gas condenses in the condenser and becomes a liquid again. Heat is radiated away from the condenser. & An emotion is expressed and released. A calming effect follows after the expression. & : 0.3 & : 0.0 & 0.0 & PP \\ \hline They left him the key to the entrance. When Tom went over he realized it was the wrong key. & They gave her the password to the website. When Jane logged in, she realized it was the wrong password. & : 1.0 & : 2.7 & 1.3 & ROC \\ \hline I was building a dresser. I had several tools to help. & I was baking a cake. I had several ingredients to help. & : 1.0 & : 3.0 & 1.5 & ROC \\ \hline It’s broken. I have to buy a new one. & It’s expired. I have to get a new one. & : 3.0 & : 3.0 & : 3.0 & 0.8L & ROC \\ \hline His cellmate tried to buildy the man. The man fought his cellmate. & His climate tried to intimidate him. The man stood his ground and refused to be bullied. & : 2.7 & : 1.5 & 0.4 & ROC \\ \hline The fight lasted until 10 am. We finally just went to bed out of exhaustion. & The argument went on until midnight. We eventually just gave up and went home in defeat. & : 1.3 & : 0.0 & 0.0 & ROC \\ \hline Foundations are poured to support the walls and roofs of buildings. The structure of the building is only as strong as it’s foundation. & Reasons are formulated to make theories. The conclusions of theories are only as dependable as their initial premises. & : 0.6 & : 1.8 & 1.1 & WA \\ \hline The ground for the building is solid and secure. This gives the building its foundation and stability. & The reasons for the theory provide a rational explanation. This informs the decision-making process that supports the theories accuracy. & : 0.4 & : 2.8 & 2.0 & WA \\ \hline His memory has broken into fragmented pieces. He can recall flashes and images of the past, but nothing concrete or clear. & His memories remain a confused mess. Nothing holds together and what he remembers don’t make sense. & : 2.7 & : 3.0 & 0.8 & WA \\ \hline Heat energy is transferred from one point to another. Transfers between different substances cause temperature changes. & Solid materials undergo phase transitions when energy is added. Changes in pressure can also result in phase transitions. & : 2.8 & : 1.0 & 0.3 & WA \\ \hline She laughed and let go off all of her worries. Her carefree attitude was liberating. & He delved into the unknown without a second thought, ignorant of the knowledge to come. & : 0.8 & : 0.0 & : 0.0 & WA \\ \hline The student opens the book and begins to read. The knowledge gained from the book is absorbed by the student. & The cat sees a mouse and begins to chase it. The cat honing its hunting skills through practice and repetition. & : 0.8 & : 1.4 & 0.8 & CN \\ \hline The trigger is pulled and the pistol shoots. The gun fires. & A meteorite impacts Saturn's surface. The planet is buffeted by these larger objects. & : 0.4 & : 2.8 & 2.0 & CN \\ \hline He knew his only way out was to commit suicide. He was determined to die, no matter what. & She figured an overdose was the only way out. Within minutes, she had taken her last breath and died. & : 3.0 & : 2.6 & 0.7 & CN \\ \end{tabular} \begin{tabular}{p{142.3pt} p{142.3pt} p{142.3pt} p{142.3pt} p{142.3pt} p{142.3pt} p{142.3pt}} She purchased a round-trip ticket for her travels. She left with the assurance that she would return. & She chose her destination for her vacation with excitement. She anticipated what her journey would bring. & : 2.8 & : 1.4 & 0.4 & CN \\ \end{tabular} 
\begin{tabular}{p{142.3pt} p{142.3pt} p{142.3pt} p{142.3pt} p{142.3pt}} The rain began to pour and gradually, the river started to overflow. It was the start of a devastating flood. & The scissors snipped away, trimming her locks until her hair was just right. She became the proud owner of a new, short hairstyle. & : 0.0 & : 0.0 & : 0.0 & CN \\ \end{tabular}
\end{table}
Table 9: Examples in StoryAnalogy with annotations from each domain. We report the EntSim® and RelSim® from crowd workers © . The Domain column indicates the source of the story pairs. “PP”, “ROC", “WA”, and “CN” are short for “ProPara”, “ROCStories”, “Word Analogy”, and “ConceptNet”, respectively.





## Chapter 1 Biggert lived on a farm. He loved bacon

### 1.1 Jane lived in the city. She loved pizza

How will you rate the topic similarity between these two stories?

Almost equivalent The entries in the two stories are overlapped or synonymous. Somewhat equivalent The two stories have different entities, but they have the same types. Somewhat related. The two stories talk about different entities, and some of them have similar or related types. Unrelated. The two stories are taking about different topics and entities of different types. At least one of these stories are ungrammatical or counterfactual.

Figure 8: The instructions used for evaluating relation similarity in human annotations.

Figure 7: The template for presenting a question regarding the evaluation of entity similarity in human annotations.



Chapter 1 The animal's heart rate and breathing rate slow. The animal loses weight more slowly than usual.

### The car's engine and exhaust system slow down. The car uses less fuel than usual.

How will you rate the relational similarity between these two stories?

Perfect Alignment: The relationships can align very well between these two stories.

Aigment with insignificant mismatches. Most of the relationships align except for some insignificant mismatches.

Aigment with significant mismatches. Some relationships align, but there are some significant mismatches.

Very poor alignment. Most or even all relationships do not align.

At least one of these stories are ungrammatical or counterfactual.