StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding
Authors: Cheng Jiayang
Lin Qiu
Tsz Ho CHAN
Tianqing Fang
Weiqi Wang
Chunkit Chan
Dongyu Ru
Qipeng Guo
Hongming Zhang
Yangqiu Song
Yue Zhang
Zheng Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Analogy
Semantic similarity

Abstract: Analogy-making between narratives is crucial for human reasoning.
In this paper, we evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus, \textsc{StoryAnalogy}, which contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory.
We design a set of tests on \textsc{StoryAnalogy}, presenting the first evaluation of story-level analogy identification and generation.
Interestingly, we find that the analogy identification tasks are incredibly difficult not only for sentence embedding models but also for the recent large language models (LLMs) such as ChatGPT and LLaMa.
ChatGPT, for example, only achieved around 30\% accuracy in multiple-choice questions (compared to over 85\% accuracy for humans). 
Furthermore, we observe that the data in \textsc{StoryAnalogy} can improve  the quality of analogy generation in LLMs, where a fine-tuned FlanT5-xxl model achieves comparable performance to zero-shot ChatGPT.

STORYANALOGY:
Deriving Story-level Analogies from Large Language
Models to Unlock Analogical Understanding
Cheng Jiayang
Lin Qiu
Tsz Ho Chan
Tianqing Fang"
Weiqi Wang'
Chunkit Chan
Dongyu Ru
Qipeng Guo
Hongming Zhang"
Yangqiu Song"
Yue Zhang'
Zheng Zhang"
The Hong Kong University of Science and Technology
WWetllkke University Ammaon AWS AI
[jchengaj, yqsong]ecse. ust. zhangyue@westlake. edu. cn
zhaz@amazon.com com
Abstract
51: The virus invades cells
Analogy-makee between narratives is crucial for human reasoning, In this paper, we evaluate the ability to identify and generate analogies by constructing sast-offits-kind large-scale story-level analogy corpus, STORYANALOGGY, which contains 24K story pairs from diverse domains with human annotations on two simi- larities from the extended Structure-Mapping Theory. We design a set of tests on STO- RYANALOGY, presenting the first evaluation story-level analogy identification and gener- ation. Interestingly, find that the analogy identification tasks are incredibly difficult not only sentence embedding models also for the recent large language models (LLMs) such as ChatGPT and LLaMa. ChatGPT, for example, only achieved around 30% accuracy in multiple-choice questions (compared to over 85% accuracy for humans). Furthermore, we observe that the data in STORYANALOOG can improve the quality of analogy generation in LLMs, where fine-tuned F-xxl model achieves comparable performance zero-shot ChatGPT.
As result, their DNA is damaged.
S2: The burgiar breaks into the house
result, the valuables inside are smashed
Figure An example of analogy hetween story $1: the invasion of cells by virus, and S2: burglar breaking into a house.
Despite its significance, there has been limited research on story analogies. One of the reasons is the lack available data and evaluation bench- marks. In contrast, the community has predom- inantly focused on word-level analogies, which involve identifying relational similarities between pairs of concepts (e.g., king man is like queen to woman) (Mikolov al., 2013; Gladkova et al., 2016; Czinczoll al., 2022).
In this work, introduce STORYANALOGY, large-scale story-level analogy corpus derived from various domains: scientific scripts, social narra- tives, word analogies, and knowledge graph triples, to facilitate the study of complex analogies. The story-level analogies we examine contain richer relational details, such as relations between enti- ties (e.g., virus, invades, cells) and between events (e.g., the virus invades cells, as a result, the virus damages DNAs).
1 Introduction
Analogy-making plays central role in human rea- soning abilities. By drawing similarities between seemingly unrelated concepts (e.g., in Figure 1, "virus" v.s. "burglar") and processes ("the virus invades cells" v.S. "the burglar breaks into the house"), we can infer that the virus infiltrates and damages cells a similar way to how a burglar breaks into house to stcal or cause harm. These seory-level analogies, which involve comparing entire narratives coherent sequences of events, enable intelligent agents to gain insights (Boden, 2009; Ding al., 2023; Bhavya al., 2023) and un- derstand complex phenomena (Webb et al., 2022).
One of the challenges in building STORYANALOGY is establishing a clear and specific way to evaluate story analogies. To address this problem, we extend the Structure- Mapping Theory (SMT; Gentner, 1983) to evaluate on longer texts. According to SMT, analogies hold (e.g., the hydrogen atom vs. the Solar System) because of the similarity in relational information (e.g., the relative motion between objects), rather than attributive information (e.g., size), between the source and target. Conversely, both types of information are similar, the source and target
'This work was done when Jiayang was an intern at Amazon AWS Al Lab. Code and data are released at: https://githun ccm//oginaway/Storyhnn.

---

Relation similantt
gap compared to human performance in terms of predicting the level of analogy between stories. We further evaluate LLMs using multiple choice ques- tions derived from the story candidates. Even the best-performing LLM still falls short of human performance by 37.7%. Furthermore, we discover that using stories in STORYANALOGY can enhance models' ability to identify and generate analogies. By employing few-shot in-context learning and finetuning on STORYANALOGY, baseline models achieve a considerable performance boost. For instance, a fine-tuned F5xxT model exhibits generation quality on par with zero-shot ChatGPT.
Anelogy
Liferal similarity
Magma goes up from the inside the planet. The magma enters volcanoe.
Food goes up from the stomach. The food enters the esophagus.
The flashlght is lurned on. Two contact strips touch one another,
These rocks become volcanoe. erupt many times.
Anomaly (dissimilarity)
Mere- Miere-appeeronce
Source:
Magma from deep earth. The magma goes into valcanos.
Entttytopic simllont
The similarity space, showing different kinds of matches in terms of the degree of relation similarity According SMT, we can classify the type of matches (Analogy, Literal similarity, Anomaly, and Mere-appearance) between the source and target story by the two similarities. The figure is an extension with story examples based on the visualization Gentner and Markman (1997).
Figure 2:
versus entity similarity
We hope that the data and evaluation settings we proposed in this study will benefit the research community in the area of story analogies.
exhibit literal similarity (e.g., the X/2 star system v.S. the Solar System). Inspired by this notion, we extend SMT to the story level ($ 2.1). We use entity and relation similarity assess the level of similarity in attributes and relations between the source and target stories, Additionally, we propose an analogy score based on these two similaritics to quantify the degree of analogy between stories. Figure 2 provides visual representation of the similarity space spanned by the two similarities.
2 STORYANALOGY
Conventional benchmarks in computational anal- ogy primarily focus on word-level analogies (e.g. word to language is like note to music). How- ever, less attention has been given to more sophisti- cated analogies. We introduce STORYANALOGY, a dataset of 24,388 pairs of stories (e.g.. "The virus invades cells and DNAs are damaged." versus "A burglar breaks into the house and smashes the valu- ables inside.") "), each annotated with two dimen- sions of similarity based to SMT.
We then collect candidate story analogies for similarity annotations. Since story analogies are scarce in free texts", we use large language models (LLMss to generate story pairs that are likely to be analogies. The stories are sourced from various domains, including scientific scripts (Dalvi et al., 2018), social commonsense stories (Mostafazadch et al., 2016), word-level analogies (Turney et al., 2003; Czinczoll et al.. 2022), and knowledge graphs (Speer et al., 2017). Next, conduct crowd-sourcing obtain similarity annotations for each candidate story pair. As a result, we create STORYANALOGY, which consists of 24K diverse story pairs, each with human annotation guided by the extended SMT.
2.1 Evaluating story analogies
To assess the degree of analogy between pair of instances, recent studies classify story pairs us- ing a set of labels. For instance, Sultan and Sha- haf (2023) use 5 labels including not-analogy. self-analogy. close-analogy, far-analogy. and sub-analogy. Nagarajah et al. (2022) use labels: shallow attribute analogy, deep attribute analogy, relational analogy, event analogy, structural analogy, and moral/purpose, However, they observed very poor agreement among annotators for most labels, which indicates a vague understanding of the task. Making comparisons across these studies are chal- lenging due to the vastly different settings.
Based on STORYANALOGY, we curate a set of tests to evaluate the analogy identification ability of models. Our findings indicate that both com- petitive encoder models (such as SimCSE (Gao et al., 2021) and OpenAl's text-embeddding-o0) and LLMs (such as ChatGPT (OpenAI, 2022) and LLaMa (Touvron et al., 2023)) have a significant
In cognitive psychology, the Structure Mapping Theory (SMT; Gentner, 1983) is well-known for its explanation of the cognitive process of mak- ing analogies between objects. SMT evaluates ob- ject comparisons from two perspectives: (a) the attributes of objects and (b) the relational struc- tures between objects. Analogies between objects
'Analogies are only present in approximately 3% of a scientific corpus (Sultan and Shahaf, 2023), and the prevalence is expected to be even lower in general texts.

---

Source story
Target story
Scores
Domain
A person grows from child into an adult. As time passes, the person experiences ongoing growth and maturation.
The stream becomes river. The river continues flow along the same path long time.
0.6
2.8
PP
They gave her the password to the website. When Jane logged in, she realized it was the wrong pass- word.
They left him the key to the entrance. When Tom went he realized it the wrong key.
1.0
2.7
ROC
Foundations are poured to support the walls and roofs buildings. The structure of the building only as strong as it's foundation.
Reasons are formulated to make theories, The con- clusions of thenries are only as dependable as their initial premises.
0.6
1.8
WA
His memory has broken fragmented pieces. He can recall flashes and images of the past, but nothing concrete or clear.
His memories remain a confused mess. Nothing holds together and what he remembers don't make sense.
2.7
3.0
WA
The student opens book and begins to read. The cat sees a mouse begins chase The knowledge gained from the book absorbed by the student.
cat honing its hunting skills through practice and repetition.
0.8
1.4 CN
Table Examples in STORYANALOGY with annotations from each domain. We report the EntSim and RelSim from crowd workers & The Domain column indicates the source of the story pairs. "PP", "ROC", "WA", and "CN" are short for "ProPara", "ROCStories", "Word Analogy", and "ConceptNet",, respectively.
occur when they have similar relational structures but dissimilar attributes (e.g., the hydrogen atom v.s. the Solar System). In contrast, literal similarity occurs when objects have both similar relational structures and attributes (e.g., the X12 star system v.s. the Solar System).
We present the established similarity space with example source and target stories in Figure 2.
Modeling the analogy score (a). We discuss pos- sible definitions of the analogy score (a). The score a should be proportional to the level of analogy be- tween a pair of stories. Defining a to be equivalent with RelSim has been adopted word analogy (Ushio et al., 2021a). However, this definition can- not distinguish analogy from literal similarity, both of them have high RelSim (Figure 2). We can alleviate this problem by introducing EntSim to the definition of according to SMT, anal- ogy happens when the RelSim between the source and target story high and the EntSim is low?. Therefore, the rest of this paper, we define a as RelSim/EntSim"
Based on SMT, we propose to compare stories by their entity and relation similarity. These mea- sures assess the degree of similarity in terms of at- tributive and relational structures, respectively. We provide necessary extensions to their definitions:
The similarity of entity and topics discussed between a pair of stories, ranging from (unrelated) to 3 (almost equivalent). This score should be high the two stories are both discussing apples and pears, even if they differ greatly in the details.
Entity similarity (EntSim
Relation similarity (RelSim D) The similarity of relational structures between a pair of stories, ranging from (very poor alignment) to 3 (align- ment). In this context, the relational structures refer to the connections between elements at dif- ferent levels. For instance, first-order relations can be regarded as the relationship between entities, such as predicates. Second-order relations, on the other hand, represent connections between higher granularity elements, such the logical connec- tion between events sentences. We encourage annotators to also consider higher-order relational similarity, such as the moral or purpose behind the stories.
2.2 Distilling story analogies from LLMs
Obtaining large number of story analogy by trieval is difficult. Evidence from Sultan and Sha- haf (2023) shows that the prevalence of analogies within a categorized dataset is around 3%. is expected that the ratio is much lower in general cor- pora. Identifying analogies by retrieving from gen- eral corpora would thus require huge human efforts, making it unrealistic to build large-scale story
A'An analogy is comparison in which relational predi- cates, but few object attributes, be mapped from base target." (Gentner, 1983)
'In practice, we compute it hy RelSim/ (1+EntSim) to ensure numerical stability.

---

analogy collection in this way. Recently observa- tions suggest that LLMs are capable of understand- ing and predicting analogies for problem-solvine (Webb et al., 2022), cross-domane creativities (Ding et al., 2023), and generating explanations for word analogies (Bhavya et al., 2022). In addition to these findings. we discover that LLMs can generate high- quality story analogies (i.e., with more than half generations being analogies). Here, we introduce the pipeline for generating story analogies. The generated analogies are further annotated by crowd annotators for verification. (Details are in $ A.I.) Curating seed examples. The first step is to cu- rate seed set of story analogies. We ask experts from our team write story analogies. To ensure diversity, the experts are required consider mul- tiple domains and are allowed to search in corpora or on the Internet. They then determine whether these story pairs are indeed analogies Examples that are not considered analogies are removed from the gold set. As result, we obtained a total of 28 story analogy examples, each containing a pair of stories and the corresponding entities.
the target story.
Generating from word pairs. Given a word analogy pair (c.g. "word", "language" and "note", "mu- sic"), together with source-target analogies with the corresponding entities from seed examples, an LLM is prompted to generate both the source and target stories.
2.3 Annotation
To evaluate each candidate story pair under the ex- tended SMT, we conduct crowd annotations on Amazon Mechanical Turk". We recruit crowd workers to annotate the entity and relation similari- ties for the collected pairs. In addition, workers are required to label an instance as "poor quality" if they find the generated content broken or toxic. The annotation consists of the following two rounds: (i) Qualification round. We first annotate 80 can- didate story pairs (20 from each domain) to curate qualification set. Three domain experts from our team are asked to read through the annotation in- struction and independently annotate EntSim and RelSim for these pairs. The Spearman's p between each annotator's prediction with the average scores of the others ranges from 93% to 96% on EntSim, and from 89% to 95% on RelSim.
Source data. To guarantee the coverage of topics, we sample from corpora of four domains to gener- ate stories, including (1) scientific scripts ProPara (Dalvi et al., 2018), (2) social commonsense stories ROCStories (Mostafazadch et al., 2016), (3) word analogy evaluation sets SAT (Turney al., 2003), U2 and U4%, and SCAN (Czinczoll al., 2022).. and (4) the commonsense KG ConceptNet" (Speer et al., 2017). Note that, source data (1) and (2) consist of stories, while (3) and (4) consist of word pairs.
We invite crowd workers who have 290% his- tory approval rates and have 21K HITs approved to attend the qualification. Workers whose predic- tions achieve 20% Spearman's p with the average scores from three experts pass the qualification. As a result, 158 and 80 workers passed the qualifica- tion for EntSim and RelSim, respectively. (ii) Main round. Qualified crowd workers are in- vited to attend the main round annotations. We assign 5 different annotators give predictions for each similarity of story pair. To guarantee the annotation quality, we follow the annotation setting in (Agirre et al., 2012). We split the main round into multiple mini-rounds, each with 1K-2K candidate pairs. After each mini-round, we filter out and disqualify workers who do not show sig- nificant correlations with the average scores of the others. They are paid more than what is required by the local wage law. In addition, experts from our team manually check the quality of annotations and write feedback to workers correspondingly.
Generating story candidates. Using the seed ex- amples and source data, we prompt LLMs to gen- erate analogies. Due to the different formats of the source data, the story pairs are generated using two different paradigms (Details are in $ A.1.):
Generating from story pairs. Given source story and several source-target story pairs sampled from the seed examples, we prompt an LLM to generate
'After manual inspection of all word analogy datasets, we do not include classic datasets such as Google (Mikolov et al., 2013), where the relations between words are relatively casier syntactic or shallow semantic relations, such as ("similar: similarly", "rare: rarely").
'ht tps: /ttp://engl/shffeee wwwpllisshorrrrrryee wwaeeeerrg//occcee Analogies. hmm
The generated contents sometimes contain hallu- cinations or toxic contents. We filter out story pairs labeled as "poor quality" by more than 10% an-
"We consider entity pairs in triples that share the same relations from ConceptNet. httpss//huggingfaat co/ http://ebbbrr/aalgyyoooooo 'OpenAl's dext-davinci-00. is used in the generation.
Phttps://www http///wwwwtrukccooo

---

the second evaluation, we reframe our dataset multiple-choiee questions and evaluate LLMs on this set ($ 3.2).
1.2
0.5
Source:
0.0
Conceptiet
ProFara
3.1
Correlation with the analogy score
1.0
Wardknalegy
Similar to the STS-style evaluation (Agirre et al., 2012), we assess whether models can predict analogy scores based on embeddings (for encoder models) or by generation (for LLMs). We use model to predict the similarity f(.:-) for two stories. For encoder models, f(s1, Encoder(s2)).
0.3
0.1
Figure 3: Distributions of EntSim and RelSim on four data domains in STORYANALOGY. Notably, the distri- butions of EntSim and RelSim on ROCStories tend skew towards higher values. This could be attributed to the fact that stories from this source primarily revolve human-focused social narratives.
Cosine (Encoder LLMs, we prompt them to predict the EntSim and RelSim for the two stories. Finally, Spearman's 's correlations between the predicted similarity and the respective scores are reported.
For
notators, which accounts for 142 instances. For each story pair, adopt the average scores from workers the predicted EntSim and RelSim.
Setups. We consider both encoder models and LLMs as baselines. Details are in $ A.2.
The encoder models we evaluate include RoBERTa (Liu et al., 2019), SimCSE (Gao et al., 2021). OpenAl-ada
2.4 Analysis of STORYANALOGY
-naatt-e-addddddda-----a--- Discourse Marker Representation (DMR) (Ru al., 2023), ReIBERT (Ushio et al., 2021b), and GloVe embeddings (Pennington et al., 2014) on nouns, verbs, or all words In addition to the unsu- pervised encoder models, we also fine-tune two models the training set: regression model, RoBERTa-Reg, which has multilayer percep- tron on top of the RoBERTa model that predicts EntSim and RelSim, and a contrastive learning- based model, ROBERTA-CL, which uses a con- trastive learning objective to optimize its represen- tations.
To assess inter-annotator agreement, we randomly sampled IK instances with 3 independent annota- tions from our dataset. The Fleiss's kappa (Fleiss, 1971) on the binarized annotations of EntSim are 47%, and 42% on RelSim, indicating moderate agreement among annotators. In addition, we ad- ditionally obtained expert annotations on 200 ran- domly sampled instances. The averaged Spear- man's correlation between crowd and expert anno- tations on EntSim and RelSim 64.7% % and 69.9%, respectively.
The final dataset consists of 24,388 story pairs four domains: ProPara (6.9K), ROCStories (4.9K), Word-Analogy (7.5K), and ConceptNet (5.0K). Sto- ries in STORYANAL.OGY have 19.94 tokens on av- erage. The distributions of EntSim and RelSim are presented in Figure 3. We randomly select 500 instances from each domain as the test set, and another 500 instances from each domain as the validation set. Examples of STORYANALOGY are shown in Table 1.
For LLMs, we test FlanT5 (Chung et al., 2022), LLaMa (Touvron al., 2023), ChatGPT (OpenAI, 2022), and GPT-3.5 (text-davinci-003) Each model input composed of three parts: the in- structions, which give explanations to the similar- ity scores; N examples, and the query story pair. We evaluate models with two instructions (short and long, where short instructions only contain the labels, and long instructions additionally have label definitions), and N is set to 0, 1, or
3 Story Analogy Identification
We begin by assessing the ability of models to iden- tify story analogies using two different setups. The first evaluation setup is similar to Semantic Tex- tual Similarity (STS) tasks (Agirre al., 2012), where we calculate the Spearman's 's correlation be- tween models' predicted similarity and the analogy scores (a) derived from annotations (8 3. For
Results. The overall evaluation results are pre- sented in Table 2. Generally, the models perform relatively poorly on the analogy score a, indicat- ing that there is still room for improvement on STORYANALOGY.
We use Stanza (Qi et al., 2020) to conduct part-of-speech tagging for words.

---

ProPara
ROCStories
Word-Analogy
ConceptNet
Mean
E
R
0
R
R
F
R
E
R
a
E
Random
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Human
54.9
64.6
67.9
82,9
70.1
55.2
67.1
69.4
58.3
53.7
75.5
68.5
64.7
69.9
62.5
Encoder models
RoBERTa
45.2
41.9
6.2
20.6
22.2
7.6
34.8
24.5
0.9
34.9
28.8
9.8
33.9
29.4
SimCSE
48.0
38.4
1.8
14.4
12.7
2.6
43.2
26.8
-2.0
30.7
21.2
3.7
34.1
24.8
1.5
OpenAI-ada
52.8
43.9
3,4
22.3
41.3
24.0
-3.8
32.3
17.8
-1.2
37.2
DMR
34.8
42.0
12.6
20.1
35.0
20.1
17.3
18.7
7.3
21.9
19.1
5.5
23.5
28.7
11.4
ReIBERT
37.9
38.8
7.5
15.6
20.6
9.1
28.6
15.5
-3.6
26.6
24.7
8.2
27.2
24.9
5.3
Glo Ve-Nou
35.2
18.5
-7.8
9.4
6.9
2.5
29.8
14.2
-5.6
27.7
13.0
-2.2
25.5
13.2
-3.3
Veere-Verb
27.3
44.8
21.3
22.7
34.2
17.3
9.6
7.0
1.3
13.0
1.0
-7.2
18.1
21.7
8.2
GloVe-All
36.3
29.0
-1.0
28.7
27.2
26.1
12.3
-5.1
18.6
-7.9
27.4
18.0
-2.3
LLMs
FlanTT-xx1
41.9
21.8
4.8
9.4
-8.7
-2.4
40.0
27.5
8.3
37.7
26.7
8.1
32.3
16.8
4.7
LLaMa-65B
16.8
3.8
-1.3
0.4
-10.2
-7.7
31.6
25.3
9.5
22.1
4.2
15.6
10.3
1.2
GPT-3.5
24.1
11.4
-6.4
-3.2
-2.8
-6.4
34.2
26.9
8.6
28.8
30.0
7.9
21.0
16.4
0,9
ChatGPT
26.9
11.9
-233
1.7
-4.4
-5.0
46.6
31.4
3.4
32.1
36.4
13.1
26.8
Finetuned models
RoBERTa-Ras
38.5
34.8
16.6
14.5
26.2
12.0
20.1
28.8
19.4
23.8
32.0
20.1
24.2
30.5
17.0
ROBERTa-CI. 25.7
53.9
35.2
29.1
47.2
28.3
33.8
40.9
21.1
26.0
30.8
15.3
28.7
43.2
25.0
Table 2: STS-style evaluation on different domains of STORYANALOGY. The values represent the Spearman's s correlation (%) between the model prediction and scores from dataset (E, and a)). Here, E, R, and a correspond to EntSim, RelSim, and the analogy score RelSim/EntSim, respectively. The LLM performance is evaluated under the "long instruction+3-shen setting.
Verb (predicates), correlate better with the anal- ogy score a. (3) Finetuning improves models' analogy identification ability. The finetuned mod- els, RoBERTa- Reg and RoBERTa-CC CL, are the top- performing models that significantly outperform all the other baselines on a. (4) Generally, LLMs do not perform well on the analogy score a. As shown in Figure 4, most LLMs can benefit from longer instructions as the extra definitions help in under- standing the scores. Moreover, find that despite its size, FlanT5-xxl is one of the rest-performing LLMs in terms of predicting EntSim and RelSim.
Long
Short
Speaimnns comelation (%6)
FlanTS-zd
LLMMa 658
ChaiPt
GPT-33
Fann5-ead
Laata -000
ChaiPi
GPT-25 25
Figure 4: Spearman's 's p (%) of LLMs, averaged across data domains. Here F, R, and R/E indicate EntSim, RelSim, and RelSim/Enttimmee (a).
3.2 Multiple choice evaluation
We construct a multiple-choice evaluation set using the annotated story pairs. First, gather story pairs with EntSim 1.0 and RelSim 2.0. For each target story, we choose 3 negative choices to form the candidates. Out of these, two (easy) negative choices are randomly selected, while one (hard) negative example chosen by retrieving sto- ries with high nounal similarity (measured by the cosine similarity of the nounal GloVe embeddings) and < 50% token overlap. An example question is provided in Table 3. To assess human performance, conduct human annotations.
We have the following observations: (1) Simi- larities from state-of-the-e sentence embedding models are not good indicators for story analogy. Encoders such as RoBERTa, SimCSE, and OpenAI- ada show relatively good correlation with EntSim and RelSim, but they perform poorly on the anal- ogy score This suggests that their embeddings are suitable for literal similarity retrieval but not analogy retrieval. (2) Relational feature-aware models are better analogy identification. Ad- ditionally, we find that encoder models aware of relation information, such as DMR (discourse rela- tion), RelBERT (inter-wort word relation), and GloVe-
We assess LLMSs on multiple-choice questions. Each model input consists an instruction, N ex- amples of multiple-choice questions, and the query

---

Which candidate story is the best creative analogy for the source story?
Target
Hard
Easy
Question:
Random
25.0
50.0
Carbanic acid rainwater breaks down rock. Plants gnow rock.
(Sultan and Shabaf, 2023)
44.9
17.8
37.2
Source:
FlanTS-xxl
45.4
37.2
17.4
LLaMa-65B
28.6
59.7
11.7
(0)
Plants and animals grow and repeoduce. The population size gets larger and larger.
ChatGPT
32.4
59.5
8.1
GPT-3.5
37.1
55.8
7.1
(I) Recyclables are placed in centralized container for the house. Recyclables are picked up hy a recycling company.
Table 5: Breakdown ratio (%) of model predictions: The table presents the percentage of different types of choices selected. "Target" refers to the ground larget, which is the analogous story. "Hard" and "Tasy" refer to the negative examples sampled using nounal similarity and random sampling, respectively.
(2) Salty ocean water crodes metal. Corals thrive on motal.
The roots of the growing plants start to break up the mek.
(3)
The plant acids dissolve the rock.
Answer:
(2)
Table 3: An example of the multiple choice question. The goal is to select candidate story that is the hest
analogy for the source story.
atives they select is less than 20%, whereas random chance would be 50%. Furthermore, more power- ful LLMs like GPT-3.5 are better at this judgement compared to LLaMa and FlanT5-xxl. (2) LLMs can be easily distracted by hard negatives, as they often have a similar or higher chance of selecting hard negative choices instead of the targets. This suggests that the models prioritize surface similar- ity over structural similarity, despite the latter being more important in identifying analogies." (3) In comparison, the baseline model from (Sultan and Shahaf, 2023) more resilient against hard nega- tive distractions. This likely due to its framework design, which captures the structural similarity be- tween stories by clustering entities and finding the mappings between clusters.
N-shot
Question template
Model
Model
A
B
FlanT5--xd
45.0
46.3
FlanT--xal
41.2
47.1
65B
27.7
28.6
29.5
LLMMa 65B
29.6
26.0
30.2
CbaGPT
3588
29.2
32.3
ChaGPT
30.1
33,3
33.9
GPT-3.5
44.2
34.1
33.1
GPT-35
33.7
33.6
44.0
Table 4: Multiple choice evaluation results. Each value represents the average accuracy (%) across different number of demonstrations (left) across three question templates (right). The random and human performance are 25% and 85.7%, respectively.
multiple-choice question. We evaluate the models using three different instructions, such as "Which candidate story is the best creative analogy for the source story?", where N can be 0, 1, or 3. As baseline, we obtain the per- formance of the analogy retrieval model in (Sultan and Shahaf, 2023) on our multiple choice questions, which achieves accuracy of 44.9%.
Story Analogy Generation
We examine whether the dataset STORYANALOGY can enhance the ability of analogy generation. We evaluate FlanTS (Chung et al., 2022), LLaMa (Tou- vron et al., 2023), ChatGPT (OpenAI, 2022), and GPT-3.5 in zero-shot and few-shot settings using 40 source stories from the test set. To explore the po- tential of smaller models in generating high-quality analogies, we fine-tuned FlanT5-xl (3B parame- ters) and FlanT5-xxl (IIB parameters) using the same template.
Results. The results are presented in Table Inter- estingly, while annotators can answer the questions correctly at an accuracy of 85.7%, LLMs struggle on selecting the most analogous story (the averaged is merely 37.1 17)..)
accuracy for Increasing the number of demonstrations does not show consistent benefits to model prediction. Also, we find that explicitly instructing models choose the "creative analogy" ($ A.3, question template B) or to provide a definition of SMT when explaining analogies (template C) yields better performance compared to simply asking models to select the best analogy (template A).
A crowd annotation is conducted to evaluate the quality of the generated stories from the models mentioned above. Workers are provided with a source story and its corresponding generated target story. They are then asked to assess the following: (1) Whether the target story is an analogy for the
We present the breakdown ratio of the percent- age of types of choices selected in Table 5. We have the following observations: (1) LLMS can can differentiate between randomly sampled easy nega- tives and other choices. The proportion of easy neg-
"This phenomenon was also observed in visual analogies (Bitton et al., 2023), where they found that that models can solve visual analogies well when the distractors are random, hut struggle with difficult distractors.

---

Setting
Model
Analogy
Generation quality
Novelty
Plausibiliity
Source
projecrile is affected by gravity. falls picks up spoed.
FlanTS-xl
52.5
48.3
92.5
FlanTS-xxl
46.7
49.2
92.5
Just as a sled sliding down a steep hill gains momentum as accelerates, so does projectile fails under the force of gravity.
7ero
I.L.Ma-65B
38.3
39.2
93.33
ChaGGT
ChatGPT
70.0
72.5
90.8
GPT-3.5
75.8
81.7
87.5
An Olympic runner is cumming a midklle distance race. GPT-3.5
FlanTS-xl
48.3
50.0
91.7
A projectile is affected by gravity. It falls and picks up speed.
65B
FlanT5-xxl
40.0
43.3
85.0
Few
LLaMa-65B
66.7
66.7
92.5
FlanTS-xl
meker affected gravity. falls and picks up speed.
CharGPT
78.3
83,3
86.7
affected gravity. It picks speed and goes fister.
GPT-3.5
77.5
79.2
88.3
FlarTS-xxl
Tuned
FlanT55xxl
65.8
79.2
88.3
Frwe-dox
F5xnT5-xxl
72.5
81.7
86.7
seed planted in fertile soil. It takes and grows, flourishing into healthy plant
CharGT
Table The crowd-annotated generation quality (%) in terms of (1) Whether the target story is considered an analogy to the source; (2) Novelty of the target story; (3) Plausibility of the generations.
An alpha partiele is buffeted by the atmosphere. It spirals eventually finds the ground.
GPT-3.5
projectile is affected by gravity. falls and picks up speed.
1.LMM-668
FlanTS-xl
A rocket is affected by gravily. It falls and picks up speed.
source (as opposed to being a literal similarity or something else); (2) whether the target story is novel compared to the source; and (3) whether the target is plausible (More details can be found in $ A.4). The average scores from three annotators are reported in Table 6. Example generations are shown in Table 7.
car affected by gravity. It accelerates and picks up speed.
FlaiT5-xal
Timed!
FlanTS-xl
A meterr is propelledd by tug. It moves and finds its way.
An idea is planted by brainstorm. It takes shope and comes
Flai5-xxl
fruition.
Table 7: Examples showing the source story and model generations under zero-shot, few-shot, and finetuning settings.
Under the zero-shot setting, we observe that FlanT55 and LLaMa struggle to generate meaning- ful analogies. They often tend to repeat patters from the source stories (e.g., only replacing one word). In contrast, ChatGPT and GPT-3.5 produce more flexible stories that are frequently considered as analogies and novel.
the development of pretrained language models (PLMs) such as BERT (Devlin et al., 201 8), there have been works utilizing PLMs to solve word analogies by LM perplexity (Ushio et al., 2021a), pretrain relational embedding on certain prompt templates (Ushio et al., 2021b), or use word analo- gies as latent restriction to implicitly probe rela- tional knowledge (Rezaee and Camacho-Collados, 2022).
Stories in STORYANALOGY can help models generate better analogies. With a few demon- strations, we observe significant improvement in the generation quality of LLaMa (+28.4% and +27.5%). Moderate improvement ChatGPT and GPT-3.5 is also observed. Notably, finetun- ing smaller LMs enhanced their generation quality. The finetuned F-anT5-xxl model performs better than the zero-shot ChatGPT and is comparable to the few-shot ChatGPT and GPT-3.5, despite hav- ing fewer parameters. Furthermore, while mod- els become more creative through finetuning and in-context learning, their generation plausibility decreases, indicating an increase in hallucination.
In this line of work, a typical evaluation setting is ranking word pairs based on their relational sim- ilarity with the source pair (Mikolov et al., 2013; Czinczoll al., 2022). For instance, given word pair A:B, the aim is to select target pair C:D such that the relation between C and is the most simi- lar to A:B among all candidates. This is similar to our multiplc-choice evaluation setting.
In comparison, only a handful of research has been done in sentence or paragraph-level analogy: Analogous text retrieval. Built on the famous structure mapping theory, SME (Falkenhainer al., 1989) and LRME (Turney, 2008) model the anal- ogy retrieval problem as an entity-mapping prob- lem. They then solve this problem through web mining. Sultan and Shahaf (2023) develops a QA- SRL based analogy retrieval method to conduct
5 Related Work
Word-level analogy. One of the famous works on word-level computational analogy was (Mikolov et al., 2013), where they found that word analogies can be predicted by word vector offsets. For in- stance, King- - Mon + Queen=Queen. With

---

entity mapping. However, these works evaluate their methods by annotating the precision of the top-ranked results, leaving no large-scale analogy evaluation benchmarks to date.
gies pairs annotated on two similarities under the extended SMT. To assess the analogy identifica- tion and generation capabilities of various mod- els, we have devised a series of tests based on STORYANALOGY. The experimental findings indi- cate that current encoder models and LLMs still fall short of human performance in analogy identifica- tion. Additionally, we demonstrate that generative models can greatly benefit from our dataset.
Analogy generation. Recently, there have been attempts at pretraining or prompting LMs for anal- ogy generation. Bhavya al. (2022); Webb al. (2022) evaluated LLMs' ability on solving word analogy tasks, where they found that large language models such as GPT-3.5 can surpass human perfor- mance on certain word analogy tasks. Ding et al. (2023) evaluated LLMs' creativity in terms of cross- domain analogies. Bhavya et al. (2022); Chen et al. (2022a) evaluated LMs' ability on generating ex- planations for word analogies. Bhavya et al. (2023) proposed novel analogy mining framework based on generation.
Limitations
We attempted to ensure dataset coverage by uti- lizing seed data from various sources. However, there are still specific domains that we were un- able to include, such as biomedical stories or aca- demic articles. We can extend the annotation to these domains using the annotation framework and evaluation metries mentioned in this paper. Ad- ditionally, we have explored applications such analogy identification (Section 3) and generation (Section 4). The potential of STORYANALOGY to be applied for creativity generation tasks (such as poetry, lyries and humor generation) has not been fully investigated. Further development on other sources and applications is left future work.
Analogy benchmarks. There are many word-level analogy datasets. Google (Mikolov et al., 2013), BATS (Gladkova et al., 2016) contain relatively casier syntactic shallow semantic relations. In contrast, U2 and U4, and Czinczoll et al. (2022) include examples with relatively more abstract re- lations. To the best of our knowledge, there is no large-scale story-level analogy data or resources as of the time of writing. The only related works here are (Li and Zhao, 2021; Zhu and de Melo, 2020), which transform word analogy pairs into sentence pairs with few templates. Nagarajah et al. (2022) tried annotate tiny scale story analogy bench- mark based on fables, but they failed achieve. Wijesiriwardene et (2023) re-organized sentence relation datasets, where they viewed such relations (e.g., entailment, negation) are analogy, which is fundamentally different from our settings.
Ethics Statement
The generated knowledge in STORYANALOGY have been carefully evaluated by crowd annota- tors to remove any possible toxic or counterfactual content. We set the threshold to be as low as 10%, such that any annotator's 's labeling an instance as toxic will lead to its removal. 142 instances are removed during this process. We conformed to rec- ognized privacy practices and rigorously followed the data usage policy. We declare that all authors of this paper acknowledge the ACM Code of Ethies and honor the code of conduct.
Analogy in other domains. In addition to analo- gies on word pairs and stories, there have been related studies on other topics. Hope al. (2017) contribute a method for analogy mining over prod- uets. Chan et al. (2018) mine analogies from re- search papers with respect to their background, pur- pose, mechanism, and findings. Gilon et al. (2018) develop search engine for expressing and abstract- ing specific design needs. Recently, Bitton et al. (2023) propose visual analogies dataset, VASR, where they found that models struggle to find out analogies when given carefully chosen distractors.
Acknowledgements
The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 19 and R6021-20) and the GRF (16211520 and 16205322) from RGC of Hong Kong. We also thank the support from the UGC Research Matching Grants (RMGS20EG01-D. RMGS20CR11. 1, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR0S, RMGS23EG08).
6 Conclusion
We introduce STORYANALOGY, a multi-domain seory-level analogy corpus with 24K story analo-

---

References
Jiangiie Chen, Rui Xu, Ziquan Fu, Wei Shi, Zhongqiao Li, Xinbo Zhang, Changzhi Sun, Lei Li, Yanghua Xiao, and Hao Zhou. 2022a. E--arr A benchmark for rationalizing natural language analogical reasoning. In Findings of the Association for Computational Linguistics: ACI. 2022, pages 3941-3955.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Conzalez-Agirre. 2012. Semeval-2012 task pilot on semantic textual similarity. In SEM 2012: The First Joins Conference on Lexical and Compu- tational Semantics- Voouuee I: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semuntic Evaluation (SemEval 2012), pages 385-
Yi Chen, Jiayang Cheng, Haiyun Jiang, Lemao Liu, Haisong Zhang, Shuming Shi, and Ruifeng Xu. 2022b. Learning from sibling mentions with scal- able graph inference in fine-grai grained entity typing, In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2076-2087.
393.
Bhavya Bhavya, Jinjun Xiong, and Chengxiang Zhai. 2022. Analogy generation by prompting large lan- guage models: A case study of instructgpt. arXiv preprint 100.:22210.0116.
Jiayang Cheng. Haiyun Jiang, Deqing Yang. and based
Yanghua Xiao. 2021. A framework for relation extraction validation. CoRR. Abs/2104.02934,
Bhavya Bhavya, Jinjun Xiong, and Chengxiang Zhai. 2023, Cam: A large language model-based creative analogy mining framework. In Proceedings of the ACM Web Conference 2023, pages 3903-3914,
Wei Chiang, Zhuohan Li, Zi Lin. Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Yonatan Bitton, Ron Yosef, Eliyahu Strugo, Dafna Sha- haf, Roy Schwartz, and Gabriel Stanovsky. 2023. Vasr: Visual analogies of situation recognition. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 37, pages 241-249.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling rnstruciionfinetume language models. arXiv prepriat arXiv:2210. 1416.
Margaret A Boden. 2009, Computer models of creativ- ity. Al Magazine, 30(3):23-23.
Chunkit Chan and Tsz Ho Chan. 2023. Discourse-aware prompt for argument impact classification. In Pro-
Li Cui, Deqing Yang. Jiayang Cheng, and Yanghua Xiao. 2021a. Incorporating syntactic information into rela- tion representations for enhanced relation extraction. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 416-428. Springer.
ceedings of the 15th International Conference on Machine Learning and Computing. ICMLC 2023, Zhuhai, China, February 17-20, 2023, pages 165- 171. ACM.
Chunkit Chan, Jiayang Cheng, Weiqi Wang, Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song. 2023a. Chatgpt evaluation on sentence level rela- tions: A focus on temporal, causal, and discourse relations. CoRR, abs/2304,14822 14827..
Li Cui, Deqing Yang. Jiaxin Yu, Chengwei Hu, Jiayang Cheng. Jingjie Yi, and Yanghua Xiao. 2021b. Refin- ing sample embeddings with relation prototypes to enhance continual relation extraction. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguisties and the I 1Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 232-243.
Chunkit Chan, Xin Liu, Tsz Ho Chan, Jiayang Cheng. Yangqiu Song, Ginny Y. Wong, and Si- mon See. 2023b. Self-consistent narrative prompts on abductive natural language inference. CoRR, abs/2309.0830
Tamara Czinczoll, Helen Yannakoudakis, Pushkar Mishra, and Ekaaerina Shutova. 2022. Scientific and creative analogies in pretrained language models. arXiv preprint arXiv:2211.1526.
Chunkit Chan, Xin Liu, Jiayang Cheng. Zihan Li, Yangqiu Song, Ginny Y. Wong, and Simon See. 2023c. Discoprompt: Path prediction prompt tun- ing for implicit discourse relation recognition. In Findings of the Association for Computational Lin- guistics: ACL. 2023, Toronto, Canada, July 9-14, 2023, pages 35-57. Association for Computational Linguistics.
Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: challenge dataset and models for process paragraph comprehension. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages 1595-1604.
Joel Chan, Joseph Chee Chang, Tom Hope, Dafna Sha- haf, and Aniket Kittur. 2018. Solvent: A mixed initia- tive system for finding analogies between research pa- pers. Proceedings of the ACM on Human- Human-Computer Computer Interaction, 21C
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 8. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint

---

Zijian Ding, Arvind Srinivasan, Stephen MacNeil, and Joel Chan. 2023. Fluid transformers and creative analogies: Exploring large language models' capacity for augmenting cross-domain analogical creativity. arXiv preprint urXiv:2302. 128.22..........
Haoran Li, Yulin Chen, Jinglong Luo, Yan Kang, Xiao- jin Zhang. Qi Hu, Chunkit Chan, and Yangqiu Song. 2023a. Privacy in large language models: Attacks, defenses and future directions.
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. 2023b. Multi-step jailbreaking pri- vacy attacks on chatgpt. CoRR, abs/2304.05197.
Brian Falkenhainer, Kenneth D Forbus, and Dedre Gen- tmer. 1989. The structure- stautuur-mapping engine: Algo- rithm and examples. Artificial intelligence. 41(1):1- 63.
Yian Li and Hai Zhao. 2021. Pre-training universal language representation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the I 11th International Joint Confer- ence Natural Language Processing (Volume 1: Long Papers), pages 5122-5133. 5133.
Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6910.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint
Dedre Gentner. 1983. Structure-mappe mapping: A theoretical framework for analogy. Cognitive science, 1(2):155- 170.
Tomas Mikolov, llya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems, 26.
Dedre Gentner and Arthur B Markman. 1997. Struc- ture mapping in analogy and similarity. American psychologist, 52(1):45.
Karni Gilon, Joel Chan, Felicia Y Ng. Hila Liifshitz- Assaf, Aniket Kiuur, and Dafna Shahaf. 2018. Anal- ogy mining for specific design needs. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1-11.
Haoran Mo, Edgar Simo-Serra, Chengying Gao, Changqing Zou, and Ruomei Wang. 2021. General virtual sketching framework for vector line art. ACM Trans. Graph., 40(4).
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen, 2016. A cor- pus and evaluation framework for deeper under- standing of commonsense stories. arXiv preprint 16xiv.160..01696
Anna Gladkova, Aleksandr Drozd, and Satoshi Mat- suoka. 2016. maalogy-bbased detection of morpholog- ical and semantic relations with word embeddings: what works and what doesn't. 't. In Proceedings of the NAACL Student Research Workshop, pages 8-15.
Tom Hope, Joel Chan, Aniket Kittur, and Dafna Sha- haf. 2017. Accelerating innovation through analogy mining. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 235-243.
Thiloshon Nagarajah, Filip Ilievski, and Jay Pujara. 2022. Understanding narratives through dimensions of analogy. arXiv preprint 07Xiv:2206.07167.
OpenAl. 2022. Introducing chatgpt.
Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. Lion: Adversarial distillation of closed-source large language model. CoRR, abs/2305. 12870.
OpenAL. 2023. abs/2303.08877.
GPT-4 technical report. CoRR,
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language P Processing (EMNLP), pages 1532-1543.
Yuxin Jiang, Linhan Zhang, and Wei Wang. 2022. Im- proved universal sentence embeddings with prompt- based contrastive learning and energy-based learning. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3021-3035. Association for Computational Linguistics.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. arXiv preprint arXiv:2003.07882.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scou Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Abs/2001.08366
Partha Pratim Ray. 2023. Chatgpt: A comprehensive review background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physice Systems.
CoRR,

---

Kiamehr Rezaee and Jose Samacho-Collados. 2022. Probing relational knowledge in language models via word analogies. In Findings of the Association for Computational Linguisties: EMNLP 2022. pages 3930-3936.
Thilini Wijesiriwardene, Ruwan Wickramarachchi, Bi- mal G. Gajera, Shreeyash Mukul Gowaikar, Chan- dan Gupla, Aman Chadha, Aishwarya Naresh Re- ganti, Amit Sheth, and Amitava Das. 2023. ANA- LOGICAL- A novel benchmark for long text anal- ogy evaluation in large language models. CoRR, abs/2305.05050.
Dongyu Ru, Lin Qiu, Xipeng Qiu, Yue Zhang, and Zheng Zhang. 2023. Distributed marker representa- tion for ambiguous discourse markers and entangled relations. CoRR, abs/2306. 10658..
Liu Zhenyuan, Michal Piovarei, Christian Hafner, Raphaël Charrondière, and Bernd Bickel. 2023. Directionality design of embroidery patterns. In Computer Graphics Forum, volume 42, pages 397- 409. Wiley Online Library.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. An open multilingual graph gen- eral knowledge. In Proceedings of the AAAI confer- ence artificial intelligence, volume 31.
Conceptnet 5.3:
Xunjie Zhu and Gerard de Melo. 2020. Sentence analo- gies: I Linguistic regularities in sentence embeddings. In Proceedings of the 28th International Conference on Compulational Linguistics, pages 3389-3400.
Oren Sultan and Dafna Shahaf. 2023. Life is circus and we are the clowns: Automatically finding analo- gies between situations and processes.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instructionnfollowviin Ilama model. https:// gi thub. wowm/ttttttaaaaaaaacaaa httpe//aaaasaaadaalllpaoaaaaa
A Appendix
A.1 Details in creating STORYANALOGY.
A.1.1 Details in generating candidates.
We prompt text-davinci-O0 model to collect the story analogy candidates. The prompt template
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint Ait.:2302.139771
is
The in-context learning seed examples are presented in Table 8. In addition the golden story analogies, we also curated the cor- responding keyword pairs with regard to each story pair. This keyword pairs are useful for prompting to generate candidate stories from the Word Anal- ogy and ConceptNet inputs, where their input data format are word-pairs (e.g. word: language :: note: music).
Demonstrations.
Peter D Turney. 2008.
The latent relation mapping engine: Algorithm and experiments. Journal of Arti- ficial Intelligence Research, 33:615-655.
Peter D Turney, Michael L Littman, Jeffrey Bigham, and Victor Shnayder. 2003. Combining independent mod- ules in lexical multiple-choice problems. In RANLP, pages 101-110.
To construct list of demonstrations for each data source, we ask experts to construct set of analogous story pairs by web-searching and revis- ing the results. Then, to ensure the diversity of the analogy, we make a list of orthogonal topics in each dataset, and randomly sampled demonstra- tions from these subtopics every time we construct prompt.
Asahi Ushio, Luis Espinosa Anke, Steven Schockaert, and Jose Camacho-Collados. 2.21a. Bert is to nlp what alexnet is to ev: Can pre-trained language mod- els identify analogies? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3609-3624.
Asahi Ushio, Jose Himacho-Collador and Steven Schockaert. 202 Distilling relation embeddings from pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9044-9062.
Prompt templates for "generating from story pairs". The template for the demonstration is: "Example: (s) story (i)))Ann analogy for story (1) be: \n) (target story(i))"
can
Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xian- gru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. 2023. Survey on factuality in large language models: Knowledge, retrieval and domain-speciffcity arXiv preprint arxivv2310.07521.
It is concatenated with prompt at the end: "Example: \n(1) story)\nAn analogy for story (1) can be:"
Prompt templates for "generating from word pairs". The prompt template for generating from word pairs is: "Write a group of 2-sentence
Taylor Webb, Keith J Holyoak, and Hongjing Lu. 2022. Emergent analogical reasoning large language models. arXiv preprini 12Xiv:2212.09190

---

Source story
Target story
person grows from a child into an adult. As time passes, the person experiences ongoing growth and maturation.
The stream becomes a river. The river continues flow along the same path for long time. ENTITY: stream, river
ENTITY: child, adult
Magma rises from deep in the earth. The magma gocs into volcanos.
Food goes up from the stomach. The food enters the csophagus.
ENTITY: magma, volcanos
ENTITY: food, esophagus
The plasma membrane encloses the animal cell. It controls the movement of materials into and out of the cell.
Security guards monitor the doors of the factory. They manage the entry and exit of personnel to and from the factory.
ENTITY: plasma membrane, cell
ENTITY: security guard, factory
The tadpole begins storing food in the tail. The tadpole develops hind legs and lives off food stored in the it's tail.
person saves money in savings account. The person relies the saved funds meet future fi- nancial obligations and sustain their lifestyle. ENTITY: human, money
ENTITY: tadpole, food
The sediment near the bottom compressed by the weight of newer sediment. The sediment becomes sedimentary rock as it is pushed together by the heavy weight. ENTITY: sediment, sedimentary rock
person's idcas and belicfs are shaped by their experiences and influences. The person's thoughts
and opinions become more solidified and defined as they are influenced by outside forces. ENTITY: belief, solidified belief
Morgan enjoyed long walks on the beach. She and her boyfriend decided to go for long walk.
Lenny liked to climb trees. He embarked on tree- climbing expedition in the woods. ENTITY: woods, clibbing trees
ENTITY:
beach, walking
He got call from his girlfriend, asking where he Frank suddenly realized he had date that night. ENTITY: call, date
She received notification on her phone, reminding her of an upcoming meeting. Jane suddenly remem- bered there was important presentation to give. ENTITY: notification, presentation
He was terrified of the upcoming job interview. Due to oversleeping on the day of the interview; he missed the appointment and thus avoided the stress. ENTITY: job interview, stress
She was petrified and prayed to get out of the test. On the last day lessons, the bus broke down and she was spared. ENTITY: test, fear
Having recently started relationship, she is grap- pling with anxiety. She becomes highly anxious whenever they have a disagreement. ENTITY: relationship, anxious
He is only two weeks into his job and he is nervous. Every time he responds to calls he gets very worried. ENTITY: job, nervous
She made sure she was quiet and respected others' space. It was strange that Wednesday, she came to the office hung over. ENTITY: introverted, getting drunk
James took care comply with the rules and demonstrate deference towards authority figures. Surprisingly, he was caught shoplifting on a Friday. ENTITY: disciplined, shoplifting
Table Seed analogy examples for generation candidates STORYANALOOGY, We sample 5 pairs from propara and 5 paris from rocstories.

---

stories in around 30 words given the keyword(s). AHint: Story should be analogous to story 1n Example 0: Vn Keywords for story 0: [keywords)"
ROBERTA-CL Details for training the con- trastive learning model. We adopt the SimCSE training script" for training our contrastive learn- ing model. The positive pairs are filtered according to the EntSim <= 1.0 and RelSim >= 2.0.
In addition, we notice that the story pairs gen- erated this way tend to have low entity similarity (since their entities are pre-given). Therefore, we additionally prompt LLMs to write set of similar keywords for the source first, and then write cor- responding story, which does not need be similar the source.
A.2.2 Prompt templates for similarity prediction.
The templates used for prompting LLMs to gener- ate similarity predictions are presented below. The "long instruction" template for EntSim pre- diction.
"Write group of 2-sentence stories in around 30 words given the ks)word
"Evaluate the entity similarity between pair of stories. Assign the pair score between and as follows:
Keywords 1: O
Story 1: O
Give set of keywords similar to keywords
The two stories are talking about different topics and entities of different types.
Unrelated.
1, and then write corresponding story (the stories do not have be similar):"
Somewhat related. The two stories talk about different entities, and some of them have similar or related types. Somewhat equivalent. The two stories have different entities, but they have the same types.
A.1.2 Annotation templates.
In this section, we showcase our templates used for annotation on the Amazon Mechanical Turk plat- form. The instructions used for evaluating entity and relation similarities are presented in Figure and Figure 8, respectively. Followed by these in- structions, questions are presented using templates shown in Figure 7 and Figure 9.
Almost equivalent. The entities in the two stories are overlapped or synonymous.
A.2
Details in $ 3.1.
Following the above instruction, evaluate the entity / topic similarity for S1 and $2 (only answer by score from 0, 1, 2, 3):
A.2.1
Baselines
Note that DMR requires two sentences as input. We use the NLTK toolkit to tokenize the stories into two sentences. In cases where there are not enough two sentences, we try split the story by the first comma. If there is comma in the story, the DMV vector is computed between the story and an empty string This only accounts for small number stories (10-100).
DMR
(N-DEMONSTRATIONS HERE) Q:
S1 (INPUT-S1)
$2 (INPUT-S2)
Score :"
The "long instruction" template for RelSim pre- diction.
"Evaluate the relation similarity between pair of stories. Assign the pair a score between and as follows:
GloVe We use the glove-840B-300D version We first use the Stanza part-of-speech (PoS) anno- tation tool to parse the PoS of words. Then, we return the summation of embeddings for all the words with the corresponding PoS. Specifically, determine nouns a word's upos is in ("PROPN", "NOUN"). We detect verbs if its upos is "VERB" or its xpos starts with "VB".
Very poor alignment. Most if not all relationships do not align.
Alignment with significant mismatches. Some of the relationships align, but there are some significant mismatches. Alignment with insignificant
mimmathes. Most of the relationships align gn except for some insignificam mismatches.
We apply an MLP on top of the RoBERTa model and output two digits, which cor- respond to the EntSim and RelSim, respectively.
RoBERTa-Reg
Alignment. The relationships can
12 "https://nIp. http://lll..st..rraa..oo stanford...oooo edu/ data/glavaa 8408. 300d.zip
13 "htttps://githie http://ttwca..eeeeeeeeeooaaaaaaa

---

align very well between the two stories. Following the above instruction, evaluate the relational similarity for and $2 (only answer by a score from 0, 1, 2, 3): (N-DEMONSTRATIONS HERE} Q:
nal settings in their code repo at https://gi .eniing..miinn..... As for the re- sult, we discovered that FMQ and FMV exhibited comparable performance (44.9% versus 44.7%) on the multiple choice dataset. The result from FMQ are reported in the main paper.
S1 (INPUT-S1)
S2 - (INPUT-S2)
A.3.1 Prompt templates for multiple-choiee evaluation.
Score :"
Here, we insert NE (0,1,3) demonstrations the "N-DEMONSTRATION HERE" and fill in the story pairs at "INPUT-S1" and "INPUT-S2". The "short instruction" templates are similar, with the only difference that the detailed definition of scores is removed. For instance, "0 Unrelated. The two stories are talking about different topics and entities of different types." is replaced with "0 Unrelated."
The prompt template used in multiple-choice eval- uation is:
(QUESTION)
Source story: 0
Candidate stories:
(0) 0
(1): 0
(2): 0
(3): (
Answer:
A.3 Details in $ 3.2.
where "QUESTION" replaced with one of the fol-
lowing questions:
To construct the multiple-choice evaluation set, we gather story analogy pairs with EntSim 1.0 and RelSim > 2.0. Next, we sample negatives for each story analogy pairs to form multiple choice ques- tions. Similar to the GloVe bascline in $ A.2, we obtain the nounal embedding for each story, and retrieve stories with high cosine similarities while have <50% overlapped tokens the hard negative choices. We manually inspect the overall qual- ity of the multiple choice questions constructed in this manner. We excluded the questions gener- ated from the ROCStories split due to their lower quality, likely because the unusual distribution of EntSim in this split made it difficult to use the same method for creating the dataset as in the other splits (Figure 3). The resulting multiple choice dataset consists of 360 questions.
A: "Select the candidate that best matches
the source story as an analogy."
B: "Which candidate story is the best creative analogy for the source story?" C: "A creative analogy should have fewer similar entities but similar relational structures to the source story. Which candidate story is the best creative analogy for the source story?"
A.4
Details in the evaluation of story analogy generation.
A.4.1 Generation setups.
The models are evaluated under zero-shot, few- shot, and instruction-tuning settings. For zero- shot and few-shot prompting, the templates are: "Write analogy for story An \nStory 1: ()nStory 2:" This template is also used in the finetuning setting. For finetuning, we employ DeepSpeed 14 to accelerate the training on single V 100 (32GB) instance.
Baselines in (Sultan and Shahaf, 2023) We ap- ply both the FMQ and FMV models, as suggested in (Sultan and Shahaf, 2023), to our story analogy identification task. To be precise, we gather the in- termediate story pair similarities generated by their models. Afterwards, we choose the option that exhibits the highest similarity to the source story. Notably, the lengths of the stories in our dataset are considerably shorter than datasets used in its paper. Therefore, when running the baseline on dataset, adjusted the threshold of the similar- ity filter to better suit our settings. We selected a threshold of 0.3 for FMQ and 0.2 for FMV. For the other implementation details, we follow the origi-
A.4.2 Annotation.
We conduct crowd annotation on AMT to evaluate the generation quality. The annotation instruction is presented in Figure 5. During the annotation, the meta information of the target generation is hidden from the annotators and the requesters, In addition, the target stories are shuffled such that
htttps://ww/ www.:wwwiic.oo project/deepspedd

---

annotators cannot find out which models are used generate the stories based on the order.
2021: Cui al., 2021b,a), where analogy could be useful angle improve automatic building per- formance (Chen al., 2022b). The data and evalua- tion metrics in this work may serve as a benchmark in evaluating one of the analogical reasoning abili- ties.
A.5
Miscellaneous
In this section, we present some discussions that took place during the reviewing process.
A.5.1 Potential applications of this work.
Why the predictions of individual scores are good, but the prediction of a is bad. Original question: How is it that models so good individually predicting EntSim and RelSim (in Section 3. but they are not that good at predicting the analogy score a??
A.5.2
Analogy Mining for Art and Design. There have been various studies focusing on building analog- ical search engines. Hope et al. (2017) contribute a method for analogy mining over products. Chan et al. (2018) mine analogies from research papers with respect to their background, purpose, mecha- nism, and findings. Gilon et al. (2018) develop search engine for expressing and abstracting spe- cific design needs. Recently, Bitton et al. (2023) propose a visual analogies dataset, VASR, where they found that models struggle to find out analo- gies when given carefully chosen distractors. In computer graphics, some graphics design algo- rithms take input image from the user, and transform it to some other types of visual designs that are similar the given image, such as em- broidery patterns (Zhenyuan et al., 2023) and vec- tor line arts (Mo et al., 2021). This category of work establishes connections between images and coplication-specitie graphics patterns. With im- ages as guidance, the complicated visual design processes are made easy and intuitive for nonpro- fessional users.
Since the analogy score is computed from both EntSim and RelSim, the prediction of the analogy score relies on predicting the gap between EntSim and RelSim, which is harder than predicting each similarity alone. A case is presented below to illus- trate this.
Suppose we have four story pairs, and their ground-truth scores are: EntSim [2, 1, 3,
0], RelSim [0, 1, 2, 3].
The corresponding predictions are: EntSim'= [0, 2, 3, 0], RelSim'= [1, 0, 2, 1]. Then, the analogy scores and the predicted anal- ogy scores can be computed from the above values RelSim
(using EntSim+1 )): [0, 0.5, 0.5, 3], a' [1, 0, 0.5, 1].
Finally, we can compute the Spearman's correla- tion coefficients as:
Corr (RelSim, RelSim =01. 316 316 Corr (EntSim, EntSim ( = Corr a') Here, though the predictions of the respective scores have medium correlation with the ground- truths, the prediction of has zero correlation.
Analogical Reasoning. Large language models (LLMss) have demonstrated impressive abilities in few-shot and zero-shot learning (Kaplan et al., 2020; OpenAI, 2022, 2023). Recently, Chat- GPT (OpenAl, 2022), GPT-4 (OpenAI, 2023), Alpaca (Taori et al., 2023) and their following works (Chiang et al., 2023; Jiang et al., 2023) have achieved remarkable performance of benchmarks. It believed that they have ac- quired certain kind of analogical reasoning ability that are not only task-specific (Webb et al., 2022: Ding et al., 2023). but also omnipresent through- out the prompting process of LLMs, and there are a lot of prompt engineering work to leverage this characteristic for downstream tasks (Jiang et al., 2022; Chan et al., 2023b,a,c; Chan and Chan, 2023). Meanwhile. it important note that LLMs also exhibit potential issues related hallucination, bi- ases, and privacy (Ray, 2023; et al., b023a,b; Wang et al., 2023). Mitigating such issues often requires building up knowledge bases (Cheng et al.,
wide range

---

Source story: The virus invades cells. As result, the DNAs damaged.
and several target storte:
Target story 1: As virus infiltrates the body, attacks healthy cells and causes disturbance the immune system.
Target story 2: Parasites invade their hosts, feed nutrients, weaken the immune system, causing damage host's body.
Target story 3: The burglar breaks into the house. As a result, the valuables inside are smashed.
Target story 4: A swarm of space bees invaded the atmosphere and caused the planets to spin backwards. As result, the gravitational pull of the universe disrupted and time travel became possible.
task 1. [Analogy?] the target story analogy for the source story?
YES/NO questions for each target story:
Here, an analogy should have similar structures rather than details. For instance, target story 2 and are analogies to the source story, while story is not.
[Novel7] the target story novel (compared the source)? This measures the novelty of the target story. Think about whether the target is engaging and interesting For instance, many may found target story 3 novel, while and 2 not.
[Plausible?] target story plausible?
Check whether the target plausible and makes For instance, target story does not really make and should be labeled plausible.
Figure The annotation instruction for generation quality evaluation.
Entity/Topic Similarity Rating
similarity you For first time readers, you recommended read through Full Instruction here. short, the defintions and examples of the four scores are provided below:
higher score,
are.
3: Almost Equivalent
Definition:
overlapped synerymous
Towins harm the body. The body triss roazt to the toxins.
Poison ontons the body. The body trfes to fight off the paison.
Bogart lived on form. loved becon.
lived on farm. She loved bacon.
Somewhat equivalent
Defnition: The stories have diflerent entiies, they have the
John walked over to talk her, Five minutes later he retured to a grll full burned burgers.
Mary went inside to grab a drink. When she came back, the vegetables on the grill were charee.
Bogart lied on a form. He loved bacon.
Jane lived the city. She loved pizza.
vapor Clouds
organined compact.
1: Somewhat related
Definiton:
different
related types.
My wife went to the store to buy moving baxes, She bought baxes that were too large.
went the grocery store to buy fruit. bought apples that were too ripe.
sluminum
bin.
Put the dirty dishes dishwasherr, The dishes transported the sinkk for cleening.
facilty.
0: Unrelated
Definiton::
stories talking about different topics and entities ditferent types.
This liquid known as magma. The magma rises to the carth's surface in volicaoos where coots and hardens.
Security guards monitor the doors the tactory. They contral the movement of people into and out of the factory.
The valcanos crupt many times, The size of the rocky area grows.
A plant grows larger over time. The plants mots spread and deepen.
Figure 6: The instructions used for evaluating entity similarity in human annotations.

---

Source story
Targel story
Domain
person grows from child adult. As time pusses, the person experiences ongoing growth and maturation.
The stream becomes a river. The river continues flow along same path for long time.
0.6
2.8
1.6
PP
Apply lotion to skin. Massage the lotion into skin.
Fertilize the soil. Mix seeds into the fortilized soil.
0.0
2.6
PP
Fill the tray with cool water. Place the in the freezer.
Fill the backet with warm water. Place the bucket in the refrigerator.
330
3.0
PP
The resulting material disappcars. The plant with the soil.
The evaporates. The liquid vapor
and disperses into the
2.0
0.4
0.1
PP
The gas condenses in the condenser and becomes a liquid again. Heat is ridiated away from the con- denser.
emotion expressed released. A calming effect follows after the expression.
0,3
0.0
0.0
PP
They gave her the password to the website. When Jane logged in, she realized it was the wrong pass- word.
They left him the key in the entrance. When Tom went over realized it was the wrong key.
1.0
ROC
was baking a cake, had several ingredients help.
I was huilding a dresser. had several lools help.
3.0
1.5
ROC
It's broken. have to buy new one.
It's expired. have new one.
3.0
:
3.0
0.81
ROC
His cellmmtte tried bully man. man lought cellmate.
His classmate tried to intimidate him. stood his ground and refused be bullied.
2.7
0.4
ROC
The fight lasted until 10 am. We finally just went to hed out of exhaustion.
The argument went on until midnight. We eventu- ally just gave up and went home defeal.
0.0
ROC
Foundations poured support the walls and roofs of buildings. The structure of the building is only as strong it's foundation.
Reasons formulated to make theories. The clusions of theorics only dependable their initial premises.
0.6
WA
The reasons the theory provide a rational expla- nation. This informs decision-making process that supports theories accuracy.
The ground for the building is solid and secure. This gives the building its foundation and stability.
0.4
2.8
WA
His memory has bruken into fragmented pieces. He can recall flashes and images of the past, but nothing concrete or clear.
His memories remain a confused mexs. Nothing holds together and what he semembers don't 't make sease.
2.7
3.0
0.8
WA
Heat energy is transferred from point another. Transfers between different substances perature changes.
Solid materials undergo phase transitions ergy is added, Changes pressure also result in phase transitions.
2.8
1..
WA
She laughed and go carefree attitude was liberating.
her worries, Her
delved into the unknown without a second thought, ignorant the knowledge to come.
0.8
WA
The student opens the book and begins to read. The knowledge gained from the hook absurbed by the student.
The cat sees mouse and begins to chase it. The cal honing its hunting skills through practice and repetition.
0.8
1.4
0.8
CN
The tigger pulled and pistel shoots. The gun fires.
metcorite impacts Saturn's surface. The planct buffeted by these larger objocts.
0..
2.0
CN
She figured an overdose was the only way out. Within minutes, she had taken her last breath and died.
He knew his only way commit suicide. He determined die, matter whar.
330
2.6
0.7
CN
She chose her destination for her vacation with citement. She anticipated what her journcy would bring.
She purchased a round-trip ticket for her travels. She left assurance that she would return.
2.8
1.4
0.4
CN
The rain began pour and gradually. river started to overflow, It was the start devastating Blond.
The scissors snipped away. trimming locks until was just right. becamee proud owner new, huirstyle.
0.0
CN
Table 9: Examples in STORYANALOGY with annotations from each domain. We report the EntSim and RelSim from crowd workers 2 The Domain column indicates the source of the story pairs. "PP", "ROC", "WA", and "CN" are short for "ProPara", "ROCStories", "Word Analogy", and "ConceptNet", respectively.

---

Pair
Bogart lived loved bacon.
Jane lived in the city. She loved pizza.
How will rate the topic similarity between these stories?
two are pverlepee synortymous, Somewhat equivaentt two
equivalent
antities they have the types.
Somewhat related. The swo stories talk about difereet enttiess and some of them have simitar or related types.
Unreisted, The two stories are talking about difierent topics and entiies of different types.
one storess ungrammatica countertacnual.
Figure 7: The template for presenting question regarding the evaluation of entity similarity human annotations.
Relational Similarity Rating
To desermine the relational similartyy you required to score from 0 The higher the score, the more close two stories short. the definiions and examples the four scores are provided below. first column denoted as Story A and the second column contains Story
Perfect Alignment
Defintion: The relationshiips can align well between the tvo storees.
breathing rate animal weight slowly than
animel's
engine
system
car
The stroam bocomes a nivr.. The river continues to flo along the same path a long time.
A plant grows from a seod into a mature piant. The plant continues grow and thrive in its crwiromment over time.
Many more doad plants sink the same area. The dead plants join rogether forming
Many more lcaves tall to the ground the same area. The leaves torming layer of muich.
Alignment with insignificant mismatches
Definiion: Most of the relationships olign except for insigrificant mismetches.. ("rssgnilican"" that the differences invalidate relational similartty the story pair.)
Magma rises from deep in the carth. The magma goes into volcano.
Oxygen goes from the lungs the biood. Blood goes to the heart (Explanztion: In story B, the oxygen goes from the lungs the heart through blod, which siihtly different from the relationship of story A.)
Sarah was on bus to her work She had to poo badly.
Tom was driving a cilent meeting. He suddenly realbed he had pressing need use the aothroom.
(Explanatin: The logioal connection (nttr-aventiicate relationshig between events can slightly aifferent. in story B, Tom "suddenly mallodd the ungent need to to the hathroom, while this missing story A.)
Alignment with significant mismatches
Definiton: Some relationships align, but there some significant mismatches.
the bat. bat
A person speaks (Explanation: Many evandistate relationsiips connor align B.)
person.
person
responds.
achood
The pans are tnansported facillyy The cans are shroddded by machine.
A package delivered to a warehouse. The package is opened and is contents are sorted by workers.
(Explanation: The reletionships in the underfined part cennot align with story A)
wonted impress his friendss He went wrestlin show:
Tom loved dencing. He (Explenatio: The relstionships the underfined part do not wlin.)
salse club.
0: Very poor alignment
Defiriton:: Must il sil relotionships slign.
Morgan enjoyed long warks on the beach. She and her boyfriend decided go long walk.
decided to take my gi fiifeend the beach. As we were walking she paused.
This liuud known as magme, The magma rises to the earth is surface in valcanoes where coots and hardens.
Security guards monitor the doors of the factory. They control the movement of people into and out of the factory.
Figure 8: The instructions used for evaluating relation similarity in human annotations.

---

Pair
animal's and breathing animal loses weight slowly than usual.
The car's engine and exhaust system slow down. The car uses less fuel than usual,
How will you rate the relational similarity between these two stories?
Pertect Algement The relationships can align very wall betwoon these two stories.
Aligmmett with insigificant mismatches, Most the relatenships align except for some insigniicant
Aligmment with signilicant mismatccess. Some relatioships align, but there are some significamt mismaiches.
Very poor aligmeet. Most or even all releiioshiips do not alignn
ingrammatical
Figure The template presenting question regarding evaluation of relation similarity in human annotations.


Reviews:
[{"review_id": "jok9TLWmk9", "reviewer": "Reviewer_8L5d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our paper. We appreciate your feedback on our work and would like to address your comments below:\n\n1. Restricted Domain Scope.\n\nWe agree that such extension will increase the value of our dataset, and we will discuss it in the Future Work section. In the future, using our annotation framework and evaluation metrics, we can extend the annotation to other domains, such as biomedical and scholarly articles, in addition to the four data domains in this study. \n\n2. Disparity in Model Performance.\n\nThank you for the suggestion on the visualization. We will emphasize the gap on the table and add visualization figures in the final version. \n\nIn terms of strategies to mitigate the discrepancy, we have shown that: \n\n(1)   (At line 380) For unsupervised encoder models, we found that models aware of relation information in their pre-training stage, such as DMR, RelBERT, and GloVe-Verb have better correlation performance than the general ones. \n\n(2)   (At line 352 and line 384) By fine-tuning encoder models with regression and contrastive learning objectives on the training set, the model can improve substantially. \n\n(3)   (At line 388) For LLMs, we also conduct experiments on using different prompting setups (Figure 4), where we found that using extra definitions as hints in the prompt can help LLMs improve their prediction. \n\nThese findings are useful for mitigating the discrepancy for different models. We will leave more detailed exploration for future work. \n\n3. Constrained Evaluation Methodology.\n\nSince story level analogy has not been well-studied yet, the goal of the experiment shown in Table 2 is to evaluate whether the embeddings generated by existing encoder models can perform well on analogy identification. Therefore, the methodologies behind the models have little influence on the conclusions we made: We can treat the encoder model as a black box. Story texts are input to the black box, and we use its output embeddings for evaluation. We select representative encoder models from all kinds, such as word embedding model (GloVe), contextualized text embedding models (RoBERTa, SimCSE, OpenAI-ada), and contextualized text embedding models pretrained with relation-aware signals (RelBERT, DMR). \n\nTherefore, OpenAI-ada still makes a fair side-by-side comparison with the other models. In fact (Table 2), we found that although OpenAI-ada performs well on EntSim and RelSim correlations respectively, it has poor performance in terms of correlation with alpha, which is even worse than the DMR model (which is of the same size of RoBERTa-base). This indicates that it is not a good choice for analogy identification. \n\n  4. Absence of Defined Practical Applications.\n\nThank you for the suggestion. We will make clear the practical applications of this work in the final version.  In our experiments, we have done applications such as analogy identification (in Sec 3) and generation (in Sec 4). In practice, analogy identification is related to applications such as creativity search engine, where the similarity scores (line 336) can be directly used for analogy passage retrieval. Analogy generation can be used for generating useful ideas or poems. There are many related works on these applications such as [Sultan and Shahaf, 2023], [Turney, 2008], [Ding et al., 2023]. \nMoreover, we are working on a demo of these applications, which will come out in the final version of the paper. We hope these efforts can help illustrate the potential practical applications. \n\n Thank you again for your constructive feedbacks. We will address these concerns and incorporate related discussion on them in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tNovelty: The approach to story-level analogies is unique and addresses a gap in existing research that has mostly focused on word-level analogies.\n\n2.\tRich Data Resource: The creation of a substantial corpus specific to story analogies provides an invaluable resource for further research and development.\n\n3.\tDetailed Evaluation: The paper offers a comprehensive assessment of current models' capabilities, providing insights into where they succeed and where they fall short.\n\n4.\tPotential for Further Research: The insights and resources presented in this paper could stimulate further research in areas like creativity-based generation tasks, including poetry, lyrics, and humor generation.\n", "reasons_to_reject": "1.\tRestricted Domain Scope: Despite the utilization of diverse sources in assembling the corpus, certain specialized fields such as biomedical narratives or scholarly articles remain outside its coverage. This exclusion may constrict the corpus's range of applicability. Furthermore, the aggregate size of the corpus might be inadequate for the comprehensive fine-tuning of a large language model (LLM) with numerous parameters.\n\n2.\tDisparity in Model Performance: The paper accentuates a significant divergence in performance levels between existing models and human abilities. However, it fails to visually represent this gap succinctly in Section “3.1 Correlation with the analogy score α,” nor does it elucidate detailed strategies to mitigate this discrepancy.\n\n3.\tConstrained Evaluation Methodology: The automated assessments employed within this paper hinge on encoder models introduced from 2014 through recent years. With the exception of the newly publicized OpenAI-ada, the inclusion of models from previous years might not be methodologically sound for side-by-side comparison, as illustrated in Table 2.\n\n4.\tAbsence of Defined Practical Applications: The paper could refine the exposition of its practical applications and the specific contexts where the methods and discoveries might be deployed. Given the breadth and complexity of tasks that an LLM can undertake, the provision of a multi-constrained, task-specific corpus may lack novelty or sufficient contribution to the enrichment of diversity or the enhancement of LLMs' overall performances.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "Topic: The paper presents an in-depth exploration of story-level analogies in the context of natural language understanding by introducing an analogy corpus aimed at enhancing the performance of LLMs in multi-domain story-level analogy generation.\n\nMain Contributions:\n\n1.\tCreation of the STORYANALOGY Corpus: The paper introduces a novel, large-scale corpus named STORYANALOGY, which comprises 24,000 pairs of story analogies from diverse domains, enriched with human annotations.\n\n2.\tDevelopment of an Evaluation Framework: The authors devise tests and evaluations to assess the identification and generation of story-level analogies, utilizing an extended version of the Structure-Mapping Theory (SMT).\n\n3.\tAnalysis of Performance: The paper uncovers the challenges faced by both sentence embedding models and large language models (LLMs) when performing story-level analogical tasks. Special emphasis is given to models such as ChatGPT and LLaMa.\n\n4.\tIntroduction of Improvement Techniques: The paper demonstrates that the inclusion of STORYANALOGY data can enhance the quality of analogy generation in models, particularly through fine-tuning with specific models like FlanT5-xxl.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XpFNaKLv5W", "reviewer": "Reviewer_yNU4", "summary": "", "questions": "Please address the weaknesses in the author response. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedbacks on this work! We hope the following discussions are helpful for addressing your concerns:\n\n1. Essential details are not presented in and moved to the supplementary section.\n\nDue to the limited space, we put details (such as the prompt templates and the model we use) into the appendix section. The name of the LLM is mentioned at line 747 in Appendix A.1. (text-davinci-003). For the final version with one more page, we will definitely move important details to the main section. \n\n2. Using a proprietary tool for analogy generation.\n\nThank you for bringing up this point. In order to minimize potential misunderstandings, we are adding further specifications as follows: \n\nWe would like to point out that which exact model to use to generate the data does not make a huge difference here (the LLM we use is text-davinci-003, at line 747), since we are presenting the annotated data, and that the data has been annotated and filtered by human workers. The generation model will not be used again after the candidate generation step.  \n\nFor replication purposes, our data will be made public in the final version. Additionally, the expansion of our data is possible because the candidate creation process and annotation guidelines have been clearly defined. \n\n3. I did not fully understand why “alpha” which is a ratio of RelSim and EntSim is a better metric to evaluate analogies. \n\nAs illustrated in the paragraph at line 143, according to the original Structure-Mapping Theory, analogy between objects holds when they have similar relational structures while dissimilar attributes, while “literal similarity” holds when both the relational structures and attributes are similar. In this work, we extend this to story level, which means story analogies should have similar relational structures (high RelSim) and dissimilar in terms of entity or topic (low EntSim). \n\nThe analogy score “alpha=RelSim/EntSim” is an instantiation of such a notion to reflect the level of analogy between a pair of stories. There are of course other ways of implementing this, such as “RelSim-EntSim”. In practice, we conducted some preliminary experiments and found these definitions do not make a big difference, so we finally decided to use the current definition. \n\nWe are not sure what it means by “entities paradigmatically related to each other”. (Could you please elaborate? Thanks!) If it means entities with close meanings, such as “Magma-Lava”. Then, according to the structure-mapping theory, such high EntSim would make it likely to be a literal similarity case (see Figure 2), and this is less desirable than analogy (“Magma-food”) which have lower EntSim. \n\n4. The font size in figures and tables are sometimes too small and illegible. Moreover, Figures and tables are not explained enough in sections. For example, what is the summary of figure 3 and what does it entail? Any interesting insights? \n\nThank you for pointing this out. We will adjust the font size of figures and tables to make them more friendly to readers. \n\nFigure 3 presents the distribution of annotations of EntSim and RelSim across different sources. From the plot, we can find that candidate pairs generated from the ROCStories have relatively high EntSim and RelSim compared to the other sources, which might be because ROCStories are a social commonsense story corpora that are mostly talking about similar social stories. This implies that genres of story source have influence on the score distribution of analogy generated. \n\nWe will further add explanations to figures and tables. \n\n5. Neither the paper not the ethics statement includes any potential shortcoming the dataset may have in terms of generating analogies that have some form of societal bias (pertaining to race, gender and occupation). At-least a detailed ethics statement should be provided. \n\nDuring the annotation (section 2.3, line 261, 294; also, in ethics statement at line 572), we required workers to label a story pair as “poor quality” if they find any toxic content. Story pairs with >10% annotators annotated with this tag are filtered out. This is a rather general instruction to remove the potentially toxic contents, so it is assumed that most possibly toxic contents have been removed in the final dataset (which accounts for 142 instances as mentioned at line 296). We will include a more detailed discussion and ethics statements on these toxic contents in the final version. \n\nThank you again for the valuable suggestions. We will address these concerns and incorporate related discussion on them in the revision to improve the presentation and flow.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting dataset, thorough qualitative and quantitative evaluation", "reasons_to_reject": "The presentation and flow could e improved a lot. See the weaknesses above. ", "questions_to_authors": "Please address the weaknesses in the author response. ", "paper_topic_and_main_contributions": "The paper presents a method for generating analogical sentence pairs using extensive, unlabeled narrative corpora and large language models. Starting with inputs sampled from scientific narratives, stories, and knowledge graph tuples (e.g., ConceptNet), the authors employ ChatGPT (text-davinci-003) to produce analogies, aided by a set of seed paired data. These generated analogies are evaluated through crowd annotation, utilizing novel metrics like EntSim, RelSim, and alpha, inspired by the Structure Mapping Theory. The resulting dataset of story analogies is established and validated, while the authors also propose diverse techniques to fine-tune and test several Large Language Models (LLMs) and language encoders for the identification and generation of analogies.\n\nStrengths: \n- Interesting approach and a valuable dataset creation for exploring and applying analogical reasoning (a powerful AI paradigm) to NLP models. \n- Clear annotation guidelines to evaluate automatically generated datasets\n- Extensive evaluation of LLMs and encoders on the curated dataset, demonstrating their capabilities and limitations \n\nWeaknesses: \n- Essential details are not presented in and moved to the supplementary section: The crux of the work lies in synthetically generating sentence-analogy pairs using LLMs, but I had to go to Appendix to know the exact implementation details. Even the name of the LLM was not mentioned in the main section. \n- Using a proprietary tool for analogy generation: Since, it’s not mentioned, I assumed that authors used text-davinci-003 in ChatGPT api to generate analogies. This is a proprietary system and not a lot has been disclosed about how it operates and what kind of pre- and post-processing steps it applies to get better generations. Additionally, proprietary systems like this change too fast, making it harder to replicate this work. \n- I did not fully understand why “alpha” which is a ratio of RelSim and EntSim is a better metric to evaluate analogies. While it may mark high RelSim and low EntSim cases better analogies, it rules out other possibilities (such as entities paradigmatically related to each other, thus having a higher EntSim but perfectly making a good analogy). \n- The font size in figures and tables are sometimes too small and illegible. Moreover, Figures and tables are not explained enough in sections. For example, what is the summary of figure 3 and what does it entail? Any interesting insights?\n- Neither the paper not the ethics statement includes any potential shortcoming the dataset may have in terms of generating analogies that have some form of societal bias (pertaining to race, gender and occupation). At-least a detailed ethics statement should be provided. \n\nIn general, the dataset holds some value, yet there's room for presentation enhancement. The paper lacks clarity and justification in some sections while excessively elaborating on others. Resolving these matters would require significant revision, hence, my inclination is toward rejecting the current version.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "set9v7dmEV", "reviewer": "Reviewer_WoNd", "summary": "", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper! We appreciate your feedback on this work, and we will add the missing references in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, this paper makes a solid contribution in terms of a new dataset, evaluation methodology, and empirical analysis on an important NLP problem. It should lead to increased community interest in complex analogical reasoning and encourage follow-up work leveraging the introduced resources. The novel dataset and experiments are also presented rigorously. This can positively impact the field by spurring research that could lead to AI systems with stronger narrative understanding, commonsense reasoning, and transfer learning abilities.\n\n1) Addresses an important but under-studied problem in AI - modeling complex story-level analogies requires deeper semantic understanding and abstraction than traditional word or sentence-level analogy tasks.\n2) Provides novel and sizable resources to the community - the dataset of over 24K annotated story analogy pairs enables new research in this direction.\n3) Presents comprehensive empirical results and analysis - the authors thoroughly evaluate strong baseline models from encoder methods to large language models, clearly showing current limitations.\n4) Promotes an evaluation methodology - the proposed analogy identification and generation tests based on structure mapping theory provides a principled way to assess progress.\n5) Demonstrates clear value of the resource - the experiments show models can benefit from the dataset through fine-tuning and few-shot learning.\n6) Well-written and structured - the background, dataset creation, experiments, and analysis are presented clearly.\n", "reasons_to_reject": "I dont see any reasons to reject", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "This paper introduces STORYANALOGY, a new dataset and set of benchmarks for evaluating story-level analogical reasoning in artificial intelligence systems. The key points are:\n\n- Story-level analogies involve comparing entire narratives or sequences of events, which is more complex than just comparing individual words or concepts as in traditional analogy datasets. However, there has been little research on modeling story analogies computationally. \n\n- The paper creates a large dataset of over 24,000 story pairs labeled for analogy using crowdsourcing. The analogies cover diverse domains like science, social narratives, and commonsense knowledge.\n\n- To systematically assess different levels of analogy, the authors extend the cognitive psychology theory of Structure Mapping to define two similarity scores - entity similarity and relational similarity. These scores characterize how much the stories align in terms of entities/topics and relational structure. \n\n- Using the dataset, the paper evaluates several state-of-the-art models (encoders like SimCSE and LLMs like ChatGPT) on story analogy identification and generation. The models significantly underperform humans, showing there is substantial room for progress.\n\n- The dataset and benchmarks enable new research on improving AI's analogical reasoning over narratives. This could enable better commonsense understanding and transfer learning.\n\nIn summary, the key contribution is creating resources to advance research on an important but relatively under-studied AI capability - story-level analogical reasoning. The empirical results show current models still have significant limitations on this task.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "\nJoel Chan, Joseph Chee Chang, Tom Hope, Dafna Shahaf, and Aniket Kittur, SOLVENT: A Mixed Initiative System for Finding Analogies between Research Papers\nCSCW 2018\n\nKarni Gilon, Felicia Ng, Joel Chan, Dafna Shahaf and Aniket Kittur, Analogy Mining for Specific Design Needs\nCHI 2018\n\nTom Hope, Joel Chan, Aniket Kittur and Dafna Shahaf, Accelerating Innovation Through Analogy Mining\nKDD 2017\n\nVASR: Visual Analogies of Situation Recognition\nYonatan Bitton, Ron Yosef, Eliyahu Strugo, Dafna Shahaf, Roy Schwartz, Gabriel Stanovsky\nAAAI 2023"}}, {"review_id": "DbBaHWPHqd", "reviewer": "Reviewer_2Qdd", "summary": "", "questions": "1. In section 3.1 for the encoder based baseline, lines 333-343 state that, analogous score was taken as cosine(encoder(s1), encoder(s2)). I am not convinced if similarity based on embeddings from just a pre-trained, off the shelf model would suffice. For example, I don't think that the scores for stories describing electrons and atoms & planets and sun would be close enough. Fine-tuning is necessary. I acknowledge that the authors present two fine-tuned models as well. I just think all other encoder models form a week baseline. \n2. Section 3.1  lines 333-343 just describe how analogous score alpha would be calculated. They don't describe how entity similarity and relation similarity scores are calculated. Or did I miss it? \n3. How is it that models are so good an individually predicting the two scores, but they are not that good at predicting the analogous score? And the fact that analogous score is such a simple combination of the other two makes is perplexing.\n4. Appendix A.1 line 760. Is there something missing at the end of the line. \" using the methods in '…' \" \n5. I think the the template mentioned in line 340 are not provided. Am I understanding correctly?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper. We appreciate your thoughtful feedbacks and questions, and we hope the following discussions can help address your concerns:\n\n1. [Reasons to reject 1, Question 1] It's not fully convincing that pre-trained off the shelf encoder models make strong enough baseline (Table 2).\n\nUsing cosine similarity based on embeddings from off-the-shelf encoders is commonly used in semantic textual similarity, word analogy, and unsupervised discourse analysis tasks. To this end, we select representative encoder models from all kinds, such as word embedding model (GloVe), contextualized text embedding models (RoBERTa, SimCSE, OpenAI-ada), and contextualized text embedding models pretrained with relation-aware signals (RelBERT, DMR). \n\nBefore the experiments, we cannot assume that the encoder models are weak baselines on our task. Also, these baselines help illustrate the relative performance improvement for our fine-tuned models. Therefore, we think that presenting results from these models would be necessary to draw conclusions. \n\n2. [Reasons to reject 2] Since it's basically a dataset paper, it would have been nice to see datacards. \n\nThank you for the suggestion. We found this a useful tool and we will try to add a datacard in the final version. At the current stage, we cannot present a datacard as we found it would reveal authors and affiliations and violate the anonymity requirement. \n\n3.  [Question 2] Section 3.1 lines 333-343 just describe how analogous score alpha would be calculated. They don't describe how entity similarity and relation similarity scores are calculated. Or did I miss it?  \n\nWe are sorry for the misunderstanding caused. We will revise the corresponding part to make these points clear: \n\n(Line 339) For LLMs, we prompt them to predict the entity similarity and relation similarity scores for a given pair of stories, respectively. The prompt templates are presented in the response to Q5. \n\n(Line 338) For encoder models (including the finetuned RoBERTa-CL), we obtain their similarity prediction with the formula f(s1, s2) = Cosine(Encoder(s1), Encoder(s2)). We compute the correlation between f(s1, s2) and EntSim or RelSim or alpha, respectively.  \n\nThe finetuned regression model RoBERTa-Reg is coupled with an MLP layer that outputs both EntSim and RelSim. \n\n4. [Question 3-1] How is it that models are so good an individually predicting the two scores, but they are not that good at predicting the analogous score?  \n\nSince the analogy score is computed from both EntSim and RelSim, the prediction of the analogy score relies on predicting the gap between EntSim and RelSim, which is harder than predicting each similarity alone. Given that the prediction of EntSim and RelSim is far from perfect for the current baselines, the prediction of the analogy score is also unstable. \n\nWe present a case here to illustrate this: \n\nEntSim = [2,1,3,0], \n\nRelSim = [0,1,2,3] \n\nEntSim-prediction = [0,2,3,0] \n\nRelSim-prediction = [1,0,2,1] \n\nThe analogy scores can be computed from the above values (alpha=R/(E+1)): \n\nAlpha = [0.,  0.5, 0.5, 3. ] \n\nAlpha-prediction = [1.,  0.,  0.5, 1. ] \n\n \nThe spearman’s correlation coefficients are: \n\nCorr(RelSim, RelSim-prediction) = 0.316 \n\nCorr(EntSim, EntSim-prediction) = 0.632 \n\nCorr(Alpha, Alpha-prediction) = 0.0 \n\nHere, though the predictions of the respective scores have medium correlation with the ground-truths, the prediction of alpha has zero correlation. \n\nWe will present a case study in the final version to help better understand this phenomenon. \n\n \n5. [Question 4] Appendix A.1 line 760. Is there something missing at the end of the line. \" using the methods in '…' \"  \n\nYes. We will fix it in the final version. The related details are also mentioned at line 218. To facilitate constructing a set of demonstrations, we encouraged experts to retrieve information either by web search or browse through the provided story corpora. \n\n6. [Question 5] I think the the template mentioned in line 340 are not provided. Am I understanding correctly? \n\nThank you for pointing this out. We will add the templates to Appendix and release all the source codes in the final version. The templates are as follows: \n\n \n\nThe “long instruction” template for entity similarity prediction (line 363): \n\n> Evaluate the entity similarity between a pair of stories. Assign the pair a score between 0 and 3 as follows: \n> \n> 0 : Unrelated. The two stories are talking about different topics and entities of different types. \n> \n> 1 : Somewhat related. The two stories talk about different entities, and some of them have similar or related types. \n> \n> 2 : Somewhat equivalent. The two stories have different entities, but they have the same types. \n> \n> 3 : Almost equivalent. The entities in the two stories are overlapped or synonymous. \n> \n> Following the above instruction, evaluate the entity / topic similarity for S1 and S2 (only answer by a score from 0, 1, 2, 3): \n> \n> {N-DEMONSTRATIONS HERE} \n> \n> Q :  \n> \n> S1 - {INPUT-S1} \n> \n> S2 - {INPUT-S2} \n> \n> Score : \n\n \n\nThe “long instruction” template for relation similarity prediction: \n\n> Evaluate the relation similarity between a pair of stories. Assign the pair a score between 0 and 3 as follows: \n> \n> 0 : Very poor alignment. Most if not all relationships do not align. \n> \n> 1 : Alignment with significant mismatches. Some of the relationships align, but there are some significant mismatches. \n> \n> 2 : Alignment with insignificant mismatches. Most of the relationships align except for some insignificant mismatches. \n> \n> 3 : Alignment. The relationships can align very well between the two stories. \n> \n> Following the above instruction, evaluate the relational similarity for S1 and S2 (only answer by a score from 0, 1, 2, 3): \n> \n> {N-DEMONSTRATIONS HERE} \n> \n> Q :  \n> \n> S1 - {INPUT-S1} \n> \n> S2 - {INPUT-S2} \n> \n> Score : \n\n \n\nWe insert N (N=0, 1, 3) demonstrations at the “N-DEMONSTRATION HERE” and input the story pairs. The“short instruction” templates has similar format to the above, except that the detailed definition of scores is removed, e.g. “0 : Unrelated. The two stories are talking about different topics and entities of different types.” is replaced with “0 : Unrelated.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a great resource of analogous stories. They have around 24k story pairs annotated with scores for them being analogous. \n2. The paper is very well written. The authors provide information about the prompts, dataset creation, human annotators in great detail. The main paper along with appendix seems well written and complete.\n4. The authors follow a well thought of process of selecting annotators (Section 2.3)\n", "reasons_to_reject": "1. It's not fully convincing that pre-trained off the shelf encoder models make strong enough baseline (Table 2). See the questions to author below for more details. \n2. Since it's basically a dataset paper, it would have been nice to see datacards. (https://sites.research.google/datacardsplaybook/, https://arxiv.org/pdf/2204.01075.pdf )", "questions_to_authors": "1. In section 3.1 for the encoder based baseline, lines 333-343 state that, analogous score was taken as cosine(encoder(s1), encoder(s2)). I am not convinced if similarity based on embeddings from just a pre-trained, off the shelf model would suffice. For example, I don't think that the scores for stories describing electrons and atoms & planets and sun would be close enough. Fine-tuning is necessary. I acknowledge that the authors present two fine-tuned models as well. I just think all other encoder models form a week baseline. \n2. Section 3.1  lines 333-343 just describe how analogous score alpha would be calculated. They don't describe how entity similarity and relation similarity scores are calculated. Or did I miss it? \n3. How is it that models are so good an individually predicting the two scores, but they are not that good at predicting the analogous score? And the fact that analogous score is such a simple combination of the other two makes is perplexing.\n4. Appendix A.1 line 760. Is there something missing at the end of the line. \" using the methods in '…' \" \n5. I think the the template mentioned in line 340 are not provided. Am I understanding correctly?\n", "paper_topic_and_main_contributions": "This paper presents a new dataset (called StoryAnalogy) which has around 24k annotations about analogies in context of stories. \nThe authors argue that the major work till now has been on word analogies, and work on sentences, and paragraph analogies has been limited. \n\nWith this background, the authors present StoryAnalogy in which authors do the following: \n1. source stories from four diverse sources\n2. use LLM to generate analogous stories corresponding to them. \n3. get those stories annotated by humans\n\nFurther, authors show that current LLMs are very far behind humans in determining if two stories are analogies. They perform experiments on various tasks (for e.g: a) predicting similarity scores, b) selecting analogous stories amongst multiple options c) generating analogous stories ). The authors find that even the current best models are far behind humans on these tasks. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses an important and under-studied NLP problem. The main contributions of this work are a substantially novel large dataset of story analogies created by crowdsourcing, a valuable evaluation framework to assess the identification and generation of story-level analogies, a method for enhancing analogy generation which makes use of the novel dataset and fine-tuning with specific models like FlanT5.  The dataset is used experimentally to benchmark the performance of several LLMs; results highlight the task is still challenging both for sentence embedding models and LLMs.  Overall, the work has the potential to help advance research in story-level analogical reasoning. However, the presentation could be improved (see reviewer 2 comment about the imbalance in sections). Although some reviewers consider as limitations the lack of coverage of some domains, my view is that the current version of the work still sufficiently supports the authors’ claims. It is recommended that authors’ incorporate all suggestions by reviewers, as also accounted for in rebuttals. Especially, “any details important for understanding the key aspects of the work should be in the paper rather than in appendices, as per ACL reviewing policies.  

**Pros.**

- The described approach to story-level analogies is novel and addresses a gap in existing NLP/AI research; 

- The paper is generally well written, and the argumentation is solid, but there is some imbalance in details among sections;  

- It states clearly its claims and goals and provides details about the prompts and the dataset creation process; 

- The dataset is extremely valuable and can stimulate future research on the task of complex analogical reasoning; 

- The annotation process is solid and carefully thought-through; 

- The paper presents extensive empirical results in evaluation of current models' capabilities and shows how tested models can benefit from the newly produced dataset; 

- The work has potential for future research.   

**Cons:**

- Important implementation details are given only in Appendix; 

- A proprietary tool is used for analogy generation. While this AC tends to agree with authors that this does not weaken the claims nor the results, it hampers full replicability; 

- The dataset might be too small for an effective fine-tuning of LLMs; 

- Missing justification/explanation of the use of α as the metric for evaluation and cornerstone for analysis;  


**Other revisions needed:**

- Fix font issues in figures and tables;  

- Better explain Figures and tables;
Decision: Accept-Main