Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction
Authors: Siyu Yuan
Jiangjie Chen
Xuyang Ge
Yanghua Xiao
Deqing Yang
Venue: EMNLP 2023 Findings (2023)
Keywords: Analogical Reasoning
Large Language Models
Resources and Benchmark
TL;DR: This paper proposes an analogical structure abduction task to evaluate LLMs, which better aligns the analogical reasoning ability of humans.

Abstract: The vital role of analogical reasoning in human cognition allows us to grasp novel concepts by linking them with familiar ones through shared relational structures. Despite the attention previous research has given to word analogies, this work suggests that Large Language Models (LLMs) often overlook the structures that underpin these analogies, raising questions about the efficacy of word analogies as a measure of analogical reasoning skills akin to human cognition. In response to this, our paper introduces a task of analogical structure abduction, grounded in cognitive psychology, designed to abduce structures that form an analogy between two systems. In support of this task, we establish a benchmark called SCAR, containing 400 scientific analogies from 13 distinct fields, tailored for evaluating analogical reasoning with structure abduction. The empirical evidence underlines the continued challenges faced by LLMs, including ChatGPT and GPT-4, in mastering this task, signifying the need for future exploration to enhance their abilities.

Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction
Siyu Yuan V Jiangjje Chen Xuyang Ge Yanghua Xiao Deqing Yang
School of Data Science, Fudan University
Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University
Fudan-Aishu Cognitive Intelligence Joint Research Center syyuan21 @m. @m. fudan. edu. cn, {ijchen19 9, xyge20, hann:hhhyyyaggeeiggguuuuuuuaa edu. cn
Abstract
Making analogy to explain how biologically eye works an engineer ...
The vital role of analogical reasoning in human cognition allows us grasp novel concepts by linking them with familiar ones through shared relational structures. Despite the attention pre- vious research has given to word analogies, this work suggests that Large Language Models (LLMs) often overlook the structures that derpin these analogies, raising questions about the efficacy of word analogies as a measure of analogical reasoning skills akin to human cognition. In response to this, our paper intro- duces task of analogical structure abduction, grounded in cognitive psychology, designed to abduce structures that form an analogy between systems. In support of this task, we estab- lish a benchmark called ScAR, containing 400 scientific analogies from distinct fields, tai- lored for evaluating analogical reasoning with structure abduction. The empirical evidence underlines the continued challenges faced by LLMs, including ChatGPT and GPT-4, in mas- tering this task, signifying the need for future exploration to enhance their abilities.!
Concept mappings
Film
Retina
Diaphragm
Iris
Aperture Pupil
Lens
Lens
Black paint Chorold
System A: Camera (Engineering)
System B: Eye (Biology)
Aperture should mapped Pupli both are channels for light to enter.
Figure I: An example of establishing an analogy be- tween systems across different domains. Based on the common relational structures, an engineer can abduct concept mappings to learn about the cross- section of eye (right) with the help of camera structure (left).
of the camera structure since both of them exhibit common relational structures.
In this paper, we aim to evaluate the analogical reasoning ability of language models (LMs) align- ing with humans. In this regard, previous work on analogical reasoning mainly focuses on word anal- ogy (e.g.. "king is to man as queen is to woman") which does not evaluate if LMs reason about the analogy between two systems in a manner akin to humans (Turney al., 2003; Mikolov al., 2013b; Boteanu and Chernova, 2015; Gladkova et al., 2016; Chen et al., 2022). There has been a paradigm shift the study of analogies, moving from examining word analogies between phrases to exploring analogies between processes (Turney, 2008; Sultan and Shahaf, 2022), e.g., the process of hurricanes can be analogous to the process of vol- cano cruptions. However, these researches remain limited to the situations within the same domains, leaving cross-domain exploration uncharted, and lack benchmarks. Large language models (LLMss, e.g., Brown et al., 2020; Ouyang et al., 2022; Ope- nAI, 2022, 2023), despite their great abilities
1 Introduction
Analogical reasoning is one of the foundations of human cognition, which helps humans under- stand complex and unfamiliar concepts by relating them to familiar ones (Gentner, 1983; Hofstadter, 2001: Hofstadter and Sander, 2013). In cognitive psychology, theories like the Structure Mapping Theory (SMT) have been proposed to explain the underlying mechanisms behind analogical reason- ing (Gentner and Markman, 1997). According to SMT, individuals gain new knowledge by establish- ing mapping relations between familiar systems unfamiliar systems (system analogy) (Bunge, 1981). As an example in Figure 1, an engineer can learn the eye cross-section by taking the analogy
*Corresponding authors. Ressurccs of this paper can he found at https: //github. com/si htt///iuuuuuscaaaaaaaaaa

---

many tasks including analogy generation (Bhavya al., 2022; Webb al., 2022: Yuan et al., 2023), the evaluation limited to simple word analogies. Little investigation has been done on system analo- gies to align with human cognition.
Word Analogy Test
Query
riverbnkkbridgee
Candidates:
(A) post office:letter
(B) floor:stairs
(C) phone:communication
(D) train:dettination
In this paper, we begin by evaluating and ana- lyzing the analogical reasoning ability of LLMs the word analogy test. Although LLMs, such GPT-4 (OpenAI, 2023), exhibit exceptional per- formance in word analogy recognition, they often fail at abducing the correct structures when solv- ing word analogies. To improve the evaluation and benchmarking of analogical reasoning for bet- ter alignment with human cognitive processes, we draw inspiration from SMT and propose an ana- logical structure abduction task. This task aims to construct the mappings between concepts in two systems based on the relational structure abduction to establish a system analogy.
II: Relational Structure Identification (RSI)
Query
riverbankkrrigg::floorsssaaii
Candidates:
(A) separate (semantic opposite distractor)
(B) linked by
(C) link (relational opposite distractor)
(D) adjacent (similar distractor)
Table Examples of word analogy and RSI task. We also give the distractor types in RSI task for better understanding.
The true answers are highlighted.
SMT suggests that analogy achieved by identi- fying common relational structures between two systems (i.e., system analogy) (Bunge, 1981; Gen- tner, 1983). Key components SMT include
For this purpose, we introduce a benchmark of SCientific Analogical Reasoning with structure abduction, i.e., ScAR, consisting of 400 system analogies across 13 domains with 1600 concept muppings, enriched with background knowledge from Wikipedia and GPT-4 generated explanations. Our experiments reveal that LLMs struggle this task, but can be improved by incorporating back- ground knowledge and explanations in a chain-of- thought (CoT) (Wei et al., 2022) manner.
1. Representation: Structured systems with con- cepts and relations;
2. Mapping:
Comparisons between two rep- resentations for commonalities, resulting in structure abduction between two systems; Evaluation: Analogies are evaluated based on the abduced structures between the two representations.
Our contributions are summarized as follows:
An example of SMT is illustrated in Figure 1. The two systems, i.e,, camera and eye, can be rep- resented into five concepts. Based on the relational structure, aperture should be mapped to pupil since both are channels for light to enter. Adhering the one-to-one mapping, this process focuses on structure abduction to foster comprehension across both domains (Bartha, 2013).
We demonstrate that word analogies do not ad- equately reflect the analogical reasoning abil- ity of LMs to align with human cognition;
We propose the analogical structure abduc- tion task evaluate LLMs from cognitive perspective to align with humans;
We develop a benchmark of scientific ana- logical reasoning with structure abduction, i.e., ScAR, and introduce a CoT prompting method to enhance model performance on this task.
2.2 How are LLMs on word analogy test?
Previous work adopts word analogy tests (i.e., A is to B as C is to D) to evaluate the analogical rea- soning ability of LMs. As illustrated in Table (I), this task can be framed as a multiple-choice question-answering (QA) challenge. We adhere to this paradigm and test the performance of LLMs, e.g., InstructGPT series (Ouyang et al., 2022), Chat- GPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023), on this test. We also adopt InstructGPT embed- dings (text-ada- deext-ada-embeddiing (00d) and follow the method proposed by Ushio et al. (2021), which con- verts word analogies into embeddings and select
2 A Analogical reasoning for LLMs
2.1 A Cognitive Perspective for Analogical Reasoning
We first introduce the cognitive foundations of ana- logical reasoning through the lens of Structure Mapping Theory (SMT), a psychological frame- work proposed by Gentner and Markman (1997).

---

Word Analogy Test
Test on E-KAR
Model
E-KAR
BATS
UNIT 2
UNIT 4 Google
SAT
Mean
Accuracy
Overlap (1)
Embedding
30.53
30.24
34.65
33.56
50.40
36.69
36.01
33.25
22.40
32.44
57.78
47.80
46.99
78.40
37.39
50.13
57.50
47,50
38,93
81.90
50.00
52.78
91.60
48.96
60.70
58.50
49.50
39.31
82.77
56.14
58.33
94.40
47.48
63.07
61.50
48,50
InstructGPTe::
41.60
88.99
62.72
63.66
98.20
$7.27
68.74
65,50
50.00
41,22
81.71
53.07
52.31
93.80
49.26
61.90
64.30
52.47
ChatGPT
44.27
81.59
59.21
55,32
94.80
55.19
65.06
68.48
53.76
53.05
92.42
76.32
71.30
98.80
74.78
77.78
71.69
60.47
GPT-4
60.36
93.97
84,21
81,71
100.00
83.68
83.99
78.50
64.90
Human
77.80
84.85
87.50
66.66
99.41
57.00
78.87
86.43
98.70
Table 2: Accuracy on the word analogy test and RSI test with k-shot learning. Overlap indicates the ratio of correct answers on both E-KAR and RSI tests. obtain human performance on word analogy benchmarks from the original papers. The best results bolded, and the second best underlined.
the answer candidate with the marginal likelihood biased perplexity.
LLMs select the relation constituting the anal- ogy from four options. We construct a bench- mark using 700 E-KAR test data, where annota- tors identify the correct relation of the word anal- ogy. For distractors, annotators write the relation opposite the golden relation (relational oppo- site distractor) and the semantic opposite rela- tion (semantic opposite distractor). Besides, we convert golden and Wikidata relations (Vrandečić and Krötzsch, 2014) into InstructGPT embeddings (text-adas weeeee-meeddinggg 002) and calculate cosine similarity. Then, annotators select an incorrect re- lation with the closest semantics as distractor (similar distractor). Two annotators annotate each data at first, and then we hire third annotator select better one the distractor. For human performance, we test two undergraduates on the RSI test with their results averaged.
We use six benchmarks and design instructions for LLMs to complete the tests. SAT (Turney et al., 2003), UNIT2 (Boteanu and Chernova, 2015) and UNIT4 (Boteanu and Chernova, 2015) come from educational resources. Google (Mikolov et al., 2013b) and BATS (Gladkova al., 2016) are de- rived for word embedding evaluation with semantic and morphological relations. E-KAR (Chen et al., 2022) is from China's Civil Service Examinations with more complex relations and structures.
The results in Table 2 show that 1) InstructGPT embeddings perform poorly the word analogy test; 2) GPT-4 achieves human-level performance and providing examples improves model perfor- mance; 3) Despite GPT-4 exceeding human perfor- mance on most word analogy benchmarks, it lags considerably behind humans on E-KAR.
We evaluate LLMs on the RSI task and calculate the Accuracy and Overlap. Overlap is metric that represents the ratio of data samples that are correctly identified in both tasks to the total number of samples correctly identified in least one of the tasks. higher overlap suggests LMs tend understand word analogies based on structure.
However, analogical reasoning relies on iden- tifying shared relational structures between two systems, but the word analogy test does not explic- itly evaluate structural abduction. Thus, the word analogy test may not reflect model performance in analogical reasoning aligned with humans.
2.3 Is the word analogy test aligned with humans?
Table 2 shows superior model performance the RSI test. However, the low overlap reveals LLMs doing well in the RSI test may not necessar- ily succeed in the word analogy test. According to SMT, analogical reasoning is based on identifying common relational structures between two systems. Such discrepancy indicates that the word analogy test is not aligned with humans, and we need a new way of evaluating and benchmarking analogical reasoning to align with humans.
To confirm our hypothesis, we explore the discern- ing relations between word analogies for LLMs. We analyze word analogies in E-KAR to explore whether LLMs can establish structural relations.
As shown in Table 1 (II), we define rela- tional structure identification (RSI) test where
Details on the benchmarks and prompt templates are vided in Appendix B.

---

3 ScAR: Scientific Analogical Reasoning with Structure Abduction
Statistic
Number
Total system analogies
400
Systems
632
3.1 Schema for Analogies in ScAR
Mappings
1615
Concepts analogies
3230
this paper, we aim to explore the analogical reasoning ability of LLMs to align with humans. Inspired by SMT, we focus on the structure abduc- tion of system analogy (Bunge, 1981). ). As shown in Figure 1, two systems are analogous based on their common relational structure. To construct the system analogy, concepts in System A (e.g., Cam- era) be mapped into corresponding concepts in System B (e.g., Eye), forming multiple one-to-one concept mappings (e.g., Aperture maps to Pupil). This process facilitates analogical reasoning and enables deeper understanding of both systems.
Domain classes
13
Word Analogy
3159
Different mappings
1555
Different concepts
2046
Average mappings in analogies
4.04
Average concept length
1,36
Average background length
148.81
Average explanation length
44.80
Table 3: Main statistics of SCAR.
to rewrite each abstract the background. We prompt ChatGPT with human-writte instructions to ensure that each revised background is limited to 500 words and encompasses all concepts in each system." 5
3.2 Data Collection
System Analogy Selection Given that analogi- cal reasoning usually used in scientific problem- solving, we construct a benchmark of SCientific Analogical Reasoning with structure abduction, i.e., SCAR. We recruit a team of five undergraduate stu- dents with different academic backgrounds to serve annotators for this benchmark.
Explanation Generation As shown in Figure 1, Film maps Retina since both capture light and trans- late it into recognizable information. To rationalize analogical reasoning, we design prompts for GPT- generate explanations for each concept map- ping. 6 To ensure the quality of explanations, we employ two annotators to evaluate the accuracy each explanation in SCAR. The annotation results indicate that 69.35% of the concept mapping expla- nations are accurate, with Fleiss's к 0.93 (Fleiss et al., 1981). Then we ask the annotators who ated the dataset to revise the wrong explanations (495 in total) with their expertise, thereby guaran- teeing the quality of the explanations.
The annotators provided with guidelines of SMT to learn about identifying potential analogies based on the relational structures of systems. To assist our annotators, we furnish them with scien- tific analogies sourced through online research. These resources contain various scientific analo- gies with detailed information and thus can prompt annotators to create system analogies with concept mappings. Overall, annotators manually curate 400 system analogies and define mappings between concepts based their domain-specific expertise. We also ask the annotators to mark the domains in each system analogy. Then, we remove duplicates to collate the benchmark and conduct review pro- cess to verify the correctness and plausibility of the analogies in the benchmark.
Bilinguality: English and Chinese To broaden the scope of this work, we also develop Chinese version of ScAR through translation. We employ three native Chinese-speaking annotators to refine the machine translation provided by Google. Fi- nally, we have a bilingual ScAR benchmark."
Background Knowledge Retrieval We incorpo- rate background knowledge into each system to facilitate the understanding of LMs and streamline the mapping process. To achieve this, we first ex- tract the encyclopedia abstracts from Wikipedia for each system. Considering that abstracts may not include all relevant concepts and could be too lengthy, use ChatGPT (OpenAI, 2022)
3.3 SCAR Analysis
Table 3 shows the main statisties of ScAR. SCAR contains total of 400 system analogies, with 632 systems and 1,614 concept mappings, indicating
'The instruction template revise backgrounds is shown in Appendix C.3. 'The instruction template to generate explanations is shown in Appendix C.4.
"The online resources are shown in Appendix C.I. 'https: h/www wike wwi.eeia..or..
"The data examples of SCAR are provided in Ap- pendix C.5.

---

Computer
1e Task prompt */
Chemical
Philosophy
For two given systems, you are required create an analogy by matching the cnncepts in each system with one another one-to-one mapping.
Geography
Biology
1e Data */
System A: Camera
Physics
System B: Eye
Concepts in System A:
Film, Diaphragm. Aperture, Lens, Black paint Concepts System
Engineering
Economy
Iris, Choroid, Lens, Retina, Pupil
Art
History
Literature
1" Question */
Question: Please establish mappings between
Figure 2: Domain transfer in ScAR benchmark
concepts. The format should be list:
(Conceptl SystemA, Conceptl SystemB),
(Concept2_SystemaA 1e Answer *f
rich and complex analogy structure that can poten- tially challenge LLMs. The benchmark spans 13 domains for evaluating the generalizability of mod- els across various domains. In addition, the bench- mark provides backgrounds and explanations for concept mappings, serving as valuable resources rationalize reasoning.
Answer: (Film, Retina), (Diaphragm, Iris), (Aperture, Pupil), (Lens, Lens), (Black paint, Choroid)
Table 4: An instruction template for LLMs the analog- ical structure abduction task. Generated texts by LLMs are highlighted.
Comparison with Previous Benchmarks We compare ScAR to existing analogy resources by transforming into word analogies. As in Fugure I, we can obtain word analogy, i.e, Film is to Retina as Aperture is Pupil. Overall, as shown in Ta- ble 3, there are 3,159 word analogies ScAR, which exhibits a larger number of word analogies than previous benchmarks.
explore the analogical reasoning ability of LLMs. Given two systems:
S^ = {41 1A
A
SB (4] B
this task involves establish mappings between cepts {4,4]: ; and {1,9, for two systems form analogy between SA and Sg. The task requires the understanding of the relational structures between concepts in both systems and creating a one-to-one mapping between them. Table shows an instruc- tion for LLMs to generate mappings. Our evalu- ation assesses the accuracy of concept mappings and system analogy. A system analogy is deemed correct if all concept mappings between the two systems are accurate.
Domain Analysis We consider the domains of the two systems in each system analogy as a pair, e.g., (Engineering, Biology), and calculate the fre- quency of each pair ScAR to derive the do- main transfer distribution of SCAR, as illustrated in Figure 2. The Figure highlights interdisciplinary aspects, with prominent cross-field relationships between Biology, Engineering, and Physics, em- phasizing their inherent inter-conmection. ScAR shows the prevalence of within-field analogies and asserts the significance of promoting interdisci- plinary connections to foster collaborative advance- ments in knowledge acquisition.
Evaluation
4.1 Evaluation Settings
3.4 Probing Task Formulation
To minimize the impact of instruction design on LLMs, we create 10 different instruction templates for LLMs in this task and select the best one evaluate model performance. Furthermore, we also explore the ability of models to use background knowledge and the CoT prompting (Kojima et al., 2022; Wei et al., 2022) with explanations in this task. The templates are shown in Table 11.
Our task draws inspiration from SMT, which sug- gests that analogical reasoning involves drawing correspondences between two systems, founded on their shared relational structure. To this end, we define the analogical structure abduction task to "The dctailed comparison is shown in Appendix C.6

---

Method
Concept Acc.
System Acc.
Avg
70
70
Acc.
InstrueGPPan
En
Zh
En
Zh
60
wr Step Expl.
60
Alpaca
4.58
0.00
1.75
0.00
1,58
50
50
w/ 1-shot
9.97
14.11
4,50
5,50
8.52
40
ChaGGT
Vicuna
9.04
26.42
4.50
0.12
10.02
30
30
w/ Step
w/ 1-shot
40.93
16.00
21.75
0.00
19.67
Expl.
20
20
InstructGPPeer
3.18
0.00
1.75
0.00
1.23
2 4
8 910
1
123
910
w/ 1-shot
8,43
5.43
2.75
2.00
4.65
Template id
Template id
InstrucGGPT>>>>
51.18
37.83
36.18
25.37
37.64
(a) Acc. on InstructGPPeas:
(b) Acc. on ChatGPT.
w/ 1-shot
54.71
48.82
40.25
33.50
44.32
w/ Backg.
51.36
54.86
34.50
41.25
45.49
70
50
w/ 1-shho-BBcckg.
55.46
54.94
41.30
49.20
60
InstructGPTes
52.49
39.91
36.25
26.50
38.79
50
w/ 1-shot
55.36
47.70
40.50
33.00
44.14
40
w/ Backg.
54.95
54.76
37.25
41.75
47.18
40
GPPT
w/ 1-shottBBackg.
58.63
53.92
42.60
43.50
49.66
A w/ Step O/ Fapl.
35
O-shou
1-shot
w/ Expl.
w/ Backg
ChatGPT
66.52
66.26
46.61
52.00
57.85
20
w/ 1-shot
69.99
70.33
51.25
56.75
62.08
2 4 5 6 910 Template id
w/ Backg.
70.78
71.97
52.50
61.25
64.13
w/ Baakk.
72.80
74.03
57.39
59.25
65.87
(c) Acc. GPT-4.
(d) Average Acc.
GPT-4
73.28
69.77
58.50
58.50
65.01
w/ 1-shot
71.66
71.96
59.25
61.75
66.16
Figure 3: Subfigures (a-c) show the system accuracy of L.LMs enhanced with different types of CoT prompt- ing. The average system accuracy of LLMs (ie., InstructGPToas. ChatGPT and GPT-4) on different tem- plates is shown in Subtigure (d). All results are from the English version of SCAR.
w/ Backg.
75.10
73.11
63.00
62.74
68.49
w/ Backkt
77.17
72.74
64.00
62.50
69.10
Human
85.94 35.94
88.46
83.37
86.36
86.03
Table 5: Main Results of different LLMs.. We compare vanilla LLMs and LLMs with one added example (w/ 1-shot) or background (w/ Backg.) ). The best results are bolded and the second best are underlined.
other in science, with their results averaged. Result in Table 5 shows that: 7) GPT-4 achieves the best performance across both languages, and adding example can enhance model performance. How- ever, its ability still lags behind that of humans; 2) Smaller models perform poorly, but training on dialogue data can improve model performance; 3) The performance of the InstructGPT series and ChatGPT in Chinese is improved significantly by adding background knowledge, highlighting the model's struggle with domain terminology in Chinese to affect their performance.
4.2 Model Choices
We choose Alpaca (Taori et al., 2023) (7B),
Vicuna (Chiang et al., 2023) (7B), In-
structGPT series (Ouyang et al., 2022) (i.e., InstructGPTourie 691 (6. (6.7B), InstructGPT ee2 (2 175B) and InstructGPTo0 (2175B)). ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023). Alpaca and Vicuna are fine-tuned from 7B LLaMA model (Touvron et al., 2023) with instructions or user-shared conversations. InstructGPT series are variants of GPT-3 (Brown et al., 2020) fine-tuned on instructions using reinforcement learning with human feedback (RLHF). ChalGPT is built InstructGPT and trained dialogue data using RLHF. GPT-4 the most advanced LLM so far.
4.4 Analysis
Will step-by-step reasoning help solve SCAR' ?? We dig into the effectiveness of the CoT prompting technique for LLMs in structure abduction. We adopt one example with the explanations of con- cept mappings to induce the LLMs to generate in- termediate steps in analogical reasoning (w/ Expl.). One instruction template is shown in Table 11 (II). We also add "Let's think step by step" before each answer (w/ Step), which improves zero-shot rea- soning for LLMs (Kojima et al., 2022). We conduct experiments on ten different templates to mitigate
4.3
Overall Performance
We first compare LLMs with O-shot, 1-shot, and background knowledge (w/ Backg.). Due the limited input length, we excluded back- ground knowledge from Alpaca, Vieuna, and InstructGPTcuris 681 For human performance, we test two graduate students, one in liberal arts and the

---

100
Art
Max-similarity
Greedy
Kuhn-Munkres
Biology
English
Chinese
230
55.
80
System Accuracy
Computer
378
150
50
423
530
0.
XX
83.
40-
Econemy
331 mm
Engineering
234
26.0
10
Geography
History
236
500
40
Sentonce
BERT
SimCSE toxt-aca- embedding-002
Sentence
BERT embedding--002
SimCSE toxt-ado-
Literature
15.5 511
Mathematics
88
520
334
FAx 90.0
Figure 5: The system accuracy of different em- bedding methods (i.e., Sentence-BERT (Reimers and Gurevych, 2019), SimCSE (Gao al., 2021). text text-ada-embeddinn 002 (Ouyang et al., 2022)).
Philosophy
230
10.3300
06.6
234
23.3
25.0
1000
20
Physics
Sport
Blolegy
Chemical
Computer
Economy
Engineering
Geography
History
Literature
Mathematics
Ihilosophyy
Physics
Sport
Нow do the beabeeddnn-maappine algorithms of concepts affect system analogies? We explore creating system analogies based on the embeddings of each concept in the systems. To achieve this, we implement three distinct mapping algorithmss, lever- aging the cosine similarity score of embeddings to facilitate the process: 1) Max-Similarity Algorithm: This algorithm maps each concept in System A to the concept from System B that exhibits the highest cosine similarity score, implying the same concept from System can map to multiple con- cepts from System A; 2) Greedy Algorithm (Zhang et al., 2000): This algorithm iteratively maps the concepts with the highest cosine similarity. In each iteration, the concepts with the highest similarity mapped and excluded from further considera- tion, generating one-to-one mappings without over- all optimality; 3) Kuhn-Munkke Algorithm (Kuhn, 1955): This combinatorial optimization algorithm generates one-to-one mappings, providing a glob- ally optimized solution." 10
Figure 4: The heatmap depicts the system accuracy of
GPT-4 across different domains.
human design bias for LLMs. Results in Figure (a-c) show that: 1) CoT prompting enhances GPT- 4 performance in structure abduction but harms ChatGPT and InstructGPTeg ee3 performance due to flawed reasoning; 2) CoT prompting with expla- nations outperforms the "Let's think step by step" approach, highlighting the importance of explana- tions CoT prompting.
Does instruction design affect the model perfor- mance? To answer this question, we calculate the average system accuracy of LLMs across all templates. Results Figure 3(d) show that LLMs are sensitive to the instruction design. However, providing an example, additional backgrounds, or utilizing CoT prompting can enhance the robust- ness of LLMs to the instruction design.
Results in Figure reveal the insufficiency of the max-similarity algorithm in creating viable sys- tem analogies in SCAR. It underscores the infer- ence that human-like analogical reasoning does not rely solely surface-level embedding similarities. However, both the Greedy and Kuhn- Munkres al- gorithms display enhanced performance, suggest- ing that the lackluster results of the LLMs on the structure abduction task might be attributed to their weak mapping capabilities. This observation in- dicates that human-like reasoning could employ embeddings alongside mapping algorithms com- plementary tools to deduce system analogies.
Do models behave differently for analogies across various domains? The heatmap in Fig- ure 4 shows the system accuracy of GPT-4 across different domains. We find that: 1) The perfor- mance varies considerably for system analogies of different domains, indicating limited sensitivity of LLMs to domain boundaries; 2) In certain domains, e.g., literature, intra-domain system analogies have low accuracy, indicating LLMs may have some difficulties in these fields; 3) System analogies be- tween similar domains (i.e., biology and chemistry) show high accuracy, demonstrating the potential for knowledge transfer.
Kuhn-Munkres
"We select the hest results of GPT-4 on the English version of SCAR to draw the heatmap.
PPlccse refer to Appendix for an in-depth explanation of the employed emhedding similarity methods.

---

Precision
Recall
F1
5 Related Work
1-Shot
Performance (%)
Analogical reasoning has been an area of interest in the AT community, primarily focusing on word analogies (Mitchell, 2021). Early researches in word analogy focus on evaluating the quality of word embeddings, which examines linear relations between words (Turney et al., 2003; Mikolov et al., 2013b; Gladkova et al., 2016) and can be effec- tively addressed through vector arithmetic for neu- ral word embeddings such as Word2 (Mikolov et al., 2013a) and GloVe (Pennington et al., 2014).
30
InstructGPT ChalGPT
GPT-4
InstructGPT ChatGPT
GPT-4
003
003
Figure 6: The performance of LLMs in the open ana- logical structure abduction task. All results are from the English version of ScAR.
4.5
Open Analogical Structure Abduction
In recent years, some studies explore anal- ogy understanding of LMs on various bench- marks (Fournier et al., 2020; Ushio et al., 2021) and fine-tuned LMs using constructed knowledge bases of analogies to improve performance (Li et al., 2018, 2020; Yuan al., 2023). New word analogy benchmarks with more complex relational structure (Chen et al., 2022; Czinezoll et al., 2022) and with multimodal elements (Zhang et al., 2023) (Zhang et al., 2023) are built to evaluate the per- formance of multilingual and multimodal models. However, gap exists between word analogy for- mats and the nature of analogical reasoning in hu- man cogmition (Hofstadter, 2001; Bartha, 2013; Gentner and Maravilla, 2017), limiting the word analogy task to reflect the LLMs' analogical rea- soning ability aligning with humans.
the above experiments, we evaluate LLMs' abil- ity of structure abduction in close setting, where the concepts of each system are given (as in Ta- ble 4). However, more intriguing question arises: Can LLMs perform analogical reasoning for sys- tems with open set of concepts, where concepts are not explicitly given? In this case, models must first identify concepts from contexts and then gen- erate concept mappings to form a system analogy. This open analogical structure abduction problem more closely simulates the process of humans dis- covering and acquiring knowledge.
We provide LLMS with the background descrip- tion texts of systems simulate an open setting. LLMs are expected to retrieve concepts that can be used create mappings from backgrounds and then establish concept mappings to form system analogies." 1I For evaluation, we automatically cal- culate the recall of concept mappings based on SCAR to measure the correctness of newly gener- ated mappings with annotated mappings (as in the close setting). Since some reasonable mappings may not be included in SCAR, we need to manu- ally evaluate the precision. We randomly sample 50 data from ScAR and let LLMs generate con- cept mappings in the open setting. Two annotators assess the precision of the generated concept map- pings with Fleiss's a = 0.86. Then Fl score can be calculated based on precision and recall.
There has been a paradigm shift toward explor- ing analogies between situations (Turney, 2008; Sultan and Shahaf, 2022). These works are in- spired by the SMT (Gentner, 1983), which aims to establish mappings between concepts in two domains based on a common relational structure. Nonetheless, Turney (2008) focuses on simple com- monsense relations. Sultan and Shahaf (2022) ar- gue that two processes with similar questioning formats can be analogies, which does not address complex structures and yield unsatisfactory perfor- mance on discovering analogies between different domains. Furthermore, some studies also explore the analogy generation of LLMs (Bhavya et al., 2022; Yuan et al., 2023; Ding et al., 2023). How- ever, they mostly evaluate word analogies or simple analogies between two sentences, leaving complex structures in analogical reasoning unstudied. Webb et al. (2022) and Hu et al. (2023) examine the ab- stract language-basee analogy task and evaluate the analogical reasoning ability of LLMs on this task. Compared to their task, our task requires the inten- sive involvement of commonsense, encyclopedic,
The results in Figure 6 show that LLMs can establish some concept mappings even when con- cepts are not directly given. Despite the relatively low recall, higher precision indicates LLMs' abil- ity to form new concept mappings, which can be utilized to further improve ScAR.
11 O'nne instruction template is shown in the Appendix D.

---

and cultural (e.g., idiom and historical) knowledge. Recent researchers study Al alignment to guide AT toward achieving human preferences and ethical principles (Li et al., 2022; Rao et al., 2023; Park et al., 2023). We explore the analogical reasoning ability of LLMs with complex structure abduction, which is more aligned with human cognition.
imum wage and consent to using the ScAR for rescarch purposes.
Acknowledgement
We thank the anonymous reviewers for their valu- able comments, and Yikai Zhang and Shuang Li from Fudan University for their useful suggestions for the manuscript. This work is supported by the Chinese NSF Major Research Plan 1122222..... Shanghai Science and Technology Innovation Ac- tion Plan (No.215111 (110000) and the Science and Technology Commission of Shanghai Municipality Grant (No. 22511105902).
6 Conclusion
In this paper, we explore the analogical reasoning ability of LLMs. We highlight word analogies ne- glect structures and thus can not evaluate L.MSs in alignment with human cognition. To better evaluate LLMs aligning with humans, we propose an analog- ical structure abduction task with a new benchmark, SCAR. Experiments show that LLMs struggle with this task, but incorporating background knowledge and CoT prompting can improve their performance. We hope the ScAR can be valuable resource to advance the research on analogical reasoning.
References
Paul Bartha. 2013. Analogy and analogical reasoning.
Bhavya Bhavya, Jinjun Xiong, and ChengXiang Zhai. 2022. Analogy generation by prompting large lan- guage models: A case study of instructgpt. In Pro- ceedings of the 15th International Conference on Nat- ural Language Generation, pages 298-312, Water- ville, Maine, USA and virtual meeting. Association for Computational Linguistics.
Limitations
We only instruct LLMs to establish concept map- pings between systems in the analogical structure abduction task, leaving the discovery of novel analogies unexplored. Such a limitation highlights the potential for future work to adopt structure ab- duction to uncover analogies and learn about new knowledge.
Adrian Boteanu and Sonia Chernova, 2015. Solving and explaining analogy questions using semantic net- works. Proceedings of the AAAI Conference on Arti- ficial Intelligence, 29(1)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates. Inc.
Another limitation of this work is that our eval- uation of the analogical structure abduction task relies on concept mappings. Although this crite- rion aligns with humans, it remains a challenge for the model. Future studies can consider designing more appropriate evaluation tasks. Additionally, although we mitigated the impact of templates on model results by designing ten templates and choos- ing the best results for evaluation, we believe there remains room for improvement in instruction de- sign to fully harness the capability of LLMs.
Mario Bunge. 1981. Analogy between systems. Inter- national Journal Of General System, 7(4):221-223.
Ethics Statement
Jiangjie Chen, Rui Xu, Ziquan Fu, Wei Shi, Zhongqiao Li, Xinbo Zhang, Changzhi Sun, Lei Li. Yanghua Xiao, and Hao Zhou. 2022. E-KAR: A benchmark for rationalizing natural language analogical reason- ing. In Findings of the Association for Computa- tional Linguistics: ACL 2022, pages 3941-3955, Dublin, Ireland. Association for Computational Lin- guistics.
All authors of this work abide by the provided Code of Ethics. Annotators recruited by our institution annotate the system analogies in SCAR. The an- notation quality is ensured through double-check strategy outlined in Section 3. We ensure that the privacy rights of all annotators are respected in the annotation process. As described in our paper, all annotators are compensated above the local min-
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E, Gonzalez, lon

---

Stoica, and Eric Xing, 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Xiaoyang Hu, Shane Storks, Richard Lewis, and Joyce Chai. 2023, In-context analogical reasoning with pre-trained language models. In Proceedings of the s Anmual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1953-1969, Toronto, Canada. Association for Com- putational Linguistics.
Tamara Czinczoll, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2022. Scientific and creative analogies in pretrained language mod- els. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2094-2100, 100, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguisties.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
Zijian Ding, Arvind Srinivasan, Stephen MacNeil, and Joel Chan. 2023. Fluid transformers and creative analogies: Exploring large language models' capacity for augmenting cross-domain analogical creativity. arXiv preprint arXiv:2302.1283: 12832.
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly, 2(1-2):3-99.
Joseph L Fleiss, Bruce Levin, Myunghee Cho Paik, al. 1981. The measurement of interrater agreement. Statistical methods for rates and proportions, 2(212- 236):22 23.
Peng- Husug-Hsunn Li. Tsan- Yu Yang, and Wei- Ma. 2020. CA-EHN: Commonsense analogy from E-HowNet. In Proceedings of the Twelfih Language Resources and Evaluation Conference, pages 2984-2990, Mar- seille, France. European Language Resources Asso- ciation.
Louis Fournier, Emmanuel Dupoux, and Ewan Dun- bar. 2020. Analogies minus analogy test: measuring regularities in word embeddings. In Proceedings of the 24th Conference on Computational Natural Lan- guage Learning, pages 365-375. Online. Association for Computational Linguistics.
Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. 2018. Analogical reasoning on Chinese morphological and semantic relations. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Pupers), pages 138-143, Melbourne, Australia. Association for Computational Linguisties.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedingy of the 2021 Conference on Empirical Methods in Natural Language Process- Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Xingxuan Li, Yutong Li, Linlin Liu, Lidong Bing, and Shafiq Joty. 2022. Is gpt-3 psychopath? evalu- ating large language models from a psychological perspective. arXiv preprint 1xXiv:2212.10529,
ing, pages
Dedre Gentner. 1983. Structure-mapinng A theoretical framework for analogy. Cognitive science, 7(2):155- 170.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jef- frey Dean, 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Dedre Gentner and Francisco Maravilla. 2017. Ana- logical reasoning. In The Routledge International Handbook of Thinking and Reasoning, pages 186- 203. Routledge.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746-751, Atlanta, Georgia. Association for Computational Linguistics.
Dedre Gentner and Arthur B Markman. 1997. Struc- ture mapping in analogy and similarity. American psychologist. 52(1):45.
Anna Gladkova, Aleksandr Drozd, and Satoshi Mat- suoka. 2016. Analogy-based detection of morpholog- ical and semantic relations with word embeddings: what works and what doesn't. 't In Proceedings of the NAACL Student Research Workshop, pages 8-15, San Diego, California. Association for Computational Linguistics.
Melanie Mitchell. 2021. Abstraction and analogy- making in artificial intelligence. Annals of the New York Academy of Sciences, 1505(1):79-101.
OpenAL. 2022. Chatgpt.
OpenAL. 2023. Gpt-4 technical report.
Douglas Hofstadter. 2001. Analogy the core of cognition. The analogical mind: Perspectives from cognitive science, pages 499-538.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with
Douglas R Hofstadter and Emmanuel Sander. 2013. Sur- faces and essences: Analogy as the fuel and fire of thinking. Basic books.

---

human feedback. In Advances Neural Information Processing Systems.
Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages -609-3624, Online. As- sociation for Computational Linguisties.
Peter S Park, Philipp Schoenegger, and Chongyang Zhu, 2023. Artificial intelligence in psychology research. arXiv preprint arXiv:2302.07267.
Denny Vrandečić and Markus Krötzsch. 2014. Wiki- data: free collaborative knowledgebase. Communi- cations of the ACM. 57(10)78 85.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Taylor Webb, Keith Holyoak, and Hongjing Lu. 2022. Emergent analogical reasoning in large language models. arXiv preprini arXiv:2212.09196
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi. Quoe V Lc, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning large language models. Advances in Neural Information Processing Systems.
Haocong Rao, Cyril Leung, and Chunyan Miao. 2023. Can chatgpt assess human personalities? a general evaluation framework. arXiv preprint arXiv:2303.01248.
Siyu Yuan, Jiangjie Chen, Changzhi Sun, Jiaqing Liang. Yanghua Xiao, and Deqing Yang. 2023, Analogyko kb: Unlocking analogical reasoning of language models with million-scal scale knowledge base. arXiv preprint arXiv:2305.0S5 059944
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguisties.
Ningyu Zhang, Lei Li, Xiang Chen, Xiaozhuan Liang, Shumin Deng, and Huajun Chen. 2023, Multimodal analogical reasoning over knowledge graphs. In The Eleventh International Conference on Learning Rep- resentations.
Oren Sultan and Dafna Shahaf. 2022. Life is a circus and the clowns: Automatically finding analo- gies between situations and processes. Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3547-3562, Abu Dhabi, United Arab Emirates, Association for Computational Linguistics.
Zheng Zhang, Scott Schwartz, Lukas Wagner, and Webb Miller. 2000. A greedy algorithm for aligning dna sequences. Journal of Compulational biology, 7(1- 2):003-214.
A Author Contributions
Siyu Yuan Lead the project, develop the original method and original code, lead the curation of the dataset, contribute the original experiments, and contribute to the original manuscript.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois. Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An cestruction-followiin Ilaaa model. https:// gi thub. com/ wat.uuulllasttttaaaaaaaaaa pacc...
Jiangjie Chen Conceptualization of the origi- nal idea, supervision of the research activity plan- ning and execution, contribution to the original manuscript and figures, and acquisition of financial support for the project.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Xuyang Ge Contribute to the experiments and data curation.
Peter D Turney. 2008 The latent relation mapping engine: Algorithm and experiments. Journal of Arti- ficial Intelligence Research, 33:615-655.
Yanghua Xiao Provide financial support for the project and revise the manuscript.
Peter Turney, Michael I. Littman, Jeffrey Bigham, and Victor Shnayder. 2003. Combining independent modules in lexical multiple-choice problems. Re- cent Advances in Natural Language Processing III: Selected Papers from RANLP, 2003:101-110.
Deging Yang Provide financial support for the project, revise the manuscript, and oversee the re- search activity execution.
Asahi Ushio, Luis Espinosa Anke, Steven Schockaert, and Jose Camacho- Collados. 2021. BERT is to NLP what AlexNet is to CV: Can pre-trained language models identify analogies? In Proceedings of the 59th Annual Meeting of the Association for Compu- lational Linguistics and the 11th International Joint
B Word Analogy Test
Camacho-Collados
B.1
Benchmark
We compare LLMs with human performance in 6 different analogy benchmarks.

---

E-KAR (Chen al., 2022): A knowledge- intensive analogical reasoning benchmark from China's Civil Service Examinations (CSE), including linguistic, commonsense, en- cyclopedic, and cultural knowledge.
Data
Analogy
Lang.
Backg. Expl.
SAT
374
En
Google
550
En
X
UNIT 2
252
En
K
UNIT
480
En
BATS
1998
En
E-KAR
1251
Fm
BATS (Gladkova et al.. 2016); This bench- mark features over 1,000 analogies in four cat- egories: lexicographic, encyclopedic, deriva- tional, and inflectional morphology.
E-KAR
1655
Zh
SCAR
3159
En
SCAR
3159
Zh
Table 6: Comparison between ScAR and previous anal- ogy data source: numbers of word analogies, language (Lang.) and whether the benchmark has background information (Backg,) and explanations (Expl.).
UNIT2 (Boteanu and Chernova, 2015): A benchmark using word analogy problems from an educational resource.
UNIT4 (Boteanu and Chernova, 2015): Sim- ilar U2, this benchmark comes from an educational resource but is more challenging.
For example, "When Rutherford (following Nagoka) conceived of the atom miniature solar system circling the nucleus planets circle the sun". These resources prompt notators to create system analogies with concept mappings.
Google (Mikolov et al., 2013b): A benchmark for intrinsic evaluation of word embeddings, containing semantic and morphological rela- tions.
C.2 Crowd-sourcing Details
We have recruited a team of five undergraduates with diverse academic backgrounds in Computer Science, History, Physics, Biology, and Chemistry. Among them, the student majoring in Computer Science has a minor in Economics, while the stu- dent majoring in History has minors in Creative Writing and Philosophy. We pay each annotator $8/h, exceeding the local minimum wage $5/h.
SAT (Turney al., 2003): benchmark de- rived from US college admission test with 374 word analogy problems.
B.2 Prompt Templates for Large Language Models
As shown in Table we design the instruction for LLMs to generate the answers.
C.3 Backgroud Rewriting Template
Details of SCAR
As shown Table 10 (I), we design the instruc- tion for ChatGPT to ensure that each revised back- ground is limited to 500 words and encompasses all concepts relevant to each system.
C.1 Resource of SCAR
To facilitate more efficient and effective annota- tion process, we furnish annotators with scientific analogies sourced through online research:
C.4 Explanation Generation Template
https://homeworkk
Table (II) shows the human-written instruction for GPT-4 to generate the explanation for each map- ping in SCAR.
s study.comm/explanttionn Mht--is-analogy-iim-cceeeeee
www. www.csun.edu/sceee analogy.
www.cum..ed//ccinne//////aaaaaa/
C.5 Data Examples of ScAR
Table 7 presents some examples of SCAR for a better understanding.
httpss//ecddnneeedduutttiirrrrrr com//tesching-sciencce//
C.6 Comparison to Previous Benchmarks
constructivistt-edaagey
aakking-tte-uunffallll-----iiiin
We compare ScAR with the resources related the problem of analogy. As the existing bench- marks are based on word analogies, we transform SCAR for appropriate comparisons. We combine
science-analogies,
www. www.engagefastllerrning.com

---

Limit Modification
Firewall
1a Task prompt *1
(Biology)
(Computer)
For two given systems, are required to create an analogy by extracting concepts from the backgrounds of systems and matching the concepts each system with another in one-to-one mapping.
Prokaryotes
Computer
Exogenous DNA
Virus
Restriction Enzyme
Antivirus Software
Cut
Intercept
Degradation
Clear
Data */
Tide
Lift
System A: Camera
(Gcography)
(Engincering)
System B: Eye
Ocean
Platform
Background of System A:
Moon
Console
camera is a device that captures visual images by...
High tide
Rise
Background of System B:
Ebb and Flow
Decline
The eye is remarkable organ that allows us...
3
Sound
Light
Question *
(Physics)
(Physics)
Question: Please extract concepts from the backgrounds of systems and establish the mappings between concepts.
Low
Red
High
Violet
The format should be a list:
Echoos
Reflects
Sooncept_Systeneee Concepti_ L_yttemB,
Loud
Bright
(Concept2_ SystemA, Concept2 SystemB),.. Answer
Quiet
Dim
Horn
Lens
Answer: (Film, Retina), (Diaphragm, Iris), (Aperture, Pupil), (Lens, Lens). (Black paint, Choroid)
4 Computer Systems
Urban
(Computer)
(Geography)
Operating System
Mayor
Process
Resident
Table 8: An instruction template for LLMs in the open analogical structure abduction task. Generated texts by LLMs are highlighted.
Resource manager
Municipal Facilities
File System
Architecture
5
Chemistry
Cooking
(Chemical)
(Art)
Temperature
Heat
exhibits the highest cosine similarity score. This method performs mapping with replacement, mean- ing that concept from System can be mapped to multiple concepts from System A; 2. Greedy algorithm (Zhang et al., 2000), which iteratively maps concepts exhibiting the highest cosine similarity in each step. In each round of iter- ations, we first calculate all cosine similarity scores between concepts in the two systems. Then, we map the concepts with the highest cosine similarity scores, and the concepts that have been mapped will not be considered in the next round of itera- tions. This strategy generates one-to-one mappings while not considering the overall optimality.
Pressure
Firepower Food Size Raw Material Dishes
Reactant Concentration
Reactant
Product
Table 7: Some data examples of SCAR. We give the system analogies (=) with concept mappings (-)).
the two concept mappings within the same system analogy to form word analogy. For instance, in Fugure 1, can obtain word analogy, i.e., film is retina as aperture is to pupil. As reported in Table 6, our method exhibits larger number of word analogies with bilingual language.
3. Kuhn-Munkres algorithm (Kuhn, 1955), combinatorial optimization technique for solving one-to-one mapping problems. Given matrix C, with each C(i.,j] representing the cost of match- ing vertex (a "worker"") to vertex j (a "job"), the objective is to find minimal cost assignment of workers to jobs. Let X be a boolean matrix, with Cii.j] = if and only if row is assigned to col- umn j. The optimal assignment cost is given by:
D Details about A Analogical Structure Abduction Task
We show the instructions combining backgrounds and the CoT prompting with explanations in Ta- ble 11 (I) and (IT). One numan-writter instruction for the open analogical structure abduction task is shown in Table 8.
E Embedding Similarity Method
min ΣΣ C.y.
(1)
We convert concepts in two systems into differ- ent embeddings with the following strategies and calculate cosine similarity between concepts:
where X is square, each row corresponds ex- actly one column, and each column corresponds to exactly one row.
1. Max-iiiilar algorithm, which establishes a mapping with the concept from System B that

---

I: Word Analogy Test
II: Relational Structure Identification (RST)
1 Task prompt */
Task prompt */
Find the most analogous candidate answer that follows the relations in the query.
What relationship is the given analogy?
/ Examples */
19 Examples *
Question: army.order.handdbam leader
Question: broom:dustpan
Choice:
Choice:
A: govern
A: lock:key
B: violate
B: frame:lens
C: obey
C: scarf-hat
D: cooperate
D: toothbrush:cup
Please choose A, B. Cor
Please choose A, B, or
Answer: C
Answer:
18 Test data */
Test data *
Query: riverbankcbriggeeffoosstta
Question: admire:respect
Choice:
Choice:
A: separate
A: like:adore
B: linked by
B: oppress:exploit
C: link
C: spouse:husband wife
D: adjacent
D: relghivss:niighbors
Please choose A, B. Co
Please choose A, B, Cor Answer:
Answer: B
Table 9: Prompt templates for LLMs in word analogy task and RSI task. Generated texts by LLMS are highlighted.
Background Revision
1+ Task prompt */
Given a deseription system with list of concepts related to the system, please generate short introduction of the system according to the description concepts within 500 words.
1* Data */
System:
Biosphere
Description:
The biosphere, also known as the ecosphere, the worldwide sum of all ecosystems. It can also... Concepts: biology. biodiversity, ecosystem
Introduction: The biosphere refers to the sum of all ecosystems on Earth, including the inteructions between living organisms and their environment...
II: Explanation Generation
1' Task prompt
Given two systems, each of which consists some concepts, establishing mappings between the concepts can make the two systems analogous. Given mappings, please explain why such mappings can form an analogy. Hint: You answer from a structural point of view.
Data
System I: Immune System
System 2: Army
Concepts in System 1:Immune Cells, Antibody, lymphoid tissue, regulatory organs
Concepts in System 2:Soldier, Weaponry, base, headquarters
Mappings: [Immune Colls,Soldier]]AAnnthoo
Wccaponry].Iyyen Ilyphhod Jissuctbase_Jreeulattn organs,heudduarters]
Explanations:
1. Immore cells correspond to soldiers: the immune system, immune cells (such as white blood cells) are the soldiers responsible for identifying and eliminating pathogens. They can quickly recognize, attack, and clear invading pathogens. In the military, soldiers are the basic force for executing tasks and defending national security. In this mapping, immune cells and soldiers both represent the executors of their respective systems.
busicforce
2. Antibodies correspond to weaponry: In the immune system, antibodies are specific proteins produced by immune cells. They can recognize and bind to pathogens, helping the immune system clear them. In the military, weaponry is the tool soldiers use to execute tasks. Antibodies and weaponry both represent specific tools used to accomplish tasks.
3. Lymphoid rissue corresponds to bases: In the inmune system, lymphoid rissue (such as the spleen and lymph nodes)
is where immune cells are produced, stored, and exchanged. In the military, bases are where soldiers live, train, and organize tasks. In this mapping. lymphoid tissue and bases are both support facilities for their respective systems.
4. Regulatory organs correspond to headquarters: In the inmune system, regulatory organs (such as the thymus and bone murrow) are responsible for coordinating the production, development, and function of immune cells, ensuring the normal operation of the inmuun system. In the military, headquarters are responsible for organizing, commanding, and coordinating military actions. Regulatory organs and headquarters both represent the munagement and coordination organizations of their respective systems.
Table 10: Prompt templates for background revision (generated by ChatGPT) and explanation generation (generated by GPT-4.. Generated texts by ChatGPT and GPT4 are highlighted.

---

I: Instruction Template with Background
1t Task prompt "/
For two given systems, you required create analogy by matching the concepts in each system with one another in one-to-one mapping.
1 Data */
System A: Camera. A camera is device that captures images by exposing photosensitive surface to light...
System B: Eye. The human eye complex optical instrument that allows to perceive the world around... Concepts in System A: Film. Diaphragm, Aperture, Lens, Black paint Concepts in System B: Iris, Choroid, Lens, Retina, Pupil Please establish the analogies between the concepts.
The format should be a list: (Concept) Saaiceptt_SSyttmmah Answer: (Film, Retina), (Diaphragm, Iris), (Aperture, Pupit), (Lens, Lens), (Black paint, Choroid)
Sanneptt_SystemB),, (Concept2_ 2_sssetm,_,t Concept2_Systemm
II: Instruction Template with Explanation (CoT)
1 Task prompt o/
For two given systems, you required create analogy by matching the concepts in each system with another in one-to-one mapping.
1 Example
System A: Immune System
System B: Army
Concepts in System A:Immunne Cells, Antihody, lymphoid tissue, regulatory organs
Concepts in System B:hasc, Weaponry, headquarters, Soldier
Please establish the analogies between the concepts. The format should a list: (Concepti_Systen) Answer: let's think step by step.
Conceptl_Systentte (Concept2_Systeeme Concept2_SSytemmB)
Immune cells correspond soldiers: the immune system, immune cells (such as white blood cells) are soldiers responsible for identifying and eliminating pathogens. They can quickly recognize, attack, and clear invading pathogens. In the military. soldiers are the basic force for executing tasks and defending national security. this mapping. immune cells and soldiers both represent the executors of their respective systems.
2. Antibodies correspond to weaponry: In the immune system, antibodics are specific proteins produced by immune cells. They can recognize and bind to pathogens, helping the immune system clear them. In the military, weaponry is the tool soldiers use to execute tasks, Antibodies and weaponry both represent specific tools used to accomplish tasks.
3. Lymphoid tissue corresponds to bases: In the immune system, lymphoid tissue (such as the spleen and lymph nodes)
is where immune cells are produced, stored, and exchanged. In the military, bases are where soldiers live, train, and organize tasks. In this mapping. lymphoid tissue and bases are both support facilities for their respective systems.
4. Regulatory organs correspond to headquarters: In the immune system, regulatory organs (such as the thymus and bone marrow) are responsible for coordinating the production, development, and function of immune cells, ensuring the normal operation of the immune system. In the military, headquarters are responsible for organizing, commanding, and coordinating military actions. Regulatory organs and headquarters hoth represent the management and coordination organizations of their respective systems.
Therefore, the final answer is: (Immune Colls,,oldier), (Antibody, Weaponry), (lymphoid tissue,hase), (regulatory headqurrar))
/+ Test Data #/ System A: Camera
System B: Eye
Concepts in System A: Film. Diaphragm, Aperture, Lens, Black paint
Concepts in System B: Iris. Choroid, Lens, Retina. Pupil
Please estahlish the analogies between the concepts.
The format should be list: (Conceptl_Systent Conceptl_Systembee (Concept__SystemmA Concept2_Systemm))
Answer: let's think step hy step-
1. Film corresponds to Retina:
Table 11: Instruction templates for LLMs in the structure mapping abduction task. Generated texts by LLMs are highlighted.


Reviews:
[{"review_id": "yuHFUXJjjQ", "reviewer": "Reviewer_6uTV", "summary": "", "questions": "It is not clear to me how the classic \"queen is to woman as king is man\" is really an instance of analogy since basically only the similarity of two vectors to one another  is being compared: how is that really analogical reasoning?   Do the authors equate semantic similarity with analogy?  This point needs to be brought out more strongly in the paper.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n## Here are the answers to your concern\nIn order to address your concern more clearly, we divide your concern into several parts and respond to each of them separately.\n> W(eakness)P(art)1: Within linguistics, language change based on analogy is pervasive, as is language acquisition based on rule formation via analogy. Analogy is also a powerful tool for story-telling, writing and argumentation. However, the authors chose none of these domains for their experiments.\n\nA(nswer)W(eakness)P(art)1: We agree that analogy is useful in applications like story-telling, writing, and argumentation. In this work, we focus on studying the fundamental structure of analogical reasoning beneath these applications, according to the Structure Mapping Theory (SMT) of analogical reasoning in humans. \nFor example, authors often use analogies such as \"love is a journey\" in novels, which could be a guide to the storyline. This analogy **essentially establishes mappings** between the concepts of \"love\" and \"journey\", where \"lover\" maps \"traveler\", \"relationship\" maps \"road\", \"stage\" maps \"milestone\", \"difficulty\" maps \"obstacle\" and \"happiness or separation\" maps \"destination\". \nThe analogical structure abduction task introduced in this paper examines the process behind such analogy-making process. This task can **simulate this mapping process and thus can explain the rationale behind the analogy-making in story-telling, writing, and argumentation**. Moreover, our benchmark includes analogies from various domains such as Philosophy, Art, and Literature, which can be a valuable resource to understand and improve LMs' abilities in writing with analogy-making.\n\n> WP2: Rather, they construct an artificial task of a slightly more complex relationship between words that looks at word similarity in that the relationship involves pairs of two words rather than a single pair of words. It is not clear how this really reflects the human process of analogy.\n\nAWP2: Our task is a simplification of modeling what actually happens beneath analogical reasoning by SMT, i.e., **identifying common relational structures** between two systems. An example (i.e., camera and eye) is illustrated in Figure 1 in our paper. Based on the relational structure, the aperture should be mapped to the pupil since both are channels for light to enter, while black paint maps the choroid since both absorb light to prevent it from bouncing and reflecting. Thus, our task is a **faithful simulation of this process**.\nFurthermore, as shown in Figure 5 in our paper, relying solely on word similarities can not achieve satisfactory performance on our task. Therefore, our tasks need **a comprehensive understanding of systems and relevant concepts and the relational structure among them** rather than only focusing on pairs of two words and looking at word similarity.\n\n> WP3:  It is not clear how it is more generally of interest beyond this rather artificially constructed set of relations between pairs of words. This is also presumably why the attempts at having LLMs work with the open list of \"analogies\" do so badly: the task is artificially defined and does not tap into higher principles/generalizations that the LLMs might have learned.\n\nAWP3: We believe this task (analogical structure abduction) is a foundation for the **development, explanation, and evaluation** of more complex analogies that underlie various creative and cognitive processes. For example, the analogy \"ideas are seeds\" forms the basis of concepts and storylines, when extended to real applications like novel writing. As \"seeds\" need \"nurturing\" to \"grow\", \"ideas\" require \"development\" to \"reach their full potential\". The process of basic word-level analogy serves as the stepping stone to **creating and explaining** more complex analogical reasoning in real-world applications.\nBy simplifying the complex analogy-making process to the fundamental analogy-making with words and concepts, we can evaluate the LLMs' ability for analogical reasoning in a **more focused and effective way**. This **disentangles confounding factors for evaluating analogical reasoning in real-world tasks**, such as narrative composition or rhetorical complexity.\nTherefore, our task forms **a part of a whole framework for assessing and bolstering the ability of LLMs to generalize and apply analogical reasoning** in more intricate, real-world situations.\n\n## Here is the answer to your question\n\n> Question: It is not clear to me how the classic \"queen is to woman as king is man\" is really an instance of analogy since basically only the similarity of two vectors to one another is being compared: how is that really analogical reasoning? Do the authors equate semantic similarity with analogy? This point needs to be brought out more strongly in the paper.\n\nAnswer: In the example \"queen is to woman as king is man\", the analogy is **not merely identifying the semantic similarity** between them. Instead, it recognizes the shared relation \"gender of a ruler\". \nIn cognitive science, analogy can be used to convey that two situations or systems share a relational structure despite differences in the objects that make up the domains. This means that common relations (**structural similarity**) are essential to analogy, not the similarity of objects. \nIn our paper, we argue that the word analogy test (*A is to B as C is to D*) does not explicitly evaluate **structural similarity** between two systems and thus propose an analogical structure abduction task to evaluate analogical reasoning of LLMs which are better aligned with human cognition.\n\n## Here is the answer to the ethical concern\n> Ethical Concern: The annotation was done by human annotators at \"local\" minimum wage. It is not said where those annotators are from or under what conditions they perform the annotation.\n\nAnswer: We recruited a team of five undergraduate students with different academic backgrounds to annotate this benchmark. The annotators are provided with guidelines of SMT to learn about identifying potential analogies based on the relational structures of systems. We pay each annotator $8/h, exceeding the local minimum wage $5/h. Please refer to **Answer C for Reviewer 1Lsr** for detailed annotation information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic of the paper is interesting given that, as the authors point out, analogy is a very basic human skill that is applied across many different domains. It is as such interesting to extend LLMs capabilities to include analogical reasoning.", "reasons_to_reject": "Within linguistics, language change based on analogy is pervasive as is language acquisition based on rule formation via analogy.  Analogy is also a powerful tool for story-telling, writing and argumentation.  Yet the authors chose none of these domains for their experiments.  Rather, they construct an artificial task of a slightly more complex relationship between words that looking at word similarity in that the relationship involves pairs of two words rather than a single pair of words.  It is not clear how this really reflects the human process of analogy or how it is more generally of interest beyond this rather artificially constructed sets of relations between pairs of words.  This is also presumably why the attempts at having LLMs work with the open list of \"analogies\" do so badly: the task is artificially defined and does not tap into higher principles/generalizations that the LLMs might have learned.", "questions_to_authors": "It is not clear to me how the classic \"queen is to woman as king is man\" is really an instance of analogy since basically only the similarity of two vectors to one another  is being compared: how is that really analogical reasoning?   Do the authors equate semantic similarity with analogy?  This point needs to be brought out more strongly in the paper.", "paper_topic_and_main_contributions": "The paper proposes to move beyond word analogies for LLMs and to enable \"analogical structure abduction\".  To this end a new data set is constructed based on the efforts of human annotators paid a local minimum wage, wherever local may be and however much that works out to be.  The results are mixed on a closed list of analogies, quite bad on an open list.  The main contributions are a move towards going beyond word analogies and the creation of a new data set.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Justification_for_Ethical_Concerns": "The annotation was done by human annotators at \"local\" minimum wage.  It is not said where those annotators are from or under what conditions they are performing the annotation. \n\nThis concern was addressed sufficiently by the authors.   They used undergraduates who were trained in this particular task."}}, {"review_id": "hzWaFpE9ty", "reviewer": "Reviewer_SVGF", "summary": "", "questions": "**Question A:** To what extent can contamination of LLM pre-training data impact the observed abilities of LLMs in analogical reasoning? For example, I can Google about the aperture of a camera and pupil of a human eye and find plenty of web content talking about how they’re analogous. Does the LLM gain this capability just by memorizing the pre-training data? \n\nI would be curious to see how thoroughly the system understands common systems or concepts across examples from ScAR - if the system can correctly answer the *aperture -> pupil* example, but can’t understand some other example about *aperture* that’s less common, this may suggest a lack of deep understanding of the concept of *aperture.* Figure 4 gives some hints about this possibility across domains, but doing such an analysis could also be a valuable addition to the work! *As a side note, this is one reason why Mitchell (2021) listed later in my review argues that we shouldn’t focus on natural language analogy problems, but I think this issue can also be explored by more thorough evaluations.*", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful suggestions, we will make the following revisions to our paper: \n1. Polish the paper to be more clear.\n    - Add the papers you mentioned in our paper.\n    - Break the paragraph in L091-L100 out into a numbered list.\n    - Put the formal problem definition into Section 3.1 before introducing our dataset.\n    - Use *\\citealp* in LaTeX to combine citations with other content in parentheses.\n    - Correct the title of Table 10: *Prompt templates for background revision (generated by ChatGPT) and explanation generation (generated by GPT-4). Generated texts by ChatGPT and GPT-4 are highlighted.*\n\n2. Reorganize Section 5 (Related Work) based on different parts of analogy-making, i.e., representation, mapping, and evaluation.\n\n3. Add a separate paragraph about abstract analogical reasoning in Section 5, and refine the statement for better clarity on Webb's work: *\"...Webb et al. (2022) examine the abstract language-based analogy task and evaluate analogical reasoning ability of LLMs on this task. Compared to their task, our task requires the intensive involvement of commonsense, encyclopedic, and cultural (e.g., idiom and historical) knowledge. \"*\n\n## Here are the answers to your concern\n> W(eakness)1: This paper doesn’t propose any new modeling techniques or methods to apply LLMs to analogy tasks. This causes the main benchmark results on the paper to not be very inspiring, as they don’t show much significant variation across existing models and prompting techniques that are explored.\n\nA(nswer)W(eakness)1: Thank you for your comments. Our work is not positioned as a method paper, but rather studies the problem of analogical reasoning for LLMs. However, to enrich the main benchmark results, we added **Llama 2 (13B and 70B)** as a baseline open-sourced LLM [1] and **self-consistency prompting** as a baseline for advanced prompting [2] on the English version of SCAR. For self-consistency prompting, we adopt two methods: \n\n(1) We set the temperature to 0.7, generate results **five times**, and take the majority vote to select the answer, which is denoted as w/ *Self-consistency (Temperature)*; \n\n(2) We use **five different instruction** templates and take the majority vote to select the answer, denoted as w/ *Self-consistency (Template)*.\n\nThe results in the following table show that: \n1. Llama 2 70B performs poorly, but self-consistency prompting can significantly improve performance. \n2. Additionally, we observe performance improvements in both ChatGPT and GPT-4 through advanced prompting techniques like self-consistency prompting.\n\n[1] Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. arXiv preprint arXiv:2307.09288, 2023.\n\n[2] Wang X, Wei J, Schuurmans D, et al. Self-consistency improves chain of thought reasoning in language models[J]. arXiv preprint arXiv:2203.11171, 2022.\n\n| Method  | Concept Acc.| System Acc. |\n|---------|:----:|:----:|\n| Llama 2 (13B)     |  36.78   |  19.50   |\n| w/ 1-shot     |  49.54   |  24.50   |\n| Llama 2 (70B)     |  52.38   |  32.50   |\n| w/ 1-shot     |   62.85  |  44.50   |\n| w/ Self-consistency (Temperature)  |  67.93   |  48.85   |\n| w/ Self-consistency (Template) |  58.14   |  46.50  |\n| ChatGPT     |  66.52   |  46.61   |\n| w/ Self-consistency (Temperature)  |  68.24   |  48.75  |\n| w/ Self-consistency (Template) |  68.68   |  53.50  |\n| GPT-4    |   73.28  |  58.50   |\n| w/ Self-consistency (Temperature)  |  77.96   |   67.00  |\n| w/ Self-consistency (Template) |  75.94   |  65.50  |\n\n\n\n\n> W2: The paper does not rigorously explore the impact of pre-training data on LLMs’ analogical reasoning.\n\nAW2: Please refer to Answer A for Question A\n\n## Here is the answer to your question\n> Question A: To what extent can contamination of LLM pre-training data impact the observed abilities of LLMs in analogical reasoning?\n\nAnswer A: Thank you for this suggestion. We randomly sample 10% (about 35M samples) of the C4 dataset, which is from Common Crawl's web corpus, a commonly used source for pre-training LLMs, to observe the **contextual (co-occurrence) associations** between the concepts of two systems in the pre-training data. \nWe aggregate the number of articles in which two systems' concepts co-occurred. In the C4 dataset, each sample corresponds to an article. If the concepts of two systems are in a single article, it might indicate a high co-occurrence between the concepts. The results are listed below. \nSystem A -> System B | System A in article| System B in article| # Co-occurred / # Total |\n|---------|:----:|:----:|:----:|\nCamera -> Eye | ~1M | ~2M | ~17K/35M   |\nCell -> Factory | ~4K | ~238K |  27/35M   |\nComputer Systems -> Urban | ~483K | ~1M |  541/35M   |\nChemistry -> Cooking | ~82K | ~853K |  93/35M   |\n\nWe find that the concepts of two systems from different domains rarely co-occurred. Yet, LLMs can still establish the mappings between concepts to form analogies and recover reasonable explanations of relational structures. The low co-occurrence between concepts suggests that LLMs may **not rely on the pre-training data's co-occurrence associations** for analogical reasoning. Therefore, we think that the observed abilities are not severely contaminated by the pre-training data. Instead, pre-training teaches LLMs **a wealth of background knowledge**, enabling them to identify and infer relational structures behind analogical reasoning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Strength A:** The paper compiles a (seemingly) high-quality benchmark dataset of interesting analogies across scientific and other domains, including consistent details about the underlying structure of analogies, and background knowledge and explanations for analogies. This resource could be useful to others in the community.\n\n**Strength B:** The creation of the benchmark is justified by a preliminary study on how word analogy tasks do not actually represent analogical reasoning capabilities. I have not seen another work demonstrate this before, and this can be motivation for the community to look at analogical reasoning more holistically than prior work has.\n\n**Strength C:** The analysis of benchmark results is thorough and fairly comprehensive, covering a lot of variations and details on the prompt engineering/LLM interface side, as well as including a thoughtful study on how analogies can be made across an open domain of concepts (which I believe is the hardest challenge in analogical reasoning). The authors also explore earlier representation/embedding-based approaches to analogy. It feels like the authors have considered a lot of questions in performing these experiments.", "reasons_to_reject": "**Weakness A:** This paper doesn’t propose any new modeling techniques or methods to apply LLMs to analogy tasks. This causes the main benchmark results on the paper to not be very inspiring, as they don’t show much significant variation across existing models and prompting techniques that are explored.\n\n**Weakness B:** The paper does not rigorously explore the impact of pre-training data on LLMs’ analogical reasoning. In natural domains like this that may be largely supported by memorization of training data, some analysis for this is important.", "questions_to_authors": "**Question A:** To what extent can contamination of LLM pre-training data impact the observed abilities of LLMs in analogical reasoning? For example, I can Google about the aperture of a camera and pupil of a human eye and find plenty of web content talking about how they’re analogous. Does the LLM gain this capability just by memorizing the pre-training data? \n\nI would be curious to see how thoroughly the system understands common systems or concepts across examples from ScAR - if the system can correctly answer the *aperture -> pupil* example, but can’t understand some other example about *aperture* that’s less common, this may suggest a lack of deep understanding of the concept of *aperture.* Figure 4 gives some hints about this possibility across domains, but doing such an analysis could also be a valuable addition to the work! *As a side note, this is one reason why Mitchell (2021) listed later in my review argues that we shouldn’t focus on natural language analogy problems, but I think this issue can also be explored by more thorough evaluations.*", "paper_topic_and_main_contributions": "This paper is about analogical reasoning in language models supported by structure abduction. Its contributions are as follows:\n\n1. The authors present a study on past analogical reasoning benchmarks to show that while LLMs can achieve high performance on simpler word analogy benchmarks, this is not always supported by an understanding of the underlying structure for the analogy (unlike humans). This motivates them to create a stronger evaluation for analogical reasoning that considers structure abduction jointly with making analogies.\n\n2. The authors create a benchmark called Scientific Analogical Reasoning (ScAR) which provides 400 analogies across systems in 13 different scientific and other domains, including 1600 possible mappings between concepts within systems. They additionally collect background knowledge and explanations to support the analogies in the dataset, making it a possibly valuable and high-quality resource for studying analogy-making in LLMs.\n\n3. The authors evaluate contemporary LLMs, both open-source and closed API-based, on ScAR under several prompting settings. Further, they perform analysis on the impact of various LLM-based aspects in the experiment and the domain. Lastly, they apply earlier word embedding-based methods for analogy as a reference, and perform a preliminary study on open analogical structure abduction, where concepts from the two analogous systems are not provided directly, and rather must be retrieved from a large list.\n\nWhile the paper does not present new methods for tackling analogical reasoning with LLMs and perhaps lacks some consideration of the impact of LLM pre-training data on this problem, it still constitutes a substantial contribution that could be valuable to the NLP community.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "ACL 2023 has at least one paper on LLMs and analogical reasoning, which looks at abstract analogical reasoning: https://aclanthology.org/2023.acl-long.109/ \n\nThis paper by Melanie Mitchell would be important to cite: https://arxiv.org/abs/2102.10717", "Typos_Grammar_Style_and_Presentation_Improvements": "**Organization/presentation suggestions:**\n\nL091: If accepted, you may want to use the extra space to break this paragraph out into a numbered list - may make it easier to read. There are some other paragraphs where you do this too, and I would recommend the same.\n\nSection 3.4 presents a formal problem definition, which might make more sense to introduce before your dataset. This would help give the reader a foundation on what task you’re looking at, and then the data collection would be easier to understand and map each step to components of the task. I’m not totally sure - would leave it up to you.\n\nI’m not sure if it’s fair to characterize Webb et al. (2022) as just looking at analogy generation with word analogies or simple sentence analogies as your Section 5 says around L529. This paper looks at abstract language-based analogy tasks and uses them to study how well LLMs can perform analogical reasoning. I might suggest adding a separate paragraph in Section 5 looking at abstract analogical reasoning rather than lumping it into the paragraph about analogies between situations. You may also think about organizing Section 5 based on how these different works focus on different parts of analogy-making, i.e., representation, mapping, and evaluation, and how your work contrasts from it.\n\n**Formatting/typo corrections:**\n\nL036, L046, L055, L060, L090, L110, L135, L344-348…: putting two sets of parentheses next to each other or nesting parentheses is improper format and a bit confusing to read. You can use \\citealp in LaTeX to combine citations with other content in parentheses, or find another way to remove one set of parentheses.\n\nL124: missing a space after period\n\nDid you use ChatGPT or GPT-4 to revise background information about systems? The paper says ChatGPT, but Table 10 says GPT-4."}}, {"review_id": "s1wsuqNeS1", "reviewer": "Reviewer_1Lsr", "summary": "", "questions": "A. Can you elaborate on the reasonableness of using the E-KAR dataset heavily in this work? The main concern is that it is a dataset taken from China’s Civil Service Examinations. The description provided in Appendix A suggests that “cultural knowledge” is part of the exam. Is it reasonable to expect that a model trained largely on English data and evaluated on English data will be able to answer these questions? I have in mind work like Santurkar et al. (2023). Whose Opinions Do Language Models Reflect. https://arxiv.org/abs/2303.17548., which demonstrate that models encode only some groups well. \n\nB. Why was GPT-4 used so extensively in the creation of the dataset (e.g., for the explanation generation)? Wouldn’t higher quality data come from having the annotators do all of this (they are already reading through them after all)? \n\nC. In the Ethics Statement it is stated that “as described in our paper, all annotators are compensated”, where is this stated in the body of the paper? It would be helpful if more details about the annotators (e.g., what are the 5 annotators backgrounds on lines 229-231) and their task were included (e.g., the instructions they were provided). This would facilitate both replication and extension of this work. \n\nD. Technique for embeddings was unclear in section 2. It is stated that the method proposed by Ushio et al. (2021) was used (line 139). However, that paper used cosine similarity in evaluating embeddings, as far as I can tell. Moreover, it is stated that the winning answer candidate was determined with the marginal likelihood biased perplexity measure which uses probability of the token in context, not the embedding. Could this method be clarified in the paper?  \n\nMINOR:\n\nE. In section 4, a few more models were evaluated. Why weren’t these models included in the other experiments? \n\nF. The role of RSI in the paper is a bit unclear. Is it part of the new proposed benchmark as well? Or is it just used as an additional source of evidence that models fail to capture the underlying structural relations in an analogy (similar to how the ability of models to fully associate the set of concepts in the benchmark evidences they systematically understand the relations)?  \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful suggestions, we will make the following revisions to our paper: \n\n1.**Transform Figure 3d into a bar chart accompanied by error bars** to better prove that providing an example, additional backgrounds, or utilizing CoT prompting can enhance the robustness of LLMs to the instruction design. Since the error bar is a specific confidence interval for the outcomes, here we report the **95% confidence interval** of the average system accuracy of LLMs with different methods for your reference:\n| Model  | 95% confidence interval| confidence interval width |\n|---------|:----:|:----:|\n| 0-shot     |[38.68, 43.54] |4.86|\n| 1-shot     |[43.13, 46.86] |3.73|\n| w/ Expl.   |[44.16, 47.08] |2.92|\n| w/ Backg   |[43.77, 46.73] |2.95|\n\n2.Consistent y-axis bounds in Figures 3, 5, and 6 to facilitate comparisons.\n\n3.Add detailed information about **annotations** to the Appendix to facilitate both replication and extension of this work (The information is shown in **Answer C**).\n\n## Here is the answer to your concern\n\n> Weakness: The linkage to human cognition (which appears to be a central aim of the paper) is a bit unclear. There are two uses of SMT in the paper: (i) as the account for analogical reasoning broadly construed, and (ii) as the account for system-level analogical reasoning. If (ii) is the correct interpretation, then it’s possible models can do simple word analogies in a human-like fashion without the ability to do system-level analogies. \n\n\nAnswer: Thank you for pointing out this confusion! We would like to clarify that both stated uses of SMT in the paper, (i) as the mechanism for broad analogical reasoning and (ii) as the mechanism for system-level analogical reasoning, are not contradictory. Instead, they demonstrate the versatility of SMT. The essence of SMT lies in identifying common relational structures among two \"systems,\" where each system can be a central concept or situation with relevant concepts. Therefore, **system-level analogies can be generalized to broader ones**. \n\nAccording to SMT, analogical reasoning is based on identifying common relational structures between two systems. However, our RSI task shows that the word analogy test **can not** evaluate the ability of LMs to identify common relational structures between two systems. Therefore, even if a model can do simple word analogies in a human-like fashion and even achieves human-level performance on the word analogy test, it is **not guaranteed** to have human-level analogical reasoning ability.\n\n\n## Here are the answers to your questions\n> Q(uestion)A: Can you elaborate on the reasonableness of using the E-KAR dataset heavily in this work? Is it reasonable to expect that a model trained largely on English data and evaluated on English data will be able to answer these questions?\n\nAnswer A: As shown in Table 6, compared to other existing analogy benchmarks, E-KAR contains more data samples with **more diverse and challenging relations**, and it is the only benchmark that **provides relational explanations** for each pair of analogy. Additionally, as stated in the E-KAR's paper, E-KAR is **bilingual**, and the authors have **eliminated culture-specific samples** in the English split of E-KAR. Therefore, we primarily adopt E-KAR to construct the relational structure identification task dataset.\n\n> QB: Why was GPT-4 used so extensively in creating the dataset (e.g., for the explanation generation)? Wouldn’t higher-quality data come from having the annotators do all of this?\n\nAnswer B: Providing human-written explanations could be costly, time-consuming and labor-intensive. Thus, we use GPT-4 to generate explanations and ask the annotators who created the dataset to revise the wrong explanations to **reduce human labor**, which has become a prevailing paradigm for data annotation nowadays (e.g., WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation [Liu et al., EMNLP Findings 2022] and Reframing Human-AI Collaboration for Generating Free-Text Explanations [Wiegreffe et al., NAACL 2022]).\n\n> QC: In the Ethics Statement, it is stated that “as described in our paper, all annotators are compensated”, where is this stated in the body of the paper? It would be helpful if more details about the annotators (e.g., what are the 5 annotators' backgrounds on lines 229-231) and their tasks were included (e.g., the instructions they were provided). \n\nAnswer C: Thank you for pointing out this confusion about annotation! Here's the detailed information about annotations for your reference, and we will add it in the revision:\n\n**Annotator Details**: \n\nWe have recruited a team of five undergraduates with diverse academic backgrounds in Computer Science, History, Physics, Biology, and Chemistry. Among them, the student majoring in Computer Science has a minor in Economics, while the student majoring in History has minors in Creative Writing and Philosophy. We pay each annotator \\\\$8/h, exceeding the local minimum wage \\\\$5/h.\n\n**Guidelines For Annotators**:\n\nInstructions\n\nThanks for participating in this HIT! Please spend some time reading this instruction and the example section to better understand our HIT! In this hit, you need first read the context of Structure Mapping Theory to learn how to identify potential analogies based on the relational structures of systems and then create analogies based on this theory as many as possible. **[Here, we introduced the structure mapping theory as in Section 2.1 of the paper. We omit it due to the length]** After reading the above context, we believe you have understood the SMT. Next, you need to create system analogies based on the relational structures of two systems. One data example is shown in Figure 1. You will complete 4 steps to create one system analogies. \n- Step 1: You need to write two systems that can form an analogy (e.g., Camera and Eye).\n- Step 2: You need to write relevant concepts of two systems as many as possible (e.g., Film, Diaphragm, Aperture, Lens, Black paint for Camera and Retina, Iris, Pupil, Lens, Choroid for Eye).\n- Step 3: You need to construct the mappings between concepts in two systems based on the relational structure abduction (e.g., aperture should be mapped to pupil).\n- Step 4: You need to remove the concepts that are not mapped.\n\nTo facilitate a more efficient and effective annotation process, we provide you with scientific analogies sourced through online research **[We omit the URLs of these sources in Appendix B.1 due to the length]**. Notes: You can SEARCH GOOGLE to help you judge whether the mappings are reasonable, especially if you are unsure about the system.\n\n> QD: Technique for embeddings was unclear in section 2. It is stated that the method proposed by Ushio et al. (2021) was used (line 139). However, that paper used cosine similarity in evaluating embeddings, as far as I can tell. Moreover, it is stated that the winning answer candidate was determined with the marginal likelihood biased perplexity measure, which uses the probability of the token in context, not the embedding. Could this method be clarified in the paper? \n\nAnswer D: We apologize for the confusion caused by this typo. We did use **cosine similarity in evaluating embeddings**. For the word embedding models, we follow the method proposed by Ushio et al. (2021) and represent word pairs by taking the difference between their InstructGPT embeddings (text-ada-embedding-002). Then, we choose the answer candidate with the highest cosine similarity to the query in terms of this vector difference.\n\n> QE: In section 4, a few more models were evaluated. Why weren’t these models included in the other experiments?\n\nAnswer E: Thanks for your suggestion, and we will report the performance of all models in each experiment in the Appendix for reference in our next version. Due to limited space, we selected the **top three performing models** in the main results (Table 5) for further investigation (i.e., InstructGPT 003, ChatGPT and GPT-4). \n\n> QF: The role of RSI in the paper is a bit unclear. Is it part of the new proposed benchmark as well? Or is it just used as an additional source of evidence that models fail to capture the underlying structural relations in an analogy?\n\nAnswer F: RSI **is not a part of** the new proposed benchmark. RSI is a task designed to have LLMs select the relation constituting the analogy, which is used to prove that the word analogy test **can not** evaluate the ability of LMs on identifying common relational structures between two systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well motivated and its aim clear. The benchmark will be of use to community and appears to address a gap in existing resources. The experiments are reasonably thorough and expose limitations with existing models, while suggesting ways they might be improved. ", "reasons_to_reject": "The problem identified by the paper is interesting and well articulated. There are a few methodological issues that weaken the rating (included under questions for the authors). Additionally, the linkage to human cognition (which appears to be a central aim of the paper) is a bit unclear. It is stated that the Structure Mapping Theory (SMT) has been proposed as a mechanism for analogical reasoning. Other times, it is stated that SMT suggests that new knowledge is gained by system level analogies. Put another way, there are two uses of SMT in the paper, (i) as the account for analogical reasoning broadly construed, and (ii) as the account for system level analogical reasoning (a more narrow form of analogy). If (ii) is the correct interpretation, then it’s possible models can do simple word analogies in a human-like fashion without the ability to do system level analogies. The lack of clarity around these possibilities makes it difficult to interpret what these results say about comparisons between models and humans. \n\nUltimately, the paper proposes a potentially useful benchmark for evaluating the underlying structural reasoning abilities of large-language models. However, some issues with the methodology and the underlying arguments motivating the conclusion depress my rating. \n", "questions_to_authors": "A. Can you elaborate on the reasonableness of using the E-KAR dataset heavily in this work? The main concern is that it is a dataset taken from China’s Civil Service Examinations. The description provided in Appendix A suggests that “cultural knowledge” is part of the exam. Is it reasonable to expect that a model trained largely on English data and evaluated on English data will be able to answer these questions? I have in mind work like Santurkar et al. (2023). Whose Opinions Do Language Models Reflect. https://arxiv.org/abs/2303.17548., which demonstrate that models encode only some groups well. \n\nB. Why was GPT-4 used so extensively in the creation of the dataset (e.g., for the explanation generation)? Wouldn’t higher quality data come from having the annotators do all of this (they are already reading through them after all)? \n\nC. In the Ethics Statement it is stated that “as described in our paper, all annotators are compensated”, where is this stated in the body of the paper? It would be helpful if more details about the annotators (e.g., what are the 5 annotators backgrounds on lines 229-231) and their task were included (e.g., the instructions they were provided). This would facilitate both replication and extension of this work. \n\nD. Technique for embeddings was unclear in section 2. It is stated that the method proposed by Ushio et al. (2021) was used (line 139). However, that paper used cosine similarity in evaluating embeddings, as far as I can tell. Moreover, it is stated that the winning answer candidate was determined with the marginal likelihood biased perplexity measure which uses probability of the token in context, not the embedding. Could this method be clarified in the paper?  \n\nMINOR:\n\nE. In section 4, a few more models were evaluated. Why weren’t these models included in the other experiments? \n\nF. The role of RSI in the paper is a bit unclear. Is it part of the new proposed benchmark as well? Or is it just used as an additional source of evidence that models fail to capture the underlying structural relations in an analogy (similar to how the ability of models to fully associate the set of concepts in the benchmark evidences they systematically understand the relations)?  \n", "paper_topic_and_main_contributions": "This paper is centrally interested in structured analogical reasoning in large language models. Existing work has focused on word analogies. This work suggests that results from that paradigm overestimate the overlap between neural models and human cognition. They introduce a task of analogical structure abduction that evaluates whether models can systemically associate the components of an analogy. The results suggest that models often miss the underlying relations, and thus, differ from humans. The main contributions are: i) a benchmark called ScAR and ii) a evaluation of LLMs on this task highlighting the need for future work to improve models. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 3d is used to argue that models are more robust in the CoT prompting scenario. The inclusion of error bars would facilitate more informed comparisons between the variation in a given condition and across conditions. \n\nConsistent y-axis bounds in Figures 3, 5, and 6 would facilitate comparisons. \n"}}]

Author rebuttal:

Meta-review: The paper explores analogical reasoning in large language models (LLMs), moving beyond word analogies to "analogical structure abduction". It introduces a benchmark, Scientific Analogical Reasoning (ScAR), with 400 analogies across 13 domains and 1600 concept mappings, supplemented with background knowledge. The study reveals LLMs, while adept at word analogies, often miss deeper structural connections, unlike humans. Using ScAR, LLMs show varied results, particularly when structures aren't directly provided. The paper's key contributions are the ScAR benchmark and highlighting the LLM-human cognition gap in analogy. It paves the way for future research in this area.

The paper's blend of a relevant topic, novel insights, a high-quality dataset, and comprehensive analysis makes it a worthy contribution to the field. In particular: 
a) The authors have meticulously compiled a benchmark dataset that encompasses a diverse range of analogies spanning scientific and other domains. This dataset is enriched with consistent details about the structural essence of analogies and is supplemented with background knowledge and explanations. Such a comprehensive resource stands to benefit the broader community.
b) A remarkable feature of the paper is its preliminary study highlighting the inadequacies of word analogy tasks in truly representing analogical reasoning capabilities. This perspective, not widely explored in previous works, offers a fresh lens to view analogical reasoning, urging the community to adopt a more holistic approach.

However, the main weaknesses of the contribution are: paper's lack of novel techniques (despite the aforementioned novel insights), insufficient exploration of pre-training data, ambiguity in its linkage to human cognition, and methodological shortcomings.
Decision: Accept-Findings