MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models
Authors: Jiachun Li
Pengfei Cao
Zhuoran Jin
Yubo Chen
Kang Liu
Jun Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: inductive reasoning
large language model
model explanation
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: Our work introduces Mirage, a synthetic dataset that assesses LLMs' inductive reasoning, showing they often fail to apply rules but excel at using neighbor examples in feature space to improve performance

Abstract: Inductive reasoning is an essential capability for large language models (LLMs) to achieve higher intelligence, which requires the model to generalize rules from observed facts and then apply them to unseen examples. We present {\scshape Mirage}, a synthetic dataset that addresses the limitations of previous work, specifically the lack of comprehensive evaluation and flexible test data. In it, we evaluate LLMs' capabilities in both the inductive and deductive stages, allowing for flexible variation in input distribution, task scenario, and task difficulty to analyze the factors influencing LLMs' inductive reasoning. Based on these multi-faceted evaluations, we demonstrate that the LLM is a poor rule-based reasoner. In many cases, when conducting inductive reasoning, they do not rely on a correct rule to answer the unseen case. From the perspectives of different prompting methods, observation numbers, and task forms, models tend to consistently conduct correct deduction without correct inductive rules. Besides, we find that LLMs are good neighbor-based reasoners. In the inductive reasoning process, the model tends to focus on observed facts that are close to the current test example in feature space. By leveraging these similar examples, the model maintains strong inductive capabilities within a localized region, significantly improving its deductive performance.

Published as a conference paper at ICLR 2025
MIRAGE: EVALUATING AND EXPLAINING INDUCTIVE REASONING PROCESS IN LANGUAGE MODELS
Jiachun Li Pengfei Cao Zhuoran Jin Yubo Chen Kang Liu Jun Zhao!
"School of Artificial Intelligence, University of Chinese Academy of Sciences The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences (jiachun.1i, pengfei.cao, zhuoran. jin, yubo yubo. chwooccnnne@n........ ia. iatpnnlprraaa........
ABSTRACT
Inductive reasoning is an essential capability for large language models (LLMs) achieve higher intelligence, which requires the model to generalize rules from observed facts and then apply them to unseen examples. We present MIRAGE, a synthetic dataset that addresses the limitations of previous work, specifically the lack of comprehensive evaluation and flexible test data. In it, we evaluate LLMs' capabilities in both the inductive and deductive stages, allowing for flexible vari- ation in input distribution, task scenario, and task difficulty to analyze the factors influencing L.MS' inductive reasoning. Based on these multi-faceted evaluations, we demonstrate that the LLM is poor rule-based reasoner. In many cases, when conducting inductive reasoning, they do not rely on a correct rule to answer the unseen case. From the perspectives of different prompting methods, observation numbers, and task forms, models tend to conduct correct deduction without cor- rect inductive rules consistently. Besides, find that LLMs are good neighbor- based reasoners. In the inductive reasoning process, the model tends to focus on observed facts that are close to the current test example in feature space. By lever- aging these similar examples, the model maintains strong inductive capabilities within localized region, significantly improving its reasoning performance.
INTRODUCTION
Inductive reasoning, known as the ability of an intelligent agent to infer abstract rules from limited observations and apply them to new examples, is crucial for large language model (LLMs) (Li al.
2023 Wen et al., 2023; Sun et al. 2024a) progressing toward artificial general intelligence (AGI) (Xu et al. 2024b Sun al., 2024b Wang al., 2024b). As illustrated in Figure [ given a set of observed facts, inductive reasoning process expect the model to generate abstract rules from the provided facts (i.e. [A,B,C] - [B+C,B+C,C] in the rule induction task) and apply these rules to answer specific new questions (i.e. [3,4.7] + [11,11,7] in the example inference task). Despite its significant research value, it has been relatively neglected compared to other types of reasoning (e.g., math reasoning, multi-hop reasoning, etc.).
Recently, some works have started to explore this problem. They primarily evaluate the model's inductive reasoning capabilities using various datasets (Shao et al.) 2024 Cheng et al. 2024 Quu
et al., 2024; Ding et al.) (024).. Though they have made great progress, their works still have two main limitations: (1) Previous works lack comprehensive evaluation. Most works have only one
evaluation task: the inductive task on collected rules (Yang et al. 2024b Shao et al. 2024 Wang al. 2025) or the deductive task on specific test samples (Chollet 2019 Xu et al. [024aa] Qiu al.
2024 Therefore, they can only evaluate the rule induction performance or final results of inductive
reasoning, instead of comprehensively analyzing the whole process (i.e. inductive deductive). (2)
Previous works lack flexible test data. Most former datasets evaluate the overall performance of
models by collecting observation and test examples under the same rules (Rule) 20200] Kim et al.)
2022 Lake et al. 2019). However, due to the absence of transformation rules, it is impossible to extend these examples, resulting in a fixed test set. This limitation makes it challenging to assess the
*Corresonndigg authors.

---

Published as a conference paper at ICLR 2025
bserved Facts
[0,1.]] - [443]
[5,2,5] - [7.7,5]
[2,1,8] - [9.98] [9,6,]] - 19,9,3]
[3,3,7] - [10,10,7]
Rule Induction Task
What tule?
What the mole?
[A,BC]] [B-C
[A - (A-4B-1
Example Inference Task
13477+
[3477++
Rule (A [НC
Caee: [3377] [1010
11,11.7
Aneaenn [11,117]]
Rule-based
Neighbor-based
Figure 1: An overview of two paradigms (i.e. rule-based and neighbor-based) in inductive reasoning.
impact of factors such as distribution, quantity, and form of input examples on the model's 's inductive reasoning, thereby hindering a deeper analysis of the model's reasoning mechanisms.
In this paper, we present MIRAGE (Meta Inductive ReAsoninG Evaluation), a dataset designed to address the two aforementioned limitations. includes both inductive and deductive evaluation tasks, while offering flexibility construct test data with various forms, arbitrary input distributions, and controllable difficulties. In detail, we first construct a rule library based on various vector oper- ations (c.g.. [A.B,C] [B+C,B+C,C] rules, we can generate facts arbitrarily through instantiation, ensuring the flexibility and scalability of the test data. Next, filter out the noise data (e.g. duplicated facts) to further improve the effectiveness and quality of our dataset. Finally, to comprehensively evaluate the inductive reason- ing process, not only design inductive and deductive questions based on the synthesized data but also construct diverse application scenarios for these tasks, including list transformations, real-world problems, code generations, and string transformations (as shown in Figure 2
shown in Figure DT Using the automatically synthesized
Based on our dataset, we perform deeper analysis of the model's inductive reasoning process, from which we draw two new conclusions about the inductive reasoning mechanisms of L.MMs: (1) Language models are poor rule-based reasoners. As shown in the left column of Figure [ in the rule-based reasoning paradigm, inductive reasoning involves first deriving the correct rule through the observation of examples and then using the inductive rule to answer new questions (like what humans do). However, we find that LLMs perform poorly this paradigm: In many cases, though they can not induce correct rule, they can still perform well on example inference tasks. Through experimentation, we observe this performance gap between induction and deduction across different prompting methods, models, observed example numbers, and scenarios. This indicates that the final performance of the L.M''s inductive reasoning rarely relies on the intermediate inductive rules. (2) Language models are good neighbor-based reasoners. Furthermore, we identify an important mechanism behind LLM's inductive reasoning, which we refer to as "neighbor- based reasoning": If some observed facts are close the test examples in feature space, the model tends leverage this similarity to improve inductive reasoning performance. For example, as shown in the right column of Figure [ even when the model cannot generate the correct rule, it can rely on the neighbor fact [3.3,7] + 10,, (here the distance between [3.3.7] and [3,4.7] is small, so we refer to them as neighbors) to successfully performs the reasoning. We demonstrate that this paradigm persists across different scenarios, models, and observed example numbers. However, it can only enhance the performance within a localized scope.
To sum up, the main contributions of our work are as follows: (1) We present a new dataset MI- RAGE, through it, we can comprehensively evaluate the LLM's inductive reasoning process under more flexible settings. (2) We find that LLM is a poor rule-based inductive reasoner. In many cases, it does not rely on inductive rules to make correct deductions. (3) We demonstrate that LLM is neighhor-based inductive reasoner. When performing inductive reasoning, models rely
2

---

Published as a conference paper at ICLR 2025
the neighbor facts in the observed fact set to get better performance. Our code is available at:
2 DATA CONSTRUCTION
In this section, we describe the whole pipeline to build MIRAGE. We start by constructing rules based five basic operations (;2.T). Next, substitute the instantiate vectors into the rules to generate facts ( and apply filtering to them ($2.)) Finally, we transform the facts into different scenarios, creating questions evaluate the LLM's inductive reasoning performance (92...
2.1 RULE GENERATION
According to previous work and relevant definitions (Huber 2017 Han et al. 2024). in inductive reasoning, for each observed fact X (x, y), the input vector a is transtormed into the output vector y according to certain rule i.e.:
f(x) y
v(o, E
(1)
where X is the observed fact under the rule f. Here f is the core of the problem, as it allows generate facts X based automatically. Conversely, inferring f from X requires significantly more effort due to the vast range of possible rules. Therefore, we first consider automating these rules' large-scale synthesis.
Based on previous representative datasets (Chollet 2019 Rule 2020 Xu et al. 2024a we summarize the main types of rules, resulting in five atomic operations this dataset: (1) Add:
The operation adds certain components together. For example: [z, v. [2,, 10------- M. 2.. (2) Copy: The operation copies some components others. For example: 22, y.
22,222] (3) Map: The operation applies linear transformation some components. For example: [ar y, =] + ky.ky + b, 2]. To avoid the interference of complex math calculations, we have & E [1,9] and h E [0, 9). (4) Pad: The operation fills certain components with constant values. For example: [ar, 2] + [z, cl.. where
[0. 9J. (5) Swap: The operation swaps certain components. For
example: [z. 2] [2, v.).
For each operation O, we randomly initialize the set index vector on which the operation applies and the index vector r where the result is output. Specifically, for i E a. E y:
[((z))) Zj.
if E if gr
y/
(2)
where r, = j and [-], represents the i-th component. Therefore, we can generate meta-rule f == (O,d,r) Through sampling (O, d, r) randomly, we can construct a meta-rule library I..
2.2 FACT GENERATION
After generating the rule library, we can randomly initialize æ, and apply a specific rule f E F' to get y. We repeat this process to generate the fact set X under the rule f. All the (æ. E X are used for the L.M to induce the rule f. is worth noting that we can control the inductive difficulty by adjusting two factors: the and
dimension of x, y and the fact number of X. As an example, in Figure [ a higher D and a smaller N tend to increase the task difficulty. Additionally, to avoid the interference of complex mathematical calculations in evaluating inductive reasoning ability, we restrict the elements in each integers between and 9 Since we synthesize any D-dimensional vector construct a fact, we flexibly control the input distribution.
5. Empirically,
2.3
DATA FILTERING
To ensure the quality dataset and the effectiveness of the evaluation, we need to filter out some noisy data. The following filtering steps are applied: (1) Filtering out duplicate facts. For any two facts in X, their input vectors are identical, one of them is removed and resampled. This ensures that for each rulc, all observed facts are unique. (2) Filtering out duplicate rules. To ensure diversity in the evaluation, we also remove duplicate rules, which have the same (O, d, r). (3) Filtering may include some trivial facts that provide little value for model induction, such as facts like æ = y. x O, or = O. We filter the data to ensure that each X contains at most one trivial fact, thereby limiting the noise that could affect the model's inductive reasoning process.
trivial facts. After random sampling,
'Our pilot experiments indicate that, under these constraints, most of the models can achieve an accuracy of nearly 100% in performing purely mathematical operations. See Appendix A,2 for details.
3

---

Published as a conference paper at ICLR 2025
List Transformation
Real-worid Problem
Code Generation
String Transformation
Rale:
Rule:
Bale
Meta Rule
Jock wen mnde neme, artitaaly bazs chars.
det fff.ABCC
[A.8 [ABBCC) B+C, CJ
[A.0.]] (0-C,BKCCC)
K.B.C BHC BHG,, C N.m.A.B..C
SCRCC
Past:
Fact:
Fact
Meta Fact [3366]] 19, 6]
Juck wast trade Hane, He crittezlly bad 3 bods. 3 bogs. 6 cheirs. ffte the tride. The has P bads, 9 bagg, & dichais.
[3.2.0] =(3.00))
0338 - 9,06
ddg >> da00a
Figure 2: Examples in four different scenarios of MIRAGE.
2.4 QUESTION GENERATION
So far, we have constructed all the metadata that we need to generate specific questions. is worth noting that both IF and X contain only abstract rules and facts, without any specific context. Therefore, they represent the fundamental inductive reasoning test data, which is why we refer to them as meta-rules and meta-facts. As shown in Figure to evaluate the practical inductive reasoning capability of models, we apply these metadata to various scenarios to generate concrete problems. Specifically, we have: (1) List Transformation (LT); List transformation is the primary format used in previous inductive reasoning tasks (Rule 2020 Xu et al. 2024a Chollet 2019), and here we adopt this approach well. We transform all fact vectors into one-dimensional
lists and require the model to inductively infer the transformation rules applied to these lists. (2) Real-world
Problem (RP): Previous datasets lack tests for inductive reasoning capabilities in real-world scenarios (Rule) 2020 Xu et al. [2024a] Qiu et al. 2024 To mitigate this gap, we populate the metadata into different natural language templates across five real-life scenarios. The example in Figure 2 describes a trading scenario, where we use different items represent different dimensions of the vector. All item transactions follow the same rule. (3) Code Generation (CG): For each fact, we use æ as the input and y as the output of a function. The model is then tasked with predicting the corresponding Python function. (4) String Transformation (ST); The former three scenarios are related to numbers. Here. we replace the basic elements in the fact vectors with characters to conduct new test. Notably, we modify the operations as follows: addition in the Add and Map operations is replaced with string concatenation, multiplication in Map replaced with character replication, zero-padding Pad becomes character deletion, and the numbers 0-9 are replaced with the characters a-j.
For humans, although the process of reasoning tends to be implicit, according to the traditional definition in logic, it can be divided into two stages: induction and deduction. Here, we focus on the objectives of these two stages: deriving the correct rules through induction and making inferences on new instances using deduction. We aim to explore the correlation between the two during the reasoning process of LLMs. Therefore, for cach scenario, we design two tasks: rule induction (RI) and example inference (EI), defined as follows:
Rule Induction Task: Given observed fact set X. this task evaluates the model's accuracy in inducing transformation rules f. It aims evaluate the model's proficiency in mastering intermediate rules during inductive reasoning (Yang et al.] (0246) (Shao et al. 2024).
Example Inference Task: Given an observed fact set X. the task provides an unseen example input æ, as input and measures the accuracy the predicted y:. aims evaluate the final performance
of the model's inductive reasoning (Chollet 2019 Xu et al. 2024a Qiu et al. 2024).
We provide all the prompts used for these tasks in Appendix A.3
3 LANGUAGE MODELS ARE POOR RULE-BASED REASONERS
3.1 OVERALL PERFORMANCES ON MIRAGE
Setup We first evaluate the overall performance of various LLMs on MIRAGE. Here, select GPT-4 (OpenAl 2023). GPT-40, Claude-3.5. 3.5, Llama3-8B (Dubey et al. 2024), and Llama2-13B (Touvron et al. 2023 as representative models For the first three models, given their strong instruction-following capabilitess, we provide only the instruction and allow them to answer the questions in a zero-shot setting. For the latter two models, to improve the format accuracy of the response, we additionally provide five examples hefore they answer the questions. Unless otherwise specified, continue to use this setup prompt the model in the subsequent experiments. For the dataset setting, we fix the size N at 5 and measure performance across four
Herre real-world scenarios refer to mathematical inductive reasoning within natural language contexts. Due the frequency limitations of API calls, we can not conduct our evaluation on the latest ol model.
4

---

Published as a conference paper at ICLR 2025
scenarios when the dimension 3, We sample 500 questions for each test. More implementation details
can found in Appendix B.I
D=3
D=5
D=8
Model
Task
LT
RP
CG
ST
LT
RP
CG
ST
LT
RP
CG
ST
RI
0.01
0.00
0.00
0.03
0.01
0.01
0.00
0.21
0.00
0.01
0.00
0.10
Llama2-13B
El
0.26
0.11
0.25
0.22
0.13
0.03
0.14
0.25
0.06
0.01
0.06
0.19
RI
0.15
0.11
0.19
0.19
0.23
0.04
0.14
0.22
0.16
0.02
0.08
0.21
Llama3-8B
El
0.30
0.15
0.25
0.25
0.20
0.12
0.25
0.29
0.09
0.11
0.16
0.24
RI
0.41
0.32
0.38
0.32
0.35
0.21
0.44
0.30
0.33
0,16
0,41
0.24
GPT-4o
Fl
0.68
0.37
0.61
0.56
0.58
0.25
0.64
0.39
0.42
0.17
0.49
0.29
GPT-4
RI
0.47
0.29
0.41
0.28
0,58
0.22
0,56
0.27
0.46
0.15
0.45
0.23
El
0.68
0.37
0.61
0.57
0.63
0.29
0.71
0.44
0.42
0.21
0.64
0.30
RI
0.44
0.35
0.34
0.46
0.22
0.20
0.38
0.33
0.24
0.13
0.38
0.26
Claudo-3.5
E[
0.79
0.45
0.62
0.58
0.65
0.33
0.76
0.45
0.46
0.24
0.59
0.30
Table 1: Overall performance (accuracy) of different models on MIRAGE. The best results in rule induction (RI) are bold, while the best results in example inference (EI) are underlined.
The results are shown in Table [ from which we can draw the following conclusions: (1) LLMs' inductive reasoning does not rely on rule induction. Given the same of observed facts, the model's performance on rule induction is noticeably worse than on example inference in almost all cases. This suggests that most of the model's s correct deductions do not depend on inducing a correct rule. (2) LLMs face difficulties in handling inductive reasoning in real-world problems. When comparing different scenarios, all models perform the worst the RP tasks. For example, GPT-4o only achieves 0.16 and 0.17 accuracy when the dimension is 8. This indicates that, compared purely symholic forms (LT, CG, ST), natural language forms pose a greater challenge for the models' inductive reasoning abilities.
Results
Supplementary Experiments In the main experiments, we find that there a significant performance gap between rule induction and example inference for LLMs. However, this gap may be caused by the dif- ference in difficulty between the two tasks. When the model is unable to perform correct inductive reasoning, it is likely to guess the correct answers for the El task more casily compared to the RI task. resulting in a higher accu- racy. We conduct this additional experi- ment eliminate the interference of this factor. Specifically, we randomly perturb one fact in X to violate rule f. Then, we observe the performance of both tasks and calculate the change rate (CR) of accuracy before and after the perturba- tion. CR represents the sensitivity of the model's performance to the input. If CR is high. it indicates a strong correlation between task performance and input, making it difficult to answer the question correctly through random guessing. Therefore, CR can serve for the task. We randomly choose 100 pieces of test data from the dataset and generate questions under the LF scemario [I The experimental results on different models are demonstrated in Table 2 We can observe that the two tasks have comparable CR for both models, indicating that the reasoning difficulty of the EI task is not lower than the RI task. The tasks themselves do not cause such a large performance gap.
Rule Induction
Example Inference
Model
BF
AF
CR
BF
AF
CR
GPT-4o
0.50
0.13
0.74
0.66
0.15
0.77
Claude-3.5
0.37
0.07
0.81
0.65
0.22
0.66
Table 2: Comparison of CR on two tasks (D 3,
N = 3). BF
and AF indicate the accuracy before and after perturbation.
indicator of the reasoning difficulty
3.2 PERFORMANCES OF ADVANCED METHOOS
In 3.1
we observe that LLMs perform poorly dataset, especially in rule induction tasks. Considering previous work has proposed numerous methods to elicit the model's reasoning abilities (Wei al. (2022 Wang
all 2023b Madaan al. 2023), we wonder whether they can boost. models' performance on MIRAGE.
Setup Since we focus on exploring the model's intrinsic capabilities, we only consider methods that do not introduce any external tools or knowledge. Specifically, the methods are as follows: Input- (10): We prompt models generate answers directly under different shots. Inductive-Deduuctive (ID): We prompt
Unless otherwise specified, this configuration will be maintained for all subsequent experiments.
5

---

Published as a conference paper at ICLR 2025
LT
RP
CG
ST
Method
RI
EI
(A)
R1
EI
(A)
R1
E1
(A)
RI
HI
(A)
10 (Q-shot)
0.46
0.76
0.30
0.43
0.72
0.28
0.39
0.46
0.08
0.47
0.70
0.23
10 (S-shot)
0.63
0.76
0.13
0.59
0.77
0.17
0,55
0.54
-0.02
0.52
0.78
0.26
ID (0-shot)
0.46
0.56
0.11
0.46
0.57
0.11
0.33
0.42
0.09
0.22
0.65
0.43
ID (5-shot)
0.59
0.68
0.08
0.57
0.66
0.08
0.47
0.54
0.08
0.48
0.69
0.21
CoT (0-shol)
0.50
0.57
0.07
0.47
0.55
0.08
0.34
0.39
0.05
0.52
0.62
0.10
CoT (5-shol)
0.56
0.59
0.04
0.41
0.55
0.13
0.37
0.40
0.03
0.45
0.64
0.20
SC (a=5)
0.59
0.74
0.15
0.49
0.62
0.14
0.38
0,45
0.07
0.57
0.68
0.10
SR (1=3)
0.48
0.64
0.16
0.42
0.67
0.25
0.36
0.49
0.13
0.53
0.67
0.14
HR (1=3, m=1)
0.56
0.68
0.12
0.45
0.71
0.26
0,41
0.53
0.11
0.43
0.71
0.27
HR (1=3, n=
0.66
0.79
0.13
0.59
0.80
0.21
0.55
0.60
0.05
0.49
0.67
0.18
Table 3: Performance of different methods on MIRAGE using GPT-4o. The best results in each column are highlighted in bold, while the second best results are underlined.
models to generate rules for RI and apply them to answer questions in El. Thain-of-Thoure (CoT) (Wei et al.
2022): We prompt models to generate rationales and answers for the two tasks. Celf-Consistency (SC) (Wang) al. 2023b); Based on CoT, we sample rationales and use the major voting strategy predict the final answer. Self-Refine (SR) (Madaan al. p We prompt the model to provide fcedback on the generated rules, and then refine the rules based on that feedback (with a maximum of itcrations). After the itcration stops, we use the latest rule to answer the RI and apply it to answer the EL. Hypothesis Refinement (HR) (Qiu al. 2024): HR is optimized version of SR, which first generates rules. In each iteration, we apply the current rules to all observed examples, compare the actual output with the expected output, and get the number of correct predictions along with the information about incorrect examples. observed facts, it is immediately returned. Otherwise, the rule with the highest number of correct predictions and the associated error information is used as input for the model to refine. generating n rules for the next round, until the maximum number of iterations is reached. We sample 200 questions for each test.
candidate rule is correct for all
We illustrate the main results on GPT-4o in Table 3 (more results in Appendix B.2 from which we can conclude that: (1) Advanced methods provide limited improvement to the model's inductive reasoning ability or may even have negative effects. For both tasks, directly answering with few-shot settings can consistently achieve the highest or second- -iighet accuracy in most cases. After applying methods like CoT, the model's s accuracy decreases by up to 18% and 22% on two tasks, respectively. It indicates that the key to optimizing inductive reasoning does not lie in refining the intermediate inductive process (as CoT-like methods do). (2) The model's disregard for abstract rules during inductive reasoning is method-agnostic. Although some methods use instructions to guide the model to focus more on the induced rules during reasoning (c.g. ID, SR, HR), there remains a significant gap in the model's 's RI and El performance. For example, in the case of SR, the model's 's example inference accuracy outperforms its rule induction accuracy by an average of 16 %.
Results
3.3 IMPACT OF INCREASING FACT SIZE
In the previous experiments, we consistently fix the observed fact numbers N. Therefore, as a supplement, we explore the impact of N on the model's 's inductive reasoning process in this section. Theoretically, the number of observed facts increases, the scope of the candidate rules narrows, which can lead the incorrect inductive process becoming correct. the reasoning process rule-based, the model is likely to generate the correct rule (inductive) before applying it correctly (deductive). In other words, the time when the LLM induces the correct rule is no later than the time it performs the correct deduction. Thus, the cumulative number of observations required for the inductive rule to change from incorrect correct should not exceed the number required for the test case to become correct. We design this experiment to validate whether holds on the LLM.
Setup Given a fact set x of size k. and fixed test input æt, we define the inductive correction threshold (ICT) and deductive correction threshold (DCT) as follows:
ICT < Vi < k M(X (2) + f M(x
DCT - k M(X ..I A A(Xx" xx(D)) = f(x:)
(4)
Here, M() are the model's outputs in RI and El tasks. We set D= 5 and vary N. then analyze the distribution of these two thresholds across 100 samples, reporting the results in Figure 3
Results Based on the result, we further demonstrate that LLM's deduction does not rely on an inductive
rule. From both of the two figures, we can observe that most points are distributed in the upper left region
6

---

following section, we focus on Published as a conference paper at ICLR 2025
6/5
67
077
046
67
66
1:44
144
021
144
6.5
64
103
1F
CG
433
1F
R7
CG
(a) GPT-4o
(b) Claude-3.5
(a) GPT-4o
(b) Claude-3.5
Figure 3: The distribution of ICT and DCT for Figure Performance on EI tasks under different scenarios of observed and test facts.
the examples across different models.
of the line y. indicating that for the vast majority of cases, DCT is smaller than ICT. Therefore, the fact numbers does not affect the conclusion we stated earlier. LLM requires fewer facts to successfully perform an example inference task compared to correct induction.
3.4 TRANSFERABILITY TEST OF INDUCTIVE RULES
Finally, we investigate the impact of different scenarios on the inductive reasoning process. For rule-based reasoning, once a rule is formed through induction, it should be transferable. That is, a rule induced in one scenario should be applicable to another scenario with the same underlying transformation. We experiment to explore whether LLMs possess this ability when performing inductive reasoning.
Setup Specifically, we exclude ST in this experiment since its basic transformations differ from the other three scenarios (see (2.1]. For the remaining three scenarios, we generate the observed facts in one scenario, and then transform test case into another scenario. Since our dataset can generate questions different scenarios based on the same meta-rule, we can easily ensure that they share the same underlying transformation.
Results From the results shown in Figure 4 we can get that: (1) LLMS lack transferability in inductive reasoning, Across different cases, the highest performance occurs when the scenarios of the observed and test facts are consistent (i.e.. the diagonal from the top left to the bottom right in the figure). The inductive reasoning process of the LLM form-related. Compared to the transfer between LT and RP (or CG and RP), the transfer between LTT' and CG demonstrates better performance. We infer that this is because the forms of L.T and CG are more similar (see Figure 2. In addition, we also design experiments under the fine-tuning paradigm compare the model's 's transfcrability (presented in Appendix [B.3). and the results remain consistent. Based on the above observations, further confirm that LLMs do not rely abstract rules when performing inductive reasoning. So, what the underlying mechanism behind addressing this question.
4
LANGUAGE MODELS ARE GOOD NEIGHBOR-BASEE REASONERS
4.1 MOTIVATION
From 3.4 we know that closer forms between the observed facts and the test case enable the model to perform inductive reasoning more effectively. However, is the positive impact brought by the similarity limited only to the form? The answer to this question is likely "No". Upon reviewing related works, we find that models tend to match various similar patterns in the context and use them to predict the next token (Olsson ct al. 2022 Wang et al. 2023a Hu et al. 2024b). Therefore, we aim to identify metric to measure some other similarities between the observed facts and the test input. Since all of our facts are transformed from vectors, we associate this similarity with the distance between these facts in feature space.
topology, if ʃ + Y a continuous function between two Euclidean spaces and N(wo, c) is e- neighborhood of the point æo in X, then we have:
Bn > 0, S.L. Vz (.(aa, f(x) (((f(aa)
In other words, continuous functions preserve the neighborhood property. If a fact input vector æ closes to the test input then their output vectors y and ye will remain close Therefore. the close distance between y1 and y may allow LLM to prediet yı based rule generation. In the following sections, we demonstrate through experiments that the model's inductive reasoning relies on this paradigm, which we refer to as neighbor-based reasoning.
y in observed facts without the need for correct
We rigorously prove in Appendix C.1 that the rules in our dataset all continuous functions.
7

---

Published as a conference paper at ICLR 2025
66
A2000
03
60
60
80
(a) GPT-4o
(b) Claude-3.5
(c) Llama3-8B
Figure Performance of different fact types on our dataset (D 3, N 5). represents the baseline accuracy.
The dashed line
N=3
N=5
N=8
Type
L.T
RP
ST
RP
ST
RP
CG
Baseline
0.52
0.19
0.78
0.42
0.66
0.36
0.71
0.46
0.76
0.34
0.80
0.54
IF Only
0.78
0.46
0.82
0.59
0.84
0.52
0.84
0.63
0.86
0.54
0.91
0.70
CF Only
0.48
0.25
0.59
0,43
0.69
0.35
0.72
0.50
0.75
0.34
0.82
0.53
OF Only
0.46
0.18
0.50
0.38
0.49
0.23
0.57
0.36
0.61
0.23
0.67
0.38
Table 4: Performance of different fact types under various settings (D = 5). The best results in each column are highlighted in bold, while the worst results are underlined.
4.2 NEIGHBOR FACTS IN INDUCTIVE REASONING
Before conducting the experiments, we first define some key concepts in our work: the distance dl and neighbor- hood In our setup, the components at corresponding positions the vectors follow the same transformation rulcs, while non-corresponding components may undergo different transformations (see Equation 2 Hence, we consider using the distance based on the corresponding components: Chebyshev distance (further discussion in Appendix C.2 Given observed fact X; = (æt, M:) and test input æt, we have:
d(x., mx) = max ( (Jr
(6)
where wu and Fi are k-th component two input vectors. Then we can define the e-neighborhood based on the distance:
Wa c) (X, d(x, xx) se
Setup According above definitions, we can divide an observed fact X2 into three categories based the distance between and the test input (1) In-neighborhood Fact (IF): If %A E EN(x,, é), call Xx is in- neighborhood fact. (2) borgho Fact (CF): If Xx 4 $ [/(x,,) <), but 3i E [1, D). s.t. [z,-Zt:1 - we consider it suboptimal neighbor fact because some its components can still contribute to the model's inductive reasoning process. In this case, we call X, is cross-neighborhood fact. (3) Out-ncighborhood Fact (OF): If vi [1, D). [x; Teil these definitions, sets, we compare the model's performance
we call X is an out-neighborhood fact. By generating examples based make the fact X contain only one type of fact. After constructing different fact tasks under these settings. Besides, we use the performance under the default fact set as the baseline, where all facts are randomly sampled without any constraints.
Results We report the results in Figure 5 It demonstrates that: (1) LLM's inductive reasoning is neighbor- based. By comparing three settings. we find that observed facts closer to the test case result in better perfor- mance (IF > CF OF) across all models. Besides, compared to the baseline (the dashed line in figures), the accuracy significantly drops after removing the neighbor
X (i.e. OF). These phenomena indicate that the model hcavily relies on neighbor facts during reasoning. (2) LLMSs have strong ability to capture neigh- boring patterns. When we set the neighborhood radius to 4, both IF and CF still contribute to high accuracy for the model. Besides, OF continues to show significant decline (compared to = 3). These observations indicate that LLMs still learn similar patterns when the observed facts relatively distant.
4.3 UNIVERSALITY OF NEIGHBOR-BBASE BBASED REASONING
We consider whether L.L'' inductive reasoning universally relies neighbor cases, hence, set to and repeat the experiment under different settings, where the baseline is the same as & 4.2 The results GPT-
8

---

Published as a conference paper at ICLR 2025
(a) D = 33c
(b) 3, inf
(c) 5,
(d) D 5, inf
Figure 7: Deductive Density (1a) of various fact types on GPT-4o under different test radius 1/ and neighborhood radius (N 5). BL represents the performance of the baseline, where we use default fact sets with no substitution.
4o are reported in Table 4 which demonstrates that the neighbor-based paradigm is universal in LLMs* inductive reasoning process. Across different scenarios and fact numbers, IF consistently gets the highest accuracy. while OF gets the lowest accuracy. The reliance of LLMs' inductive reasoning neighbor facts independent the specific task scenarios, models, or fact numbers. present more results in Appendix
4.4 EFFECTIVE SCOPE OF NEIGHBOR-BASED REASONING
We have demonstrated that neighbor exam- ples in observed facts significantly affect the the test case. How-
model's performance ever, what is the effective scope of it? it only pattern matching on single test example or reasoning an implicit rule that affects more ex- amples? To answer the question, we first make three assumptions about its possible scope and show them Figure 6 For individual scope, the model can only answer the test case o; (c.g. [3,4,7] in the figure), for all other cases, the ac- curacy of the prediction is very low. For lo- calized scope, the model also answer cases close to æ, (i.e. the neighbor facts ax).. For global scope, model can answer all cases with high accuracy.
Space
[1.41
Space
(1,44]
Spoce
[U04]]
Naighbochood
Nestiooatsood
Neighbothood
14.30 O
[A388
1A.0 O
BAA
[347
B36
[AA.1]
[BA11
(99111
Tretivichal
Localized
Globsal
Figure 6: Examples for three different effective scopes.
Setup In our experiment, we sample n test cases X, (here n. = 5) in each El task r and define the accuracy a, for this particular task as:
Σ xI f(x)]
(8)
# EEX:
Here x is the observed fact set of the task. Let T denote the set of all EI tasks (we set [T] 100), we define deductive density La
La To.
Σ aaI[MM(X,, ax f
TET
[Ta| = Σ T[M( = fxx))
(10)
where is the origin test input in task 7. We use this metric to indicate the impact of a successful deduction (i.e. [3, Figure on reasoning over other examples in the test region N (x,,, 17). A high Ia indicates that the model performs well most cases within this region, while a low I4 suggests that the model's reasoning more localized or even individual. For comparison, we set the test radius ") to 1, 2, 3, and infinity (i.c. the full test space), and calculate the corresponding La for the model, Besides, we also vary the neighborhood radius examine the impact of different distributions of neighbor facts their effective scope (here region to full space). repeat the experiment five times to eliminate interference of random errors, and the results are illustrated Figure
the test
Results We can draw conclusions as follows: (1) LLM conducts localized reasoning through the
neighbor-based paradigm. From Figure Pa 7c we observe that the Ta of IF and CF decreases continuously the radius of the test domain expands. These neighhor cases highly effective within the neighborhood of

---

Published as a conference paper at ICLR 2025
at. For example, in Figure 7a when 7) = 1, the model can achieve over 0.9 La- However, this impact dimin- ishes for test cases that are farther from 2t. As example, Figure 7c the model only gets around 0.5 la full test space. (2) The effective scope of neighbor facts is proportional to their distance from the test case, According to Figure [b the La of IF and CF (particularly IF) increases as e becomes large. When the neighborhood radius increases, the distribution of these facts becomes more dispersed. We can infer that dispersed distribution of neighbor facts tends to make the effective scope more global.
5 LIMITATIONS AND DISCUSSIONS
Interpretation Methods Most model interpretation studies delve into the internal of models (e.g. neurons, attention layers), providing comprehensive explanation of the working mechanisms (Romera- et al. 2024 et al. 2024). However, our work does not conduct internal analysis but instead relies on performance comparisons under different settings. There are two main considerations for this: On one hand, this work aims to identify mechanisms that are applicable to black-box models. Since we do not have access to the internal parameters of these models, we are unable to use previous methods. On the other hand, white-box models exhibit poor inductive reasoning capabilities according Table [ Therefore, conducting in-depth interpretations based on white-hox models may introduce noise to the conclusions.
Experimental Settings The goal of this paper evaluate and explain the inductive reasoning process LLMs, rather than to improve the task performance. Therefore, we do not meticulously design the prompts used in the experiments, nor do we use the best-performing inductive reasoning methods throughout the analysis. We believe that the experimental setup of 0-shot IO with simple instructions is more aligned with real-world application scenarios, making our evaluation and explanation results more meaningful.
Future Directions Our study demonstrates that LLMs perform poorly in tule-based reasoning but excel using neighbor facts for reasoning. Future work could explore methods to encourage the model to follow rules more closely during reasoning or to further optimize the model's 's inductive reasoning abilities based on this neighbor-matching finding.
6 RELATED WORK
Evaluating Inductive Reasoning Abilities of LLMs. Existing studies on evaluating L.M's inductive reasoning capabilities mainly use only single task. On one hand, some works assess the model's 's rule induction ability by evaluating the inference accuracy on unseen examples (Moskvichev et al. 2023 Tang et al. 2023] Gendron al. 2023 Xu et al. 2024b Qiu al. 2024). However, since the model's deduction does not always rely on inducing the correct rule, this indirect eval evaluation method can introduce some inaccuracies. On the other hand, some studies directly evaluate the correctness of the generated rules to assess inductive reasoning ability (Shao et al. 2024] Cheng ct al.] 2024] Yang et al.) [024b) Wang et al.) (022). These studies lack evaluation on test examples, making difficult to confirm the model's mastery of the inductive rules. Our work evaluates both aspects, providing comprehensive analysis of the model's inductive reasoning process.
Mechanism Analysis on LLM's Reasoning. A growing body of interpretability research has begun amalyzing the rcasoning mechanisms of LLMs, aiming deepen our understanding of how these models func- tion. Some studies explore the mechanisms behind mathematical reasoning (Zhang et al.] [0044 Hu el all 2024b Rooera-Parreess et al. 2024; (Stllo el al., (202)), some works investigate multi-hop reasoning (Wang) al. 2024a| Hou et al. 2023 Yang et al. 2024al Biran et al. 2024), and some focus other types of rea- 2024a). However, there is currently lack of analysis the mechanisms
soming (Li al. 2024 Hu
of inductive reasoning. work mitigates this gap and uncovers the neighbor- based paradigms LLMs follow
when performing inductive reasoning.
7 CONCLUSION
In this paper, we focus on evaluating and explaining the inductive reasoning process of LLMs. First, we con- struct datasct MIRAGE, which provides both inductive and deductive evaluation tasks, with the ficxibility to generate test examples in any distribution, different difficulties, and various forms. Based on it, we demon- strated that LLM is a poor rule-based reasoner, it does not need rely inductive rules when performing inductive reasoning. Compared correct induction, the model can perform successful deduction with fewer observations, and this deduction closely related to the form of the input. Furthermore, we identify a key paradigm of LLM inductive reasoning: neighbor-basee that are close to the test examples in feature space for inductive reasoning. Through it, the model can achieve strong inductive reasoning capabilities within localized scope and apply this ability to make inferences on unseen examples.
reasoning. The model tends to leverage observed facts
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation China (No.U24A20335, No. 62176257, No. 62406321). This work is also supported by Beijing Natural Science Foundation (L243006). This work is also supported by the Youth Innovation Promotion Association CAS and the China Postdoctoral Science Foundation under Grant Number 2024M753500.
REFERENCES
Eden Biran, Daniela Gottesman. Sohee Yang, Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models on multi-hop queries. CoRR. abs/2406. 12775. 2024, doi: 10.48550/ /ttppstss:///...crr
ARXIV
48550/arXiv 240...iwii......... 12775
Kewei Cheng. Jingfeng Yang. Haoming Jiang, Zhengyang Wang, Binxuan Huang, Ruirui Li, Shiyang Li, Zheng Li, Yifan Gao, Xian Li. Bing Yin, and Yizhou Sun. Inductive deductive? rethinking the fundamen- tal reasoming abilities of llms. CoRR, abs/2408.001 14, 2024. doi: 10.48550/RRXI22488.00111 URL
https: http://ccc...gg////// crg/10. 48550/arXiv. 2408. 00114
François Chollet. On the measure of intelligence. CoRR, abs/1911.01157,, 2019. URL http://arxiv.
2cg/abs/ 19911.01541
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev. Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang. Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferror, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis An- derson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evti- mov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Ji- awen Liu, Wang, Jiccao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua John- stun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Ken- neth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR. abs/2407. 21783, 2024, doi: 1783. URL (ttpttts::/ http::::///...oog//.././
wwt.:/44550........0 http://axxvv....... 21782
Gaël Gendron, Qiming Bao, Michael Withrock, and Gillian Dobbie. Large language models are not abstract reasoners. CoRR, abs/2305. 19555, 2023. doi: 02448500ARXXIV22255.1555 10. (13555//aX1i10 2305. 19555
(Rtlssttp:://// settps://doi.ooe org/
Simon Jerome Han, Keith J Ransom, Andrew Perfors, and Charles Kemp. Inductive reasoning in humans and large language models. Cognitive Systems Research, 83:101 155, 2024.
Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng. Antoine Bosselut, and Mrinmaya Sachan. Towards a mechanistic interpretation of multi-step reasoning capabilities of language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, FMNIIP 2023, Singapore, December 6-10, 2023, pp. 4902-4919. Association for Computational I. Linguisties, 2023. doi: 10. (M66:////223.MN..PMMMMM
Yebowen Hu, Kaigiang Song, Sangwoo Cho, Xiaoyang Wang, Wenlin Yao, Hassan Foroosh, Dong Yu, and Fei Liu. When reasoning meets information aggregation: A study with sports narratives. CoRR, 0bs/2400.12088. 2024a.
doi: 10.45550/ARXV.2246022222 12084. URL /tttps://do..rgg//
arxi v. 2406.
Yi Hu, Xiaojuan Tang, Haotong Yang, and Muhan Zhang. Case-based rule-based: How do transformers do math? Forty-fist International Conference on Machine Learning, ICML 2024, Vieuna, Austria, July 21-27, 2024. OpenReview.net, 2024b. URL httpsst:sttt http:p:s/////peeeviieeee wwt.eeewwweeeeeeeeeeeeeeeeeeeeee wtt.eewww.eweeeetteee/ee (oorrmmii-44/r8sRFr.
Franz Huber. On the justification of deduction and induction. European Journal for Philosophy of Science, 7: 507-534, 2017.
11

---

Published as a conference paper at ICLR 2025
Yifan Jiang, Jiarui Zhang, Kexuan Sun, Zhivar Sourati, Kian Ahrabian, Kaixin Ma, Filip Ilievski, and Jay Pu- jara. MARVEL: multidimensional abstraction and reasoning through visual evaluation and learning. CoRR, abs/2404.1355 13591, 2024. doi: 1044850/ARXIV.2044.55555 URL htttps://dll.rgg
arxiv arXiv.2404. 13591
Subin Kim, Prin Phunyaphibarn, Donghyun Ahn, and Sundong Kim. Playgrounds for abstraction and reason- ing. In NeurlPS 2022 Workshop on Neuro Causal and Symbolic Al (nCSI), 2022.
Brenden Lake, Tal Linzen, Marco Baroni. Human few-shot learning of compositional instructions. In Ashok K. Goel, Colleen M. Seifert, and Christian Freksa (eds.), Proceedings of the 41th Annual Meeting of the Cognitive Science Society, CogSci 2019: Creativity + Cognition Computation. Montreal, Canada, July 24-27. 2019, pp. 611 617. cognitivesciencssccieyy..gg 2019. URL https: http::://mmmmmmmmmmmmm////m Ling org/ Cogsci 201 10200/9papppe wttpp//ppppppa////
ndex. html
Jiachun Li, Pengfei Cao, Chenhao Wang. Zhuoran Jin, Yubo Chen, Daojian Zeng. Kang Liu, and Jun Zhao. Facus on your question! interpreting and mitigating toxic cot problems in commonsense reasoning. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Mceting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok. Thailand, August 11-16, 2024, pp. 9206-9230 Association for Computational Linguistics. 2024. doi: 10.18653/V 1/2024. ACL-LONG.499, URL https: //wi//do..ooo/////// wwt.oooooggoggoooooooooooeeeoooe 1tt.:......53/11111
acl long................ 499
Linhan Li, Huaping Zhang, Chunjin Li, Haowen You, and Wenyao Cui. Evaluation on chatgpt for chinese language understanding. Data Intell., 5(4):885-908 2023, doi: AA) URL https:
//doi.org/100 11tp///6t_aaaaa_eee 03232
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazılabakhssh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. CoRR, abs/2303.1763 2023. doi: 10448550/ARXV22303... arX1 v 2303.
URL
Arsenii Moskvichev, Victor Vikram Odouard, and Melanic Mitchell. The conceptare benchmark: Evaluating understanding and generalization in the ARC domain. Trans. Mach. Learn. Res., 2023, 2023. URL bttpst
//openreview net. wwt./eetofrtti BeerrummmddByyyyy...
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, 7 Haffield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish. and Chris Olah. In-context learning doi: 10448500ARRX1.2200.11$5 URL https:
and induction heads. CoRR, abs/2209. 11895, 2022.
//doi.org 10. 18w50//a553301 1189. 1895
OpenAl. GPT-4 technical report. CoRR, whtp/3/3..0/77/// 2023, doi: 10.48550/AAXXXV2233..0777 URL
/ettos://dc..orge 20t.0///////...2.....0.1171. 2303.
Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang. Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, 2024. URL https://cpenreviess net/ forum? 1d=bNt 70a 112a
Beraariio Comera-Parcdes Mohammadamin Barckatain, Alexander Novikov, Matej Balog, M. Pawan Ku- mar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. doi: 10. 10.0088S511366-00--699444
Nat., 625(7995):468-4775 2024.
URL
1038/ 0205--886-22----- -23-05924- 06924-6
Joshua Stewart Rule. The child as hacker: building more human-like models of learning. PhD thesis, Mas- sachusetts Institute of Technology, 2020.
Yunfan Shao, Linyang Li, Yichuan Ma, Peiji Li, Demin Song. Qinyuan Cheng, Shimin Li, Xiaonan Li, Pengyu Wang. Qipeng Guo, Hang Yan, Xipeng Qiu, Xuanjing Huang, and Dahua Lin. Case2code: Learning induc- tive reasoning with synthetic data. CoRR, ahs/2407. 12504, 2024. doi: 10.48850/AAXX1..40...22 URL. https:://cii www./errggo... wttp//...850///2330... 2407.
12

---

Published as a conference paper at ICLR 2025
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic rea- soning language models using causal mediation analysis. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 7035- 7052.705. Association for Computational Linguis- tics, 2023, doi:
/TT////0.......MA..AA...... URL https://doi...... ww..orgg1000 18653/v1 2023. emm1p mammnp-mainn. main. 435
Tianxiang Sun, Xiaotian Zhang, Zhengfu Hc, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, al. Moss: An open conversational large language model. Machine Intelligence Research, pp. 1-18, 2024a.
Wangtao Sun, Haotian Xu, Xuanqing Yu, Pei Chen, Shizhu He, Jun Zhao, and Kang Liu. Itd: Large language models can teach themselves induction through deduction. Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics Association
(Volume 1: Long Papers), ACL 2024, Bangkok. Thailand. August 11-16, 2024, pp. for Computational Linguistics, 2024b. doi: 10.18653W112224444LLLLL00 150. URL https://doi.
acl:0-24.cccc. long. 150
Xiaojuan Tang. Zilong Zheng, Jiaqi Li, Fanxu Meng. Song-Chun Zhu, Yitao Liang, and Muhan Zhang, Large language models are in-context semantic reasoners rather than symbolic reasoners. CoRR, (bs/2235.11425, 2023. doi: 00445500ARXXVV2305..455
URL tlttps://doos wtaedii.org..oooooooo
httpss//rrxvvv.. 2305.
14825
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cyn- thia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khahsa, Isahel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thihaut Lavril, Jenya Lee, Diana Liskavich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet. Todor Mihaylov. Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- ging Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanic Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Sto- jnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine tuned chat models. CoRR, abs/2307.09 092888 2023. doi: 4448500/AXI..230700009000
URL httts:://dot org/ 4855C/
arXiv 2307 09288
Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic journey the edge of generalization. CoRR, a55/2405.15071, 2024a. doi: (0-48550/ARXXI22245.55555
ptRlhhttss://// wwaeddc...rg//// 000 48550/ 2405. 15071
Chenhao Wang, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. A survey recent advances in commonsense knowledge acquisition: Methods and resources. Machine Intelligence Research, pp. 1-18, 2025. doi: 10. 1007/s1 16331023-3--00221--333
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore. December 6-10, 2023, pp. 9840-9855. Association for Computational Linguistics, 2023a. doi: 10.18653/V1/2023..0 WEW.EEMNNNNIIIIEEE
URL http::htt::////
org/10 3zg/10.1865333
cmn emnip- ma in..n.... 609
Ruocheng Wang, Eric Zelikman, Gabriel Poesia. Yewen Pu, Nick Haber, and Noah D. Goodman. Hypothesis search: Inductive reasoning with language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL https: 7/openreview net www./eeoooueeee
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quee V. Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations. ICLR 2023, Kieali. Rwanda, May 1-5, 2023. OpenReview.net, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter. Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. tf-uughh prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho. and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28
13

---

Published as a conference paper at ICLR 2025
December 2022. 2022. https:///papesssnnssscccpppeeeee http://ppprrrileee/ppp//////////
Chaojie Wen, Xudong Jia, and Tao Chen. Improving extraction of chinese open relations using pre-trained language model and knowledge enhancement. Data Intell., 5(4):922 989, 2023. doi: 10.1162/DINT) _00227. http:tppee/.oooo..//1111111111..
2aapnntt_________ 00227
Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, Elias Boutros Khalil. Lims and abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations. Trans. Mach. Learn. Res., 2024, 2024a. B
net forum? 10- E6m88m8ySS
Yudong Xu, Wenhao Li, Pashootan Vaczipoor, Scott Sanner, and Elias Boutros Khalil. Lims and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations. Trans. Mach. Learn Res., 2024, 2024b. URL httpttttttt//ptooeeeeetoo
net forum? wewtrrrmmmmd---ieeee 1d-E8n8y5vVV
Sohee Yang. Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language mod- els latently perform multi-hop reasoning? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Mecting of the Association for Computational Linguistics (Volume Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 10210-10229 Association for Compu- URL pettps::/doi.org/
lational Linguistics, 2024a. doi:
18653/v1/ 2024. http:////222.....111 acl- Long. 550
Zonglin Yang. Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. Language models as inductive reasoners. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 Volume Long Papers, Julian's ss Malta, March 17-22. 2024, pp. 209-20922222 Association for Computational Linguistics, 2024b. URL hctpssss:: / 1 wtw.org...oooo.oo...o........... 2024 eacl c1 1ae.4eeec--1oo.1111
Wei Zhang, Chaoqun Wan, Yonggang Zhang, ftu-ming&& Cheung, Xinmei Tian, Xu Shen, and Jieping Interpreting and improving large language models in arithmetic calculation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
URI/htttss///penrree net. forum? d-C.ft i
14

---

Published as a conference paper at ICLR 2025
A MORE DETAILS FOR DATASET CONSTRUCTION
A.1 COMPARISON OF DATASETS BETWEEN RELATED WORK AND OUR STUDY
Since our work is conducted entirely on our MIRAGE dataset, we aim to provide a detailed comparison with representative datasets from related studies to demonstrate its effectiveness. Specifically, report the com- most of the operations and forms
parison in Table 5 From the results, previous datascts. For example, the transformation examples in the ID-ARC dataset shown in the table equivalent to our PAD operation. Therefore, we demonstrate that our dataset an effective dataset for inductive reasoning. Moreover, based on its coverage, conducting experiments solely on it is sufficient.
that dataset
Dataset
Fact Form
Main Operation
Example
input: [[0,1,0],[1,1,0], 10.110110.111111111101 [1.1,0]]
ARC (Chollet 2019
List (2D)
Fill, Move, Pile
output: [[0.2.0][[222,0]] 10.22,0,002222210....0000 0,2000000.2200110.22... [0.2,0]
input: her sneury voirk
MiniSCAN (akee et 2019
String
String Translation
output: GREEN BLUE input: [4.7,6,9.0]
ListFunctions Rulle 2020
List
List Operation
output:[48,66990)
input: [[1,1,5,6,8] [0.1.566.66[15555555555. [777.5.4.41117.55504444]
MiniARC (Kim al. 2022
List (2D)
Fill, Move, Pile
output: [[1,6,0,0,0]. [7..00000001000000000000 [0,0,0.0.0], (0.0000000)
input: [0.0.2.000...0222.. 0,0,0]
ID-ARC
Xu al.
2024a
List
Fill, Move, Pile
output: [0.0,2.222,2.2222. 0,0,0]
input: dict(no=2)
Casc2Code
Shao ct al.
2024
Code
Python Function
output: [2]
MIRAGE
All above RP
All above
See Figure 2
Table 5: Comparison between some representative datasets and ours.
A.2 EVALUATION OF MATHEMATICAL OPERATION DIFFICULTY IN MIRAGE
Our data introduces some mathematical operations in Add and Map, here we aim to demonstrate that these calculations are inherently simple for LLMs, ensuring that they do not interfere with evaluation of the model's inductive reasoning performance. Specifically, randomly construct linear operations in Add and operations included in this paper) and observe
Map with single-digit operands (cover all of each model on 100 questions. The results are reported Table We can observe that most of the modes can achieve very high accuracy on these math operations. Specifically, all closed closed-source source models achieve 100% accuracy. This indicates that dataset construction effectively eliminates noise introduced by mathematical calculations in most cases.
accuracy
15

---

Published as a conference paper at ICLR 2025
Operation
Llama2-13B
Llama3-8B
GPT-4o
GPT-4
Claude-3.5
Map Add
0.96
0.93
1.00
1.00
1.00
0.51
0.99
1.00
1.00
1.00
Table 6: Accuracy of basic mathematical computations across different models in our dataset.
A.3 EVALUATION TEMPLATES FOR DIFFERENT TASKS AND SCENARIOS
In Tables 14 and 13
we report the evaluation prompts used to evaluate the models in our work. In Table 16 we provide the templates used for constructing different scenarios RP.
B MORE DETAILS FOR RULE-BASED REASONING EVALUATION
B. IMPLEMENT DETAILS FOR MAIN EXPERIMENTS
For model version,
select Llama-2- 1-1bb-chatt--- Mata-LLlma-3-S8 IBB-Instu and cl --one--------t---------
All experiments are conducted on NVIDIA GeForce RTX 3090 GPUs, the sake simplicity, we include the prompts used in this work in the supplementary materials.
gpt-4-0613, 9pt-40-2624-05-1
B.2 MORE EXPERIMENTS ON OTHER MODELS
In this section, we repeat the experiments in 3.2 on the Llama3-8B model and Llama2- 13B model, the results are shown in Table 7 g
Here sct D 3, N = 5. We observe that the results in main text still hold
on these models.
not apply HR on them, since these two models have difficulty in evaluating the rule based on given templates under 0-shot settings.
LT
RP
CG
ST
Method
RI
FI
(A)
RI
EI
(A)
RI
EI
(A)
RI
FI
(A)
10 (0-shot)
0.21
0,54
0.33
0.03
0.48
0.45
0.17
0.22
0.05
0.07
0.45
0.38
10 (5-shot)
0.32
0.41
0.09
0.35
0.40
0.05
0.14
0.24
0.10
0.25
0.35
0.10
(0-shot)
0.10
0.24
0.14
0.12
0.30
0.18
0.10
0.17
0.07
0.01
0.35
0.34
(5-shot)
0.35
0.44
0.09
0.30
0.40
0.10
0.01
0.04
0.03
0.31
0.33
0.02
CoT (0-shot)
0.25
0.39
0.14
0.13
0.39
0.26
0.16
0.14
-0.02
0.16
0.42
0.26
CoT (5-shot)
0.54
0.59
0.05
0.36
0.40
0.04
0.41
0.55
0.14
0.47
0.19
SC (n=5)
0.47
0.54
0.07
0.35
0.37
0.02
0.45
0.55
0.10
0.51
0.62
0.11
SR (t=3)
0.16
0.26
0.10
0.09
0.27
0.18
0.08
0.20
0.12
0.05
0.39
0.34
Table 7: Performance of different methods MIRAGE using Llama3-8B (100 examples).
LT
RP
CG
ST
Method
RI
EI
(A)
RI
EI
(A)
RI
EI
(A)
RI
El
(A)
10 (U-shot)
0.01
0.29
0.28
0.01
0.41
0.40
0.01
0.15
0.15
0.18
0.40
0.22
10 (S-shot)
0.02
0.37
0.35
0.01
0.34
0.33
0.01
0.14
0.14
0.05
0.30
0.25
ID (0-shot)
0.02
0.02
0.00
0.03
0.03
0.00
0.06
0.06
0.17
0.20
0.02
ID (5-shot)
0.02
0.12
0.10
0.01
0.14
0.14
0.00
0.01
0.01
0.01
0.30
0.29
CoT (0-shot)
0.03
0.10
0.08
0.03
0.20
0.17
0.00
0.13
0.13
0.06
0.13
0.08
CoT (5-shot)
0.01
0.24
0.23
0.02
0.14
0.12
0.00
0.14
0.14
0.07
0.10
0.03
SC (n=5)
0.07
0.24
0.17
0.08
0.49
0.41
0.01
0.34
0.33
0.17
0.37
0.20
SR (1=3)
0.00
0.03
0.03
0.01
0.07
0.06
0.00
0.06
0.06
0.01
0.10
0.09
Table 8: Performance of different methods on MIRAGE using Llama2-133 (200 examples).
16

---

Published as a conference paper at ICLR 2025
B.3 EXPERIMENTS ON FINE-TUNING METHOD
the main text, we primarily use in-context learning to evaluate the performance of various models. Here, we supplement the evaluation with the performance of tueettund models on MIRAGE. Specifically, we use 3-sB-Inst ruct as the backbone, setting N = and D = 5, and train the model on 8,000 samples. For the training parameters, set the learning rate to 0,0001, the batch size to 1, and the number of epochs 10. Additionally, LoRA employed train people on different types of tasks. The performances 100 test examples are presented Table demonstrates that fine-tuning can effectively enhance the model's inductive reasoning capabilities. Compared to the 5-shot ICL, the model performs better on both reasoning tasks, even when not trained on tasks of the same format. However, consistent with the conclusion in $3.44 the results of the training do exhibit good formal generalization. The model tends perform relatively poorly on test sets with significantly different forms from the training set (c.g., LT and RP).
LT
RP
CG
ST
Method
RI
EI
RI
EI
RI
EI
RI
El
5-shot ICL
0.31
0.27
0.04
0.17
0.16
0.28
0.21
0.32
LT Training
0.89
0.82
0.22
0.24
0.34
0.80
0.26
0.38
RP Training
0.51
0)44
0.78
0.74
0.42
0.69
0.24
0.35
CG Training
0.76
0.75
0.22
0.25
0.86
0.80
0.26
0.37
ST Training
0.51
0.52
0.19
0.21
0.33
0.61
0.50
0.35
Table 9: Performance of the fine-tuned model on MIRAGE. highlighted in bold.
The best results in each column are
C MORE DETAILS FOR NEIGHBOR-BASED REASONING EVALUATION
C.1 PROOF CONTINUES FUNCTIONS
Here, we prove that the five basic vector operations MIRAGE are all continuous functions:
Theorem (Add Operation Continuity). Let (a), .2.... On R.. Define a mapping f R" R such that for a fixed index & (1,,2....,17 n and a fixed subset I C (1,2.....71) we have
f(A) (a).... ak-1; Σ IEI
as Oa+11--..- an);
where & & I. Then f is continuous functiion.
Proof. Consider two vectors A, B E B":
A (a1 a2, an), B (bi, .., bw.)
The mapping f replaces the k-th element of the vector with the sum of elements indexed by the subset I. Thus,
((A) (ai..... ak-1,
Σ
as ak+1,..., an).
EET
f(B) (bi,.... b
b bati,..., bn).
1E
The distance between the images of A and B under is
(If(A) /(B]]
Σ
(a,
b,) +
Σ a: Σ bx
j=Lj
Let us focus on the term involving the sums:
Σ
b,
(a: b2).
vel
17

---

Published as a conference paper at ICLR 2025
By the triangle inequality, we have
(a:
bi)
5
a; bil.
121
EE7
Therefore,
Σ
Σ
aa bil
FET
EET
ic
Using the Cauchy-Schwarz inequality, we gel
2
Σ
la bal
17| > (a, - bb)²
EE
where |/] is the cardinality of the set I. Therefore,
(IS(A) [(B)Il -
Σ
(a, b,)3 + + _
(a: 6.)2.
j=1JA
iel
This can be bounded as
IIf(A) f(B]] 3 C|| BB],
where constant depending on and |4]. Therefore, for any e >> choose
&. A B|| < 6, then
(A) /(BII C8= =6
Hence, f is continuous.
Theorem 2 (Copy Operation Continuity). Let A = (01, 02,, .... .0n) E RR. Define mapping R + R such that for fixed indices J C (1,2,..., n. and a fixed index E {1, (1, n3, have
S(A) (b,, b2. ......... bu.)
where
b. ak ffE
otherwise.
Then f is continuous function.
Proof. Consider two vectors A, B R":
(a) d2.......,,., (n).
B = (b1, ba....,b...
The mapping f replaces each element of at the positions indexed by with the value the element at index k. Specifically,
S(A) = e2, (c.)))
where
G au if
otherwise.
Similarly,
f(B) = (d1, dn),
where
bu if J, di bi otherwise.
The distance between the images of A and B under ff is given by
[[(A) - S(B]] =
[( (a - d.)².
18

---

Published as a conference paper at ICLR 2025
By the definition of f. have Therefore, the distance can be rewritten
a# by
if E J, otherwise.
d:
b.
(I((A) - [BBII
Σ VEU
ax
bu)a + Σ (a: b.)2
Since the sum over J/ has |J| terms that are all equal to (az - be)² this simplifies
- S(B)I = |luika== bx)2 + E(ar (as b.)2 -
This can bounded
(I((A) - [[B)| - C - BlI.
where C' is constant depending and |J].
Therefore, for any choose 6 &. If A B|| < 6, then
(A) [(B)]] C8 =c.
Hence, f is continuous.
Theorem 3 (Map Operation Continuity). Let = (a1, 02 ... an Define a mapping f such that for fixed indices J S (1,2...., n2 and fixed scalars k b E B, we have
R"
[(A) (b1,b,... bm).
where
b; kai +b if
otherwise.
Then f continuous function.
Proof. Consider two vectors A, E R
A (a1: 42, an): B (b1, b2.................. bn).
The mapping f applics the linear transformation Ar + b the elements of indexed by J and leaves the other elements unchanged:
f(A) = (CI, Ca),
where
kai + ifi J, otherwise.
C =
Similarly,
f(B) (di,dd....,da) dn),
where
kby
+b if
d,
b otherwise.
The distance between the images of A and B under ff is given by
[[(A)- - S(B)]I
E(a d.)2.
By the definition of f, have
k(as b.) if a: h
C: d;
otherwise.
19

---

Published as a conference paper at ICLR 2025
Therefore, the distance be rewritten
(I(A)
/(B)II =
Σ
(k(a:
b:])2 + D
(a: 6.)2.
iEJ
This simplifies to
II((A) f(B)II
Σ
(as bi)² + Σ (ai - 6.)
SCT
iEJ
Let C = max(1, (e].. Then
I[((A) - S(B)II CC
(a. b.)² = CIIA- - Bl|.
Therefore, for any 6 choose g. If A B|| < 6, then
(A( [(B)I =8
Hence, f is continuous.
Theorem 4 (Pad Operation Continuity). Let A (01, ....... anl E R". Define mapping f : RR" + R" such that for fixed subset J S .1, and fixed constant
have
f(A) (b1, bu.................. ..,, bw).
where
c if J. 04 otherwise.
b
Then f is a continuous function.
Proof. Consider vectors A, B R":
A (a) 02....;; an), B (b1, b2,..., bn).
The mapping replaces each element of A the positions indexed by with the constant and leaves the other elements unchanged:
S(A) = (ci, cz. cn))
where
fc if
G a {a as otherwise.
Similarly,
S(B) = (di, .a.,...,,,,, dn),
where
C if d. b. otherwise.
The distance between the images of A and B under f is given by
((A)-f(BB)) [( d.)².
By the definition of f. we have
i J, ai b, otherwise.
C4 d,
Therefore, the distance can he rewritten as
(I((A) /(B]|
Σ as bz)
20

---

Published as a conference paper at ICLR 2025
Note that the is only the indices not in J. This is because the elements replaced by the constant C, and thus their difference zero.
Since
I[(A) M(B]I & |A - -
for any e 0, choose 5 = e. A B|| < 6, then
(IS(A) [(B)Il <
Therefore, f is continuous.
Theorem 5 (Swap Operation Continuity). Let A (a1, 02 18..... (n.. E R". Define mapping f R RR" such that for fixed disjoint subsets 1,J S (1.2.....) n with 1 [nJ] indexed by are swapped with the elements indexed by Then f is a continuous fonciuoossuunciion
and [1] [J], the elements of A
Proef. Let A (a1, a2.............................. ................................ an)) E IR", and let I = (i,, i2. im} and J [i1, f2. pm be two Define the mapping f such that swaps the elements of A
disjoint subsets of indices with |1| = |J] indexed by and Specifically, for any A. mapping produces a vector f (A) given by
ajz-
if for some r 1, 1,,,...,7. if j. some 1,2,...,,, otherwise.
by
ai,
ak
Consider two vectors A, C
A = (01, 02 ................................ on),
C= (c.,.)
Applying the mapping f to both vectors, obtain
f(A) = (bı, b2,..., bu), f(C) = (di, d2, dn,))
dn);
where
Agr
if for 1, if j. for some = 1....... .....
ai,
ak
otherwise,
and similarly,
Cjr
if i, for some 1,2, ,. if j- for some = 1.2,...
du =
Ck
otherwise.
The distance between f(A) and f(C) is given by
(I(A) - ((C) Σ (bu - d.)
Since only swaps the elements indexed by and J, we have
aj. Cjr
if h i, for some
by dr
Cap
if = j. for some otherwise.
ak Ck
Therefore, the norm becomes
(I((A) f(C)]l
Σ
(a,, Cz2 )2
(a,, 2 Σ KETUJ
(ax cx)2
Rearranging the terms, we have
- f(C)]
(as cx)² -A-C) ClJ.
Therefore, for 0, choose 8 = c. If |
C|| < 8, then
[[(A) f(C)] A ClI <c
Hence, ff is continuous.
21

---

Published as a conference paper at ICLR 2025
C.2 COMPARISON WITH OTHER DISTANCE METRIC
aim explore whether using different distance metrics to define neighbor facts would also influence the model's inductive reasoning. Therefore, we additionally introduce three other distance metrics: Euclidean distance dene, Manhattan distance dman, and Minkowski distance dmin- Like Equation 6 we have:
denc (Xs, Fias
Fiik xo6)2
(11)
dman (Ths 2s))
Σ
Iisk 3ekc
(12)
k=1
dmin (Ki, Dss)
Σ
Nek
(13)
where we set p = 3. We can generate three distinct new neighborhoods (xx <) by incorporating these distances into Equation 7 thereby constructing three new kinds of OF. Therefore, we compare the model's performance El tasks when using only these different OFs, and the results shown in Figure g From the that neighhorhood construction outperforms those constructed using other distance metries. EI performance of other three OFs across different radií is similar that removing neighbor facts constructed using these methods does not influence the model's 's inductive reason- ing ability. In contrast, our constructed OF leads to significant decline in accuracy, proving the validity of our neighborhood construction.
figure,
baseline, indicating
0.8
0.7
Man
Man
Mic
0.7
Min:
0.6
Our
Our
0.6
Accuracy
0.5
Accurcy
0.4
0.4
0.3
0.3
0.2
0.1
0.1
0.0
0.0
2
Radius
Radius
(a) GPT-4o
(b) Claude-3.5
Figure 8: Performance comparison of the impact of different OFs. The dashed line represents the baseline accuracy using default fact sets. Euc represents Euclidean distance, Man represents Manhattan distance and Min represents Minkowski distance.
C.3
MORE EXPERIMENTS ON OTHER MODELS
8BB The results are shown in Table We repeat the experiments in & 4.3 on Llama2- 13B, Claude-3.5 and Llama3-8B.
T0 Ii [2] Besides, we also repeat the experiments in 8 4.4 on Claude-3.5 and report the results in Figure 9 The results of all these additional experiments are consistent with those in the main text.
C.4 SUPPLEMENTARY EXPERIMENT FOR MAIN EXPERIMENT
We observe that, in the experiment of 222 though the performance of OF significantly decreases compared the baseline,
models still able maintain around 40% accuracy, facts. infer that models are likely conduct based reasoning in these cases. Hence, we design extra experiment for supplementary. ID in (3.2) on these cases in Table 13 From the table, we can observe that the model's deductive accuracy using the rule exceeds 70% when there are fewer neighbor facts in the context. This demonstrates that the rule-hased induction if there is less neighbor-based matching.
with only distant observed
it, we prompt LLMs to induct rules and finish example inference tasks (i.e.
model tends rely
22

---

Published as a conference paper at ICLR 2025
N=3
N=5
N=8
Type
LT
RP
CG
ST
LT
RP
CG
ST
LT
RP
CG
ST
Baseline
0.18
0.05
0.14
0.30
0.15
0.05
0.17
0.23
0.00
0.17
0.31
IF Only
0.43
0.14
0.35
0.34
0.48
0.03
0.49
0.36
0.46
0.02
0.52
0.36
CF Only
0.17
0.06
0.14
0.28
0.15
0.04
0.22
0.25
0.14
0.00
0.17
0.31
OF Only
0.16
0.04
0.13
0.27
0.09
0.02
0.09
0.26
0.10
0.01
0.10
0.22
Table 10: Performance of different fact types under various settings on Llama2-13B (D
N=3
N=5
N=8
Type
LT
RP
CG
ST
LT
RP
CG
ST
LT
RP
CG
ST
Baseline
0.49
0.23
0.49
0.38
0.68
0.39
0.80
0,53
0.78
0,51
0.81
0.56
IF Only
0.76
0.42
0.84
0.61
0.90
0.46
0.87
0.61
0.89
0.62
0.93
0.76
CF Only
0.54
0.23
0.59
0.36
0.67
0.36
0.81
0.56
0.76
0.42
0.85
0.53
OF Only
0.50
0.24
0.49
0.45
0.60
0.30
0.71
0.39
0.66
0.26
0.77
0.49
Table 11: Performance of different fact types under various settings on Claude-3.5 (D= 5).
N=3
N=5
N=8
Type
LT
RP
CG
ST
RP
ST
RP
CG
ST
Baseline
0.19
0.12
0.26
0.29
0.27
0.17
0.28
0.32
0.24
0.26
0.37
0.20
IF Only
0.55
0.22
0.55
0.38
0.65
0.40
0.65
0.46
0.68
0.41
0.75
0.41
CF Only
0.21
0.07
0.29
0.29
0.27
0.14
0.31
0.31
0.22
0.39
0.19
OF Only
0.24
0.06
0.34
0.26
0.23
0.08
0.24
0.25
0.18
0.11
0.23
0.15
Table 12:
Performance of different fact types under various settings on Llama3-8B (D 5).
0.9
IF
IF
Deductive Density
0.8
Deductive Densily
0.8
CF
OF
OF
0.7
0.7
0.6
0.6
0.5
3
inf
7
(b) 1 inf
Figure 9: Deductive Density of various fact types on Claude-3.5 under different test radius and neighborhood radius (D = N 5).
Model
e=1
c=2
e=3
Avg
GPT-4o
0.74
0.72
0.76
0.74
Claude-3.5
0.76
0.72
0.73
0.74
Table 13: Performance on the correct case of OF. We use the 0-shot setting and vary the radius
23

---

Published as a conference paper at ICLR 2025
Scenario
Prompt
Please summarize the rules of the list transformation based the given facts.
Your reply should strictly follow the following format:
Rule: [A, B, C] +
[<<expression>>, <<<xpression>>>, <<expression>>>>
LT
Fact Input: {INPUT)
Output: {OUTPUT)
Fact Input: (INPUT) Output: (OUTPUT)
Please generate the rule of list transformation based on the former facts.
Please summarize the rules of the (TASS_TYPE) based on the given facts.
Your reply should strictly follow the following format:
Rule: If there are A {OBJ1). B (OBI2), (OB13). After the (TASK.TYPE).
there are <<expression>>> (OBJI), <<expression>>>> (OBJ2), <<expression>>> (OBJ3).
Fact I: Input: (INPUT) Output: (OUTPUT)
Fact n: Input: (INPUT) Output: (OUTPUT)
Please generate the rule of (TASK_TYPE) based on former facts.
Please summarize the rules of the function based on the given facts.
Your reply should strictly follow the following format:
Rule:
def f(A, B, C):
A, B, C= <<expression>>> <<expression>>> <<expression>>>
CG
return A, B.
Fact Input: (INPUT)
Output: (OUTPUT)
Faet Input: (INPUT) Output: (OUTPUT)
Please generate the rule of function based on the former facts.
Please summarize the rules of the string transformation based on the given facts.
Your reply should strictly follow the following format:
Rule: ABC +
ST
Fact Input: (INPUT)
Output: (OUTPUT)
Fact Input: (INPUT) Output: (OUTPUT)
Please generate the rule of string transformation based on the former facts.
Table 14: Prompts for rule induction tasks (D = 3).
24

---

Published as a conference paper at ICLR 2025
Scenario
Prompt
Please answer the question based on rules of the list transformation in the given facts.
Your reply should strictly follow the following format:
Answer: <<<xxpessioo>>>
<<expression>>> <<expressio>>>>
LT
Fact 1: Input: (INPUT) Output: (OUTPUT)
Fact n: Input: (INPUT) Output: {OUTPUT)
Question: Input: (TEST INPUT)
Please answer the question based on rules of the (TASK_TYPE) in the given facts.
Your reply should strictly follow the following format:
Answer: <<expression>>> (OBJI), <<expression>>>> (OBJ2), <<expression>>> (OBJ3).
RP
Fact I: Input: (INPUT) Output: (OUTPUT)
Fact n: Input: (INPUT) Output: (OUTPUT)
Question: Input: (TEST_INPUT)
Please answer the question based on rules of functioon in the given facts.
Your reply should strictly follow the following format:
Answer: <<expression>>>
<<cxpression>>, <<expression>>>
CG
Fact I: Input: (INPUT)
Output: (OUTPUT)
Fact n: Input: (INPUT)
Output: (OUTPUT)
Question: Input: (TEST INPUT)
Please answer question on rules of string transformation the given facts.
Your reply should strictly follow the following format:
Answer:
Fact 1: Input: (INPUT)
Output: (OUTPUT)
Fact n: Input: (INPUT) Output: (OUTPUT)
Question: Input: (TEST INPUT)
Table 15: Prompts for example inference tasks (D = 3).
25

---

Published as a conference paper at ICLR 2025
Scenario
Template
Objects
[NAME] went to the market to trade items based on the rule.
Trade
He originally had {obj (xppexxpression)
chairs, tables, pens
After the trade, he had (obj expression]
[NAME} adjusted his dict plan according to the expert's advice.
Diet
He originally planned to take in [obj expression)
apples. bananas, oranges
After the adjustment, he had expressir)ss
[NAME) was performing card magic trick.
Magic
Initially, he had {obj expression]
Spade 5s, Jokers, Hearts 6s
After performing the magic, he ended up with {obj expression)
(NAME} adjusted the investment amount of cach asset based on criteria.
Invest
Initially, he invested {obj expression]
stocks, bonds, funds
After the adjustment, he invested {obj obj_exxprsssoon
(NAME) adjusted the students' courses according to certain rules.
Course
Initially, the weekly course schedule (obj.expression)
math, science, history
After the adjustment, the weekly course schedule was: (ohj_expression)
Table 16: Prompts for real-world problems construction.
26


Reviews:
[{"review_id": "x69zRbdQfT", "reviewer": "Reviewer_6pHZ", "summary": "This paper evaluates and analyzes LLM's ability for inductive reasoning. Compared to previous works in inductive reasoning, it adopts a new setting, which is to compare the performance of LLMs under two settings: (1) specific --> general --> specific, and (2) specific --> specific.\nIt is interesting to see that the performance of LLMs under the two settings is comparable. The result indicates that setting (2) will have a better performance, even though traditionally we think a general rule is important to make further inferences to unseen specific examples.\n\nAs for results, the main findings of this paper are that \n(1) setting (2) outperforms setting (1);\n(2) many techniques such as self-consistency can't improve performance in the proposed task; and \n(3) in-context demonstrations that are closer to the input can help more in terms of the accuracy of output.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The main finding is insightful, that setting (2) outperforms setting (1), which can bring in more discussions on how LLMs perform inductive reasoning.\n2. The analysis in terms of neighbor-based reasoning, especially the investigation into the scope is also interesting. However, it might not be surprising, since the similarity of in-context demonstrations has been discussed extensively in previous papers.", "weaknesses": "1. The writing is not clear. Specifically,\na). what does the \"substitution\" in line 414 and line \"443\" mean?\nb). after finding IF, CF, and OF, how are they used?\nc). what is the metric in Table 1?\n\n2. The claim \"We prove that LLM is a poor rule-based inductive reasoner\" is too strong and unacceptable. What is the definition of \"poor\"? And have the authors proved it mathematically? \n\n3. \"We prove that LLM is a neighbor-based inductive reasoner.\": Have the authors proved it mathematically? When LLM is a neighbor-based inductive reasoner?\n\n4. Although the performance that setting (2) outperforms setting (1) is provided. There might lack of a more in-depth analysis of when LLMs use neighbor-based reasoning, and when LLMs use the general pattern thinking? Or is it that LLMs can purely do neighbor-based reasoning, and never use the general pattern thinking at all? It seems that although each experiment is interesting, but I don't know what is the takeaway knowledge from this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VbmU9T5134", "reviewer": "Reviewer_DD2X", "summary": "This paper presents a new dataset for evaluating the induction and deduction ability of LLMs. The authors provide a very detailed explanation of the generation method of the dataset and make extensive experiments to evaluate current state-of-the-art models. The authors also propose two hypotheses about LLMs' reasoning behaviours, and the experiments are designed to verify these hypotheses. Detailed explanations and discussion are provided, and the supplementary material also gives good support to the conclusion.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper covered a broad range of related works and provided another valuable benchmark for evaluating LLMs' reasoning power. The tasks has been formulated in four representations to ensure the diversity.\n- The authors make two interesting hypotheses on LLM's behaviour, and carefully designed several experiments to verify them.\n- In order to evaluate LLMs' behaviours, the authors designed several metrics specifically, most of which make sense to me. Detailed discussions are provided, and all the charts and tables are explained well.", "weaknesses": "- First of all, all the experiments are behavioural studies and empirical evaluations, and there is no mathematical evidence about LLM's induction/deduction power. IMHO, if there is no mathematical proof, then claims such as \"we prove that LLMs are poor in inductive reasoning\" would be too strong.\n- It is fine to evaluate the models' induction ability by using the designed tasks. However, I wouldn't call the other task as \"deduction\" since the rules are not provided. From the definition from logic, a deductive inference needs both input facts and the logic rules for deducing the outputs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cqTFtQ1IQR", "reviewer": "Reviewer_ctTK", "summary": "This paper investigates the inductive reasoning abilities of LLMs, introducing Mirage, a synthetic dataset designed to comprehensively evaluate LLMs' inductive and deductive reasoning from various perspectives. Using Mirage, the authors demonstrate that LLMs are weak at rule-based reasoning but perform well with neighbor-based reasoning. These findings suggest that LLMs' deductive reasoning can be enhanced by leveraging similar examples to support a robust inductive reasoning process.", "questions": "Questions:\n- Could the authors offer additional insights or empirical evidence on the model's performance with real-world datasets or in less controlled environments?\n- Additionally, could the authors also check if fine-tuning over the synthetic data helps the inductive reasoning of LLMs?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n- Present a synthetic data generation framework to evaluate inductive and deductive reasoning processes. The authors provide clear steps detailing the data generation process, specifically outlining the rule generation and question generation procedures.\n- Various advanced prompting techniques, including SR and HR, along with the latest LLMs, are tested in the experiments.\n- The effective scope analysis for neighbor reasoning supports the claim that LLMs are effective neighbor-based reasoners.", "weaknesses": "Weaknesses:\n- In the section on neighbor reasoning, the authors use the term \"Feature Space\" to denote the space of observed facts and test samples. However, it’s unclear where the features in this space originate.\n- The data setting may lack sufficient complexity to effectively demonstrate inductive reasoning capabilities in real-life scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eYMDjHmjNc", "reviewer": "Reviewer_cTBS", "summary": "Authors created a synthetic dataset MIRAGE to evaluate the reasoning ability of LLMs, with two conclusions: (1) LLMs are not good at deductive reasoning, (2) LLMs are good at performing inference based on neighborhood information, thus, they are \"neighbor-based inductive reasoner\". \n\nAuthors try to distinguish the ability of deductive reasoning from the ability of inductive reasoning. The missing experiment(s) is/are to distinguish the ability of reasoning from the ability of pattern matching. For example, it is necessary to show the so-called \"neighbor-based inductive reasoner\" is the ability of reasoning, instead of the ability of pattern-matching.", "questions": "1. why do you only mention inductive reasoning in the title? you also report experiments on deductive reasoning of LLMs. \n\n2. do you assume any causal relations between the input and the output of rules, or just associative relations? If only associative relations, how do you distinguish deductive from inductive reasoning?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "A new synthetic dataset is created and contributed to the reasoning evaluation of LLMs. \nAuthors conclude that LLMs are poor in rule-based reasoning, and good at neighbor-based reasoning.", "weaknesses": "Authors coined the nice term \"neighbor-based reasoning\". But, is it reasoning or just pattern-matching? I am afraid there is no term of \"neighbor-based inductive reasoning\" in the literature of logic or psychology. \n\nIt is not surprising that LLMs are good at \"predicting\" based on neighborhood context, as this is one of the important methods to train them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)