HypoBootstrap: A Bootstrapping Framework for Inductive Reasoning
Authors: Si Chen
Yifei Li
Richong Zhang
Venue: NeurIPS 2025 poster ()
Keywords: inductive reasoning
inductive logic
hypothesis generation
hypothesis confirmation
Primary area: deep_learning

Abstract: Inductive reasoning infers general rules from observed evidence, which is one of the most critical intelligence abilities.
Previous works have succeeded in formal languages but suffer from onerous and error-prone conversions between a particular formal language and the working language.
As large language models (LLMs) have emerged, direct reasoning with various kinds of languages, especially natural languages, without formal language involvement has become feasible.
However, existing LLM-based inductive reasoning usually relies on LLM's intrinsic generation ability, which is prone to LLM's hallucination and lacks systematic guidance according to the nature of inductive reasoning.
To this end, we propose HypoBootstrap, an integrated framework for inductive reasoning that generates and confirms hypotheses both in a bootstrapping manner.
Regarding hypothesis generation, we propose a novel bootstrapping generation strategy, bootstrapping object hypotheses, relational hypotheses, and functional hypotheses successively, which assists LLM in observing the evidence from trivial patterns to non-trivial patterns.
Regarding hypothesis confirmation, we utilize Glymour's theory of bootstrap confirmation, a hypothesis confirmation theory from the philosophy of science that can confirm a set of hypotheses.
We use its principles to confirm the object hypotheses, relational hypotheses, and functional hypotheses.
Empirical studies on four inductive reasoning scenarios of different natures, involving causal induction, concept learning, grammar learning, and abstract reasoning, demonstrate that HypoBootstrap significantly outperforms existing methods.

HypoBootstrap: A Bootstrapping Framework for Inductive Reasoning
Chen', Yifei Li", Richong Zhang"
SKLCCSE, Beihang University, Beijing, China
chen. si@buaa. edu. en, 1yf08620buaa. edu. en, zhangrc@act. buaa, edu, en
Abstract
Inductive reasoning infers general rules from observed evidence, which is one of the most critical intelligence abilities. Previous works have succeeded in formal languages but suffer from onerous and error-prone conversions between particular formal language and the working language. As large language models (LLMs) have emerged, direct reasoning with various kinds of languages, especially natural languages, without formal language involvement has become feasible. However, existing LLM-based inductive reasoning usually relies on LLM's intrinsic genera- tion ability, which is prone to LLM's s hallucination and lacks systematic guidance according the nature of inductive reasoning. To this end, propose HypoBoot- strap, an integrated framework for inductive reasoning that generates and confirms hypotheses both in bootstrapping manner. Regarding hypothesis generation, we propose novel bootstrapping generation strategy, bootstrapping object hypotheses, relational hypotheses, and functional hypotheses successively, which assists LLM in observing the evidence from trivial patterns to non-trivial patterns. Regarding hypothesis confirmation, we utilize Glymour's theory of bootstrap confirmation, a hypothesis confirmation theory from the philosophy of science that can confirm a set of hypotheses. We use its principles to confirm tional hypotheses, and functional hypotheses. Empirical studies on four inductive reasoning scenarios of different natures, involving causal induction, concept learn- ing, grammar learning, and abstract reasoning, demonstrate that HypoBootstrap significantly outperforms existing methods.
object hypotheses, rela-
1 Introduction
Inductive reasoning emblems high-level human intelligence [8
While deductive reasoning
is critically required ability for intelligence, inductive reasoning is more significant, by which means humans infer general rules from observed evidence. These rules are the basis of the soundness of deductive reasoning. Figure D iilluttrates an inductive reasoning task. In contrast to deductive reasoning, where we reason over given rules, inductive reasoning has no rules follow and requires creative thinking to generate various possible hypotheses.
In the past few decades, Inductive Logic Programming (ILP) was the major research subject for inductive reasoning in the Al community. An ILP system can learn symbolic rules from symbolic evidence, typically expressed in first-order language. An example hypothesis in first-order language is depicted in Figure I While ILP can efficiently learn interpretable rules in formal language, it mandatorily requires the formalization of evidence, preventing it from inductive reasoning over more common natural-language evidence. Besides, since II. cannot produce natural-languanl rules, translating formal-land language rules into natural language is another tremendous burden.
I Equal contribution
Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Evidence
x: [2,3,6,4] y: [6]
Inductive
Reasoning
Natural-Language Hypothesis
output
the input.
len(y): deriving the length of
Implicidy
Explicitly
contain(x,y) contains
Stated
elen(x; 1): deriving the i-th element of
Formal-Language Hypothesis
1s is_list(y) lon(y)=1 A contain(x,y) A elen(x, 30alooyy)
equality first-order logic
No
Hi
Figure I: Comparison of natural-language hypothesis with formal- lormaa-language hypothesis given the same evidence. Natural-lanuuag hypotheses are typically imprecise and implicitly imply some statements, which are usually considered trivial and redundant in daily conversation. In contrast, formal-language hypotheses are precise as possible;
trivial things should be expressed explicitly. HypoBootstrap first generates the trivial hypotheses (i.e., object hypothesis Ho and relational hypothesis HR), which are much casier to observe, and final hypotheses (i.e., functional hypothesis F) are generated based on these hypotheses.
With the rise of large language models (LLMss 28), direct reasoning in non-formal languages becomes feasible, especially in natural languages, which avoid the troublesome mutual conversion between non-formal language and formal language. Existing works reveal some degree of ability to propose natural-language hypotheses via LLM's powerful generation ability [20] 25]. However, due to the unavoidable hallucination, LLMs usually unconsciously propose fallacious hypotheses, losing the
rigorous nature of formal-language reasoning.
To alleviate LLM's hallucination and provide systematic guidance for inductive reasoning, we propose HypoBootstrap, novel agent framework to boost LLM's inductive reasoning performance. Hypo- Bootstrap enhances inductive reasoning from two tightly interacting aspects. hypothesis generation (i.e., hypothesis proposing) and hypothesis confirmation, the two indispensable aspects of inductive reasoning. On the one hand, HypoBootstrap enhances hypothesis generation by bootstrapping hy- potheses from trivial to non-trivial. The trivial hypotheses, corresponding to the implicitly stated hypotheses of Figure [ (pink), are easy to observe and help generate more non-trivial hypotheses, corresponding the explicitly stated hypotheses of Figure (blue). Specifically, inspired by first- order languages, HypoBootstrap successively generates object hypotheses, relational hypotheses, and functional hypotheses, bootstrapping the latter from the former. On the other hand, HypoBootstrap confirms hypotheses based on Glymour's theory of bootstrap confirmation [6. a widely-use confir- mation theory from the philosophy of science that is advantageous to confirm set of hypotheses. HypoBootstrap adopts this confirmation method to confirm the set of object hypotheses, relational hypotheses, and functional hypotheses. Implementing such confirmation theory originated from first-order languages via LLM agents is not direct; hence, we deliberately design a confirmation procedure and simultaneously integrate it into bootstrapping generation procedure, which can alleviate the LLM's hallucination during hypothesis generation.
We conduct experiments four inductive reasoning scenarios with varying nature: causal induction. concept learning, grammar learning, and abstract reasoning. Empirical results demonstrate significant improvement of HypoBootstrap compared with previous works and verify that both the bootstrap generation and bootstrap confirmation are effective. In addition, HypoBootstrap can be further improved via human priors, demonstrating its broader potential in inductive reasoning. The main limitation of HypoBootstrap is the high inference cost. In practice, the trade-off between performance and cost should be considered seriously.
Related work
2.1 Inductive reasoning via language models
Language model- based inductive reasoning enables inferring hypotheses in various languages, espe- cially natural languages, and does not require formalizing evidence. Existing research is divided into
"Code available at /ttttss///ithub http:///ggttttcccoooo/oo/a//t/tt

---

two lines. One line requires parameter fine-tuning [24], which is costly for LLMs, even impossible for clored-source but powerful models. The fine-tuning may even require annotated rules as training data [22 26]. which is costly, error-prone, and intractable for broader applications. Another line does not rely on fine-tuning, where language models are directly prompted to infer rule hypotheses from evidence, as in Hypothesis Refinement [20]]. The hypotheses are confirmed via programs, which are translated from natural-language hypotheses via LLMs. The hypotheses can be refined by iteratively generating with error feedback. Hypothesis Search [25] attempts to inject human priors into prompts, which can further stimulate LLMs' inductive reasoning ability. MoC [3] prompts LLM first generate relevant concepts for each inductive reasoning task, and then generate hypotheses according to these concepts, which enriches the diversity of generated hypotheses. However, none of these works explicitly alleviates hallucinations during hypothesis generation. Our work falls into the second line and alleviates LLM's hallucination in inductive reasoning. Further, our proposed framework considers the nature of inductive reasoning and provides systematic guidance for LLMss.
2.2 Confirmation theory
Confirmation theory studies the logic that a hypothesis may be confirmed or disconfirmed by given evidence. Various confirmation theories are used in the philosophy of science, such as the Hempelian positive-instance account confirmation, the hypotheticoo-deductiee method, and the probabilistic method [17]. Glymour's theory of bootstrap confirmation [6] is the one that is, implicitly explicitly, often used in various significant scientific breakthroughs, such as Newton's 's universal gravitation, general theory of relativity [6), and quantum theory [SS [6]. We adopt Glymour's bootstrap confirmation since it is designed and appropriate for scenarios where multiple hypotheses should be confirmed together.
3 Preliminary
3.1 Inductive reasoning
Following previous works (20 [25 consider practical induction problem of inferring mapping rule from an input space X' to an output space y. All the elements of the spaces are assumed to be represented in some language, and all the inputs and outputs passed to LLMS refer to their language representation. To have coherent notations throughout the paper, we deliberately express this problem in first-order logic. D Specifically, for each task, we have observed evidence, consistent set of binary ground atoms E (f)))))))))))))))))))))))))))[ = ..... (.).))(((())
}. where z(1), ,...,.""" x(() E X, y!)).... ..... ( y, f is the mapping function to be inferred, and = denotes the binary predicate equality. The goal is to infer a rule s.t. Va E X,y y E y,
H f(x)) =y
if f(x) is equal to y, f(r not equal to y-
H -f
For instance, given evidence (f([4.2,3, 1)) = 1,, 4]}. where input and output are both lists of numbers, we may infer H as is the sorted list of in ascending order. The evaluation of a rule is conducted on unobserved evidence, a separate set of binary ground atoms generated from the same mapping f. The hold-out evidence ensures that itself would not be an acceptable rule and good inductive reasoner should have inferred the underlying mapping behind the observed evidence.
The hypotheses and evidence involved throughout the paper are all supposed to be expressible in a first-order language, though they never explicitly converted into it; i.e., we conduct induction directly in various working languages that modern LLMs support, such as natural language. This assumption is acceptable since knowledge has been, to some extent, successfully represented by first-order languages in AI research, such as knowledge graph (9] and logic programming [14]. We also give examples of translating our experimental data into first-order language (Appendix D
3.2 Glymour's theory of bootstrap confirmation
While Glymour's 's original work 66 leaves some space for interpretation, several formalizations exist.
We adopt the formalization from 4 to conduct rigorous discussion.
We assume basic knowledge about first-order logic.
3

---

Definition 3.1 (Glymour's Bootstrap Confirmation (simplified version)). Given set of hypotheses
H [....... Hm the evidence E' bootstrap-confimm H if
1. HUEX1
(consistency)
2. for each i E m.) there is a H' & s.t. s.t. H, & H' and E. confirms H; with respect to H'
(confirmation)
3. there is possible, but not observed, evidence E' s.t. E' disconfirms H
(non-triviality)
See Appendix A for more details. In addition, Definition 3.1 requires auxiliary (non-bootstrap) confirmation & disconfirmation method. Glymour's theory tends to be neutral across any auxiliary confirmation & disconfirmation. In practice, hypothetico-deductive (HD) confirmation & disconfir- mation is one of the common choices, which we defined Definition 3.2 & 3.3 following [7]. The exact auxiliary confirmation & disconfirmation HypoBootstrap uses is imp emplemtttiioon feasible by LLM and will be illustrated in Section 4.2.2 & 4.2.3
Definition 3.2 Cyypothettco-Deduccin Confirmation), Given hypothesis H, the evidence E confirms H with respect to H' if
H U H' 1 (consistency)
2. HHHH" +
(entailment)
3. H' y E (necessity)
Definition 3.3 Plypotheticoo-eeductie Disconfirmation). Given hypothesis H, the evidence E hypothetico-deduutivvey disconfirms H if
1. H. (consistency)
2. H. + --E (entailment)
4 HypoBootstrap
This section presents HypoBootstrap, an inductive reasoning framework that utilizes LLM agents to implement hypothesis bootstrap generation and Glymour's bootstrap confirmation. The pseudo-code is given in Appendix [BI]] As depicted in Figure 2 the framework generates the object hypothesis, relational hypothesis, and functional hypothesis in bootstrapping manner, with confirmations embedded into the generation procedure. If not confirmed, the framework iteratively regenerates functional hypothesis according to error feedback from the last iteration. See Appendix BB3 for implementation details of the agents in Figure 2
4.1 Bootstrap generation
Inspired by first-order languages that mainly deal with objects (i.e., constants), relations (i.e., pred- icates), and functions, the bootstrap generation is divided into three steps: generating the object hypothesis Ho, the relational hypothesis HR. and the functional hypothesis Hp. Intuitively, the difficulties of generating these three hypotheses increase in order. While object hypothesis and relational hypothesis correspond
the implicit and trivial statements in Figure the functional hypotheses correspond more to the explicit and non-trivial statements.
4.1.1 Object hypothesis generation
In the first step, we generate hypotheses that describe the patterns of each object in the evidence (i.e., input or output). Such hypotheses may be relatively simple and specific to particular object, but they are the starting points of bootstrap generation. Specifically, for each input zx() (resp., output ()),
saseerateenrtoooo identifies a list of object patterns H (resp., H(
ielliimioooooooooommm is used to eliminate those patterns inconsistent with its relevant object from the pattern list. The filtered object hypothesis of H (resp., ()))) is denoted by ( (resp., ( The object hypothesis Ho contains all these filtered object patterns. Formally, Ho e N H!! A (())) In addition to the LLM's hallucination, another benefit of eliminating inconsistency ensuring Ho U E is consistent as possible. If each (()) or ((()) is consistent (with its corresponding object), Ho E will be consistent, since ( (1(0) are object-level (i.e., cannot contradict with E, which is function-level) and object-dependent (i.e., cannot contradict each other). The consistency of Ho contributes the consistency clause of Definition 3.1 detailed in Section 4.2.1
Since L.LM may generate fallacious patterns, Ob ObjectHlyootheise hec.lllppptesssssscossssssseteee

---

Hypothesis Generation
Hypothesis Confirmation
Ag ,x0 y
Evidence [f(x = y0u ((t00)) you)
Object Hypothesis Generator
Object
Object Hypothesis
Hypathesis
Inconsistency Eliminator
Hn
R
A
A
ycn
m .yo
Bootstrapping
X0x, o Mg
N1D Relational Hypothes1s Generator
Relational Hypothesis Inconsistency Elimtinator
Relational Hypothesis
yl My
A00 x/A) yfa)
Hg
ao M/ (. M
F Functional Hypothesis Generator
Auxiliary Confirmer
Functional Hypothesis
Hr
Hy
oor y0u, 2 M
Hp
Feedback
Consisteney Validator
Confirmed Hypothesis
Mg.
N (Mo.
Figure 2: Overview of HypoBootstrap.. The orange rectangles represent agents. Figure 3 and
Appendix C.2 for concrete examples.
4.1.2 Relational hypothesis generation
In the second step, generate hypotheses for each input-output pair, which describe its relational patterns, by bootstrapping from the hypotheses for each object. Specifically, for each pair (x) ( delatiionallypotheesseeee
identifies a list of patterns H between z(9) and y based on H .1,5) (1) (0) For example, the input and oulput both number matrices of 5 columns and rows. At this step. the hypotheses are closer to functional hypotheses generated at the next step, since they already involve the relation between objects and the relational hypotheses are still specific to particular anput-output pair and may be unable to form function, i.e., is probably impossible determine the unique output for certain input.
merely object-specific. However,
The relational hypothesis Relationsnl
HR contains the relational patterns Hay filtered by Formally, HR A N H. Simi- larly, if each ( is consistent (with its corresponding input-output pair), Hn U E will be consistent, since H are pair-dependent (i.e., cannot contradict each other). Besides, we assume (Ho, Hn) is consistent enough, so is (Ho, HR, , E} (justified Appendix B.2).
4.1.3 Functional hypothesis generation
In the third step, bootstrapping from the object and relational hypotheses, we generate functional hypotheses for all input-output pairs, which describe the general transformation rule from input to output. Specifically, Functi oowaltypotheesseeeeee based on {Ho, RR E}. As illustrated in Figure 1 HF may be natural-language hypothesis and imprecise. Combining with object and relational hypotheses, Ho. H Hr expected to constitute a precise hypothesis and match the underlying formal hypothesis. We describe the confirmation of и (Ho, HR, Hp] below.
generates the functional hypothesis H
4.2 Bootstrap confirmation
In this section, we bootstrap confirm H = (Ho, Hn, Hr) using Definition 3.1 A crucial difference, terms of hypothesis confirmation, against existing works (20 25 is that, roughly speaking, existing works confirm H which is typically imprecise as illustrated in Figure but HypoBootstrap confirms
5

---

= {Ho, HR, H that is deliberately -enerrated and is expected to be as precise as formal language.
Adapting Definition 3.1, defined in the first-order language, to LLMs, natural language- models, is problematic. The pivotal issue is also LLM's hallucination, which makes strict confirmation rarely possible. Therefore, in the following subsections, we attempt to simplify Definition 3.1 with reasonable justification. alleviate the impact of LLM''s hallucination in our bootstrap confirmation. Note that it is our deliberately designed bootstrap generation strategy that permits most of the simplifications. Specifically, we validate the consistency clause, the confirmation clause, and the non-triviality clause of Definition 3.1 in Section 4.2.1 Section 4.2.2 and Section 4.2.3 respectively.
4.2.1 Consistency
The consistency of H U E. where H (Ho, HR, is guaranteed in multiple parts. In Section 4.1.I and Section [4112] we have already ensured (as much as possible) the consis- tency of (oo Hn, F). The consistency check between Hr and (Ho, Hn, E} is ensured by To reduce experimental cost, ConsistencyVallddtto is used after AuxiliaryConf irmer (see Section 4.2.2), since the latter costs less. If AuxiliaryCon irmer failed, consistencyValidaate will not be used.
ConsistencyValidatte
The inconsistency elimination used in Section 4.1.1 and Section 4.1.2 vastly reduces the burden of Consistency dator and the whole framework. we did not eliminate inconsistency when generating Ho and HR, ConsistencyValideee in E, i.e. H H H Hr, which would intuitively cause severe hallucination. Besides, also profiting from the inconsistency elimination, if ConsistencyVall idator detect inconsistency in HF, we can only regenerate HF.
would check the consistency among everything
4.2.2 Confirmation
Recall that the confirmation clause of Definition 3.1 lacks an auxiliary confirmation method and HypoBootstrap uses HD-like confirmation (i.e., similar Definition 3.2), which define here.
Definition 4.1 (HypoBootstrap's Auxiliary Confirmation). Given a hypothesis H, the evidence E confirms H with respect to H' if the program function, translated from {H, H') via LLMs, passes the unit test with E as test cases.
This definition is good approximation for the entailment clause of Definition 3.2 since it attempts to infer E from { H, H'} deductive way. The motivation of utilizing programs instead of LLMs is that LLMs are relatively not good at deduction with given rules, and can often produce correct programs if the rule is not too complex 20). Definition 4.1 is spirit similar to the hypothesis confirmation method used in 20 25 in the case without H' (i.e., E. confirms H, without respect to any auxiliary hypothesis).
In Glymour's 's theory of bootstrap confirmation (Definition 3.I. checking the consistency clause of Definition 3.2 is superfluous since it is implied by the consistency clause of Definition 3.1 See
Appendix A for detailed justification. What is lacking for Definition 3.2 is only the necessity clause
Before discussing the necessity clause, must first discuss the implementation details of the confir- mation clause in Definition 3.1] According to the clause, we should find for each of Ho. HR. He proper subset of H (Ho. H H auxiliary hypotheses Bootstrap, it is probably that verifying Ho or H according to the entailment clause of Definition 3.2 the core part of the confirmation clause, requires the help of Hr since object hypothesis and relational hypothesis usually cannot imply function -enerin-geeneaaeed evidence. Hence, the verification of the entailment clause in Definition 3.2 for Ho or HR is implicitly accomplished we verify He for
help E confirm it. In Hypo-
B.3 indicates that the prompt of ConsistencyVallidttoo is much longer than AuxiliaryCan irmer. Our codebase provide real cases, showing that the prompt and response of ConsisteencyValldator is much longer than AuxiliaryConfirra Hence, Consisteenyyallldtoor much more costly than AuxiliaryConf irmer.
'Appendix
'These works do not use Glymour's 's bootstrap confirmation. All they use is method similar to Definition which is only one part of our confirmation See Appendix A.I for more discussion.

---

the entailment clause with the help of {Ho. HR) (i.e., we have (Ho, HR. HF) E). Therefore, for simplicity, we only verify Hr for the confirmation clause. f
Now we go back to the necessity clause Definition 3.2 What we still need to check, according to the final method given above, is (oo, Hn) Y F. However, this check is unnecessary since Ho and H have little chance of completely describing the mapping f that generates E.
It is worth recalling and emphasizing that although the final confirmation method, after quite few simplifications, seems much simpler than the initial version, is because of our elaborately designed bbotstrap-generation method that we reasonably permitted do the simplification above.
4.2.3 Non-triviality
In the problem setting described in Section 3.1 checking the non-triviality clause of Definition 3.
not critical if the confirmation clause of Definition 3.1 successfully verified via Definition 4.1 Specifically, successfully verified via Definition 4.1 indicates that for all x E X, there is unique y Y such that f(x) = y is true, and that for any y' * y. f(x) y is false. 6 Analogous to
Definition 4.1 we then have H + -f(x)) y'. By Definition 3.3 f(x) = y is thus possible, but
not observed, evidence that disconfirms H.
4.3 Refinement
As depicted Figure and mentioned in Section 4.2.2 if H or H is not confirmed, regenerate He with error feedback. The feedback contains the wrong case from Definition 4. and its corre- sponding object hypotheses and relational hypotheses. We use to denote the number of refinement iterations (T indicates no refinement).
5 Experiments
We compare HypoBootstrap (HB) against Hypothesis Refinement (HR) (0 and MoC [13 to demon- strate effectiveness of both bootstrap generation and bootstrap confirmation. Hypothesis Search (HS) [25] manually adds human priors into prompts; hence, it is not directly comparable to other methods, and cannot be easily extended to new tasks. Instead, we imitate HS to add human priors into agents
whether the HypoBootstrap's performance can be further improved. We also conduct ablation studies and case studies to further demonstrate the effectiveness of HypoBootstrap's components.
5.1
Setup
Our experimental setup follows HR 20 detailed in this subsection. All experiments use GPT-4 18
We also include results on DeepSeek-V3 V3 B Appendix C.I For a fair comparison, we ensure that the number of functional hypotheses generated across different methods remains the same, i.e., N in HR and K in MoC. This also ensures that the frequency of unit testing in training In addition, LLM'' decoding temperature in HypoBootstrap
remains the same across all methods.
g
set to 0, i.e., greedy decoding.
5.1.1 Evaluation
Evaluation is conducted on hold-out of unobserved evidence, separated from the observed evidence used to infer rules. We use raw accuracy and task accuracy to compare inductive reasoning
'In practice, we found the LLM agent AuriliaryCon irmfe can generate a well-defined program solely hased on Hr, and thus the helper hypothesis (Ho, Hr) in Definition 4.1 (corresponding to the entailment clause) are unnecessary. Generating the program additionally hased on To. HR) may further increase the correctness. but may augment hallucination due the long context. We choose to ignore them reduce experimental cost.
'The only exception is that the program function involves randomness, preventing from producing outputs uniquely. We did not observe such in experiments.
McC doesn't 't involve iterations. See their original paper for the definition N and
Mooe precisely, HR and HB may have one less unit test since not necessary in the final refinement iteration.
7

---

Table I: Main results. T" refers to the number of refinement iterations. K = T for MoC.
ACRE
List Fns
MiniSCAN
MiniARC
Avg.
T
Met.
Raw
Task
Raw
Task
Raw
Task
Raw
Task
Raw
Task
HR
78.3
45.0
51.6
42.4
77.0
46.0
5.9
3.8
53.2
34.3
MoC
74.0
34.0
53.0
43.2
60.4
21.0
6.7
4.6
48.5(-4.7)
25.7(-8.6)
1
НB
84.8
57.0
53.5
42.0
92.9
75.0
9.5
60.2(+7.0)
44.7(+10.4)
HB
84.8
$7.0
57.9
47.6
94.9
82.0
11.0
6.9
62.2(+9.0)
48.4(+14.1)
HR
77.8
47.0
61.7
52.8
98.2
95.0
10.1
6,9
62.0
50,4
MoC
73.0
39.0
61.9
50.8
87.3
64.0
10.8
8.5
58.3(-3.7)
40.6(-9.8)
3
HB
79.0
56.0
59.9
50.4
93.2
78.0
12.6
61.2(-0.8)
48.0(-2.4)
HB
79.3
56.0
61.8
50.4
96.2
79.0
12.3
7.7
62.4(+0.4)
48.3(-2.1)
HB
78.5
55.0
62.4
54.0
96.6
89.0
11.0
7.7
62.1(+0.1)
51.4(+1.0)
HB
79.0
55.0
64.3
53.6
98.2
89.0
12.1
8.5
63.4(+14)
51.5(+1..)
Table 2: Ablation results List Functions.
Table 3: Results with human prior.
T=1
T=3
MiniARC
Method
Raw
Task
Raw
Task
Method
Raw
Task
HB
57.9
47.6
62.4
54.0
HB (T=1)
11.0
6.9
w/o o Ho
57.2
45.6
62.4
52.0
w human prior
11.0
7.7
w/o HR
53.4
44.0
60.5
51.6
HB (T=3)
11.0
7.7
HB
53.5
42.0
59.9
50.4
w human prior
12.6
9.2
methods. An input-output pair is considered accurately predicted by hypothesis if the program function, translated from the evaluating hypothesis via LLMs, passes the unit test with this pair as the test case (similar to Definition 4.1). The raw accuracy of task is the average accuracy input-output pairs, and the raw accuracy of a dataset is averaged over all tasks involved. The task accuracy of dataset is the percentage of tasks whose input-output pairs all accurately predicted.
all
5.1.2 Datasets
We use four inductive reasoning datasets with varying natures: causal induction, concept learning, grammar learning, and abstract reasoning. See Qiu al. (20) and our codebase for examples and more details.
The Abstract Causal REasoning dataset (ACRE) 27 is diagnostic benchmark for causal induction. The presence of certain objects triggers a machine, and the goal is to find which objects can trigger it. Sampled 100 tasks from the original dataset are used. Following Gendron et al. (5], instead of images, evidence is represented in language. List Functions dataset [2T) is initially designed for psychological investigation of concept learning, which requires establishing mapping from input lists to output lists. We use the original 250 tasks. MiniSCAN 12 requires the ability of sequence-tor so-suuueee learning. As in HR, quasi-symchronous context-free grammar 23 used to represent -oreeuueet-- rules and pseudowords to represent inputs. We use 100 generated tasks. The Abstract Reasoning Corpus (ARC) [1] is advanced benchmark for measuring general fluid intelligence. MiniARC [10] is a small-scale version of ARC, where inputs and outputs are 5x5 visual grids. The remaining 130 tasks after heuristically filtering are used. The visual grids are transformed into text data by mapping the grid cells to corresponding integers.
5.2
Main results
Table
shows the main results. HB removes all the inconsistency elimination (Section 4.1.1
and consistency validation (Section 4.2.T) from HypoBootstrap. The only improvement of
HB from HR is generating object hypothesis and relational hypothesis before generating functional hypothesis. Experimental results show that, when T = 1, the bootstrap generation gains 7.0 percentage point improvement on raw accuracy, averaged over all four datasets, and a 10.4 percentage

---

number sequerce larger prevacuss
Outipu:: (0, 21, 66,85)
di hypothesis Generator
dojject Hypothesis Enconsiscancy
2ol abfeereeenne the qaino aoaa -al- -i---
4.The
regalive data
(Ounipu: (5, 60)
nator
The number aspears
The cutput sequence only includes numbers from the input sequence that are larger than the previcus number the
Aciational Hyputhesis Generator
Relational Hypothests Tnronsl steery
2,The ouiput sequence does nos inclued any ofehe smaller-umbrre from she sltenatinn pattom the leput soquence.
culpui ss sequense, induating pattern of incrasinn values.
Ellxinator
the three elements Input, regardless their values or pattern form. 2he frequerry of
curput
imput not
presence the outpput
Functional hypothests Generator
From the input velect the fist number and then welet the neet number larger than the precinualy welected number. Continue this prozesa the end of
Figure 3: A real case from HypoBootstrap on List Functions.
point improvement on task accuracy, significantly demonstrating the effectiveness the bootstrap generation. Besides, the hootstrap confirmation can further improve performance. When T = HypoBootstrap gains an 9.0 & 14.1 percentage point improvement on raw accuracy & task accuracy, significantly demonstrating the effectiveness of the whole framework.
When using iterative refinement (T = 3). HypoBootstrap still outperforms HR on average. While HR, HB, HB stop refinement if the training task accuracy achieves one (without validating by ConsistencyValin Waitor) and use results from the last iteration if no early stopping. HB* and HB* use the results from the iteration with the best training raw accuracy. Results show that consistency- related agents are indispensable for iterative refinement. We also found that HypoBootstrap slightly underperforms HR on MiniSCAN when T 3, caused by LLM's degraded long-context capability (Appendix C.3)
MoC is unstable across tasks. On ListFins and MiniARC, MoC outperforms HR achieves compara- ble performance, which is in line with the results reported in MoC's paper. However, on ACRE and MiniSCAN, MoC shows poor performance. MoC is even significantly worse than HR in several met- rics. The latter two tasks both require generating hypotheses under a specific format, and the correct hypotheses are usually highly composite, i.c.. the output must be obtained by applying the rules in combination. This significantly differs from ListFns & MiniARC and makes ACRE & MiniSCAN challenging benchmark. The comparison between the robust performance of HypoBootstrap and the unstable performance of MoC reveals the benefit of our framework.
5.3
Ablation study
The main results have already shown the individual effectiveness of bootstrap generation and bootstrap confirmation, respectively. In this section, further analyze the effect of object hypothesis and relational hypothesis. Table the indispensability of both object hypothesis and relational hypothesis. However, the consistency-related agents in bootstrap confirmation is the most important according to Table
5.4 Adding human prior
Similar to HS [25], we add human priors into agents' prompts, e.g., reasoning hints (Appendix B.4).. Table shows shows the improvement, and believe more deliberately designed human priors (e.g., don't 't' use selected in-context examples as in HS) would further improve performance.
5.5 Case study
Figure gives real case HypoBootstrap from our experiments. First, list of object patterns is generated (in the light yellow box at the upper right) for each input output. Inconsistent patterns are eliminated (struck through with a deletion line) while preserving the consistent ones (highlighted in green). The filtered object hypothesis is used to bootstrap-generate the relational hypotheses (within the light yellow box positioned centrally on the right side), list of relational patterns of each oututt pair. As for object hypotheses, inconsistent relational patterns filtered out.

---

Table 4: Token consumption for DeepSeck-V3, P for the number of prompt tokens and C for the number of completion tokens. T for MoC. (Unit: thousand tokens)
ACRE
List Fns
MiniSCAN
MiniARC
Avg.
T
Met.
P
C
C
C
C
C
HR
22
6
161
34
43
29
622
120
212
47
MoC
46
24
209
67
306
417
152
50
178
140
HB
401
69
2464
664
1201
245
991
283
1264
315
HR
56
10
539
76
331
77
1064
208
498
93
MoC
98
23
439
82
454
464
317
90
327
165
HB*
572
82
3242
840
1642
305
1616
407
1768
409
Finally, based on the object and relational hypotheses above, the functional hypothesis is bootstrap- generated for all input-output pairs. More importantly, compared with HR [20], whose generated functional hypothesis (i.e., rule) in this case is removing all even numbers and zero from the list, method successfully generates the functional hypothesis since the pre- generated object and relational hypotheses automatically provide hints. See Appendix C.2 for more study.
Limitations
Although HypoBootstrap achieves the best results on inductive reasoning, as shown in Table 1 still cannot completely solve LLM's hallucination problem in inductive reasoning. For example, C shows wrongly generated rule after refinement iterations. The main reason is that LLMs cannot strictly adhere to formal logic.
Appendix[C.. CC3
Another limitation is that HypoBootstrap has higher inference cost compared to baselines. Table shows the token consumption of each method. On average, compared with HR & MoC, HypoBoot- strap consumes
7 times the prompt tokens and 2 times the completion tokens for iteration, and & 5 times the prompt tokens and 4 & times the completion tokens for 3 iterations. We argue that the stable performance gain well justify the token consumption. In addition, HypoBoostrap has decreasing marginal cost in terms of refinement iterations, since only Hr is regenerated in each refinement loop and the agents related to Ho, HR are only used once at the very beginning.
Besides, HypoBoostrap and all baselines require way to verify generated hypotheses, e.g., unit test. Although this available in research benchmarks, verification not always available in real- world scenario, such open-ended text-based induction. In scientific research, which is one of the largest applications of inductive reasoning, hypothesis verification may be costly due to expensive experimental instruments. Hence, reducing the need for verification is worth exploring.
Conclusion
LLM-based inductive reasoning allows direct reasoning in natural languages, without the need to convert into specific formal language. To alleviate hallucinations and provide systematic guidance for LLM-based inductive reasoning, we propose HypoBootstrap, a framework integrating hypothesis bootstrap generation and hypothesis bootstrap confirmation. Experimental results four different scenarios, including causal induction, concept learning, grammar learning, and abstract reasoning, demonstrate the effectiveness of both bootstrap generation and bootstrap confirmation. However, HypoBootstrap still cannot completely solve LLM's hallucination problem in inductive reasoning, though it achieves the best results. this end, we emphasize that using HypoBootstrap in applications must consider the potential fallacious rules it produces.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (No. U2433212), in part by the Beijing Municipal Science and Technology Program (No. Z221100007 22003), in
10

---

part by the Fundamental Research Funds for the Central Universities, and in part by the State Key Laboratory of Complex & Critical Software Environment.
References
[i] Chollet. On the measure intelligence.
CoRR, abs/1911.01547, 2019. URL http
7/arxiv. org/ httg/abs/tte abs/1911. 1911.01547
[2] A. Cropper and S. Dumancic. Inductive logic programming 30: A new introduction. J. Artif. Intell. Res., 74:765-850, 2022. doi: 10.161 10.1613/AAIR.11 3/A.RR1..350.. URL httpt://doi..orrg
10. 10.1613/jair... 13507
[3] DeepSeck-Al. Deepseek-v3 technical report, 2024. URL httptts:/aaxiiv.or//aaaaaa 19437
[4] I. Douven and W. Meijs. Bootstrap confirmation made quantitative. Synthese, 149(1):97-132. 2006. doi: 10.1007/s1 1229-0024--222 000-00000-222222
[5] G. Gendron, Q. Bao, M. Witbrock, and G. Dobbie. Large language models are not strong abstract reasoners, 2024. URL httthts////rxiii www.iv.org//bbb ab0.aa/...... 19555
[6] C. N. Glymour. Theory and evidence. Princeton University Press, Princeton, New Jersey, 1980.
[7] T. R. Grimes. Truth, content, and the hypothetico--eeduutive method. Philosophy of Science, 57 (3)5514522,, 1990.
[8] D. Hume. A treatise of human nature. Oxford University Press, 2000.
[9] S. Ji, S. Pan, E. Cambria, Marttinen, and S. Y. Philip. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33(2):494-514 2021.
[10] Kim, P. Phunyaphibarn, D. Ahn, and S. Kim. Playgrounds for abstraction and reasoning. In NeurlPS 2022 Workshop Neuro Causal and Symbolic Al (nCSI), 2022. URL https //openreview. wwwtfeeuue/itiee FdirFiddpFoRBooPPPPP
[11] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
[12] B. M. Lake, T. Linzen, and M. Baroni. Human few-shot learning of compositional in- structions. In Annual Meeting of the Cognitive Science Society, 2019. URL https / /aapi www.steiamoooooo..m...
Org/CorppsIDD 58006558
[13] K.-i. Lee, H. Koh, D. Lee, S. Yoon, M. Kim, and K. Jung. Generating diverse hypotheses for inductive reasoning. L. Chiruzzo, A. Ritter, and L. Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Pupers), pages 8461-8474, Albuquerque, New Mexico, Apr. 2025. Association for Computational Linguistics. ISBN
199-8899176-18-66 URL htttpss//cclantholooyoror.g
nww.aaacllllloo long. 429/
[14] J. W. Lloyd. Foundations of logic programming. Springer Science & Business Media, 2012.
[15] J. D. Norton. The determination of theory by evidence: The case for quantum discontinuity, 1900-1915. Synthese, 97:1-31, 1993.
[16] J. D. Norton. How we know about electrons. In After Popper, Kuhn and Feyerabend: recent issues in theories of scientific method, pages 67-97. Springer, 2000.
[17] J. D. Norton. A little survey of induction. April 2003. URRl:attss///aacchllarraippppppp
[18] OpenAL. GPT-4 technical report. CoRR, atw.23/3..0/77//44 2023. doi: 10-48550/ARXIV.2330.3 08774. URL https::::/ :/ //ooi. org 48550/ www.:i....3....300 08774

---

[19] C. S. Peirce. Questions concerning certain faculties claimed for man. The Journal of Speculative Philosophy, 2(2):103-114 1868.
[20] L. Qiu, L. Jiang, X. Lu, M. Sclar, V. Pyatkin, C. Bhagavatula, B. Wang, Kim, Y. Choi, N. Dziri, and X. Ren. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httLtttttp//perreeeeee
net/ forrmm/rrmm//// ddoimmddddbttoa2222.
[21] J. S. Rule. The child as hacker: building more human-like models of learning. PhD thesis, Massachusetts Institute of Technology, 2020.
[22] K. Sinha, S. Sodhani, J. Dong, J. Pineau, and W. L. Hamilton. CLUTRR: diagnostic benchmark for inductive reasoning from text. In K. Inui, J. Jiang, Ng, and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4506-4515, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v//D199 1458. URL
org/ 1458/
[23] D. Smith and J. Eisner. Quasi-synchronous grammars: Alignment by soft projection of syntactic dependencies. In Koehn and C. Monz, editors, Proceedings on the Workshop on Statistical Machine Translation, pages 23-30, New York City, June 2006. Association for Computational Linguistics. URL https: http::::/aaaaaahhooooooo
org/w06- MtT///066330////
[24] W. Sun, H. Xu, X. Yu, P. Chen, S. He, J. Zhao, and K. Liu. ItD: Large language models can teach themselves induction through deduction. In L.-W. Ku, A. Martins, and V. Sriku- mar, editors, Proceedings of the 62nd Annual Meeting of the Association, for Computational Linguistics (Volume Long Papers). pages 2719-2731, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 1/2044 https://aclanthan
URL
2004///222...22 acl- -oog..n..... 150/
[25] R. Wang, E. Zelikman, G. Poesia, Y. Pu, N. Haber, and N. D. Goodman. Hypothesis search: Inductive reasoning with language models. In The Twelfth International Conference Learning Representations, ICI. 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL. https://openreviie net. htte//orrmm?cc dittummiImd
[26] Z. Yang, L. Dong, X. Du, H. Cheng, E. Cambria, X. Liu, J. Gao, and F. Wei. Language models as inductive reasoners. In Y. Graham and M. Purver, editors, Proceedings of the 18th Conference of the European Chapter the Association for Computational Linguisties (Volume 1: Long Papers). pages 209-225, St. Julian's, s, Malta, Mar. 2024. Association for Computational Linguisties. URL wtttttts:///aaaaaaaaha
org/ 20eeg/2224.20 10244eacl-long 13/
[27] C. Zhang, B. Jia, M. Edmonds, S. Zhu, and Zhu. ACRE: abstract causal reasoning beyond covariation. In IEEE Conference on Computer Vision and Pat-
tern Recognition. CVPR 2021, virtual, June 19-25, 2021, pages 10643-10653.
Computer Vision Foundation / IEFF, 2021. doi: 10.1109/CVVPR443372222
01050. URL https://openaccees. thecvf. cwm/ htmp///attom////p2222///////attt
ACRE_ TRAa_Abstract_Caae Beaoaaaannnooonnnoooendddadd Covariation CVPR__CCP_________________ 2021 2a1p1r_ppprr.. html
[28] W. X. Zhao, K. Zhou, Li, T. Tang, X. Wang. Y. Hou, Y. Min. B. Zhang. J. Zhang. Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J. Nic, and
J. Wen. A survey of large language models. CoRR, abs/2303. 18223, 2023. doi: 10.48550/
ARXIV.2303.18223 URL httLhttt://// :/do:i:///..../ http:///oio..o///////////// 48550/arXiv 2303. 18223
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims are clearly written in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
Section
Appendix
A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section Appendix
B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification:
Code is released and the url is given in the main text.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
www.sssssooooiiiyooo ollicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/t.pss.ccppullc//uuii / Sow.llesss/oSSSeemiiseeee
www.SSubbisiooooiiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section
Appendix
B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Repeated experiments on GPT-4 are too expensive. Besides, since we use greedy decoding for LLMs, there is no randomness in our algorithm. However, note that the API service providers do not guarantee reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: Since we mainly use public API services, the computer resources cannot be determined. Instead, we have discussed the token consumption in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: No ethic concerns.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We mainly use public API services. No risk happens from our side.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have check the licenses.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We give README.md and LICENSE for our code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: LLM is the core component of our proposed framework, and the usage of LLM is clearly described Scction
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
19

---

We include various details in this appendix, including a discussion on Glymour's theory of bootstrap
confirmation (A). HypoBootstrap details (B more experimental results ( examples of converting
natural-language statements to formal-language statements (D), and broader impacts
(E.
More specifically, Appendix includes the pseudo-code of HypoBootstrap (B.1) justification for
relational hypothesis generation (B2). agents' details (B.3), prompt with human prior [B.4). Appendix
includes DeepSeek-Ve results [C]] more case study (C.2) and failure analysis (C33)
A Glymour's theory of bootstrap confirmation
In this section, we introduce the first-order formalization of Glymour's 's theory of bootstrap confirma- tion introduced by Douven and Meijs 4 and justify how to transform it to Definition 3.1 in the case of using hypothetico-deductive confirmation (Definition 3.2) disconfirmation (Definition B.3 the HypoBootstrap framework. Since HypoBootstrap uses a hypothetico byypoheriio-dedduttiiiss -ww.dddiiis--liiiiiieiei like confirmation (Definition 4.1) disconfirmation (Section 4.2.3) using Definition 3.1 is rational approximation.
Definition A.1 (Douven and Meijs [4]ss Formalization Glymour's Theory of Bootstrap Confirma- tion). Given hypothesis H [H1 H2,..... ................................ Hn), }. evidence E bootstrap-confirms H if
1. HUEXI
2. for each H;, there exists H' C H s.. H; & H' and
(a) E confirms H; with respect to H' and
(b) there is possible evidence E", but not actually observed, such that E' disconfirms H; with respect to H'
3. for each Hi there is no H" C H such that E disconfirms H; with respect to H"
The confirm & disconfirm refer to an auxiliary mnon-bootstrap) confirmation & disconfirmation method. Glymour's s original theory tends to be neutral across auxiliary (non-bootstrap) confirmation methods ( We select hypothetico-deduucive confirmation & disconfirmation, where evidence confirms a hypothesis if the hypothesis can deductively entail the evidence, since it is one of the popular confirmation methods scientific research. The formalization of hypothetico-dedducive confirmation & disconfirmation has been given in Definition 3.2 & 3.3 In Definition (32 & 3.3 if E'
serves as deduction consequence, it is understood the conjunction of its elements. For example, H U H' + E is identical to H U H' A (x)(0) ())
Since Glymour's bootstrap confirmation often seems circular at first sight, is worth mentioning the significance of the clause (b) of Definition A. which corresponds to the non-triviality clause of Definition 3.1 This clause makes the Definition A.I not a circular confirmation. For a discussion, see Comment (6) for Definition 2.1 in Douven and Meijs 'ss work.
Now we specify how to transform Definition A.1 to Definition 3.1 Douven and Meijs
4 indicates
that using -epooheeicoodeeuctive confirmation can eliminate the third clause of Definition A. since it is entirely redundant. See Comment (3) for Definition 2.1 in Douven and Meijs 4 's' work for proof. Hence, we only consider clauses 1, 2.))), and 2b,b), which, respectively, correspond to the consistency clause, confirmation clause, and non-triviality clause of Definition 3.1
Simplification of non-triviality clause The non-triviality clause of Definition 3.1 is independent of H' and different from the clause 2.(b). The non-triviality clause is sufficient condition of the clause 2b)b) when the clause 2.()) holds. We do such simplification because separately determining E' for object hypothesis, relational hypothesis, and functional hypothesis are difficult and costly for LLMs. To see the sufficiency, recall we consider HD disconfirmation (Definition 3.3), where checking E' disconfirms H is check H EE' (the consistency of H checked by the clause 1). Similarly, checking E' disconfirms H; with respect to H' is to check H; U H' + EE. By these definitions, if E' disconfirms H. then for each H;, disconfirms H; with respect to H H \ H; Hence, E' is possible evidence that disconfirms any hypothesis. On the other hand, according to the clause (.()), we already have H; U H'' F. for some H'. By the monoticity of first-order logic and since H' H", we have H. U H | E. we set H' to H" then the clause (.(a) and 2.(b) both hold.
20

---

Algorithm HypoBootstrap
1: input: evidence E {{(xx))) = y()) ,(,))).,,((()))))
) = = maximum iteration of refinement
T
2: output: functional hypothesis F
3: 1* Object Hypothesis */
4: for i = 1 to n do
5:
H -- Oyjjccilppcceessseennerraaoraa)"
6:
H3 + Dbjjcchtpocheessseenerrrttrr///)
7:
H
oijeetppppaeessslooossssreegaaan ,2(0)
8:
H
obeechhpochhsssstcoossssergggahn (1) y(00)
9: end for
10: Ho - H AH
1* Relational Hypothesis */
12: for 1 to n do
H f- Ralattooolllpphhhsssssssrrrrrrrd HH, ,0(0 (()))
14: Hzg
Raaedomnliyyceessssooossttteeetn x((1 (0(())
end for
16: HR + N H!0
17: 1* Functional Hypothesis */
18: Ewrong =
19: repeat
20: if' it is the first iteration then
21:
Hr +- Phactioalliyppaasssssnnnaaaarrrr Hn)
22:
else
23:
HF +- Fuactlooallkypcaees.snnnrrrorrrr Ho, Ha, HF, Ewrong)
24:
end if
25:
Ewrong + buxiliaryConffrmerr (Hr
26: if Ewring is not empty then
27:
continue
28: end if
29:
if Comsisteney/Vlliatttortt.t Ho. HR, HF) then
break
31:
end if
until reach the T-th iteration
Consistency clause of Definition 3.2 is superftuous. In the context of Definition 3.1 only the confirmation clause uses an auxiliary confirmation method, and the hypotheses in the confirmation clause (i.e., H4, H') are element subset of the whole hypothesis to be bootstrap-confirmee
(i.c., H). The consistency clause in Definition 3.1 already verifies E K I, which accomplished in Section 4.2. By the monotonicity, if (1, H' 1. then H U E + I. Hence, don' need to check the consistency clause Definition
A.1 Relation to existing works
The confirmation used by Hypothesis Refinement (20). Hypothesis Search (25), and MoC [13 is Definition 4.1 This is only one part of our confirmation since they only generate functional hypothesis and do not generate object hypothesis and relational hypothesis.
HypoBootstrap bootstrap- getrupptggerrtessss functional hypothesis based on object hypothesis and relational hypothesis, and all these hypotheses together form a complete hypothesis for the transformation rule. Consequently, addition to using Definition 4.1 which accomplished by AuxiliaryConfirmee immer HypoBootstrap also
uses
ObjectHypothesi Inconsssteencyyllnnii. and Consistencyyallllato to confirm
Relat i ional the hypothesis.
21

---

B HypoBootstrap details
This section shows more details of HypoBootstrap. including the pseudo-code (B. justification for relational hypothesis generation (B.2 agents' details [B.3) and prompt with human prior (..4)
B.1 Pseudo-code
The pseudo-code of HypoBootstrap is given in Algorithm Algorithm Sttos can pass MoxiliaryConf irmer (i.e., Fwrong empty) and ConsistencyVal Valatooo... Slightly different from Algorithm [] the variant HB* selects the functional hypothesis from the iteration with the best raw accuracy on observed evidence.
stops if all training evidence
B.2 Justification for relational hypothesis generation
One may question the assumption that {Ho. } is consistent enough since we do not eliminate the inconsistency between H H and Hay.. We argue that this inconsistency less severe since H is generated based on H() Hy and H!! H Hay are, respectively, already consistent with E. reduce the cost, we do not explicitly eliminate the inconsistency between H H and H comments on inconsistency elimination, which htw.:llwppppttesssssooossaaaaaaa
We also have two extra accomplished
and First, these two agents remove in- consistent object patterns from the pattern list or remove inconsistent relational patterns from the relational pattern list. Filtering in this way does not increase inconsistency, even if they may accidentally remove consistent patterns. Second, eliminating patterns in Ho, HR may reduce the hallucination of Functi FunctionalHypont maaliyyothhessseeeerrtt since LLMs are usually error-prone to long context.
by
by decreasing the length of the input context,
B.3 Agents
The agents' prompts used for GPT-4 and List Functions dataset are shown in Table 5 where the auxiliary prompt templates are shown Table The agents for other datasets are similar, and the differences arise from different task characteristics. For example, the Auxiliaa AuxiliaryConf irmer for MiniSCAN and ACRE accomplished via symbolic validators since their functional hypotheses are organized in formal structures. Refer to codebase for more details on all datasets and all LLMs. Below, we describe the entire procedure with List Functions task.
Objecttlyoothesseeneeee GGenerator generates list of object hypotheses of given data, where {object} refers the input or output of each task. The object hypotheses are then fed into Obje wwwpehhessssnnnssssssseeeei ates their numbers, and filters them based on the generated numbers.
eillimmtooo,,oo which identifies inconsistent hypotheses, gener-
Similarly, Relational www.oooyltheeeieeeeeeeeeeeee sGenerator generates potheses between given input and output according identifies inconsistent hypotheses,
list of relational hy- their object hypothesis.
Hyyaaiieeesyootheessii htwpiiesiisnnnonnnssnnnnnneee inniiiinateyylliiaaoo generates their numbers, and filters them out based on the generated numbers.
The Funct i mmalliipothesiseeeereee feedback and with feedback. The generation without feedback takes all the input-output pairs as examples, along with all the consistent object hypotheses and relational hypotheses generated in previous steps. The generation with feedback leverages the previous functional hypothesis, unit test feedback, and the corresponding object hypotheses and relational hypotheses of the wrong examples.
generates the functional hypothesis in two modes: without
The Auxil 1 aryConf irmer evaluates the functional hypothesis by unit tests. After passing the unit test on observed evidence, the aemssstencyaalieee Valittor checks whether the rule is logically consistent with the input-output pairs and their object hypothesis and relational hypothesis.
22

---

Table 5: Prompts used for List Functions.
Agent
Prompt
Observe the following data, which consists of integers. Systematically analyze its patterns.
Provide up to three most important patterns your analysis. Data: {object}
Object Hypothesis Generator
Please format your analysis follows:
2.
3.
Observe the data and its patterns below.
Data: {object)
Object Hypothesis
Patterns:
hypotheshppotsssis>
Inconsistency Eliminator
Which patterns are logically inconsistent with the other patterns?
Please respond with pattern numbers follows: Inconsistent patterna: <number>, <number>, <number>,
Observe the input-outppl pair and its patterns below. Systematically analyze the correlations between the input and the output.
Provide up to three most important correlations in your amalysis.
Input: {input}
Output: {output)
Patterns of the input:
Relational Hypothesis Generator
Patterns of the output:
Coutput.objacct.bpotteess
Please format your analysis as follows:
2.
3.
Observe the input-outpen pair and the correlations between the input and the output given below.
Input: {input}
Output: foutput}
Relational
Hypothesis
Correlations:
Inconsistency Eliminator
deelational_hypoheeet
Which correlations are logically inconsistent with the input-output pair and other correlations?
Please reapond with correlation numbers follows: Inconsistent correlations: <number>, <number?, <number>,
23

---

Table 5: continued
Agent
Prompt
Generate transformation that converts each the given input lists into their corresponding output lists.
{examples}
Please format your rule as follows: Rule: <Your rule>
Functional
Hypothesis Generator
Below, collect the patterns and correlations of several input-output pairs mentioned above.
(w/o feedhack)
hyyothessis
Based on your previous rule and the feedback, generate revised transformation rule that converts each of the given input lists into their corresponding output lists.
Your previous rule: maaetiioal.hypotheiii.
Feedback:
[feedback}
Functional
Hypothesis
Please format your revised rule as follows: Rule:
Generator
<Your rule>
(w feedback)
Below, collect the patterns and correlations of several onput-output pairs mentioned above.
(feedback.uith_ htg.aaahw..thc.oooe...moooooooad
Write a Python function for following rule, where the input must be a list of integers and the output must also be a list of integers.
Auxiliary Confirmer
Rule: (functional hypothessi)
Is the target statement logically consistent with the input-output pairs their patterns and correlations?
Target statement: iayeciional.ypothhssii
Consistency Validator
Please reapond with Yes or No.
24

---

Table 6: Auxiliary prompt templates for List Functions.
Template
Prompt
Input: (input)
example_template
Output: {output)
Pair (number)
Input: (input)
Output: (output)
Patterns of the input:
example_witt wettehttthhhhtteie object_tooee and heairaal_yoothhsstteeee
Patterns of the output: Suutput_object_uyytttsstt
Correlations between input and output:
inarrional.hypoohesiie
Input: (input} Expected output: foutput)
feedback_templee
Predicted output: (prediction)
Input: (input> Expected output: {output>
Predicted output: (prediction)
Patterns the input: hunut__bbjet_bbyptteee
feedback with with_object, and iaatiinal_yypohheiis template
Patterns of the output: houpput_ohject__ypoteesss
Correlations between input and expected output: Reelational_hypotheee
B.4 Prompt with human prior
The prompt with human prior for MiniARC (used in Section 5.4 is shown in Table 7
25

---

Table 8: Results on DeepSeck-V2. V3.
ACRE
List Fns
MiniSCAN
MiniARC
Avg.
T
Met.
Raw
Task
Raw
Task
Raw
Task
Raw
Task
Raw
Task
72.0
37.0
44.4
32.8
25.8
21.0
9.7
38.0
24.4
1
MoC
70.5
34.0
52.5
40.8
53.8
44.0
9.2
6.2
46.5(+8.5)
31.3(+6.9)
HB
75.8
48.0
55.6
42.0
69.8
47.0
16.7
10.0
54.5(+16.5)
36.8(+12.4)
HR
77.3
53.0
52.2
41.6
34.3
24.0
16.2
11.5
45.0
32.5
3
MoC
73.5
43.0
59.2
48.4
80.0
61.0
14.9
11.5
56.9(+11.9)
41(+8.5)
HB*
77.5
53.0
62.8
48.8
74.0
53.0
21.8
14.6
59.0(+14.0)
42.4(+9.9)
Table 7: Prompt with human prior for MiniARC.
Agent
Prompt
Generate tramsformation rule that converta each of the given input matrices their corresponding output matrices. Each input and output is grid of numbers representing a 5x5 visual grid.
The transformation rule may involve counting sorting objects (e.g. sorting by size), comparing numbers (e.g. which shape symbol appears the most? Which is largest object? Which objects are the same size?), or repeating pattern for a fixed number of time.
There are other concepts that may be relevant.
Lines, rectangular shapes
Flipping objects
Rotating objects
Translating objects
Shape upscaling downscaling, elastic distortions, Containing being contained being inside outside a perimeter.
Functional Hypothesis
Generator
Drawing lines, connecting points, orthogonal projections.
(w/o feedback)
Copying, repeating objects.
Pushing objects side
(examples}
Please format your rule follows: Rule: <Your rule>
Below, we collect the patterns and correlations of several input-output pairs mentioned above.
witplees_stthhhbecttteeeeee red_tenational obap
C More experimental results
This section includes DeepSeek-V3 results C.I more case study (C.2). and failure analysis (C.3
C.1 Results on DeepSeck-V? V3
In addition to GPT-4, we include experimental results on DeepSeek-Ve V3 in Table We can draw similar conclusions from GPT-4 results.
26

---

Evidence
Input: [7, S, 4]
Object Hypothesis Inconsistency Elininator
Relational Hypothesis Inconsistency Eliminator
Output: [5, 4]
Object Hypothesis Generator
Relational Hypothesis Generator
Inpuut 8,5,
Output: (6, 3.
Input: [1, 4,
AuxilLaryConfirmer
Feedback
Expected Output:
Functional Hypothesis Generator
first element from the input list.
"python
Actual Output: [9, 4, 7, 3]
fn([st):
len((ss) Ist(1:]
Input: [7, 3, 5, 2]
else: return
Expected Output:
Feedback
Actual Output: [3, 0,
Figure 4: A case for HypoBootstrap's Auxi AuxiliaryConfirre from List Functions.
C.2
More case study
Figure 4 shows a case of HypoBootstrap's AuxiliaryConfirmes from List Functions. The functional hypotheses overfit to certain observed evidence and are inconsistent with other ob- served evidence. The partially correct generated functional hypotheses were evaluated by muxiliaryConfinn irmer imeer but did not pass the unit test with observed evidence. The feedback contain- ing the failure cases of observed evidence is collected for the to generate revised functional hypotheses. Without auxiliaryConf irmer, HypoBootstrap is in- capable of rectifying incorrect functional hypotheses since it cannot identify the failure cases from observed evidence.
More real cases can be found in our codebase.
C.3 Failure analysis
An example of failure case when 7' 3 from MiniSCAN shows that our method requires robust
long-come long-context comprehension capabilities, which current LLMs still struggle to achieve. This results in slightly inferior performance compared to HR (20 MiniSCAN when T 3. As shown in this example, the generated rules did not follow the prompt's format due to the long context.
User (Prompt for Functional Hypothesis Generation)
Generate granmar rules that the folloving inputs to their corresponding
outputs. Your graunar rules should follow the format "<imput> -> <output>", Use the prefix "*p" to denote nonterninel syabel. For instance, *#4Λ twice -> ##1 #* A", *88A exap #AB -> #NB **л", The left-hand side cannot contain repetitive or adjacent nonterninal synbols; i.e., rules like *#*Λ -> N&B WAA" are not alloved. Easure that the number of unique nonterninal syabols your
tvice" "4нΛ ##3
on left For each rule, assign integer as its priority. A higher priority indicates that the rule should be considered first when generating parses. Ensure that unnеcesиary colors preučovord(s) in the left hand side. make your
matches
on
right-
replaced by nonterninal syabole and that each rule has
mininal passible.
Input:
Output: "RED"
Input: "szeaurt" Output: "BLUE"
Input: "fiact"
Output: "GREEN"
Input: "cyclaiopa"
Output: "YELLOW"
Input: "szeaurt bingly"
Output: "BLUE BLUE BLUE
Input: "nccluiss bingly'
Output: "RED RED RED"
Input:
rioct"
27

---

Output: "BLUE GREEN BLUE"
Input: "fiect staioff mecluise'
Output: "GREEN RED GREEX"
Input: "szeaurt biouck fiact"
Output: "GREEN BLUK
Input: "nccluise biouck Szeaurt" RED"
Output:
Input: "szeaurt bingly biouck fioct"
Output: "GREEN BLUE BLUE BLUE"
Input: "fiact biouck aecluiss staioff szeaurt"
Output: "RED BLUE RED GREEN"
Input:
biouck riact bingly"
Output: "GREEN GREEN GREEN BLUE"
Input:
Output: "BLUE GREEE RED GREEN"
Please format your rules follous:
<input> <output> Priority 1: <Your priority>
(Balov, collect the patterns correlations several input output pairs nentioned above.)
Pair "szeaurt bingly' "BLUE BLUE BLUE
Input:
Output:
Patterue
input:
The data censists of two separate psoudovords: 'szeaurt' and "bingly". psoudouords single words, not sentences phrases. 3. There is
1.
ssparating the two pseudouords.
Patterns
output:
1. The data consiste of a single repeated element, which is color. repeated The rapatition occurs three times.
"BLUE".
3.
Correlations between imput and output:
1. The autput does not diractly correspend to the individunl paeudororda in the transtornation
simple one-to-one napping. 2. The output is repetition of single color word, "BLUE", three times,
input, auggesting
regardless
input.
3. The specific coler in the output does not seen to be influenced by the apecific paeudovorda
imput, auggesting that transformation night independent of the specific input content.
Input: "nccluisa bingly'
Output: "RED RED RED"
Patterns of the input:
consiste of peeudoxords,
2. There are tuo distinct pseudovords separated by space. pseudouord
single, unbreken string letters, suggesting they may represent individual units or entities.
Patterns of the output:
1. The data consists of a single repeated clanent, which is color.
repeated 3. The repatition occurs three times.
"AED".
Correlations betreen input and output:
1. The input data, regardless of its content or structure, consistently results in the output of the color "RED".
2. The numher of vards in the input does not affect the nunber of tines "RED" in is alvays three.
repeated
output, 3. The specific pseudovords used in the input do not seen to influence the color output, auggeating that the systen does not differentiate betueen different inputa
28

---

Pair
Input: "szeaurt staieff fioct"
Output:
"BLUE GREEN BLUE"
Patterue of the input:
of peeudoxorde,
Each pseudouord is seven letters long
3, All pseudowords are separated by space.
Patterns of the output: data consists
2. The sequence of colors is repetitive, with "BLUE" appearing twice and "GREEX"
3. The pattern of the data "BLUE", "GREEX", "BLUE",
Correlations betreen input and output:
1. Each psaudavord in the imput corresponda to a color in the autput. The first third paeudoxords correspond "BLUE", while corresponde to "GREEN". The pacudezords to the colors they represent. The correlation night be based on predefined or coded systen.
paeudoword
not
phonetic alphabetic correlation
The repatition of the coler "BLUE" output corresponds the repetition of the first paeudosord in the imput, auggesting consistent one-ta-one napping specific peeudovords
Pair
Input: *fiact staioff mecluiss'
Output: "GREEN RED GREEX"
Patterus of the input:
1.
consists of paeudoxorda, Each pseudouord is separated by space.
3. The pseudozords vary in length, ranging from 5 to characters.
Patterue of the output:
consists colors, pseudovorás.
2. The sequence of colors is repetitive, with "GREEY" appearing tuice and "RED" appaaring
3. The data does not show any progression gradient terus color shades bues.
Correlations between imput and output: preudosord
input corresponds napping between psoudovords and colors.
output, suggesting
2. The paeudozord "fiect" and "aceluiss" both naр the color "GREEN", indicating correspond the same color. 3. The pacudoxord "staioff" napa to the color "RED", shozing that each paaudovord specific color.
that different peeudovorde
Pair
Input: "szeaurt biouck fiece"
Output: "GREEN BLUE*
Patteras of the input:
consists of pseudorords, 2. Each pseudovord is separated by space.
3. There are three pseudowords the data.
Patterns of the output:
separated by space.
2. Both elements in the data are colors. The colors represented are prinary (blue) secondary (grean) colors.
Correlatione between imput and output:
Each preudovord the input corresponds and second pseudorords correspond to "GREEK" and "BLUE" respectively. The third output.
color the output. The first
corresponding color 2. The pseadoxords in the imput do not seen to have direct phonetic alphabetic correlation to the colors in the output. correlation night be hidden coded syeten.
preudovord
29

---

Pair 10
Input: "mecluiss biouck
Output: "BLUE RED"
Patterne of the input:
consists peeudoxorde,
2. All paeudoxords are aingle worda, not separated by hyphens or other punctuation
Each preudavord unique, with repetition of the psoudoword.
Patterns output:
1. The data consiste of two distinct elementa separated by space.. specifically "BLUE" "RED". 3. The data does not contain paoudovords.
Correlations betreen imput and output:
1. The output colars "BLUE" and "RED" are derivad fron the lettars present in the input. paeudoword "biouck" BLUE" and "accluiss" contains all the letters for "RED". order psoudezords in the input. The first psaudovord "accluiss" correaponds to the second color "RED", and the second pseudouord "biouck" corresponds to the first color "BLUE".
output
correspond
3. The paeudezord "azeaurt* doea not contribute to the output, auggeating that not imput pseudowords
to generate output.
Pair 11
Input: "szeaurt bingly biauck fiect"
Output: "GREEN BLUE BLUE BLUE"
Patterus of the input:
1.
consists entirely pseudowords, colors. Each pseudouord is separated by space, indicating diatinct units.
3. All pseudoxords are of varying lengtha, ranging from to 7 letters.
Patterue of the output:
colors.
2. The color "BLUE" is repeated three times.
sequence starta
by "BLUE".
Correlations batveen imput and output:
paeudovord the input corresponds one-to-one nspping betveen pgeudovorde and colors.
output, indicating
The pacudexord "szeaurt" correspends biauck", and "fioct" all correspand to the color "BLUE* output suggeata that different
"GREEK", "bingly",
3. The repetition the coler "BLUE" preudovords correspond the color the output.
Input: *fiact biouck necluiss staioff azeaurt"
Output: "RED BLUE RED GREEN"
Patterns of the input:
pseudowords 2. Each preudovord in the data set consists of seven letters. paeudozords consonant placement.
consistent pattern
Patterns of the output:
1. The color "RED" is repeated tvice in the data set.
sequence colors discernible pattern based on coler properties.
follov
spectrun any
Correlations betreen imput and output:
1. The output colore seen to be determined by the third and fourth letters in each pseudovord of the input. If the third and fourth letters locl, the *RED". If they are 'ou', the output is *BLEE'. they are Pai', the autput is GREEN
2. The number of psoudovords in the input matches the number of colors in the output, auggeating one-to-one correspondence betueen paoudouords and colors. The order of the colors in the output natches the order of the pseudoxords in the imput, suggesting that the sequence of pseudovorde determines the sequence of colors.
30

---

Pair 13
biauck riact bingly" Output: "GREEN GREEN GREEN BLUE"
Input:
Patterns of the input:
1.
The data consiste of peeudoxords, not colore.
Each preudovord is separated by space.
3. The paeudozords vary in length, ranging from 5 to letters.
Patterue of the output:
"GREEF" 2. The word "ELUE" appears only once and it is at the end of the sequence. All data points colors.
primarily consiste consecutively.
repetition
Correlations between imput output:
1. The first three рseudoxords in the input correspond to the color "GREEK" in the output,
preudovord correspends "BUUE". suggests one napping hetueen psaudavords and colors.
Input: "fiect staioff mecluiss biouck szeaurt" CREEY"
Output:
Patterns of the input:
The data consiats entirely of paaudowords, colors.
2. All paeudoxorda in the data set are composed of conbination of consonanta and
3. The pseudewords
in length, ranging from 5 to 8 characters.
Patterns of the output:
1. The data consists of colors, not psoudovords.
repeated, specifically "GREEK" appeara
3. The sequence of colors does not follow standard color spectrun or rsinbov
Correlations
imput output: 1. Each pseudouord in the psaudavord "fiact" corresponds to "BLUE", "staioff' and "szeaurt" correspond to GREEN", "accluisa" corresponds
corresponds
color in the output. The
correlation seems *RED".
2. The to pseudowords do not seen to have direct phonetic or spelling correlation they represent. Instead, predefined mapping.
3. The order the pseudovords in the input deternines the order the colors in the output. The first preudoword corresponds to the first color, the second pseudavord to the second color, and so on.
Assistant (Response from Functional Hypothesis Generation)
Based on the patterns and correlations identified, the folloving grannar rules can proposed:
"nccluiss" Priority:
Rule 2: "szeaurt" "BLUE*
Priority:
Rule 3: "fioct" >> "CREN""
Priority:
Rule 4: "cyclaiops" >> "YELLOV*
Priority:
*НNА bingly"
Priority:
Rule 6: *Н4А staioff #ND" "НAА BAB AMA"
Priority:
31

---

Rule 7: *B*a biouck #EB *BAB P#A"
Priority:
Rule B: *asA bingly biauck #EB* 182B aEA ASA REA"
Priority:
Bule 9: *#*A blouek #BB staloff #EC > "EAB HEA BAB #AC"
Priority:
Bule 10: "=аλ biouck ##B bingly" >> *BBB #*B #aB #EA"
Priority:
Priority:
Note: The nonterminal symbol "*aA", *eaB", *#*0" represent psaudavord that be replaced by a color according to the rules 1-4. The priority of the rulas is based complexity rule, priority.
complex rules having higher
D Converting natural-language rules into first-order languages
This section gives examples to convert natural- -anrruggl statements into first-order languages as in Figure I The natural-languae statements related to used datasets.
MiniARC Fill the output grid with the number of non-zero elements the input can converted into the following first-order statement
5,5
elem y, i, gg)))
i=1.==
where elem(y, i,j) is the element of the output in the i-th row and j-th column, g(x) is a function that counts the number of non-zero elements in the input æ.
List Functions See Figure I
ACRE The red cylinder can trigger the machine, but the black cube cannot trigger the machine can be converted into the following first-order statement
trigger(a) -trigger(b)
where a refers the object red cylinder, refers the object back cube, and trigger(C) asserts whether the object can trigger the machine.
MiniSCAN
The symbol gsecua swaps its adjacent two words can be converted into the following
first-order statement
elem(z, æz i) = gsecua + elem(y, - 1) = elem(r, i 1) A elem (y. i 1) = elem (z.
where elem(a: (z. i) is the i-th element of the input sequence y and y the output sequence. This example is simplified since we don't consider the influence of other words in the sequence and retain the symbol gsecua the output.
E Broader impacts
In real-world applications, method contribute scientific hypothesis proposing, daily- life inductive reasoning, etc. If properly used, HypoBootstrap can enhance the inductive reasoning performance. However, emphasize that using HypoBootstrap must consider the potential fallacious rules it produces and that one shouldn't 't completely believe in the results produced by HypoBootstrap.
32


Reviews:
[{"review_id": "hPgqGDKotB", "reviewer": "Reviewer_vRQJ", "summary": "The paper proposed a framework addressing two core steps in inductive reasoning: hypothesis generation and hypothesis confirmation. For hypothesis generation, it incrementally constructs object-level, relational, and then functional hypotheses in a bottom-up manner, helping LLMs move from simple patterns to more complex generalizations. For hypothesis confirmation, it employs Glymour’s bootstrap confirmation theory, a philosophy-of-science-based method that ensures sets of hypotheses are jointly consistent, supported by evidence, and non-trivial. Experiments across diverse reasoning tasks show that HypoBootstrap significantly reduces hallucinations and improves accuracy over prior LLM-based methods.", "questions": "The main points are covered in the strength and weakness. Some actionable items could be \n1. Experiment with some open source models and reveal if the solution is generalizable. If the open-source models have gap, it would be interesting to see some analysis on the gap. \n2. Some error case study would be interesting, especially the hallucination issues before and after the solution is applied.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength 1: Motivation is clear. The paper starts by clearly identifying a key challenge: LLMs are capable of generating hypotheses in natural language but tend to hallucinate and lack structured guidance during inductive reasoning. Though I have some hypothesis that the inductive reasoning in natural language might already be sufficient in the era of LLM. \n\nStrength 2: The paper uses Glymour’s theory of bootstrap instead of relying on ad hoc metrics, which is unique and principled. The authors adopt a confirmation method rooted in the philosophy of science, ensuring hypotheses are consistent, non-trivial, and jointly supported by evidence.\nStrength 3: The framework is evaluated on four types of inductive reasoning tasks—causal induction, concept learning, grammar learning, and abstract reasoning. HypoBootstrap outperforms strong baselines with a clear performance gain. Code repo is available to publish, which confirms the reproducibility of the paper.\n\nThe weakness of the paper:\nW1: The experiment mainly adopts GPT4, instead of open source models. What are the limitations of open source models? It would be interesting to see such study. \nW2: While ablations are provided for object/relational hypotheses, there is no detailed ablation isolating the impact of Glymour’s bootstrap confirmation. It’s hard to tell how much of the gain comes from confirmation vs generation improvements.\nW3. Some study in failure cases or hallucination types is missing. Though the paper aims to reduce hallucination, it does not provide a systematic analysis of when and how hallucinations still occur, nor categorize failure modes.", "was_revised": false, "final_justification": "", "rebuttal": "# Reviewer vRQJ\n\nWe sincerely thank you for the insightful comments and constructive feedback, which are highly valuable in improving our manuscript.\nWe also appreciate your recognition of the principled foundation, uniqueness, and high effectiveness of our method.\n\nIn the following, we address each of your questions in detail, point by point.\n\n\n> **Question 1**: Experiment with some open source models and reveal if the solution is generalizable. If the open-source models have gap, it would be interesting to see some analysis on the gap.\n>\n> **Weakness 1**: The experiment mainly adopts GPT4, instead of open source models. What are the limitations of open source models? It would be interesting to see such study.\n\n\nWe add experiments on open-source DeepSeek-V3 (671B-A37B).\nDue to the limited time of rebuttal period, we are still running experiments on other LLMs, but the results on DeepSeek-V3 is already convincing to show the generalization ability of our method.\n\n\n\n\n[$T=1$]\n| Method | ACRE (Raw\\Task) | List Fns (Raw\\Task) | MiniSCAN (Raw\\Task) | MiniARC (Raw\\Task) | Avg. (Raw\\Task) |\n| - | - | - | - | - | - |\n| HR | 72.0\\37.0 | 46.4\\36.4 | 25.8\\21.0 | 9.5\\7.7 | 38.4\\25.5 |\n| HB (ours) | **74.5**\\\\**47.0** | **55.6**\\\\**42.0** | **69.8**\\\\**47.0** | **16.7**\\\\**10** | **54.2(+15.8)\\36.5(+11.0)** |\n\n\n\n[$T=3$]\n| Method | ACRE (Raw\\Task) | List Fns (Raw\\Task) | MiniSCAN (Raw\\Task) | MiniARC (Raw\\Task) | Avg. (Raw\\Task) |\n| - | - | - | - | - | - |\n| HR | 77.3\\\\**53.0** | 52.9\\43.6 | 34.3\\24.0 | 11.4\\9.2 | 44.0\\32.5 |\n| HB* (ours) | **77.5**\\\\**53.0** | **62.8**\\\\**48.8** | **74.0**\\\\**53.0** | **21.8**\\\\**14.6** | **59.0(+15.0)\\42.4(+9.9)** |\n\n\nThese results not only show the significant performance gain for $T=1$ (the same for GPT-4), but also show significant performance gain for $T=3$.\nThe robustness and generalization ability across models and datasets are well proven.\n\n\n\n\n\nSee our response to Reviewer tLyE for results of MoC (if it is interesting for you), which we misunderstood as concurrent work but actually not.\n\n\n\n\n\n\n\n> **Question 2**: Some error case study would be interesting, especially the hallucination issues before and after the solution is applied.\n>\n> **Weakness 3**: Some study in failure cases or hallucination types is missing. Though the paper aims to reduce hallucination, it does not provide a systematic analysis of when and how hallucinations still occur, nor categorize failure modes.\n\nDue to the 'black box' nature of current LLMs and the complexity of natural language, it is difficult to identify the patterns and mechanisms behind hallucination. We have uncovered some potential patterns by analyzing the categories of error cases.\n\nWe examined the failure cases and categorized them into distinct failure modes.\nNote: The citation number is the same as in the paper.\n\n**1.ACRE**\nThe reasoning query modes in the ACRE dataset can be categorized into four types[26], including: direct, indirect, screening-off, and backward-blocking. We found several failure cases in backward-blocking reasoning  (i.e., in addition to being a Blicket or non-Blicket, an object’s Blicketness could also be undetermined, which occurs when the object is not directly tested, but can activate the machine together with other potential Blickets).\nAn example is given below:\n\n{\"idx\": 14, train\":[{\"input\": [\"blue rubber cube\"], \"output\": \"off\"}, {\"input\": [\"red rubber cube\"], \"output\": \"on\"}, {\"input\": [\"red rubber cube\", \"blue rubber cube\"], \"output\": \"on\"}, {\"input\": [\"red metal cube\", \"cyan rubber cube\"], \"output\": \"off\"}, {\"input\": [\"gray metal cube\", \"blue rubber sphere\", \"red metal cube\"], \"output\": \"on\"}, {\"input\": [\"yellow rubber cylinder\"], \"output\": \"off\"}], \n\"test\": [{\"input\": [\"cyan rubber cube\"], \"output\": \"off\", \"type\": \"direct\"}, {\"input\": [\"gray metal cube\"], \"output\": \"undetermined\", \"type\": \"potential\"}, {\"input\": [\"blue rubber cube\", \"blue rubber sphere\"], \"output\": \"undetermined\", \"type\": \"potential\"}, {\"input\": [\"blue rubber cube\", \"gray metal cube\"], \"output\": \"undetermined\", \"type\": \"potential\"}]}\n\nGenerated Rule: {\"blue rubber sphere\": \"off\"} \nCorrect Rule: {\"blue rubber sphere\": \"undetermined\"} \n\nHallucinations occur when the model analyzes the fifth task:{\"input\": [\"gray metal cube\", \"blue rubber sphere\", \"red metal cube\"], \"output\": \"on\"}. While the 'blue rubber sphere' is not directly tested, it could potentially activate the machine together with other Blickets (like the 'gray metal cube'). Therefore, the Blicketness of the 'blue rubber sphere' should be 'undetermined'.\n\n**2.List Functions**\nList Functions tasks are organized into different groups [20], containing problems of varying difficulty to test for differences in:\n(1) recursive and non-recursive reasoning\n(2) the ability to handle exceptions\n(3) biases toward elements occurring early in a list.\nWe found that hallucinations occur when dealing with biases toward elements occurring early in a list. An example is given below:\n\n{\"idx\": 84, \"train\": [{\"input\": \"[2, 1, 40, 92, 26, 95, 45]\", \"output\": \"[40]\"}, {\"input\": \"[1, 8]\", \"output\": \"[8]\"}, {\"input\": \"[1, 4, 5, 80, 76, 0, 2, 71, 26]\", \"output\": \"[4]\"}, {\"input\": \"[1, 14, 3, 2, 78, 9]\", \"output\": \"[14]\"}, {\"input\": \"[6, 2, 9, 8, 44, 7, 3]\", \"output\": \"[3]\"}, {\"input\": \"[1, 53]\", \"output\": \"[53]\"}, {\"input\": \"[3, 6, 4, 93, 43]\", \"output\": \"[93]\"}, {\"input\": \"[3, 7, 55, 75, 5, 24, 14, 1]\", \"output\": \"[75]\"}], \"test\": [{\"input\": \"[9, 60, 11, 93, 4, 5, 8, 24, 1, 97]\", \"output\": \"[97]\"}, {\"input\": \"[2, 8, 47, 60, 0, 18, 5, 7, 3]\", \"output\": \"[47]\"}, {\"input\": \"[3, 72, 2, 75, 27, 6, 7, 4]\", \"output\": \"[75]\"}, {\"input\": \"[1, 62, 4]\", \"output\": \"[62]\"}, {\"input\": \"[5, 17, 1, 4, 3, 88, 68]\", \"output\": \"[88]\"}, {\"input\": \"[1, 56, 7, 30]\", \"output\": \"[56]\"}, {\"input\": \"[4, 6, 71, 0, 9, 86, 59, 29, 3]\", \"output\": \"[9]\"}, {\"input\": \"[5, 64, 3, 7, 65, 83, 54, 27, 4]\", \"output\": \"[83]\"}]}\n\nGenerated Rule: The output is the second highest number in the input list.\nCorrect Rule: Remove all but element N + 1, N = element 1.\n\n**3.MiniSCAN**\nTasks of MiniSCAN dataset contain two types of pseudowords, some pseudowords were primitive instructions that correspond to a single output symbol, while other pseudowords are interpreted as functions that need to be applied to arguments to construct the output [11].\nFunctions can be categorized into three types: \n(1) Function 1 takes the preceding primitive as an argument and repeats its output three times; (2) Function 2 takes both the preceding primitive and following primitive as arguments, producing their outputs in a specific alternating sequence. \n(3) Function 3 takes both the preceding and following strings as input, processes them, and concatenates their outputs in reverse order.\nWe found that hallucinations occur when the model extracts rules like Function 3 in cases where its arguments are generated by other functions. For example, the model generates the incorrect rule \"##A spiirth ##B shruw ##C\" → \"##C ##A ##B ##C\" when the correct rule should be \"##A spiirth ##B shruw ##C\" → \"##C ##A ##B ##A.\"\n\n**4.MiniARC**\nAs the most challenging dataset, MiniARC[1] aims to evaluate LLMs' human cognitive priors including:\n(1) Objectness priors (object cohesion, object persistence, and object influence via contact)\n(2) Goal-directedness prior\n(3) Numbers and Counting priors\n(4) Basic Geometry and Topology priors\nWe found that our approach still underperforms on these tasks, as the inputs are limited to text-based matrices. In future work, we will explore whether incorporating visual information through multimodal LLMs could improve performance.\n\n> **Weakness 2**: While ablations are provided for object/relational hypotheses, there is no detailed ablation isolating the impact of Glymour’s bootstrap confirmation. It’s hard to tell how much of the gain comes from confirmation vs generation improvements.\n\nWe also separately study the impact of the confirmation part and the generation part of HypoBootstrap in Section 5.2.\n\n- **Independent impact of the proposed generation in HypoBootstrap**:\nThe variant $\\widetilde{\\textrm{HB}}$ removes all the inconsistency elimination and consistency validation from HypoBootstrap.\nThe only remaining component of the confirmation part is the unit test ($\\texttt{AuxiliaryConfirmer}$), which is also used in HR (the baseline).\n    - For $T=1$, the unit test isn't used at all; hence, the results of $\\widetilde{\\textrm{HB}}$ under $T=1$ in Table 1 shows the independent impact of the proposed generation method. Since $\\widetilde{\\textrm{HB}}$ outperforms HR significantly (+10.4 task accuracy), our proposed generation is very effective.\n    - For $T=3$, $\\widetilde{\\textrm{HB}}$ underperforms HR slightly, showing that the proposed generation is not good enough in multi-turn refinement setup.\n\n- **Impact of the proposed confirmation in HypoBootstrap**:\n    - From $T=1$, the results of HB (add back the inconsistency elimination and consistency validation) and $\\widetilde{\\textrm{HB}}$, show that the consistency related part of our proposed confirmation method is effective (+3.7 task accuracy).\n    - From $T=3$, comparing HR, $\\widetilde{\\textrm{HB}}$, and HB (or HB*), we found the confirmation part is indispensable for our proposed generation method.\n\nWe will further revised the paper to clearly state these observations.\n\n\n\n\n\n\nThank you again for the thorough review. We would be glad to provide any additional information if needed."}, {"review_id": "1aZEqE4Lkq", "reviewer": "Reviewer_tLyE", "summary": "The paper focuses on the task of inductive reasoning, where a model needs to infer a pattern (usually a mapping) from a given set of observations (typically multiple input-output pairs). Inspired by a hypothesis confirmation theory from the philosophy of science, the authors propose an LLM-based framework in which hypotheses of varying granularity are proposed and verified progressively. Specifically, in the first stage, the LLM is asked to describe each input or output individually. Then, it is asked to identify the patterns between a single input-output pair. Finally, the pattern governing all input-output pairs is inferred based on the information from the previous stages. Experiments have shown that the method is effective across various inductive reasoning benchmarks.", "questions": "1. It is important to report the average computation cost for each problem across all datasets considered.\n2. Comparison with MoC would be helpful to show the effectiveness of the method.\n3. It is also valuable to test the proposed method on more LLMs to see the robustness of the method.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n- The paper is mostly clearly written.\n- Leveraging ideas from hypothesis confirmation theory is interesting.\n\n# Weaknesses\n1. It is uncertain whether the comparison with the baseline is fair. It is unclear how many hypotheses the proposed method (HB) and the baseline method (HR) generate. Since the two methods differ significantly in how hypotheses are generated, it is essential to report the computational cost of each method for a given problem. This is a good measurement of the compute needed to solve each problem, considering the context length and the number of generated tokens. But unfortunately, this information is missing in the paper. \n2. The method shows only marginal improvement (1.1% on average) over the simple baseline HR when the number of iterations increases to 3. This suggests that the effectiveness of the proposed algorithm is limited. Given that the method is relatively complex, this also raises questions about its generalizability to other domains or its performance when provided with more compute.\n3. Although MoC was published one month before the NeurIPS submission deadline, its arXiv version appeared in December 2024. According to the policy stated here, this does not qualify as concurrent work. While I believe the methods are significantly different, MoC would still serve as a simpler and strong baseline for comparison.\n\n## Minor Points\n- The experiments section is poorly written:\n\n  - The meaning of T is not defined. It likely refers to the number of refinement rounds, as in previous work (HR), but this is not explained.\n\n  - The number of hypotheses generated for the HR baseline is not reported.", "was_revised": true, "final_justification": "The authors propose an effective method for inductive reasoning using large language models, drawing inspiration from the hypothesis confirmation theory. Extensive experiments have demonstrated the effectiveness of the method on a wide range of inductive reasoning tasks using different large language models.", "rebuttal": "We sincerely thank you for the insightful comments and constructive feedback, which are highly valuable in improving our manuscript.\nWe also appreciate your recognition of the integration of hypothesis confirmation theory and of the demonstrated effectiveness of our method.\n\nIn the following, we address each of your questions in detail, point by point.\n\n\n\n\n> **Weakness Minor Points**:\n> The experiments section is poorly written:\n> - The meaning of T is not defined. It likely refers to the number of refinement rounds, as in previous work (HR), but this is not explained.\n> - The number of hypotheses generated for the HR baseline is not reported.\n\nWe apologize for the unclearness. Here are clarifications.\n- $T$ refers to the number of refinement loop, which is defined in Section 4.3. We will emphasize the meaning of $T$ in experiments section.\n- The number of hypotheses generated for the HR is 1 per refinement iteration. In Section 5.1, we claim that *we only allow one functional hypothesis to be generated per iteration*. In the revised paper, we will clarify that this is also valid for baselines.\n\n\n\n\n> **Question 2**: Comparison with MoC would be helpful to show the effectiveness of the method.\n> \n> **Weakness 3**: Although MoC was published one month before the NeurIPS submission deadline, its arXiv version appeared in December 2024. According to the policy stated here, this does not qualify as concurrent work. While I believe the methods are significantly different, MoC would still serve as a simpler and strong baseline for comparison.\n>\n> **Question 3**: It is also valuable to test the proposed method on more LLMs to see the robustness of the method.\n\nThanks for pointing out our misunderstanding of the definition of concurrent work.\nWe add experiments on MoC.\nIn addition, also according to your suggestion, we also add experiments on DeepSeek-V3 (671B-A37B). Due to the limited time of rebuttal period, we are still running experiments on other LLMs, but the results on DeepSeek-V3 is already convincing to show the robustness of our method.\n\nThe results are shown below.\nTo conduct fair comparison, we let all methods generate the same number of functional hypotheses (i.e., set $K=3$ for MoC).\nThis number is linearly correlated with the number of required unit tests in all methods, which is an important limitation in some real-world applications (see our response to your Weakness 2).\n\n\n\n\n[GPT-4; 1 hypothesis]\n| Method | ACRE (Raw\\Task) | List Fns (Raw\\Task) | MiniSCAN (Raw\\Task) | MiniARC (Raw\\Task) | Avg. (Raw\\Task) |\n| - | - | - | - | - | - |\n| HR | 78.2\\45.0 | 51.6\\42.4 | 77.0\\46.0 | 5.9\\3.8 | 53.2\\34.3 |\n| MoC | 74.0\\34.0 | 53.0\\43.2 | 60.4\\21.0 | 6.7\\4.6 | 48.5(-4.7)\\25.7(-8.6) |\n| HB (ours) | **84.8**\\\\**57.0** | **57.9**\\\\**47.6** | **94.9**\\\\**82.0** | **11.0**\\\\**6.9** | **62.1(+8.9)**\\\\**48.4(+14.1)** |\n\n\n\n[GPT-4; 3 hypotheses]\n| Method | ACRE (Raw\\Task) | List Fns (Raw\\Task) | MiniSCAN (Raw\\Task) | MiniARC (Raw\\Task) | Avg. (Raw\\Task) |\n| - | - | - | - | - | - |\n| HR | 77.8\\47.0 | 61.7\\52.8 | **98.2**\\\\**95.0** | 10.1\\6.9 | 62.0\\50.4 |\n| MoC | 73.0\\39.0 | 61.9\\50.8 | 87.3\\64.0 | 10.8\\\\**8.5** | 58.3(-3.7)\\40.6(-9.8) |\n| HB* (ours) | **79.0**\\\\**55.0** | **64.3**\\\\**53.6** | **98.2**\\89.0 | **12.1**\\\\**8.5** | **63.4(+1.4)\\51.5(+1.1)** |\n\n\n\n\n[DeepSeek-V3; 1 hypothesis]\n| Method | ACRE (Raw\\Task) | List Fns (Raw\\Task) | MiniSCAN (Raw\\Task) | MiniARC (Raw\\Task) | Avg. (Raw\\Task) |\n| - | - | - | - | - | - |\n| HR | 72.0\\37.0 | 46.4\\36.4 | 25.8\\21.0 | 9.5\\7.7 | 38.4\\25.5 |\n| MoC | 70.5\\34.0 | 52.5\\40.8 | 30.2\\11.0 | 9.2\\6.2 | 40.6(+2.2)\\23.0(-2.5) |\n| HB (ours) | **74.5**\\\\**47.0** | **55.6**\\\\**42.0** | **69.8**\\\\**47.0** | **16.7**\\\\**10** | **54.2(+15.8)\\36.5(+11.0)** |\n\n\n\n[DeepSeek-V3; 3 hypotheses]\n| Method | ACRE (Raw\\Task) | List Fns (Raw\\Task) | MiniSCAN (Raw\\Task) | MiniARC (Raw\\Task) | Avg. (Raw\\Task) |\n| - | - | - | - | - | - |\n| HR | 77.3\\\\**53.0** | 52.9\\43.6 | 34.3\\24.0 | 11.4\\9.2 | 44.0\\32.5 |\n| MoC | 73.5\\43.0 | 59.2\\48.4 | 71.9\\51.0 | 14.9\\11.5 | 54.9(+10.9)\\38.5(+6.0) |\n| HB* (ours) | **77.5**\\\\**53.0** | **62.8**\\\\**48.8** | **74.0**\\\\**53.0** | **21.8**\\\\**14.6** | **59.0(+15.0)\\42.4(+9.9)** |\n\n\nWe can draw the following additional conclusions:\n- **Our framework significantly outperforms HR and MoC**: Compared to HR, HB achieves an average improvement in task accuracy across four datasets of: +14.1 (GPT-4; 1 hypo), +11.0 (DeepSeek-V3; 1 hypo), +9.9 (DeepSeek-V3; 3 hypo).\nCompared to MoC, HB's improvement is much higher on GPT-4 and [DeepSeek-V3; 1 hypo]. For [DeepSeek-V3; 3 hypos], HB also outperforms MoC about 4 percentage points.\n- **Our framework is robust across datasets and models**: Among all the results above, HB achieves the best performance across all datasets and models for both raw accuracy and task accuracy, except for GPT-4's task accuracy on MiniSCAN.\n- **MoC is unstable across tasks and models**: It is interesting to see the unstable performance of MoC. Since MoC did not release its code, we implement it according to its paper published in NAACL 2025. On ListFns and MiniARC, which are also used in MoC's paper, MoC outperforms HR or achieves comparable performance, which is inline with the results in their paper. However, On ACRE and MiniSCAN, which are not used in MoC's paper, MoC shows very unstable performance across tasks and models. Sometimes MoC is even significantly worse than HR.\n    - ACRE and MiniSCAN all require generating hypotheses under a specific format and the correct hypotheses are usually highly composite, i.e., the output (in evidence) must be obtained by applying the rules in combination. This significantly differs from ListFns and MiniARC, which makes ACRE and MiniSCAN a better benckmark.\n    - **The comparison between the robust performance of our framework and the unstable performance of MoC reveals the benefit of our framework.**\n\n\n\n\n\n\n\n\n\n\n\n> **Question 1**: It is important to report the average computation cost for each problem across all datasets considered.\n>\n> **Weakness 1**: It is uncertain whether the comparison with the baseline is fair. It is unclear how many hypotheses the proposed method (HB) and the baseline method (HR) generate. Since the two methods differ significantly in how hypotheses are generated, it is essential to report the computational cost of each method for a given problem. This is a good measurement of the compute needed to solve each problem, considering the context length and the number of generated tokens. But unfortunately, this information is missing in the paper.\n\nHere is the token consumed by all methods.\n(It's slightly simpler to count tokens on DeepSeek-V3, considering the limited rebuttal period.)\n\n\n\n[DeepSeek-V3; 1 hypothesis] (P for prompt tokens; C for completion tokens)\n| Method | ACRE (P\\C) | List Fns (P\\C) | MiniSCAN (P\\C) | MiniARC (P\\C) |\n| - | - | - | - | - |\n| HR | 22\\6 | 103\\120 | 43\\29 | 99\\215 |\n| MoC | 46\\24 | 209\\67 | 80\\72 | 152\\50 |\n| HB (ours) | 400\\69 | 2409\\633 | 1201\\245 | 991\\283 |\n\n\n\n[DeepSeek-V3; 3 hypotheses]\n| Method | ACRE (P\\C) | List Fns (P\\C) | MiniSCAN (P\\C) | MiniARC (P\\C) |\n| - | - | - | - | - |\n| HR | 56\\10 | 477\\352 | 331\\77 | 565\\532 |\n| MoC | 98\\23 | 439\\81 | 177\\142 | 317\\90 |\n| HB* (ours) | 572\\82 | 3121\\745 | 1642\\305 | 1616\\407 |\n\n\n\nOur framework does consume about 1~10 times tokens than baseline methods.\nHowever, this is not surprising since\n- **Our framework is much more robust across models and datasets than baselines**, as demonstrated above.\n- **It is common that the models performing better on any reasoning tasks typically consume much more tokens.** Similar conclusions can be confirmed in the literatures on Math reasoning, code generation, etc.\n- **Our framework has decreasing marginal cost in terms of refinement iterations**, since only $H_F$ is regenerated in each refinement loop and the agents related to $H_O, H_R$ only use once at the very beginning.\n\n\n\n\nIn addition, **the experimental comparison is fair since the number of functional hypotheses generated by HR, MoC, and our framework are the same**, as stated above.\nAn important reason for ensuring same number of functional hypotheses is that, for all methods reported, this number is linearly correlated with the number of required unit tests in all methods, which is an important limitation in some real-world applications (see our response to your Weakness 2).\n\n\n\n\n\n\n\n\n\n> **Weakness 2**: The method shows only marginal improvement (1.1% on average) over the simple baseline HR when the number of iterations increases to 3. This suggests that the effectiveness of the proposed algorithm is limited. Given that the method is relatively complex, this also raises questions about its generalizability to other domains or its performance when provided with more compute.\n\nAccording to our responses above, we can draw the following conclusions.\n- **Our framework shows significant performance gain on DeepSeek-V3 for $T=3$**, revealing that the relative small gain on GPT-4 may be model-related.\n- **Performance on $T=1$ is more important in the sense that no hypothesis verification is required.** As pointed out by Reviewer 6r5u, the unit test, which is used as hypothesis verification in all methods reported in the paper, is not always available in real-world applications. In scientific research, which is one of the largest applications of inductive reasoning, hypothesis verification is also costly due to expensive experimental instruments. Hence, generating high-quality hypothesis without verification is a critical ability. We apologize for not emphasizing this importance in the paper and will revised the paper accordingly.\n\n\n\n\n\n\n\n\n\n\n\nThank you again for the thorough review. We would be glad to provide any additional information if needed."}, {"review_id": "OzDNg0kLbZ", "reviewer": "Reviewer_38v8", "summary": "The paper provides a formal logic description of a pipeline of LLMs for inductive reasoning - finding the underlying patterns given examples.\nThe LLMs first generate hypotheses, then verify them themselves.\nHypotheses are first generated for each input object, then for each input pair, finally for all the pairs together.\nEvaluation is performed on 4 inductive reasoning datasets.", "questions": "What do you mean by \"We use the 100 tasks filtered by HR\"?", "limitations": "Limitations are not discussed. The paper only mentions that the proposed method \"cannot completely solve LLM’s hallucination problem\".", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The pipeline of iterative hypothesis generation and self-checking seems to be a good idea.\nResults show improvements over the chosen baseline.\n\nThe paper presentation is quite difficult to follow. \nEverything is presented in an overly complex formal logic representation and notation, while the system is just a sequence of prompted LLMs and there is no way to guarantee that the LLM outputs actually follow that formal logic.\nThe structure of the paper could also be improved. \n\nWhile MoC is claimed to be concurrent work, it has been on arxiv since December 2024. NeurIPS guidelines specify that appearing on arxiv more than 2 months earlier does not count as concurrect work. Therefore a more thorough comparison should be performed to that work.", "was_revised": true, "final_justification": "Thank you for the response. It's great that you provided the additional results with MoC. I think it's important to also highlight the differences and contributions in the main text. The clarity aspect is more difficult to assess based on the response, a new version of the paper would be needed. My score already leans positive and I will keep it the same.", "rebuttal": "# Reviewer 38v8\n\nWe sincerely thank you for the insightful comments and constructive feedback, which are highly valuable in improving our manuscript.\nWe also appreciate your recognition of our idea and of the experimental improvement of our method.\n\nIn the following, we address each of your questions in detail, point by point.\n\n\n\n\n> **Question**: What do you mean by \"We use the 100 tasks filtered by HR\"?\n\nTo conduct fair comparison, our experimental setup, including datasets, exactly follows Hypothesis Refinement (HR) [19].\nIn the Appendix A of HR, HR's author mentioned that they filtered a evaluation subset from the original SCAN dataset [11] via some heuristic methods.\nThe filtered evaluation subset has 100 SCAN tasks.\n\nWe will further clarify this in the revised paper.\n\n\n\n> The paper presentation is quite difficult to follow. Everything is presented in an overly complex formal logic representation and notation, while the system is just a sequence of prompted LLMs and there is no way to guarantee that the LLM outputs actually follow that formal logic. The structure of the paper could also be improved.\n\nThanks for pointing out the clarity issue and the important limitation that LLMs cannot strictly adhere to formal logic.\n- For the clarity issue, we will try our best to revised the structure of the paper and make the description of our method easier to follow.\n- For the limitation, we acknowledge that LLMs cannot strictly adhere to formal logic, and that is one of the major reasons why our method cannot achieve 100% accuracy. In Section 6 and Appendix F, we already acknowledge that our method cannot completely tackles LLM's hallucination in inductive reasoning. We will further emphasize the limitation you pointed out in the revised paper.\n\nIn addition, although formalizing our method in formal logic increases difficulty in understanding, it has several advantages:\n\n1. Our proposed hypothesis confirmation method comes from a theoretical principle from philosophy of science. i.e., Glymour's theory of bootstrap confirmation, **which is originally rigorously discussed in formal logic**. This principle confirmation method makes our method theoretically-grounded and not simply heuristic.\nMoreover, our proposed hypothesis generation method comes from the observation of the difference between natural language and formal language.\nHence, formalizing our method in formal logic makes sense.\n2. **Formalizing inductive reasoning in formal logic is common in AI** (i.e., Inductive Logic Programming; see Line 29-35) **and philosophy of science**. It is easier for researchers from other domains related to inductive reasoning to understand our method.\n\nWe will try our best to make our paper easier to follow for broader audience.\n\n\n\n> While MoC is claimed to be concurrent work, it has been on arxiv since December 2024. NeurIPS guidelines specify that appearing on arxiv more than 2 months earlier does not count as concurrect work. Therefore a more thorough comparison should be performed to that work.\n\nThanks for pointing out our misunderstanding of the definition of concurrent work.\nWe add experiments on MoC.\n\n\n\n[GPT-4; $T=1$]\n| Method | ACRE (Raw\\Task) | List Fns (Raw\\Task) | MiniSCAN (Raw\\Task) | MiniARC (Raw\\Task) | Avg. (Raw\\Task) |\n| - | - | - | - | - | - |\n| HR | 78.2\\45.0 | 51.6\\42.4 | 77.0\\46.0 | 5.9\\3.8 | 53.2\\34.3 |\n| MoC | 74.0\\34.0 | 53.0\\43.2 | 60.4\\21.0 | 6.7\\4.6 | 48.5(-4.7)\\25.7(-8.6) |\n| HB (ours) | **84.8**\\\\**57.0** | **57.9**\\\\**47.6** | **94.9**\\\\**82.0** | **11.0**\\\\**6.9** | **62.1(+8.9)**\\\\**48.4(+14.1)** |\n\n\n\n[GPT-4; $T=3$]\n| Method | ACRE (Raw\\Task) | List Fns (Raw\\Task) | MiniSCAN (Raw\\Task) | MiniARC (Raw\\Task) | Avg. (Raw\\Task) |\n| - | - | - | - | - | - |\n| HR | 77.8\\47.0 | 61.7\\52.8 | **98.2**\\\\**95.0** | 10.1\\6.9 | 62.0\\50.4 |\n| MoC | 73.0\\39.0 | 61.9\\50.8 | 87.3\\64.0 | 10.8\\\\**8.5** | 58.3(-3.7)\\40.6(-9.8) |\n| HB* (ours) | **79.0**\\\\**55.0** | **64.3**\\\\**53.6** | **98.2**\\89.0 | **12.1**\\\\**8.5** | **63.4(+1.4)\\51.5(+1.1)** |\n\n\nIt is interesting to see the unstable performance of MoC.\nSince MoC did not release its code, we implement it according to its paper published in NAACL 2025.\nOn ListFns and MiniARC, which are also used in MoC's paper, MoC outperforms HR or has comparable performance, which is inline with the results in their paper.\nHowever, On ACRE and MiniSCAN, which are not used in MoC's paper, MoC show very unstable performance across tasks and models. Sometimes MoC is even significantly worse than HR.\n- ACRE and MiniSCAN all require generating hypotheses under a specific format and the correct hypotheses are usually highly composite, i.e., the output (in evidence) must be obtained by applying the rules in combination. This significantly differs from ListFns and MiniARC, which makes ACRE and MiniSCAN a better benckmark.\n- **The comparison between the robust performance of our framework and the unstable performance of MoC reveals the benefit of our framework.**\n\n\n\nSee our response to Reviewer tLyE for results on DeepSeek-V3, If this is of interest to you.\nMoC outperforms HR but underperforms our framework on DeepSeek-V3.\n\n\n\n\n> **Limitations**: Limitations are not discussed. The paper only mentions that the proposed method \"cannot completely solve LLM’s hallucination problem\".\n\nThanks for pointing out this limitation.\nHere are limitations summarized from our new observation and other reviewers.\n1. Our framework (and all baselines) requires a way to verify generated hypotheses, e.g., unit tests.\nHowever, unit test is not always available in real-world scenario, such as open-ended text-based induction.\nIn scientific research, which is one of the largest applications of inductive reasoning, hypothesis verification is costly due to expensive experimental instruments.\n2. Our framework consumes more tokens than baselines.\n    - However, the robust performance across tasks and models deserves more tokens. (see our response to Reviewer tLyE for detailed discussion if this is of interest to you)\n\nWe will emphasize these limitations in the revised paper.\n\n\n\n\nThank you again for the thorough review. We would be glad to provide any additional information if needed."}, {"review_id": "oQpF8ZNlz1", "reviewer": "Reviewer_6r5u", "summary": "This paper addresses the challenge of enhancing the inductive reasoning capabilities of Large Language Models (LLMs) while mitigating their propensity for hallucination. To this end, the authors propose HypoBootstrap, a novel framework that employs a structured, multi-stage process to generate and validate hypotheses. The generation process is progressive, beginning with the identification of simple Object Hypotheses concerning individual data points, followed by the establishment of Relational Hypotheses (HR​) between input-output pairs, and culminating in the general Functional Hypothesis (HF​). This pipeline is tightly integrated with a bootstrap confirmation mechanism, an adaptation of Glymour's confirmation theory from the philosophy of science. The role of this mechanism is to validate the entire set of hypotheses for logical consistency and correctness, for instance, by eliminating inconsistent statements and programmatically verifying the final hypothesis against the evidence. The authors provide a comprehensive empirical evaluation on four diverse inductive reasoning benchmarks, where HypoBootstrap is shown to significantly outperform existing methods.", "questions": "1. The framework seems to have a high computational overhead. Can you provide a more direct quantitative comparison of this cost against the baseline and discuss how the approach is expected to scale? \n\n2. How robust is the framework to flawed or uninformative intermediate hypotheses (HO,HR)? Since the refinement loop only targets HF​, are there cases where feedback should instead revise the more foundational HO or HR ?", "limitations": "1. Quantify computational and environmental costs.\nWhile Appendix D.1 reports per-module resource usage, the paper lacks a holistic comparison of total LLM calls, wall-clock time, API cost against baseline methods—and no analysis of how this overhead grows with dataset size or input length.\n\n2. Clarify the framework’s scope and generality. \nHypoBootstrap rests on converting hypotheses into executable unit tests. The authors should explicitly state this dependency and discuss its applicability (or breakdown) when such tests cannot be constructed—for example, in open-ended text-only inductive tasks—so readers understand where the method may fail or need adaptation.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Positive Aspects: \n1. The work's primary strength is its originality in adapting Glymour's bootstrap confirmation theory from the philosophy of science to guide inductive reasoning in LLMs. This provides a novel, theoretically-grounded alternative to purely heuristic methods.\n2. The proposed framework is technically robust. The multi-stage generation process is a logical and effective way to decompose the reasoning task. Furthermore, the experimental validation is rigorous, demonstrating significant performance gains on four diverse benchmarks and using insightful ablation studies to confirm the contribution of each component.\n3. The paper is clearly written and well-presented. The authors effectively use figures to illustrate the framework's complex architecture and provide a compelling case study of its practical advantages.\nWeaknesses：\n1. The empirical results are reported without error bars or statistical significance tests. The authors note this is due to the high cost of replicating experiments on GPT-4.\n2. The framework's multi-step process, involving numerous LLM agent calls, likely incurs a significant computational overhead compared to baseline methods. This trade-off is not explicitly analyzed in the main paper.\n3. There is a minor inaccuracy in the submitted checklist regarding the declaration of LLM usage , which should be corrected to reflect that LLMs are a core component of the methodology.", "was_revised": true, "final_justification": "We thank the authors for their response, which we have carefully reviewed.  With my main concerns addressed, I’ll maintain my evaluation.", "rebuttal": "# Reviewer 6r5u\n\nWe sincerely thank you for the insightful comments and constructive feedback, which are highly valuable in improving our manuscript.\nWe also appreciate your recognition of the novelty, principled foundation, technically robustness, and high effectiveness of our method.\n\n\nIn the following, we address each of your questions in detail, point by point.\n\n\n> **Question 1**: The framework seems to have a high computational overhead. Can you provide a more direct quantitative comparison of this cost against the baseline and discuss how the approach is expected to scale?\n>\n> **Weakness 5**: The framework's multi-step process, involving numerous LLM agent calls, likely incurs a significant computational overhead compared to baseline methods. This trade-off is not explicitly analyzed in the main paper.\n>\n> **Limitation 1**: Quantify computational and environmental costs. While Appendix D.1 reports per-module resource usage, the paper lacks a holistic comparison of total LLM calls, wall-clock time, API cost against baseline methods—and no analysis of how this overhead grows with dataset size or input length.\n\nA fair way to compare the computational overhead for LLMs is the token consumption.\nHere is the number of tokens consumed by all methods.\n(It's slightly simpler to count tokens on DeepSeek-V3, considering the limited rebuttal period. And other reviewers requeust us to conduct experiments on open-source models)\n\n\n\n[DeepSeek-V3; $T=1$] (P for prompt tokens; C for completion tokens)\n| Method | ACRE (P\\C) | List Fns (P\\C) | MiniSCAN (P\\C) | MiniARC (P\\C) |\n| - | - | - | - | - |\n| HR | 22\\6 | 103\\120 | 43\\29 | 99\\215 |\n| MoC | 46\\24 | 209\\67 | 80\\72 | 152\\50 |\n| HB (ours) | 400\\69 | 2409\\633 | 1201\\245 | 991\\283 |\n\n\n\n[DeepSeek-V3; $T=3$]\n| Method | ACRE (P\\C) | List Fns (P\\C) | MiniSCAN (P\\C) | MiniARC (P\\C) |\n| - | - | - | - | - |\n| HR | 56\\10 | 477\\352 | 331\\77 | 565\\532 |\n| MoC | 98\\23 | 439\\81 | 177\\142 | 317\\90 |\n| HB* (ours) | 572\\82 | 3121\\745 | 1642\\305 | 1616\\407 |\n\n\n\nOur framework does consume about 1~10 times tokens than baseline methods.\nHowever, this is not surprising since\n- **Our framework is much more robust across models and datasets than baselines**. See our response to Reviewer tLyE for results on DeepSeek-V3 and MoC (if this is of interest to you), which show HypoBoostrap's robustness across datasets and models.\n- **It is common that the models performing better on any reasoning tasks typically consume much more tokens.** Similar conclusions can be confirmed in the literatures on Math reasoning, code generation, etc.\n- **Our framework has decreasing marginal cost in terms of refinement iterations**, since only $H_F$ is regenerated in each refinement loop and the agents related to $H_O, H_R$ only use once at the very beginning. Note that HR consumes more tokens than us on MiniARC for $T=3$.\n\n\n\n\n\n> **Question 2**: How robust is the framework to flawed or uninformative intermediate hypotheses (HO,HR)? Since the refinement loop only targets HF​, are there cases where feedback should instead revise the more foundational HO or HR ?\n\nOur framework may indeed produce flawed or uninformative intermediate HO \\& HR due to LLMs' hallucination, but $\\texttt{ObjectHypothesisInconsistencyEliminator}$ and $\\texttt{RelationalHypothesisInconsistencyEliminator}$ can eliminate several flawed HO \\& HR (see Figure 3 for examples).\nWe choose to eliminating them, instead of refining them, to reduce costs.\nHowever, the remaining flawed and uninformative HO \\& HR indeed may still exist.\n\nTo further improve these *bad* HO \\& HR, we may have two choices, each of which has drawbacks.\n1. If refining them in the current refinement loop, HO \\& HR has no explicit optimizing signal since the feedback (passing the unit test or not) does not directly tell which HO \\& HR are *bad* and how bad they are.\n2. If refining them by simply ask LLMs to refine them, we also don't know how to tell which are *good* HO \\& HR.\n\nHence, it may be an interesting future work to enhance these intermediate hypotheses.\n\n\n\n\n\n> **Weakness 4**: The empirical results are reported without error bars or statistical significance tests. The authors note this is due to the high cost of replicating experiments on GPT-4.\n\nWe apologize for the oversight in omitting the reporting of the decoding temperature of the LLMs.\nIn our framework, all the LLMs using temperature 0 for generating texts.\nTherefore, **there is no randomness in our framework**.\nError bars or statistical significance are thus unnecessary.\n\nOur small-scale preliminary experiments also show that our framework is stable across repeated experiments, and that baselines have little variance in experimental results.\n\n\n\n\n\n> **Weakness 6**: There is a minor inaccuracy in the submitted checklist regarding the declaration of LLM usage , which should be corrected to reflect that LLMs are a core component of the methodology.\n\nThanks for pointing out our negligence in the checklist. We will correct it according to your suggestion.\n\n\n\n> **Limitation 2**: Clarify the framework’s scope and generality. HypoBootstrap rests on converting hypotheses into executable unit tests. The authors should explicitly state this dependency and discuss its applicability (or breakdown) when such tests cannot be constructed—for example, in open-ended text-only inductive tasks—so readers understand where the method may fail or need adaptation.\n\nThanks for pointing out this important limitation.\nOur framework (and all the baselines) leverages unit tests, or more generally, a way to verify the correctness of the proposed hypotheses.\nIn some real-world scenarios of inductive reasoning, such as scientific research (which is one of the largest applications of inductive reasoning), verifying the correctness of hypotheses can be costly due to the expensive experimental instruments.\nAnd, as you mentioned, in open-ended text-only inductive tasks, unit tests may be impossible.\n\nWe will thoroughly discuss this limitation in the revised paper.\n\n\n\n\n\n\n\nThank you again for the thorough review. We would be glad to provide any additional information if needed."}]

Author rebuttal:

Meta-review: The paper presents a novel approach that adapts Glymour's bootstrap confirmation theory to guide inductive reasoning in LLMs. All reviewers are positive with the originality, technically robustness, rigorous experimental validation, and clear presentation.
Decision: Accept (poster)