From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Models
Authors: Kaiyu He
Zhiyu Chen
Venue: Accepted by TMLR ()

Abstract: Since the advent of Large Language Models (LLMs), efforts have largely focused on improving their instruction-following and deductive reasoning abilities, leaving open the question of whether these models can truly discover new knowledge. In pursuit of artificial general intelligence (AGI), there is a growing need for models that not only execute commands or retrieve information but also learn, reason, and generate new knowledge by formulating novel hypotheses and theories that deepen our understanding of the world. Guided by Peirce's framework of abduction, deduction, and induction, this survey offers a structured lens to examine LLM-based hypothesis discovery. We synthesize existing work in hypothesis generation, application, and validation, identifying both key achievements and critical gaps. By unifying these threads, we illuminate how LLMs might evolve from mere ``information executors'' into engines of genuine innovation, potentially transforming research, science, and real-world problem solving.

Published in Transactions on Machine Learning Research (04/2025)
From Reasoning to Learning: A Survey on Hypothesis Dis- covery and Rule Learning with Large Language Models
Kaiyu He
koigu.he@rtdallls.eedd
Deportment of Computer Science University of Teros Dallas
Zhiyu Chen
Zhiyu. Chen2@eudallas. edu
Department of Computer Science University of Teras at Dallas
Reviewed on OpenReview: openrevvee.....ww....mme net/ net/forum? id= d7W38UzUg0
Abstract
Since the advent of Large Language Models (LLMs), efforts have largely focused on improv- ing their -ottuuttoo-followinng and deductive reasoning abilitics, leaving open the question of whether these models can truly discover new knowledge. In pursuit of artificial general intelligence (AGT), there is growing need for models that not only execute commands or retrieve information but also learn, reason, and generate new knowledge by formulat- ing novel hypotheses and theories that deepen understanding of the world. Guided by Peirce's framework of abduction, deduction, and induction, this survey offers a structured lens to examine LLM-based hypothesis discovery. We synthesize existing work in hypothesis generation, application, and validation, identifying both key achievements and critical gaps. By unifying these threads, we illuminate how LLMs might evolve from mere "information exccutors" into engines of genuine innovation, potentially transforming rescarch, science, and world problem solving.
1 Introduction
One major pillar of human intelligence is the capacity to discover hypotheses and learning rules. We call this capability hypothesss discovery (or rule learning)- Earlier AT systems struggled with because formal symbolic methods lacked the commonsense background needed for inventive rule formation (Yu et al.] 2024a). Recent advances in natural language processing (NLP) have produced LLMs pretrained extensive text corpora that embed substantial commonsense knowledge. These models now enable tasks that demand rich background knowledge, such as formulating new hypotheses and deriving novel conclusions.
Hypothesis discovery inherently relies on a blend of reasoning that includes abduction, induction, and de- duction, each defined differently by various scholars. For instance, Gilbert H. Harman considers induction be a special case of abduction, describing as "inference to the best explanation" " (IBE) (Harman) 1965; Douven 2021). However, while this definition is easy understand, it oversimplifies key aspects of hypoth- esis discovery. In particular, the notion of the "best" explanation is ambiguous and often requires additional assumptions that vary by context. Moreover, this framework does not fully capture real-world scenarios, where a "best" explanation is rarely reached immediately; rather, we continually experiment, gather new observations, and refine our hypotheses. Based on these considerations, we adopt Charles Peirce's defini- tion of hypothesis discovery and reasoning, which posits that hypothesis discovery begins with forming an explanatory hypothesis explain observations through abduction, proceeds with itcratively apply hypoth- esis to solve problem or derive new knowledge with deduction, and validate hypothesis through induction
[Frankfurt] 1958
Peirce 1974
Burks 1946.
Minnameier 2004) (See explanation in Figure 2

---

Published in Transactions on Machine Learning Research (04/2025)
Pronpt Method
Wingnelle es
2072
2021
Yann et
Natural Language Hyputbesis
RAG Mothhed
Hu es al.
2024
Sang st ml
2005
40244
Kion at ul.
3024
Clu
2124
blathod L1
Tumes suppocted
Zhto
2024
al.
2024
Pu al.
poas
Ferual observation
Young et
Nauyyn et
Formal Language Hypothons
W; NL. ulserraaiim
Cheng <t m)
A/21
Wane et All
(qm et al.
2014
Norva dt
Hypotlaosi
Generaztion
Prodcction Based
Siala
20.0
Woston
20.5
Balegur al.
2/24
Abvitoation
Natural Tangrage Hypothecks
Gromel Truth Based
Yong ct
2/24o
12 ct
2024
2025
rang
Humua Evaluation
Zhan ati a
Yang et al
2i Hu et al
Q et all
Evsimitin
Tang
3325
Predection Based
Chvllie
2019
Formall Langunge Hypotireex
Gronnd Truth Based
Bowen ei
2014
His es al.
2026
Young et al.
2022
Rossoting for Hyypeshesis Diseovery
LLM as ontobormallzer
Fun et al.
2023 Olanzoon
2024
Kabyanpur et al.
2024
Natural Language Hypothesia
Flue-Tuee Based
Sun
Mltthodi
Prompt Desed
Hypathoesis Appllcation Dediection
Formal Langrigge Hypothons
N/A
Natural Longuage Hypottacss
Prodiction Based
Yi
2024h
Emaluntion
Formal Language Dypothons
N/A
Prompt-Hoow Hraee
aumpisson nl.
2022 Stn
2024
Natural Longuage Hypothacsa
Mathodi
Fine-Tum Resssl
Zhao
2023
2011
Typothosts Validitioon w Fnwhnntion g/u
Formal Lengrage Hypotheess
Promyt- Baard
Te &
2004
Trun:Pabar Rossd
Lampizam et al. 2022
Rudinge et al.
20200
Zhaung et al.
2025
Natural Language Hypothorss
Emaluntion S 6
Selection Road
Blagaattil
20:00 Jisng
2003
Del & Pickni
2023
Furnul Languagy Hyputlasis
Grounel Trush Basel
Hr
2011
Prosion Hepeothoai Diccorry
et al.
Ahora et
2021
Eutioe
[ypothoots
Proartive Hlppothesis Dexoverr
7.0
Li al
2000
X021
Lung mi al
20222 Stall &e Sieelin
Dacovery
Hypothoois Diaowarr
Worki Simmlation
2022
2013
Figure 1: Taxonomy for Hypothesis Discovery with LLMs. Our survey categorizes work into four topics based Peirce's ' definition of hypothesis discovery: Generation (creating hypotheses that explain given observations with ab- duction), Application (deducing new observations from established hypotheses with deduction), Validation (verifying and refining hypotheses against new evidence with induction), and Integrated Hypothesis Discovery (examining the dynamic interdependencies among these components in a continuous, iterative process).
The rest of the survey is organized as follows. Section 2 presents background knowledge on hypothesis discov- ery using LLMs, including different forms of reasoning and representations involved in the process. Section 3 examines prior surveys on LLM reasoning and hypothesis discovery, highlighting their narrow emphasis on deductive tasks or application-speeift methods. Section reviews methods for forming hypotheses (Abduc-
tion). Section 5 then covers approaches for applying these hypotheses (Deduction). and Section 0 focuses
explores the entire hypothesis-discover cycle by examining the interdependencies among these reasoning steps and showing how abduction, deduction, and induction can be iteratively used to refine more robust hypotheses. For cach stage, we discuss methods, benchmarks, evaluations, and identify limitations and future directions. A high-level taxonomy guiding this survey is shown in Figure]
techniques for validating given hypotheses with new observations (Induction). Finally, Section 7
2 Background
Before LLMs, most AI systems stored knowledge as handcrafted symbols and rules. This paradigm originated in the 1960s, in which the DENDRAL (Lindsay et al. 1993) system pioncered expert deductive reasoning in chemistry but required scientists to manually encode their knowledge into computer-readabin rules. Following
2

---

Published in Transactions on Machine Learning Research (04/2025)
a)
Abduction:
2
Deduction: Apply hypothesis/ Conduct experiments to derive new observations
b)
During every luner eclipee, the shadow cast Earnannnn the Moon Is grdee..
Generate hypotheses that can explain the observations
Generate plausible hypothesis that axplan this observation (Abduction)
Earth vs8
3
All Possible Hypotheses
New observations
Keep sailing sraight acrozs the ocean (Deduction)
Observations
Eventually back
Induction:Vallddtee hypotheses by updating their confidence with observations
Weaken
more Ikkly Stargthen sphere (Induction)
Heraaion 2
hypothesis with obeervations (Abduction)
Update Obsarvations and Continue 200
On the left-hand side, a) illustrates Peirce's framework for hypothesis discovery through abduction, deduction, and induction. The process begins with abduction, which generales explanatory hypotheses based an initial set of observations. Deduction then used to apply these hypotheses and derive predictions. Induction evaluates how well the predicted observations align with actual outcomes, updating the confidence of the hypotheses or rejecting those that are no longer valid. This process is iterative: validated hypotheses may be refined through further rounds of abduction using updated observations, gradually leading to more robust theories. On the right-hand side, b) provides a simple example that illustrates this process.
Figure 2:
programs like BACON (Laggley 1977 and Inductive Logic Programming (Muggleton 1991 were limited and obstacled by this fundamental knowledge representation problem. That format works well for deduction, because most of the problems we need to solve with symbolic Al systems work with limited premises and countable task-specific knowledge; even in the most recent work, for example, questions in the ProofWriter (Tafford et al. 2021) and FOLIO (Han et al. 2024) benchmarks are limited to fewer than a hundred premises. However, abduction and induction are different: they call for generating and validating many tentative explanations inspired by vast commonsense or expert domain knowledge (such weather patterns, social norms, or physics) and for updating beliefs new observations arrive. Handling these reasoning tasks with symbolic Al meant writing and maintaining vast, interlocking rule bases, effort so costly that few projects moved beyond toy domains (Yang et al. 2024ch. Consequently, the research landscape remained dominated
by deductive tasks (Yu et al. 2024a Liu et al. 2025 Huang & Chang 2023).
Later in the 2000s, when probabilistic models took charge, the Bayesian model seemed a better solution compared with symbolic AI (Pearl, 2009). However, due the limitations of computational power, model structure, and the training data use, small-scale probabilistic models relied heavily on human-abstracted clean features to build models. Without the main step of feature engineering by humans, automated hy- pothesis discovery was inposssible with early probabilistic models. LLMs have transformed this landscape. Advanced model structure with billions of parameters trained on vast corpora, they implicitly absorb broad commonsense and domain knowledge, cxhibiting strong reasoning abilitics on complex, natural-language tasks Yang et al., 2024ch. With a simple text prompt, we can now ask them to carry out abduction or induction and even inspect their intermediate reasoning steps (Li et al., 2024 Jung et al. 2022), exposing the latent information they rely on. This advancement has made it practical to study and deploy defeasible reasoning (Yang al. 2024c). Defeasible reasoning refers to forms of reasoning, such as abduction and ion, that. yield probable conclusions that remain open to revision as new evidence emerges. This shift
has fueled a wave of NLP research that places such flexible reasoning at the heart of Al progress (Liu et al.
2025
Huang & Chang 2023).
2.1 Hypothesis Discovery
Throughout this survey, we treat hypotheses, theories, and rules as compressed, human-learned knowledge for understanding and predicting the world. A hypothesis is a tentative claim that may, with sufficient empirical
3

---

Published in Transactions on Machine Learning Research (04/2025)
Hypothesis: (Darwin's hypothesis of evolution)
Theory: (Darwin's theory of evolution)
Experiment, application, experience
Can Be
Rules: If... then
Can Be
Represented as
Represented as
Figure 3: Relationship between hypothesis, theory, and rule.
support, mature into a theory a more firmly established hypothesis. Both hypotheses and theories can often be cast as rules, typically written in the form "if then 3 (See in Figure 3
This process of Hypothesis discovery or Rule learning, the cyclical process of formulating hypotheses, gath- ering evidence, validating refuting them, and ultimately establishing robust theories, lies at the heart of scientific progress (Eger et al. 2025). Early humans, for example, hypothesized that the Earth was flat based everyday observations. Later, Eratosthenes measured shadow angles at different locations, obtaining ev- idence that suggested the Earth's surface was curved. This evidence challenged the flat Earth hypothesis, and subsequent findings, notably Magellan's circumnavigation, conclusively confirmed the Earth's round- ness. Even with today's sophisticated instruments, researchers continue iterate this loop new domains, validating and refining theories as new data emerges.
Today, there is growing interest in whether LLMs can autonomously generate, apply, and validate hypotheses from natural language represented observations, mirroring this iterative process to achieve interpretable and adaptive hypothesis discovery. Although many studies have explored individual steps of hypot discovery, their efforts tend to be scattered across abduction, deduction, and induction, with insufficient attention to how these forms of reasoning interconnect drive genuinely iterative, hypothesis-driven discovery.
2.2 Reasoning
Reasoning is central to hypothesis discovery. Researchers have historically debated how best to categorize reasoning into clear, operational types. Different frameworks each have strengths and limitations (Harman
1965 Douven 2021 Bacon 1878 Laudan 1971 Mill, 2024 Stadler [011 Popper
2005
Okoli 2023)
In this survey, we adopt Charles Peirce's definition of reasoning (Peiree 1974 Burks
1946
Minnamcior
2004), emphasizing abduction, deduction, and induction as separale but interrelated processes. We choose
Peirce's framework for three main reasons. First, clarity: Unlike many other approaches, Peirce explicitly
differentiates among the three reasoning types, preventing confusion, such as the common conflation of
abduction and induction. Second, practicality: Peirce's 's model aligns each form of reasoning directly with distinct phase in the hypothesis discovery these hypotheses, and induction for validating them. This clear mapping makes his framework particularly suitable for systematically studying the entire process of hypothesis discovery, rather than isolated reasoning components. Finally, granularity: Peirce's framework breaks down the scientific discovery process into well- defined, finer-grained steps, facilitating detailed analysis and enabling more structured evaluation.
for generating hypotheses, deduction for applying
Abductive Reasoning is the process of forming explanatory hypotheses to make sense of observed phe- nomena. It is the only form of reasoning that generates entirely new ideas or explanations (Peirce) 1974 Frankfurt 1958). Given set of observations, one uses creative thinking and recalls necessary knowledge to come up with hypotheses that plausibly explain these observations. Importantly, a single set of observations can lead to multiple possible explanations. For instance, you come home and find the floor wet, you might form several possible explanations: perhaps pipe lcaked, or somcone spilled water accidentally. Without additional evidence or testing, you can't know for sure which explanation is correct. This illustrates how abduction helps generate potential explanations, which then must be tested further.
4

---

Published in Transactions on Machine Learning Research (04/2025)
Inductive Reasoning is the process of testing whether the hypothesis and deduced consequences really obtain and evaluating to what extent they obtain [Minnameier 2004 Peirce 1974). In practice, induction up- dates a hypothesis's confidence based on new observations, including rejecting it outright, or selects the most convincing candidate from a set of competing hypotheses. Consider the claim "Swans are (100%) white," " or linguistically, "All swans are white," " formed afler observing 99 white swans Texas. Encountering a black swan in New York contradicts that hypothesis. Through induction, we recognize this contradiction and lower our confidence in the original claim, adjusting it to "Swans are (99%) white," " linguistically expressed as "Almost all swans white." In this example, although the hypothesis appears "revised," the change is limited to its confidence level; no new explanatory perspective is introduced, and we do not actually form a new hypothesis. By contrast, abduction can lead us to a fresh explanatory hypothesis with new observations, e.g.: "Swans' color depends on their habitat," or " All swans in Texas are white," which introduces new ideas and is not a case of induction. Thus, inductive reasoning verifies or refines existing hypotheses (in terms of confidence) based on accumulating evidence.
Deductive Reasoning is the process of logically deriving specific conclusions from general hypotheses rules. If the initial hypotheses are true, deduction guarantees that the derived conclusions must also be truc. For instance, from the general rule " "All swans are white" and the obsorvation "This bird is a swan," we logically conclude "This bird must be white". While traditional deductive reasoming tasks, such as instruction-ffolooingg and standard problem-solving, have been extensively studied with LLMs (Pan et al. 2023 Wei et al.] 2022 Liu et al.) 2025 Huang & Chang, 2023), deductive reasoning in the context of hypothesis discovery poses unique challenges. Specifically, emphasizes inferential nllefollowing, requiring models to consistently apply hypotheses or rules to derive new and potentially unfamiliar conclusions, even when these hypotheses are counterfactual, unfamiliar, incorrect. For example, when flawed hypothesis is introduced in an unfamiliar domain, inferential rule-following requires us to strictly derive its predicted consequence, even if that consequence itself is incorrect. By comparing this consequence with experimental data, we can directly assess the hypothesis's validity and guide its revision. Conversely, if the deductive process is unreliable, we may overlook real contradictions and thus retain invalid hypotheses or discard valid ones. Tndeed, recent. work shows that although LLMs can demonstrate strong deductive performance on in- distribution tasks, they rely heavily on surface-level pattern matching and fail to generalize their inferential
rule-following to novel counterfactual scenarios (Pu ct al. 2025 Mirzadeh et al. 2024 Kang et al. 2024
Yan et al. 2025b).
There are also other types of reasoning, such as analogical reasoning Yuan et al. 2023 Jiayang et al. 2023 However, their function in hypothesis discovery is generally covered by abduction and induction. We will include these additional forms when we encounter a relevant case in the following section.
2.3 Rule Representation: Formal Language vs Natural Language
Table 1:
Comparison of Natural vs. Formal Language Representations for the Hypothesis "Sam is dragon". In natural language, commonsense knowledge is implicitly embedded, and derived knowledge relies on extensive commonsense, polentially resulting in different interpretations depending on background kuowledge and context. In formal languages (e.g., first-order logic (FOL) or code), the knowledge base must be defined explicitly and cannot fully capture all commonsense knowledge, however, the derived conclusions are deterministic and precise.
Representation Hypothesis
Knowledge Base
Derived Knowledge
English commonsense of 'dragon" "
English: 'Sam is a dragon" "
Sam is dangerous
Natural Language
Chinese commonsense of 'dragon" (r<(aagon(z) Fly(z))
Sam brings good fortune and a bountiful harvest.
Chinese: 'Sam is a dragon"
FOL: Dragon(Sam)
Fly(Sam)
Formal Language
Class Dragon:
Code: Sam Dragon
def fly (self):
Sam S fly ()
5

---

Published in Transactions on Machine Learning Research (04/2025)
There are many ways to represent hypotheses and rules, which we broadly divide into two categories: formal languages (FL) and natural languages (NL). Formal languages, such as first-order logic and program- ming languages, are systematic and rule-bound. After real-world entities are encoded as explicit literals, procise inference rules yield provably correct and sound conclusions, making these systems well suited to deductive reasoning. Yet the encoding process strips away many subtle semantic relationships and com- monsense knowledge, limiting the system's 's ability to handle the creative, defcasible reasoning required for abduction and induction (MeCarthy & Hayes 1981 Reiter 1980 Hanks & McDermott 1987 Liu et al,, 2025 Yu et al., 2024a Huang & Chang 2023). Natural language preserves those nuances and aligns more closely with human cognition, so it is better suited to abductive and inductive tasks. However, its mean- ings are implicit and context-dependent, making it difficult to define a deterministic reasoning pipeline and reducing the reliability of the resulting inferences (See in Table D Accordingly, the following sections treat langal- and natural- languagg approaches separately, emphasizing how their reasoning methods and evaluation protocols differ.
3
Related Surveys
Most existing work assessing LLM reasoning, both survey syntheses and popular benchmarks such as GSMSK et al.) 2021), centres almost exclusively on multi-step deductive tasks, leaving abduction and induction, the engines of hypothesis discovery, largely unexplored. Surveys of the field Yu et al.] (2024a) Liu et al.] (2025); Huang & Chang (2023) highlight the absence of systematic study and clear analytical frameworks for these modes, while benchmark analyses likewise show that abductive and inductive inference receive limited attention (Plaat et al. 2024 Li et al.] 2025b). This imbalance obscures our understand- ing of whether, and to what extent, L.MSs can perform the creative, evidence-based reasoning required for hypothesis-driven discovery.
On the other hand, research in the AI for Science domain takes a distinctly horizontal, application-driven approach. This body of work emphasizes practical tasks such as generating research ideas, conducting exper- iments, and synthesizing reports, often employing domain-specific pipelines tailored to individual scientific fields. However, these studies usually lack a generalizable reasoning framework applicable across different scientific contexts. Furthermore, their evaluation metrics, typically novelty, creativity, or consistency, tend to be subjoctive, human-centric, and thus difficult to generalize, offering limited thcoretical insight into the
underlying reasoning mechanisms involved in scientific discovery (Movva et al. 2025 Alkan et al. 2025
Reddy & Shojace 2025 Gridach et al. 2025 Bazgir et al. 2025).
Our survey adopts vertical, reasoning-centerel perspective grounded in Peirce's classical framework. It integrates three modes of reasoning into unified view of hypothesis discovery: abduction for hypothesis generation, deduction for hypothesis application, and induction for hypothesis validation. Unlike prior surveys that emphasize primarily deductive tasks, we concentrate on the entire reasoning process involved in hypothesis discovery, explicitly covering both defcasible reasoning (abduction and induction) and deductive reasoning. By clearly defining each reasoning mode and explaining its role within each stage of the discovery process, we provide a structured basis for designing principled, model-agoosicc benchmarks and evaluation (asks. Compared to existing application-oriented surveys, our framework thus offers a more abstract, system- atic, and theoretically informed approach to understanding and enhancing the role of LLMs in automated scientific discovery.
4 Hypothesis Generation with Abduction
Every scientific discovery begins with set of observations, denoted as 0= (0r, ... o, that we aim to explain. Let h represent the generated explanation hypothesis. The hypothesis generation task can be defined as generating an h such that:
h レ (o) A A ... on)
This notation means that h logically entails the observations. In other words, assuming h holds, it guarantees that all observations or A oz A ... A On follow. In this survey, we follow Peirce's definitions for reasoning.
6

---

Published in Transactions on Machine Learning Research (04/2025)
Accordingly, the primary process used in hypothesis generation is abduction, the method of formulating explanatory hypotheses to account for observed phenomena.
4.1 Method
Despite LLMs' demonstrated prowess in tasks like summarization or code generation, devising robust meth- ods to guide them in hypothesis generation remains an active area of research. Recent work has sought to leverage LLMs' in-context learning and natural language understanding to produce novel or domain-specific hypotheses, spurring the development of new techniques aimed at improving both the quality and appli- cability of generated hypotheses (Yang et al.] 2024c). In this section, review these methods, spanning approaches that rely solely on prompting, those that integrate external knowledge sources, and those that incorporate human expertise in the loop.
4.1.1 Natural Language Hypothesis Generation with LLMs
Prompt-Based Methods:
Due to the lack of large-scale, domain-specific data for hypothesis generation, most abduction approaches rely on prompt-based methods that are casy to deploy and don't require ex- tensive additional data. For instance, when provided with observations expressed in natural language and asked to generate plausible hypothesis that explains them, both Wiegreffe et al. (2022) and Qi et al. (2024) employ few-shot prompting to guide LLMs in generating hypotheses. Specifically, Wiegreffe et al. (2022)
construets few-shot examples using a triplet format (question, answer, explanation). In solving a task of generating biomedical hypotheses with given observations, Qi et. al. (2024) embeds a small set of indepen- dent observation-tor to-oon-ttt-ppohhsstt pairs in the prompt. By showing how each block of biomedical background obscrvations maps its corresponding hypothesis, the model learns extract relevant domain cues and generale novel biomedical hypotheses. Their findings indicate that including more examples in the prompt tends to reduce the novelty of the generated hypotheses while increasing their correctness. Furthermore, Yang et al. (024a) propose pipeline for hypothesis generation that involves five prompt-based modules: one to generate hypotheses, one to test deductive consistency, one to verify that the hypothesis is not merely a copy of the given context, one to assess its generalizability, and one to determine whether the hypothesis is trivial.
RAG-Based Methods: Labeling massive corpora for pre-training costly, but assembling a small or medium dataset for Retrieval-Augmented Generation (RAG) is practical, and several studies follow similar iterative threc-step pattern: (i) retrieve task-specific documents, (ii) let LLM generate refine hypothe- ses, and (iii) iterate with LLM feedback. For instance, after a user supplies seed paper and asks the LLM to generate a worthwhile hypothesis to pursue in research, Hu et al. (2024) query the Scholar API for related work, then repeatedly generate and critique hypotheses, gradually expanding a web of novel idcas. Yang eet al. 2025) apply the same loop to 51 lop-tier chemistry papers from 2024: experts first segment each pa- per into background, inspiration, and hypothesis; an I.I.M-baeed multi-agent system (MOOSF-Chem) then retrieves relevant snippets, drafts hypotheses, and scores them for originality,
similar pipeline appears in Yang et al. (2024b), where 50 conference papers are annotated in the same three fields, augmented with thematically similar web documents and 14 survey papers so that the LLM can judge both relevance and novelty.
Two variants enrich the retricval step with structured or fine-tuned knowledge. Xiong et al. 2024 ground each hepothesis in a domain knowledge graph: entities mentioned during generation are checked against graph relations, cnsuring the final claims remain fact-consistent. In contrast, Chai et al. (2024) finc-tune a T5 model (Ralfel et al., 2020) on curated scientific abstracts and, during inference, retrieve citation contexts and related data; novelty-guided loop then re-generates until the candidate is both coherent and inventive, outperforming standard transformer basclines.
Human-in-the-loog Hypothesis Generation with LLM: Recent studies show that combining humans with LLM support yields higher-quality, more novel hypotheses than either party working alone. Zhao et al. (2024) report that a human+LLMM pipeline surpasses both human-only and LLM-only basclines. Human annotators first draft hypotheses for uncommon observations; carefully designed prompts then guide the LLM to refine each draft by adding details and improving logical flow. Low-quality hypotheses generated
7

---

Published in Transactions on Machine Learning Research (04/2025)
by LLM are filtered with human evaluations and automated metrics such as BERTScore, and the resulting h.IA collaboration produces the strongest hypotheses. Similarly, Si et al. 2024) involve more than 100 NLP researchers in a three-condition study: LLM-only generation, human-only generation, and LLM generation reranked by humans. Human evaluations rate the human-rcrankod LLM hypotheses best on Novelty, Excitement, Feasibility, and Effectiveness. Pu el al. (2025) move beyond controlled experiments by introducing IdeaSynth, a copilot-like framework that assists users throughout hypothesis formulation. When a user supplies a high-level hypothesis, IdeaSynth retrieves relevant papers through an API and summarizes key information needed for development. Users interactively edit these summaries with LLM help, adding details and improving clarity. The system then aggregates all refined nodes, employs an LLM to craft a final applicable hypothesis, and supplies suggestions for follow-up experiments.
The quality of natural-language hypothesis generation largely depends on the inherent capabilities of LLMs. Because these models excel at in-context learning, prompt strategies such as Chain-of-Thought (CoT) and Reflexion (Wei et al. 2022; Shinn et al. 2023) can be applicd directly to this task. However, unlike computer- vision research, which gained rapid momentum from the ImageNet, benchmark, hypothesis generation lacks comparable, widely recognized task set. The main challenge is therefore the absence of reliable evaluation task and benchmark for natural-language hypotheses, an issuc examined further in Section 4.2
4.1.2 Formal Language Hypothesis Generation with LLM
One major advantage of formal hypotheses is that once formal language hypothesis is obtained, we can directly perform inference on it with guarantees of soundness and correctness. Depending on whether obser- vations are represented in formal or natural language, methods for proposing a formal language hypothesis need to be discussed separately.
Formal Language Observations: When observations are encoded in formal language, dedicated formal language solvers typically yield clear, white-box solutions that outperform language models. Consequently, using an LLM for these tasks is generally not preferred. Nevertheless, few early studies in the LLM era have explored this approach. For example, Young et al. (2022) trained a transformer model on FOL abduction tasks, demonstrating that the model can generate FOL. hypotheses from formal observations. Similarly, Nguyen et al. (2023) fine-tuned state-of-the-art legal transformers on FOL abduction tasks and found that models pre-trained on natural language legal abduction tasks do not show any performance improvements on FOL hypothesis generation problems.
When observations are represented in natural language, traditional symbolic solvers struggle to extract the key information needed for hypothesis generation. With LLMs, however, we can directly generate formal hypotheses. A popular formal language for this purpose is code, as it offers greater flexibility than other symbolic representations like FOL, and LLMs excel at coding.
Natural Language Obscrvations:
The simplest variant prompls an LLM with an observation set and asks it to produce executable functions as hypotheses that match the input-output pairs; Cheng et al. (2024) follow this pattern, treating each observation an (x. y) example and evaluating the generated function by execution. Extending this idca, Wang et all (0224); Qiu et al. (2024) have the LLM create multiple executable hypotheses, run them the observations, feed the results back to the model, and iterate, discarding weak candidates and refining promising ones until one covers all examples. To encourage diversity, il Lee et al. (2025) first ask the model for a single-word "main concept," " then use that concept to steer subsequent code generation, avoiding the similarity of low-temperature outputs and the degeneration of high-temperature sampling while still producing coherent hypothescs.
Instead of prompting an LLM to generate hypotheses from sentences, more grounded approach is to understand how the LLM's internal representations give rise to such hypotheses. However, due to the superposition problem (Elhage et al. 2022), the same neuron may be activated for semantically unrelated concepts, making this approach challenging. Fortunately, recent work by Bricken et al. (2023) leverages sparse autoencoders (SAEs) trained on LLM activations to mitigate this issue. This enables a plausible mothod for directly interpreting the internal "thoughts" of LLMs via their hidden representations. Building on this line of work, Movva et al. (2025) trained an SAE on an LLM and isolated neurons activated when the model predicted the click-through rate of Twitter posts. They found that neurons associated with "surprise"
8

---

Published in Transactions on Machine Learning Research (04/2025)
"shock" positively contributed to the prediction, supporting the hypothesis that surprising or shocking content tends to attract more clicks.
4.2 Evaluation for Hypothesis Generation
Due to LLMs' strong reasoning abilities and natural language interface, many methods have been proposed for hypothesis generation, and numerous ideas based on everyday human reasoning can be adapted for this purpose (Niu et al., 2024). However, a major challenge remains in establishing a grounded and convincing way to evaluate the quality of the generated hypotheses.
4.2.1 Natural Language Hypothesis Evaluation
Although prompting LLMs to generate natural language hypotheses straightforward, evaluating the qual- ity of these hypotheses is challenging due to the anbiguily inherent in natural language representations. Consequently, a common evaluation method involves either human evaluation using an LLM to assess the
generated hypothesos' validity (Zhao et al. 2024 Yang et al. 2024b Hu et al. 2024 Qi et al. 2024 Yang
let al. 2025) While human evaluation can provide valuable insights without relying on predefined answers, it is inherently subjective, less reproducible, expensive, and sometimes not entirely convincing. Therefore, alternative evaluation strategies are needed.
Implicit Prediction-based Evaluation: Early benchmarks often relied on question-answering (QA) tasks that required the model to implicitly form hypothesis answer a question (Sinha et. al., 2019 Weston et al. 2015). For example, consider the observation: "Lily is swan, Lily is white, Bernhard is green, Gery is swan. What color is Greg?" To answer correctly, one must infer an implicit hypothesis, such as "All swans are white" or "Most swans are white," " based on the fact that Lily is both a swan and white. Thus, the correct answer is "white." By verifying whether the model's answer is "white," " one can indirectly assess its ability to form an appropriate hypothesis and perform reasoning. Similarly, recent work shows that prompting LLMs to generate an intermediate hypothesis and then using that hypothesis for inference yields higher performance on complex tasks (Balepur et al., 2024 Shi et al.) 2023 Wang et al., 2025). However, this approach is problematic: the hypothesis may be formed incorrectly, the subsequent inference could be flawed, and the model might arrive al the correct answer through memorization or random guessing rather than proper abductive reasoning, Therefore, success in these tasks does not directly imply that the model possesses superior abductive capabilitics, making them unsuitable for reliably evaluating hypothesis generation.
Ground Truth-based Evaluation: Some studies build benchmarks with labeled hypotheses so that out- puts of LLM can be matched directly against references. DEER (Yang al. 2024a) supplics 1,200 fact-rule pairs, all written in natural language by experts across six topics-zoology, botany, geology, astronomy, his- tory, and physics. Generated hypotheses are compared with the gold rules using token-level mapping metrics like METEOR (Banerjee & Lavie 2005). In biomedicine, Qi et al. (2024) curate a benchmark with both seen and unseen samples: the seen split contains 2,700 background bookground-hppoteestt pairs collected before Jamuary 2023, whereas the unseen split has 200 pairs collected after that date. Outputs are evaluated against the ground truth with BLEU and ROUGE (Papineni et al. 2002 Lin 2004). On synthetic corpora such as WIKT and BILLS (Pham et al. 2024 Zhong et al., (2024), Movva al. (2025) treat hypothesis generation identifying the key features that drive prediction. The model proposes feature sets, which are judged by how well they match and cover the ground-truth features, thereby quantifying the LLM's ability to isolate causal signals.
Despite these efforts, Yang et al. 2024b) note that reference-based metrics such as BLEU, ROUGE, and ME- TEOR assume a single correct answer and therefore struggle capture the open-ended nature of hypothesis generation; developing fair, reliable metrics remains an open challenge.
4.2.2 Formal Language Hypothesis Evaluation
Unlike natural language hypotheses, formal hypotheses evaluations are more grounded due to their clarity and unambiguous semantics.
9

---

Published in Transactions on Machine Learning Research (04/2025)
Prediction based Hypothesis validation
NL
Initial Observation
Generated hypothesi
Deduction
Generated Prediction
Ground truth prediction
Ground truth hypothesis
134
Sam Dragon
Sam is cute
daaws
Sam cute
Sam
Class Dragon:
Sam = Dragon()
def fierce():
Sam is fierce
FL
Figure 4: Directly evaluating a hypothesis is often impractical. A reliable alternative is prediction-based evaluation, which operates on key principle of formal logic: deductive reasoning from correct hypothesis must yield correct predictions. Therefore, single incorrect prediction is sufficient to definitively falsify the hypothesis. However, this reliability fails for non-deterministic LLMss evaluating NT. hypotheses. The process is fundamentally flawed because an incorrect hypothesis can still generate a correct prediction.
Ground Truth-based Evaluation: Generated formal hypotheses can be evaluated against pre-defined ground truth hypotheses. Unlike natural language evaluation, where ground truth is often written by do- main experts and evaluated using token-level metrics like BLEU or ROUGE, formal hypotheses can be evaluated procedurally using solvers. This allows us to verify correctness deterministically. For example, Bowen et al. (2024) designed formal representations for synthetic grouping tasks to evaluate formal language hypothesis generation. Hua et al. (2025) constructed their benchmark based deterministic regular fune- tions, providing a procedural framework for evaluating formal hypotheses. Similarly, Young et al. (2022) used FOL representations, where LLMs were tasked with generaling FOL hypotheses to explain given lacts, and the outputs were evaluated by comparing them against ground truth hypotheses verified by solvers.
Prediction-based Evaluation: Since inference on formal hypotheses is deterministic, common evaluation method is test whether the generated hypothesis produces correct outcomes on held-out examples (See in Figure 4 For instance, Rule 2020) propose the list function task, where LLMs generate hypothesis function from observed (x, y) pairs, and evaluation is based on how well the hypothesis predicts hidden pairs. Similarly, Chollet (2019) introduces the Abstraction and Reasoning Corpus (ARC), where tasks involve transforming input grids of colored cells into output grids. The generated function is executed on test inputs, and correctness is determined by exact matches with the target output grids, including grid dimensions. Liu al. (2024) further propose a benchmark consisting of arithmetic calculations, color token mapping, and Kalamang vocabulary tasks, all evaluated in the same way. Additionally, Li et al. (2025a) construct diverse application scenarios, including list transformations, real-world problems, code generation, and string transformations, where the generated hypothesis is executed both seen and test observations and the final score aggregates performance across both sets. In more realistic setting, Chen et al. (0024) extract 102 tasks from 44 peer-reviewed publications. Based on a concise instruction and the corresponding dataset for each task, the agent must generate a formal hypothesis in the form self-contained Python program- that proposes a complete methodology to reproduce the original scientific analysis. The validity of this generated hypothesis then directly evaluated against expert-validatee test cases and through fine- grained, rubric-based human evaluations.
While the output of formal hypothesis generation can be an executable program, making it procedurally similar to program synthesis, the two paradigms are distinguished by their fundamental objectives. Pro- gram synthesis addresses computationally concrete problems where the specifications are clearly defined and the causal relationship between inputs and outputs is explicit. The core challenge one of implemen- tation-writing code to satisfy known requirement. In contrast, formal hypothesis generation confronts holistic, semantically rich scenarios where the underlying rules are unknown. Here, the central challenge is interpretation and discovery, crucial step that precedes any implementation. The agent must. first
10

---

Published in Transactions on Machine Learning Research (04/2025)
analyze phenomenon, abstract the necessary features from information-rich inputs using domain knowl- edge and commonsense, and formulate the problem itself. For instance, even with structured inputs like the ARC benchmark, success hinges on first deducing the semantic logic which patterns are relevant and how-before a transformation can be coded. Therefore, formal hypothesis generation crucially includes the preliminary scientific lask of discovering an unknown rule, slep that is already assumed to be complete the well-defined world of program synthesis.
bogic-wiich
4.3 Discussion and Future Directions in Hypothesis Generation
There exists significant gap between formal and natural language approaches hypothesis generation. In natural language hypothesis generation, observations typically originate from recent research papers, and generated hypotheses can potentially inspire novel research ideas with tangible real-world impacts (Eger ct al. 2025). However, rigorous and reliable evaluation methods for such hypotheses remain underdeveloped. Token-based metrics, such as BLEU or ROUGE, do not effectively capture the qualitative aspects of open- ended hesos generation (Yang el al. 2024b). Meanwhile, alternative approaches involving human or LLM- based evaluations are costly, subjective, and prone to incousistencies.
Conversely, formal language hypothesis generation bencfits from grounded, objective evaluation mothods. Nevertheless, existing formal lasks oflen involve simplified or artificial scenarios that fail to reflect the com- plexity and nuance inherent in real-world applications. Consequently, the field faces trade-off: formal reprosentations facilitate robust evaluation but risk omitting critical real-world muances, while natural lan- guage representations capture real-world complexity yet lack rigorous evaluation mechanisms.
To address this challenge, future research in hypothesis generation could focus on two key directions. Firstly, there is an urgent need to develop novel evaluation methodologies for natural language hypothesis genera- tion. Current methods, such implicit prediction and ground truth comparisons using token-level similarity, are inadequate. Alternative strategies like multi-dimensional human assessments, structured feedback, and hybrid evaluation frameworks should be explored. A deeper insight gained from our survey is that recent evaluation methods are built upon an casy-to-overlook assumption: that for any given problem, there ex- ists a single "golden" hypothesis. Consequently, evaluation is performed by directly comparing a model's output to this predefined "golden" hypothosis or to the conclusions drawn from it. This approach is not aligned with scientific practice. In the real world, many published hypotheses based on limited observations are ultimately proven wrong, and many deviate from what is known to be "correct" today. We cannot, however, simply classify the abductive process that generates these hypotheses as flawed or incorrect. Fur- thermore, different perspectives It is a mistake to conclude that model lacks strong abductive reasoning simply because it generates a plausible hypothesis that doesn't 't match the predefined "golden" answer. Therefore, before designing new benchmarks, it is essential to first step back and establish a clearer definition of what constitutes a "good hypothesis" and "good abductive reasoning." " based on how hypothesis is generated by experts in practice. Secondly, bridging the gap between formal and natural language hypothesis generation is crucial. Leverag- ing code as an intermediate representation expressive capability. However, existing code-based benchmarks often focus oversimplified problems that lack relevance to practical scenarios. Thus, a compelling direction for future research is to develop realistic, code-based genneraiononn tasks grounded in established rescarch papers, real-world datasots, and open-source repositories.
give rise to multiple, semantically different, yet plausible hypotheses.
promising path forward, it combines evaluative rigor with
5 Hypothesis Application with Deduction
Given a hypothesis h; hypothesis application is defined as the derivation of a new observation Onew such that:
h
Onew
In some cases, the hypothesis may depend on a context c, so that h can be viewed as a function of c. In this context-dependent formulation, hypothesis application defined as deriving a new observation Onew such that:
h((c)
Onew
11

---

Published in Transactions on Machine Learning Research (04/2025)
In our work, we follow Peirce's definitions for reasoning. Accordingly, the primary process used in hypothesis application is deduction, the method of deriving necessary consequences from a given hypothesis.
Notably, while much research trains LLMs on formal language corpora to improve their general reasoning abilities, this survey focuses on how LLMs can advance the hypothesis discovery. For the specific task of applying a formal hypothesis, a dedicated solver (e.g., a FOL prover) is the superior tool, as its deterministic nature guarantees correct and sound prediction. A probabilistic LLM can, at best, only be trained to approximate the performance of perfect solver in this closed setting: it cannot be more correct offer better solution. Therefore, since a reliable and transparent method already exists, there is little motivation to use an LLM for deductive application of formal hypotheses. This section, consequently, will focus on the application and evaluation of hypotheses in natural language.
5.1
Method
LLM as autoformalizer: Since formal symbolic solvers yield sound and correct predictions, Pan et al. 2023 Olausson et al. (2023); Kalyanpur et al. (2024) treat LLMs as autoformalizers, using them to translate natural language hypotheses into formal reprosentations like FOL and code before applying a formal inference procedure. This translation significantly improves deductive correctness. However, these methods have primarily been evaluated on benchmarks such as Proo/Writer (Tafjord et al., 2021 and FOLIO (Han et al. 2024), where the questions are already closely aligned with formal language. For example, given the input "Factl: Eric is young, Fact2: Dave is white, Rule 10: if someone is young and not kind then they are big", translating this into FOL. is relatively straightforward. It remains unclear whether LLMs can reliably parse more complex, everyday natural language into formal representations.
Fine-tuning is a common approach to improve model performance when corresponding training data is available. [Sun et al. (2024) proposed a synthetic "StringGame" task which ground truth hypotheses and answers are provided. Leveraging the CoT approach, a. LLM is prompted to generate multiple candidate hypothesis application trajectories along with their results. By comparing these results with the ground truth, the trajectories that produce correct outcomes are identified as correct and stored for fine-tuning. The resulting fine-tuned model then demonstrates improved performance in both hypothesis application and instruction following.
Fine- Tane-TTTnng-Baeed Method:
Prompt-Based Method: Although CoT prompting has improved performance on multi-hop question answering tasks, [Snn et al. (2024) found that does not directly enhance performance in hypothesis ap- plication. Therefore, new prompting methods have been designed specifically for this purpose. Inspired by mathematical induction, Cai et al. (2025) propose quantifying the difficulty of question so that the LLM can solve it incrementally, from simpler versions to more complex ones, ultimately arriving at the correct answer. In another approach, [Ling et al.] (2023) design a pipcline that supervise the correctness of cach rea- soning step during hypothesis application. First, the LLM indexes all premises; then it is asked to label the minimal set of premises required to derive new facts. This pipeline generates multiple candidate hypothesis application trajectories, and by having the LLM vote on each step, the most convincing deductive trajectory is selected.
5.2 Evaluation for Hypothesis Application
Although many benchmarks and evaluation methods exist for general deductive reasoning, such as question answering and mathematical tasks like GSM-8k (Cobbe et al. (001),, these question types do not explicitly test the formation of new facts based on given hypotheses or rules. Evaluating the correctness of a natural- language deductive trajectory is challenging because annotated reasoning paths for hypothesis application are scarce, and the same result can follow from different reasoning paths. As a result, most evaluations use prediction-based chocks. We assume that, given a correct hypothesis and a known ground-truth result, a valid deduction will reproduce that result. By comparing the model's deduced outcome with the ground truth, we can judge whether its deduction is correct. For example, take the hypothesis "Coin flips are independent and identically distributed (f.i.d) with a 50 percent chance of heads." When askod, "After three consecutive heads, what is the probability of a tail on the fourth flip? " flawed model might claim the chance of tail has increased. In fact, under the i.i.d. assumption, the probability remains 50 percent. Supplying
12

---

Published in Transactions on Machine Learning Research (04/2025)
the correct hypothesis and comparing the model's answer to the true result lets us evaluate whether its deductive reasoning is valid.
Several benchmarks have been developed to evaluate hypothesis application. Inspired by the "Turtle Soup" game, Yu et al. (204b)) created TURTLEBENCH, where an LLM must answer "True," " "False," " or "Not Relevant" " to a series of questions to test its ability to follow a story. The RULES benchmark, introduced by Mu et al. (2024), provides 14 rule-following scenarios with programmatic evaluation to objectively assess adherence to specified rules. Expanding
complexity, the Holiday Puzzle benchmark (Cai et al., 2025) features multiple holiday schedule scenarios and uses test cases to verify correct computation. To specifically test inferential rule-follwiin, Sun et al.] (2024) constructed RuleBench, which evaluates model's ability to apply both factual and counterfactual rules; their experiments reveal that while LLMs achieve near-perfect accuracy on factual rules, their performance drops dramatically under counterfactual scenarios, revealing a significant capability gap.
5.3 Discussion and Future Directions in Hypothesis Application
While traditional deductive reasoning tasks (c.g., question answering, problem solving) in LLMs have been widely studied, the capability for hypothesis application remains significantly underexplored. According to Sun et al. (2024), hypothesis application involves inferential rule-following, requiring models to consistently apply given hypotheses to derive novel knowledge in unfamiliar domains. Robust hypothesis application is critical to hypothesis discovery, as hypotheses must gencralize to scenarios with unseen observations. However, existing LLMs frequently struggle to extend hypotheses beyond familiar contexts, thus limiling the evaluation of hypothesis generation.
Future research could therefore focus on rigorously evaluating LLMs' hypothesis application, both factual and counterfactual, in novel scenarios. Developing benchmarks explicitly designed for hypothesis-driven inference in unfamiliar domains could reveal important insights into model adaptability and generalization. Additionally, current evaluations of hypothesis application mainly rely on outcome-based correctness, com- paring predicted results to ground truth given correct hypotheses. However, incorrect reasoning may still lead to correct predictions in natural-language contexts. Although Ling et al. (2023) propose improving hypothesis application by intervening in reasoning trajectorics, a large-scale benchmark specifically designed to evaluate trajectory-based hypothesis application remains absent.
Although as stated in the beginning of the section, if all premises and hypothesis need to be applied are already represented in formal language like FOL or code. The hypothesis application tasks with formal language do not bring gain with LLM. There still cases where LLM may help improve this problem
6 Hypothesis Validation with Induction
According to Peirce, induction validates a hypothesis by updating its confidence when new evidence appears. However, in studies that focus exclusively on induction, tasks are typically onc-off: a hypothesis (or set of hypotheses) and collection of observations are provided, and there is iterative updating of confidence. A simplified framework for hypothesis validation treats it as a multiple-choice problem: given observations the model selects the most possible
O = {or, o2,..,, on} and a set of hypothesis IH = hypothesis. In simpler scenarios, where only one hypothesis is provided, the model determines whether the hypothesis correctly explains the observations. Tn the next section, when combined with deduction and abduction, induction can subsequently be used to itcratively update the confidence in the hypothesis.
Natural language representations add significant complexity to induction. In formal language settings, all necessary information is explicitly provided, and reasoning follows rigorous, well-defined steps. In contrast, validating a natural language hypothesis often requires commonsense knowledge and interpretation of nu- anced language. For example, consider the observations "Neil wanted to see the mountains of Asia" and "Neil loved being so close to the mountains in Nepal," " with candidate hypotheses "Neil booked a trip online" and "Neil took a trip to see the Rocky Mountains instead." Here, the nuanced mcaning of the term "instcad" and the geographic relationships require careful analysis and may lead to different conclusions. Indeod, Zhang et al. 2020 reports that, when verifying their dataset where five annotators judged the plausibility of hand-
13

---

Published in Transactions on Machine Learning Research (04/2025)
written hypotheses, disagrcements occurred in 62.34% of 1,365 explanations, underscoring the challenge of natural language hypothesis validation.
6.1 Method
6.1.1
Formal Language Hypothesis Validation
He Lu (2024) introduce the CauscJudger framework, which leverages LLMs every stage to validato candidate hypotheses. First, LLM transforms the natural language inputs into an FOL-based repre- sentation by integrating each candidate hypothesis into the premises. Next, LLM filters out irrelevant premises and rules. Finally, another LLM performs forward reasoning to decide which hypothesis explains the observations.
6.1.2 Natural Language Hypothesis Validation
Prompt-Based Method: Lampinen et al. (0022; ; Sun et al. (2024) employ a few-shot prompting approach for hypothesis validation. In this method, case triplets, consisting of an observation, a hypothesis, and its corresponding validity, are provided to the model, which then answers hypothesis validation question. Although this approach improves performance, [Sun al. (2024) reports that the performance boost is limited. Their experiments further indicate that fine-tuning oulperforms few-shot prompting.
Fine- Time-Tuning-Bare Based Method:
Since hypothesis validation essentially constitutes classification problem, many Natural Language Inference (NLI) datasets can be adapted into hypothosis validation tasks. Conse-
quently, fine-tuning is a popular method in this context. For example, Zhao et al. (2023); Chan et al. 20023
Sun et al. (2024) fine-tune models to select the correct hypothesis from a set of hypotheses based on new observations.
6.2
Evaluation for Hypothesis Validation
6.2.1 Formal Language Evaluation
Along with the CauseJudger framework, He Tu 20024) also proposed the CauseTogics dataset. Based on the required formal reasoning depth, the dataset is divided into four difficulty levels for hypothesis validation tasks, with 50,000 samples per level. Each hypothesis assigned a binary ground-truth label indicating whether it correctly explains the observations.
6.2.2 Natural language Evaluation
Binary-Clase Classiifiatiinnn-aaan -ased Evaluation: [ampinen al. (2022) chose a subset of tasks from the crowd-sourced benchmark BIG-bench (beuch authors] (023) and constructed their own benchmark specifi- cally for hypothesis validation. Each data sample consists of an observation, its corrosponding hypothesis, and ground truth label indicating whether the hypothesis truly explains the observation.
Hypothesis validation using natural language is inherently challenging because the implicit information and required common-sense background are not explicitly stated. This often leads different individuals to draw different conclusions when validating a hypothesis based solely on recalled information. Rudinger et al. (2020) mitigate this issue by adopting different strategy. Instead of asking annotators to directly judge whether an observation explains a hypothesis, they ask the model determine if a given observation weakens strengthens the hypothesis. Specifically, they sample observation abservationn-hypoteesit pairs from existing datasets and then manually craft two types of sentences: that acts as "strengthener" (increasing the likelihood of the hypothesis) and one that acts as "weakener" (decrcasing the likelihood of the hypothesis). A quality check of the labels confirmed high inter-amnolator agreement. on the strenglhening and weakening effects. During evaluation, the model is required to decide whether a new observation strengthens or weakens the hypothesis. This approach aligns with the paper's goal of modeling defeasible inference by leveraging explicit contextual updates rather than relying on potentially variable human interpretations of implicit information. Furthermore, Zhang al. (225)) extended this task to include visual observations. In their extension, given
14

---

Published in Transactions on Machine Learning Research (04/2025)
visual observation and natural language hypothesis, LLM is tasked to determine whether the provided sentence serves as strengthener or a weakener.
Siriple-Choice-Bassee Evaluation: Bhagavatula et al. 2020) introduce the ART benchmark, compris-
ing roughly 20k narrative contexts where each sample includes two time-ordered observations, one depicting a story's 's start (01) and the other its outcome (02), alongside two hypotheses: a plausible explanation (h+) and a less plausible one (h-), challenging models to choose the best explanatory hypothesis and enabling adaptation to prrothesis-generation tasks evaluated against ground-truth explanations. Similarly, [aian le al. 2023) present the BRAINTEASER benchmark of about 1.1k lateral-thinking puzzles, each offering a question with multiple-choice answers, one that defies commonsense and several conventional distractors, in both sentence (narrative) and word (meeaiin--altertionn) formats to test creative reasoning, with addi- tional semantic and context reconstruction variants assessing reasoning consistency and robustness across formulations. Moreover, Del & Fishel (2023) introduced the True Detective benchmark for deep hypothesis validation, featuring 191 long-form detective puzzles (=s 1200 words cach) from the "5 Minute Mystery" plat- form, where models (and humans) select. the correct explanation from 4-5 options, human accuracy averages 47%, top solvers exceed 80%, and each puzzle includes golden chain-of thought explanations detailing the reasoning steps that lead to the correct answer.
6.3 Discussion and Future Directions in Hypothesis Validation
Previous literature often conflates hypothesis generation and hypothesis validation, primarily due to am- biguily inherent in the inference to the best explanation (IBE) paradigm. Within IBE-based approaches, hypothesis validation typically appears as an implicit intermediate step, where selecting the "best" hypothe- sis is frequently based on unclear or subjective criteria without dedicated, independent evaluation. However, adopting Peirce's explicit distinction between abduction, deduction, and induction clearly separates valida- tion from generation, underscoring the need for dedicated research on validating hypotheses against newly observed evidence.
Current validation methodologies predominantly adopt end-to-end metrics that only assess final correctness, neglecting the reasoning processes and commonsense knowledge required to validate hypotheses in realistic settings. The subjective nature of natural language, coupled with different interpretations of observations, highlights the necessily for richer evaluative frameworks. Future benchmarks should incorporate detailed intermediate Chain-of-Thoute pothoscs, such as recalling relevant commonsense knowledge and performing nuanced inference. Evaluations should then emphasize consistency between the reasoning process and available commonsense context rather than relying solely on superficial similarity to reference answers. Such benchmarks would greatly enhance our understanding of hypothesis validation and better reflect the complexities of human-like reasoning.
data, capturing explicit reasoning steps humans take when validating hy-
7 Hypothesis Discovery
Although many works introduced in the previous sections propose methods and evaluation metries, they mainly focus on individual phases of Hypothesis Discovery-Hypoothess generation (Abduction 4. Hy- pothesis application (Deduction 5). and Hypothesis validation (Induction @. However, in real-life Hypoth- esis Discovery, these reasoning stages are not independent and must be treated holistically. Initially, we form hypotheses based on limited observations using abduction, which subsequently informs the application of these hypotheses through deduction, enabling the collection of further evidence. Concurrently, induction continuously evaluates and resolves inconsistencies arising between newly obtained observations and earlier hypotheses. This iterative interplay means that each hypothesis formulated, action taken, observation gath- ered, and inconsistency identified dynamically shapes and reshapes our evolving understanding, influencing subscquent rcasoning steps and contributing to diverse interpretations of the world. Treating any single reasoning phase in isolation oversimplifies hypothesis discovery. For example, although Bowen el al. (2024) evaluated every reasoning, they handled each step separately and thus failed to assess the true rule-learning capability of LLMs. Consoquently, integrating abduction, deduction, and induction into a unified lcarning loop remains both challenging and largely understudied, yet it is the ultimate goal for constructing and-to-end agents capable of scientific discovery.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Passive hypothesis discovery
Proactive hypothesis discovery
Real world simulation
Epwisstert
Simuiated Brivtormen
Obsenvatios
observations
Abduction
Abduction & Induction
Abduction & Induction
Induction
Hypotheses
Hepothears
pass hypofhesi diseovery. observations drown without ciservations neot. resuti deductivv ressionng not strictly involved..
In proessive hypothesis discovery, observation dotleed retfoved from paramenic produced simple tools that direclly ganerate deservatiors.
enstrommet through defined action
meaningfu observations sequence acdion requied, so the LLM must plnn it moves strtegrially beltmn acting
Figure 5: Differences and similarities among different types of hypothesis discovery tasks
Despite few studies that acknowledge the interdependence among reasoning types and allow models to refine hypotheses iteratively, they still overlook two decisive aspects of real-world hypothesis discovery. First, most benchmarks remain static and passive: they hand agents a fixed set of observations deemed sufficient to reach the correct. hypothesis, whereas real-life hypothesis discovery requires actively seeking additional evidence. Second, even in settings that allow proactive information gathering, the granularity the action space is still too coarse: agents fetch observations via onc-shot "recall" or "web-scarch" commands, whereas rcal scientists must strategically plan and carry out precisely staged experiments often designing specialized equipment at each step. Recognizing these limitations, categorize existing hypothesis-discovery research into three classes (see Fig. 5
7.1 Passive Hypothesis Discovery
In this type of study, LLMs generate, apply, and validate hypotheses iteratively. However, the observations are provided by fixed dataset. The LLM does not need worry about which observations it will receive. Instead, it simply reasons based on the given data, passively receiving and processing the information provided.
Zhu el al. 2024 proposed the Hypothesss-to-Thheee
(HCT) Framework to generate formal hypotheses (e.g., "if A then "" by leveraging existing benchmarks (Sinha et al. 2019 Wang et al. 2022b Rule 2020). In HtT, LLMs generate a hypothesis and propose learned rules to solve each question. When a new question is received, the model first formulates a preliminary hypothesis based on the context. then proposes candidate rules that might lead to the correct answer. These candidate rules are applied to the problem and verified against the ground truth. Rules that consistently yield correct predictions are retained and added to the rule library, while ineffective ones discarded. Iteratively, after processing all questions in the benchmark, the LLM builds rule library containing effective rules for solving the questions.
Zhou et al. (2024) proposed the HypoGeniC framework. Unlike HtT, HypoGeniC is evaluated on more realistic datasets such as Shoe Sales, Deceptive Reviews (Granhag & Vrij 2005). Headline Popularity (Matias et al. 2021), and Tweet Popularity (Tan et al.] 2014). Due to the complexity of real-world data, the gencrated hypotheses are more nuanced and expressed in natural language. Similar HtT, HypoGeniC begins by generating a set of candidate hypotheses from small number of examples. As new observations are processed, each hypothesis is used to make predictions and is assigned reward based on its accuracy. The system dynamically updates the confidence of each hypothesis; those that consistently perform poorly are removed from the hypothesis bank. New hypotheses are generated from examples that existing hypotheses fail to explain, allowing the model to refine and expand its understanding over time.
Both HypoGeniC and HtT simplify hypothesis discovery by relying on benchmark questions that include ground-truth answers. This configuration allows an external algorithm, not the LLMs themselves, to validate generated hypotheses and update their confidence based on the correctness of predictions. In real-world
16

---

Published in Transactions on Machine Learning Research (04/2025)
scenarios, where no ground-truth answers are available, these frameworks become inapplicable and would require substantial adaptation.
7.2 Proactive Hypothesis Discovery
In real-life hypothesis discovery, we do not start with predefined set of observations that continuously propose new insights. Instead, once an initial hypothesis is formed, we proactively recall our memories or explore further to gather new observations that either strengthen or weaken the hypothesis, allowing us to verify and refine our ideas.
Given a hypothesis, Li et al. (2024) and Jung et al. (2022) propose two proactive methods for hypothesis discovery that both leverage the LLM's parametric memory to generate evidence that either strengthens or weakens the hypothesis. In Hypothesis Testing Prompling, the model directly uses its internal reasoning to evaluate the generated evidence, determining which pieces are more convincing, and then decides whether the hypothesis is correct based on the balance of evidence that strengthens or weakens it. In contrast, Maieutic Prompting iteratively constructs a tree of evidence by generating both strengthening and weakening explanations. It then employs the LLM to assign a belief score (reflecting the model's confidence in the evidence) and a consistency score (measuring how well the evidence aligns with the hypothesis). Finally, a MAX-SAT solver is applied to select the subset of evidence that maximizes the overall scores, thereby determining whether to accept reject the hypothesis.
Different from relying solely an LLM's parametric memory to generate new evidence, [Salls & Shalin (2024) propose minimal setting for proactive hypothesis discovery. Inspired by the Wason Task from cognitive science, this lask challenges LLMs to prove a formal language hypothesis of the form "if p then q" Here, both p and are objects described in natural language, for example, "if a person is a man, then he drinks alcohol." The task provides four cards, each with two sides representing different attributes. Initially, one side of each card shown, displaying p. q, p, and =q, while the other side reveals the state of another attribute. To rigorously validate the hypothesis "if p then q, " one must flip the p card to confirm that its hidden side is g (modus ponens) and flip the -g card to check that its hidden side is "p (modus tollens). Flipping only these two cards provides sulficient evidence for the hypothesis, while the other two cards do not offer the necessary information. Thus, in this benchmark, by proactively flipping two cards, we can determine whether the LLM can correctly identify natural language expressions of and and validato the hypothesis using a minimal action space.
Moreover, Conti et al. (2024) propose APEx, a multimodal automatic benchmarking framework that eval- uates hypotheses about large multimodal models in fully automated and iterative fashion. For example, to lest a hypothesis such as "a model is able to identify graffiti-styeed images," " APEx first leverages texi- to-image retrieval and generation tools to create tailored set of test images. It then employs a range of transformation tools to perform image augmentation, introducing variations that challenge the models' robustness. In an iterative experimental loop, the framework executes these experiments on a library of models, analyzes the results, and refines the testing protocol accordingly.
7.3 Complete Loop: Real-World Hypothesis Discovery Simulation
Other works equip LLM agents with interactive environments that more closely mirror the complexity of real-world hypothesis discovery by combining planning, acting, and evidence collection. For example, Xu] et al. (2033) construct a Minecraft-like world in which a "vandal" agent performs up to 26 types of actions (e.g., moving, eating, crafting) to achieve a hidden goal (such as collecting lava or crafting a particular item) and leaves behind tracks evidence. A detective agent agent-driven by reinforcement learning to maximize information gain-then gathers those tracks and presents them to an LLM,, which must answer multiple- choice question about the vandal's original objective. Because evidence collection relies on an RL policy rather than LLM planning, this setup evaluates only the model's capacity to interpret evidence, not. its ability to proactively generate and test hypotheses in a dynamic setting.
Building on this approach, Wang et al. (2022a) introduce 30 scientific tasks drawn from five topics in fifth- grade curricula, ranging from measuring the friction coefficient of inclined plane to testing electrical
17

---

Published in Transactions on Machine Learning Research (04/2025)
conductivity. Here, agents must execute long action sequences and apply deductive reasoning grounded in established theories and definitions to complete each task. Likewise, Jansen et al. (2024) design 120 experiments across eight subjects (e.g., Chemistry, Archaeology), each with three difficulty levels, and allow 14 coarso-grained actions (such as "take," " "put," " and "move"). Agents are evaluated on (1) task completion, (2) execution of key experimental sleps, and (3) accurate hypothesis discovery compared to a ground truth. While these virtual labs simulate multi-step procedures and test hypothesis application, their restricted action spaces support only qualitative inference and preclude the fine-grained interventions needed for quantitative rule-learning
To address these limitations, He et al. (022)) propose puzzle environments in which agents can input arbi- trary integers or letters and receive tailored feedback based on a hidden rule. In this framework, an LLM must iteratively probe the environment, uncover the underlying quantitative rule, and solve the puzzle. Performance is assessed not only by whether the agent solves the puzzle but also by human judgments of the clarity and rigor of its reasoning steps, thereby offering a finer-grained evaluation of both quantitative hypothesis generation and the quality of the model's deductive process.
7.4 Discussion and Future Directions in Hypothesis Discovery
Hypothesis discovery fundamentally differs from isolated reasoning tasks by requiring iterative learning and continuous refinement of hypotheses within dynamic, evolving contexts. Particularly in Real-World simulation scenarios, the decisions and actions taken by an LLM may lead to entirely different trajectories of observation collection, varied learning efficiencies, and alternative hypotheses.
Building effective benchmarks for hypothesis discovery requires constructing rich, realistic environments capable of simulating real-world complexities. These environments should contain diverse, comprehensive action spaces and varied observational feedback mechanisms. Compared to traditional static, label-based datasets, ercating such benchmarks is significantly more labor-intensive, demanding at least two key com- ponents: 1, A set of rules unknown to the LLM that can be learned within the environment. 2, A sufficiently expressive action space that allows the LLM to interact with the environment, receive feed- back, and gather new information. Based on our survey, we provide the following insights to satisfy these two components.
Firstly, since current. LLMs are already trained on a vast corpus of text, finding a rule that is unknown to them is tricky. Based on our survey, we have identified two valid methods. One approach, as proposed by He et al. (2024); Jansen et al. (2024), is to use self-defined, fictitious rules. These rules are novel to LLMs; however, this introduces limitations. The handmade, fictitious rules are less complex compared to real-world rules. A second method is to test LLMs on newly founded hypotheses from the most recent work, which would not have been part of the LLM's training corpus (Qi et al. 2024). However, although this approach eliminates the rule leakage problem, it introduces unique challenges. As previously noted, the work by Qi et al. (024)) limited to hypothesis generation; additional work needed to adapt suitable research into interactive environments for hypothesis discovery.
Secondly, regarding a sufficiently expressive action space, it is costly to construct an interactive environment from scratch. Luckily, recent hot topic involves leveraging LLM code agents as researchers to reproduce the code and methods in new research papers (Huang et al.] 2023 Chan et al. 2024 Yan et al. 2025a Although the aim of these studies is limited to reproducing the code program (similar to program senthesis), this presents a possible and valuable direction. One could translate an annotated paper into an interactive environment. It would not be necessary to define the interactions, as the code generated by the LLM agent already provides sufficiently fine- grained and expressive action space. One would then mask the hypothesis and results of the paper and test whether LLM can reproduce the statements and hypotheses proposed.
8 Summary
In this survey, we have presented a comprehensive and structured framework for hypothesis discovery using LLMs, guided by Peirce's reasoning paradigm of abduction, deduction, and induction. We systematically ex-
18

---

Published in Transactions on Machine Learning Research (04/2025)
plored current methods and benchmarks across the three core components: hypothesis generation, hypothesis application, and hypothesis validation.
A central finding of our analysis is the significant trade-off between formal (FL) and natural language (NL) representations. While FL enables rigorous, objective evaluation, it is often confined to simplified, artificial scenarios that lack real-world complexity. Conversely, NL captures the nuanced complexity of real-world reasoning but suffers from critical lack of reliable and reproducible evaluation metrics. Most fundamentally, current evaluation paradigms are often biased by the underlying assumption of single "golden" hypothesis, which ignores that multiple, semantically distinct theories can be plausible and valid a stark deviation from authentic scientific practice.
Furthermore, we advocate for a paradigm shift away from isolated tasks and loward integrated, dynamic environments that faithfully model the scientific process. Building such benchmarks requires two key features: a set of rules genuinely unknown to the LLM and a sufficiently expressive action space for interaction. A promising path forward involves creating these environments by adapting recent research papers, whose findings post-date the LLM's training corpus. This strategy ensures the underlying scientific principles are genuinely unknown to the model. Within this setup, the LLM agent's objective is to rediscover the paper's s central hypothesis by generating code to conduct experiments. The act of code generation itself serves as a naturally expressive and fine-grained action space, shifting the focus from mere code reproduction to genuine hypothesis discovery.
By developing these holistic environments that demand proactive hypothesis generation, robust application, and continuous validation against evolving evidence, future research will significantly advance the ability of LLMs to not merely exccute instructions but to autonomously generate, refine, and validate hypotheses, thus realizing their potential true engines of discovery and innovation.
References
Atilla Kaan Alkan, Shashwat Sourav, Maja Jablonska, Simone Astarita, Rishabh Chakrabarty, Nikhil Garuda, Pranav Khetarpal, Maciej Pióro, Dimitrios Tanoglidis, Kartheik G. Iyer, Mugdha S. Polimera, Michael J. Smith, Tirthankar Ghosal, Mare Kuertas-Compal Sandor Kruk, Kevin Schawinski, and Ioana Ciucă. A survey on hypothesis generation for scientific discovery in the era of large language models, 2025.
URL https: httpps///axvv..o org/ Saarg/bbs//5500 a50/ 2505/0490400
Francis Bacon. Novum organam. Clarendon press, 1878.
Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. Artifacts or abduction: How do LLMs an- swer multiple-choice questions without the question? In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (ods.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10308-10330, Bangkok, Thailand, August 2024. Association for Computational Lin- guisties. doi: 10.18653/vv w1 acl.o...lloooo5555 555. URL
555/
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yer Yew Lin, and Clare Voss (eds.), Proceedings of the ACI. Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pp. 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL htttp://acclanthollee wwwp:/hhoooooyoooo/0-0--00-00
Adib Bazgir, Rama chandra Praneeth Madugula, and Yuwen Zhang. Agentichyppohesss: A survey on hypoth- esis generation using LLM systems. In Towards Agentic AT for Science: Hypothesis Generation, Compre- hension, Quantification, and Validation, 2025. URL http::///ppppooeeeeoeoooottttttp
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL httpp://openrrvvier. net/foorm?id-uyyLLBBBW
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. Abductive commonsense reasoning, 2020. URL http://////aaaaao.r/////////////
19

---

Published in Transactions on Machine Learning Research (04/2025)
Chen Bowen, Rune Setrr,, and Yusuke Miyao. A comprehensive evaluation of inductive reasoning capabilities and problem solving in large language models. In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguistics: EACL 2024, pp. 323 339, St. Julian's is, Malta, March 2024. Association for Computational Linguistics. URL https://aclanttoooogy org/2024. 2024. f findings-eacl. 22/
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Saffield-Dodda Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monose- manticity: Decomposing language models with dictionary learning. Transfommer Circuits Thread, 2023. httpa///neaiarraii/ao//////////a /tt////mmanii--eaaamumum iitss/iidexx.mmlll
Arthur W. Burks. Peirce's theory of abduction. Philosophy of Science, 13(4):301-30 1946. ISSN 00318248,
1539767X. URL http::/ jttp//rw...strrr.oo.ooo.oooorr. org/stabbee/185211 htt0:///www...ooooooo/////////11
Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, Serge Belongie, and Lei Li. The role of deductive and inductive reasoning in large language models, 2025.
URL https://arxiv, .rg.iv..rrggooo org/ att///ass///.../////////////////
Miaosen Chai, Emily Herron, Erick Cervantes, and Tirthankar Ghosal. Exploring scientific hypothesis generation with mamba. In Lotem Peled-Cohen, Nitay Calderon, Shir Lissak, and Roi Reichart (eds.), Proceedings of the 1st Workshop on NLP for Science (NLP4Science), pp- 197-207, Miami, FL, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/vv 2011853/v112000 /ttp:///2222..//ciioooooooo URL https://acllanthologr
org/2024. 2024. wwwpg/2224..llssiieeeee nlp4science-1.117
Chunkit Chan, Xin Liu, Tsz Ho Chan, Jiayang Cheng, Yangqiu Song, Ginny Wong, and Simon See. Self- consistent narrative prompts on abductive natural language inference. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.), Proceedings of the 13th International Joind Conference on Nalural Language Processing and the 3rd Conference of the Chopter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1040- 1057, Nusa Dua, Bali, November 2023. Association for Computational Linguistics. doi: 10.18653/v1/2022 jjeulp-main.fe URL https://aclantholl.e http:://ocolatooooooooog//////// 1jcllp main.667
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. ai preprinl wxXi:2410.070095 2024.
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xuc, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. Scienceagentteecch: Toward rigorous assessment of language agents for data-driven scientific discovery, 2024. URL. htttp:://araa
htts//bb/22111 05080,
Kewei Cheng, Jingfeng Yang, Haoming Jiang, Zhengyang Wang, Binxuan Huang, Ruirui Li, Shiyang Li, Zheng Li, Yifan Gao, Xian Li, Bing Yin, and Yizhou Sun. Inductive or doductive? rethinking the fundamental reasoning abilities of lims, 2024. URL
François Chollet. On the measure of intelligence, 2019. URL
Karl Cobbo, Vineot Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021. URL https:://axii...ot org/ http//bb/2210000 14168
Alessandro Conti, Enrico Fini, Paolo Rota, Yiming Wang, Massimiliano Mancini, and Elisa Ricci. Automatic benchmarking of large multimodal models via iterative experiment programming, 2024. URL https: //aaxii.org/abae..... 12321
20

---

Published in Transactions on Machine Learning Research (04/2025)
Maksym Del and Mark Fishel. True detective: A deep abductive reasoning benchmark undoable for GPT- 3 and challenging for GPT-4. In Alexis Palmer and Jose Canacho-collaados (eds.), Proceedings of the 12th Joint Conference on Levical and Computational Semontics (*SEM 2023), pp. 314 322, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v https://acllatthol.gg orgg/0233 starsem-1 28/
1.22.... URL
Igor Douven. Abduction. In Edward N. Zalta (cd.), The Stanford Encyclopedia of Philosophy. Mctaphysics Research Lab, Stanford University, Summer 2021 edition, 2021.
Steffen Eger, Yong Cao, Jemnifer D'Souza, Andreas Geiger, Christian Greisinger, Stephanie Gross, Yufang Hou, Brigitte Krenn, Anne Lauscher, Yizhi Li, Chenghua Lin, Nafise Sadat Moosavi, Wei Zhao, and Tristan Miller. Transforming science with large language models: A survey on ai-assisted scientific discovery, experimentation, content generation, and evaluation, 2025. URL http:////aaxiw../ab///////aa
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatficl-Doddds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Cireuits Thread, 2022. http:://tttansfome-rrrreeee
pub/2022 odel oode///nneerrrrr/./ee/ /in//dddddtttt htmml.
Harry G. Frankfurt. Peirce's notion of abduction. The Journal of Philosophy, 55(14):593 597, 1958. ISSN 0022362X. URL
jstor http://jjssoooorrra//a/////////e
Pär Anders Granhag and Aldert. Vrij. Deception detection. Psychology and law: An empirical perspective, pp. 43-92, 2005.
Mourad Gridach, Jay Nanavati, Khaldoun Zine El Abidine, Lenon Mendes, and Christina Mack. Agentic ai for scientific discovery: A survey of progress, challenges, and future directions, 2025. URL https:
//axxiv.orr atgg/bbs/////.../// 08979 08979
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alex Wardle-Solano, Hannah Szabo, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander R. Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, and Dragomir Radev. Folio: Natural language reasoning with first-order logie, 2024. URL https://aaxiis wrgg absaa/as///// 0tt.//229.00000
Steve Hanks and Drew McDermott. Nonmonotonic logic and temporal projection. Artificial intelligence, 33 (3):379 412, 1987.
Gilbert H. Harman. The inference to the best explanation. The Philosophical Review, 74(1):88 95, 1965. ISSN 00318108, 15581470. URL http::/www.jstrrrrc http:///tttrrrg/ttaaeee/e/////ee
Jinwei He and Feng Lu. Causejudger: Identifying the cause with Ilms for abductive logical reasoning, 2024.
URL htttt: http://axxiv.oorrbb///////
Kaiyu He, Mian Zhang, Shuo Yan, Peilin Wu, and Zhiyu Zoey Chen. Idea: Enhancing the rule learning
ability of large language model agent through induction, deduction, and abduction, 2024. URL https:
//axxiv.org org.obggg/a............... http///ss/224....// 2208 10455.
Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang, Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan, and Zhenzhong Lan. Nova: An iterative planning and search approach to enhance novelty and diversity of Ilm generated ideas, 2024. URL htttps://arxii http::///xxvv.ooo/abbeee
htrrg/abs/24100 14255.
Wenyue Hua, Tyler Wong, Sun Fei, Liangming Pan, Adam Jardine, and William Yang Wang. Inductionbench: Lims fail in the simplest complexity class, 2025. URL https::/aaxx....oo wttp:///.w.r///////222..2/ 15823
Jie Huang and Kevin Chen-Chhan Chang. Towards reasoning in large language models: A survey. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (ods.), Findings of the Association for Computational Lin- guistics: ACL 2023, pp. 1049-1065, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ /1/53//v//// 111/22...nnninng..... findings-acl.67. URL httpss//acclaaaoooooooooo//////p wrg.://ooooooro//22/////// http://022...iiiinsaaaaall. fidiings-acl..677
21

---

Published in Transactions on Machine Learning Research (04/2025)
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents machine learning experimentation. arXiv preprint arXiv:2310.03302 2023.
Kang il Lee, Hyukhan Koh, Dongryeol Lee, Seunghyun Yoon, Minsung Kim, and Kyomin Jung. Generating diverse hypotheses for inductive reasoning, 2025. URL
abtp./////////.....///// 13422
Peter Jansen, Mare-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafford, and Peter Clark. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents. Advances in Neural Information Processing Systems, 37: 10088 10116, 2024.
Yifan Jiang, Filip Ilievski, Kaixin Ma, and Zhivar Sourati. Brainteaser: Lateral thinking puzzles for large language models, 2023. URL https://arxii.. wwtp://.ggg/bbbaaeaeee 2310. 05057
Cheng Jiayang, Lin Qiu, Tsz Chan, Tianqing Fang, Weiqi Wang. Chunkit Chan, Dongyu Ru, Qipeng Guo, Hongming Zhang, Yangqiu Song, Yue Zhang, and Zheng Zhang. Story Analogy: Deriving story-level analogies from large language models to unlock analogical understanding. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11518-11537, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18663/v1/202. emnlp-main.706 URL httph::/t///atct httpps///aclanthology oooo/rg///0 emnlp-main. 706/
Jachun Jung, Lianhui Qin, Scan Wellock, Facze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yojin Choi. Maieutie prompting: Logically consistent reasoning with recursive explanations. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (ods.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1266 1279, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 00.18653/v112000 1000222emlpmmma.n... URL https://aclanttholl org/
2022. wwt..mmnllppmmm main. m21..82/
Aditya Kalyanpur, Kailash Karthik Saravanakumar, Victor Barres, Jemnifer Chu-Carroll, David Melville, and David Ferrucci. T.m-arc: Enhancing 1lms with an automated reasoning critie, 2024. URL https: //rxivv...v
htg/abs/2406. 17663
Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is vidco generation from world model: A physical law perspective. arXiv preprint (tt::2411022900 2024.
Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antomia Creswell, James McClelland, Jane Wang, and Felix Hill. Can language models learn from explanations in context? Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Pindings of the Association for Compulational Linguistics: FMNLP 2022, pp. 337-563, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 1118663/w122022fnniingg Sudinggs-emnlp.36 URL https://aclanthoooooo oogg 2022. seindiings-mnllp 38/
Pat Langley. Bacon: A production system that discovers empirical laws. In IJCAI, pp. 344, 1977.
Larry Landan. William whewell on the consilience of inductions. The Monist, pp. 391- 1971.
Jiachun Li, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. Mirage: Evaluating and explaining inductive reasoning process language models, 2025a. URL
Yilian Li, Jidong Tian, Hao He, and Yaohui Jin. Hypothesis testing prompting improves deductive reasoning in large language models, 2024. URL /ttps::/aarxvv wwwp://.//.bbbbboooo/ooo
Zhong-2M Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhijiang Guo, Le Song, and Cheng-Lin Liu. From system to system A survey of reasoning large language models, 2025b. URL
https://arxxv.org// htrg/ab//502.... 17419
Chin-Yew Lin. ROUGE: package for automatic evaluation of summarics. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguisties. URL https:
22

---

Published in Transactions on Machine Learning Research (04/2025)
Robert K Lindsay, Bruce G Buchanan, Edward A Feigenbaum, and Joshua Lederberg. Dendral: case study of the first expert system for scientific hypothesis formation. Artāficial intelligence, (1(2)2209-2) 261, 1993.
Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. Deductive verification of thain-of-thought reasoning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (cds.), Advances in Neural Information Processing Systems, volume 36, pp. 36407-36433 Curran Associates, Inc., 2023. URL / nwurips. httppppprr..ileaepppp/////////// 7799044772555bbbb440977772333333 Paw.://eeebbbbaaaaaa--aa-eee Pape pdf
Emmy Liu, Graham Neubig, and Jacob Andreas. An incomplete loop: Instruction inference, instruction following, and in-context learning in language models, 2024. URL https://aaxiv..rrg/ab////// 03028
Hanmeng Liu, Zhizhang Fu, Mengru Ding, Ruoxi Ning, Chaoli Zhang, Xiaozhang Liu, and Yue Zhang. Logical reasoning in large language models: A survey, 2025. URL
09100
Jorge Nathan Matias, Kevin Munger, Marianne Aubin Le Quere, and Charles R. Ebersole. The upworthy research archive, time series of 32,487 experiments in U.S. media. Scientific Data, 8, 2021. URL https:
plapi..semantcsschoo.
Trg/CorpusIDD 236883026
John McCarthy and Patrick J Hayes. Some philosophical problems from the standpoint of artificial intelligence. In Readings in artificial intelligence, pp. 431 450. Elsevier, 1981.
John Stuart Mill. A System of Logic, Ratiocinative and Inductive: Being a Connected View of the Principles of Evidence, and Methods of Scientific Investigation: Vol. I. BoD Books on Demand, 2024.
Gerhard Minnameier. Peirce-suit of truth why inference to the best explanation and abduction ought not URL https:
to be confused. Erkenntnis, 60(1):75- 2004. doi:
//doi.or//10 10. 1023/B:200BB8 ERKE, ERKE. 0000005162.520522
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint ar XxXx:2410.05229, 2024.
Rajiv Movva, Kenny Peng, Nikhil Garg, Jon Kleinberg, and Emma Pierson. Sparse autoencoders for hypothesis generation, 2025. URL
2502.. 04382
Norman Mu, Sarah Chen, Zifan Wang, Sizhe Chen, David Karamardian, Lulwa Aljeraisy, Basel Alomair, Dan Hendrycks, and David Wagner. Can llms follow simple rules?, 2024. URL https://arxii: org/ abs/2311, 04235
Stephen Muggleton. Inductive logic programming. New generation computing, 8:295-318, 1991.
Ha-Thanh Nguyen, Randy Goebel, Francesca Toni, Kostas Stathis, and Ken Satoh. How well do sota legal reasoning models support abductive reasoning?, 2023. URL https://arxiv httpss////aaxvv.oo/a/bbbbsee/
Qian Niu, Junyu Liu, Ziqian Bi, Pohsun Feng, Benji Peng, Keyu Chen, Ming Li, Lawrence KQ Yan, Yichao Zhang, Caitlyn Heqi Yin, Cheng Fei, Tianyang Wang, Yunze Wang, Silin Chen, and Ming Liu. Large language models and cognitive science: A comprehensive review of similarities, differences, and challenges, 2024. URL httts:///axxi org/ absp///9.//0000/// 02387
Chitu Okoli. Inductive, abductive and deductive theorising. International Journal of Management Concepts and Philosophy, 16(3):302-316, 2023.
Thco Olausson, Alex Gu, Ben Lipkin, Codegao Zhang, Armando Solar-Lzzama, Joshua Tenenbaum, and Roger Levy. LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 5153-5176, Singapore, December 2023. Association for Computational Linguistics. doi: 1118633/v11202..mm/l main.mm.mm......... URL https: www.aaoooooooooo/o2222222 wmw.p-main.nn.m
23

---

Published in Transactions on Machine Learning Research (04/2025)
Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.); Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3806 3824, Singapore, De- cember 2023. Association for Computational Linguistics. doi: 10.18653/v1 http://ww//2...nnnnnnnmmm// findings-emnll.22 URL
findings-emnlp... 248/
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.31 10.3115/10730833 1073135. URL https:
//aclanthology www.:aatthooogy.orr//0-2----00 wrtp:/2-100///0///////
Judea Pearl. Causality. Cambridge university press, 2009.
Charles Sanders Peirce. Collected papers of charles sanders peince, volume 5. Harvard University Press, 1974.
Chau Minh Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, and Mohit lyyer. TopicGPT: A prompt- basod topic modcling framework. In Kevin Duh, Helena Gomez, and Steven Bethard (ods.), Proceed- ings of the 2024 Conference of the North American Chapler of the Association for Compulational Lin- guistics: Humon Longuage Technologies (Volume 1: Long Papers), pp. 2956-2984, Mexico City, Mex- ico, June 2024. Association for Computational Linguistics. doi: 10.18653/vll 11/2022. long. URL https://aclaathology) wrg/2024........... naacl lonellloonn.... 164/
Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, a survey, 2024. URL https://arxiv.org/ hwtp:/orggaab/////aaee 11511
Karl Popper. The logic of scienlific discovery. Routledge, 2005.
Kevin Pu, KJ Kevin Feng, Tovi Grossman, Tom Hope, Bhavana Dalvi Mishra, Matt Latzke, Jonathan Bragg, Joseph Chee Chang, and Pao Siangliulue. Ideasynth: Iterative research idea development through evolving and composing idea facets with litcraturo-grounded feodback. In Proceedings of the 2025 CHI Conference on Human Factors in Compuling Systems, pp. 1 11, 2025.
Biqing Qi, Kaiyan Zhang, Kai Tian, Haoxiang Li, Zhang- Rhang-Ren Chen, Sihang Zeng, Ermo Hua, Hu Jinfang, and Bowen Zhou. Large language models as biomedical hypothesis generators: A comprehensive evaluation, 2024. URL htttp:://axxv..oro
/0bpb//040000400 08940
Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-lo-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Chandan K Reddy and Parshin Shojace. Towards scientific discovery with generative ai: Progress, oppor- tunitics, and challenges. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 28601 28609, 2025.
Raymond Reiter. A logic for default reasoning. Artificial intelligence, 13(1-2):81-132, 1980.
Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. Thinking like a skeptic: Defeasible inference in natural language. In Trevor Cohn, Yulan He, and Yang Liu (ods.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4661-4675, Online, November 2020. Association for Computational Linguistics. doi: 10. NO.18653/v1 20003//112200.../ 0t05:///12200..nding-mmoo www.:oiinnnngnenn...... URL httttss://claathtttteea org/2000 wwwiooisnniinnnnnnnnn...
24

---

Published in Transactions on Machine Learning Research (04/2025)
Joshua Stewart Rule. The child hacker: building more human-like models of learning. PhD thesis, Mas- sachusetts Institute of Technology, 2020.
Scals and Valerie Shalin. Evaluating the deductive competence of large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 8614 0614-8630, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v11 w1//222..maa. sanall-long4766 URL httts:://aclatthollll .rg 2024/024. naacl- 1oallloon..47000 4766
Xiaoming Shi, Siqiao Xue, Kangrui Wang, Fan Zhou, James Zhang, Jun Zhou, Chenhao Tan, and Hongyuan Mei. Language models can improve event prediction by few-shot abductive reasoning. Advances in Newral Information Processing Systems, 11-29532-29555 2023.
Noah Shimn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement. learning. Adoances in Neural Information Processing Systems, 36:8634- 8652, 2023.
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers, 2024. URL https://arxiv.. http:/awwwvvrrrrrr/////// awwpaa/9..00000
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pincau, and William L. Hamilton. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLI) pp. 4506 4515, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19 1458. URL http::///cccaaaoooooooooorr/////
Friedrich Stadler. The road experience and prediction from within: Hans reichenbach's scientific correspon- dence from berlin to istanbul. Synthese, 181:137 155, 2011.
Wangtao Sun, Chenxiang Zhang, YueYon Zhang, Xuanqing Yu, Ziyang Huang, Pei Chen, Haotian Xu, Shizhu He, Jun Zhao, and Kang Liu. Beyond instruction following: Evaluating inferential rule following of large language models, 2024. URL https://arxiv.. http://aaxiv..o//aaaa////a///
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. Proo/Writer: Generating implications, proofs, and abductive statements over natural language. In Chengqing Zong, Foi Xia, Wenjie Li, and Roberto Navigli (ods.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3621 3634, Online, August 2021. Association for Computational Linguistics. doi: 111/653//1/2222.fnnningss. https: haap:s://allathhoog/y org 2021/202..00 findings-acl.317
URL
Chenhao Tan, Lillian Lee, and Bo Pang. The effect of wording message propagation: Topic- and author- controlled natural experiments twitter. In Proceedings of ACL, 2014.
Fuchun Wang, Xian Zhou, Wenpeng Hu, Zhunchen Luo, Wei Luo, and Xiaoying Bai. Lim assists hypothesis generation and testing for deliberative questions. In Derck F. Wong, Zhongyu Wei, and Muyun Yang (eds.), Natural Language Processing and Chinese Compuling, pp. 424 436, Singapore, 2025. Springer Nature Singapore. ISBN 978-981-99-99844-8
Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D. Goodman. Hypothesis search: Inductive reasoning with language models, 2024. URL httpp::/aarvv..or//abaaaae 008399005660
Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader?, 2022a. URL
Ruoyao Wang, Peter Jansen, Mlex-Alex Côté, and Ammanabrriiammmmammo Scienceworld; Is your agent smarter than 5th grader?, 2022b. URL httt:://aaxx..oog/////22.......
25

---

Published in Transactions on Machine Learning Research (04/2025)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 248277 Curran Associates, Inc., 2022. URLitttps://proceedngg hww.eeeeddinnnnn..ip...... http://ppppppp/////..////////aaa
pdf
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks, 2015. URL
https://arxiv.. wwtpa/iiivvoogaaabaaabeee h/////b///// 150.. a02///////50000886
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. Reframing human-AT collaboration for generating free-text explanations. In Marine Carpuat, Marie-Cathern de Marneffe, and Ivan Vladimir Meza Ruiz (ods.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 682-658, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/ /2/05//1/2222..22 www.:aaaaalla...... URL org/ 2022. naa.2nnacc--aa meaecclmain..411 47/
Guangzhi Xiong, Eric Xie, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang, Improving scientific hypothesis generation with knowledge grounded large language models, 2024. URL
https://aarxvv.org// abtaaabs/111 2411.02382
Manjie Xu, Guangyuan Jiang, Wei Liang, Chi Zhang, and Yixin Zhu. Active reasoning in an open-world environment. Advances in Newral Information Processing Systems, 06:11776-11736, 2023.
Shuo Yan, Ruochen Li, Ziming Luo, Zimu Wang, Daoyang Li, Liqiang Jing, Kaiyu He, Peilin Wu, George Michalopoulos, Yue Zhang, al. Tamr-bench: Evaluating 11 agent's 's ability on reproducing language mod- eling research. arXiv preprint ar 2rXiv:2506.1783 17335, 2025a.
Yang Yan, Yu Lu, Renjun Xu, and Zhenzhong Lan. Do led-level Iims truly grasp elementary addition? probing rule learning vs. memorization in large language models. arXic preprint arx:222004 2552225000022,
05262, 2025b.
Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. Language models as inductive reasoners. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 209-225, 225, St. Julian's, Malta, March 2024a. Association for Computational Linguistics. URL https://aclanthollogg www.gogg..ooggoooe wrg/ www.2222...11oomooe 13/
Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. Large language models for automated open-domain scientific hypotheses discovery. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Pindings of the Association for Compulational Linguisties: ACL 2024, pp. 13545 13565, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653//v 222....inn findings-aadd waaiinggs-cll.00. a04.
URL httpps://alaatthlllge http://aaaaaabblolooooo//r///.a Fondero.
Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and Erik Cambria. Logical reasoning over natural language as knowledge representation: A survey, 2024c. URL https://arxiv: http///aavvvrrraraa/aa///a org/abs//3/3/ 12023
Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou Mooso-chem: Large language models for rediscovering unseon chemistry scientific hypotheses, 2025. URL http::://aaxxv.org///
Nathan Young, Qiming Bao, Joshua Bensemann, and Michael Witbrock. AbductionRules:: Training trans- formers to explain unexpected inputs. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Compulational Linguistics: ACL 2022, pp. land, May 2022. Association for Computational Linguistics. doi: attps://acclaathoooog..o//////// wwwp:/gg...rg/222o...e. http://222...iiinnnaaca...a-.e
Dublin, Tre- URL
Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. Natural language reasoning, a survey. ACM Comput. Surv., 56(12), October 2024a. ISSN 0360-0300. doi: 10.1145/366411 3664194
URL https:: sttps:///ol.. htt///i..org/..11///// 1145/
26

---

Published in Transactions on Machine Learning Research (04/2025)
Qingchen Yu, Shichao Song, Ke Fang, Yunfeng Shi, Zifan Zheng, Hanyu Wang. Simin Niu, and Zhiyu Li. Turtlebench: Evaluating top language models via real-world ves/no puzzles, 2024b. URL 3Tg/abs/2410.0522
Siyu Yuan, Jiangjie Chen, Xuyang Ge, Yanghua Xiao, and Deqing Yang. Beneath surface similarity: Large language models make reasonable scientific analogies after structure abduction. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Associalion for Compulational Linguislies: EMNLP 2023, pp. Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023 mudings-emnll.100 160. URL https:/ hatp:///llaan org/2023 indings-emnlp 160/
Hongming Zhang, Xinran Zhao, and Yangqiu Song. WhnoWhy: A deep diagnosis of essential commonsense knowledge for answering Winograd schema challenge. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5736-5745, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v /2000/000clmmmmmm
508. URL https://acclantholpgs. org.eoogoooo///220 org/2020 acl-main...m 508/
Yue Zhang, Liqiang Jing, and Vibhav Gogate. Defeasible visual entailment: Benchmark, evaluator, and reward-driven optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 25976 25984, 2025.
Wenting Zhao, Justin T. Chiu, Claire Cardie, and Alexander M. Rush. Abductive commonsense reasoning exploiting mutually exclusive explanations, 2023. URL https://arxiv, wttp.aa///....11005555
Wenting Zhao, Justin Chiu, Jena Hwang, Faeze Brahman, Jack Hessel, Sanjiban Choudhury, Yejin Choi, Xiang Li, and Alane Suhr. UNcommonsense reasoning: Abductive reasoning about uncommon situations. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Proceedings of the 2024 Conference of the North American Chopter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 8487-8505, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653 naccl-log... URL htttp::/accaattholog. http://ccaaathoooogoo////22/22/ maacl-long.e. 469/
Ruiqi Zhong, Heng Wang, Dan Klein, and Jacob Steinhardt. Explaining datasets in words: Statistical models with natural language parameters. Advances in Neural Information Processing Systems, 37:79350-79380, 2024.
Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. Hypothesis generation with large language models. Tn Proceedings of the 1st Workshop NLP for Science (NIP4Sccience) pp. 117-139. Association for Computational Linguisties, 2024. doi: http://dx.doi: http//xxxxii...o wrgg11..1.633/112222444 http:/2000...ppsccccceee.1....
URL
Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large language models learn rules, 2024. URL httph:://///// httpa//wwvvooooaa//////aaaaa///a
27


Reviews:
[{"review_id": "mblsuBj6A3", "reviewer": "Reviewer_G3US", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nI think that adopting Peirce's framework not only provides a clear organization for the paper, but also gives it a sound theoretical basis that helps pinpoint the gaps between real-world hypothesis discovery and current practice. For example, Section 6 discusses the one-off nature of induction and the lack of iterative updating of confidence in current practice.\n\nI found the discussions interesting and valuable.\n\nThe paper is very clear and easy to read.\n\n## Weaknesses\n\nEven though the survey is about LLMs, it could reference some prior foundational work or perhaps summarize it to give some context. Currently it feels somewhat isolated from the rest of the literature and the vast non-LLM prior work.\n\n5.1 talks about LLMs as formal language parsers. I believe that the natural language is parsed into formal language here, so the LLMs are used as natural language parsers or autoformalizers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper surveys the literature of hypothesis discovery and rule learning with LLMs. The authors adopt Peirce's framework for hypothesis discovery, and organize the survey according to its stages of abduction, deduction, and induction (the corresponding sections are hypothesis generation, application, and validation). Each of the three sections review relevant methods and evaluation strategies for both formal and natural languages, then concludes with a discussion and future directions.\n\nSection 7, Hypothesis Discovery, synthesizes the three stages into holistic hypothesis discovery, and categorizes discovery into three different types which more and more resemble real-world hypothesis discovery. In passive discovery, observations are drawn from a fixed dataset, and the LLM is passive: it doesn't have any control over which observations come next. Proactive discovery generates new observations through deduction either from the LLM's parametric memory, or by simple tools. Lastly, the LLM interacts with the environment and gathers meaningful observations through the sequence of its actions in a simulation of the real world.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The last paragraph of 4.1.2 cites Bricken, et al., which is about mechanistic interpretability. I believe the intended citation is sparse autoencoders, which is Makhzani, A., & Frey, B. (2013). K-sparse autoencoders. arXiv preprint arXiv:1312.5663.\n\nThe summary of Chen et al. in 4.2.2 sounds very similar to program synthesis, it would help to detail and emphasize the difference.\n\nThe last paragraph of the first part of Section 5 says that there is little motivation to leverage LLMs for deductive reasoning in formal hypotheses. That's not strictly so because some work tries to improve LLM's deductive reasoning capabilites, for example, Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2023, July). Learning deductive reasoning from synthetic corpus based on formal logic. In International Conference on Machine Learning (pp. 25254-25274). PMLR.\n\nNaming sections similarly to Figure 1 (like \"Hypothesis Generation with Abduction\", for example) would further emphasize that the paper is organized according to Peirce's framework.\n\nThe sentence structures in the second paragraph of 5.2 could be improved as they are very similar to each other. They start with \"Similarly\", \"In addition\", \"Moreover\".\n\nIn the first part of 6.2.2, the sentences starting with \"Their validation process showed\"  and \"During evaluation\" are probably in the wrong order.\n\nSome abbreviations are not defined:\n- First Order Logic (FOL)\n- the IBE paradigm is defined on page 1, but then used next on page 13\n\nTypos:\n- To encourage diversity, il Lee et al. (2025)\n- \"Abstract Reasoning Corpus\" => \"Abstraction and Reasoning Corpus\"", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t67t9jWmvi", "reviewer": "Reviewer_1sKv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The paper topic is interesting, and the number of hypothesis discovery surveys is still limited.\n* The paper structure is clear, as presented in Figure 1.\n* This work reviews enough number of works to conduct the survey. The survey covers the area of knowledge discovery.\n\nWeakness:\n* There are some types. For example, the \"Multiple-Choice-Based Evaluation\" has additional next line space.\n* More discussion of the future work and the insights of knowledge discovery. The current survey only lists papers and introduces them in different sections. However, the discussion of future work and the insight into knowledge discovery is limited.\n* For the summary, it is better to give more insight into this area.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work takes a comprehensive survey of knowledge discovery. The paper structure is clear. And this work review is enough paper for each section. In the future, the paper could help summarize the progress of hypothesis discovery and rule learning with large language models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m74E5TYs25", "reviewer": "Reviewer_ywQH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The breadth of the survey is appropriate. The authors have managed to include all the major relevant works in the field which I am aware of, as well as some more obscure ones that nicely complement their results. At the same time, I never got the sense that they were including \"throw-away\" citations just to pump the numbers up.\n- The paper is exceptionally well written from a linguistic point of view.\n- The structure of the paper makes it easy to follow. Indeed I think it is testament to the strength of the paper that within the first 5 minutes of skimming the manuscript I felt like I had already developed a more structured mental model of the literature.\n- The use of Pierce's framework of hypothesis discovery shines a new light on recent application-focused research in the LLM literature. I believe practitioners and researchers alike will find themselves coming up with plenty of new ideas just by reading this survey.\n\nWeaknesses:\n- My only complaint is that the survey is quite light on figures and tables, which might make the paper a bit more digestible. Luckily the text is written so well that it nonetheless remains quite manageable to simply read it in full.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper forms an exhaustive survey of what the authors collectively label as \"hypothesis discovery\" and \"rule learning\" in systems that are predominantly driven by a language model. The authors do a good job of synthesizing results from across the LLM reasoning literature, managing to achieve quite a bit of breadth without ever making me feel like they had to \"shoehorn\" a paper to fit into their framework. The primary contributions of the paper are thus:\n- Collecting and summarizing a large number of results from the very fast-moving (and, let's be honest, rather disorganized) LLM reasoning literature\n- Identifying common strengths and weaknesses across different papers from the literature in terms of how far they go to evaluate LLM's ability to discover hypotheses and learn rules\n- Presenting a unified framework through which to view progress in the field. Though the framework (by Pierce) itself is well-established in the philosophy of logic, I think it is a novel and interesting way to frame research in this field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- (Not critical): Consider whether adding more visual summarizations of the survey would help the user digest your insights. These could potentially go in the appendix if you do not want to lengthen the paper itself too much.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)